[GitHub] [spark] AmplabJenkins removed a comment on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

AmplabJenkins removed a comment on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643830992 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

AmplabJenkins commented on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643830992 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

SparkQA commented on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643830855 **[Test build #124017 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124017/testReport)** for PR 28828 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28829: [WIP][SQL] Benchmark the EXCEPTION rebase mode

AmplabJenkins removed a comment on pull request #28829: URL: https://github.com/apache/spark/pull/28829#issuecomment-643828984 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28829: [WIP][SQL] Benchmark the EXCEPTION rebase mode

AmplabJenkins commented on pull request #28829: URL: https://github.com/apache/spark/pull/28829#issuecomment-643828984 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28829: [WIP][SQL] Benchmark the EXCEPTION rebase mode

SparkQA commented on pull request #28829: URL: https://github.com/apache/spark/pull/28829#issuecomment-643828884 **[Test build #124016 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124016/testReport)** for PR 28829 at commit

[GitHub] [spark] MaxGekk opened a new pull request #28829: [WIP][SQL] Benchmark the EXCEPTION rebase mode

MaxGekk opened a new pull request #28829: URL: https://github.com/apache/spark/pull/28829 ### What changes were proposed in this pull request? ### Why are the changes needed? ### Does this PR introduce _any_ user-facing change? ### How

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

AmplabJenkins removed a comment on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643827609 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

AmplabJenkins commented on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643827609 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

SparkQA commented on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643827509 **[Test build #124015 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124015/testReport)** for PR 28828 at commit

[GitHub] [spark] dongjoon-hyun closed pull request #26901: [SPARK-29152][CORE][2.4] Executor Plugin shutdown when dynamic allocation is enabled

dongjoon-hyun closed pull request #26901: URL: https://github.com/apache/spark/pull/26901 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] SparkQA commented on pull request #28616: [SPARK-31798][SHUFFLE][API] Shuffle Writer API changes to return custom map output metadata

SparkQA commented on pull request #28616: URL: https://github.com/apache/spark/pull/28616#issuecomment-643826770 **[Test build #124014 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124014/testReport)** for PR 28616 at commit

[GitHub] [spark] srowen commented on pull request #28786: [SPARK-31925][ML] Summary.totalIterations greater than maxIters

srowen commented on pull request #28786: URL: https://github.com/apache/spark/pull/28786#issuecomment-643826632 @zhengruifeng if you don't strongly object to https://github.com/apache/spark/pull/28786#discussion_r439608878 I think this one can be merged. Just needs a rebase at your

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

AmplabJenkins removed a comment on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643826208 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] dongjoon-hyun commented on pull request #28787: [SPARK-31959][SQL][test-java11] Fix Gregorian-Julian micros rebasing while switching standard time zone offset

dongjoon-hyun commented on pull request #28787: URL: https://github.com/apache/spark/pull/28787#issuecomment-643826389 Hi, All. As I commented on the backporting PR, this causes a weird consistent failure on `research-jenkins-worker-09`. Could you take a look at that? -

[GitHub] [spark] SparkQA removed a comment on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

SparkQA removed a comment on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643826003 **[Test build #124013 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124013/testReport)** for PR 28828 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

AmplabJenkins removed a comment on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643826206 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

AmplabJenkins commented on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643826206 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

AmplabJenkins removed a comment on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643826119 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] SparkQA commented on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

SparkQA commented on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643826204 **[Test build #124013 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124013/testReport)** for PR 28828 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

AmplabJenkins commented on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643826119 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] srowen commented on pull request #28752: [SPARK-31642][FOLLOWUP] Fix Sorting for duration column and make Status column sortable

srowen commented on pull request #28752: URL: https://github.com/apache/spark/pull/28752#issuecomment-643826026 Oh I see, this was attached to the JIRA for pagination. It does make sense to have a separate JIRA for _that_. Let me retroactively fix up the links.

[GitHub] [spark] SparkQA commented on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

SparkQA commented on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643826003 **[Test build #124013 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124013/testReport)** for PR 28828 at commit

[GitHub] [spark] srowen commented on pull request #28752: [SPARK-31642][FOLLOWUP] Fix Sorting for duration column and make Status column sortable

srowen commented on pull request #28752: URL: https://github.com/apache/spark/pull/28752#issuecomment-643825958 Merged to master. I'll look at the backport. This is an automated message from the Apache Git Service. To

[GitHub] [spark] srowen closed pull request #28752: [SPARK-31642][FOLLOWUP] Fix Sorting for duration column and make Status column sortable

srowen closed pull request #28752: URL: https://github.com/apache/spark/pull/28752 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] HeartSaVioR commented on a change in pull request #28607: [SPARK-24634][SS] Add a new metric regarding number of inputs later than watermark plus allowed delay

HeartSaVioR commented on a change in pull request #28607: URL: https://github.com/apache/spark/pull/28607#discussion_r439870225 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala ## @@ -77,6 +77,8 @@ trait StateStoreWriter

[GitHub] [spark] HeartSaVioR commented on pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

HeartSaVioR commented on pull request #28828: URL: https://github.com/apache/spark/pull/28828#issuecomment-643825765 cc. @zsxwing This is an automated message from the Apache Git Service. To respond to the message, please

[GitHub] [spark] HeartSaVioR opened a new pull request #28828: [SPARK-24634][SS][FOLLOWUP] Rename the variable from "numLateInputs" to "numDropppedRowsByWatermark"

HeartSaVioR opened a new pull request #28828: URL: https://github.com/apache/spark/pull/28828 ### What changes were proposed in this pull request? This PR renames the variable from "numLateInputs" to "numDropppedRowsByWatermark" so that it becomes self-explanation. ### Why

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28809: [SPARK-31959][SQL][3.0] Fix Gregorian-Julian micros rebasing while switching standard time zone offset

dongjoon-hyun commented on a change in pull request #28809: URL: https://github.com/apache/spark/pull/28809#discussion_r439869464 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/RebaseDateTimeSuite.scala ## @@ -409,4 +409,40 @@ class

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28809: [SPARK-31959][SQL][3.0] Fix Gregorian-Julian micros rebasing while switching standard time zone offset

dongjoon-hyun commented on a change in pull request #28809: URL: https://github.com/apache/spark/pull/28809#discussion_r439869464 ## File path: sql/catalyst/src/test/scala/org/apache/spark/sql/catalyst/util/RebaseDateTimeSuite.scala ## @@ -409,4 +409,40 @@ class

[GitHub] [spark] dongjoon-hyun closed pull request #28801: [SPARK-31970][CORE] Make MDC configuration step be consistent between setLocalProperty and log4j.properties

dongjoon-hyun closed pull request #28801: URL: https://github.com/apache/spark/pull/28801 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to

[GitHub] [spark] SparkQA commented on pull request #27066: [SPARK-31317][SQL] Add withField method to Column

SparkQA commented on pull request #27066: URL: https://github.com/apache/spark/pull/27066#issuecomment-643823823 **[Test build #124012 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124012/testReport)** for PR 27066 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing

AmplabJenkins removed a comment on pull request #24525: URL: https://github.com/apache/spark/pull/24525#issuecomment-643818140 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing

AmplabJenkins commented on pull request #24525: URL: https://github.com/apache/spark/pull/24525#issuecomment-643818140 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing

SparkQA removed a comment on pull request #24525: URL: https://github.com/apache/spark/pull/24525#issuecomment-643786010 **[Test build #124009 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124009/testReport)** for PR 24525 at commit

[GitHub] [spark] SparkQA commented on pull request #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing

SparkQA commented on pull request #24525: URL: https://github.com/apache/spark/pull/24525#issuecomment-643817928 **[Test build #124009 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124009/testReport)** for PR 24525 at commit

[GitHub] [spark] iRakson commented on pull request #26901: [SPARK-29152][CORE][2.4] Executor Plugin shutdown when dynamic allocation is enabled

iRakson commented on pull request #26901: URL: https://github.com/apache/spark/pull/26901#issuecomment-643816792 #28254 has been closed. This is an automated message from the Apache Git Service. To respond to the message,

[GitHub] [spark] iRakson closed pull request #28254: [SPARK-31478][CORE]Call `StopExecutor` before killing executors

iRakson closed pull request #28254: URL: https://github.com/apache/spark/pull/28254 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to

[GitHub] [spark] iRakson commented on pull request #28254: [SPARK-31478][CORE]Call `StopExecutor` before killing executors

iRakson commented on pull request #28254: URL: https://github.com/apache/spark/pull/28254#issuecomment-643816527 For the time being we are continuing with the current approach only. Closing this PR. This is an automated

[GitHub] [spark] iRakson edited a comment on pull request #26901: [SPARK-29152][CORE][2.4] Executor Plugin shutdown when dynamic allocation is enabled

iRakson edited a comment on pull request #26901: URL: https://github.com/apache/spark/pull/26901#issuecomment-642632099 @dongjoon-hyun @vanzin I have tested this a couple of times, older approach of adding shutdownhook is working fine. Last failure is unrelated to this. Also about the

[GitHub] [spark] iRakson commented on pull request #26901: [SPARK-29152][CORE][2.4] Executor Plugin shutdown when dynamic allocation is enabled

iRakson commented on pull request #26901: URL: https://github.com/apache/spark/pull/26901#issuecomment-643815823 > Hi, @iRakson . So, this is your final decision and are you going to close #28254 ? > This PR looks like to be reverted into the original patch. In that case, we don't need

[GitHub] [spark] dongjoon-hyun commented on pull request #26901: [SPARK-29152][CORE][2.4] Executor Plugin shutdown when dynamic allocation is enabled

dongjoon-hyun commented on pull request #26901: URL: https://github.com/apache/spark/pull/26901#issuecomment-643814681 Hi, @iRakson . So, this is your final decision and are you going to close #28254 ? This PR looks like to be reverted into the original patch. In that case, we don't

[GitHub] [spark] HeartSaVioR commented on a change in pull request #28607: [SPARK-24634][SS] Add a new metric regarding number of inputs later than watermark plus allowed delay

HeartSaVioR commented on a change in pull request #28607: URL: https://github.com/apache/spark/pull/28607#discussion_r439861427 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala ## @@ -77,6 +77,8 @@ trait StateStoreWriter

[GitHub] [spark] iRakson commented on pull request #26901: [SPARK-29152][CORE][2.4] Executor Plugin shutdown when dynamic allocation is enabled

iRakson commented on pull request #26901: URL: https://github.com/apache/spark/pull/26901#issuecomment-643811881 @dongjoon-hyun @vanzin Can we get this in 2.4 ? I think now this is fine. It should not break the code. I have already tested this multiple times.

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28825: [SPARk-31950][SQL][FOLLOW-UP][MINOR] Better error message on SPARK_HOME or…

AmplabJenkins removed a comment on pull request #28825: URL: https://github.com/apache/spark/pull/28825#issuecomment-643809567 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28825: [SPARk-31950][SQL][FOLLOW-UP][MINOR] Better error message on SPARK_HOME or…

AmplabJenkins commented on pull request #28825: URL: https://github.com/apache/spark/pull/28825#issuecomment-643809567 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28825: [SPARk-31950][SQL][FOLLOW-UP][MINOR] Better error message on SPARK_HOME or…

SparkQA commented on pull request #28825: URL: https://github.com/apache/spark/pull/28825#issuecomment-643809409 **[Test build #124011 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124011/testReport)** for PR 28825 at commit

[GitHub] [spark] agrawaldevesh commented on a change in pull request #28817: [WIP][SPARK-31197][CORE] Exit the executor once all tasks and migrations are finished built on top of on top of spark20629

agrawaldevesh commented on a change in pull request #28817: URL: https://github.com/apache/spark/pull/28817#discussion_r439859171 ## File path: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala ## @@ -258,26 +262,60 @@ private[spark] class

[GitHub] [spark] holdenk commented on a change in pull request #28817: [WIP][SPARK-31197][CORE] Exit the executor once all tasks and migrations are finished built on top of on top of spark20629

holdenk commented on a change in pull request #28817: URL: https://github.com/apache/spark/pull/28817#discussion_r439857534 ## File path: core/src/main/scala/org/apache/spark/executor/CoarseGrainedExecutorBackend.scala ## @@ -258,26 +262,60 @@ private[spark] class

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28827: [SPARK-31989][SQL] Generate JSON rebasing files w/ 30 minutes step

AmplabJenkins removed a comment on pull request #28827: URL: https://github.com/apache/spark/pull/28827#issuecomment-643806345 Test FAILed. Refer to this link for build results (access rights to CI server needed):

[GitHub] [spark] SparkQA removed a comment on pull request #28827: [SPARK-31989][SQL] Generate JSON rebasing files w/ 30 minutes step

SparkQA removed a comment on pull request #28827: URL: https://github.com/apache/spark/pull/28827#issuecomment-643804463 **[Test build #124010 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124010/testReport)** for PR 28827 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28827: [SPARK-31989][SQL] Generate JSON rebasing files w/ 30 minutes step

AmplabJenkins removed a comment on pull request #28827: URL: https://github.com/apache/spark/pull/28827#issuecomment-643806342 Merged build finished. Test FAILed. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins commented on pull request #28827: [SPARK-31989][SQL] Generate JSON rebasing files w/ 30 minutes step

AmplabJenkins commented on pull request #28827: URL: https://github.com/apache/spark/pull/28827#issuecomment-643806342 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA commented on pull request #28827: [SPARK-31989][SQL] Generate JSON rebasing files w/ 30 minutes step

SparkQA commented on pull request #28827: URL: https://github.com/apache/spark/pull/28827#issuecomment-643806320 **[Test build #124010 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124010/testReport)** for PR 28827 at commit

[GitHub] [spark] MaxGekk commented on pull request #28816: [SPARK-31986][SQL] Fix Julian-Gregorian micros rebasing of overlapping local timestamps

MaxGekk commented on pull request #28816: URL: https://github.com/apache/spark/pull/28816#issuecomment-643804818 @cloud-fan @HyukjinKwon Please, review the fix. This is an automated message from the Apache Git Service. To

[GitHub] [spark] MaxGekk commented on pull request #28824: [SPARK-31984][SQL] Make micros rebasing functions via local timestamps pure

MaxGekk commented on pull request #28824: URL: https://github.com/apache/spark/pull/28824#issuecomment-643804694 @cloud-fan @HyukjinKwon Please, review this PR. This is an automated message from the Apache Git Service. To

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28827: [SPARK-31989][SQL] Generate JSON rebasing files w/ 30 minutes step

AmplabJenkins removed a comment on pull request #28827: URL: https://github.com/apache/spark/pull/28827#issuecomment-643804601 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28827: [SPARK-31989][SQL] Generate JSON rebasing files w/ 30 minutes step

AmplabJenkins commented on pull request #28827: URL: https://github.com/apache/spark/pull/28827#issuecomment-643804601 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] MaxGekk commented on pull request #28827: [SPARK-31989][SQL] Generate JSON rebasing files w/ 30 minutes step

MaxGekk commented on pull request #28827: URL: https://github.com/apache/spark/pull/28827#issuecomment-643804614 @cloud-fan @HyukjinKwon After merging this bug fix https://github.com/apache/spark/pull/28816 and make the rebase functions pure by https://github.com/apache/spark/pull/28824,

[GitHub] [spark] SparkQA commented on pull request #28827: [SPARK-31989][SQL] Generate JSON rebasing files w/ 30 minutes step

SparkQA commented on pull request #28827: URL: https://github.com/apache/spark/pull/28827#issuecomment-643804463 **[Test build #124010 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124010/testReport)** for PR 28827 at commit

[GitHub] [spark] MaxGekk opened a new pull request #28827: [SPARK-31989][SQL] Generate JSON rebasing files w/ 30 minutes step

MaxGekk opened a new pull request #28827: URL: https://github.com/apache/spark/pull/28827 ### What changes were proposed in this pull request? 1. Change the max step from 1 week to 30 minutes in the tests `RebaseDateTimeSuite`.`generate 'gregorian-julian-rebase-micros.json'` and

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28685: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function

AmplabJenkins removed a comment on pull request #28685: URL: https://github.com/apache/spark/pull/28685#issuecomment-643800833 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28685: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function

AmplabJenkins commented on pull request #28685: URL: https://github.com/apache/spark/pull/28685#issuecomment-643800833 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28685: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function

SparkQA removed a comment on pull request #28685: URL: https://github.com/apache/spark/pull/28685#issuecomment-643767795 **[Test build #124008 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124008/testReport)** for PR 28685 at commit

[GitHub] [spark] SparkQA commented on pull request #28685: [SPARK-27951][SQL] Support ANSI SQL NTH_VALUE window function

SparkQA commented on pull request #28685: URL: https://github.com/apache/spark/pull/28685#issuecomment-643800607 **[Test build #124008 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124008/testReport)** for PR 28685 at commit

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

dongjoon-hyun commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r439853527 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerMaster.scala ## @@ -14,7 +14,6 @@ * See the License for the specific

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

dongjoon-hyun commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r439853479 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

dongjoon-hyun commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r439853387 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

dongjoon-hyun commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r439853387 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache

[GitHub] [spark] zsxwing commented on a change in pull request #28607: [SPARK-24634][SS] Add a new metric regarding number of inputs later than watermark plus allowed delay

zsxwing commented on a change in pull request #28607: URL: https://github.com/apache/spark/pull/28607#discussion_r439853232 ## File path: sql/core/src/main/scala/org/apache/spark/sql/execution/streaming/statefulOperators.scala ## @@ -77,6 +77,8 @@ trait StateStoreWriter

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

dongjoon-hyun commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r439853227 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

dongjoon-hyun commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r439853174 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManagerDecommissioner.scala ## @@ -0,0 +1,281 @@ +/* + * Licensed to the Apache

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

dongjoon-hyun commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r439852892 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ## @@ -650,6 +658,23 @@ private[spark] class BlockManager(

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

dongjoon-hyun commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r439852796 ## File path: core/src/main/scala/org/apache/spark/storage/BlockManager.scala ## @@ -650,6 +658,23 @@ private[spark] class BlockManager(

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

dongjoon-hyun commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r439852528 ## File path: core/src/main/scala/org/apache/spark/scheduler/MapStatus.scala ## @@ -33,9 +33,11 @@ import org.apache.spark.util.Utils * task ran

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

dongjoon-hyun commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r439852420 ## File path: core/src/main/scala/org/apache/spark/internal/config/package.scala ## @@ -420,6 +420,21 @@ package object config {

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

dongjoon-hyun commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r439852162 ## File path: core/src/main/scala/org/apache/spark/internal/config/package.scala ## @@ -420,6 +420,21 @@ package object config {

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

dongjoon-hyun commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r439852124 ## File path: core/src/main/scala/org/apache/spark/internal/config/package.scala ## @@ -420,6 +420,21 @@ package object config {

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

dongjoon-hyun commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r439851962 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -775,7 +802,12 @@ private[spark] class MapOutputTrackerMaster(

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

dongjoon-hyun commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r439851900 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -479,6 +497,15 @@ private[spark] class MapOutputTrackerMaster(

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

dongjoon-hyun commented on a change in pull request #28708: URL: https://github.com/apache/spark/pull/28708#discussion_r439851831 ## File path: core/src/main/scala/org/apache/spark/MapOutputTracker.scala ## @@ -121,12 +121,28 @@ private class ShuffleStatus(numPartitions: Int)

[GitHub] [spark] dongjoon-hyun commented on pull request #28708: [SPARK-20629][CORE][K8S] Copy shuffle data when nodes are being shutdown

dongjoon-hyun commented on pull request #28708: URL: https://github.com/apache/spark/pull/28708#issuecomment-643797896 The K8s integration test failure is irrelevant to this PR. ``` - Run SparkPi with no resources *** FAILED *** ```

[GitHub] [spark] dongjoon-hyun commented on pull request #28814: [SPARK-31968][SQL]Duplicate partition columns check when writing data

dongjoon-hyun commented on pull request #28814: URL: https://github.com/apache/spark/pull/28814#issuecomment-643797145 @TJX2014 . I mean your JIRA account ID. I need to assign SPARK-31968 to you. :) This is an automated

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28819: [SPARK-31980][SQL]Function sequence() fails if start and end of range are equal dates

AmplabJenkins removed a comment on pull request #28819: URL: https://github.com/apache/spark/pull/28819#issuecomment-643791390 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28819: [SPARK-31980][SQL]Function sequence() fails if start and end of range are equal dates

AmplabJenkins commented on pull request #28819: URL: https://github.com/apache/spark/pull/28819#issuecomment-643791390 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28819: [SPARK-31980][SQL]Function sequence() fails if start and end of range are equal dates

SparkQA removed a comment on pull request #28819: URL: https://github.com/apache/spark/pull/28819#issuecomment-643758259 **[Test build #124007 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124007/testReport)** for PR 28819 at commit

[GitHub] [spark] SparkQA commented on pull request #28819: [SPARK-31980][SQL]Function sequence() fails if start and end of range are equal dates

SparkQA commented on pull request #28819: URL: https://github.com/apache/spark/pull/28819#issuecomment-643791166 **[Test build #124007 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124007/testReport)** for PR 28819 at commit

[GitHub] [spark] AmplabJenkins removed a comment on pull request #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing

AmplabJenkins removed a comment on pull request #24525: URL: https://github.com/apache/spark/pull/24525#issuecomment-643786152 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing

AmplabJenkins commented on pull request #24525: URL: https://github.com/apache/spark/pull/24525#issuecomment-643786152 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28826: [SPARK-31988] - schema pruning may discard attribute metadata

AmplabJenkins removed a comment on pull request #28826: URL: https://github.com/apache/spark/pull/28826#issuecomment-643785951 Can one of the admins verify this patch? This is an automated message from the Apache Git

[GitHub] [spark] AmplabJenkins commented on pull request #28826: [SPARK-31988] - schema pruning may discard attribute metadata

AmplabJenkins commented on pull request #28826: URL: https://github.com/apache/spark/pull/28826#issuecomment-643786098 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To

[GitHub] [spark] SparkQA commented on pull request #24525: [SPARK-27633][SQL] Remove redundant aliases in NestedColumnAliasing

SparkQA commented on pull request #24525: URL: https://github.com/apache/spark/pull/24525#issuecomment-643786010 **[Test build #124009 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124009/testReport)** for PR 24525 at commit

[GitHub] [spark] AmplabJenkins commented on pull request #28826: [SPARK-31988] - schema pruning may discard attribute metadata

AmplabJenkins commented on pull request #28826: URL: https://github.com/apache/spark/pull/28826#issuecomment-643785951 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To

[GitHub] [spark] guykhazma opened a new pull request #28826: [SPARK-31988] - schema pruning may discard attribute metadata

guykhazma opened a new pull request #28826: URL: https://github.com/apache/spark/pull/28826 ### What changes were proposed in this pull request? Fixing the `getRootFields` function to preserve attribute metadata ### Why are the changes needed? This can lead to a

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28816: [SPARK-31986][SQL] Fix Julian-Gregorian micros rebasing of overlapping local timestamps

AmplabJenkins removed a comment on pull request #28816: URL: https://github.com/apache/spark/pull/28816#issuecomment-643784950 This is an automated message from the Apache Git Service. To respond to the message, please log on

[GitHub] [spark] AmplabJenkins commented on pull request #28816: [SPARK-31986][SQL] Fix Julian-Gregorian micros rebasing of overlapping local timestamps

AmplabJenkins commented on pull request #28816: URL: https://github.com/apache/spark/pull/28816#issuecomment-643784950 This is an automated message from the Apache Git Service. To respond to the message, please log on to

[GitHub] [spark] SparkQA removed a comment on pull request #28816: [SPARK-31986][SQL] Fix Julian-Gregorian micros rebasing of overlapping local timestamps

SparkQA removed a comment on pull request #28816: URL: https://github.com/apache/spark/pull/28816#issuecomment-643751654 **[Test build #124006 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124006/testReport)** for PR 28816 at commit

[GitHub] [spark] SparkQA commented on pull request #28816: [SPARK-31986][SQL] Fix Julian-Gregorian micros rebasing of overlapping local timestamps

SparkQA commented on pull request #28816: URL: https://github.com/apache/spark/pull/28816#issuecomment-643784685 **[Test build #124006 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/124006/testReport)** for PR 28816 at commit

[GitHub] [spark] MaxGekk commented on a change in pull request #28821: [SPARK-31981][SQL] Keep TimestampType when taking an average of a Timestamp

MaxGekk commented on a change in pull request #28821: URL: https://github.com/apache/spark/pull/28821#discussion_r439842079 ## File path: sql/core/src/test/scala/org/apache/spark/sql/test/SQLTestData.scala ## @@ -73,6 +74,17 @@ private[sql] trait SQLTestData { self => df

[GitHub] [spark] AmplabJenkins removed a comment on pull request #28825: [SPARk-31950][SQL][FOLLOW-UP][MINOR] Better error message on SPARK_HOME or…

AmplabJenkins removed a comment on pull request #28825: URL: https://github.com/apache/spark/pull/28825#issuecomment-643782881 This is an automated message from the Apache Git Service. To respond to the message, please log on

<    1   2   3   4   5   >