[GitHub] [spark] yaooqinn commented on a diff in pull request #36052: [SPARK-38777][YARN] Add `bin/spark-submit --kill / --status` support for yarn

yaooqinn commented on code in PR #36052: URL: https://github.com/apache/spark/pull/36052#discussion_r841461428 ## resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnSparkSubmitOperation.scala: ## @@ -0,0 +1,136 @@ +/* + * Licensed to the Apache Software

[GitHub] [spark] monkeyboy123 commented on a diff in pull request #35984: [MINOR][SQL] Show debug log for `AnalysisException` in Analyzer

monkeyboy123 commented on code in PR #35984: URL: https://github.com/apache/spark/pull/35984#discussion_r841430645 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala: ## @@ -1761,7 +1761,9 @@ class Analyzer(override val catalogManager:

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36038: [SPARK-38759][PYTHON][SS] Add StreamingQueryListener support in PySpark

HyukjinKwon commented on code in PR #36038: URL: https://github.com/apache/spark/pull/36038#discussion_r841351936 ## python/docs/source/reference/pyspark.ss.rst: ## @@ -30,10 +30,10 @@ Core Classes DataStreamReader DataStreamWriter -ForeachBatchFunction Review

[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #35856: [SPARK-38550][SQL][CORE] Use a disk-based store to save more debug information for live UI

dongjoon-hyun commented on code in PR #35856: URL: https://github.com/apache/spark/pull/35856#discussion_r841422549 ## sql/core/src/main/scala/org/apache/spark/sql/internal/SharedState.scala: ## @@ -118,6 +119,12 @@ private[sql] class SharedState( statusStore } +

[GitHub] [spark] dongjoon-hyun commented on a diff in pull request #35856: [SPARK-38550][SQL][CORE] Use a disk-based store to save more debug information for live UI

dongjoon-hyun commented on code in PR #35856: URL: https://github.com/apache/spark/pull/35856#discussion_r841421049 ## core/src/main/scala/org/apache/spark/internal/config/Status.scala: ## @@ -70,4 +70,11 @@ private[spark] object Status { .version("3.0.0")

[GitHub] [spark] yaooqinn commented on a diff in pull request #36053: [SPARK-38778][INFRA][BUILD] Replace http with https for project url in pom

yaooqinn commented on code in PR #36053: URL: https://github.com/apache/spark/pull/36053#discussion_r841373575 ## pom.xml: ## @@ -29,7 +29,7 @@ 3.4.0-SNAPSHOT pom Spark Project Parent POM - http://spark.apache.org/ Review Comment: updated, thanks. -- This is

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36038: [SPARK-38759][PYTHON][SS] Add StreamingQueryListener support in PySpark

HyukjinKwon commented on code in PR #36038: URL: https://github.com/apache/spark/pull/36038#discussion_r841351936 ## python/docs/source/reference/pyspark.ss.rst: ## @@ -30,10 +30,10 @@ Core Classes DataStreamReader DataStreamWriter -ForeachBatchFunction Review

[GitHub] [spark] wangyum commented on a diff in pull request #36053: [SPARK-38778][INFRA][BUILD] Replace http with https for project url in pom

wangyum commented on code in PR #36053: URL: https://github.com/apache/spark/pull/36053#discussion_r841345386 ## pom.xml: ## @@ -29,7 +29,7 @@ 3.4.0-SNAPSHOT pom Spark Project Parent POM - http://spark.apache.org/ Review Comment: Line 53?

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #36038: [SPARK-38759][PYTHON][SS] Add StreamingQueryListener support in PySpark

HyukjinKwon commented on code in PR #36038: URL: https://github.com/apache/spark/pull/36038#discussion_r84177 ## python/pyspark/sql/streaming/listener.py: ## @@ -0,0 +1,666 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license

[GitHub] [spark] monkeyboy123 commented on a diff in pull request #35984: [MINOR] Log AnalysisException output for debug and tracing

monkeyboy123 commented on code in PR #35984: URL: https://github.com/apache/spark/pull/35984#discussion_r841321778 ## sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/Analyzer.scala: ## @@ -1761,7 +1761,9 @@ class Analyzer(override val catalogManager:

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #35979: [SPARK-38664][CORE] Support compact EventLog when there are illegal characters in the path

HyukjinKwon commented on code in PR #35979: URL: https://github.com/apache/spark/pull/35979#discussion_r841300801 ## core/src/main/scala/org/apache/spark/deploy/history/EventLogFileCompactor.scala: ## @@ -221,5 +221,5 @@ private class CompactedEventLogFileWriter(

[GitHub] [spark] zhengruifeng commented on a diff in pull request #36048: [SPARK-38774][PYTHON] Implement Series.autocorr

zhengruifeng commented on code in PR #36048: URL: https://github.com/apache/spark/pull/36048#discussion_r841305123 ## python/pyspark/pandas/series.py: ## @@ -2937,6 +2937,73 @@ def add_suffix(self, suffix: str) -> "Series": DataFrame(internal.with_new_sdf(sdf,

[GitHub] [spark] zhengruifeng commented on a diff in pull request #36049: [SPARK-38775][ML] cleanup validation functions

zhengruifeng commented on code in PR #36049: URL: https://github.com/apache/spark/pull/36049#discussion_r841303261 ## mllib/src/main/scala/org/apache/spark/ml/util/DatasetUtils.scala: ## @@ -138,4 +140,61 @@ private[spark] object DatasetUtils { case Row(point: Vector) =>

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #35979: [SPARK-38664][CORE] Support compact EventLog when there are illegal characters in the path

HyukjinKwon commented on code in PR #35979: URL: https://github.com/apache/spark/pull/35979#discussion_r841300801 ## core/src/main/scala/org/apache/spark/deploy/history/EventLogFileCompactor.scala: ## @@ -221,5 +221,5 @@ private class CompactedEventLogFileWriter(

[GitHub] [spark] HyukjinKwon commented on a diff in pull request #35979: [SPARK-38664][CORE] Support compact EventLog when there are illegal characters in the path

HyukjinKwon commented on code in PR #35979: URL: https://github.com/apache/spark/pull/35979#discussion_r841300801 ## core/src/main/scala/org/apache/spark/deploy/history/EventLogFileCompactor.scala: ## @@ -221,5 +221,5 @@ private class CompactedEventLogFileWriter(

[GitHub] [spark] huaxingao commented on a diff in pull request #36043: [SPARK-38768][SQL] Remove `Limit` from plan if complete push down limit to data source.

huaxingao commented on code in PR #36043: URL: https://github.com/apache/spark/pull/36043#discussion_r841285841 ## sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/v2/V2ScanRelationPushDown.scala: ## @@ -380,27 +380,32 @@ object V2ScanRelationPushDown extends

[GitHub] srowen closed pull request #166: Further expand and update the merge and commit process for committers

srowen closed pull request #166: Further expand and update the merge and commit process for committers URL: https://github.com/apache/spark-website/pull/166 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of

[GitHub] dongjoon-hyun commented on issue #166: Further expand and update the merge and commit process for committers

dongjoon-hyun commented on issue #166: Further expand and update the merge and commit process for committers URL: https://github.com/apache/spark-website/pull/166#issuecomment-453167761 Great! Thank you for updating, @srowen .

[GitHub] dongjoon-hyun commented on a change in pull request #166: Further expand and update the merge and commit process for committers

dongjoon-hyun commented on a change in pull request #166: Further expand and update the merge and commit process for committers URL: https://github.com/apache/spark-website/pull/166#discussion_r246836939 ## File path: committers.md ## @@ -127,13 +127,41 @@ Git history for

[GitHub] srowen opened a new pull request #166: Further expand and update the merge and commit process for committers

srowen opened a new pull request #166: Further expand and update the merge and commit process for committers URL: https://github.com/apache/spark-website/pull/166 Following up on https://github.com/apache/spark-website/commit/eb0aa14df472cff092b35ea1b894a0d880185561#r31886611 with

[GitHub] srowen closed pull request #165: Update 2018-12-19-spark-ai-summit-apr-2019-agenda-posted.md

srowen closed pull request #165: Update 2018-12-19-spark-ai-summit-apr-2019-agenda-posted.md URL: https://github.com/apache/spark-website/pull/165 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance:

[GitHub] jzhuge commented on issue #165: Update 2018-12-19-spark-ai-summit-apr-2019-agenda-posted.md

jzhuge commented on issue #165: Update 2018-12-19-spark-ai-summit-apr-2019-agenda-posted.md URL: https://github.com/apache/spark-website/pull/165#issuecomment-452937118 Sure. On Wed, Jan 9, 2019 at 5:18 PM Sean Owen wrote: > Oops, good catch @jzhuge

[GitHub] srowen commented on issue #165: Update 2018-12-19-spark-ai-summit-apr-2019-agenda-posted.md

srowen commented on issue #165: Update 2018-12-19-spark-ai-summit-apr-2019-agenda-posted.md URL: https://github.com/apache/spark-website/pull/165#issuecomment-452934048 Oops, good catch @jzhuge . Can you run `jekyll build` locally to also update the HTML? if it's any trouble I can do it

[GitHub] jzhuge opened a new pull request #165: Update 2018-12-19-spark-ai-summit-apr-2019-agenda-posted.md

jzhuge opened a new pull request #165: Update 2018-12-19-spark-ai-summit-apr-2019-agenda-posted.md URL: https://github.com/apache/spark-website/pull/165 This is an automated message from the Apache Git Service. To respond

[GitHub] srowen closed pull request #164: Suggest new Apache repo for committers

srowen closed pull request #164: Suggest new Apache repo for committers URL: https://github.com/apache/spark-website/pull/164 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign

[GitHub] rxin commented on issue #164: Suggest new Apache repo for committers

rxin commented on issue #164: Suggest new Apache repo for committers URL: https://github.com/apache/spark-website/pull/164#issuecomment-452047166 LGTM This is an automated message from the Apache Git Service. To respond to

[GitHub] srowen opened a new pull request #164: Suggest new Apache repo for committers

srowen opened a new pull request #164: Suggest new Apache repo for committers URL: https://github.com/apache/spark-website/pull/164 This suggests to committers that they should use the new github remote to push to Apache.

[GitHub] srowen commented on a change in pull request #163: Announce the schedule of 2019 Spark+AI summit at SF

srowen commented on a change in pull request #163: Announce the schedule of 2019 Spark+AI summit at SF URL: https://github.com/apache/spark-website/pull/163#discussion_r243353800 ## File path: site/sitemap.xml ## @@ -139,657 +139,661 @@ -

[GitHub] gatorsmile commented on a change in pull request #163: Announce the schedule of 2019 Spark+AI summit at SF

gatorsmile commented on a change in pull request #163: Announce the schedule of 2019 Spark+AI summit at SF URL: https://github.com/apache/spark-website/pull/163#discussion_r243158425 ## File path: site/sitemap.xml ## @@ -139,657 +139,661 @@ -

[GitHub] ueshin commented on a change in pull request #163: Announce the schedule of 2019 Spark+AI summit at SF

ueshin commented on a change in pull request #163: Announce the schedule of 2019 Spark+AI summit at SF URL: https://github.com/apache/spark-website/pull/163#discussion_r243132369 ## File path: site/sitemap.xml ## @@ -139,657 +139,661 @@ -

[GitHub] gatorsmile commented on issue #163: Announce the schedule of 2019 Spark+AI summit at SF

gatorsmile commented on issue #163: Announce the schedule of 2019 Spark+AI summit at SF URL: https://github.com/apache/spark-website/pull/163#issuecomment-448825575 Thanks! Merged to master. This is an automated message from

[GitHub] ueshin commented on a change in pull request #163: Announce the schedule of 2019 Spark+AI summit at SF

ueshin commented on a change in pull request #163: Announce the schedule of 2019 Spark+AI summit at SF URL: https://github.com/apache/spark-website/pull/163#discussion_r243130975 ## File path: site/sitemap.xml ## @@ -139,657 +139,661 @@ -

[GitHub] gatorsmile commented on a change in pull request #163: Announce the schedule of 2019 Spark+AI summit at SF

gatorsmile commented on a change in pull request #163: Announce the schedule of 2019 Spark+AI summit at SF URL: https://github.com/apache/spark-website/pull/163#discussion_r243128948 ## File path: site/mailing-lists.html ## @@ -12,7 +12,7 @@ -

[GitHub] gatorsmile commented on issue #163: Announce the schedule of 2019 Spark+AI summit at SF

gatorsmile commented on issue #163: Announce the schedule of 2019 Spark+AI summit at SF URL: https://github.com/apache/spark-website/pull/163#issuecomment-448815820 cc @rxin @yhuai @cloud-fan @srowen This is an automated

[GitHub] gatorsmile opened a new pull request #163: Announce the schedule of Spark+AI summit at SF 2019

gatorsmile opened a new pull request #163: Announce the schedule of Spark+AI summit at SF 2019 URL: https://github.com/apache/spark-website/pull/163 ![screen shot 2018-12-19 at 4 59 12

[GitHub] HyukjinKwon closed pull request #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools

HyukjinKwon closed pull request #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools URL: https://github.com/apache/spark-website/pull/162 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below

[GitHub] HyukjinKwon commented on issue #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools

HyukjinKwon commented on issue #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools URL: https://github.com/apache/spark-website/pull/162#issuecomment-448164740 Thanks guys! This is an

[GitHub] HyukjinKwon commented on issue #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools

HyukjinKwon commented on issue #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools URL: https://github.com/apache/spark-website/pull/162#issuecomment-448075651 adding @squito as well FYI

[GitHub] HyukjinKwon opened a new pull request #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools

HyukjinKwon opened a new pull request #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools URL: https://github.com/apache/spark-website/pull/162 I received some feedback about running PySpark tests via private emails. Unlike SBT or Maven testing,

[GitHub] HyukjinKwon commented on issue #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools

HyukjinKwon commented on issue #162: Add a note about Spark build requirement at PySpark testing guide in Developer Tools URL: https://github.com/apache/spark-website/pull/162#issuecomment-448075198 adding @cloud-fan and @srowen.