[spark] branch master updated (0660a05 -> 15b7333)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 0660a05 [SPARK-32525][DOCS] The layout of monitoring.html is broken add 15b7333 [SPARK-32507][DOCS][PYTHON] Add main page for PySpark documentation No new revisions were added by this update. Summary of changes: docs/img/pyspark-components.png | Bin 0 -> 32727 bytes docs/img/pyspark-components.pptx | Bin 0 -> 354673 bytes python/docs/source/index.rst | 36 3 files changed, 36 insertions(+) create mode 100644 docs/img/pyspark-components.png create mode 100644 docs/img/pyspark-components.pptx - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (0660a05 -> 15b7333)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 0660a05 [SPARK-32525][DOCS] The layout of monitoring.html is broken add 15b7333 [SPARK-32507][DOCS][PYTHON] Add main page for PySpark documentation No new revisions were added by this update. Summary of changes: docs/img/pyspark-components.png | Bin 0 -> 32727 bytes docs/img/pyspark-components.pptx | Bin 0 -> 354673 bytes python/docs/source/index.rst | 36 3 files changed, 36 insertions(+) create mode 100644 docs/img/pyspark-components.png create mode 100644 docs/img/pyspark-components.pptx - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (0660a05 -> 15b7333)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 0660a05 [SPARK-32525][DOCS] The layout of monitoring.html is broken add 15b7333 [SPARK-32507][DOCS][PYTHON] Add main page for PySpark documentation No new revisions were added by this update. Summary of changes: docs/img/pyspark-components.png | Bin 0 -> 32727 bytes docs/img/pyspark-components.pptx | Bin 0 -> 354673 bytes python/docs/source/index.rst | 36 3 files changed, 36 insertions(+) create mode 100644 docs/img/pyspark-components.png create mode 100644 docs/img/pyspark-components.pptx - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (0660a05 -> 15b7333)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 0660a05 [SPARK-32525][DOCS] The layout of monitoring.html is broken add 15b7333 [SPARK-32507][DOCS][PYTHON] Add main page for PySpark documentation No new revisions were added by this update. Summary of changes: docs/img/pyspark-components.png | Bin 0 -> 32727 bytes docs/img/pyspark-components.pptx | Bin 0 -> 354673 bytes python/docs/source/index.rst | 36 3 files changed, 36 insertions(+) create mode 100644 docs/img/pyspark-components.png create mode 100644 docs/img/pyspark-components.pptx - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (0660a05 -> 15b7333)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 0660a05 [SPARK-32525][DOCS] The layout of monitoring.html is broken add 15b7333 [SPARK-32507][DOCS][PYTHON] Add main page for PySpark documentation No new revisions were added by this update. Summary of changes: docs/img/pyspark-components.png | Bin 0 -> 32727 bytes docs/img/pyspark-components.pptx | Bin 0 -> 354673 bytes python/docs/source/index.rst | 36 3 files changed, 36 insertions(+) create mode 100644 docs/img/pyspark-components.png create mode 100644 docs/img/pyspark-components.pptx - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch branch-2.4 updated: [SPARK-32003][CORE][2.4] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost
This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch branch-2.4 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-2.4 by this push: new 39d31dc [SPARK-32003][CORE][2.4] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost 39d31dc is described below commit 39d31dcba362c1c34995d15495d6b3753f8355b5 Author: Wing Yew Poon AuthorDate: Tue Aug 4 14:36:59 2020 -0500 [SPARK-32003][CORE][2.4] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost ### What changes were proposed in this pull request? If an executor is lost, the `DAGScheduler` handles the executor loss by removing the executor but does not unregister its outputs if the external shuffle service is used. However, if the node on which the executor runs is lost, the shuffle service may not be able to serve the shuffle files. In such a case, when fetches from the executor's outputs fail in the same stage, the `DAGScheduler` again removes the executor and by right, should unregister its outputs. It doesn't because the epoch used to track the executor failure has not increased. We track the epoch for failed executors that result in lost file output separately, so we can unregister the outputs in this scenario. The idea to track a second epoch is due to Attila Zsolt Piros. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New unit test. This test fails without the change and passes with it. Closes #29182 from wypoon/SPARK-32003-2.4. Authored-by: Wing Yew Poon Signed-off-by: Imran Rashid --- .../org/apache/spark/scheduler/DAGScheduler.scala | 100 +--- .../apache/spark/scheduler/DAGSchedulerSuite.scala | 101 + 2 files changed, 151 insertions(+), 50 deletions(-) diff --git a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala index d0d12d8..18baa0b 100644 --- a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala +++ b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala @@ -171,13 +171,34 @@ private[spark] class DAGScheduler( */ private val cacheLocs = new HashMap[Int, IndexedSeq[Seq[TaskLocation]]] - // For tracking failed nodes, we use the MapOutputTracker's epoch number, which is sent with - // every task. When we detect a node failing, we note the current epoch number and failed - // executor, increment it for new tasks, and use this to ignore stray ShuffleMapTask results. - // - // TODO: Garbage collect information about failure epochs when we know there are no more - // stray messages to detect. - private val failedEpoch = new HashMap[String, Long] + /** + * Tracks the latest epoch of a fully processed error related to the given executor. (We use + * the MapOutputTracker's epoch number, which is sent with every task.) + * + * When an executor fails, it can affect the results of many tasks, and we have to deal with + * all of them consistently. We don't simply ignore all future results from that executor, + * as the failures may have been transient; but we also don't want to "overreact" to follow- + * on errors we receive. Furthermore, we might receive notification of a task success, after + * we find out the executor has actually failed; we'll assume those successes are, in fact, + * simply delayed notifications and the results have been lost, if the tasks started in the + * same or an earlier epoch. In particular, we use this to control when we tell the + * BlockManagerMaster that the BlockManager has been lost. + */ + private val executorFailureEpoch = new HashMap[String, Long] + + /** + * Tracks the latest epoch of a fully processed error where shuffle files have been lost from + * the given executor. + * + * This is closely related to executorFailureEpoch. They only differ for the executor when + * there is an external shuffle service serving shuffle files and we haven't been notified that + * the entire worker has been lost. In that case, when an executor is lost, we do not update + * the shuffleFileLostEpoch; we wait for a fetch failure. This way, if only the executor + * fails, we do not unregister the shuffle data as it can still be served; but if there is + * a failure in the shuffle service (resulting in fetch failure), we unregister the shuffle + * data only once, even if we get many fetch failures. + */ + private val shuffleFileLostEpoch = new HashMap[String, Long] private [scheduler] val outputCommitCoordinator = env.outputCommitCoordinator @@ -1389,7 +1410,8 @@ private[spark] class DAGScheduler( val status = e
[spark] branch branch-3.0 updated: [SPARK-32003][CORE][3.0] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost
This is an automated email from the ASF dual-hosted git repository. irashid pushed a commit to branch branch-3.0 in repository https://gitbox.apache.org/repos/asf/spark.git The following commit(s) were added to refs/heads/branch-3.0 by this push: new fd445cb [SPARK-32003][CORE][3.0] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost fd445cb is described below commit fd445cb46ce940fbc68f85d2db14367d9f58739a Author: Wing Yew Poon AuthorDate: Tue Aug 4 11:33:56 2020 -0500 [SPARK-32003][CORE][3.0] When external shuffle service is used, unregister outputs for executor on fetch failure after executor is lost ### What changes were proposed in this pull request? If an executor is lost, the `DAGScheduler` handles the executor loss by removing the executor but does not unregister its outputs if the external shuffle service is used. However, if the node on which the executor runs is lost, the shuffle service may not be able to serve the shuffle files. In such a case, when fetches from the executor's outputs fail in the same stage, the `DAGScheduler` again removes the executor and by right, should unregister its outputs. It doesn't because the epoch used to track the executor failure has not increased. We track the epoch for failed executors that result in lost file output separately, so we can unregister the outputs in this scenario. The idea to track a second epoch is due to Attila Zsolt Piros. ### Why are the changes needed? Without the changes, the loss of a node could require two stage attempts to recover instead of one. ### Does this PR introduce _any_ user-facing change? No. ### How was this patch tested? New unit test. This test fails without the change and passes with it. Closes #29193 from wypoon/SPARK-32003-3.0. Authored-by: Wing Yew Poon Signed-off-by: Imran Rashid --- .../org/apache/spark/scheduler/DAGScheduler.scala | 100 +--- .../apache/spark/scheduler/DAGSchedulerSuite.scala | 101 + 2 files changed, 151 insertions(+), 50 deletions(-) diff --git a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala index ec6eff3..51445bf 100644 --- a/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala +++ b/core/src/main/scala/org/apache/spark/scheduler/DAGScheduler.scala @@ -168,13 +168,34 @@ private[spark] class DAGScheduler( */ private val cacheLocs = new HashMap[Int, IndexedSeq[Seq[TaskLocation]]] - // For tracking failed nodes, we use the MapOutputTracker's epoch number, which is sent with - // every task. When we detect a node failing, we note the current epoch number and failed - // executor, increment it for new tasks, and use this to ignore stray ShuffleMapTask results. - // - // TODO: Garbage collect information about failure epochs when we know there are no more - // stray messages to detect. - private val failedEpoch = new HashMap[String, Long] + /** + * Tracks the latest epoch of a fully processed error related to the given executor. (We use + * the MapOutputTracker's epoch number, which is sent with every task.) + * + * When an executor fails, it can affect the results of many tasks, and we have to deal with + * all of them consistently. We don't simply ignore all future results from that executor, + * as the failures may have been transient; but we also don't want to "overreact" to follow- + * on errors we receive. Furthermore, we might receive notification of a task success, after + * we find out the executor has actually failed; we'll assume those successes are, in fact, + * simply delayed notifications and the results have been lost, if the tasks started in the + * same or an earlier epoch. In particular, we use this to control when we tell the + * BlockManagerMaster that the BlockManager has been lost. + */ + private val executorFailureEpoch = new HashMap[String, Long] + + /** + * Tracks the latest epoch of a fully processed error where shuffle files have been lost from + * the given executor. + * + * This is closely related to executorFailureEpoch. They only differ for the executor when + * there is an external shuffle service serving shuffle files and we haven't been notified that + * the entire worker has been lost. In that case, when an executor is lost, we do not update + * the shuffleFileLostEpoch; we wait for a fetch failure. This way, if only the executor + * fails, we do not unregister the shuffle data as it can still be served; but if there is + * a failure in the shuffle service (resulting in fetch failure), we unregister the shuffle + * data only once, even if we get many fetch failures. + */ + private val shuffleFileLostEpoch = new HashMap[String, Long] private [sche
[spark] branch master updated (7eb6f45 -> 0660a05)
This is an automated email from the ASF dual-hosted git repository.
gengliang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 7eb6f45 [SPARK-32499][SQL] Use `{}` in conversions maps and structs
to strings
add 0660a05 [SPARK-32525][DOCS] The layout of monitoring.html is broken
No new revisions were added by this update.
Summary of changes:
docs/monitoring.md | 2 ++
1 file changed, 2 insertions(+)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7eb6f45 -> 0660a05)
This is an automated email from the ASF dual-hosted git repository.
gengliang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 7eb6f45 [SPARK-32499][SQL] Use `{}` in conversions maps and structs
to strings
add 0660a05 [SPARK-32525][DOCS] The layout of monitoring.html is broken
No new revisions were added by this update.
Summary of changes:
docs/monitoring.md | 2 ++
1 file changed, 2 insertions(+)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7eb6f45 -> 0660a05)
This is an automated email from the ASF dual-hosted git repository.
gengliang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 7eb6f45 [SPARK-32499][SQL] Use `{}` in conversions maps and structs
to strings
add 0660a05 [SPARK-32525][DOCS] The layout of monitoring.html is broken
No new revisions were added by this update.
Summary of changes:
docs/monitoring.md | 2 ++
1 file changed, 2 insertions(+)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7eb6f45 -> 0660a05)
This is an automated email from the ASF dual-hosted git repository.
gengliang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 7eb6f45 [SPARK-32499][SQL] Use `{}` in conversions maps and structs
to strings
add 0660a05 [SPARK-32525][DOCS] The layout of monitoring.html is broken
No new revisions were added by this update.
Summary of changes:
docs/monitoring.md | 2 ++
1 file changed, 2 insertions(+)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7eb6f45 -> 0660a05)
This is an automated email from the ASF dual-hosted git repository.
gengliang pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 7eb6f45 [SPARK-32499][SQL] Use `{}` in conversions maps and structs
to strings
add 0660a05 [SPARK-32525][DOCS] The layout of monitoring.html is broken
No new revisions were added by this update.
Summary of changes:
docs/monitoring.md | 2 ++
1 file changed, 2 insertions(+)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (171b7d5 -> 7eb6f45)
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 171b7d5 [SPARK-23431][CORE] Expose stage level peak executor metrics
via REST API
add 7eb6f45 [SPARK-32499][SQL] Use `{}` in conversions maps and structs
to strings
No new revisions were added by this update.
Summary of changes:
docs/sql-migration-guide.md| 2 +
python/pyspark/ml/stat.py | 4 +-
python/pyspark/sql/functions.py| 28 +++
.../spark/sql/catalyst/expressions/Cast.scala | 20 +++--
.../org/apache/spark/sql/internal/SQLConf.scala| 9 +++
.../spark/sql/catalyst/expressions/CastSuite.scala | 88 --
.../test/resources/sql-tests/results/pivot.sql.out | 10 +--
.../sql-tests/results/udf/udf-pivot.sql.out| 8 +-
.../org/apache/spark/sql/DataFrameSuite.scala | 6 +-
.../scala/org/apache/spark/sql/DatasetSuite.scala | 4 +-
.../test/scala/org/apache/spark/sql/UDFSuite.scala | 4 +-
11 files changed, 105 insertions(+), 78 deletions(-)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (171b7d5 -> 7eb6f45)
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 171b7d5 [SPARK-23431][CORE] Expose stage level peak executor metrics
via REST API
add 7eb6f45 [SPARK-32499][SQL] Use `{}` in conversions maps and structs
to strings
No new revisions were added by this update.
Summary of changes:
docs/sql-migration-guide.md| 2 +
python/pyspark/ml/stat.py | 4 +-
python/pyspark/sql/functions.py| 28 +++
.../spark/sql/catalyst/expressions/Cast.scala | 20 +++--
.../org/apache/spark/sql/internal/SQLConf.scala| 9 +++
.../spark/sql/catalyst/expressions/CastSuite.scala | 88 --
.../test/resources/sql-tests/results/pivot.sql.out | 10 +--
.../sql-tests/results/udf/udf-pivot.sql.out| 8 +-
.../org/apache/spark/sql/DataFrameSuite.scala | 6 +-
.../scala/org/apache/spark/sql/DatasetSuite.scala | 4 +-
.../test/scala/org/apache/spark/sql/UDFSuite.scala | 4 +-
11 files changed, 105 insertions(+), 78 deletions(-)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (171b7d5 -> 7eb6f45)
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 171b7d5 [SPARK-23431][CORE] Expose stage level peak executor metrics
via REST API
add 7eb6f45 [SPARK-32499][SQL] Use `{}` in conversions maps and structs
to strings
No new revisions were added by this update.
Summary of changes:
docs/sql-migration-guide.md| 2 +
python/pyspark/ml/stat.py | 4 +-
python/pyspark/sql/functions.py| 28 +++
.../spark/sql/catalyst/expressions/Cast.scala | 20 +++--
.../org/apache/spark/sql/internal/SQLConf.scala| 9 +++
.../spark/sql/catalyst/expressions/CastSuite.scala | 88 --
.../test/resources/sql-tests/results/pivot.sql.out | 10 +--
.../sql-tests/results/udf/udf-pivot.sql.out| 8 +-
.../org/apache/spark/sql/DataFrameSuite.scala | 6 +-
.../scala/org/apache/spark/sql/DatasetSuite.scala | 4 +-
.../test/scala/org/apache/spark/sql/UDFSuite.scala | 4 +-
11 files changed, 105 insertions(+), 78 deletions(-)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (171b7d5 -> 7eb6f45)
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 171b7d5 [SPARK-23431][CORE] Expose stage level peak executor metrics
via REST API
add 7eb6f45 [SPARK-32499][SQL] Use `{}` in conversions maps and structs
to strings
No new revisions were added by this update.
Summary of changes:
docs/sql-migration-guide.md| 2 +
python/pyspark/ml/stat.py | 4 +-
python/pyspark/sql/functions.py| 28 +++
.../spark/sql/catalyst/expressions/Cast.scala | 20 +++--
.../org/apache/spark/sql/internal/SQLConf.scala| 9 +++
.../spark/sql/catalyst/expressions/CastSuite.scala | 88 --
.../test/resources/sql-tests/results/pivot.sql.out | 10 +--
.../sql-tests/results/udf/udf-pivot.sql.out| 8 +-
.../org/apache/spark/sql/DataFrameSuite.scala | 6 +-
.../scala/org/apache/spark/sql/DatasetSuite.scala | 4 +-
.../test/scala/org/apache/spark/sql/UDFSuite.scala | 4 +-
11 files changed, 105 insertions(+), 78 deletions(-)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (171b7d5 -> 7eb6f45)
This is an automated email from the ASF dual-hosted git repository.
wenchen pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git.
from 171b7d5 [SPARK-23431][CORE] Expose stage level peak executor metrics
via REST API
add 7eb6f45 [SPARK-32499][SQL] Use `{}` in conversions maps and structs
to strings
No new revisions were added by this update.
Summary of changes:
docs/sql-migration-guide.md| 2 +
python/pyspark/ml/stat.py | 4 +-
python/pyspark/sql/functions.py| 28 +++
.../spark/sql/catalyst/expressions/Cast.scala | 20 +++--
.../org/apache/spark/sql/internal/SQLConf.scala| 9 +++
.../spark/sql/catalyst/expressions/CastSuite.scala | 88 --
.../test/resources/sql-tests/results/pivot.sql.out | 10 +--
.../sql-tests/results/udf/udf-pivot.sql.out| 8 +-
.../org/apache/spark/sql/DataFrameSuite.scala | 6 +-
.../scala/org/apache/spark/sql/DatasetSuite.scala | 4 +-
.../test/scala/org/apache/spark/sql/UDFSuite.scala | 4 +-
11 files changed, 105 insertions(+), 78 deletions(-)
-
To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org
For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (6d69068 -> 171b7d5)
This is an automated email from the ASF dual-hosted git repository. gengliang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6d69068 [SPARK-32521][SQL] Bug-fix: WithFields Expression should not be foldable add 171b7d5 [SPARK-23431][CORE] Expose stage level peak executor metrics via REST API No new revisions were added by this update. Summary of changes: .../apache/spark/status/AppStatusListener.scala| 43 +- .../org/apache/spark/status/AppStatusStore.scala | 3 +- .../scala/org/apache/spark/status/LiveEntity.scala | 10 +- .../scala/org/apache/spark/status/api/v1/api.scala | 10 +- .../scala/org/apache/spark/ui/jobs/JobPage.scala | 3 +- .../application_list_json_expectation.json | 15 + .../completed_app_list_json_expectation.json | 15 + .../limit_app_list_json_expectation.json | 30 +- .../minDate_app_list_json_expectation.json | 45 +- .../minEndDate_app_list_json_expectation.json | 15 + .../stage_list_with_peak_metrics_expectation.json | 204 + .../stage_with_peak_metrics_expectation.json | 998 + .../resources/spark-events/app-20200706201101-0003 | 124 +++ .../spark/deploy/history/HistoryServerSuite.scala | 2 + .../spark/status/AppStatusListenerSuite.scala | 90 +- .../scala/org/apache/spark/ui/StagePageSuite.scala | 3 +- dev/.rat-excludes | 1 + 17 files changed, 1558 insertions(+), 53 deletions(-) create mode 100644 core/src/test/resources/HistoryServerExpectations/stage_list_with_peak_metrics_expectation.json create mode 100644 core/src/test/resources/HistoryServerExpectations/stage_with_peak_metrics_expectation.json create mode 100644 core/src/test/resources/spark-events/app-20200706201101-0003 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (6d69068 -> 171b7d5)
This is an automated email from the ASF dual-hosted git repository. gengliang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6d69068 [SPARK-32521][SQL] Bug-fix: WithFields Expression should not be foldable add 171b7d5 [SPARK-23431][CORE] Expose stage level peak executor metrics via REST API No new revisions were added by this update. Summary of changes: .../apache/spark/status/AppStatusListener.scala| 43 +- .../org/apache/spark/status/AppStatusStore.scala | 3 +- .../scala/org/apache/spark/status/LiveEntity.scala | 10 +- .../scala/org/apache/spark/status/api/v1/api.scala | 10 +- .../scala/org/apache/spark/ui/jobs/JobPage.scala | 3 +- .../application_list_json_expectation.json | 15 + .../completed_app_list_json_expectation.json | 15 + .../limit_app_list_json_expectation.json | 30 +- .../minDate_app_list_json_expectation.json | 45 +- .../minEndDate_app_list_json_expectation.json | 15 + .../stage_list_with_peak_metrics_expectation.json | 204 + .../stage_with_peak_metrics_expectation.json | 998 + .../resources/spark-events/app-20200706201101-0003 | 124 +++ .../spark/deploy/history/HistoryServerSuite.scala | 2 + .../spark/status/AppStatusListenerSuite.scala | 90 +- .../scala/org/apache/spark/ui/StagePageSuite.scala | 3 +- dev/.rat-excludes | 1 + 17 files changed, 1558 insertions(+), 53 deletions(-) create mode 100644 core/src/test/resources/HistoryServerExpectations/stage_list_with_peak_metrics_expectation.json create mode 100644 core/src/test/resources/HistoryServerExpectations/stage_with_peak_metrics_expectation.json create mode 100644 core/src/test/resources/spark-events/app-20200706201101-0003 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (6d69068 -> 171b7d5)
This is an automated email from the ASF dual-hosted git repository. gengliang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6d69068 [SPARK-32521][SQL] Bug-fix: WithFields Expression should not be foldable add 171b7d5 [SPARK-23431][CORE] Expose stage level peak executor metrics via REST API No new revisions were added by this update. Summary of changes: .../apache/spark/status/AppStatusListener.scala| 43 +- .../org/apache/spark/status/AppStatusStore.scala | 3 +- .../scala/org/apache/spark/status/LiveEntity.scala | 10 +- .../scala/org/apache/spark/status/api/v1/api.scala | 10 +- .../scala/org/apache/spark/ui/jobs/JobPage.scala | 3 +- .../application_list_json_expectation.json | 15 + .../completed_app_list_json_expectation.json | 15 + .../limit_app_list_json_expectation.json | 30 +- .../minDate_app_list_json_expectation.json | 45 +- .../minEndDate_app_list_json_expectation.json | 15 + .../stage_list_with_peak_metrics_expectation.json | 204 + .../stage_with_peak_metrics_expectation.json | 998 + .../resources/spark-events/app-20200706201101-0003 | 124 +++ .../spark/deploy/history/HistoryServerSuite.scala | 2 + .../spark/status/AppStatusListenerSuite.scala | 90 +- .../scala/org/apache/spark/ui/StagePageSuite.scala | 3 +- dev/.rat-excludes | 1 + 17 files changed, 1558 insertions(+), 53 deletions(-) create mode 100644 core/src/test/resources/HistoryServerExpectations/stage_list_with_peak_metrics_expectation.json create mode 100644 core/src/test/resources/HistoryServerExpectations/stage_with_peak_metrics_expectation.json create mode 100644 core/src/test/resources/spark-events/app-20200706201101-0003 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (6d69068 -> 171b7d5)
This is an automated email from the ASF dual-hosted git repository. gengliang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6d69068 [SPARK-32521][SQL] Bug-fix: WithFields Expression should not be foldable add 171b7d5 [SPARK-23431][CORE] Expose stage level peak executor metrics via REST API No new revisions were added by this update. Summary of changes: .../apache/spark/status/AppStatusListener.scala| 43 +- .../org/apache/spark/status/AppStatusStore.scala | 3 +- .../scala/org/apache/spark/status/LiveEntity.scala | 10 +- .../scala/org/apache/spark/status/api/v1/api.scala | 10 +- .../scala/org/apache/spark/ui/jobs/JobPage.scala | 3 +- .../application_list_json_expectation.json | 15 + .../completed_app_list_json_expectation.json | 15 + .../limit_app_list_json_expectation.json | 30 +- .../minDate_app_list_json_expectation.json | 45 +- .../minEndDate_app_list_json_expectation.json | 15 + .../stage_list_with_peak_metrics_expectation.json | 204 + .../stage_with_peak_metrics_expectation.json | 998 + .../resources/spark-events/app-20200706201101-0003 | 124 +++ .../spark/deploy/history/HistoryServerSuite.scala | 2 + .../spark/status/AppStatusListenerSuite.scala | 90 +- .../scala/org/apache/spark/ui/StagePageSuite.scala | 3 +- dev/.rat-excludes | 1 + 17 files changed, 1558 insertions(+), 53 deletions(-) create mode 100644 core/src/test/resources/HistoryServerExpectations/stage_list_with_peak_metrics_expectation.json create mode 100644 core/src/test/resources/HistoryServerExpectations/stage_with_peak_metrics_expectation.json create mode 100644 core/src/test/resources/spark-events/app-20200706201101-0003 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (6d69068 -> 171b7d5)
This is an automated email from the ASF dual-hosted git repository. gengliang pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 6d69068 [SPARK-32521][SQL] Bug-fix: WithFields Expression should not be foldable add 171b7d5 [SPARK-23431][CORE] Expose stage level peak executor metrics via REST API No new revisions were added by this update. Summary of changes: .../apache/spark/status/AppStatusListener.scala| 43 +- .../org/apache/spark/status/AppStatusStore.scala | 3 +- .../scala/org/apache/spark/status/LiveEntity.scala | 10 +- .../scala/org/apache/spark/status/api/v1/api.scala | 10 +- .../scala/org/apache/spark/ui/jobs/JobPage.scala | 3 +- .../application_list_json_expectation.json | 15 + .../completed_app_list_json_expectation.json | 15 + .../limit_app_list_json_expectation.json | 30 +- .../minDate_app_list_json_expectation.json | 45 +- .../minEndDate_app_list_json_expectation.json | 15 + .../stage_list_with_peak_metrics_expectation.json | 204 + .../stage_with_peak_metrics_expectation.json | 998 + .../resources/spark-events/app-20200706201101-0003 | 124 +++ .../spark/deploy/history/HistoryServerSuite.scala | 2 + .../spark/status/AppStatusListenerSuite.scala | 90 +- .../scala/org/apache/spark/ui/StagePageSuite.scala | 3 +- dev/.rat-excludes | 1 + 17 files changed, 1558 insertions(+), 53 deletions(-) create mode 100644 core/src/test/resources/HistoryServerExpectations/stage_list_with_peak_metrics_expectation.json create mode 100644 core/src/test/resources/HistoryServerExpectations/stage_with_peak_metrics_expectation.json create mode 100644 core/src/test/resources/spark-events/app-20200706201101-0003 - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7fec6e0 -> 6d69068)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7fec6e0 [SPARK-32524][SQL][TESTS] CachedBatchSerializerSuite should clean up InMemoryRelation.ser add 6d69068 [SPARK-32521][SQL] Bug-fix: WithFields Expression should not be foldable No new revisions were added by this update. Summary of changes: .../catalyst/expressions/complexTypeCreator.scala | 2 -- .../apache/spark/sql/ColumnExpressionSuite.scala | 32 ++ 2 files changed, 32 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7fec6e0 -> 6d69068)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7fec6e0 [SPARK-32524][SQL][TESTS] CachedBatchSerializerSuite should clean up InMemoryRelation.ser add 6d69068 [SPARK-32521][SQL] Bug-fix: WithFields Expression should not be foldable No new revisions were added by this update. Summary of changes: .../catalyst/expressions/complexTypeCreator.scala | 2 -- .../apache/spark/sql/ColumnExpressionSuite.scala | 32 ++ 2 files changed, 32 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7fec6e0 -> 6d69068)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7fec6e0 [SPARK-32524][SQL][TESTS] CachedBatchSerializerSuite should clean up InMemoryRelation.ser add 6d69068 [SPARK-32521][SQL] Bug-fix: WithFields Expression should not be foldable No new revisions were added by this update. Summary of changes: .../catalyst/expressions/complexTypeCreator.scala | 2 -- .../apache/spark/sql/ColumnExpressionSuite.scala | 32 ++ 2 files changed, 32 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7fec6e0 -> 6d69068)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7fec6e0 [SPARK-32524][SQL][TESTS] CachedBatchSerializerSuite should clean up InMemoryRelation.ser add 6d69068 [SPARK-32521][SQL] Bug-fix: WithFields Expression should not be foldable No new revisions were added by this update. Summary of changes: .../catalyst/expressions/complexTypeCreator.scala | 2 -- .../apache/spark/sql/ColumnExpressionSuite.scala | 32 ++ 2 files changed, 32 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (7fec6e0 -> 6d69068)
This is an automated email from the ASF dual-hosted git repository. wenchen pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 7fec6e0 [SPARK-32524][SQL][TESTS] CachedBatchSerializerSuite should clean up InMemoryRelation.ser add 6d69068 [SPARK-32521][SQL] Bug-fix: WithFields Expression should not be foldable No new revisions were added by this update. Summary of changes: .../catalyst/expressions/complexTypeCreator.scala | 2 -- .../apache/spark/sql/ColumnExpressionSuite.scala | 32 ++ 2 files changed, 32 insertions(+), 2 deletions(-) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (005ef3a -> 7fec6e0)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 005ef3a [SPARK-32468][SS][TESTS][FOLLOWUP] Provide "default.api.timeout.ms" as well when specifying "request.timeout.ms" on replacing "default.api.timeout.ms" add 7fec6e0 [SPARK-32524][SQL][TESTS] CachedBatchSerializerSuite should clean up InMemoryRelation.ser No new revisions were added by this update. Summary of changes: .../spark/sql/execution/columnar/InMemoryRelation.scala | 3 +++ .../sql/execution/columnar/CachedBatchSerializerSuite.scala | 11 +++ 2 files changed, 14 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (005ef3a -> 7fec6e0)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 005ef3a [SPARK-32468][SS][TESTS][FOLLOWUP] Provide "default.api.timeout.ms" as well when specifying "request.timeout.ms" on replacing "default.api.timeout.ms" add 7fec6e0 [SPARK-32524][SQL][TESTS] CachedBatchSerializerSuite should clean up InMemoryRelation.ser No new revisions were added by this update. Summary of changes: .../spark/sql/execution/columnar/InMemoryRelation.scala | 3 +++ .../sql/execution/columnar/CachedBatchSerializerSuite.scala | 11 +++ 2 files changed, 14 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (005ef3a -> 7fec6e0)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 005ef3a [SPARK-32468][SS][TESTS][FOLLOWUP] Provide "default.api.timeout.ms" as well when specifying "request.timeout.ms" on replacing "default.api.timeout.ms" add 7fec6e0 [SPARK-32524][SQL][TESTS] CachedBatchSerializerSuite should clean up InMemoryRelation.ser No new revisions were added by this update. Summary of changes: .../spark/sql/execution/columnar/InMemoryRelation.scala | 3 +++ .../sql/execution/columnar/CachedBatchSerializerSuite.scala | 11 +++ 2 files changed, 14 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (005ef3a -> 7fec6e0)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 005ef3a [SPARK-32468][SS][TESTS][FOLLOWUP] Provide "default.api.timeout.ms" as well when specifying "request.timeout.ms" on replacing "default.api.timeout.ms" add 7fec6e0 [SPARK-32524][SQL][TESTS] CachedBatchSerializerSuite should clean up InMemoryRelation.ser No new revisions were added by this update. Summary of changes: .../spark/sql/execution/columnar/InMemoryRelation.scala | 3 +++ .../sql/execution/columnar/CachedBatchSerializerSuite.scala | 11 +++ 2 files changed, 14 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
[spark] branch master updated (005ef3a -> 7fec6e0)
This is an automated email from the ASF dual-hosted git repository. gurwls223 pushed a change to branch master in repository https://gitbox.apache.org/repos/asf/spark.git. from 005ef3a [SPARK-32468][SS][TESTS][FOLLOWUP] Provide "default.api.timeout.ms" as well when specifying "request.timeout.ms" on replacing "default.api.timeout.ms" add 7fec6e0 [SPARK-32524][SQL][TESTS] CachedBatchSerializerSuite should clean up InMemoryRelation.ser No new revisions were added by this update. Summary of changes: .../spark/sql/execution/columnar/InMemoryRelation.scala | 3 +++ .../sql/execution/columnar/CachedBatchSerializerSuite.scala | 11 +++ 2 files changed, 14 insertions(+) - To unsubscribe, e-mail: commits-unsubscr...@spark.apache.org For additional commands, e-mail: commits-h...@spark.apache.org
