Matt Cheah created SPARK-32504:
--
Summary: Shuffle Storage API: Dynamic updates of shuffle metadata
Key: SPARK-32504
URL: https://issues.apache.org/jira/browse/SPARK-32504
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-28210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168287#comment-17168287
]
Matt Cheah commented on SPARK-28210:
[~devaraj] [~tianczha] Thanks for expressing interest in this!
Matt Cheah created SPARK-31801:
--
Summary: Register shuffle map output metadata with a shuffle
output tracker
Key: SPARK-31801
URL: https://issues.apache.org/jira/browse/SPARK-31801
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-31798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-31798:
---
Description:
Part of the overall sub-push for shuffle metadata management as proposed in
[this
[
https://issues.apache.org/jira/browse/SPARK-31798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-31798:
---
Description:
Part of the overall sub-push for shuffle metadata management as proposed in
[this
[
https://issues.apache.org/jira/browse/SPARK-31798?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-31798:
---
Description:
Part of the overall sub-push for shuffle metadata management as proposed in
[this
Matt Cheah created SPARK-31798:
--
Summary: Return map output metadata from shuffle writers
Key: SPARK-31798
URL: https://issues.apache.org/jira/browse/SPARK-31798
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-29072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-29072:
---
Description: From SPARK-28209, SPARK-28570, and SPARK-28571, we used the
new shuffle writer plugin
[
https://issues.apache.org/jira/browse/SPARK-29072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-29072:
---
Description:
>From SPARK-28209, SPARK-28570, and SPARK-28571, we used the new shuffle writer
[
https://issues.apache.org/jira/browse/SPARK-29072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-29072:
---
Description:
>From SPARK-28209, SPARK-28570, and SPARK-28571, we used the new shuffle writer
Matt Cheah created SPARK-29072:
--
Summary: Properly track shuffle write time with refactor
Key: SPARK-29072
URL: https://issues.apache.org/jira/browse/SPARK-29072
Project: Spark
Issue Type: Bug
Matt Cheah created SPARK-28764:
--
Summary: Remove unnecessary writePartitionedFile method from
ExternalSorter
Key: SPARK-28764
URL: https://issues.apache.org/jira/browse/SPARK-28764
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-28607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-28607:
---
Description: SPARK-28209 introduced the new shuffle writer API and its
usage in
Matt Cheah created SPARK-28607:
--
Summary: Don't hold a reference to two partitionLengths arrays
Key: SPARK-28607
URL: https://issues.apache.org/jira/browse/SPARK-28607
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-28209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-28209:
---
Description: Adds the write-side API for storing shuffle data in arbitrary
storage systems. Also
Matt Cheah created SPARK-28571:
--
Summary: Shuffle storage API: Use API in SortShuffleWriter
Key: SPARK-28571
URL: https://issues.apache.org/jira/browse/SPARK-28571
Project: Spark
Issue Type:
Matt Cheah created SPARK-28570:
--
Summary: Shuffle Storage API: Use writer API in UnsafeShuffleWriter
Key: SPARK-28570
URL: https://issues.apache.org/jira/browse/SPARK-28570
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-28209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-28209:
---
Summary: Shuffle Storage API: Writer API and usage in
BypassMergeSortShuffleWriter (was: Shuffle
[
https://issues.apache.org/jira/browse/SPARK-28209?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-28209:
---
Summary: Shuffle Storage API: Writer API (was: Shuffle Storage API: Writes)
> Shuffle Storage API:
Matt Cheah created SPARK-28568:
--
Summary: Make Javadoc in org.apache.spark.shuffle.api visible
Key: SPARK-28568
URL: https://issues.apache.org/jira/browse/SPARK-28568
Project: Spark
Issue Type:
Matt Cheah created SPARK-28238:
--
Summary: DESCRIBE TABLE for Data Source V2 tables
Key: SPARK-28238
URL: https://issues.apache.org/jira/browse/SPARK-28238
Project: Spark
Issue Type: New Feature
Matt Cheah created SPARK-28212:
--
Summary: Shuffle Storage API: Shuffle Cleanup
Key: SPARK-28212
URL: https://issues.apache.org/jira/browse/SPARK-28212
Project: Spark
Issue Type: Sub-task
Matt Cheah created SPARK-28211:
--
Summary: Shuffle Storage API: Driver Lifecycle
Key: SPARK-28211
URL: https://issues.apache.org/jira/browse/SPARK-28211
Project: Spark
Issue Type: Sub-task
Matt Cheah created SPARK-28210:
--
Summary: Shuffle Storage API: Reads
Key: SPARK-28210
URL: https://issues.apache.org/jira/browse/SPARK-28210
Project: Spark
Issue Type: Sub-task
[
https://issues.apache.org/jira/browse/SPARK-25299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-25299:
---
Description:
In Spark, the shuffle primitive requires Spark executors to persist data to the
local
[
https://issues.apache.org/jira/browse/SPARK-25299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875106#comment-16875106
]
Matt Cheah commented on SPARK-25299:
I also noticed the SPIP document wasn't ever posted on this
Matt Cheah created SPARK-28209:
--
Summary: Shuffle Storage API: Writes
Key: SPARK-28209
URL: https://issues.apache.org/jira/browse/SPARK-28209
Project: Spark
Issue Type: Sub-task
[
https://issues.apache.org/jira/browse/SPARK-25299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16875101#comment-16875101
]
Matt Cheah commented on SPARK-25299:
Let's start by making sub-issues. I have a patch staged for
[
https://issues.apache.org/jira/browse/SPARK-26874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-26874.
Resolution: Not A Problem
I did some more digging with [~rdblue] and we discovered that this is a
[
https://issues.apache.org/jira/browse/SPARK-26874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-26874:
---
Description:
This issue will only come up when Spark upgrades its Parquet dependency to the
latest
[
https://issues.apache.org/jira/browse/SPARK-26874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-26874:
---
Priority: Critical (was: Major)
> When we upgrade Parquet to 1.11+, Spark can erroneously write
[
https://issues.apache.org/jira/browse/SPARK-26874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-26874:
---
Description:
This issue will only come up when Spark upgrades its Parquet dependency to the
[
https://issues.apache.org/jira/browse/SPARK-26874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16767736#comment-16767736
]
Matt Cheah commented on SPARK-26874:
[~rdblue] [~cloud_fan] - was wondering if you had any thoughts
Matt Cheah created SPARK-26874:
--
Summary: When we upgrade Parquet to 1.11+, Spark can erroneously
write empty pages
Key: SPARK-26874
URL: https://issues.apache.org/jira/browse/SPARK-26874
Project: Spark
[
https://issues.apache.org/jira/browse/SPARK-26625?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-26625.
Resolution: Fixed
Fix Version/s: 3.0.0
> spark.redaction.regex should include oauthToken
>
[
https://issues.apache.org/jira/browse/SPARK-25877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-25877.
Resolution: Fixed
Fix Version/s: 3.0.0
> Put all feature-related code in the feature step
Matt Cheah created SPARK-26301:
--
Summary: Consider switching from putting secret in environment
variable directly to using secret reference
Key: SPARK-26301
URL: https://issues.apache.org/jira/browse/SPARK-26301
[
https://issues.apache.org/jira/browse/SPARK-26194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-26194.
Resolution: Fixed
Fix Version/s: 3.0.0
> Support automatic spark.authenticate secret in
[
https://issues.apache.org/jira/browse/SPARK-26239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16707962#comment-16707962
]
Matt Cheah commented on SPARK-26239:
It could work in client mode but is less useful there overall
[
https://issues.apache.org/jira/browse/SPARK-25876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-25876.
Resolution: Fixed
Fix Version/s: 3.0.0
> Simplify configuration types in k8s backend
>
[
https://issues.apache.org/jira/browse/SPARK-26239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705273#comment-16705273
]
Matt Cheah edited comment on SPARK-26239 at 11/30/18 8:59 PM:
--
Could we add
[
https://issues.apache.org/jira/browse/SPARK-26239?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16705273#comment-16705273
]
Matt Cheah commented on SPARK-26239:
Would a simple addition just to point to file paths for the
[
https://issues.apache.org/jira/browse/SPARK-25957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-25957.
Resolution: Fixed
Fix Version/s: 3.0.0
> Skip building spark-r docker image if spark
[
https://issues.apache.org/jira/browse/SPARK-26078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-26078:
---
Labels: Correctness (was: )
> WHERE .. IN fails to filter rows when used in combination with UNION
[
https://issues.apache.org/jira/browse/SPARK-26078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-26078:
---
Labels: Correctness correctness (was: Correctness)
> WHERE .. IN fails to filter rows when used in
[
https://issues.apache.org/jira/browse/SPARK-26078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-26078:
---
Target Version/s: 2.4.1, 3.0.0
> WHERE .. IN fails to filter rows when used in combination with
[
https://issues.apache.org/jira/browse/SPARK-26078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-26078:
---
Labels: correctness (was: Correctness correctness)
> WHERE .. IN fails to filter rows when used in
[
https://issues.apache.org/jira/browse/SPARK-26078?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-26078:
---
Priority: Blocker (was: Major)
> WHERE .. IN fails to filter rows when used in combination with
[
https://issues.apache.org/jira/browse/SPARK-26078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16688460#comment-16688460
]
Matt Cheah commented on SPARK-26078:
As per other correctness issues we have seen as of late and as
[
https://issues.apache.org/jira/browse/SPARK-25821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685705#comment-16685705
]
Matt Cheah commented on SPARK-25821:
Should we call this out in release notes? It seems pretty
[
https://issues.apache.org/jira/browse/SPARK-25862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-25862:
---
Labels: release-notes (was: )
> Remove rangeBetween APIs introduced in SPARK-21608
>
[
https://issues.apache.org/jira/browse/SPARK-25862?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16685704#comment-16685704
]
Matt Cheah commented on SPARK-25862:
Adding release-notes label because this seems like a breaking
[
https://issues.apache.org/jira/browse/SPARK-25908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-25908:
---
Labels: release-notes (was: )
> Remove old deprecated items in Spark 3
>
[
https://issues.apache.org/jira/browse/SPARK-24834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16680170#comment-16680170
]
Matt Cheah commented on SPARK-24834:
[~srowen] - I know this is an old ticket but I wanted to
[
https://issues.apache.org/jira/browse/SPARK-25875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-25875.
Resolution: Fixed
> Merge code to set up driver features for different languages
>
[
https://issues.apache.org/jira/browse/SPARK-25809?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-25809.
Resolution: Fixed
> Support additional K8S cluster types for integration tests
>
[
https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-24434.
Resolution: Fixed
> Support user-specified driver and executor pod templates
>
[
https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16650524#comment-16650524
]
Matt Cheah commented on SPARK-18278:
The fork is no longer being maintained, because Kubernetes
[
https://issues.apache.org/jira/browse/SPARK-18278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16648208#comment-16648208
]
Matt Cheah commented on SPARK-18278:
[~liushaohui] a significantly large prerequisite for dynamic
[
https://issues.apache.org/jira/browse/SPARK-23429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-23429.
Resolution: Fixed
> Add executor memory metrics to heartbeat and expose in executors REST API
>
[
https://issues.apache.org/jira/browse/SPARK-25262?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16606522#comment-16606522
]
Matt Cheah commented on SPARK-25262:
For [https://github.com/apache/spark/pull/22323] we allow using
[
https://issues.apache.org/jira/browse/SPARK-25222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-25222.
Resolution: Fixed
> Spark on Kubernetes Pod Watcher dumps raw container status
>
[
https://issues.apache.org/jira/browse/SPARK-25299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16603807#comment-16603807
]
Matt Cheah commented on SPARK-25299:
(Changed the title to "remote storage" for a little more
[
https://issues.apache.org/jira/browse/SPARK-25299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-25299:
---
Summary: Use remote storage for persisting shuffle data (was: Use
distributed storage for
[
https://issues.apache.org/jira/browse/SPARK-24434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599697#comment-16599697
]
Matt Cheah commented on SPARK-24434:
Everyone, thank you for your contribution to this discussion.
[
https://issues.apache.org/jira/browse/SPARK-25299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599428#comment-16599428
]
Matt Cheah commented on SPARK-25299:
Note that SPARK-1529 was a much earlier feature request that
[
https://issues.apache.org/jira/browse/SPARK-25299?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16599428#comment-16599428
]
Matt Cheah edited comment on SPARK-25299 at 9/1/18 12:27 AM:
-
Note that
Matt Cheah created SPARK-25299:
--
Summary: Use distributed storage for persisting shuffle data
Key: SPARK-25299
URL: https://issues.apache.org/jira/browse/SPARK-25299
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-25264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-25264.
Resolution: Fixed
Fix Version/s: 2.4.0
> Fix comma-delineated arguments passed into
Matt Cheah created SPARK-25152:
--
Summary: Enable Spark on Kubernetes R Integration Tests
Key: SPARK-25152
URL: https://issues.apache.org/jira/browse/SPARK-25152
Project: Spark
Issue Type: Test
[
https://issues.apache.org/jira/browse/SPARK-24433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-24433:
---
Fix Version/s: 2.4.0
> Add Spark R support
> ---
>
> Key:
[
https://issues.apache.org/jira/browse/SPARK-24433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-24433.
Resolution: Fixed
> Add Spark R support
> ---
>
> Key:
[
https://issues.apache.org/jira/browse/SPARK-24960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-24960.
Resolution: Fixed
Fix Version/s: 2.4.0
> k8s: explicitly expose ports on driver container
[
https://issues.apache.org/jira/browse/SPARK-24963?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-24963.
Resolution: Fixed
Fix Version/s: 2.4.0
> Integration tests will fail if they run in a
[
https://issues.apache.org/jira/browse/SPARK-23146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-23146.
Resolution: Fixed
Fix Version/s: 2.4.0
> Support client mode for Kubernetes cluster
[
https://issues.apache.org/jira/browse/SPARK-24825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16547137#comment-16547137
]
Matt Cheah commented on SPARK-24825:
We're looking into this now, this particular phase was built
[
https://issues.apache.org/jira/browse/SPARK-24825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-24825:
---
Issue Type: Bug (was: Improvement)
> [K8S][TEST] Kubernetes integration tests don't trace the
Matt Cheah created SPARK-24825:
--
Summary: [K8S][TEST] Kubernetes integration tests don't trace the
maven project dependency structure
Key: SPARK-24825
URL: https://issues.apache.org/jira/browse/SPARK-24825
[
https://issues.apache.org/jira/browse/SPARK-24825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah updated SPARK-24825:
---
Priority: Critical (was: Major)
> [K8S][TEST] Kubernetes integration tests don't trace the maven
[
https://issues.apache.org/jira/browse/SPARK-24683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-24683.
Resolution: Fixed
Fix Version/s: 2.4.0
> SparkLauncher.NO_RESOURCE doesn't work with Java
Matt Cheah created SPARK-24683:
--
Summary: SparkLauncher.NO_RESOURCE doesn't work with Java
applications
Key: SPARK-24683
URL: https://issues.apache.org/jira/browse/SPARK-24683
Project: Spark
Matt Cheah created SPARK-24655:
--
Summary: [K8S] Custom Docker Image Expectations and Documentation
Key: SPARK-24655
URL: https://issues.apache.org/jira/browse/SPARK-24655
Project: Spark
Issue
[
https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516368#comment-16516368
]
Matt Cheah commented on SPARK-24248:
I've summarized what we ended up going with after some
[
https://issues.apache.org/jira/browse/SPARK-24584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516300#comment-16516300
]
Matt Cheah commented on SPARK-24584:
Related to https://issues.apache.org/jira/browse/SPARK-24248
>
Matt Cheah created SPARK-24584:
--
Summary: [K8s] More efficient storage of executor pod state
Key: SPARK-24584
URL: https://issues.apache.org/jira/browse/SPARK-24584
Project: Spark
Issue Type:
[
https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-24248.
Resolution: Fixed
Fix Version/s: 2.4.0
> [K8S] Use the Kubernetes cluster as the backing
[
https://issues.apache.org/jira/browse/SPARK-23010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-23010.
Resolution: Fixed
Fix Version/s: 2.4.0
Some tests are missing but we got a basic set of
[
https://issues.apache.org/jira/browse/SPARK-23984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Cheah resolved SPARK-23984.
Resolution: Fixed
Fix Version/s: 2.4.0
> PySpark Bindings for K8S
>
[
https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471295#comment-16471295
]
Matt Cheah commented on SPARK-24248:
I see - I suppose if the watch connection drops, we should try
[
https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471280#comment-16471280
]
Matt Cheah edited comment on SPARK-24248 at 5/10/18 11:18 PM:
--
I thought
[
https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471280#comment-16471280
]
Matt Cheah commented on SPARK-24248:
I thought about it a bit more, and believe that we can do most
[
https://issues.apache.org/jira/browse/SPARK-24248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471206#comment-16471206
]
Matt Cheah commented on SPARK-24248:
[~foxish] [~liyinan926] curious as to what you think about this
Matt Cheah created SPARK-24248:
--
Summary: [K8S] Use the Kubernetes cluster as the backing store for
the state of pods
Key: SPARK-24248
URL: https://issues.apache.org/jira/browse/SPARK-24248
Project:
Matt Cheah created SPARK-24247:
--
Summary: [K8S] currentNodeToLocalTaskCount is unused in
KubernetesClusterSchedulerBackend
Key: SPARK-24247
URL: https://issues.apache.org/jira/browse/SPARK-24247
[
https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464467#comment-16464467
]
Matt Cheah commented on SPARK-24135:
Put up the PR< see above - created a separate setting for this
[
https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462106#comment-16462106
]
Matt Cheah edited comment on SPARK-24135 at 5/3/18 8:35 AM:
Not necessarily -
[
https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16462106#comment-16462106
]
Matt Cheah commented on SPARK-24135:
Not necessarily - if the pods fail to start up, we should retry
[
https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461188#comment-16461188
]
Matt Cheah edited comment on SPARK-24135 at 5/2/18 3:37 PM:
{quote}Restarting
[
https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460047#comment-16460047
]
Matt Cheah edited comment on SPARK-24135 at 5/2/18 3:37 PM:
{quote}But I'm
[
https://issues.apache.org/jira/browse/SPARK-24135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16461188#comment-16461188
]
Matt Cheah commented on SPARK-24135:
> Restarting seems like it would eventually be limited by the
1 - 100 of 270 matches
Mail list logo
