[jira] [Assigned] (SPARK-35917) Disable push-based shuffle until the feature is complete
[
https://issues.apache.org/jira/browse/SPARK-35917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan reassigned SPARK-35917:
---
Assignee: Mridul Muralidharan
> Disable push-based shuffle until the feature is complete
>
>
> Key: SPARK-35917
> URL: https://issues.apache.org/jira/browse/SPARK-35917
> Project: Spark
> Issue Type: Sub-task
> Components: Shuffle, Spark Core
>Affects Versions: 3.1.0
>Reporter: Chandni Singh
>Assignee: Mridul Muralidharan
>Priority: Major
>
> Push-based shuffle is partially merged in apache master but some of the tasks
> are still incomplete. Since 3.2 is going to cut soon, we will not be able to
> get the pending tasks reviewed and merged. Few of the pending tasks make
> protocol changes to the push-based shuffle protocols, so we would like to
> prevent users from enabling push-based shuffle both on the client and the
> server until push-based shuffle implementation is complete.
> We can prevent push-based shuffle to be used by throwing
> {{UnsupportedOperationException}} (or something like that) both on the client
> and the server when the user tries to enable it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-35917) Disable push-based shuffle until the feature is complete
[
https://issues.apache.org/jira/browse/SPARK-35917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan resolved SPARK-35917.
-
Resolution: Won't Fix
> Disable push-based shuffle until the feature is complete
>
>
> Key: SPARK-35917
> URL: https://issues.apache.org/jira/browse/SPARK-35917
> Project: Spark
> Issue Type: Sub-task
> Components: Shuffle, Spark Core
>Affects Versions: 3.1.0
>Reporter: Chandni Singh
>Priority: Major
>
> Push-based shuffle is partially merged in apache master but some of the tasks
> are still incomplete. Since 3.2 is going to cut soon, we will not be able to
> get the pending tasks reviewed and merged. Few of the pending tasks make
> protocol changes to the push-based shuffle protocols, so we would like to
> prevent users from enabling push-based shuffle both on the client and the
> server until push-based shuffle implementation is complete.
> We can prevent push-based shuffle to be used by throwing
> {{UnsupportedOperationException}} (or something like that) both on the client
> and the server when the user tries to enable it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-35917) Disable push-based shuffle until the feature is complete
[
https://issues.apache.org/jira/browse/SPARK-35917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17391313#comment-17391313
]
Mridul Muralidharan commented on SPARK-35917:
-
Closing this Jira - as push based shuffle has been merged.
Thanks for the ping [~Gengliang.Wang] !
> Disable push-based shuffle until the feature is complete
>
>
> Key: SPARK-35917
> URL: https://issues.apache.org/jira/browse/SPARK-35917
> Project: Spark
> Issue Type: Sub-task
> Components: Shuffle, Spark Core
>Affects Versions: 3.1.0
>Reporter: Chandni Singh
>Priority: Major
>
> Push-based shuffle is partially merged in apache master but some of the tasks
> are still incomplete. Since 3.2 is going to cut soon, we will not be able to
> get the pending tasks reviewed and merged. Few of the pending tasks make
> protocol changes to the push-based shuffle protocols, so we would like to
> prevent users from enabling push-based shuffle both on the client and the
> server until push-based shuffle implementation is complete.
> We can prevent push-based shuffle to be used by throwing
> {{UnsupportedOperationException}} (or something like that) both on the client
> and the server when the user tries to enable it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-36266) Rename classes in shuffle RPC used for block push operations
[ https://issues.apache.org/jira/browse/SPARK-36266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-36266. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 33340 [https://github.com/apache/spark/pull/33340] > Rename classes in shuffle RPC used for block push operations > > > Key: SPARK-36266 > URL: https://issues.apache.org/jira/browse/SPARK-36266 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Priority: Major > Fix For: 3.2.0 > > > In the current implementation of push-based shuffle, we are reusing certain > code between both block fetch and block push. > This is generally good except that certain classes that are meant to be used > for both block fetch and block push now have names that indicate they are > only for block fetches, which is confusing. > This ticket renames these classes to be more generic to be reused across both > block fetch and block push. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-35546) Enable push-based shuffle when multiple app attempts are enabled and manage concurrent access to the state in a better way
[ https://issues.apache.org/jira/browse/SPARK-35546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35546: --- Assignee: Ye Zhou > Enable push-based shuffle when multiple app attempts are enabled and manage > concurrent access to the state in a better way > --- > > Key: SPARK-35546 > URL: https://issues.apache.org/jira/browse/SPARK-35546 > Project: Spark > Issue Type: Sub-task > Components: Shuffle >Affects Versions: 3.1.0 >Reporter: Ye Zhou >Assignee: Ye Zhou >Priority: Major > Fix For: 3.2.0 > > > In the current implementation of RemoteBlockPushResolver, two > ConcurrentHashmap are used to store #1 applicationId -> > mergedShuffleLocalDirPath #2 applicationId+attemptId+shuffleID -> > mergedShuffleParitionInfo. As there are four types of messages: > ExecutorRegister, PushBlocks, FinalizeShuffleMerge and ApplicationRemove, > will trigger different types of operations within these two hashmaps, it is > required to maintain strong consistency about the informations stored in > these two hashmaps. Otherwise, either there will be data > corruption/correctness issues or memory leak in shuffle server. > We should come up with systematic way to resolve this, other than spot fixing > the potential issues. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-35276) Write checksum files for shuffle
[ https://issues.apache.org/jira/browse/SPARK-35276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35276: --- Assignee: wuyi > Write checksum files for shuffle > - > > Key: SPARK-35276 > URL: https://issues.apache.org/jira/browse/SPARK-35276 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.2.0 >Reporter: wuyi >Assignee: wuyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-35276) Write checksum files for shuffle
[ https://issues.apache.org/jira/browse/SPARK-35276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35276. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32401 [https://github.com/apache/spark/pull/32401] > Write checksum files for shuffle > - > > Key: SPARK-35276 > URL: https://issues.apache.org/jira/browse/SPARK-35276 > Project: Spark > Issue Type: Sub-task > Components: Spark Core >Affects Versions: 3.2.0 >Reporter: wuyi >Assignee: wuyi >Priority: Major > Fix For: 3.2.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-32922) Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures
[ https://issues.apache.org/jira/browse/SPARK-32922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32922. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32140 [https://github.com/apache/spark/pull/32140] > Add support for ShuffleBlockFetcherIterator to read from merged shuffle > partitions and to fallback to original shuffle blocks if encountering failures > -- > > Key: SPARK-32922 > URL: https://issues.apache.org/jira/browse/SPARK-32922 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Chandni Singh >Priority: Major > Fix For: 3.2.0 > > > With the extended MapOutputTracker, the reducers can now get the task input > data from the merged shuffle partitions for more efficient shuffle data > fetch. The reducers should also be able to fallback to fetching the original > unmarked blocks if it encounters failures when fetching the merged shuffle > partitions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-32922) Add support for ShuffleBlockFetcherIterator to read from merged shuffle partitions and to fallback to original shuffle blocks if encountering failures
[ https://issues.apache.org/jira/browse/SPARK-32922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32922: --- Assignee: Chandni Singh > Add support for ShuffleBlockFetcherIterator to read from merged shuffle > partitions and to fallback to original shuffle blocks if encountering failures > -- > > Key: SPARK-32922 > URL: https://issues.apache.org/jira/browse/SPARK-32922 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Chandni Singh >Priority: Major > > With the extended MapOutputTracker, the reducers can now get the task input > data from the merged shuffle partitions for more efficient shuffle data > fetch. The reducers should also be able to fallback to fetching the original > unmarked blocks if it encounters failures when fetching the merged shuffle > partitions. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-35258) Enhance ESS ExternalBlockHandler with additional block rate-based metrics and histograms
[
https://issues.apache.org/jira/browse/SPARK-35258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan resolved SPARK-35258.
-
Fix Version/s: 3.2.0
Resolution: Fixed
Issue resolved by pull request 32388
[https://github.com/apache/spark/pull/32388]
> Enhance ESS ExternalBlockHandler with additional block rate-based metrics and
> histograms
>
>
> Key: SPARK-35258
> URL: https://issues.apache.org/jira/browse/SPARK-35258
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle, YARN
>Affects Versions: 3.1.1
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Fix For: 3.2.0
>
>
> Today the {{ExternalBlockHandler}} component of ESS exposes some useful
> metrics, but is lacking around metrics for the rate of block transfers. We
> have {{blockTransferRateBytes}} to tell us the rate of _bytes_, but no metric
> to tell us the rate of _blocks_, which is especially relevant when running
> the ESS on HDDs that are sensitive to random reads. Many small block
> transfers can have a negative impact on performance, but won't show up as a
> spike in {{blockTransferRateBytes}} since the sizes are small.
> We can also enhance {{YarnShuffleServiceMetrics}} to expose histogram-style
> metrics from the {{Timer}} instances within {{ExternalBlockHandler}} -- today
> it is only exposing the count and rate, but not timing information from the
> {{Snapshot}}.
> These two changes can make it easier to monitor the health of the ESS.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-35258) Enhance ESS ExternalBlockHandler with additional block rate-based metrics and histograms
[
https://issues.apache.org/jira/browse/SPARK-35258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan reassigned SPARK-35258:
---
Assignee: Erik Krogen
> Enhance ESS ExternalBlockHandler with additional block rate-based metrics and
> histograms
>
>
> Key: SPARK-35258
> URL: https://issues.apache.org/jira/browse/SPARK-35258
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle, YARN
>Affects Versions: 3.1.1
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
>
> Today the {{ExternalBlockHandler}} component of ESS exposes some useful
> metrics, but is lacking around metrics for the rate of block transfers. We
> have {{blockTransferRateBytes}} to tell us the rate of _bytes_, but no metric
> to tell us the rate of _blocks_, which is especially relevant when running
> the ESS on HDDs that are sensitive to random reads. Many small block
> transfers can have a negative impact on performance, but won't show up as a
> spike in {{blockTransferRateBytes}} since the sizes are small.
> We can also enhance {{YarnShuffleServiceMetrics}} to expose histogram-style
> metrics from the {{Timer}} instances within {{ExternalBlockHandler}} -- today
> it is only exposing the count and rate, but not timing information from the
> {{Snapshot}}.
> These two changes can make it easier to monitor the health of the ESS.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-35836) Remove reference to spark.shuffle.push.based.enabled in ShuffleBlockPusherSuite
[
https://issues.apache.org/jira/browse/SPARK-35836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan resolved SPARK-35836.
-
Fix Version/s: 3.2.0
Resolution: Fixed
Issue resolved by pull request 32992
[https://github.com/apache/spark/pull/32992]
> Remove reference to spark.shuffle.push.based.enabled in
> ShuffleBlockPusherSuite
> ---
>
> Key: SPARK-35836
> URL: https://issues.apache.org/jira/browse/SPARK-35836
> Project: Spark
> Issue Type: Bug
> Components: Shuffle, Spark Core
>Affects Versions: 3.1.0
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Trivial
> Fix For: 3.2.0
>
>
> The test suite for ShuffleBlockPusherSuite was added with SPARK-32917 and in
> this suite, the configuration for push-based shuffle is incorrectly
> referenced as {{spark.shuffle.push.based.enabled}}. We need to remove this
> config from here.
> {{ShuffleBlockPusher}} is created only when push based shuffle is enabled and
> this suite is for {{ShuffleBlockPusher}}, so no other change is required.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-35836) Remove reference to spark.shuffle.push.based.enabled in ShuffleBlockPusherSuite
[
https://issues.apache.org/jira/browse/SPARK-35836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan reassigned SPARK-35836:
---
Assignee: Chandni Singh
> Remove reference to spark.shuffle.push.based.enabled in
> ShuffleBlockPusherSuite
> ---
>
> Key: SPARK-35836
> URL: https://issues.apache.org/jira/browse/SPARK-35836
> Project: Spark
> Issue Type: Bug
> Components: Shuffle, Spark Core
>Affects Versions: 3.1.0
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Trivial
>
> The test suite for ShuffleBlockPusherSuite was added with SPARK-32917 and in
> this suite, the configuration for push-based shuffle is incorrectly
> referenced as {{spark.shuffle.push.based.enabled}}. We need to remove this
> config from here.
> {{ShuffleBlockPusher}} is created only when push based shuffle is enabled and
> this suite is for {{ShuffleBlockPusher}}, so no other change is required.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-35671) Add Support in the ESS to serve merged shuffle block meta and data to executors
[ https://issues.apache.org/jira/browse/SPARK-35671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35671: --- Assignee: Chandni Singh > Add Support in the ESS to serve merged shuffle block meta and data to > executors > --- > > Key: SPARK-35671 > URL: https://issues.apache.org/jira/browse/SPARK-35671 > Project: Spark > Issue Type: Sub-task > Components: Shuffle >Affects Versions: 3.1.0 >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > > With push-based shuffle enabled, the reducers send 2 different requests to > the ESS: > 1. Request to fetch the metadata of the merged shuffle block. > 2. Requests to fetch the data of the merged shuffle block in chunks which > are by default 2MB in size. > The ESS should be able to serve these requests and this Jira targets all the > changes in the network-common and network-shuffle modules to be able to > support this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-35671) Add Support in the ESS to serve merged shuffle block meta and data to executors
[ https://issues.apache.org/jira/browse/SPARK-35671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35671. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32811 [https://github.com/apache/spark/pull/32811] > Add Support in the ESS to serve merged shuffle block meta and data to > executors > --- > > Key: SPARK-35671 > URL: https://issues.apache.org/jira/browse/SPARK-35671 > Project: Spark > Issue Type: Sub-task > Components: Shuffle >Affects Versions: 3.1.0 >Reporter: Chandni Singh >Assignee: Chandni Singh >Priority: Major > Fix For: 3.2.0 > > > With push-based shuffle enabled, the reducers send 2 different requests to > the ESS: > 1. Request to fetch the metadata of the merged shuffle block. > 2. Requests to fetch the data of the merged shuffle block in chunks which > are by default 2MB in size. > The ESS should be able to serve these requests and this Jira targets all the > changes in the network-common and network-shuffle modules to be able to > support this. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-34898) Send ExecutorMetricsUpdate EventLog appropriately
[
https://issues.apache.org/jira/browse/SPARK-34898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan resolved SPARK-34898.
-
Fix Version/s: 3.2.0
Resolution: Fixed
Issue resolved by pull request 31992
[https://github.com/apache/spark/pull/31992]
> Send ExecutorMetricsUpdate EventLog appropriately
> -
>
> Key: SPARK-34898
> URL: https://issues.apache.org/jira/browse/SPARK-34898
> Project: Spark
> Issue Type: Sub-task
> Components: Spark Core
>Affects Versions: 3.2.0
>Reporter: angerszhu
>Assignee: Apache Spark
>Priority: Major
> Fix For: 3.2.0
>
>
> In current EventLoggingListener, we won't write
> SparkListenerExecutorMetricsUpdate message at all
> {code:java}
> override def onExecutorMetricsUpdate(event:
> SparkListenerExecutorMetricsUpdate): Unit = {
> if (shouldLogStageExecutorMetrics) {
> event.executorUpdates.foreach { case (stageKey1, newPeaks) =>
> liveStageExecutorMetrics.foreach { case (stageKey2, metricsPerExecutor)
> =>
> // If the update came from the driver, stageKey1 will be the dummy
> key (-1, -1),
> // so record those peaks for all active stages.
> // Otherwise, record the peaks for the matching stage.
> if (stageKey1 == DRIVER_STAGE_KEY || stageKey1 == stageKey2) {
> val metrics = metricsPerExecutor.getOrElseUpdate(
> event.execId, new ExecutorMetrics())
> metrics.compareAndUpdatePeakValues(newPeaks)
> }
> }
> }
> }
> }
> {code}
> It causes this effect that we can't get driver peakMemoryMetrics in SHS. We
> can get executor's since it will update with TaskEnd events.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-35613) Cache commonly occurring strings from SQLMetrics, JsonProtocol and AccumulatorV2
[ https://issues.apache.org/jira/browse/SPARK-35613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35613. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32754 [https://github.com/apache/spark/pull/32754] > Cache commonly occurring strings from SQLMetrics, JsonProtocol and > AccumulatorV2 > > > Key: SPARK-35613 > URL: https://issues.apache.org/jira/browse/SPARK-35613 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 3.2.0 >Reporter: Venkata krishnan Sowrirajan >Assignee: Venkata krishnan Sowrirajan >Priority: Major > Fix For: 3.2.0 > > > SQLMetrics allows creation of different metrics like sizing, timing metrics > etc. some of those names for the metric can be duplicated and along with that > the `Some` wrapper objects adds additional memory overhead. In our internal > platform, this has caused huge memory usage on driver causing Full GC to kick > in every so often. > cc [~mridulm80] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-35613) Cache commonly occurring strings from SQLMetrics, JsonProtocol and AccumulatorV2
[ https://issues.apache.org/jira/browse/SPARK-35613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35613: --- Assignee: Venkata krishnan Sowrirajan > Cache commonly occurring strings from SQLMetrics, JsonProtocol and > AccumulatorV2 > > > Key: SPARK-35613 > URL: https://issues.apache.org/jira/browse/SPARK-35613 > Project: Spark > Issue Type: Bug > Components: Spark Core, SQL >Affects Versions: 3.2.0 >Reporter: Venkata krishnan Sowrirajan >Assignee: Venkata krishnan Sowrirajan >Priority: Major > > SQLMetrics allows creation of different metrics like sizing, timing metrics > etc. some of those names for the metric can be duplicated and along with that > the `Some` wrapper objects adds additional memory overhead. In our internal > platform, this has caused huge memory usage on driver causing Full GC to kick > in every so often. > cc [~mridulm80] -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-33350) Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data
[
https://issues.apache.org/jira/browse/SPARK-33350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan reassigned SPARK-33350:
---
Assignee: Ye Zhou
> Add support to DiskBlockManager to create merge directory and to get the
> local shuffle merged data
> --
>
> Key: SPARK-33350
> URL: https://issues.apache.org/jira/browse/SPARK-33350
> Project: Spark
> Issue Type: Sub-task
> Components: Shuffle
>Affects Versions: 3.1.0
>Reporter: Chandni Singh
>Assignee: Ye Zhou
>Priority: Major
>
> DiskBlockManager should be able to create the {{merge_manager}} directory,
> where the push-based merged shuffle files are written and also create
> sub-dirs under it.
> It should also be able to serve the local merged shuffle data/index/meta
> files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-33350) Add support to DiskBlockManager to create merge directory and to get the local shuffle merged data
[
https://issues.apache.org/jira/browse/SPARK-33350?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan resolved SPARK-33350.
-
Fix Version/s: 3.2.0
Resolution: Fixed
Issue resolved by pull request 32007
[https://github.com/apache/spark/pull/32007]
> Add support to DiskBlockManager to create merge directory and to get the
> local shuffle merged data
> --
>
> Key: SPARK-33350
> URL: https://issues.apache.org/jira/browse/SPARK-33350
> Project: Spark
> Issue Type: Sub-task
> Components: Shuffle
>Affects Versions: 3.1.0
>Reporter: Chandni Singh
>Assignee: Ye Zhou
>Priority: Major
> Fix For: 3.2.0
>
>
> DiskBlockManager should be able to create the {{merge_manager}} directory,
> where the push-based merged shuffle files are written and also create
> sub-dirs under it.
> It should also be able to serve the local merged shuffle data/index/meta
> files.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage
[ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32920. - Resolution: Fixed > Add support in Spark driver to coordinate the finalization of the push/merge > phase in push-based shuffle for a given shuffle and the initiation of the > reduce stage > --- > > Key: SPARK-32920 > URL: https://issues.apache.org/jira/browse/SPARK-32920 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Venkata krishnan Sowrirajan >Priority: Major > Fix For: 3.2.0 > > > With push-based shuffle, we are currently decoupling map task executions from > the shuffle block push process. Thus, when all map tasks finish, we might > want to wait for some small extra time to allow more shuffle blocks to get > pushed and merged. This requires some extra coordination in the Spark driver > when it transitions from a shuffle map stage to the corresponding reduce > stage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage
[ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-32920: Fix Version/s: 3.2.0 > Add support in Spark driver to coordinate the finalization of the push/merge > phase in push-based shuffle for a given shuffle and the initiation of the > reduce stage > --- > > Key: SPARK-32920 > URL: https://issues.apache.org/jira/browse/SPARK-32920 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Venkata krishnan Sowrirajan >Priority: Major > Fix For: 3.2.0 > > > With push-based shuffle, we are currently decoupling map task executions from > the shuffle block push process. Thus, when all map tasks finish, we might > want to wait for some small extra time to allow more shuffle blocks to get > pushed and merged. This requires some extra coordination in the Spark driver > when it transitions from a shuffle map stage to the corresponding reduce > stage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-32920) Add support in Spark driver to coordinate the finalization of the push/merge phase in push-based shuffle for a given shuffle and the initiation of the reduce stage
[ https://issues.apache.org/jira/browse/SPARK-32920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32920: --- Assignee: Venkata krishnan Sowrirajan > Add support in Spark driver to coordinate the finalization of the push/merge > phase in push-based shuffle for a given shuffle and the initiation of the > reduce stage > --- > > Key: SPARK-32920 > URL: https://issues.apache.org/jira/browse/SPARK-32920 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Venkata krishnan Sowrirajan >Priority: Major > > With push-based shuffle, we are currently decoupling map task executions from > the shuffle block push process. Thus, when all map tasks finish, we might > want to wait for some small extra time to allow more shuffle blocks to get > pushed and merged. This requires some extra coordination in the Spark driver > when it transitions from a shuffle map stage to the corresponding reduce > stage. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-35263) Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code
[
https://issues.apache.org/jira/browse/SPARK-35263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan resolved SPARK-35263.
-
Fix Version/s: 3.2.0
Resolution: Fixed
Issue resolved by pull request 32389
[https://github.com/apache/spark/pull/32389]
> Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code
> ---
>
> Key: SPARK-35263
> URL: https://issues.apache.org/jira/browse/SPARK-35263
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle, Tests
>Affects Versions: 3.1.1
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Fix For: 3.2.0
>
>
> {{ShuffleFetcherBlockIteratorSuite}} has tons of duplicate code, like:
> {code}
> val iterator = new ShuffleBlockFetcherIterator(
> taskContext,
> transfer,
> blockManager,
> blocksByAddress,
> (_, in) => in,
> 48 * 1024 * 1024,
> Int.MaxValue,
> Int.MaxValue,
> Int.MaxValue,
> true,
> false,
> metrics,
> false)
> {code}
> It's challenging to tell what the interesting parts are vs. what is just
> being set to some default/unused value.
> Similarly but not as bad, there are 10 calls like:
> {code}
> verify(transfer, times(1)).fetchBlocks(any(), any(), any(), any(), any(),
> any())
> {code}
> and 7 like
> {code}
> when(transfer.fetchBlocks(any(), any(), any(), any(), any(),
> any())).thenAnswer ...
> {code}
> This can result in about 10% reduction in both lines and characters in the
> file:
> {code}
> # Before
> > wc
> > core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala
> 10633950 43201
> core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala
> # After
> > wc
> > core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala
> 9283609 39053
> core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala
> {code}
> It also helps readability:
> {code}
> val iterator = createShuffleBlockIteratorWithDefaults(
> transfer,
> blocksByAddress,
> maxBytesInFlight = 1000L
> )
> {code}
> Now I can clearly tell that {{maxBytesInFlight}} is the main parameter we're
> interested in here.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-35263) Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code
[
https://issues.apache.org/jira/browse/SPARK-35263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan reassigned SPARK-35263:
---
Assignee: Erik Krogen
> Refactor ShuffleBlockFetcherIteratorSuite to reduce duplicated code
> ---
>
> Key: SPARK-35263
> URL: https://issues.apache.org/jira/browse/SPARK-35263
> Project: Spark
> Issue Type: Improvement
> Components: Shuffle, Tests
>Affects Versions: 3.1.1
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
>
> {{ShuffleFetcherBlockIteratorSuite}} has tons of duplicate code, like:
> {code}
> val iterator = new ShuffleBlockFetcherIterator(
> taskContext,
> transfer,
> blockManager,
> blocksByAddress,
> (_, in) => in,
> 48 * 1024 * 1024,
> Int.MaxValue,
> Int.MaxValue,
> Int.MaxValue,
> true,
> false,
> metrics,
> false)
> {code}
> It's challenging to tell what the interesting parts are vs. what is just
> being set to some default/unused value.
> Similarly but not as bad, there are 10 calls like:
> {code}
> verify(transfer, times(1)).fetchBlocks(any(), any(), any(), any(), any(),
> any())
> {code}
> and 7 like
> {code}
> when(transfer.fetchBlocks(any(), any(), any(), any(), any(),
> any())).thenAnswer ...
> {code}
> This can result in about 10% reduction in both lines and characters in the
> file:
> {code}
> # Before
> > wc
> > core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala
> 10633950 43201
> core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala
> # After
> > wc
> > core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala
> 9283609 39053
> core/src/test/scala/org/apache/spark/storage/ShuffleBlockFetcherIteratorSuite.scala
> {code}
> It also helps readability:
> {code}
> val iterator = createShuffleBlockIteratorWithDefaults(
> transfer,
> blocksByAddress,
> maxBytesInFlight = 1000L
> )
> {code}
> Now I can clearly tell that {{maxBytesInFlight}} is the main parameter we're
> interested in here.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-32921) Extend MapOutputTracker to support tracking and serving the metadata about each merged shuffle partitions for a given shuffle in push-based shuffle scenario
[ https://issues.apache.org/jira/browse/SPARK-32921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32921. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 30480 [https://github.com/apache/spark/pull/30480] > Extend MapOutputTracker to support tracking and serving the metadata about > each merged shuffle partitions for a given shuffle in push-based shuffle > scenario > > > Key: SPARK-32921 > URL: https://issues.apache.org/jira/browse/SPARK-32921 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Venkata krishnan Sowrirajan >Priority: Major > Fix For: 3.2.0 > > > Similar to MapStatus, which tracks the metadata about each map task's shuffle > output, we also need to track the metadata about each merged shuffle > partition with push-based shuffle. We currently term this as MergeStatus. > Since MergeStatus tracks metadata from the perspective of reducer tasks, it's > not efficient to break up the metadata tracked in a MergeStatus and spread it > across multiple MapStatus. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-32921) Extend MapOutputTracker to support tracking and serving the metadata about each merged shuffle partitions for a given shuffle in push-based shuffle scenario
[ https://issues.apache.org/jira/browse/SPARK-32921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32921: --- Assignee: Venkata krishnan Sowrirajan > Extend MapOutputTracker to support tracking and serving the metadata about > each merged shuffle partitions for a given shuffle in push-based shuffle > scenario > > > Key: SPARK-32921 > URL: https://issues.apache.org/jira/browse/SPARK-32921 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Venkata krishnan Sowrirajan >Priority: Major > > Similar to MapStatus, which tracks the metadata about each map task's shuffle > output, we also need to track the metadata about each merged shuffle > partition with push-based shuffle. We currently term this as MergeStatus. > Since MergeStatus tracks metadata from the perspective of reducer tasks, it's > not efficient to break up the metadata tracked in a MergeStatus and spread it > across multiple MapStatus. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-35049) Remove unused MapOutputTracker in BlockStoreShuffleReader
[ https://issues.apache.org/jira/browse/SPARK-35049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-35049: --- Assignee: angerszhu > Remove unused MapOutputTracker in BlockStoreShuffleReader > - > > Key: SPARK-35049 > URL: https://issues.apache.org/jira/browse/SPARK-35049 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.2.0 >Reporter: angerszhu >Assignee: angerszhu >Priority: Major > > Remove unused MapOutputTracker in BlockStoreShuffleReader > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-35049) Remove unused MapOutputTracker in BlockStoreShuffleReader
[ https://issues.apache.org/jira/browse/SPARK-35049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-35049. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 32148 [https://github.com/apache/spark/pull/32148] > Remove unused MapOutputTracker in BlockStoreShuffleReader > - > > Key: SPARK-35049 > URL: https://issues.apache.org/jira/browse/SPARK-35049 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 3.2.0 >Reporter: angerszhu >Assignee: angerszhu >Priority: Major > Fix For: 3.2.0 > > > Remove unused MapOutputTracker in BlockStoreShuffleReader > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-34949) Executor.reportHeartBeat reregisters blockManager even when Executor is shutting down
[ https://issues.apache.org/jira/browse/SPARK-34949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-34949: --- Assignee: Sumeet > Executor.reportHeartBeat reregisters blockManager even when Executor is > shutting down > - > > Key: SPARK-34949 > URL: https://issues.apache.org/jira/browse/SPARK-34949 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.2.0, 3.1.1 > Environment: Resource Manager: K8s >Reporter: Sumeet >Assignee: Sumeet >Priority: Major > Labels: Executor, heartbeat > Fix For: 3.2.0, 3.1.2 > > > *Problem:* > I was testing Dynamic Allocation on K8s with about 300 executors. While doing > so, when the executors were torn down due to > "spark.dynamicAllocation.executorIdleTimeout", I noticed all the executor > pods being removed from K8s, however, under the "Executors" tab in SparkUI, I > could see some executors listed as alive. > [spark.sparkContext.statusTracker.getExecutorInfos.length|https://github.com/apache/spark/blob/65da9287bc5112564836a555cd2967fc6b05856f/core/src/main/scala/org/apache/spark/SparkStatusTracker.scala#L100] > also returned a value greater than 1. > > *Cause:* > * "CoarseGrainedSchedulerBackend" issues RemoveExecutor on a > "executorEndpoint" and publishes "SparkListenerExecutorRemoved" on the > "listenerBus" > * "CoarseGrainedExecutorBackend" starts the executor shutdown > * "HeartbeatReceiver" picks the "SparkListenerExecutorRemoved" event and > removes the executor from "executorLastSeen" > * In the meantime, the executor reports a Heartbeat. Now "HeartbeatReceiver" > cannot find the "executorId" in "executorLastSeen" and hence responds with > "HeartbeatResponse(reregisterBlockManager = true)" > * The Executor now calls "env.blockManager.reregister()" and reregisters > itself thus creating inconsistency > > *Proposed Solution:* > The "reportHeartBeat" method is not aware of the fact that Executor is > shutting down, it should check "executorShutdown" before reregistering. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-34949) Executor.reportHeartBeat reregisters blockManager even when Executor is shutting down
[ https://issues.apache.org/jira/browse/SPARK-34949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-34949. - Fix Version/s: 3.1.2 3.2.0 Resolution: Fixed Issue resolved by pull request 32043 [https://github.com/apache/spark/pull/32043] > Executor.reportHeartBeat reregisters blockManager even when Executor is > shutting down > - > > Key: SPARK-34949 > URL: https://issues.apache.org/jira/browse/SPARK-34949 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.2.0, 3.1.1 > Environment: Resource Manager: K8s >Reporter: Sumeet >Priority: Major > Labels: Executor, heartbeat > Fix For: 3.2.0, 3.1.2 > > > *Problem:* > I was testing Dynamic Allocation on K8s with about 300 executors. While doing > so, when the executors were torn down due to > "spark.dynamicAllocation.executorIdleTimeout", I noticed all the executor > pods being removed from K8s, however, under the "Executors" tab in SparkUI, I > could see some executors listed as alive. > [spark.sparkContext.statusTracker.getExecutorInfos.length|https://github.com/apache/spark/blob/65da9287bc5112564836a555cd2967fc6b05856f/core/src/main/scala/org/apache/spark/SparkStatusTracker.scala#L100] > also returned a value greater than 1. > > *Cause:* > * "CoarseGrainedSchedulerBackend" issues RemoveExecutor on a > "executorEndpoint" and publishes "SparkListenerExecutorRemoved" on the > "listenerBus" > * "CoarseGrainedExecutorBackend" starts the executor shutdown > * "HeartbeatReceiver" picks the "SparkListenerExecutorRemoved" event and > removes the executor from "executorLastSeen" > * In the meantime, the executor reports a Heartbeat. Now "HeartbeatReceiver" > cannot find the "executorId" in "executorLastSeen" and hence responds with > "HeartbeatResponse(reregisterBlockManager = true)" > * The Executor now calls "env.blockManager.reregister()" and reregisters > itself thus creating inconsistency > > *Proposed Solution:* > The "reportHeartBeat" method is not aware of the fact that Executor is > shutting down, it should check "executorShutdown" before reregistering. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-34840) Fix cases of corruption in merged shuffle blocks that are pushed
[
https://issues.apache.org/jira/browse/SPARK-34840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan resolved SPARK-34840.
-
Fix Version/s: 3.1.2
3.2.0
Resolution: Fixed
Issue resolved by pull request 31934
[https://github.com/apache/spark/pull/31934]
> Fix cases of corruption in merged shuffle blocks that are pushed
>
>
> Key: SPARK-34840
> URL: https://issues.apache.org/jira/browse/SPARK-34840
> Project: Spark
> Issue Type: Bug
> Components: Shuffle
>Affects Versions: 3.1.0
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
> Fix For: 3.2.0, 3.1.2
>
>
> The {{RemoteBlockPushResolver}} which handles the shuffle push blocks and
> merges them was introduced in
> [#30062|https://github.com/apache/spark/pull/30062]. We have identified 2
> scenarios where the merged blocks get corrupted:
> # {{StreamCallback.onFailure()}} is called more than once. Initially we
> assumed that the onFailure callback will be called just once per stream.
> However, we observed that this is called twice when a client connection is
> reset. When the client connection is reset then there are 2 events that get
> triggered in this order.
> * {{exceptionCaught}}. This event is propagated to {{StreamInterceptor}}.
> {{StreamInterceptor.exceptionCaught()}} invokes
> {{callback.onFailure(streamId, cause)}}. This is the first time
> StreamCallback.onFailure() will be invoked.
> * {{channelInactive}}. Since the channel closes, the {{channelInactive}}
> event gets triggered which again is propagated to {{StreamInterceptor}}.
> {{StreamInterceptor.channelInactive()}} invokes
> {{callback.onFailure(streamId, new ClosedChannelException())}}. This is the
> second time StreamCallback.onFailure() will be invoked.
> # The flag {{isWriting}} is set prematurely to true. This introduces an edge
> case where a stream that is trying to merge a duplicate block (created
> because of a speculative task) may interfere with an active stream if the
> duplicate stream fails.
> Also adding additional changes that improve the code.
> # Using positional writes all the time because this simplifies the code and
> with microbenchmarking haven't seen any performance impact.
> # Additional minor changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-34840) Fix cases of corruption in merged shuffle blocks that are pushed
[
https://issues.apache.org/jira/browse/SPARK-34840?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan reassigned SPARK-34840:
---
Assignee: Chandni Singh
> Fix cases of corruption in merged shuffle blocks that are pushed
>
>
> Key: SPARK-34840
> URL: https://issues.apache.org/jira/browse/SPARK-34840
> Project: Spark
> Issue Type: Bug
> Components: Shuffle
>Affects Versions: 3.1.0
>Reporter: Chandni Singh
>Assignee: Chandni Singh
>Priority: Major
>
> The {{RemoteBlockPushResolver}} which handles the shuffle push blocks and
> merges them was introduced in
> [#30062|https://github.com/apache/spark/pull/30062]. We have identified 2
> scenarios where the merged blocks get corrupted:
> # {{StreamCallback.onFailure()}} is called more than once. Initially we
> assumed that the onFailure callback will be called just once per stream.
> However, we observed that this is called twice when a client connection is
> reset. When the client connection is reset then there are 2 events that get
> triggered in this order.
> * {{exceptionCaught}}. This event is propagated to {{StreamInterceptor}}.
> {{StreamInterceptor.exceptionCaught()}} invokes
> {{callback.onFailure(streamId, cause)}}. This is the first time
> StreamCallback.onFailure() will be invoked.
> * {{channelInactive}}. Since the channel closes, the {{channelInactive}}
> event gets triggered which again is propagated to {{StreamInterceptor}}.
> {{StreamInterceptor.channelInactive()}} invokes
> {{callback.onFailure(streamId, new ClosedChannelException())}}. This is the
> second time StreamCallback.onFailure() will be invoked.
> # The flag {{isWriting}} is set prematurely to true. This introduces an edge
> case where a stream that is trying to merge a duplicate block (created
> because of a speculative task) may interfere with an active stream if the
> duplicate stream fails.
> Also adding additional changes that improve the code.
> # Using positional writes all the time because this simplifies the code and
> with microbenchmarking haven't seen any performance impact.
> # Additional minor changes.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-24818) Ensure all the barrier tasks in the same stage are launched together
[ https://issues.apache.org/jira/browse/SPARK-24818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-24818: --- Assignee: wuyi > Ensure all the barrier tasks in the same stage are launched together > > > Key: SPARK-24818 > URL: https://issues.apache.org/jira/browse/SPARK-24818 > Project: Spark > Issue Type: New Feature > Components: Spark Core >Affects Versions: 3.1.0 >Reporter: Xingbo Jiang >Assignee: wuyi >Priority: Major > Fix For: 3.2.0 > > > When some executors/hosts are blacklisted, it may happen that only a part of > the tasks in the same barrier stage can be launched. We shall detect the case > and revert the allocated resource offers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-24818) Ensure all the barrier tasks in the same stage are launched together
[ https://issues.apache.org/jira/browse/SPARK-24818?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-24818. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 30650 [https://github.com/apache/spark/pull/30650] > Ensure all the barrier tasks in the same stage are launched together > > > Key: SPARK-24818 > URL: https://issues.apache.org/jira/browse/SPARK-24818 > Project: Spark > Issue Type: New Feature > Components: Spark Core >Affects Versions: 3.1.0 >Reporter: Xingbo Jiang >Priority: Major > Fix For: 3.2.0 > > > When some executors/hosts are blacklisted, it may happen that only a part of > the tasks in the same barrier stage can be launched. We shall detect the case > and revert the allocated resource offers. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-34069) Kill barrier tasks should respect SPARK_JOB_INTERRUPT_ON_CANCEL
[ https://issues.apache.org/jira/browse/SPARK-34069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-34069: --- Assignee: ulysses you > Kill barrier tasks should respect SPARK_JOB_INTERRUPT_ON_CANCEL > --- > > Key: SPARK-34069 > URL: https://issues.apache.org/jira/browse/SPARK-34069 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.2.0 >Reporter: ulysses you >Assignee: ulysses you >Priority: Major > > We should interrupt task thread if user set local property > `SPARK_JOB_INTERRUPT_ON_CANCEL` to true. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-34069) Kill barrier tasks should respect SPARK_JOB_INTERRUPT_ON_CANCEL
[ https://issues.apache.org/jira/browse/SPARK-34069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-34069. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 31127 [https://github.com/apache/spark/pull/31127] > Kill barrier tasks should respect SPARK_JOB_INTERRUPT_ON_CANCEL > --- > > Key: SPARK-34069 > URL: https://issues.apache.org/jira/browse/SPARK-34069 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.2.0 >Reporter: ulysses you >Assignee: ulysses you >Priority: Major > Fix For: 3.1.0 > > > We should interrupt task thread if user set local property > `SPARK_JOB_INTERRUPT_ON_CANCEL` to true. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-32917) Add support for executors to push shuffle blocks after successful map task completion
[ https://issues.apache.org/jira/browse/SPARK-32917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32917: --- Assignee: Chandni Singh > Add support for executors to push shuffle blocks after successful map task > completion > - > > Key: SPARK-32917 > URL: https://issues.apache.org/jira/browse/SPARK-32917 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Chandni Singh >Priority: Major > Fix For: 3.2.0 > > > This is the shuffle write path for push-based shuffle, where the executors > would leverage the RPC protocol to push shuffle blocks to remote shuffle > services. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-32917) Add support for executors to push shuffle blocks after successful map task completion
[ https://issues.apache.org/jira/browse/SPARK-32917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32917. - Fix Version/s: 3.2.0 Resolution: Fixed Issue resolved by pull request 30312 [https://github.com/apache/spark/pull/30312] > Add support for executors to push shuffle blocks after successful map task > completion > - > > Key: SPARK-32917 > URL: https://issues.apache.org/jira/browse/SPARK-32917 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Priority: Major > Fix For: 3.2.0 > > > This is the shuffle write path for push-based shuffle, where the executors > would leverage the RPC protocol to push shuffle blocks to remote shuffle > services. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-33669) Wrong error message from YARN application state monitor when sc.stop in yarn client mode
[
https://issues.apache.org/jira/browse/SPARK-33669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan reassigned SPARK-33669:
---
Assignee: Su Qilong
> Wrong error message from YARN application state monitor when sc.stop in yarn
> client mode
>
>
> Key: SPARK-33669
> URL: https://issues.apache.org/jira/browse/SPARK-33669
> Project: Spark
> Issue Type: Bug
> Components: YARN
>Affects Versions: 2.4.3, 3.0.1
>Reporter: Su Qilong
>Assignee: Su Qilong
>Priority: Minor
> Fix For: 3.1.0
>
>
> For YarnClient mode, when stopping YarnClientSchedulerBackend, it first tries
> to interrupt Yarn application monitor thread. In MonitorThread.run() it
> catches InterruptedException to gracefully response to stopping request.
> But client.monitorApplication method also throws InterruptedIOException when
> the hadoop rpc call is calling. In this case, MonitorThread will not know it
> is interrupted, a Yarn App failed is returned with "Failed to contact YARN
> for application x; YARN application has exited unexpectedly with state
> x" is logged with error level. which confuse user a lot.
> We Should take considerate InterruptedIOException here to make it the same
> behavior with InterruptedException.
> {code:java}
> private class MonitorThread extends Thread {
> private var allowInterrupt = true
> override def run() {
> try {
> val YarnAppReport(_, state, diags) =
> client.monitorApplication(appId.get, logApplicationReport = false)
> logError(s"YARN application has exited unexpectedly with state $state!
> " +
> "Check the YARN application logs for more details.")
> diags.foreach { err =>
> logError(s"Diagnostics message: $err")
> }
> allowInterrupt = false
> sc.stop()
> } catch {
> case e: InterruptedException => logInfo("Interrupting monitor thread")
> }
> }
>
> {code}
> {code:java}
> // wrong error message
> 2020-12-05 03:06:58,000 ERROR [YARN application state monitor]:
> org.apache.spark.deploy.yarn.Client(91) - Failed to contact YARN for
> application application_1605868815011_1154961.
> java.io.InterruptedIOException: Call interrupted
> at org.apache.hadoop.ipc.Client.call(Client.java:1466)
> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy38.getApplicationReport(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:187)
> at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
> at com.sun.proxy.$Proxy39.getApplicationReport(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:408)
> at
> org.apache.spark.deploy.yarn.Client.getApplicationReport(Client.scala:327)
> at
> org.apache.spark.deploy.yarn.Client.monitorApplication(Client.scala:1039)
> at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:116)
> 2020-12-05 03:06:58,000 ERROR [YARN application state monitor]:
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend(70) - YARN
> application has exited unexpectedly with state FAILED! Check the YARN
> application logs for more details.
> 2020-12-05 03:06:58,001 ERROR [YARN application state monitor]:
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend(70) -
> Diagnostics message: Failed to contact YARN for application
> application_1605868815011_1154961.
> {code}
>
> {code:java}
> // hadoop ipc code
> public Writable call(RPC.RpcKind rpcKind, Writable rpcRequest,
> ConnectionId remoteId, int serviceClass,
> AtomicBoolean fallbackToSimpleAuth) throws IOException {
> final Call call = createCall(rpcKind, rpcRequest);
> Connection connection = getConnection(remoteId, call, serviceClass,
> fallbackToSimpleAuth);
> try {
> connection.sendRpcRequest(call); // send the rpc request
> } catch (RejectedExecutionException e) {
> throw new IOException("connection has been closed", e);
> } catch (InterruptedException e) {
>
[jira] [Resolved] (SPARK-33669) Wrong error message from YARN application state monitor when sc.stop in yarn client mode
[
https://issues.apache.org/jira/browse/SPARK-33669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan resolved SPARK-33669.
-
Fix Version/s: 3.1.0
Resolution: Fixed
Issue resolved by pull request 30617
[https://github.com/apache/spark/pull/30617]
> Wrong error message from YARN application state monitor when sc.stop in yarn
> client mode
>
>
> Key: SPARK-33669
> URL: https://issues.apache.org/jira/browse/SPARK-33669
> Project: Spark
> Issue Type: Bug
> Components: YARN
>Affects Versions: 2.4.3, 3.0.1
>Reporter: Su Qilong
>Priority: Minor
> Fix For: 3.1.0
>
>
> For YarnClient mode, when stopping YarnClientSchedulerBackend, it first tries
> to interrupt Yarn application monitor thread. In MonitorThread.run() it
> catches InterruptedException to gracefully response to stopping request.
> But client.monitorApplication method also throws InterruptedIOException when
> the hadoop rpc call is calling. In this case, MonitorThread will not know it
> is interrupted, a Yarn App failed is returned with "Failed to contact YARN
> for application x; YARN application has exited unexpectedly with state
> x" is logged with error level. which confuse user a lot.
> We Should take considerate InterruptedIOException here to make it the same
> behavior with InterruptedException.
> {code:java}
> private class MonitorThread extends Thread {
> private var allowInterrupt = true
> override def run() {
> try {
> val YarnAppReport(_, state, diags) =
> client.monitorApplication(appId.get, logApplicationReport = false)
> logError(s"YARN application has exited unexpectedly with state $state!
> " +
> "Check the YARN application logs for more details.")
> diags.foreach { err =>
> logError(s"Diagnostics message: $err")
> }
> allowInterrupt = false
> sc.stop()
> } catch {
> case e: InterruptedException => logInfo("Interrupting monitor thread")
> }
> }
>
> {code}
> {code:java}
> // wrong error message
> 2020-12-05 03:06:58,000 ERROR [YARN application state monitor]:
> org.apache.spark.deploy.yarn.Client(91) - Failed to contact YARN for
> application application_1605868815011_1154961.
> java.io.InterruptedIOException: Call interrupted
> at org.apache.hadoop.ipc.Client.call(Client.java:1466)
> at org.apache.hadoop.ipc.Client.call(Client.java:1409)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
> at com.sun.proxy.$Proxy38.getApplicationReport(Unknown Source)
> at
> org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.getApplicationReport(ApplicationClientProtocolPBClientImpl.java:187)
> at sun.reflect.GeneratedMethodAccessor22.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
> at com.sun.proxy.$Proxy39.getApplicationReport(Unknown Source)
> at
> org.apache.hadoop.yarn.client.api.impl.YarnClientImpl.getApplicationReport(YarnClientImpl.java:408)
> at
> org.apache.spark.deploy.yarn.Client.getApplicationReport(Client.scala:327)
> at
> org.apache.spark.deploy.yarn.Client.monitorApplication(Client.scala:1039)
> at
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend$MonitorThread.run(YarnClientSchedulerBackend.scala:116)
> 2020-12-05 03:06:58,000 ERROR [YARN application state monitor]:
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend(70) - YARN
> application has exited unexpectedly with state FAILED! Check the YARN
> application logs for more details.
> 2020-12-05 03:06:58,001 ERROR [YARN application state monitor]:
> org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend(70) -
> Diagnostics message: Failed to contact YARN for application
> application_1605868815011_1154961.
> {code}
>
> {code:java}
> // hadoop ipc code
> public Writable call(RPC.RpcKind rpcKind, Writable rpcRequest,
> ConnectionId remoteId, int serviceClass,
> AtomicBoolean fallbackToSimpleAuth) throws IOException {
> final Call call = createCall(rpcKind, rpcRequest);
> Connection connection = getConnection(remoteId, call, serviceClass,
> fallbackToSimpleAuth);
> try {
> connection.sendRpcRequest(call); // send the rpc request
> } catch (RejectedExecutionException e) {
> throw new
[jira] [Resolved] (SPARK-33185) YARN: Print direct links to driver logs alongside application report in cluster mode
[
https://issues.apache.org/jira/browse/SPARK-33185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan resolved SPARK-33185.
-
Resolution: Fixed
Issue resolved by pull request 30450
[https://github.com/apache/spark/pull/30450]
> YARN: Print direct links to driver logs alongside application report in
> cluster mode
>
>
> Key: SPARK-33185
> URL: https://issues.apache.org/jira/browse/SPARK-33185
> Project: Spark
> Issue Type: Improvement
> Components: YARN
>Affects Versions: 3.0.1
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Fix For: 3.1.0
>
>
> Currently when run in {{cluster}} mode on YARN, the Spark {{yarn.Client}}
> will print out the application report into the logs, to be easily viewed by
> users. For example:
> {code}
> INFO yarn.Client:
>client token: Token { kind: YARN_CLIENT_TOKEN, service: }
>diagnostics: N/A
>ApplicationMaster host: X.X.X.X
>ApplicationMaster RPC port: 0
>queue: default
>start time: 1602782566027
>final status: UNDEFINED
>tracking URL: http://hostname:/proxy/application_/
>user: xkrogen
> {code}
> Typically, the tracking URL can be used to find the logs of the
> ApplicationMaster/driver while the application is running. Later, the Spark
> History Server can be used to track this information down, using the
> stdout/stderr links on the Executors page.
> However, in the situation when the driver crashed _before_ writing out a
> history file, the SHS may not be aware of this application, and thus does not
> contain links to the driver logs. When this situation arises, it can be
> difficult for users to debug further, since they can't easily find their
> driver logs.
> It is possible to reach the logs by using the {{yarn logs}} commands, but the
> average Spark user isn't aware of this and shouldn't have to be.
> I propose adding, alongside the application report, some additional lines
> like:
> {code}
> Driver Logs (stdout):
> http://hostname:8042/node/containerlogs/container_/xkrogen/stdout?start=-4096
> Driver Logs (stderr):
> http://hostname:8042/node/containerlogs/container_/xkrogen/stderr?start=-4096
> {code}
> With this information available, users can quickly jump to their driver logs,
> even if it crashed before the SHS became aware of the application. This has
> the additional benefit of providing a quick way to access driver logs, which
> often contain useful information, in a single click (instead of navigating
> through the Spark UI).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-32918) RPC implementation to support control plane coordination for push-based shuffle
[ https://issues.apache.org/jira/browse/SPARK-32918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32918: --- Assignee: Ye Zhou > RPC implementation to support control plane coordination for push-based > shuffle > --- > > Key: SPARK-32918 > URL: https://issues.apache.org/jira/browse/SPARK-32918 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Ye Zhou >Priority: Major > Fix For: 3.1.0 > > > RPCs to facilitate coordination of shuffle map/reduce stages. Notifications > to external shuffle services to finalize shuffle block merge for a given > shuffle are carried through this RPC. It also respond back the metadata about > a merged shuffle partition back to the caller. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-32918) RPC implementation to support control plane coordination for push-based shuffle
[ https://issues.apache.org/jira/browse/SPARK-32918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32918. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30163 [https://github.com/apache/spark/pull/30163] > RPC implementation to support control plane coordination for push-based > shuffle > --- > > Key: SPARK-32918 > URL: https://issues.apache.org/jira/browse/SPARK-32918 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Priority: Major > Fix For: 3.1.0 > > > RPCs to facilitate coordination of shuffle map/reduce stages. Notifications > to external shuffle services to finalize shuffle block merge for a given > shuffle are carried through this RPC. It also respond back the metadata about > a merged shuffle partition back to the caller. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-32919) Add support in Spark driver to coordinate the shuffle map stage in push-based shuffle by selecting external shuffle services for merging shuffle partitions
[ https://issues.apache.org/jira/browse/SPARK-32919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32919: --- Assignee: Venkata krishnan Sowrirajan > Add support in Spark driver to coordinate the shuffle map stage in push-based > shuffle by selecting external shuffle services for merging shuffle partitions > --- > > Key: SPARK-32919 > URL: https://issues.apache.org/jira/browse/SPARK-32919 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Venkata krishnan Sowrirajan >Priority: Major > Fix For: 3.1.0 > > > In the beginning of a shuffle map stage, driver needs to select external > shuffle services as the mergers of the shuffle partitions for the > corresponding shuffle. > We currently leverage the immediate available information about current and > past executor location information for this selection purpose. Ideally, this > would be behind a pluggable interface so that we can potentially leverage > information tracked outside of a Spark application for better load balancing > or for a disaggregate deployment environment. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-32919) Add support in Spark driver to coordinate the shuffle map stage in push-based shuffle by selecting external shuffle services for merging shuffle partitions
[ https://issues.apache.org/jira/browse/SPARK-32919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32919. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30164 [https://github.com/apache/spark/pull/30164] > Add support in Spark driver to coordinate the shuffle map stage in push-based > shuffle by selecting external shuffle services for merging shuffle partitions > --- > > Key: SPARK-32919 > URL: https://issues.apache.org/jira/browse/SPARK-32919 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Priority: Major > Fix For: 3.1.0 > > > In the beginning of a shuffle map stage, driver needs to select external > shuffle services as the mergers of the shuffle partitions for the > corresponding shuffle. > We currently leverage the immediate available information about current and > past executor location information for this selection purpose. Ideally, this > would be behind a pluggable interface so that we can potentially leverage > information tracked outside of a Spark application for better load balancing > or for a disaggregate deployment environment. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-31069) high cpu caused by chunksBeingTransferred in external shuffle service
[ https://issues.apache.org/jira/browse/SPARK-31069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-31069: --- Assignee: angerszhu > high cpu caused by chunksBeingTransferred in external shuffle service > - > > Key: SPARK-31069 > URL: https://issues.apache.org/jira/browse/SPARK-31069 > Project: Spark > Issue Type: Improvement > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Xiaoju Wu >Assignee: angerszhu >Priority: Major > Fix For: 3.1.0 > > > "shuffle-chunk-fetch-handler-2-40" #250 daemon prio=5 os_prio=0 > tid=0x02ac nid=0xb9b3 runnable [0x7ff20a1af000] >java.lang.Thread.State: RUNNABLE > at > java.util.concurrent.ConcurrentHashMap$Traverser.advance(ConcurrentHashMap.java:3339) > at > java.util.concurrent.ConcurrentHashMap$ValueIterator.next(ConcurrentHashMap.java:3439) > at > org.apache.spark.network.server.OneForOneStreamManager.chunksBeingTransferred(OneForOneStreamManager.java:184) > at > org.apache.spark.network.server.ChunkFetchRequestHandler.channelRead0(ChunkFetchRequestHandler.java:85) > at > org.apache.spark.network.server.ChunkFetchRequestHandler.channelRead0(ChunkFetchRequestHandler.java:51) > at > org.spark_project.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) > at > org.spark_project.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) > at > org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) > at > org.spark_project.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) > at > org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) > at > org.spark_project.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) > at java.lang.Thread.run(Thread.java:748) > > > > "shuffle-chunk-fetch-handler-2-48" #235 daemon prio=5 os_prio=0 > tid=0x7ff2302ec800 nid=0xb9ad runnable [0x7ff20a7b4000] >java.lang.Thread.State: RUNNABLE > at > org.apache.spark.network.server.OneForOneStreamManager.chunksBeingTransferred(OneForOneStreamManager.java:186) > at > org.apache.spark.network.server.ChunkFetchRequestHandler.channelRead0(ChunkFetchRequestHandler.java:85) > at > org.apache.spark.network.server.ChunkFetchRequestHandler.channelRead0(ChunkFetchRequestHandler.java:51) > at > org.spark_project.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) > at > org.spark_project.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) > at > org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) > at > org.spark_project.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) > at > org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) > at > org.spark_project.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-31069) high cpu caused by chunksBeingTransferred in external shuffle service
[ https://issues.apache.org/jira/browse/SPARK-31069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-31069: --- Assignee: angerszhu (was: angerszhu) > high cpu caused by chunksBeingTransferred in external shuffle service > - > > Key: SPARK-31069 > URL: https://issues.apache.org/jira/browse/SPARK-31069 > Project: Spark > Issue Type: Improvement > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Xiaoju Wu >Assignee: angerszhu >Priority: Major > Fix For: 3.1.0 > > > "shuffle-chunk-fetch-handler-2-40" #250 daemon prio=5 os_prio=0 > tid=0x02ac nid=0xb9b3 runnable [0x7ff20a1af000] >java.lang.Thread.State: RUNNABLE > at > java.util.concurrent.ConcurrentHashMap$Traverser.advance(ConcurrentHashMap.java:3339) > at > java.util.concurrent.ConcurrentHashMap$ValueIterator.next(ConcurrentHashMap.java:3439) > at > org.apache.spark.network.server.OneForOneStreamManager.chunksBeingTransferred(OneForOneStreamManager.java:184) > at > org.apache.spark.network.server.ChunkFetchRequestHandler.channelRead0(ChunkFetchRequestHandler.java:85) > at > org.apache.spark.network.server.ChunkFetchRequestHandler.channelRead0(ChunkFetchRequestHandler.java:51) > at > org.spark_project.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) > at > org.spark_project.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) > at > org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) > at > org.spark_project.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) > at > org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) > at > org.spark_project.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) > at java.lang.Thread.run(Thread.java:748) > > > > "shuffle-chunk-fetch-handler-2-48" #235 daemon prio=5 os_prio=0 > tid=0x7ff2302ec800 nid=0xb9ad runnable [0x7ff20a7b4000] >java.lang.Thread.State: RUNNABLE > at > org.apache.spark.network.server.OneForOneStreamManager.chunksBeingTransferred(OneForOneStreamManager.java:186) > at > org.apache.spark.network.server.ChunkFetchRequestHandler.channelRead0(ChunkFetchRequestHandler.java:85) > at > org.apache.spark.network.server.ChunkFetchRequestHandler.channelRead0(ChunkFetchRequestHandler.java:51) > at > org.spark_project.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) > at > org.spark_project.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) > at > org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) > at > org.spark_project.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) > at > org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) > at > org.spark_project.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-31069) high cpu caused by chunksBeingTransferred in external shuffle service
[ https://issues.apache.org/jira/browse/SPARK-31069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-31069. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30139 [https://github.com/apache/spark/pull/30139] > high cpu caused by chunksBeingTransferred in external shuffle service > - > > Key: SPARK-31069 > URL: https://issues.apache.org/jira/browse/SPARK-31069 > Project: Spark > Issue Type: Improvement > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Xiaoju Wu >Priority: Major > Fix For: 3.1.0 > > > "shuffle-chunk-fetch-handler-2-40" #250 daemon prio=5 os_prio=0 > tid=0x02ac nid=0xb9b3 runnable [0x7ff20a1af000] >java.lang.Thread.State: RUNNABLE > at > java.util.concurrent.ConcurrentHashMap$Traverser.advance(ConcurrentHashMap.java:3339) > at > java.util.concurrent.ConcurrentHashMap$ValueIterator.next(ConcurrentHashMap.java:3439) > at > org.apache.spark.network.server.OneForOneStreamManager.chunksBeingTransferred(OneForOneStreamManager.java:184) > at > org.apache.spark.network.server.ChunkFetchRequestHandler.channelRead0(ChunkFetchRequestHandler.java:85) > at > org.apache.spark.network.server.ChunkFetchRequestHandler.channelRead0(ChunkFetchRequestHandler.java:51) > at > org.spark_project.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) > at > org.spark_project.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) > at > org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) > at > org.spark_project.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) > at > org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) > at > org.spark_project.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) > at java.lang.Thread.run(Thread.java:748) > > > > "shuffle-chunk-fetch-handler-2-48" #235 daemon prio=5 os_prio=0 > tid=0x7ff2302ec800 nid=0xb9ad runnable [0x7ff20a7b4000] >java.lang.Thread.State: RUNNABLE > at > org.apache.spark.network.server.OneForOneStreamManager.chunksBeingTransferred(OneForOneStreamManager.java:186) > at > org.apache.spark.network.server.ChunkFetchRequestHandler.channelRead0(ChunkFetchRequestHandler.java:85) > at > org.apache.spark.network.server.ChunkFetchRequestHandler.channelRead0(ChunkFetchRequestHandler.java:51) > at > org.spark_project.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:38) > at > org.spark_project.io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:353) > at > org.spark_project.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163) > at > org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:403) > at > org.spark_project.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:463) > at > org.spark_project.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) > at > org.spark_project.io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:138) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-32915) RPC implementation to support pushing and merging shuffle blocks
[ https://issues.apache.org/jira/browse/SPARK-32915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32915: --- Assignee: Min Shen > RPC implementation to support pushing and merging shuffle blocks > > > Key: SPARK-32915 > URL: https://issues.apache.org/jira/browse/SPARK-32915 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Min Shen >Priority: Major > Fix For: 3.1.0 > > > RPC implementation for the basic functionality in network-common and > network-shuffle module to enable pushing blocks on the client side and > merging received blocks on the server side. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-32916) Add support for external shuffle service in YARN deployment mode to leverage push-based shuffle
[ https://issues.apache.org/jira/browse/SPARK-32916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32916. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 30062 [https://github.com/apache/spark/pull/30062] > Add support for external shuffle service in YARN deployment mode to leverage > push-based shuffle > --- > > Key: SPARK-32916 > URL: https://issues.apache.org/jira/browse/SPARK-32916 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core, YARN >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Chandni Singh >Priority: Major > Fix For: 3.1.0 > > > Integration needed to bootstrap external shuffle service in YARN deployment > mode. Properly create the necessary dirs and initialize the relevant > server-side components in the RPC layer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-32916) Add support for external shuffle service in YARN deployment mode to leverage push-based shuffle
[ https://issues.apache.org/jira/browse/SPARK-32916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan reassigned SPARK-32916: --- Assignee: Chandni Singh > Add support for external shuffle service in YARN deployment mode to leverage > push-based shuffle > --- > > Key: SPARK-32916 > URL: https://issues.apache.org/jira/browse/SPARK-32916 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core, YARN >Affects Versions: 3.1.0 >Reporter: Min Shen >Assignee: Chandni Singh >Priority: Major > > Integration needed to bootstrap external shuffle service in YARN deployment > mode. Properly create the necessary dirs and initialize the relevant > server-side components in the RPC layer. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-33185) YARN: Print direct links to driver logs alongside application report in cluster mode
[
https://issues.apache.org/jira/browse/SPARK-33185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan reassigned SPARK-33185:
---
Assignee: Erik Krogen
> YARN: Print direct links to driver logs alongside application report in
> cluster mode
>
>
> Key: SPARK-33185
> URL: https://issues.apache.org/jira/browse/SPARK-33185
> Project: Spark
> Issue Type: Improvement
> Components: YARN
>Affects Versions: 3.0.1
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
>
> Currently when run in {{cluster}} mode on YARN, the Spark {{yarn.Client}}
> will print out the application report into the logs, to be easily viewed by
> users. For example:
> {code}
> INFO yarn.Client:
>client token: Token { kind: YARN_CLIENT_TOKEN, service: }
>diagnostics: N/A
>ApplicationMaster host: X.X.X.X
>ApplicationMaster RPC port: 0
>queue: default
>start time: 1602782566027
>final status: UNDEFINED
>tracking URL: http://hostname:/proxy/application_/
>user: xkrogen
> {code}
> Typically, the tracking URL can be used to find the logs of the
> ApplicationMaster/driver while the application is running. Later, the Spark
> History Server can be used to track this information down, using the
> stdout/stderr links on the Executors page.
> However, in the situation when the driver crashed _before_ writing out a
> history file, the SHS may not be aware of this application, and thus does not
> contain links to the driver logs. When this situation arises, it can be
> difficult for users to debug further, since they can't easily find their
> driver logs.
> It is possible to reach the logs by using the {{yarn logs}} commands, but the
> average Spark user isn't aware of this and shouldn't have to be.
> I propose adding, alongside the application report, some additional lines
> like:
> {code}
> Driver Logs (stdout):
> http://hostname:8042/node/containerlogs/container_/xkrogen/stdout?start=-4096
> Driver Logs (stderr):
> http://hostname:8042/node/containerlogs/container_/xkrogen/stderr?start=-4096
> {code}
> With this information available, users can quickly jump to their driver logs,
> even if it crashed before the SHS became aware of the application. This has
> the additional benefit of providing a quick way to access driver logs, which
> often contain useful information, in a single click (instead of navigating
> through the Spark UI).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-33185) YARN: Print direct links to driver logs alongside application report in cluster mode
[
https://issues.apache.org/jira/browse/SPARK-33185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan resolved SPARK-33185.
-
Fix Version/s: 3.1.0
Resolution: Fixed
Issue resolved by pull request 30096
[https://github.com/apache/spark/pull/30096]
> YARN: Print direct links to driver logs alongside application report in
> cluster mode
>
>
> Key: SPARK-33185
> URL: https://issues.apache.org/jira/browse/SPARK-33185
> Project: Spark
> Issue Type: Improvement
> Components: YARN
>Affects Versions: 3.0.1
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
> Fix For: 3.1.0
>
>
> Currently when run in {{cluster}} mode on YARN, the Spark {{yarn.Client}}
> will print out the application report into the logs, to be easily viewed by
> users. For example:
> {code}
> INFO yarn.Client:
>client token: Token { kind: YARN_CLIENT_TOKEN, service: }
>diagnostics: N/A
>ApplicationMaster host: X.X.X.X
>ApplicationMaster RPC port: 0
>queue: default
>start time: 1602782566027
>final status: UNDEFINED
>tracking URL: http://hostname:/proxy/application_/
>user: xkrogen
> {code}
> Typically, the tracking URL can be used to find the logs of the
> ApplicationMaster/driver while the application is running. Later, the Spark
> History Server can be used to track this information down, using the
> stdout/stderr links on the Executors page.
> However, in the situation when the driver crashed _before_ writing out a
> history file, the SHS may not be aware of this application, and thus does not
> contain links to the driver logs. When this situation arises, it can be
> difficult for users to debug further, since they can't easily find their
> driver logs.
> It is possible to reach the logs by using the {{yarn logs}} commands, but the
> average Spark user isn't aware of this and shouldn't have to be.
> I propose adding, alongside the application report, some additional lines
> like:
> {code}
> Driver Logs (stdout):
> http://hostname:8042/node/containerlogs/container_/xkrogen/stdout?start=-4096
> Driver Logs (stderr):
> http://hostname:8042/node/containerlogs/container_/xkrogen/stderr?start=-4096
> {code}
> With this information available, users can quickly jump to their driver logs,
> even if it crashed before the SHS became aware of the application. This has
> the additional benefit of providing a quick way to access driver logs, which
> often contain useful information, in a single click (instead of navigating
> through the Spark UI).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-33088) Enhance ExecutorPlugin API to include methods for task start and end events
[
https://issues.apache.org/jira/browse/SPARK-33088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan resolved SPARK-33088.
-
Fix Version/s: 3.1.0
Resolution: Fixed
Issue resolved by pull request 29977
[https://github.com/apache/spark/pull/29977]
> Enhance ExecutorPlugin API to include methods for task start and end events
> ---
>
> Key: SPARK-33088
> URL: https://issues.apache.org/jira/browse/SPARK-33088
> Project: Spark
> Issue Type: New Feature
> Components: Spark Core
>Affects Versions: 3.1.0
>Reporter: Samuel Souza
>Priority: Major
> Fix For: 3.1.0
>
>
> On [SPARK-24918|https://issues.apache.org/jira/browse/SPARK-24918]'s
> [SIPP|https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/view#|https://docs.google.com/document/d/1a20gHGMyRbCM8aicvq4LhWfQmoA5cbHBQtyqIA2hgtc/edit#],
> it was raised to potentially add methods to ExecutorPlugin interface on task
> start and end:
> {quote}The basic interface can just be a marker trait, as that allows a
> plugin to monitor general characteristics of the JVM (eg. monitor memory or
> take thread dumps). Optionally, we could include methods for task start and
> end events. This would allow more control on monitoring – eg., you could
> start polling thread dumps only if there was a task from a particular stage
> that had been taking too long. But anything task related is a bit trickier to
> decide the right api. Should the task end event also get the failure reason?
> Should those events get called in the same thread as the task runner, or in
> another thread?
> {quote}
> The ask is to add exactly that. I've put up a draft PR [in our fork of
> spark|https://github.com/palantir/spark/pull/713] and I'm happy to push it
> upstream. Also happy to receive comments on what's the right interface to
> expose - not opinionated on that front, tried to expose the simplest
> interface for now.
> The main reason for this ask is to propagate tracing information from the
> driver to the executors
> ([SPARK-21962|https://issues.apache.org/jira/browse/SPARK-21962] has some
> context). On
> [HADOOP-15566|https://issues.apache.org/jira/browse/HADOOP-15566] I see we're
> discussing how to add tracing to the Apache ecosystem, but my problem is
> slightly different: I want to use this interface to propagate tracing
> information to my framework of choice. If the Hadoop issue gets solved we'll
> have a framework to communicate tracing information inside the Apache
> ecosystem, but it's highly unlikely that all Spark users will use the same
> common framework. Therefore we should still provide plugin interfaces where
> the tracing information can be propagated appropriately.
> To give more color, in our case the tracing information is [stored in a
> thread
> local|https://github.com/palantir/tracing-java/blob/4.9.0/tracing/src/main/java/com/palantir/tracing/Tracer.java#L61],
> therefore it needs to be set in the same thread which is executing the task.
> [*]
> While our framework is specific, I imagine such an interface could be useful
> in general. Happy to hear your thoughts about it.
> [*] Something I did not mention was how to propagate the tracing information
> from the driver to the executors. For that I intend to use 1. the driver's
> localProperties, which 2. will be eventually propagated to the executors'
> TaskContext, which 3. I'll be able to access from the methods above.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-32915) RPC implementation to support pushing and merging shuffle blocks
[ https://issues.apache.org/jira/browse/SPARK-32915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32915. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 29855 [https://github.com/apache/spark/pull/29855] > RPC implementation to support pushing and merging shuffle blocks > > > Key: SPARK-32915 > URL: https://issues.apache.org/jira/browse/SPARK-32915 > Project: Spark > Issue Type: Sub-task > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Priority: Major > Fix For: 3.1.0 > > > RPC implementation for the basic functionality in network-common and > network-shuffle module to enable pushing blocks on the client side and > merging received blocks on the server side. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32738) thread safe endpoints may hang due to fatal error
[ https://issues.apache.org/jira/browse/SPARK-32738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-32738: Fix Version/s: 2.4.8 > thread safe endpoints may hang due to fatal error > - > > Key: SPARK-32738 > URL: https://issues.apache.org/jira/browse/SPARK-32738 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.4, 2.4.6, 3.0.0 >Reporter: Zhenhua Wang >Assignee: Zhenhua Wang >Priority: Major > Fix For: 2.4.8, 3.0.2, 3.1.0 > > > Processing for `ThreadSafeRpcEndpoint` is controlled by 'numActiveThreads' in > `Inbox`. Now if any fatal error happens during `Inbox.process`, > 'numActiveThreads' is not reduced. Then other threads can not process > messages in that inbox, which causes the endpoint to "hang". > This problem is more serious in previous Spark 2.x versions since the driver, > executor and block manager endpoints are all thread safe endpoints. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-32738) thread safe endpoints may hang due to fatal error
[ https://issues.apache.org/jira/browse/SPARK-32738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-32738: Fix Version/s: 3.0.2 > thread safe endpoints may hang due to fatal error > - > > Key: SPARK-32738 > URL: https://issues.apache.org/jira/browse/SPARK-32738 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.4, 2.4.6, 3.0.0 >Reporter: Zhenhua Wang >Assignee: Zhenhua Wang >Priority: Major > Fix For: 3.0.2, 3.1.0 > > > Processing for `ThreadSafeRpcEndpoint` is controlled by 'numActiveThreads' in > `Inbox`. Now if any fatal error happens during `Inbox.process`, > 'numActiveThreads' is not reduced. Then other threads can not process > messages in that inbox, which causes the endpoint to "hang". > This problem is more serious in previous Spark 2.x versions since the driver, > executor and block manager endpoints are all thread safe endpoints. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-30602) SPIP: Support push-based shuffle to improve shuffle efficiency
[ https://issues.apache.org/jira/browse/SPARK-30602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17183622#comment-17183622 ] Mridul Muralidharan commented on SPARK-30602: - SPIP proposal document: [https://docs.google.com/document/d/1mYzKVZllA5Flw8AtoX7JUcXBOnNIDADWRbJ7GI6Y71Q/edit|https://docs.google.com/document/d/1mYzKVZllA5Flw8AtoX7JUcXBOnNIDADWRbJ7GI6Y71Q/edit] > SPIP: Support push-based shuffle to improve shuffle efficiency > -- > > Key: SPARK-30602 > URL: https://issues.apache.org/jira/browse/SPARK-30602 > Project: Spark > Issue Type: Improvement > Components: Shuffle, Spark Core >Affects Versions: 3.1.0 >Reporter: Min Shen >Priority: Major > Attachments: Screen Shot 2020-06-23 at 11.31.22 AM.jpg, > vldb_magnet_final.pdf > > > In a large deployment of a Spark compute infrastructure, Spark shuffle is > becoming a potential scaling bottleneck and a source of inefficiency in the > cluster. When doing Spark on YARN for a large-scale deployment, people > usually enable Spark external shuffle service and store the intermediate > shuffle files on HDD. Because the number of blocks generated for a particular > shuffle grows quadratically compared to the size of shuffled data (# mappers > and reducers grows linearly with the size of shuffled data, but # blocks is # > mappers * # reducers), one general trend we have observed is that the more > data a Spark application processes, the smaller the block size becomes. In a > few production clusters we have seen, the average shuffle block size is only > 10s of KBs. Because of the inefficiency of performing random reads on HDD for > small amount of data, the overall efficiency of the Spark external shuffle > services serving the shuffle blocks degrades as we see an increasing # of > Spark applications processing an increasing amount of data. In addition, > because Spark external shuffle service is a shared service in a multi-tenancy > cluster, the inefficiency with one Spark application could propagate to other > applications as well. > In this ticket, we propose a solution to improve Spark shuffle efficiency in > above mentioned environments with push-based shuffle. With push-based > shuffle, shuffle is performed at the end of mappers and blocks get pre-merged > and move towards reducers. In our prototype implementation, we have seen > significant efficiency improvements when performing large shuffles. We take a > Spark-native approach to achieve this, i.e., extending Spark’s existing > shuffle netty protocol, and the behaviors of Spark mappers, reducers and > drivers. This way, we can bring the benefits of more efficient shuffle in > Spark without incurring the dependency or overhead of either specialized > storage layer or external infrastructure pieces. > > Link to dev mailing list discussion: > http://apache-spark-developers-list.1001551.n3.nabble.com/Enabling-push-based-shuffle-in-Spark-td28732.html -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-32663) TransportClient getting closed when there are outstanding requests to the server
[
https://issues.apache.org/jira/browse/SPARK-32663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan reassigned SPARK-32663:
---
Assignee: Attila Zsolt Piros
> TransportClient getting closed when there are outstanding requests to the
> server
>
>
> Key: SPARK-32663
> URL: https://issues.apache.org/jira/browse/SPARK-32663
> Project: Spark
> Issue Type: Bug
> Components: Shuffle
>Affects Versions: 3.0.0
>Reporter: Chandni Singh
>Assignee: Attila Zsolt Piros
>Priority: Major
>
> The implementation of {{removeBlocks}} and {{getHostLocalDirs}} in
> {{ExternalBlockStoreClient}} closes the client after processing a response in
> the callback.
> This is a cached client which will be re-used for other responses. There
> could be other outstanding request to the shuffle service, so it should not
> be closed after processing a response.
> Seems like this is a bug introduced with SPARK-27651 and SPARK-27677.
> The older methods {{registerWithShuffleServer}} and {{fetchBlocks}} didn't
> close the client.
> cc [~attilapiros] [~vanzin] [~mridulm80]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-32663) TransportClient getting closed when there are outstanding requests to the server
[
https://issues.apache.org/jira/browse/SPARK-32663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan resolved SPARK-32663.
-
Fix Version/s: 3.1.0
3.0.1
Resolution: Fixed
Issue resolved by pull request 29492
[https://github.com/apache/spark/pull/29492]
> TransportClient getting closed when there are outstanding requests to the
> server
>
>
> Key: SPARK-32663
> URL: https://issues.apache.org/jira/browse/SPARK-32663
> Project: Spark
> Issue Type: Bug
> Components: Shuffle
>Affects Versions: 3.0.0
>Reporter: Chandni Singh
>Assignee: Attila Zsolt Piros
>Priority: Major
> Fix For: 3.0.1, 3.1.0
>
>
> The implementation of {{removeBlocks}} and {{getHostLocalDirs}} in
> {{ExternalBlockStoreClient}} closes the client after processing a response in
> the callback.
> This is a cached client which will be re-used for other responses. There
> could be other outstanding request to the shuffle service, so it should not
> be closed after processing a response.
> Seems like this is a bug introduced with SPARK-27651 and SPARK-27677.
> The older methods {{registerWithShuffleServer}} and {{fetchBlocks}} didn't
> close the client.
> cc [~attilapiros] [~vanzin] [~mridulm80]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-32119) ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars
[ https://issues.apache.org/jira/browse/SPARK-32119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-32119. - Fix Version/s: 3.1.0 Resolution: Fixed Issue resolved by pull request 28939 [https://github.com/apache/spark/pull/28939] > ExecutorPlugin doesn't work with Standalone Cluster and Kubernetes with --jars > -- > > Key: SPARK-32119 > URL: https://issues.apache.org/jira/browse/SPARK-32119 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 3.0.1, 3.1.0 >Reporter: Kousuke Saruta >Assignee: Kousuke Saruta >Priority: Major > Fix For: 3.1.0 > > > ExecutorPlugin can't work with Standalone Cluster and Kubernetes > when a jar which contains plugins and files used by the plugins are added by > --jars and --files option with spark-submit. > This is because jars and files added by --jars and --files are not loaded on > Executor initialization. > I confirmed it works with YARN because jars/files are distributed as > distributed cache. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25594) OOM in long running applications even with UI disabled
[ https://issues.apache.org/jira/browse/SPARK-25594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17150729#comment-17150729 ] Mridul Muralidharan commented on SPARK-25594: - Given regression in functionality if this is merged, closing bug. See comment: https://github.com/apache/spark/pull/22609#issuecomment-426405757 > OOM in long running applications even with UI disabled > -- > > Key: SPARK-25594 > URL: https://issues.apache.org/jira/browse/SPARK-25594 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0, 2.4.0 >Reporter: Mridul Muralidharan >Assignee: Mridul Muralidharan >Priority: Major > > Typically for long running applications with large number of tasks it is > common to disable UI to minimize overhead at driver. > Earlier, with spark ui disabled, only stage/job information was kept as part > of JobProgressListener. > As part of history server scalability fixes, particularly SPARK-20643, > inspite of disabling UI - task information continues to be maintained in > memory. > In our long running tests against spark thrift server, this eventually > results in OOM. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-25594) OOM in long running applications even with UI disabled
[ https://issues.apache.org/jira/browse/SPARK-25594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-25594. - Resolution: Won't Fix > OOM in long running applications even with UI disabled > -- > > Key: SPARK-25594 > URL: https://issues.apache.org/jira/browse/SPARK-25594 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0, 2.4.0 >Reporter: Mridul Muralidharan >Assignee: Mridul Muralidharan >Priority: Major > > Typically for long running applications with large number of tasks it is > common to disable UI to minimize overhead at driver. > Earlier, with spark ui disabled, only stage/job information was kept as part > of JobProgressListener. > As part of history server scalability fixes, particularly SPARK-20643, > inspite of disabling UI - task information continues to be maintained in > memory. > In our long running tests against spark thrift server, this eventually > results in OOM. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29302) dynamic partition overwrite with speculation enabled
[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086124#comment-17086124 ] Mridul Muralidharan commented on SPARK-29302: - I agree with [~feiwang], it looks like newTaskTempFile is not robust to speculative execution and task failures when dynamicPartitionOverwrite is enabled IMO. This will need to be fixed - it is currently using the same path irrespective of which attempt it is. > dynamic partition overwrite with speculation enabled > > > Key: SPARK-29302 > URL: https://issues.apache.org/jira/browse/SPARK-29302 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.4 >Reporter: feiwang >Priority: Major > Attachments: screenshot-1.png, screenshot-2.png > > > Now, for a dynamic partition overwrite operation, the filename of a task > output is determinable. > So, if speculation is enabled, would a task conflict with its relative > speculation task? > Would the two tasks concurrent write a same file? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Issue Comment Deleted] (SPARK-29302) dynamic partition overwrite with speculation enabled
[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-29302: Comment: was deleted (was: Drive by observations: * Speculative execution does not run the speculative task concurrently on the same node running the task (so definitely not same executor). * Do we have additional details about the tasks which are failing ? Did they fail before (for unrelated reasons) before failing for path conflict ? ) > dynamic partition overwrite with speculation enabled > > > Key: SPARK-29302 > URL: https://issues.apache.org/jira/browse/SPARK-29302 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.4 >Reporter: feiwang >Priority: Major > Attachments: screenshot-1.png, screenshot-2.png > > > Now, for a dynamic partition overwrite operation, the filename of a task > output is determinable. > So, if speculation is enabled, would a task conflict with its relative > speculation task? > Would the two tasks concurrent write a same file? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-29302) dynamic partition overwrite with speculation enabled
[ https://issues.apache.org/jira/browse/SPARK-29302?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17086117#comment-17086117 ] Mridul Muralidharan commented on SPARK-29302: - Drive by observations: * Speculative execution does not run the speculative task concurrently on the same node running the task (so definitely not same executor). * Do we have additional details about the tasks which are failing ? Did they fail before (for unrelated reasons) before failing for path conflict ? > dynamic partition overwrite with speculation enabled > > > Key: SPARK-29302 > URL: https://issues.apache.org/jira/browse/SPARK-29302 > Project: Spark > Issue Type: Bug > Components: SQL >Affects Versions: 2.4.4 >Reporter: feiwang >Priority: Major > Attachments: screenshot-1.png, screenshot-2.png > > > Now, for a dynamic partition overwrite operation, the filename of a task > output is determinable. > So, if speculation is enabled, would a task conflict with its relative > speculation task? > Would the two tasks concurrent write a same file? -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26688) Provide configuration of initially blacklisted YARN nodes
[ https://issues.apache.org/jira/browse/SPARK-26688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750477#comment-16750477 ] Mridul Muralidharan commented on SPARK-26688: - If this is a legitimate usecase, we should get yarn team to enhance node label support. > Provide configuration of initially blacklisted YARN nodes > - > > Key: SPARK-26688 > URL: https://issues.apache.org/jira/browse/SPARK-26688 > Project: Spark > Issue Type: Improvement > Components: YARN >Affects Versions: 3.0.0 >Reporter: Attila Zsolt Piros >Priority: Major > > Introducing new config for initially blacklisted YARN nodes. > This came up in the apache spark user mailing list: > [http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Yarn-is-it-possible-to-manually-blacklist-nodes-before-running-spark-job-td34395.html] > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-26688) Provide configuration of initially blacklisted YARN nodes
[ https://issues.apache.org/jira/browse/SPARK-26688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748915#comment-16748915 ] Mridul Muralidharan commented on SPARK-26688: - What is the usecase for this ? As others have mentioned in the news group, nodelabels is the typical way to prevent (or require) allocation requests to satisfy some constraint. > Provide configuration of initially blacklisted YARN nodes > - > > Key: SPARK-26688 > URL: https://issues.apache.org/jira/browse/SPARK-26688 > Project: Spark > Issue Type: Improvement > Components: YARN >Affects Versions: 3.0.0 >Reporter: Attila Zsolt Piros >Priority: Major > > Introducing new config for initially blacklisted YARN nodes. > This came up in the apache spark user mailing list: > [http://apache-spark-user-list.1001560.n3.nabble.com/Spark-on-Yarn-is-it-possible-to-manually-blacklist-nodes-before-running-spark-job-td34395.html] > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25732) Allow specifying a keytab/principal for proxy user for token renewal
[
https://issues.apache.org/jira/browse/SPARK-25732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16651301#comment-16651301
]
Mridul Muralidharan commented on SPARK-25732:
-
[~vanzin] With long running applications (not necessarily streaming) needing
access (read/write) to various data sources (not just hdfs), is there a way to
do this even assuming livy rpc was augmented to support it ? For example, livy
server would not know which data sources to fetch tokens for (since that will
be part of user application jars/config).
For the specific usecase [~mgaido] detailed, proxy principal (foo)/keytab would
be present and distinct from zeppelin or livy principal/keytab.
The 'proxy' part would simply be for livy to submit the application as the
proxied user 'foo' - once application comes up, it will behave as though it was
submitted by the user 'foo' with specified keytab (from hdfs) - acquire/renew
tokens for user 'foo' from its keytab.
[~tgraves] I do share your concern; unfortunately for the usecase Marco is
targeting, there does not seem to be an alternative; livy server is man in the
middle here (w.r.t submitting client).
Having said that, if there is an alternative, I would definitely prefer that
over sharing keytabs - even if it is over secured hdfs.
> Allow specifying a keytab/principal for proxy user for token renewal
> -
>
> Key: SPARK-25732
> URL: https://issues.apache.org/jira/browse/SPARK-25732
> Project: Spark
> Issue Type: Improvement
> Components: Deploy
>Affects Versions: 2.4.0
>Reporter: Marco Gaido
>Priority: Major
>
> As of now, application submitted with proxy-user fail after 2 week due to the
> lack of token renewal. In order to enable it, we need the the
> keytab/principal of the impersonated user to be specified, in order to have
> them available for the token renewal.
> This JIRA proposes to add two parameters {{--proxy-user-principal}} and
> {{--proxy-user-keytab}}, and the last letting a keytab being specified also
> in a distributed FS, so that applications can be submitted by servers (eg.
> Livy, Zeppelin) without needing all users' principals being on that machine.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-25594) OOM in long running applications even with UI disabled
[ https://issues.apache.org/jira/browse/SPARK-25594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16635122#comment-16635122 ] Mridul Muralidharan commented on SPARK-25594: - Task level information is required only when spark UI is enabled, and tends to be expensive to maintain (TaskDataWrapper was occupying bulk of the heap at OOM). We can avoid maintaining task information when it is a live UI (not history server) and spark UI has been disabled (spark.ui.enabled = false) > OOM in long running applications even with UI disabled > -- > > Key: SPARK-25594 > URL: https://issues.apache.org/jira/browse/SPARK-25594 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0, 2.4.0 >Reporter: Mridul Muralidharan >Assignee: Mridul Muralidharan >Priority: Major > > Typically for long running applications with large number of tasks it is > common to disable UI to minimize overhead at driver. > Earlier, with spark ui disabled, only stage/job information was kept as part > of JobProgressListener. > As part of history server scalability fixes, particularly SPARK-20643, > inspite of disabling UI - task information continues to be maintained in > memory. > In our long running tests against spark thrift server, this eventually > results in OOM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-25594) OOM in long running applications even with UI disabled
Mridul Muralidharan created SPARK-25594: --- Summary: OOM in long running applications even with UI disabled Key: SPARK-25594 URL: https://issues.apache.org/jira/browse/SPARK-25594 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.3.0, 2.4.0 Reporter: Mridul Muralidharan Assignee: Mridul Muralidharan Typically for long running applications with large number of tasks it is common to disable UI to minimize overhead at driver. Earlier, with spark ui disabled, only stage/job information was kept as part of JobProgressListener. As part of history server scalability fixes, particularly SPARK-20643, inspite of disabling UI - task information continues to be maintained in memory. In our long running tests against spark thrift server, this eventually results in OOM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-24948) SHS filters wrongly some applications due to permission check
[ https://issues.apache.org/jira/browse/SPARK-24948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan resolved SPARK-24948. - Resolution: Fixed > SHS filters wrongly some applications due to permission check > - > > Key: SPARK-24948 > URL: https://issues.apache.org/jira/browse/SPARK-24948 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.3.1 >Reporter: Marco Gaido >Priority: Blocker > Fix For: 2.4.0 > > > SHS filters the event logs it doesn't have permissions to read. > Unfortunately, this check is quite naive, as it takes into account only the > base permissions (ie. user, group, other permissions). For instance, if ACL > are enabled, they are ignored in this check; moreover, each filesystem may > have different policies (eg. they can consider spark as a superuser who can > access everything). > This results in some applications not being displayed in the SHS, despite the > Spark user (or whatever user the SHS is started with) can actually read their > ent logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-24948) SHS filters wrongly some applications due to permission check
[ https://issues.apache.org/jira/browse/SPARK-24948?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-24948: Fix Version/s: 2.4.0 > SHS filters wrongly some applications due to permission check > - > > Key: SPARK-24948 > URL: https://issues.apache.org/jira/browse/SPARK-24948 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.3.1 >Reporter: Marco Gaido >Priority: Blocker > Fix For: 2.4.0 > > > SHS filters the event logs it doesn't have permissions to read. > Unfortunately, this check is quite naive, as it takes into account only the > base permissions (ie. user, group, other permissions). For instance, if ACL > are enabled, they are ignored in this check; moreover, each filesystem may > have different policies (eg. they can consider spark as a superuser who can > access everything). > This results in some applications not being displayed in the SHS, despite the > Spark user (or whatever user the SHS is started with) can actually read their > ent logs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark
[
https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569363#comment-16569363
]
Mridul Muralidharan edited comment on SPARK-24375 at 8/5/18 4:42 AM:
-
{quote} We've thought hard on the issue and don't feel we can make it unless we
force users to explicitly set a number in a barrier() call (actually it's not a
good idea because it brings more borden to manage the code).{quote}
I am not sure where the additional burden exists.
Make it an optional param to barrier.
* If not defined, it would be analogous to what exists right now.
* If specified, fail the stage if different tasks in stage end up waiting on
different barrier names (or some have a name and others dont).
In example usecases I have seen, there is usually partition specific code paths
(if partition 0, do some initialization/teardown, etc) - which results in
divergent codepaths : and so increases potential for this issue.
It will be very difficult to reason about the state when this happens.
was (Author: mridulm80):
{quote} We've thought hard on the issue and don't feel we can make it unless we
force users to explicitly set a number in a barrier() call (actually it's not a
good idea because it brings more borden to manage the code).{quote}
I am not sure where the additional burden exists.
Make it an optional param to barrier.
* If not defined, it would be analogous to what exists right now.
* If specified, fail the stage if different tasks in stage end up waiting on
different barrier names (or some have a name and others dont).
In example usecases I have seen, there is usually partition specific code paths
(if partition 0, do some initialization/teardown, etc) - which results in
divergent codepaths : and so increases potential for this issue.
It will be very difficult to reason about the state what happens.
> Design sketch: support barrier scheduling in Apache Spark
> -
>
> Key: SPARK-24375
> URL: https://issues.apache.org/jira/browse/SPARK-24375
> Project: Spark
> Issue Type: Story
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Xiangrui Meng
>Assignee: Jiang Xingbo
>Priority: Major
>
> This task is to outline a design sketch for the barrier scheduling SPIP
> discussion. It doesn't need to be a complete design before the vote. But it
> should at least cover both Scala/Java and PySpark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark
[
https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16569363#comment-16569363
]
Mridul Muralidharan commented on SPARK-24375:
-
{quote} We've thought hard on the issue and don't feel we can make it unless we
force users to explicitly set a number in a barrier() call (actually it's not a
good idea because it brings more borden to manage the code).{quote}
I am not sure where the additional burden exists.
Make it an optional param to barrier.
* If not defined, it would be analogous to what exists right now.
* If specified, fail the stage if different tasks in stage end up waiting on
different barrier names (or some have a name and others dont).
In example usecases I have seen, there is usually partition specific code paths
(if partition 0, do some initialization/teardown, etc) - which results in
divergent codepaths : and so increases potential for this issue.
It will be very difficult to reason about the state what happens.
> Design sketch: support barrier scheduling in Apache Spark
> -
>
> Key: SPARK-24375
> URL: https://issues.apache.org/jira/browse/SPARK-24375
> Project: Spark
> Issue Type: Story
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Xiangrui Meng
>Assignee: Jiang Xingbo
>Priority: Major
>
> This task is to outline a design sketch for the barrier scheduling SPIP
> discussion. It doesn't need to be a complete design before the vote. But it
> should at least cover both Scala/Java and PySpark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark
[
https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16568664#comment-16568664
]
Mridul Muralidharan commented on SPARK-24375:
-
{quote}
It's not desired behavior to catch exception thrown by TaskContext.barrier()
silently. However, in case this really happens, we can detect that because we
have `epoch` both in driver side and executor side, more details will go to the
design doc of BarrierTaskContext.barrier() SPARK-24581
{quote}
The current 'barrier' function does not identify 'which' barrier it is from a
user point of view.
Here, due to exceptions raised (not necessarily from barrier(), but could be
from user code as well), different tasks are waiting on different barriers.
{code}
try {
... snippet A ...
// Barrier 1
context.barrier()
... snippet B ...
} catch { ... }
... snippet C ...
// Barrier 2
context.barrier()
{code}
T1 waits on barrier 1, T2 could have raised exception in snippet A and ends up
waiting on Barrier 2 (having never seen Barrier 1).
In this scenario, how is spark making progress ?
(And ofcourse, when T1 reaches barrier 2, when T2 has moved past it).
I did not see this clarified in the design or in the implementation.
> Design sketch: support barrier scheduling in Apache Spark
> -
>
> Key: SPARK-24375
> URL: https://issues.apache.org/jira/browse/SPARK-24375
> Project: Spark
> Issue Type: Story
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Xiangrui Meng
>Assignee: Jiang Xingbo
>Priority: Major
>
> This task is to outline a design sketch for the barrier scheduling SPIP
> discussion. It doesn't need to be a complete design before the vote. But it
> should at least cover both Scala/Java and PySpark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24615) Accelerator-aware task scheduling for Spark
[ https://issues.apache.org/jira/browse/SPARK-24615?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16551198#comment-16551198 ] Mridul Muralidharan commented on SPARK-24615: - [~tgraves] This was indeed a recurring issue - the ability to modulate ask's to RM based on current requirements. What you bring out is an excellent point - changing resource requirements would be very useful - particularly for applications with heterogenous resource needs. Even currently when executor_memory/executor_cores does not allign well with stage requirements, we end up with OOM - resulting in over-provisioning memory needs; resulting in suboptimal use. GPU/accelerator aware scheduler is an extension of the same - where we have other resources to consider. I agree with [~tgraves] that a more general way to model this would look at all resources (when declaratively specified ofcourse) and use the information to allocate resources (from RM) and for task schedule (within spark). > Accelerator-aware task scheduling for Spark > --- > > Key: SPARK-24615 > URL: https://issues.apache.org/jira/browse/SPARK-24615 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.4.0 >Reporter: Saisai Shao >Assignee: Saisai Shao >Priority: Major > Labels: Hydrogen, SPIP > > In the machine learning area, accelerator card (GPU, FPGA, TPU) is > predominant compared to CPUs. To make the current Spark architecture to work > with accelerator cards, Spark itself should understand the existence of > accelerators and know how to schedule task onto the executors where > accelerators are equipped. > Current Spark’s scheduler schedules tasks based on the locality of the data > plus the available of CPUs. This will introduce some problems when scheduling > tasks with accelerators required. > # CPU cores are usually more than accelerators on one node, using CPU cores > to schedule accelerator required tasks will introduce the mismatch. > # In one cluster, we always assume that CPU is equipped in each node, but > this is not true of accelerator cards. > # The existence of heterogeneous tasks (accelerator required or not) > requires scheduler to schedule tasks with a smart way. > So here propose to improve the current scheduler to support heterogeneous > tasks (accelerator requires or not). This can be part of the work of Project > hydrogen. > Details is attached in google doc. It doesn't cover all the implementation > details, just highlight the parts should be changed. > > CC [~yanboliang] [~merlintang] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24755) Executor loss can cause task to not be resubmitted
[ https://issues.apache.org/jira/browse/SPARK-24755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16536016#comment-16536016 ] Mridul Muralidharan commented on SPARK-24755: - Go for it - thanks [~hthuynh2] ! > Executor loss can cause task to not be resubmitted > -- > > Key: SPARK-24755 > URL: https://issues.apache.org/jira/browse/SPARK-24755 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0 >Reporter: Mridul Muralidharan >Priority: Major > > As part of SPARK-22074, when an executor is lost, TSM.executorLost currently > checks for "if (successful(index) && !killedByOtherAttempt(index))" to decide > if task needs to be resubmitted for partition. > Consider following: > For partition P1, tasks T1 and T2 are running on exec-1 and exec-2 > respectively (one of them being speculative task) > T1 finishes successfully first. > This results in setting "killedByOtherAttempt(P1) = true" due to running T2. > We also end up killing task T2. > Now, exec-1 if/when goes MIA. > executorLost will no longer schedule task for P1 - since > killedByOtherAttempt(P1) == true; even though P1 was hosted on T1 and there > is no other copy of P1 around (T2 was killed when T1 succeeded). > I noticed this bug as part of reviewing PR# 21653 for SPARK-13343 > Essentially, SPARK-22074 causes a regression (which I dont usually observe > due to shuffle service, sigh) - and as such the fix is broken IMO. > I dont have a PR handy for this, so if anyone wants to pick it up, please do > feel free ! > +CC [~XuanYuan] who fixed SPARK-22074 initially. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-24755) Executor loss can cause task to not be resubmitted
[ https://issues.apache.org/jira/browse/SPARK-24755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-24755: Description: As part of SPARK-22074, when an executor is lost, TSM.executorLost currently checks for "if (successful(index) && !killedByOtherAttempt(index))" to decide if task needs to be resubmitted for partition. Consider following: For partition P1, tasks T1 and T2 are running on exec-1 and exec-2 respectively (one of them being speculative task) T1 finishes successfully first. This results in setting "killedByOtherAttempt(P1) = true" due to running T2. We also end up killing task T2. Now, exec-1 if/when goes MIA. executorLost will no longer schedule task for P1 - since killedByOtherAttempt(P1) == true; even though P1 was hosted on T1 and there is no other copy of P1 around (T2 was killed when T1 succeeded). I noticed this bug as part of reviewing PR# 21653 for SPARK-13343 Essentially, SPARK-22074 causes a regression (which I dont usually observe due to shuffle service, sigh) - and as such the fix is broken IMO. I dont have a PR handy for this, so if anyone wants to pick it up, please do feel free ! +CC [~XuanYuan] who fixed SPARK-22074 initially. was: As part of SPARK-22074, when an executor is lost, TSM.executorLost currently checks for "if (successful(index) && !killedByOtherAttempt(index))" to decide if task needs to be resubmitted for partition. Consider following: For partition P1, tasks T1 and T2 are running on exec-1 and exec-2 respectively (one of them being speculative task) T1 finishes successfully first. This results in setting "killedByOtherAttempt(P1) = true" due to running T2. We also end up killing task T2. Now, exec-1 if/when goes MIA. executorLost will no longer schedule task for P1 - since killedByOtherAttempt(P1) == true; even though P1 was hosted on T1 and there is no other copy of P1 around (T2 was killed when T1 succeeded). I noticed this bug as part of reviewing PR# 21653 for SPARK-13343 Essentially, SPARK-22074 causes a regression (which I dont usually observe due to shuffle service, sigh) - and as such the fix is broken IMO : I believe it got introduced as part of the review (the original change looked fine to me - but I did not look at it in detail). I dont have a PR handy for this, so if anyone wants to pick it up, please do feel free ! +CC [~XuanYuan] who fixed SPARK-22074 initially. > Executor loss can cause task to not be resubmitted > -- > > Key: SPARK-24755 > URL: https://issues.apache.org/jira/browse/SPARK-24755 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0 >Reporter: Mridul Muralidharan >Priority: Major > > As part of SPARK-22074, when an executor is lost, TSM.executorLost currently > checks for "if (successful(index) && !killedByOtherAttempt(index))" to decide > if task needs to be resubmitted for partition. > Consider following: > For partition P1, tasks T1 and T2 are running on exec-1 and exec-2 > respectively (one of them being speculative task) > T1 finishes successfully first. > This results in setting "killedByOtherAttempt(P1) = true" due to running T2. > We also end up killing task T2. > Now, exec-1 if/when goes MIA. > executorLost will no longer schedule task for P1 - since > killedByOtherAttempt(P1) == true; even though P1 was hosted on T1 and there > is no other copy of P1 around (T2 was killed when T1 succeeded). > I noticed this bug as part of reviewing PR# 21653 for SPARK-13343 > Essentially, SPARK-22074 causes a regression (which I dont usually observe > due to shuffle service, sigh) - and as such the fix is broken IMO. > I dont have a PR handy for this, so if anyone wants to pick it up, please do > feel free ! > +CC [~XuanYuan] who fixed SPARK-22074 initially. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-24755) Executor loss can cause task to not be resubmitted
[ https://issues.apache.org/jira/browse/SPARK-24755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-24755: Summary: Executor loss can cause task to not be resubmitted (was: Executor loss can cause task to be not resubmitted) > Executor loss can cause task to not be resubmitted > -- > > Key: SPARK-24755 > URL: https://issues.apache.org/jira/browse/SPARK-24755 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0 >Reporter: Mridul Muralidharan >Priority: Major > > As part of SPARK-22074, when an executor is lost, TSM.executorLost currently > checks for "if (successful(index) && !killedByOtherAttempt(index))" to decide > if task needs to be resubmitted for partition. > Consider following: > For partition P1, tasks T1 and T2 are running on exec-1 and exec-2 > respectively (one of them being speculative task) > T1 finishes successfully first. > This results in setting "killedByOtherAttempt(P1) = true" due to running T2. > We also end up killing task T2. > Now, exec-1 if/when goes MIA. > executorLost will no longer schedule task for P1 - since > killedByOtherAttempt(P1) == true; even though P1 was hosted on T1 and there > is no other copy of P1 around (T2 was killed when T1 succeeded). > I noticed this bug as part of reviewing PR# 21653 for SPARK-13343 > Essentially, SPARK-22074 causes a regression (which I dont usually observe > due to shuffle service, sigh) - and as such the fix is broken IMO : I believe > it got introduced as part of the review (the original change looked fine to > me - but I did not look at it in detail). > I dont have a PR handy for this, so if anyone wants to pick it up, please do > feel free ! > +CC [~XuanYuan] who fixed SPARK-22074 initially. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-24755) Executor loss can cause task to be not resubmitted
[ https://issues.apache.org/jira/browse/SPARK-24755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-24755: Description: As part of SPARK-22074, when an executor is lost, TSM.executorLost currently checks for "if (successful(index) && !killedByOtherAttempt(index))" to decide if task needs to be resubmitted for partition. Consider following: For partition P1, tasks T1 and T2 are running on exec-1 and exec-2 respectively (one of them being speculative task) T1 finishes successfully first. This results in setting "killedByOtherAttempt(P1) = true" due to running T2. We also end up killing task T2. Now, exec-1 if/when goes MIA. executorLost will no longer schedule task for P1 - since killedByOtherAttempt(P1) == true; even though P1 was hosted on T1 and there is no other copy of P1 around (T2 was killed when T1 succeeded). I noticed this bug as part of reviewing PR# 21653 for SPARK-13343 Essentially, SPARK-22074 causes a regression (which I dont usually observe due to shuffle service, sigh) - and as such the fix is broken IMO : I believe it got introduced as part of the review (the original change looked fine to me - but I did not look at it in detail). I dont have a PR handy for this, so if anyone wants to pick it up, please do feel free ! +CC [~XuanYuan] who fixed SPARK-22074 initially. was: As part of SPARK-22074, when an executor is lost, TSM.executorLost currently checks for "if (successful(index) && !killedByOtherAttempt(index))" to decide if task needs to be resubmitted for partition. Consider following: For partition P1, tasks T1 and T2 are running on exec-1 and exec-2 respectively (one of them being speculative task) T1 finishes successfully first. This results in setting "killedByOtherAttempt(P1) = true" due to running T2. We also end up killing task T2. Now, exec-1 if/when goes MIA. executorLost will no longer schedule task for P1 - since killedByOtherAttempt(P1) == true; even though P1 was hosted on T1 and there is no other copy of P1 around (T2 was killed, not T1 - which was successful). I noticed this bug as part of reviewing PR# 21653 for SPARK-13343 Essentially, SPARK-22074 causes a regression (which I dont usually observe due to shuffle service, sigh) - and as such the fix is broken IMO : I believe it got introduced as part of the review (the original change looked fine to me - but I did not look at it in detail). I dont have a PR handy for this, so if anyone wants to pick it up, please do feel free ! +CC [~XuanYuan] who fixed SPARK-22074 initially. > Executor loss can cause task to be not resubmitted > -- > > Key: SPARK-24755 > URL: https://issues.apache.org/jira/browse/SPARK-24755 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.3.0 >Reporter: Mridul Muralidharan >Priority: Major > > As part of SPARK-22074, when an executor is lost, TSM.executorLost currently > checks for "if (successful(index) && !killedByOtherAttempt(index))" to decide > if task needs to be resubmitted for partition. > Consider following: > For partition P1, tasks T1 and T2 are running on exec-1 and exec-2 > respectively (one of them being speculative task) > T1 finishes successfully first. > This results in setting "killedByOtherAttempt(P1) = true" due to running T2. > We also end up killing task T2. > Now, exec-1 if/when goes MIA. > executorLost will no longer schedule task for P1 - since > killedByOtherAttempt(P1) == true; even though P1 was hosted on T1 and there > is no other copy of P1 around (T2 was killed when T1 succeeded). > I noticed this bug as part of reviewing PR# 21653 for SPARK-13343 > Essentially, SPARK-22074 causes a regression (which I dont usually observe > due to shuffle service, sigh) - and as such the fix is broken IMO : I believe > it got introduced as part of the review (the original change looked fine to > me - but I did not look at it in detail). > I dont have a PR handy for this, so if anyone wants to pick it up, please do > feel free ! > +CC [~XuanYuan] who fixed SPARK-22074 initially. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-24755) Executor loss can cause task to be not resubmitted
Mridul Muralidharan created SPARK-24755: --- Summary: Executor loss can cause task to be not resubmitted Key: SPARK-24755 URL: https://issues.apache.org/jira/browse/SPARK-24755 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.3.0 Reporter: Mridul Muralidharan As part of SPARK-22074, when an executor is lost, TSM.executorLost currently checks for "if (successful(index) && !killedByOtherAttempt(index))" to decide if task needs to be resubmitted for partition. Consider following: For partition P1, tasks T1 and T2 are running on exec-1 and exec-2 respectively (one of them being speculative task) T1 finishes successfully first. This results in setting "killedByOtherAttempt(P1) = true" due to running T2. We also end up killing task T2. Now, exec-1 if/when goes MIA. executorLost will no longer schedule task for P1 - since killedByOtherAttempt(P1) == true; even though P1 was hosted on T1 and there is no other copy of P1 around (T2 was killed, not T1 - which was successful). I noticed this bug as part of reviewing PR# 21653 for SPARK-13343 Essentially, SPARK-22074 causes a regression (which I dont usually observe due to shuffle service, sigh) - and as such the fix is broken IMO : I believe it got introduced as part of the review (the original change looked fine to me - but I did not look at it in detail). I dont have a PR handy for this, so if anyone wants to pick it up, please do feel free ! +CC [~XuanYuan] who fixed SPARK-22074 initially. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark
[
https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516411#comment-16516411
]
Mridul Muralidharan edited comment on SPARK-24375 at 6/18/18 10:17 PM:
---
[~jiangxb1987] A couple of comments based on the document and your elaboration
above:
* Is the 'barrier' logic pluggable ? Instead of only being a global sync point.
* Dynamic resource allocation (dra) triggers allocation of additional resources
based on pending tasks - hence the comment _We may add a check of total
available slots before scheduling tasks from a barrier stage taskset._ does not
necessarily work in that context.
* Currently DRA in spark uniformly allocates resources - are we envisioning
changes as part of this effort to allocate heterogenous executor resources
based on pending tasks (atleast initially for barrier support for gpu's) ?
* How is fault tolerance handled w.r.t waiting on incorrect barriers ? Any way
to identify the barrier ? Example:
{code}
try {
... snippet A ...
// Barrier 1
context.barrier()
... snippet B ...
} catch { ... }
... snippet C ...
// Barrier 2
context.barrier()
{code}
** In face of exceptions, some tasks will wait on barrier 2 and others on
barrier 1 : causing issues.
* Can you elaborate more on leveraging TaskContext.localProperties ? Is it
expected to be sync'ed after 'barrier' returns ? What gaurantees are we
expecting to provide ?
was (Author: mridulm80):
[~jiangxb1987] A couple of comments based on the document and your elaboration
above:
* Is the 'barrier' logic pluggable ? Instead of only being a global sync point.
* Dynamic resource allocation (dra) triggers allocation of additional resources
based on pending tasks - hence the comment _We may add a check of total
available slots before scheduling tasks from a barrier stage taskset._ does not
necessarily work in that context.
* Currently DRA in spark uniformly allocates resources - are we envisioning
changes as part of this effort to allocate heterogenous executor resources
based on pending tasks (atleast initially for barrier support for gpu's) ?
* How is fault tolerance handled w.r.t waiting on incorrect barriers ? Any way
to identify the barrier ? Example:
{code}
try {
... snippet A ...
// Barrier 1
context.barrier()
... snippet B ...
} catch { ... }
... snippet C ...
// Barrier 2
context.barrier()
{code}
** In face of exceptions, some tasks will wait on barrier 2 and others on
barrier 1 : causing issues.
*
> Design sketch: support barrier scheduling in Apache Spark
> -
>
> Key: SPARK-24375
> URL: https://issues.apache.org/jira/browse/SPARK-24375
> Project: Spark
> Issue Type: Story
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Xiangrui Meng
>Assignee: Jiang Xingbo
>Priority: Major
>
> This task is to outline a design sketch for the barrier scheduling SPIP
> discussion. It doesn't need to be a complete design before the vote. But it
> should at least cover both Scala/Java and PySpark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark
[
https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516411#comment-16516411
]
Mridul Muralidharan commented on SPARK-24375:
-
[~jiangxb1987] A couple of comments based on the document and your elaboration
above:
* Is the 'barrier' logic pluggable ? Instead of only being a global sync point.
* Dynamic resource allocation (dra) triggers allocation of additional resources
based on pending tasks - hence *We may add a check of total available slots
before scheduling tasks from a barrier stage taskset.* does not necessarily
work in that context.
* Currently DRA in spark uniformly allocates resources - are we envisioning
changes as part of this effort to allocate heterogenous executor resources
based on pending tasks (atleast initially for barrier support for gpu's) ?
* How is fault tolerance handled w.r.t waiting on incorrect barriers ? Any way
to identify the barrier ? Example:
{code}
try {
... snippet A ...
// Barrier 1
context.barrier()
... snippet B ...
} catch { ... }
... snippet C ...
// Barrier 2
context.barrier()
{code}
** In face of exceptions, some tasks will wait on barrier 2 and others on
barrier 1 : causing issues.
*
> Design sketch: support barrier scheduling in Apache Spark
> -
>
> Key: SPARK-24375
> URL: https://issues.apache.org/jira/browse/SPARK-24375
> Project: Spark
> Issue Type: Story
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Xiangrui Meng
>Assignee: Jiang Xingbo
>Priority: Major
>
> This task is to outline a design sketch for the barrier scheduling SPIP
> discussion. It doesn't need to be a complete design before the vote. But it
> should at least cover both Scala/Java and PySpark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-24375) Design sketch: support barrier scheduling in Apache Spark
[
https://issues.apache.org/jira/browse/SPARK-24375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16516411#comment-16516411
]
Mridul Muralidharan edited comment on SPARK-24375 at 6/18/18 10:15 PM:
---
[~jiangxb1987] A couple of comments based on the document and your elaboration
above:
* Is the 'barrier' logic pluggable ? Instead of only being a global sync point.
* Dynamic resource allocation (dra) triggers allocation of additional resources
based on pending tasks - hence the comment _We may add a check of total
available slots before scheduling tasks from a barrier stage taskset._ does not
necessarily work in that context.
* Currently DRA in spark uniformly allocates resources - are we envisioning
changes as part of this effort to allocate heterogenous executor resources
based on pending tasks (atleast initially for barrier support for gpu's) ?
* How is fault tolerance handled w.r.t waiting on incorrect barriers ? Any way
to identify the barrier ? Example:
{code}
try {
... snippet A ...
// Barrier 1
context.barrier()
... snippet B ...
} catch { ... }
... snippet C ...
// Barrier 2
context.barrier()
{code}
** In face of exceptions, some tasks will wait on barrier 2 and others on
barrier 1 : causing issues.
*
was (Author: mridulm80):
[~jiangxb1987] A couple of comments based on the document and your elaboration
above:
* Is the 'barrier' logic pluggable ? Instead of only being a global sync point.
* Dynamic resource allocation (dra) triggers allocation of additional resources
based on pending tasks - hence *We may add a check of total available slots
before scheduling tasks from a barrier stage taskset.* does not necessarily
work in that context.
* Currently DRA in spark uniformly allocates resources - are we envisioning
changes as part of this effort to allocate heterogenous executor resources
based on pending tasks (atleast initially for barrier support for gpu's) ?
* How is fault tolerance handled w.r.t waiting on incorrect barriers ? Any way
to identify the barrier ? Example:
{code}
try {
... snippet A ...
// Barrier 1
context.barrier()
... snippet B ...
} catch { ... }
... snippet C ...
// Barrier 2
context.barrier()
{code}
** In face of exceptions, some tasks will wait on barrier 2 and others on
barrier 1 : causing issues.
*
> Design sketch: support barrier scheduling in Apache Spark
> -
>
> Key: SPARK-24375
> URL: https://issues.apache.org/jira/browse/SPARK-24375
> Project: Spark
> Issue Type: Story
> Components: Spark Core
>Affects Versions: 3.0.0
>Reporter: Xiangrui Meng
>Assignee: Jiang Xingbo
>Priority: Major
>
> This task is to outline a design sketch for the barrier scheduling SPIP
> discussion. It doesn't need to be a complete design before the vote. But it
> should at least cover both Scala/Java and PySpark.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (YARN-7935) Expose container's hostname to applications running within the docker container
[ https://issues.apache.org/jira/browse/YARN-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16418029#comment-16418029 ] Mridul Muralidharan commented on YARN-7935: --- [~eyang] I think there is some confusion here. Spark does not require user defined networks - I dont think it was mentioned that this was required. Taking a step back: With "host" networking mode, we get it to work without any changes to the application code at all - giving us all the benefits of isolation without any loss in existing functionality (modulo specifying the env variables required ofcourse). When used with bridge/overlay/user defined networks/etc, the container hostname passed to spark AM via allocation request is that of nodemanager, and not the actual container hostname used in the docker container. This patch exposes the container hostname as an env variable - just as we have other container and node specific env variables exposed to the container (CONTAINER_ID, NM_HOST, etc). Do you see any concern with exposing this variable ? I want to make sure I am not missing something here. What spark (or any other application) does with this variable is their implementation detail; I can go into details of why this is required in the case of spark specifically if required, but that might digress from the jira. > Expose container's hostname to applications running within the docker > container > --- > > Key: YARN-7935 > URL: https://issues.apache.org/jira/browse/YARN-7935 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Major > Attachments: YARN-7935.1.patch, YARN-7935.2.patch, YARN-7935.3.patch > > > Some applications have a need to bind to the container's hostname (like > Spark) which is different from the NodeManager's hostname(NM_HOST which is > available as an env during container launch) when launched through Docker > runtime. The container's hostname can be exposed to applications via an env > CONTAINER_HOSTNAME. Another potential candidate is the container's IP but > this can be addressed in a separate jira. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7935) Expose container's hostname to applications running within the docker container
[ https://issues.apache.org/jira/browse/YARN-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415223#comment-16415223 ] Mridul Muralidharan commented on YARN-7935: --- [~eyang] Using YARN service to run spark AM is not feasible; IMO requiring use of yarn service to leverage docker support in yarn must not be a requirement. > Expose container's hostname to applications running within the docker > container > --- > > Key: YARN-7935 > URL: https://issues.apache.org/jira/browse/YARN-7935 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Major > Attachments: YARN-7935.1.patch, YARN-7935.2.patch, YARN-7935.3.patch > > > Some applications have a need to bind to the container's hostname (like > Spark) which is different from the NodeManager's hostname(NM_HOST which is > available as an env during container launch) when launched through Docker > runtime. The container's hostname can be exposed to applications via an env > CONTAINER_HOSTNAME. Another potential candidate is the container's IP but > this can be addressed in a separate jira. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (SPARK-23721) Enhance BlockManagerId to include container's underlying host machine hostname
[
https://issues.apache.org/jira/browse/SPARK-23721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan updated SPARK-23721:
Summary: Enhance BlockManagerId to include container's underlying host
machine hostname (was: Enhance BlockManagerId to include container's
underlying host machie hostname)
> Enhance BlockManagerId to include container's underlying host machine hostname
> --
>
> Key: SPARK-23721
> URL: https://issues.apache.org/jira/browse/SPARK-23721
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.4.0
>Reporter: Mridul Muralidharan
>Priority: Major
>
> In spark, host and rack locality computation is based on BlockManagerId's
> hostname - which is the container's hostname.
> When running in containerized environment's like kubernetes, docker support
> in hadoop 3, mesos docker support, etc; the hostname reported by container is
> not the actual 'host' the container is running on.
> This results in spark getting affected in multiple ways.
> h3. Suboptimal schedules
> Due to host name mismatch between different containers on same physical host,
> spark will treat all containers as running on own host.
> Effectively, there is no host-locality schedule at all due to this.
> In addition, depending on how sophisticated locality script is, it can also
> lead to either suboptimal rack locality computation all the way to no
> rack-locality schedule entirely.
> Hence the performance degradation in scheduler can be significant - only
> PROCESS_LOCAL schedules dont get affected.
> h3. HDFS reads
> This is closely related to "suboptimal schedules" above.
> Block locations for hdfs files refer to the datanode hostnames - and not the
> container's hostname.
> This effectively results in spark ignoring hdfs data placement entirely for
> scheduling tasks - resulting in very heavy cross-node/cross-rack data
> movement.
> h3. Speculative execution
> Spark schedules speculative tasks on a different host - in order to minimize
> the cost of node failures for expensive tasks.
> This gets effectively disabled, resulting in speculative tasks potentially
> running on the same actual host.
> h3. Block replication
> Similar to "speculative execution" above, block replication minimizes
> potential cost of node loss by typically leveraging another host; which gets
> effectively disabled in this case.
> Solution for the above is to enhance BlockManagerId to also include the
> node's actual hostname via 'nodeHostname' - which should be used for usecases
> above instead of the container hostname ('host').
> When not relevant, nodeHostname == hostname : which should ensure all
> existing functionality continues to work as expected with regressions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23721) Enhance BlockManagerId to include container's underlying host machie hostname
[
https://issues.apache.org/jira/browse/SPARK-23721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan updated SPARK-23721:
Summary: Enhance BlockManagerId to include container's underlying host
machie hostname (was: Enhance BlockManagerId to include container's underlying
host name)
> Enhance BlockManagerId to include container's underlying host machie hostname
> -
>
> Key: SPARK-23721
> URL: https://issues.apache.org/jira/browse/SPARK-23721
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.4.0
>Reporter: Mridul Muralidharan
>Priority: Major
>
> In spark, host and rack locality computation is based on BlockManagerId's
> hostname - which is the container's hostname.
> When running in containerized environment's like kubernetes, docker support
> in hadoop 3, mesos docker support, etc; the hostname reported by container is
> not the actual 'host' the container is running on.
> This results in spark getting affected in multiple ways.
> h3. Suboptimal schedules
> Due to host name mismatch between different containers on same physical host,
> spark will treat all containers as running on own host.
> Effectively, there is no host-locality schedule at all due to this.
> In addition, depending on how sophisticated locality script is, it can also
> lead to either suboptimal rack locality computation all the way to no
> rack-locality schedule entirely.
> Hence the performance degradation in scheduler can be significant - only
> PROCESS_LOCAL schedules dont get affected.
> h3. HDFS reads
> This is closely related to "suboptimal schedules" above.
> Block locations for hdfs files refer to the datanode hostnames - and not the
> container's hostname.
> This effectively results in spark ignoring hdfs data placement entirely for
> scheduling tasks - resulting in very heavy cross-node/cross-rack data
> movement.
> h3. Speculative execution
> Spark schedules speculative tasks on a different host - in order to minimize
> the cost of node failures for expensive tasks.
> This gets effectively disabled, resulting in speculative tasks potentially
> running on the same actual host.
> h3. Block replication
> Similar to "speculative execution" above, block replication minimizes
> potential cost of node loss by typically leveraging another host; which gets
> effectively disabled in this case.
> Solution for the above is to enhance BlockManagerId to also include the
> node's actual hostname via 'nodeHostname' - which should be used for usecases
> above instead of the container hostname ('host').
> When not relevant, nodeHostname == hostname : which should ensure all
> existing functionality continues to work as expected with regressions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23721) Enhance BlockManagerId to include container's underlying host name
[
https://issues.apache.org/jira/browse/SPARK-23721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mridul Muralidharan updated SPARK-23721:
Summary: Enhance BlockManagerId to include container's underlying host name
(was: Use actual node's hostname for host and rack locality computation)
> Enhance BlockManagerId to include container's underlying host name
> --
>
> Key: SPARK-23721
> URL: https://issues.apache.org/jira/browse/SPARK-23721
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.4.0
>Reporter: Mridul Muralidharan
>Priority: Major
>
> In spark, host and rack locality computation is based on BlockManagerId's
> hostname - which is the container's hostname.
> When running in containerized environment's like kubernetes, docker support
> in hadoop 3, mesos docker support, etc; the hostname reported by container is
> not the actual 'host' the container is running on.
> This results in spark getting affected in multiple ways.
> h3. Suboptimal schedules
> Due to host name mismatch between different containers on same physical host,
> spark will treat all containers as running on own host.
> Effectively, there is no host-locality schedule at all due to this.
> In addition, depending on how sophisticated locality script is, it can also
> lead to either suboptimal rack locality computation all the way to no
> rack-locality schedule entirely.
> Hence the performance degradation in scheduler can be significant - only
> PROCESS_LOCAL schedules dont get affected.
> h3. HDFS reads
> This is closely related to "suboptimal schedules" above.
> Block locations for hdfs files refer to the datanode hostnames - and not the
> container's hostname.
> This effectively results in spark ignoring hdfs data placement entirely for
> scheduling tasks - resulting in very heavy cross-node/cross-rack data
> movement.
> h3. Speculative execution
> Spark schedules speculative tasks on a different host - in order to minimize
> the cost of node failures for expensive tasks.
> This gets effectively disabled, resulting in speculative tasks potentially
> running on the same actual host.
> h3. Block replication
> Similar to "speculative execution" above, block replication minimizes
> potential cost of node loss by typically leveraging another host; which gets
> effectively disabled in this case.
> Solution for the above is to enhance BlockManagerId to also include the
> node's actual hostname via 'nodeHostname' - which should be used for usecases
> above instead of the container hostname ('host').
> When not relevant, nodeHostname == hostname : which should ensure all
> existing functionality continues to work as expected with regressions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-23721) Use actual node's hostname for host and rack locality computation
Mridul Muralidharan created SPARK-23721:
---
Summary: Use actual node's hostname for host and rack locality
computation
Key: SPARK-23721
URL: https://issues.apache.org/jira/browse/SPARK-23721
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 2.4.0
Reporter: Mridul Muralidharan
In spark, host and rack locality computation is based on BlockManagerId's
hostname - which is the container's hostname.
When running in containerized environment's like kubernetes, docker support in
hadoop 3, mesos docker support, etc; the hostname reported by container is not
the actual 'host' the container is running on.
This results in spark getting affected in multiple ways.
h3. Suboptimal schedules
Due to host name mismatch between different containers on same physical host,
spark will treat all containers as running on own host.
Effectively, there is no host-locality schedule at all due to this.
In addition, depending on how sophisticated locality script is, it can also
lead to either suboptimal rack locality computation all the way to no
rack-locality schedule entirely.
Hence the performance degradation in scheduler can be significant - only
PROCESS_LOCAL schedules dont get affected.
h3. HDFS reads
This is closely related to "suboptimal schedules" above.
Block locations for hdfs files refer to the datanode hostnames - and not the
container's hostname.
This effectively results in spark ignoring hdfs data placement entirely for
scheduling tasks - resulting in very heavy cross-node/cross-rack data movement.
h3. Speculative execution
Spark schedules speculative tasks on a different host - in order to minimize
the cost of node failures for expensive tasks.
This gets effectively disabled, resulting in speculative tasks potentially
running on the same actual host.
h3. Block replication
Similar to "speculative execution" above, block replication minimizes potential
cost of node loss by typically leveraging another host; which gets effectively
disabled in this case.
Solution for the above is to enhance BlockManagerId to also include the node's
actual hostname via 'nodeHostname' - which should be used for usecases above
instead of the container hostname ('host').
When not relevant, nodeHostname == hostname : which should ensure all existing
functionality continues to work as expected with regressions.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-23720) Leverage shuffle service when running in non-host networking mode in hadoop 3 docker support
Mridul Muralidharan created SPARK-23720: --- Summary: Leverage shuffle service when running in non-host networking mode in hadoop 3 docker support Key: SPARK-23720 URL: https://issues.apache.org/jira/browse/SPARK-23720 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 2.4.0 Reporter: Mridul Muralidharan In current external shuffle service integration, hostname of the executor and the shuffle service is the same while the port's are different (shuffle service port vs block manager port). When running in non-host networking mode under docker, in yarn, the shuffle service runs on the NM_HOST while the docker container run's under its own (ephemeral and generated) hostname. We should make use of the container's host machine's hostname for shuffle service and not the container hostname, when external shuffle is enabled. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23718) Document using docker in host networking mode in hadoop 3
[ https://issues.apache.org/jira/browse/SPARK-23718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-23718: Issue Type: Documentation (was: Task) > Document using docker in host networking mode in hadoop 3 > - > > Key: SPARK-23718 > URL: https://issues.apache.org/jira/browse/SPARK-23718 > Project: Spark > Issue Type: Documentation > Components: Documentation >Affects Versions: 2.4.0 >Reporter: Mridul Muralidharan >Priority: Major > > Document the configuration options required to be specified to run Apache > Spark application on Hadoop 3 docker support in host networking mode. > There is no code changes required to leverage the same, giving us package > isolation with all other functionality at-par with what currently exists. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-23719) Use correct hostname in non-host networking mode in hadoop 3 docker support
Mridul Muralidharan created SPARK-23719: --- Summary: Use correct hostname in non-host networking mode in hadoop 3 docker support Key: SPARK-23719 URL: https://issues.apache.org/jira/browse/SPARK-23719 Project: Spark Issue Type: Improvement Components: Spark Core, YARN Affects Versions: 2.4.0 Reporter: Mridul Muralidharan Hostname (node-id's hostname field) specified by RM in allocated containers is the NM_HOST and not the hostname which will be used by the container when running in docker container executor : the actual container hostname is generated at runtime. Due to this spark executor's are unable to launch in non-host networking mode when leveraging docker support in hadoop 3 - due to bind failures as hostname they are trying to bind to is of the host machine and not the container. We can leverage YARN-7935 to fetch the container's hostname (when available) else fallback to existing mechanism - when running executors. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23718) Document using docker in host networking mode in hadoop 3
[ https://issues.apache.org/jira/browse/SPARK-23718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-23718: Issue Type: Improvement (was: Documentation) > Document using docker in host networking mode in hadoop 3 > - > > Key: SPARK-23718 > URL: https://issues.apache.org/jira/browse/SPARK-23718 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 2.4.0 >Reporter: Mridul Muralidharan >Priority: Major > > Document the configuration options required to be specified to run Apache > Spark application on Hadoop 3 docker support in host networking mode. > There is no code changes required to leverage the same, giving us package > isolation with all other functionality at-par with what currently exists. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-23718) Document using docker in host networking mode in hadoop 3
[ https://issues.apache.org/jira/browse/SPARK-23718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mridul Muralidharan updated SPARK-23718: Issue Type: Task (was: Improvement) > Document using docker in host networking mode in hadoop 3 > - > > Key: SPARK-23718 > URL: https://issues.apache.org/jira/browse/SPARK-23718 > Project: Spark > Issue Type: Task > Components: Documentation >Affects Versions: 2.4.0 >Reporter: Mridul Muralidharan >Priority: Major > > Document the configuration options required to be specified to run Apache > Spark application on Hadoop 3 docker support in host networking mode. > There is no code changes required to leverage the same, giving us package > isolation with all other functionality at-par with what currently exists. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-23718) Document using docker in host networking mode in hadoop 3
Mridul Muralidharan created SPARK-23718: --- Summary: Document using docker in host networking mode in hadoop 3 Key: SPARK-23718 URL: https://issues.apache.org/jira/browse/SPARK-23718 Project: Spark Issue Type: Improvement Components: Documentation Affects Versions: 2.4.0 Reporter: Mridul Muralidharan Document the configuration options required to be specified to run Apache Spark application on Hadoop 3 docker support in host networking mode. There is no code changes required to leverage the same, giving us package isolation with all other functionality at-par with what currently exists. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-23717) Leverage docker support in Hadoop 3
Mridul Muralidharan created SPARK-23717: --- Summary: Leverage docker support in Hadoop 3 Key: SPARK-23717 URL: https://issues.apache.org/jira/browse/SPARK-23717 Project: Spark Issue Type: Improvement Components: Spark Core, YARN Affects Versions: 2.4.0 Reporter: Mridul Muralidharan The introduction of docker support in Apache Hadoop 3 can be leveraged by Apache Spark for resolving multiple long standing shortcoming - particularly related to package isolation. It also allows for network isolation, where applicable, allowing for more sophisticated cluster configuration/customization. This jira will track the various tasks for enhancing spark to leverage container support. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (YARN-7935) Expose container's hostname to applications running within the docker container
[ https://issues.apache.org/jira/browse/YARN-7935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16373202#comment-16373202 ] Mridul Muralidharan commented on YARN-7935: --- Hi [~tgraves], Initial expectation was that there will be no change in client application with introduction of docker support in yarn. Unfortunately, in our experiments, we find the need to make application changes - the scope of which is still not know. For example, with 'host' networking mode, there is actually no change required and things work seemlessly ! (and give us package isolation we so badly require) But with bridge or overlay, YARN-7935 will be required (among other things - which is still under investigation). >From spark side of things, I am still unsure about what all other changes are >needed and we are still investigating there too. We use "--hostname" not just to specify where to bind to, but also what to report back to driver. Driver uses this for comm as well as locality computation (HOST_LOCAL and RACK_LOCAL). Hence it is not simple case of replacing --hostname with CONTAINER_HOSTNAME (if available) - that will break HOST locality computation. I wanted to create an a jira in spark once the scope was more concrete, but I can do that sooner if it helps to collaborate better. If you are doing some experiments, would be great to touch base ! > Expose container's hostname to applications running within the docker > container > --- > > Key: YARN-7935 > URL: https://issues.apache.org/jira/browse/YARN-7935 > Project: Hadoop YARN > Issue Type: Sub-task > Components: yarn >Reporter: Suma Shivaprasad >Assignee: Suma Shivaprasad >Priority: Major > Attachments: YARN-7935.1.patch, YARN-7935.2.patch > > > Some applications have a need to bind to the container's hostname (like > Spark) which is different from the NodeManager's hostname(NM_HOST which is > available as an env during container launch) when launched through Docker > runtime. The container's hostname can be exposed to applications via an env > CONTAINER_HOSTNAME. Another potential candidate is the container's IP but > this can be addressed in a separate jira. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
