[jira] [Created] (SPARK-21501) Spark shuffle index cache size should be memory based
Thomas Graves created SPARK-21501: - Summary: Spark shuffle index cache size should be memory based Key: SPARK-21501 URL: https://issues.apache.org/jira/browse/SPARK-21501 Project: Spark Issue Type: Bug Components: Shuffle Affects Versions: 2.0.0 Reporter: Thomas Graves Right now the spark shuffle service has a cache for index files. It is based on a # of files cached (spark.shuffle.service.index.cache.entries). This can cause issues if people have a lot of reducers because the size of each entry can fluctuate based on the # of reducers. We saw an issues with a job that had 17 reducers and it caused NM with spark shuffle service to use 700-800MB or memory in NM by itself. We should change this cache to be memory based and only allow a certain memory size used. When I say memory based I mean the cache should have a limit of say 100MB. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21460) Spark dynamic allocation breaks when ListenerBus event queue runs full
[ https://issues.apache.org/jira/browse/SPARK-21460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095546#comment-16095546 ] Thomas Graves commented on SPARK-21460: --- I didn't think that was the case, but took a look at the code and I guess I was wrong, it definitely appears to be reliant on the listener bus. That is really bad in my opinion. We are intentionally dropping events and we know that will cause issues. > Spark dynamic allocation breaks when ListenerBus event queue runs full > -- > > Key: SPARK-21460 > URL: https://issues.apache.org/jira/browse/SPARK-21460 > Project: Spark > Issue Type: Bug > Components: Scheduler, YARN >Affects Versions: 2.0.0, 2.0.2, 2.1.0, 2.1.1, 2.2.0 > Environment: Spark 2.1 > Hadoop 2.6 >Reporter: Ruslan Dautkhanov >Priority: Critical > Labels: dynamic_allocation, performance, scheduler, yarn > > When ListenerBus event queue runs full, spark dynamic allocation stops > working - Spark fails to shrink number of executors when there are no active > jobs (Spark driver "thinks" there are active jobs since it didn't capture > when they finished) . > ps. What's worse it also makes Spark flood YARN RM with reservation requests, > so YARN preemption doesn't function properly too (we're on Spark 2.1 / Hadoop > 2.6). -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-21243) Limit the number of maps in a single shuffle fetch
[ https://issues.apache.org/jira/browse/SPARK-21243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-21243. --- Resolution: Fixed Fix Version/s: 2.3.0 > Limit the number of maps in a single shuffle fetch > -- > > Key: SPARK-21243 > URL: https://issues.apache.org/jira/browse/SPARK-21243 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.1.0, 2.1.1 >Reporter: Dhruve Ashar >Assignee: Dhruve Ashar >Priority: Minor > Fix For: 2.3.0 > > > Right now spark can limit the # of parallel fetches and also limits the > amount of data in one fetch, but one fetch to a host could be for 100's of > blocks. In one instance we saw 450+ blocks. When you have 100's of those and > 1000's of reducers fetching that becomes a lot of metadata and can run the > Node Manager out of memory. We should add a config to limit the # of maps per > fetch to reduce the load on the NM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-21243) Limit the number of maps in a single shuffle fetch
[ https://issues.apache.org/jira/browse/SPARK-21243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-21243: - Assignee: Dhruve Ashar > Limit the number of maps in a single shuffle fetch > -- > > Key: SPARK-21243 > URL: https://issues.apache.org/jira/browse/SPARK-21243 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Affects Versions: 2.1.0, 2.1.1 >Reporter: Dhruve Ashar >Assignee: Dhruve Ashar >Priority: Minor > Fix For: 2.3.0 > > > Right now spark can limit the # of parallel fetches and also limits the > amount of data in one fetch, but one fetch to a host could be for 100's of > blocks. In one instance we saw 450+ blocks. When you have 100's of those and > 1000's of reducers fetching that becomes a lot of metadata and can run the > Node Manager out of memory. We should add a config to limit the # of maps per > fetch to reduce the load on the NM. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-15703) Make ListenerBus event queue size configurable
[ https://issues.apache.org/jira/browse/SPARK-15703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16091531#comment-16091531 ] Thomas Graves commented on SPARK-15703: --- It should not be breaking dynamic allocation, if it is, its definitely a bug. The scheduler and critical pieces are not supposed to be relying on those messages. If you are seeing issues with that please file a jira and if you know where its broke go ahead and submit pr to fix. > Make ListenerBus event queue size configurable > -- > > Key: SPARK-15703 > URL: https://issues.apache.org/jira/browse/SPARK-15703 > Project: Spark > Issue Type: Improvement > Components: Scheduler, Web UI >Affects Versions: 2.0.0 >Reporter: Thomas Graves >Assignee: Dhruve Ashar >Priority: Minor > Fix For: 2.0.1, 2.1.0 > > Attachments: Screen Shot 2016-06-01 at 11.21.32 AM.png, Screen Shot > 2016-06-01 at 11.23.48 AM.png, spark-dynamic-executor-allocation.png, > SparkListenerBus .png > > > The Spark UI doesn't seem to be showing all the tasks and metrics. > I ran a job with 10 tasks but Detail stage page says it completed 93029: > Summary Metrics for 93029 Completed Tasks > The Stages for all jobs pages list that only 89519/10 tasks finished but > its completed. The metrics for shuffled write and input are also incorrect. > I will attach screen shots. > I checked the logs and it does show that all the tasks actually finished. > 16/06/01 16:15:42 INFO TaskSetManager: Finished task 59880.0 in stage 2.0 > (TID 54038) in 265309 ms on 10.213.45.51 (10/10) > 16/06/01 16:15:42 INFO YarnClusterScheduler: Removed TaskSet 2.0, whose tasks > have all completed, from pool -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-21321) Spark very verbose on shutdown confusing users
[
https://issues.apache.org/jira/browse/SPARK-21321?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves resolved SPARK-21321.
---
Resolution: Fixed
Assignee: Jong Yoon Lee
Fix Version/s: 2.3.0
2.2.1
> Spark very verbose on shutdown confusing users
> --
>
> Key: SPARK-21321
> URL: https://issues.apache.org/jira/browse/SPARK-21321
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.1.1
>Reporter: Jong Yoon Lee
>Assignee: Jong Yoon Lee
>Priority: Minor
> Fix For: 2.2.1, 2.3.0
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> On shutdown, spark can be very verbose and can spit out errors that causes
> the user to be confused.
> If possible, we should not print those out and just ignore them by changing
> the changing the log level from WARNING to DEBUG.
> Also shutdown of spark can take a long time because of the backlog of events
> in the event queue ( warnings like: Message $message dropped.
> ${e.getMessage}).
> This happens more with dynamic allocation on.
> I am suggesting to
> 1. change the log level when the shutdown is happening and the RPC
> connections are closed(RpcEnvStoppedException).
> 2. Clear the event queue when RPC module is stopped and spark is shutting
> down.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode
[
https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084458#comment-16084458
]
Thomas Graves commented on SPARK-21376:
---
so you are referring to the org.apache.spark.launcher.SparkLauncher code that
is launching a yarn cluster mode job? or what do you mean by "local yarn
launcher process"?
> Token is not renewed in yarn client process in cluster mode
> ---
>
> Key: SPARK-21376
> URL: https://issues.apache.org/jira/browse/SPARK-21376
> Project: Spark
> Issue Type: Bug
> Components: YARN
>Affects Versions: 2.1.1, 2.2.0
>Reporter: Yesha Vora
>Priority: Minor
>
> STR:
> * Set below config in spark-default.conf
> {code}
> spark.yarn.security.credentials.hbase.enabled true
> spark.hbase.connector.security.credentials.enabled false{code}
> * Set below config in hdfs-site.xml
> {code}
> 'dfs.namenode.delegation.token.max-lifetime':'4320'
> 'dfs.namenode.delegation.token.renew-interval':'2880' {code}
> * Run HDFSWordcount streaming app in yarn-cluster mode for 25 hrs.
> After 25 hours, noticing that HDFS Wordcount job is hitting
> HDFS_DELEGATION_TOKEN renewal issue.
> {code}
> 17/06/28 10:49:47 WARN Client: Exception encountered while connecting to the
> server :
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 230 for hrt_qa) is expired
> 17/06/28 10:49:47 WARN Client: Failed to cleanup staging dir
> hdfs://mycluster0/user/hrt_qa/.sparkStaging/application_1498539861056_0015
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 230 for hrt_qa) is expired
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1554)
> at org.apache.hadoop.ipc.Client.call(Client.java:1498){code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21376) Token is not renewed in yarn client process in cluster mode
[
https://issues.apache.org/jira/browse/SPARK-21376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16084075#comment-16084075
]
Thomas Graves commented on SPARK-21376:
---
Can you please clarify the title and description? What do you mean by "in yarn
client process in cluster mode"? I assume you were running in yarn cluster
mode but what is the yarn client process? the application master?
> Token is not renewed in yarn client process in cluster mode
> ---
>
> Key: SPARK-21376
> URL: https://issues.apache.org/jira/browse/SPARK-21376
> Project: Spark
> Issue Type: Bug
> Components: YARN
>Affects Versions: 2.1.1, 2.2.0
>Reporter: Yesha Vora
>Priority: Minor
>
> STR:
> * Set below config in spark-default.conf
> {code}
> spark.yarn.security.credentials.hbase.enabled true
> spark.hbase.connector.security.credentials.enabled false{code}
> * Set below config in hdfs-site.xml
> {code}
> 'dfs.namenode.delegation.token.max-lifetime':'4320'
> 'dfs.namenode.delegation.token.renew-interval':'2880' {code}
> * Run HDFSWordcount streaming app in yarn-cluster mode for 25 hrs.
> After 25 hours, noticing that HDFS Wordcount job is hitting
> HDFS_DELEGATION_TOKEN renewal issue.
> {code}
> 17/06/28 10:49:47 WARN Client: Exception encountered while connecting to the
> server :
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 230 for hrt_qa) is expired
> 17/06/28 10:49:47 WARN Client: Failed to cleanup staging dir
> hdfs://mycluster0/user/hrt_qa/.sparkStaging/application_1498539861056_0015
> org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
> token (HDFS_DELEGATION_TOKEN token 230 for hrt_qa) is expired
> at org.apache.hadoop.ipc.Client.getRpcResponse(Client.java:1554)
> at org.apache.hadoop.ipc.Client.call(Client.java:1498){code}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-21383) YARN can allocate to many executors
[
https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083321#comment-16083321
]
Thomas Graves edited comment on SPARK-21383 at 7/12/17 2:17 AM:
Note we saw this with dynamic allocation off. Its easy to reproduce if you
modify runAllocatedContainers to have something like:
{noformat}
new ExecutorRunnable(
Some(container),
conf,
sparkConf,
driverUrl,
executorId,
executorHostname,
executorMemory,
executorCores,
appAttemptId.getApplicationId.toString,
securityMgr,
localResources
).run()
logInfo("Delaying before updating internal to reproduce bug")
try {
Thread.sleep(1)
} catch {
case e => logError("exception", e)
}
updateInternalState()
{noformat}
was (Author: tgraves):
Note we saw this with dynamic allocation off. Its easy to reproduce if you
modify runAllocatedContainers to have something like:
new ExecutorRunnable(
Some(container),
conf,
sparkConf,
driverUrl,
executorId,
executorHostname,
executorMemory,
executorCores,
appAttemptId.getApplicationId.toString,
securityMgr,
localResources
).run()
logInfo("Delaying before updating internal to reproduce bug")
try {
Thread.sleep(1)
} catch {
case e => logError("exception", e)
}
updateInternalState()
> YARN can allocate to many executors
> ---
>
> Key: SPARK-21383
> URL: https://issues.apache.org/jira/browse/SPARK-21383
> Project: Spark
> Issue Type: Bug
> Components: YARN
>Affects Versions: 2.0.0
>Reporter: Thomas Graves
>
> The YarnAllocator doesn't properly track containers being launched but not
> yet running. If it takes time to launch the containers on the NM they don't
> show up as numExecutorsRunning, but they are already out of the Pending list,
> so if the allocateResources call happens again it can think it has missing
> executors even when it doesn't (they just haven't been launched yet).
> This was introduced by SPARK-12447
> Where it check for missing:
> https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala#L297
> Only updates the numRunningExecutors after NM has started it:
> https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala#L524
> Thus if the NM is slow or the network is slow, it can miscount and start
> additional executors.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21383) YARN can allocate to many executors
[
https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16083321#comment-16083321
]
Thomas Graves commented on SPARK-21383:
---
Note we saw this with dynamic allocation off. Its easy to reproduce if you
modify runAllocatedContainers to have something like:
new ExecutorRunnable(
Some(container),
conf,
sparkConf,
driverUrl,
executorId,
executorHostname,
executorMemory,
executorCores,
appAttemptId.getApplicationId.toString,
securityMgr,
localResources
).run()
logInfo("Delaying before updating internal to reproduce bug")
try {
Thread.sleep(1)
} catch {
case e => logError("exception", e)
}
updateInternalState()
> YARN can allocate to many executors
> ---
>
> Key: SPARK-21383
> URL: https://issues.apache.org/jira/browse/SPARK-21383
> Project: Spark
> Issue Type: Bug
> Components: YARN
>Affects Versions: 2.0.0
>Reporter: Thomas Graves
>
> The YarnAllocator doesn't properly track containers being launched but not
> yet running. If it takes time to launch the containers on the NM they don't
> show up as numExecutorsRunning, but they are already out of the Pending list,
> so if the allocateResources call happens again it can think it has missing
> executors even when it doesn't (they just haven't been launched yet).
> This was introduced by SPARK-12447
> Where it check for missing:
> https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala#L297
> Only updates the numRunningExecutors after NM has started it:
> https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala#L524
> Thus if the NM is slow or the network is slow, it can miscount and start
> additional executors.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-21383) YARN can allocate to many executors
[ https://issues.apache.org/jira/browse/SPARK-21383?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-21383: -- Summary: YARN can allocate to many executors (was: YARN: can allocate to many containers) > YARN can allocate to many executors > --- > > Key: SPARK-21383 > URL: https://issues.apache.org/jira/browse/SPARK-21383 > Project: Spark > Issue Type: Bug > Components: YARN >Affects Versions: 2.0.0 >Reporter: Thomas Graves > > The YarnAllocator doesn't properly track containers being launched but not > yet running. If it takes time to launch the containers on the NM they don't > show up as numExecutorsRunning, but they are already out of the Pending list, > so if the allocateResources call happens again it can think it has missing > executors even when it doesn't (they just haven't been launched yet). > This was introduced by SPARK-12447 > Where it check for missing: > https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala#L297 > Only updates the numRunningExecutors after NM has started it: > https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala#L524 > Thus if the NM is slow or the network is slow, it can miscount and start > additional executors. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-21383) YARN: can allocate to many containers
Thomas Graves created SPARK-21383: - Summary: YARN: can allocate to many containers Key: SPARK-21383 URL: https://issues.apache.org/jira/browse/SPARK-21383 Project: Spark Issue Type: Bug Components: YARN Affects Versions: 2.0.0 Reporter: Thomas Graves The YarnAllocator doesn't properly track containers being launched but not yet running. If it takes time to launch the containers on the NM they don't show up as numExecutorsRunning, but they are already out of the Pending list, so if the allocateResources call happens again it can think it has missing executors even when it doesn't (they just haven't been launched yet). This was introduced by SPARK-12447 Where it check for missing: https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala#L297 Only updates the numRunningExecutors after NM has started it: https://github.com/apache/spark/blob/master/resource-managers/yarn/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocator.scala#L524 Thus if the NM is slow or the network is slow, it can miscount and start additional executors. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19659) Fetch big blocks to disk when shuffle-read
[ https://issues.apache.org/jira/browse/SPARK-19659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16080340#comment-16080340 ] Thomas Graves commented on SPARK-19659: --- can you be more specific here, do you mean that if you want to use this feature you have to use the shuffle service from 2.2.0? Is that shuffle service compatible with older versions of Spark? ie spark 1.6 can still talk to 2.2.0 shuffle service. > Fetch big blocks to disk when shuffle-read > -- > > Key: SPARK-19659 > URL: https://issues.apache.org/jira/browse/SPARK-19659 > Project: Spark > Issue Type: Improvement > Components: Shuffle >Affects Versions: 2.1.0 >Reporter: jin xing >Assignee: jin xing > Fix For: 2.2.0 > > Attachments: SPARK-19659-design-v1.pdf, SPARK-19659-design-v2.pdf > > > Currently the whole block is fetched into memory(offheap by default) when > shuffle-read. A block is defined by (shuffleId, mapId, reduceId). Thus it can > be large when skew situations. If OOM happens during shuffle read, job will > be killed and users will be notified to "Consider boosting > spark.yarn.executor.memoryOverhead". Adjusting parameter and allocating more > memory can resolve the OOM. However the approach is not perfectly suitable > for production environment, especially for data warehouse. > Using Spark SQL as data engine in warehouse, users hope to have a unified > parameter(e.g. memory) but less resource wasted(resource is allocated but not > used), > It's not always easy to predict skew situations, when happen, it make sense > to fetch remote blocks to disk for shuffle-read, rather than > kill the job because of OOM. This approach is mentioned during the discussion > in SPARK-3019, by [~sandyr] and [~mridulm80] -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21122) Address starvation issues when dynamic allocation is enabled
[ https://issues.apache.org/jira/browse/SPARK-21122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16075147#comment-16075147 ] Thomas Graves commented on SPARK-21122: --- I agree with Sean on this. if you are aiming this at yarn, this is not the right approach. One application shouldn't be concerned with the resource usage of other applications on the cluster. yarn already has configs for limiting per user resources, queue capacities, etc. It also has preemption and will notify the AM that it needs resources before killing them (in AllocateResponse). See https://github.com/apache/hadoop/blob/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/PreemptionMessage.java We could modify spark on yarn to account for that and to choose what to kill or to save off state. I don't remember all the specifics on this, but I believe yarn give the AM a certain amount of time to choose before it choses for you and just kills what it thinks is best. I don't see anything in the design that supports your statement that applications shouldn't know details about other applications. You say the EAM should poll the RM to see if applications are being starved. This would be a YARN api change. > Address starvation issues when dynamic allocation is enabled > > > Key: SPARK-21122 > URL: https://issues.apache.org/jira/browse/SPARK-21122 > Project: Spark > Issue Type: Improvement > Components: Spark Core, YARN >Affects Versions: 2.2.0, 2.3.0 >Reporter: Craig Ingram > Attachments: Preventing Starvation with Dynamic Allocation Enabled.pdf > > > When dynamic resource allocation is enabled on a cluster, it’s currently > possible for one application to consume all the cluster’s resources, > effectively starving any other application trying to start. This is > particularly painful in a notebook environment where notebooks may be idle > for tens of minutes while the user is figuring out what to do next (or eating > their lunch). Ideally the application should give resources back to the > cluster when monitoring indicates other applications are pending. > Before delving into the specifics of the solution. There are some workarounds > to this problem that are worth mentioning: > * Set spark.dynamicAllocation.maxExecutors to a small value, so that users > are unlikely to use the entire cluster even when many of them are doing work. > This approach will hurt cluster utilization. > * If using YARN, enable preemption and have each application (or > organization) run in a separate queue. The downside of this is that when YARN > preempts, it doesn't know anything about which executor it's killing. It > would just as likely kill a long running executor with cached data as one > that just spun up. Moreover, given a feature like > https://issues.apache.org/jira/browse/SPARK-21097 (preserving cached data on > executor decommission), YARN may not wait long enough between trying to > gracefully and forcefully shut down the executor. This would mean the blocks > that belonged to that executor would be lost and have to be recomputed. > * Configure YARN to use the capacity scheduler with multiple scheduler > queues. Put high-priority notebook users into a high-priority queue. Prevents > high-priority users from being starved out by low-priority notebook users. > Does not prevent users in the same priority class from starving each other. > Obviously any solution to this problem that depends on YARN would leave other > resource managers out in the cold. The solution proposed in this ticket will > afford spark clusters the flexibly to hook in different resource allocation > policies to fulfill their user's needs regardless of resource manager choice. > Initially the focus will be on users in a notebook environment. When > operating in a notebook environment with many users, the goal is fair > resource allocation. Given that all users will be using the same memory > configuration, this solution will focus primarily on fair sharing of cores. > The fair resource allocation policy should pick executors to remove based on > three factors initially: idleness, presence of cached data, and uptime. The > policy will favor removing executors that are idle, short-lived, and have no > cached data. The policy will only preemptively remove executors if there are > pending applications or cores (otherwise the default dynamic allocation > timeout/removal process is followed). The policy will also allow an > application's resource consumption to expand based on cluster utilization. > For example if there are 3 applications running but 2 of them are idle, the > policy will allow a busy application with pending tasks to consume more than > 1/3rd of the
[jira] [Resolved] (SPARK-13669) Job will always fail in the external shuffle service unavailable situation
[ https://issues.apache.org/jira/browse/SPARK-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-13669. --- Resolution: Fixed Fix Version/s: 2.3.0 > Job will always fail in the external shuffle service unavailable situation > -- > > Key: SPARK-13669 > URL: https://issues.apache.org/jira/browse/SPARK-13669 > Project: Spark > Issue Type: Bug > Components: Spark Core, YARN >Reporter: Saisai Shao >Assignee: Saisai Shao > Fix For: 2.3.0 > > > Currently we are running into an issue with Yarn work preserving enabled + > external shuffle service. > In the work preserving enabled scenario, the failure of NM will not lead to > the exit of executors, so executors can still accept and run the tasks. The > problem here is when NM is failed, external shuffle service is actually > inaccessible, so reduce tasks will always complain about the “Fetch failure”, > and the failure of reduce stage will make the parent stage (map stage) rerun. > The tricky thing here is Spark scheduler is not aware of the unavailability > of external shuffle service, and will reschedule the map tasks on the > executor where NM is failed, and again reduce stage will be failed with > “Fetch failure”, and after 4 retries, the job is failed. > So here the main problem is that we should avoid assigning tasks to those bad > executors (where shuffle service is unavailable). Current Spark's blacklist > mechanism could blacklist executors/nodes by failure tasks, but it doesn't > handle this specific fetch failure scenario. So here propose to improve the > current application blacklist mechanism to handle fetch failure issue > (especially with external shuffle service unavailable issue), to blacklist > the executors/nodes where shuffle fetch is unavailable. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-13669) Job will always fail in the external shuffle service unavailable situation
[ https://issues.apache.org/jira/browse/SPARK-13669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-13669: - Assignee: Saisai Shao > Job will always fail in the external shuffle service unavailable situation > -- > > Key: SPARK-13669 > URL: https://issues.apache.org/jira/browse/SPARK-13669 > Project: Spark > Issue Type: Bug > Components: Spark Core, YARN >Reporter: Saisai Shao >Assignee: Saisai Shao > Fix For: 2.3.0 > > > Currently we are running into an issue with Yarn work preserving enabled + > external shuffle service. > In the work preserving enabled scenario, the failure of NM will not lead to > the exit of executors, so executors can still accept and run the tasks. The > problem here is when NM is failed, external shuffle service is actually > inaccessible, so reduce tasks will always complain about the “Fetch failure”, > and the failure of reduce stage will make the parent stage (map stage) rerun. > The tricky thing here is Spark scheduler is not aware of the unavailability > of external shuffle service, and will reschedule the map tasks on the > executor where NM is failed, and again reduce stage will be failed with > “Fetch failure”, and after 4 retries, the job is failed. > So here the main problem is that we should avoid assigning tasks to those bad > executors (where shuffle service is unavailable). Current Spark's blacklist > mechanism could blacklist executors/nodes by failure tasks, but it doesn't > handle this specific fetch failure scenario. So here propose to improve the > current application blacklist mechanism to handle fetch failure issue > (especially with external shuffle service unavailable issue), to blacklist > the executors/nodes where shuffle fetch is unavailable. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-20898) spark.blacklist.killBlacklistedExecutors doesn't work in YARN
[ https://issues.apache.org/jira/browse/SPARK-20898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-20898: - Assignee: Saisai Shao > spark.blacklist.killBlacklistedExecutors doesn't work in YARN > - > > Key: SPARK-20898 > URL: https://issues.apache.org/jira/browse/SPARK-20898 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.2.0 >Reporter: Thomas Graves >Assignee: Saisai Shao > Fix For: 2.3.0 > > > I was trying out the new spark.blacklist.killBlacklistedExecutors on YARN but > it doesn't appear to work. Everytime I get: > 17/05/26 16:28:12 WARN BlacklistTracker: Not attempting to kill blacklisted > executor id 4 since allocation client is not defined > Even though dynamic allocation is on. Taking a quick look, I think the way > it creates the blacklisttracker and passes the allocation client is wrong. > The scheduler backend is > not set yet so it never passes the allocation client to the blacklisttracker > correctly. Thus it will never kill. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-20898) spark.blacklist.killBlacklistedExecutors doesn't work in YARN
[ https://issues.apache.org/jira/browse/SPARK-20898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-20898. --- Resolution: Fixed Fix Version/s: 2.3.0 > spark.blacklist.killBlacklistedExecutors doesn't work in YARN > - > > Key: SPARK-20898 > URL: https://issues.apache.org/jira/browse/SPARK-20898 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.2.0 >Reporter: Thomas Graves >Assignee: Saisai Shao > Fix For: 2.3.0 > > > I was trying out the new spark.blacklist.killBlacklistedExecutors on YARN but > it doesn't appear to work. Everytime I get: > 17/05/26 16:28:12 WARN BlacklistTracker: Not attempting to kill blacklisted > executor id 4 since allocation client is not defined > Even though dynamic allocation is on. Taking a quick look, I think the way > it creates the blacklisttracker and passes the allocation client is wrong. > The scheduler backend is > not set yet so it never passes the allocation client to the blacklisttracker > correctly. Thus it will never kill. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-21082) Consider Executor's memory usage when scheduling task
[ https://issues.apache.org/jira/browse/SPARK-21082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16056118#comment-16056118 ] Thomas Graves commented on SPARK-21082: --- Yes executor should not oom if you are trying to cache to much data there. OOM generally comes from the task side processing data. Like mentioned there are a lot of reasons for cached data to be skewed. How are you proposing spark to figure this out? Most of the time spark is not going to know how much data a task is going to generate and cache. Generally its just a users tasks that reads some data and then caching it. That could just be naturally skewed if the user doesn't have customer partitioner or handle that skew. I guess what you are saying is to try to distribute the tasks equally among the executors which COULD result in a more equal distribution of cached data. I would normally expect this to happen if you already have the executors and you don't have locality wait on. What is the reason you are getting the skewed cached results? Are you really asking for spark to handle skewed data better? > Consider Executor's memory usage when scheduling task > -- > > Key: SPARK-21082 > URL: https://issues.apache.org/jira/browse/SPARK-21082 > Project: Spark > Issue Type: Improvement > Components: Scheduler, Spark Core >Affects Versions: 2.3.0 >Reporter: DjvuLee > > Spark Scheduler do not consider the memory usage during dispatch tasks, this > can lead to Executor OOM if the RDD is cached sometimes, because Spark can > not estimate the memory usage well enough(especially when the RDD type is not > flatten), scheduler may dispatch so many tasks on one Executor. > We can offer a configuration for user to decide whether scheduler will > consider the memory usage. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16049228#comment-16049228 ] Thomas Graves commented on SPARK-20589: --- [~Robin Shao] can you please clarify your comment? What do you mean by simultaneous task number?what contradiction are you referring to > Allow limiting task concurrency per stage > - > > Key: SPARK-20589 > URL: https://issues.apache.org/jira/browse/SPARK-20589 > Project: Spark > Issue Type: Improvement > Components: Scheduler >Affects Versions: 2.1.0 >Reporter: Thomas Graves > > It would be nice to have the ability to limit the number of concurrent tasks > per stage. This is useful when your spark job might be accessing another > service and you don't want to DOS that service. For instance Spark writing > to hbase or Spark doing http puts on a service. Many times you want to do > this without limiting the number of partitions. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-20970) Deprecate TaskMetrics._updatedBlockStatuses
Thomas Graves created SPARK-20970: - Summary: Deprecate TaskMetrics._updatedBlockStatuses Key: SPARK-20970 URL: https://issues.apache.org/jira/browse/SPARK-20970 Project: Spark Issue Type: Improvement Components: Spark Core Affects Versions: 2.2.0 Reporter: Thomas Graves TaskMetrics._updatedBlockStatuses isn't used anywhere internally by spark. It could be used by users though since its exposed by SparkListenerTaskEnd. We made it configurable to turn off the tracking of it since it uses a lot of memory in https://issues.apache.org/jira/browse/SPARK-20923. That config is still true for backwards compatibility. We should turn that to false in next release and deprecate that api altogether. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20923) TaskMetrics._updatedBlockStatuses uses a lot of memory
[ https://issues.apache.org/jira/browse/SPARK-20923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16030059#comment-16030059 ] Thomas Graves commented on SPARK-20923: --- taking a quick look at the history of the _updatedBlockStatuses it looks like this used to be used for StorageStatusListener but it has been since changed to do this on the SparkListenerBlockUpdated event. That BlockUpdated event is coming from the BlockManagerMaster.updateBlockInfo which is called by the executors. So I'm not seeing anything use _updatedBlockStatuses. I'll start to rip it out and see what I hit. > TaskMetrics._updatedBlockStatuses uses a lot of memory > -- > > Key: SPARK-20923 > URL: https://issues.apache.org/jira/browse/SPARK-20923 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.0 >Reporter: Thomas Graves > > The driver appears to use a ton of memory in certain cases to store the task > metrics updated block status'. For instance I had a user reading data form > hive and caching it. The # of tasks to read was around 62,000, they were > using 1000 executors and it ended up caching a couple TB's of data. The > driver kept running out of memory. > I investigated and it looks like there was 5GB of a 10GB heap being used up > by the TaskMetrics._updatedBlockStatuses because there are a lot of blocks. > The updatedBlockStatuses was already removed from the task end event under > SPARK-20084. I don't see anything else that seems to be using this. Anybody > know if I missed something? > If its not being used we should remove it, otherwise we need to figure out a > better way of doing it so it doesn't use so much memory. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20923) TaskMetrics._updatedBlockStatuses uses a lot of memory
[ https://issues.apache.org/jira/browse/SPARK-20923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029917#comment-16029917 ] Thomas Graves commented on SPARK-20923: --- [~joshrosen] [~zsxwing] [~eseyfe] I think you have looked at this fairly recently, do you know if this is used by anything or anybody? I'm not finding it used anywhere in the code or UI but maybe I'm missing some obscure reference > TaskMetrics._updatedBlockStatuses uses a lot of memory > -- > > Key: SPARK-20923 > URL: https://issues.apache.org/jira/browse/SPARK-20923 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.0 >Reporter: Thomas Graves > > The driver appears to use a ton of memory in certain cases to store the task > metrics updated block status'. For instance I had a user reading data form > hive and caching it. The # of tasks to read was around 62,000, they were > using 1000 executors and it ended up caching a couple TB's of data. The > driver kept running out of memory. > I investigated and it looks like there was 5GB of a 10GB heap being used up > by the TaskMetrics._updatedBlockStatuses because there are a lot of blocks. > The updatedBlockStatuses was already removed from the task end event under > SPARK-20084. I don't see anything else that seems to be using this. Anybody > know if I missed something? > If its not being used we should remove it, otherwise we need to figure out a > better way of doing it so it doesn't use so much memory. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20923) TaskMetrics._updatedBlockStatuses uses a lot of memory
[ https://issues.apache.org/jira/browse/SPARK-20923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029701#comment-16029701 ] Thomas Graves commented on SPARK-20923: --- [~rdblue] with SPARK-20084, did you see anything using these updatedblockStatuses in TaskMetrics? > TaskMetrics._updatedBlockStatuses uses a lot of memory > -- > > Key: SPARK-20923 > URL: https://issues.apache.org/jira/browse/SPARK-20923 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.0 >Reporter: Thomas Graves > > The driver appears to use a ton of memory in certain cases to store the task > metrics updated block status'. For instance I had a user reading data form > hive and caching it. The # of tasks to read was around 62,000, they were > using 1000 executors and it ended up caching a couple TB's of data. The > driver kept running out of memory. > I investigated and it looks like there was 5GB of a 10GB heap being used up > by the TaskMetrics._updatedBlockStatuses because there are a lot of blocks. > The updatedBlockStatuses was already removed from the task end event under > SPARK-20084. I don't see anything else that seems to be using this. Anybody > know if I missed something? > If its not being used we should remove it, otherwise we need to figure out a > better way of doing it so it doesn't use so much memory. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-20923) TaskMetrics._updatedBlockStatuses uses a lot of memory
Thomas Graves created SPARK-20923: - Summary: TaskMetrics._updatedBlockStatuses uses a lot of memory Key: SPARK-20923 URL: https://issues.apache.org/jira/browse/SPARK-20923 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.1.0 Reporter: Thomas Graves The driver appears to use a ton of memory in certain cases to store the task metrics updated block status'. For instance I had a user reading data form hive and caching it. The # of tasks to read was around 62,000, they were using 1000 executors and it ended up caching a couple TB's of data. The driver kept running out of memory. I investigated and it looks like there was 5GB of a 10GB heap being used up by the TaskMetrics._updatedBlockStatuses because there are a lot of blocks. The updatedBlockStatuses was already removed from the task end event under SPARK-20084. I don't see anything else that seems to be using this. Anybody know if I missed something? If its not being used we should remove it, otherwise we need to figure out a better way of doing it so it doesn't use so much memory. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures
[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16029425#comment-16029425 ] Thomas Graves commented on SPARK-20178: --- Yeah I think we should do something here. I never looked at the code details closely in https://github.com/apache/spark/pull/17088 but you can resurrect and we can look at in more detail. [~joshrosen] are you ok with that or did you have something else in progress? > Improve Scheduler fetch failures > > > Key: SPARK-20178 > URL: https://issues.apache.org/jira/browse/SPARK-20178 > Project: Spark > Issue Type: Epic > Components: Scheduler >Affects Versions: 2.1.0 >Reporter: Thomas Graves > > We have been having a lot of discussions around improving the handling of > fetch failures. There are 4 jira currently related to this. > We should try to get a list of things we want to improve and come up with one > cohesive design. > SPARK-20163, SPARK-20091, SPARK-14649 , and SPARK-19753 > I will put my initial thoughts in a follow on comment. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-20898) spark.blacklist.killBlacklistedExecutors doesn't work in YARN
Thomas Graves created SPARK-20898: - Summary: spark.blacklist.killBlacklistedExecutors doesn't work in YARN Key: SPARK-20898 URL: https://issues.apache.org/jira/browse/SPARK-20898 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.2.0 Reporter: Thomas Graves I was trying out the new spark.blacklist.killBlacklistedExecutors on YARN but it doesn't appear to work. Everytime I get: 17/05/26 16:28:12 WARN BlacklistTracker: Not attempting to kill blacklisted executor id 4 since allocation client is not defined Even though dynamic allocation is on. Taking a quick look, I think the way it creates the blacklisttracker and passes the allocation client is wrong. The scheduler backend is not set yet so it never passes the allocation client to the blacklisttracker correctly. Thus it will never kill. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures
[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16021199#comment-16021199 ] Thomas Graves commented on SPARK-20178: --- | My understanding of today's code is that a single FetchFailed task will trigger a stage failure and parent stage retry and that the task which experienced the fetch failure will not be retried within the same task set that scheduled it. I'm basing this off the comment at https://github.com/apache/spark/blob/9b09101938399a3490c3c9bde9e5f07031140fdf/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala#L77 and the code at https://github.com/apache/spark/blob/9b09101938399a3490c3c9bde9e5f07031140fdf/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala#L770 where the TSM prevents re-attempts of FetchFailed tasks. That is correct but that doesn't mean we can't track the fetch failures on a host across stages. You may or may not get multiple fetch failures in the first stage before it is aborted (very timing dependent) so you are correct that you can't rely on that. But if you track those across stage attempts and if the max is set to 2 or 3 then it will clear the entire host before the 4 default stage failures. This might give us a little more confidence its a hard failure vs a transient failure. But that does take extra tracking and right now I don't have a good measure of metrics to tell me how many of different kinds of failures. So to get the robustness for now I'm fine with just invalidating it immediately and see how that works. > Improve Scheduler fetch failures > > > Key: SPARK-20178 > URL: https://issues.apache.org/jira/browse/SPARK-20178 > Project: Spark > Issue Type: Epic > Components: Scheduler >Affects Versions: 2.1.0 >Reporter: Thomas Graves > > We have been having a lot of discussions around improving the handling of > fetch failures. There are 4 jira currently related to this. > We should try to get a list of things we want to improve and come up with one > cohesive design. > SPARK-20163, SPARK-20091, SPARK-14649 , and SPARK-19753 > I will put my initial thoughts in a follow on comment. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures
[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16019650#comment-16019650 ] Thomas Graves commented on SPARK-20178: --- | when the DAGScheduler is notified of a FetchFailure from a node then mark that shuffle's output locations on that node as unavailable (rather than all shuffles' outputs on that node) Can you please clarify this? Do you mean only the map stage output locations for that particular shuffle id that got the fetch failure? | the likelihood of the FetchFailure being a transient failure is relatively small Unfortunately this is not what we have seen in the past with mapreduce and tez. Both of those used to be very aggressive about invalidating outputs and such and we had to tune those back because we do see a lot of transient failures and caused a lot of extra work and delays. But I do agree that this one is a lot of work and will take some time to implement and based on the way things work now we should do something shorter term as well. I was actually just looking at this more last friday because I am seeing more of these fetch failures cause job failures so something needs to be done. Based on the way we handle fetch failures now invalidating all for that stage and causing it to take longer is better then job failure. I was thinking we could invalidate all the map outputs on that host for that map stage after a certain number of failures, which could be across attempts. I think this is basically what you are proposing except waiting for a configurable amount of failures rather then doing it immediately. Thoughts? I also think adding blacklisting after a certain number of fetch failures would be good for those cases where the YARN NM crashes but there could still be executors running on the node. That one isn't as difficult, you just have to track it in the BlacklistTracker. Work started in SPARK-13669 it would just need tracking again of multiple failures. > Improve Scheduler fetch failures > > > Key: SPARK-20178 > URL: https://issues.apache.org/jira/browse/SPARK-20178 > Project: Spark > Issue Type: Epic > Components: Scheduler >Affects Versions: 2.1.0 >Reporter: Thomas Graves > > We have been having a lot of discussions around improving the handling of > fetch failures. There are 4 jira currently related to this. > We should try to get a list of things we want to improve and come up with one > cohesive design. > SPARK-20163, SPARK-20091, SPARK-14649 , and SPARK-19753 > I will put my initial thoughts in a follow on comment. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19354) Killed tasks are getting marked as FAILED
[
https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16006549#comment-16006549
]
Thomas Graves commented on SPARK-19354:
---
just an fyi, filed https://issues.apache.org/jira/browse/SPARK-20713 for the
other issue I mention.
> Killed tasks are getting marked as FAILED
> -
>
> Key: SPARK-19354
> URL: https://issues.apache.org/jira/browse/SPARK-19354
> Project: Spark
> Issue Type: Bug
> Components: Scheduler, Spark Core
>Reporter: Devaraj K
>
> When we enable speculation, we can see there are multiple attempts running
> for the same task when the first task progress is slow. If any of the task
> attempt succeeds then the other attempts will be killed, during killing the
> attempts those attempts are getting marked as failed due to the below error.
> We need to handle this error and mark the attempt as KILLED instead of FAILED.
> ||93 ||214 ||1 (speculative) ||FAILED||ANY ||1 /
> xx.xx.xx.x2
> stdout
> stderr||2017/01/24 10:30:44 ||0.2 s ||0.0 B / 0 ||8.0 KB / 400
> ||java.io.IOException: Failed on local exception:
> java.nio.channels.ClosedByInterruptException; Host Details : local host is:
> node2/xx.xx.xx.x2; destination host is: node1:9000;
> +details||
> {code:xml}
> 17/01/23 23:54:32 INFO Executor: Executor is trying to kill task 93.1 in
> stage 1.0 (TID 214)
> 17/01/23 23:54:32 INFO FileOutputCommitter: File Output Committer Algorithm
> version is 1
> 17/01/23 23:54:32 ERROR Executor: Exception in task 93.1 in stage 1.0 (TID
> 214)
> java.io.IOException: Failed on local exception:
> java.nio.channels.ClosedByInterruptException; Host Details : local host is:
> "stobdtserver3/10.224.54.70"; destination host is: "stobdtserver2":9000;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> at org.apache.hadoop.ipc.Client.call(Client.java:1479)
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy17.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy18.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
> at
> org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
> at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:90)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1133)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1124)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:88)
> at org.apache.spark.scheduler.Task.run(Task.scala:114)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at
[jira] [Created] (SPARK-20713) Speculative task that got CommitDenied exception shows up as failed
Thomas Graves created SPARK-20713: - Summary: Speculative task that got CommitDenied exception shows up as failed Key: SPARK-20713 URL: https://issues.apache.org/jira/browse/SPARK-20713 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.1.1 Reporter: Thomas Graves When running speculative tasks you can end up getting a task failure on a speculative task (the other task succeeded) because that task got a CommitDenied exception when really it was "killed" by the driver. It is a race between when the driver kills and when the executor tries to commit. I think ideally we should fix up the task state on this to be killed because the fact that this task failed doesn't matter since the other speculative task succeeded. tasks showing up as failure confuse the user and could make other scheduler cases harder. This is somewhat related to SPARK-13343 where I think we should be correctly account for speculative tasks. only one of the 2 tasks really succeeded and commited, and the other should be marked differently. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-19354) Killed tasks are getting marked as FAILED
[
https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves closed SPARK-19354.
-
Resolution: Duplicate
> Killed tasks are getting marked as FAILED
> -
>
> Key: SPARK-19354
> URL: https://issues.apache.org/jira/browse/SPARK-19354
> Project: Spark
> Issue Type: Bug
> Components: Scheduler, Spark Core
>Reporter: Devaraj K
>
> When we enable speculation, we can see there are multiple attempts running
> for the same task when the first task progress is slow. If any of the task
> attempt succeeds then the other attempts will be killed, during killing the
> attempts those attempts are getting marked as failed due to the below error.
> We need to handle this error and mark the attempt as KILLED instead of FAILED.
> ||93 ||214 ||1 (speculative) ||FAILED||ANY ||1 /
> xx.xx.xx.x2
> stdout
> stderr||2017/01/24 10:30:44 ||0.2 s ||0.0 B / 0 ||8.0 KB / 400
> ||java.io.IOException: Failed on local exception:
> java.nio.channels.ClosedByInterruptException; Host Details : local host is:
> node2/xx.xx.xx.x2; destination host is: node1:9000;
> +details||
> {code:xml}
> 17/01/23 23:54:32 INFO Executor: Executor is trying to kill task 93.1 in
> stage 1.0 (TID 214)
> 17/01/23 23:54:32 INFO FileOutputCommitter: File Output Committer Algorithm
> version is 1
> 17/01/23 23:54:32 ERROR Executor: Exception in task 93.1 in stage 1.0 (TID
> 214)
> java.io.IOException: Failed on local exception:
> java.nio.channels.ClosedByInterruptException; Host Details : local host is:
> "stobdtserver3/10.224.54.70"; destination host is: "stobdtserver2":9000;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> at org.apache.hadoop.ipc.Client.call(Client.java:1479)
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy17.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy18.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
> at
> org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
> at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:90)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1133)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1124)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:88)
> at org.apache.spark.scheduler.Task.run(Task.scala:114)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:659)
> at
>
[jira] [Commented] (SPARK-19354) Killed tasks are getting marked as FAILED
[
https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16006464#comment-16006464
]
Thomas Graves commented on SPARK-19354:
---
thanks for pointing those out, that does fix this issue, I will dup this to
that. To bad they didn't pull that back to 2.1.
There is still one case tasks show up as failed when killed, which is sometimes
with TaskCommitDenied. It doesn't affect the blacklisting since it doesn't
countTowardsTaskFailures though. I'll look at this again and maybe file a
separate jira for that if it seems like something we should fix.
> Killed tasks are getting marked as FAILED
> -
>
> Key: SPARK-19354
> URL: https://issues.apache.org/jira/browse/SPARK-19354
> Project: Spark
> Issue Type: Bug
> Components: Scheduler, Spark Core
>Reporter: Devaraj K
>
> When we enable speculation, we can see there are multiple attempts running
> for the same task when the first task progress is slow. If any of the task
> attempt succeeds then the other attempts will be killed, during killing the
> attempts those attempts are getting marked as failed due to the below error.
> We need to handle this error and mark the attempt as KILLED instead of FAILED.
> ||93 ||214 ||1 (speculative) ||FAILED||ANY ||1 /
> xx.xx.xx.x2
> stdout
> stderr||2017/01/24 10:30:44 ||0.2 s ||0.0 B / 0 ||8.0 KB / 400
> ||java.io.IOException: Failed on local exception:
> java.nio.channels.ClosedByInterruptException; Host Details : local host is:
> node2/xx.xx.xx.x2; destination host is: node1:9000;
> +details||
> {code:xml}
> 17/01/23 23:54:32 INFO Executor: Executor is trying to kill task 93.1 in
> stage 1.0 (TID 214)
> 17/01/23 23:54:32 INFO FileOutputCommitter: File Output Committer Algorithm
> version is 1
> 17/01/23 23:54:32 ERROR Executor: Exception in task 93.1 in stage 1.0 (TID
> 214)
> java.io.IOException: Failed on local exception:
> java.nio.channels.ClosedByInterruptException; Host Details : local host is:
> "stobdtserver3/10.224.54.70"; destination host is: "stobdtserver2":9000;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> at org.apache.hadoop.ipc.Client.call(Client.java:1479)
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy17.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy18.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
> at
> org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
> at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:90)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1133)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1124)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:88)
> at org.apache.spark.scheduler.Task.run(Task.scala:114)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
[jira] [Commented] (SPARK-19354) Killed tasks are getting marked as FAILED
[
https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005817#comment-16005817
]
Thomas Graves commented on SPARK-19354:
---
Right from what I've seen not a blacklisting bug. Bug with speculative tasks
being marked as failed rather then killed which then leads to the executor
being blacklisted.
Not sure on the oom part never saw that.
> Killed tasks are getting marked as FAILED
> -
>
> Key: SPARK-19354
> URL: https://issues.apache.org/jira/browse/SPARK-19354
> Project: Spark
> Issue Type: Bug
> Components: Scheduler, Spark Core
>Reporter: Devaraj K
>
> When we enable speculation, we can see there are multiple attempts running
> for the same task when the first task progress is slow. If any of the task
> attempt succeeds then the other attempts will be killed, during killing the
> attempts those attempts are getting marked as failed due to the below error.
> We need to handle this error and mark the attempt as KILLED instead of FAILED.
> ||93 ||214 ||1 (speculative) ||FAILED||ANY ||1 /
> xx.xx.xx.x2
> stdout
> stderr||2017/01/24 10:30:44 ||0.2 s ||0.0 B / 0 ||8.0 KB / 400
> ||java.io.IOException: Failed on local exception:
> java.nio.channels.ClosedByInterruptException; Host Details : local host is:
> node2/xx.xx.xx.x2; destination host is: node1:9000;
> +details||
> {code:xml}
> 17/01/23 23:54:32 INFO Executor: Executor is trying to kill task 93.1 in
> stage 1.0 (TID 214)
> 17/01/23 23:54:32 INFO FileOutputCommitter: File Output Committer Algorithm
> version is 1
> 17/01/23 23:54:32 ERROR Executor: Exception in task 93.1 in stage 1.0 (TID
> 214)
> java.io.IOException: Failed on local exception:
> java.nio.channels.ClosedByInterruptException; Host Details : local host is:
> "stobdtserver3/10.224.54.70"; destination host is: "stobdtserver2":9000;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> at org.apache.hadoop.ipc.Client.call(Client.java:1479)
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy17.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy18.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
> at
> org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
> at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:90)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1133)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1124)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:88)
> at org.apache.spark.scheduler.Task.run(Task.scala:114)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.channels.ClosedByInterruptException
> at
>
[jira] [Comment Edited] (SPARK-19354) Killed tasks are getting marked as FAILED
[
https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005189#comment-16005189
]
Thomas Graves edited comment on SPARK-19354 at 5/10/17 6:37 PM:
[~irashid] wondering if you have seen the issue with blacklisting or perhaps
there is another fix for that already?
was (Author: tgraves):
[~squito] wondering if you have seen the issue with blacklisting or perhaps
there is another fix for that already?
> Killed tasks are getting marked as FAILED
> -
>
> Key: SPARK-19354
> URL: https://issues.apache.org/jira/browse/SPARK-19354
> Project: Spark
> Issue Type: Bug
> Components: Scheduler, Spark Core
>Reporter: Devaraj K
>
> When we enable speculation, we can see there are multiple attempts running
> for the same task when the first task progress is slow. If any of the task
> attempt succeeds then the other attempts will be killed, during killing the
> attempts those attempts are getting marked as failed due to the below error.
> We need to handle this error and mark the attempt as KILLED instead of FAILED.
> ||93 ||214 ||1 (speculative) ||FAILED||ANY ||1 /
> xx.xx.xx.x2
> stdout
> stderr||2017/01/24 10:30:44 ||0.2 s ||0.0 B / 0 ||8.0 KB / 400
> ||java.io.IOException: Failed on local exception:
> java.nio.channels.ClosedByInterruptException; Host Details : local host is:
> node2/xx.xx.xx.x2; destination host is: node1:9000;
> +details||
> {code:xml}
> 17/01/23 23:54:32 INFO Executor: Executor is trying to kill task 93.1 in
> stage 1.0 (TID 214)
> 17/01/23 23:54:32 INFO FileOutputCommitter: File Output Committer Algorithm
> version is 1
> 17/01/23 23:54:32 ERROR Executor: Exception in task 93.1 in stage 1.0 (TID
> 214)
> java.io.IOException: Failed on local exception:
> java.nio.channels.ClosedByInterruptException; Host Details : local host is:
> "stobdtserver3/10.224.54.70"; destination host is: "stobdtserver2":9000;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> at org.apache.hadoop.ipc.Client.call(Client.java:1479)
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy17.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy18.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
> at
> org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
> at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:90)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1133)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1124)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:88)
> at org.apache.spark.scheduler.Task.run(Task.scala:114)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at
[jira] [Updated] (SPARK-19354) Killed tasks are getting marked as FAILED
[
https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves updated SPARK-19354:
--
Priority: Major (was: Minor)
> Killed tasks are getting marked as FAILED
> -
>
> Key: SPARK-19354
> URL: https://issues.apache.org/jira/browse/SPARK-19354
> Project: Spark
> Issue Type: Bug
> Components: Scheduler, Spark Core
>Reporter: Devaraj K
>
> When we enable speculation, we can see there are multiple attempts running
> for the same task when the first task progress is slow. If any of the task
> attempt succeeds then the other attempts will be killed, during killing the
> attempts those attempts are getting marked as failed due to the below error.
> We need to handle this error and mark the attempt as KILLED instead of FAILED.
> ||93 ||214 ||1 (speculative) ||FAILED||ANY ||1 /
> xx.xx.xx.x2
> stdout
> stderr||2017/01/24 10:30:44 ||0.2 s ||0.0 B / 0 ||8.0 KB / 400
> ||java.io.IOException: Failed on local exception:
> java.nio.channels.ClosedByInterruptException; Host Details : local host is:
> node2/xx.xx.xx.x2; destination host is: node1:9000;
> +details||
> {code:xml}
> 17/01/23 23:54:32 INFO Executor: Executor is trying to kill task 93.1 in
> stage 1.0 (TID 214)
> 17/01/23 23:54:32 INFO FileOutputCommitter: File Output Committer Algorithm
> version is 1
> 17/01/23 23:54:32 ERROR Executor: Exception in task 93.1 in stage 1.0 (TID
> 214)
> java.io.IOException: Failed on local exception:
> java.nio.channels.ClosedByInterruptException; Host Details : local host is:
> "stobdtserver3/10.224.54.70"; destination host is: "stobdtserver2":9000;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> at org.apache.hadoop.ipc.Client.call(Client.java:1479)
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy17.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy18.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
> at
> org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
> at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:90)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1133)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1124)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:88)
> at org.apache.spark.scheduler.Task.run(Task.scala:114)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:659)
> at
>
[jira] [Commented] (SPARK-19354) Killed tasks are getting marked as FAILED
[
https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005189#comment-16005189
]
Thomas Graves commented on SPARK-19354:
---
[~squito] wondering if you have seen the issue with blacklisting or perhaps
there is another fix for that already?
> Killed tasks are getting marked as FAILED
> -
>
> Key: SPARK-19354
> URL: https://issues.apache.org/jira/browse/SPARK-19354
> Project: Spark
> Issue Type: Bug
> Components: Scheduler, Spark Core
>Reporter: Devaraj K
>
> When we enable speculation, we can see there are multiple attempts running
> for the same task when the first task progress is slow. If any of the task
> attempt succeeds then the other attempts will be killed, during killing the
> attempts those attempts are getting marked as failed due to the below error.
> We need to handle this error and mark the attempt as KILLED instead of FAILED.
> ||93 ||214 ||1 (speculative) ||FAILED||ANY ||1 /
> xx.xx.xx.x2
> stdout
> stderr||2017/01/24 10:30:44 ||0.2 s ||0.0 B / 0 ||8.0 KB / 400
> ||java.io.IOException: Failed on local exception:
> java.nio.channels.ClosedByInterruptException; Host Details : local host is:
> node2/xx.xx.xx.x2; destination host is: node1:9000;
> +details||
> {code:xml}
> 17/01/23 23:54:32 INFO Executor: Executor is trying to kill task 93.1 in
> stage 1.0 (TID 214)
> 17/01/23 23:54:32 INFO FileOutputCommitter: File Output Committer Algorithm
> version is 1
> 17/01/23 23:54:32 ERROR Executor: Exception in task 93.1 in stage 1.0 (TID
> 214)
> java.io.IOException: Failed on local exception:
> java.nio.channels.ClosedByInterruptException; Host Details : local host is:
> "stobdtserver3/10.224.54.70"; destination host is: "stobdtserver2":9000;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> at org.apache.hadoop.ipc.Client.call(Client.java:1479)
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy17.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy18.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
> at
> org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
> at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:90)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1133)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1124)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:88)
> at org.apache.spark.scheduler.Task.run(Task.scala:114)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at
[jira] [Updated] (SPARK-19354) Killed tasks are getting marked as FAILED
[
https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves updated SPARK-19354:
--
Issue Type: Bug (was: Improvement)
> Killed tasks are getting marked as FAILED
> -
>
> Key: SPARK-19354
> URL: https://issues.apache.org/jira/browse/SPARK-19354
> Project: Spark
> Issue Type: Bug
> Components: Scheduler, Spark Core
>Reporter: Devaraj K
>Priority: Minor
>
> When we enable speculation, we can see there are multiple attempts running
> for the same task when the first task progress is slow. If any of the task
> attempt succeeds then the other attempts will be killed, during killing the
> attempts those attempts are getting marked as failed due to the below error.
> We need to handle this error and mark the attempt as KILLED instead of FAILED.
> ||93 ||214 ||1 (speculative) ||FAILED||ANY ||1 /
> xx.xx.xx.x2
> stdout
> stderr||2017/01/24 10:30:44 ||0.2 s ||0.0 B / 0 ||8.0 KB / 400
> ||java.io.IOException: Failed on local exception:
> java.nio.channels.ClosedByInterruptException; Host Details : local host is:
> node2/xx.xx.xx.x2; destination host is: node1:9000;
> +details||
> {code:xml}
> 17/01/23 23:54:32 INFO Executor: Executor is trying to kill task 93.1 in
> stage 1.0 (TID 214)
> 17/01/23 23:54:32 INFO FileOutputCommitter: File Output Committer Algorithm
> version is 1
> 17/01/23 23:54:32 ERROR Executor: Exception in task 93.1 in stage 1.0 (TID
> 214)
> java.io.IOException: Failed on local exception:
> java.nio.channels.ClosedByInterruptException; Host Details : local host is:
> "stobdtserver3/10.224.54.70"; destination host is: "stobdtserver2":9000;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> at org.apache.hadoop.ipc.Client.call(Client.java:1479)
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy17.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy18.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
> at
> org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
> at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:90)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1133)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1124)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:88)
> at org.apache.spark.scheduler.Task.run(Task.scala:114)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: java.nio.channels.ClosedByInterruptException
> at
> java.nio.channels.spi.AbstractInterruptibleChannel.end(AbstractInterruptibleChannel.java:202)
> at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:659)
> at
>
[jira] [Commented] (SPARK-19354) Killed tasks are getting marked as FAILED
[
https://issues.apache.org/jira/browse/SPARK-19354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16005182#comment-16005182
]
Thomas Graves commented on SPARK-19354:
---
This is definitely causing issues with blacklisting. speculative tasks can
come back with failed like:
17/05/10 17:14:38 ERROR Executor: Exception in task 71476.0 in stage 52.0 (TID
317274)
java.nio.channels.ClosedByInterruptException
Then executor gets marked as blacklisted:
https://github.com/apache/spark/blob/branch-2.1/core/src/main/scala/org/apache/spark/scheduler/TaskSetManager.scala#L793
> Killed tasks are getting marked as FAILED
> -
>
> Key: SPARK-19354
> URL: https://issues.apache.org/jira/browse/SPARK-19354
> Project: Spark
> Issue Type: Improvement
> Components: Scheduler, Spark Core
>Reporter: Devaraj K
>Priority: Minor
>
> When we enable speculation, we can see there are multiple attempts running
> for the same task when the first task progress is slow. If any of the task
> attempt succeeds then the other attempts will be killed, during killing the
> attempts those attempts are getting marked as failed due to the below error.
> We need to handle this error and mark the attempt as KILLED instead of FAILED.
> ||93 ||214 ||1 (speculative) ||FAILED||ANY ||1 /
> xx.xx.xx.x2
> stdout
> stderr||2017/01/24 10:30:44 ||0.2 s ||0.0 B / 0 ||8.0 KB / 400
> ||java.io.IOException: Failed on local exception:
> java.nio.channels.ClosedByInterruptException; Host Details : local host is:
> node2/xx.xx.xx.x2; destination host is: node1:9000;
> +details||
> {code:xml}
> 17/01/23 23:54:32 INFO Executor: Executor is trying to kill task 93.1 in
> stage 1.0 (TID 214)
> 17/01/23 23:54:32 INFO FileOutputCommitter: File Output Committer Algorithm
> version is 1
> 17/01/23 23:54:32 ERROR Executor: Exception in task 93.1 in stage 1.0 (TID
> 214)
> java.io.IOException: Failed on local exception:
> java.nio.channels.ClosedByInterruptException; Host Details : local host is:
> "stobdtserver3/10.224.54.70"; destination host is: "stobdtserver2":9000;
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:776)
> at org.apache.hadoop.ipc.Client.call(Client.java:1479)
> at org.apache.hadoop.ipc.Client.call(Client.java:1412)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:229)
> at com.sun.proxy.$Proxy17.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.create(ClientNamenodeProtocolTranslatorPB.java:296)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:497)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:191)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy18.create(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream.newStreamForCreate(DFSOutputStream.java:1648)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1689)
> at org.apache.hadoop.hdfs.DFSClient.create(DFSClient.java:1624)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:448)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$7.doCall(DistributedFileSystem.java:444)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:459)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.create(DistributedFileSystem.java:387)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:804)
> at
> org.apache.hadoop.mapred.TextOutputFormat.getRecordWriter(TextOutputFormat.java:123)
> at org.apache.spark.SparkHadoopWriter.open(SparkHadoopWriter.scala:90)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1133)
> at
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1124)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:88)
> at org.apache.spark.scheduler.Task.run(Task.scala:114)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:322)
> at
>
[jira] [Resolved] (SPARK-20355) Display Spark version on history page
[
https://issues.apache.org/jira/browse/SPARK-20355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves resolved SPARK-20355.
---
Resolution: Fixed
Assignee: Sanket Reddy
Fix Version/s: 2.3.0
> Display Spark version on history page
> -
>
> Key: SPARK-20355
> URL: https://issues.apache.org/jira/browse/SPARK-20355
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, Web UI
>Affects Versions: 2.1.0
>Reporter: Sanket Reddy
>Assignee: Sanket Reddy
>Priority: Minor
> Fix For: 2.3.0
>
>
> Spark Version for a specific application is not displayed on the history page
> now. It should be nice to switch the spark version on the UI when we click on
> the specific application.
> Currently there seems to be way as SparkListenerLogStart records the
> application version. So, it should be trivial to listen to this event and
> provision this change on the UI.
> {"Event":"SparkListenerLogStart","Spark
> Version":"1.6.2.0_2.7.2.7.1604210306_161643"}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19112) add codec for ZStandard
[ https://issues.apache.org/jira/browse/SPARK-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000705#comment-16000705 ] Thomas Graves commented on SPARK-19112: --- also can you put some more details about the benchmark itself and your setup just so we have reference. > add codec for ZStandard > --- > > Key: SPARK-19112 > URL: https://issues.apache.org/jira/browse/SPARK-19112 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Thomas Graves >Priority: Minor > > ZStandard: https://github.com/facebook/zstd and > http://facebook.github.io/zstd/ has been in use for a while now. v1.0 was > recently released. Hadoop > (https://issues.apache.org/jira/browse/HADOOP-13578) and others > (https://issues.apache.org/jira/browse/KAFKA-4514) are adopting it. > Zstd seems to give great results => Gzip level Compression with Lz4 level CPU. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19112) add codec for ZStandard
[ https://issues.apache.org/jira/browse/SPARK-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16000703#comment-16000703 ] Thomas Graves commented on SPARK-19112: --- Execution time is definitely worse, did you get the compression ratio or disk space used during these? How big were the intermediate files?I would think someone would want to use this would be mostly interested in the space aspect and keep the run time small while getting the higher compression. > add codec for ZStandard > --- > > Key: SPARK-19112 > URL: https://issues.apache.org/jira/browse/SPARK-19112 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Thomas Graves >Priority: Minor > > ZStandard: https://github.com/facebook/zstd and > http://facebook.github.io/zstd/ has been in use for a while now. v1.0 was > recently released. Hadoop > (https://issues.apache.org/jira/browse/HADOOP-13578) and others > (https://issues.apache.org/jira/browse/KAFKA-4514) are adopting it. > Zstd seems to give great results => Gzip level Compression with Lz4 level CPU. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18971) Netty issue may cause the shuffle client hang
[
https://issues.apache.org/jira/browse/SPARK-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998531#comment-15998531
]
Thomas Graves commented on SPARK-18971:
---
[~zsxwing]have you seen any issues with the new netty version? We have hit a
similar issue?
> Netty issue may cause the shuffle client hang
> -
>
> Key: SPARK-18971
> URL: https://issues.apache.org/jira/browse/SPARK-18971
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Reporter: Shixiong Zhu
>Assignee: Shixiong Zhu
>Priority: Minor
> Fix For: 2.2.0
>
>
> Check https://github.com/netty/netty/issues/6153 for details
> You should be able to see the following similar stack track in the executor
> thread dump.
> {code}
> "shuffle-client-7-4" daemon prio=5 tid=97 RUNNABLE
> at io.netty.util.Recycler$Stack.scavengeSome(Recycler.java:504)
> at io.netty.util.Recycler$Stack.scavenge(Recycler.java:454)
> at io.netty.util.Recycler$Stack.pop(Recycler.java:435)
> at io.netty.util.Recycler.get(Recycler.java:144)
> at
> io.netty.buffer.PooledUnsafeDirectByteBuf.newInstance(PooledUnsafeDirectByteBuf.java:39)
> at
> io.netty.buffer.PoolArena$DirectArena.newByteBuf(PoolArena.java:727)
> at io.netty.buffer.PoolArena.allocate(PoolArena.java:140)
> at
> io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:271)
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:177)
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:168)
> at
> io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:129)
> at
> io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-18971) Netty issue may cause the shuffle client hang
[
https://issues.apache.org/jira/browse/SPARK-18971?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15998531#comment-15998531
]
Thomas Graves edited comment on SPARK-18971 at 5/5/17 4:31 PM:
---
[~zsxwing]have you seen any issues with the new netty version? We have hit
this same issue.
was (Author: tgraves):
[~zsxwing]have you seen any issues with the new netty version? We have hit a
similar issue?
> Netty issue may cause the shuffle client hang
> -
>
> Key: SPARK-18971
> URL: https://issues.apache.org/jira/browse/SPARK-18971
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Reporter: Shixiong Zhu
>Assignee: Shixiong Zhu
>Priority: Minor
> Fix For: 2.2.0
>
>
> Check https://github.com/netty/netty/issues/6153 for details
> You should be able to see the following similar stack track in the executor
> thread dump.
> {code}
> "shuffle-client-7-4" daemon prio=5 tid=97 RUNNABLE
> at io.netty.util.Recycler$Stack.scavengeSome(Recycler.java:504)
> at io.netty.util.Recycler$Stack.scavenge(Recycler.java:454)
> at io.netty.util.Recycler$Stack.pop(Recycler.java:435)
> at io.netty.util.Recycler.get(Recycler.java:144)
> at
> io.netty.buffer.PooledUnsafeDirectByteBuf.newInstance(PooledUnsafeDirectByteBuf.java:39)
> at
> io.netty.buffer.PoolArena$DirectArena.newByteBuf(PoolArena.java:727)
> at io.netty.buffer.PoolArena.allocate(PoolArena.java:140)
> at
> io.netty.buffer.PooledByteBufAllocator.newDirectBuffer(PooledByteBufAllocator.java:271)
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:177)
> at
> io.netty.buffer.AbstractByteBufAllocator.directBuffer(AbstractByteBufAllocator.java:168)
> at
> io.netty.buffer.AbstractByteBufAllocator.ioBuffer(AbstractByteBufAllocator.java:129)
> at
> io.netty.channel.AdaptiveRecvByteBufAllocator$HandleImpl.allocate(AdaptiveRecvByteBufAllocator.java:104)
> at
> io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:117)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:652)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:575)
> at
> io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:489)
> at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:451)
> at
> io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:140)
> at
> io.netty.util.concurrent.DefaultThreadFactory$DefaultRunnableDecorator.run(DefaultThreadFactory.java:144)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20589) Allow limiting task concurrency per stage
[ https://issues.apache.org/jira/browse/SPARK-20589?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15995650#comment-15995650 ] Thomas Graves commented on SPARK-20589: --- thanks for the suggestions [~mridulm80]. Note just for reference tez did a similar thing under https://issues.apache.org/jira/browse/TEZ-2914, may be useful for the yarn side of things here. > Allow limiting task concurrency per stage > - > > Key: SPARK-20589 > URL: https://issues.apache.org/jira/browse/SPARK-20589 > Project: Spark > Issue Type: Improvement > Components: Scheduler >Affects Versions: 2.1.0 >Reporter: Thomas Graves > > It would be nice to have the ability to limit the number of concurrent tasks > per stage. This is useful when your spark job might be accessing another > service and you don't want to DOS that service. For instance Spark writing > to hbase or Spark doing http puts on a service. Many times you want to do > this without limiting the number of partitions. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-20589) Allow limiting task concurrency per stage
Thomas Graves created SPARK-20589: - Summary: Allow limiting task concurrency per stage Key: SPARK-20589 URL: https://issues.apache.org/jira/browse/SPARK-20589 Project: Spark Issue Type: Improvement Components: Scheduler Affects Versions: 2.1.0 Reporter: Thomas Graves It would be nice to have the ability to limit the number of concurrent tasks per stage. This is useful when your spark job might be accessing another service and you don't want to DOS that service. For instance Spark writing to hbase or Spark doing http puts on a service. Many times you want to do this without limiting the number of partitions. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-20426) OneForOneStreamManager occupies too much memory.
[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-20426: -- Fix Version/s: 2.2.1 > OneForOneStreamManager occupies too much memory. > > > Key: SPARK-20426 > URL: https://issues.apache.org/jira/browse/SPARK-20426 > Project: Spark > Issue Type: Improvement > Components: Shuffle >Affects Versions: 2.1.0 >Reporter: jin xing >Assignee: jin xing > Fix For: 2.2.1, 2.3.0 > > Attachments: screenshot-1.png, screenshot-2.png > > > Spark jobs are running on yarn cluster in my warehouse. We enabled the > external shuffle service(*--conf spark.shuffle.service.enabled=true*). > Recently NodeManager runs OOM now and then. Dumping heap memory, we find that > *OneFroOneStreamManager*'s footprint is huge. NodeManager is configured with > 5G heap memory. While *OneForOneManager* costs 2.5G and there are 5503233 > *FileSegmentManagedBuffer* objects. Is there any suggestions to avoid this > other than just keep increasing NodeManager's memory? Is it possible to stop > *registerStream* in OneForOneStreamManager? Thus we don't need to cache so > many metadatas(i.e. StreamState). -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-20426) OneForOneStreamManager occupies too much memory.
[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-20426: -- Fix Version/s: (was: 2.2.1) > OneForOneStreamManager occupies too much memory. > > > Key: SPARK-20426 > URL: https://issues.apache.org/jira/browse/SPARK-20426 > Project: Spark > Issue Type: Improvement > Components: Shuffle >Affects Versions: 2.1.0 >Reporter: jin xing >Assignee: jin xing > Fix For: 2.3.0 > > Attachments: screenshot-1.png, screenshot-2.png > > > Spark jobs are running on yarn cluster in my warehouse. We enabled the > external shuffle service(*--conf spark.shuffle.service.enabled=true*). > Recently NodeManager runs OOM now and then. Dumping heap memory, we find that > *OneFroOneStreamManager*'s footprint is huge. NodeManager is configured with > 5G heap memory. While *OneForOneManager* costs 2.5G and there are 5503233 > *FileSegmentManagedBuffer* objects. Is there any suggestions to avoid this > other than just keep increasing NodeManager's memory? Is it possible to stop > *registerStream* in OneForOneStreamManager? Thus we don't need to cache so > many metadatas(i.e. StreamState). -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Assigned] (SPARK-20426) OneForOneStreamManager occupies too much memory.
[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves reassigned SPARK-20426: - Assignee: jin xing > OneForOneStreamManager occupies too much memory. > > > Key: SPARK-20426 > URL: https://issues.apache.org/jira/browse/SPARK-20426 > Project: Spark > Issue Type: Improvement > Components: Shuffle >Affects Versions: 2.1.0 >Reporter: jin xing >Assignee: jin xing > Fix For: 2.2.0, 2.3.0 > > Attachments: screenshot-1.png, screenshot-2.png > > > Spark jobs are running on yarn cluster in my warehouse. We enabled the > external shuffle service(*--conf spark.shuffle.service.enabled=true*). > Recently NodeManager runs OOM now and then. Dumping heap memory, we find that > *OneFroOneStreamManager*'s footprint is huge. NodeManager is configured with > 5G heap memory. While *OneForOneManager* costs 2.5G and there are 5503233 > *FileSegmentManagedBuffer* objects. Is there any suggestions to avoid this > other than just keep increasing NodeManager's memory? Is it possible to stop > *registerStream* in OneForOneStreamManager? Thus we don't need to cache so > many metadatas(i.e. StreamState). -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-20426) OneForOneStreamManager occupies too much memory.
[ https://issues.apache.org/jira/browse/SPARK-20426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-20426. --- Resolution: Fixed Fix Version/s: 2.3.0 2.2.0 > OneForOneStreamManager occupies too much memory. > > > Key: SPARK-20426 > URL: https://issues.apache.org/jira/browse/SPARK-20426 > Project: Spark > Issue Type: Improvement > Components: Shuffle >Affects Versions: 2.1.0 >Reporter: jin xing > Fix For: 2.2.0, 2.3.0 > > Attachments: screenshot-1.png, screenshot-2.png > > > Spark jobs are running on yarn cluster in my warehouse. We enabled the > external shuffle service(*--conf spark.shuffle.service.enabled=true*). > Recently NodeManager runs OOM now and then. Dumping heap memory, we find that > *OneFroOneStreamManager*'s footprint is huge. NodeManager is configured with > 5G heap memory. While *OneForOneManager* costs 2.5G and there are 5503233 > *FileSegmentManagedBuffer* objects. Is there any suggestions to avoid this > other than just keep increasing NodeManager's memory? Is it possible to stop > *registerStream* in OneForOneStreamManager? Thus we don't need to cache so > many metadatas(i.e. StreamState). -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Closed] (SPARK-20480) FileFormatWriter hides FetchFailedException from scheduler
[
https://issues.apache.org/jira/browse/SPARK-20480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves closed SPARK-20480.
-
Resolution: Duplicate
> FileFormatWriter hides FetchFailedException from scheduler
> --
>
> Key: SPARK-20480
> URL: https://issues.apache.org/jira/browse/SPARK-20480
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Thomas Graves
>Priority: Critical
>
> I was running a large job where it was getting faiures, noticed they were
> listed as "SparkException: Task failed while writing rows", but when I looked
> further they were really caused by FetchFailure exceptions. This is a
> problem because the scheduler handles Fetch Failures differently then normal
> exception. This can affect things like blacklisting.
> {noformat}
> 17/04/26 20:08:59 ERROR Executor: Exception in task 2727.0 in stage 4.0 (TID
> 102902)
> org.apache.spark.SparkException: Task failed while writing rows
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:204)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.spark.shuffle.FetchFailedException: Failed to connect
> to host1.com:7337
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:357)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:332)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:54)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
> at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.sort_addToSorter$(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$2.hasNext(WholeStageCodegenExec.scala:396)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)
> at
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1341)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)
> ... 8 more
> Caused by: java.io.IOException: Failed to connect to host1.com:7337
> at
>
[jira] [Commented] (SPARK-20480) FileFormatWriter hides FetchFailedException from scheduler
[
https://issues.apache.org/jira/browse/SPARK-20480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985590#comment-15985590
]
Thomas Graves commented on SPARK-20480:
---
ah it looks like it should, hadn't seen that jira, I'll give that fix a try.
> FileFormatWriter hides FetchFailedException from scheduler
> --
>
> Key: SPARK-20480
> URL: https://issues.apache.org/jira/browse/SPARK-20480
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Thomas Graves
>Priority: Critical
>
> I was running a large job where it was getting faiures, noticed they were
> listed as "SparkException: Task failed while writing rows", but when I looked
> further they were really caused by FetchFailure exceptions. This is a
> problem because the scheduler handles Fetch Failures differently then normal
> exception. This can affect things like blacklisting.
> {noformat}
> 17/04/26 20:08:59 ERROR Executor: Exception in task 2727.0 in stage 4.0 (TID
> 102902)
> org.apache.spark.SparkException: Task failed while writing rows
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:204)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.spark.shuffle.FetchFailedException: Failed to connect
> to host1.com:7337
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:357)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:332)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:54)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
> at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.sort_addToSorter$(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$2.hasNext(WholeStageCodegenExec.scala:396)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)
> at
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1341)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)
> ... 8 more
> Caused by: java.io.IOException: Failed to
[jira] [Commented] (SPARK-20480) FileFormatWriter hides FetchFailedException from scheduler
[
https://issues.apache.org/jira/browse/SPARK-20480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985571#comment-15985571
]
Thomas Graves commented on SPARK-20480:
---
Note with blacklisting on this caused the job to fail:
Job aborted due to stage failure: Aborting TaskSet 4.0 because task 21
(partition 21) cannot run anywhere due to node and executor blacklist.
Blacklisting behavior can be configured via spark.blacklist.*.
> FileFormatWriter hides FetchFailedException from scheduler
> --
>
> Key: SPARK-20480
> URL: https://issues.apache.org/jira/browse/SPARK-20480
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Thomas Graves
>Priority: Critical
>
> I was running a large job where it was getting faiures, noticed they were
> listed as "SparkException: Task failed while writing rows", but when I looked
> further they were really caused by FetchFailure exceptions. This is a
> problem because the scheduler handles Fetch Failures differently then normal
> exception. This can affect things like blacklisting.
> {noformat}
> 17/04/26 20:08:59 ERROR Executor: Exception in task 2727.0 in stage 4.0 (TID
> 102902)
> org.apache.spark.SparkException: Task failed while writing rows
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:204)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.spark.shuffle.FetchFailedException: Failed to connect
> to host1.com:7337
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:357)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:332)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:54)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
> at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.sort_addToSorter$(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$2.hasNext(WholeStageCodegenExec.scala:396)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)
> at
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1341)
> at
>
[jira] [Commented] (SPARK-20480) FileFormatWriter hides FetchFailedException from scheduler
[
https://issues.apache.org/jira/browse/SPARK-20480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985567#comment-15985567
]
Thomas Graves commented on SPARK-20480:
---
exception in task manager looks like:
17/04/26 20:09:21 INFO TaskSetManager: Lost task 3516.0 in stage 4.0 (TID
103691) on gsbl521n33.blue.ygrid.yahoo.com, exec
utor 4516: org.apache.spark.SparkException (Task failed while writing rows)
[duplicate 22]
> FileFormatWriter hides FetchFailedException from scheduler
> --
>
> Key: SPARK-20480
> URL: https://issues.apache.org/jira/browse/SPARK-20480
> Project: Spark
> Issue Type: Bug
> Components: SQL
>Affects Versions: 2.1.0
>Reporter: Thomas Graves
>Priority: Critical
>
> I was running a large job where it was getting faiures, noticed they were
> listed as "SparkException: Task failed while writing rows", but when I looked
> further they were really caused by FetchFailure exceptions. This is a
> problem because the scheduler handles Fetch Failures differently then normal
> exception. This can affect things like blacklisting.
> {noformat}
> 17/04/26 20:08:59 ERROR Executor: Exception in task 2727.0 in stage 4.0 (TID
> 102902)
> org.apache.spark.SparkException: Task failed while writing rows
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:204)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)
> at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
> at org.apache.spark.scheduler.Task.run(Task.scala:99)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.spark.shuffle.FetchFailedException: Failed to connect
> to host1.com:7337
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:357)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:332)
> at
> org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:54)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
> at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
> at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at
> org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
> at
> org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
> at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.sort_addToSorter$(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
> Source)
> at
> org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
> Source)
> at
> org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
> at
> org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$2.hasNext(WholeStageCodegenExec.scala:396)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)
> at
> org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)
> at
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1341)
> at
>
[jira] [Updated] (SPARK-20480) FileFormatWriter hides FetchFailedException from scheduler
[
https://issues.apache.org/jira/browse/SPARK-20480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves updated SPARK-20480:
--
Description:
I was running a large job where it was getting faiures, noticed they were
listed as "SparkException: Task failed while writing rows", but when I looked
further they were really caused by FetchFailure exceptions. This is a problem
because the scheduler handles Fetch Failures differently then normal exception.
This can affect things like blacklisting.
{noformat}
17/04/26 20:08:59 ERROR Executor: Exception in task 2727.0 in stage 4.0 (TID
102902)
org.apache.spark.SparkException: Task failed while writing rows
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:204)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.spark.shuffle.FetchFailedException: Failed to connect to
host1.com:7337
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:357)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:332)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:54)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.sort_addToSorter$(Unknown
Source)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
Source)
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
Source)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
Source)
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$2.hasNext(WholeStageCodegenExec.scala:396)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)
at
org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1341)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)
... 8 more
Caused by: java.io.IOException: Failed to connect to host1.com:7337
at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:228)
at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:179)
at
org.apache.spark.network.shuffle.ExternalShuffleClient$1.createAndStart(ExternalShuffleClient.java:105)
at
org.apache.spark.network.shuffle.RetryingBlockFetcher.fetchAllOutstanding(RetryingBlockFetcher.java:140)
at
org.apache.spark.network.shuffle.RetryingBlockFetcher.access$200(RetryingBlockFetcher.java:43)
[jira] [Created] (SPARK-20480) FileFormatWriter hides FetchFailedException from scheduler
Thomas Graves created SPARK-20480:
-
Summary: FileFormatWriter hides FetchFailedException from scheduler
Key: SPARK-20480
URL: https://issues.apache.org/jira/browse/SPARK-20480
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 2.1.0
Reporter: Thomas Graves
Priority: Critical
I was running a large job where it was getting faiures, noticed they were
listed as "SparkException: Task failed while writing rows", but when I looked
further they were really caused by FetchFailure exceptions. This is a problem
because the scheduler handles Fetch Failures differently then normal exception.
This can affect things like blacklisting.
{noformat}
17/04/26 20:08:59 ERROR Executor: Exception in task 2727.0 in stage 4.0 (TID
102902)
org.apache.spark.SparkException: Task failed while writing rows
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:204)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:129)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$write$1$$anonfun$3.apply(FileFormatWriter.scala:128)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:282)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.apache.spark.shuffle.FetchFailedException: Failed to connect to
gsbl546n07.blue.ygrid.yahoo.com/10.213.43.94:7337
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.throwFetchFailedException(ShuffleBlockFetcherIterator.scala:357)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:332)
at
org.apache.spark.storage.ShuffleBlockFetcherIterator.next(ShuffleBlockFetcherIterator.scala:54)
at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
at scala.collection.Iterator$$anon$12.nextCur(Iterator.scala:434)
at scala.collection.Iterator$$anon$12.hasNext(Iterator.scala:440)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at
org.apache.spark.util.CompletionIterator.hasNext(CompletionIterator.scala:32)
at
org.apache.spark.InterruptibleIterator.hasNext(InterruptibleIterator.scala:39)
at scala.collection.Iterator$$anon$11.hasNext(Iterator.scala:408)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.sort_addToSorter$(Unknown
Source)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
Source)
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$8$$anon$1.hasNext(WholeStageCodegenExec.scala:377)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.findNextInnerJoinRows$(Unknown
Source)
at
org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIterator.processNext(Unknown
Source)
at
org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
at
org.apache.spark.sql.execution.WholeStageCodegenExec$$anonfun$10$$anon$2.hasNext(WholeStageCodegenExec.scala:396)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$SingleDirectoryWriteTask.execute(FileFormatWriter.scala:243)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:190)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$$anonfun$org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask$3.apply(FileFormatWriter.scala:188)
at
org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1341)
at
org.apache.spark.sql.execution.datasources.FileFormatWriter$.org$apache$spark$sql$execution$datasources$FileFormatWriter$$executeTask(FileFormatWriter.scala:193)
... 8 more
Caused by: java.io.IOException: Failed to connect to
gsbl546n07.blue.ygrid.yahoo.com/10.213.43.94:7337
at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:228)
at
org.apache.spark.network.client.TransportClientFactory.createClient(TransportClientFactory.java:179)
at
[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures
[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15985447#comment-15985447 ] Thomas Graves commented on SPARK-20178: --- Another thing we should tie in here is handling preempted containers better. This kind of matches with my point above "Improve logic around deciding which node is actually bad when you get a fetch failures." but a little bit of a special case. If the containers gets preempted on the yarn side we need to properly detect that and not count that as a normal fetch failure. Right now that seems pretty difficult with the way we handle stage failures but I guess you would just line that up and not caught that as a normal stage failure. > Improve Scheduler fetch failures > > > Key: SPARK-20178 > URL: https://issues.apache.org/jira/browse/SPARK-20178 > Project: Spark > Issue Type: Epic > Components: Scheduler >Affects Versions: 2.1.0 >Reporter: Thomas Graves > > We have been having a lot of discussions around improving the handling of > fetch failures. There are 4 jira currently related to this. > We should try to get a list of things we want to improve and come up with one > cohesive design. > SPARK-20163, SPARK-20091, SPARK-14649 , and SPARK-19753 > I will put my initial thoughts in a follow on comment. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-19812) YARN shuffle service fails to relocate recovery DB across NFS directories
[ https://issues.apache.org/jira/browse/SPARK-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-19812. --- Resolution: Fixed Fix Version/s: 2.3.0 2.2.0 > YARN shuffle service fails to relocate recovery DB across NFS directories > - > > Key: SPARK-19812 > URL: https://issues.apache.org/jira/browse/SPARK-19812 > Project: Spark > Issue Type: Bug > Components: YARN >Affects Versions: 2.0.1 >Reporter: Thomas Graves >Assignee: Thomas Graves > Fix For: 2.2.0, 2.3.0 > > > The yarn shuffle service tries to switch from the yarn local directories to > the real recovery directory but can fail to move the existing recovery db's. > It fails due to Files.move not doing directories that have contents. > 2017-03-03 14:57:19,558 [main] ERROR yarn.YarnShuffleService: Failed to move > recovery file sparkShuffleRecovery.ldb to the path > /mapred/yarn-nodemanager/nm-aux-services/spark_shuffle > java.nio.file.DirectoryNotEmptyException:/yarn-local/sparkShuffleRecovery.ldb > at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:498) > at > sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) > at java.nio.file.Files.move(Files.java:1395) > at > org.apache.spark.network.yarn.YarnShuffleService.initRecoveryDb(YarnShuffleService.java:369) > at > org.apache.spark.network.yarn.YarnShuffleService.createSecretManager(YarnShuffleService.java:200) > at > org.apache.spark.network.yarn.YarnShuffleService.serviceInit(YarnShuffleService.java:174) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:143) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:262) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:357) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:636) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:684) > This used to use f.renameTo and we switched it in the pr due to review > comments and it looks like didn't do a final real test. The tests are using > files rather then directories so it didn't catch. We need to fix the test > also. > history: > https://github.com/apache/spark/pull/14999/commits/65de8531ccb91287f5a8a749c7819e99533b9440 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19812) YARN shuffle service fails to relocate recovery DB across NFS directories
[ https://issues.apache.org/jira/browse/SPARK-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-19812: -- Summary: YARN shuffle service fails to relocate recovery DB across NFS directories (was: YARN shuffle service fails to relocate recovery DB directories) > YARN shuffle service fails to relocate recovery DB across NFS directories > - > > Key: SPARK-19812 > URL: https://issues.apache.org/jira/browse/SPARK-19812 > Project: Spark > Issue Type: Bug > Components: YARN >Affects Versions: 2.0.1 >Reporter: Thomas Graves >Assignee: Thomas Graves > > The yarn shuffle service tries to switch from the yarn local directories to > the real recovery directory but can fail to move the existing recovery db's. > It fails due to Files.move not doing directories that have contents. > 2017-03-03 14:57:19,558 [main] ERROR yarn.YarnShuffleService: Failed to move > recovery file sparkShuffleRecovery.ldb to the path > /mapred/yarn-nodemanager/nm-aux-services/spark_shuffle > java.nio.file.DirectoryNotEmptyException:/yarn-local/sparkShuffleRecovery.ldb > at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:498) > at > sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) > at java.nio.file.Files.move(Files.java:1395) > at > org.apache.spark.network.yarn.YarnShuffleService.initRecoveryDb(YarnShuffleService.java:369) > at > org.apache.spark.network.yarn.YarnShuffleService.createSecretManager(YarnShuffleService.java:200) > at > org.apache.spark.network.yarn.YarnShuffleService.serviceInit(YarnShuffleService.java:174) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:143) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:262) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:357) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:636) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:684) > This used to use f.renameTo and we switched it in the pr due to review > comments and it looks like didn't do a final real test. The tests are using > files rather then directories so it didn't catch. We need to fix the test > also. > history: > https://github.com/apache/spark/pull/14999/commits/65de8531ccb91287f5a8a749c7819e99533b9440 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19812) YARN shuffle service fails to relocate recovery DB directories
[ https://issues.apache.org/jira/browse/SPARK-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15979349#comment-15979349 ] Thomas Graves commented on SPARK-19812: --- Sorry wasn't clear in the original description, it errors when moving across nfs mounts. I was wrong on the unit test comment. Unit test is fine except it doesn't test across mounts which would be really difficult to do. > YARN shuffle service fails to relocate recovery DB directories > -- > > Key: SPARK-19812 > URL: https://issues.apache.org/jira/browse/SPARK-19812 > Project: Spark > Issue Type: Bug > Components: YARN >Affects Versions: 2.0.1 >Reporter: Thomas Graves >Assignee: Thomas Graves > > The yarn shuffle service tries to switch from the yarn local directories to > the real recovery directory but can fail to move the existing recovery db's. > It fails due to Files.move not doing directories that have contents. > 2017-03-03 14:57:19,558 [main] ERROR yarn.YarnShuffleService: Failed to move > recovery file sparkShuffleRecovery.ldb to the path > /mapred/yarn-nodemanager/nm-aux-services/spark_shuffle > java.nio.file.DirectoryNotEmptyException:/yarn-local/sparkShuffleRecovery.ldb > at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:498) > at > sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) > at java.nio.file.Files.move(Files.java:1395) > at > org.apache.spark.network.yarn.YarnShuffleService.initRecoveryDb(YarnShuffleService.java:369) > at > org.apache.spark.network.yarn.YarnShuffleService.createSecretManager(YarnShuffleService.java:200) > at > org.apache.spark.network.yarn.YarnShuffleService.serviceInit(YarnShuffleService.java:174) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:143) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:262) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:357) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:636) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:684) > This used to use f.renameTo and we switched it in the pr due to review > comments and it looks like didn't do a final real test. The tests are using > files rather then directories so it didn't catch. We need to fix the test > also. > history: > https://github.com/apache/spark/pull/14999/commits/65de8531ccb91287f5a8a749c7819e99533b9440 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20391) Properly rename the memory related fields in ExecutorSummary REST API
[
https://issues.apache.org/jira/browse/SPARK-20391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975115#comment-15975115
]
Thomas Graves commented on SPARK-20391:
---
> My proposal was to add 2 extra fields which duplicate the existing ones, so
> that the memory metrics are together and hopefully the meaning is clear.
> totalManagedMemory would be the same as maxMemory; usedStorageMemory would be
> the same as memoryUsed. But I'm not super firm on that, and its definitely
> not "must do" for 2.2.
yep, makes sense I would think it is easy enough to do, we should just do it
here.
> Properly rename the memory related fields in ExecutorSummary REST API
> -
>
> Key: SPARK-20391
> URL: https://issues.apache.org/jira/browse/SPARK-20391
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.2.0
>Reporter: Saisai Shao
>Priority: Blocker
>
> Currently in Spark we could get executor summary through REST API
> {{/api/v1/applications//executors}}. The format of executor summary
> is:
> {code}
> class ExecutorSummary private[spark](
> val id: String,
> val hostPort: String,
> val isActive: Boolean,
> val rddBlocks: Int,
> val memoryUsed: Long,
> val diskUsed: Long,
> val totalCores: Int,
> val maxTasks: Int,
> val activeTasks: Int,
> val failedTasks: Int,
> val completedTasks: Int,
> val totalTasks: Int,
> val totalDuration: Long,
> val totalGCTime: Long,
> val totalInputBytes: Long,
> val totalShuffleRead: Long,
> val totalShuffleWrite: Long,
> val isBlacklisted: Boolean,
> val maxMemory: Long,
> val executorLogs: Map[String, String],
> val onHeapMemoryUsed: Option[Long],
> val offHeapMemoryUsed: Option[Long],
> val maxOnHeapMemory: Option[Long],
> val maxOffHeapMemory: Option[Long])
> {code}
> Here are 6 memory related fields: {{memoryUsed}}, {{maxMemory}},
> {{onHeapMemoryUsed}}, {{offHeapMemoryUsed}}, {{maxOnHeapMemory}},
> {{maxOffHeapMemory}}.
> These all 6 fields reflects the *storage* memory usage in Spark, but from the
> name of this 6 fields, user doesn't really know it is referring to *storage*
> memory or the total memory (storage memory + execution memory). This will be
> misleading.
> So I think we should properly rename these fields to reflect their real
> meanings. Or we should will document it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20391) Properly rename the memory related fields in ExecutorSummary REST API
[
https://issues.apache.org/jira/browse/SPARK-20391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15975029#comment-15975029
]
Thomas Graves commented on SPARK-20391:
---
I agree that if its been released we can't change it, the on/off heap we need
to change asap before a release. If we want to change the names of the other 2
we could simply add 2 extra fields with a more appropriate name and leave the
other 2 not sure that is necessary at this point though.
It think we should document rest api better and I think that page would be fine
or link to another page, but that might be a separate jira if this is to change
names still. Its an api and we should have had that from the beginning.
example of yarn rest api docs:
https://hadoop.apache.org/docs/r2.7.2/hadoop-yarn/hadoop-yarn-site/ResourceManagerRest.html
I'm sure there are better examples too.
I think making it a separate ExecutorMemoryMetrics makes sense so we can more
easily extend in the future.. I assume managed memory here is
spark.memory.fraction on heap + spark.memory.offHeap.size?
> Properly rename the memory related fields in ExecutorSummary REST API
> -
>
> Key: SPARK-20391
> URL: https://issues.apache.org/jira/browse/SPARK-20391
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
>Affects Versions: 2.2.0
>Reporter: Saisai Shao
>Priority: Minor
>
> Currently in Spark we could get executor summary through REST API
> {{/api/v1/applications//executors}}. The format of executor summary
> is:
> {code}
> class ExecutorSummary private[spark](
> val id: String,
> val hostPort: String,
> val isActive: Boolean,
> val rddBlocks: Int,
> val memoryUsed: Long,
> val diskUsed: Long,
> val totalCores: Int,
> val maxTasks: Int,
> val activeTasks: Int,
> val failedTasks: Int,
> val completedTasks: Int,
> val totalTasks: Int,
> val totalDuration: Long,
> val totalGCTime: Long,
> val totalInputBytes: Long,
> val totalShuffleRead: Long,
> val totalShuffleWrite: Long,
> val isBlacklisted: Boolean,
> val maxMemory: Long,
> val executorLogs: Map[String, String],
> val onHeapMemoryUsed: Option[Long],
> val offHeapMemoryUsed: Option[Long],
> val maxOnHeapMemory: Option[Long],
> val maxOffHeapMemory: Option[Long])
> {code}
> Here are 6 memory related fields: {{memoryUsed}}, {{maxMemory}},
> {{onHeapMemoryUsed}}, {{offHeapMemoryUsed}}, {{maxOnHeapMemory}},
> {{maxOffHeapMemory}}.
> These all 6 fields reflects the *storage* memory usage in Spark, but from the
> name of this 6 fields, user doesn't really know it is referring to *storage*
> memory or the total memory (storage memory + execution memory). This will be
> misleading.
> So I think we should properly rename these fields to reflect their real
> meanings. Or we should will document it.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-14245) webUI should display the user
[ https://issues.apache.org/jira/browse/SPARK-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971365#comment-15971365 ] Thomas Graves commented on SPARK-14245: --- see the commend in the PR, I think there was a race with the SparkContext version: I just pushed an update. I changed how I get the username so it uses the same method for both active and history. This method actually pulls the information from the same source as both sc.sparkUser and the API do at their root, the system env. Except with this method we make sure we always get the correct user; I noticed a race condition when using sc.sparkUser while finding this solution. After starting up the master when two spark applications are started at once, one by the user who started the master and one who didn't, both would appear to been started by the user who started the master. This would only happened on the first apps started, any apps started after would behave as expected. This race condition doesn't exist with my change. > webUI should display the user > - > > Key: SPARK-14245 > URL: https://issues.apache.org/jira/browse/SPARK-14245 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 1.6.1 >Reporter: Thomas Graves >Assignee: Alex Bozarth > Fix For: 2.0.0 > > > It would be nice if the Spark UI (both active and history) showed the user > who ran the application somewhere when you are in the application view. > Perhaps under the Jobs view by total uptime and scheduler mode. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20340) Size estimate very wrong in ExternalAppendOnlyMap from CoGroupedRDD, cause OOM
[ https://issues.apache.org/jira/browse/SPARK-20340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15971071#comment-15971071 ] Thomas Graves commented on SPARK-20340: --- Right, I figured it was probably for performance, the thing is that when its wrong it causes the job to fail and this could be unexpectedly. Meaning a production job was running fine for months and then the data it gets in is all of a sudden differently/skewed due to say high traffic day and then a critical job fails. This to me is not good for a production environment, if we want Spark to continue to be adopted by larger companies in production environments this sort of thing has to be very reliable. It looks very bad for Spark that my user said this runs fine on PIG. It makes users very wary about switching to Spark as they have doubts about this scalability and reliability. Anyway, I think this either needs a sanity at some point to say its estimates are really off so get the real size or switch it to always use the real size. For shuffle data we should know what the size is as we just transferred it. that is abstracted away a git at this point in the code though so need to understand the code more. [~rxin] [~joshrosen] any thoughts on this? > Size estimate very wrong in ExternalAppendOnlyMap from CoGroupedRDD, cause OOM > -- > > Key: SPARK-20340 > URL: https://issues.apache.org/jira/browse/SPARK-20340 > Project: Spark > Issue Type: Bug > Components: Spark Core >Affects Versions: 2.1.0 >Reporter: Thomas Graves > > I had a user doing a basic join operation. The values are image's binary > data(in base64 format) and widely vary in size. > The job failed with out of memory. Originally failed on yarn with using to > much overhead memory, turned spark.shuffle.io.preferDirectBufs to false > then failed with out of heap memory. I debugged it down to during the > shuffle when CoGroupedRDD putting things into the ExternalAppendOnlyMap, it > computes an estimated size to determine when to spill. In this case > SizeEstimator handle arrays such that if it is larger then 400 elements, it > samples 100 elements. The estimate is coming back as GB's different from the > actual size. It claims 1GB when it is actually using close to 5GB. > Temporary work around it to increase the memory to be very large (10GB > executors) but that isn't really acceptable here. User did the same thing in > pig and it easily handled the data with 1.5GB of memory. > It seems risky to be using an estimate in such a critical thing. If the > estimate is wrong you are going to run out of memory and fail the job. > I'm looking closer at the users data still to get more insights. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-20340) Size estimate very wrong in ExternalAppendOnlyMap from CoGroupedRDD, cause OOM
Thomas Graves created SPARK-20340: - Summary: Size estimate very wrong in ExternalAppendOnlyMap from CoGroupedRDD, cause OOM Key: SPARK-20340 URL: https://issues.apache.org/jira/browse/SPARK-20340 Project: Spark Issue Type: Bug Components: Spark Core Affects Versions: 2.1.0 Reporter: Thomas Graves I had a user doing a basic join operation. The values are image's binary data(in base64 format) and widely vary in size. The job failed with out of memory. Originally failed on yarn with using to much overhead memory, turned spark.shuffle.io.preferDirectBufsto false then failed with out of heap memory. I debugged it down to during the shuffle when CoGroupedRDD putting things into the ExternalAppendOnlyMap, it computes an estimated size to determine when to spill. In this case SizeEstimator handle arrays such that if it is larger then 400 elements, it samples 100 elements. The estimate is coming back as GB's different from the actual size. It claims 1GB when it is actually using close to 5GB. Temporary work around it to increase the memory to be very large (10GB executors) but that isn't really acceptable here. User did the same thing in pig and it easily handled the data with 1.5GB of memory. It seems risky to be using an estimate in such a critical thing. If the estimate is wrong you are going to run out of memory and fail the job. I'm looking closer at the users data still to get more insights. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures
[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15969388#comment-15969388 ] Thomas Graves commented on SPARK-20178: --- One thing I ran into today which is somewhat related to this is a combination of failure types. In this case it was broadcast fetch failures combined with shuffle fetch failures which lead to 4 task failures and failed the job. I believe they were all from the same host and happened really quickly (within 3 seconds). This seems like this should fall under the fetch failure case as well. Failed to get broadcast_646_piece0 of broadcast_646 at org.apache.spark.util.Utils$.tryOrIOException(Utils.scala:1222) at org.apache.spark.broadcast.TorrentBroadcast.readBroadcastBlock(TorrentBroadcast.scala:165) at org.apache.spark.broadcast.TorrentBroadcast._value$lzycompute(TorrentBroadcast.scala:64) at org.apache.spark.broadcast.TorrentBroadcast._value(TorrentBroadcast.scala:64) at org.apache.spark.broadcast.TorrentBroadcast.getValue(TorrentBroadcast.scala:88) at org.apache.spark.broadcast.Broadcast.value(Broadcast.scala:70) at org.apache.spark.MapOutputTracker$.deserializeMapStatuses(MapOutputTracker.scala:691) at org.apache.spark.MapOutputTracker.getStatuses(MapOutputTracker.scala:204) at org.apache.spark.MapOutputTracker.getMapSizesByExecutorId(MapOutputTracker.scala:143) at org.apache.spark.shuffle.BlockStoreShuffleReader.read(BlockStoreShuffleReader.scala:47) at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:147) at org.apache.spark.rdd.CoGroupedRDD$$anonfun$compute$2.apply(CoGroupedRDD.scala:136) at scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772) at scala.collection.immutable.List.foreach(List.scala:318) at scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:771) at org.apache.spark.rdd.CoGroupedRDD.compute(CoGroupedRDD.scala:136) at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:306) > Improve Scheduler fetch failures > > > Key: SPARK-20178 > URL: https://issues.apache.org/jira/browse/SPARK-20178 > Project: Spark > Issue Type: Epic > Components: Scheduler >Affects Versions: 2.1.0 >Reporter: Thomas Graves > > We have been having a lot of discussions around improving the handling of > fetch failures. There are 4 jira currently related to this. > We should try to get a list of things we want to improve and come up with one > cohesive design. > SPARK-20163, SPARK-20091, SPARK-14649 , and SPARK-19753 > I will put my initial thoughts in a follow on comment. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-20178) Improve Scheduler fetch failures
[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15950917#comment-15950917 ] Thomas Graves edited comment on SPARK-20178 at 3/31/17 1:53 PM: Overall what I would like to accomplish is not throwing away work and making the failure case very performant. More and more people are running spark on larger clusters, this means failures are going to occur more. We need those failures to be as fast as possible. We need to be careful here and make sure we handle the node totally down case, the nodemanager totally down, and the nodemanager or node is just having intermittent issue. Generally I see the last where the issue is just intermittent but some people recently have had more of the nodemanager totally down case in which case you want to fail all maps on that node quickly. The decision on what to rerun is hard now because it could be very costly to rerun more, but at the same time it could be very costly to not rerun all immediately because you can fail all 4 stage attempts. This really depends on how long the maps and reduces run. A lot of discussion on https://github.com/apache/spark/pull/17088 related to that. - We should not kill the Reduce tasks on fetch failure. Leave the Reduce tasks running since it could have done useful work already like fetching X number of map outputs. It can simply fail that map output which would cause the map to be rerun and only that specific map output would need to be refetched. This does require checking to make sure there are enough resource to run the map and if not possibly killing a reducer or getting more resources if dynamic allocation. - Improve logic around deciding which node is actually bad when you get a fetch failures. Was it really the node the reduce was on or the node the map was on. You can do something here like a % of reducers failed to fetch from map output node. - We should only rerun the maps that are necessary. Other maps could have already been fetched (with bullet one) so no need to rerun those immediately. Since the reduce tasks keep running, other fetch failures can happen in parallel and that would just cause other maps to be rerun. At some point based on bullet 2 above we can decide entire node is bad or to invalidate all output on that node. Make sure to think about intermittent failures vs shuffle handler totally down and not coming back. Use that in determining logic - Improve the blacklisting based on the above improvements - make sure to think about how this plays into the stage attempt max failures (4, now settable) - try to not waste resources. ie right now we can have 2 of the same reduce tasks running which is using twice the resources and there are a bunch of different conditions that can occur as to whether this work is actually useful. Question: - should we consider having it fetch all map output from a host at once (rather then per executor). This could improve fetching times (but would have to test) as well as fetch failure handling. This could cause it to fail more maps which is somewhat contradictory to bullet 3 above, need to think about this more. - Do we need pluggable interface or how do we not destabilize current scheduler? Bonus or future: - Decision on when and how many maps to rerun is cost based estimate. If maps only take a few seconds to run could rerun all maps on the host immediately - option to prestart reduce tasks so that they can start fetching while last few maps are failing (if you have long tail maps) was (Author: tgraves): Overall what I would like to accomplish is not throwing away work and making the failure case very performant. More and more people are running spark on larger clusters, this means failures are going to occur more. We need those failures to be as fast as possible. We need to be careful here and make sure we handle the node totally down case, the nodemanager totally down, and the nodemanager or node is just having intermittent issue. Generally I see the last where the issue is just intermittent but some people recently have had more of the nodemanager totally down case in which case you want to fail all maps on that node quickly. The decision on what to rerun is hard now because it could be very costly to rerun more, but at the same time it could be very costly to not rerun all immediately because you can fail all 4 stage attempts. This really depends on how long the maps and reduces run. A lot of discussion on https://github.com/apache/spark/pull/17088 related to that. - We should not kill the Reduce tasks on fetch failure. Leave the Reduce tasks running since it could have done useful work already like fetching X number of map outputs. It can simply fail that map output which would cause the map to be rerun and only that specific map output would need to be
[jira] [Commented] (SPARK-20178) Improve Scheduler fetch failures
[ https://issues.apache.org/jira/browse/SPARK-20178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15950917#comment-15950917 ] Thomas Graves commented on SPARK-20178: --- Overall what I would like to accomplish is not throwing away work and making the failure case very performant. More and more people are running spark on larger clusters, this means failures are going to occur more. We need those failures to be as fast as possible. We need to be careful here and make sure we handle the node totally down case, the nodemanager totally down, and the nodemanager or node is just having intermittent issue. Generally I see the last where the issue is just intermittent but some people recently have had more of the nodemanager totally down case in which case you want to fail all maps on that node quickly. The decision on what to rerun is hard now because it could be very costly to rerun more, but at the same time it could be very costly to not rerun all immediately because you can fail all 4 stage attempts. This really depends on how long the maps and reduces run. A lot of discussion on https://github.com/apache/spark/pull/17088 related to that. - We should not kill the Reduce tasks on fetch failure. Leave the Reduce tasks running since it could have done useful work already like fetching X number of map outputs. It can simply fail that map output which would cause the map to be rerun and only that specific map output would need to be refetched. This does require checking to make sure there are enough resource to run the map and if not possibly killing a reducer or getting more resources if dynamic allocation. - Improve logic around deciding which node is actually bad when you get a fetch failures. Was it really the node the reduce was on or the node the map was on. You can do something here like a % of reducers failed to fetch from map output node. - We should only rerun the maps that failed (or have been logic around how to make this decision), other maps could have already been fetch (with bullet one) so no need to rerun if all reducers already fetched. Since the reduce tasks keep running, other fetch failures can happen in parallel and that would just cause other maps to be rerun. At some point based on bullet 2 above we can decide entire node is bad. - Improve the blacklisting based on the above improvements - make sure to think about how this plays into the stage attempt max failures (4, now settable) - try to not waste resources. ie right now we can have 2 of the same reduce tasks running which is using twice the resources and there are a bunch of different conditions that can occur as to whether this work is actually useful. Question: - should we consider having it fetch all map output from a host at once (rather then per executor). This could improve fetching times (but would have to test) as well as fetch failure handling. This could cause it to fail more maps which is somewhat contradictory to bullet 3 above, need to think about this more. - Do we need pluggable interface or how do we not destabilize current scheduler? Bonus or future: - Decision on when and how many maps to rerun is cost based estimate. If maps only take a few seconds to run could rerun all maps on the host immediately - option to prestart reduce tasks so that they can start fetching while last few maps are failing (if you have long tail maps) > Improve Scheduler fetch failures > > > Key: SPARK-20178 > URL: https://issues.apache.org/jira/browse/SPARK-20178 > Project: Spark > Issue Type: Epic > Components: Scheduler >Affects Versions: 2.1.0 >Reporter: Thomas Graves > > We have been having a lot of discussions around improving the handling of > fetch failures. There are 4 jira currently related to this. > We should try to get a list of things we want to improve and come up with one > cohesive design. > SPARK-20163, SPARK-20091, SPARK-14649 , and SPARK-19753 > I will put my initial thoughts in a follow on comment. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-20178) Improve Scheduler fetch failures
Thomas Graves created SPARK-20178: - Summary: Improve Scheduler fetch failures Key: SPARK-20178 URL: https://issues.apache.org/jira/browse/SPARK-20178 Project: Spark Issue Type: Epic Components: Scheduler Affects Versions: 2.1.0 Reporter: Thomas Graves We have been having a lot of discussions around improving the handling of fetch failures. There are 4 jira currently related to this. We should try to get a list of things we want to improve and come up with one cohesive design. SPARK-20163, SPARK-20091, SPARK-14649 , and SPARK-19753 I will put my initial thoughts in a follow on comment. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)
[
https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15944046#comment-15944046
]
Thomas Graves commented on SPARK-19143:
---
Yes I can be Shephard.
> API in Spark for distributing new delegation tokens (Improve delegation token
> handling in secure clusters)
> --
>
> Key: SPARK-19143
> URL: https://issues.apache.org/jira/browse/SPARK-19143
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, YARN
>Affects Versions: 2.0.2, 2.1.0
>Reporter: Ruslan Dautkhanov
>
> Spin off from SPARK-14743 and comments chain in [recent comments|
> https://issues.apache.org/jira/browse/SPARK-5493?focusedCommentId=15802179=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15802179]
> in SPARK-5493.
> Spark currently doesn't have a way for distribution new delegation tokens.
> Quoting [~vanzin] from SPARK-5493
> {quote}
> IIRC Livy doesn't yet support delegation token renewal. Once it reaches the
> TTL, the session is unusable.
> There might be ways to hack support for that without changes in Spark, but
> I'd like to see a proper API in Spark for distributing new delegation tokens.
> I mentioned that in SPARK-14743, but although that bug is closed, that
> particular feature hasn't been implemented yet.
> {quote}
> Other thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19904) SPIP Add Spark Project Improvement Proposal doc to website
[ https://issues.apache.org/jira/browse/SPARK-19904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943799#comment-15943799 ] Thomas Graves commented on SPARK-19904: --- Is this done or what is this waiting on? > SPIP Add Spark Project Improvement Proposal doc to website > -- > > Key: SPARK-19904 > URL: https://issues.apache.org/jira/browse/SPARK-19904 > Project: Spark > Issue Type: Improvement > Components: Documentation >Affects Versions: 2.1.0 >Reporter: Cody Koeninger >Assignee: Cody Koeninger > Labels: SPIP > > see > http://apache-spark-developers-list.1001551.n3.nabble.com/Spark-Improvement-Proposals-td19268.html -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)
[
https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15943284#comment-15943284
]
Thomas Graves commented on SPARK-19143:
---
I assume this needs to go through the new spip process:
http://spark.apache.org/improvement-proposals.html
> API in Spark for distributing new delegation tokens (Improve delegation token
> handling in secure clusters)
> --
>
> Key: SPARK-19143
> URL: https://issues.apache.org/jira/browse/SPARK-19143
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, YARN
>Affects Versions: 2.0.2, 2.1.0
>Reporter: Ruslan Dautkhanov
>
> Spin off from SPARK-14743 and comments chain in [recent comments|
> https://issues.apache.org/jira/browse/SPARK-5493?focusedCommentId=15802179=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15802179]
> in SPARK-5493.
> Spark currently doesn't have a way for distribution new delegation tokens.
> Quoting [~vanzin] from SPARK-5493
> {quote}
> IIRC Livy doesn't yet support delegation token renewal. Once it reaches the
> TTL, the session is unusable.
> There might be ways to hack support for that without changes in Spark, but
> I'd like to see a proper API in Spark for distributing new delegation tokens.
> I mentioned that in SPARK-14743, but although that bug is closed, that
> particular feature hasn't been implemented yet.
> {quote}
> Other thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Comment Edited] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)
[
https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905247#comment-15905247
]
Thomas Graves edited comment on SPARK-19143 at 3/10/17 3:20 PM:
Made some comments in the design doc.
My original idea was to add rpc between client and driver so it works in both
yarn client and cluster mode. that rpc can be used for other things as well
(like a -kill command or get status straight from driver). For yarn side the
credentials are there and available from RM for a client to talk to driver, we
just have to make that token work with our rpc which I don't think should not
be to difficult just needs to be done. we can continue to support the
--keytab/--principal arguments and have the AM update the tokens. It would be
best to update that to push to executors via the rpc too.
was (Author: tgraves):
Made some comments in the design doc.
My original idea was to add rpc between client and driver so it works in both
yarn client and cluster mode. that rpc can be used for other things as well
(like a -kill command or get status straight from driver. For yarn side the
credentials are there and available from RM for a client to talk to driver, we
just have to make that token work with our rpc which I don't think should not
be to difficult just needs to be done. we can continue to support the
--keytab/--principal arguments and have the AM update the tokens. It would be
best to update that to push to executors via the rpc too.
> API in Spark for distributing new delegation tokens (Improve delegation token
> handling in secure clusters)
> --
>
> Key: SPARK-19143
> URL: https://issues.apache.org/jira/browse/SPARK-19143
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, YARN
>Affects Versions: 2.0.2, 2.1.0
>Reporter: Ruslan Dautkhanov
>
> Spin off from SPARK-14743 and comments chain in [recent comments|
> https://issues.apache.org/jira/browse/SPARK-5493?focusedCommentId=15802179=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15802179]
> in SPARK-5493.
> Spark currently doesn't have a way for distribution new delegation tokens.
> Quoting [~vanzin] from SPARK-5493
> {quote}
> IIRC Livy doesn't yet support delegation token renewal. Once it reaches the
> TTL, the session is unusable.
> There might be ways to hack support for that without changes in Spark, but
> I'd like to see a proper API in Spark for distributing new delegation tokens.
> I mentioned that in SPARK-14743, but although that bug is closed, that
> particular feature hasn't been implemented yet.
> {quote}
> Other thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)
[
https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15905247#comment-15905247
]
Thomas Graves commented on SPARK-19143:
---
Made some comments in the design doc.
My original idea was to add rpc between client and driver so it works in both
yarn client and cluster mode. that rpc can be used for other things as well
(like a -kill command or get status straight from driver. For yarn side the
credentials are there and available from RM for a client to talk to driver, we
just have to make that token work with our rpc which I don't think should not
be to difficult just needs to be done. we can continue to support the
--keytab/--principal arguments and have the AM update the tokens. It would be
best to update that to push to executors via the rpc too.
> API in Spark for distributing new delegation tokens (Improve delegation token
> handling in secure clusters)
> --
>
> Key: SPARK-19143
> URL: https://issues.apache.org/jira/browse/SPARK-19143
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, YARN
>Affects Versions: 2.0.2, 2.1.0
>Reporter: Ruslan Dautkhanov
>
> Spin off from SPARK-14743 and comments chain in [recent comments|
> https://issues.apache.org/jira/browse/SPARK-5493?focusedCommentId=15802179=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15802179]
> in SPARK-5493.
> Spark currently doesn't have a way for distribution new delegation tokens.
> Quoting [~vanzin] from SPARK-5493
> {quote}
> IIRC Livy doesn't yet support delegation token renewal. Once it reaches the
> TTL, the session is unusable.
> There might be ways to hack support for that without changes in Spark, but
> I'd like to see a proper API in Spark for distributing new delegation tokens.
> I mentioned that in SPARK-14743, but although that bug is closed, that
> particular feature hasn't been implemented yet.
> {quote}
> Other thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19812) YARN shuffle service fails to relocate recovery DB directories
[ https://issues.apache.org/jira/browse/SPARK-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15894584#comment-15894584 ] Thomas Graves commented on SPARK-19812: --- note that it will go ahead and start using the recovery db, it just doesn't copy over the old one so anything running gets lost. > YARN shuffle service fails to relocate recovery DB directories > -- > > Key: SPARK-19812 > URL: https://issues.apache.org/jira/browse/SPARK-19812 > Project: Spark > Issue Type: Bug > Components: YARN >Affects Versions: 2.0.1 >Reporter: Thomas Graves >Assignee: Thomas Graves > > The yarn shuffle service tries to switch from the yarn local directories to > the real recovery directory but can fail to move the existing recovery db's. > It fails due to Files.move not doing directories that have contents. > 2017-03-03 14:57:19,558 [main] ERROR yarn.YarnShuffleService: Failed to move > recovery file sparkShuffleRecovery.ldb to the path > /mapred/yarn-nodemanager/nm-aux-services/spark_shuffle > java.nio.file.DirectoryNotEmptyException:/yarn-local/sparkShuffleRecovery.ldb > at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:498) > at > sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) > at java.nio.file.Files.move(Files.java:1395) > at > org.apache.spark.network.yarn.YarnShuffleService.initRecoveryDb(YarnShuffleService.java:369) > at > org.apache.spark.network.yarn.YarnShuffleService.createSecretManager(YarnShuffleService.java:200) > at > org.apache.spark.network.yarn.YarnShuffleService.serviceInit(YarnShuffleService.java:174) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:143) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:262) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:357) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:636) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:684) > This used to use f.renameTo and we switched it in the pr due to review > comments and it looks like didn't do a final real test. The tests are using > files rather then directories so it didn't catch. We need to fix the test > also. > history: > https://github.com/apache/spark/pull/14999/commits/65de8531ccb91287f5a8a749c7819e99533b9440 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19812) YARN shuffle service fails to relocate recovery DB directories
[ https://issues.apache.org/jira/browse/SPARK-19812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-19812: -- Summary: YARN shuffle service fails to relocate recovery DB directories (was: YARN shuffle service fix moving recovery DB directories) > YARN shuffle service fails to relocate recovery DB directories > -- > > Key: SPARK-19812 > URL: https://issues.apache.org/jira/browse/SPARK-19812 > Project: Spark > Issue Type: Bug > Components: YARN >Affects Versions: 2.0.1 >Reporter: Thomas Graves >Assignee: Thomas Graves > > The yarn shuffle service tries to switch from the yarn local directories to > the real recovery directory but can fail to move the existing recovery db's. > It fails due to Files.move not doing directories that have contents. > 2017-03-03 14:57:19,558 [main] ERROR yarn.YarnShuffleService: Failed to move > recovery file sparkShuffleRecovery.ldb to the path > /mapred/yarn-nodemanager/nm-aux-services/spark_shuffle > java.nio.file.DirectoryNotEmptyException:/yarn-local/sparkShuffleRecovery.ldb > at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:498) > at > sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) > at java.nio.file.Files.move(Files.java:1395) > at > org.apache.spark.network.yarn.YarnShuffleService.initRecoveryDb(YarnShuffleService.java:369) > at > org.apache.spark.network.yarn.YarnShuffleService.createSecretManager(YarnShuffleService.java:200) > at > org.apache.spark.network.yarn.YarnShuffleService.serviceInit(YarnShuffleService.java:174) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:143) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:262) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:357) > at > org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:636) > at > org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:684) > This used to use f.renameTo and we switched it in the pr due to review > comments and it looks like didn't do a final real test. The tests are using > files rather then directories so it didn't catch. We need to fix the test > also. > history: > https://github.com/apache/spark/pull/14999/commits/65de8531ccb91287f5a8a749c7819e99533b9440 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-19812) YARN shuffle service fix moving recovery DB directories
Thomas Graves created SPARK-19812: - Summary: YARN shuffle service fix moving recovery DB directories Key: SPARK-19812 URL: https://issues.apache.org/jira/browse/SPARK-19812 Project: Spark Issue Type: Bug Components: YARN Affects Versions: 2.0.1 Reporter: Thomas Graves Assignee: Thomas Graves The yarn shuffle service tries to switch from the yarn local directories to the real recovery directory but can fail to move the existing recovery db's. It fails due to Files.move not doing directories that have contents. 2017-03-03 14:57:19,558 [main] ERROR yarn.YarnShuffleService: Failed to move recovery file sparkShuffleRecovery.ldb to the path /mapred/yarn-nodemanager/nm-aux-services/spark_shuffle java.nio.file.DirectoryNotEmptyException:/yarn-local/sparkShuffleRecovery.ldb at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:498) at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262) at java.nio.file.Files.move(Files.java:1395) at org.apache.spark.network.yarn.YarnShuffleService.initRecoveryDb(YarnShuffleService.java:369) at org.apache.spark.network.yarn.YarnShuffleService.createSecretManager(YarnShuffleService.java:200) at org.apache.spark.network.yarn.YarnShuffleService.serviceInit(YarnShuffleService.java:174) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.containermanager.AuxServices.serviceInit(AuxServices.java:143) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:262) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:357) at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:636) at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:684) This used to use f.renameTo and we switched it in the pr due to review comments and it looks like didn't do a final real test. The tests are using files rather then directories so it didn't catch. We need to fix the test also. history: https://github.com/apache/spark/pull/14999/commits/65de8531ccb91287f5a8a749c7819e99533b9440 -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18769) Spark to be smarter about what the upper bound is and to restrict number of executor when dynamic allocation is enabled
[ https://issues.apache.org/jira/browse/SPARK-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15892318#comment-15892318 ] Thomas Graves commented on SPARK-18769: --- I definitely understand there is an actual problem here, but I think the problem is more with Spark and its event processing/synchronization then the fact we are asking for more containers.Like I mention I agree with doing the jira I just want to clarify why we are doing it and make sure we do it such that it doesn't hurt our container allocation. Its always good to play nice in the yarn environment and not ask for more containers then the entire cluster can handle for instance, but at the same time if we are limiting the container requests early on, yarn could easily free up resource and make them available for you. If you don't have your request in yarn could give those to someone else. There are a lot of configs in the yarn schedulers and different situations.If you look at some other apps on yarn (MR and TEZ), both immediately ask for all of their resource. MR is definitely different since it doesn't reuse containers, TEZ does. With asking for everything immediately you can definitely hit issues where if your tasks run really fast then you don't need all of those containers, but the exponential ramp up on our allocation now gets you their really quickly anyway and I think you can hit the same issue. Note that in our clusters we set the upper limit by default to something reasonable (couple thousand) and if someone has really large job they can reconfigure. > Spark to be smarter about what the upper bound is and to restrict number of > executor when dynamic allocation is enabled > > > Key: SPARK-18769 > URL: https://issues.apache.org/jira/browse/SPARK-18769 > Project: Spark > Issue Type: New Feature >Reporter: Neerja Khattar > > Currently when dynamic allocation is enabled max.executor is infinite and > spark creates so many executor and even exceed the yarn nodemanager memory > limit and vcores. > It should have a check to not exceed more that yarn resource limit. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18769) Spark to be smarter about what the upper bound is and to restrict number of executor when dynamic allocation is enabled
[ https://issues.apache.org/jira/browse/SPARK-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15890629#comment-15890629 ] Thomas Graves commented on SPARK-18769: --- [~yuming] I already made a comment on that, I don't think we should be looking at queue configs to do this. We can use the resources returned from the call to allocate to get a more accurate picture without having to know the internals of the RM queues.The max capacity of the queue could be the entire cluster which could be much much larger then a user can actually get based on other queue configs and who else is using the queue. [~vanzin] that makes sense but I'm not sure there is anyway around that or that it matters that much. The yarn api requires 1 container request per container.It makes sense to try to limit if we don't really need those or the cluster has no where near those resources. The reply from resource manager isn't based on the # of requests, it just updates the requests/releases and returns what the scheduler has allocated to it in between heartbeats. Scheduler runs async to that that updates the allocations for that application. ignoring the cluster capacity issue, the hard thing about the dynamic allocation is determine if the tasks will be quick and thus might not need all the containers because tasks finish faster then we can allocate executors and use them. > Spark to be smarter about what the upper bound is and to restrict number of > executor when dynamic allocation is enabled > > > Key: SPARK-18769 > URL: https://issues.apache.org/jira/browse/SPARK-18769 > Project: Spark > Issue Type: New Feature >Reporter: Neerja Khattar > > Currently when dynamic allocation is enabled max.executor is infinite and > spark creates so many executor and even exceed the yarn nodemanager memory > limit and vcores. > It should have a check to not exceed more that yarn resource limit. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18769) Spark to be smarter about what the upper bound is and to restrict number of executor when dynamic allocation is enabled
[
https://issues.apache.org/jira/browse/SPARK-18769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15888430#comment-15888430
]
Thomas Graves commented on SPARK-18769:
---
{quote}
A little clarification in case the summary is not clear: Spark's dynamic
allocation will keep growing the number of requested executors until it reaches
the upper limit, even when the cluster manager hasn't really been allocating
new executors. This is sub-optimal, especially since, I believe, this increases
the memory usage in Spark unnecessarily, and might also put unnecessary burden
on the cluster manager.
{quote}
What do you mean by this? Spark can change its asks to yarn at any time, this
doesn't affect its actual usage of resources until things get allocated by the
yarn resourcemanager. As far as more pressure on the cluster manager, that
should only happen if we are increasing our heartbeats or interval at which we
ask for resources, which does happen more frequently as we ask for more
containers but it backs off pretty quickly. It does leave one more application
the yarn schedulers list to look at but if Spark could actually use more
containers I don't see a problem with this. If we are purely asking for more
resources then we can use then we definitely shouldn't do it. I know there
were a few issues with older versions of hadoop, I thought they were more
around releasing containers but I'll have to double check on that.
I think it would be good to be a little smarter here but at the same time if
you don't request things fast enough you just make things slower. Yarn can
handle you requesting more resources then it can give you as it handles all the
queue limits and such and if space does free up then you would get more
resources.
What is the real problem here that we are trying to solve or what are we trying
to enhance? I do definitely see issues with yarn giving spark many containers
and spark just taking a long time to bring them up and utilize them. This
definitely wastes resources. Spark obviously has an issue with event
processing and synchronization which I think causes some of that.I haven't
had time to investigate it further but if you have it would be great to hear
what you have found.
I also think its a bit weird the way we ramp up containers. I'm not sure why
we don't just ask for the number we need immediately. This would reduce the
number of changes to asks and number of messages flying around. I know we ramp
up pretty quickly but again that might just be adding overhead.jk
> Spark to be smarter about what the upper bound is and to restrict number of
> executor when dynamic allocation is enabled
>
>
> Key: SPARK-18769
> URL: https://issues.apache.org/jira/browse/SPARK-18769
> Project: Spark
> Issue Type: New Feature
>Reporter: Neerja Khattar
>
> Currently when dynamic allocation is enabled max.executor is infinite and
> spark creates so many executor and even exceed the yarn nodemanager memory
> limit and vcores.
> It should have a check to not exceed more that yarn resource limit.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15849965#comment-15849965 ] Thomas Graves commented on SPARK-18085: --- Are the ui updates just updates to use new api or are you also changing to use datatable or other? > Better History Server scalability for many / large applications > --- > > Key: SPARK-18085 > URL: https://issues.apache.org/jira/browse/SPARK-18085 > Project: Spark > Issue Type: Umbrella > Components: Spark Core, Web UI >Affects Versions: 2.0.0 >Reporter: Marcelo Vanzin > Attachments: spark_hs_next_gen.pdf > > > It's a known fact that the History Server currently has some annoying issues > when serving lots of applications, and when serving large applications. > I'm filing this umbrella to track work related to addressing those issues. > I'll be attaching a document shortly describing the issues and suggesting a > path to how to solve them. -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] (SPARK-16742) Kerberos support for Spark on Mesos
Title: Message Title Thomas Graves commented on SPARK-16742 Re: Kerberos support for Spark on Mesos I skimmed over the existing design doc. From a security point of view it would be much better if you can just send tokens and not keytabs or TGT's. If keytabs are compromised user can use forever, if TGT is compromised user can use for X amount of time but can get other tokens for other services, if token is compromised its only for x amount of time for that particular service. I assume you already have something on the client side pushing new TGT's? If so it should be trivial to change to do tokens. Definitely sending over rpc to executors is better. I want to do both of those things on the yarn side (send over rpc and send only tokens). Add Comment This message was sent by Atlassian JIRA (v6.3.15#6346-sha1:dbc023d)
[jira] [Resolved] (SPARK-18750) spark should be able to control the number of executor and should not throw stack overslow
[
https://issues.apache.org/jira/browse/SPARK-18750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves resolved SPARK-18750.
---
Resolution: Fixed
Assignee: Marcelo Vanzin
Fix Version/s: 2.2.0
2.1.1
> spark should be able to control the number of executor and should not throw
> stack overslow
> --
>
> Key: SPARK-18750
> URL: https://issues.apache.org/jira/browse/SPARK-18750
> Project: Spark
> Issue Type: Bug
>Reporter: Neerja Khattar
>Assignee: Marcelo Vanzin
> Fix For: 2.1.1, 2.2.0
>
>
> When running Sql queries on large datasets. Job fails with stack overflow
> warning and it shows it is requesting lots of executors.
> Looks like there is no limit to number of executors or not even having an
> upperbound based on yarn available resources.
> {noformat}
> 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
> bdtcstr61n5.svr.us.jpmchase.net:8041
> 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
> bdtcstr61n8.svr.us.jpmchase.net:8041
> 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
> bdtcstr61n2.svr.us.jpmchase.net:8041
> 16/11/29 15:47:47 INFO yarn.YarnAllocator: Driver requested a total number of
> 32770 executor(s).
> 16/11/29 15:47:47 INFO yarn.YarnAllocator: Will request 24576 executor
> containers, each with 1 cores and 6758 MB memory including 614 MB overhead
> 16/11/29 15:49:11 INFO yarn.YarnAllocator: Driver requested a total number of
> 52902 executor(s).
> 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
> bdtcstr61n5.svr.us.jpmchase.net:8041
> 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
> bdtcstr61n8.svr.us.jpmchase.net:8041
> 16/11/29 15:47:47 INFO impl.ContainerManagementProtocolProxy: Opening proxy :
> bdtcstr61n2.svr.us.jpmchase.net:8041
> 16/11/29 15:47:47 INFO yarn.YarnAllocator: Driver requested a total number of
> 32770 executor(s).
> 16/11/29 15:47:47 INFO yarn.YarnAllocator: Will request 24576 executor
> containers, each with 1 cores and 6758 MB memory including 614 MB overhead
> 16/11/29 15:49:11 INFO yarn.YarnAllocator: Driver requested a total number of
> 52902 executor(s).
> 16/11/29 15:49:11 WARN yarn.ApplicationMaster: Reporter thread fails 1
> time(s) in a row.
> java.lang.StackOverflowError
> at scala.collection.immutable.HashMap.$plus(HashMap.scala:57)
> at scala.collection.immutable.HashMap.$plus(HashMap.scala:36)
> at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:28)
> at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:24)
> at
> scala.collection.generic.Growable$$anonfun$$plus$plus$eq$1.apply(Growable.scala:48)
> at
> scala.collection.generic.Growable$$anonfun$$plus$plus$eq$1.apply(Growable.scala:48)
> at
> scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224)
> at
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
> at
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
> at
> scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:48)
> at
> scala.collection.mutable.MapBuilder.$plus$plus$eq(MapBuilder.scala:24)
> at
> scala.collection.TraversableLike$class.$plus$plus(TraversableLike.scala:156)
> at
> scala.collection.AbstractTraversable.$plus$plus(Traversable.scala:105)
> at scala.collection.immutable.HashMap.$plus(HashMap.scala:60)
> at scala.collection.immutable.Map$Map4.updated(Map.scala:172)
> at scala.collection.immutable.Map$Map4.$plus(Map.scala:173)
> at scala.collection.immutable.Map$Map4.$plus(Map.scala:158)
> at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:28)
> at scala.collection.mutable.MapBuilder.$plus$eq(MapBuilder.scala:24)
> at
> scala.collection.TraversableLike$$anonfun$filter$1.apply(TraversableLike.scala:264)
> at
> scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
> at
> scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
> at
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
> at
> scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
> at
> scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
> at
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
> at
> scala.collection.MapLike$MappedValues$$anonfun$foreach$3.apply(MapLike.scala:245)
> at
>
[jira] [Resolved] (SPARK-19179) spark.yarn.access.namenodes description is wrong
[ https://issues.apache.org/jira/browse/SPARK-19179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-19179. --- Resolution: Fixed Fix Version/s: 2.2.0 > spark.yarn.access.namenodes description is wrong > > > Key: SPARK-19179 > URL: https://issues.apache.org/jira/browse/SPARK-19179 > Project: Spark > Issue Type: Bug > Components: YARN >Affects Versions: 2.0.2 >Reporter: Thomas Graves >Assignee: Saisai Shao >Priority: Minor > Fix For: 2.2.0 > > > The description and name of spark.yarn.access.namenodesis off. It > says this is for HDFS namenodes when really this is to specify any hadoop > filesystems. It gets the credentials for those filesystems. > We should at least update the description on it to be more generic. We could > change the name on it but we would have to deprecated it and keep around > current name as many people use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19179) spark.yarn.access.namenodes description is wrong
[ https://issues.apache.org/jira/browse/SPARK-19179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-19179: -- Assignee: Saisai Shao > spark.yarn.access.namenodes description is wrong > > > Key: SPARK-19179 > URL: https://issues.apache.org/jira/browse/SPARK-19179 > Project: Spark > Issue Type: Bug > Components: YARN >Affects Versions: 2.0.2 >Reporter: Thomas Graves >Assignee: Saisai Shao >Priority: Minor > > The description and name of spark.yarn.access.namenodesis off. It > says this is for HDFS namenodes when really this is to specify any hadoop > filesystems. It gets the credentials for those filesystems. > We should at least update the description on it to be more generic. We could > change the name on it but we would have to deprecated it and keep around > current name as many people use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-19179) spark.yarn.access.namenodes description is wrong
Thomas Graves created SPARK-19179: - Summary: spark.yarn.access.namenodes description is wrong Key: SPARK-19179 URL: https://issues.apache.org/jira/browse/SPARK-19179 Project: Spark Issue Type: Bug Components: YARN Affects Versions: 2.0.2 Reporter: Thomas Graves Priority: Minor The description and name of spark.yarn.access.namenodes is off. It says this is for HDFS namenodes when really this is to specify any hadoop filesystems. It gets the credentials for those filesystems. We should at least update the description on it to be more generic. We could change the name on it but we would have to deprecated it and keep around current name as many people use it. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-19021) Generailize HDFSCredentialProvider to support non HDFS security FS
[
https://issues.apache.org/jira/browse/SPARK-19021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves resolved SPARK-19021.
---
Resolution: Fixed
Assignee: Saisai Shao
Fix Version/s: 2.2.0
> Generailize HDFSCredentialProvider to support non HDFS security FS
> --
>
> Key: SPARK-19021
> URL: https://issues.apache.org/jira/browse/SPARK-19021
> Project: Spark
> Issue Type: Improvement
> Components: YARN
>Affects Versions: 2.1.0
>Reporter: Saisai Shao
>Assignee: Saisai Shao
>Priority: Minor
> Fix For: 2.2.0
>
>
> Currently Spark can only get token renewal interval from security HDFS
> (hdfs://), if Spark runs with other security file systems like webHDFS
> (webhdfs://), wasb (wasb://), ADLS, it will ignore these tokens and not get
> token renewal intervals from these tokens. These will make Spark unable to
> work with these security clusters. So instead of only checking HDFS token, we
> should generalize to support different {{DelegationTokenIdentifier}}.
> This is a follow-up work of SPARK-18840.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19143) API in Spark for distributing new delegation tokens (Improve delegation token handling in secure clusters)
[
https://issues.apache.org/jira/browse/SPARK-19143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15815078#comment-15815078
]
Thomas Graves commented on SPARK-19143:
---
[~mridulm80] You say "we added", you are saying you have already implemented
this?
It would definitely be nice to support pushing tokens from say a gateway so you
don't have to ship the keytab because shipping the keytab is much less secure.
It would also be nice to not use HDFS to store and transfer the tokens. My
initial thought was to create an rpc between the client on gateways and the
driver/AM (running on yarn node) and transfer the new tokens that way. ideally
it would also then be transferred to the executors via rpc vs again storing in
hdfs. All that would be more secure then storing in hdfs.
We could add a command to spark-submit to get and push new credentials. For
long running jobs it would have to happen periodically (< every 24 hours) but
for initial that could be done via cron or other mechanism.
> API in Spark for distributing new delegation tokens (Improve delegation token
> handling in secure clusters)
> --
>
> Key: SPARK-19143
> URL: https://issues.apache.org/jira/browse/SPARK-19143
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core, YARN
>Affects Versions: 2.0.2, 2.1.0
>Reporter: Ruslan Dautkhanov
>
> Spin off from SPARK-14743 and comments chain in [recent comments|
> https://issues.apache.org/jira/browse/SPARK-5493?focusedCommentId=15802179=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15802179]
> in SPARK-5493.
> Spark currently doesn't have a way for distribution new delegation tokens.
> Quoting [~vanzin] from SPARK-5493
> {quote}
> IIRC Livy doesn't yet support delegation token renewal. Once it reaches the
> TTL, the session is unusable.
> There might be ways to hack support for that without changes in Spark, but
> I'd like to see a proper API in Spark for distributing new delegation tokens.
> I mentioned that in SPARK-14743, but although that bug is closed, that
> particular feature hasn't been implemented yet.
> {quote}
> Other thoughts?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19112) add codec for ZStandard
[ https://issues.apache.org/jira/browse/SPARK-19112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15811935#comment-15811935 ] Thomas Graves commented on SPARK-19112: --- For any sort of Hadoop output formats we get it for free but for the internal and shuffle outputs I think we have to add support to spark.io.compression.codec for this. I'm also wondering if there are other output formats like parquet that we would want to add this option to (spark.sql.parquet.compression.codec), but I haven't investigated those enough. > add codec for ZStandard > --- > > Key: SPARK-19112 > URL: https://issues.apache.org/jira/browse/SPARK-19112 > Project: Spark > Issue Type: Improvement > Components: Spark Core >Reporter: Thomas Graves >Priority: Minor > > ZStandard: https://github.com/facebook/zstd and > http://facebook.github.io/zstd/ has been in use for a while now. v1.0 was > recently released. Hadoop > (https://issues.apache.org/jira/browse/HADOOP-13578) and others > (https://issues.apache.org/jira/browse/KAFKA-4514) are adopting it. > Zstd seems to give great results => Gzip level Compression with Lz4 level CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Created] (SPARK-19112) add codec for ZStandard
Thomas Graves created SPARK-19112: - Summary: add codec for ZStandard Key: SPARK-19112 URL: https://issues.apache.org/jira/browse/SPARK-19112 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Thomas Graves ZStandard: https://github.com/facebook/zstd and http://facebook.github.io/zstd/ has been in use for a while now. v1.0 was recently released. Hadoop (https://issues.apache.org/jira/browse/HADOOP-13578) and others (https://issues.apache.org/jira/browse/KAFKA-4514) are adopting it. Zstd seems to give great results => Gzip level Compression with Lz4 level CPU. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-19033) HistoryServer still uses old ACLs even if ACLs are updated
[
https://issues.apache.org/jira/browse/SPARK-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves resolved SPARK-19033.
---
Resolution: Fixed
Fix Version/s: 2.2.0
2.1.1
> HistoryServer still uses old ACLs even if ACLs are updated
> --
>
> Key: SPARK-19033
> URL: https://issues.apache.org/jira/browse/SPARK-19033
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.1.0
>Reporter: Saisai Shao
>Assignee: Saisai Shao
>Priority: Minor
> Fix For: 2.1.1, 2.2.0
>
>
> In the current implementation of HistoryServer, Application ACLs is picked
> from event log rather than configuration:
> {code}
> val uiAclsEnabled =
> conf.getBoolean("spark.history.ui.acls.enable", false)
> ui.getSecurityManager.setAcls(uiAclsEnabled)
> // make sure to set admin acls before view acls so they are
> properly picked up
>
> ui.getSecurityManager.setAdminAcls(appListener.adminAcls.getOrElse(""))
> ui.getSecurityManager.setViewAcls(attempt.sparkUser,
> appListener.viewAcls.getOrElse(""))
>
> ui.getSecurityManager.setAdminAclsGroups(appListener.adminAclsGroups.getOrElse(""))
>
> ui.getSecurityManager.setViewAclsGroups(appListener.viewAclsGroups.getOrElse(""))
> {code}
> This will become a problem when ACLs is updated (newly added admin), only the
> new application can be effected, the old applications were still using the
> old ACLs. So these new admin still cannot check the logs of old applications.
> It is hard to say this is a bug, but in our scenario this is not the expected
> behavior we wanted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-19033) HistoryServer still uses old ACLs even if ACLs are updated
[
https://issues.apache.org/jira/browse/SPARK-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves updated SPARK-19033:
--
Assignee: Saisai Shao
> HistoryServer still uses old ACLs even if ACLs are updated
> --
>
> Key: SPARK-19033
> URL: https://issues.apache.org/jira/browse/SPARK-19033
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.1.0
>Reporter: Saisai Shao
>Assignee: Saisai Shao
>Priority: Minor
>
> In the current implementation of HistoryServer, Application ACLs is picked
> from event log rather than configuration:
> {code}
> val uiAclsEnabled =
> conf.getBoolean("spark.history.ui.acls.enable", false)
> ui.getSecurityManager.setAcls(uiAclsEnabled)
> // make sure to set admin acls before view acls so they are
> properly picked up
>
> ui.getSecurityManager.setAdminAcls(appListener.adminAcls.getOrElse(""))
> ui.getSecurityManager.setViewAcls(attempt.sparkUser,
> appListener.viewAcls.getOrElse(""))
>
> ui.getSecurityManager.setAdminAclsGroups(appListener.adminAclsGroups.getOrElse(""))
>
> ui.getSecurityManager.setViewAclsGroups(appListener.viewAclsGroups.getOrElse(""))
> {code}
> This will become a problem when ACLs is updated (newly added admin), only the
> new application can be effected, the old applications were still using the
> old ACLs. So these new admin still cannot check the logs of old applications.
> It is hard to say this is a bug, but in our scenario this is not the expected
> behavior we wanted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-19033) HistoryServer still uses old ACLs even if ACLs are updated
[
https://issues.apache.org/jira/browse/SPARK-19033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15795863#comment-15795863
]
Thomas Graves commented on SPARK-19033:
---
this should actually be an improvement to allow admin acls from history server
config to also apply. I've been meaning to do this for a while, thought I filed
jira for it but don't see it now.
Currently it uses what the application sets for acls, which in general is
correct because you don't want the user level acls being applied differently
then what the application had originally set, this would be a security hole and
you would be applying one set of history acls across all applications. The
exception to that is the admin level acls, the idea is that these are people
with special privileges and like you mention this could change. Although I
would also encourage you to use groups which helps with this issue as you
shouldn't need a huge list of individual users but its still also possible for
a user to blank out the admin acls so it would be good to allow the history
server to add those back.
So this jira should just add another option to append history server admin acls
to application ones. Something like spark.history.ui.admin.acls and
spark.history.ui.admin.acls.groups
> HistoryServer still uses old ACLs even if ACLs are updated
> --
>
> Key: SPARK-19033
> URL: https://issues.apache.org/jira/browse/SPARK-19033
> Project: Spark
> Issue Type: Bug
> Components: Spark Core
>Affects Versions: 2.1.0
>Reporter: Saisai Shao
>Priority: Minor
>
> In the current implementation of HistoryServer, Application ACLs is picked
> from event log rather than configuration:
> {code}
> val uiAclsEnabled =
> conf.getBoolean("spark.history.ui.acls.enable", false)
> ui.getSecurityManager.setAcls(uiAclsEnabled)
> // make sure to set admin acls before view acls so they are
> properly picked up
>
> ui.getSecurityManager.setAdminAcls(appListener.adminAcls.getOrElse(""))
> ui.getSecurityManager.setViewAcls(attempt.sparkUser,
> appListener.viewAcls.getOrElse(""))
>
> ui.getSecurityManager.setAdminAclsGroups(appListener.adminAclsGroups.getOrElse(""))
>
> ui.getSecurityManager.setViewAclsGroups(appListener.viewAclsGroups.getOrElse(""))
> {code}
> This will become a problem when ACLs is updated (newly added admin), only the
> new application can be effected, the old applications were still using the
> old ACLs. So these new admin still cannot check the logs of old applications.
> It is hard to say this is a bug, but in our scenario this is not the expected
> behavior we wanted.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18733) Spark history server file cleaner excludes in-progress files
[ https://issues.apache.org/jira/browse/SPARK-18733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726701#comment-15726701 ] Thomas Graves commented on SPARK-18733: --- oh nevermind its looking at lastupdated time. > Spark history server file cleaner excludes in-progress files > > > Key: SPARK-18733 > URL: https://issues.apache.org/jira/browse/SPARK-18733 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.0.2 >Reporter: Ergin Seyfe > > When we restart history server, it does spend a lot of time to load/replay > incomplete applications which mean the inprogress log files in the log folder. > We have already enabled "spark.history.fs.cleaner.enabled" but seems like > it's skipping the inprogress files. > I checked the log folder and saw that there are many old orphan files. > Probably files left over due to spark-driver failures or OOMs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18733) Spark history server file cleaner excludes in-progress files
[ https://issues.apache.org/jira/browse/SPARK-18733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15726671#comment-15726671 ] Thomas Graves commented on SPARK-18733: --- yes looks like a dup but I'm not sure on current solution. max age could easily be set to say 1 week and a job could easily run longer then 1 week. I'll post comment on PR about that though. > Spark history server file cleaner excludes in-progress files > > > Key: SPARK-18733 > URL: https://issues.apache.org/jira/browse/SPARK-18733 > Project: Spark > Issue Type: Bug > Components: Web UI >Affects Versions: 2.0.2 >Reporter: Ergin Seyfe > > When we restart history server, it does spend a lot of time to load/replay > incomplete applications which mean the inprogress log files in the log folder. > We have already enabled "spark.history.fs.cleaner.enabled" but seems like > it's skipping the inprogress files. > I checked the log folder and saw that there are many old orphan files. > Probably files left over due to spark-driver failures or OOMs. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-17843) History Server gives no feedback about the application list being incomplete
[ https://issues.apache.org/jira/browse/SPARK-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated SPARK-17843: -- Fix Version/s: 2.0.3 > History Server gives no feedback about the application list being incomplete > > > Key: SPARK-17843 > URL: https://issues.apache.org/jira/browse/SPARK-17843 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 2.0.0, 2.0.1 >Reporter: Vinayak Joshi >Assignee: Vinayak Joshi >Priority: Minor > Fix For: 2.0.3, 2.1.0, 2.2.0 > > > A user who lands on the history server's application listing page before the > history server's background processing has finished replaying the existing > event logs complains why a particular application is not included in the > list. There is no indication on the UI about the possible reason, or that > refreshing the page after some time may show up the application the user > expects to see. This problem is very noticeable when there are large sized > event logs that take a long time to finish replaying. > The usability problems with large (number or size) event logs and the history > server have been known. Particularly SPARK-5522, SPARK-13988 and other issues > referenced within them highlight the problems and the improvements done so > far. To improve the history server startup time and reduce the impact of > large event log files, the event logs are now processed (replayed) via a pool > of threads. This allows a user to browse to the application listing page > before the event logs have finished replaying. After history server startup, > a user expects to see any old completed applications to appear in the > application list page. But unless the corresponding event logs have finished > replaying, the application wont be in the list, and the user complains. There > is no feedback to the user about this on the UI, hence this JIRA to try and > address this problem. > Idea is to give some indication of the number of event logs that are pending > replay to the user. Note that the way the replay is currently designed, one > cycle of "check for logs that need to be replayed > replay the logs > update > application info" needs to complete before a new one begins. Therefore, it > should be possible for the FsApplicationHistoryProvider to send info about > the number of logs that are currently pending processing. This in turn would > address the user anxiety of not seeing the application they expect to see. > I will be attaching a pull request with my initial take on implementing this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Commented] (SPARK-18085) Better History Server scalability for many / large applications
[ https://issues.apache.org/jira/browse/SPARK-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15667226#comment-15667226 ] Thomas Graves commented on SPARK-18085: --- I think at a high level its fine. You mention using a /v2 api for now, is this just for development or are you going to change it to v1 once its all working? Or are you going to wait to see if the api changes? It seems to me the only reason to keep a /v2 would be if api changes. User shouldn't really care about the backend so ideally wouldn't have to change api for that. > Better History Server scalability for many / large applications > --- > > Key: SPARK-18085 > URL: https://issues.apache.org/jira/browse/SPARK-18085 > Project: Spark > Issue Type: Umbrella > Components: Spark Core, Web UI >Affects Versions: 2.0.0 >Reporter: Marcelo Vanzin > Attachments: spark_hs_next_gen.pdf > > > It's a known fact that the History Server currently has some annoying issues > when serving lots of applications, and when serving large applications. > I'm filing this umbrella to track work related to addressing those issues. > I'll be attaching a document shortly describing the issues and suggesting a > path to how to solve them. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-17843) History Server gives no feedback about the application list being incomplete
[ https://issues.apache.org/jira/browse/SPARK-17843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves resolved SPARK-17843. --- Resolution: Fixed Assignee: Vinayak Joshi Fix Version/s: 2.2.0 2.1.0 > History Server gives no feedback about the application list being incomplete > > > Key: SPARK-17843 > URL: https://issues.apache.org/jira/browse/SPARK-17843 > Project: Spark > Issue Type: Improvement > Components: Web UI >Affects Versions: 2.0.0, 2.0.1 >Reporter: Vinayak Joshi >Assignee: Vinayak Joshi >Priority: Minor > Fix For: 2.1.0, 2.2.0 > > > A user who lands on the history server's application listing page before the > history server's background processing has finished replaying the existing > event logs complains why a particular application is not included in the > list. There is no indication on the UI about the possible reason, or that > refreshing the page after some time may show up the application the user > expects to see. This problem is very noticeable when there are large sized > event logs that take a long time to finish replaying. > The usability problems with large (number or size) event logs and the history > server have been known. Particularly SPARK-5522, SPARK-13988 and other issues > referenced within them highlight the problems and the improvements done so > far. To improve the history server startup time and reduce the impact of > large event log files, the event logs are now processed (replayed) via a pool > of threads. This allows a user to browse to the application listing page > before the event logs have finished replaying. After history server startup, > a user expects to see any old completed applications to appear in the > application list page. But unless the corresponding event logs have finished > replaying, the application wont be in the list, and the user complains. There > is no feedback to the user about this on the UI, hence this JIRA to try and > address this problem. > Idea is to give some indication of the number of event logs that are pending > replay to the user. Note that the way the replay is currently designed, one > cycle of "check for logs that need to be replayed > replay the logs > update > application info" needs to complete before a new one begins. Therefore, it > should be possible for the FsApplicationHistoryProvider to send info about > the number of logs that are currently pending processing. This in turn would > address the user anxiety of not seeing the application they expect to see. > I will be attaching a pull request with my initial take on implementing this. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Resolved] (SPARK-18357) YARN --files/--archives broke
[
https://issues.apache.org/jira/browse/SPARK-18357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves resolved SPARK-18357.
---
Resolution: Fixed
Fix Version/s: 2.2.0
2.1.0
> YARN --files/--archives broke
> -
>
> Key: SPARK-18357
> URL: https://issues.apache.org/jira/browse/SPARK-18357
> Project: Spark
> Issue Type: Bug
> Components: YARN
>Affects Versions: 2.1.0
>Reporter: Thomas Graves
>Priority: Blocker
> Fix For: 2.1.0, 2.2.0
>
>
> SPARK-18099 broke --files and --archives options. The check should be ==null
> instead of !=:
> if (localizedPath != null) {
> +throw new IllegalArgumentException(s"Attempt to add ($file)
> multiple times" +
> + " to the distributed cache.")
> + }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
[jira] [Updated] (SPARK-18357) YARN --files/--archives broke
[
https://issues.apache.org/jira/browse/SPARK-18357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Thomas Graves updated SPARK-18357:
--
Assignee: Kishor Patil
> YARN --files/--archives broke
> -
>
> Key: SPARK-18357
> URL: https://issues.apache.org/jira/browse/SPARK-18357
> Project: Spark
> Issue Type: Bug
> Components: YARN
>Affects Versions: 2.1.0
>Reporter: Thomas Graves
>Assignee: Kishor Patil
>Priority: Blocker
> Fix For: 2.1.0, 2.2.0
>
>
> SPARK-18099 broke --files and --archives options. The check should be ==null
> instead of !=:
> if (localizedPath != null) {
> +throw new IllegalArgumentException(s"Attempt to add ($file)
> multiple times" +
> + " to the distributed cache.")
> + }
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
-
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
