[jira] [Resolved] (YARN-218) Distiguish between failed and killed app attempts

[ https://issues.apache.org/jira/browse/YARN-218?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White resolved YARN-218. Resolution: Duplicate Fixed in YARN-614. Distiguish between failed and killed app attempts

[jira] [Commented] (YARN-1029) Allow embedding leader election into the RM

[ https://issues.apache.org/jira/browse/YARN-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13849016#comment-13849016 ] Tom White commented on YARN-1029: - RM HA uses ZK itself for shared storage, so it already

[jira] [Commented] (YARN-1028) Add FailoverProxyProvider like capability to RMProxy

[ https://issues.apache.org/jira/browse/YARN-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13847432#comment-13847432 ] Tom White commented on YARN-1028: - Thanks for addressing the feedback I gave. +1 for the

[jira] [Commented] (YARN-1028) Add FailoverProxyProvider like capability to RMProxy

[ https://issues.apache.org/jira/browse/YARN-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13846395#comment-13846395 ] Tom White commented on YARN-1028: - Thanks for the explanation of how failover works,

[jira] [Commented] (YARN-1028) Add FailoverProxyProvider like capability to RMProxy

[ https://issues.apache.org/jira/browse/YARN-1028?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13845471#comment-13845471 ] Tom White commented on YARN-1028: - It looks like the behaviour in this patch differs from

[jira] [Commented] (YARN-1029) Allow embedding leader election into the RM

[ https://issues.apache.org/jira/browse/YARN-1029?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13845491#comment-13845491 ] Tom White commented on YARN-1029: - Implementing ActiveStandbyElector sounds like a good

[jira] [Commented] (YARN-1144) Unmanaged AMs registering a tracking URI should not be proxy-fied

[ https://issues.apache.org/jira/browse/YARN-1144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13761732#comment-13761732 ] Tom White commented on YARN-1144: - +1 Unmanaged AMs registering a

[jira] [Commented] (YARN-884) AM expiry interval should be set to smaller of {am, nm}.liveness-monitor.expiry-interval-ms

[ https://issues.apache.org/jira/browse/YARN-884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13706826#comment-13706826 ] Tom White commented on YARN-884: The approach seems reasonable to me. +1 for the patch.

[jira] [Commented] (YARN-803) factor out scheduler config validation from the ResourceManager to each scheduler implementation

[ https://issues.apache.org/jira/browse/YARN-803?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683454#comment-13683454 ] Tom White commented on YARN-803: Looks good to me. Is there a reason

[jira] [Commented] (YARN-789) Enable zero capabilities resource requests in fair scheduler

[ https://issues.apache.org/jira/browse/YARN-789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13683466#comment-13683466 ] Tom White commented on YARN-789: This looks good to me, +1. The increment setting only

[jira] [Commented] (YARN-689) Add multiplier unit to resourcecapabilities

[ https://issues.apache.org/jira/browse/YARN-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13677937#comment-13677937 ] Tom White commented on YARN-689: +1 to Hitesh and Bikas' points about minimum (and

[jira] [Commented] (YARN-689) Add multiplier unit to resourcecapabilities

[ https://issues.apache.org/jira/browse/YARN-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13676843#comment-13676843 ] Tom White commented on YARN-689: The multiplier seems to be one approach to do this, though

[jira] [Commented] (YARN-689) Add multiplier unit to resourcecapabilities

[ https://issues.apache.org/jira/browse/YARN-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13675988#comment-13675988 ] Tom White commented on YARN-689: I had a look at the latest patch. Why is

[jira] [Commented] (YARN-689) Add multiplier unit to resourcecapabilities

[ https://issues.apache.org/jira/browse/YARN-689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13673582#comment-13673582 ] Tom White commented on YARN-689: It seems that there's a concern about impacting CS with

[jira] [Commented] (YARN-595) Refactor fair scheduler to use common Resources

[ https://issues.apache.org/jira/browse/YARN-595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13640612#comment-13640612 ] Tom White commented on YARN-595: +1 Refactor fair scheduler to use common

[jira] [Commented] (YARN-142) Change YARN APIs to throw IOException

[ https://issues.apache.org/jira/browse/YARN-142?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573516#comment-13573516 ] Tom White commented on YARN-142: I think it'd be useful to have the APIs throw IOException

[jira] [Commented] (YARN-371) Consolidate resource requests in AM-RM heartbeat

[ https://issues.apache.org/jira/browse/YARN-371?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13570109#comment-13570109 ] Tom White commented on YARN-371: Looks like there's a misunderstanding here - Sandy talks

[jira] [Commented] (YARN-333) On app submission, have RM ask scheduler for queue name

[ https://issues.apache.org/jira/browse/YARN-333?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13559548#comment-13559548 ] Tom White commented on YARN-333: This is not a problem in MR is it, since the queue is

[jira] [Commented] (YARN-319) Submit a job to a queue that not allowed in fairScheduler, client will hold forever.

[ https://issues.apache.org/jira/browse/YARN-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13559583#comment-13559583 ] Tom White commented on YARN-319: When waiting for the final application status to be

[jira] [Commented] (YARN-319) Submit a job to a queue that not allowed in fairScheduler, client will hold forever.

[ https://issues.apache.org/jira/browse/YARN-319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13553942#comment-13553942 ] Tom White commented on YARN-319: Is it possible to write a unit test for this?

[jira] [Commented] (YARN-307) NodeManager should log container launch command.

[ https://issues.apache.org/jira/browse/YARN-307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13542993#comment-13542993 ] Tom White commented on YARN-307: Lohit, if you set yarn.nodemanager.delete.debug-delay-sec

[jira] [Commented] (YARN-286) Add a YARN ApplicationClassLoader

[ https://issues.apache.org/jira/browse/YARN-286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13542060#comment-13542060 ] Tom White commented on YARN-286: The patch is needed by MAPREDUCE-1700 for class isolation

[jira] [Commented] (YARN-103) Add a yarn AM - RM client module

[ https://issues.apache.org/jira/browse/YARN-103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536929#comment-13536929 ] Tom White commented on YARN-103: The interface shouldn't really be implemented by anyone

[jira] [Commented] (YARN-103) Add a yarn AM - RM client module

[ https://issues.apache.org/jira/browse/YARN-103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13536990#comment-13536990 ] Tom White commented on YARN-103: Some feedback on the current patch: * Why not use the

[jira] [Commented] (YARN-272) Fair scheduler log messages try to print objects without overridden toString methods

[ https://issues.apache.org/jira/browse/YARN-272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13534981#comment-13534981 ] Tom White commented on YARN-272: It would be better to add the toString() methods since then

[jira] [Commented] (YARN-103) Add a yarn AM - RM client module

[ https://issues.apache.org/jira/browse/YARN-103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13535045#comment-13535045 ] Tom White commented on YARN-103: +1, this is really helpful if we can get the APIs right.

[jira] [Commented] (YARN-230) Make changes for RM restart phase 1

[ https://issues.apache.org/jira/browse/YARN-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13534044#comment-13534044 ] Tom White commented on YARN-230: Arun, yes it looks good to me, +1. We can address any

[jira] [Commented] (YARN-230) Make changes for RM restart phase 1

[ https://issues.apache.org/jira/browse/YARN-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13532371#comment-13532371 ] Tom White commented on YARN-230: OK, in which case please make the default state store the

[jira] [Commented] (YARN-255) Support secure AM launch for unmanaged AM's

[ https://issues.apache.org/jira/browse/YARN-255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13532414#comment-13532414 ] Tom White commented on YARN-255: Per discussion on YARN-252, the ACL methods might be better

[jira] [Created] (YARN-273) Add an unmanaged AM client for in-process AMs

Tom White created YARN-273: -- Summary: Add an unmanaged AM client for in-process AMs Key: YARN-273 URL: https://issues.apache.org/jira/browse/YARN-273 Project: Hadoop YARN Issue Type: New Feature

[jira] [Updated] (YARN-252) Unmanaged AMs should not have to set the AM's ContainerLaunchContext

[ https://issues.apache.org/jira/browse/YARN-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated YARN-252: --- Attachment: YARN-252.patch Another patch with some comments about ACLs. Would you please register your ideas

[jira] [Updated] (YARN-252) Unmanaged AMs should not have to set the AM's ContainerLaunchContext

[ https://issues.apache.org/jira/browse/YARN-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated YARN-252: --- Summary: Unmanaged AMs should not have to set the AM's ContainerLaunchContext (was: Unmanaged AMs should not

[jira] [Commented] (YARN-230) Make changes for RM restart phase 1

[ https://issues.apache.org/jira/browse/YARN-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13531147#comment-13531147 ] Tom White commented on YARN-230: Thanks for addressing my feedback Bikas. The

[jira] [Commented] (YARN-253) Container launch may fail if no files were localized

[ https://issues.apache.org/jira/browse/YARN-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13530009#comment-13530009 ] Tom White commented on YARN-253: Thanks Vinod. Looking at LCE, it seems to handle this case

[jira] [Updated] (YARN-253) Container launch may fail if no files were localized

[ https://issues.apache.org/jira/browse/YARN-253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated YARN-253: --- Attachment: YARN-253.patch New patch which removes code from TestContainerManagerSecurity.

[jira] [Commented] (YARN-253) Container launch may fail if no files were localized

[ https://issues.apache.org/jira/browse/YARN-253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13508685#comment-13508685 ] Tom White commented on YARN-253: I see that you removed the change in MAPREDUCE-4427 because

[jira] [Updated] (YARN-252) Unmanaged AMs should not have to set the ContainerLaunchContext

[ https://issues.apache.org/jira/browse/YARN-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated YARN-252: --- Attachment: YARN-252.patch Bikas, thanks for taking a look. I was thinking that since there is no AM

[jira] [Commented] (YARN-170) NodeManager stop() gets called twice on shutdown

[ https://issues.apache.org/jira/browse/YARN-170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13507289#comment-13507289 ] Tom White commented on YARN-170: +1 Vinod, do you have any more comments before we commit

[jira] [Updated] (YARN-72) NM should handle cleaning up containers when it shuts down

[ https://issues.apache.org/jira/browse/YARN-72?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated YARN-72: -- Summary: NM should handle cleaning up containers when it shuts down (was: NM should handle cleaning up

[jira] [Commented] (YARN-72) NM should handle cleaning up containers when it shuts down ( and kill containers from an earlier instance when it comes back up after an unclean shutdown )

[ https://issues.apache.org/jira/browse/YARN-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13507290#comment-13507290 ] Tom White commented on YARN-72: --- +1 NM should handle cleaning up containers

[jira] [Updated] (YARN-251) Proxy URI generation fails for blank tracking URIs

[ https://issues.apache.org/jira/browse/YARN-251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated YARN-251: --- Attachment: YARN-251.patch With this fix applications that don't set a tracking URI won't get a confusing

[jira] [Created] (YARN-252) Unmanaged AMs should not have to set the ContainerLaunchContext

Tom White created YARN-252: -- Summary: Unmanaged AMs should not have to set the ContainerLaunchContext Key: YARN-252 URL: https://issues.apache.org/jira/browse/YARN-252 Project: Hadoop YARN Issue

[jira] [Updated] (YARN-252) Unmanaged AMs should not have to set the ContainerLaunchContext

[ https://issues.apache.org/jira/browse/YARN-252?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated YARN-252: --- Attachment: YARN-252.patch Simple fix. Unmanaged AMs should not have to set the

[jira] [Commented] (YARN-252) Unmanaged AMs should not have to set the ContainerLaunchContext

[ https://issues.apache.org/jira/browse/YARN-252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13507442#comment-13507442 ] Tom White commented on YARN-252: Testing is covered by the existing TestUnmanagedAMLauncher.

[jira] [Created] (YARN-253) Container launch may fail if no files were localized

Tom White created YARN-253: -- Summary: Container launch may fail if no files were localized Key: YARN-253 URL: https://issues.apache.org/jira/browse/YARN-253 Project: Hadoop YARN Issue Type: Bug

[jira] [Updated] (YARN-253) Container launch may fail if no files were localized

[ https://issues.apache.org/jira/browse/YARN-253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated YARN-253: --- Attachment: YARN-253-test.patch This change to the DS unit test exposes the problem by changing the mini YARN

[jira] [Updated] (YARN-253) Container launch may fail if no files were localized

[ https://issues.apache.org/jira/browse/YARN-253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated YARN-253: --- Attachment: YARN-253.patch This fixes the problem by creating parent directories if they don't already exist.

[jira] [Commented] (YARN-187) Add hierarchical queues to the fair scheduler

[ https://issues.apache.org/jira/browse/YARN-187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506408#comment-13506408 ] Tom White commented on YARN-187: Sandy, thanks for adding the javadoc. It would be good to

[jira] [Commented] (YARN-248) Restore RMDelegationTokenSecretManager state on restart

[ https://issues.apache.org/jira/browse/YARN-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13506574#comment-13506574 ] Tom White commented on YARN-248: I thought about it more, and delegation tokens from the

[jira] [Commented] (YARN-72) NM should handle cleaning up containers when it shuts down ( and kill containers from an earlier instance when it comes back up after an unclean shutdown )

[ https://issues.apache.org/jira/browse/YARN-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505398#comment-13505398 ] Tom White commented on YARN-72: --- I don't think it needs to be configurable, since it's a best

[jira] [Commented] (YARN-187) Add hierarchical queues to the fair scheduler

[ https://issues.apache.org/jira/browse/YARN-187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505402#comment-13505402 ] Tom White commented on YARN-187: License headers are missing on the new files. (BTW I filed

[jira] [Commented] (YARN-231) Add persistent store implementation for RMStateStore

[ https://issues.apache.org/jira/browse/YARN-231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505417#comment-13505417 ] Tom White commented on YARN-231: I was assuming that the store was HA since it would be

[jira] [Commented] (YARN-230) Make changes for RM restart phase 1

[ https://issues.apache.org/jira/browse/YARN-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505427#comment-13505427 ] Tom White commented on YARN-230: bq. But it could make sense to remove application attempts

[jira] [Created] (YARN-248) Restore RMDelegationTokenSecretManager state on restart

Tom White created YARN-248: -- Summary: Restore RMDelegationTokenSecretManager state on restart Key: YARN-248 URL: https://issues.apache.org/jira/browse/YARN-248 Project: Hadoop YARN Issue Type:

[jira] [Commented] (YARN-248) Restore RMDelegationTokenSecretManager state on restart

[ https://issues.apache.org/jira/browse/YARN-248?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13505726#comment-13505726 ] Tom White commented on YARN-248: I can think of a couple of options: i) persist

[jira] [Commented] (YARN-230) Make changes for RM restart phase 1

[ https://issues.apache.org/jira/browse/YARN-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13504765#comment-13504765 ] Tom White commented on YARN-230: {quote} The store and remove methods have been made mirrors

[jira] [Commented] (YARN-229) Remove old code for restart

[ https://issues.apache.org/jira/browse/YARN-229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13503178#comment-13503178 ] Tom White commented on YARN-229: Can you make

[jira] [Commented] (YARN-230) Make changes for RM restart phase 1

[ https://issues.apache.org/jira/browse/YARN-230?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13503228#comment-13503228 ] Tom White commented on YARN-230: Overall, this looks great. General feedback: * Can we make

[jira] [Commented] (YARN-72) NM should handle cleaning up containers when it shuts down ( and kill containers from an earlier instance when it comes back up after an unclean shutdown )

[ https://issues.apache.org/jira/browse/YARN-72?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13500195#comment-13500195 ] Tom White commented on YARN-72: --- Sandy, this looks like a good start, hooking in the code for

[jira] [Commented] (YARN-128) Resurrect RM Restart

[ https://issues.apache.org/jira/browse/YARN-128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13500434#comment-13500434 ] Tom White commented on YARN-128: I had a quick look at the new patches and

[jira] [Commented] (YARN-128) Resurrect RM Restart

[ https://issues.apache.org/jira/browse/YARN-128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13498825#comment-13498825 ] Tom White commented on YARN-128: Bikas, this looks good so far. Thanks for working on it. A

[jira] [Commented] (YARN-128) Resurrect RM Restart

[ https://issues.apache.org/jira/browse/YARN-128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13498883#comment-13498883 ] Tom White commented on YARN-128: You are right about there being no race - I missed the

[jira] [Commented] (YARN-169) Update log4j.appender.EventCounter to use org.apache.hadoop.log.metrics.EventCounter

[ https://issues.apache.org/jira/browse/YARN-169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13492378#comment-13492378 ] Tom White commented on YARN-169: +1 pending Jenkins. Update

[jira] [Commented] (YARN-129) Simplify classpath construction for mini YARN tests

[ https://issues.apache.org/jira/browse/YARN-129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13492416#comment-13492416 ] Tom White commented on YARN-129: The test failure is unrelated - see MAPREDUCE-4774. bq.

[jira] [Commented] (YARN-145) Add a Web UI to the fair share scheduler

[ https://issues.apache.org/jira/browse/YARN-145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13474035#comment-13474035 ] Tom White commented on YARN-145: +1 I ran a single node cluster and monitored the fair

[jira] [Commented] (YARN-146) Add unit tests for computing fair share in the fair scheduler

[ https://issues.apache.org/jira/browse/YARN-146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13473329#comment-13473329 ] Tom White commented on YARN-146: Looks good to me. Nit: please change the test to use JUnit

[jira] [Assigned] (YARN-146) Add unit tests for computing fair share in the fair scheduler

[ https://issues.apache.org/jira/browse/YARN-146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White reassigned YARN-146: -- Assignee: Sandy Ryza Add unit tests for computing fair share in the fair scheduler

[jira] [Created] (YARN-129) Simplify classpath construction for mini YARN tests

Tom White created YARN-129: -- Summary: Simplify classpath construction for mini YARN tests Key: YARN-129 URL: https://issues.apache.org/jira/browse/YARN-129 Project: Hadoop YARN Issue Type:

[jira] [Updated] (YARN-129) Simplify classpath construction for mini YARN tests

[ https://issues.apache.org/jira/browse/YARN-129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom White updated YARN-129: --- Attachment: YARN-129.patch Here's a patch that implements this idea. Simplify classpath