[GitHub] twill issue #23: (TWILL-181) allow setting the maximum number of retries per...
Github user hsaputra commented on the issue: https://github.com/apache/twill/pull/23 Looks like not related to this PR. Will merge EOD if no more comment. Thanks again for the hard work, @serranom --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] twill issue #23: (TWILL-181) allow setting the maximum number of retries per...
Github user serranom commented on the issue: https://github.com/apache/twill/pull/23 I looked at the latest CI run and the failure is related to some `twill-zookeeper` project issue -- not yarn --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] twill issue #23: (TWILL-181) allow setting the maximum number of retries per...
Github user hsaputra commented on the issue: https://github.com/apache/twill/pull/23 I am having problem applying the patch, could you kindly rebase the patch into single commit? Thanks --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] twill issue #23: (TWILL-181) allow setting the maximum number of retries per...
Github user serranom commented on the issue: https://github.com/apache/twill/pull/23 My pleasure. It was a good way to get to know twill code base. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] twill issue #23: (TWILL-181) allow setting the maximum number of retries per...
Github user hsaputra commented on the issue: https://github.com/apache/twill/pull/23 Thanks much for working on the patch and for the update, @serranom. Did a pass for review. LGTM in general, add mini comment. Other than that +1 will merge after comment addressed. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] twill issue #23: (TWILL-181) allow setting the maximum number of retries per...
Github user serranom commented on the issue: https://github.com/apache/twill/pull/23 I found that there is still a race condition in ApplicationMasterService.launchRunnable. If the number of instances is increased right after the original request is fullfilled, the current logic can result in the original request not being polled resulting in future requests hanging. This seems to be a fairly unlikely case in the real world, but I'll file a JIRA for it. For now, I've reworked the test to wait until launchRunnable has done the polling for the original request. This PR is now complete. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] twill issue #23: (TWILL-181) allow setting the maximum number of retries per...
Github user serranom commented on the issue: https://github.com/apache/twill/pull/23 ugh. I'll take a look, I may have messed up the merge. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] twill issue #23: (TWILL-181) allow setting the maximum number of retries per...
Github user serranom commented on the issue: https://github.com/apache/twill/pull/23 I think this is set now. I've updated the PR with instance-based maxRetries changes. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] twill issue #23: (TWILL-181) allow setting the maximum number of retries per...
Github user serranom commented on the issue: https://github.com/apache/twill/pull/23 I have new changes I will add to this PR once https://github.com/apache/twill/pull/29 is resolved since the tests will fail intermittently without that fix. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] twill issue #23: (TWILL-181) allow setting the maximum number of retries per...
Github user serranom commented on the issue:
https://github.com/apache/twill/pull/23
I have discovered the cause of the intermittent failure.
This code (starting on line 703 of ApplicationMasterService, comments added
by me):
/*
* The provisionRequest will either contain a single container
(ALLOCATE_ONE_INSTANCE_AT_A_TIME), or all the
* containers to satisfy the expectedContainers count. In the later
case, the provision request is complete once
* all the containers have run at which point we poll() to remove the
provisioning request.
*/
if (expectedContainers.getExpected(runnableName) ==
runningContainers.count(runnableName) ||
provisioning.peek().getType().equals(AllocationSpecification.Type.ALLOCATE_ONE_INSTANCE_AT_A_TIME))
{
provisioning.poll();
}
There is a case when instances are failing (but not simultaneously) where
the retries for the instances will be spread over two invocations of
`ApplicationMasterService.handleCompleted`. This means they will be part of
separate `RunnableContainerRequests` and thus will be provisioned separately.
But because the code above does not anticipate this case, the first
provisionRequest will never appear to be satisfied, never be polled and the
total can never be met. This is an existing bug, since I think it could happen
before my new code -- its just that my test tickles it. If there is agreement,
I can file a new JIRA and PR request or just fix it with this PR. Let me know.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] twill issue #23: (TWILL-181) allow setting the maximum number of retries per...
Github user serranom commented on the issue: https://github.com/apache/twill/pull/23 @hsaputra , that is the failure I see as intermittent. My comment above has a relevant log excerpt. I'm planning on trying to dig into it today or tomorrow. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] twill issue #23: (TWILL-181) allow setting the maximum number of retries per...
Github user hsaputra commented on the issue: https://github.com/apache/twill/pull/23 Seems like one of the tests failed related to new one you added: ` MaxRetriesTestRun.maxRetriesTwoInstances` Could you quickly take a peek? https://s3.amazonaws.com/archive.travis-ci.org/jobs/192819935/log.txt --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] twill issue #23: (TWILL-181) allow setting the maximum number of retries per...
Github user serranom commented on the issue: https://github.com/apache/twill/pull/23 @hsaputra, since I have a bug to fix, the updated request will trigger the rebuild. thanks. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] twill issue #23: (TWILL-181) allow setting the maximum number of retries per...
Github user serranom commented on the issue: https://github.com/apache/twill/pull/23 Thanks Henry! No rush on my account, I was just checking in. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
