[
https://issues.apache.org/jira/browse/TWILL-213?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15856669#comment-15856669
]
Martin Serrano commented on TWILL-213:
--------------------------------------
I've updated the description with what I think is happening.
> Increase of instances while starting up may lead to ignored retries and
> instance increases
> ------------------------------------------------------------------------------------------
>
> Key: TWILL-213
> URL: https://issues.apache.org/jira/browse/TWILL-213
> Project: Apache Twill
> Issue Type: Bug
> Components: yarn
> Affects Versions: 0.9.0
> Reporter: Martin Serrano
>
> As seen in the test development for TWILL-181, if the number of instances for
> a container is increased before the {{ApplicationMasterService}} has observed
> the original request as being satisfied, the instance increase and any
> subsequent retries will be blocked. This is because in {{launchRunnable}}:
> {code}
> TwillContainerLauncher launcher = new TwillContainerLauncher(
> twillSpec.getRunnables().get(runnableName),
> processLauncher.getContainerInfo(), launchContext,
> ZKClients.namespace(zkClient, getZKNamespace(runnableName)),
> containerCount, jvmOpts, reservedMemory, getSecureStoreLocation());
> runningContainers.start(runnableName,
> processLauncher.getContainerInfo(), launcher);
> // Need to call complete to workaround bug in YARN AMRMClient
> if (provisionRequest.containerAcquired()) {
> amClient.completeContainerRequest(provisionRequest.getRequestId());
> }
> /*
> * The provisionRequest will either contain a single container
> (ALLOCATE_ONE_INSTANCE_AT_A_TIME), or all the
> * containers to satisfy the expectedContainers count. In the later
> case, the provision request is complete once
> * all the containers have run at which point we poll() to remove the
> provisioning request.
> */
> if (expectedContainers.getExpected(runnableName) ==
> runningContainers.count(runnableName) ||
>
> provisioning.peek().getType().equals(AllocationSpecification.Type.ALLOCATE_ONE_INSTANCE_AT_A_TIME))
> {
> provisioning.poll();
> }
> {code}
> There is a race condition. The sequence:
> * *Thread A*: {{runningContainers.start}} is called and 2 instances are
> started
> * *Thread B*: The runnable from {{createSetInstanceRunnable}} executes, sees
> the 2 instances are started and updates the expected count to 3.
> * *Thread A*: Gets to if check comparing expectedContainers (3) to
> runningContainers.count (2). Since this fails, {{poll}} is not called and
> this provision request is not satisfied.
> Subsequent calls will try to provision the 3rd container because it seems
> like the first provision request is not yet satisfied.
> The {{MaxRetriesTestRun.maxRetriesWithIncreasedInstances}} method can be used
> to reproduce this case intermittently by changing the {{allRunning.await}}
> check to something that does a countdown latch {{onRunning}} as
> {{EchoServerTestRun}} does.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
