[JIRA] [ec2] (JENKINS-17257) EC2 plugin fails to launch slave (IndexOutOfBoundsException)

Tue, 02 Apr 2013 11:14:38 -0700

t r commented on Bug JENKINS-17257

We see this error as well. It happens quite often when trying to launch a bunch of new slaves at the same time. When you work with the ec2 api you have to assume that every call can fail, and even calls that succeed might not return the response that you expect. In this case the instance was launched but the subsequent status-polling returns an empty list where you'd expected our new instance to be.

The fix is to make the retry loop in EC2Computer.java check for the case where the DescribeInstancesRequest succeeds but returns an empty instance list. This same defensive approach should be done for all remote calls that might return lists of things. Don't assume that list.get(0) is safe.

I'll post the local hack that we're using to get by while we wait for the official fixes in the github master.

private Instance _describeInstance() throws AmazonClientException, InterruptedException {
        // Sometimes even after a successful RunInstances, DescribeInstances
        // returns an error for a few seconds. We do a few retries instead of
        // failing instantly. See [JENKINS-15319].
    	
    	Instance myInstance = null;
    	
        for (int i = 0; i < 10; i++) {
        	
            myInstance = _describeInstanceOnce();
            if (myInstance == null) {
                // retry in 5 seconds.
                Thread.sleep(5000);
                continue;
            }
        }

    	if (myInstance == null)
    		throw new AmazonClientException("No reservation for " + getNode().getInstanceId());
        
        return myInstance;
    }

    private Instance _describeInstanceOnce() {
    	Instance myInstance = null;
    	try {
	        DescribeInstancesRequest request = new DescribeInstancesRequest();
	        request.setInstanceIds(Collections.<String>singletonList(getNode().getInstanceId()));
	        List<Reservation> reservations = EC2Cloud.get().connect().describeInstances(request).getReservations();
	        if (reservations != null && reservations.size() > 0) {
	        	Reservation reservation = reservations.get(0);
	        	List<Instance> instances = reservation.getInstances();
	        	if (instances != null && instances.size() > 0) {
	        		myInstance = instances.get(0);
	        	}
	        }
    	} catch (Exception e) {
    		e.printStackTrace();
    	}
	        	        
        return myInstance;
    }
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira

--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
 
 

Reply via email to