[JIRA] (JENKINS-35524) new ec2 slave is terminated or terminating during launch
Title: Message Title
Daniel Starling edited a comment on JENKINS-35524
Re: new ec2 slave is terminated or terminating during launch
We ran into this issue today. I have not yet determined the cause (was not likely EC2 instance limits or EBS volume limits), but it was a bit alarming that in a few minutes it started 25 instances that were all terminated when I checked on them. If I understand EC2 billing correctly, each of those instances counts as 1 hour of EC2 time. After an hour or so, the issue has magically evaporated. We saw the same error ("is terminated or terminating, aborting launch") as in logs here.It seems like it would be good to have a low-level configurable throttle -- rapidly cycling instances up/down at too great a rate smells like a condition you might want to halt everything for. Or at a minimum, set a limit on how many instances can be created per hour. This might be a good preventative measure to deal with instance-creation problems.EDIT: This is helpful: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_InstanceStraightToTerminated.html -- turns out we did hit "State transition reasonClient.VolumeLimitExceeded: Volume limit exceeded"
Add Comment
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-35524) new ec2 slave is terminated or terminating during launch
Title: Message Title
Daniel Starling edited a comment on JENKINS-35524
Re: new ec2 slave is terminated or terminating during launch
We ran into this issue today. I have not yet determined the cause (was not likely EC2 instance limits or EBS volume limits), but it was a bit alarming that in a few minutes it started 25 instances that were all terminated when I checked on them. If I understand EC2 billing correctly, each of those instances counts as 1 hour of EC2 time. After an hour or so, the issue has magically evaporated. We saw the same error ("is terminated or terminating, aborting launch") as in logs here.It seems like it would be good to have a low-level configurable throttle -- rapidly cycling instances up/down at too great a rate smells like a condition you might want to halt everything for. Or at a minimum, set a limit on how many instances can be created per hour. This might be a good preventative measure to deal with instance-creation problems. EDIT: This is helpful: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/Using_InstanceStraightToTerminated.html
Add Comment
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-35524) new ec2 slave is terminated or terminating during launch
Title: Message Title
Daniel Starling edited a comment on JENKINS-35524
Re: new ec2 slave is terminated or terminating during launch
We ran into this issue today. I have not yet determined the cause (was not likely EC2 instance limits or EBS volume limits), but it was a bit alarming that in a few minutes it started 25 instances that were all terminated when I checked on them. If I understand EC2 billing correctly, each of those instances counts as 1 hour of EC2 time. After an hour or so, the issue has magically evaporated. We saw the same error ("is terminated or terminating, aborting launch") as in logs here.It seems like it would be good to have a low-level configurable throttle -- rapidly cycling instances up/down at too great a rate smells like a condition you might want to halt everything for. Or at a minimum, set a limit on how many instances can be created per hour. This might be a good preventative measure to deal with instance-creation bugs problems .
Add Comment
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] (JENKINS-35524) new ec2 slave is terminated or terminating during launch
Title: Message Title
Daniel Starling commented on JENKINS-35524
Re: new ec2 slave is terminated or terminating during launch
We ran into this issue today. I have not yet determined the cause (was not likely EC2 instance limits or EBS volume limits), but it was a bit alarming that in a few minutes it started 25 instances that were all terminated when I checked on them. If I understand EC2 billing correctly, each of those instances counts as 1 hour of EC2 time. After an hour or so, the issue has magically evaporated. We saw the same error ("is terminated or terminating, aborting launch") as in logs here. It seems like it would be good to have a low-level configurable throttle – rapidly cycling instances up/down at too great a rate smells like a condition you might want to halt everything for. Or at a minimum, set a limit on how many instances can be created per hour. This might be a good preventative measure to deal with instance-creation bugs.
Add Comment
This message was sent by Atlassian JIRA (v7.1.7#71011-sha1:2526d7c)
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
