[JIRA] [ec2-plugin] (JENKINS-32779) Spot instance hit cap when sharing a AMI
Title: Message Title James Judd edited a comment on JENKINS-32779 Re: Spot instance hit cap when sharing a AMI Also seeing this since 1.312 ec2 configs in the same cloud sharing the same AMI. First config with limit 4. Second config with limit 2.2 instances of config 1 are launched -> cannot launch any of config 2Change instance limit of config 2 to 3+ -> config 2 launchesMy guess is that the limits are read per config but the total when launching an instance from a config is calculated using from all instances using that AMI. Note: If it makes any difference, the instances above were all spot instances. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-32779) Spot instance hit cap when sharing a AMI
Title: Message Title James Judd edited a comment on JENKINS-32779 Re: Spot instance hit cap when sharing a AMI Also seeing this since 1.31 2 ec2 configs in the same cloud sharing the same AMI. First config with limit 4. Second config with limit 2.2 instances of config 1 are launched -> cannot launch any of config 2Change instance limit of config 2 to 3+ -> config 2 launchesMy guess is that the limits are read per config but the total when launching an instance from a config is calculated using all instances using that AMI. Note: If it makes any difference, the instances above were all spot instances. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-32779) Spot instance hit cap when sharing a AMI
Title: Message Title James Judd commented on JENKINS-32779 Re: Spot instance hit cap when sharing a AMI Also seeing this since 1.31 Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-29851) global instance cap not respected for spot instances
Title: Message Title James Judd commented on JENKINS-29851 Re: global instance cap not respected for spot instances We have run into this as well. Probably won't have time to look into this before January. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title James Judd commented on JENKINS-26854 Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Francis Upton Just curious when 1.28 will be released. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title James Judd commented on JENKINS-26854 Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" It's been almost 24 hours and I have not had any expirations. Created PR #148 Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title
James Judd edited a comment on JENKINS-26854
Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Thanks for the info Martin.I spent some more time looking into this tonight and I think I found the cause. Even better, I think the fix is quite simple. At the moment, in EC2Cloud: we create an AmazonEC2Client like so{code:Java}AmazonEC2 client = new AmazonEC2Client(credentialsProvider.getCredentials(), config);{code}According to the [Amazon SDK source|https://github.com/aws/aws-sdk-java/blob/84adaca80ee2b974f76816f5d4d9be90a5aa8543/aws-java-sdk-ec2/src/main/java/com/amazonaws/services/ec2/AmazonEC2Client.java#L134-138] this creates a [StaticCredentialsProvider|https://github.com/puppetlabs/aws-sdk-for-java/blob/master/src/main/java/com/amazonaws/internal/StaticCredentialsProvider.java] using the given credentials. From what I can tell, StaticCredentialsProvider never refreshes its credentials, leading to expiration.Instead, you can create an [AmazonEC2Client with a credentials provider directly|https://github.com/aws/aws-sdk-java/blob/84adaca80ee2b974f76816f5d4d9be90a5aa8543/aws-java-sdk-ec2/src/main/java/com/amazonaws/services/ec2/AmazonEC2Client.java#L172-174]. This should, as far as I can tell, refresh the credentials as needed.{code:Java}AmazonEC2 client = new AmazonEC2Client(credentialsProvider, config);{code}This is further supported by [this amazon documentation|http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-roles.html], which states{quote}*Important*The automatic credentials refresh happens only when you use the default client constructor, which creates its own InstanceProfileCredentialsProvider as part of the default provider chain, _ or _or when you pass an InstanceProfileCredentialsProvider instance directly to the client constructor_. If you use another method to obtain or pass instance profile credentials, you are responsible for checking for and refreshing expired credentials.{quote} [emphasis mine]I just uploaded a version of the plugin with this change to our Jenkins server. I'll let it run and report back tomorrow if I see any errors. If it works, I'll create a pull request.
Add Comment
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title James Judd commented on JENKINS-26854 Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Thanks for the info Martin. I spent some more time looking into this tonight and I think I found the cause. Even better, I think the fix is quite simple. At the moment, in EC2Cloud: we create an AmazonEC2Client like so AmazonEC2 client = new AmazonEC2Client(credentialsProvider.getCredentials(), config); According to the Amazon SDK source this creates a StaticCredentialsProvider using the given credentials. From what I can tell, StaticCredentialsProvider never refreshes its credentials, leading to expiration. Instead, you can create an AmazonEC2Client with a credentials provider directly. This should, as far as I can tell, refresh the credentials as needed. AmazonEC2 client = new AmazonEC2Client(credentialsProvider, config); This is further supported by this amazon documentation, which states Important The automatic credentials refresh happens only when you use the default client constructor, which creates its own InstanceProfileCredentialsProvider as part of the default provider chain,_ or when you pass an InstanceProfileCredentialsProvider instance directly to the client constructor_. If you use another method to obtain or pass instance profile credentials, you are responsible for checking for and refreshing expired credentials. [emphasis mine] I just uploaded a version of the plugin with this change to our Jenkins server. I'll let it run and report back tomorrow if I see any errors. If it works, I'll create a pull request. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title
James Judd edited a comment on JENKINS-26854
Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
I resolved the conflicts on ([#131|https://github.com/jenkinsci/ec2-plugin/pull/131]) and opened a new pull request ([#147|https://github.com/jenkinsci/ec2-plugin/pull/147] ) . Ran into issues using this fix. I think removing the cached connection in EC2Cloud might cause requests to be made too frequently to the metadata service. Our log file is full of these errors and jenkins stops working. The line {noformat} INFO: Excess workload after pending Spot instances: 23 {noformat} is strange, because we are not using spot instances.Going to try applying only the first commit from ([#131|https://github.com/jenkinsci/ec2-plugin/pull/131]) and seeing if that works, sans the fix that martin made.{noformat} com.amazonaws.auth.InstanceProfileCredentialsProvider handleErrorSEVERE: Unable to load credentials from Amazon EC2 metadata servicecom.amazonaws.AmazonClientException: Unable to load credentials from Amazon EC2 metadata serviceat com.amazonaws.auth.InstanceProfileCredentialsProvider.handleError(InstanceProfileCredentialsProvider.java:244)at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:225)at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:124)at com.amazonaws.auth.InstanceProfileCredentialsProvider$1.run(InstanceProfileCredentialsProvider.java:104)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)at java.lang.Thread.run(Thread.java:745)Caused by: java.io.IOException: Server returned HTTP response code: 429 for URL: http://169.254.169.254/latest/meta-data/iam/security-credentials/at sun.reflect.GeneratedConstructorAccessor460.newInstance(Unknown Source)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:526)at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1676)at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1674)at java.security.AccessController.doPrivileged(Native Method)at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1672)at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1245)at com.amazonaws.internal.EC2MetadataClient.readResponse(EC2MetadataClient.java:113)at com.amazonaws.internal.EC2MetadataClient.readResource(EC2MetadataClient.java:92)at com.amazonaws.internal.EC2MetadataClient.getDefaultCredentials(EC2MetadataClient.java:55)at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:186)... 9 moreCaused by: java.io.IOException: Server returned HTTP response code: 429 for URL: http://169.254.169.254/latest/meta-data/iam/security-credentials/at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1627)at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)at com.amazonaws.internal.EC2MetadataClient.readResponse(EC2MetadataClient.java:110)
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title James Judd commented on JENKINS-26854 Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" I resolved the conflicts on (#131) and opened a new pull request (#147. Ran into issues using this fix. I think removing the cached connection in EC2Cloud might cause requests to be made too frequently to the metadata service. Our log file is full of these errors and jenkins stops working. The line INFO: Excess workload after pending Spot instances: 23 is strange, because we are not using spot instances. Going to try applying only the first commit from (#131) and seeing if that works, sans the fix that martin made. com.amazonaws.auth.InstanceProfileCredentialsProvider handleError SEVERE: Unable to load credentials from Amazon EC2 metadata service com.amazonaws.AmazonClientException: Unable to load credentials from Amazon EC2 metadata service at com.amazonaws.auth.InstanceProfileCredentialsProvider.handleError(InstanceProfileCredentialsProvider.java:244) at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:225) at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:124) at com.amazonaws.auth.InstanceProfileCredentialsProvider$1.run(InstanceProfileCredentialsProvider.java:104) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Server returned HTTP response code: 429 for URL: http://169.254.169.254/latest/meta-data/iam/security-credentials/ at sun.reflect.GeneratedConstructorAccessor460.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1676) at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1674) at java.security.AccessController.doPrivileged(Native Method) at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1672) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1245) at com.amazonaws.internal.EC2MetadataClient.readResponse(EC2MetadataClient.java:113) at com.amazonaws.internal.EC2MetadataClient.readResource(EC2MetadataClient.java:92) at com.amazonaws.internal.EC2MetadataClient.getDefaultCredentials(EC2MetadataClient.java:55) at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:186) ... 9 more Caused by: java.io.IOException: Server returned HTTP respons
