[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title Francis Upton closed an issue as Fixed Fixed in 1.28. Jenkins / JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Change By: Francis Upton Status: Resolved Closed Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title Francis Upton resolved as Fixed Jenkins / JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Change By: Francis Upton Status: Open Resolved Resolution: Fixed Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title Vincent Rivellino commented on JENKINS-26854 Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" I updated to 1.28 yesterday and have disabled my auto-restart job. After 19 hours of uptime, no expired credentials. I think things are copacetic. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title Francis Upton commented on JENKINS-26854 Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" I think the release actually worked this time, so by tomorrow it should be present on the wiki page and available. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title James Judd commented on JENKINS-26854 Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Francis Upton Just curious when 1.28 will be released. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title Francis Upton commented on JENKINS-26854 Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" @Ximon, I had intended to do the release and I think something went wrong. I will try to fix this today. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title Ximon Eighteen edited a comment on JENKINS-26854 Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" I see that there is a [1.28 release tag|https://github.com/jenkinsci/ec2-plugin/releases/tag/ec2-1.28] which includes fixes for this issue. However I don't see a 1.28 release at the Jenkins updates site or on the [ec2 plugin home page|https://wiki.jenkins-ci.org/display/JENKINS/Amazon+EC2+Plugin]. Also this issue is still marked as 'Open' - has the 1.28 build been tested by anyone following this issue? When will this 1.28 version be released? Additional: Can I test using this HPI? https://buildhive.cloudbees.com/job/jenkinsci/job/ec2-plugin/115/org.jenkins-ci.plugins$ec2/artifact/org.jenkins-ci.plugins/ec2/1.28-SNAPSHOT/ec2-1.28-SNAPSHOT.hpi Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title Ximon Eighteen commented on JENKINS-26854 Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" I see that there is a 1.28 release tag which includes fixes for this issue. However I don't see a 1.28 release at the Jenkins updates site or on the ec2 plugin home page. Also this issue is still marked as 'Open' - has the 1.28 build been tested by anyone following this issue? When will this 1.28 version be released? Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title SCM/JIRA link daemon commented on JENKINS-26854 Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Code changed in jenkins User: James Judd Path: src/main/java/hudson/plugins/ec2/EC2Cloud.java http://jenkins-ci.org/commit/ec2-plugin/2855d1d925dcfe92043ec6ce0b58111c116eb330 Log: Creates an AmazonEC2Client with an AWSCredentialsProvider instead of the AWSCredentials directly. This is done so the credentials will refresh instead of expire. Resolves JENKINS-26854. At the moment, in EC2Cloud we create an AmazonEC2Client by passing in the credentials directly. This creates a StaticCredentialsProvider using the given credentials. StaticCredentialsProvider never refreshes its credentials, leading to expiration. Instead, you can create an AmazonEC2Client with a credentials provider directly. This refreshes the credentials as needed. See http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-roles.html for more information.AWSCredentials directly. This is done so the credentials will refresh instead of expire. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title SCM/JIRA link daemon commented on JENKINS-26854 Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Code changed in jenkins User: Francis Upton Path: src/main/java/hudson/plugins/ec2/EC2Cloud.java http://jenkins-ci.org/commit/ec2-plugin/1fa2ee1126e007d874cb40d5dee25a031746a635 Log: Merge pull request #148 from jjudd/request-expired JENKINS-26854: Fixing 'RequestExpired' Compare: https://github.com/jenkinsci/ec2-plugin/compare/b576bb3163db...1fa2ee1126e0 Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title James Judd commented on JENKINS-26854 Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" It's been almost 24 hours and I have not had any expirations. Created PR #148 Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title
James Judd edited a comment on JENKINS-26854
Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Thanks for the info Martin.I spent some more time looking into this tonight and I think I found the cause. Even better, I think the fix is quite simple. At the moment, in EC2Cloud: we create an AmazonEC2Client like so{code:Java}AmazonEC2 client = new AmazonEC2Client(credentialsProvider.getCredentials(), config);{code}According to the [Amazon SDK source|https://github.com/aws/aws-sdk-java/blob/84adaca80ee2b974f76816f5d4d9be90a5aa8543/aws-java-sdk-ec2/src/main/java/com/amazonaws/services/ec2/AmazonEC2Client.java#L134-138] this creates a [StaticCredentialsProvider|https://github.com/puppetlabs/aws-sdk-for-java/blob/master/src/main/java/com/amazonaws/internal/StaticCredentialsProvider.java] using the given credentials. From what I can tell, StaticCredentialsProvider never refreshes its credentials, leading to expiration.Instead, you can create an [AmazonEC2Client with a credentials provider directly|https://github.com/aws/aws-sdk-java/blob/84adaca80ee2b974f76816f5d4d9be90a5aa8543/aws-java-sdk-ec2/src/main/java/com/amazonaws/services/ec2/AmazonEC2Client.java#L172-174]. This should, as far as I can tell, refresh the credentials as needed.{code:Java}AmazonEC2 client = new AmazonEC2Client(credentialsProvider, config);{code}This is further supported by [this amazon documentation|http://docs.aws.amazon.com/AWSSdkDocsJava/latest/DeveloperGuide/java-dg-roles.html], which states{quote}*Important*The automatic credentials refresh happens only when you use the default client constructor, which creates its own InstanceProfileCredentialsProvider as part of the default provider chain, _ or _or when you pass an InstanceProfileCredentialsProvider instance directly to the client constructor_. If you use another method to obtain or pass instance profile credentials, you are responsible for checking for and refreshing expired credentials.{quote} [emphasis mine]I just uploaded a version of the plugin with this change to our Jenkins server. I'll let it run and report back tomorrow if I see any errors. If it works, I'll create a pull request.
Add Comment
This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title James Judd commented on JENKINS-26854 Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Thanks for the info Martin. I spent some more time looking into this tonight and I think I found the cause. Even better, I think the fix is quite simple. At the moment, in EC2Cloud: we create an AmazonEC2Client like so AmazonEC2 client = new AmazonEC2Client(credentialsProvider.getCredentials(), config); According to the Amazon SDK source this creates a StaticCredentialsProvider using the given credentials. From what I can tell, StaticCredentialsProvider never refreshes its credentials, leading to expiration. Instead, you can create an AmazonEC2Client with a credentials provider directly. This should, as far as I can tell, refresh the credentials as needed. AmazonEC2 client = new AmazonEC2Client(credentialsProvider, config); This is further supported by this amazon documentation, which states Important The automatic credentials refresh happens only when you use the default client constructor, which creates its own InstanceProfileCredentialsProvider as part of the default provider chain,_ or when you pass an InstanceProfileCredentialsProvider instance directly to the client constructor_. If you use another method to obtain or pass instance profile credentials, you are responsible for checking for and refreshing expired credentials. [emphasis mine] I just uploaded a version of the plugin with this change to our Jenkins server. I'll let it run and report back tomorrow if I see any errors. If it works, I'll create a pull request. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265)
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title martinfr62 commented on JENKINS-26854 Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" James - just an fyi the message about pending spot instances is actually n othing to do with spot instances. Weird I know - it just means that you have a pending queue size of 23, after all spot instances have been launched - as you dont have any spot instances - you just have a queue size of 23. That's all the message means. Nothing to do really with spot instances. Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title
James Judd edited a comment on JENKINS-26854
Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
I resolved the conflicts on ([#131|https://github.com/jenkinsci/ec2-plugin/pull/131]) and opened a new pull request ([#147|https://github.com/jenkinsci/ec2-plugin/pull/147] ) . Ran into issues using this fix. I think removing the cached connection in EC2Cloud might cause requests to be made too frequently to the metadata service. Our log file is full of these errors and jenkins stops working. The line {noformat} INFO: Excess workload after pending Spot instances: 23 {noformat} is strange, because we are not using spot instances.Going to try applying only the first commit from ([#131|https://github.com/jenkinsci/ec2-plugin/pull/131]) and seeing if that works, sans the fix that martin made.{noformat} com.amazonaws.auth.InstanceProfileCredentialsProvider handleErrorSEVERE: Unable to load credentials from Amazon EC2 metadata servicecom.amazonaws.AmazonClientException: Unable to load credentials from Amazon EC2 metadata serviceat com.amazonaws.auth.InstanceProfileCredentialsProvider.handleError(InstanceProfileCredentialsProvider.java:244)at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:225)at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:124)at com.amazonaws.auth.InstanceProfileCredentialsProvider$1.run(InstanceProfileCredentialsProvider.java:104)at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)at java.lang.Thread.run(Thread.java:745)Caused by: java.io.IOException: Server returned HTTP response code: 429 for URL: http://169.254.169.254/latest/meta-data/iam/security-credentials/at sun.reflect.GeneratedConstructorAccessor460.newInstance(Unknown Source)at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)at java.lang.reflect.Constructor.newInstance(Constructor.java:526)at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1676)at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1674)at java.security.AccessController.doPrivileged(Native Method)at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1672)at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1245)at com.amazonaws.internal.EC2MetadataClient.readResponse(EC2MetadataClient.java:113)at com.amazonaws.internal.EC2MetadataClient.readResource(EC2MetadataClient.java:92)at com.amazonaws.internal.EC2MetadataClient.getDefaultCredentials(EC2MetadataClient.java:55)at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:186)... 9 moreCaused by: java.io.IOException: Server returned HTTP response code: 429 for URL: http://169.254.169.254/latest/meta-data/iam/security-credentials/at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1627)at java.net.HttpURLConnection.getResponseCode(HttpURLConnection.java:468)at com.amazonaws.internal.EC2MetadataClient.readResponse(EC2MetadataClient.java:110)
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Title: Message Title James Judd commented on JENKINS-26854 Re: EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" I resolved the conflicts on (#131) and opened a new pull request (#147. Ran into issues using this fix. I think removing the cached connection in EC2Cloud might cause requests to be made too frequently to the metadata service. Our log file is full of these errors and jenkins stops working. The line INFO: Excess workload after pending Spot instances: 23 is strange, because we are not using spot instances. Going to try applying only the first commit from (#131) and seeing if that works, sans the fix that martin made. com.amazonaws.auth.InstanceProfileCredentialsProvider handleError SEVERE: Unable to load credentials from Amazon EC2 metadata service com.amazonaws.AmazonClientException: Unable to load credentials from Amazon EC2 metadata service at com.amazonaws.auth.InstanceProfileCredentialsProvider.handleError(InstanceProfileCredentialsProvider.java:244) at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:225) at com.amazonaws.auth.InstanceProfileCredentialsProvider.getCredentials(InstanceProfileCredentialsProvider.java:124) at com.amazonaws.auth.InstanceProfileCredentialsProvider$1.run(InstanceProfileCredentialsProvider.java:104) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) Caused by: java.io.IOException: Server returned HTTP response code: 429 for URL: http://169.254.169.254/latest/meta-data/iam/security-credentials/ at sun.reflect.GeneratedConstructorAccessor460.newInstance(Unknown Source) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) at java.lang.reflect.Constructor.newInstance(Constructor.java:526) at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1676) at sun.net.www.protocol.http.HttpURLConnection$6.run(HttpURLConnection.java:1674) at java.security.AccessController.doPrivileged(Native Method) at sun.net.www.protocol.http.HttpURLConnection.getChainedException(HttpURLConnection.java:1672) at sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1245) at com.amazonaws.internal.EC2MetadataClient.readResponse(EC2MetadataClient.java:113) at com.amazonaws.internal.EC2MetadataClient.readResource(EC2MetadataClient.java:92) at com.amazonaws.internal.EC2MetadataClient.getDefaultCredentials(EC2MetadataClient.java:55) at com.amazonaws.auth.InstanceProfileCredentialsProvider.loadCredentials(InstanceProfileCredentialsProvider.java:186) ... 9 more Caused by: java.io.IOException: Server returned HTTP respons
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Vincent Rivellino commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Ximon - not sure if you noticed, but your pull request (#131) has merge conflicts now. I'd love to see this issue resolved! This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Unfortunately https://github.com/jenkinsci/ec2-plugin/pull/131 has now been closed without merging because the closer thought that ec2-1.27 fixed the issue. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Interesting. It seems that JENKINS-26414 has updated to the latest Java AWS SDK and added the signer override. Is this issue now obsolete? This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Interesting. It seems that JENKINS-26414 has updated to the latest Java AWS SDK and added the signer override. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" So having worked with Ximon offline, the signing step was the issue. I didnt notice it so didnt have it set. I should have made it the default as that was effectively the behaviour prior to this new version of the plugin. However, I think that automated self configuration as suggested by martinfr62 is a better idea. Sadly I haven't had any time for this issue for the last few weeks as the urgency on my side decreased the moment we managed to find a work around, i.e. a custom build of the plugin containing the fix documented in this issue. However, this isn't a sustainable long term solution so I hope to get back to this issue and increase the quality of my/our pull request. It bothers me however that there has been no obvious interest from the original plugin authors/maintainers in this issue... if I do improve the pull request I hope that it will be accepted and merged into a new release of the plugin. The plugin does seem to be actively maintained as there was a new 1.27 release a few days ago so there is hope I suppose. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" So having worked with Ximon offline, the signing step was the issue. I didnt notice it so didnt have it set. I should have made it the default as that was effectively the behaviour prior to this new version of the plugin. However, I think that automated self configuration as suggested by martinfr62 is a better idea. Sadly I haven't had any time for this issue for the last few weeks as the urgency on my side decreased the moment we managed to find a work around, i.e. a custom build of the plugin containing the fix documented in this issue. However, this isn't a sustainable long term solution, this PR is already one version behind the official version as 1.27 was just released and the PR is based on 1.26, so I hope to get back to this issue and increase the quality of my/our pull request. It bothers me however that there has been no obvious interest from the original plugin authors/maintainers in this issue... if I do improve the pull request I hope that it will be accepted and merged into a new release of the plugin. The plugin does seem to be actively maintained as there was a new 1.27 release a few days ago so there is hope I suppose. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
martinfr62 commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" So having worked with Ximon offline, the signing step was the issue. I didnt notice it so didnt have it set. Would prefer that test connection generate an error and indicate to try changing the signing to see if that works - or test both if flag isnt set and indicate that it requires setting. So user 'has control'. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" martinfr62 contacted me privately, we'll work out what the problem is. Separately to that, it occurred to me that end users won't want to know or care about request signing mechanism, or even having to specify whether or not instance profile credentials should be used. Instead both could be determined by the plugin by testing the connection, checking if it works with instance profile credentials, checking if it works with old or new signing algorithm, and just use what works. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
martinfr62 commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Here's the stack trace I get using us-east-1 with instance profile enabled - hit test connection and get this in the log. Process I followed - clone master repo, sync to head, pull down 131 patch and apply using tortoisegit (using windows).Build plugin and deploy to working jenkins. Hit test connection and it fails as below. Failed to check EC2 credential com.amazonaws.AmazonServiceException: AWS was not able to validate the provided access credentials (Service: AmazonEC2; Status Code: 401; Error Code: AuthFailure; Request ID: 43877fc7-b5e0-43ac-8dda-b9f802645343) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:8665) at hudson.plugins.ec2.EC2Cloud$DescriptorImpl.doTestConnection(EC2Cloud.java:598) at hudson.plugins.ec2.AmazonEC2Cloud$DescriptorImpl.doTestConnection(AmazonEC2Cloud.java:159) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:746) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:876) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:649) at org.kohsuke.stapler.Stapler.service(Stapler.java:238) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:123) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter$1.call(ScmSyncConfigurationFilter.java:46) at hudson.plugins.scm_sync_configuration.ScmSyncConfigurationDataProvider.provideRequestDuring(ScmSyncConfigurationDataProvider.java:103) at hudson.plugins.scm_sync_configuration.extensions.ScmSyncConfigurationFilter.doFilter(ScmSyncConfigurationFilter.java:42) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:120) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:114) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:135) at hud
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Hi martinfr62, I have verified that I have not left any files uncommitted, and that the pull request latest commit ID and commit message match what I have locally. I am now using that pull request on several Jenkins instances in AWS and they are all now working. The pull request introduced a "Use old signing mechanism for authenticating request" tick box in the Amazon EC2 Cloud configuration, above the secret key and access key, did you tick that box? Without ticking that box I get authentication errors. It was implied by the original AWS staff member who responded to the forum request about the v4 vs query text signer algorithm issue that the default (v4) signer algorithm should work in most cases and thus I have not made the box ticked by default. However, thinking about it, the previous behaviour of the EC2 plugin was to use the old signer algorithm, so perhaps the old behaviour should be the default? There is currently no description or help for the tick box, I should add that. 'Test Connection' and 'Check AMI' both work for me when the box is ticked. My Jenkins instances are in the EU (Ireland) AWS region. If you could give me the error message and stack trace that would be very appreciated, and please verify that you have tested both with and without ticking the tick box. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Hi martinfr62, I have verified that I have not left any files uncommitted, and that the pull request latest commit ID and commit message match what I have locally. I am now using that pull request on several Jenkins instances in AWS and they are all now working. The pull request introduced a "Use old signing mechanism for authenticating request" tick box in the Amazon EC2 Cloud configuration, above the secret key and access key, did you tick that box? Without ticking that box I get authentication errors, but it was implied by the original AWS staff member who responded to the forum request about the v4 vs query text signer algorithm issue that the default (v4) signer algorithm should work in most cases and thus I have not made the box ticked by default. There is currently no description or help for the tick box, I should add that. 'Test Connection' and 'Check AMI' both work for me when the box is ticked. My Jenkins instances are in the EU (Ireland) AWS region. If you could give me the error message and stack trace that would be very appreciated, and please verify that you have tested both with and without ticking the tick box. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Hi martinfr62, I have verified that I have not left any files uncommitted, and that the pull request latest commit ID and commit message match what I have locally. I am now using that pull request on several Jenkins instances in AWS and they are all now working. The pull request introduced a "Use old signing mechanism for authenticating request" tick box in the Amazon EC2 Cloud configuration, above the secret key and access key, did you tick that box? Without ticking that box I get authentication errors, but it was implied by the original AWS staff member who responded to the forum request about the v4 vs query text signer algorithm issue that the default (v4) signer algorithm should work in most cases and thus I have not made the box ticked by default. 'Test Connection' and 'Check AMI' both work for me when the box is ticked. My Jenkins instances are in the EU (Ireland) AWS region. If you could give me the error message and stack trace that would be very appreciated, and please verify that you have tested both with and without ticking the tick box. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
martinfr62 edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Tried pulling 131 and patching current master - it fails to connect to aws. Pulled current master - built and depoyed - connects okay (but has normal bug in it) Pulled 131 and split the patch in two - first half (ie prior 131) works - can connect to aws Apply second patch as a straight merge - and it builds - but cannot connect Next tried pull master, apply first half - and then apply my patch alone - this works. Looks like something in the second patch causes aws to fail. Let me know if you need a stack trace or anything if you cannot reproduce - but happens every time for me. Try executing 'Test Connection' from jenkins config page for ec2 - fails immediately for both profile and real keys. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
martinfr62 commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Tried pulling 131 and patching current master - it fails to connect to aws. Pulled current master - built and depoyed - connects okay (but has normal bug in it) Pulled 131 and split the patch in two - first half (ie prior 131) works - can connect to aws Apply second patch as a straight merge - and it builds - but cannot connect Next tried pull master, apply firt half - and then apply my patch alone - this works. Looks like something in the second patch causes aws to fail. Let me know if you need a stack trace or anything if you cannot reproduce - but happens every time for me. Try executing 'Test Connection' from jenkins config page for ec2 - fails immediately for both profile and real keys. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" I've updated my pull request with a new boolean EC2 Cloud parameter to enable using the query string signer, and the fix from Martin (though I just realized I only mentioned the query string signer fix in the commit message, wasn't intentional not to mention that...). Just thought it would be handy to put the changes in one place for anyone else reading this issue until the fix makes it into a proper release build, and to make sure that if the query string signer type is needed that users can enable it. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
chris fair
commented on JENKINS-26854
EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
I built Ximon's branch above, along with Martin's changes, and haven't seen this issue occur again. I did have to add 'config.setSignerOverride("QueryStringSignerType");' and I'm in region us-east-1.
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Vincent Rivellino commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" FYI: I'm running into the same issue on Jenkins ver. 1.580.3 with Amazon EC2 Plugin v1.26 on Amazon Linux with an IAM Instance Profile assigned to the instance. Would be great to get this fixed so we don't have to restart our Jenkins master every 6 hours. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
martinfr62
commented on JENKINS-26854
EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Okay - here's what i did in patch form against the pull 131 (so includes your Ximon).
Removed the transient connection object and created a new one every time in connect() and then updated/added some more logging so i could tell when the terminate was about to fire. I also found a typo in the log by doing this in EC2RetentionStrategy - was treating freeSecondsLeft as milliseconds - not seconds - so fixed that also.
—
src/main/java/hudson/plugins/ec2/EC2Cloud.java | 9 +
src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java | 6 +-
2 files changed, 10 insertions, 5 deletions
diff --git a/src/main/java/hudson/plugins/ec2/EC2Cloud.java b/src/main/java/hudson/plugins/ec2/EC2Cloud.java
index 8891999..ca102e6 100644
— a/src/main/java/hudson/plugins/ec2/EC2Cloud.java
+++ b/src/main/java/hudson/plugins/ec2/EC2Cloud.java
@@ -113,7 +113,7 @@ public abstract class EC2Cloud extends Cloud {
private final List templates;
private transient KeyPair usableKeyPair;
protected transient AmazonEC2 connection;
+ // protected transient AmazonEC2 connection;
private static AWSCredentialsProvider awsCredentialsProvider;
@@ -460,9 +460,9 @@ public abstract class EC2Cloud extends Cloud {
*/
public synchronized AmazonEC2 connect() throws AmazonClientException {
try {
if (connection == null) {
-connection = connect(createCredentialsProvider(), getEc2EndpointUrl());
-}
+ // if (connection == null) {
+ AmazonEC2connection = connect(createCredentialsProvider(), getEc2EndpointUrl());
+ // }
return connection;
} catch (IOException e) {
throw new AmazonClientException("Failed to retrieve the endpoint",e);
@@ -476,6 +476,7 @@ public abstract class EC2Cloud extends Cloud {
public synchronized static AmazonEC2 connect(AWSCredentialsProvider credentialsProvider, URL endpoint) {
awsCredentialsProvider = credentialsProvider;
ClientConfiguration config = new ClientConfiguration();
+config.setSignerOverride("QueryStringSignerType");
ProxyConfiguration proxyConfig = Jenkins.getInstance().proxy;
Proxy proxy = proxyConfig == null ? Proxy.NO_PROXY : proxyConfig.createProxy(endpoint.getHost());
if (! proxy.equals(Proxy.NO_PROXY) && proxy.address() instanceof InetSocketAddress) {
diff --git a/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java b/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java
index 22e185a..52d8f49 100644
a/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java
+++ b/src/main/java/hudson/plugins/ec2/EC2RetentionStrategy.java
@@ -94,6 +94,8 @@ public class EC2RetentionStrategy extends RetentionStrategy {
final long idleMilliseconds = System.currentTimeMillis() - c.getIdleStartMilliseconds();
if (idleTerminationMinutes > 0) {
// TODO: really think about the right strategy here, see JENKINS-23792
+LOGGER.fine("Test Idle timeout of "+c.getName() + " after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes");
+
if (idleMilliseconds > TimeUnit2.MINUTES.toMillis(idleTerminationMinutes)) {
LOGGER.info("Idle timeout of "+c.getName() + " after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes");
c.getNode().idleTimeout();
@@ -110,8 +112,10 @@ public class EC2RetentionStrategy extends RetentionStrategy Unknown macro: { final int freeSecondsLeft = (60*60) - (int)(TimeUnit2.SECONDS.convert(uptime, TimeUnit2.MILLISECONDS) % (60*60)); // if we have less "free" (aka already paid for) time left than our idle time, stop/terminate the instance // See JENKINS-23821+LOGGER.fine("Test Idle timeout of "+c.getName()+" after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes, with " + TimeUnit2.SECONDS.toMinutes(freeSecondsLeft) + " minutes remaining in billing period");+ if (freeSecondsLeft <= (Math.abs(idleTerminationMinutes*60))) {
-LOGGER.info("Idle timeout of "+c.getName()+" after " + TimeUnit2.MILLISECONDS.toMinutes(idleMilliseconds) + " idle minutes, with " + TimeUnit2.MILLISECONDS.toMinutes(fr
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
martinfr62 commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Looks like EC2Cloud.connect() caches the AmazonEC2 connection - testing patch which obtains the connection every time. Will taker a while, have to let the terminate event fire for the instances. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
martinfr62 commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" So still seeing the issue - this time though when the instance is terminated after it times out - instance then is left running in AWS, but dropped from jenkins, and cannot start any more instances until the aws instances have been terminated. Failed to fetch EC2 instance: i-2107a6db com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 1b435e23-f37f-4433-9526-f7226801cab9) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541) at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:208) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:354) at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:326) at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:42) at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:53) at java.lang.Thread.run(Thread.java:745) Feb 12, 2015 6:31:26 AM INFO hudson.plugins.ec2.EC2SlaveMonitor execute EC2 instance is dead: i-2107a6db Feb 12, 2015 6:31:28 AM WARNING hudson.plugins.ec2.EC2AbstractSlave getInstance Failed to fetch EC2 instance: i-2107a6db com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 8e53415f-6d34-4669-a76e-536d9cedcf37) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541) at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:208) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:354) at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:326) at hudson.plugins.ec2.EC2OndemandSlave.terminate(EC2OndemandSlave.java:63) at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:44) at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:53) at java.lang.Thread.run(Thread.java:745) Feb 12, 2015 6:31:28 AM INFO hudson.plugins.ec2.EC2OndemandSlave terminate EC2 instance already terminated: i-2107a6db Feb 12, 2015 6:31:28 AM INFO hudson.plugins.ec2.EC2OndemandSlave terminate Removed EC2 instance from jenkins master: i-2107a6db Feb 12, 2015 6:31:30 AM WARNING hudson.plugins.ec2.EC2AbstractSlave getInstance Failed to fetch EC2 instance: i-1a04a5e0 com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 4a7feb85-c0ff-4dda-9f88-f605513408a8) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4541) at hudson.plugins.ec2.EC2AbstractSlave.getInstance(EC2AbstractSlave.java:208) at hudson.plugins.ec2.EC2AbstractSlave.fetchLiveInstanceData(EC2AbstractSlave.java:354) at hudson.plugins.ec2.EC2AbstractSlave.isAlive(EC2AbstractSlave.java:326) at hudson.plugins.ec2.EC2SlaveMonitor.execute(EC2SlaveMonitor.java:42) at hudson.model.AsyncPeriodicWork$1.run(AsyncPeriodicWork.java:53) at java.lang.Thread.run(Thread.java:745) Feb 12, 2015 6:31:30 AM INFO hudson.plugins.ec2.EC2SlaveMonitor execute EC2 instance is
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
martinfr62 commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Pulled copy of 131 and built locally - just realized I had to apply above change as well. Will rebuild, deploy and test myself. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen
commented on JENKINS-26854
EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
To be clear, I have a working (well seems to work, I must admit it's not thoroughly tested yet) solution, but I'm not sure if it's wise for others to use. My solution is the pull request here, with the following tiny modification:
└─(.../scratch/ximon-ec2-plugin)─> git diff
diff --git a/src/main/java/hudson/plugins/ec2/EC2Cloud.java b/src/main/java/hudson/plugins/ec2/EC2Cloud.java
index 8891999..7feed81 100644
--- a/src/main/java/hudson/plugins/ec2/EC2Cloud.java
+++ b/src/main/java/hudson/plugins/ec2/EC2Cloud.java
@@ -476,6 +476,7 @@ public abstract class EC2Cloud extends Cloud {
public synchronized static AmazonEC2 connect(AWSCredentialsProvider credentialsProvider, URL endpoint) {
awsCredentialsProvider = credentialsProvider;
ClientConfiguration config = new ClientConfiguration();
+config.setSignerOverride("QueryStringSignerType");
ProxyConfiguration proxyConfig = Jenkins.getInstance().proxy;
Proxy proxy = proxyConfig == null ? Proxy.NO_PROXY : proxyConfig.createProxy(endpoint.getHost());
if (! proxy.equals(Proxy.NO_PROXY) && proxy.address() instanceof InetSocketAddress) {
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
martinfr62 edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Seeing same issue, restarting jenkins from the web browser cleared the issue, so clearly some sort of issue in the ec2 plugin itself, not an issue with the date on the instance hosting jenkins. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
martinfr62 commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Seeing same issue, restarting jenkins from the web browser cleared the issue, so clearly some sort of issue in the ec2 plugin itself, not an issue with the date on the instance hosting jenkins. My guess - the request object used to talk to amazon is not being created for every request, rather is being cached and reused. Thus it 'ages out' and is unusable after some period of time. Will take a look at the source code later today to see if I can find the issue. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Ah, the AuthFailure may affect others too. See: https://forums.aws.amazon.com/thread.jspa?messageID=574914&tstart=0. I'm seeing this issue in the eu-west-1 region. Update: Applying the force previous signer configuration solved this issue for me, that is this part of the forum article that I referred to: clientConfiguration.setSignerOverride(“QueryStringSignerType”); AmazonEC2 ec2 = new AmazonEC2Client(configuration); This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Ah, the AuthFailure may affect others too. See: https://forums.aws.amazon.com/thread.jspa?messageID=574914&tstart=0. I'm seeing this issue in the eu-west-1 region. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Ah, the AuthFailure may affect others too. See: https://forums.aws.amazon.com/thread.jspa?messageID=574914&tstart=0 This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" I have created a minimal pull request. See: https://github.com/jenkinsci/ec2-plugin/pull/131 This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" I tried building the EC2 plugin with AWS Java SDK 1.8.11, the first stable release after 1.8.10 that introduced the new InstanceProfileCredentialsProvider(true) functionality. This solves the missing refetch of credentials, causing the SDK to check them once a minute. However, it fails with the Auth error I referred to above: com.amazonaws.AmazonServiceException: AWS was not able to validate the provided access credentials (Service: AmazonEC2; Status Code: 401; Error Code: AuthFailure; Request ID: 0ad2caa1-4f05-41e7-b168-3dc37940265b) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) at com.amazonaws.services.ec2.AmazonEC2Client.describeKeyPairs(AmazonEC2Client.java:6321) at com.amazonaws.services.ec2.AmazonEC2Client.describeKeyPairs(AmazonEC2Client.java:8879) at hudson.plugins.ec2.EC2PrivateKey.find(EC2PrivateKey.java:135) at hudson.plugins.ec2.SlaveTemplate.getKeyPair(SlaveTemplate.java:719) at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:303) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:287) at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:283) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:745) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:875) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:745) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:875) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:648) at org.kohsuke.stapler.Stapler.service(Stapler.java:237) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcess
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" I tried building the EC2 plugin with AWS Java SDK 1.8.11, the first stable release after 1.8.10 that introduced the new InstanceProfileCredentialsProvider(true) functionality. This solves the missing refetch of credentials, causing the SDK to check them once a minute. However, it fails with the Auth error I referred to above: com.amazonaws.AmazonServiceException: AWS was not able to validate the provided access credentials (Service: AmazonEC2; Status Code: 401; Error Code: AuthFailure; Request ID: 0ad2caa1-4f05-41e7-b168-3dc37940265b) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) at com.amazonaws.services.ec2.AmazonEC2Client.describeKeyPairs(AmazonEC2Client.java:6321) at com.amazonaws.services.ec2.AmazonEC2Client.describeKeyPairs(AmazonEC2Client.java:8879) at hudson.plugins.ec2.EC2PrivateKey.find(EC2PrivateKey.java:135) at hudson.plugins.ec2.SlaveTemplate.getKeyPair(SlaveTemplate.java:719) at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:303) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:287) at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:283) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:745) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:875) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:745) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:875) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:648) at org.kohsuke.stapler.Stapler.service(Stapler.java:237) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.ja
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" I have no reason to think this is the cause of my problem but I just noticed that the plugin is built using v1.8.3 of the Java SDK while the latest version is 1.9.17, in theory at least there could be a bug fix in the newer versions. I looked through the release notes of the interim versions but didn't find an obvious bug fix that could be related to this issue. Update: Actually the release 1.8.10 of the Java SDK added the InstanceProfilerCredentialsProvider(true) behaviour I refer to above, but this was not directly mentioned in the release notes. This release was not a happy release, two hot fixes 1.8.10.1 and 1.8.10.2 were released in the following week. Update: If I build the EC2 plugin with the latest 1.9.17 SDK version it fails with an HTTP 401 Auth error which I haven't tracked down yet. I suspect this is because my AWS IAM InstanceProfile role does not include a permission which newer versions of the SDK require, but I haven't determined which permission is missing yet. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" I'm going to put some logging into a subclass of InstanceProfilerCredentialsProvider() because I suspect that for some reason the credentials are not being refreshed by this class... will let you know what I find out. Update: Indeed the EC2 credential refresh functionality of the AWS Java SDK is not invoked. See attached jenkins.log (look for lines containing "Ximon:") and gitdiff.txt. Rough highlights from the log (might not be entirely accurate, trying to remember what I did last night): 11:33:00 UTC: Jenkins finished responding to a /safeRestart request that I performed. 11:34:36 UTC: I instructed Jenkins to launch a new EC2 slave. The EC2 plugin fetched the EC2 credentials as part of launching the instance. 11:45:46 UTC: I instructed Jenkins to terminate the EC2 slave. The EC2 plugin did not refetch the credentials. 11:46:09 UTC: I instructed Jenkins to launch a new EC2 slave. The EC2 plugin did not refetch the credentials. 12:22:46 UTC: The EC2 plugin correctly stopped the EC2 slave instance after the idle timeout expired. The EC2 plugin did not refetch the credentials. 04:32:58 UTC: UNRELATED BUG: Jenkins logged "Making (i-a2a41545) offline because it’s not responding". Why is this logged over 4 hours after the EC2 plugin stopped the slave? 05:30:22 UTC: I refreshed the Jenkins log web page which seems to have caused the EC2 plugin to attempt to update its knowledge about the state of the slave. By this point the EC2 credentials have expired, but the EC2 plugin did not refetch the credentials. The call to the EC2 API failed with HTTP 400 AmazonServiceException "Request has expired". This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" I'm going to put some logging into a subclass of InstanceProfilerCredentialsProvider() because I suspect that for some reason the credentials are not being refreshed by this class... will let you know what I find out. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" My investigation into NTP hasn't found any problems: ubuntu@ip-172-30-0-149:~$ ntpq -pn remote refid st t when poll reach delay offset jitter == -87.232.1.40 62.231.32.35 4 u 1037 1024 3774.120 -1.394 2.634 +78.143.174.10 193.1.219.1162 u 382 1024 377 71.766 -7.342 48.358 -86.43.77.42 193.120.10.3 2 u 624 1024 337 40.7238.783 0.787 +85.91.1.180 195.66.241.2 2 u 94 1024 3771.812 -3.254 0.835 *91.189.89.199 192.93.2.20 2 u 741 1024 377 10.945 -1.322 0.847 The * shows the NTP server being used, and the reach 377 value shows that the NTP daemon was repeatedly able to contact the remote time server, the stratum number is low which is good, and the jitter and offset values are low which is good. Executing 'date' in both the Ubuntu host and the Docker container yields the same date and time. Unrelated to the NTP investigation, but related to my point about restarting Jenkins in the initial post, if I invoke http:///safeRestart then the slave is started correctly after Jenkins restarts, without my fixing any clocks. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" I have no reason to think this is the cause of my problem but I just noticed that the plugin is built using v1.8.3 of the Java SDK while the latest version is 1.9.17, in theory at least there could be a bug fix in the newer versions. I looked through the release notes of the interim versions but didn't find an obvious bug fix that could be related to this issue. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" My investigating into NTP hasn't found any problems. If I invoke http:///safeRestart then the slave is started correctly after Jenkins restarts, without my fixing any clocks. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" My investigation into NTP hasn't found any problems. If I invoke http:///safeRestart then the slave is started correctly after Jenkins restarts, without my fixing any clocks. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" I've been looking at the underlying AWS SDK code and it looks like it already has built in support for refreshing the credentials before they expire, and that this support is already being used by the Jenkins EC2 plugin. The only things I can think of are either that NTP isn't working properly and clock drift causes the problem, or that the synchronous mode of credential refresh being used by the Jenkins EC2 plugin doesn't work for some reason and perhaps the asynchronous background thread mode needs to be used? I will investigate NTP on my side. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen updated JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Change By: Ximon Eighteen (09/Feb/15 5:46 AM) Environment: * EC2 plugin version 1.26.* Jenkins 1.580.2 running inside the official Jenkins Docker LTS image , running inside .* Host O/S: Ubuntu 14.04 LTS 64-bit on an EC2 master. * EC2 rights are conferred via an EC2 InstanceProfile. EC2 plugin version 1.26. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen
updated JENKINS-26854
EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Change By:
Ximon Eighteen
(09/Feb/15 5:45 AM)
Environment:
Jenkins 1.580.2 running inside the official Jenkins Docker LTS image, running inside Ubuntu 14.04 LTS 64-bit on an EC2 master.
EC2 rights are conferred via an EC2 InstanceProfile.EC2 plugin version 1.26.
Description:
After Jenkins first starts it is able to launch EC2 slaves, both manually and when jobs indicate they need to use the slave label.A few hours later (not sure how long, maybe 24 hours?) slaves no longer start, manually or automatically. In "Manage Jenkins -> System Log -> All Jenkins Logs" the following error occurs repeatedly. Restarting Jenkins solves the problem.{code}Started EC2 alive slaves monitorFeb 09, 2015 5:14:47 AM INFO hudson.model.AsyncPeriodicWork$1 runFinished EC2 alive slaves monitor. 0 msFeb 09, 2015 5:15:51 AM INFO hudson.plugins.ec2.EC2Cloud provisionExcess workload after pending Spot instances: 1Feb 09, 2015 5:15:53 AM WARNING hudson.plugins.ec2.EC2Cloud provisionFailed to count the # of live instances on EC2com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 59f7935f-15f0-455c-a6f1-f6057f5ffc77) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:886) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:484) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:256) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:8798) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4137) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:8087) at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:228) at hudson.plugins.ec2.EC2Cloud.addProvisionedSlave(EC2Cloud.java:299) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:389) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:281) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:368) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745){code}I also don't understand the log statement {{Excess workload after pending Spot instances: 1}} as I have not ticked the "Use Spot instance" tick box.
In my cloud settings I have ticked the "Use EC2 instance profile to obtain credentials" and have set both the access key and secret key values to "THIS VALUE IS NOT USED - THE INSTANCE PROFILE IS USED INSTEAD".
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
--
You received this message because you are subscribed to the Google Groups "Jenkins Issues" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException "Request has expired"
Ximon Eighteen created JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException "Request has expired" Issue Type: Bug Assignee: Francis Upton Components: ec2-plugin Created: 09/Feb/15 5:18 AM Description: After Jenkins first starts it is able to launch EC2 slaves, both manually and when jobs indicate they need to use the slave label. A few hours later (not sure how long, maybe 24 hours?) slaves no longer start, manually or automatically. In "Manage Jenkins -> System Log -> All Jenkins Logs" the following error occurs repeatedly. Restarting Jenkins solves the problem. Started EC2 alive slaves monitor Feb 09, 2015 5:14:47 AM INFO hudson.model.AsyncPeriodicWork$1 run Finished EC2 alive slaves monitor. 0 ms Feb 09, 2015 5:15:51 AM INFO hudson.plugins.ec2.EC2Cloud provision Excess workload after pending Spot instances: 1 Feb 09, 2015 5:15:53 AM WARNING hudson.plugins.ec2.EC2Cloud provision Failed to count the # of live instances on EC2 com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 59f7935f-15f0-455c-a6f1-f6057f5ffc77) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:886) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:484) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:256) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:8798) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4137) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:8087) at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:228) at hudson.plugins.ec2.EC2Cloud.addProvisionedSlave(EC2Cloud.java:299) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:389) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:281) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:368) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) I also don't understand the log statement Excess workload after pending Spot instances: 1 as I have not ticked the "Use Spot instance" tick box. Environment: Jenkins 1.580.2 running inside the official Jenkins Docker LTS image, running inside Ubuntu 14.04 LTS 64-bit on an EC2 master. Project: Jenkins Priority: Blocker Reporter: Ximon Eighteen This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups "Jenkins Issues" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenki
