[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Title: Message Title Ximon Eighteen edited a comment on JENKINS-26854 Re: EC2 slave launch stops working after a while with AmazonServiceException Request has expired Iseethatthereisa[1.28releasetag|https://github.com/jenkinsci/ec2-plugin/releases/tag/ec2-1.28]whichincludesfixesforthisissue.HoweverIdon'tseea1.28releaseattheJenkinsupdatessiteoronthe[ec2pluginhomepage|https://wiki.jenkins-ci.org/display/JENKINS/Amazon+EC2+Plugin].Alsothisissueisstillmarkedas'Open'-hasthe1.28buildbeentestedbyanyonefollowingthisissue?Whenwillthis1.28versionbereleased? Additional:CanItestusingthisHPI?https://buildhive.cloudbees.com/job/jenkinsci/job/ec2-plugin/115/org.jenkins-ci.plugins$ec2/artifact/org.jenkins-ci.plugins/ec2/1.28-SNAPSHOT/ec2-1.28-SNAPSHOT.hpi Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Title: Message Title Ximon Eighteen commented on JENKINS-26854 Re: EC2 slave launch stops working after a while with AmazonServiceException Request has expired I see that there is a 1.28 release tag which includes fixes for this issue. However I don't see a 1.28 release at the Jenkins updates site or on the ec2 plugin home page. Also this issue is still marked as 'Open' - has the 1.28 build been tested by anyone following this issue? When will this 1.28 version be released? Add Comment This message was sent by Atlassian JIRA (v6.4.2#64017-sha1:e244265) -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired Unfortunately https://github.com/jenkinsci/ec2-plugin/pull/131 has now been closed without merging because the closer thought that ec2-1.27 fixed the issue. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired Interesting. It seems that JENKINS-26414 has updated to the latest Java AWS SDK and added the signer override. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired So having worked with Ximon offline, the signing step was the issue. I didnt notice it so didnt have it set. I should have made it the default as that was effectively the behaviour prior to this new version of the plugin. However, I think that automated self configuration as suggested by martinfr62 is a better idea. Sadly I haven't had any time for this issue for the last few weeks as the urgency on my side decreased the moment we managed to find a work around, i.e. a custom build of the plugin containing the fix documented in this issue. However, this isn't a sustainable long term solution, this PR is already one version behind the official version as 1.27 was just released and the PR is based on 1.26, so I hope to get back to this issue and increase the quality of my/our pull request. It bothers me however that there has been no obvious interest from the original plugin authors/maintainers in this issue... if I do improve the pull request I hope that it will be accepted and merged into a new release of the plugin. The plugin does seem to be actively maintained as there was a new 1.27 release a few days ago so there is hope I suppose. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired So having worked with Ximon offline, the signing step was the issue. I didnt notice it so didnt have it set. I should have made it the default as that was effectively the behaviour prior to this new version of the plugin. However, I think that automated self configuration as suggested by martinfr62 is a better idea. Sadly I haven't had any time for this issue for the last few weeks as the urgency on my side decreased the moment we managed to find a work around, i.e. a custom build of the plugin containing the fix documented in this issue. However, this isn't a sustainable long term solution so I hope to get back to this issue and increase the quality of my/our pull request. It bothers me however that there has been no obvious interest from the original plugin authors/maintainers in this issue... if I do improve the pull request I hope that it will be accepted and merged into a new release of the plugin. The plugin does seem to be actively maintained as there was a new 1.27 release a few days ago so there is hope I suppose. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired Interesting. It seems that JENKINS-26414 has updated to the latest Java AWS SDK and added the signer override. Is this issue now obsolete? This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired martinfr62 contacted me privately, we'll work out what the problem is. Separately to that, it occurred to me that end users won't want to know or care about request signing mechanism, or even having to specify whether or not instance profile credentials should be used. Instead both could be determined by the plugin by testing the connection, checking if it works with instance profile credentials, checking if it works with old or new signing algorithm, and just use what works. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired Hi martinfr62, I have verified that I have not left any files uncommitted, and that the pull request latest commit ID and commit message match what I have locally. I am now using that pull request on several Jenkins instances in AWS and they are all now working. The pull request introduced a "Use old signing mechanism for authenticating request" tick box in the Amazon EC2 Cloud configuration, above the secret key and access key, did you tick that box? Without ticking that box I get authentication errors, but it was implied by the original AWS staff member who responded to the forum request about the v4 vs query text signer algorithm issue that the default (v4) signer algorithm should work in most cases and thus I have not made the box ticked by default. 'Test Connection' and 'Check AMI' both work for me when the box is ticked. My Jenkins instances are in the EU (Ireland) AWS region. If you could give me the error message and stack trace that would be very appreciated, and please verify that you have tested both with and without ticking the tick box. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired Hi martinfr62, I have verified that I have not left any files uncommitted, and that the pull request latest commit ID and commit message match what I have locally. I am now using that pull request on several Jenkins instances in AWS and they are all now working. The pull request introduced a "Use old signing mechanism for authenticating request" tick box in the Amazon EC2 Cloud configuration, above the secret key and access key, did you tick that box? Without ticking that box I get authentication errors. It was implied by the original AWS staff member who responded to the forum request about the v4 vs query text signer algorithm issue that the default (v4) signer algorithm should work in most cases and thus I have not made the box ticked by default. However, thinking about it, the previous behaviour of the EC2 plugin was to use the old signer algorithm, so perhaps the old behaviour should be the default? There is currently no description or help for the tick box, I should add that. 'Test Connection' and 'Check AMI' both work for me when the box is ticked. My Jenkins instances are in the EU (Ireland) AWS region. If you could give me the error message and stack trace that would be very appreciated, and please verify that you have tested both with and without ticking the tick box. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired Hi martinfr62, I have verified that I have not left any files uncommitted, and that the pull request latest commit ID and commit message match what I have locally. I am now using that pull request on several Jenkins instances in AWS and they are all now working. The pull request introduced a "Use old signing mechanism for authenticating request" tick box in the Amazon EC2 Cloud configuration, above the secret key and access key, did you tick that box? Without ticking that box I get authentication errors, but it was implied by the original AWS staff member who responded to the forum request about the v4 vs query text signer algorithm issue that the default (v4) signer algorithm should work in most cases and thus I have not made the box ticked by default. There is currently no description or help for the tick box, I should add that. 'Test Connection' and 'Check AMI' both work for me when the box is ticked. My Jenkins instances are in the EU (Ireland) AWS region. If you could give me the error message and stack trace that would be very appreciated, and please verify that you have tested both with and without ticking the tick box. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired I've updated my pull request with a new boolean EC2 Cloud parameter to enable using the query string signer, and the fix from Martin (though I just realized I only mentioned the query string signer fix in the commit message, wasn't intentional not to mention that...). Just thought it would be handy to put the changes in one place for anyone else reading this issue until the fix makes it into a proper release build, and to make sure that if the query string signer type is needed that users can enable it. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen
commented on JENKINS-26854
EC2 slave launch stops working after a while with AmazonServiceException Request has expired
To be clear, I have a working (well seems to work, I must admit it's not thoroughly tested yet) solution, but I'm not sure if it's wise for others to use. My solution is the pull request here, with the following tiny modification:
└─(.../scratch/ximon-ec2-plugin)─ git diff
diff --git a/src/main/java/hudson/plugins/ec2/EC2Cloud.java b/src/main/java/hudson/plugins/ec2/EC2Cloud.java
index 8891999..7feed81 100644
--- a/src/main/java/hudson/plugins/ec2/EC2Cloud.java
+++ b/src/main/java/hudson/plugins/ec2/EC2Cloud.java
@@ -476,6 +476,7 @@ public abstract class EC2Cloud extends Cloud {
public synchronized static AmazonEC2 connect(AWSCredentialsProvider credentialsProvider, URL endpoint) {
awsCredentialsProvider = credentialsProvider;
ClientConfiguration config = new ClientConfiguration();
+config.setSignerOverride("QueryStringSignerType");
ProxyConfiguration proxyConfig = Jenkins.getInstance().proxy;
Proxy proxy = proxyConfig == null ? Proxy.NO_PROXY : proxyConfig.createProxy(endpoint.getHost());
if (! proxy.equals(Proxy.NO_PROXY) proxy.address() instanceof InetSocketAddress) {
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
--
You received this message because you are subscribed to the Google Groups Jenkins Issues group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired I tried building the EC2 plugin with AWS Java SDK 1.8.11, the first stable release after 1.8.10 that introduced the new InstanceProfileCredentialsProvider(true) functionality. This solves the missing refetch of credentials, causing the SDK to check them once a minute. However, it fails with the Auth error I referred to above: com.amazonaws.AmazonServiceException: AWS was not able to validate the provided access credentials (Service: AmazonEC2; Status Code: 401; Error Code: AuthFailure; Request ID: 0ad2caa1-4f05-41e7-b168-3dc37940265b) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) at com.amazonaws.services.ec2.AmazonEC2Client.describeKeyPairs(AmazonEC2Client.java:6321) at com.amazonaws.services.ec2.AmazonEC2Client.describeKeyPairs(AmazonEC2Client.java:8879) at hudson.plugins.ec2.EC2PrivateKey.find(EC2PrivateKey.java:135) at hudson.plugins.ec2.SlaveTemplate.getKeyPair(SlaveTemplate.java:719) at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:303) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:287) at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:283) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:745) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:875) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:745) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:875) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:648) at org.kohsuke.stapler.Stapler.service(Stapler.java:237) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired I have created a minimal pull request. See: https://github.com/jenkinsci/ec2-plugin/pull/131 This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired I have no reason to think this is the cause of my problem but I just noticed that the plugin is built using v1.8.3 of the Java SDK while the latest version is 1.9.17, in theory at least there could be a bug fix in the newer versions. I looked through the release notes of the interim versions but didn't find an obvious bug fix that could be related to this issue. Update: Actually the release 1.8.10 of the Java SDK added the InstanceProfilerCredentialsProvider(true) behaviour I refer to above, but this was not directly mentioned in the release notes. This release was not a happy release, two hot fixes 1.8.10.1 and 1.8.10.2 were released in the following week. Update: If I build the EC2 plugin with the latest 1.9.17 SDK version it fails with an HTTP 401 Auth error which I haven't tracked down yet. I suspect this is because my AWS IAM InstanceProfile role does not include a permission which newer versions of the SDK require, but I haven't determined which permission is missing yet. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired I tried building the EC2 plugin with AWS Java SDK 1.8.11, the first stable release after 1.8.10 that introduced the new InstanceProfileCredentialsProvider(true) functionality. This solves the missing refetch of credentials, causing the SDK to check them once a minute. However, it fails with the Auth error I referred to above: com.amazonaws.AmazonServiceException: AWS was not able to validate the provided access credentials (Service: AmazonEC2; Status Code: 401; Error Code: AuthFailure; Request ID: 0ad2caa1-4f05-41e7-b168-3dc37940265b) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:1032) at com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:687) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:441) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:292) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:9225) at com.amazonaws.services.ec2.AmazonEC2Client.describeKeyPairs(AmazonEC2Client.java:6321) at com.amazonaws.services.ec2.AmazonEC2Client.describeKeyPairs(AmazonEC2Client.java:8879) at hudson.plugins.ec2.EC2PrivateKey.find(EC2PrivateKey.java:135) at hudson.plugins.ec2.SlaveTemplate.getKeyPair(SlaveTemplate.java:719) at hudson.plugins.ec2.SlaveTemplate.provisionOndemand(SlaveTemplate.java:303) at hudson.plugins.ec2.SlaveTemplate.provision(SlaveTemplate.java:287) at hudson.plugins.ec2.EC2Cloud.doProvision(EC2Cloud.java:283) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:298) at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:161) at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:96) at org.kohsuke.stapler.MetaClass$1.doDispatch(MetaClass.java:121) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:745) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:875) at org.kohsuke.stapler.MetaClass$6.doDispatch(MetaClass.java:249) at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:53) at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:745) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:875) at org.kohsuke.stapler.Stapler.invoke(Stapler.java:648) at org.kohsuke.stapler.Stapler.service(Stapler.java:237) at javax.servlet.http.HttpServlet.service(HttpServlet.java:848) at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:686) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1494) at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:96) at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:88) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:48) at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1482) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:84) at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:51) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at jenkins.security.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:117) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.providers.anonymous.AnonymousProcessingFilter.doFilter(AnonymousProcessingFilter.java:125) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.rememberme.RememberMeProcessingFilter.doFilter(RememberMeProcessingFilter.java:142) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at org.acegisecurity.ui.AbstractProcessingFilter.doFilter(AbstractProcessingFilter.java:271) at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:87) at
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired Ah, the AuthFailure may affect others too. See: https://forums.aws.amazon.com/thread.jspa?messageID=574914tstart=0 This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired Ah, the AuthFailure may affect others too. See: https://forums.aws.amazon.com/thread.jspa?messageID=574914tstart=0. I'm seeing this issue in the eu-west-1 region. Update: Applying the force previous signer configuration solved this issue for me, that is this part of the forum article that I referred to: clientConfiguration.setSignerOverride(“QueryStringSignerType”); AmazonEC2 ec2 = new AmazonEC2Client(configuration); This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired Ah, the AuthFailure may affect others too. See: https://forums.aws.amazon.com/thread.jspa?messageID=574914tstart=0. I'm seeing this issue in the eu-west-1 region. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired I'm going to put some logging into a subclass of InstanceProfilerCredentialsProvider() because I suspect that for some reason the credentials are not being refreshed by this class... will let you know what I find out. Update: Indeed the EC2 credential refresh functionality of the AWS Java SDK is not invoked. See attached jenkins.log (look for lines containing "Ximon:") and gitdiff.txt. Rough highlights from the log (might not be entirely accurate, trying to remember what I did last night): 11:33:00 UTC: Jenkins finished responding to a /safeRestart request that I performed. 11:34:36 UTC: I instructed Jenkins to launch a new EC2 slave. The EC2 plugin fetched the EC2 credentials as part of launching the instance. 11:45:46 UTC: I instructed Jenkins to terminate the EC2 slave. The EC2 plugin did not refetch the credentials. 11:46:09 UTC: I instructed Jenkins to launch a new EC2 slave. The EC2 plugin did not refetch the credentials. 12:22:46 UTC: The EC2 plugin correctly stopped the EC2 slave instance after the idle timeout expired. The EC2 plugin did not refetch the credentials. 04:32:58 UTC: UNRELATED BUG: Jenkins logged "Making NODE NAME (i-a2a41545) offline because it’s not responding". Why is this logged over 4 hours after the EC2 plugin stopped the slave? 05:30:22 UTC: I refreshed the Jenkins log web page which seems to have caused the EC2 plugin to attempt to update its knowledge about the state of the slave. By this point the EC2 credentials have expired, but the EC2 plugin did not refetch the credentials. The call to the EC2 API failed with HTTP 400 AmazonServiceException "Request has expired". This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired I've been looking at the underlying AWS SDK code and it looks like it already has built in support for refreshing the credentials before they expire, and that this support is already being used by the Jenkins EC2 plugin. The only things I can think of are either that NTP isn't working properly and clock drift causes the problem, or that the synchronous mode of credential refresh being used by the Jenkins EC2 plugin doesn't work for some reason and perhaps the asynchronous background thread mode needs to be used? I will investigate NTP on my side. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired My investigating into NTP hasn't found any problems. If I invoke http://jenkins/safeRestart then the slave is started correctly after Jenkins restarts, without my fixing any clocks. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired My investigation into NTP hasn't found any problems. If I invoke http://jenkins/safeRestart then the slave is started correctly after Jenkins restarts, without my fixing any clocks. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired I have no reason to think this is the cause of my problem but I just noticed that the plugin is built using v1.8.3 of the Java SDK while the latest version is 1.9.17, in theory at least there could be a bug fix in the newer versions. I looked through the release notes of the interim versions but didn't find an obvious bug fix that could be related to this issue. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen edited a comment on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired My investigation into NTP hasn't found any problems: ubuntu@ip-172-30-0-149:~$ ntpq -pn remote refid st t when poll reach delay offset jitter == -87.232.1.40 62.231.32.35 4 u 1037 1024 3774.120 -1.394 2.634 +78.143.174.10 193.1.219.1162 u 382 1024 377 71.766 -7.342 48.358 -86.43.77.42 193.120.10.3 2 u 624 1024 337 40.7238.783 0.787 +85.91.1.180 195.66.241.2 2 u 94 1024 3771.812 -3.254 0.835 *91.189.89.199 192.93.2.20 2 u 741 1024 377 10.945 -1.322 0.847 The * shows the NTP server being used, and the reach 377 value shows that the NTP daemon was repeatedly able to contact the remote time server, the stratum number is low which is good, and the jitter and offset values are low which is good. Executing 'date' in both the Ubuntu host and the Docker container yields the same date and time. Unrelated to the NTP investigation, but related to my point about restarting Jenkins in the initial post, if I invoke http://jenkins/safeRestart then the slave is started correctly after Jenkins restarts, without my fixing any clocks. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen commented on JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired I'm going to put some logging into a subclass of InstanceProfilerCredentialsProvider() because I suspect that for some reason the credentials are not being refreshed by this class... will let you know what I find out. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen updated JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired Change By: Ximon Eighteen (09/Feb/15 5:46 AM) Environment: *EC2pluginversion1.26.* Jenkins1.580.2runninginsidetheofficialJenkinsDockerLTSimage ,runninginside .*HostO/S: Ubuntu14.04LTS64-bitonanEC2master. * EC2rightsareconferredviaanEC2InstanceProfile. EC2pluginversion1.26. This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen
updated JENKINS-26854
EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Change By:
Ximon Eighteen
(09/Feb/15 5:45 AM)
Environment:
Jenkins1.580.2runninginsidetheofficialJenkinsDockerLTSimage,runninginsideUbuntu14.04LTS64-bitonanEC2master.
EC2rightsareconferredviaanEC2InstanceProfile.EC2pluginversion1.26.
Description:
AfterJenkinsfirststartsitisabletolaunchEC2slaves,bothmanuallyandwhenjobsindicatetheyneedtousetheslavelabel.Afewhourslater(notsurehowlong,maybe24hours?)slavesnolongerstart,manuallyorautomatically.InManageJenkins-SystemLog-AllJenkinsLogsthefollowingerroroccursrepeatedly.RestartingJenkinssolvestheproblem.{code}StartedEC2aliveslavesmonitorFeb09,20155:14:47AMINFOhudson.model.AsyncPeriodicWork$1runFinishedEC2aliveslavesmonitor.0msFeb09,20155:15:51AMINFOhudson.plugins.ec2.EC2CloudprovisionExcessworkloadafterpendingSpotinstances:1Feb09,20155:15:53AMWARNINGhudson.plugins.ec2.EC2CloudprovisionFailedtocountthe#ofliveinstancesonEC2com.amazonaws.AmazonServiceException:Requesthasexpired.(Service:AmazonEC2;StatusCode:400;ErrorCode:RequestExpired;RequestID:59f7935f-15f0-455c-a6f1-f6057f5ffc77) atcom.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:886) atcom.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:484) atcom.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:256) atcom.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:8798) atcom.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4137) atcom.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:8087) athudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:228) athudson.plugins.ec2.EC2Cloud.addProvisionedSlave(EC2Cloud.java:299) athudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:389) athudson.slaves.NodeProvisioner.update(NodeProvisioner.java:281) athudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51) athudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:368) athudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) atjava.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) atjava.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) atjava.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) atjava.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) atjava.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) atjava.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) atjava.lang.Thread.run(Thread.java:745){code}Ialsodontunderstandthelogstatement{{ExcessworkloadafterpendingSpotinstances:1}}asIhavenottickedtheUseSpotinstancetickbox.
InmycloudsettingsIhavetickedtheUseEC2instanceprofiletoobtaincredentialsandhavesetboththeaccesskeyandsecretkeyvaluestoTHISVALUEISNOTUSED-THEINSTANCEPROFILEISUSEDINSTEAD.
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators.
For more information on JIRA, see: http://www.atlassian.com/software/jira
--
You received this message because you are subscribed to the Google Groups Jenkins Issues group.
To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-issues+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
[JIRA] [ec2-plugin] (JENKINS-26854) EC2 slave launch stops working after a while with AmazonServiceException Request has expired
Ximon Eighteen created JENKINS-26854 EC2 slave launch stops working after a while with AmazonServiceException Request has expired Issue Type: Bug Assignee: Francis Upton Components: ec2-plugin Created: 09/Feb/15 5:18 AM Description: After Jenkins first starts it is able to launch EC2 slaves, both manually and when jobs indicate they need to use the slave label. A few hours later (not sure how long, maybe 24 hours?) slaves no longer start, manually or automatically. In "Manage Jenkins - System Log - All Jenkins Logs" the following error occurs repeatedly. Restarting Jenkins solves the problem. Started EC2 alive slaves monitor Feb 09, 2015 5:14:47 AM INFO hudson.model.AsyncPeriodicWork$1 run Finished EC2 alive slaves monitor. 0 ms Feb 09, 2015 5:15:51 AM INFO hudson.plugins.ec2.EC2Cloud provision Excess workload after pending Spot instances: 1 Feb 09, 2015 5:15:53 AM WARNING hudson.plugins.ec2.EC2Cloud provision Failed to count the # of live instances on EC2 com.amazonaws.AmazonServiceException: Request has expired. (Service: AmazonEC2; Status Code: 400; Error Code: RequestExpired; Request ID: 59f7935f-15f0-455c-a6f1-f6057f5ffc77) at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:886) at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:484) at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:256) at com.amazonaws.services.ec2.AmazonEC2Client.invoke(AmazonEC2Client.java:8798) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:4137) at com.amazonaws.services.ec2.AmazonEC2Client.describeInstances(AmazonEC2Client.java:8087) at hudson.plugins.ec2.EC2Cloud.countCurrentEC2Slaves(EC2Cloud.java:228) at hudson.plugins.ec2.EC2Cloud.addProvisionedSlave(EC2Cloud.java:299) at hudson.plugins.ec2.EC2Cloud.provision(EC2Cloud.java:389) at hudson.slaves.NodeProvisioner.update(NodeProvisioner.java:281) at hudson.slaves.NodeProvisioner.access$000(NodeProvisioner.java:51) at hudson.slaves.NodeProvisioner$NodeProvisionerInvoker.doRun(NodeProvisioner.java:368) at hudson.triggers.SafeTimerTask.run(SafeTimerTask.java:54) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) I also don't understand the log statement Excess workload after pending Spot instances: 1 as I have not ticked the "Use Spot instance" tick box. Environment: Jenkins 1.580.2 running inside the official Jenkins Docker LTS image, running inside Ubuntu 14.04 LTS 64-bit on an EC2 master. Project: Jenkins Priority: Blocker Reporter: Ximon Eighteen This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators. For more information on JIRA, see: http://www.atlassian.com/software/jira -- You received this message because you are subscribed to the Google Groups Jenkins Issues group. To unsubscribe from this group and stop receiving emails from it, send an email to
