Re: Upgrade 1.598 -> 1.609.1 breaks jobs on 1 slave, revert doesn't fix

2015-06-11 Thread Ross Oliver
Root cause turned out to be an outdated commons-io jar I had installed for 
another project in the Jenkins user's ~/Library/Java/Extensions directory. 
 When I updated to the latest version, both Jenkins and the other project 
were happy.  The problem manifested at the time of the upgrade attempt 
because that was the first time the Jenkins slave on that host had been 
restarted since the installation of the outdated jar.

On Wednesday, June 10, 2015 at 3:16:14 PM UTC-7, Ross Oliver wrote:
>
> Greetings,
>
> I am running a Jenkins master and several slaves all on Mac OS 10.10. 
>  Yesterday I attempted an upgrade from 1.598 to 1.609.1, and immediately 
> all jobs on one slave started failing, unable to check out from git.  The 
> other slaves were unaffected an continued to operate normally.  After some 
> basic troubleshooting, I decided I needed to get these jobs working again, 
> so I reverted to 1.598.  I also had to manually restore the config.xml from 
> the previous day, as the storage of node data in 1.609.1 was incompatible 
> with 1.598.  The revert seems successful, but the jobs on the 1 slave were 
> still failing with the same errors.  The jobs would run fine if moved to 
> another slave.  Still other than the slave.jar file, I could not find 
> anything on the slave that the upgrade/revert might have changed.  For 
> troubleshooting, I created a very simple freestyle job that executed a 
> single shell command "echo this is here"  That job failed with the 
> following log message (slave host name removed per company confidentiality 
> policy):
>
> Started by user ross_oliver <http://mobilejenkins.apple.com/user/ross_oliver>
> [EnvInject] - Loading node environment variables.
> Building remotely on  (java7 master2) in workspace 
> 
> FATAL: command execution failedjava.io.IOException 
> <http://stacktrace.jenkins-ci.org/search?query=java.io.IOException>: Remote 
> call on  failed
>   at hudson.remoting.Channel.call(Channel.java:760) 
> <http://stacktrace.jenkins-ci.org/search/?query=hudson.remoting.Channel.call&entity=method>
>   at hudson.Launcher$RemoteLauncher.launch(Launcher.java:916) 
> <http://stacktrace.jenkins-ci.org/search/?query=hudson.Launcher$RemoteLauncher.launch&entity=method>
>   at hudson.Launcher$ProcStarter.start(Launcher.java:381) 
> <http://stacktrace.jenkins-ci.org/search/?query=hudson.Launcher$ProcStarter.start&entity=method>
>   at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:97) 
> <http://stacktrace.jenkins-ci.org/search/?query=hudson.tasks.CommandInterpreter.perform&entity=method>
>   at hudson.tasks.CommandInterpreter.perform(CommandInterpreter.java:66) 
> <http://stacktrace.jenkins-ci.org/search/?query=hudson.tasks.CommandInterpreter.perform&entity=method>
>   at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20) 
> <http://stacktrace.jenkins-ci.org/search/?query=hudson.tasks.BuildStepMonitor$1.perform&entity=method>
>   at 
> hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:761)
>  
> <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.AbstractBuild$AbstractBuildExecution.perform&entity=method>
>   at hudson.model.Build$BuildExecution.build(Build.java:199) 
> <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.Build$BuildExecution.build&entity=method>
>   at hudson.model.Build$BuildExecution.doRun(Build.java:160) 
> <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.Build$BuildExecution.doRun&entity=method>
>   at 
> hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:536) 
> <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.AbstractBuild$AbstractBuildExecution.run&entity=method>
>   at hudson.model.Run.execute(Run.java:1718) 
> <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.Run.execute&entity=method>
>   at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43) 
> <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.FreeStyleBuild.run&entity=method>
>   at hudson.model.ResourceController.execute(ResourceController.java:89) 
> <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.ResourceController.execute&entity=method>
>   at hudson.model.Executor.run(Executor.java:240) 
> <http://stacktrace.jenkins-ci.org/search/?query=hudson.model.Executor.run&entity=method>
> Caused by: java.lang.NoSuchFieldError 
> <http://stacktrace.jenkins-ci.org/search?query=java.lang.NoSuchFieldError>: 
> NULL_OUTPUT_STREAM
>   at hudson.Launcher$ProcStarter.(Launcher.java:152) 
> <http://stacktrace.je

Upgrade 1.598 -> 1.609.1 breaks jobs on 1 slave, revert doesn't fix

2015-06-10 Thread Ross Oliver
 at hudson.remoting.UserRequest.perform(UserRequest.java:49) 
<http://stacktrace.jenkins-ci.org/search/?query=hudson.remoting.UserRequest.perform&entity=method>
at hudson.remoting.Request$2.run(Request.java:324) 
<http://stacktrace.jenkins-ci.org/search/?query=hudson.remoting.Request$2.run&entity=method>
at 
hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:68)
 
<http://stacktrace.jenkins-ci.org/search/?query=hudson.remoting.InterceptingExecutorService$1.call&entity=method>
at java.util.concurrent.FutureTask.run(FutureTask.java:266) 
<http://stacktrace.jenkins-ci.org/search/?query=java.util.concurrent.FutureTask.run&entity=method>
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) 
<http://stacktrace.jenkins-ci.org/search/?query=java.util.concurrent.ThreadPoolExecutor.runWorker&entity=method>
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) 
<http://stacktrace.jenkins-ci.org/search/?query=java.util.concurrent.ThreadPoolExecutor$Worker.run&entity=method>
at java.lang.Thread.run(Thread.java:745) 
<http://stacktrace.jenkins-ci.org/search/?query=java.lang.Thread.run&entity=method>
at ..remote call to (Native Method)
at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1356)
at hudson.remoting.UserResponse.retrieve(UserRequest.java:221)
at hudson.remoting.Channel.call(Channel.java:752)
... 13 more
Build step 'Execute shell' marked build as failure
Finished: FAILURE


This job also succeeds on other slaves.  I've tried disconnecting and 
reconnecting the slave several times, and verified the slave.jar file 
removed and replaced each time.  I cleared the .jenkins jar cache on both 
the master and slave.  A lot of Google searching hasn't turned up anything 
directly relavant.  I've run out of ideas as to where to look for what else 
might have changed.  Any suggestions for additional troubleshooting would 
be greatly appreciated.

Thanks,
Ross Oliver

-- 
You received this message because you are subscribed to the Google Groups 
"Jenkins Users" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to jenkinsci-users+unsubscr...@googlegroups.com.
To view this discussion on the web visit 
https://groups.google.com/d/msgid/jenkinsci-users/4d82a68d-97ee-43a1-87a4-5502a4d99016%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.