Re: Sporadic MSBuild errors / possible durable task issue?
On Friday, February 1, 2019 at 12:04:26 AM UTC, Daniel Becroft wrote: > > We had a similar issue, and it ended up being related to MSBuild itself, > not Jenkins. The jobs seemed to work correctly when run from a command > line, but it took a while and they eventually started failing as well. > > Our issue was due to the maximum number of temp files in %TEMP% being > exceeded (see the GetTempFileNameA[1] function, it has an internal limit of > 65,535). Once we cleared those out (and resolved the bug in our code that > wasn't cleaning them up), the builds worked again. > > [1] - > https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-gettempfilenamea > > That sounds like a weird one. I had a look at %TEMP% and there were only about 50 or so files at a time there. The build agents are on-demand so they usually only run a handful of builds before being torn down. -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/00594064-0c54-432d-a1d7-7bf938c9989a%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Sporadic MSBuild errors / possible durable task issue?
On Fri, Feb 1, 2019 at 2:31 AM Steven Foster wrote: > After updating Jenkins with plugins from early April 2018, I'm seeing > inconsistent failures in my pipelines' calls to MSBuild which I cannot > reproduce outside of Jenkins. It occurs in both powershell and bat steps. > > The error presents as > > 0>MSBUILD : error MSB4166: Child node "14" exited prematurely. Shutting > down. Diagnostic information may be found in files in the temporary files > directory named MSBuild_*.failure.txt. > > There is no such extra information file. This looks a bit like an > interrupt to me, is there a way I can reproduce how Jenkins sends > interrupts so I can verify that? > Are there any advanced durable task debugging techniques I could use to > find out what could be causing an interrupt or something else? > We had a similar issue, and it ended up being related to MSBuild itself, not Jenkins. The jobs seemed to work correctly when run from a command line, but it took a while and they eventually started failing as well. Our issue was due to the maximum number of temp files in %TEMP% being exceeded (see the GetTempFileNameA[1] function, it has an internal limit of 65,535). Once we cleared those out (and resolved the bug in our code that wasn't cleaning them up), the builds worked again. [1] - https://docs.microsoft.com/en-us/windows/desktop/api/fileapi/nf-fileapi-gettempfilenamea -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/CAJM0iSQNCSNi3Z0oQSfZTJD9ZmbjKwCyNO8VNuhgimiXDwLk8Q%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
Re: Sporadic MSBuild errors / possible durable task issue?
On Thursday, January 31, 2019 at 4:45:40 PM UTC, Simon Richter wrote: > > > There are a few things going on: > > - MSBuild sticks around after a build if the same project is rebuilt > - Jenkins kills processes belonging to a build after it believes the > build has finished > - MSBuild has serious issues with memory management > > All of these are somewhat unpredictable. > > I've found that it becomes a bit better if you set an environment variable > > MSBUILDDISABLENODEREUSE=1 > > as suggested[1] by the VS team, disable the Jenkins Process Tree > Killer[2] and try to use the 64 bit version of MSBuild[3] to avoid > running out of virtual memory (which MSBuild sometimes, but not always > reports as the reason for termination in the log files in %APPDATA%\Temp) > Thanks for the leads, I will look into those. Just to note that the issue only appeared after updating Jenkins (identical build agent images) and the builds were running fine before. -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/e7a25c32-ed5d-489b-8f63-f87902b12e57%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
Re: Sporadic MSBuild errors / possible durable task issue?
Hi, On 31.01.19 17:31, Steven Foster wrote: > There is no such extra information file. This looks a bit like an > interrupt to me, is there a way I can reproduce how Jenkins sends > interrupts so I can verify that? There are a few things going on: - MSBuild sticks around after a build if the same project is rebuilt - Jenkins kills processes belonging to a build after it believes the build has finished - MSBuild has serious issues with memory management All of these are somewhat unpredictable. I've found that it becomes a bit better if you set an environment variable MSBUILDDISABLENODEREUSE=1 as suggested[1] by the VS team, disable the Jenkins Process Tree Killer[2] and try to use the 64 bit version of MSBuild[3] to avoid running out of virtual memory (which MSBuild sometimes, but not always reports as the reason for termination in the log files in %APPDATA%\Temp) Simon [1] https://developercommunity.visualstudio.com/content/problem/301840/msbuild-not-terminating.html [2] https://wiki.jenkins.io/display/JENKINS/ProcessTreeKiller [3] https://gitlab.kitware.com/cmake/cmake/issues/18219 -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/115f850e-abd8-f380-0157-9a6676f382c6%40hogyros.de. For more options, visit https://groups.google.com/d/optout. signature.asc Description: OpenPGP digital signature
Sporadic MSBuild errors / possible durable task issue?
After updating Jenkins with plugins from early April 2018, I'm seeing inconsistent failures in my pipelines' calls to MSBuild which I cannot reproduce outside of Jenkins. It occurs in both powershell and bat steps. The error presents as 0>MSBUILD : error MSB4166: Child node "14" exited prematurely. Shutting down . Diagnostic information may be found in files in the temporary files directory named MSBuild_*.failure.txt. There is no such extra information file. This looks a bit like an interrupt to me, is there a way I can reproduce how Jenkins sends interrupts so I can verify that? Are there any advanced durable task debugging techniques I could use to find out what could be causing an interrupt or something else? -- You received this message because you are subscribed to the Google Groups "Jenkins Users" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-users+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-users/6e8d5f7f-7713-4052-9054-220d92b66a6f%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
