Hi,
I'm getting random time outs from various executable tasks. For
example, one executes:
<exec>
<executable>$(ComSpec)</executable>
<buildArgs>/c dir /O-S *.*</buildArgs>
<description>"List file sizes"</description>
</exec>
Which lists the file sizes of the built files in a particular
directory so they appear in the logs.
Over the weekend this ran and the log says:
<buildresults>
<message>Command line 'C:\Windows\system32\cmd.exe /c dir /O-S
*.*'
timed out after 600 seconds.</message>
</buildresults>
And yet the log contains the output of the command, so it looks like
the command ran successfully. There are only 207 files in that
directory, not millions, so 'dir' should not time out listing that
many files in 10 minutes; I would expect it to run almost instantly.
The last couple of lines of this are what I would expect to see when
the dir command finished, and it does not look like it has stalled:
<message> 207 File(s) 2,593,893,817 bytes</message>
<message> 7 Dir(s) 752,638,242,816 bytes free</
message>
</buildresults>
This appears to happen randomly for various tasks, about 1 in every
1000 tasks, but can happen in batches once it starts happening. We
have approx 100 projects and each one has a few tasks so we see these
time outs several times a day and people are starting to lose
confidence that they have actually broken the build because they just
think it is another time out, so we need to find a solution to the
problem.
The build machine is a Dell T3500 (4 cores, hyperthreaded, 12Gb RAM,
dual RAID mirrored HDDs - no h/w errors I could find) running Win 7/64
Pro. We are using the latest CCNET built from GIT source. I have tried
a number of things to fix this, including turning on Debug logging.
For a different task that timed out I did notice from the logs that
although the process had finished the output/error streams never
closed and so the exec task still timed out waiting on these. I even
tried changing the code to not wait on these streams closing (only the
process ending, then 'cancelled' the streams) but that hasn't fixed
the issue. Since that is how the code currently is then in the case of
the above 'dir' task the process must never have exited.
Anyone seen this type of thing and have a solution / explanation? I
don't currently have any ideas other than speculative wild guesses at
anti-virus software (Symmantec Corporate) or windows cmd process
spawning itself, but this seems unlikely.
Thanks
Phil
