So 2 hours is a hard cap on how long a build can run. Okie doke.

Perhaps then I'll wrap the run-tests step as you suggest and limit it to
100 minutes or something, and cleanly report if it times out.

Sound good?


On Fri, Aug 15, 2014 at 4:43 PM, Patrick Wendell <pwend...@gmail.com> wrote:

> Hey Nicholas,
>
> Yeah so Jenkins has it's own timeout mechanism and it will just kill the
> entire build after 120 minutes. But since run-tests is sitting in the
> middle of the tests, it can't actually post a failure message.
>
> I think run-tests-jenkins should just wrap the call to run-tests in a call
> in its own timeout. It might be possible to just use this:
>
> http://linux.die.net/man/1/timeout
>
> - Patrick
>
>
> On Fri, Aug 15, 2014 at 1:31 PM, Nicholas Chammas <
> nicholas.cham...@gmail.com> wrote:
>
>> OK, I've captured this in SPARK-3076
>> <https://issues.apache.org/jira/browse/SPARK-3076>.
>>
>> Patrick,
>>
>> Is the problem that this run-tests
>> <https://github.com/apache/spark/blob/0afe5cb65a195d2f14e8dfcefdbec5dac023651f/dev/run-tests-jenkins#L151>
>>  step
>> times out, and that is currently not handled gracefully? To be more
>> specific, it hangs for 120 minutes, times out, but the parent script for
>> some reason is also terminated. Does that sound right?
>>
>> Nick
>>
>>
>> On Fri, Aug 15, 2014 at 3:33 PM, Shivaram Venkataraman <
>> shiva...@eecs.berkeley.edu> wrote:
>>
>>> Jenkins runs for this PR https://github.com/apache/spark/pull/1960
>>> timed out without notification. The relevant Jenkins logs are at
>>>
>>>
>>> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18588/consoleFull
>>>
>>> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18592/consoleFull
>>>
>>> https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/18597/consoleFull
>>>
>>>
>>> On Fri, Aug 15, 2014 at 11:44 AM, Nicholas Chammas <
>>> nicholas.cham...@gmail.com> wrote:
>>>
>>>> Shivaram,
>>>>
>>>> Can you point us to an example of that happening? The Jenkins console
>>>> output, that is.
>>>>
>>>> Nick
>>>>
>>>>
>>>> On Fri, Aug 15, 2014 at 2:28 PM, Shivaram Venkataraman <
>>>> shiva...@eecs.berkeley.edu> wrote:
>>>>
>>>>> Also I think Jenkins doesn't post build timeouts to github. Is there
>>>>> anyway
>>>>> we can fix that ?
>>>>> On Aug 15, 2014 9:04 AM, "Patrick Wendell" <pwend...@gmail.com> wrote:
>>>>>
>>>>> > Hi All,
>>>>> >
>>>>> > I noticed that all PR tests run overnight had failed due to
>>>>> timeouts. The
>>>>> > patch that updates the netty shuffle I believe somehow inflated to
>>>>> the
>>>>> > build time significantly. That patch had been tested, but one change
>>>>> was
>>>>> > made before it was merged that was not tested.
>>>>> >
>>>>> > I've reverted the patch for now to see if it brings the build times
>>>>> back
>>>>> > down.
>>>>> >
>>>>> > - Patrick
>>>>> >
>>>>>
>>>>
>>>>
>>>
>>
>

Reply via email to