> What about the case where the rebase *lessens* the improvement? That
is, you're expecting these 10 cases to improve, but after a rebase, only
1 improves. That's news! But a blanket "accept improvements" won't tell you.
I don't think that scenario currently triggers a CI failure. So this
wouldn'
Yes, this is exactly one of the issues that marge might run into as well,
the aggregate ends up performing differently from the individual ones. Now
we have marge to ensure that at least the aggregate builds together, which
is the whole point of these merge trains. Not to end up in a situation
wher
What about the case where the rebase *lessens* the improvement? That is, you're
expecting these 10 cases to improve, but after a rebase, only 1 improves.
That's news! But a blanket "accept improvements" won't tell you.
I'm not hard against this proposal, because I know precise tracking has its o
After the idea of letting marge accept unexpected perf improvements and
looking at https://gitlab.haskell.org/ghc/ghc/-/merge_requests/4759
which failed because of a single test, for a single build flavour
crossing the
improvement threshold where CI fails after rebasing I wondered.
When would acc
Simon Peyton Jones via ghc-devs writes:
> > We need to do something about this, and I'd advocate for just not making
> > stats fail with marge.
>
> Generally I agree. One point you don’t mention is that our perf tests
> (which CI forces us to look at assiduously) are often pretty weird
> cases.
ion I'm not entirely sure
> if this is not just a witch-hunt hurting/beating mostly most active GHC
> developers. Another idea may be to give up on CI doing perf reg testing
> at all and invest saved resources into proper investigation of
> GHC/Haskell programs performance. Not sure
sion I'm not
entirely sure
if this is not just a witch-hunt hurting/beating mostly
most active GHC
developers. Another idea may be to give up on CI doing
perf reg testing
at all and invest saved resources into prope
e even on lightly loaded workstation.
>>> Not sure if CI is not run on some shared cloud resources where it may be
>>> even higher.
>>>
>>> I've done simple experiment of pining ghc compiling ghc-cabal and I've
>>> been able to "speed" it up
d cloud resources where it may be
>> even higher.
>>
>> I've done simple experiment of pining ghc compiling ghc-cabal and I've
>> been able to "speed" it up by 5-10% on W-2265.
>>
>> Also following this CI/performance regs discussion I'm no
g ghc-cabal and I've
> been able to "speed" it up by 5-10% on W-2265.
>
> Also following this CI/performance regs discussion I'm not entirely sure
> if this is not just a witch-hunt hurting/beating mostly most active GHC
> developers. Another idea may be to give up on C
t hurting/beating mostly most active GHC
developers. Another idea may be to give up on CI doing perf reg testing
at all and invest saved resources into proper investigation of
GHC/Haskell programs performance. Not sure, if this would not be more
beneficial longer term.
Just one random number thrown t
On 17 Mar 2021, at 16:16, Andreas Klebinger wrote:
>
> While I fully agree with this. We should *always* want to know if a small
> syntetic benchmark regresses by a lot.
> Or in other words we don't want CI to accept such a regression for us ever,
> but the developer of a patch should need to e
> I'd be quite happy to accept a 25% regression on T9872c if it yielded
a 1% improvement on compiling Cabal. T9872 is very very very strange!
(Maybe if *all* the T9872 tests regressed, I'd be more worried.)
While I fully agree with this. We should *always* want to know if a
small syntetic benchma
Yes, I think the counter point of "automating what Ben does" so people
besides Ben can do it is very important. In this case, I think a good
thing we could do is asynchronously build more of master post-merge,
such as use the perf stats to automatically bisect anything that is
fishy, including
Re: Performance drift: I opened
https://gitlab.haskell.org/ghc/ghc/-/issues/17658 a while ago with an idea
of how to measure drift a bit better.
It's basically an automatically checked version of "Ben stares at
performance reports every two weeks and sees that T9872 has regressed by
10% since 9.0"
> On Mar 17, 2021, at 6:18 AM, Moritz Angermann
> wrote:
>
> But what do we expect of patch authors? Right now if five people write
> patches to GHC, and each of them eventually manage to get their MRs green,
> after a long review, they finally see it assigned to marge, and then it
> starts
I am not advocating to drop perf tests during merge requests, I just want
them to not be fatal for marge batches. Yes this means that a bunch of
unrelated merge requests all could be fine wrt to the perf checks per merge
request, but the aggregate might fail perf. And then subsequently the next
MR
Ah, so it was really two identical pipelines (one for the branch where
Margebot batches commits, and one for the MR that Margebot creates before
merging). That's indeed a non-trivial amount of purely wasted
computer-hours.
Taking a step back, I am inclined to agree with the proposal of not
checkin
aking something worse, even after it lands.
Simon
From: ghc-devs On Behalf Of Moritz Angermann
Sent: 17 March 2021 03:00
To: ghc-devs
Subject: On CI
Hi there!
Just a quick update on our CI situation. Ben, John, Davean and I have been
discussion on CI yesterday, and what we can do about it, as we
*why* is a very good question. The MR fixing it is here:
https://gitlab.haskell.org/ghc/ghc/-/merge_requests/5275
On Wed, Mar 17, 2021 at 4:26 PM Spiwack, Arnaud
wrote:
> Then I have a question: why are there two pipelines running on each merge
> batch?
>
> On Wed, Mar 17, 2021 at 9:22 AM Moritz
Then I have a question: why are there two pipelines running on each merge
batch?
On Wed, Mar 17, 2021 at 9:22 AM Moritz Angermann
wrote:
> No it wasn't. It was about the stat failures described in the next
> paragraph. I could have been more clear about that. My apologies!
>
> On Wed, Mar 17, 20
No it wasn't. It was about the stat failures described in the next
paragraph. I could have been more clear about that. My apologies!
On Wed, Mar 17, 2021 at 4:14 PM Spiwack, Arnaud
wrote:
>
> and if either of both (see below) failed, marge's merge would fail as well.
>>
>
> Re: “see below” is th
> and if either of both (see below) failed, marge's merge would fail as well.
>
Re: “see below” is this referring to a missing part of your email?
___
ghc-devs mailing list
ghc-devs@haskell.org
http://mail.haskell.org/cgi-bin/mailman/listinfo/ghc-devs
Hi there!
Just a quick update on our CI situation. Ben, John, Davean and I have been
discussion on CI yesterday, and what we can do about it, as well as some
minor notes on why we are frustrated with it. This is an open invitation to
anyone who in earnest wants to work on CI. Please come forward
*To:* ghc-devs mailto:ghc-devs@haskell.org>>
*Subject:* Re: On CI
I'm not opposed to some effort going into this, but I would
strongly opposite putting all our effort there. Incremental CI can
cut multiple hours to < mere minutes, especially with the test
suite being
n
>
>
>
> *From:* ghc-devs *On Behalf Of *John
> Ericson
> *Sent:* 22 February 2021 05:53
> *To:* ghc-devs
> *Subject:* Re: On CI
>
>
>
> I'm not opposed to some effort going into this, but I would strongly
> opposite putting all our effort there. Inc
r 10) I think we need to
do less compiling - hence incremental CI.
Simon
From: ghc-devs On Behalf Of John Ericson
Sent: 22 February 2021 05:53
To: ghc-devs
Subject: Re: On CI
I'm not opposed to some effort going into this, but I would strongly opposite
putting all our effort there. Incr
the
aggressive caching scheme.
Just my 2p
Josef
*From:* ghc-devs mailto:ghc-devs-boun...@haskell.org>> on behalf of Simon Peyton
Jones via ghc-devs mailto:ghc-devs@haskell.org>>
*Sent:* F
es" and restart building the libraries. That way we can
> validate that the build failure was a true build failure and not just due to
> the aggressive caching scheme.
>
> Just my 2p
>
> Josef
>
> From: ghc-devs <mailto:ghc-devs-boun...@haskell.org>> on
---
> *From:* ghc-devs on behalf of Simon Peyton
> Jones via ghc-devs
> *Sent:* Friday, February 19, 2021 8:57 AM
> *To:* John Ericson ; ghc-devs <
> ghc-devs@haskell.org>
> *Subject:* RE: On CI
>
>
>1. Building and testing happen together. When test
nt: Friday, February 19, 2021 8:57 AM
To: John Ericson ; ghc-devs
Subject: RE: On CI
1. Building and testing happen together. When tests failure spuriously, we
also have to rebuild GHC in addition to re-running the tests. That's pure
waste.
https://gitlab.haskell.org/ghc/ghc/-/i
Simon Peyton Jones via ghc-devs writes:
>> 1. Building and testing happen together. When tests failure
>> spuriously, we also have to rebuild GHC in addition to re-running
>> the tests. That's pure waste.
>> https://gitlab.haskell.org/ghc/ghc/-/issues/13897 tracks this more
>> or less.
imon
From: ghc-devs On Behalf Of John Ericson
Sent: 19 February 2021 03:19
To: ghc-devs
Subject: Re: On CI
I am also wary of us to deferring checking whole platforms and what not. I
think that's just kicking the can down the road, and will result in more
variance and uncertainty. It might
I am also wary of us to deferring checking whole platforms and what not.
I think that's just kicking the can down the road, and will result in
more variance and uncertainty. It might be alright for those authoring
PRs, but it will make Ben's job keeping the system running even more
grueling.
Moritz Angermann writes:
> At this point I believe we have ample Linux build capacity. Darwin looks
> pretty good as well the ~4 M1s we have should in principle also be able to
> build x86_64-darwin at acceptable speeds. Although on Big Sur only.
>
> The aarch64-Linux story is a bit constraint by
Apologies for the latency here. This thread has required a fair amount of
reflection.
Sebastian Graf writes:
> Hi Moritz,
>
> I, too, had my gripes with CI turnaround times in the past. Here's a
> somewhat radical proposal:
>
>- Run "full-build" stage builds only on Marge MRs. Then we can as
I'm glad to report that my math was off. But it was off only because I
assumed that we'd successfully build all
windows configurations, which we of course don't. Thus some builds fail
faster.
Sylvain also provided a windows machine temporarily, until it expired.
This led to a slew of new windows w
At this point I believe we have ample Linux build capacity. Darwin looks
pretty good as well the ~4 M1s we have should in principle also be able to
build x86_64-darwin at acceptable speeds. Although on Big Sur only.
The aarch64-Linux story is a bit constraint by powerful and fast CI
machines but p
Hi Moritz,
I, too, had my gripes with CI turnaround times in the past. Here's a
somewhat radical proposal:
- Run "full-build" stage builds only on Marge MRs. Then we can assign to
Marge much earlier, but probably have to do a bit more of (manual)
bisecting of spoiled Marge batches.
Friends,
I've been looking at CI recently again, as I was facing CI turnaround times
of 9-12hs; and this just keeps dragging out and making progress hard.
The pending pipeline currently has 2 darwin, and 15 windows builds waiting.
Windows builds on average take ~220minutes. We have five builders,
40 matches
Mail list logo
