Re: [Tails-dev] Automated builds specification
On Wed, Jun 17, 2015 at 10:30:28AM +0200, anonym wrote: > On 06/16/2015 02:41 PM, bertagaz wrote: > > On Fri, May 29, 2015 at 01:59:06PM +0200, intrigeri wrote: > >> anonym wrote (30 Mar 2015 07:48:28 GMT) : > >>> On 29/03/15 15:04, bertagaz wrote: > > >>> Wild (possibly unrelated) idea: instead of only notifying the author of > >>> the last commit of a topic branch, what about notifying all authors of > >>> the topic branch since it deviated from the base branch? > [...] > >>> Also, I guess we need to filter out authors that are not Tails "core" > >>> developers, so they do not get build failure notifications. This applies > >>> to both packages uploaded (when we upload a package built by a 3rd > >>> party), and Git (patches). Hmm. > >>sure. > >> Why? > > > > I think on the contrary it might be useful for people that are not core > > devs to get notifications on build failure. > > I'm not sure that contributors will appreciate these notifications. > > Any way, at least some "core" member *must* be notified too since they > have the power to actually fix it so... > > >>> This makes me think that we should perhaps look at Git committer > >>> name/email in Git instead of the author. > >> > >> Indeed, Git has separate committer and author "metadata fields" for > >> each commit. But I don't understand what exactly you're suggesting we > >> use them for -- may you please elaborate on this idea? > > > > I don't think it's that important. The only use case I see where it would > > change who gets the notification would be when one of us import a patch > > we received. Then, it is interesting still to use the author field, as it > > means that the notification would be sent to the one who actually wrote > > the patch, and not just to the one who merged it. Or maybe we want both of > > them to be notified? > > ... notifying both author and committer seems like a very nice idea. Ok, as it's quite a change to our current specifications, and is a longer discussion that might need to see live action going on to decide something, I've tracked this discussion in a separate ticket: https://labs.riseup.net/code/issues/9615 Thanks! bert. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
On 06/16/2015 02:41 PM, bertagaz wrote: > On Fri, May 29, 2015 at 01:59:06PM +0200, intrigeri wrote: >> anonym wrote (30 Mar 2015 07:48:28 GMT) : >>> On 29/03/15 15:04, bertagaz wrote: >>> Wild (possibly unrelated) idea: instead of only notifying the author of >>> the last commit of a topic branch, what about notifying all authors of >>> the topic branch since it deviated from the base branch? [...] >>> Also, I guess we need to filter out authors that are not Tails "core" >>> developers, so they do not get build failure notifications. This applies >>> to both packages uploaded (when we upload a package built by a 3rd >>> party), and Git (patches). Hmm. >>sure. >> Why? > > I think on the contrary it might be useful for people that are not core > devs to get notifications on build failure. I'm not sure that contributors will appreciate these notifications. Any way, at least some "core" member *must* be notified too since they have the power to actually fix it so... >>> This makes me think that we should perhaps look at Git committer >>> name/email in Git instead of the author. >> >> Indeed, Git has separate committer and author "metadata fields" for >> each commit. But I don't understand what exactly you're suggesting we >> use them for -- may you please elaborate on this idea? > > I don't think it's that important. The only use case I see where it would > change who gets the notification would be when one of us import a patch > we received. Then, it is interesting still to use the author field, as it > means that the notification would be sent to the one who actually wrote > the patch, and not just to the one who merged it. Or maybe we want both of > them to be notified? ... notifying both author and committer seems like a very nice idea. Cheers! ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi,
Just realized while searching for leftovers in this thread that this last
bit of the discussion was still not over.
On Fri, May 29, 2015 at 01:59:06PM +0200, intrigeri wrote:
>
> [This is about builds triggered by uploads to our APT repository.]
>
> anonym wrote (30 Mar 2015 07:48:28 GMT) :
> > On 29/03/15 15:04, bertagaz wrote:
> >> However, one point it raises (added to the blueprint) is determining who
> >> would be notified of this kind of build on failure.
>
> Given the way we've chosen to implement post-APT-upload builds for
> this first iteration, I have my doubts wrt. whether we can distinguish
> such builds from other ones. It may be that we simply can't, so
> perhaps it doesn't make much sense to discuss failure notification for
> this case in isolation from the general case... because we won't be
> able to implement a different behaviour (yet). Did I miss something?
Agree it doesn't make much sense.
> So, below I'll discuss the general case of build failure notification.
>
> > Wild (possibly unrelated) idea: instead of only notifying the author of
> > the last commit of a topic branch, what about notifying all authors of
> > the topic branch since it deviated from the base branch? E.g.
>
> > git log --pretty="format:%an <%ae>" ${BASE_BRANCH}.. | sort -u
>
> Interesting. I seem to remember having seen something like that
> available as an option in Jenkins' Git plugin. I suggest we start by
> notifying the author of the last commit, and keep this alternate idea
> in mind if that's not enough. Thoughts?
Added it to the blueprint as a future possible enhancement if we feel it's
needed.
> > Also, I guess we need to filter out authors that are not Tails "core"
> > developers, so they do not get build failure notifications. This applies
> > to both packages uploaded (when we upload a package built by a 3rd
> > party), and Git (patches). Hmm.
>
> Why?
I think on the contrary it might be useful for people that are not core
devs to get notifications on build failure.
> > This makes me think that we should perhaps look at Git committer
> > name/email in Git instead of the author.
>
> Indeed, Git has separate committer and author "metadata fields" for
> each commit. But I don't understand what exactly you're suggesting we
> use them for -- may you please elaborate on this idea?
I don't think it's that important. The only use case I see where it would
change who gets the notification would be when one of us import a patch
we received. Then, it is interesting still to use the author field, as it
means that the notification would be sent to the one who actually wrote
the patch, and not just to the one who merged it. Or maybe we want both of
them to be notified?
bert.
___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to
tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi,
[This is about builds triggered by uploads to our APT repository.]
anonym wrote (30 Mar 2015 07:48:28 GMT) :
> On 29/03/15 15:04, bertagaz wrote:
>> However, one point it raises (added to the blueprint) is determining who
>> would be notified of this kind of build on failure.
Given the way we've chosen to implement post-APT-upload builds for
this first iteration, I have my doubts wrt. whether we can distinguish
such builds from other ones. It may be that we simply can't, so
perhaps it doesn't make much sense to discuss failure notification for
this case in isolation from the general case... because we won't be
able to implement a different behaviour (yet). Did I miss something?
So, below I'll discuss the general case of build failure notification.
>> Multiple options are maybe to consider:
>>
>> * Notify every core devs that has upload access on our reprepro.
> I think notifying the last Git committer would be correct 90+% of the
> time, instead of spamming everyone 100% of the time (=> will make people
> ignore build failure notifications). [...]
Full ACK. That's what the blueprint says for the general case already.
> Wild (possibly unrelated) idea: instead of only notifying the author of
> the last commit of a topic branch, what about notifying all authors of
> the topic branch since it deviated from the base branch? E.g.
> git log --pretty="format:%an <%ae>" ${BASE_BRANCH}.. | sort -u
Interesting. I seem to remember having seen something like that
available as an option in Jenkins' Git plugin. I suggest we start by
notifying the author of the last commit, and keep this alternate idea
in mind if that's not enough. Thoughts?
>> * Change our packaging habits and put our emails in the changelog.
> It sounds good, but not without problems. Sometimes we upload packages
> we haven't built ourselves, e.g. I often upload I2P packages built by
> KillYourTV.
We're actually sending to reprepro the "who did the upload"
information, as part of the signature on the .changes file. However,
I think we're ditching this information as soon as the package is
imported. If we want to use this information in any way, we'll have to
keep it around somehow. Anyway, IMO that's long-term, and no idea if
it's worth bothering yet. Time will tell.
> Also, I guess we need to filter out authors that are not Tails "core"
> developers, so they do not get build failure notifications. This applies
> to both packages uploaded (when we upload a package built by a 3rd
> party), and Git (patches). Hmm.
Why?
> This makes me think that we should perhaps look at Git committer
> name/email in Git instead of the author.
Indeed, Git has separate committer and author "metadata fields" for
each commit. But I don't understand what exactly you're suggesting we
use them for -- may you please elaborate on this idea?
Cheers!
--
intrigeri
___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to
tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, On Mon, Apr 06, 2015 at 06:53:35PM +0200, intrigeri wrote: > bertagaz wrote (06 Apr 2015 16:44:30 GMT) : > > > Was there any other topics to discuss on this subject? I kinda remember > > this was the last review round, and we managed to answer to all the raised > > questions. > > No idea. I suggest quickly skimming over the thread again, looking for > anything that hasn't been captured in the blueprint. (We may have the > big picture in mind right now — wait, actually not apparently — but > who knows where we'll be in a year or so?) We should maybe call that landscapes. :) I already did a last skimming before anonym had his last crazy good ideas. I'll take another look at it quickly, maybe note the interesting msg-ids in the blueprint, but I was asking because to me we're quite done on this. bert. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
bertagaz wrote (06 Apr 2015 16:44:30 GMT) : > How did you know I was confused? Long running threads are a bit hard to > follow, and I tend to have more and more troubles getting the full > picture. :) > Removed it. :) > Was there any other topics to discuss on this subject? I kinda remember > this was the last review round, and we managed to answer to all the raised > questions. No idea. I suggest quickly skimming over the thread again, looking for anything that hasn't been captured in the blueprint. (We may have the big picture in mind right now — wait, actually not apparently — but who knows where we'll be in a year or so?) ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, On Mon, Mar 30, 2015 at 09:28:08PM +0200, intrigeri wrote: > > Scenario 15 looks good. > > I don't understand what scenario 14 is about, especially right after > the discussion we've had on this thread. How could base branches be > possibly affected by changes in topic branches? > > Did I miss anything, or were you a little bit confused? How did you know I was confused? Long running threads are a bit hard to follow, and I tend to have more and more troubles getting the full picture. :) Removed it. Was there any other topics to discuss on this subject? I kinda remember this was the last review round, and we managed to answer to all the raised questions. bert. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, bertagaz wrote (29 Mar 2015 13:04:11 GMT) : > On Wed, Mar 25, 2015 at 04:44:32PM +0100, intrigeri wrote: >> => bertagaz and Jurre, may you please capture this problem, and the >> long-term solution ideas we had, in the "Future ideas" section of >> the blueprint? > Done in scenarios 14 and 15, please review. Scenario 15 looks good. I don't understand what scenario 14 is about, especially right after the discussion we've had on this thread. How could base branches be possibly affected by changes in topic branches? * by a commit to the Git topic branch: that's not the case unless one then merges the topic branch into the base one, but then this adds new commits to the topic branch, and is already dealt with by other scenarios * by a package upload: as said earlier in this thread, one MUST NOT upload new packages to a topic branch's APT suite once it has been merged, so this should not happen (instead, I think we need "something" to let us remember we should simply forbid such dangerous actions with technical means) Did I miss anything, or were you a little bit confused? Cheers, -- intrigeri ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, anonym wrote (25 Mar 2015 16:30:26 GMT) : > On 25/03/15 16:44, intrigeri wrote: >> anonym wrote (24 Mar 2015 16:50:06 GMT) : >>> However, I think the reason I asked for this feature was to trigger >>> rebuilds when the feature branch's APT suite has been updated. From >>> reading the blueprint it only mentions "Git", so I assume the "watchdog" >>> won't look at the feature branch's APT suite state? >> [...] >> Long-term: it seems quite clear to me that any upload to an APT suite >> should trigger a rebuild of the *directly* affected base branch. > I assume you didn't intend "base" to be there, right? Right. > This should fix all issues I came up with Cool :) > Sorry for the noise. No problem, it forced me to clarify my vision, and thus gives more ample opportunity to others to disagree while it's not too late. (I'd hate to see the "OMG WTF?" effect in N months, when all this is ready to go into production. I'm still bitter from last time it happened, so better avoid it :) Cheers! -- intrigeri ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
On 29/03/15 15:04, bertagaz wrote:
> Hi,
>
> Sorry, lagging a bit one my emails.
>
> On Wed, Mar 25, 2015 at 04:44:32PM +0100, intrigeri wrote:
>> Hi,
>>
>> anonym wrote (24 Mar 2015 16:50:06 GMT) :
>>> However, I think the reason I asked for this feature was to trigger
>>> rebuilds when the feature branch's APT suite has been updated. From
>>> reading the blueprint it only mentions "Git", so I assume the "watchdog"
>>> won't look at the feature branch's APT suite state?
>
> That's right, good catch. I tried to add in the blueprint that we want
> builds triggered by uploads on APT suites.
>
> However, one point it raises (added to the blueprint) is determining who
> would be notified of this kind of build on failure.
>
> Multiple options are maybe to consider:
>
> * Notify every core devs that has upload access on our reprepro.
I think notifying the last Git committer would be correct 90+% of the
time, instead of spamming everyone 100% of the time (=> will make people
ignore build failure notifications). After all, most of the time only
one person works on any given branch, and with #8654 one will be forced
to modify Git in order to create (or at least activate) a branche's APT
suite, so there will be no "empty" Git topic branches that only modify
the APT suite any more.
Wild (possibly unrelated) idea: instead of only notifying the author of
the last commit of a topic branch, what about notifying all authors of
the topic branch since it deviated from the base branch? E.g.
git log --pretty="format:%an <%ae>" ${BASE_BRANCH}.. | sort -u
> * Change our packaging habits and put our emails in the changelog.
It sounds good, but not without problems. Sometimes we upload packages
we haven't built ourselves, e.g. I often upload I2P packages built by
KillYourTV. Since it's KillYourTV's email in the last changelog entry,
should he,the package builder, be notified, or I, the branch owner? If
only one person should be notified, I think that should be me, the
branch owner, but both would be better, perhaps. I think that'd go well
with the "notify all authors of topic branch" idea I proposed above.
Also, I guess we need to filter out authors that are not Tails "core"
developers, so they do not get build failure notifications. This applies
to both packages uploaded (when we upload a package built by a 3rd
party), and Git (patches). Hmm. This makes me think that we should
perhaps look at Git committer name/email in Git instead of the author.
Cheers!
___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to
tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, Sorry, lagging a bit one my emails. On Wed, Mar 25, 2015 at 04:44:32PM +0100, intrigeri wrote: > Hi, > > anonym wrote (24 Mar 2015 16:50:06 GMT) : > > However, I think the reason I asked for this feature was to trigger > > rebuilds when the feature branch's APT suite has been updated. From > > reading the blueprint it only mentions "Git", so I assume the "watchdog" > > won't look at the feature branch's APT suite state? That's right, good catch. I tried to add in the blueprint that we want builds triggered by uploads on APT suites. However, one point it raises (added to the blueprint) is determining who would be notified of this kind of build on failure. Multiple options are maybe to consider: * Notify every core devs that has upload access on our reprepro. * Change our packaging habits and put our emails in the changelog. > Short-term mitigation: > > 1. if anyone really want to trigger an immediate rebuild, they can do > it manually in the Jenkins interface (people who have upload > rights to our APT repo also have powerful-enough access to Jenkins > to trigger builds); > 2. the proposal says that all active branches are built daily, in > addition to post-Git-push => worst case, the branch will be > rebuilt within 24h; > 3. if in a hurry, or for whatever reason, you can still force > a rebuild by pushing a dummy commit (ugly, but it works). > > Long-term: it seems quite clear to me that any upload to an APT suite > should trigger a rebuild of the *directly* affected base branch. > > And also, more generally: at some point, whenever a base branch's > current build is marked as outdated and needing to be rebuilt, be it > because the base branch was updated in Git or via its APT suite, we'll > want to trigger rebuilds of indirectly affected active branches as > well (e.g. topic branches based on that base branch) somehow. > Note that our resource estimates (and thus, our last round of hardware > shopping) didn't take this cascade of triggered builds, so we simply > can't implement that at the moment. > > BTW, I've read a bit about Zuul > (http://ci.openstack.org/zuul/zuul.html) recently, and this made me > aware of quite a few issues similar to the one you're raising here. > Lots of food for thought, forwarded to bertagaz already. > > Now, let's put things back into perspective: what bertagaz and Jurre > are working on so far is expending what we currently have to all > active branches. Of course it doesn't cover everything that should > ideally be done yet, simply because what we have so far itself > doesn't. That's merely yet another step towards implementing the ideal > CI system we need :) > > => bertagaz and Jurre, may you please capture this problem, and the > long-term solution ideas we had, in the "Future ideas" section of > the blueprint? Done in scenarios 14 and 15, please review. bert. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
On 25/03/15 16:44, intrigeri wrote: > Hi, > > anonym wrote (24 Mar 2015 16:50:06 GMT) : >> However, I think the reason I asked for this feature was to trigger >> rebuilds when the feature branch's APT suite has been updated. From >> reading the blueprint it only mentions "Git", so I assume the "watchdog" >> won't look at the feature branch's APT suite state? > > Short-term mitigation: > > 1. if anyone really want to trigger an immediate rebuild, they can do > it manually in the Jenkins interface (people who have upload > rights to our APT repo also have powerful-enough access to Jenkins > to trigger builds); Of course! Can't believe I forgot about this! > [...] > Long-term: it seems quite clear to me that any upload to an APT suite > should trigger a rebuild of the *directly* affected base branch. I assume you didn't intend "base" to be there, right? I mean, whatever branch we're dealing with, feature or base, it should be rebuilt when its APT suite is modified. If that's what you meant, great! > And also, more generally: at some point, whenever a base branch's > current build is marked as outdated and needing to be rebuilt, be it > because the base branch was updated in Git or via its APT suite, we'll > want to trigger rebuilds of indirectly affected active branches as > well (e.g. topic branches based on that base branch) somehow. > Note that our resource estimates (and thus, our last round of hardware > shopping) didn't take this cascade of triggered builds, so we simply > can't implement that at the moment. Cool! [...] > On a more general note, all this discussion and thinking convinces me > more and more that we should really capture in Git everything that > matters for a given branch/build, including the state of the custom > source packages it needs. Lots of technical details (Git submodules, > when/where packages shall be built, etc.) clearly need more thinking, > but the general concept, once implemented, will make most of this > discussion moot. Agreed. >> Another thing that *just* struck me (and has a lot of consequences... >> sorry for coming with this at this late stage) is that updates to the >> base branch's APT suite will affect all feature branches based on it, so >> a rebuild for all of them would make sense immediately after such an >> update. > > I agree in principle, but we can't do the "immediately" part yet, as > explained above in more general terms (since the same is true for Git > changes in a base branch). Right. Take my "immediately":s with a grain of salt. >> OTOH, they may lack important Git state (since the base branch >> wasn't merged back) that otherwise will make the builds break. Luckily, >> if we have #8654 and never modify any base branch's APT suite directly >> but only add APT suites to the base branche's config file *in* *Git*, we >> should be fine since a Git merge into the feature branch is needed to >> make it aware of the APT changes. > > (In what follows, I'm assuming we will have #8654 soonish, since > otherwise the rest of the extended autobuild system wouldn't be > very useful.) For the record, everything I wrote was with the assumption that we have #8654. > I don't get what's the problem you're raising here. My understanding > is that there are few potential problems: > > a) Changes made directly in the base branch' APT suite are not taken > into account by a rebuild of a topic branch. > b) Changes made directly in the base branch' APT suite don't trigger > a rebuild of the affected topic branches. > c) Changes made in the base branch' APT suites config file are not > taken into account by a rebuild of a topic branch. > d) Changes made in the base branch' APT suites config file don't > trigger a rebuild of the affected topic branches. > > Triggering rebuilds (b and d) has been discussed above, and in short: > we can't do that yet, too bad. > > I believe that (a) and (c) are non-issues once we have #8654. Urgh. I had forgotten this part of the build specification: A topic branch may be lagging behind the base branch it's based upon. What we're interested in is generally not whether a (possibly outdated) topic branch builds fine, but whether it would build fine once merged into its base branch: This should fix all issues I came up with (except that already-merged branches should be frozen and not reused). Sorry for the noise. I had already written some answers below before realizing this, so I leave them, but you can just ignore them. >> There's a similar case that also needs some consideration, and changes >> in how we work: imagine we have a feature branch F based on base branch >> B. We upload a package to F's APT suite, all looks good at the moment, >> and we merge F into B. Then a feature branch E has B merged into. All >> good so far. However, at this point we discover an issue with F and have >> to rebuild the package, or otherwise change F's APT suite. Then we >> cannot do anythin
Re: [Tails-dev] Automated builds specification
Hi, anonym wrote (24 Mar 2015 16:50:06 GMT) : > However, I think the reason I asked for this feature was to trigger > rebuilds when the feature branch's APT suite has been updated. From > reading the blueprint it only mentions "Git", so I assume the "watchdog" > won't look at the feature branch's APT suite state? Short-term mitigation: 1. if anyone really want to trigger an immediate rebuild, they can do it manually in the Jenkins interface (people who have upload rights to our APT repo also have powerful-enough access to Jenkins to trigger builds); 2. the proposal says that all active branches are built daily, in addition to post-Git-push => worst case, the branch will be rebuilt within 24h; 3. if in a hurry, or for whatever reason, you can still force a rebuild by pushing a dummy commit (ugly, but it works). Long-term: it seems quite clear to me that any upload to an APT suite should trigger a rebuild of the *directly* affected base branch. And also, more generally: at some point, whenever a base branch's current build is marked as outdated and needing to be rebuilt, be it because the base branch was updated in Git or via its APT suite, we'll want to trigger rebuilds of indirectly affected active branches as well (e.g. topic branches based on that base branch) somehow. Note that our resource estimates (and thus, our last round of hardware shopping) didn't take this cascade of triggered builds, so we simply can't implement that at the moment. BTW, I've read a bit about Zuul (http://ci.openstack.org/zuul/zuul.html) recently, and this made me aware of quite a few issues similar to the one you're raising here. Lots of food for thought, forwarded to bertagaz already. Now, let's put things back into perspective: what bertagaz and Jurre are working on so far is expending what we currently have to all active branches. Of course it doesn't cover everything that should ideally be done yet, simply because what we have so far itself doesn't. That's merely yet another step towards implementing the ideal CI system we need :) => bertagaz and Jurre, may you please capture this problem, and the long-term solution ideas we had, in the "Future ideas" section of the blueprint? On a more general note, all this discussion and thinking convinces me more and more that we should really capture in Git everything that matters for a given branch/build, including the state of the custom source packages it needs. Lots of technical details (Git submodules, when/where packages shall be built, etc.) clearly need more thinking, but the general concept, once implemented, will make most of this discussion moot. > Another thing that *just* struck me (and has a lot of consequences... > sorry for coming with this at this late stage) is that updates to the > base branch's APT suite will affect all feature branches based on it, so > a rebuild for all of them would make sense immediately after such an > update. I agree in principle, but we can't do the "immediately" part yet, as explained above in more general terms (since the same is true for Git changes in a base branch). > OTOH, they may lack important Git state (since the base branch > wasn't merged back) that otherwise will make the builds break. Luckily, > if we have #8654 and never modify any base branch's APT suite directly > but only add APT suites to the base branche's config file *in* *Git*, we > should be fine since a Git merge into the feature branch is needed to > make it aware of the APT changes. (In what follows, I'm assuming we will have #8654 soonish, since otherwise the rest of the extended autobuild system wouldn't be very useful.) I don't get what's the problem you're raising here. My understanding is that there are few potential problems: a) Changes made directly in the base branch' APT suite are not taken into account by a rebuild of a topic branch. b) Changes made directly in the base branch' APT suite don't trigger a rebuild of the affected topic branches. c) Changes made in the base branch' APT suites config file are not taken into account by a rebuild of a topic branch. d) Changes made in the base branch' APT suites config file don't trigger a rebuild of the affected topic branches. Triggering rebuilds (b and d) has been discussed above, and in short: we can't do that yet, too bad. I believe that (a) and (c) are non-issues once we have #8654. So, what's the remaining problem we should solve *now*? > There's a similar case that also needs some consideration, and changes > in how we work: imagine we have a feature branch F based on base branch > B. We upload a package to F's APT suite, all looks good at the moment, > and we merge F into B. Then a feature branch E has B merged into. All > good so far. However, at this point we discover an issue with F and have > to rebuild the package, or otherwise change F's APT suite. Then we > cannot do anything with F's APT suite, because that would affect E > without i
Re: [Tails-dev] Automated builds specification
On 01/03/15 12:49, intrigeri wrote: > hi, > > bertagaz wrote (27 Feb 2015 10:07:03 GMT) : >> On Thu, Feb 26, 2015 at 08:46:17PM +0100, bertagaz wrote: >>> On Thu, Feb 26, 2015 at 05:21:23PM +0100, anonym wrote: On 11/01/15 20:07, Jurre van Bergen wrote: > Developers should be able to trigger automatic builds for a branch > whose build was dropped (eg. last commit too old) by pushing a dumb > commit on a timestamp file in that branch. Can't we make builds trigger via signed email too? Polluting the Git repo like that seems ugly to me. > [...] >> While discussing last night about this interesting point you raised, we >> thought of an easy and obvious fix to that history pollution. > >> In fact, that's even already implemented in the way we often work: after a >> release, people working on a topic branch often merge stable or testing >> back in it. So there *will* be new commits on their branches. > >> Does that sound better to you? > > At least it does sound better to me: the problem at hand is detecting > the transition of a branch between inactive and active state. It sounds better to me too. However, I think the reason I asked for this feature was to trigger rebuilds when the feature branch's APT suite has been updated. From reading the blueprint it only mentions "Git", so I assume the "watchdog" won't look at the feature branch's APT suite state? I think I mentioned something about this during the sprint. Another thing that *just* struck me (and has a lot of consequences... sorry for coming with this at this late stage) is that updates to the base branch's APT suite will affect all feature branches based on it, so a rebuild for all of them would make sense immediately after such an update. OTOH, they may lack important Git state (since the base branch wasn't merged back) that otherwise will make the builds break. Luckily, if we have #8654 and never modify any base branch's APT suite directly but only add APT suites to the base branche's config file *in* *Git*, we should be fine since a Git merge into the feature branch is needed to make it aware of the APT changes. There's a similar case that also needs some consideration, and changes in how we work: imagine we have a feature branch F based on base branch B. We upload a package to F's APT suite, all looks good at the moment, and we merge F into B. Then a feature branch E has B merged into. All good so far. However, at this point we discover an issue with F and have to rebuild the package, or otherwise change F's APT suite. Then we cannot do anything with F's APT suite, because that would affect E without it merging B again. Hence, the fixup for F has to be made in a new feature branch, say F-fixup. When F-fixup is merged into B, E won't be affected (and need a rebuild) until B is merged into E. To conclude, we should update our merge policy (or whatever) with the following rules: * Never upload packages directly to any base branch, not even during during release time. Never! * Once a feature branch has been merged into a base branch, the feature branch's APT suite should be frozen forever. Any fixups that affect the feature branch's APT suite *must* be done in the APT suite of a *new* feature branch. I also think we have some problems around release time when we merge all APT suites of the feature branches that *have* been Git-merged into the base branch being released (I'm gonna refer to this procedure simply as "squashing" below). Specifically, all feature branches based on the to-be-released base branch that has *not* been merged will be affected by this. Example: 1. Let F1 and F2 be divergent feature branches based on base branch B. 2. F1 is merged into B. 3. B is released, so F1's APT suite is merged into B's APT suite, i.e. the "squashing". 4. Note that at no point was F1 or B merged into F2, but right now it's like the APT suite of F1 is merged into F2. Also note that F1's Git is not merged into F2. Now we have two issues: * F2 has (indirectly, via the base branch's APT suite) been modified without its Git state being changed at all. * However, F2 doesn't have the Git changes made in F1, and it could very well be that the APT stuff introduced by F1's APT suite depend on those Git changes in a way so that builds will break without them. The only way I see to resolve all this reliably is that: after we "squash", then all branches based on B that have *not* been merged yet must immediately have B merged into them, and all branches that *have* been merged are killed (or at least marked in a way so they are not rebuilt). This is a new step in our release process. However, it's quite unfortunate to add this procedure at that stage, which already is a bit stressful, so perhaps the "squashing" can be done in the APT suite T that's created for the release tag, and then we reset the B to T's state and merge B into the remaining feature branches based on B at post-release time, when things are a bit calmer. T
Re: [Tails-dev] Automated builds specification
Hi, On Sun, Mar 01, 2015 at 12:49:35PM +0100, intrigeri wrote: > bertagaz wrote (27 Feb 2015 10:07:03 GMT) : > > Does that sound better to you? > > At least it does sound better to me: the problem at hand is detecting > the transition of a branch between inactive and active state. > > When coming back to a branch after not having worked on it since 6+ > weeks, the first thing I do is indeed to make the branch up-to-date > wrt. changes that happened in the last 6+ weeks, then making sure it > still builds locally, and after that I would push the branch and it > will become active again (from Jenkins' perspective), as > bertagaz explained. > > bertagaz, DrWhax: please explain this on the blueprint -- you now > have two proposed wordings already so it should be easy :) Done in e5d21e8. bert. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
hi, bertagaz wrote (27 Feb 2015 10:07:03 GMT) : > On Thu, Feb 26, 2015 at 08:46:17PM +0100, bertagaz wrote: >> On Thu, Feb 26, 2015 at 05:21:23PM +0100, anonym wrote: >> > On 11/01/15 20:07, Jurre van Bergen wrote: >> > > Developers should be able to trigger automatic builds for a branch >> > > whose build was dropped (eg. last commit too old) by pushing a dumb >> > > commit on a timestamp file in that branch. >> > >> > Can't we make builds trigger via signed email too? Polluting the Git >> > repo like that seems ugly to me. [...] > While discussing last night about this interesting point you raised, we > thought of an easy and obvious fix to that history pollution. > In fact, that's even already implemented in the way we often work: after a > release, people working on a topic branch often merge stable or testing > back in it. So there *will* be new commits on their branches. > Does that sound better to you? At least it does sound better to me: the problem at hand is detecting the transition of a branch between inactive and active state. When coming back to a branch after not having worked on it since 6+ weeks, the first thing I do is indeed to make the branch up-to-date wrt. changes that happened in the last 6+ weeks, then making sure it still builds locally, and after that I would push the branch and it will become active again (from Jenkins' perspective), as bertagaz explained. bertagaz, DrWhax: please explain this on the blueprint -- you now have two proposed wordings already so it should be easy :) Cheers, -- intrigeri ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, On Thu, Feb 26, 2015 at 08:46:17PM +0100, bertagaz wrote: > On Thu, Feb 26, 2015 at 05:21:23PM +0100, anonym wrote: > > On 11/01/15 20:07, Jurre van Bergen wrote: > > > Developers should be able to trigger automatic builds for a branch > > > whose build was dropped (eg. last commit too old) by pushing a dumb > > > commit on a timestamp file in that branch. > > > > Can't we make builds trigger via signed email too? Polluting the Git > > repo like that seems ugly to me. > > It surely isn't the best for the git history or commit point of view, but > OTOH it express in git's semantic the fact that a branch is active for > more people than Jenkins and the sender of the email. So for example the > release manager can also easily get that info and add the branch on his > radar. > > Signed email could be an optional feature, like nice to have, but maybe > not a blocker to deploy the automated builds. It will surely be a bit more > tricky and might delay things a bit. While discussing last night about this interesting point you raised, we thought of an easy and obvious fix to that history pollution. In fact, that's even already implemented in the way we often work: after a release, people working on a topic branch often merge stable or testing back in it. So there *will* be new commits on their branches. Does that sound better to you? bert. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
On Thu, Feb 26, 2015 at 05:21:23PM +0100, anonym wrote: > On 11/01/15 20:07, Jurre van Bergen wrote: > > https://tails.boum.org/blueprint/automated_builds_and_tests/autobuild_specs/ > > Sorry for joining in so late. All looks good, and I only have minor > concerns. > > > Developers should be able to trigger automatic builds for a branch > > whose build was dropped (eg. last commit too old) by pushing a dumb > > commit on a timestamp file in that branch. > > Can't we make builds trigger via signed email too? Polluting the Git > repo like that seems ugly to me. It surely isn't the best for the git history or commit point of view, but OTOH it express in git's semantic the fact that a branch is active for more people than Jenkins and the sender of the email. So for example the release manager can also easily get that info and add the branch on his radar. Signed email could be an optional feature, like nice to have, but maybe not a blocker to deploy the automated builds. It will surely be a bit more tricky and might delay things a bit. > > How to build it > > [...] > > when building topic branch F, we need to build from branch F once > > merged into branch B. However, this merge must only be done locally, > > at least because Jenkins doesn't have push access to our Git repo. > > [...] > > This locally-merge-before-building process requires ticket #8654 to > > be implemented [...] > > The thing with #8654 is that it's gonna introduce a lot of merge > conflicts. They will be simple for a human to resolve, so I'm not > worried about that, but since jenkins will bail out if there's any > conflict in the ocally-merge-before-building process, this will be annoying. > > Here's an example of how the config/APT_suites may evolve for devel, > feature/x and feature/y, which both have APT suite: > > [...] > > In other words, for all branches based on B that modifies > config/APT_suites, whenever one of them is merged, it will reduce merge > conflicts for all other branches based on B. This will get problematic, > so we probably will need a custom git merge conflict handler. It could > probably be quite simple, since most of the time branches will just be > added (not removed, renamed) so they can be automatically resolved by > removing the merge conflict tags. It seems that this kind of easy merges could be resolved by configuring .gitattributes to use the union merge for config/APT_suites. https://code.google.com/p/endgame-singularity/source/browse/.gitattributes I'm not sure how it handles corner cases though, it might deserve some tests to see how it fits. Otherwise intrigeri was mentionning another possibility in Git. Thanks for the meaningful feedbacks! bert. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
On 11/01/15 20:07, Jurre van Bergen wrote: > https://tails.boum.org/blueprint/automated_builds_and_tests/autobuild_specs/ Sorry for joining in so late. All looks good, and I only have minor concerns. > Developers should be able to trigger automatic builds for a branch > whose build was dropped (eg. last commit too old) by pushing a dumb > commit on a timestamp file in that branch. Can't we make builds trigger via signed email too? Polluting the Git repo like that seems ugly to me. > How to build it > [...] > when building topic branch F, we need to build from branch F once > merged into branch B. However, this merge must only be done locally, > at least because Jenkins doesn't have push access to our Git repo. > [...] > This locally-merge-before-building process requires ticket #8654 to > be implemented [...] The thing with #8654 is that it's gonna introduce a lot of merge conflicts. They will be simple for a human to resolve, so I'm not worried about that, but since jenkins will bail out if there's any conflict in the ocally-merge-before-building process, this will be annoying. Here's an example of how the config/APT_suites may evolve for devel, feature/x and feature/y, which both have APT suite: devel: -- devel feature/x: -- devel feature-x feature/y: -- devel feature-x In this state, the locally-merge-before-building process will work fine for all branches. However, let's say feature/x is merged (for real, not locally!) into devel: devel: -- devel feature-x When jenkins want to build feature/y, it will locally merge feature/y into devel, and then we'll have a merge conflict in config/APT_suites: devel devel feature-x feature-y feature/y The resolution is simple: devel feature-x feature-y In other words, for all branches based on B that modifies config/APT_suites, whenever one of them is merged, it will reduce merge conflicts for all other branches based on B. This will get problematic, so we probably will need a custom git merge conflict handler. It could probably be quite simple, since most of the time branches will just be added (not removed, renamed) so they can be automatically resolved by removing the merge conflict tags. Cheers! ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
bertagaz wrote (24 Feb 2015 16:04:14 GMT) : > Yesterday, 40 ISOs were build on Jenkins... I bet that the majority of those were triggered mistakenly, due to https://labs.riseup.net/code/issues/8692. I recommend working on this during the 1.3.1 cycle if possible, so that we have more useful data to reason about. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
bertagaz wrote (24 Feb 2015 14:19:02 GMT) : > On Sat, Feb 21, 2015 at 05:26:09PM +0100, intrigeri wrote: >> bertagaz: I think it can now be clarified accordingly on the blueprint :) > Done. Thanks. I've clarified on the blueprint the fact that we're talking about a *local* merge, since that's precisely where the misunderstanding I had to clarify apparently was. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, bertagaz wrote (24 Feb 2015 14:12:14 GMT) : > On Fri, Feb 20, 2015 at 11:56:02AM +0100, intrigeri wrote: >> It currently reads: >> >> For base branches: >> >> * When failing after a git push, notify the last commit author. >> * When failing after the daily trigger, notify the RM. >> >> I say let's just always the RM to start with. > I'm not sure about that still: if we proceed this way, we're not > implementing the reviewer scenario, because the RM will be notified when a > branch is merged and breaks the build, and not the one who did this merge. It seems there's a serious misunderstanding here. The way I understand the reviewer scenario has no specific provision regarding whatever should happen *after* branch F was *really* merged into branch B by the reviewer. Reviewing happens *before* merging, and part of the review work is ensuring that the resulting branch B (after branch F is merged into it) will still build fine. Of course reviewers must not do this part of their job *after* merging and pushing to the official repo, so this is only meaningful if done *before*. IMO the reviewer scenario we have is meant to address this specific problem, to make the reviewers' work easier (hence the "I might want to download the resulting ISO" part for example, which only makes sense before merging for real), thanks to Jenkins and to your work. The way I understand it (and the way I think it should be), in: As a reviewer When I'm asked to review branch F into branch B Then I need to know if branch F builds fine once merged into branch B (fresh results!) And I should be notified of the results And if the build succeeded I might want to download the resulting ISO I might want to get the pkg list I want the Redmine ticket to be notified (optional) Otherwise if it fails the developer who proposed the merge should be notified And the developer *needs* to see the build logs And the ticket should be reassigned to the branch submitter And QA check should be set to "Dev needed" ... "once merged into branch B" is a local merge, done in Jenkins only, and *not* pushed to the official repo. Otherwise the remaining steps don't make any sense to me: if the build fails, we don't want to say "oh too bad, we have merged branch F already, so our build is broken, but let's mark the ticket as dev needed and reassign to the developer". Instead, we want to simply *not* merge this branch yet, and then mark the ticket as dev needed and reassign to the developer. And, if the reviewer mistakenly merges something that breaks the build, then of course I would consider it's their responsibility for fixing it, ideally... but we can't count on reviewers to be around, notice the failure and fix it in a timely manner: as you know, reviewers come and go, and sometimes disappear for a week or more. The only person we have on formal duty for such things, who's taking care of the state of our base branches on which the next release(s) will be based, is the RM. So IMO the only reliable solution is to notify the RM whenever this situation happens (which should be pretty rare anyway :) Thoughts? Cheers, -- intrigeri ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
On Wed, Feb 18, 2015 at 09:51:48AM +, sajolida wrote: > > The Global build stats Jenkins plugin [1] seems interesting to display the > > stats once more logs are kept. Shall I install it? > > I would love to have stats about the number of branches and ISO built > and tested each day to put in our reports to our funders. Done, one that has access to our jenkins instance can go read some stats there, and create new charts. I've also updated the stat page in the blueprint section [1] Yesterday, 40 ISOs were build on Jenkins... bert. [1] https://tails.boum.org/blueprint/automated_builds_and_tests/autobuild_stats/ ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
On Sat, Feb 21, 2015 at 05:26:09PM +0100, intrigeri wrote: > bertagaz: I think it can now be clarified accordingly on the blueprint :) Done. bert. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, On Fri, Feb 20, 2015 at 11:56:02AM +0100, intrigeri wrote: > bertagaz wrote (19 Feb 2015 18:15:35 GMT) : > > On Tue, Feb 17, 2015 at 10:32:59PM +0100, intrigeri wrote: > > > I like this idea, simple and effective. :) > > > So for the base branches, the RM would be the contact point for daily and > > on push build failure notifications. And for topic branches, that would be > > the last commiter. > > Agreed. It seems that the blueprint doesn't reflect that, though. Yeah, I think I was confused because we had different conlusions about this in the two different sub-threads of this discussion that talked about it. > It currently reads: > > For base branches: > > * When failing after a git push, notify the last commit author. > * When failing after the daily trigger, notify the RM. > > I say let's just always the RM to start with. I'm not sure about that still: if we proceed this way, we're not implementing the reviewer scenario, because the RM will be notified when a branch is merged and breaks the build, and not the one who did this merge. So, shall we ignore this, and assume that having the reviewer building locally on her hardware is enough, meaning the RM is the contact point for all build failures on the base branches? bert. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Alan wrote (21 Feb 2015 13:36:41 GMT) : >> Most of us (including me) do the same as Alan, but IMO that's almost >> irrelevant to the topic at hand. This same scenario reads: >> >>And I need to know if my branch build is broken by something else >> possibly weeks after my last commit (by e.g Debian changes, >> changes in branch B, ...) >> ^^^ >> >> ... and we cannot possibly get this without locally merging the topic >> branch F into B before building. >> >> The important point here may be the *locally* word. This merge would >> only be done in Jenkins own temporary Git checkout, and wouldn't >> affect how we're handling our branches' Git history. >> >> > But I agree this is not the best way to go, so if Alan doesn't come up with >> > a block on this, I agree to add the clarification. >> >> OK. But apparently, either I misunderstood why Alan had trouble with >> this idea, or Alan has misunderstood the idea, so IMO it would be good >> to have his opinion now that I've clarified what I think we should do, >> and why. Alan? >> > I do not see any issue with this local merge by our autobuilder. Looks > to me like the right thing to do. Thanks for the clarification! bertagaz: I think it can now be clarified accordingly on the blueprint :) Cheers, -- intrigeri ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, intrigeri wrote: > bertagaz wrote (16 Feb 2015 14:32:57 GMT) : > > On Wed, Feb 11, 2015 at 11:49:02PM +0100, intrigeri wrote: > >> I have a few concerns, though: > >> > >> * "Scenario 2 : developer" doesn't make it clear if branch T is > >> build *after being locally merged into branch B*, or not. > >> Given that's what we're really interested in, and given > >> "Scenario 1 : reviewer" is clearer (answer is: yes), I guess the > >> answer is yes here too, but this should be clarified. > > > IIRC that was something Alan had troubles with, as not being the way she > > usually work on a feature branch, which I think was more close to "Work > > work work on the branch, and then when the feature is ready, merge the base > > branch in it." So she usually do not merge the base branch very often. > > Most of us (including me) do the same as Alan, but IMO that's almost > irrelevant to the topic at hand. This same scenario reads: > >And I need to know if my branch build is broken by something else > possibly weeks after my last commit (by e.g Debian changes, > changes in branch B, ...) > ^^^ > > ... and we cannot possibly get this without locally merging the topic > branch F into B before building. > > The important point here may be the *locally* word. This merge would > only be done in Jenkins own temporary Git checkout, and wouldn't > affect how we're handling our branches' Git history. > > > But I agree this is not the best way to go, so if Alan doesn't come up with > > a block on this, I agree to add the clarification. > > OK. But apparently, either I misunderstood why Alan had trouble with > this idea, or Alan has misunderstood the idea, so IMO it would be good > to have his opinion now that I've clarified what I think we should do, > and why. Alan? > I do not see any issue with this local merge by our autobuilder. Looks to me like the right thing to do. Cheers ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
bertagaz wrote (19 Feb 2015 22:52:20 GMT) : >> If you prefer, in the future I can dump such research results into the >> blueprint instead of here. Just tell me where it should go. > The blueprint/automated_builds_and_tests/jenkins page seems perfect for > that. Right => done. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, bertagaz wrote (19 Feb 2015 18:15:35 GMT) : > On Tue, Feb 17, 2015 at 10:32:59PM +0100, intrigeri wrote: >> Now, I wonder if we could solve this quite simply by having the RM be >> notified for *base* branch build failure only. The RM doesn't care >> about topic branches that haven't been submitted for merging >> yet, right? > I like this idea, simple and effective. :) > So for the base branches, the RM would be the contact point for daily and > on push build failure notifications. And for topic branches, that would be > the last commiter. Agreed. It seems that the blueprint doesn't reflect that, though. It currently reads: For base branches: * When failing after a git push, notify the last commit author. * When failing after the daily trigger, notify the RM. I say let's just always the RM to start with. Cheers, -- intrigeri ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, bertagaz wrote (19 Feb 2015 21:39:25 GMT) : > On Mon, Feb 16, 2015 at 04:39:22PM +0100, intrigeri wrote: >> bertagaz wrote (16 Feb 2015 12:03:12 GMT) : >> > Ack, sounds reasonable. However from what I've seen, it sometimes means a >> > lot of branches so I wonder if we scaled our infra enough for that, as we >> > didn't include this branches in our maths since the beginning of the >> > discussion. >> >> This seems like a serious bug in this discussion process. May you >> please then provide example numbers that match what the proposed >> algorithm would output, so that we can reason about it? > I've added to the statistic page the doc branches that have been merged. > Part of them were not in the output of the script because they often get > merged in master. It seems to add 4 to 6 branches per release, but their > development and integration flaw is a bit different. I've modified my count-merges script locally to take into account merges into master too, but it didn't catch any more merges. No surprise, given e.g.: git log 1.1..1.3-rc1 --merges --pretty=oneline | grep master | grep doc ... returns no such merge. Possibly the doc branches are merged without --no-ff into master? sajolida, any idea? > So in the end, I'm not sure they'll add a lot more load, but we have to > count them in our maths. Yep. Too bad we can't really do that with good stats. Cheers, -- intrigeri ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
On Tue, Feb 17, 2015 at 09:47:51PM +0100, intrigeri wrote: > bertagaz wrote (16 Feb 2015 14:37:56 GMT) : > > On Thu, Feb 12, 2015 at 12:22:45AM +0100, intrigeri wrote: > >> Here's one more: the proposed notification mechanism makes sense to me > >> for topic branches. But for "base" branches, it's more complicated: > > >> Thoughts? > > > Agree with that as stated in my previous email. > > OK, good. Then I'll let you capture in the specification that problem, > and our currently preferred solution. Done. > What follows is only interesting for Jenkins folks. I guess almost > everyone but DrWhax and bertagaz can stop here. > > > I think it is doable to differenciate them, by splitting the daily > > and on git push job definiton maybe. > > => I think this trick would make the overall thing hard to understand > and draw conclusions from, for anyone not deeply involved in our > Jenkins stuff. That's my opinion too. > > Having a look at plugins might help, as well as how other projects > > do that (as you stated elsewhere). > > Yep. At least the obvious candidate (Email-ext plugin) doesn't seem > able to email different recipients depending on what triggered the > build out-of-the-box. But apparently, one can set up two 'Script - > After Build' email triggers in the Email-ext configuration: one emails > the culprit, the other emails the RM. And then, they do something or > not depending on a variable we set during the build, based on what > triggered the build. Likely the cleaner and simpler solution. > > Otherwise, we could have Jenkins email some pipe script that will > forward to the right person depending on 1. whether it's a base > branch; and 2. whether the build was triggered by a push or by > something else. This should work if we can get the email notification > to pass the needed info in it. E.g. the full console output currently > has "Started by timer" or "Started by an SCM change", but this is not > part of the email notification. Could work, but a bit hackish and all > kinds of things can go wrong. > > Also, I've seen lots of people documenting crazy similar things with > some of these plugins: "Run Condition", "Conditional BuildStep", > "Flexible Publish" and "Any Build step". But then it gets too > complicated for me to dive into it right now. Yes, feel also quite lost in all this big Jenkins ecosystem sometimes. But it's certainly doable. I'll spend time digging into this too. > If you prefer, in the future I can dump such research results into the > blueprint instead of here. Just tell me where it should go. The blueprint/automated_builds_and_tests/jenkins page seems perfect for that. bert. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, On Mon, Feb 16, 2015 at 04:39:22PM +0100, intrigeri wrote: > bertagaz wrote (16 Feb 2015 12:03:12 GMT) : > > Ack, sounds reasonable. However from what I've seen, it sometimes means a > > lot of branches so I wonder if we scaled our infra enough for that, as we > > didn't include this branches in our maths since the beginning of the > > discussion. > > This seems like a serious bug in this discussion process. May you > please then provide example numbers that match what the proposed > algorithm would output, so that we can reason about it? I've added to the statistic page the doc branches that have been merged. Part of them were not in the output of the script because they often get merged in master. It seems to add 4 to 6 branches per release, but their development and integration flaw is a bit different. So in the end, I'm not sure they'll add a lot more load, but we have to count them in our maths. bert. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, On Tue, Feb 17, 2015 at 10:32:59PM +0100, intrigeri wrote: > bertagaz wrote (16 Feb 2015 14:32:57 GMT) : > > On Wed, Feb 11, 2015 at 11:49:02PM +0100, intrigeri wrote: > > > That's a logical awesome idea I'm ashamed not to have had sooner. > > Still, it seems that it comes too late, after some searching it appears > > that our Jenkins don't keep much stats. > > As discussed on IRC, we have about 10 days of logs on the filesystem > currently, so you can still retrieve recent stats about it. > The earlier, the better, as we're frozen and the more we wait, the > less the numbers will be meaningful. I've just pushed a new page in the blueprint section, that contains the number of branches merged per release (as your script told us), and the number of builds per day that happened since 2015-02-10. https://tails.boum.org/blueprint/automated_builds_and_tests/autobuild_stats/ > > The Global build stats Jenkins plugin [1] seems interesting to display the > > stats once more logs are kept. Shall I install it? > > Let's give it a try, yes. I'll do that soon then. > >> * "Scenario 3 : RM" says "I need to know when a branch FTBFS", but > >> I see no notification mechanism planned, so... how is the RM > >> supposed to know. Also, whenever stalled topic branches start > >> failing, this can imply an avalanche of daily email to the RM, who > >> will of course start ignoring all email from Jenkins and then > >> we've lost. I'm particularly worried about this topic since anonym > >> (our RM most of the time) didn't comment on this proposal yet, and > >> has already expressed concerns about this kind of issues. > > > I think anonym did comment on this proposal, he did a review of this > > blueprint already. > > OK. Let's keep in mind that he may not have thought of this potential > problem yet. > > Now, I wonder if we could solve this quite simply by having the RM be > notified for *base* branch build failure only. The RM doesn't care > about topic branches that haven't been submitted for merging > yet, right? I like this idea, simple and effective. :) So for the base branches, the RM would be the contact point for daily and on push build failure notifications. And for topic branches, that would be the last commiter. > > One thing that this question also raise is the fact that the RM is not > > always the same person, so I'm wondering how to have jenkins notifying the > > right email depending on who is the RM for the next release. First thing > > that come in top of my mind is... a schleuder address. :) > > Indeed, that's the kind of aliases the RMs can trivially update > themselves, without asking the infra team to do anything. Great, I'll add that to the specs and will open the corresponding tickets. bert. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
bertagaz: >> * I see no statistics about how many ISOs we are *currently* >> building each day. So it's not clear to me if our current setup >> can support building N more branches, once a day each. It would be >> useful to have this number (e.g. raw number per day during the 1.3 >> release cycle, and daily average). Of course we can fix that later >> (either by throwing more hardware at it, or by tweaking the branch >> selection algorithm a bit, or by decreasing the build frequency >> a bit for some branches based on some criteria), but if we can >> know *right now* that the designed plan cannot possibly work, then >> I would find it a bit sad to invest time into it. > > That's a logical awesome idea I'm ashamed not to have had sooner. > Still, it seems that it comes too late, after some searching it appears > that our Jenkins don't keep much stats. I'll extend the jjb setting that > remove build logs after 1 day. > > The Global build stats Jenkins plugin [1] seems interesting to display the > stats once more logs are kept. Shall I install it? I would love to have stats about the number of branches and ISO built and tested each day to put in our reports to our funders. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, bertagaz wrote (16 Feb 2015 14:32:57 GMT) : > On Wed, Feb 11, 2015 at 11:49:02PM +0100, intrigeri wrote: >> I have a few concerns, though: >> >> * "Scenario 2 : developer" doesn't make it clear if branch T is >> build *after being locally merged into branch B*, or not. >> Given that's what we're really interested in, and given >> "Scenario 1 : reviewer" is clearer (answer is: yes), I guess the >> answer is yes here too, but this should be clarified. > IIRC that was something Alan had troubles with, as not being the way she > usually work on a feature branch, which I think was more close to "Work > work work on the branch, and then when the feature is ready, merge the base > branch in it." So she usually do not merge the base branch very often. Most of us (including me) do the same as Alan, but IMO that's almost irrelevant to the topic at hand. This same scenario reads: And I need to know if my branch build is broken by something else possibly weeks after my last commit (by e.g Debian changes, changes in branch B, ...) ^^^ ... and we cannot possibly get this without locally merging the topic branch F into B before building. The important point here may be the *locally* word. This merge would only be done in Jenkins own temporary Git checkout, and wouldn't affect how we're handling our branches' Git history. > But I agree this is not the best way to go, so if Alan doesn't come up with > a block on this, I agree to add the clarification. OK. But apparently, either I misunderstood why Alan had trouble with this idea, or Alan has misunderstood the idea, so IMO it would be good to have his opinion now that I've clarified what I think we should do, and why. Alan? >> * I see no statistics about how many ISOs we are *currently* >> building each day. So it's not clear to me if our current setup >> can support building N more branches, once a day each. It would be >> useful to have this number (e.g. raw number per day during the 1.3 >> release cycle, and daily average). Of course we can fix that later >> (either by throwing more hardware at it, or by tweaking the branch >> selection algorithm a bit, or by decreasing the build frequency >> a bit for some branches based on some criteria), but if we can >> know *right now* that the designed plan cannot possibly work, then >> I would find it a bit sad to invest time into it. > That's a logical awesome idea I'm ashamed not to have had sooner. > Still, it seems that it comes too late, after some searching it appears > that our Jenkins don't keep much stats. As discussed on IRC, we have about 10 days of logs on the filesystem currently, so you can still retrieve recent stats about it. The earlier, the better, as we're frozen and the more we wait, the less the numbers will be meaningful. > The Global build stats Jenkins plugin [1] seems interesting to display the > stats once more logs are kept. Shall I install it? Let's give it a try, yes. >> * "Scenario 3 : RM" says "I need to know when a branch FTBFS", but >> I see no notification mechanism planned, so... how is the RM >> supposed to know. Also, whenever stalled topic branches start >> failing, this can imply an avalanche of daily email to the RM, who >> will of course start ignoring all email from Jenkins and then >> we've lost. I'm particularly worried about this topic since anonym >> (our RM most of the time) didn't comment on this proposal yet, and >> has already expressed concerns about this kind of issues. > I think anonym did comment on this proposal, he did a review of this > blueprint already. OK. Let's keep in mind that he may not have thought of this potential problem yet. Now, I wonder if we could solve this quite simply by having the RM be notified for *base* branch build failure only. The RM doesn't care about topic branches that haven't been submitted for merging yet, right? (So I'm skipping the rest of the discussion, that may actually be moot.) > One thing that this question also raise is the fact that the RM is not > always the same person, so I'm wondering how to have jenkins notifying the > right email depending on who is the RM for the next release. First thing > that come in top of my mind is... a schleuder address. :) Indeed, that's the kind of aliases the RMs can trivially update themselves, without asking the infra team to do anything. Cheers, -- intrigeri ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, bertagaz wrote (16 Feb 2015 14:37:56 GMT) : > On Thu, Feb 12, 2015 at 12:22:45AM +0100, intrigeri wrote: >> Here's one more: the proposed notification mechanism makes sense to me >> for topic branches. But for "base" branches, it's more complicated: >> >> * when building after a Git push, [...] >> >> * when building daily, on failure I don't see how it can be useful to >> notify the last person who merged something to into a base branch; >> so, who is responsible to keep these branches in a buildable >> state? [...] >> Thoughts? > Agree with that as stated in my previous email. OK, good. Then I'll let you capture in the specification that problem, and our currently preferred solution. What follows is only interesting for Jenkins folks. I guess almost everyone but DrWhax and bertagaz can stop here. > I think it is doable to differenciate them, by splitting the daily > and on git push job definiton maybe. Looks a bit ugly a solution, and it would make the Jenkins config, interface and reporting perhaps a bit too complicated. And we would have two separate sources of information to answer the "does branch B currently build fine?" question, which feels conceptually wrong. E.g. consider the following scenario: 1. Someone pushes something that breaks the build 2. The post-push job is triggered, fails, and the person who pushed gets notified; so far, so good, except the daily job still pretends everything is fine 3. The person who pushed doesn't fix their stuff before the next daily build 4. The next daily build fails, and then the RM is notified; in the case of a base branch, this may be good to have the RM as a fallback here. But then, we have two jobs in fail state in Jenkins. Weird. 5. The RM fixes the build and pushes. 6. The post-push job is triggered, passes, and then the RM (who just pushed) is notified that the build is back to normal, *but* the person who pushed initially isn't notified that the problem has been fixed (so might do duplicate work), and the daily job is still in failing state until it is run again. => I think this trick would make the overall thing hard to understand and draw conclusions from, for anyone not deeply involved in our Jenkins stuff. > Having a look at plugins might help, as well as how other projects > do that (as you stated elsewhere). Yep. At least the obvious candidate (Email-ext plugin) doesn't seem able to email different recipients depending on what triggered the build out-of-the-box. But apparently, one can set up two 'Script - After Build' email triggers in the Email-ext configuration: one emails the culprit, the other emails the RM. And then, they do something or not depending on a variable we set during the build, based on what triggered the build. Likely the cleaner and simpler solution. Otherwise, we could have Jenkins email some pipe script that will forward to the right person depending on 1. whether it's a base branch; and 2. whether the build was triggered by a push or by something else. This should work if we can get the email notification to pass the needed info in it. E.g. the full console output currently has "Started by timer" or "Started by an SCM change", but this is not part of the email notification. Could work, but a bit hackish and all kinds of things can go wrong. Also, I've seen lots of people documenting crazy similar things with some of these plugins: "Run Condition", "Conditional BuildStep", "Flexible Publish" and "Any Build step". But then it gets too complicated for me to dive into it right now. If you prefer, in the future I can dump such research results into the blueprint instead of here. Just tell me where it should go. Cheers, -- intrigeri ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
bertagaz wrote (16 Feb 2015 12:03:12 GMT) : > Ack, sounds reasonable. However from what I've seen, it sometimes means a > lot of branches so I wonder if we scaled our infra enough for that, as we > didn't include this branches in our maths since the beginning of the > discussion. This seems like a serious bug in this discussion process. May you please then provide example numbers that match what the proposed algorithm would output, so that we can reason about it? ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
On Wed, Feb 11, 2015 at 11:29:45PM +0100, intrigeri wrote: > Hi, > > bertagaz wrote (18 Jan 2015 16:39:28 GMT) : > > 0. Do we think we might also need or want a mechanism to blacklist a branch, > > or we should just assume that our algos will only select the right ones > > and not hit any corner cases? > > I think this is a good question, that deserves more thought. > > Unfortunately I've not found any discussion about it on the blueprint > nor on the list, not even an example use case for such a blacklist, so > right now — sorry to be harsh — it looks like a technical idea that's > looking for a problem it might help solving. It's just that, something that did pop up in my head while writing the blueprint, but I didn't find much corresponding branches while watching the unmerged ones during the 1.3 release cycle. > So, what would be the use cases? I can think of a few potential ones: > > 1. A branch that satisfies the criteria for automatic builds, but >starts failing continuously, e.g. because its developer is on >vacation for 2 weeks. > >=> as far as I understood, only that branch's developer is nagged >by failure notifications, so... who cares if it fails to build for >2 weeks? > > 2. Branches that modify only the doc or website > >=> indeed, at first glance it seems a bit sad to spend CPU cycles >on building and potentially testing such branches. OTOH, these >branches, like any other, must not break the build, otherwise >they're not fit for merging, so it makes sense to build an ISO >(yes, an ISO, not only the website: we have quite a few things in >the ISO build process that somewhat depend on the website, run `git >grep usr/share/doc/tails/website -- config' if unconvinced). >And they must not break functionality either, so IMO it makes sense >to run the automated test suite on it too (again, we have quite >a few runtime functionality that depends on the bundled static >website). Ack, sounds reasonable. However from what I've seen, it sometimes means a lot of branches so I wonder if we scaled our infra enough for that, as we didn't include this branches in our maths since the beginning of the discussion. > bertagaz, any other use case you were thinking of? Not really at the moment. bert. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
On Thu, Feb 12, 2015 at 12:22:45AM +0100, intrigeri wrote: > Here's one more: the proposed notification mechanism makes sense to me > for topic branches. But for "base" branches, it's more complicated: > > * when building after a Git push, it does make sense to notify the > committer on failure -- that is, in theory, the person who merged > something into the branch, and apparently forgot to make sure it > builds before pushing > > * when building daily, on failure I don't see how it can be useful to > notify the last person who merged something to into a base branch; > so, who is responsible to keep these branches in a buildable state? > In practice, quite often, by default it's me -- and I never > volunteered to do that. I'd rather see this formally put on the > release manager's plate, since these branches are only useful to > prepare the next release. So, IMO the RM should be notified in > this case. > > Now, if it's too complicated, implementation-wise, to differenciate > these two situations, and we have to choose between committer, RM, and > tails-dev@, then I'd be in favour of notifying either the RM or > tails-dev@, who are more likely to be/feel responsible for the failure > than the last committer. > > Thoughts? Agree with that as stated in my previous email. I think it is doable to differenciate them, by splitting the daily and on git push job definiton maybe. Having a look at plugins might help, as well as how other projects do that (as you stated elsewhere). For consistency with our roles, I think it make more sense to have the RM notified, but don't block on the tails-dev choice. bert. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, Thx for the extensive review! On Wed, Feb 11, 2015 at 11:49:02PM +0100, intrigeri wrote: > > We're already drafted some scenario's on: > > https://tails.boum.org/blueprint/automated_builds_and_tests/autobuild_specs/ > > I have a few concerns, though: > > * "Scenario 2 : developer" doesn't make it clear if branch T is > build *after being locally merged into branch B*, or not. > Given that's what we're really interested in, and given > "Scenario 1 : reviewer" is clearer (answer is: yes), I guess the > answer is yes here too, but this should be clarified. IIRC that was something Alan had troubles with, as not being the way she usually work on a feature branch, which I think was more close to "Work work work on the branch, and then when the feature is ready, merge the base branch in it." So she usually do not merge the base branch very often. But I agree this is not the best way to go, so if Alan doesn't come up with a block on this, I agree to add the clarification. > * I see no statistics about how many ISOs we are *currently* > building each day. So it's not clear to me if our current setup > can support building N more branches, once a day each. It would be > useful to have this number (e.g. raw number per day during the 1.3 > release cycle, and daily average). Of course we can fix that later > (either by throwing more hardware at it, or by tweaking the branch > selection algorithm a bit, or by decreasing the build frequency > a bit for some branches based on some criteria), but if we can > know *right now* that the designed plan cannot possibly work, then > I would find it a bit sad to invest time into it. That's a logical awesome idea I'm ashamed not to have had sooner. Still, it seems that it comes too late, after some searching it appears that our Jenkins don't keep much stats. I'll extend the jjb setting that remove build logs after 1 day. The Global build stats Jenkins plugin [1] seems interesting to display the stats once more logs are kept. Shall I install it? > * "Scenario 3 : RM" says "I need to know when a branch FTBFS", but > I see no notification mechanism planned, so... how is the RM > supposed to know. Also, whenever stalled topic branches start > failing, this can imply an avalanche of daily email to the RM, who > will of course start ignoring all email from Jenkins and then > we've lost. I'm particularly worried about this topic since anonym > (our RM most of the time) didn't comment on this proposal yet, and > has already expressed concerns about this kind of issues. I think anonym did comment on this proposal, he did a review of this blueprint already. But I agree the RM notification part is not very clear. We could make it so that the RM is emailed when a daily build job fails, the "build failed on git push" one being sent to the commit author anyway, according to our scenarios. Of course the commit author would also be notified for the daily ones too. If that's still too much from the RM point of view, we could have the RM notified only the first time a daily job fails. That seems to make sense to me: the RM gets a notification in the mailbox once a day for failed jobs, and then he/she either fix it, or ask someone to work on that. To have a developer claiming to Jenkins he/she is now responsible of that branch (and thus is getting the next notifications), he/she would just have to do a dumb commit on the branch. One thing that this question also raise is the fact that the RM is not always the same person, so I'm wondering how to have jenkins notifying the right email depending on who is the RM for the next release. First thing that come in top of my mind is... a schleuder address. :) > I assume these concerns (except the 2nd one probably) are not blocking > wrt. starting to implement stuff, so: Go! Great! > To end with, I find two things confusing in the rest of the document: > > * Scenario numbering is reset in the "Future ideas" section, so one > cannot simply refer to "scenario 2" unambiguously, without making > it clear if they're speaking of "scenario 2 in the Scenarios > section", or of "scenario 2 in the Future ideas section"; > I suggest you avoid assigning duplicated scenario identifiers. Fixed. > * The "tag T" notation (undefined) is somewhat conflicting with the > "branch T" one that's defined earlier. Fixed. bert. [1] https://wiki.jenkins-ci.org/display/JENKINS/Global+Build+Stats+Plugin ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
intrigeri wrote (11 Feb 2015 22:49:02 GMT) : > I have a few concerns, though: Here's one more: the proposed notification mechanism makes sense to me for topic branches. But for "base" branches, it's more complicated: * when building after a Git push, it does make sense to notify the committer on failure -- that is, in theory, the person who merged something into the branch, and apparently forgot to make sure it builds before pushing * when building daily, on failure I don't see how it can be useful to notify the last person who merged something to into a base branch; so, who is responsible to keep these branches in a buildable state? In practice, quite often, by default it's me -- and I never volunteered to do that. I'd rather see this formally put on the release manager's plate, since these branches are only useful to prepare the next release. So, IMO the RM should be notified in this case. Now, if it's too complicated, implementation-wise, to differenciate these two situations, and we have to choose between committer, RM, and tails-dev@, then I'd be in favour of notifying either the RM or tails-dev@, who are more likely to be/feel responsible for the failure than the last committer. Thoughts? ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, > We're already drafted some scenario's on: > https://tails.boum.org/blueprint/automated_builds_and_tests/autobuild_specs/ I've finally looked at it. I've pushed minor typo/formatting fixes and rephrasing, nothing important. I'm very happy with basically everything in the "2. Scenarios" section, which seems to be the most important one. Yay! I have a few concerns, though: * "Scenario 2 : developer" doesn't make it clear if branch T is build *after being locally merged into branch B*, or not. Given that's what we're really interested in, and given "Scenario 1 : reviewer" is clearer (answer is: yes), I guess the answer is yes here too, but this should be clarified. * I see no statistics about how many ISOs we are *currently* building each day. So it's not clear to me if our current setup can support building N more branches, once a day each. It would be useful to have this number (e.g. raw number per day during the 1.3 release cycle, and daily average). Of course we can fix that later (either by throwing more hardware at it, or by tweaking the branch selection algorithm a bit, or by decreasing the build frequency a bit for some branches based on some criteria), but if we can know *right now* that the designed plan cannot possibly work, then I would find it a bit sad to invest time into it. * "Scenario 3 : RM" says "I need to know when a branch FTBFS", but I see no notification mechanism planned, so... how is the RM supposed to know. Also, whenever stalled topic branches start failing, this can imply an avalanche of daily email to the RM, who will of course start ignoring all email from Jenkins and then we've lost. I'm particularly worried about this topic since anonym (our RM most of the time) didn't comment on this proposal yet, and has already expressed concerns about this kind of issues. I assume these concerns (except the 2nd one probably) are not blocking wrt. starting to implement stuff, so: Go! To end with, I find two things confusing in the rest of the document: * Scenario numbering is reset in the "Future ideas" section, so one cannot simply refer to "scenario 2" unambiguously, without making it clear if they're speaking of "scenario 2 in the Scenarios section", or of "scenario 2 in the Future ideas section"; I suggest you avoid assigning duplicated scenario identifiers. * The "tag T" notation (undefined) is somewhat conflicting with the "branch T" one that's defined earlier. Cheers, -- intrigeri ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, bertagaz wrote (18 Jan 2015 16:39:28 GMT) : > 0. Do we think we might also need or want a mechanism to blacklist a branch, > or we should just assume that our algos will only select the right ones > and not hit any corner cases? I think this is a good question, that deserves more thought. Unfortunately I've not found any discussion about it on the blueprint nor on the list, not even an example use case for such a blacklist, so right now — sorry to be harsh — it looks like a technical idea that's looking for a problem it might help solving. So, what would be the use cases? I can think of a few potential ones: 1. A branch that satisfies the criteria for automatic builds, but starts failing continuously, e.g. because its developer is on vacation for 2 weeks. => as far as I understood, only that branch's developer is nagged by failure notifications, so... who cares if it fails to build for 2 weeks? 2. Branches that modify only the doc or website => indeed, at first glance it seems a bit sad to spend CPU cycles on building and potentially testing such branches. OTOH, these branches, like any other, must not break the build, otherwise they're not fit for merging, so it makes sense to build an ISO (yes, an ISO, not only the website: we have quite a few things in the ISO build process that somewhat depend on the website, run `git grep usr/share/doc/tails/website -- config' if unconvinced). And they must not break functionality either, so IMO it makes sense to run the automated test suite on it too (again, we have quite a few runtime functionality that depends on the bundled static website). bertagaz, any other use case you were thinking of? Cheers! ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, On Mon, Jan 26, 2015 at 01:33:25PM +0100, bertagaz wrote: > > The blueprint has been updated with some proposalsi for each question. If > you want to consider them, or propose another one, please have a look. I've updated the blueprint again. Revelant changes: +Given a branch take around 45 minutes to be build on lizard (worth case), +with two builders, lizard will be able to build something like 64 ISOs a +day. +As of 2015-02-02, there are 26 branches that would be automatically +build as part of the next 1.3 release, following the for now defined +criterias [complete list in the blueprint] (this statistic has been collected with the script posted in https://labs.riseup.net/code/issues/8657#note-7) bert. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, On Mon, Jan 19, 2015 at 08:14:55PM +0100, bertagaz wrote: > > On Sun, Jan 18, 2015 at 08:07:46PM +, sajolida wrote: > > bertagaz: > > > 0. We might also want a mechanism for devs to pro-actively state they > > > want to > > > keep their branches being build even if the last commit was older than the > > > last release. IIRC > > > > If I understand correctly, adding a dummy commit would bring back my > > branch in the set of branches that are automatically built. So maybe we > > don't need a dedicated mechanism handle those rare cases. > > That's the idea, having a timestamp file that would be use for this dummy > commit. But it comes for free, sure. :) I had Alan having a look at the blueprint too, she fixed some issues in the scenario and added one for the future. The blueprint has been updated with some proposalsi for each question. If you want to consider them, or propose another one, please have a look. bert. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi, On Sun, Jan 18, 2015 at 08:07:46PM +, sajolida wrote: > bertagaz: > > 0. We might also want a mechanism for devs to pro-actively state they want > > to > > keep their branches being build even if the last commit was older than the > > last release. IIRC > > If I understand correctly, adding a dummy commit would bring back my > branch in the set of branches that are automatically built. So maybe we > don't need a dedicated mechanism handle those rare cases. That's the idea, having a timestamp file that would be use for this dummy commit. But it comes for free, sure. :) bert. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
bertagaz: > 0. We might also want a mechanism for devs to pro-actively state they want to > keep their branches being build even if the last commit was older than the > last release. IIRC If I understand correctly, adding a dummy commit would bring back my branch in the set of branches that are automatically built. So maybe we don't need a dedicated mechanism handle those rare cases. -- sajolida ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Automated builds specification
Hi,
Some thoughts and questions, raised in parts from past IRL discussions.
Consider it like a ping for this thread. :)
On Sun, Jan 11, 2015 at 08:07:13PM +0100, Jurre van Bergen wrote:
> Hi,
>
> For the first iteration, which is automatically build of interesting
> branches, we need to specify:
>
> * Which branches we want to build?
We might want to use several algorithms / tricks to select the branches to
be build automatically.
One of this algo could be:
0. Build all {feature,bugfix} branch that are not merged in
stable, devel and testing, and had new commits since the last release.
I have a script that does this roughly and it says that currently, that
means 11 branches. [1]
That would make sense, but might require to have some stats about how much
it has meant for the past releases. I'm not sure how to do that though.
0. We might also want a mechanism for devs to pro-actively state they want to
keep their branches being build even if the last commit was older than the
last release. IIRC
0. Do we think we might also need or want a mechanism to blacklist a branch,
or we should just assume that our algos will only select the right ones
and not hit any corner cases?
Thoughts, answers, ideas?
bert.
[1] bugfix/8680-git-without-polipo
bugfix/8714-tor-is-ready-robustness
feature/6297-save-packages-list
feature/6992-whisperback-email-address
feature/7450-openpgp-applet-exit
feature/7779-revisit-touchpad-settings
feature/8681-test-suite-ramdisk
feature/8719-mount-output-in-bug-reports
feature/jessie
feature/linux-3.18
feature/torbrowser-alpha
___
Tails-dev mailing list
Tails-dev@boum.org
https://mailman.boum.org/listinfo/tails-dev
To unsubscribe from this list, send an empty email to
tails-dev-unsubscr...@boum.org.
[Tails-dev] Automated builds specification
Hi, As some of you might know, we'll spend quite some time this year seting up automated builds and test using our Jenkins platform and make it easier to extract information and data from it. For the first iteration, which is automatically build of interesting branches, we need to specify: * Which branches we want to build? We already build the base branches (stables, testing, devel and experimental + feature/jessie). So the questions raised is mostly concern the feature/* and bugfix/* branches (so topic branches) The criterias to automatically select the branches to be build could be: branches which are not merged to devel but has new commits since N weeks time, 15 days or the previous release. * Which regularity if needed? (assuming it will be build when a push is made on it) - Is a branch built everyday enough? Should it be built more often or less often? * How should you be notified? - Direct e-mail? - IRC channel? We're already drafted some scenario's on: https://tails.boum.org/blueprint/automated_builds_and_tests/autobuild_specs/ Some of you might want to see other one's implemented , so please update the blueprint accordingly. Scenarios can range from release managers, developers, reviewers, testers. Some of these suggestions might not make it this year but will be looked at in the following year. Need to keep in mind that the same question will be raised when we'll design the automated tests. All the best, CI team ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
