Re: [Tails-dev] Releasing automated tests
[Moving discussion to tails-dev@] Given the trimming that has happened, some context may have been lost. The discussion is about that we now, in our Jenkins setup, automatically test images built from doc/ and web/ branches, which wastes a lot of time on our isotesters. > From: intrigeri > Date: Tue, 20 Oct 2015 13:08:31 +0200 > >> From: bertagaz >> Date: Mon, 19 Oct 2015 12:29:26 +0200 > >>> From: anonym >> >>> Still, once we release 1.7, then all doc/ and web/ branches will become >>> tested. I suspect we will need a permanent fix for only building (not >>> testing) these branches -- it's useless to test them 99.9% of the time, >>> and they will block (for ~5 hours) test runs from relevant branches that >>> got something pushed to them right after them. > >> That's something we didn't decide when during the design round. Sounds >> doable, but I wonder if there are still some valid points to still test >> that branches. True, but there's an overwhelming amount of them, and their modifications are limited to something that is completely isolated from most of Tails, the OS, meaning that a huge part of each test run on them is just a (possibly out-dated) re-run of master/devel/stable depending on which branch it was based on. That is unlike feature/, bugfix/ and test/ branches, that need a full run in general. Perhaps you can see where I'm going: As an optimization, we could introduce @web and @doc tags, and run the test suite with `--tag @web` or `--tag @doc` for doc/ and web/ branches, respectively. Then we could even have automated tests of @web changes before deploying them by browsing the local wiki in Tails. :) Note that I may not have the correct understanding of the doc/ vs web/ distinction, so I'd like a clarification just in case so we don't design something stupid. I suspect that since we don't have any automated tests for the *website* (as opposed to the docs) we only care about doc/ and only need to test web/ if we want to start testing the website. > FTR I dislike the idea of blacklisting such branches based on their > name. I'm not going to debate it here [...] Please do it here, then! :) I guess it's related to that you want at least doc/ branches to be tested, since our automated test suite actually depend on the documentation (see two quotes below for more on this). I wonder if we could have a configuration filem like config/jenkins_testing, which has settings controlling how the automated test suite is invoked on Jenkins. I imagine something like this: ENABLE = {true,false} Whether to run the test suite at all. Hence, when merging another base branch into master as part of the release process, we just make sure to set this to `false` and we're done. When merging master into a base branch, it must be set to `true`. So it's similar to config/base_branch, except that it includes master as a special case. Is this problematic? Won't merge conflicts save us? If we prefer it, we could implement my tagging idea with this: CUCUMBER_ARGUMENTS = This makes Jenkins invoke run_test_suite with extra CUCUMBER_ARGUMENTS. And it could be used for the @web/@doc idea above if we have it set to `--tag @doc` or whatever in master. We could even test master then, in the same way. We just have to update the release process in the same way as for the ENABLE setting, but for CUCUMBER_ARGUMENTS. Since this affects how Jenkins will invoke a script, there's a security concern vs the shell that has to be considered. Also arbitrary paths can be given. Hence we probably may want to have this option *instead*: RUN_WITH_TAGS = So if it's set to ["tag1", "~tag2,tag3"] it would be translated to these distinct arguments: ['--tag', 'tag1', '--tag', '~tag2,tag3'] in the usual safe (i.e. shell-free) list syntax used for exec in many programming languages (python, ruby, perl...). This would also improve the situation for @fragile, and also be subject to the same release process update as ENABLE above. Note that ENABLE gives an alternative implementation for the problem that #10389 wants to solve, i.e. "don't run tests for WIP branches". I think I prefer this approach, so we avoid excessive branch re-naming. Also note that with the flexibility from CUCUMBER_ARGUMENTS/CUCUMBER_TAGS we can remove `--tag ~@fragile` from the Jenkins wrapper script (#10288) by setting it in the config instead. Using CUCUMBER_ARGUMENTS/CUCUMBER_TAGS the true power users could even do "focus" testing of specific features/scenarios via a temporary tag and then making Jenkins only test that tag -- now the test suite will not waste time on other scenarios. FWIW. I don't think this will play nicely with our future ideas about Jenkins updating tickets in Redmine given the test results. That could be fixed if we added another option: JENKINS_REVIEW = {true,false} It would be true by default (I guess), and would then enable Jenkins "reviewing" (i.e. ticket updating). So when doing focus testing like descri
Re: [Tails-dev] Releasing automated tests
anonym: > [Moving discussion to tails-dev@] Meta: I really don't want to understand everything that's in this email but I felt you would want me to answer this one. But if you think that you can have this discussion without me I would be super happy as well. > Given the trimming that has happened, some context may have been lost. > The discussion is about that we now, in our Jenkins setup, automatically > test images built from doc/ and web/ branches, which wastes a lot of > time on our isotesters. > >> From: intrigeri >> Date: Tue, 20 Oct 2015 13:08:31 +0200 >> >>> From: bertagaz >>> Date: Mon, 19 Oct 2015 12:29:26 +0200 >> From: anonym >>> Still, once we release 1.7, then all doc/ and web/ branches will become tested. I suspect we will need a permanent fix for only building (not testing) these branches -- it's useless to test them 99.9% of the time, and they will block (for ~5 hours) test runs from relevant branches that got something pushed to them right after them. >> >>> That's something we didn't decide when during the design round. Sounds >>> doable, but I wonder if there are still some valid points to still test >>> that branches. > > True, but there's an overwhelming amount of them, and their > modifications are limited to something that is completely isolated from > most of Tails, the OS, meaning that a huge part of each test run on them > is just a (possibly out-dated) re-run of master/devel/stable depending > on which branch it was based on. That is unlike feature/, bugfix/ and > test/ branches, that need a full run in general. Perhaps you can see > where I'm going: > > As an optimization, we could introduce @web and @doc tags, and run the > test suite with `--tag @web` or `--tag @doc` for doc/ and web/ branches, > respectively. Then we could even have automated tests of @web changes > before deploying them by browsing the local wiki in Tails. :) > > Note that I may not have the correct understanding of the doc/ vs web/ > distinction, so I'd like a clarification just in case so we don't design > something stupid. I suspect that since we don't have any automated tests > for the *website* (as opposed to the docs) we only care about doc/ and > only need to test web/ if we want to start testing the website. > >> FTR I dislike the idea of blacklisting such branches based on their >> name. I'm not going to debate it here [...] The prefixes doc/ and web/ are used loosely to differentiate work on the "documentation" (/doc /support) and the "website" in general (structure, stuff not in /doc, etc.) but the difference is not strict. I also don't think they should be tested. Maybe you could exclude them from testing by their diff to their base branch. If all the diff is under wiki/src then don't test that branch. > Please do it here, then! :) I guess it's related to that you want at > least doc/ branches to be tested, since our automated test suite > actually depend on the documentation (see two quotes below for more on > this). > [...] Skipping tons of stuff I don't understand. >> [...] making sure that the workaround is not worst than the problem >> it fixes > > The only effect should be that we won't get automated tests of the few > scenarios that looks at the wiki shipped inside Tails. I think that > currently is one scenario: The Report an Error launcher will open the > support documentation in supported non-English locales. So, if some wiki > change makes the 'the support documentation page opens in Tor Browser' > step fail. The only way that can happen is if: > > * the url to the "Help and Support" section changes so the desktop > shortcut is wrong > > * there's a change to the "Help and Support" section, or its German > translation > > * we change the website layout of the "parentlinks" thing, or the "home" > icon there > > Only in the first case would our test suite find an actual error, the > other two just implies that our test is now out of sync and must be > updated. Clearly it's not worth wasting that much isotester time on this > at the moment. Agreed. If I understand correctly, these scenarios have a *dependency* on what is on the local copy of the website, but they are actually testing the Tails software (the "Report an Error" launcher for example). They are not testing the content of the local copy of the website itself. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Releasing automated tests
sajolida: > anonym: >> [Moving discussion to tails-dev@] > > Meta: I really don't want to understand everything that's in this email > but I felt you would want me to answer this one. But if you think that > you can have this discussion without me I would be super happy as well. I believe you have answered the question that was (mostly) directed to you, but you also added an interesting idea, so... :) >> Given the trimming that has happened, some context may have been lost. >> The discussion is about that we now, in our Jenkins setup, automatically >> test images built from doc/ and web/ branches, which wastes a lot of >> time on our isotesters. >> >>> From: intrigeri >>> Date: Tue, 20 Oct 2015 13:08:31 +0200 >>> From: bertagaz Date: Mon, 19 Oct 2015 12:29:26 +0200 >>> > From: anonym > Still, once we release 1.7, then all doc/ and web/ branches will become > tested. I suspect we will need a permanent fix for only building (not > testing) these branches -- it's useless to test them 99.9% of the time, > and they will block (for ~5 hours) test runs from relevant branches that > got something pushed to them right after them. >>> That's something we didn't decide when during the design round. Sounds doable, but I wonder if there are still some valid points to still test that branches. >> >> True, but there's an overwhelming amount of them, and their >> modifications are limited to something that is completely isolated from >> most of Tails, the OS, meaning that a huge part of each test run on them >> is just a (possibly out-dated) re-run of master/devel/stable depending >> on which branch it was based on. That is unlike feature/, bugfix/ and >> test/ branches, that need a full run in general. Perhaps you can see >> where I'm going: >> >> As an optimization, we could introduce @web and @doc tags, and run the >> test suite with `--tag @web` or `--tag @doc` for doc/ and web/ branches, >> respectively. Then we could even have automated tests of @web changes >> before deploying them by browsing the local wiki in Tails. :) >> >> Note that I may not have the correct understanding of the doc/ vs web/ >> distinction, so I'd like a clarification just in case so we don't design >> something stupid. I suspect that since we don't have any automated tests >> for the *website* (as opposed to the docs) we only care about doc/ and >> only need to test web/ if we want to start testing the website. >> >>> FTR I dislike the idea of blacklisting such branches based on their >>> name. I'm not going to debate it here [...] > > The prefixes doc/ and web/ are used loosely to differentiate work on the > "documentation" (/doc /support) and the "website" in general (structure, > stuff not in /doc, etc.) but the difference is not strict. ACK, as I expected, than. > I also don't think they should be tested. Maybe you could exclude them > from testing by their diff to their base branch. If all the diff is > under wiki/src then don't test that branch. I guess you mean the diff against the base branch (but base branches themselves would *always* build, of course). Hm. Technically we'd have to do something slightly different since a `git diff` would show changes in the base branch since the point they diverged. We'd have to look at all files touches in `git log $base_branch..` or something like that. It's an interesting approach, which I think I like. >>> [...] making sure that the workaround is not worst than the problem >>> it fixes >> >> The only effect should be that we won't get automated tests of the few >> scenarios that looks at the wiki shipped inside Tails. [...] > > Agreed. If I understand correctly, these scenarios have a *dependency* > on what is on the local copy of the website, but they are actually > testing the Tails software (the "Report an Error" launcher for example). Correct so far. > They are not testing the content of the local copy of the website itself. Actually, they *are* testing the content of the local copy, e.g. that the support page is properly localized. Hence there could be a subset of automated tests that would make sense to run for doc/ and web/ branches. Cheers! ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Releasing automated tests
anonym: > sajolida: >> anonym: >>> [Moving discussion to tails-dev@] >> >> Meta: I really don't want to understand everything that's in this email >> but I felt you would want me to answer this one. But if you think that >> you can have this discussion without me I would be super happy as well. > > I believe you have answered the question that was (mostly) directed to > you, but you also added an interesting idea, so... :) > >>> Given the trimming that has happened, some context may have been lost. >>> The discussion is about that we now, in our Jenkins setup, automatically >>> test images built from doc/ and web/ branches, which wastes a lot of >>> time on our isotesters. >>> From: intrigeri Date: Tue, 20 Oct 2015 13:08:31 +0200 > From: bertagaz > Date: Mon, 19 Oct 2015 12:29:26 +0200 >> From: anonym > >> Still, once we release 1.7, then all doc/ and web/ branches will become >> tested. I suspect we will need a permanent fix for only building (not >> testing) these branches -- it's useless to test them 99.9% of the time, >> and they will block (for ~5 hours) test runs from relevant branches that >> got something pushed to them right after them. > That's something we didn't decide when during the design round. Sounds > doable, but I wonder if there are still some valid points to still test > that branches. >>> >>> True, but there's an overwhelming amount of them, and their >>> modifications are limited to something that is completely isolated from >>> most of Tails, the OS, meaning that a huge part of each test run on them >>> is just a (possibly out-dated) re-run of master/devel/stable depending >>> on which branch it was based on. That is unlike feature/, bugfix/ and >>> test/ branches, that need a full run in general. Perhaps you can see >>> where I'm going: >>> >>> As an optimization, we could introduce @web and @doc tags, and run the >>> test suite with `--tag @web` or `--tag @doc` for doc/ and web/ branches, >>> respectively. Then we could even have automated tests of @web changes >>> before deploying them by browsing the local wiki in Tails. :) >>> >>> Note that I may not have the correct understanding of the doc/ vs web/ >>> distinction, so I'd like a clarification just in case so we don't design >>> something stupid. I suspect that since we don't have any automated tests >>> for the *website* (as opposed to the docs) we only care about doc/ and >>> only need to test web/ if we want to start testing the website. >>> FTR I dislike the idea of blacklisting such branches based on their name. I'm not going to debate it here [...] >> >> The prefixes doc/ and web/ are used loosely to differentiate work on the >> "documentation" (/doc /support) and the "website" in general (structure, >> stuff not in /doc, etc.) but the difference is not strict. > > ACK, as I expected, than. > >> I also don't think they should be tested. Maybe you could exclude them >> from testing by their diff to their base branch. If all the diff is >> under wiki/src then don't test that branch. > > I guess you mean the diff against the base branch (but base branches > themselves would *always* build, of course). Hm. Technically we'd have > to do something slightly different since a `git diff` would show changes > in the base branch since the point they diverged. We'd have to look at > all files touches in `git log $base_branch..` or something like that. > It's an interesting approach, which I think I like. Did you took into account the '...' (THREE DOTS) operator which is slightly different than '..' and I *think* might be helpful here to diff only the changes that happened on this branch. I'm not sure what it does exactly (couldn't understand the man page). [...] making sure that the workaround is not worst than the problem it fixes >>> >>> The only effect should be that we won't get automated tests of the few >>> scenarios that looks at the wiki shipped inside Tails. [...] >> >> Agreed. If I understand correctly, these scenarios have a *dependency* >> on what is on the local copy of the website, but they are actually >> testing the Tails software (the "Report an Error" launcher for example). > > Correct so far. > >> They are not testing the content of the local copy of the website itself. > > Actually, they *are* testing the content of the local copy, e.g. that > the support page is properly localized. Hence there could be a subset of > automated tests that would make sense to run for doc/ and web/ branches. I didn't know that, sorry! ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Releasing automated tests
sajolida: > anonym: >> sajolida: >>> anonym: [Moving discussion to tails-dev@] >>> >>> Meta: I really don't want to understand everything that's in this email >>> but I felt you would want me to answer this one. But if you think that >>> you can have this discussion without me I would be super happy as well. >> >> I believe you have answered the question that was (mostly) directed to >> you, but you also added an interesting idea, so... :) >> Given the trimming that has happened, some context may have been lost. The discussion is about that we now, in our Jenkins setup, automatically test images built from doc/ and web/ branches, which wastes a lot of time on our isotesters. > From: intrigeri > Date: Tue, 20 Oct 2015 13:08:31 +0200 > >> From: bertagaz >> Date: Mon, 19 Oct 2015 12:29:26 +0200 > >>> From: anonym >> >>> Still, once we release 1.7, then all doc/ and web/ branches will become >>> tested. I suspect we will need a permanent fix for only building (not >>> testing) these branches -- it's useless to test them 99.9% of the time, >>> and they will block (for ~5 hours) test runs from relevant branches that >>> got something pushed to them right after them. > >> That's something we didn't decide when during the design round. Sounds >> doable, but I wonder if there are still some valid points to still test >> that branches. True, but there's an overwhelming amount of them, and their modifications are limited to something that is completely isolated from most of Tails, the OS, meaning that a huge part of each test run on them is just a (possibly out-dated) re-run of master/devel/stable depending on which branch it was based on. That is unlike feature/, bugfix/ and test/ branches, that need a full run in general. Perhaps you can see where I'm going: As an optimization, we could introduce @web and @doc tags, and run the test suite with `--tag @web` or `--tag @doc` for doc/ and web/ branches, respectively. Then we could even have automated tests of @web changes before deploying them by browsing the local wiki in Tails. :) Note that I may not have the correct understanding of the doc/ vs web/ distinction, so I'd like a clarification just in case so we don't design something stupid. I suspect that since we don't have any automated tests for the *website* (as opposed to the docs) we only care about doc/ and only need to test web/ if we want to start testing the website. > FTR I dislike the idea of blacklisting such branches based on their > name. I'm not going to debate it here [...] >>> >>> The prefixes doc/ and web/ are used loosely to differentiate work on the >>> "documentation" (/doc /support) and the "website" in general (structure, >>> stuff not in /doc, etc.) but the difference is not strict. >> >> ACK, as I expected, than. >> >>> I also don't think they should be tested. Maybe you could exclude them >>> from testing by their diff to their base branch. If all the diff is >>> under wiki/src then don't test that branch. >> >> I guess you mean the diff against the base branch (but base branches >> themselves would *always* build, of course). Hm. Technically we'd have >> to do something slightly different since a `git diff` would show changes >> in the base branch since the point they diverged. We'd have to look at >> all files touches in `git log $base_branch..` or something like that. >> It's an interesting approach, which I think I like. > > Did you took into account the '...' (THREE DOTS) operator which is > slightly different than '..' and I *think* might be helpful here to diff > only the changes that happened on this branch. I'm not sure what it does > exactly (couldn't understand the man page). Neat! That's is a very useful Git feature. Thanks for letting me know about it! Then I think we can combine the "..." operator with another fancy Git feature I recently found, namely Git pathspec "magic signatures". So we could do: BASE_BRANCH_DIFF="$(git diff $base_branch...$commit -- \ '*' \ ':!/wiki' \ ':!/ikiwiki.setup' \ ':!/ikiwiki-cgi.setup')" if [ -z "${BASE_BRANCH_DIFF}" ]; then CUCUMBER_ARGS="${CUCUMBER_ARGS} --tag @doc" fi where $commit is the commit we test, before merging the base branch locally. Interesting! Cheers! ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Releasing automated tests
Hi, Sorry for the late reply. On Wed, Oct 28, 2015 at 04:52:31PM +, anonym wrote: > sajolida: > > anonym: > >> sajolida: > > From: intrigeri > > Date: Tue, 20 Oct 2015 13:08:31 +0200 > > > > FTR I dislike the idea of blacklisting such branches based on their > > name. I'm not going to debate it here [...] Intrigeri, what's on your opinion on the proposed way to blacklist them made in this thread by anonym and sajolida (see below)? It's not based on their name anymore. > >>> I also don't think they should be tested. Maybe you could exclude them > >>> from testing by their diff to their base branch. If all the diff is > >>> under wiki/src then don't test that branch. That seems to be the consensus here then, unless someone else raises. > Then I think we can combine the "..." operator with another fancy Git > feature I recently found, namely Git pathspec "magic signatures". So we > could do: > >BASE_BRANCH_DIFF="$(git diff $base_branch...$commit -- \ >'*' \ >':!/wiki' \ >':!/ikiwiki.setup' \ >':!/ikiwiki-cgi.setup')" >if [ -z "${BASE_BRANCH_DIFF}" ]; then >CUCUMBER_ARGS="${CUCUMBER_ARGS} --tag @doc" >fi > > where $commit is the commit we test, before merging the base branch > locally. Interesting! Agree, I like this way to encode it. Sounds at least like a nice baby step to begin with, simple and effective. This whole thread remembered me the discussion we had earlier about feature branches, and made me think (again) we could as well use the cucumber tag facility to lighten the number of tests: if building from a feature|test branch, run with the --tag $branch_name, so developers can choose which test they want their branch to be tested against. May be useful for some branches were we know that it'll break some tests. But we dropped this idea. So if everyone agree on the proposal made here by anonym and sajolida, I guess that's now a new "Code" ticket. bert. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Releasing automated tests
bertagaz wrote (18 Nov 2015 12:34:34 GMT) : > Intrigeri, what's on your opinion on [...] Please give me an explicit deadline. Thanks for asking! ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Releasing automated tests
Hi, On Wed, Nov 18, 2015 at 02:38:00PM +0100, intrigeri wrote: > bertagaz wrote (18 Nov 2015 12:34:34 GMT) : > > Intrigeri, what's on your opinion on [...] > > Please give me an explicit deadline. I probably won't be able to work on it before week 49 anyway, so no rush, but an answer before would be cool. > Thanks for asking! Well, you express a strong blocking position, can't get a real consensus without an update from you :) Bert. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Releasing automated tests
Hi, [too bad it's too late to rename this thread, now that there have been 6 replies, to it.] anonym wrote (21 Oct 2015 12:46:37 GMT) : >> FTR I dislike the idea of blacklisting such branches based on their >> name. I'm not going to debate it here [...] > Please do it here, then! :) I guess it's related to that you want at > least doc/ branches to be tested, since our automated test suite > actually depend on the documentation (see two quotes below for more on > this). My concerns about skipping branches based on their name are essentially the same as the ones I've just posted in another sub-thread of this one (https://mailman.boum.org/pipermail/tails-dev/2015-December/009836.html). Besides, I think they are addressed by the content-based filtering (as opposed to name-based filtering) that seems to be the most current proposal in this thread, so no need for me to elaborate about it now. > Ok, let's forget about this now and return to the "workaround", i.e. > skipping tests depending on branch name: >> Still, this idea is probably OK as a short-term workaround if we need >> it for some reason, but then we need a clear description of the >> problem, > The problem, as I see it, is: we run the full automated test suite for > doc/ and web/ branches, which wastes a lot of valuable isotester time > given that all but a few tests are completely orthogonal to changes in > the wiki sub directory of Tails Git sources. About isotester time, I computed some stats in order to make my opinion a bit better informed: in November, among 314 non-aborted ISO test suite runs, 24% (75) were about branches whose name starts with "doc" or "web". So yes, that's wasting a lot of resources. It will probably be a problem, we'll see on #9264, so all the work put into this thread is definitely useful :) It seems that the proposed solution also addresses the problem that was raised initially, which was rather about latency for testing non-doc branches, so it sounds like we can avoid introducing clever stuff like prioritization + quiet time for a bit longer (#9760 will likely need it anyway, though). Yay! >> [...] making sure that the workaround is not worst than the problem >> it fixes > The only effect should be that we won't get automated tests of the few > scenarios that looks at the wiki shipped inside Tails. I think that > currently is one scenario: The Report an Error launcher will open the > support documentation in supported non-English locales. > So, if some wiki > change makes the 'the support documentation page opens in Tor Browser' > step fail. The only way that can happen is if: > [...] > Only in the first case would our test suite find an actual error, the > other two just implies that our test is now out of sync and must be > updated. Clearly it's not worth wasting that much isotester time on this > at the moment. Thanks for gathering all the data I need to be convinced :) I'll comment on this topic, about the actual proposed solution, later on this thread if needed. Cheers, -- intrigeri ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Releasing automated tests
Hi, sajolida wrote (22 Oct 2015 12:19:04 GMT) : >>> FTR I dislike the idea of blacklisting such branches based on their >>> name. I'm not going to debate it here [...] > The prefixes doc/ and web/ are used loosely to differentiate work on the > "documentation" (/doc /support) and the "website" in general (structure, > stuff not in /doc, etc.) but the difference is not strict. Indeed. I've also seen the "contrib/", "about/" and "faq/" prefixes being used. Branch naming is quite arbitrary, and sometimes creative, so yay, instead of relying on a hard-coded list of prefixes: > I also don't think they should be tested. Maybe you could exclude them > from testing by their diff to their base branch. If all the diff is > under wiki/src then don't test that branch. I like the idea! Cheers, -- intrigeri ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Releasing automated tests
Hi, anonym wrote (28 Oct 2015 16:52:31 GMT) : > Then I think we can combine the "..." operator with another fancy Git > feature I recently found, namely Git pathspec "magic signatures". So we > could do: >BASE_BRANCH_DIFF="$(git diff $base_branch...$commit -- \ >'*' \ >':!/wiki' \ >':!/ikiwiki.setup' \ >':!/ikiwiki-cgi.setup')" >if [ -z "${BASE_BRANCH_DIFF}" ]; then >CUCUMBER_ARGS="${CUCUMBER_ARGS} --tag @doc" >fi > where $commit is the commit we test, before merging the base branch > locally. Interesting! So, it seems that the current proposal is to skip the temporary workaround (name-based filtering), and instead directly do content-based filtering. Fine by me! What's missing is: * the list of scenarios that shall be tagged @doc: at least the one anonym mentioned earlier in this thread; plus "check all PO files" I guess; the WhisperBack tests once they have been written; more? * some process to make sure we tag scenarios @doc whenever relevant; I think we'll forget from time to time, and it's fine, because we'll be adding such scenarios very rarely; so perhaps a yearly check by the RM (e.g. 1st release of the year) is good enough? * and finally, some Redmine tickets describing this decision and the timeline of its implementation. bertagaz: once we have all that, please reassign #10492 to me for review. Cheers, -- intrigeri ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Releasing automated tests
intrigeri: > sajolida wrote (22 Oct 2015 12:19:04 GMT) : FTR I dislike the idea of blacklisting such branches based on their name. I'm not going to debate it here [...] > >> The prefixes doc/ and web/ are used loosely to differentiate work on the >> "documentation" (/doc /support) and the "website" in general (structure, >> stuff not in /doc, etc.) but the difference is not strict. > > Indeed. I've also seen the "contrib/", "about/" and "faq/" prefixes > being used. Branch naming is quite arbitrary, and sometimes creative, > so yay, instead of relying on a hard-coded list of prefixes: Yes, amongst writers and translators pretty much everybody has their own nomenclature :) >> I also don't think they should be tested. Maybe you could exclude them >> from testing by their diff to their base branch. If all the diff is >> under wiki/src then don't test that branch. > > I like the idea! I'll be very happy to receive less Jenkins alerts as well, so I can concentrate on the few relevant ones. I don't think I need to follow-up on anything else in this thread. Please tell me if I should. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Releasing automated tests
Hi, On Tue, Dec 01, 2015 at 09:28:13PM +0100, intrigeri wrote: > > So, it seems that the current proposal is to skip the temporary > workaround (name-based filtering), and instead directly do > content-based filtering. Fine by me! Yay! Happy to get a consensus. > What's missing is: > > * the list of scenarios that shall be tagged @doc: at least the one >anonym mentioned earlier in this thread; plus "check all PO files" >I guess; the WhisperBack tests once they have been written; more? I've added a ticket (#10706) and a branch containing what I've found to be relevant to be tagged as such. > * some process to make sure we tag scenarios @doc whenever relevant; >I think we'll forget from time to time, and it's fine, because >we'll be adding such scenarios very rarely; so perhaps a yearly >check by the RM (e.g. 1st release of the year) is good enough? Added that to the RM role in commit 4b176e2. > * and finally, some Redmine tickets describing this decision and the >timeline of its implementation. I've also added #10707 which is about the actual implementation of this feature. It has a dedicated branch with a commit doing so. I don't see what other tickets we may want to add for this. > bertagaz: once we have all that, please reassign #10492 to me for > review. Done. bert. ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.
Re: [Tails-dev] Releasing automated tests
Hi, bertagaz wrote (03 Dec 2015 15:40:09 GMT) : > On Tue, Dec 01, 2015 at 09:28:13PM +0100, intrigeri wrote: >> * the list of scenarios that shall be tagged @doc: at least the one >>anonym mentioned earlier in this thread; plus "check all PO files" >>I guess; the WhisperBack tests once they have been written; more? > I've added a ticket (#10706) and a branch containing what I've found to > be relevant to be tagged as such. Cool! Merged, and added a note on #10270 :) >> * some process to make sure we tag scenarios @doc whenever relevant; >>I think we'll forget from time to time, and it's fine, because >>we'll be adding such scenarios very rarely; so perhaps a yearly >>check by the RM (e.g. 1st release of the year) is good enough? > Added that to the RM role in commit 4b176e2. Thanks. I felt that the room for interpretation (and for skipping this task forever) the phrasing left was way too big so I made it much smaller ⇒ please review recent changes I did on that page. >> * and finally, some Redmine tickets describing this decision and the >>timeline of its implementation. > I've also added #10707 which is about the actual implementation of this > feature. It has a dedicated branch with a commit doing so. Replied on the ticket, let's follow-up there. > I don't see what other tickets we may want to add for this. OK (⇒ closed #10492! :) Cheers, -- intrigeri ___ Tails-dev mailing list Tails-dev@boum.org https://mailman.boum.org/listinfo/tails-dev To unsubscribe from this list, send an empty email to tails-dev-unsubscr...@boum.org.