Re: Thinktank: CI's, compiler lists, and project automation
On Friday, 2 March 2018 at 06:07:25 UTC, Nick Sabalausky (Abscissa) wrote: (I'm posting this here instead of D.learn because it isn't really a "How to do XYZ in D?", but rather to invite discussion on high-level solutions to a problem.) Here's a common problem: 1. A project (ex, some library) uses travis-ci (and/or other) to ensure compatibility with a range of compiler versions. The travis configuration file includes a manually-curated list of compiler versions. 2. CI tests are triggered by new commits/merges/PRs in the project. 3. New compiler versions are released. Well, just for the record - imho adding your project to the project tester is the best way to prevent regressions a priori: add your project to the Project Tester (ci.dlang.io). The Project Tester runs the testsuite of currently ~40 packages on every PR at dmd, druntime, phobos, tools and dub An example: https://ci.dlang.io/blue/organizations/jenkins/dlang-org%2Fphobos/detail/PR-6259/2/pipeline Imho it's te best way as a regression is caught _before_ the change is merged and the need for the interaction is on the side of the person who would introduce the regression (i.e. the one understanding the change). New projects can be added either by making a PR to the file to [1] or pinging me. [1] https://github.com/dlang/ci/blob/master/vars/runPipeline.groovy but there are still limitations and downsides (ex: D tester has limited resources FWIW it's not so limited and highly parallizable. It's already configured to automatically create new temporary worker machines in the Google Cloud [2] whenever there's more traffic/demand. Considering the upsides of knowing that your project can _never ever_break again is so huge that imho even if more workers are needed, it's well-justified as there's quite a net gain. [2] It currently uses the cheap preemptible VM machines in the Google Cloud (https://cloud.google.com/compute/docs/instances/preemptible), but Martin plans to migrate to Hetzner soon as there are too many problems with those VMs.
Re: Thinktank: CI's, compiler lists, and project automation
On 03/04/2018 03:42 AM, Jacob Carlborg wrote: Aha, you mean like that. Seems a bit difficult to fix. Perhaps specifying a range of compiler versions would do? Yea, there's really no two ways around it: Ultimately, each new compiler release will need to get added to .travis.yml (or any other CI's equivalent) either sooner or later. (Unless the project eschews anything beyond "latest versions ONLY", but that comes with its own downsides, especially for libraries.) Luckily, this is definitely automatable, at least as far as auto-submitted PRs, if nothing else. The devil is just in the details.
Re: Thinktank: CI's, compiler lists, and project automation
On 03/05/2018 02:42 AM, Nick Sabalausky (Abscissa) wrote: On 03/04/2018 02:05 AM, Nick Sabalausky (Abscissa) wrote: On 03/03/2018 10:22 AM, Jacob Carlborg wrote: You should get an email notification about failed builds [1] [2], Weird. Never worked for me. Will have to check into that. Ahh, it looks like travis disregards the github email address (perhaps it can't access it, or indirectly tell github to email me?), and the only way to receive travis email notifications is to expose the desired email address to the general public via either .travis.yml or the git commit author. (Un)fortunately, I have a far more effective, reliable method of filtering spam than the "state of the art" gmail heuristics: I run a mail server, create domain-specific email addresses, and delete any accounts that get spammed (knowing full well who it was that leaked my email). I was never stupid enough to configure git with a legitimate email address I actually check...I have better things to do than fight spam and regularly check some "spam folder". Travis (not to mention Git itself) doesn't appear to support the post-1980 reality of email, ie, "Publicly-posted addressed WILL be spammed, hard"), thus, even though I get GitHub notifications just fine, I haven't been getting notifications from travis even though its linked to my github account. Hooray "web 2.0" :/
Re: Thinktank: CI's, compiler lists, and project automation
On 03/04/2018 02:05 AM, Nick Sabalausky (Abscissa) wrote: On 03/03/2018 10:22 AM, Jacob Carlborg wrote: You should get an email notification about failed builds [1] [2], Weird. Never worked for me. Will have to check into that. Ahh, it looks like travis disregards the github email address (perhaps it can't access it, or indirectly tell github to email me?), and the only way to receive travis email notifications is to expose the desired email address to the general public via either .travis.yml or the git commit author. (Un)fortunately, I have a far more effective, reliable method of filtering spam than the "state of the art" gmail heuristics: I run a mail server, create domain-specific email addresses, and delete any accounts that get spammed (knowing full well who it was that leaked my email). I was never stupid enough to configure git with a legitimate email address I actually check...I have better things to do than fight spam and regularly check some "spam folder". Travis (not to mention Git itself) doesn't appear to support the post-1980 reality of email, ie, "Publicly-posted addressed WILL be spammed, hard"), thus, even though I get GitHub notifications just fine, I haven't been getting notifications from travis even though its linked to my github account.
Re: Thinktank: CI's, compiler lists, and project automation
On 2018-03-04 08:05, Nick Sabalausky (Abscissa) wrote: 3. Release of DMD 2.081.0 occurs. "dmd" now points to 2.081.0. Unless .travis.yml has been manually updated, it still contains: - dmd # Ie, dmd-2.081.0, now # WTF?!?! dmd-2.080.0 is no longer being tested - dmd-2.079.0 - dmd-2.078.2 - dmd-2.078.1 - dmd-2.078.0 - dmd-2.077.1 - dmd-2.077.0 Summary: CI *stops* testing dmd-2.080.0. FAIL. Aha, you mean like that. Seems a bit difficult to fix. Perhaps specifying a range of compiler versions would do? -- /Jacob Carlborg
Re: Thinktank: CI's, compiler lists, and project automation
On 03/04/2018 02:05 AM, Nick Sabalausky (Abscissa) wrote: 1. Suppose a library supports DMD v2.071.0 and up. Because of this, .travis.yml includes: - dmd # Ie, dmd-2.079.0, ATM - dmd-2.079.0 - dmd-2.078.2 - dmd-2.078.1 - dmd-2.078.0 - dmd-2.077.1 - dmd-2.077.0 Summary: CI now tests 2.076.1 through 2.079.0. Oops, proofread failures. Naturally, I meant this (fixes emphasized): - 1. Suppose a library supports ***DMD v2.077.0*** and up. Because of this, .travis.yml includes: - dmd # Ie, dmd-2.079.0, ATM - dmd-2.079.0 - dmd-2.078.2 - dmd-2.078.1 - dmd-2.078.0 - dmd-2.077.1 - dmd-2.077.0 Summary: CI now tests ***2.077.1*** through 2.079.0.
Re: Thinktank: CI's, compiler lists, and project automation
On 03/04/2018 02:23 AM, Nick Sabalausky (Abscissa) wrote: Oops, proofread failures. Naturally, I meant this (fixes emphasized): - 1. Suppose a library supports ***DMD v2.077.0*** and up. Because of this, .travis.yml includes: - dmd # Ie, dmd-2.079.0, ATM - dmd-2.079.0 - dmd-2.078.2 - dmd-2.078.1 - dmd-2.078.0 - dmd-2.077.1 - dmd-2.077.0 Summary: CI now tests ***2.077.1*** through 2.079.0. Summary: CI now tests ***2.077.0*** through 2.079.0. (Argh, maybe it's bedtime... ;))
Re: Thinktank: CI's, compiler lists, and project automation
On 03/04/2018 02:05 AM, Nick Sabalausky (Abscissa) wrote: No. That is only sufficient *temporarily*, that is, until the next time the label "dmd" is updated to another new release once again. At *some* point, .travis.yml will still need to be manually updated... Scenario 1-2-3: 1. Suppose a library supports DMD v2.071.0 and up. Because of this, .travis.yml includes: - dmd # Ie, dmd-2.079.0, ATM - dmd-2.079.0 - dmd-2.078.2 - dmd-2.078.1 - dmd-2.078.0 - dmd-2.077.1 - dmd-2.077.0 Summary: CI now tests 2.076.1 through 2.079.0. 2. Release of DMD 2.080.0 occurs. "dmd" now points to 2.080.0 (and forget reproducibility: if 2.080.0 contains an unexpected regression then using travis's "Restart Job" on the "dmd" job will break overall reproducibility.) Summary: CI now tests 2.076.1 through 2.080.0. So far so good, right? Until... 3. Release of DMD 2.081.0 occurs. "dmd" now points to 2.081.0. Unless .travis.yml has been manually updated, it still contains: - dmd # Ie, dmd-2.081.0, now # WTF?!?! dmd-2.080.0 is no longer being tested - dmd-2.079.0 - dmd-2.078.2 - dmd-2.078.1 - dmd-2.078.0 - dmd-2.077.1 - dmd-2.077.0 Summary: CI *stops* testing dmd-2.080.0. FAIL. Additionally, what happens when said project above doesn't receive commits/merges between steps 1 and 3 (ie, during the time period where 2.080.0 is the latest release)? Any incompatibilities with 2.080.0 go completely unnoticed because CI never ends up testing with 2.080.0. The "dmd" label does NOT solve the problem.
Re: Thinktank: CI's, compiler lists, and project automation
On 03/03/2018 10:22 AM, Jacob Carlborg wrote: On 2018-03-03 07:39, Nick Sabalausky (Abscissa) wrote: - Maybe there's a simple setting I've overlooked, but when a build job fails on travis, the author does not get proactively notified. The author only finds out next time they go into travis. (I've been surprised many times to discover failed builds that had occured several days ago, or more.) You should get an email notification about failed builds [1] [2], Weird. Never worked for me. Will have to check into that. - A project author will still need to: 1. Actively notice new compiler releases and 2. Manually update the .travis.yml files for each of their projects. Certainly there's room for more automation here. No. If you specify "dmd" (without any version) in the list of compilers [5], it will build the latest version. For example, this build [3], which ran 23 hours ago uses DMD 2.078.3, while this one [4], 29 days ago, uses 2.078.2. As you can see on the commit in Travis CI, it's the same commit. No. That is only sufficient *temporarily*, that is, until the next time the label "dmd" is updated to another new release once again. At *some* point, .travis.yml will still need to be manually updated... Scenario 1-2-3: 1. Suppose a library supports DMD v2.071.0 and up. Because of this, .travis.yml includes: - dmd # Ie, dmd-2.079.0, ATM - dmd-2.079.0 - dmd-2.078.2 - dmd-2.078.1 - dmd-2.078.0 - dmd-2.077.1 - dmd-2.077.0 Summary: CI now tests 2.076.1 through 2.079.0. 2. Release of DMD 2.080.0 occurs. "dmd" now points to 2.080.0 (and forget reproducibility: if 2.080.0 contains an unexpected regression then using travis's "Restart Job" on the "dmd" job will break overall reproducibility.) Summary: CI now tests 2.076.1 through 2.080.0. So far so good, right? Until... 3. Release of DMD 2.081.0 occurs. "dmd" now points to 2.081.0. Unless .travis.yml has been manually updated, it still contains: - dmd # Ie, dmd-2.081.0, now # WTF?!?! dmd-2.080.0 is no longer being tested - dmd-2.079.0 - dmd-2.078.2 - dmd-2.078.1 - dmd-2.078.0 - dmd-2.077.1 - dmd-2.077.0 Summary: CI *stops* testing dmd-2.080.0. FAIL.
Re: Thinktank: CI's, compiler lists, and project automation
On 2018-03-03 07:39, Nick Sabalausky (Abscissa) wrote: Certainly a possible approach, but has downsides: - Maybe there's a simple setting I've overlooked, but when a build job fails on travis, the author does not get proactively notified. The author only finds out next time they go into travis. (I've been surprised many times to discover failed builds that had occured several days ago, or more.) You should get an email notification about failed builds [1] [2], - A project author will still need to: 1. Actively notice new compiler releases and 2. Manually update the .travis.yml files for each of their projects. Certainly there's room for more automation here. No. If you specify "dmd" (without any version) in the list of compilers [5], it will build the latest version. For example, this build [3], which ran 23 hours ago uses DMD 2.078.3, while this one [4], 29 days ago, uses 2.078.2. As you can see on the commit in Travis CI, it's the same commit. - Except when "nightly" is desired, this leads to many unnecessary redundant builds/tests, especially across all the various D projects. (Though I don't know how much that would matter on travis. Maybe it'd be a drop in their bucket.) [1] https://docs.travis-ci.com/user/notifications/#Default-notification-settings [2] https://docs.travis-ci.com/user/notifications/#Missing-build-notifications [3] https://travis-ci.org/jacob-carlborg/dstep/jobs/348275710 [4] https://travis-ci.org/jacob-carlborg/dstep/jobs/340527251 [5] https://github.com/jacob-carlborg/dstep/blob/master/.travis.yml#L4 -- /Jacob Carlborg
Re: Thinktank: CI's, compiler lists, and project automation
On Saturday, 3 March 2018 at 06:39:42 UTC, Nick Sabalausky (Abscissa) wrote: On 03/02/2018 02:24 AM, Jacob Carlborg wrote: Travis CI allows you to specify a D compiler in the following ways: * - the latest version of the specified compiler * -beta - the latest beta * -nightly - the nightly build * - - a specific version of the compiler Where is "dmd", "ldc" or "gdc". Combine the above with a scheduled build that builds once every day you're pretty well covered. For an example, see the Travis CI configuration for DStep [1]. I'm building the latest version of DMD and LDC on Linux and macOS, but the betas and nightly are only built on Linux, due the the limited resources of macOS builds on Travis CI. [1] https://github.com/jacob-carlborg/dstep/blob/master/.travis.yml Certainly a possible approach, but has downsides: - Maybe there's a simple setting I've overlooked, but when a build job fails on travis, the author does not get proactively notified. The author only finds out next time they go into travis. (I've been surprised many times to discover failed builds that had occured several days ago, or more.) I receive emails from them, after it broke, each time it builds and that the build is not repaired and finally a last one when the build is repaired. I've just checked and there is no settings to activate this. - A project author will still need to: 1. Actively notice new compiler releases and 2. Manually update the .travis.yml files for each of their projects. Certainly there's room for more automation here. - Except when "nightly" is desired, this leads to many unnecessary redundant builds/tests, especially across all the various D projects. (Though I don't know how much that would matter on travis. Maybe it'd be a drop in their bucket.) I agree. "Nightly" is useful but "beta" not, or rather _not always_. "beta" is useful when there's a new beta and until it get released (usually 2-3 weeks), after what this wastes computing resources.
Re: Thinktank: CI's, compiler lists, and project automation
On Fri, 02 Mar 2018 01:07:25 -0500, Nick Sabalausky (Abscissa) wrote: > Another sample point of discussion: One possible approach is to have a > bot generate PRs to update project's compiler lists. But that leads to > other questions: How/when is the bot triggered? On what machine does it > run? Is there an existing jumping-off point for creating such a bot? How > does it approach the task of modifying `.travis.yml` (how smart/dumb > does it need to be)? Any security concerns to be mindful of? Etc. Once I catch up on the other things I've said I'm going to do, I'm planning on doing something like pyup-bot[1] for Dub projects. pyup does this with project dependencies for python - you get a pull request with dependency updates, links to their changelog/documentation, etc. Adding compiler updates might make sense, as the compiler is a dependency. Configuration would need to allow for the various decisions people make concerning what versions they support, so it's not a trivial addition, but doable. --Ryan [1]: pyup.io/
Re: Thinktank: CI's, compiler lists, and project automation
On 03/02/2018 02:24 AM, Jacob Carlborg wrote: Travis CI allows you to specify a D compiler in the following ways: * - the latest version of the specified compiler * -beta - the latest beta * -nightly - the nightly build * - - a specific version of the compiler Where is "dmd", "ldc" or "gdc". Combine the above with a scheduled build that builds once every day you're pretty well covered. For an example, see the Travis CI configuration for DStep [1]. I'm building the latest version of DMD and LDC on Linux and macOS, but the betas and nightly are only built on Linux, due the the limited resources of macOS builds on Travis CI. [1] https://github.com/jacob-carlborg/dstep/blob/master/.travis.yml Certainly a possible approach, but has downsides: - Maybe there's a simple setting I've overlooked, but when a build job fails on travis, the author does not get proactively notified. The author only finds out next time they go into travis. (I've been surprised many times to discover failed builds that had occured several days ago, or more.) - A project author will still need to: 1. Actively notice new compiler releases and 2. Manually update the .travis.yml files for each of their projects. Certainly there's room for more automation here. - Except when "nightly" is desired, this leads to many unnecessary redundant builds/tests, especially across all the various D projects. (Though I don't know how much that would matter on travis. Maybe it'd be a drop in their bucket.)
Re: Thinktank: CI's, compiler lists, and project automation
On Friday, 2 March 2018 at 06:07:25 UTC, Nick Sabalausky (Abscissa) wrote: (I'm posting this here instead of D.learn because it isn't really a "How to do XYZ in D?", but rather to invite discussion on high-level solutions to a problem.) Here's a common problem: 1. A project (ex, some library) uses travis-ci (and/or other) to ensure compatibility with a range of compiler versions. The travis configuration file includes a manually-curated list of compiler versions. 2. CI tests are triggered by new commits/merges/PRs in the project. 3. New compiler versions are released. 4. Compiler updates may or may not trigger new tests with the new compiler, depending on the project. 5. Project maintainer must manually update the list of compilers. (And this isn't one task, but rather O(n), where n=number of projects the given person is maintaining). The challenge: What is the best way to automate this? (ie, #4 and #5) Travis CI allows you to specify a D compiler in the following ways: * - the latest version of the specified compiler * -beta - the latest beta * -nightly - the nightly build * - - a specific version of the compiler Where is "dmd", "ldc" or "gdc". Combine the above with a scheduled build that builds once every day you're pretty well covered. For an example, see the Travis CI configuration for DStep [1]. I'm building the latest version of DMD and LDC on Linux and macOS, but the betas and nightly are only built on Linux, due the the limited resources of macOS builds on Travis CI. [1] https://github.com/jacob-carlborg/dstep/blob/master/.travis.yml -- /Jacob Carlborg
Thinktank: CI's, compiler lists, and project automation
(I'm posting this here instead of D.learn because it isn't really a "How to do XYZ in D?", but rather to invite discussion on high-level solutions to a problem.) Here's a common problem: 1. A project (ex, some library) uses travis-ci (and/or other) to ensure compatibility with a range of compiler versions. The travis configuration file includes a manually-curated list of compiler versions. 2. CI tests are triggered by new commits/merges/PRs in the project. 3. New compiler versions are released. 4. Compiler updates may or may not trigger new tests with the new compiler, depending on the project. 5. Project maintainer must manually update the list of compilers. (And this isn't one task, but rather O(n), where n=number of projects the given person is maintaining). The challenge: What is the best way to automate this? (ie, #4 and #5) We do already have certain approaches to at least certain facets of the problem (ex: some third party libs are included in DMD's CI testing), but there are still limitations and downsides (ex: D tester has limited resources, and it doesn't address manually updating the travis config's list of compilers or notifying authors their project's compiler list needs updated.) Another sample point of discussion: One possible approach is to have a bot generate PRs to update project's compiler lists. But that leads to other questions: How/when is the bot triggered? On what machine does it run? Is there an existing jumping-off point for creating such a bot? How does it approach the task of modifying `.travis.yml` (how smart/dumb does it need to be)? Any security concerns to be mindful of? Etc.
