On Wednesday, May 16, 2018 at 5:16:56 PM UTC+1, Jesse Glick wrote: > > My response would mostly follow Daniel’s but I will add more > background—something I can write up into a subsection of > > > https://github.com/jenkinsci/jep/blob/master/jep/305/README.adoc#other-version-schemes > > > once the dust settles. > > James Nord & Stephen Connolly wrote: > > as it stands in JEP-305 the versions are > > releases and these are expected to be garbage collected. > > Technically this is from IEP-9, but as that attracted little notice > (indeed I only stumbled across it after it was finalized and > implemented), it makes sense to discuss in the context of JEP-305. > > So this is the big sticking point and what Stephen and I are mostly concerned about. If that little "gem" was to be removed then I think all other issues (except for disk usage!) would just fall by the wayside (and you can skip the rest of this message and get some time back to <insert favourite passtime here>)
> > Anyone also wanting to consume incrementals via a proxy would need to > write > > tooling to garbage collect releases > > I think GC is the crux of the question. Yes it is. > For the record, I would have > recommended a rather more conservative collection schedule in IEP-9 > (basically keep everything until storage costs start to raise > eyebrows); I am not actually sure whether the collection scheme was > even implemented, or if so, whether this is custom tooling or some > stock feature of Artifactory. Daniel could speak to that. > > There are several things to consider about GC, and none have anything > to do with whether the version number happens to end in `-SNAPSHOT` or > not, so adding that suffix solves no practical problem. > Actually it does, as SNAPSHOTs are expected be be garbage collected, releases are not, and that is fundamental to the Maven philosophy. First, how likely is it that a newly run build of a downstream > component will refer to an incremental version that has been expired > from Artifactory? This is hard to know until JEP-305 gets broader > usage, but my expectation would be that new builds will usually refer > to pretty recent dependencies, especially if something like > JENKINS-47498 is adopted so that merely picking the latest parent POM > for a component will switch `<dependencyManagement>` to recent > `master` commits (or MRP-tagged releases). There will always be some > cases when an old line of development is suddenly resurrected and the > developer declines to start by refreshing dependencies: > > mvn incrementals:update > > Second, what is the impact of expiration? For builds on ci.jenkins.io, > possibly none: even though local repositories will typically start > empty, the build will download incremental releases from > > https://repo.azure.jenkins.io/incrementals/ > > which (Olivier correct me if I am mistaken) uses an LRU cache. > Therefore even an old dependency will remain in the cache indefinitely > if it continues to be used actively in builds. You could still get a > broken build in cases like that mentioned above, where the old > dependency is used by a hitherto-dormant branch of some downstream > component that has not gotten a refresh. > Ahh so now you have the debuging environments when things behave a little differently. Ie a unit test fails on CI, and you can not even build the thing locally anymore. This is not a good thing for a CI server, and you have now replaced "it works on my machine" with "well it works on CI" Third, can the release be reconstructed? For use from local developer > builds, pretty easily: just check out the specified commit of the > upstream component and > > mvn -DskipTests -Dset.changelist clean install > Alas I am Skeptical about this. For exapmple say I depended on BlueOcean. If I tried to check that out and build it I am left with something that is conmpletely dead on arival as it produces garbage on windows.And the argument that the CI also does a build on windows doesn't mean that that build is any good, BO builds on windows, and the unit tests pass, but the result is garbage, so unless you run the ATH on windows and Linux (which you don't) I actually hit a bug on a closed source plugin yesterday that the build failed on windows (that dreaded npm..) You may also at points be able to replace Windows with Mac or some Linux (or even a specific version of the JDK - which has also caused builds to break in the past, or even maven). incrementals does not fix the environment so reproducability on a different environment is never guaranteed (and hence why you should not mess with releases because whilst you hope the build is reproducable you are probably not recording everything you need to know about the env in order to guarantee it) > For use from CI builds, currently there is no mechanism in place to > re`deploy` old incremental releases on demand, such as jitpack.io > provides. One idea however is that we could copy back artifacts still > present in the Azure mirror, or alternately decline to GC artifacts > known to be in that mirror. We could also scan for incremental release > versions textually mentioned in reference heads of various > repositories (akin to “conservative” GC in some programming > languages). > or just not delete them... > Either way, incremental releases are if anything _safer_ to recreate > from sources than MRP-tagged releases: a tag is easily overwritten, or > just lost during an incomplete `git-clone`, whereas a 12-digit commit > hash prefix is forgeable only with deliberate malice. Is the CI the only place that can deploy them? (I don't recall but it may well be). If building an incremental from a PR then the hash is likely not enough (as that may/will not be pushed back upstream) > Note that > incremental releases always deploy a flattened POM containing an > `<scm>` section with a valid `<url>` and `<tag>` (actually full commit > hash, not technically a “tag”, but then neither is `HEAD`). > > > the impact this will have > > on other pipelines that do canary builds by picking up the latest > *release* > > of a artifacts/plugins > > Do not run `mvn versions:use-latest-releases` if your build > configuration refers to the official Incrementals repository or a > mirror thereof. The problem is when it is consumed in mirrors downstream. Many people will just have a corporate single URL and many people (maven repo admins) when asked to add a mirror of a upstream release will just add it in to the main mirror - which will as you say likely break builds that are already using tooling. > Use `incrementals:update` if you wish to get > incremental updates (this is more nuanced—pays attention to Git > topology), or just omit this repository if you do not. (Currently > governed by the presence of `-Pconsume-incrementals` in > `.mvn/maven.config`.) > > > in order to do bleeding edge testing in our > > pipelines > > Of course a reasonable interpretation of “bleeding-edge testing” is > that you _do_ want incremental versions! > for some definitions of bleeding edge... think of a slightly less bleeding edge if you will. > > the ability once it is in our mirror > > for things to accidentally depend on these releases. > > JENKINS-50804 if you are concerned. The most obvious harm would be if > a tagged release of a downstream plugin were to express a dependency > on an incremental release of an upstream plugin, since the resulting > update center would then be internally inconsistent if it were built > from the Nexus index of the traditional release repository. Of course > this can already happen quite easily with upstream release versions > marked `-alpha` or `-beta`, or in cases where the upstream plugin is > excluded from the UC. The best solution is for the UC generation code > to actually perform a consistency check before publishing > anything—that seems like the very least we could do to prevent > accidents. > > > we believe that you can accomplish this with > > SNAPSHOTS. > > You do not need to use timestamped snapshots (which is the main > objection in > > JEP-305 against SNAPSHOTS) > > Timestamped snapshots are indeed especially problematic, but snapshots > generally do not match the use case for Incrementals. Besides the > practical annoyance of Maven checking daily for updates which should > not be there (we forbid redeployment), philosophically it does not > follow what I think should be the distinction between snapshots and > releases. > You can configure maven to check as you want on a per repo basis. ie in the repo definition in you pom use never (same as releases) so only new versions are looked for. in a mirror nexus (I would also guess artifactory etc) you can also set the check time (and again set it to have an infinite not found cache. > A release is something which is reconstructible from a specific > commit. It is a globally identifiable version of that component which > you may in turn refer to in other commits and be assured that other > developers or CI systems will not encounter ambiguity. You can create > permalinks to its sources. > Nah. this is FUD. I can create a release can call it whatever I want and deploy it (subject to access control on the repo server). if you make it such that only the CI can deploy these so you can ensure that your above statement is true, then you can by inference also do it for SNAPSHOTs. A snapshot might be built from a commit, or a commit with some local > modifications, or even from an unversioned source tree—we do not know > or care. You may refer to a `*-SNAPSHOT` version temporarily in the > course of testing some changes locally, but it should never be > deployed to a public server, nor should such a version ever be > mentioned in a committed POM/manifest/whatever. (Just like you should > never refer to `LATEST`/`RELEASE` or a branch name.) > > An incremental release is just a kind of release that is friendlier to > CI/CD. It is marked as a “release candidate” since every commit is > indeed a candidate for deployment. The selection of which versions are > actually used in production after running a gantlet of tests is just > left to a different layer of tooling, such as a versioned bill of > materials, rather than the awkward dance of MRP and (worse) staging > repositories. > Erm, so if you are making these release canditates pass the full suite, you are then using them in production via promotion, but the version in them is still a release candidate? the above is just confusing? -- You received this message because you are subscribed to the Google Groups "Jenkins Developers" group. To unsubscribe from this group and stop receiving emails from it, send an email to jenkinsci-dev+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/jenkinsci-dev/4a21e6d6-4df0-489b-ba62-efa198ff865c%40googlegroups.com. For more options, visit https://groups.google.com/d/optout.
