> Bin compatibility checks for 1.1.0 can be done against 1.0.x jars like we already do today.
The problem is not whether it can be done, the problem is that its a manual process that if not done can introduce binary incompatible changes which actually already happened in the past, i.e. we forgot to update the latest binary version that MIMA checks when 1.0.x branch happened which means that theoretically speaking someone could have introduced bincompat changes. Just in general, there is a reason why https://github.com/sbt/sbt-dynver#previous-version-detection exists, it's because of previous learnings by OS projects that doing things manually greatly increases the chance of problems seeping through. > I don't really like the idea of having milestone releases and all the extra voting and uptake work that this entails. I don't like having the do a vote either, but the ASF process mandates that even though the milestone is in essence not a release it still needs to be voted on > I am even less enthusiastic about a 1.1.0-M0 release that has no meaningful 1.1.0 changes and that is basically a copy of the 1.0.1 release. Then we shouldn't be creating the `-M0` tag when the major is bumped because we either are a milestone or not, and whether binaries come from that is an irrelevant detail. That's like saying snapshots of -M0 after a branch are useless because they contain no meaningful changes and the reason we put the tag there in the first place is because of the snapshots in the first place. Also this is the reason why it's -M0 and not -M1, the 0 indicates that there are no changes, a hypothetical -M1 release indicates the first milestone release that actually has changes. In summary, if a tag exists there should be a binary version of it. On Wed, Aug 30, 2023 at 12:53 PM PJ Fanning <[email protected]> wrote: > I don't really like the idea of having milestone releases and all the extra > voting and uptake work that this entails. > > I am even less enthusiastic about a 1.1.0-M0 release that has no meaningful > 1.1.0 changes and that is basically a copy of the 1.0.1 release. > > Bin compatibility checks for 1.1.0 can be done against 1.0.x jars like we > already do today. > > On Wed 30 Aug 2023, 08:29 Matthew de Detrich, > <[email protected]> wrote: > > > So I want to revive this topic now that the storm has settled a bit and > the > > modules > > which the community wants released are now underway. > > > > I still think that this idea has merit for the reasons I stated (which I > > don't want to repeat) > > so I think going forward its just to do a vote for these milestone jar's. > > > > @PJ Fanning how do you feel about creating a milestone -M0 jar for pekko > > core right at > > the point when the branching happened, creating an ASF release for the > > process (I > > don't imagine there should be any issues here) so we can at least unblock > > https://github.com/apache/incubator-pekko/pull/532 . > > > > We can also create another milestone in Pekko around now for downstream > > projects > > such as pekko-http to depend on, as there will be some significant > changes > > later on > > so I think that right now marks a good point in time. > > > > > > On Mon, Aug 7, 2023 at 5:00 PM Matthew de Detrich < > > [email protected]> wrote: > > > > > So interestingly it's actually possible to have a mono-repo which > > contains > > > all of the pekko projects as > > > they are now defined with each project being a sbt module within its > own > > > sub-folder AND each of those > > > sbt modules having their own version lifecycle. Furthermore it should > > also > > > be possible for those sub > > > folders to be loaded independently just by cd'ing into the folder. > > > > > > If its acceptable for Apache in the source distribution to only > provide a > > > subset of the sources which > > > are relevant to the module being released at the time, given what was > > said > > > earlier about each project > > > sub folder being able to loaded on its own this compromise may be able > to > > > give us the best of both > > > worlds. > > > > > > On Mon, Aug 7, 2023 at 3:29 PM PJ Fanning <[email protected]> > wrote: > > > > > >> One radical solution that we could consider is starting to merge repos > > >> like pekko-http into the main pekko repo. I know this has drawbacks > but > > it > > >> actually has a few advantages. > > >> > > >> Cons > > >> * This main drawback is that we need to release all the modules in the > > >> main repo at the same time. If we need to fix one module, we need to > > >> release all the modules. Note that many of our repos already have lots > > of > > >> modules, so all that changes is the number of modules being released > at > > the > > >> same time. > > >> * We would need to refactor the CI so that we don't end up with > > >> individual jobs running too much. We don't want to end up with a build > > that > > >> requires hours to verify a PR. > > >> > > >> Pros > > >> * For users, it simplifies working out which versions of pekko modules > > to > > >> use. From the traffic in our online mailing lists and GitHub > > discussions - > > >> I think a lot of users are confused about why a module like > > >> pekko-connectors-s3 is released separately from pekko-stream and has a > > >> different version number. > > >> * For testing the modules and how they work together, all the jars can > > be > > >> built into mavenLocal. > > >> * No longer do we need to worry about how to get a pekko-stream > change > > >> tested in all the connectors > > >> * For release voting, voters will not to have to work out the specific > > >> circumstances of all the individual git repos that we have. > > >> > > >> This is a lot of work but it may be worth considering. > > >> > > >> > > >> On 2023/08/07 11:17:18 Matthew de Detrich wrote: > > >> > I made my case quite clear before, it breaks past builds if people > > want > > >> to > > >> > load > > >> > past versions of the project (and when I mean break I really do mean > > >> > break, as in the build won't even load when you check out the > source) > > >> > and additionally there are strong arguments that this is a misuse of > > >> > snapshots. > > >> > > > >> > The whole point of milestone is to have a persistent/permanent (this > > is > > >> > critical) > > >> > artifact that regardless of when someone resolves a build at a > > specific > > >> > point > > >> > in time they will always be able to load it whether it is now, 5 > > days, 5 > > >> > months or > > >> > 5 years. If the main reason why uses will want to be loading past > > >> builds to > > >> > try and > > >> > bisect some bug/issue in the past, it's imperative that they do > > actually > > >> > load the > > >> > code that is relevant back then, not what is published now (because > > >> > snapshots by > > >> > definition only maintain the latest X versions of an artifact's > > version > > >> Y). > > >> > Of course > > >> > there are ways around this, but this basically involves users having > > to > > >> > carefully > > >> > and transitively build all relevant dependencies in the dependency > > tree > > >> and > > >> > the > > >> > careful part here is significant because they need to build the > > project > > >> the > > >> > same > > >> > way it was built back then (think JDK versions here). > > >> > > > >> > This is why at least to me, using pinned snapshots as core > > dependencies > > >> > in Pekko modules is not negotiable from my side, I would much prefer > > >> > publishing milestones as part of an Apache's Release process or just > > >> > leave the core dependencies of Pekko as full release versions and > > create > > >> > a separate branch/CI matrix that dynamically builds the dependencies > > >> > from source and/or loads snapshots to detect regressions (but even > > >> > that has its own host of problems). > > >> > > > >> > The thing is none of the contributors have the capacity to do such > > >> extensive > > >> > changes, the whole point of the milstone solution is it is a trivial > > and > > >> > clean > > >> > solution to many problems (not just this one, bringing up > > >> > > > >> > > > https://github.com/apache/incubator-pekko/pull/532#issuecomment-1667527994 > > >> > again). It massively simplifies the code, makes it trivially easy > for > > >> pekko > > >> > contributors to understand what is going on (i.e. no complex > > dynamically > > >> > finding > > >> > out the latest snapshot as is done in > > >> pekko-http/pekko-persistence-dynamodb) > > >> > and doesn't have hair pulling gotchas like "why isn't this build > > >> loading, > > >> > oh great > > >> > the snapshot is deleted because it was pruned so now I have to > figure > > >> out > > >> > how > > >> > that snapshot was built and do it all myself". > > >> > > > >> > Other Apache projects can use snapshots in this way, and that's fine > > but > > >> > Pekko > > >> > already has enough issues with overburdening complexity and gotchas > > >> that is > > >> > making the project less approachable for contributors and using > pinned > > >> > snapshots in this way just makes it worse. > > >> > > > >> > On Mon, Aug 7, 2023 at 12:14 PM Claude Warren, Jr > > >> > <[email protected]> wrote: > > >> > > > >> > > Matthew, your argument makes no sense to me. Let me see if I can > > >> restate > > >> > > it to be sure I have the salient points. > > >> > > > > >> > > 1. There are two modules, let's call them "core" and "child". > > >> > > 2. There is a version where core and child work together. > Let's > > >> call > > >> > > this 1.0.0 for both. I realize they can be different, but I > want > > >> to > > >> > > keep > > >> > > this as simple as possible. > > >> > > 3. Now core develops a new version that has a change that may > > >> cause a > > >> > > regression. let's call this core-1.1.0-S1. > > >> > > 4. The problem is we want "child" to be able to test against > > >> > > core-1.1.0-S1 to verify that there are no regression issues. > So > > >> for the > > >> > > immediate problem:, > > >> > > 1. "child" can create a branch that tests against > > core-1.1.0-S1. > > >> > > Let's call this version child-1.1.0-regression (it has to be > > >> the next > > >> > > version of child because 1.0.0 is already released -- it > could > > >> > > be 1.0.1 but > > >> > > I'm not going there) > > >> > > 5. If there is a regression: > > >> > > 1. that information will flow back to core dev and > > >> core-1.1.0-S2 will > > >> > > be created > > >> > > 2. "child" can then test against that in > > child-1.1.0-regression. > > >> > > 3. In this case child-1.1.0-regression wants to test with > the > > >> latest > > >> > > core-1.1.0-Sx version. > > >> > > 6. If there was no regression there might be when core-1.1.0-S2 > > is > > >> > > published. > > >> > > 1. In this case child-1.1.0-regression should be run > > >> > > against core-1.1.0-S2 > > >> > > 2. again "child" wants to run against the latest > > >> "core-1.1.0-Sx". > > >> > > 7. For both of the above paths the standard core-1.1.0-SNAPSHOT > > >> fits the > > >> > > bill. > > >> > > 8. At some point "child" dev has to decide to go with > core-1.1.0 > > >> (after > > >> > > it is released) or core-1.0.0 but that is a decision for > "child" > > >> and > > >> > > either > > >> > > will work (assuming child has no dependency on functionality > > >> introduced > > >> > > in > > >> > > core-1.1.0). > > >> > > 9. After core-1.1.0 is released core-1.1.0-Sx is deleted from > the > > >> > > repository, because it is a snapshot. And this leads to the > > issue. > > >> > > 10. At some later time someone wants to checkout > > >> > > "child-1.1.0-regression" and have it build, but can not because > > >> > > "core-1.1.0-Sx" is no longer found. There are 2 cases here: > > >> > > > > >> > > > > >> > > 1. There is some issue being checked that does not involve the > > >> changes > > >> > > in core-1.1.0 at this moment in time, in which case > core-1.1.0 > > >> > > (release) > > >> > > can replace core-1.1.0-Sx. > > >> > > 2. There is some issue being checked with the regression > tests > > >> for > > >> > > core-1.1.0-Sx. This feels like a very low probability issue > > >> but, > > >> > > let's > > >> > > move forward with this discussion. > > >> > > 1. At this point, the developer can checkout the > core-1.1.0 > > >> code > > >> > > for the time frame in question and build it thus > providing > > >> the > > >> > > library > > >> > > necessary for the analysis they are performing. > > >> > > 1. Is this solution optimal - no. > > >> > > 2. Is it acceptable? In my opinion looking at > > >> cost/benefit, I > > >> > > think so. The probability that anyone will want to > > >> rebuild > > >> > > child-1.1.0-regression is low. In every case I can > > think > > >> of > > >> > > 1. working with child-1.1.0 is the better solution > as > > >> that > > >> > > is the code that was finally accepted to release, > or > > >> > > 2. if not released at least the accepted snapshot. > > >> > > 3. If development on "child" has stalled and > someone > > is > > >> > > picking it up again and the most advanced code is > in > > >> > > "child-1.1.0-regression" then they need to move the > > >> > > dependency to > > >> > > core-1.1.0 (if not later) and work forward from > > there. > > >> > > > > >> > > I make some assumptions here: > > >> > > > > >> > > - Snapshot versions are retained in the repository by count as > > >> well as > > >> > > date so the last 5 or last 5 days whichever is greater sort of > > >> thing. > > >> > > - Snapshot versions are deleted when a release is made. > > >> > > - I do not assume that child-1.1.0 will be released just > > >> > > because core-1.1.0 is released, the regression check was just > > that > > >> and > > >> > > does > > >> > > not imply the version of core that will be used in child-1.1.0. > > >> There > > >> > > is > > >> > > some argument to be made that the child regression check should > > be > > >> > > core-child-1.1.0-regression and that it should be the > > >> responsibility of > > >> > > the > > >> > > core developers to make the test. But that is neither here nor > > >> there as > > >> > > the test is assumed to have been made for purposes of this > > >> discussion. > > >> > > - There is nothing stopping the "child" team from preserving > any > > >> version > > >> > > of core-1.1.0-Sx that they wish to retain for testing or other > > >> uses, > > >> > > provided they don't release it. > > >> > > > > >> > > So if there is any disagreement please let me know which bullet > > number > > >> > > (and/or sub numbers) you disagree with and why. > > >> > > > > >> > > > > >> > > On Mon, Aug 7, 2023 at 8:49 AM Matthew de Detrich > > >> > > <[email protected]> wrote: > > >> > > > > >> > > > > I think that there is a communication probleem; crossed wires > in > > >> > > > terminology. Matthew talks about "general public" and most of > us > > >> grey > > >> > > > heads (trying not to be gender specific here) in the ASF hear > > >> "release". > > >> > > > What I think Matthew is talking about is a wider testing > audience. > > >> > > People > > >> > > > who understand that the code is not "released". If that is > > >> > > > indeed the case, and I am certain Matthew will tell me if I am > > >> wrong, > > >> > > then > > >> > > > the following should make sense. > > >> > > > > > >> > > > Yes this is completely correct > > >> > > > > > >> > > > > Use the snapshot repositories to their fullest. > > >> > > > > > >> > > > This is actually where the problem lies. We are already > deploying > > >> into > > >> > > > snapshots > > >> > > > (see > > >> > > > > > >> > > https://repository.apache.org/content/groups/snapshots/org/apache/pekko/ > > >> > > ). > > >> > > > The problem is that due to the way snapshots are being deployed > > >> currently > > >> > > > they > > >> > > > are not being pruned and INFRA has actually approached us saying > > >> that at > > >> > > > some point we need to fix this. In summary we cannot rely on > > >> snapshots > > >> > > > being > > >> > > > persistent, they are designed to be pruned. > > >> > > > > > >> > > > Now while this may be fine for a wider general testing audience, > > it > > >> does > > >> > > > pose > > >> > > > other issues if we were to use snapshots to solve other issues, > > i.e. > > >> > > > testing > > >> > > > merged changes in pekko main in other modules. Let me > demonstrate > > >> what I > > >> > > > mean by this > > >> > > > > > >> > > > * Pekko merges some significant feature, lets say the upgrade > from > > >> netty3 > > >> > > > to > > >> > > > netty4 (which is a very good example) i.e. > > >> > > > https://github.com/apache/incubator-pekko/pull/539 > > >> > > > * Since this is a major we want to make sure that it doesn't > cause > > >> > > > unintended > > >> > > > regressions in other pekko modules, such as pekko-management > > >> > > > * This means that ideally we would like to update the pekko > > version > > >> in > > >> > > > pekko-management to a version of pekko that has the > netty3->netty4 > > >> change > > >> > > > so we can catch problems well before a release is made > > >> > > > > > >> > > > Now we could update the pekko dependency in pekko-management to > > use > > >> > > > a snapshot but as stated before the snapshot's are not meant to > be > > >> > > > persistent. > > >> > > > This leaves us with the next best solution which is to use a > > >> milestone > > >> > > > which > > >> > > > is designed to persist (unlike a snapshot). When enough > > significant > > >> > > changes > > >> > > > get merged into Pekko, one of the PPMC/committers will notify > > >> developers > > >> > > > that a milestone is being published and when a milestone is > > >> published we > > >> > > > can just update the dependency in pekko-management to that > > >> milestone. > > >> > > > > > >> > > > Now of course there are other solutions to this, i.e. pekko-http > > and > > >> > > > pekko-persistence-dynamodb happen to have code that will > > >> > > > dynamically find the latest snapshot from Apache Maven Snapshots > > >> > > > repository but this causes its own problems (see the > conversation > > >> > > > at > > >> > > > > > >> > > > > > >> > > > > >> > > > https://github.com/apache/incubator-pekko-http/issues/293#issuecomment-1666893217 > > >> > > > ). > > >> > > > The other solutions mentioned, i.e. using the nightlies > repository > > >> also > > >> > > are > > >> > > > less than ideal solutions i.e. publishing to nightlies > repository > > >> > > > requires us to manually deploy locally and move the jars via > rsync > > >> which > > >> > > is > > >> > > > quite > > >> > > > brittle (i.e. the build will break whenever a folder structure > > >> changes). > > >> > > > More > > >> > > > generally projects have actually been moving away from deploying > > >> jars > > >> > > into > > >> > > > the nightlies repository because it's not the correct place to > put > > >> > > them[1], > > >> > > > the > > >> > > > nighlies repository is more suited to binary applications builds > > >> (think > > >> > > > executables for Apache OpenOffice) > > >> > > > > > >> > > > Deploying milestone jars will also happen to solve > > >> > > > > > >> > > > > >> > > > https://github.com/apache/incubator-pekko/pull/532#issuecomment-1664741546 > > >> > > > very > > >> > > > cleanly. > > >> > > > > > >> > > > This is what I meant earlier when I said these are largely > > technical > > >> > > > problems. > > >> > > > > > >> > > > [1] > > https://lists.apache.org/thread/t6598v7lsdb74q65x9grwhxqt38ntq2s > > >> > > > > > >> > > > On Mon, Aug 7, 2023 at 8:17 AM Claude Warren, Jr > > >> > > > <[email protected]> wrote: > > >> > > > > > >> > > > > Here is my take on this discussion. Please stick with me for > a > > >> bit. > > >> > > > > > > >> > > > > I think that there is a communication probleem; crossed wires > in > > >> > > > > terminology. Matthew talks about "general public" and most of > > us > > >> grey > > >> > > > > heads (trying not to be gender specific here) in the ASF hear > > >> > > "release". > > >> > > > > What I think Matthew is talking about is a wider testing > > audience. > > >> > > > People > > >> > > > > who understand that the code is not "released". If that is > > >> > > > > indeed the case, and I am certain Matthew will tell me if I am > > >> wrong, > > >> > > > then > > >> > > > > the following should make sense. > > >> > > > > > > >> > > > > > > >> > > > > - Pekko can build and place into the SNAPSHOT repository, > > >> nightly > > >> > > > > builds. These builds should be named > > >> > > <pekko-module>-x.y.z-SNAPSHOT. > > >> > > > > The > > >> > > > > repository will take care of adding dates to them to ensure > > the > > >> > > > > latest SNAPSHOT is returned. (The repository should also > > >> > > > automatically > > >> > > > > retain only 5 or so snapshots for pekko-module-x.y.z but > > that > > >> is a > > >> > > > > different issue) > > >> > > > > - Pekko can build and place into the SNAPSHOT repository, a > > >> longer > > >> > > > > period (weekly,monthly) or manually triggered build that > has > > a > > >> name > > >> > > > like > > >> > > > > <pekko-module>-x.y.z-M-SNAPSHOT. Where "M" is the literal > > 'M'. > > >> > > This > > >> > > > > will > > >> > > > > have the same retention properties as > > >> <pekko-module>-x.y.z-SNAPSHOT. > > >> > > > > - The "wider testing audience" needs to be pointed to the > dev > > >> > > mailing > > >> > > > > list. There are several reasons for this, most notably: > They > > >> have > > >> > > > > access > > >> > > > > to the discussion about what is changing, they can be > > notified > > >> when > > >> > > an > > >> > > > > "M" > > >> > > > > release is created, they can contribute ideas, suggestions > > and > > >> > > > concepts > > >> > > > > back to the community, they become part of the community. > > >> > > > > - The "wider testing audience" can depend upon > > >> > > > > <pekko-module>-x.y.z.M-SNAPSHOT during their builds, > provided > > >> they > > >> > > add > > >> > > > > the > > >> > > > > ASF snapshot repository. > > >> > > > > - There is no "release" here, all work is in the "dev" side > > of > > >> the > > >> > > > > house. > > >> > > > > - The pekko dev team can produce Milestone ("M") builds > when > > >> they > > >> > > > > desire, but may need to do manual cleanup of the repository > > >> when the > > >> > > > > actual > > >> > > > > release is made. > > >> > > > > - The pekko dev team can produce Regression ("R") builds > that > > >> > > perform > > >> > > > > extensive regression testing if so desired. > > >> > > > > - The pekko dev team can produce Foo ("F") builds to ensure > > foo > > >> > > > > correctness (whatever that is) if so desired. > > >> > > > > - The pekko dev team will need to clean up all the lettered > > >> versions > > >> > > > > when the release is made or Infra will become very annoyed. > > >> > > > > - The pekko dev team should try to limit the number of > > lettered > > >> > > > versions > > >> > > > > so that infra does not become annoyed. > > >> > > > > > > >> > > > > The short bit here is that as long as the work is development > > work > > >> > > > > (building, testing, and verification) and not for the "general > > >> public" > > >> > > > > (grey head definition) then it can safely be done and > "released" > > >> > > > (non-grey > > >> > > > > head definition) within the dev environment (source, build, > > >> repository, > > >> > > > and > > >> > > > > community tools). > > >> > > > > > > >> > > > > My recommendations are: > > >> > > > > > > >> > > > > - Bring the "wider testing audience" into the dev > > >> conversations. > > >> > > > > - Announce milestone builds and internal releases in the > dev > > >> list. > > >> > > > > - Use the snapshot repositories to their fullest. > > >> > > > > - expand the pool of potential team members. > > >> > > > > > > >> > > > > That last point brings me to the other topic of PPMC members, > > and > > >> I > > >> > > > > apologize for not seeing and paying attention to the > discussion > > >> on the > > >> > > > > private list and will endeavour to do better in future. > > >> > > > > > > >> > > > > Claude > > >> > > > > > > >> > > > > On Sun, Aug 6, 2023 at 4:01 PM Matthew de Detrich > > >> > > > > <[email protected]> wrote: > > >> > > > > > > >> > > > > > > I agree with all of Justin McLean's points. > > >> > > > > > > > >> > > > > > > I care about the ASF rules > > >> > > > > > > > >> > > > > > I am still waiting for an actual reference to a rule on this > > and > > >> > > Justin > > >> > > > > > appears to have misunderstood what was being suggested (but > I > > >> > > > > > will wait for his answer) > > >> > > > > > > > >> > > > > > > and norms. > > >> > > > > > > > >> > > > > > I have seen more than non negligible amount of so-called > > "norms" > > >> > > > > > which turn out to be either false or some convention which > > >> projects > > >> > > > > > copy from other projects just because. Pekko is also not a > > >> typical > > >> > > > > > project in this specific regard, which by definition means > its > > >> not > > >> > > > > > normal so I in this case I don't about norms unless there > is a > > >> rule > > >> > > > > > behind it. > > >> > > > > > > > >> > > > > > > > >> > > > > > On Sun, Aug 6, 2023 at 3:48 PM PJ Fanning < > > [email protected] > > >> > > > >> > > > wrote: > > >> > > > > > > > >> > > > > > > I agree with all of Justin McLean's points. > > >> > > > > > > > > >> > > > > > > I absolutely don't care what non-ASF projects do. I care > > >> about the > > >> > > > ASF > > >> > > > > > > rules and norms. > > >> > > > > > > > > >> > > > > > > On Sun, 6 Aug 2023 at 14:44, Matthew de Detrich > > >> > > > > > > <[email protected]> wrote: > > >> > > > > > > > > > >> > > > > > > > > I remain strongly opposed to publishing to maven > central > > >> > > without > > >> > > > a > > >> > > > > > > > full release process with 2 phase voting. > > >> > > > > > > > > > >> > > > > > > > Can you elaborate on this opposition? There is plenty of > > >> software > > >> > > > > that > > >> > > > > > is > > >> > > > > > > > distributed on maven using commonly established > milestone > > >> suffix > > >> > > > > > > > (i.e. -M0, -M1 etc etc) and I would argue it's far from > > >> > > > controversial > > >> > > > > > to > > >> > > > > > > > state that it's extremely clear for users that such an > > >> artifact > > >> > > is > > >> > > > a > > >> > > > > > > > milestone, this is already established. > > >> > > > > > > > > > >> > > > > > > > I just want to know where you are coming from, because > if > > >> it's > > >> > > just > > >> > > > > due > > >> > > > > > > > to it being a rule/Apache process this is what I want to > > >> clarify > > >> > > > (if > > >> > > > > > > there > > >> > > > > > > > is a clear specific rule on this that this is not > allowed; > > >> which > > >> > > > has > > >> > > > > > yet > > >> > > > > > > > to be provided yet then yes I will drop it). > > >> > > > > > > > > > >> > > > > > > > On Sun, Aug 6, 2023 at 3:30 PM PJ Fanning < > > >> [email protected]> > > >> > > > > wrote: > > >> > > > > > > > > > >> > > > > > > > > I remain strongly opposed to publishing to maven > central > > >> > > without > > >> > > > > full > > >> > > > > > > > > release process with 2 phase voting. > > >> > > > > > > > > > > >> > > > > > > > > I can live with putting milestone jars elsewhere like > > >> > > > > > > > > nightlies.apache.org. We used to have a CI job that > > >> published > > >> > > > jars > > >> > > > > > to > > >> > > > > > > > > nightlies.apache.org. This can easily be dusted down > > and > > >> > > > > repurposed. > > >> > > > > > > > > My vague recollection was that the jars were published > > in > > >> a > > >> > > Maven > > >> > > > > > repo > > >> > > > > > > > > compatible data structure so that they were easily > > >> consumable > > >> > > in > > >> > > > > > > > > mvn/gradle/sbt builds. > > >> > > > > > > > > > > >> > > > > > > > > On Sun, 6 Aug 2023 at 14:18, Matthew de Detrich > > >> > > > > > > > > <[email protected]> wrote: > > >> > > > > > > > > > > > >> > > > > > > > > > > And we can release voted on milestones. > > >> > > > > > > > > > > > >> > > > > > > > > > We can but for reasons stated earlier this to me > seems > > >> like a > > >> > > > > > bandaid > > >> > > > > > > > > > and as you pointed out in [1] we are having issues > > >> currently > > >> > > > with > > >> > > > > > > getting > > >> > > > > > > > > > enough votes for releases. Hence my biggest > complaint > > >> here is > > >> > > > > that > > >> > > > > > > > > > requiring votes sends the wrong signals, after all > one > > >> of the > > >> > > > > main > > >> > > > > > > points > > >> > > > > > > > > > of my suggested use of milestones is to test > published > > >> > > > milestones > > >> > > > > > in > > >> > > > > > > > > > downstream modules to confirm that they are suitable > > for > > >> > > > general > > >> > > > > > > public > > >> > > > > > > > > > consumption which is the point that Justin raised > > >> earlier. > > >> > > > > > > > > > > > >> > > > > > > > > > > In the meantime, we have: > > >> > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > https://cwiki.apache.org/confluence/display/PEKKO/Testing+with+Pekko+Snapshot+Jars > > >> > > > > > > > > > > > >> > > > > > > > > > > Which is not ideal but it does provide some help. > > >> > > > > > > > > > > > >> > > > > > > > > > While this is (for now) fine at least for one case > > (i.e. > > >> > > > > > > users/developers > > >> > > > > > > > > > testing quick changes which is intended use for > > >> snapshots) it > > >> > > > > > doesn't > > >> > > > > > > > > > practically alleviate the other case regarding > testing > > >> > > > downstream > > >> > > > > > > pekko > > >> > > > > > > > > > modules with more recent upstream pekko changes. > > >> > > > > > > > > > > > >> > > > > > > > > > Using snapshots for this means we have to > continuously > > >> > > > add/remove > > >> > > > > > > > > > the Apache Nexus snapshot repository in the > downstream > > >> > > modules > > >> > > > > > build > > >> > > > > > > > > > files. > > >> > > > > > > > > > > > >> > > > > > > > > > This point in general actually becoming increasingly > > >> more > > >> > > > > important > > >> > > > > > > now > > >> > > > > > > > > > considering we are getting significant changes > merged > > >> > > upstream > > >> > > > in > > >> > > > > > the > > >> > > > > > > > > > Pekko 1.1.x series and our current approach of only > > >> bringing > > >> > > in > > >> > > > > the > > >> > > > > > > > > > changes when a release candidate is being voted on > > >> means that > > >> > > > > > > > > > we will be bringing in ~6-12 months of changes all > at > > >> once at > > >> > > > an > > >> > > > > > > > > > inopportune time (i.e. a release is being made) > > >> > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > [1] > > >> > > > > > > > >> https://lists.apache.org/thread/b57nmyrg5xhgdv3gnbq2o9s6wsqf31q6 > > >> > > > > > > > > > > > >> > > > > > > > > > On Sun, Aug 6, 2023 at 2:57 PM PJ Fanning < > > >> > > > [email protected]> > > >> > > > > > > wrote: > > >> > > > > > > > > > > > >> > > > > > > > > > > And we can release voted on milestones. > > >> > > > > > > > > > > > > >> > > > > > > > > > > In the meantime, we have: > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > https://cwiki.apache.org/confluence/display/PEKKO/Testing+with+Pekko+Snapshot+Jars > > >> > > > > > > > > > > > > >> > > > > > > > > > > Which is not ideal but it does provide some help. > > >> > > > > > > > > > > > > >> > > > > > > > > > > On Sun, 6 Aug 2023 at 13:38, Matthew de Detrich > > >> > > > > > > > > > > <[email protected]> wrote: > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Nobody is threatening to delete our recent > > >> snapshots. > > >> > > > There > > >> > > > > > is > > >> > > > > > > a > > >> > > > > > > > > > > > question about what to do about older snapshots > - > > >> which > > >> > > is > > >> > > > > > > separate > > >> > > > > > > > > > > > from this topic. > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > I am not saying that people will be deleting our > > >> > > snapshots, > > >> > > > > but > > >> > > > > > > > > rather > > >> > > > > > > > > > > > there is a current problem with them being > pruned > > >> (or > > >> > > more > > >> > > > > > > correctly > > >> > > > > > > > > > > > not being pruned as expected) which needs to be > > >> resolved. > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Anyone who wants to test the latest pekko > 1.1.0 > > >> preview > > >> > > > can > > >> > > > > > > track > > >> > > > > > > > > down > > >> > > > > > > > > > > > versions at > > >> > > > > > > > > > > > The core point is while its possible for users > to > > >> > > manually > > >> > > > > > track > > >> > > > > > > > > > > snapshots > > >> > > > > > > > > > > > and update them as new ones get released/old > ones > > >> get > > >> > > > deleted > > >> > > > > > > this > > >> > > > > > > > > is not > > >> > > > > > > > > > > > practical for the use case I mentioned earlier > > >> where we > > >> > > > want > > >> > > > > to > > >> > > > > > > set > > >> > > > > > > > > the > > >> > > > > > > > > > > > downstream Pekko modules versions to a more > > >> frequently > > >> > > > built > > >> > > > > > non > > >> > > > > > > > > release > > >> > > > > > > > > > > > version so we can catch regressions and/or make > > >> sure that > > >> > > > > newly > > >> > > > > > > > > merged > > >> > > > > > > > > > > > features work as expected. > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > If we use snapshots for this then we have the > > >> overhead of > > >> > > > > > having > > >> > > > > > > to > > >> > > > > > > > > > > > manually update snapshots versions when builds > > >> break due > > >> > > to > > >> > > > > > > > > > > > snapshots not existing anymore (note that this > > >> hasn't > > >> > > been > > >> > > > a > > >> > > > > > > problem > > >> > > > > > > > > for > > >> > > > > > > > > > > now > > >> > > > > > > > > > > > because the snapshot pruning is not working > > >> correctly > > >> > > which > > >> > > > > is > > >> > > > > > > the > > >> > > > > > > > > > > problem > > >> > > > > > > > > > > > I was referring to before). > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > Furthermore if developers want to test newly > > merged > > >> > > > features > > >> > > > > > into > > >> > > > > > > > > > > > development/staging/prod systems to see that > there > > >> aren't > > >> > > > > > > > > regressions, > > >> > > > > > > > > > > > assuming the snapshots are working correctly > > (which > > >> they > > >> > > > > > > currently > > >> > > > > > > > > > > aren't) > > >> > > > > > > > > > > > they can also hit the same issues regarding the > > >> snapshots > > >> > > > > being > > >> > > > > > > > > pruned. > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > At least to me, this is one of the core problems > > >> that the > > >> > > > > > > milestone > > >> > > > > > > > > > > > concept is intended to solve, snapshots are too > > >> > > > > > > frequent/granular and > > >> > > > > > > > > > > > releases/release candidates are too infrequent. > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > On Sun, Aug 6, 2023 at 2:23 PM PJ Fanning < > > >> > > > > > [email protected]> > > >> > > > > > > > > wrote: > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > > Nobody is threatening to delete our recent > > >> snapshots. > > >> > > > There > > >> > > > > > is > > >> > > > > > > a > > >> > > > > > > > > > > > > question about what to do about older > snapshots > > - > > >> which > > >> > > > is > > >> > > > > > > separate > > >> > > > > > > > > > > > > from this topic. > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > Anyone who wants to test the latest pekko > 1.1.0 > > >> preview > > >> > > > can > > >> > > > > > > track > > >> > > > > > > > > down > > >> > > > > > > > > > > > > versions at > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > https://repository.apache.org/content/groups/snapshots/org/apache/pekko/pekko-actor-typed_2.13/ > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > 1.1.0-M0+5-964dcf53-SNAPSHOT is the most > recent > > - > > >> but > > >> > > > there > > >> > > > > > > will be > > >> > > > > > > > > > > > > new snapshots published most nights (any day > > that > > >> has > > >> > > > > commits > > >> > > > > > > to > > >> > > > > > > > > the > > >> > > > > > > > > > > > > main branch). > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > On Sun, 6 Aug 2023 at 13:08, Matthew de > Detrich > > >> > > > > > > > > > > > > <[email protected]> wrote: > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > The question to ask is, will these > artefacts > > >> be > > >> > > used > > >> > > > by > > >> > > > > > > users > > >> > > > > > > > > or > > >> > > > > > > > > > > > > intended > > >> > > > > > > > > > > > > > for their use? i.e. people who are not > > >> subscribed to > > >> > > > the > > >> > > > > > > mailing > > >> > > > > > > > > > > list or > > >> > > > > > > > > > > > > > not committers or PMC members? If so, then > by > > >> ASF > > >> > > > > > definition, > > >> > > > > > > > > that > > >> > > > > > > > > > > > > artefact > > >> > > > > > > > > > > > > > is a release and needs to be voted on by the > > >> > > PPMC/IPMC. > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > No they aren't, as I said they are treated > in > > >> the > > >> > > exact > > >> > > > > > same > > >> > > > > > > way > > >> > > > > > > > > > > > > snapshots > > >> > > > > > > > > > > > > > are currently treated. The **ONLY** > difference > > >> is > > >> > > that > > >> > > > > they > > >> > > > > > > will > > >> > > > > > > > > be > > >> > > > > > > > > > > > > > distributed to the Apache Nexus main > > repository > > >> and > > >> > > > this > > >> > > > > is > > >> > > > > > > > > mainly > > >> > > > > > > > > > > due to > > >> > > > > > > > > > > > > > technical reasons stated before (i.e. > > snapshots > > >> are > > >> > > > > > > > > semi-permanent > > >> > > > > > > > > > > since > > >> > > > > > > > > > > > > > they need to be pruned). > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > The primary motivation for the milestone > > >> artifacts > > >> > > are > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > * Internal testing between the Pekko > > >> dependencies > > >> > > (i.e. > > >> > > > > > > > > publishing an > > >> > > > > > > > > > > > > > artifact of Pekko core and using that > artifact > > >> in in > > >> > > > > other > > >> > > > > > > Pekko > > >> > > > > > > > > > > modules > > >> > > > > > > > > > > > > to > > >> > > > > > > > > > > > > > catch regressions before a release is > marked) > > >> > > > > > > > > > > > > > * For Pekko developers to have the ability > to > > >> test > > >> > > > their > > >> > > > > > > features > > >> > > > > > > > > > > which > > >> > > > > > > > > > > > > > were merged into main in their production > > >> systems > > >> > > > without > > >> > > > > > > having > > >> > > > > > > > > to > > >> > > > > > > > > > > > > > manually build Pekko (and possibly all of > > >> Pekko's > > >> > > > > > > dependencies) > > >> > > > > > > > > > > > > themselves, > > >> > > > > > > > > > > > > > which also as stated previously is quite > > >> difficult > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > There is no intention of the milestones > having > > >> any > > >> > > > formal > > >> > > > > > > release > > >> > > > > > > > > > > > > > announcement. > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Have other incubating projects made > non-ASF > > >> > > releases > > >> > > > > > while > > >> > > > > > > in > > >> > > > > > > > > > > > > incubation? > > >> > > > > > > > > > > > > > In a small number of cases, yes, mostly > while > > >> they > > >> > > were > > >> > > > > > > getting > > >> > > > > > > > > their > > >> > > > > > > > > > > > > code > > >> > > > > > > > > > > > > > base in order but not after they had made an > > ASF > > >> > > > release, > > >> > > > > > and > > >> > > > > > > > > there > > >> > > > > > > > > > > we > > >> > > > > > > > > > > > > very > > >> > > > > > > > > > > > > > clearly labelled as non-ASF releases. > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Yes and this is not meant to be a formal ASF > > >> release, > > >> > > > > hence > > >> > > > > > > the > > >> > > > > > > > > > > previous > > >> > > > > > > > > > > > > > suggestion of making this clear even in the > > >> > > DISCLAIMER > > >> > > > > file > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Re official documentation/process that > > >> describes > > >> > > this > > >> > > > > > > > > distinction? > > >> > > > > > > > > > > Yes, > > >> > > > > > > > > > > > > > that policy page sets that out. See, for > > >> instance [1] > > >> > > > and > > >> > > > > > > [2] for > > >> > > > > > > > > > > why it > > >> > > > > > > > > > > > > > needs to be this way. > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > [1] Is talking about source packages, this > > >> discussion > > >> > > > is > > >> > > > > > > about > > >> > > > > > > > > binary > > >> > > > > > > > > > > > > > artifacts. In the last line they mention > this > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Nightly Builds that are not release > > >> candidates can > > >> > > be > > >> > > > > > > hosted at > > >> > > > > > > > > > > > > > ci.apache.org projects area, just file an > > INFRA > > >> > > > ticket. > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > The links to ci.apache.org aren't even > > working > > >> and I > > >> > > > > doubt > > >> > > > > > > we > > >> > > > > > > > > can > > >> > > > > > > > > > > even > > >> > > > > > > > > > > > > use > > >> > > > > > > > > > > > > > such a repository since we are dealing with > > JVM > > >> jar's > > >> > > > > > > > > specifically > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Re official documentation/process that > > >> describes > > >> > > this > > >> > > > > > > > > distinction? > > >> > > > > > > > > > > Yes, > > >> > > > > > > > > > > > > > that policy page sets that out. See, for > > >> instance [1] > > >> > > > and > > >> > > > > > > [2] for > > >> > > > > > > > > > > why it > > >> > > > > > > > > > > > > > needs to be this way. The Incubator > > distribution > > >> > > > > guideline > > >> > > > > > > also > > >> > > > > > > > > > > covers > > >> > > > > > > > > > > > > this > > >> > > > > > > > > > > > > > [3]. You see that at [4] "Release > candidates, > > >> > > nightlys > > >> > > > > and > > >> > > > > > > > > snapshots > > >> > > > > > > > > > > must > > >> > > > > > > > > > > > > > not be advertised to the general public.” > and > > >> you can > > >> > > > > read > > >> > > > > > > [5] > > >> > > > > > > > > for > > >> > > > > > > > > > > why. > > >> > > > > > > > > > > > > The > > >> > > > > > > > > > > > > > release distribution policy also touches on > > >> this e.g. > > >> > > > [6] > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > As was clarified earlier, milestones are > > **NOT** > > >> > > > > releases, > > >> > > > > > > they > > >> > > > > > > > > are > > >> > > > > > > > > > > > > treated > > >> > > > > > > > > > > > > > the exact same way as snapshots, nightlies > or > > >> release > > >> > > > > > > candidates > > >> > > > > > > > > > > with the > > >> > > > > > > > > > > > > > **ONLY** critical difference being that they > > are > > >> > > > deployed > > >> > > > > > in > > >> > > > > > > a > > >> > > > > > > > > > > repository > > >> > > > > > > > > > > > > > that is not snapshots and they are published > > >> less > > >> > > > > > frequently > > >> > > > > > > > > then a > > >> > > > > > > > > > > > > > snapshot/nightly is. > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > With this being said I am increasingly of > the > > >> opinion > > >> > > > > that > > >> > > > > > > this > > >> > > > > > > > > is > > >> > > > > > > > > > > more > > >> > > > > > > > > > > > > of > > >> > > > > > > > > > > > > > a technical INFRA question than an ASF > policy > > >> > > question > > >> > > > > > since > > >> > > > > > > the > > >> > > > > > > > > > > issues > > >> > > > > > > > > > > > > > being discussed are technical in nature. > There > > >> was > > >> > > > never > > >> > > > > a > > >> > > > > > > > > > > suggestion of > > >> > > > > > > > > > > > > > making an alternative formal type of ASF > > >> release. > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > On Sun, Aug 6, 2023 at 10:51 AM Justin > Mclean > > < > > >> > > > > > > > > > > [email protected]> > > >> > > > > > > > > > > > > > wrote: > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Hi, > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > The question to ask is, will these > artefacts > > >> be > > >> > > used > > >> > > > by > > >> > > > > > > users > > >> > > > > > > > > or > > >> > > > > > > > > > > > > intended > > >> > > > > > > > > > > > > > > for their use? i.e. people who are not > > >> subscribed > > >> > > to > > >> > > > > the > > >> > > > > > > > > mailing > > >> > > > > > > > > > > list > > >> > > > > > > > > > > > > or > > >> > > > > > > > > > > > > > > not committers or PMC members? If so, then > > by > > >> ASF > > >> > > > > > > definition, > > >> > > > > > > > > that > > >> > > > > > > > > > > > > artefact > > >> > > > > > > > > > > > > > > is a release and needs to be voted on by > the > > >> > > > PPMC/IPMC. > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Have other incubating projects made > non-ASF > > >> > > releases > > >> > > > > > while > > >> > > > > > > in > > >> > > > > > > > > > > > > incubation? > > >> > > > > > > > > > > > > > > In a small number of cases, yes, mostly > > while > > >> they > > >> > > > were > > >> > > > > > > getting > > >> > > > > > > > > > > their > > >> > > > > > > > > > > > > code > > >> > > > > > > > > > > > > > > base in order but not after they had made > an > > >> ASF > > >> > > > > release, > > >> > > > > > > and > > >> > > > > > > > > > > there we > > >> > > > > > > > > > > > > very > > >> > > > > > > > > > > > > > > clearly labelled as non-ASF releases. > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Re official documentation/process that > > >> describes > > >> > > this > > >> > > > > > > > > distinction? > > >> > > > > > > > > > > Yes, > > >> > > > > > > > > > > > > > > that policy page sets that out. See, for > > >> instance > > >> > > [1] > > >> > > > > and > > >> > > > > > > [2] > > >> > > > > > > > > for > > >> > > > > > > > > > > why > > >> > > > > > > > > > > > > it > > >> > > > > > > > > > > > > > > needs to be this way. The Incubator > > >> distribution > > >> > > > > > guideline > > >> > > > > > > also > > >> > > > > > > > > > > covers > > >> > > > > > > > > > > > > this > > >> > > > > > > > > > > > > > > [3]. You see that at [4] "Release > > candidates, > > >> > > > nightlys > > >> > > > > > and > > >> > > > > > > > > > > snapshots > > >> > > > > > > > > > > > > must > > >> > > > > > > > > > > > > > > not be advertised to the general public.” > > and > > >> you > > >> > > can > > >> > > > > > read > > >> > > > > > > [5] > > >> > > > > > > > > for > > >> > > > > > > > > > > > > why. The > > >> > > > > > > > > > > > > > > release distribution policy also touches > on > > >> this > > >> > > e.g. > > >> > > > > [6] > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > Kind Regards, > > >> > > > > > > > > > > > > > > Justin > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > 1. > > >> > > > > > > https://www.apache.org/legal/release-policy.html#host-rc > > >> > > > > > > > > > > > > > > 2. > > >> > > > > https://www.apache.org/legal/release-policy.html#why > > >> > > > > > > > > > > > > > > 3. > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > https://cwiki.apache.org/confluence/display/INCUBATOR/Distribution+Guidelines > > >> > > > > > > > > > > > > > > 4. > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > >> > > > > > > >> > > > > >> > https://incubator.apache.org/guides/distribution.html#release_platforms > > >> > > > > > > > > > > > > > > 5. > > >> > > > > > > > > > > > > >> > > > > > > https://incubator.apache.org/guides/distribution.html#motivation > > >> > > > > > > > > > > > > > > 6. > > >> > > > > > > https://infra.apache.org/release-distribution.html#maven > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > >> > > > > --------------------------------------------------------------------- > > >> > > > > > > > > > > > > > > To unsubscribe, e-mail: > > >> > > > > [email protected] > > >> > > > > > > > > > > > > > > For additional commands, e-mail: > > >> > > > > > [email protected] > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > -- > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Matthew de Detrich > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > *Aiven Deutschland GmbH* > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu > > >> Valtonen > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > *m:* +491603708037 > > >> > > > > > > > > > > > > > > > >> > > > > > > > > > > > > > *w:* aiven.io *e:* > [email protected] > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > >> --------------------------------------------------------------------- > > >> > > > > > > > > > > > > To unsubscribe, e-mail: > > >> > > [email protected] > > >> > > > > > > > > > > > > For additional commands, e-mail: > > >> > > > [email protected] > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > -- > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > Matthew de Detrich > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > *Aiven Deutschland GmbH* > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu > Valtonen > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > *m:* +491603708037 > > >> > > > > > > > > > > > > > >> > > > > > > > > > > > *w:* aiven.io *e:* [email protected] > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > >> > > > > --------------------------------------------------------------------- > > >> > > > > > > > > > > To unsubscribe, e-mail: > > >> [email protected] > > >> > > > > > > > > > > For additional commands, e-mail: > > >> [email protected] > > >> > > > > > > > > > > > > >> > > > > > > > > > > > > >> > > > > > > > > > > > >> > > > > > > > > > -- > > >> > > > > > > > > > > > >> > > > > > > > > > Matthew de Detrich > > >> > > > > > > > > > > > >> > > > > > > > > > *Aiven Deutschland GmbH* > > >> > > > > > > > > > > > >> > > > > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > >> > > > > > > > > > > > >> > > > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > >> > > > > > > > > > > > >> > > > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > >> > > > > > > > > > > > >> > > > > > > > > > *m:* +491603708037 > > >> > > > > > > > > > > > >> > > > > > > > > > *w:* aiven.io *e:* [email protected] > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > >> --------------------------------------------------------------------- > > >> > > > > > > > > To unsubscribe, e-mail: > > [email protected] > > >> > > > > > > > > For additional commands, e-mail: > > >> [email protected] > > >> > > > > > > > > > > >> > > > > > > > > > > >> > > > > > > > > > >> > > > > > > > -- > > >> > > > > > > > > > >> > > > > > > > Matthew de Detrich > > >> > > > > > > > > > >> > > > > > > > *Aiven Deutschland GmbH* > > >> > > > > > > > > > >> > > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > >> > > > > > > > > > >> > > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > >> > > > > > > > > > >> > > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > >> > > > > > > > > > >> > > > > > > > *m:* +491603708037 > > >> > > > > > > > > > >> > > > > > > > *w:* aiven.io *e:* [email protected] > > >> > > > > > > > > >> > > > > > > > > >> > > > > --------------------------------------------------------------------- > > >> > > > > > > To unsubscribe, e-mail: [email protected] > > >> > > > > > > For additional commands, e-mail: > [email protected] > > >> > > > > > > > > >> > > > > > > > > >> > > > > > > > >> > > > > > -- > > >> > > > > > > > >> > > > > > Matthew de Detrich > > >> > > > > > > > >> > > > > > *Aiven Deutschland GmbH* > > >> > > > > > > > >> > > > > > Immanuelkirchstraße 26, 10405 Berlin > > >> > > > > > > > >> > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > >> > > > > > > > >> > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > >> > > > > > > > >> > > > > > *m:* +491603708037 > > >> > > > > > > > >> > > > > > *w:* aiven.io *e:* [email protected] > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > > > >> > > > -- > > >> > > > > > >> > > > Matthew de Detrich > > >> > > > > > >> > > > *Aiven Deutschland GmbH* > > >> > > > > > >> > > > Immanuelkirchstraße 26, 10405 Berlin > > >> > > > > > >> > > > Amtsgericht Charlottenburg, HRB 209739 B > > >> > > > > > >> > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > >> > > > > > >> > > > *m:* +491603708037 > > >> > > > > > >> > > > *w:* aiven.io *e:* [email protected] > > >> > > > > > >> > > > > >> > > > >> > > > >> > -- > > >> > > > >> > Matthew de Detrich > > >> > > > >> > *Aiven Deutschland GmbH* > > >> > > > >> > Immanuelkirchstraße 26, 10405 Berlin > > >> > > > >> > Amtsgericht Charlottenburg, HRB 209739 B > > >> > > > >> > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > >> > > > >> > *m:* +491603708037 > > >> > > > >> > *w:* aiven.io *e:* [email protected] > > >> > > > >> > > >> --------------------------------------------------------------------- > > >> To unsubscribe, e-mail: [email protected] > > >> For additional commands, e-mail: [email protected] > > >> > > >> > > > > > > -- > > > > > > Matthew de Detrich > > > > > > *Aiven Deutschland GmbH* > > > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > > > *m:* +491603708037 > > > > > > *w:* aiven.io *e:* [email protected] > > > > > > > > > -- > > > > Matthew de Detrich > > > > *Aiven Deutschland GmbH* > > > > Immanuelkirchstraße 26, 10405 Berlin > > > > Amtsgericht Charlottenburg, HRB 209739 B > > > > Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen > > > > *m:* +491603708037 > > > > *w:* aiven.io *e:* [email protected] > > > -- Matthew de Detrich *Aiven Deutschland GmbH* Immanuelkirchstraße 26, 10405 Berlin Amtsgericht Charlottenburg, HRB 209739 B Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen *m:* +491603708037 *w:* aiven.io *e:* [email protected]
