Thanks David for the detailed write up. Replies inline…
> We tried in-tree for in-jvm dtest and found that this broke every other > commit… maintaining the APIs across all our supported branches was too hard > to do and moving it outside of the tree helped make the upgrade tests more > stable (there were breakage but less frequent)…. > The in-jvm dtest-api library is unique in this way. I would not use it as reasoning that other libraries should not be in-tree. > We tried to do snapshot builds where the version contained the SHA, but > this has the issue that snapshot builds “may” go away over time and made > older SHAs no longer building… > Only keeping the last snapshot in repository.a.o is INFRA's policy (i've found out). We can ask INFRA to set up a separate snapshots repository just for us, with a longer expiry policy. I'd rather not create extra work for infra if there's other ways we can do this, and this approach would always require some fallback approach to rebuilding the dedepency's SHA from scratch. > We break python-dtest when cross-cutting changes are added as CI is hard > to do correctly or not supported (testing downstream users (our 4 supported > branches) is rarely done). > python dtests' is also in a different category, (the context and consumption in a different direction, i.e. it's not a library used within the in-tree). > * [nice to have] be able to work with all subprojects in one IDE and not > have to switch between windows while making cross-cutting changes > Isn't it only IntelliJ that suffers this problem? (That doesn't invalidate it, just asking…) > - same with `git pull …` (easy to be left with out-of-sync submodules) > > > Correct, if you use submodules/script you have a text file saying what we > “should” use, but this does not enforce actually using them… again we could > make sure build.xml does the right thing, > If we try this approach out, I'm definitely in favour of any build.xml command immediately failing if `git submodule status` != `git submodule status --cached` > but this can be confusing for people who mainly build in IDE and don’t > depend on build.xml until later in development… this is something we should > think about… > Again, isn't this only IntelliJ? > A project I am familiar with has their build auto-inject git hooks to make > sure things “just work”, we may be able to solve this in a similar way? > I'd like to hear/see more! - permanence from a git SHA no longer exists > > > Why is this? The SHA points to other SHAs, so it is still immutable. If > we claim that pointing to other SHAs doesn’t count then why do library > versions? Both are immutable snapshots of code at a specific point in time? > This, and a number of the other points, is already resolved (that submodule's are on fixed SHAs, not floating HEAD). - our releases get more complicated (our source tarballs are the asf > releases) > > > We don’t include our dependencies do we? If so, then does it really? If > Accord is a library we use, why would we include it’s source in the build? > Isn’t it just another library from this point of view? > The build of the source tarball must work. If the source tarball release switches how it does things, from building the submodule to including a dependency then we're back to having to make releases (and introducing a risk, and we don't ourselves work frequently with the source tarballs). > switching between branches, > > > This is a pain point that I feel should be handled by git hooks. We have > this issue when you try to reuse the same directory for different release > branches, and its super annoying when you go back in time when jars were > in-tree as you need to cleanup after switching back…. I do agree that we > should flesh this out as its going to be the common case, so how do we “do > the right thing” should be handled > +1 > Rather that you need to know in advance when the SHA is not HEAD. > > > Do you? Or do you really need to know which “branch” it is following? > For example, lets say we release 5.0 then 5.1 then 5.2, and there are > accord versions for each: 1.0, 1.2, 2.0… do we not need to really know > which branch it is following, and only when you are trying to do a > cross-cutting change? > I'm still a little confused here. If a submodule is following a branch, is that floating? Then a parent SHA isn't fixed to a submodule SHA? Say trunk is using accord:a12 where a12 is a SHA on its trunk. Other non-cassandra people using accord make commits, but our in-tree trunk isn't moved forward. Then someone in-tree does some dev that touches accord, they work away but late in the dev cycle find out that in-tree trunk isn't on the latest accord trunk and there's a conflict rebasing their work onto the latest accord. Is this an accurate description? Hope that all makes sense.
