On Fri, Aug 12, 2016 at 9:11 AM, Nate Finch <nate.fi...@canonical.com> wrote:
> The problem really stems from two go-isms - > > 1.) The standard for Go projects is to have the path on disk > representative of the URL to the project. > 2.) Every file that uses a package has an import statement that references > the path on disk. > > If either one of these were not true, we could have avoided your problem. > > There's nothing we can do about #2. Building with the go tool requires > that if you use a package in a file, you must have an import statement for > it, and if you import "foo/bar/gnuflag", that code must live in > $GOPATH/src/foo/bar/gnuflag". > > For #1, *in theory* we could change things. Only "go get" knows that > $GOPATH/src/launchpad.net/gnuflag corresponds to > http://launchpad.net/gnuflag. When you compile and build, all the go > tool cares about is the path on disk and what code is in that folder. We > *could* detach the idea of an import path from the URL it corresponds > to. We could git clone github.com/juju/gnuflag into $GOPATH/src/gnuflag, > and the go tool would build it just fine. the import statement would then > be import "gnuflag" and this would work just fine. We'd need to maintain a > mapping file of repo to folder on disk, similar to what we do now with > dependencies.tsv, but it wouldn't be the end of the world. > > I'm not entirely convinced this is a good idea, but it would make updating > dependencies a lot easier. > This is interesting, and more or less what I was after in my mail. I wasn't sure if there were any out of the box ideas that would make life easier, but I hoped to surface some for discussion. Thanks! I suppose my follow-up question is, why or why not is this a good idea? Having to change all of the imports is painful, but as you say, it is an intentional design decision by the Go Community. I don't have any real feeling or inkling on what other Go projects do, but I suspect at least some of them follow this idea of importing all dependencies and updating them only if needed. However, by nature of them simply cloning the source into the repo, I would also guess they are not able to easily update a dependency. > > The source code hack to mgo is due to this problem as well. In theory we > can just fork mgo and make our fix, but that would again require us to > change a lot of import statements. > > Part of the problem was that you were changing two very common > dependencies that were imported 4 levels deep (i.e. a imports b imports c > imports d, and they all import the dependency). This wouldn't change no > matter what we do - you'd still need to start the changes at d and roll > them up the stack. That would be true in any language. > Yes, I told of my pain, but as painful as it was, I got lots of help from many of you. Thank you again for helping land this. > > The "circular" dependency (in quotes because you can't have a true > circular dependency in go, but you can have two repos that both depend on > each other) shouldn't have landed that way. Two repos should definitely > not depend on each other. No other repo should ever depend on > github.com/juju/juju. We explicitly make no claims for backwards > compatibility, so the thing you're depending on could just disappear one > day. If you need something in there, factor it out into its own repo with > a backwards compatibility guarantee first. > > It *is* a problem that dependencies under juju can be imported in broken > states because of our use of dependencies.tsv. There's nothing that says > we couldn't run all the unit tests for all packages juju uses in the > landing job for github.com/juju/juju (and any other repo we desire)... > just run go test ./... from $GOPATH/src and it'll test everything we have > downloaded to build juju with. That's an easy win, and something we should > start doing ASAP in my opinion. > This was my initial takeaway. We don't have testing in the subpackages when landing, and we should. This I agree should be straightforward enough to fix. I was curious if there were other low hanging fruit like this. The other oddity was keeping common dependencies in sync. That is to say, if juju/utils needs version 2016-03-05, juju/juju can request the same depends, but use version 2015-12-10. They don't have to be the same, and in reality aren't. This makes sense as they are separate projects. It would seem unless someone (like me) needs to update something, they don't. However, this ends up causing you to cascade dependency updates you didn't plan on when you attempt to make a change. There might not be anything to change here. > > Finally... a lot of the reasons these things hurt so much is because juju > is a huge project - over 500,000 lines of code. What is a minor annoyance > in a small project becomes much worse in a large project. Obviously, part > of that is inescapable - we can't make juju smaller. I do think it's worth > considering ways to make our lives easier. > > -Nate > > On Thu, Aug 11, 2016 at 7:08 PM Casey Marshall < > casey.marsh...@canonical.com> wrote: > >> On Thu, Aug 11, 2016 at 5:44 PM, Nicholas Skaggs < >> nicholas.ska...@canonical.com> wrote: >> >>> This is a simple story of a man and a simple mission. Eliminate the >>> final 2 dependencies that are in bazaar and launchpad. It makes juju and >>> it's dependencies live completely in git. A notable goal, and one that I >>> desired for getting snaps to build with launchpad. >>> >>> I don't feel I need to explain the pain that making a no-source change >>> update to a dependency (point juju and friends to it's new location) has >>> been. I've had to carefully craft a set of PR's, and land them in a certain >>> order. I've encountered contention (because I have to change hundreds of >>> imports), unit test issues (because juju's dependencies aren't tested when >>> merged, so they can be incompatible with the latest juju without knowing >>> it), and circular dependencies that require some magic and wishful thinking >>> to workaround. >>> >>> I'm still not finished landing the change, but I hope to do so *soon*. >>> It must be close now! >>> >>> All of this to say, I think it would be useful to have a discussion on >>> how we manage dependencies within the project. From a release perspective, >>> it can be quite cumbersome as dependencies are picked up and dropped. It's >>> also recently made a release (And critical bugfix) really difficult at >>> times. From my newly experience contributor perspective, I would really >>> think twice before attempting to make an update :-) I suspect I'm not alone. >>> >>> I've heard ideas in the past about cleaning this up, and some things >>> like circular dependencies between romulus and juju are probably best >>> described as tech debt. But there also is some pain in the larger scheme of >>> things. For example, we are currently hacking a patch to juju's source for >>> the mgo dependency since updating the source or vendoring or any other >>> option is way too painful. It's time to really fix this. Ideas? >>> >> >> My team's been chipping away at romulus and it'll be sorted out soon >> enough. We've already moved the terms API client out to >> github.com/juju/terms-client, and we'll be doing something similar for >> the other APIs. As for the commands.. these probably need to find a better >> home closer to the command base types they extend, in cmd/juju/... >> >> One thing that occurred to me today though is most of our dependencies >> also have tests (well, they should!). We don't often run *those* tests >> as part of Juju CI, but you could run into some cases where some >> dependencies share common dependencies, but are tested with different >> common dependency versions than those specified by Juju's dependencies.tsv. >> >> >>> -- >>> Juju-dev mailing list >>> Juju-dev@lists.ubuntu.com >>> Modify settings or unsubscribe at: https://lists.ubuntu.com/ >>> mailman/listinfo/juju-dev >>> >>> -- >> Juju-dev mailing list >> Juju-dev@lists.ubuntu.com >> Modify settings or unsubscribe at: https://lists.ubuntu.com/ >> mailman/listinfo/juju-dev >> >
-- Juju-dev mailing list Juju-dev@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/juju-dev
