That makes perfect sense Stephen. Quick question regarding the last point (testing environments and projects). Do you know what apache policies are in regards to maintaining multiple repos for "official" plugins/GLVs? This might make streamlining testing and updating the package in dependency management systems easier (it's actually mandatory for php composer). If you don't I can always bring it up with infra.
On Sat, Aug 6, 2016 at 6:47 AM, Stephen Mallette <spmalle...@gmail.com> wrote: > Let me first address this piece: > > > My understanding around the driver needing modifications is -I'm > guessing- mostly to accommodate for the updates in TINKERPOP-1278 rather > than from drivers needing to be specific for GLVs. If not let me know. > > The modifications are mostly to trim down the size of gremlinclient and > simplify. gremlinclient allowed for pluggable webscocket implementations > and other stuff that adds some complexity/code we could be better without > in TinkerPop. There are some specific updates needed for full support of > GLVs. (i.e. collecting side-effects which i think can remain optional) but > (I think) David's work can already submit bytecode (if not, then we'll need > to change that too, but that's pretty trivial. > > As for the rest, which largely boils down to "consider depending on > third-party drivers": > > I should have included some reasoning around your suggestion in my original > post as this was a position in my mind that was considered. The problem > with depending on third-party projects are the downsides you laid out and I > think those are more than we should bear. I would also add to that list > that the dependency is cyclical. The third party project depends on us then > TinkerPop depends back on them. I don't imagine that this will necessarily > cause a "compilation" problem but it doesn't seem quite natural/right. > > > Would it not be better to have GLV's simply depend on third party > projects when they exist and are decently up to date)? > > The part in parenthesis is part of the problem in the long run. Code gets > stale, people move on. It's easy to write a gremlin-driver on the weekend, > but to build a driver on-par with the performance/features of the java > driver - that's a difference story. Note that neo4j i think discovered this > which is why they in-housed their drivers for bolt. > > > If features are lacking the community can provide patches, etc. > > We could try and I would think most driver developers would be amenable to > that, however some might not. Also the direction of an external project, > might not match the direction TinkerPop wants. Apache governance doesn't > extend to third-party projects so in the worst case, if we hit a wall on > some technical issue that the owner of the project doesn't agree with, we'd > be left with "find something else". In that case, TinkerPop has already > spent a lot of time sending patches to that third-party and our time would > have be wasted. > > > If a project dies we can always switch it out for another one. > > I don't know if it's that simple. Again, we'd (TinkerPop) presumably > invested time getting a project on par with the java driver in terms of > features and performance. So first of all, is there another project to > switch to (right now there's not many second options for different > languages)? And second, how much effort will there be to re-invest in a > different driver to get it consistent with the features/performance of the > java driver? > > > I just think that as we get more GLVs, providing CI with the correct > environments > to run various lower level driver tests in various languages is going to be > a headache. > > Agreed. I worried about that too. However, this wont' be a fast process. > We're not on a tear to build as many GLVs as possible. We're trying to get > the model right with gremlin-python, set a high bar for the build > automation needed, and then look at how to apply that elsewhere. If we can > attain that same high bar, then we include a GLV. If we can't, then we wait > until we can figure out how to. At least that's how I think about it. > > I'm not sure if any of that changes your position, but I believe the risks > of depending on third-party drivers is a bit too high - at least higher > than us having the extra code and build automation problems of owning the > drivers ourselves. > > > On Sat, Aug 6, 2016 at 1:26 AM, Dylan Millikin <dylan.milli...@gmail.com> > wrote: > > > If I may suggest an alternative. Most (all?) languages have dependency > > management. > > Would it not be better to have GLV's simply depend on third party > projects > > (when they exist and are decently up to date)? > > These dependencies can be pulled during testing. If features are lacking > > the community can provide patches, etc. If a project dies we can always > > switch it out for another one. > > When it comes to users, they simply download their GLV and go. Ideally > the > > GLV would exist as a package in their dependency management system of > > choice. (as a side note apt-get/yum install gremlin-server would be > > amazing) > > > > My understanding around the driver needing modifications is -I'm > guessing- > > mostly to accommodate for the updates in TINKERPOP-1278 rather than from > > drivers needing to be specific for GLVs. If not let me know. > > > > This obviously comes with it's own downsides. If a project dies and > there's > > no replacement we basically have to fall back to your proposal by forking > > and maintaining it ourselves. I also don't know how licensing would work > in > > this case, but I'm guessing a license change could be akin to the project > > dying in regards to TP's dependance on it. > > > > I just think that as we get more GLVs, providing CI with the correct > > environments to run various lower level driver tests in various languages > > is going to be a headache. It would be much better to keep them as > separate > > projects. > > > > If I completely missed the original intent of the thread also do let me > > know. > > > > > > On Fri, Aug 5, 2016 at 4:57 PM, Stephen Mallette <spmalle...@gmail.com> > > wrote: > > > > > I mentioned this thread to David Brown and asked if he thought that > > > gremlinclient would be a good fit for this issue. He said that he > thought > > > it would be with some modifications. He's agreed to simplify > gremliclient > > > with the intent of making it the driver for TinkerPop's gremlin-python. > > > Assuming no objections, he said that he could look to have a pull > request > > > to TINKERPOP-1278 by end of next week. > > > > > > > > > > > > > > > On Thu, Aug 4, 2016 at 1:39 PM, Stephen Mallette <spmalle...@gmail.com > > > > > wrote: > > > > > > > TinkerPop has long held to the idea that "drivers" would be a bit > like > > > > graph implementations. TinkerPop builds reference implementation of a > > > graph > > > > and third-parties maintain "other" implementations separately. The > same > > > > idea has been held for "drivers" in that we have our reference > > > > implementation of the Java driver and third-parties produce > > > implementations > > > > for other languages. > > > > > > > > GLVs sorta changed things though. Now we have a model where we > maintain > > > > GLVs of other languages in TinkerPop itself. While somewhat risky to > > > > include the added code/complexity, maintaining the GLV in TinkerPop > > has a > > > > lot of good advantages for users and for growing our community. We > also > > > > have come up with ways to manage the risk, by establishing a really > > nice > > > > initial reference implementation of gremlin-python which sets a high > > bar > > > > for new GLVs to come in (i.e. hooked into tinkerpop test suite, > > automated > > > > build mixed with maven, non-native JVM language focused, etc.). > > > > > > > > Of course, by introducing GLVs to TinkerPop though, we now have the > > issue > > > > of conflict with TinkerPop only having a single "reference > > > implementation" > > > > for drivers because without a driver the GLV is pretty useless. We've > > > > mitigated that in gremlin-python so far with a simple REST > > implementation > > > > of a driver, but as we advance the capabilities of GLVs that approach > > > won't > > > > be good for the long term. > > > > > > > > So - it seems like we (TinkerPop) should have a driver to go with > each > > > > GLV? To me, the upsides are big for users: > > > > > > > > 1. Users don't have to hunt for drivers elsewhere - they just grab > > their > > > > GLV and go. I suppose there is still opportunity for third-party > > drivers > > > to > > > > be built, but at least users have an immediate easy option within the > > > > Apache TinkerPop. > > > > 2. Our docs for GLV usage are completely in Apache TinkerPop and are > > > > "production ready" (i.e. not REST based then go switch to something > > > > third-party) > > > > > > > > There are some downsides which basically boil down to "Mo' code Mo' > > > > problems" but basically: > > > > > > > > 1. I believe that it will be expected that we maintain some parity in > > > > terms of features and performance with the drivers. We can't have the > > > java > > > > driver be 10x over the python driver > > > > 2. The GLV code base becomes more complex. A high performing driver > > will > > > > not be trivial. > > > > 3. A whole class of downsides related to GLVs in general where we > don't > > > > have the necessary language expertise to maintain it - though - > again, > > > > mitigated by the high bar we set with our reference implementation > for > > > > gremlin-python. Also, rather than a negative, one could also see this > > as > > > an > > > > opportunity to grow the TinkerPop community. > > > > > > > > Ultimately, I think all of this is outweighed by the simple fact that > > we > > > > need a driver in Apache TinkerPop to fully test GLVs. Without that > > we're > > > > kinda of stuck ourselves. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
