Yeah - (1) is stinky. The more I think about it the more I don't think there is value in validating gremlin-python over Jython, if Jython is not up to date and, perhaps more importantly, not the environment users will run their code in. Doesn't seem to have much value in that case. Perhaps for 3.2.2 we don't have to try to tackle this testing issue completely.
On Tue, Aug 16, 2016 at 9:31 AM, Dylan Millikin <dylan.milli...@gmail.com> wrote: > For 1) maybe now is a good time to look into using docker for testing? That > would have it's own issues in the fact that testing would have to be > written directly in python so you would loose reusability for future GLVs > though. > > I'm not sure how to build a test suit that would either work across all > languages or enforce behaviors in all GLVs. I know this came up before I > just can't seem to find those issues. Might not be searching properly. > > > On Mon, Aug 15, 2016 at 4:29 PM, Stephen Mallette <spmalle...@gmail.com> > wrote: > > > After some review and tweaks, this PR from David just merged which got > the > > Python RemoteConnection going on TINKERPOP-1278: > > > > https://github.com/apache/tinkerpop/pull/379 > > > > Still some work to do here: > > > > 1. Proper testing for Python's RemoteConnection implementation - ran into > > problems getting everything running under jython which is sorta out of > date > > with the latest python stuff. still trying to figure out what we want to > do > > there with respect to testing. basically we get some dependency errors > > around Python's fcntl, possibly related to: http://bugs.jython.org/ > > issue1943 > > - not completely sure what to do about that. On one hand we're using > Jython > > to test client side operations in the JVM environment so that we can get > > some re-use out of the full gremlin test suite (that's the good side). On > > the downside, no one will likely execute their gremlin-python code in > > Jython, we get bound to python 2.x where Jython appears largely stuck and > > we can't really work our way out of that bug anyway. Anyway, Jython > still > > remains useful on the server where it will let us process python lambdas > > embedded in traversals - so even though you'd be bound to jython and > python > > 2.x for those lambdas that's still pretty neat. Solving this testing > issue > > is a top priority I think. > > > > 2. Making an actual "driver". Dave's PR focused on getting a Python > > RemoteConnection built, but it would be nice if the code that sent the > > request to the server was extracted a bit so that users could > instantiate a > > low-level driver to submit raw scripts and custom request messages. In > that > > way it would be in line with how gremlin-driver works for Java and we'd > > have some consistency for our first two languages in this department. > > > > 3. Returning side-effects. I documented the new TraversalOpProcessor in > the > > TINKERPOP-1278 branch ( https://github.com/apache/ > > tinkerpop/blob/TINKERPOP-1278/docs/src/dev/provider/index. > > asciidoc#traversal-opprocessor ) and it describes how Traversal > > side-effects are treated. We need to get that piece working > > > > 4. I sense that GraphSON 2.0 is going to be important for this branch. We > > really need embedded types and the GraphSON 1.0 ones are clunky. I'd hate > > to take us down a path of trying to build GLVs on the clunkiness only to > > have that clunkiness removed in the very near future. > > > > If anyone has any thoughts on any of this, especially on the issues we > face > > with item 1, please feel free to participate. > > > > > > > > On Sat, Aug 6, 2016 at 10:52 AM, Stephen Mallette <spmalle...@gmail.com> > > wrote: > > > > > Coming from TinkerPop 2.x with lots of repos, I think we'd want to try > to > > > avoid that (even though this situation is a little different). Before > we > > go > > > that route I think it would be helpful to understand what the > challenges > > > are in integrating a particular build environment into TinkerPop's. For > > > example, I imagine that a C# GLV would present some "interesting" > issues > > to > > > overcome. Perhaps you already have PHP in mind as a "problem". Anyway, > I > > > just think that we should deal with those problems as we identify them > > and > > > focus on the less troublesome to integrate languages first. I'm sure > > that > > > we can figure out something with Apache when/if the time comes that the > > > build complexity consumes us. Maybe docker can help solve some of our > > > build/test environment problems?. > > > > > > On Sat, Aug 6, 2016 at 10:28 AM, Dylan Millikin < > > dylan.milli...@gmail.com> > > > wrote: > > > > > >> That makes perfect sense Stephen. Quick question regarding the last > > point > > >> (testing environments and projects). Do you know what apache policies > > are > > >> in regards to maintaining multiple repos for "official" plugins/GLVs? > > This > > >> might make streamlining testing and updating the package in dependency > > >> management systems easier (it's actually mandatory for php composer). > > >> If you don't I can always bring it up with infra. > > >> > > >> On Sat, Aug 6, 2016 at 6:47 AM, Stephen Mallette < > spmalle...@gmail.com> > > >> wrote: > > >> > > >> > Let me first address this piece: > > >> > > > >> > > My understanding around the driver needing modifications is -I'm > > >> > guessing- mostly to accommodate for the updates in TINKERPOP-1278 > > rather > > >> > than from drivers needing to be specific for GLVs. If not let me > know. > > >> > > > >> > The modifications are mostly to trim down the size of gremlinclient > > and > > >> > simplify. gremlinclient allowed for pluggable webscocket > > implementations > > >> > and other stuff that adds some complexity/code we could be better > > >> without > > >> > in TinkerPop. There are some specific updates needed for full > support > > of > > >> > GLVs. (i.e. collecting side-effects which i think can remain > optional) > > >> but > > >> > (I think) David's work can already submit bytecode (if not, then > we'll > > >> need > > >> > to change that too, but that's pretty trivial. > > >> > > > >> > As for the rest, which largely boils down to "consider depending on > > >> > third-party drivers": > > >> > > > >> > I should have included some reasoning around your suggestion in my > > >> original > > >> > post as this was a position in my mind that was considered. The > > problem > > >> > with depending on third-party projects are the downsides you laid > out > > >> and I > > >> > think those are more than we should bear. I would also add to that > > list > > >> > that the dependency is cyclical. The third party project depends on > us > > >> then > > >> > TinkerPop depends back on them. I don't imagine that this will > > >> necessarily > > >> > cause a "compilation" problem but it doesn't seem quite > natural/right. > > >> > > > >> > > Would it not be better to have GLV's simply depend on third party > > >> > projects when they exist and are decently up to date)? > > >> > > > >> > The part in parenthesis is part of the problem in the long run. Code > > >> gets > > >> > stale, people move on. It's easy to write a gremlin-driver on the > > >> weekend, > > >> > but to build a driver on-par with the performance/features of the > java > > >> > driver - that's a difference story. Note that neo4j i think > discovered > > >> this > > >> > which is why they in-housed their drivers for bolt. > > >> > > > >> > > If features are lacking the community can provide patches, etc. > > >> > > > >> > We could try and I would think most driver developers would be > > amenable > > >> to > > >> > that, however some might not. Also the direction of an external > > project, > > >> > might not match the direction TinkerPop wants. Apache governance > > doesn't > > >> > extend to third-party projects so in the worst case, if we hit a > wall > > on > > >> > some technical issue that the owner of the project doesn't agree > with, > > >> we'd > > >> > be left with "find something else". In that case, TinkerPop has > > already > > >> > spent a lot of time sending patches to that third-party and our time > > >> would > > >> > have be wasted. > > >> > > > >> > > If a project dies we can always switch it out for another one. > > >> > > > >> > I don't know if it's that simple. Again, we'd (TinkerPop) presumably > > >> > invested time getting a project on par with the java driver in terms > > of > > >> > features and performance. So first of all, is there another project > to > > >> > switch to (right now there's not many second options for different > > >> > languages)? And second, how much effort will there be to re-invest > in > > a > > >> > different driver to get it consistent with the features/performance > of > > >> the > > >> > java driver? > > >> > > > >> > > I just think that as we get more GLVs, providing CI with the > correct > > >> > environments > > >> > to run various lower level driver tests in various languages is > going > > >> to be > > >> > a headache. > > >> > > > >> > Agreed. I worried about that too. However, this wont' be a fast > > process. > > >> > We're not on a tear to build as many GLVs as possible. We're trying > to > > >> get > > >> > the model right with gremlin-python, set a high bar for the build > > >> > automation needed, and then look at how to apply that elsewhere. If > we > > >> can > > >> > attain that same high bar, then we include a GLV. If we can't, then > we > > >> wait > > >> > until we can figure out how to. At least that's how I think about > it. > > >> > > > >> > I'm not sure if any of that changes your position, but I believe the > > >> risks > > >> > of depending on third-party drivers is a bit too high - at least > > higher > > >> > than us having the extra code and build automation problems of > owning > > >> the > > >> > drivers ourselves. > > >> > > > >> > > > >> > On Sat, Aug 6, 2016 at 1:26 AM, Dylan Millikin < > > >> dylan.milli...@gmail.com> > > >> > wrote: > > >> > > > >> > > If I may suggest an alternative. Most (all?) languages have > > dependency > > >> > > management. > > >> > > Would it not be better to have GLV's simply depend on third party > > >> > projects > > >> > > (when they exist and are decently up to date)? > > >> > > These dependencies can be pulled during testing. If features are > > >> lacking > > >> > > the community can provide patches, etc. If a project dies we can > > >> always > > >> > > switch it out for another one. > > >> > > When it comes to users, they simply download their GLV and go. > > Ideally > > >> > the > > >> > > GLV would exist as a package in their dependency management system > > of > > >> > > choice. (as a side note apt-get/yum install gremlin-server would > be > > >> > > amazing) > > >> > > > > >> > > My understanding around the driver needing modifications is -I'm > > >> > guessing- > > >> > > mostly to accommodate for the updates in TINKERPOP-1278 rather > than > > >> from > > >> > > drivers needing to be specific for GLVs. If not let me know. > > >> > > > > >> > > This obviously comes with it's own downsides. If a project dies > and > > >> > there's > > >> > > no replacement we basically have to fall back to your proposal by > > >> forking > > >> > > and maintaining it ourselves. I also don't know how licensing > would > > >> work > > >> > in > > >> > > this case, but I'm guessing a license change could be akin to the > > >> project > > >> > > dying in regards to TP's dependance on it. > > >> > > > > >> > > I just think that as we get more GLVs, providing CI with the > correct > > >> > > environments to run various lower level driver tests in various > > >> languages > > >> > > is going to be a headache. It would be much better to keep them as > > >> > separate > > >> > > projects. > > >> > > > > >> > > If I completely missed the original intent of the thread also do > let > > >> me > > >> > > know. > > >> > > > > >> > > > > >> > > On Fri, Aug 5, 2016 at 4:57 PM, Stephen Mallette < > > >> spmalle...@gmail.com> > > >> > > wrote: > > >> > > > > >> > > > I mentioned this thread to David Brown and asked if he thought > > that > > >> > > > gremlinclient would be a good fit for this issue. He said that > he > > >> > thought > > >> > > > it would be with some modifications. He's agreed to simplify > > >> > gremliclient > > >> > > > with the intent of making it the driver for TinkerPop's > > >> gremlin-python. > > >> > > > Assuming no objections, he said that he could look to have a > pull > > >> > request > > >> > > > to TINKERPOP-1278 by end of next week. > > >> > > > > > >> > > > > > >> > > > > > >> > > > > > >> > > > On Thu, Aug 4, 2016 at 1:39 PM, Stephen Mallette < > > >> spmalle...@gmail.com > > >> > > > > >> > > > wrote: > > >> > > > > > >> > > > > TinkerPop has long held to the idea that "drivers" would be a > > bit > > >> > like > > >> > > > > graph implementations. TinkerPop builds reference > implementation > > >> of a > > >> > > > graph > > >> > > > > and third-parties maintain "other" implementations separately. > > The > > >> > same > > >> > > > > idea has been held for "drivers" in that we have our reference > > >> > > > > implementation of the Java driver and third-parties produce > > >> > > > implementations > > >> > > > > for other languages. > > >> > > > > > > >> > > > > GLVs sorta changed things though. Now we have a model where we > > >> > maintain > > >> > > > > GLVs of other languages in TinkerPop itself. While somewhat > > risky > > >> to > > >> > > > > include the added code/complexity, maintaining the GLV in > > >> TinkerPop > > >> > > has a > > >> > > > > lot of good advantages for users and for growing our > community. > > We > > >> > also > > >> > > > > have come up with ways to manage the risk, by establishing a > > >> really > > >> > > nice > > >> > > > > initial reference implementation of gremlin-python which sets > a > > >> high > > >> > > bar > > >> > > > > for new GLVs to come in (i.e. hooked into tinkerpop test > suite, > > >> > > automated > > >> > > > > build mixed with maven, non-native JVM language focused, > etc.). > > >> > > > > > > >> > > > > Of course, by introducing GLVs to TinkerPop though, we now > have > > >> the > > >> > > issue > > >> > > > > of conflict with TinkerPop only having a single "reference > > >> > > > implementation" > > >> > > > > for drivers because without a driver the GLV is pretty > useless. > > >> We've > > >> > > > > mitigated that in gremlin-python so far with a simple REST > > >> > > implementation > > >> > > > > of a driver, but as we advance the capabilities of GLVs that > > >> approach > > >> > > > won't > > >> > > > > be good for the long term. > > >> > > > > > > >> > > > > So - it seems like we (TinkerPop) should have a driver to go > > with > > >> > each > > >> > > > > GLV? To me, the upsides are big for users: > > >> > > > > > > >> > > > > 1. Users don't have to hunt for drivers elsewhere - they just > > grab > > >> > > their > > >> > > > > GLV and go. I suppose there is still opportunity for > third-party > > >> > > drivers > > >> > > > to > > >> > > > > be built, but at least users have an immediate easy option > > within > > >> the > > >> > > > > Apache TinkerPop. > > >> > > > > 2. Our docs for GLV usage are completely in Apache TinkerPop > and > > >> are > > >> > > > > "production ready" (i.e. not REST based then go switch to > > >> something > > >> > > > > third-party) > > >> > > > > > > >> > > > > There are some downsides which basically boil down to "Mo' > code > > >> Mo' > > >> > > > > problems" but basically: > > >> > > > > > > >> > > > > 1. I believe that it will be expected that we maintain some > > >> parity in > > >> > > > > terms of features and performance with the drivers. We can't > > have > > >> the > > >> > > > java > > >> > > > > driver be 10x over the python driver > > >> > > > > 2. The GLV code base becomes more complex. A high performing > > >> driver > > >> > > will > > >> > > > > not be trivial. > > >> > > > > 3. A whole class of downsides related to GLVs in general where > > we > > >> > don't > > >> > > > > have the necessary language expertise to maintain it - though > - > > >> > again, > > >> > > > > mitigated by the high bar we set with our reference > > implementation > > >> > for > > >> > > > > gremlin-python. Also, rather than a negative, one could also > see > > >> this > > >> > > as > > >> > > > an > > >> > > > > opportunity to grow the TinkerPop community. > > >> > > > > > > >> > > > > Ultimately, I think all of this is outweighed by the simple > fact > > >> that > > >> > > we > > >> > > > > need a driver in Apache TinkerPop to fully test GLVs. Without > > that > > >> > > we're > > >> > > > > kinda of stuck ourselves. > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > > >> > > > > > >> > > > > >> > > > >> > > > > > > > > >
