After some review and tweaks, this PR from David just merged which got the Python RemoteConnection going on TINKERPOP-1278:
https://github.com/apache/tinkerpop/pull/379 Still some work to do here: 1. Proper testing for Python's RemoteConnection implementation - ran into problems getting everything running under jython which is sorta out of date with the latest python stuff. still trying to figure out what we want to do there with respect to testing. basically we get some dependency errors around Python's fcntl, possibly related to: http://bugs.jython.org/issue1943 - not completely sure what to do about that. On one hand we're using Jython to test client side operations in the JVM environment so that we can get some re-use out of the full gremlin test suite (that's the good side). On the downside, no one will likely execute their gremlin-python code in Jython, we get bound to python 2.x where Jython appears largely stuck and we can't really work our way out of that bug anyway. Anyway, Jython still remains useful on the server where it will let us process python lambdas embedded in traversals - so even though you'd be bound to jython and python 2.x for those lambdas that's still pretty neat. Solving this testing issue is a top priority I think. 2. Making an actual "driver". Dave's PR focused on getting a Python RemoteConnection built, but it would be nice if the code that sent the request to the server was extracted a bit so that users could instantiate a low-level driver to submit raw scripts and custom request messages. In that way it would be in line with how gremlin-driver works for Java and we'd have some consistency for our first two languages in this department. 3. Returning side-effects. I documented the new TraversalOpProcessor in the TINKERPOP-1278 branch ( https://github.com/apache/ tinkerpop/blob/TINKERPOP-1278/docs/src/dev/provider/index. asciidoc#traversal-opprocessor ) and it describes how Traversal side-effects are treated. We need to get that piece working 4. I sense that GraphSON 2.0 is going to be important for this branch. We really need embedded types and the GraphSON 1.0 ones are clunky. I'd hate to take us down a path of trying to build GLVs on the clunkiness only to have that clunkiness removed in the very near future. If anyone has any thoughts on any of this, especially on the issues we face with item 1, please feel free to participate. On Sat, Aug 6, 2016 at 10:52 AM, Stephen Mallette <spmalle...@gmail.com> wrote: > Coming from TinkerPop 2.x with lots of repos, I think we'd want to try to > avoid that (even though this situation is a little different). Before we go > that route I think it would be helpful to understand what the challenges > are in integrating a particular build environment into TinkerPop's. For > example, I imagine that a C# GLV would present some "interesting" issues to > overcome. Perhaps you already have PHP in mind as a "problem". Anyway, I > just think that we should deal with those problems as we identify them and > focus on the less troublesome to integrate languages first. I'm sure that > we can figure out something with Apache when/if the time comes that the > build complexity consumes us. Maybe docker can help solve some of our > build/test environment problems?. > > On Sat, Aug 6, 2016 at 10:28 AM, Dylan Millikin <dylan.milli...@gmail.com> > wrote: > >> That makes perfect sense Stephen. Quick question regarding the last point >> (testing environments and projects). Do you know what apache policies are >> in regards to maintaining multiple repos for "official" plugins/GLVs? This >> might make streamlining testing and updating the package in dependency >> management systems easier (it's actually mandatory for php composer). >> If you don't I can always bring it up with infra. >> >> On Sat, Aug 6, 2016 at 6:47 AM, Stephen Mallette <spmalle...@gmail.com> >> wrote: >> >> > Let me first address this piece: >> > >> > > My understanding around the driver needing modifications is -I'm >> > guessing- mostly to accommodate for the updates in TINKERPOP-1278 rather >> > than from drivers needing to be specific for GLVs. If not let me know. >> > >> > The modifications are mostly to trim down the size of gremlinclient and >> > simplify. gremlinclient allowed for pluggable webscocket implementations >> > and other stuff that adds some complexity/code we could be better >> without >> > in TinkerPop. There are some specific updates needed for full support of >> > GLVs. (i.e. collecting side-effects which i think can remain optional) >> but >> > (I think) David's work can already submit bytecode (if not, then we'll >> need >> > to change that too, but that's pretty trivial. >> > >> > As for the rest, which largely boils down to "consider depending on >> > third-party drivers": >> > >> > I should have included some reasoning around your suggestion in my >> original >> > post as this was a position in my mind that was considered. The problem >> > with depending on third-party projects are the downsides you laid out >> and I >> > think those are more than we should bear. I would also add to that list >> > that the dependency is cyclical. The third party project depends on us >> then >> > TinkerPop depends back on them. I don't imagine that this will >> necessarily >> > cause a "compilation" problem but it doesn't seem quite natural/right. >> > >> > > Would it not be better to have GLV's simply depend on third party >> > projects when they exist and are decently up to date)? >> > >> > The part in parenthesis is part of the problem in the long run. Code >> gets >> > stale, people move on. It's easy to write a gremlin-driver on the >> weekend, >> > but to build a driver on-par with the performance/features of the java >> > driver - that's a difference story. Note that neo4j i think discovered >> this >> > which is why they in-housed their drivers for bolt. >> > >> > > If features are lacking the community can provide patches, etc. >> > >> > We could try and I would think most driver developers would be amenable >> to >> > that, however some might not. Also the direction of an external project, >> > might not match the direction TinkerPop wants. Apache governance doesn't >> > extend to third-party projects so in the worst case, if we hit a wall on >> > some technical issue that the owner of the project doesn't agree with, >> we'd >> > be left with "find something else". In that case, TinkerPop has already >> > spent a lot of time sending patches to that third-party and our time >> would >> > have be wasted. >> > >> > > If a project dies we can always switch it out for another one. >> > >> > I don't know if it's that simple. Again, we'd (TinkerPop) presumably >> > invested time getting a project on par with the java driver in terms of >> > features and performance. So first of all, is there another project to >> > switch to (right now there's not many second options for different >> > languages)? And second, how much effort will there be to re-invest in a >> > different driver to get it consistent with the features/performance of >> the >> > java driver? >> > >> > > I just think that as we get more GLVs, providing CI with the correct >> > environments >> > to run various lower level driver tests in various languages is going >> to be >> > a headache. >> > >> > Agreed. I worried about that too. However, this wont' be a fast process. >> > We're not on a tear to build as many GLVs as possible. We're trying to >> get >> > the model right with gremlin-python, set a high bar for the build >> > automation needed, and then look at how to apply that elsewhere. If we >> can >> > attain that same high bar, then we include a GLV. If we can't, then we >> wait >> > until we can figure out how to. At least that's how I think about it. >> > >> > I'm not sure if any of that changes your position, but I believe the >> risks >> > of depending on third-party drivers is a bit too high - at least higher >> > than us having the extra code and build automation problems of owning >> the >> > drivers ourselves. >> > >> > >> > On Sat, Aug 6, 2016 at 1:26 AM, Dylan Millikin < >> dylan.milli...@gmail.com> >> > wrote: >> > >> > > If I may suggest an alternative. Most (all?) languages have dependency >> > > management. >> > > Would it not be better to have GLV's simply depend on third party >> > projects >> > > (when they exist and are decently up to date)? >> > > These dependencies can be pulled during testing. If features are >> lacking >> > > the community can provide patches, etc. If a project dies we can >> always >> > > switch it out for another one. >> > > When it comes to users, they simply download their GLV and go. Ideally >> > the >> > > GLV would exist as a package in their dependency management system of >> > > choice. (as a side note apt-get/yum install gremlin-server would be >> > > amazing) >> > > >> > > My understanding around the driver needing modifications is -I'm >> > guessing- >> > > mostly to accommodate for the updates in TINKERPOP-1278 rather than >> from >> > > drivers needing to be specific for GLVs. If not let me know. >> > > >> > > This obviously comes with it's own downsides. If a project dies and >> > there's >> > > no replacement we basically have to fall back to your proposal by >> forking >> > > and maintaining it ourselves. I also don't know how licensing would >> work >> > in >> > > this case, but I'm guessing a license change could be akin to the >> project >> > > dying in regards to TP's dependance on it. >> > > >> > > I just think that as we get more GLVs, providing CI with the correct >> > > environments to run various lower level driver tests in various >> languages >> > > is going to be a headache. It would be much better to keep them as >> > separate >> > > projects. >> > > >> > > If I completely missed the original intent of the thread also do let >> me >> > > know. >> > > >> > > >> > > On Fri, Aug 5, 2016 at 4:57 PM, Stephen Mallette < >> spmalle...@gmail.com> >> > > wrote: >> > > >> > > > I mentioned this thread to David Brown and asked if he thought that >> > > > gremlinclient would be a good fit for this issue. He said that he >> > thought >> > > > it would be with some modifications. He's agreed to simplify >> > gremliclient >> > > > with the intent of making it the driver for TinkerPop's >> gremlin-python. >> > > > Assuming no objections, he said that he could look to have a pull >> > request >> > > > to TINKERPOP-1278 by end of next week. >> > > > >> > > > >> > > > >> > > > >> > > > On Thu, Aug 4, 2016 at 1:39 PM, Stephen Mallette < >> spmalle...@gmail.com >> > > >> > > > wrote: >> > > > >> > > > > TinkerPop has long held to the idea that "drivers" would be a bit >> > like >> > > > > graph implementations. TinkerPop builds reference implementation >> of a >> > > > graph >> > > > > and third-parties maintain "other" implementations separately. The >> > same >> > > > > idea has been held for "drivers" in that we have our reference >> > > > > implementation of the Java driver and third-parties produce >> > > > implementations >> > > > > for other languages. >> > > > > >> > > > > GLVs sorta changed things though. Now we have a model where we >> > maintain >> > > > > GLVs of other languages in TinkerPop itself. While somewhat risky >> to >> > > > > include the added code/complexity, maintaining the GLV in >> TinkerPop >> > > has a >> > > > > lot of good advantages for users and for growing our community. We >> > also >> > > > > have come up with ways to manage the risk, by establishing a >> really >> > > nice >> > > > > initial reference implementation of gremlin-python which sets a >> high >> > > bar >> > > > > for new GLVs to come in (i.e. hooked into tinkerpop test suite, >> > > automated >> > > > > build mixed with maven, non-native JVM language focused, etc.). >> > > > > >> > > > > Of course, by introducing GLVs to TinkerPop though, we now have >> the >> > > issue >> > > > > of conflict with TinkerPop only having a single "reference >> > > > implementation" >> > > > > for drivers because without a driver the GLV is pretty useless. >> We've >> > > > > mitigated that in gremlin-python so far with a simple REST >> > > implementation >> > > > > of a driver, but as we advance the capabilities of GLVs that >> approach >> > > > won't >> > > > > be good for the long term. >> > > > > >> > > > > So - it seems like we (TinkerPop) should have a driver to go with >> > each >> > > > > GLV? To me, the upsides are big for users: >> > > > > >> > > > > 1. Users don't have to hunt for drivers elsewhere - they just grab >> > > their >> > > > > GLV and go. I suppose there is still opportunity for third-party >> > > drivers >> > > > to >> > > > > be built, but at least users have an immediate easy option within >> the >> > > > > Apache TinkerPop. >> > > > > 2. Our docs for GLV usage are completely in Apache TinkerPop and >> are >> > > > > "production ready" (i.e. not REST based then go switch to >> something >> > > > > third-party) >> > > > > >> > > > > There are some downsides which basically boil down to "Mo' code >> Mo' >> > > > > problems" but basically: >> > > > > >> > > > > 1. I believe that it will be expected that we maintain some >> parity in >> > > > > terms of features and performance with the drivers. We can't have >> the >> > > > java >> > > > > driver be 10x over the python driver >> > > > > 2. The GLV code base becomes more complex. A high performing >> driver >> > > will >> > > > > not be trivial. >> > > > > 3. A whole class of downsides related to GLVs in general where we >> > don't >> > > > > have the necessary language expertise to maintain it - though - >> > again, >> > > > > mitigated by the high bar we set with our reference implementation >> > for >> > > > > gremlin-python. Also, rather than a negative, one could also see >> this >> > > as >> > > > an >> > > > > opportunity to grow the TinkerPop community. >> > > > > >> > > > > Ultimately, I think all of this is outweighed by the simple fact >> that >> > > we >> > > > > need a driver in Apache TinkerPop to fully test GLVs. Without that >> > > we're >> > > > > kinda of stuck ourselves. >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > > >> > > > >> > > >> > >> > >
