> >but I think > >that's a bit different than TinkerPop promoting that concept as a way for extension
> I do not understand what you mean. Could you collaborate more? I just meant that it's one thing for providers to take it upon themselves to modify the grammar today as a way of extending it but it's different when TinkerPop officially recommends that approach. In this latter case, TinkerPop has to provide support around its recommendation in terms of provider/user documentation, code changes/maintenance, community support, etc. On Wed, Aug 27, 2025 at 10:39 AM Andrii Lomakin <[email protected]> wrote: > Hi. > > >Are you suggesting a different packaging than what is already present in > gremlin-language? isn't that a separate artifact already? > > I propose making it more visible for developers by outlining this in > the documentation. > Sorry, I was not clear enough on this topic. > > >Also, could you say a bit more about how developers would "extend it with > new step definitions by contract"? > > The fact that new steps can be added should be considered during the > design of this grammar. > I have made a quick check, and I do not see any obstacles to adding > new steps right now. > Probably, guidelines should also be provided, which, if developers > follow them, guarantee future compatibility, at least between minor > versions. > I will do those activities anyway. I can take those steps on my own > and share the results in the current thread. > > >but I think > >that's a bit different than TinkerPop promoting that concept as a way for > extension > > I do not understand what you mean. Could you collaborate more? > > > I think Yang is exploring > how the same mechanism could work for custom types (e.g. an OrientDB Record > Id). > > I would be really glad to read about it in a separate thread, so as > not to pollute the current one. > I hope Yang will not mind sharing it with me, by email, or in our > Zulip chat (I will not put the link here to avoid ads). > > > is that it would give tool developers (likeG.V()) better specific graph > database support for their users. > > That was precisely the intention to allow G.V() to use our grammar > extensions with zero effort. > > > perhaps > what you're suggesting is indeed the best way. It would be great to figure > this out. Thanks for putting some thought to it. > > Thank you for such positive feedback. We will continue to work in this > direction, incorporating feedback from this thread and continuing to > drive it. > Will be glad to see feedback from as many participants and Gremlin > users as possible :-) > > On Wed, Aug 27, 2025 at 4:11 PM Stephen Mallette <[email protected]> > wrote: > > > > > 1. ANTLR grammar is distributed as a separate artifact, and users > are > > allowed to extend it with new step definitions by contract. I am not > sure > > about predicates, but that is likely also possible. > > > > Are you suggesting a different packaging than what is already present in > > gremlin-language? isn't that a separate artifact already? > > > > Also, could you say a bit more about how developers would "extend it with > > new step definitions by contract"? I suppose I'm wary of there being lots > > of variations of the grammar out there and might open abuses, > configuration > > challenges and other things that confuse users. I feel like call() may > > already allow for too much independence in the community and less > > collaboration to instill more generalized (but extendable ) first-class > > features in TinkerPop. I suppose providers would be free to modify the > > grammar today and distribute their own versions as things are, but I > think > > that's a bit different than TinkerPop promoting that concept as a way for > > extension. > > > > I do like that you are suggesting mechanisms for "handy" and > > "developer-friendly" interfaces that can simplify extension. Ideally, > we'd > > keep the grammar as it is but figure out guardrailed approaches to offer > > step extension. Note that we do have something along those lines with > > custom strategies where providers can register any custom > TraversalStrategy > > in such a way that the grammar/parser will automatically recognize it and > > it requires no changes to the grammar itself. In this way, > > g.withoutStrategies(CustomStrategy) just works. I think Yang is exploring > > how the same mechanism could work for custom types (e.g. an OrientDB > Record > > Id). I've often wondered if the same would work for custom steps, where > > providers would just register custom steps and the grammar/parser would > > recognize them and convert them to call() somehow internally. At that > point > > providers could process the call() directly or use strategies to convert > > them to concrete custom steps. I presume if that worked for steps it > would > > work for P as well. Perhaps that's worth exploring as an option. > > > > Perhaps one advantage to letting providers extend the grammar in the way > I > > think you are describing, is that it would give tool developers (like > > G.V()) better specific graph database support for their users. Something > > like g.ytdbStep() could be recognized within the tool itself. Of course, > > that's a narrow group of users, but on the other hand they serve a large > > number of end Gremlin users themselves. > > > > All that said, this issue is a problem that needs a solution and perhaps > > what you're suggesting is indeed the best way. It would be great to > figure > > this out. Thanks for putting some thought to it. > > > > On Wed, Aug 27, 2025 at 9:36 AM Andrii Lomakin > > <[email protected]> wrote: > > > > > Good day. > > > > > > > > > This question has already been raised several times, but typically, it > was > > > discussed in the context of a specific extension of steps. > > > > > > > > > Here, I want to propose a general approach that we are going to use in > > > YouTrackDB to extend the Gremlin language. > > > > > > > > > Our intention in this discussion is to receive initial feedback from > the TP > > > team. If they agree, we will eventually contribute part of this > approach or > > > the complete approach in the next versions of the TP framework. > > > > > > > > > We used the following core principle when developing this approach: > > > existing Gremlin tools should be able to take advantage of all the > > > functionality provided by Gremlin extensions. The only difference is > that > > > it > > > will not be as handy for users as using the new features provided. > > > > > > > > > As a result, we paid attention to the two tools already at hand for TP > > > users: > > > > > > 1. DSL annotation processor. > > > 2. The call step. > > > > > > > > > So first of all, if we can not convert our DSL using the previously > > > mentioned annotation processor into the standard steps, we convert it > into > > > the `call` of the service. This service is actually a stub, not a real > > > service. > > > > > > The first parameter of this `call` step is the new step name, and the > rest > > > are arguments specified in free form, supported by TP, and > understandable > > > to humans, as well as the optimization strategy that will convert them > to > > > the real step in the phase of traversal preprocessing. This approach > allows > > > for the addition of new steps and types of arguments, such as > predicates. > > > > > > > > > Let us look at a fake example: If I call `g.addSchemaClass("Test")`, > DSL > > > converts it into the standard step -> `g.call("extDSL", > Map.of("stepName", > > > "addSchemaClass", "className", "Test"))`, which, during the Traversal > > > preprocessing phase, is converted to the AddSchemaClass step by the > related > > > optimization strategy. > > > > > > > > > This approach: > > > > > > 1. Allows the use of new steps in already existing tools. > > > 2. The straightforward design to follow to extend Gremlin drivers > > > allows for > > > quick incorporation of those drivers by the tool's developers and > > > application developers. > > > 3. It allows DB vendors to add new steps themselves easily. > > > > > > > > > However, this does not solve the issue when DSL is passed as text. > > > > > > To solve this last-mile issue, we propose the following approach: > > > > > > 1. ANTLR grammar is distributed as a separate artifact, and users > are > > > allowed to extend it with new step definitions by contract. I am not > > > sure > > > about predicates, but that is likely also possible. > > > 2. A new Customizer that provides instances of GremlinLexer, > > > GermlinParser > > > and a new interface, GremlinASTTransformer, if specified is > introduced. > > > 3. The responsibility of a new GremlinASTTransformer class would be > to > > > transform the AST provided by the extended parser into the standard > AST > > > generated by the standard Gremlin ANTLR grammar. > > > 4. After that, the same optimization routine that converts fake > service > > > into the new Step instance is applied. > > > > > > > > > This routine can also be made more developer-friendly. > > > > > > > > > For example, there could be handy automations that will allow you not > to > > > write AST transformation code manually, but merely map it to the > > > specialized TraversalStrategy interface implementation that will > provide > > > all information needed for both transformations: > > > > > > 1. Forward -> from the `call` step to the new Step instance > conversion. > > > 2. Backward -> from the Gremllin step in the extended AST to the > `call ` > > > step in Traversal. > > > > > > Moreover, a new step, "stepExt," can be introduced to make the Gremlin > > > extension more semantically clear. In such a case, a new type of > > > TraversalStrategy can be specialized to work only with a given type of > > > step. > > > > > > > > > To recap, after this massive wall of text, the benefits of this > approach > > > are: > > > > > > 1. Despite being somewhat convoluted, extending the Gremlin language > > > will not require spending a large amount of resources for developers > > > with > > > knowledge of ANTLR, especially if AbstractGremlinASTTransformer and > > > StepExtensionTraversalStrategy are implemented, which will do all > the > > > heavy > > > lifting of step transformation. > > > 2. Existing gremlin tools can be used to call new steps, though less > > > conveniently than in the presence of new steps in Traveral. > > > > > > Looking forward to reading your feedback. > > > >
