Re: code generation and RDF support in TinkerPop 4

[Joshua Shinavier]

Hi Pieter,

You give some good motivation for a formal schema language. My proposal for
an abstract data model for TinkerPop was, and is Algebraic Property Graphs (
paper ), of which Dragon's data model is
an extension. APG is broader than typical property graphs (e.g. allowing
hyperelements, nested data, and other features which are uncommon or
unknown in connection with TinkerPop), so the best answer to your question
is probably "a variant of APG with restrictions".

Given a formal specification of TinkerPop's data model, we can be very
flexible with respect to concrete syntaxes. Dragon has its YAML syntax, and
the new framework will probably support a slightly different YAML syntax,
but you can specify graph schemas in a variety of languages (the current
tooling will read schemas expressed in YAML, JSON, Thrift, or Protobuf),
and you can express graph data in a variety of languages. What the formal
specification of the data model, and the mappings, give you is the ability
to map schemas and data transparently between the formats, so you can use
whatever is most appropriate to your application.

Btw. at some point, you'll see a schema for property graph features appear
in the branch -- a kind of TP4 successor to Graph.Features
.
This will be a small language for declaring the specific refinement of APG
/ the TinkerPop data model which is supported by a given property graph
implementation. That will help you understand not only a single graph, but
also the characteristic class of graphs for a given vendor, adapter, etc.

Josh




On Thu, Jun 3, 2021 at 11:43 AM pieter gmail 
wrote:

> Hi,
>
> I kinda lost track of what we discussed previously.
> Did we come to a decision regarding what language we are going to use to
> describe the structure of the graph.
>
> yaml,xsd,uml,yang or some category theory based language?
>
> From my understanding this would be the biggest change in tp4. A TinkerPop
> graph will no be longer a tangle of endless vertices and edges but instead
> can, optionally, be well defined and constrained. This way an engineer can,
> long after the original creators of a graph have left, immediately
> understand the graph, without needing to write a single query.
>
> Thanks
> Pieter
>
>
>
>
> On Thu, 2021-06-03 at 09:59 -0700, Joshua Shinavier wrote:
>
> Hi Pieter,
>
>
> On Thu, Jun 3, 2021 at 9:40 AM pieter gmail 
> wrote:
>
> Hi,
>
> Just to understand a bit better whats going on.
>
> Did you hand write the dragon yaml with the antlr grammar as input?
>
>
>
> Yes, the YAML was written by hand, and based pretty closely on Gremlin.g4.
> You can see Stephen's ANTLR definitions inline with the YAML as comments. I
> also took some direction from the Java API.
>
>
>
>
> Did you generate the java classes from the yaml using dragon or
> something else?
>
>
>
> Yes, the Java classes are currently generated using Dragon. I'm limiting
> the generated code to Java for now (other possible targets being Scala and
> Haskell) just to keep diffs to a reasonable size, and because a new,
> open-source solution is needed to replace Dragon. My current thinking is
> that the new transformation framework will be separate from TinkerPop, as
> it will serve non-graph as well as graph use cases. For now, you can think
> of the code generation as a bootstrapping strategy.
>
> Josh
>
>
>
>
>
> Thanks
> Pieter
>
> On Thu, 2021-06-03 at 07:48 -0700, Joshua Shinavier wrote:
> > Hello all,
> >
> > I would like to take some concrete steps toward the TinkerPop 4
> > interoperability goals I've stated a few times (e.g. see TinkerPop
> > 2020
> > from last year). At
> > a
> > meetup  a
> > couple
> > of months ago, I demonstrated an approach for generating TinkerPop
> > APIs
> > consistently into different languages. I have started to check in
> > some of
> > that generated code in a branch (see my commits here
> > <
> https://github.com/apache/tinkerpop/commits/TINKERPOP-2563-language/gremlin-language
> > >)
> > and add bits and pieces for RDF support, as well.
> >
> > The Apache Software Foundation asks us to discuss any significant
> > changes
> > to the code base on the dev list. Since these steps toward TP4 will
> > be
> > major changes if and when they are merged into the master branch, I
> > will
> > start discussing them here. Expect occasional emails from me about
> > the
> > various things I will be doing in the branch. I absolutely invite
> > comments,
> > feedback, and actual discussion on these design proposals, but even
> > if it's
> > just me issuing self-affirming statements into the void like the King
> > of
> > Pointland, I will just carry on, because that's how this process
> > works.
> >
> > A brief summary of the changes so far:
> >
> >
> >- *Abstract 

[jira] [Commented] (TINKERPOP-2507) Remove requirement that Graph implementations must filter on homogeneous identifiers

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TINKERPOP-2507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17356661#comment-17356661
 ] 

ASF GitHub Bot commented on TINKERPOP-2507:
---

spmallette opened a new pull request #1426:
URL: https://github.com/apache/tinkerpop/pull/1426


   https://issues.apache.org/jira/browse/TINKERPOP-2507
   
   Upgrade docs in this commit explain this change fairly well. There doesn't 
seem to be a need to be so prescriptive about the types that can be filtered on 
and it should be up to each graph to decide how they handle such input. Removed 
the general exception, tests and updated the behavior of graphs to support the 
change.
   
   Decided to put this on 3.6.0 rather than bring what amounts to a behavior 
change in the middle of 3.4.x/3.5.x. 
   
   All tests pass with `docker/build.sh -t -n -i`
   
   VOTE +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Remove requirement that Graph implementations must filter on homogeneous 
> identifiers
> 
>
> Key: TINKERPOP-2507
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2507
> Project: TinkerPop
>  Issue Type: Bug
>  Components: structure
>Affects Versions: 3.4.9
>Reporter: Christopher Smith
>Assignee: Stephen Mallette
>Priority: Major
>  Labels: breaking
>
> In {{TinkerGraph#validateHomogenousIds(List)}}, TinkerGraph _explicitly_ 
> checks for identical ID types among all IDs passed in and provides a 
> misleading error message if they mismatch. In fact, this is a semantic error; 
> TinkerGraph quite happily supports heterogeneous element ID types.
> In my application, vertex IDs are generally explicitly assigned using 
> UUID-as-string values, and so the IDs are primarily of type {{String}}. 
> However, my test infrastructure has a few bookkeeping nodes that I'm not 
> manually writing IDs to, and TinkerGraph assigns them {{Long}} IDs. These 
> mixed IDs work _just fine_, but when I try to pass an {{Object...}} to {{V}}, 
> TinkerGraph throws.
> Mixing ID types is perfectly legal (if not, in most cases, practically 
> advisable), and TinkerGraph should not throw when presented with a mixed 
> array of valid IDs.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: code generation and RDF support in TinkerPop 4

[pieter gmail]

Hi,

I kinda lost track of what we discussed previously.
Did we come to a decision regarding what language we are going to use
to describe the structure of the graph.

yaml,xsd,uml,yang or some category theory based language?

>From my understanding this would be the biggest change in tp4. A
TinkerPop graph will no be longer a tangle of endless vertices and
edges but instead can, optionally, be well defined and constrained.
This way an engineer can, long after the original creators of a graph
have left, immediately understand the graph, without needing to write a
single query.

Thanks
Pieter




On Thu, 2021-06-03 at 09:59 -0700, Joshua Shinavier wrote:
> Hi Pieter,
> 
> 
> On Thu, Jun 3, 2021 at 9:40 AM pieter gmail 
> wrote:
> > Hi,
> > 
> > Just to understand a bit better whats going on.
> > 
> > Did you hand write the dragon yaml with the antlr grammar as input?
> > 
> 
> 
> 
> Yes, the YAML was written by hand, and based pretty closely on
> Gremlin.g4. You can see Stephen's ANTLR definitions inline with the
> YAML as comments. I also took some direction from the Java API.
> 
> 
>  
> > Did you generate the java classes from the yaml using dragon or
> > something else?
> > 
> 
> 
> 
> Yes, the Java classes are currently generated using Dragon. I'm
> limiting the generated code to Java for now (other possible targets
> being Scala and Haskell) just to keep diffs to a reasonable size, and
> because a new, open-source solution is needed to replace Dragon. My
> current thinking is that the new transformation framework will be
> separate from TinkerPop, as it will serve non-graph as well as graph
> use cases. For now, you can think of the code generation as a
> bootstrapping strategy.
> 
> Josh
> 
> 
>  
> > 
> > Thanks
> > Pieter
> > 
> > On Thu, 2021-06-03 at 07:48 -0700, Joshua Shinavier wrote:
> > > Hello all,
> > > 
> > > I would like to take some concrete steps toward the TinkerPop 4
> > > interoperability goals I've stated a few times (e.g. see
> > TinkerPop
> > > 2020
> > > from last
> > year). At
> > > a
> > > meetup 
> > a
> > > couple
> > > of months ago, I demonstrated an approach for generating
> > TinkerPop
> > > APIs
> > > consistently into different languages. I have started to check in
> > > some of
> > > that generated code in a branch (see my commits here
> > >
> >
>  > > >)
> > > and add bits and pieces for RDF support, as well.
> > > 
> > > The Apache Software Foundation asks us to discuss any significant
> > > changes
> > > to the code base on the dev list. Since these steps toward TP4
> > will
> > > be
> > > major changes if and when they are merged into the master branch,
> > I
> > > will
> > > start discussing them here. Expect occasional emails from me
> > about
> > > the
> > > various things I will be doing in the branch. I absolutely invite
> > > comments,
> > > feedback, and actual discussion on these design proposals, but
> > even
> > > if it's
> > > just me issuing self-affirming statements into the void like the
> > King
> > > of
> > > Pointland, I will just carry on, because that's how this process
> > > works.
> > > 
> > > A brief summary of the changes so far:
> > > 
> > > 
> > >    - *Abstract specification of Gremlin traversals*. I have
> > turned
> > >    Stephen's Gremlin.g4
> > >   
> > >
> >
>  > > >
> > >    ANTLR grammar into an abstract specification of Gremlin
> > traversal
> > > syntax
> > >    using the Dragon (YAML-based) format. Unfortunately, it is
> > looking
> > > very
> > >    unlikely that Dragon will become available as open-source
> > > software, so you
> > >    can expect this YAML format to change just slightly once we
> > have a
> > > new
> > >    Dragon-like tool for schema and data transformations. More on
> > that
> > > later.
> > >    Right now, the syntax specification can be found here
> > >   
> > >
> >
>  > > >,
> > >    although the file path might change in the future.
> > > 
> > > 
> > >    - *Traversal DTOs*. Based on the abstract specification, I
> > have
> > >    generated Java classes for building and working with
> > traversals.
> > > The
> > >    generated files can currently be found here
> > >   
> > >
> >
>  > > >.
> > >    These are essentially POJOs or DTO classes, with special
> > > boilerplate
> > >    methods for equality, pattern matching over alternative
> > > constructors, and
> > >    modification by copying (since the instances are 

Re: code generation and RDF support in TinkerPop 4

[Joshua Shinavier]

Hi Pieter,


On Thu, Jun 3, 2021 at 9:40 AM pieter gmail  wrote:

> Hi,
>
> Just to understand a bit better whats going on.
>
> Did you hand write the dragon yaml with the antlr grammar as input?
>


Yes, the YAML was written by hand, and based pretty closely on Gremlin.g4.
You can see Stephen's ANTLR definitions inline with the YAML as comments. I
also took some direction from the Java API.




> Did you generate the java classes from the yaml using dragon or
> something else?
>


Yes, the Java classes are currently generated using Dragon. I'm limiting
the generated code to Java for now (other possible targets being Scala and
Haskell) just to keep diffs to a reasonable size, and because a new,
open-source solution is needed to replace Dragon. My current thinking is
that the new transformation framework will be separate from TinkerPop, as
it will serve non-graph as well as graph use cases. For now, you can think
of the code generation as a bootstrapping strategy.

Josh




>
> Thanks
> Pieter
>
> On Thu, 2021-06-03 at 07:48 -0700, Joshua Shinavier wrote:
> > Hello all,
> >
> > I would like to take some concrete steps toward the TinkerPop 4
> > interoperability goals I've stated a few times (e.g. see TinkerPop
> > 2020
> > from last year). At
> > a
> > meetup  a
> > couple
> > of months ago, I demonstrated an approach for generating TinkerPop
> > APIs
> > consistently into different languages. I have started to check in
> > some of
> > that generated code in a branch (see my commits here
> > <
> https://github.com/apache/tinkerpop/commits/TINKERPOP-2563-language/gremlin-language
> > >)
> > and add bits and pieces for RDF support, as well.
> >
> > The Apache Software Foundation asks us to discuss any significant
> > changes
> > to the code base on the dev list. Since these steps toward TP4 will
> > be
> > major changes if and when they are merged into the master branch, I
> > will
> > start discussing them here. Expect occasional emails from me about
> > the
> > various things I will be doing in the branch. I absolutely invite
> > comments,
> > feedback, and actual discussion on these design proposals, but even
> > if it's
> > just me issuing self-affirming statements into the void like the King
> > of
> > Pointland, I will just carry on, because that's how this process
> > works.
> >
> > A brief summary of the changes so far:
> >
> >
> >- *Abstract specification of Gremlin traversals*. I have turned
> >Stephen's Gremlin.g4
> >
> > <
> https://github.com/apache/tinkerpop/blob/TINKERPOP-2563-language/gremlin-language/src/main/antlr4/Gremlin.g4
> > >
> >ANTLR grammar into an abstract specification of Gremlin traversal
> > syntax
> >using the Dragon (YAML-based) format. Unfortunately, it is looking
> > very
> >unlikely that Dragon will become available as open-source
> > software, so you
> >can expect this YAML format to change just slightly once we have a
> > new
> >Dragon-like tool for schema and data transformations. More on that
> > later.
> >Right now, the syntax specification can be found here
> >
> > <
> https://github.com/apache/tinkerpop/tree/TINKERPOP-2563-language/gremlin-language/src/main/yaml/org/apache/tinkerpop/gremlin/language/model
> > >,
> >although the file path might change in the future.
> >
> >
> >- *Traversal DTOs*. Based on the abstract specification, I have
> >generated Java classes for building and working with traversals.
> > The
> >generated files can currently be found here
> >
> > <
> https://github.com/apache/tinkerpop/tree/TINKERPOP-2563-language/gremlin-language/src/gen/java/org/apache/tinkerpop/gremlin/language/model
> > >.
> >These are essentially POJOs or DTO classes, with special
> > boilerplate
> >methods for equality, pattern matching over alternative
> > constructors, and
> >modification by copying (since the instances are immutable). These
> > classes
> >allow you to build traversals in a declarative way, while all of
> > the logic
> >for evaluating traversals goes elsewhere. Support for
> > serialization and
> >deserialization for traversals is to be added in the future -- and
> > the same
> >goes for all other classes generated in this way.
> >
> >
> >- *RDF 1.1 concepts model*. RDF support was part of TinkerPop from
> > the
> >beginning, but it was de-emphasized for TinkerPop 3 due to other
> > priorities
> >such as OLAP. For years, developers have been asking us for better
> >interoperability with RDF. While we do have some query-level
> > support for
> >RDF these days in sparql-gremlin, we no longer have any data-level
> > support,
> >e.g. supporting loading RDF data into a property graph and getting
> > it back
> >out, evaluating Gremlin traversals over RDF datasets, etc. These
> > things are
> >not especially hard to do, in certain limited ways, but 

Re: code generation and RDF support in TinkerPop 4

[pieter gmail]

Hi,

Just to understand a bit better whats going on.

Did you hand write the dragon yaml with the antlr grammar as input?
Did you generate the java classes from the yaml using dragon or
something else?

Thanks
Pieter

On Thu, 2021-06-03 at 07:48 -0700, Joshua Shinavier wrote:
> Hello all,
> 
> I would like to take some concrete steps toward the TinkerPop 4
> interoperability goals I've stated a few times (e.g. see TinkerPop
> 2020
> from last year). At
> a
> meetup  a
> couple
> of months ago, I demonstrated an approach for generating TinkerPop
> APIs
> consistently into different languages. I have started to check in
> some of
> that generated code in a branch (see my commits here
>  >)
> and add bits and pieces for RDF support, as well.
> 
> The Apache Software Foundation asks us to discuss any significant
> changes
> to the code base on the dev list. Since these steps toward TP4 will
> be
> major changes if and when they are merged into the master branch, I
> will
> start discussing them here. Expect occasional emails from me about
> the
> various things I will be doing in the branch. I absolutely invite
> comments,
> feedback, and actual discussion on these design proposals, but even
> if it's
> just me issuing self-affirming statements into the void like the King
> of
> Pointland, I will just carry on, because that's how this process
> works.
> 
> A brief summary of the changes so far:
> 
> 
>    - *Abstract specification of Gremlin traversals*. I have turned
>    Stephen's Gremlin.g4
>   
>  >
>    ANTLR grammar into an abstract specification of Gremlin traversal
> syntax
>    using the Dragon (YAML-based) format. Unfortunately, it is looking
> very
>    unlikely that Dragon will become available as open-source
> software, so you
>    can expect this YAML format to change just slightly once we have a
> new
>    Dragon-like tool for schema and data transformations. More on that
> later.
>    Right now, the syntax specification can be found here
>   
>  >,
>    although the file path might change in the future.
> 
> 
>    - *Traversal DTOs*. Based on the abstract specification, I have
>    generated Java classes for building and working with traversals.
> The
>    generated files can currently be found here
>   
>  >.
>    These are essentially POJOs or DTO classes, with special
> boilerplate
>    methods for equality, pattern matching over alternative
> constructors, and
>    modification by copying (since the instances are immutable). These
> classes
>    allow you to build traversals in a declarative way, while all of
> the logic
>    for evaluating traversals goes elsewhere. Support for
> serialization and
>    deserialization for traversals is to be added in the future -- and
> the same
>    goes for all other classes generated in this way.
> 
> 
>    - *RDF 1.1 concepts model*. RDF support was part of TinkerPop from
> the
>    beginning, but it was de-emphasized for TinkerPop 3 due to other
> priorities
>    such as OLAP. For years, developers have been asking us for better
>    interoperability with RDF. While we do have some query-level
> support for
>    RDF these days in sparql-gremlin, we no longer have any data-level
> support,
>    e.g. supporting loading RDF data into a property graph and getting
> it back
>    out, evaluating Gremlin traversals over RDF datasets, etc. These
> things are
>    not especially hard to do, in certain limited ways, but our old
> approach of
>    writing adapters like GraphSail
>   
> ,
>    SailGraph
>   
> ,
> and
>    PropertyGraphSail
>   
>  >
>    in Java, with no support for other languages, does not seem
> appropriate for
>    TinkerPop 4. Also, those early mappings were extremely
> underspecified in a
>    formal sense -- good enough for some practical applications, but
> not good
>    enough for anything requiring inference, optimization, or
> composition with
>    other mappings. To that end, I am starting to add abstract
> specifications
>    for RDF along the lines of the Gremlin specifications I described
> above.
>    The first of these, a specification of RDF 1.1 Concepts, can
> currently be
>    found here
>   
> 

[jira] [Closed] (TINKERPOP-2551) Setup scripts to publish Gremint to npm

[Stephen Mallette (Jira)]

 [ 
https://issues.apache.org/jira/browse/TINKERPOP-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Mallette closed TINKERPOP-2551.
---
Fix Version/s: 3.6.0
   3.5.1
 Assignee: Stephen Mallette
   Resolution: Done

> Setup scripts to publish Gremint to npm
> ---
>
> Key: TINKERPOP-2551
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2551
> Project: TinkerPop
>  Issue Type: Improvement
>  Components: gremlint
>Affects Versions: 3.5.0
>Reporter: Øyvind Sæbø
>Assignee: Stephen Mallette
>Priority: Major
> Fix For: 3.5.1, 3.6.0
>
>
> Currently The gremlint web app at tinkerpop.apache.org/gremlint still 
> installs the gremlint library from github:OyvindSabo/gremlint#master. In the 
> tinkerpop repo, however, gremlint is not located in the root directory, so 
> installing it as a package directly from GitHub with npm is as far as I know 
> not possible.
> It would be nice to get gremlint published to npm so that it's easier for 
> users to use it in JavaScript projects, and so that the gremlint web app can 
> use the latest code from tinkerpop/gremlint, rather than installing the no 
> longer maintained code from OyvindSabo/gremlint.
> I assume this will have to be done similarly to how it's done for 
> gremlin-javascript, although I haven't had time to look closely at how that's 
> done yet.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TINKERPOP-2551) Setup scripts to publish Gremint to npm

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TINKERPOP-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17356505#comment-17356505
 ] 

ASF GitHub Bot commented on TINKERPOP-2551:
---

spmallette merged pull request #1424:
URL: https://github.com/apache/tinkerpop/pull/1424


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Setup scripts to publish Gremint to npm
> ---
>
> Key: TINKERPOP-2551
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2551
> Project: TinkerPop
>  Issue Type: Improvement
>  Components: gremlint
>Affects Versions: 3.5.0
>Reporter: Øyvind Sæbø
>Priority: Major
>
> Currently The gremlint web app at tinkerpop.apache.org/gremlint still 
> installs the gremlint library from github:OyvindSabo/gremlint#master. In the 
> tinkerpop repo, however, gremlint is not located in the root directory, so 
> installing it as a package directly from GitHub with npm is as far as I know 
> not possible.
> It would be nice to get gremlint published to npm so that it's easier for 
> users to use it in JavaScript projects, and so that the gremlint web app can 
> use the latest code from tinkerpop/gremlint, rather than installing the no 
> longer maintained code from OyvindSabo/gremlint.
> I assume this will have to be done similarly to how it's done for 
> gremlin-javascript, although I haven't had time to look closely at how that's 
> done yet.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (TINKERPOP-2576) Setup automatic updates via Dependabot for Gremlin.NET

[Stephen Mallette (Jira)]

 [ 
https://issues.apache.org/jira/browse/TINKERPOP-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Mallette updated TINKERPOP-2576:

Component/s: build-release

> Setup automatic updates via Dependabot for Gremlin.NET
> --
>
> Key: TINKERPOP-2576
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2576
> Project: TinkerPop
>  Issue Type: Improvement
>  Components: build-release, dotnet
>Affects Versions: 3.5.0
>Reporter: Florian Hockmann
>Priority: Major
>
> We had [some discussions in the 
> past|https://lists.apache.org/thread.html/1cc0a5d79c3c66b6294c5f27c40c440df761e64e723732d058f114b6%40%3Cdev.tinkerpop.apache.org%3E]
>  already about Dependabot. One of the problems that we had with it was that 
> we mainly wanted to enable it for the GLVs like Gremlin.NET but we used to 
> generate the {{.csproj}} file for that from a template so Dependabot couldn't 
> create meaningful PRs as it would not update the template.
> Since we have already removed this generation on {{3.5-dev}} and {{master}}, 
> we can now set up Dependabot for these two branches.
> I want to start using Dependabot for Gremlin.NET but we can of course later 
> on also use it for other parts if we want to.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (TINKERPOP-2576) Setup automatic updates via Dependabot for Gremlin.NET

[Stephen Mallette (Jira)]

 [ 
https://issues.apache.org/jira/browse/TINKERPOP-2576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Mallette updated TINKERPOP-2576:

Affects Version/s: 3.5.0

> Setup automatic updates via Dependabot for Gremlin.NET
> --
>
> Key: TINKERPOP-2576
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2576
> Project: TinkerPop
>  Issue Type: Improvement
>  Components: dotnet
>Affects Versions: 3.5.0
>Reporter: Florian Hockmann
>Priority: Major
>
> We had [some discussions in the 
> past|https://lists.apache.org/thread.html/1cc0a5d79c3c66b6294c5f27c40c440df761e64e723732d058f114b6%40%3Cdev.tinkerpop.apache.org%3E]
>  already about Dependabot. One of the problems that we had with it was that 
> we mainly wanted to enable it for the GLVs like Gremlin.NET but we used to 
> generate the {{.csproj}} file for that from a template so Dependabot couldn't 
> create meaningful PRs as it would not update the template.
> Since we have already removed this generation on {{3.5-dev}} and {{master}}, 
> we can now set up Dependabot for these two branches.
> I want to start using Dependabot for Gremlin.NET but we can of course later 
> on also use it for other parts if we want to.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (TINKERPOP-2576) Setup automatic updates via Dependabot for Gremlin.NET

[Florian Hockmann (Jira)]

Florian Hockmann created TINKERPOP-2576:
---

 Summary: Setup automatic updates via Dependabot for Gremlin.NET
 Key: TINKERPOP-2576
 URL: https://issues.apache.org/jira/browse/TINKERPOP-2576
 Project: TinkerPop
  Issue Type: Improvement
  Components: dotnet
Reporter: Florian Hockmann


We had [some discussions in the 
past|https://lists.apache.org/thread.html/1cc0a5d79c3c66b6294c5f27c40c440df761e64e723732d058f114b6%40%3Cdev.tinkerpop.apache.org%3E]
 already about Dependabot. One of the problems that we had with it was that we 
mainly wanted to enable it for the GLVs like Gremlin.NET but we used to 
generate the {{.csproj}} file for that from a template so Dependabot couldn't 
create meaningful PRs as it would not update the template.

Since we have already removed this generation on {{3.5-dev}} and {{master}}, we 
can now set up Dependabot for these two branches.

I want to start using Dependabot for Gremlin.NET but we can of course later on 
also use it for other parts if we want to.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

code generation and RDF support in TinkerPop 4

[Joshua Shinavier]

Hello all,

I would like to take some concrete steps toward the TinkerPop 4
interoperability goals I've stated a few times (e.g. see TinkerPop 2020
from last year). At a
meetup  a couple
of months ago, I demonstrated an approach for generating TinkerPop APIs
consistently into different languages. I have started to check in some of
that generated code in a branch (see my commits here
)
and add bits and pieces for RDF support, as well.

The Apache Software Foundation asks us to discuss any significant changes
to the code base on the dev list. Since these steps toward TP4 will be
major changes if and when they are merged into the master branch, I will
start discussing them here. Expect occasional emails from me about the
various things I will be doing in the branch. I absolutely invite comments,
feedback, and actual discussion on these design proposals, but even if it's
just me issuing self-affirming statements into the void like the King of
Pointland, I will just carry on, because that's how this process works.

A brief summary of the changes so far:


   - *Abstract specification of Gremlin traversals*. I have turned
   Stephen's Gremlin.g4
   

   ANTLR grammar into an abstract specification of Gremlin traversal syntax
   using the Dragon (YAML-based) format. Unfortunately, it is looking very
   unlikely that Dragon will become available as open-source software, so you
   can expect this YAML format to change just slightly once we have a new
   Dragon-like tool for schema and data transformations. More on that later.
   Right now, the syntax specification can be found here
   
,
   although the file path might change in the future.


   - *Traversal DTOs*. Based on the abstract specification, I have
   generated Java classes for building and working with traversals. The
   generated files can currently be found here
   
.
   These are essentially POJOs or DTO classes, with special boilerplate
   methods for equality, pattern matching over alternative constructors, and
   modification by copying (since the instances are immutable). These classes
   allow you to build traversals in a declarative way, while all of the logic
   for evaluating traversals goes elsewhere. Support for serialization and
   deserialization for traversals is to be added in the future -- and the same
   goes for all other classes generated in this way.


   - *RDF 1.1 concepts model*. RDF support was part of TinkerPop from the
   beginning, but it was de-emphasized for TinkerPop 3 due to other priorities
   such as OLAP. For years, developers have been asking us for better
   interoperability with RDF. While we do have some query-level support for
   RDF these days in sparql-gremlin, we no longer have any data-level support,
   e.g. supporting loading RDF data into a property graph and getting it back
   out, evaluating Gremlin traversals over RDF datasets, etc. These things are
   not especially hard to do, in certain limited ways, but our old approach of
   writing adapters like GraphSail
   ,
   SailGraph
   , and
   PropertyGraphSail
   

   in Java, with no support for other languages, does not seem appropriate for
   TinkerPop 4. Also, those early mappings were extremely underspecified in a
   formal sense -- good enough for some practical applications, but not good
   enough for anything requiring inference, optimization, or composition with
   other mappings. To that end, I am starting to add abstract specifications
   for RDF along the lines of the Gremlin specifications I described above.
   The first of these, a specification of RDF 1.1 Concepts, can currently be
   found here
   
,
   with generated Java classes here
   
.
   This gives us a way of working with RDF data in a language-neutral and
   framework-neutral way (whereas we were previously tied to Java and to the
   RDF4j, nee Sesame, API). Mappings into and out of RDF will be defined with
   respect to these 

[jira] [Commented] (TINKERPOP-2551) Setup scripts to publish Gremint to npm

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TINKERPOP-2551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17356471#comment-17356471
 ] 

ASF GitHub Bot commented on TINKERPOP-2551:
---

FlorianHockmann commented on a change in pull request #1424:
URL: https://github.com/apache/tinkerpop/pull/1424#discussion_r644839842



##
File path: gremlint/pom.xml
##
@@ -0,0 +1,214 @@
+
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+org.apache.tinkerpop
+tinkerpop
+3.5.1-SNAPSHOT
+
+gremlint
+Apache TinkerPop :: Gremlint
+
+false
+${maven.test.skip}
+6.14.6
+v10.22.0
+
+
+${basedir}/target
+${project.artifactId}-${project.version}
+
+
+
+com.github.eirslett
+frontend-maven-plugin
+1.10.0
+
+
+
+
+
+
+http://maven.apache.org/POM/4.0.0;
+ xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance;
+ xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 
http://maven.apache.org/xsd/maven-4.0.0.xsd;>
+4.0.0
+
+org.apache.tinkerpop
+tinkerpop
+3.5.1-SNAPSHOT
+
+gremlint
+Apache TinkerPop :: Gremlint
+
+false
+${maven.test.skip}
+6.14.6
+v10.22.0
+
+
+${basedir}/target
+${project.artifactId}-${project.version}
+
+
+
+com.github.eirslett
+frontend-maven-plugin
+1.10.0
+
+
+
+
+
+
+org.codehaus.gmavenplus
+gmavenplus-plugin
+
+
+org.codehaus.groovy
+groovy-all
+${groovy.version}
+pom
+runtime
+
+
+
+
+update-version
+generate-sources
+
+execute
+
+
+
+
+def mavenVersion = "${project.version}"
+def versionForJs = 
mavenVersion.replace("-SNAPSHOT", "-alpha1")
+def file = new 
File("${project.basedir}/package.json")
+
file.write(file.getText("UTF-8").replaceFirst(/"version": "(.*)",/, 
"\"version\": \"" + versionForJs + "\","))
+
+
+
+
+
+
+
+org.apache.maven.plugins
+maven-clean-plugin
+
+
+
+clean
+
+
+
+
+
+com.github.eirslett
+frontend-maven-plugin
+
+
+install node and npm
+generate-resources
+
+install-node-and-npm
+
+
+
+npm install
+generate-resources
+
+npm
+
+
+install
+
+
+
+npm test
+test
+
+npm
+
+
+${skipTests}
+test --exit
+true
+
+
+
+
+${node.version}
+${npm.version}
+
+
+
+
+org.apache.maven.plugins
+maven-javadoc-plugin
+
+true
+
+
+
+org.apache.maven.plugins
+maven-source-plugin
+
+true
+
+
+
+  

[jira] [Commented] (TINKERPOP-2358) Potential connection leak on client disposing

[ASF GitHub Bot (Jira)]

[ 
https://issues.apache.org/jira/browse/TINKERPOP-2358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17356445#comment-17356445
 ] 

ASF GitHub Bot commented on TINKERPOP-2358:
---

FlorianHockmann opened a new pull request #1425:
URL: https://github.com/apache/tinkerpop/pull/1425


   https://issues.apache.org/jira/browse/TINKERPOP-2358
   
   If the `ConnectionPool` was disposed while it was in parallel creating new 
connections (e.g., to replace closed connections), those
   connections could be leaked. `Dispose()` could not dispose them yet as they 
were not completely established so they could be added to the pool.
   
   We now check after creating new connections whether the pool has been 
disposed in the meantime and then dispose these connections directly again.
   
   In addition to that, the pool now also has a `CancellationTokenSource` which 
allows us to cancel all active creations of new connections in `Dispose()`. So 
we don't have to wait until they are created only so we can then dispose them 
if we can already cancel the connection establishment.
   
   VOTE +1


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> Potential connection leak on client disposing
> -
>
> Key: TINKERPOP-2358
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2358
> Project: TinkerPop
>  Issue Type: Bug
>  Components: dotnet
>Affects Versions: 3.4.6
>Reporter: Dzmitry L
>Assignee: Florian Hockmann
>Priority: Minor
>
> Potential issue may happen if you decide to call "Dispose" on a client while 
> "PopulatePoolAsync" is active. "Dispose" closes all connections from the 
> pool, but does not stop "PopulatePoolAsync". So after "PopulatePoolAsync" 
> finish you will have unclosed connections.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TINKERPOP-2573) Gremlin.NET: Retry on CreateNewConnectionAsync() failures and avoid disposing successful connections on a failure

[Florian Hockmann (Jira)]

[ 
https://issues.apache.org/jira/browse/TINKERPOP-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17356410#comment-17356410
 ] 

Florian Hockmann commented on TINKERPOP-2573:
-

Thanks for reporting this and giving your thoughts on how we could improve 
this, [~olivertowers]!
{quote}Gremlin server configurations can include a load-balancer in-front of 
the server nodes (Azure Cosmos DB is one example).
{quote}
That is definitely something that we haven't accounted for enough. The driver 
was originally built with the idea that a {{ConnectionPool}} holds connections 
to a single server. TINKERPOP-1765 should add support for multiple servers and 
then each server could have its own {{ConnectionPool}}. That's also how the 
Java driver operates if I'm not mistaken. But that of course doesn't help if 
multiple servers are behind a load balancer which is a setup that is getting 
more and more common in cloud environments or with containers in general.

 
{quote}1. Modify {{FillPoolAsync}} to handle connection failures individually, 
doing 'best-effort' replacement of missing connections.
 2 If {{FillPoolAsync}} observes an exception, still throw but do not dispose 
of connection tasks that were successful (and check that state of those 
successful connections and dispose if they are in error?).
{quote}
Aren't these two points basically the same? We could solve this by iterating 
over all connection tasks in the {{catch}} block, adding all connections from 
successfully finished tasks (maybe after also checking their connection state) 
to the pool and then throwing the exception.

This would of course leave the pool in a state where it doesn't have its 
configured size, but the missing connections can still be added later, e.g., 
through the retry policy as you also already mentioned.

 
{quote}3. In ConnectionPool constructor, apply reconnect retry policy to 
ReplaceDeadConnectionAsync() invocation
{quote}
I'm not sure about this one yet. If the server is already unavailable when the 
user creates a {{GremlinClient}}, then I think that it's best to fail fast to 
inform the user about the problem. I think it's more likely in this case that 
the user supplied invalid connection parameters or that the server is 
completely unreachable (e.g., due to a missing firewall exception) than a 
transient error where a retry policy could help.

I also would like to avoid that calling a constructor can take a long time. 
Creating the connections in the constructor of course also already takes some 
time and is beyond of what a constructor should usually do. Maybe we could move 
this logic in some {{ConnectAsync}} method in the future. That method could be 
a static factory method and we could then make the constructor private, but 
that would of course be a breaking change so it would have to wait for 3.6.0.

These are the two reasons I see against a retry policy in the constructor, but 
I'm also not completely against this. Do you frequently see the situation where 
calling the constructor throws due to a transient error or maybe an error with 
just one of many servers behind a load balancer?

 
{quote}To allow the underlying errors to be observed by callers, a 
callback/event handler could be added that is invoked in the 
{{ReplaceClosedConnectionsAsync}} throws. This would allow for the any 
exceptions to be observed and collected in some manner.
{quote}
That's true. My take so far was that we should add logging to the driver 
(TINKERPOP-2471) to make things like these transparent to the user. Do you 
think that callbacks / an event handler would be a better approach than 
logging? I think logging would be a lot easier to use as users would only have 
to provide a logger.
  
{quote}Further refinement of this: Have a way of populating 
{{ServerUnavailableException}} inner exception with the last observed 
replacement exception.
{quote}
Interesting idea. We currently don't do that as the connection replacement is 
handled asynchronously in the background by a different task, but we could of 
course save the last thrown exception and use that as the inner exception.

> Gremlin.NET: Retry on CreateNewConnectionAsync() failures and avoid disposing 
> successful connections on a failure
> -
>
> Key: TINKERPOP-2573
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2573
> Project: TinkerPop
>  Issue Type: Improvement
>  Components: dotnet
>Affects Versions: 3.4.11
>Reporter: Oliver Towers
>Priority: Minor
>
> [FillPoolAsync|https://github.com/apache/tinkerpop/blob/dc4a5d8795e836a35ce4b61247b9693fa0cea8cb/gremlin-dotnet/src/Gremlin.Net/Driver/ConnectionPool.cs#L119]
>  attempts to create connections to fill the pool up to the target 
> {{PoolSize}},