Re: mm-ADT to TinkerPop3

[Dmitry Novikov]

Hello,

>From users perspective, I am very excited about 1 and 2:

1. Type system makes sense, currently, there is JIRA about introducing types in 
TP 3.5 - TINKERPOP-2234
2. Making instruction set more consistent:
  * Currently `select` step is used for getting values from maps and labeled 
steps within a path, which from my perspective is confusing. Making `values` 
step recommended way to access map, would simplify `select`.
  * Representing Vertex and Edge as maps would simplify serialization. Also, it 
would make writing queries simpler as there would be fewer steps to remember.

Question about 4. Can not fully understand how references are connected with 
schemas. Could you please explain it in more detail, or point to explanation if 
already exists?

Regards,
Dmitry

On 2019/06/14 16:21:55, Marko Rodriguez  wrote: 
> Hey,
> 
> > One thing I wonder at the moment which I don't think has come up in
> > relation to mm-ADT discussion yet is DSLs. By every account, people are
> > either using DSLs now or as soon as they learn about them, they immediately
> > see the value and start to organize their code around them. So, any
> > thoughts yet on how DSLs work under mm-ADT (in relation to TP3 and/or
> > future) or is the model largely the same as what we do now?
> 
> mm-ADT is a bytecode specification. While we have a human readable/writable 
> text representation (currently being called mm-ADT-bc), mm-ADT is primarily 
> for machine consumption. Thus, when it comes to higher-level languages like 
> Gremlin or a custom DSL, they would compile to mm-ADT bytecode.  Thus, if 
> Gremlin compiles to mm-ADT, then all the Gremlin DSL infrastructure would 
> just work as is. However, things can get a more interesting.
> 
> You can create derived types of arbitrary complexity in mm-ADT.
> 
> [define,person,[name:@string,age:@int,knows:@person*]]
> 
> From a DSL perspective, users can make their own objects. Look at the friends 
> field. It is not container, but just zero or more person objects 
> (sequence/stream). When this model is embedded in a graph database (and there 
> are different ways to specify the embedding), those people could be 
> referenced via a “knows"-edge.
> 
> As you can see, there is nothing “graph” here. No vertices, no edges… just a 
> domain model.  But with mm-ADT-bc, you can create processes over that domain 
> model and thus, traverse the “graph”:
> 
> [db][values,people]  // people is defined, I just don’t show it in this 
> email
> [has,name,eq,marko]
> [values,knows]
> [value,age]
> [sum]
> 
> There is nothing pretty about mm-ADT-bc to a human user, but that is where 
> DSLs would come in. Languages that make it easy to write mm-ADT-bc.
> 
> If Gremlin were the higher-level language, the following traversal would 
> create the above bytecode:
>   g.V().has(‘person',‘name’,’marko’).out(‘knows’).values(‘age’).sum()
> 
> How do you see this being used from your perspective?
> 
> Marko.
> 
> http://rredux.com
> 
> 
> 
> > 
> > 
> > On Thu, Jun 13, 2019 at 6:25 PM Marko Rodriguez  > >
> > wrote:
> > 
> >> Hello,
> >> 
> >> Various stakeholders in Apache TinkerPop have been wondering weather
> >> mm-ADT can be leveraged in TinkerPop3. While I originally planned for
> >> mm-ADT to form the foundation of TinkerPop4, there are a subset of features
> >> in mm-ADT that could really help TP3 moving forward. Here is a preliminary
> >> outline of the mm-ADT features that could push the TP3 roadmap.
> >> 
> >>1. Type system: mm-ADT has a nominal type system for the built-in
> >> types and a structural type system for all derived types. Bytecode
> >> instructions that CRUD on database data can by statically typed and
> >> reasoned on at compile time.
> >> 
> >>2. Strategies: mm-ADT has a completely different approach to query
> >> optimization than TP3. While there are compile-time strategies for
> >> manipulating a query into a semantically equivalent, though computationally
> >> more efficient form, the concept of “provider strategies” (indices) goes
> >> out the window in favor of reference graphs. The primary benefit of the
> >> mm-ADT model is that the implementation for providers will be much simpler,
> >> less error prone, doesn’t require custom instructions, and is able to
> >> naturally capitalize on other internal provider optimizations such as
> >> schemas, denormalizations, views, etc.
> >> 
> >>3. Instruction Set: mm-ADT’s instruction set is less adhoc than
> >> TP3. Relational operators are polymorphic. Math operators are polymorphic.
> >> Container (collection) operators are polymorphic. Unlike TP3, a “vertex” is
> >> just a map like any other map. Thus, has(), value(), where(), select(),
> >> etc. operate across all such derivations. Moreover, mm-ADT’s instruction
> >> set greatly reduces the number of ways in which an expression can be
> >> represented, relying primarily on reference graphs (see #2 above)

[jira] [Created] (TINKERPOP-2243) Add user-agent to RequestOptions

[Bryn Cooke (JIRA)]

Bryn Cooke created TINKERPOP-2243:
-

 Summary: Add user-agent to RequestOptions
 Key: TINKERPOP-2243
 URL: https://issues.apache.org/jira/browse/TINKERPOP-2243
 Project: TinkerPop
  Issue Type: Improvement
  Components: driver, server
Affects Versions: 3.4.2
Reporter: Bryn Cooke


It would be useful to know the 'user-agent' that is sending a request so that 
the server can react in different ways. For instance:
 * To log differently.
 * To give different error messages.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (TINKERPOP-2243) Add user-agent to RequestOptions

[Bryn Cooke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TINKERPOP-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bryn Cooke updated TINKERPOP-2243:
--
Description: 
It would be useful to know the 'user-agent' that is sending a request so that 
the server can react in different ways. For instance:
 * To log differently.
 * To give different error messages.

For instance, if the user is connecting via gremlin console then error messages 
can include gremlin console specific help.

  was:
It would be useful to know the 'user-agent' that is sending a request so that 
the server can react in different ways. For instance:
 * To log differently.
 * To give different error messages.

 


> Add user-agent to RequestOptions
> 
>
> Key: TINKERPOP-2243
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2243
> Project: TinkerPop
>  Issue Type: Improvement
>  Components: driver, server
>Affects Versions: 3.4.2
>Reporter: Bryn Cooke
>Priority: Major
>
> It would be useful to know the 'user-agent' that is sending a request so that 
> the server can react in different ways. For instance:
>  * To log differently.
>  * To give different error messages.
> For instance, if the user is connecting via gremlin console then error 
> messages can include gremlin console specific help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: mm-ADT to TinkerPop3

[Dmitry Novikov]

Sorry, at first I've read this with the wrong context. Nevermind my question. 
It makes sense now.

On 2019/06/17 14:08:40, Marko Rodriguez  wrote: 
> Hello,
> 
> > Question about 4. Can not fully understand how references are connected 
> > with schemas. Could you please explain it in more detail, or point to 
> > explanation if already exists?
> 
> Wow. That is funny you bring this up. I worked all weekend to unify 
> “references” and “types” in mm-ADT. The short to your question is that a 
> schema is just the unification of all the defined types. For a better 
> understanding on how references are just types, I've attached a screenshot of 
> the introduction to the mm-ADT type system.
> 
> Any questions, please ask.
> 
> *** SIDENOTE: I don’t know of any programming language that has realized 
> pointers as being fundamentally types. In mm-ADT where “anonymous types” 
> (lambda types) are prevalent, it was a short step to realize that, in fact, 
> any “grouping” is ultimately a new type.
> 
> Take care,
> Marko.
> 
> http://rredux.com
> 
> 
> 
> 
> > 
> > Regards,
> > Dmitry
> > 
> > On 2019/06/14 16:21:55, Marko Rodriguez  > > wrote: 
> >> Hey,
> >> 
> >>> One thing I wonder at the moment which I don't think has come up in
> >>> relation to mm-ADT discussion yet is DSLs. By every account, people are
> >>> either using DSLs now or as soon as they learn about them, they 
> >>> immediately
> >>> see the value and start to organize their code around them. So, any
> >>> thoughts yet on how DSLs work under mm-ADT (in relation to TP3 and/or
> >>> future) or is the model largely the same as what we do now?
> >> 
> >> mm-ADT is a bytecode specification. While we have a human 
> >> readable/writable text representation (currently being called mm-ADT-bc), 
> >> mm-ADT is primarily for machine consumption. Thus, when it comes to 
> >> higher-level languages like Gremlin or a custom DSL, they would compile to 
> >> mm-ADT bytecode.  Thus, if Gremlin compiles to mm-ADT, then all the 
> >> Gremlin DSL infrastructure would just work as is. However, things can get 
> >> a more interesting.
> >> 
> >> You can create derived types of arbitrary complexity in mm-ADT.
> >> 
> >> [define,person,[name:@string,age:@int,knows:@person*]]
> >> 
> >> From a DSL perspective, users can make their own objects. Look at the 
> >> friends field. It is not container, but just zero or more person objects 
> >> (sequence/stream). When this model is embedded in a graph database (and 
> >> there are different ways to specify the embedding), those people could be 
> >> referenced via a “knows"-edge.
> >> 
> >> As you can see, there is nothing “graph” here. No vertices, no edges… just 
> >> a domain model.  But with mm-ADT-bc, you can create processes over that 
> >> domain model and thus, traverse the “graph”:
> >> 
> >> [db][values,people]  // people is defined, I just don’t show it in 
> >> this email
> >>[has,name,eq,marko]
> >>[values,knows]
> >>[value,age]
> >>[sum]
> >> 
> >> There is nothing pretty about mm-ADT-bc to a human user, but that is where 
> >> DSLs would come in. Languages that make it easy to write mm-ADT-bc.
> >> 
> >> If Gremlin were the higher-level language, the following traversal would 
> >> create the above bytecode:
> >>g.V().has(‘person',‘name’,’marko’).out(‘knows’).values(‘age’).sum()
> >> 
> >> How do you see this being used from your perspective?
> >> 
> >> Marko.
> >> 
> >> http://rredux.com
> >> 
> >> 
> >> 
> >>> 
> >>> 
> >>> On Thu, Jun 13, 2019 at 6:25 PM Marko Rodriguez  >>>   >>> >>
> >>> wrote:
> >>> 
>  Hello,
>  
>  Various stakeholders in Apache TinkerPop have been wondering weather
>  mm-ADT can be leveraged in TinkerPop3. While I originally planned for
>  mm-ADT to form the foundation of TinkerPop4, there are a subset of 
>  features
>  in mm-ADT that could really help TP3 moving forward. Here is a 
>  preliminary
>  outline of the mm-ADT features that could push the TP3 roadmap.
>  
>    1. Type system: mm-ADT has a nominal type system for the built-in
>  types and a structural type system for all derived types. Bytecode
>  instructions that CRUD on database data can by statically typed and
>  reasoned on at compile time.
>  
>    2. Strategies: mm-ADT has a completely different approach to query
>  optimization than TP3. While there are compile-time strategies for
>  manipulating a query into a semantically equivalent, though 
>  computationally
>  more efficient form, the concept of “provider strategies” (indices) goes
>  out the window in favor of reference graphs. The primary benefit of the
>  mm-ADT model is that the implementation for providers will be much 
>  simpler,
>  less error prone, doesn’t require custom instructions, and is able to
>  

[jira] [Updated] (TINKERPOP-2213) Replace scriptEvaluationTimeout in favor of something more suitable to bytecode

[stephen mallette (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TINKERPOP-2213?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

stephen mallette updated TINKERPOP-2213:

 Labels: breaking  (was: deprecation)
Description: {{scriptEvaluationTimeout}} doesn't fit well with bytecode 
based requests because they aren't actually "scripts" - consider just 
{{evaluationTimeout}}?  (was: {{scriptEvaluationTimeout}} doesn't fit well with 
bytecode based requests because they aren't actually "scripts" - consider 
deprecating for something else, perhaps just {{evaluationTimeout}}?)
Summary: Replace scriptEvaluationTimeout in favor of something more 
suitable to bytecode  (was: Deprecate scriptEvaluationTimeout in favor of 
something more suitable to bytecode)

Deprecation doesn't look easy to do because the setting is configured by way of 
a public field in {{Settings}}i can't think of a way to have a change in 
one trigger replication to the other. I'm thinking we just look to remove the 
old configuration in 3.5.0 and just have this be a breaking change.

> Replace scriptEvaluationTimeout in favor of something more suitable to 
> bytecode
> ---
>
> Key: TINKERPOP-2213
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2213
> Project: TinkerPop
>  Issue Type: Improvement
>  Components: server
>Affects Versions: 3.3.6
>Reporter: stephen mallette
>Assignee: stephen mallette
>Priority: Major
>  Labels: breaking
>
> {{scriptEvaluationTimeout}} doesn't fit well with bytecode based requests 
> because they aren't actually "scripts" - consider just {{evaluationTimeout}}?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Work started] (TINKERPOP-2238) Fix remaining iterator leaks marked by @IgnoreIteratorLeak

[stephen mallette (JIRA)]

 [ 
https://issues.apache.org/jira/browse/TINKERPOP-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on TINKERPOP-2238 started by stephen mallette.
---
> Fix remaining iterator leaks marked by @IgnoreIteratorLeak
> --
>
> Key: TINKERPOP-2238
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2238
> Project: TinkerPop
>  Issue Type: Bug
>  Components: process
>Affects Versions: 3.3.7, 3.4.2
>Reporter: Divij Vaidya
>Assignee: stephen mallette
>Priority: Major
>
> As noted in the PR [https://github.com/apache/tinkerpop/pull/1118/] some of 
> the tests do not pass the leak detector and need to be fixed. These tests 
> have been annotated with @IgnoreIteratorLeak.
> After the tests (and associated code if required) is fixed, please delete the 
> annotation as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TINKERPOP-2238) Fix remaining iterator leaks marked by @IgnoreIteratorLeak

[stephen mallette (JIRA)]

[ 
https://issues.apache.org/jira/browse/TINKERPOP-2238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16865965#comment-16865965
 ] 

stephen mallette commented on TINKERPOP-2238:
-

Started a branch for this work named the same as this issue number.

> Fix remaining iterator leaks marked by @IgnoreIteratorLeak
> --
>
> Key: TINKERPOP-2238
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2238
> Project: TinkerPop
>  Issue Type: Bug
>  Components: process
>Affects Versions: 3.3.7, 3.4.2
>Reporter: Divij Vaidya
>Assignee: stephen mallette
>Priority: Major
>
> As noted in the PR [https://github.com/apache/tinkerpop/pull/1118/] some of 
> the tests do not pass the leak detector and need to be fixed. These tests 
> have been annotated with @IgnoreIteratorLeak.
> After the tests (and associated code if required) is fixed, please delete the 
> annotation as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (TINKERPOP-2243) Add user-agent to RequestOptions

[Divij Vaidya (JIRA)]

[ 
https://issues.apache.org/jira/browse/TINKERPOP-2243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16866209#comment-16866209
 ] 

Divij Vaidya commented on TINKERPOP-2243:
-

I would take it a step further and say that driver should have an option where 
the provider can add specific custom fields to the request header. It might be 
a user-agent or an auth token or encoding of the request data. It could be done 
by allowing the providers to provide a JAR which would contain a netty handler 
adhering to a specific interface and would be auto-magically picked up by the 
client on startup.

> Add user-agent to RequestOptions
> 
>
> Key: TINKERPOP-2243
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2243
> Project: TinkerPop
>  Issue Type: Improvement
>  Components: driver, server
>Affects Versions: 3.4.2
>Reporter: Bryn Cooke
>Priority: Major
>
> It would be useful to know the 'user-agent' that is sending a request so that 
> the server can react in different ways. For instance:
>  * To log differently.
>  * To give different error messages.
> For instance, if the user is connecting via gremlin console then error 
> messages can include gremlin console specific help.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (TINKERPOP-2244) Provide examples for driver usage patterns

[Divij Vaidya (JIRA)]

Divij Vaidya created TINKERPOP-2244:
---

 Summary: Provide examples for driver usage patterns
 Key: TINKERPOP-2244
 URL: https://issues.apache.org/jira/browse/TINKERPOP-2244
 Project: TinkerPop
  Issue Type: Improvement
  Components: documentation, driver
Affects Versions: 3.3.7
Reporter: Divij Vaidya


Some examples for the following use cases would be very helpful for the end 
user of the client.
 # Example on how to submit a query asynchronously (using the less known but 
useful g.V().promise() approach). This helps in answering the question, "How do 
I send the query asynchronously using bytecode?"
 # Example of how to submit a query specifying per query options such as 
timeout or custom UUID. This helps in answering the question, "Can I set a 
per-query level timeout?"
 # Example highlighting various use cases when you would need to change the 
values of maxSimultaneousUsage and maxInProcess parameters. This helps in 
answering the question, "How do I know what values shall I set for these 
parameters? How does it balance with maxConnectionPoolSize? How do I debug 
Timeout while obtaining a connection from the client code?"
 # Example clarifying the materialization of properties in DetachedVertex 
(string based submission) vs ReferenceVertex (bytecode submission) scenarios. 
This helps in answering the question, "Is there any disadvantage in sending the 
queries as string? Which method is more efficient?"
 # Example clarifying that the query actually gets sent to server when we call 
next(). This helps in answering the question, "When does the query get sent to 
the server? What role does batchSize play? How can I optimize my query to just 
retrieve one result out of many from the server?"
 # Example clarifying that the Client object or the GraphTraversalSource object 
is thread safe and should be reused across multiple requests. This helps in 
answering the question, "Shall I create a separate client object per query or 
re-use one across multiple queries?"
 # Example clarifying that the same GraphTraversal object cannot be re-used for 
sending multiple instances of the same query.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (TINKERPOP-2245) [Refactor] Consolidate the executor for bytecode & string based client

[Divij Vaidya (JIRA)]

Divij Vaidya created TINKERPOP-2245:
---

 Summary: [Refactor] Consolidate the executor for bytecode & string 
based client
 Key: TINKERPOP-2245
 URL: https://issues.apache.org/jira/browse/TINKERPOP-2245
 Project: TinkerPop
  Issue Type: Improvement
  Components: server
Reporter: Divij Vaidya


We have two code paths in the server which perform (more or less) the same 
functions. One is the executor for string based queries and other is the 
executor for bytecode. This code can be refactored together so that the logic 
to handle timeout, handle exception during execution, handle exception before 
execution and others can consolidated. 

[https://github.com/apache/tinkerpop/blob/master/gremlin-groovy/src/main/java/org/apache/tinkerpop/gremlin/groovy/engine/GremlinExecutor.java#L246]

and

[https://github.com/apache/tinkerpop/blob/master/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/op/traversal/TraversalOpProcessor.java#L333]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (TINKERPOP-2246) Consolidate the error propagation to the client

[Divij Vaidya (JIRA)]

Divij Vaidya created TINKERPOP-2246:
---

 Summary: Consolidate the error propagation to the client
 Key: TINKERPOP-2246
 URL: https://issues.apache.org/jira/browse/TINKERPOP-2246
 Project: TinkerPop
  Issue Type: Improvement
  Components: server
Reporter: Divij Vaidya


Some places in the code base use the ResponseHandlerContext which ensures that 
only a single response is sent for a particular request [1] but other places do 
not [2]. This task is to use the ResponseContext consistently across the code 
base to prevent cases where client gets two messages for the same request (this 
can break the client when it has removed the RequestQueue thinking that the 
query is complete and yet a message comes for that request queue).

[1] 
[https://github.com/apache/tinkerpop/blob/d1a3fa147d1f009ae57274827c9b59426dfc6e58/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/op/AbstractOpProcessor.java#L100]

[[2] 
https://github.com/apache/tinkerpop/blob/master/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/op/traversal/TraversalOpProcessor.java#L315|https://github.com/apache/tinkerpop/blob/master/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/op/traversal/TraversalOpProcessor.java#L315]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (TINKERPOP-2247) Server should respect charset specified in request header

[Divij Vaidya (JIRA)]

Divij Vaidya created TINKERPOP-2247:
---

 Summary: Server should respect charset specified in request header
 Key: TINKERPOP-2247
 URL: https://issues.apache.org/jira/browse/TINKERPOP-2247
 Project: TinkerPop
  Issue Type: Wish
Reporter: Divij Vaidya


Currently, the server uses UTF-8 as its default charset and that is hardcoded 
in the system [1], irrespective of the content-type provided in the request. 
This task is to read the charset from the HTTP Content-Type header and use the 
charset to encode the response string.

 
[1][https://github.com/apache/tinkerpop/blob/master/gremlin-server/src/main/java/org/apache/tinkerpop/gremlin/server/handler/HttpGremlinEndpointHandler.java#L254]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)