[jira] [Created] (TINKERPOP-2424) Reduce chance for OOME with large results to Java driver

[Stephen Mallette (Jira)]

Stephen Mallette created TINKERPOP-2424:
---

 Summary: Reduce chance for OOME with large results to Java driver
 Key: TINKERPOP-2424
 URL: https://issues.apache.org/jira/browse/TINKERPOP-2424
 Project: TinkerPop
  Issue Type: Improvement
  Components: driver
Affects Versions: 3.4.8
Reporter: Stephen Mallette


Originally mentioned here:

https://groups.google.com/g/gremlin-users/c/I4HQC9JkzSo/m/fYfd5o0UAQAJ

and pretty easy to create with an empty TinkerGraph in Gremlin Server with 
{{evaluationTimeout}} set to something large using this script in the Gremlin 
Console with {{-Xmx512}}:

{code}
cluster = Cluster.open()
client = cluster.connect()
client.submit("g.addV().as('a').addE('self').iterate()")
rs = 
client.submit("g.V().emit().repeat(out()).valueMap(true).limit(1000)");[]
iterator = rs.iterator();[]
x = 0
while(iterator.hasNext()) {
  x++
  if (x % 1 == 0) {
System.out.println(x + "-[" + rs.getAvailableItemCount() + "]-"+ 
iterator.next());
  }
}
{code}

The {{LinkedBlockingQueue}} of the {{ResultQueue}} is unbounded and can fill 
faster than can be consumed and on a system with limited memory an OOME can 
loom. 

While we tend to discourage iteration of large result sets {{e.g. g.V()}} I 
suppose we should do what we can to keep users out of OOME situations if we 
can. Not sure of the best way to do this but some simple experimentation showed 
that bounding the queue helps (tried with 10) but does require the adding 
of new results to be blocked until more are consumed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (TINKERPOP-2423) hashCode collision for steps with different attributes

[Saikiran Boga (Jira)]

 [ 
https://issues.apache.org/jira/browse/TINKERPOP-2423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Saikiran Boga updated TINKERPOP-2423:
-
Description: 
The {{hashCode computation}} for PropertiesStep collides for multiple steps 
when there are repeating keys. For example {{.properties()}} and 
{{.properties("a","a")}}, {{.properties("a")}} and .{{properties("a","b","b")}} 
have the same hash code because of xor of just properties here 
[https://github.com/apache/tinkerpop/blob/cff4c161615f2b50bda27b6ba523c7f52b833532/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/PropertiesStep.java#L80-L87].
 Basically, the even keys cancel out the xor computation.

 

The same is also true for {{AbstractStep.hashCode}} which takes xor of labels.

{{GraphStep}} does the same thing using ids. so {{g.V("1", "1")}} and {{g.V()}} 
would collide during comparison. 

  was:
The {{hashCode computation}} for PropertiesStep collides for multiple steps 
when there are repeating keys. For example {{.properties()}} and 
{{.properties("a","a")}}, {{.properties("a")}} and .{{properties("a","b","b")}} 
have the same hash code because of xor of just properties here 
[https://github.com/apache/tinkerpop/blob/cff4c161615f2b50bda27b6ba523c7f52b833532/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/PropertiesStep.java#L80-L87].
 Basically, the even keys cancel out the xor computation.

 

The same is also true for {{AbstractStep.hashCode}} which takes xor of labels.

{{}}

{{GraphStep}} does the same thing using ids. so {{g.V("1", "1")}} and {{g.V()}} 
would collide during comparison. 


> hashCode collision for steps with different attributes
> --
>
> Key: TINKERPOP-2423
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2423
> Project: TinkerPop
>  Issue Type: Bug
>  Components: server
>Reporter: Saikiran Boga
>Priority: Major
>
> The {{hashCode computation}} for PropertiesStep collides for multiple steps 
> when there are repeating keys. For example {{.properties()}} and 
> {{.properties("a","a")}}, {{.properties("a")}} and 
> .{{properties("a","b","b")}} have the same hash code because of xor of just 
> properties here 
> [https://github.com/apache/tinkerpop/blob/cff4c161615f2b50bda27b6ba523c7f52b833532/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/PropertiesStep.java#L80-L87].
>  Basically, the even keys cancel out the xor computation.
>  
> The same is also true for {{AbstractStep.hashCode}} which takes xor of labels.
> {{GraphStep}} does the same thing using ids. so {{g.V("1", "1")}} and 
> {{g.V()}} would collide during comparison. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (TINKERPOP-2423) hashCode collision for steps with different attributes

[Saikiran Boga (Jira)]

Saikiran Boga created TINKERPOP-2423:


 Summary: hashCode collision for steps with different attributes
 Key: TINKERPOP-2423
 URL: https://issues.apache.org/jira/browse/TINKERPOP-2423
 Project: TinkerPop
  Issue Type: Bug
  Components: server
Reporter: Saikiran Boga


The {{hashCode computation}} for PropertiesStep collides for multiple steps 
when there are repeating keys. For example {{.properties()}} and 
{{.properties("a","a")}}, {{.properties("a")}} and .{{properties("a","b","b")}} 
have the same hash code because of xor of just properties here 
[https://github.com/apache/tinkerpop/blob/cff4c161615f2b50bda27b6ba523c7f52b833532/gremlin-core/src/main/java/org/apache/tinkerpop/gremlin/process/traversal/step/map/PropertiesStep.java#L80-L87].
 Basically, the even keys cancel out the xor computation.

 

The same is also true for {{AbstractStep.hashCode}} which takes xor of labels.

{{}}

{{GraphStep}} does the same thing using ids. so {{g.V("1", "1")}} and {{g.V()}} 
would collide during comparison. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Accepting gremlint donation

[Øyvind Sæbø]

Yes, I'm following along. Cool to hear that we can move forward with this.

I and Ardoq (the company that the project will be donated on behalf of)
will start filling out the required ICLA
 and CCLA
.

Regarding the points you mentioned:

   1. I can create a GitHub issue for adding the required headers in all
   the source files.
   2. There should be no issues there.
   3. The project has zero dependencies, so there should be no issues there
   either.


tir. 15. sep. 2020 kl. 15:48 skrev Stephen Mallette :

> I've set up the IP Clearance form for incubator here (website hasn't
> generated the HTML yet I guess):
>
>
> https://svn.apache.org/repos/asf/incubator/public/trunk/content/ip-clearance/tinkerpop-gremlint.xml
>
> In the checklist of items there there are few items pertaining to the code
> base itself:
>
> 1. Check and make sure that the files that have been donated have been
> updated to reflect the new ASF copyright
> 2. Check and make sure that for all items included with the distribution
> that is not under the Apache license, we have the right to combine with
> Apache-licensed code and redistribute.
> 3. Check and make sure that all items depended upon by the project is
> covered by one or more of the following approved licenses: Apache, BSD,
> Artistic, MIT/X, MIT/W3C, MPL 1.1, or something with essentially the same
> terms.
>
> For item 1 I assume that means the code base state at which we accept the
> it should have the ASF license header in it with it an appropriate NOTICE
> file if necessary:
>
> https://www.apache.org/legal/src-headers.html
>
> For 2 and 3, I don't think we have any issues there but would need to
> confirm.
>
> Øyvind, I believe you're on the list following along - could you please
> comment on the above for us?
>
>
>
>
> On Tue, Sep 15, 2020 at 7:30 AM Stephen Mallette 
> wrote:
>
> > As there haven't been any objections here, it sounds like we can go ahead
> > with this process. I believe that we will need to go through the IP
> > Clearance process in incubator:
> >
> > https://incubator.apache.org/ip-clearance/
> >
> > and engage Apache Infra about a gremlint.com domain transfer. And then
> of
> > course we will need to figure out "how" we make it part of the code base
> > (where it goes, how it fits in the release process, etc.) - my preference
> > would be to see it come in on 3.4.x so that we can immediately have an
> > official release of it, but we'll see how it goes. I suppose we will
> > continue to use this thread for all this sort of discussion for now
> unless
> > it gets too busy in which case we can spawn off other threads as needed..
> >
> > On Thu, Sep 10, 2020 at 6:31 PM David Bechberger 
> > wrote:
> >
> >> I definitely agree that having this sort of tool freely available would
> be
> >> very helpful to the community as a whole.
> >>
> >> I also would be able to help create the translator between GLV's and
> text
> >> representations as this is something I and many others have struggled
> with
> >> many times.
> >>
> >> Thanks,
> >> Dave
> >>
> >> On Thu, Sep 10, 2020 at 2:14 PM Kelvin Lawrence
>  >> >
> >> wrote:
> >>
> >> > I really like the idea of having an Apache TinkerPop hosted linter and
> >> > style guide "enforcer". I have spent many wasted hours hand formatting
> >> long
> >> > Gremlin queries people have asked me to look at over the years and the
> >> > latest version of Gremlint makes that so much easier. I also really
> like
> >> > the idea of extending the tool in the direction of "Gremlin
> converter".
> >> I
> >> > hear from a lot of users that porting from (let's say) Gremlin Console
> >> to
> >> > Gremlin Python (and vice-versa) is painful and hard to get right.
> >> >
> >> > Cheers
> >> > Kelvin
> >> >
> >> > On 2020/09/10 11:07:46, Stephen Mallette  wrote:
> >> > > I've had some on and off discussions with the developer of>
> >> > >
> >> > > https://gremlint.com/>
> >> > >
> >> > > whose name is Øyvind btw, about the possibility of making gremlint
> >> the>
> >> > > canonical Gremlin code stylizer, and by that I mean, his donating
> the
> >> > code>
> >> > > (and domain name) to the Apache Software Foundation under our
> project.
> >> > That>
> >> > > situation seems possible now and has been offered to the TinkerPop>
> >> > > Community.>
> >> > >
> >> > > I like the idea of bringing this code here because I feel our style
> >> > guide>
> >> > > in Gremlin Recipes can only suggest convention and I'd much like to
> >> see
> >> > a>
> >> > > code library managed by our project enforce a proper Gremlin style
> >> that
> >> > can>
> >> > > be used not only as the neat single HTML page of gremlint.com but
> >> also
> >> > as a>
> >> > > code library to Gremlin tool builders.>
> >> > >
> >> > > I also see the opportunity to grow this tool into an online
> translator
> >> > to>
> >> > > get you from one Gremlin dialect to another which is a consta

Re: [DISCUSS] Accepting gremlint donation

[Stephen Mallette]

I've set up the IP Clearance form for incubator here (website hasn't
generated the HTML yet I guess):

https://svn.apache.org/repos/asf/incubator/public/trunk/content/ip-clearance/tinkerpop-gremlint.xml

In the checklist of items there there are few items pertaining to the code
base itself:

1. Check and make sure that the files that have been donated have been
updated to reflect the new ASF copyright
2. Check and make sure that for all items included with the distribution
that is not under the Apache license, we have the right to combine with
Apache-licensed code and redistribute.
3. Check and make sure that all items depended upon by the project is
covered by one or more of the following approved licenses: Apache, BSD,
Artistic, MIT/X, MIT/W3C, MPL 1.1, or something with essentially the same
terms.

For item 1 I assume that means the code base state at which we accept the
it should have the ASF license header in it with it an appropriate NOTICE
file if necessary:

https://www.apache.org/legal/src-headers.html

For 2 and 3, I don't think we have any issues there but would need to
confirm.

Øyvind, I believe you're on the list following along - could you please
comment on the above for us?




On Tue, Sep 15, 2020 at 7:30 AM Stephen Mallette 
wrote:

> As there haven't been any objections here, it sounds like we can go ahead
> with this process. I believe that we will need to go through the IP
> Clearance process in incubator:
>
> https://incubator.apache.org/ip-clearance/
>
> and engage Apache Infra about a gremlint.com domain transfer. And then of
> course we will need to figure out "how" we make it part of the code base
> (where it goes, how it fits in the release process, etc.) - my preference
> would be to see it come in on 3.4.x so that we can immediately have an
> official release of it, but we'll see how it goes. I suppose we will
> continue to use this thread for all this sort of discussion for now unless
> it gets too busy in which case we can spawn off other threads as needed..
>
> On Thu, Sep 10, 2020 at 6:31 PM David Bechberger 
> wrote:
>
>> I definitely agree that having this sort of tool freely available would be
>> very helpful to the community as a whole.
>>
>> I also would be able to help create the translator between GLV's and text
>> representations as this is something I and many others have struggled with
>> many times.
>>
>> Thanks,
>> Dave
>>
>> On Thu, Sep 10, 2020 at 2:14 PM Kelvin Lawrence > >
>> wrote:
>>
>> > I really like the idea of having an Apache TinkerPop hosted linter and
>> > style guide "enforcer". I have spent many wasted hours hand formatting
>> long
>> > Gremlin queries people have asked me to look at over the years and the
>> > latest version of Gremlint makes that so much easier. I also really like
>> > the idea of extending the tool in the direction of "Gremlin converter".
>> I
>> > hear from a lot of users that porting from (let's say) Gremlin Console
>> to
>> > Gremlin Python (and vice-versa) is painful and hard to get right.
>> >
>> > Cheers
>> > Kelvin
>> >
>> > On 2020/09/10 11:07:46, Stephen Mallette  wrote:
>> > > I've had some on and off discussions with the developer of>
>> > >
>> > > https://gremlint.com/>
>> > >
>> > > whose name is Øyvind btw, about the possibility of making gremlint
>> the>
>> > > canonical Gremlin code stylizer, and by that I mean, his donating the
>> > code>
>> > > (and domain name) to the Apache Software Foundation under our project.
>> > That>
>> > > situation seems possible now and has been offered to the TinkerPop>
>> > > Community.>
>> > >
>> > > I like the idea of bringing this code here because I feel our style
>> > guide>
>> > > in Gremlin Recipes can only suggest convention and I'd much like to
>> see
>> > a>
>> > > code library managed by our project enforce a proper Gremlin style
>> that
>> > can>
>> > > be used not only as the neat single HTML page of gremlint.com but
>> also
>> > as a>
>> > > code library to Gremlin tool builders.>
>> > >
>> > > I also see the opportunity to grow this tool into an online translator
>> > to>
>> > > get you from one Gremlin dialect to another which is a constant
>> problem
>> > for>
>> > > users who are working with the tons of code examples in Gremlin Groovy
>> > but>
>> > > need to get a fast conversion to Python or Javascript. I know I've had
>> > that>
>> > > problem myself on many occasions. An easy to use translator like this
>> > would>
>> > > help users realize that Gremlin in Groovy isn't so different from
>> > Gremlin>
>> > > in any other language and that once you know the subtleties of the>
>> > > differences you can easily learn Gremlin from any example in any
>> > language.>
>> > >
>> > > I've not given a ton of thought to how gremlint would be brought in,
>> > where>
>> > > it would go in our source code tree, or how it fits in to>
>> > > release/deployment, but all that discussion can come later. I think
>> > that>
>> > > for right now, we should simply find consensus on

Re: [DISCUSS] Accepting gremlint donation

[Stephen Mallette]

As there haven't been any objections here, it sounds like we can go ahead
with this process. I believe that we will need to go through the IP
Clearance process in incubator:

https://incubator.apache.org/ip-clearance/

and engage Apache Infra about a gremlint.com domain transfer. And then of
course we will need to figure out "how" we make it part of the code base
(where it goes, how it fits in the release process, etc.) - my preference
would be to see it come in on 3.4.x so that we can immediately have an
official release of it, but we'll see how it goes. I suppose we will
continue to use this thread for all this sort of discussion for now unless
it gets too busy in which case we can spawn off other threads as needed..

On Thu, Sep 10, 2020 at 6:31 PM David Bechberger 
wrote:

> I definitely agree that having this sort of tool freely available would be
> very helpful to the community as a whole.
>
> I also would be able to help create the translator between GLV's and text
> representations as this is something I and many others have struggled with
> many times.
>
> Thanks,
> Dave
>
> On Thu, Sep 10, 2020 at 2:14 PM Kelvin Lawrence  >
> wrote:
>
> > I really like the idea of having an Apache TinkerPop hosted linter and
> > style guide "enforcer". I have spent many wasted hours hand formatting
> long
> > Gremlin queries people have asked me to look at over the years and the
> > latest version of Gremlint makes that so much easier. I also really like
> > the idea of extending the tool in the direction of "Gremlin converter". I
> > hear from a lot of users that porting from (let's say) Gremlin Console to
> > Gremlin Python (and vice-versa) is painful and hard to get right.
> >
> > Cheers
> > Kelvin
> >
> > On 2020/09/10 11:07:46, Stephen Mallette  wrote:
> > > I've had some on and off discussions with the developer of>
> > >
> > > https://gremlint.com/>
> > >
> > > whose name is Øyvind btw, about the possibility of making gremlint the>
> > > canonical Gremlin code stylizer, and by that I mean, his donating the
> > code>
> > > (and domain name) to the Apache Software Foundation under our project.
> > That>
> > > situation seems possible now and has been offered to the TinkerPop>
> > > Community.>
> > >
> > > I like the idea of bringing this code here because I feel our style
> > guide>
> > > in Gremlin Recipes can only suggest convention and I'd much like to see
> > a>
> > > code library managed by our project enforce a proper Gremlin style that
> > can>
> > > be used not only as the neat single HTML page of gremlint.com but also
> > as a>
> > > code library to Gremlin tool builders.>
> > >
> > > I also see the opportunity to grow this tool into an online translator
> > to>
> > > get you from one Gremlin dialect to another which is a constant problem
> > for>
> > > users who are working with the tons of code examples in Gremlin Groovy
> > but>
> > > need to get a fast conversion to Python or Javascript. I know I've had
> > that>
> > > problem myself on many occasions. An easy to use translator like this
> > would>
> > > help users realize that Gremlin in Groovy isn't so different from
> > Gremlin>
> > > in any other language and that once you know the subtleties of the>
> > > differences you can easily learn Gremlin from any example in any
> > language.>
> > >
> > > I've not given a ton of thought to how gremlint would be brought in,
> > where>
> > > it would go in our source code tree, or how it fits in to>
> > > release/deployment, but all that discussion can come later. I think
> > that>
> > > for right now, we should simply find consensus on whether or not to go
> > down>
> > > this path.>
> > >
> > > Are there any thoughts or concerns about accepting this code base and
> > the>
> > > gremlint.com domain?>
> > >
> > Cheers,
> > Kelvin
> >
> >
> >
>