[jira] [Created] (TINKERPOP-2417) Add possibility to supply io step with InputStream instead of a file path

[Oleksandr Porunov (Jira)]

Oleksandr Porunov created TINKERPOP-2417:


 Summary: Add possibility to supply io step with InputStream 
instead of a file path
 Key: TINKERPOP-2417
 URL: https://issues.apache.org/jira/browse/TINKERPOP-2417
 Project: TinkerPop
  Issue Type: New Feature
  Components: io
Reporter: Oleksandr Porunov


Currently it is possible to supply *io* step with file path to read a file from 
file system.

It would be very convenient if we would be able to read files from input stream 
instead of file path. 

For example, it could look something like:

*g.io(graphSonInputStream).with(IO.reader, IO.graphson).read().iterate()*

Of course it wouldn't work as expected if we would use *write* instead of read 
here because we won't be able to write to the input stream. Thus, we can fail 
fast if the user supplied an input stream into the *io* step but tried to use 
*write*.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Commented] (TINKERPOP-2417) Add possibility to supply io step with InputStream instead of a file path

[Oleksandr Porunov (Jira)]

[ 
https://issues.apache.org/jira/browse/TINKERPOP-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17193539#comment-17193539
 ] 

Oleksandr Porunov commented on TINKERPOP-2417:
--

[~spmallette] graph.io(Builder) allows to create a custom reader / writer which 
can be supplied with InputStream or OutputStream.

I.e. something like:

graph.io(IoCore.graphson()).reader().create().readGraph(graphSonInputStream), 
graph)

 

The deprecation says to use GraphTraversalSource.io(String) which isn't the 
same as it expects filesystem path which sometimes might be very inconvenient.

Examples:
 * The user receives InputStream of GraphSON but the user don't have write 
access to the underlying file system to write GraphSON file temporary and then 
remove it after it was ingested to the graph.
 * Server disks don't have enough capacity to write a temporary file for 
ingestion.
 * Server disks are too slow thus limiting ingestion performance.

 

Would it be better to undeprecate graph.io(Builder) until there is a solution 
which replaces it? Is it planned to remove graph.io(Builder) soon?

> Add possibility to supply io step with InputStream instead of a file path
> -
>
> Key: TINKERPOP-2417
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2417
> Project: TinkerPop
>  Issue Type: Improvement
>  Components: io
>Reporter: Oleksandr Porunov
>Priority: Major
>
> Currently it is possible to supply *io* step with file path to read a file 
> from file system.
> It would be very convenient if we would be able to read files from input 
> stream instead of file path. 
> For example, it could look something like:
> *g.io(graphSonInputStream).with(IO.reader, IO.graphson).read().iterate()*
> Of course it wouldn't work as expected if we would use *write* instead of 
> read here because we won't be able to write to the input stream. Thus, we can 
> fail fast if the user supplied an input stream into the *io* step but tried 
> to use *write*.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (TINKERPOP-2633) Support Gremlin Console on Java 17

[Oleksandr Porunov (Jira)]

Oleksandr Porunov created TINKERPOP-2633:


 Summary: Support Gremlin Console on Java 17
 Key: TINKERPOP-2633
 URL: https://issues.apache.org/jira/browse/TINKERPOP-2633
 Project: TinkerPop
  Issue Type: New Feature
  Components: console
 Environment: Java 17
Reporter: Oleksandr Porunov


Currently Gremlin Console cannot be started on Java 17 following the next error:

java.lang.IllegalArgumentException: Unsupported class file major version 61

 

It would be great if Gremlin Console supports Java 17 as it is the newest LTS 
release after Java 11 and most likely people will start switching to Java 17 
sooner or later.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Updated] (TINKERPOP-2633) Support Gremlin Console on Java 17

[Oleksandr Porunov (Jira)]

 [ 
https://issues.apache.org/jira/browse/TINKERPOP-2633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Oleksandr Porunov updated TINKERPOP-2633:
-
Affects Version/s: 3.5.1

> Support Gremlin Console on Java 17
> --
>
> Key: TINKERPOP-2633
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2633
> Project: TinkerPop
>  Issue Type: New Feature
>  Components: console
>Affects Versions: 3.5.1
> Environment: Java 17
>Reporter: Oleksandr Porunov
>Priority: Major
>
> Currently Gremlin Console cannot be started on Java 17 following the next 
> error:
> java.lang.IllegalArgumentException: Unsupported class file major version 61
>  
> It would be great if Gremlin Console supports Java 17 as it is the newest LTS 
> release after Java 11 and most likely people will start switching to Java 17 
> sooner or later.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (TINKERPOP-2924) Refactor PropertyMapStep to be able to overwrite map method

[Oleksandr Porunov (Jira)]

Oleksandr Porunov created TINKERPOP-2924:


 Summary: Refactor PropertyMapStep to be able to overwrite map 
method
 Key: TINKERPOP-2924
 URL: https://issues.apache.org/jira/browse/TINKERPOP-2924
 Project: TinkerPop
  Issue Type: Improvement
  Components: driver
Affects Versions: 3.6.2
Reporter: Oleksandr Porunov


We would like to extend `PropertyMapStep` and overwrite some of it's 
functionality (in `map` method), so that we could leverage multi-query 
optimization in JanusGraph for `PropertyMapStep`. 

Unfortunately, some of it's utility methods and  fields are `private`. Thus, we 
should duplicate `includeToken` logic and we don't have any access to 
`traversalRing` because it's private and is created inside the constructor.

I would suggest making all of those private fields / methods as protected 
(similarly as it's done in `PropertiesStep`), so that there would be a 
possibility for overwrite without logic duplication.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (TINKERPOP-2927) Make all Steps extensible and overridable

[Oleksandr Porunov (Jira)]

Oleksandr Porunov created TINKERPOP-2927:


 Summary: Make all Steps extensible and overridable 
 Key: TINKERPOP-2927
 URL: https://issues.apache.org/jira/browse/TINKERPOP-2927
 Project: TinkerPop
  Issue Type: Improvement
  Components: driver
Affects Versions: 3.6.2
Reporter: Oleksandr Porunov


Related issue (fixed): https://issues.apache.org/jira/browse/TINKERPOP-2924

 

Working on optimization strategies sometimes require replacing steps with 
extended version of those steps. At this moment not all steps can be extended 
due to being `final` (like `ProjectStep`, `PropertyKeyStep`, 
`PropertyValueStep`, `RangeLocalStep`, `SumLocalStep`, and many more). Thus, it 
requires creating a similar step and duplicate some logic there instead of 
simply extending a specific step.

 

For those steps which are non-final there are sometimes private fields without 
any getter methods (for example `private CallbackRegistry 
callbackRegistry` in `DropStep` is `private`. Thus, the caller needs to use 
Reflaction API to retrieve it's value).

 

In JanusGraph we replace some steps with the extended version of those steps. 

For example, we completely overwrite `flatMap` step of `PropertiesStep` which 
is an anti-pattern, but in the case when it's hard to extend specific logic 
parts such anti-pattern might be a a good solution I guess.

 

I think it would make sense to let Graph developers to extend any step and has 
access to it's fields / utility methods.

In such case we could do similar with `ProjectStep` and make it query data in 
parallel (see issue: [https://github.com/JanusGraph/janusgraph/issues/3559] ).

 

I'm also good not doing it in case anyone can suggest other patterns to follow 
for those optimizations instead of overwriting logic. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (TINKERPOP-2490) RangeGlobalStep touches next traverser when high limit is already hit

[Oleksandr Porunov (Jira)]

[ 
https://issues.apache.org/jira/browse/TINKERPOP-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17715490#comment-17715490
 ] 

Oleksandr Porunov commented on TINKERPOP-2490:
--

This bug still exists in TinkerPop (checked version 3.6.2). An example test 
which shows when this behavior is not expected is below:
{code:java}
@Test
public void testLimitedFilterNotChecksElementsOverTheLimit(){
TinkerGraph tinkerGraph = TinkerGraph.open();
List vertices = new ArrayList<>();
for(int i=0; i<3; i++){
Vertex v1 = tinkerGraph.addVertex();
Vertex v2 = tinkerGraph.addVertex();
v1.addEdge("connects", v2);
vertices.add(v1);
}
tinkerGraph.traversal().V(vertices.get(0), vertices.get(1), vertices.get(2))
.where(__.out("connects").count().is(P.gte(1))).limit(1).toList();

TraversalMetrics traversalMetrics = tinkerGraph.traversal()
.V(vertices.get(0), vertices.get(1), vertices.get(2))
.where(__.out("connects").count().is(P.gte(1)))
.limit(1)
.profile().next();

Long filterTraversalCount = 
traversalMetrics.getMetrics().stream().filter(metrics -> 
metrics.getName().startsWith(TraversalFilterStep.class.getSimpleName()))
.findFirst().get().getCount(TraversalMetrics.TRAVERSER_COUNT_ID);

Assertions.assertEquals(1, filterTraversalCount);
} {code}
In the above test the filter first checks the first provided vertex (which runs 
some potentially expensive check) then when limit is satisfied the check is 
executed again for the second provided vertex (even so it's unnecessary). The 
3rd vertex is not evaluated.

Thus, basically, we execute this computation always for 1 more unnecessary 
vertex.

> RangeGlobalStep touches next traverser when high limit is already hit
> -
>
> Key: TINKERPOP-2490
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2490
> Project: TinkerPop
>  Issue Type: Bug
>  Components: process
>Affects Versions: 3.4.8
>Reporter: Guo Junshi
>Priority: Major
>
> In FilterStep, the processNextStart() method will first retrieve next 
> traverser and then apply filtering logic. But for RangleGlobalStep, if high 
> limit is already hit, there will be no need to get next traverser.
> {code:java}
> @Override
> protected Traverser.Admin processNextStart() {
> while (true) {
> final Traverser.Admin traverser = this.starts.next();
> if (this.filter(traverser))
> return traverser;
> }
> }
> {code}
> An example would be limit step: g.V().limit(1). This query will touch 2 
> vertices although only 1 vertex will be returned.
> This extra data loading will cause performance defects if DB data loading is 
> involved. It is not a functionality bug, but for better performance, we'd 
> better check high range limit first before touching next traversal.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Comment Edited] (TINKERPOP-2490) RangeGlobalStep touches next traverser when high limit is already hit

[Oleksandr Porunov (Jira)]

[ 
https://issues.apache.org/jira/browse/TINKERPOP-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17715490#comment-17715490
 ] 

Oleksandr Porunov edited comment on TINKERPOP-2490 at 4/23/23 4:25 PM:
---

This bug still exists in TinkerPop (checked version 3.6.2). An example test 
which shows when this behavior is not expected is below:
{code:java}
@Test
public void testLimitedFilterNotChecksElementsOverTheLimit(){
TinkerGraph tinkerGraph = TinkerGraph.open();
List vertices = new ArrayList<>();
for(int i=0; i<3; i++){
Vertex v1 = tinkerGraph.addVertex();
Vertex v2 = tinkerGraph.addVertex();
v1.addEdge("connects", v2);
vertices.add(v1);
}
tinkerGraph.traversal().V(vertices.get(0), vertices.get(1), vertices.get(2))
.where(__.out("connects").count().is(P.gte(1))).limit(1).toList();

TraversalMetrics traversalMetrics = tinkerGraph.traversal()
.V(vertices.get(0), vertices.get(1), vertices.get(2))
.where(__.out("connects").count().is(P.gte(1)))
.limit(1)
.profile().next();

Long filterTraversalCount = 
traversalMetrics.getMetrics().stream().filter(metrics -> 
metrics.getName().startsWith(TraversalFilterStep.class.getSimpleName()))
.findFirst().get().getCount(TraversalMetrics.TRAVERSER_COUNT_ID);

Assertions.assertEquals(1, filterTraversalCount);
} {code}
In the above test the filter first checks the first provided vertex (which runs 
some potentially expensive check) then when limit is satisfied the check is 
executed again for the second provided vertex (even so it's unnecessary). The 
3rd vertex is not evaluated.

Thus, basically, we execute this computation always for 1 more unnecessary 
vertex.

The above assertEquals will throw:

 
{code:java}
org.opentest4j.AssertionFailedError: 
Expected :1
Actual   :2 {code}
 

 


was (Author: porunov):
This bug still exists in TinkerPop (checked version 3.6.2). An example test 
which shows when this behavior is not expected is below:
{code:java}
@Test
public void testLimitedFilterNotChecksElementsOverTheLimit(){
TinkerGraph tinkerGraph = TinkerGraph.open();
List vertices = new ArrayList<>();
for(int i=0; i<3; i++){
Vertex v1 = tinkerGraph.addVertex();
Vertex v2 = tinkerGraph.addVertex();
v1.addEdge("connects", v2);
vertices.add(v1);
}
tinkerGraph.traversal().V(vertices.get(0), vertices.get(1), vertices.get(2))
.where(__.out("connects").count().is(P.gte(1))).limit(1).toList();

TraversalMetrics traversalMetrics = tinkerGraph.traversal()
.V(vertices.get(0), vertices.get(1), vertices.get(2))
.where(__.out("connects").count().is(P.gte(1)))
.limit(1)
.profile().next();

Long filterTraversalCount = 
traversalMetrics.getMetrics().stream().filter(metrics -> 
metrics.getName().startsWith(TraversalFilterStep.class.getSimpleName()))
.findFirst().get().getCount(TraversalMetrics.TRAVERSER_COUNT_ID);

Assertions.assertEquals(1, filterTraversalCount);
} {code}
In the above test the filter first checks the first provided vertex (which runs 
some potentially expensive check) then when limit is satisfied the check is 
executed again for the second provided vertex (even so it's unnecessary). The 
3rd vertex is not evaluated.

Thus, basically, we execute this computation always for 1 more unnecessary 
vertex.

> RangeGlobalStep touches next traverser when high limit is already hit
> -
>
> Key: TINKERPOP-2490
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2490
> Project: TinkerPop
>  Issue Type: Bug
>  Components: process
>Affects Versions: 3.4.8
>Reporter: Guo Junshi
>Priority: Major
>
> In FilterStep, the processNextStart() method will first retrieve next 
> traverser and then apply filtering logic. But for RangleGlobalStep, if high 
> limit is already hit, there will be no need to get next traverser.
> {code:java}
> @Override
> protected Traverser.Admin processNextStart() {
> while (true) {
> final Traverser.Admin traverser = this.starts.next();
> if (this.filter(traverser))
> return traverser;
> }
> }
> {code}
> An example would be limit step: g.V().limit(1). This query will touch 2 
> vertices although only 1 vertex will be returned.
> This extra data loading will cause performance defects if DB data loading is 
> involved. It is not a functionality bug, but for better performance, we'd 
> better check high range limit first before touching next traversal.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Commented] (TINKERPOP-2490) RangeGlobalStep touches next traverser when high limit is already hit

[Oleksandr Porunov (Jira)]

[ 
https://issues.apache.org/jira/browse/TINKERPOP-2490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17715539#comment-17715539
 ] 

Oleksandr Porunov commented on TINKERPOP-2490:
--

Maybe I'm missing something, but I doubt that it's `CountStrategy. Even without 
`count` step at all (let's replace it with `has` step for example) the behavior 
will be the same. `has` step will be executed for 2 vertices even so, `limit` 
step says that `1` is enough. Could be I'm missing some logic behind those step 
usages, but in my understanding the filter query should be executed for a 
single vertex if that vertex matches the filter and the limit is reached.

I.e. `V(v1, v2, v3) .where(${some_filter_which_is_always_true}).limit(1)` in my 
testing `some_filter_which_is_always_true` is executed 2 times, but I would 
expect it to be executed only once because limit says that we don't need more 
elements then 1.

As for count query above, notice that I use `.gte` and not `gt`. Thus it should 
be grater or equal and thus `1` should satisfy the requirements. I don't see a 
point of checking a second element to find out that the amount of elements is 
more if equal to 1 satisfies the requirements as well. Nevertheless, my issue 
isn't in the Count query, but in the filter step which is executed more than 
needed (from my point of view).

Here is a simplified version of the test:
{code:java}
@Test
public void testLimitedFilterNotChecksElementsOverTheLimit(){
TinkerGraph tinkerGraph = TinkerGraph.open();
List vertices = new ArrayList<>();
for(int i=0; i<3; i++){
Vertex v1 = tinkerGraph.addVertex();
Vertex v2 = tinkerGraph.addVertex();
v1.addEdge("connects", v2);
vertices.add(v1);
}

TraversalMetrics traversalMetrics = tinkerGraph.traversal()
.V(vertices.get(0), vertices.get(1), vertices.get(2))
.where(__.inject(true))
.limit(1)
.profile().next();

Long filterTraversalCount = 
traversalMetrics.getMetrics().stream().filter(metrics -> 
metrics.getName().startsWith(TraversalFilterStep.class.getSimpleName()))
.findFirst().get().getCount(TraversalMetrics.TRAVERSER_COUNT_ID);

Assertions.assertEquals(1, filterTraversalCount);
} {code}
 

> RangeGlobalStep touches next traverser when high limit is already hit
> -
>
> Key: TINKERPOP-2490
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2490
> Project: TinkerPop
>  Issue Type: Bug
>  Components: process
>Affects Versions: 3.4.8
>Reporter: Guo Junshi
>Priority: Major
>
> In FilterStep, the processNextStart() method will first retrieve next 
> traverser and then apply filtering logic. But for RangleGlobalStep, if high 
> limit is already hit, there will be no need to get next traverser.
> {code:java}
> @Override
> protected Traverser.Admin processNextStart() {
> while (true) {
> final Traverser.Admin traverser = this.starts.next();
> if (this.filter(traverser))
> return traverser;
> }
> }
> {code}
> An example would be limit step: g.V().limit(1). This query will touch 2 
> vertices although only 1 vertex will be returned.
> This extra data loading will cause performance defects if DB data loading is 
> involved. It is not a functionality bug, but for better performance, we'd 
> better check high range limit first before touching next traversal.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)