[
https://issues.apache.org/jira/browse/TINKERPOP-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephen Mallette updated TINKERPOP-2424:
----------------------------------------
Priority: Major (was: Critical)
> Reduce chance for OOME with large results to Java driver
> --------------------------------------------------------
>
> Key: TINKERPOP-2424
> URL: https://issues.apache.org/jira/browse/TINKERPOP-2424
> Project: TinkerPop
> Issue Type: Improvement
> Components: driver
> Affects Versions: 3.4.8
> Reporter: Stephen Mallette
> Priority: Major
>
> Originally mentioned here:
> https://groups.google.com/g/gremlin-users/c/I4HQC9JkzSo/m/fYfd5o0UAQAJ
> and pretty easy to create with an empty TinkerGraph in Gremlin Server with
> {{evaluationTimeout}} set to something large using this script in the Gremlin
> Console with {{-Xmx512}}:
> {code}
> cluster = Cluster.open()
> client = cluster.connect()
> client.submit("g.addV().as('a').addE('self').iterate()")
> rs =
> client.submit("g.V().emit().repeat(out()).valueMap(true).limit(10000000)");[]
> iterator = rs.iterator();[]
> x = 0
> while(iterator.hasNext()) {
> x++
> if (x % 10000 == 0) {
> System.out.println(x + "-[" + rs.getAvailableItemCount() + "]-"+
> iterator.next());
> }
> }
> {code}
> The {{LinkedBlockingQueue}} of the {{ResultQueue}} is unbounded and can fill
> faster than can be consumed and on a system with limited memory an OOME can
> loom.
> While we tend to discourage iteration of large result sets {{e.g. g.V()}} I
> suppose we should do what we can to keep users out of OOME situations if we
> can. Not sure of the best way to do this but some simple experimentation
> showed that bounding the queue helps (tried with 100000) but does require the
> adding of new results to be blocked until more are consumed.
> {code}
> +++
> b/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/Connection.java
> @@ -233,7 +233,7 @@ final class Connection {
>
> cluster.executor().submit(() ->
> resultQueueSetup.completeExceptionally(f.cause()));
> } else {
> - final LinkedBlockingQueue<Result>
> resultLinkedBlockingQueue = new LinkedBlockingQueue<>();
> + final LinkedBlockingQueue<Result>
> resultLinkedBlockingQueue = new LinkedBlockingQueue<>(100000);
> final CompletableFuture<Void> readCompleted = new
> CompletableFuture<>();
>
> readCompleted.whenCompleteAsync((v, t) -> {
> diff --git
> a/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/ResultQueue.java
>
> b/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/ResultQueue.java
> index 29a6453431..4b52ae1671 100644
> ---
> a/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/ResultQueue.java
> +++
> b/gremlin-driver/src/main/java/org/apache/tinkerpop/gremlin/driver/ResultQueue.java
> @@ -70,7 +70,7 @@ final class ResultQueue {
> * @param result a return value from the {@link Traversal} or script
> submitted for execution
> */
> public void add(final Result result) {
> - this.resultLinkedBlockingQueue.offer(result);
> + while(!this.resultLinkedBlockingQueue.offer(result)) {}
> tryDrainNextWaiting(false);
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
