Thanks for the advice! I will try it out and let you know how it goes. The stack overflow answer is here: http://stackoverflow.com/questions/37126965/iterating-through-relationships-from-query-result-while-adding-new-relationship/37133192#37133192
So I just copied that part of the code and modified to suit my need. I might have done some incorrect modification though. I really appreciate your help, thank you very much! Kind regards, Cherie On 13 May, 2016 12:38 pm, "'Michael Hunger' via Neo4j" < neo4j@googlegroups.com> wrote: > Filter in cypher what you return, and yes only return the props needed > > For the batching you can also send in a list of maps on which you can > store additional props or information. > > For concurrent execution neither guarantees uniqueness of rels but you > execute one after another, so you have a guarantee with merge and create > unique > > Where did you find that answer on SO? > > Von meinem iPhone gesendet > > Am 13.05.2016 um 13:31 schrieb Cherie Pun <cherie.cy....@gmail.com>: > > Hi, > > Thanks for the quick reply. I returned the relationship because the > property of the transitive relationship that I am trying to create depends > on the property of the original two relationship. Should I have just > returned the properties and node id to reduce data? There are two different > types of transitive relationship I want to create: 1) assign the property > of bc to ac 2) assign the product of the trust strength of ab and bc to ac. > > The tx-handling code was written according to answers from stack overflow > as I don't have much knowledge in batch executing cypher queries. Thanks > for correcting it. > > I have read that merge does not guarantee uniqueness of relationship, so > when should I use create unique and when I should I use merge? Thanks. > > Kind regards, > Cherie > On 13 May, 2016 12:16 pm, "'Michael Hunger' via Neo4j" < > neo4j@googlegroups.com> wrote: > > > Your tx-handling code looks wrong. > > You should execute the whole batch of creating the connections in the > other thread, Just grabbing the transaction there is very dangerous as the > tx will be used that's attached to the current thread instead. > > I also recommend batching in cypher, that makes it much easier: > Just send in a batch (e.g. 100k) list of pairs of id's as parameters and > then do > > pool.submit( () -> > Iterables.<Long>singleOrNull(db.execute(" > UNWIND {data} as pair > MATCH (a),(b) WHERE id(a) = pair[0] and id(b) = pair[1] > MERGE (a)-[:TRUST]->(b) RETURN count(*) as c; > ", batchOfIds()).columnAs("c")); > )).get() > > If your return ab, bc then that's relationships? but in the other query > you match nodes? > > You should only return id(a),id(c) to minimize the data created. > > > > Am 13.05.2016 um 12:49 schrieb Cherie Pun <cherie.cy....@gmail.com>: > > > > Hi, > > > > Thanks for your reply. By actual code do you mean all my files that are > related? I have attached the main part of the code that does the querying > underneath the stacktrace. The other files just defined strings for the > property names and enum type for relationship and node. > > > > The query that I ran first was : > > private final String FIND_TRANSITIVE_RELATION_QUERY = > > "MATCH (a:" + TwitterLabels.USERS + ") " + > > "- [ab:" + TwitterRelationships.TRUST + "] -> " + > > "(b:" + TwitterLabels.USERS + ") " + > > "- [bc:" + TwitterRelationships.TRUST + "] -> " + > > "(c:" + TwitterLabels.USERS + ")\n" + > > "WHERE a <> c\n" + > > "return ab, bc"; > > Iterating through the result from the query, I then perform a another > query that create a unique trust relationship between node a and c: > > private final String CREATE_UNIQUE_TRUST_RELATION_QUERY = > > "MATCH (a:" + TwitterLabels.USERS + "),(b:" + > TwitterLabels.USERS + ")\n" + > > "WHERE id(a) = {startNode} AND id(b) = {endNode} \n" + > > "CREATE UNIQUE (a) - [r:" + TwitterRelationships.TRUST + > "] -> (b) \n" + > > "return r"; > > > > The second query is being executed on another thread, and the thread > commits the transaction for every 1000 queries. > > > > This executes fine until it processed 200,000 result and crashes because > of gc limit reached. I did a heap dump and found out that the > relationshipproxy objects were taking up most of the space. > > > > I hope that gives you sufficient information, please let me know if you > need more explanation of my code. > > > > Kind regards, > > Cherie > > > > > > Can you share your actual code and queries > > > > Von meinem iPhone gesendet > > > > Am 12.05.2016 um 18:52 schrieb Cherie Pun <cherie.cy....@gmail.com>: > > > > > I am trying to iterate through the results from the first query and > execute an extra cypher query for each result to either create or find the > existing relationship. However I encountered GC overhead error and the > stack trace is as followed. I thought running the queries in batch of 100 > will solve the memory problem but it seems that either it's using up the > memory too quickly or not releasing the memory quick enough. It broke down > after processing 200000 queries after I increased the batch size to 1000. > Is there a way to optimise my query to reduce the memory used, or is there > an alternative way to achieve the same result? Thanks. > > > > > > Exception in thread "main" java.lang.OutOfMemoryError: GC overhead > limit exceeded > > > at > scala.collection.mutable.OpenHashMap.foreachUndeletedEntry(OpenHashMap.scala:226) > > > at > scala.collection.mutable.OpenHashMap.foreach(OpenHashMap.scala:219) > > > at > org.neo4j.cypher.internal.compiler.v2_3.ExecutionContext.foreach(ExecutionContext.scala:46) > > > at > scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:155) > > > at > org.neo4j.cypher.internal.compiler.v2_3.ExecutionContext.foldLeft(ExecutionContext.scala:33) > > > at > org.neo4j.cypher.internal.frontend.v2_3.helpers.Eagerly$.mapToBuilder(Eagerly.scala:44) > > > at > org.neo4j.cypher.internal.frontend.v2_3.helpers.Eagerly$.immutableMapValues(Eagerly.scala:37) > > > at > org.neo4j.cypher.internal.compiler.v2_3.ClosingIterator$$anonfun$next$1.apply(ResultIterator.scala:76) > > > at > org.neo4j.cypher.internal.compiler.v2_3.ClosingIterator$$anonfun$next$1.apply(ResultIterator.scala:72) > > > at > org.neo4j.cypher.internal.compiler.v2_3.ClosingIterator$$anonfun$failIfThrows$1.apply(ResultIterator.scala:121) > > > at > org.neo4j.cypher.internal.compiler.v2_3.ClosingIterator.decoratedCypherException(ResultIterator.scala:130) > > > at > org.neo4j.cypher.internal.compiler.v2_3.ClosingIterator.failIfThrows(ResultIterator.scala:119) > > > at > org.neo4j.cypher.internal.compiler.v2_3.ClosingIterator.next(ResultIterator.scala:72) > > > at > org.neo4j.cypher.internal.compiler.v2_3.ClosingIterator.next(ResultIterator.scala:50) > > > at > org.neo4j.cypher.internal.compiler.v2_3.PipeExecutionResult.next(PipeExecutionResult.scala:77) > > > at > org.neo4j.cypher.internal.compiler.v2_3.PipeExecutionResult$$anon$2.next(PipeExecutionResult.scala:70) > > > at > org.neo4j.cypher.internal.compiler.v2_3.PipeExecutionResult$$anon$2.next(PipeExecutionResult.scala:68) > > > at > org.neo4j.cypher.internal.compatibility.ExecutionResultWrapperFor2_3$$anon$1$$anonfun$next$1.apply(CompatibilityFor2_3.scala:234) > > > at > org.neo4j.cypher.internal.compatibility.ExecutionResultWrapperFor2_3$$anon$1$$anonfun$next$1.apply(CompatibilityFor2_3.scala:234) > > > at > org.neo4j.cypher.internal.compatibility.exceptionHandlerFor2_3$.runSafely(CompatibilityFor2_3.scala:116) > > > at > org.neo4j.cypher.internal.compatibility.ExecutionResultWrapperFor2_3$$anon$1.next(CompatibilityFor2_3.scala:234) > > > at > org.neo4j.cypher.internal.compatibility.ExecutionResultWrapperFor2_3$$anon$1.next(CompatibilityFor2_3.scala:229) > > > at > org.neo4j.cypher.javacompat.ExecutionResult.next(ExecutionResult.java:233) > > > at > org.neo4j.cypher.javacompat.ExecutionResult.next(ExecutionResult.java:55) > > > at java.util.Iterator.forEachRemaining(Iterator.java:116) > > > at > Neo4j.TrustCalculator.iterateAndExecuteBatchedInSeparateThread(TrustCalculator.java:239) > > > at > Neo4j.TrustCalculator.insertSimpleTransitiveTrust(TrustCalculator.java:127) > > > at Neo4j.TrustCalculator.main(TrustCalculator.java:265) > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > > > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > > at java.lang.reflect.Method.invoke(Method.java:483) > > > > > > private final String CREATE_UNIQUE_TRUST_RELATION_QUERY = > > > "MATCH (a:" + TwitterLabels.USERS + "),(b:" + > TwitterLabels.USERS + ")\n" + > > > "WHERE id(a) = {startNode} AND id(b) = {endNode} \n" + > > > "CREATE UNIQUE (a) - [r:" + TwitterRelationships.TRUST > + "] -> (b) \n" + > > > "return r"; > > > > > > private final String FIND_TRANSITIVE_RELATION_QUERY = > > > "MATCH (a:" + TwitterLabels.USERS + ") " + > > > "- [ab:" + TwitterRelationships.TRUST + "] -> " + > > > "(b:" + TwitterLabels.USERS + ") " + > > > "- [bc:" + TwitterRelationships.TRUST + "] -> " + > > > "(c:" + TwitterLabels.USERS + ")\n" + > > > "WHERE a <> c\n" + > > > "return ab, bc"; > > > > > > private ExecutorService createSinglePool() { > > > return Executors.newSingleThreadExecutor(); > > > } > > > > > > private void iterateAndExecuteBatchedInSeparateThread(int batchsize, > Iterator iterator, Consumer consumer) { > > > final int[] opsCount = {0}; > > > final int[] batchesCount = {0}; > > > ExecutorService executorService = createSinglePool(); > > > try { > > > final Transaction[] workerTransaction = > {executorService.submit(() -> graphDb.beginTx()).get()}; > > > > > > iterator.forEachRemaining(t -> executorService.submit(() -> { > > > consumer.accept(t); > > > if ((++opsCount[0]) % batchsize == 0) { > > > batchesCount[0]++; > > > workerTransaction[0].success(); > > > workerTransaction[0].close(); > > > workerTransaction[0] = graphDb.beginTx(); > > > } > > > })); > > > executorService.submit(() -> { > > > workerTransaction[0].success(); > > > workerTransaction[0].close(); > > > }).get(); > > > > > > } catch (InterruptedException | ExecutionException e) { > > > throw new RuntimeException(e); > > > } > > > } > > > > > > > > > public void insertSimpleTransitiveTrust(){ > > > try ( Transaction tx = graphDb.beginTx() ) > > > { > > > Result results = graphDb.execute( > FIND_TRANSITIVE_RELATION_QUERY, new HashMap<String, Object>() ); > > > > > > iterateAndExecuteBatchedInSeparateThread(1000, results, > result-> insertOneSimpleTransitiveTrust((Map<String, Object>) result)); > > > > > > tx.success(); > > > } > > > } > > > > > > private void insertOneSimpleTransitiveTrust(Map<String, Object> map){ > > > Object object = map.get("ab"); > > > Relationship trustAb = (Relationship) object; > > > object = map.get("bc"); > > > Relationship trustBc = (Relationship) object; > > > if(trustAb.hasProperty(TwitterProperties.CONVERSATIONAL_TRUST) && > trustBc.hasProperty(TwitterProperties.CONVERSATIONAL_TRUST)){ > > > Node nodeA = trustAb.getStartNode(); > > > Node nodeB = trustAb.getEndNode(); > > > Node nodeC = trustBc.getEndNode(); > > > > > > Map<String, Object> params = new HashMap<String, Object>(); > > > params.put("startNode", nodeA.getId()); > > > params.put("endNode", nodeC.getId()); > > > Result results = graphDb.execute( > CREATE_UNIQUE_TRUST_RELATION_QUERY, params ); > > > if(results.hasNext()){ > > > Map<String, Object> result = results.next(); > > > object = result.get("r"); > > > Relationship transitiveTrust = (Relationship) object; > > > > transitiveTrust.setProperty(TwitterProperties.SIMPLE_TRANSITIVE_TRUST, > trustBc.getProperty(TwitterProperties.CONVERSATIONAL_TRUST)); > > > } > > > } > > > } > > > > > > > > > > > > -- > > > You received this message because you are subscribed to the Google > Groups "Neo4j" group. > > > To unsubscribe from this group and stop receiving emails from it, send > an email to neo4j+unsubscr...@googlegroups.com. > > > For more options, visit https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to a topic in the > Google Groups "Neo4j" group. > > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/neo4j/sjFDz496AwA/unsubscribe. > > To unsubscribe from this group and all its topics, send an email to > neo4j+unsubscr...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > > > -- > > You received this message because you are subscribed to the Google > Groups "Neo4j" group. > > To unsubscribe from this group and stop receiving emails from it, send > an email to neo4j+unsubscr...@googlegroups.com. > > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Neo4j" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/neo4j/sjFDz496AwA/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > neo4j+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to neo4j+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Neo4j" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/neo4j/sjFDz496AwA/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > neo4j+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.