If you perform deletes in parallel, it can be worth investing some time in making the code smart enough to choose disjoint data sets in transactions that run in parallel; e.g. no node should be start or end node in more than one parallel transaction at a time. That way they won’t contend on locks, or worse, run into deadlocks and roll back.
-- Chris Vest System Engineer, Neo Technology [ skype: mr.chrisvest, twitter: chvest ] > On 19 Jun 2016, at 01:20, 'Michael Hunger' via Neo4j <neo4j@googlegroups.com> > wrote: > > Shouldn't be slow. Faster disk. Concurrent batches would help. > > Von meinem iPhone gesendet > > Am 18.06.2016 um 22:29 schrieb John Fry <frydom.j...@gmail.com > <mailto:frydom.j...@gmail.com>>: > >> >> Clark - this works. It is still slow. I guess multithreading may help >> some.... >> >> >> >> Transaction tx = db.beginTx(); >> >> //try ( Transaction tx = db.beginTx() ) { >> >> for (int i=0; i<deletedLinks.size(); i++) { >> >> Relationship rel = >> db.getRelationshipById(deletedLinks.get(i)); >> >> rel.delete(); >> >> txc++; >> >> if (txc>50000) { >> >> txc=0; >> >> tx.success(); >> >> tx.close(); >> >> tx = db.beginTx(); >> >> } >> >> } >> >> tx.success(); >> >> tx.close(); >> >> //} >> >> //catch (Exception e) { >> >> // System.out.println("Exception link deletion: " + >> e.getMessage()); >> >> //} >> >> >> >> >> >> >> On Saturday, June 18, 2016 at 2:03:33 PM UTC-7, John Fry wrote: >> Thanks Clark - is there any good/recommended way to nest the commits? >> >> Thx JF >> >> On Saturday, June 18, 2016 at 1:43:19 PM UTC-7, Clark Richey wrote: >> You need to periodically commit. Holding that many transactions in memory >> isn't efficient. >> >> Sent from my iPhone >> >> On Jun 18, 2016, at 16:41, John Fry <frydo...@gmail.com <>> wrote: >> >>> Hello All, >>> >>> I have a graph of about 200M relationships and often I need to delete a >>> larges amount of them. >>> For the proxy code below I am seeing huge memory usage and memory thrashing >>> when deleting about 15M relationships. >>> >>> When it hits tx.close() I see all CPU cores start working at close to 100% >>> util and thrash for > 30mins. >>> I need this to work in <5mins ideally. >>> >>> (note when I execute large amounts of changes to properties or create large >>> amounts of new properties I don't have such issues) >>> >>> Any advice? Why is this happening? >>> >>> Regards, John. >>> >>> >>> >>> int txc = 0; >>> >>> // serially delete the links >>> >>> try ( Transaction tx = db.beginTx() ) { >>> >>> for (int i=0; i<deletedLinks.size(); i++) { >>> >>> Relationship rel = >>> db.getRelationshipById(deletedLinks.get(i)); >>> >>> rel.delete(); >>> >>> txc++; >>> >>> if (txc>50000) { >>> >>> txc=0; >>> >>> tx.success(); >>> >>> } >>> >>> } >>> >>> tx.success(); >>> >>> tx.close(); >>> >>> } >>> >>> catch (Exception e) { >>> >>> System.out.println("Exception link deletion: " + >>> e.getMessage()); >>> >>> } >>> >>> >>> -- >>> You received this message because you are subscribed to the Google Groups >>> "Neo4j" group. >>> To unsubscribe from this group and stop receiving emails from it, send an >>> email to neo4j+un...@googlegroups.com <>. >>> For more options, visit https://groups.google.com/d/optout >>> <https://groups.google.com/d/optout>. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Neo4j" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to neo4j+unsubscr...@googlegroups.com >> <mailto:neo4j+unsubscr...@googlegroups.com>. >> For more options, visit https://groups.google.com/d/optout >> <https://groups.google.com/d/optout>. > > > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to neo4j+unsubscr...@googlegroups.com > <mailto:neo4j+unsubscr...@googlegroups.com>. > For more options, visit https://groups.google.com/d/optout > <https://groups.google.com/d/optout>. -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
