On 27 September 2016 at 08:03, Phillip Henry <phillhe...@gmail.com> wrote:
> Hi, Luca. > > I've now tried OGraphBatchInsert. It is indeed much faster at about 4.5 > hours for the billion payments. Slower than Neo but we can live with that. > Hi Phillip, Good to know it worked. Are you using last v2.2.10 right? > However, I'm having trouble getting a full run. > > I'm getting OutOfMemory errors with -XX:MaxDirectMemorySize=512G and > combinations of: > > -Xmx51443M -Dstorage.diskCache.bufferSize=60059 > -Xmx90G -Dstorage.diskCache.bufferSize=10240 > -Xmx90G -Dstorage.diskCache.bufferSize=8192 > > So, I've now increased MaxDirectMemorySize to 999G but with no success > (this box has 128GB of memory). > > On a box with about 250GB where I can increase the heap, it ran to the > end. There was some SEVEREs at the beginning that said "Previous maximum > cache size was 3474813 current maximum cache is 278528. Cache state for > storage /home/d3956122/OrientDB/databases/MyPayments3 will not be > restored" and some "Exception during commit of active transaction... > Database /home/d3956122/OrientDB/databases/MyPayments3 is closed". But > after that things seem to go well. I hope these initial errors are not too > serious? > So It looks like 128GB wasn't enough. About the two exception, let me ask to the team first. About the second one "Exception during commit of active transaction... Database /home/d3956122/OrientDB/databases/MyPayments3 is closed" do you have the stack trace? > > Unfortunately, there was a hiccup in my coding where the app doesn't > naturally die when all the writing is over (I never stop the thread pool - > oops). However, it was a day or two later when I killed it. After I killed > the processes, I was surprised to see only a few hundred vertices in the > plocal database when I was expecting to see one million (the number of > edges was much closer to what I expected). At what point are the vertices > flushed? Can I flush them via the API? > Do you mean with the batch API? Did you call the end() ? > Regards, > > Phillip > Best Regards, Luca Garulli Founder & CEO OrientDB LTD <http://orientdb.com/> Want to share your opinion about OrientDB? Rate & review us at Gartner's Software Review <https://www.gartner.com/reviews/survey/home> > > > On Friday, September 23, 2016 at 6:27:02 PM UTC+1, l.garulli wrote: >> >> On 23 September 2016 at 11:23, Phillip Henry <phill...@gmail.com> wrote: >> >>> > will there not be potential contention when the "to" vertex is updated? >>> >>> Ah, just re-read your post and you've already answered this. My >>> apologies. >>> >> >> Yes, the idea is that with millions and mullions of vertices, the chance >> to have a collision with the target nodes is very low, unless you have >> supernodes that recurs in most of the relationships. >> >> >>> >>>> > OGraphBatchInsert ... keeps everything in RAM before flushing >>>> >>>> I assume I will still have to write retry code in the event of a >>>> collision (see above)? >>>> >>> >> No in this case, the batch insert will manage this for you. >> >> >> Luca >> >> > -- > > --- > You received this message because you are subscribed to the Google Groups > "OrientDB" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to orient-database+unsubscr...@googlegroups.com. > For more options, visit https://groups.google.com/d/optout. > -- --- You received this message because you are subscribed to the Google Groups "OrientDB" group. To unsubscribe from this group and stop receiving emails from it, send an email to orient-database+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
