Sorry i forgot to add - in addition the application sits idle for many minutes with close to zero cpu activity and presents a very slow increase in memory.
On Wednesday, May 4, 2016 at 10:30:23 AM UTC-7, John Fry wrote: > > Hi All, > > I am seeing slow and worsening memory performance and ~6 hour run times > for the application detailed below. > I am running to close to using all 32G of RAM and often the AWS > stalls/fails due to memory allocation issues. This is despite liming the > application/JVM to 24G. > Garbage collection rates worsen as the application progresses. > > What I need help with is the following: > > - given the description below - is 6 hours run-time and 32G footprint > about correct? > - how do you know when you have tuned a single instance embedded use > of neo4j to optimal memory performance? (is there a benchmark to tune > against?) > - is there anything obviously wrong or naive with the approach below? > - what other tuning options are available for me to try? > > Let me know if you need see any code. > > Many thanks in advance for help, John. > > Environment: > > - AWS m4.2large - 16 VCPUs; 32G RAM > - Application is using neo4j embedded in Java > - > > neo4j-community-2.3.0 > > > Graph Size: > > - 15M Nodes - with properties: a name/string; some floating point > values > - 170M Relationships - with properties: 5 floating point values > - approximately 15G of data > > > Algorithm Intent: > > - Fetch every source node in turn (all 15M), its out going > relationships and connecting destination nodes > - Calculate some statistics and parameters based from the properties > in the source and destination nodes and their connecting relationships > - For every outgoing relationship update and write back the 5 floating > point properties > > Implementation & Runtime Details: > > - Using 8 threads in a thread pool to queue up the algorithm in batches > - Batch sizes of 50,000 nodes > - tx.success is therefore posted every 50k iterations of the algorithm > after touching 50k sources nodes and about 500k destination nodes and 500k > relationships (~1000k objects + properties) > - JVM params: -Xms16000m -Xmx24000m -XX:NewRatio=1 -XX:+UseG1G > - neo4j-properties - everything commented out including - > #dbms.pagecache.memory=10g > > > > > -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
