Ok, it's the two create statements in a row if you split it up it will not happen:
// *!! Eager Loading kills your import, everything is pulled from the file into memory after the first create!* profile LOAD CSV WITH HEADERS FROM " https://gist.githubusercontent.com/jexp/909e92c65e715ea91062/raw/cbe617cdf551ef51111fff349f1b1826a7a52d3d/appln.txt"; AS csvLine WITH csvLine LIMIT 1 MATCH (appln:Appln) WHERE appln.ID = toInt(csvLine.appln_id) CREATE (title:Title { title:csvLine.appln_title }) CREATE (appln)-[:HAS_TITLE]->(title); +-------------+------+--------+---------------------+-------------------------------------------------------+ | Operator | Rows | DbHits | Identifiers | Other | +-------------+------+--------+---------------------+-------------------------------------------------------+ | EmptyResult | 0 | 0 | | | | UpdateGraph | 0 | 0 | title, UNNAMED304 | CreateNode; CreateRelationship | *| Eager | 0 | 0 | | |* | SchemaIndex | 0 | 1 | appln, appln | ToIntFunction(Property(csvLine,appln_id)); :Appln(ID) | | Slice | 1 | 0 | | { AUTOINT0} | | LoadCSV | 1 | 0 | csvLine | | +-------------+------+--------+---------------------+-------------------------------------------------------+ *// Do it like this:* CREATE INDEX ON :Appln(ID); CREATE INDEX ON :Title(title); profile LOAD CSV WITH HEADERS FROM " https://gist.githubusercontent.com/jexp/909e92c65e715ea91062/raw/cbe617cdf551ef51111fff349f1b1826a7a52d3d/appln.txt"; AS csvLine WITH csvLine LIMIT 1 CREATE (title:Title { title:csvLine.appln_title }); +-------------+------+--------+-------------+--------------+ | Operator | Rows | DbHits | Identifiers | Other | +-------------+------+--------+-------------+--------------+ | EmptyResult | 0 | 0 | | | | UpdateGraph | 1 | 5 | title | CreateNode | | Slice | 1 | 0 | | { AUTOINT0} | | LoadCSV | 1 | 0 | csvLine | | +-------------+------+--------+-------------+--------------+ profile LOAD CSV WITH HEADERS FROM " https://gist.githubusercontent.com/jexp/909e92c65e715ea91062/raw/cbe617cdf551ef51111fff349f1b1826a7a52d3d/appln.txt"; AS csvLine WITH csvLine LIMIT 1 MATCH (appln:Appln {ID: csvLine.appln_id}) MATCH (title:Title { title:csvLine.appln_title }) CREATE (appln)-[:HAS_TITLE]->(title); +----------------+------+--------+--------------+----------------------------------------------+ | Operator | Rows | DbHits | Identifiers | Other | +----------------+------+--------+--------------+----------------------------------------------+ | EmptyResult | 0 | 0 | | | | UpdateGraph | 0 | 0 | UNNAMED285 | CreateRelationship | | SchemaIndex(0) | 0 | 0 | title, title | Property(csvLine,appln_title); :Title(title) | | SchemaIndex(1) | 0 | 1 | appln, appln | Property(csvLine,appln_id); :Appln(ID) | | Slice | 1 | 0 | | { AUTOINT0} | | LoadCSV | 1 | 0 | csvLine | | +----------------+------+--------+--------------+----------------------------------------------+ On Sun, Sep 21, 2014 at 9:03 PM, Michael Hunger < michael.hun...@neotechnology.com> wrote: > Ah, cool, then we can do a call this week, perhaps tomorrow or on Thu. > > That's really weird. It shouldn't build up memory in 2.1.4 as the team > changed the eagerness requirement for queries where we determine > independence. > > Can you do a a limit 10 and prefix your query with profile? There should > be no "Eager"-pipe in between the match and merge. > > > On Sun, Sep 21, 2014 at 8:31 PM, 'Curtis Mosters' via Neo4j < > neo4j@googlegroups.com> wrote: > >> Sure y. And testing with LIMIT 1000000 works and needs 1 min or so. >> >> So this is just a Heap Stack thingi. >> >> I'm from Germany like you ;) >> >> Am Sonntag, 21. September 2014 19:25:15 UTC+2 schrieb Michael Hunger: >>> >>> In what Timezone are you located? Perhaps someone can help you and have >>> a look at your setup? via Skype >>> >>> But you have an index or constraint on the thing you match? >>> >>> CREATE INDEX ON :Appln(ID); >>> >>> >>> >>> On Sun, Sep 21, 2014 at 4:08 PM, 'Curtis Mosters' via Neo4j < >>> ne...@googlegroups.com> wrote: >>> >>>> Jup, same issue with >>>> >>>> *-Xmx512M*So you have any other tips? Observing the IO values, it >>>> seems to idle =/ >>>> >>>> Am Samstag, 20. September 2014 12:58:33 UTC+2 schrieb Curtis Mosters: >>>> >>>>> Well I tried it with >>>>> >>>>> *-Xms4096m -Xmx15000m* >>>>> >>>>> but the import did not finish in now 16 hours. >>>>> >>>>> Now I try your >>>>> >>>>> *-Xmx512M* >>>>> >>>>> Am Freitag, 19. September 2014 16:08:42 UTC+2 schrieb Michael Hunger: >>>>>> >>>>>> If you specifiy it I'd keep it to a few MB (e.g. -Xmx512M) as the >>>>>> shell against a running server is just a terminal. >>>>>> It's different if when you run the shell against a on-disk directory >>>>>> then it fires up the database itself. >>>>>> >>>>>> Your memory config looks ok otherwise, I'd only swap mapped memory >>>>>> settings for nodes (to 50MB) and rels (to 500MB) >>>>>> >>>>>> I think there was one memory issue in 2.1.3 could you try it with >>>>>> 2.1.4 ? >>>>>> >>>>>> Michael >>>>>> >>>>>> >>>>>> On Fri, Sep 19, 2014 at 4:00 PM, 'Curtis Mosters' via Neo4j < >>>>>> ne...@googlegroups.com> wrote: >>>>>> >>>>>>> Here is the complete file, well I still don't know which settings >>>>>>> are the best in my case. >>>>>>> >>>>>>> Should I delete the Xmx thing? Is it better to not specify it? >>>>>>> >>>>>>> Am Donnerstag, 18. September 2014 23:14:09 UTC+2 schrieb Michael >>>>>>> Hunger: >>>>>>>> >>>>>>>> If you connect to the server it uses the servers RAM >>>>>>>> The shell then needs much less like 512m >>>>>>>> >>>>>>>> Can you check the server's graph.db/messages.log for the root cause >>>>>>>> error? >>>>>>>> >>>>>>>> Which version? >>>>>>>> >>>>>>>> Sent from mobile device >>>>>>>> >>>>>>>> Am 18.09.2014 um 14:28 schrieb "'Curtis Mosters' via Neo4j" < >>>>>>>> ne...@googlegroups.com>: >>>>>>>> >>>>>>>> Well I was thinking that the Xmx is the heap and it should be not >>>>>>>> above 4 GB, so far I have learned. >>>>>>>> >>>>>>>> I currently let the import run with "COMMIT 100" so every 100 >>>>>>>> lines, and it's still running. >>>>>>>> >>>>>>>> But looking on the IO it seems to idle... >>>>>>>> >>>>>>>> I don't know, but could you tell me exactly where to set which >>>>>>>> setting? >>>>>>>> >>>>>>>> As I said I have 30 GB RAM. To avoid more complications let's maybe >>>>>>>> set the Heap 4 GB and the maximum RAM to 20 GB. >>>>>>>> >>>>>>>> And again I use the Neo4jShell for the import. It's loading a file >>>>>>>> containing LOAD CSV. >>>>>>>> >>>>>>>> Thanks. >>>>>>>> >>>>>>>> Am Donnerstag, 18. September 2014 13:42:34 UTC+2 schrieb Clark >>>>>>>> Richey: >>>>>>>>> >>>>>>>>> According to the screenshot you don't give it 30G of ram. You only >>>>>>>>> provide 3G. Is that your intent? >>>>>>>>> >>>>>>>>> Sent from my iPhone >>>>>>>>> >>>>>>>>> On Sep 18, 2014, at 02:51, 'Curtis Mosters' via Neo4j < >>>>>>>>> ne...@googlegroups.com> wrote: >>>>>>>>> >>>>>>>>> Well somehow the behaviour of the Shell is odd. >>>>>>>>> >>>>>>>>> I imported 76 Mio lines without any problems. But now I have >>>>>>>>> started a new import over night and now I watched the process and it >>>>>>>>> said >>>>>>>>> Java Heap Space Error. >>>>>>>>> >>>>>>>>> But that cannot be. I have set the Neo4jShell.bat like this: >>>>>>>>> >>>>>>>>> %JAVACMD% -Xmx3G *-Dstorage.diskCache.bufferSize=20000* %JAVA_OPTS >>>>>>>>> % %EXTRA_JVM_ARGUMENTS% -classpath %CLASSPATH_PREFIX%;%CLASSPATH% >>>>>>>>> -Dapp.name="neo4j-shell" -Dapp.repo="%REPO%" -Dbasedir="%BASEDIR%" >>>>>>>>> org.neo4j.shell.StartClient %CMD_LINE_ARGS% >>>>>>>>> >>>>>>>>> That is a great ratio of heap. But yeah why did I get this error? >>>>>>>>> I have 30 GB RAM. This happens so random that a productive import is >>>>>>>>> nearly >>>>>>>>> impossible. >>>>>>>>> >>>>>>>>> So my question is, what have I done wrong? Where do I need to set >>>>>>>>> the Java Settings correctly? Were my edited settings active? >>>>>>>>> >>>>>>>>> Here the first file or let's say script i'm loading in and where >>>>>>>>> this error happens: >>>>>>>>> >>>>>>>>> USING PERIODIC COMMIT 2000 >>>>>>>>> LOAD CSV WITH HEADERS FROM "file:///C:/data/tls202_part01.txt" AS >>>>>>>>> csvLine WITH csvLine LIMIT 1000 >>>>>>>>> MATCH (appln:Appln) >>>>>>>>> WHERE appln.ID = toInt(csvLine.appln_id) >>>>>>>>> CREATE (title:Title { title:csvLine.appln_title }) >>>>>>>>> CREATE (appln)-[:HAS_TITLE]->(title); >>>>>>>>> >>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "Neo4j" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to neo4j+un...@googlegroups.com. >>>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>>> >>>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "Neo4j" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to neo4j+un...@googlegroups.com. >>>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>>> >>>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "Neo4j" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to neo4j+un...@googlegroups.com. >>>>>>> For more options, visit https://groups.google.com/d/optout. >>>>>>> >>>>>> >>>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "Neo4j" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to neo4j+un...@googlegroups.com. >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "Neo4j" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to neo4j+unsubscr...@googlegroups.com. >> For more options, visit https://groups.google.com/d/optout. >> > > -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to neo4j+unsubscr...@googlegroups.com. For more options, visit https://groups.google.com/d/optout.
