Check with profile if there is an eager pipe in there If you use the shell with a running server can you show me again neo4-wrapper.conf ?
I ping you on skype Von meinem iPhone gesendet > Am 23.09.2014 um 11:03 schrieb 'Curtis Mosters' via Neo4j > <[email protected]>: > > I don't know but it's really buggy. The Shell also seems to buffer my batch > files. Needed to restartet die Server and recreate the database before he > took the new content. > > Anyways I will try this one now: > > USING PERIODIC COMMIT 4000 > LOAD CSV WITH HEADERS FROM "file:///C:/data/tls202_part01.txt" AS csvLine > WITH csvLine > MATCH (appln:Appln) > WHERE appln.ID = toInt(csvLine.appln_id) > CREATE (appln)-[:HAS_TITLE]->(title:Title { title:csvLine.appln_title }); > > I think this is the best solution for my case. No double CREATE. and works at > least with a small LIMIT =/ > > Let's see how is goes with Big Data! > > Am Dienstag, 23. September 2014 09:25:16 UTC+2 schrieb Curtis Mosters: >> >> I think I have the reason. >> >> Somehow my Neo4j does not want to index anything anymore. Look this: >> >> Define Index... >> +--------------------------------------------+ >> | No data returned, and nothing was changed. | >> +--------------------------------------------+ >> 1341 ms >> +--------------------------------------------+ >> | No data returned, and nothing was changed. | >> +--------------------------------------------+ >> 0 ms >> +--------------------------------------------+ >> | No data returned, and nothing was changed. | >> +--------------------------------------------+ >> 15 ms >> +--------------------------------------------+ >> | No data returned, and nothing was changed. | >> +--------------------------------------------+ >> 0 ms >> Press any key to continue . . . >> >> >> >> After the try to add 4 indexes. Very strange. I can remember that it worked >> for me in 2.1.3. >> >> And before you ask. I did not change anything in the config. It's a fresh >> 2.1.4 that you gave me here. Database created when server started. So yeah >> that should be the reason for my issues. >> But the strange thing is, even with LIMIT 1 he never finishes. I think I >> have some real trouble. >> >> When will you be available today? >> >> If you want to add me on skype it is "kwoxer" (gelbe Glühbirne) >> >> Bd >> >> Am Montag, 22. September 2014 15:56:30 UTC+2 schrieb Curtis Mosters: >> Alright I splitted it now as I said. >> >> The first part of simply inserting the Titles is like the Appln's I already >> have in there. >> >> So with >> >> USING PERIODIC COMMIT 4000 >> LOAD CSV WITH HEADERS FROM "file:///C:/data/tls202_part01.txt" AS csvLine >> WITH csvLine LIMIT 1000 >> MATCH (appln:Appln {ID: csvLine.appln_id}) >> MATCH (title:Title {title: csvLine.appln_title}) >> CREATE (appln)-[:HAS_TITLE]->(title); >> >> it's already needing minutes. That's surely buggy. I also checked the >> indexes: >> >> CREATE INDEX ON :Appln(ID); >> CREATE INDEX ON :Title(title); >> CREATE INDEX ON :Person(ID); >> >> It seems all fine. There must be an issue with the matches. >> >> Am Montag, 22. September 2014 14:46:02 UTC+2 schrieb Curtis Mosters: >> Alright now testing with new version and splitted Title. Thanks. >> >> Am Montag, 22. September 2014 12:16:24 UTC+2 schrieb Michael Hunger: >> *sigh*, this should work: >> http://neo4j.com/artifact.php?name=neo4j-community-2.1.4-windows.zip >> >> On Mon, Sep 22, 2014 at 11:58 AM, 'Curtis Mosters' via Neo4j >> <[email protected]> wrote: >> Yeah that site seems new. And there is nowhere the Older Version thing. >> >> I think the updater just forget that section... >> >> Or can you give me the direct download link for the latest binary zip please? >> >> Am Montag, 22. September 2014 08:22:36 UTC+2 schrieb Michael Hunger: >> Can you check neo4j.com/download? >> >> Von meinem iPhone gesendet >> >> Am 22.09.2014 um 06:48 schrieb 'Curtis Mosters' via Neo4j >> <[email protected]>: >> >> Sure Michael! I'll be available then. >> >> Well but there are more things broken. And I still have the 2.1.3 version >> because I'm not able to download the 2.1.4 via >> http://www.neo4j.org/download/other_versions >> I always get the download page but not any download. >> >> So could you please fix the download and then I can try the separeted >> import. Maybe that's it already! Would be great. >> >> Am Sonntag, 21. September 2014 21:22:46 UTC+2 schrieb Michael Hunger: >> Ok, it's the two create statements in a row if you split it up it will not >> happen: >> >> // !! Eager Loading kills your import, everything is pulled from the file >> into memory after the first create! >> >> profile >> LOAD CSV >> WITH HEADERS FROM >> "https://gist.githubusercontent.com/jexp/909e92c65e715ea91062/raw/cbe617cdf551ef51111fff349f1b1826a7a52d3d/appln.txt"; >> AS csvLine >> WITH csvLine >> LIMIT 1 >> MATCH (appln:Appln) >> WHERE appln.ID = toInt(csvLine.appln_id) >> CREATE (title:Title { title:csvLine.appln_title }) >> CREATE (appln)-[:HAS_TITLE]->(title); >> >> +-------------+------+--------+---------------------+-------------------------------------------------------+ >> | Operator | Rows | DbHits | Identifiers | >> Other | >> +-------------+------+--------+---------------------+-------------------------------------------------------+ >> | EmptyResult | 0 | 0 | | >> | >> | UpdateGraph | 0 | 0 | title, UNNAMED304 | >> CreateNode; CreateRelationship | >> | Eager | 0 | 0 | | >> | >> | SchemaIndex | 0 | 1 | appln, appln | >> ToIntFunction(Property(csvLine,appln_id)); :Appln(ID) | >> | Slice | 1 | 0 | | >> { AUTOINT0} | >> | LoadCSV | 1 | 0 | csvLine | >> | >> +-------------+------+--------+---------------------+-------------------------------------------------------+ >> >> >> // Do it like this: >> >> CREATE INDEX ON :Appln(ID); >> CREATE INDEX ON :Title(title); >> >> profile >> LOAD CSV >> WITH HEADERS FROM >> "https://gist.githubusercontent.com/jexp/909e92c65e715ea91062/raw/cbe617cdf551ef51111fff349f1b1826a7a52d3d/appln.txt"; >> AS csvLine >> WITH csvLine >> LIMIT 1 >> CREATE (title:Title { title:csvLine.appln_title }); >> >> +-------------+------+--------+-------------+--------------+ >> | Operator | Rows | DbHits | Identifiers | Other | >> +-------------+------+--------+-------------+--------------+ >> | EmptyResult | 0 | 0 | | | >> | UpdateGraph | 1 | 5 | title | CreateNode | >> | Slice | 1 | 0 | | { AUTOINT0} | >> | LoadCSV | 1 | 0 | csvLine | | >> +-------------+------+--------+-------------+--------------+ >> >> profile >> LOAD CSV >> WITH HEADERS FROM >> "https://gist.githubusercontent.com/jexp/909e92c65e715ea91062/raw/cbe617cdf551ef51111fff349f1b1826a7a52d3d/appln.txt"; >> AS csvLine >> WITH csvLine >> LIMIT 1 >> MATCH (appln:Appln {ID: csvLine.appln_id}) >> MATCH (title:Title { title:csvLine.appln_title }) >> CREATE (appln)-[:HAS_TITLE]->(title); >> >> +----------------+------+--------+--------------+----------------------------------------------+ >> | Operator | Rows | DbHits | Identifiers | >> Other | >> +----------------+------+--------+--------------+----------------------------------------------+ >> | EmptyResult | 0 | 0 | | >> | >> | UpdateGraph | 0 | 0 | UNNAMED285 | >> CreateRelationship | >> | SchemaIndex(0) | 0 | 0 | title, title | >> Property(csvLine,appln_title); :Title(title) | >> | SchemaIndex(1) | 0 | 1 | appln, appln | >> Property(csvLine,appln_id); :Appln(ID) | >> | Slice | 1 | 0 | | >> { AUTOINT0} | >> | LoadCSV | 1 | 0 | csvLine | >> | >> +----------------+------+--------+--------------+----------------------------------------------+ >> >> >> On Sun, Sep 21, 2014 at 9:03 PM, Michael Hunger >> <[email protected]> wrote: >> Ah, cool, then we can do a call this week, perhaps tomorrow or on Thu. >> >> That's really weird. It shouldn't build up memory in 2.1.4 as the team >> changed the eagerness requirement for queries where we determine >> independence. >> >> Can you do a a limit 10 and prefix your query with profile? There should be >> no "Eager"-pipe in between the match and merge. >> >> >> On Sun, Sep 21, 2014 at 8:31 PM, 'Curtis Mosters' via Neo4j >> <[email protected]> wrote: >> Sure y. And testing with LIMIT 1000000 works and needs 1 min or so. >> >> So this is just a Heap Stack thingi. >> >> I'm from Germany like you ;) >> >> Am Sonntag, 21. September 2014 19:25:15 UTC+2 schrieb Michael Hunger: >> In what Timezone are you located? Perhaps someone can help you and have a >> look at your setup? via Skype >> >> But you have an index or constraint on the thing you match? >> >> CREATE INDEX ON :Appln(ID); >> >> >> >> On Sun, Sep 21, 2014 at 4:08 PM, 'Curtis Mosters' via Neo4j >> <[email protected]> wrote: >> Jup, same issue with -Xmx512M >> >> So you have any other tips? Observing the IO values, it seems to idle =/ >> >> Am Samstag, 20. September 2014 12:58:33 UTC+2 schrieb Curtis Mosters: >> Well I tried it with >> >> -Xms4096m -Xmx15000m >> >> but the import did not finish in now 16 hours. >> >> Now I try your >> >> -Xmx512M >> >> Am Freitag, 19. September 2014 16:08:42 UTC+2 schrieb Michael Hunger: >> If you specifiy it I'd keep it to a few MB (e.g. -Xmx512M) as the shell >> against a running server is just a terminal. >> It's different if when you run the shell against a on-disk directory then it >> fires up the database itself. >> >> Your memory config looks ok otherwise, I'd only swap mapped memory settings >> for nodes (to 50MB) and rels (to 500MB) >> >> I think there was one memory issue in 2.1.3 could you try it with 2.1.4 ? >> >> Michael >> >> >> On Fri, Sep 19, 2014 at 4:00 PM, 'Curtis Mosters' via Neo4j >> <[email protected]> wrote: >> Here is the complete file, well I still don't know which settings are the >> best in my case. >> >> Should I delete the Xmx thing? Is it better to not specify it? >> >> Am Donnerstag, 18. September 2014 23:14:09 UTC+2 schrieb Michael Hunger: >> If you connect to the server it uses the servers RAM >> The shell then needs much less like 512m >> >> Can you check the server's graph.db/messages.log for the root cause error? >> >> Which version? >> >> Sent from mobile device >> >> Am 18.09.2014 um 14:28 schrieb "'Curtis Mosters' via Neo4j" >> <[email protected]>: >> >> Well I was thinking that the Xmx is the heap and it should be not above 4 >> GB, so far I have learned. >> >> I currently let the import run with "COMMIT 100" so every 100 lines, and >> it's still running. >> >> But looking on the IO it seems to idle... >> >> I don't know, but could you tell me exactly where to set which setting? >> >> As I said I have 30 GB RAM. To avoid more complications let's maybe set the >> Heap 4 GB and the maximum RAM to 20 GB. >> >> And again I use the Neo4jShell for the import. It's loading a file >> containing LOAD CSV. >> >> Thanks. >> >> Am Donnerstag, 18. September 2014 13:42:34 UTC+2 schrieb Clark Richey: >> According to the screenshot you don't give it 30G of ram. You only provide >> 3G. Is that your intent? >> >> Sent from my iPhone >> >> On Sep 18, 2014, at 02:51, 'Curtis Mosters' via Neo4j >> <[email protected]> wrote: >> >> Well somehow the behaviour of the Shell is odd. >> >> I imported 76 Mio lines without any problems. But now I have started a new >> import over night and now I watched the process and it said Java Heap Space >> Error. >> >> But that cannot be. I have set the Neo4jShell.bat like this: >> >> %JAVACMD% -Xmx3G -Dstorage.diskCache.bufferSize=20000 %JAVA_OPTS% >> %EXTRA_JVM_ARGUMENTS% -classpath %CLASSPATH_PREFIX%;%CLASSPATH% >> -Dapp.name="neo4j-shell" -Dapp.repo="%REPO%" -Dbasedir="%BASEDIR%" >> org.neo4j.shell.StartClient %CMD_LINE_ARGS% >> >> That is a great ratio of heap. But yeah why did I get this error? I have 30 >> GB RAM. This happens so random that a productive import is nearly impossible. >> >> So my question is, what have I done wrong? Where do I need to set the Java >> Settings correctly? Were my edited settings active? >> >> Here the first file or let's say script i'm loading in and where this error >> happens: >> >> USING PERIODIC COMMIT 2000 >> LOAD CSV WITH HEADERS FROM "file:///C:/data/tls202_part01.txt" AS csvLine >> WITH csvLine LIMIT 1000 >> MATCH (appln:Appln) >> WHERE appln.ID = toInt(csvLine.appln_id) >> CREATE (title:Title { title:csvLine.appln_title }) >> CREATE (appln)-[:HAS_TITLE]->(title); >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Neo4j" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> -- >> You received this message because you are subscribed to the Google Groups >> "Neo4j" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> -- >> You received this message because you are subscribed to the Google Groups >> "Neo4j" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Neo4j" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Neo4j" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> >> >> -- >> You received this message because you are subscribed to the Google Groups >> "Neo4j" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/d/optout. >> -- >> You received this messa >> ... > > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > For more options, visit https://groups.google.com/d/optout. -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
