ah wow, so looks like i probably have a lot more optimization i can still squeeze out of neo4j. michael, i am importing on my laptop, which has a ton of resources available. can you suggest some neo4j configuration settings which do not care about other services running on the system, but can be used to give as much resources as possible to neo4j? thanks
On Mon, Feb 24, 2014 at 4:42 PM, Michael Hunger < [email protected]> wrote: > With the batch-inserter you should get up to 1M nodes per second and > depending on your memory mapping (mmio) settings for the rel-file between > 100k and 500k rels/second. > > Not doing index lookups that is. > > Michael > > Am 24.02.2014 um 23:08 schrieb Javad Karabi <[email protected]>: > > Sotiris Beis, check out my project: > github.com/karabijavad/cadet > an example is at: > https://github.com/karabijavad/congress-graph > > so, for example, with cadet: > > db = Cadet::BatchInserter::Session.open("neo4j-community-2.0.1/data/graph.db") > > > db.constraint :Legislator, :name > > l = db.get_node(:Legislator, :thomas_id, leg["id"]["thomas"].to_i) > gender = db.get_node(:Gender, :name, leg["bio"]["gender"]) > l.outgoing(:gender) << gender > > db.close > > > i implement an index in ruby, so you can still find nodes based on > label/key/value. > > personally, i get about 2k rows a second for importing my csv data, where > each row can then create up to 10 other nodes and 10 other rels. > > hope this helps > > On Monday, February 24, 2014 8:53:44 AM UTC-6, Sotiris Beis wrote: >> >> I don't want to insert my data into batch mode, because the needs of the >> experiment I want to conduct, as I have explained. >> >> On 02/24/2014 04:51 PM, Michael Hunger wrote: >> >> Just DON'T use unforced() which is anyway not suited and thought for live >> usage. >> >> Create sensible batches of operations that you execute at once, that's >> the best solution for now. >> >> Michael >> >> Am 24.02.2014 um 15:47 schrieb Sotiris Beis <[email protected]>: >> >> if i choose this value keep_logical_logs to false is the same as >> this Transaction tx = ((GraphDatabaseAPI)neo4jGraph).tx().unforced().begin(); >> ? >> Τη Δευτέρα, 3 Φεβρουαρίου 2014 6:41:18 μ.μ. UTC+2, ο χρήστης Michael >> Hunger έγραψε: >> >>> Why do you do individual inserts when you have blocks of data? >>> >>> You can often aggregate events on the application level to be inserted >>> as a bigger batch. >>> see: http://maxdemarzi.com/2013/09/05/scaling-writes/ >>> >>> Otherwise you can also release the force-write-to-log constraint and >>> use >>> >>> Transaction tx = ((GraphDatabaseAPI)neo4jGraph) >>> .tx().unforced().begin(); >>> >>> Instead of neo4jGraph.beginTx(); >>> >>> >>> Am 03.02.2014 um 15:40 schrieb Sotiris Beis <[email protected]>: >>> >>> In addition to my previous question is there any suggestion to improve >>> the performance of the following code. I use this to simulate the needed >>> time to create a graph if the graph is created by single insertions >>> (incrementally not batch). So I commit every transaction and I measure the >>> time that a block needs to be inserted. A block consists of a 1,000 nodes >>> and their edges. Here is the code: >>> >>> import java.io.BufferedReader; >>> import java.io.FileInputStream; >>> import java.io.IOException; >>> import java.io.InputStreamReader; >>> import java.util.ArrayList; >>> import java.util.List; >>> >>> import org.neo4j.graphdb.GraphDatabaseService; >>> import org.neo4j.graphdb.Node; >>> import org.neo4j.graphdb.RelationshipType; >>> import org.neo4j.graphdb.Transaction; >>> import org.neo4j.graphdb.factory.GraphDatabaseFactory; >>> import org.neo4j.graphdb.factory.GraphDatabaseSetting; >>> import org.neo4j.graphdb.index.Index; >>> >>> public class Neo4jSingleInsertion implements Insertion { >>> >>> public static String INSERTION_TIMES_OUTPUT_PATH = >>> "data/neo4j.insertion.times"; >>> >>> private static int count; >>> >>> private GraphDatabaseService neo4jGraph = null; >>> private Index<Node> nodeIndex = null; >>> >>> private static enum RelTypes implements RelationshipType { >>> SIMILAR >>> } >>> >>> public static void main(String args[]) { >>> Neo4jSingleInsertion test = new Neo4jSingleInsertion(); >>> test.startup("data/neo4j"); >>> test.createGraph("data/enronEdges.txt"); >>> test.shutdown(); >>> } >>> >>> public void startup(String neo4jDBDir) { >>> System.out.println("The Neo4j database is now starting . . . ." >>> ); >>> neo4jGraph = new GraphDatabaseFactory().newEmbeddedDatabase( >>> neo4jDBDir); >>> nodeIndex = neo4jGraph.index().forNodes("nodes"); >>> } >>> >>> public void shutdown() { >>> System.out.println("The Neo4j database is now shuting down . . >>> . ."); >>> if(neo4jGraph != null) { >>> neo4jGraph.shutdown(); >>> nodeIndex = null; >>> } >>> } >>> >>> public void createGraph(String datasetDir) { >>> count++; >>> System.out.println("Incrementally creating the Neo4j database . >>> . . ."); >>> List<Double> insertionTimes = new ArrayList<Double>(); >>> try { >>> BufferedReader reader = new BufferedReader(new >>> InputStreamReader(new FileInputStream(datasetDir))); >>> String line; >>> int nodesCounter = 0; >>> int lineCounter = 1; >>> Transaction tx = null; >>> long start = System.currentTimeMillis(); >>> long duration; >>> while((line = reader.readLine()) != null) { >>> if(lineCounter > 4) { >>> String[] parts = line.split("\t"); >>> >>> Node srcNode = nodeIndex.get("nodeId", parts[0]). >>> getSingle(); >>> if(srcNode == null) { >>> tx = neo4jGraph.beginTx(); >>> srcNode = neo4jGraph.createNode(); >>> srcNode.setProperty("nodeId", parts[0]); >>> nodeIndex.add(srcNode, "nodeId", parts[0]); >>> tx.success(); >>> tx.finish(); >>> nodesCounter++; >>> } >>> >>> if(nodesCounter == 1000) { >>> duration = System.currentTimeMillis() - start; >>> insertionTimes.add((double) duration); >>> nodesCounter = 0; >>> start = System.currentTimeMillis(); >>> } >>> >>> Node dstNode = nodeIndex.get("nodeId", parts[1]). >>> getSingle(); >>> if(dstNode == null) { >>> tx = neo4jGraph.beginTx(); >>> dstNode = neo4jGraph.createNode(); >>> dstNode.setProperty("nodeId", parts[1]); >>> nodeIndex.add(dstNode, "nodeId", parts[1]); >>> tx.success(); >>> tx.finish(); >>> nodesCounter++; >>> } >>> >>> tx = neo4jGraph.beginTx(); >>> srcNode.createRelationshipTo(dstNode, RelTypes. >>> SIMILAR); >>> tx.success(); >>> tx.finish(); >>> >>> if(nodesCounter == 1000) { >>> duration = System.currentTimeMillis() - start; >>> insertionTimes.add((double) duration); >>> nodesCounter = 0; >>> start = System.currentTimeMillis(); >>> } >>> } >>> lineCounter++; >>> } >>> duration = System.currentTimeMillis() - start; >>> insertionTimes.add((double) duration); >>> reader.close(); >>> } >>> catch (IOException e) { >>> e.printStackTrace(); >>> } >>> Utils utils = new Utils(); >>> utils.writeTimes(insertionTimes, Neo4jSingleInsertion.INSERTION >>> _TIMES_OUTPUT_PATH+"."+count); >>> } >>> >>> } >>> >>> Thanks, >>> Sotiris >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Neo4j" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> For more options, visit https://groups.google.com/groups/opt_out. >>> >>> >>> >> -- >> You received this message because you are subscribed to the Google Groups >> "Neo4j" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> -- >> You received this message because you are subscribed to a topic in the >> Google Groups "Neo4j" group. >> To unsubscribe from this topic, visit https://groups.google.com/d/ >> topic/neo4j/uZqRgCBc9lg/unsubscribe. >> To unsubscribe from this group and all its topics, send an email to >> [email protected]. >> For more options, visit https://groups.google.com/groups/opt_out. >> >> >> > -- > You received this message because you are subscribed to the Google Groups > "Neo4j" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > > For more options, visit https://groups.google.com/groups/opt_out. > > > -- > You received this message because you are subscribed to a topic in the > Google Groups "Neo4j" group. > To unsubscribe from this topic, visit > https://groups.google.com/d/topic/neo4j/uZqRgCBc9lg/unsubscribe. > To unsubscribe from this group and all its topics, send an email to > [email protected]. > For more options, visit https://groups.google.com/groups/opt_out. > -- You received this message because you are subscribed to the Google Groups "Neo4j" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/groups/opt_out.
