Re: [Neo4j] OSGi and Neo4j - superbundle approach
Hi Peter, Thanks for initiating this work on OSGi bundle. I've forked you repository and tried to make is work using this sample activator [1]. My goal is to make my activator work when deployed on Felix with the neo4j superbundle. First of all, I had troubles to mvn clean install the bundle module. maven-bundle-plugin is complaining about bundle.namespace property not set. I've fixed it. Then when deploying it on Felix I had an error saying that javax.transaction couldn't be imported. Because this package is already included in superbundle, I've removed it explicitally from the Import-Package directive. You can see the changes here [2]. Then I tried to make my bundle activated using felix and the following bundles installed : g! lb START LEVEL 1 ID|State |Level|Name 0|Active |0|System Bundle (3.2.2) 1|Active |1|Neo4j OSGi default bundle (0.1.0.SNAPSHOT) 2|Active |1|Apache Felix Bundle Repository (1.6.2) 3|Active |1|Apache Felix Gogo Command (0.8.0) 4|Active |1|Apache Felix Gogo Runtime (0.8.0) 5|Active |1|Apache Felix Gogo Shell (0.8.0) 6|Resolved |1|EscapeK - Neo4j OSGi how-to - Embedded client (0.0.1.SNAPSHOT) There are only felix core, neo4j superbundle and my test bundle. So when starting felix I get the following trace : Welcome to Apache Felix Gogo g! Opening database in embedded mode: 17 juil. 2011 14:57:08 org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog doInternalRecovery INFO: Non clean shutdown detected on log [graphdb/nioneo_logical.log.1]. Recovery started ... OK Populating it ... OK Hello, brave Neo4j world! ERROR: Bundle org.escapek.neo4j-osgi-howto.embedded-client [6] Error starting file:/Users/nico/Dev/felix-framework-3.2.2/bundle/org.escapek.neo4j-osgi-howto.embedded-client_0.0.1-SNAPSHOT.jar (org.osgi.framework.BundleException: Activator start error in bundle org.escapek.neo4j-osgi-howto.embedded-client [6].) java.lang.IllegalArgumentException: No index provider 'lucene' found at org.neo4j.kernel.IndexManagerImpl.getIndexProvider(IndexManagerImpl.java:76) at org.neo4j.kernel.IndexManagerImpl.findIndexConfig(IndexManagerImpl.java:116) at org.neo4j.kernel.IndexManagerImpl.getOrCreateIndexConfig(IndexManagerImpl.java:178) at org.neo4j.kernel.IndexManagerImpl.getOrCreateNodeIndex(IndexManagerImpl.java:267) at org.neo4j.kernel.IndexManagerImpl.forNodes(IndexManagerImpl.java:255) at org.neo4j.kernel.IndexManagerImpl.forNodes(IndexManagerImpl.java:249) at org.escapek.neo4jOsgiHowto.embeddedClient.EmbeddedClientActivator.start(EmbeddedClientActivator.java:65) at org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:629) at org.apache.felix.framework.Felix.activateBundle(Felix.java:1904) at org.apache.felix.framework.Felix.startBundle(Felix.java:1822) at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1192) at org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:266) at java.lang.Thread.run(Thread.java:680) So everything works fine (open database, populate it and register service) until the activator tries to register IndexProvider service. I then get the previous exception which reminds me of something ... May be you can have a look at the stack trace to look for the error cause. NJ [1] - https://github.com/njouanin/neo4j-osgi-howto/blob/master/embedded-client/src/main/java/org/escapek/neo4jOsgiHowto/embeddedClient/EmbeddedClientActivator.java [2] - https://github.com/njouanin/neo4j-osgi/commit/4ec3eaf58b514d806fde3e48dd942c00fdfddf70 Le 5 juil. 2011 à 22:41, Peter Neubauer a écrit : Hi all, I just pushed an initial working bundle-build to https://github.com/neo4j/neo4j-osgi. You can see the working test in https://github.com/neo4j/neo4j-osgi/blob/master/examples/src/test/java/org/neo4j/examples/osgi/OSGiTest.java that waits for the Activator to instantiate and register a GraphDatabaseService and a LuceneIndex. This should be tweaked, but maybe you can use the bundle project https://github.com/neo4j/neo4j-osgi/blob/master/bundle/pom.xml , which is just one pom.xml, as a blueprint for your OSGi environment Neo4j bundle. Right now JTA and others are drawn into the bundle, which might be too much, but I think you get the picture. This works even with the Java Service Loader architecture. Feel free to fork and tune the bundles exports, and add more tests to it. WDYT? Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/- Öresund - Innovation happens HERE. http://www.thoughtmade.com -
[Neo4j] How can you have transactions with Lucene?
I like Neo4j and am interested in using it in a number of projects (once I figure out a good way to make Neo play nice in Glassfish) ... but I am confused about something. I apologize if this is a stupid question. My understanding is that Lucene does *not* support transactions. Supposedly that lack of transaction support was a prime motivator for the Compass project, and is a selling point for the Lucene competitor, Sphinx. So, how can Neo4j provide real transaction support if it relies on Lucene? I am confused. Thanks -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/How-can-you-have-transactions-with-Lucene-tp3176918p3176918.html Sent from the Neo4J Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Officially supported mechanism for J2EE
I really like Neo, but am concerned that there is so little official company support for J2EE. I want to use Neo as a standard data source in J2EE, with container managed transactions, etc. I don't want to use the REST interface - I want a standard data source interface. There does not appear to be an officially provided mechanism to use Neo as a data source in J2EE. I've seen a couple of people post their code to help someone get started, but shouldn't Neo provide a real official sanctioned mechanism? P.S. I don't want this thread to turn into a religious war over REST. If you like REST and want to defend it then please start a thread for that purpose. Please just accept that I don't want to use the REST interface - I want a standard DB interface for my DB. -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/Officially-supported-mechanism-for-J2EE-tp3176938p3176938.html Sent from the Neo4J Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] How can you have transactions with Lucene?
Neo4j wraps Lucene in a thin layer which makes it transactional. It's fully ACID and can participate in the same transactions as neo4j operations. It's modeled as its own data source in an XA environment (the neo4j graph is another XA data source) with a logical log for commands so that it's safe for crashes and will participate in the recovery process just like any other XA data source, f.ex. the neo4j graph. 2011/7/17 eialbur ronalb...@gmail.com I like Neo4j and am interested in using it in a number of projects (once I figure out a good way to make Neo play nice in Glassfish) ... but I am confused about something. I apologize if this is a stupid question. My understanding is that Lucene does *not* support transactions. Supposedly that lack of transaction support was a prime motivator for the Compass project, and is a selling point for the Lucene competitor, Sphinx. So, how can Neo4j provide real transaction support if it relies on Lucene? I am confused. Thanks -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/How-can-you-have-transactions-with-Lucene-tp3176918p3176918.html Sent from the Neo4J Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Index returns all nodes for value=undefined!
No, there's no such undefined constant which will return everything. It definitely sounds like a JavaScript issue, and that it turns nulls into undefined. And perhaps your code which tries to index null values. 2011/7/16 Aseem Kishore aseem.kish...@gmail.com We're seeing this crazy behavior where our exact index is returning all nodes if we pass a lookup value of undefined! E.g.: http://localhost:7474/db/data/index/node/nodes/username/undefined Is this a bug in Neo4j / REST API? Or is this a symptom of some bug we have in our own code? It could very well be our code, since we're using JS, which turns undefined values into the string undefined if you try to add it to a string (definitely a wtfjs candidate), but I wanted to see if others are seeing this in Neo4j too! Thanks, Aseem ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] How can you have transactions with Lucene?
Sorry, but how does this thin layer control how Lucene interacts with the disk (since Lucene isn't transactional). Does it somehow buffer the disk I/O? Does it coerce Lucene to write to a different file and then update the real file at the end of the transaction? I'm not trying to be difficult, but I am interested in Lucene for my project as well, and need it to behave properly regarding transactions. -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/How-can-you-have-transactions-with-Lucene-tp3176918p3176951.html Sent from the Neo4J Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] NoSuchElementException
get/query methods are only guaranteed to return stuff added before the latest call to flush() (it's all there in the javadocs). I think that's why you probably get stale results from get/query. flush(), or at least the first get/query after a flush() is expensive so try to minimize the number of calls to flush(). Do this by having different steps in your batch insertion, f.ex. one where you insert all nodes and index them... call flush() and then insert relationships. 2011/7/15 sulabh choudhury sula...@gmail.com Well while inserting the nodes I keep a check batchInserter.nodeExists(node1) so I would guess a node would not be duplicated. Otherwise during a traversal I guess duplicate nodes. IIs there a way I can look into the BtachInsert data before I flush so that I can make sure no duplicate data has been inserted ? On Sat, Jun 25, 2011 at 10:49 AM, sulabh choudhury sula...@gmail.comwrote: Thank you Jim. I will wait for 1.5 and hope it resolves the issue :) On Fri, Jun 24, 2011 at 7:11 PM, sulabh choudhury sula...@gmail.comwrote: Hi, I just downloaded the neo4j-community-1.4.M04. I stumbled into the java.util.NoSuchElementException: More than one element in org.neo4j.index.impl.lucene.LuceneIndex$1@396cbd97. First element is 'Node[3]' and the second element is 'Node[2]' at org.neo4j.helpers.collection.IteratorUtil.singleOrNull(IteratorUtil.java:114) ~[working_graphGen.jar:na] at org.neo4j.index.impl.lucene.IdToEntityIterator.getSingle(IdToEntityIterator.java:88) ~[working_graphGen.jar:na] at org.neo4j.index.impl.lucene.IdToEntityIterator.getSingle(IdToEntityIterator.java:32) ~[working_graphGen.jar:na] I looked up and found that this is a bug and has been fixed. I was wondering if the fix has been incorporated in the latest Milestone or not? -- -- Thanks and Regards, Sulabh Choudhury -- -- Thanks and Regards, Sulabh Choudhury -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] How can you have transactions with Lucene?
It just waits to write stuff until the transaction committs. Application crashes in between are guarded by the logical log, because all operations are written in the prepare-phase to a logical log (before the commit starts). Neo4j doesn't do any magical tweaking of lucene at all because it isn't necessary... it's fully ACID and transactional with the graph as it is. 2011/7/17 eialbur ronalb...@gmail.com Sorry, but how does this thin layer control how Lucene interacts with the disk (since Lucene isn't transactional). Does it somehow buffer the disk I/O? Does it coerce Lucene to write to a different file and then update the real file at the end of the transaction? I'm not trying to be difficult, but I am interested in Lucene for my project as well, and need it to behave properly regarding transactions. -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/How-can-you-have-transactions-with-Lucene-tp3176918p3176951.html Sent from the Neo4J Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] How can you have transactions with Lucene?
Lucene itself says it doesn't properly support ACID or XA transaction support. (Bug 1149 - Major - New Feature - Reopened - Unresolved). -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/How-can-you-have-transactions-with-Lucene-tp3176918p3176981.html Sent from the Neo4J Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] OSGi and Neo4j - superbundle approach
Hi Nicolas, Have you been able to run https://github.com/njouanin/neo4j-osgi/blob/master/examples/src/test/java/org/neo4j/examples/osgi/OSGiTest.java properly? It actually is running on Equinox, so let me try it with Felix ... back in a moment. Also, in order to pull your changes, could you sign up as a committer, and send a mail according to http://wiki.neo4j.org/content/About_Contributor_License_Agreement ? Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/ - Öresund - Innovation happens HERE. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Sun, Jul 17, 2011 at 3:02 PM, Nicolas Jouanin nicolas.joua...@gmail.com wrote: Hi Peter, Thanks for initiating this work on OSGi bundle. I've forked you repository and tried to make is work using this sample activator [1]. My goal is to make my activator work when deployed on Felix with the neo4j superbundle. First of all, I had troubles to mvn clean install the bundle module. maven-bundle-plugin is complaining about bundle.namespace property not set. I've fixed it. Then when deploying it on Felix I had an error saying that javax.transaction couldn't be imported. Because this package is already included in superbundle, I've removed it explicitally from the Import-Package directive. You can see the changes here [2]. Then I tried to make my bundle activated using felix and the following bundles installed : g! lb START LEVEL 1 ID|State |Level|Name 0|Active | 0|System Bundle (3.2.2) 1|Active | 1|Neo4j OSGi default bundle (0.1.0.SNAPSHOT) 2|Active | 1|Apache Felix Bundle Repository (1.6.2) 3|Active | 1|Apache Felix Gogo Command (0.8.0) 4|Active | 1|Apache Felix Gogo Runtime (0.8.0) 5|Active | 1|Apache Felix Gogo Shell (0.8.0) 6|Resolved | 1|EscapeK - Neo4j OSGi how-to - Embedded client (0.0.1.SNAPSHOT) There are only felix core, neo4j superbundle and my test bundle. So when starting felix I get the following trace : Welcome to Apache Felix Gogo g! Opening database in embedded mode: 17 juil. 2011 14:57:08 org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog doInternalRecovery INFO: Non clean shutdown detected on log [graphdb/nioneo_logical.log.1]. Recovery started ... OK Populating it ... OK Hello, brave Neo4j world! ERROR: Bundle org.escapek.neo4j-osgi-howto.embedded-client [6] Error starting file:/Users/nico/Dev/felix-framework-3.2.2/bundle/org.escapek.neo4j-osgi-howto.embedded-client_0.0.1-SNAPSHOT.jar (org.osgi.framework.BundleException: Activator start error in bundle org.escapek.neo4j-osgi-howto.embedded-client [6].) java.lang.IllegalArgumentException: No index provider 'lucene' found at org.neo4j.kernel.IndexManagerImpl.getIndexProvider(IndexManagerImpl.java:76) at org.neo4j.kernel.IndexManagerImpl.findIndexConfig(IndexManagerImpl.java:116) at org.neo4j.kernel.IndexManagerImpl.getOrCreateIndexConfig(IndexManagerImpl.java:178) at org.neo4j.kernel.IndexManagerImpl.getOrCreateNodeIndex(IndexManagerImpl.java:267) at org.neo4j.kernel.IndexManagerImpl.forNodes(IndexManagerImpl.java:255) at org.neo4j.kernel.IndexManagerImpl.forNodes(IndexManagerImpl.java:249) at org.escapek.neo4jOsgiHowto.embeddedClient.EmbeddedClientActivator.start(EmbeddedClientActivator.java:65) at org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:629) at org.apache.felix.framework.Felix.activateBundle(Felix.java:1904) at org.apache.felix.framework.Felix.startBundle(Felix.java:1822) at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1192) at org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:266) at java.lang.Thread.run(Thread.java:680) So everything works fine (open database, populate it and register service) until the activator tries to register IndexProvider service. I then get the previous exception which reminds me of something ... May be you can have a look at the stack trace to look for the error cause. NJ [1] - https://github.com/njouanin/neo4j-osgi-howto/blob/master/embedded-client/src/main/java/org/escapek/neo4jOsgiHowto/embeddedClient/EmbeddedClientActivator.java [2] - https://github.com/njouanin/neo4j-osgi/commit/4ec3eaf58b514d806fde3e48dd942c00fdfddf70 Le 5 juil. 2011 à 22:41, Peter Neubauer a écrit : Hi all, I just pushed an initial working bundle-build to https://github.com/neo4j/neo4j-osgi. You can see the working test in https://github.com/neo4j/neo4j-osgi/blob/master/examples/src/test/java/org/neo4j/examples/osgi/OSGiTest.java that
Re: [Neo4j] OSGi and Neo4j - superbundle approach
Nicolas, just pushed some more initialization along your lines, and have the test running with the output: [~/code/neo/neo4j-osgi/examples] $mvn clean test [INFO] Scanning for projects... [INFO] [INFO] Building Neo4j OSGi examples [INFO]task-segment: [clean, test] [INFO] [INFO] [clean:clean {execution: default-clean}] [INFO] Deleting /Users/peterneubauer/code/neo/neo4j-osgi/examples/target [WARNING] POM for 'biz.aQute:bndlib:pom:1.43.0:test' is invalid. Its dependencies (if any) will NOT be available to the current build. [WARNING] POM for 'biz.aQute:bndlib:pom:1.15.0:test' is invalid. Its dependencies (if any) will NOT be available to the current build. [INFO] [enforcer:enforce {execution: enforce-maven}] [INFO] [license:check {execution: check-licenses}] [INFO] Checking licenses... [INFO] [resources:resources {execution: default-resources}] [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/peterneubauer/code/neo/neo4j-osgi/examples/src/main/resources [INFO] skip non existing resourceDirectory /Users/peterneubauer/code/neo/neo4j-osgi/examples/src/main/resources/META-INF [INFO] Copying 0 resource to META-INF [INFO] [compiler:compile {execution: default-compile}] [INFO] No sources to compile [INFO] [bundle:manifest {execution: bundle-manifest}] [WARNING] Warning in manifest for org.neo4j.examples:neo4j-osgi-examples:jar:0.1-SNAPSHOT : Superfluous export-package instructions: [null.*] [WARNING] Warning in manifest for org.neo4j.examples:neo4j-osgi-examples:jar:0.1-SNAPSHOT : Did not find matching referal for * [INFO] [resources:testResources {execution: default-testResources}] [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] skip non existing resourceDirectory /Users/peterneubauer/code/neo/neo4j-osgi/examples/src/test/resources [INFO] Copying 0 resource to META-INF [INFO] [compiler:testCompile {execution: default-testCompile}] [INFO] Compiling 3 source files to /Users/peterneubauer/code/neo/neo4j-osgi/examples/target/test-classes [INFO] [surefire:test {execution: default-test}] [INFO] Surefire report directory: /Users/peterneubauer/code/neo/neo4j-osgi/examples/target/surefire-reports --- T E S T S --- Running org.neo4j.examples.osgi.OSGiTest 300 [main] INFO org.ops4j.pax.exam.spi.DefaultExamSystem - Pax Exam System (Version: 2.1.0) created. Jul 17, 2011 5:52:07 PM org.ops4j.pax.url.mvn.internal.AetherBasedResolver resolve INFO: Resolved (org.ops4j.pax.exam:pax-exam-container-rbc:jar:2.1.0) as /Users/peterneubauer/.m2/repository/org/ops4j/pax/exam/pax-exam-container-rbc/2.1.0/pax-exam-container-rbc-2.1.0.jar Jul 17, 2011 5:52:07 PM org.ops4j.pax.url.mvn.internal.AetherBasedResolver resolve INFO: Resolved (org.ops4j.pax.exam:pax-exam-extender-service:jar:2.1.0) as /Users/peterneubauer/.m2/repository/org/ops4j/pax/exam/pax-exam-extender-service/2.1.0/pax-exam-extender-service-2.1.0.jar Jul 17, 2011 5:52:07 PM org.ops4j.pax.url.mvn.internal.AetherBasedResolver resolve INFO: Resolved (org.osgi:org.osgi.compendium:jar:4.2.0) as /Users/peterneubauer/.m2/repository/org/osgi/org.osgi.compendium/4.2.0/org.osgi.compendium-4.2.0.jar Jul 17, 2011 5:52:07 PM org.ops4j.pax.url.mvn.internal.AetherBasedResolver resolve INFO: Resolved (org.ops4j.pax.logging:pax-logging-api:jar:1.6.2) as /Users/peterneubauer/.m2/repository/org/ops4j/pax/logging/pax-logging-api/1.6.2/pax-logging-api-1.6.2.jar Jul 17, 2011 5:52:07 PM org.ops4j.pax.url.mvn.internal.AetherBasedResolver resolve INFO: Resolved (org.ops4j.pax.logging:pax-logging-service:jar:1.6.2) as /Users/peterneubauer/.m2/repository/org/ops4j/pax/logging/pax-logging-service/1.6.2/pax-logging-service-1.6.2.jar Jul 17, 2011 5:52:07 PM org.ops4j.pax.url.mvn.internal.AetherBasedResolver resolve INFO: Resolved (org.neo4j:neo4j-osgi-bundle:jar:0.1-SNAPSHOT) as /Users/peterneubauer/.m2/repository/org/neo4j/neo4j-osgi-bundle/0.1-SNAPSHOT/neo4j-osgi-bundle-0.1-SNAPSHOT.jar [org.ops4j.pax.exam.rbc.internal.Activator] : Name, port or host is null. So this RBC remains inactive. org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] : Enabling SLF4J API support. org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] : Enabling Jakarta Commons Logging API support. org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] : Enabling Log4J API support. org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] : Enabling Avalon Logger API support. org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator] : Enabling JULI Logger API support. [Framework Event Dispatcher] DEBUG org.ops4j.pax.logging.pax-logging-service - BundleEvent STARTED [Framework Event
[Neo4j] How to create a graph database out of a huge dataset?
Hi all, I'm new to neo4j and graph databases. To create my graph database I got two questions for you: 1. I want to create a graph database out of a huge CSV file. The problem is, that i need to index the nodes I have already created, so that I don't create duplicate nodes. My CSV file looks like this: Article A, Timestamp, Author A Article A, Timestamp, Author B Article A, Timestamp, Author C Article B, Timestamp, Author A Article B, Timestamp, Author B Article B, Timestamp, Author D As you can see I need to access nodes I have already created and connect them to the next Article. Right now I'm using the LuceneIndex, but with around 6M authors this is getting really slow. Is there any other possibility to access nodes that I have already created? BatchInserter also doesn't work, because there you can't access nodes which you have created before. 2. Is it possible to use neo4j in a distributed system? If it is possible, are there any guides or tutorials how to realize that? Thanks for your help, Stephan -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/How-to-create-a-graph-database-out-of-a-huge-dataset-tp3177076p3177076.html Sent from the Neo4J Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Django + Neo4j seamless installation error: No module named CallbackFilter
On Fri, Jul 15, 2011 at 2:33 AM, Richard Schwab endur...@gmail.com wrote: Hello, I'm having difficulty integrating Django and Neo4j. I've tried it on several different systems including Mac OSX 10.6.8 and Ubuntu 11.04. Perhaps someone can give me some feedback or suggestions? I'm good with Django but haven't played with VirtualEnv, Jython or Neo4j before. Here is the error I'm getting trying the steps at thobe.org's seamless neo4j integration in django: == Error == ValueError: Unable to configure filter 'require_debug_false': Cannot resolve 'django.utils.log.CallbackFilter': No module named CallbackFilter == Steps == Mainly using http://journal.thobe.org/2009/12/seamless-neo4j-integration-in-django.html , I'm following these steps and I apologize if these are unnecessary, I'm just trying to give more info to get to an answer quickly. 1. VirtualEnv 1. Install virtualenv ($ sudo apt-get install virtualenv) 2. Set up a virtualenv ($ virtualenv TestEnv) 3. Activate the virtualenv ($ source TestEnv/bin/activate) 2. Jython - 2.5.2 (I've also tried 2.53b and no luck) 1. Download jython (sourceforge.net) uncompress 2. Copy jython to /usr/lib, make symbolic link ($ ln -s /usr/lib/jython2.5.2/lib/jython /usr/bin/jython) 3. Neo4j - 1.4 1. Download Neo4j ($ svn co https://svn.neo4j.org/components/neo4j.py/trunk neo4j-python) 2. Install Neo4j ($ sudo jython setup.py install) 3. Tested: jython: import neo4j works 4. Django - 1.3.0 1. Download extract Django (https://www.djangoproject.com/download/ ) 2. Install Django ($ sudo jython setup.py install) 3. Tested: jython: import django works 5. Django-Jython - 1.2.0 1. Download extract Django-Jython ( http://code.google.com/p/django-jython/downloads/list) 2. Install Django-Jython ($ sudo jython setup.py install) 6. Created a Django-Admin-Jy ln ($ alias django-admin-jy=jython /usr/lib/jython2.5.2/Lib/site-packages/django/bin/django-admin.py) 7. Created a Django Project ($ django-admin-jy startproject neo4django) 8. Created a Django App ($ jython neo4django/manage.py startapp business) 9. Edited settings.py 1. Edit settings.py NEO4J_RESOURCE_URI = '/var/neo4j/neo4django' # NEO4J_RESOURCE_URI should be the path to where #you want to store the Neo4j database. NEO4J_OPTIONS = { # this is optional and can be used to specify # extra startup parameters for Neo4j, such as # the classpath to load Neo4j from. } 10. Edited business/models.py 1. from neo4j.model import django_model as model class Person(model.NodeModel): first_name = model.Property() last_name = model.Property() def __unicode__(self): return u%s %s % (self.first_name, self.last_name) class Company(model.NodeModel): name = model.Property(indexed=True) owners = model.Relationship(Person, type=model.Outgoing.OWNED_BY, related_name=owns, ) employees = model.Relationship(Person, type=model.Incoming.WORKS_AT, related_name=employer, related_single=True, # Only allow Persons to work at one Company ) def __unicode__(self): return self.name 11. Tested it out: $ jython neo4django/manage.py shell results in the error (same as written above) - ValueError: Unable to configure filter 'require_debug_false': Cannot resolve 'django.utils.log.CallbackFilter': No module named CallbackFilter So all that said if anyone has any ideas, I'd appreciate it immensely. I was wondering also if my versioning was correct, or if I should drop to some older versions of django perhaps? It doesn't seem like there's been much work on this since 2009. That is a very good suspicion. That Django has changed since I wrote that code, and that article. Trying an older version of Django might do the trick. Sadly I can't find which version of Django I used back then. However, there is no use of CallbackFilter in my code, which seems to imply that this would be a problem in Django. Looking at the Django documentation: https://docs.djangoproject.com/en/dev/topics/logging/#id3 the CallbackFilter feature seems to not be released yet. How it got into your settings.py (I'm guessing that is where 'require_debug_false' is configures) I don't know. If I had not had your complete description of everything you did, I would have guessed that you either: a) Followed the documentation of the wrong version of Django from their website (the dev version is easier to find then the ones for released versions). Or b) Used a newer version of Django to auto-generate the settings.py file, and then an older version to run the code. But since neither of those steps are in your
[Neo4j] neo4j in a distributed system
Hi all, I would like to know if it is possible to use neo4j in a distributed system. If it is possible, are there any guides or tutorials how to realize that? For my huge database it would be nice to use it in distributed system. Thanks for your help, Stephan ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Fwd: RDF Examples not working - parent-pom
Anfang der weitergeleiteten E-Mail: Von: datablend i...@datablend.be Datum: 17. Juli 2011 17:40:39 MESZ An: Michael Hunger michael.hun...@neotechnology.com Betreff: Re: [Neo4j] RDF Examples not working - parent-pom Hi Michael, Thanks for the information. Wrote a new article on the use of Tinkerpop for importing and querying RDF data: http://datablend.be/?p=554 Greetings, Davy On 14 Jul 2011, at 17:50, Michael Hunger wrote: Davy, you used the neo4j-sail project for your examples, tools and demos. Unfortunately that one is no longer under active development, so things like the parent pom issue pop up. Would you mind having a look at the much more complete and up to date sail implementation from tinkerpop to run on top of neo4j? Thanks so much and great work ! Michael Anfang der weitergeleiteten E-Mail: Von: Thomas Fritz fritz...@gmail.com Datum: 14. Juli 2011 17:33:06 MESZ An: Neo4j user discussions user@lists.neo4j.org Betreff: Re: [Neo4j] RDF Examples not working - parent-pom Antwort an: Neo4j user discussions user@lists.neo4j.org Hi, No, it is of course not in my own pom.xml - the projects pom.xml - it is in the Neo4J dependencies (rdf, sparql, sail). I think it has to be fixed from Neo4J and has to be updated on maven central? Kind regards --- Thomas FRITZ web http://fritzthomas.com twitter http://twitter.com/thomasf 2011/7/14 Michael Hunger michael.hun...@neotechnology.com: It should be directly in your pom.xml at the top, there is an xml section with the parent element. Cheers Michael If that is not in your local pom, try to pull that project from svn remove that section and build it locally. Am 14.07.2011 um 14:49 schrieb Thomas Fritz: Thanks for answering! How can i remove the parent section from the pom.xml? It is in the dependency project maven tries to resolve, right? So how and where can i edit it? Kind regards --- Thomas FRITZ web http://fritzthomas.com twitter http://twitter.com/thomasf 2011/7/14 Michael Hunger michael.hun...@neotechnology.com: You can just try to remove the parent ... /parent block, that should work. Those are just meant for being built integrated in a larger system build. Cheers Michael Am 14.07.2011 um 13:23 schrieb Thomas Fritz: Hi, Who can fix this? As i can see one of the neo4j-rdf dependencys pulls that dependency in. Or can i bypass and fix it locally until it is fixed? Thanks Kind regards --- Thomas FRITZ web http://fritzthomas.com twitter http://twitter.com/thomasf 2011/7/14 Peter Neubauer neubauer.pe...@gmail.com: Not sure but parent - pom should maybe be parent - central? /peter Sent from my phone. On Jul 14, 2011 12:23 PM, Thomas Fritz fritz...@gmail.com wrote: I have tried the code example from here: http://datablend.be/?p=411 . I downloaded the source from github - https://github.com/datablend/neo4j-sail-test . I have also tried this example: https://github.com/aduric/Neo-SPARQL-Test They both can not be build with maven. The parent-pom can not be found by maven. I am using netbeans 7 with bundled maven 3.0.3: code Building Unnamed - neo4j-sail-test:neo4j-sail-test:jar:1.0 task-segment: [clean, install] [clean:clean] Deleting directory /home/thomas/Projekte/Neo4J-Tests/neo4j-sail-test/target [resources:resources] Using platform encoding (UTF-8 actually) to copy filtered resources, i.e. build is platform dependent! Copying 1 resource snapshot org.neo4j:neo4j-rdf-sail:0.6-SNAPSHOT: checking for updates from central snapshot org.neo4j:neo4j-rdf-sail:0.6-SNAPSHOT: checking for updates from aduna-repo snapshot org.neo4j:neo4j-kernel:1.1-SNAPSHOT: checking for updates from aduna-repo snapshot org.neo4j:neo4j-kernel:1.1-SNAPSHOT: checking for updates from neo4j-snapshot-repository snapshot org.neo4j:neo4j-kernel:1.1-SNAPSHOT: checking for updates from sonatype-nexus-snapshots snapshot org.neo4j:neo4j-rdf:0.7-SNAPSHOT: checking for updates from aduna-repo snapshot org.neo4j:neo4j-rdf:0.7-SNAPSHOT: checking for updates from sonatype-nexus-snapshots snapshot org.neo4j:neo4j-meta-model:0.9-SNAPSHOT: checking for updates from aduna-repo snapshot org.neo4j:neo4j-meta-model:0.9-SNAPSHOT: checking for updates from sonatype-nexus-snapshots snapshot org.neo4j:parent-pom:7-SNAPSHOT: checking for updates from aduna-repo snapshot org.neo4j:parent-pom:7-SNAPSHOT: checking for updates from sonatype-nexus-snapshots Downloading: http://m2.neo4j.org/snapshots/org/neo4j/parent-pom/7-SNAPSHOT/parent-pom-7-20100729.145222-31.pom BUILD ERROR Error building POM (may
Re: [Neo4j] Officially supported mechanism for J2EE
Hi, do you mean a javax.sql.DataSource or an JCA adaptor ? Could you please explain your architecture and usage scenario where this would be necessary. You can use Neo4j in embedded mode inside any J2EE container without issues. If the container comes with its own XA-TM then you can integrate with that. Cheers Michael Am 17.07.2011 um 17:03 schrieb eialbur: I really like Neo, but am concerned that there is so little official company support for J2EE. I want to use Neo as a standard data source in J2EE, with container managed transactions, etc. I don't want to use the REST interface - I want a standard data source interface. There does not appear to be an officially provided mechanism to use Neo as a data source in J2EE. I've seen a couple of people post their code to help someone get started, but shouldn't Neo provide a real official sanctioned mechanism? P.S. I don't want this thread to turn into a religious war over REST. If you like REST and want to defend it then please start a thread for that purpose. Please just accept that I don't want to use the REST interface - I want a standard DB interface for my DB. -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/Officially-supported-mechanism-for-J2EE-tp3176938p3176938.html Sent from the Neo4J Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Index returns all nodes for value=undefined!
Perhaps it is also an issue of your REST-client? If the value when rendered as part of a string evaluates to the empty string the resulting URL would be the same as the one that retrieves all nodes. i.e. http://localhost:7474/db/data/index/node/nodes/username Just an idea, could you verify the REST url that is used? Am 17.07.2011 um 17:06 schrieb Mattias Persson: No, there's no such undefined constant which will return everything. It definitely sounds like a JavaScript issue, and that it turns nulls into undefined. And perhaps your code which tries to index null values. 2011/7/16 Aseem Kishore aseem.kish...@gmail.com We're seeing this crazy behavior where our exact index is returning all nodes if we pass a lookup value of undefined! E.g.: http://localhost:7474/db/data/index/node/nodes/username/undefined Is this a bug in Neo4j / REST API? Or is this a symptom of some bug we have in our own code? It could very well be our code, since we're using JS, which turns undefined values into the string undefined if you try to add it to a string (definitely a wtfjs candidate), but I wanted to see if others are seeing this in Neo4j too! Thanks, Aseem ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] How to create a graph database out of a huge dataset?
Stephan, This is a common thing when inserting data. You should be able to use lucene in both settings (6M authors is not that much). Please have a look at your heap memory settings (and in transactional mode also your memory-map settings for neo4j). For batch inserter. You can query the index after you called flush on the index. That should return a node-id which you then can use to create relationship to that looked up node. If you could share the code that you've already written we can have a look at it. The simplest approach is to just take a HashMap for the authors and their name as key. (while indexing them for later use anyway). Just make sure your heap is large enough to hold the map and the objects created during the insert. #2 can you describe your distributed system? You can run a cluster of neo4j-HA instances that each of your distributed engines connects to. (HA instances can run embedded or as Neo4j-servers depending on your needs). (see here: http://docs.neo4j.org/chunked/stable/ha.html) Cheers Michael Am 17.07.2011 um 18:31 schrieb st3ven: Hi all, I'm new to neo4j and graph databases. To create my graph database I got two questions for you: 1. I want to create a graph database out of a huge CSV file. The problem is, that i need to index the nodes I have already created, so that I don't create duplicate nodes. My CSV file looks like this: Article A, Timestamp, Author A Article A, Timestamp, Author B Article A, Timestamp, Author C Article B, Timestamp, Author A Article B, Timestamp, Author B Article B, Timestamp, Author D As you can see I need to access nodes I have already created and connect them to the next Article. Right now I'm using the LuceneIndex, but with around 6M authors this is getting really slow. Is there any other possibility to access nodes that I have already created? BatchInserter also doesn't work, because there you can't access nodes which you have created before. 2. Is it possible to use neo4j in a distributed system? If it is possible, are there any guides or tutorials how to realize that? Thanks for your help, Stephan -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/How-to-create-a-graph-database-out-of-a-huge-dataset-tp3177076p3177076.html Sent from the Neo4J Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] neo4j in a distributed system
Great Michael! can you share with us the performance evaluation results with code of the 32 billion nodes and relationships. This be a great proof for neo4js high performance. On 7/17/11, Michael Hunger michael.hun...@neotechnology.com wrote: with huge database you mean that you want to shard the graph? And how large is your database? Please note that neo4j can handle 32 billion nodes and relationships in a single instance. If your database is larger than that you have to apply some kind of domain level sharding. (See Jims article: http://jim.webber.name/2011/02/23/abe72f61-27fb-4c1b-8ce1-d0db7583497b.aspx and http://jim.webber.name/2011/02/16/3b8f4b3d-c884-4fba-ae6b-7b75a191fa22.aspx) Am 17.07.2011 um 18:50 schrieb st3...@web.de: Hi all, I would like to know if it is possible to use neo4j in a distributed system. If it is possible, are there any guides or tutorials how to realize that? For my huge database it would be nice to use it in distributed system. Thanks for your help, Stephan ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Aliabbas Petiwala M.Tech CSE ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] RDF data wrapper/triple store options for fast traversal
Hello! I'm new to Neo4j and need some advice regarding my project and how I should proceed. The purpose of the project is to find the shortest path between two things (similar to RelFinder) as quickly as possible. So far I've loaded a few million dbpedia triples using NxParser and neo4j-rdf component as well as tried neo4j's pathfinding capabilities. Everything works fine! However, I have noticed that neo4j-rdf's DenseTripleStore stores object literals and their predicates within the subject node as a set of properties. Since, above all, I'm after fast traversals I'm wondering, whether this incurs any performance penalties to pathfinding algorithms? If I were to store those literals as separate nodes, would _that_ incur penalties? Is neo4j-rdf still in development or should I try out TinkerPop's Blueprints (seems very popular lately!)? Whilst I like the features these high-level wrappers provide, I'm really only using pathfinding and indexing. Should I write my own lightweight wrapper? But in the end is it really worth it, or should I just use existing wrappers? Thank you! ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] RDF data wrapper/triple store options for fast traversal
Jan, I would suggest to use the Tinkerpop RDF wrapper, then examine the resulting graph in neo4j, and se if raw traversals can find the relationships for you, After all, you can always drop down to the raw graph even while using the RDF implementation. That is one of the points of Neo4j dealing with raw graphs. Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/ - Öresund - Innovation happens HERE. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Sun, Jul 17, 2011 at 8:20 PM, Jan Bezget jan.bez...@uni-mb.si wrote: Hello! I'm new to Neo4j and need some advice regarding my project and how I should proceed. The purpose of the project is to find the shortest path between two things (similar to RelFinder) as quickly as possible. So far I've loaded a few million dbpedia triples using NxParser and neo4j-rdf component as well as tried neo4j's pathfinding capabilities. Everything works fine! However, I have noticed that neo4j-rdf's DenseTripleStore stores object literals and their predicates within the subject node as a set of properties. Since, above all, I'm after fast traversals I'm wondering, whether this incurs any performance penalties to pathfinding algorithms? If I were to store those literals as separate nodes, would _that_ incur penalties? Is neo4j-rdf still in development or should I try out TinkerPop's Blueprints (seems very popular lately!)? Whilst I like the features these high-level wrappers provide, I'm really only using pathfinding and indexing. Should I write my own lightweight wrapper? But in the end is it really worth it, or should I just use existing wrappers? Thank you! ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] How to create a graph database out of a huge dataset?
Hi, thanks for your fast answer. Right now I'm using lucene for 6M authors, but my whole dataset consists of nearly 25M authors. Can i use lucene there also, because I think this getting really slow to check if a user already exists. How can I change my heap memory settings and my memory-map settings, cause I'm using the transactional mode? Cause I think with 25M authors I will get a OutOfMemory Exception. Here is my code that I have already written so far: import java.io.BufferedReader; import java.io.FileReader; import java.io.IOException; import org.neo4j.graphdb.GraphDatabaseService; import org.neo4j.graphdb.Node; import org.neo4j.graphdb.Relationship; import org.neo4j.graphdb.Transaction; import org.neo4j.graphdb.index.Index; import org.neo4j.graphdb.index.IndexHits; import org.neo4j.graphdb.index.IndexManager; import org.neo4j.kernel.EmbeddedGraphDatabase; public class WikiGraphRegUser { /** * @param args */ public static void main(String[] args) throws IOException { BufferedReader bf = new BufferedReader(new FileReader( E:/wiki0.csv)); WikiGraphRegUser wgru = new WikiGraphRegUser(); wgru.createGraphDatabase(bf); } private String articleName = ; private GraphDatabaseService db; private IndexManager index; private IndexNode authorList; private int transactionCounter = 0; private Node article; private boolean isFirstAuthor = false; private Node author; private Relationship relationship; private int node; private void createGraphDatabase(BufferedReader bf) { db = new EmbeddedGraphDatabase(target/db); index = db.index(); authorList = index.forNodes(Author); String zeile; Transaction tx = db.beginTx(); try { // reads lines of CSV-file while ((zeile = bf.readLine()) != null) { if (transactionCounter++ % 5 == 0) { tx.success(); tx.finish(); tx = db.beginTx(); } // String[] looks like this: Article%;% Timestamp%;% Author String[] artikelinfo = zeile.split(%;% ); if (artikelinfo.length != 3) { System.out.println(ERROR: check CSV); for (int i = 0; i artikelinfo.length; i++) { System.out.println(artikelinfo[i]); } return; } if (articleName == ) { // create Article and connect with ReferenceNode article = createArticle(artikelinfo[0], db.getReferenceNode(), MyRelationshipTypes.ARTICLE); articleName = artikelinfo[0]; isFirstAuthor = true; } else if (!articleName.equals(artikelinfo[0])) { // create Article and connect with ReferenceNode article = createArticle(artikelinfo[0], db.getReferenceNode(), MyRelationshipTypes.ARTICLE); articleName = artikelinfo[0]; isFirstAuthor = true; } // checks if author already exists IndexHitsNode hits = authorList.get(Author, artikelinfo[2]); // if new author if (hits.size() == 0) { if (isFirstAuthor) { // creates author and connects him with an article author = createAndConnectNode(artikelinfo[2], article, MyRelationshipTypes.WROTE, artikelinfo[1]); isFirstAuthor = false; } else { author = createAndConnectNode(artikelinfo[2], article, MyRelationshipTypes.EDIT, artikelinfo[1]); } }
Re: [Neo4j] OSGi and Neo4j - superbundle approach
I've sent the CLA signup. The OSGiTest is working fine for me too. Problems come when taking the felix framework out of the box and deploying manually neo4j-osgi bundle with another bundle containing the same activator class. I made sure to deploy same bundles but I get java.lang.IllegalArgumentException: No index provider 'lucene' found when registring the IndexService. I think i miss a bundle which is created automatically by pax during the test case. May be lucene is missing. What do you think Le 17 juil. 2011 à 17:33, Peter Neubauer a écrit : Hi Nicolas, Have you been able to run https://github.com/njouanin/neo4j-osgi/blob/master/examples/src/test/java/org/neo4j/examples/osgi/OSGiTest.java properly? It actually is running on Equinox, so let me try it with Felix ... back in a moment. Also, in order to pull your changes, could you sign up as a committer, and send a mail according to http://wiki.neo4j.org/content/About_Contributor_License_Agreement ? Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/- Öresund - Innovation happens HERE. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Sun, Jul 17, 2011 at 3:02 PM, Nicolas Jouanin nicolas.joua...@gmail.com wrote: Hi Peter, Thanks for initiating this work on OSGi bundle. I've forked you repository and tried to make is work using this sample activator [1]. My goal is to make my activator work when deployed on Felix with the neo4j superbundle. First of all, I had troubles to mvn clean install the bundle module. maven-bundle-plugin is complaining about bundle.namespace property not set. I've fixed it. Then when deploying it on Felix I had an error saying that javax.transaction couldn't be imported. Because this package is already included in superbundle, I've removed it explicitally from the Import-Package directive. You can see the changes here [2]. Then I tried to make my bundle activated using felix and the following bundles installed : g! lb START LEVEL 1 ID|State |Level|Name 0|Active |0|System Bundle (3.2.2) 1|Active |1|Neo4j OSGi default bundle (0.1.0.SNAPSHOT) 2|Active |1|Apache Felix Bundle Repository (1.6.2) 3|Active |1|Apache Felix Gogo Command (0.8.0) 4|Active |1|Apache Felix Gogo Runtime (0.8.0) 5|Active |1|Apache Felix Gogo Shell (0.8.0) 6|Resolved |1|EscapeK - Neo4j OSGi how-to - Embedded client (0.0.1.SNAPSHOT) There are only felix core, neo4j superbundle and my test bundle. So when starting felix I get the following trace : Welcome to Apache Felix Gogo g! Opening database in embedded mode: 17 juil. 2011 14:57:08 org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog doInternalRecovery INFO: Non clean shutdown detected on log [graphdb/nioneo_logical.log.1]. Recovery started ... OK Populating it ... OK Hello, brave Neo4j world! ERROR: Bundle org.escapek.neo4j-osgi-howto.embedded-client [6] Error starting file:/Users/nico/Dev/felix-framework-3.2.2/bundle/org.escapek.neo4j-osgi-howto.embedded-client_0.0.1-SNAPSHOT.jar (org.osgi.framework.BundleException: Activator start error in bundle org.escapek.neo4j-osgi-howto.embedded-client [6].) java.lang.IllegalArgumentException: No index provider 'lucene' found at org.neo4j.kernel.IndexManagerImpl.getIndexProvider(IndexManagerImpl.java:76) at org.neo4j.kernel.IndexManagerImpl.findIndexConfig(IndexManagerImpl.java:116) at org.neo4j.kernel.IndexManagerImpl.getOrCreateIndexConfig(IndexManagerImpl.java:178) at org.neo4j.kernel.IndexManagerImpl.getOrCreateNodeIndex(IndexManagerImpl.java:267) at org.neo4j.kernel.IndexManagerImpl.forNodes(IndexManagerImpl.java:255) at org.neo4j.kernel.IndexManagerImpl.forNodes(IndexManagerImpl.java:249) at org.escapek.neo4jOsgiHowto.embeddedClient.EmbeddedClientActivator.start(EmbeddedClientActivator.java:65) at org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:629) at org.apache.felix.framework.Felix.activateBundle(Felix.java:1904) at org.apache.felix.framework.Felix.startBundle(Felix.java:1822) at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1192) at org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:266) at java.lang.Thread.run(Thread.java:680) So everything works fine (open database, populate it and register service) until the activator tries to register IndexProvider service. I then get the previous exception which reminds me of something ... May be you can have a look at the stack trace to
Re: [Neo4j] How can you have transactions with Lucene?
Exactly, that's what the neo4j integration layer adds to it. 2011/7/17 eialbur ronalb...@gmail.com Lucene itself says it doesn't properly support ACID or XA transaction support. (Bug 1149 - Major - New Feature - Reopened - Unresolved). -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/How-can-you-have-transactions-with-Lucene-tp3176918p3176981.html Sent from the Neo4J Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] use one index per key or one index for all keys?
Hi Boris, If you store them both in one index you get the benefit of asking queries including both those keys... however the index will be bigger and performance may drop slightly compared to having those two in separate indices. But to get a more accurate answer you should really ask this in the Lucene forums. 2011/7/15 Boris Kizelshteyn bo...@popcha.com I have an index called UniqueIndex, here I am storing emails and ids which you guessed it have to be unique. Is there any performance difference if I store both keys on the same index vs. storing them on two different indexes? Many thanks! ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] OSGi and Neo4j - superbundle approach
Nicolas, the best would be to be able to emulate this in a test, too. Maybe classloading issues? Maybe forking the PaxExam test will reproduce the issue, or running you Felix setup as a testcase to trigger this? Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/ - Öresund - Innovation happens HERE. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Sun, Jul 17, 2011 at 9:21 PM, Nicolas Jouanin nicolas.joua...@gmail.com wrote: I've sent the CLA signup. The OSGiTest is working fine for me too. Problems come when taking the felix framework out of the box and deploying manually neo4j-osgi bundle with another bundle containing the same activator class. I made sure to deploy same bundles but I get java.lang.IllegalArgumentException: No index provider 'lucene' found when registring the IndexService. I think i miss a bundle which is created automatically by pax during the test case. May be lucene is missing. What do you think Le 17 juil. 2011 à 17:33, Peter Neubauer a écrit : Hi Nicolas, Have you been able to run https://github.com/njouanin/neo4j-osgi/blob/master/examples/src/test/java/org/neo4j/examples/osgi/OSGiTest.java properly? It actually is running on Equinox, so let me try it with Felix ... back in a moment. Also, in order to pull your changes, could you sign up as a committer, and send a mail according to http://wiki.neo4j.org/content/About_Contributor_License_Agreement ? Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/ - Öresund - Innovation happens HERE. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Sun, Jul 17, 2011 at 3:02 PM, Nicolas Jouanin nicolas.joua...@gmail.com wrote: Hi Peter, Thanks for initiating this work on OSGi bundle. I've forked you repository and tried to make is work using this sample activator [1]. My goal is to make my activator work when deployed on Felix with the neo4j superbundle. First of all, I had troubles to mvn clean install the bundle module. maven-bundle-plugin is complaining about bundle.namespace property not set. I've fixed it. Then when deploying it on Felix I had an error saying that javax.transaction couldn't be imported. Because this package is already included in superbundle, I've removed it explicitally from the Import-Package directive. You can see the changes here [2]. Then I tried to make my bundle activated using felix and the following bundles installed : g! lb START LEVEL 1 ID|State |Level|Name 0|Active | 0|System Bundle (3.2.2) 1|Active | 1|Neo4j OSGi default bundle (0.1.0.SNAPSHOT) 2|Active | 1|Apache Felix Bundle Repository (1.6.2) 3|Active | 1|Apache Felix Gogo Command (0.8.0) 4|Active | 1|Apache Felix Gogo Runtime (0.8.0) 5|Active | 1|Apache Felix Gogo Shell (0.8.0) 6|Resolved | 1|EscapeK - Neo4j OSGi how-to - Embedded client (0.0.1.SNAPSHOT) There are only felix core, neo4j superbundle and my test bundle. So when starting felix I get the following trace : Welcome to Apache Felix Gogo g! Opening database in embedded mode: 17 juil. 2011 14:57:08 org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog doInternalRecovery INFO: Non clean shutdown detected on log [graphdb/nioneo_logical.log.1]. Recovery started ... OK Populating it ... OK Hello, brave Neo4j world! ERROR: Bundle org.escapek.neo4j-osgi-howto.embedded-client [6] Error starting file:/Users/nico/Dev/felix-framework-3.2.2/bundle/org.escapek.neo4j-osgi-howto.embedded-client_0.0.1-SNAPSHOT.jar (org.osgi.framework.BundleException: Activator start error in bundle org.escapek.neo4j-osgi-howto.embedded-client [6].) java.lang.IllegalArgumentException: No index provider 'lucene' found at org.neo4j.kernel.IndexManagerImpl.getIndexProvider(IndexManagerImpl.java:76) at org.neo4j.kernel.IndexManagerImpl.findIndexConfig(IndexManagerImpl.java:116) at org.neo4j.kernel.IndexManagerImpl.getOrCreateIndexConfig(IndexManagerImpl.java:178) at org.neo4j.kernel.IndexManagerImpl.getOrCreateNodeIndex(IndexManagerImpl.java:267) at org.neo4j.kernel.IndexManagerImpl.forNodes(IndexManagerImpl.java:255) at org.neo4j.kernel.IndexManagerImpl.forNodes(IndexManagerImpl.java:249) at org.escapek.neo4jOsgiHowto.embeddedClient.EmbeddedClientActivator.start(EmbeddedClientActivator.java:65)
Re: [Neo4j] How to create a graph database out of a huge dataset?
Stephan, can you perhaps share your csv file or give at least a few sample lines and a typical distribution (articles per author etc). You tested this with 20M arcticles and 6M authors? What is the current runtime of that import with which kind of hardware? (when working on a similar test I noticed that a lot of time was spent in the parsing, so it might be sensible to write a small converter to convert that csv file to a file with one entry per line and at the same time filter out invalid entries (wite them to a separate file). (If I read your code correctly you also have a number of authors per article on following lines duplicating the article name just with another author ? That might be also handled by the separate file, which could contain the # of author or have a blank line between each new article. I don't know if your input data is presorted in any way. So if you might have recurring authors within a certain set you might also enable the caching of author index lookups. ((LuceneIndex)authorList).setCacheCapacity(Author,1); you just specifiy your java heap memory with -Xmx3G or such. For your memory mapped files, it would be good to know how much memory your machine has. Otherwise see here: (http://docs.neo4j.org/chunked/1.4/configuration.html) Something that might incur performance is creating the super-category nodes for articles and authors which then have 20M+ relationships. Perhaps it is better to split them by some sharding key to subnodes, or use an index for those too. It might also be better to reduce your tx size a bit to 10k. Something I noticed. You do the index get, check for the size and do getSingle() afterwards. I'm not sure if not closing the result iterator in the other case incurs any problems. So probably change that to Node author= authorList.get(Author, artikelinfo[2]).getSingle(); which automatically closes the result and returns null if there is none. Regarding database size. What is the size and format of the timestamp string? It might be sensible to convert that to a long before storing it in the graph or at least make sure it fits into the shortstring compression: (http://docs.neo4j.org/chunked/1.4/short-strings.html) depends on how you will use that in traversals. I'll take your code and will run it locally to see where the performance issues are. Cheers Michael Am 17.07.2011 um 20:53 schrieb st3ven: Hi, thanks for your fast answer. Right now I'm using lucene for 6M authors, but my whole dataset consists of nearly 25M authors. Can i use lucene there also, because I think this getting really slow to check if a user already exists. How can I change my heap memory settings and my memory-map settings, cause I'm using the transactional mode? Cause I think with 25M authors I will get a OutOfMemory Exception. Here is my code that I have already written so far: import java.io.BufferedReader; import java.io.FileReader; import java.io.IOException; import org.neo4j.graphdb.GraphDatabaseService; import org.neo4j.graphdb.Node; import org.neo4j.graphdb.Relationship; import org.neo4j.graphdb.Transaction; import org.neo4j.graphdb.index.Index; import org.neo4j.graphdb.index.IndexHits; import org.neo4j.graphdb.index.IndexManager; import org.neo4j.kernel.EmbeddedGraphDatabase; public class WikiGraphRegUser { /** * @param args */ public static void main(String[] args) throws IOException { BufferedReader bf = new BufferedReader(new FileReader( E:/wiki0.csv)); WikiGraphRegUser wgru = new WikiGraphRegUser(); wgru.createGraphDatabase(bf); } private String articleName = ; private GraphDatabaseService db; private IndexManager index; private IndexNode authorList; private int transactionCounter = 0; private Node article; private boolean isFirstAuthor = false; private Node author; private Relationship relationship; private int node; private void createGraphDatabase(BufferedReader bf) { db = new EmbeddedGraphDatabase(target/db); index = db.index(); authorList = index.forNodes(Author); String zeile; Transaction tx = db.beginTx(); try { // reads lines of CSV-file while ((zeile = bf.readLine()) != null) { if (transactionCounter++ % 5 == 0) { tx.success(); tx.finish(); tx = db.beginTx(); } // String[] looks like this: Article%;% Timestamp%;% Author String[] artikelinfo = zeile.split(%;% ); if (artikelinfo.length != 3) {
Re: [Neo4j] How can you have transactions with Lucene?
Very Kewl. Thanks Mattias. Can you send your code over to Lucene so they can add effective transactions. I know a number of people use Lucene for full text search in conjunction with MySql, and the lack of transactions is a major issue. Thanks again -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/How-can-you-have-transactions-with-Lucene-tp3176918p3177747.html Sent from the Neo4J Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user