Re: [Neo4j] Neo4j with Tomcat...need a transaction manager
Hey Jaf, On Sat, Jul 16, 2011 at 4:23 AM, etc3 wrote: > Michael, > > If I use the default trx manager in neo4j, what connection pool driver > should be used in Tomcat? The internal implementation of Neo4j's transaction manager is not currently exposed for use by 3rd party resources. This means that, while capable of coordinating a 2PC you cannot use it to do so as a standalone TM like JOTM. The work we have done is actually the reverse of that - it substitutes Neo4's TM for an externally provided one, since both our transactional resources (neo store and lucene indexes) are XA compatible. For you this means that you have to use an external TM (like Atomikos and JBossTM that you mentioned - we have also tested SpringSource's implementation and JOTM and found all four of them to work) to coordinate your XA datasources. Doing so for Tomcat however is something we have not tried yet - the blog posts Michael mentioned should cover a lot of ground nevertheless, the only thing not covered is integration with the container, the rest should work. I will try and replicate your scenario locally and see what trouble may rise but I am really interested to see how it went for you. Hope that helped a bit, CG > -Original Message- > From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On > Behalf Of Michael Hunger > Sent: Friday, July 15, 2011 9:05 PM > To: Neo4j user discussions > Subject: Re: [Neo4j] Neo4j with Tomcat...need a transaction manager > > Yes they were added. > > The XA-TX-Manager is configurable in Neo4j (defaults to our own but if you > have an external one like JOTM or Atomikos) then you can configure those to > be used. > > Example implementation for JOTM: > https://github.com/digitalstain/JOTMServiceProvider > > Implementation in neo4j is here: > https://github.com/neo4j/community/tree/master/kernel/src/main/java/org/neo4 > j/kernel/impl/transaction > > Cheers > > Michael > > Am 16.07.2011 um 02:43 schrieb etc3: > >> Thanks, Micheal. >> >> The blog post from Chris was from last year; were those changes for >> JOTM ever added to neo4j? >> >> -Original Message- >> From: user-boun...@lists.neo4j.org >> [mailto:user-boun...@lists.neo4j.org] On Behalf Of Michael Hunger >> Sent: Friday, July 15, 2011 8:00 PM >> To: Neo4j user discussions >> Subject: Re: [Neo4j] Neo4j with Tomcat...need a transaction manager >> >> Springsource also provides/is working on their own tomcat based XA TM. >> Perhaps you'd like to check that out. >> (http://forum.springsource.org/showthread.php?76843-JTA-for-Tc-Server&; >> p=2583 >> 83#post258383) >> >> Otherwise there is only JOTM left. >> >> Neo4j can integrate with those TM. >> See also these blog posts by our team member Chris Gioran: >> http://digitalstain.blogspot.com/2010/11/using-jotm-as-transactionmana >> ger-in >> .html >> http://digitalstain.blogspot.com/2011/02/springy-ouside-graphy-inside. >> html >> http://digitalstain.blogspot.com/2010/11/neo4j-internals-transactions- >> part-3 >> -as.html >> >> Internally it uses also 2PC with the nioneo store and lucene. >> >> Cheers >> >> Michael >> >> Am 16.07.2011 um 01:37 schrieb etc3: >> >>> We plan on using neo4j as the database for our website, which will >>> run on Tomcat. Also, we'll be using ActiveMQ for sending messages >>> within neo4j transactions, so that leaves the question of what to use >>> for the transaction manager. Atomikos, JBossTM comes to mind, are we >>> missing >> anything obvious? >>> Does neo4j provide anything out-of-box for transactions management? >>> >>> >>> >>> Thanks >>> >>> Jaf >>> >>> ___ >>> Neo4j mailing list >>> User@lists.neo4j.org >>> https://lists.neo4j.org/mailman/listinfo/user >> >> ___ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user >> >> ___ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] How can you have transactions with Lucene?
Very Kewl. Thanks Mattias. Can you send your code over to Lucene so they can add effective transactions. I know a number of people use Lucene for full text search in conjunction with MySql, and the lack of transactions is a major issue. Thanks again -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/How-can-you-have-transactions-with-Lucene-tp3176918p3177747.html Sent from the Neo4J Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] How to create a graph database out of a huge dataset?
Stephan,
can you perhaps share your csv file or give at least a few sample lines and a
typical distribution (articles per author etc). You tested this with 20M
arcticles and 6M authors? What is the current runtime of that import with which
kind of hardware? (when working on a similar test I noticed that a lot of time
was spent in the parsing, so it might be sensible to write a small converter to
convert that csv file to a file with one entry per line and at the same time
filter out invalid entries (wite them to a separate file). (If I read your code
correctly you also have a number of authors per article on following lines
duplicating the article name just with another author ? That might be also
handled by the separate file, which could contain the # of author or have a
blank line between each new article.
I don't know if your input data is presorted in any way. So if you might have
recurring authors within a certain set you might also enable the caching of
author index lookups.
((LuceneIndex)authorList).setCacheCapacity("Author",1);
you just specifiy your java heap memory with -Xmx3G or such.
For your memory mapped files, it would be good to know how much memory your
machine has. Otherwise see here:
(http://docs.neo4j.org/chunked/1.4/configuration.html)
Something that might incur performance is creating the super-category nodes for
articles and authors which then have 20M+ relationships. Perhaps it is better
to split them by some sharding key to subnodes, or use an index for those too.
It might also be better to reduce your tx size a bit to 10k.
Something I noticed. You do the index get, check for the size and do
getSingle() afterwards. I'm not sure if not closing the result iterator in the
other case incurs any problems. So probably change that to
Node author= authorList.get("Author", artikelinfo[2]).getSingle();
which automatically closes the result and returns null if there is none.
Regarding database size. What is the size and format of the timestamp string?
It might be sensible to convert that to a long before storing it in the graph
or at least make sure it
fits into the shortstring compression:
(http://docs.neo4j.org/chunked/1.4/short-strings.html) depends on how you will
use that in traversals.
I'll take your code and will run it locally to see where the performance issues
are.
Cheers
Michael
Am 17.07.2011 um 20:53 schrieb st3ven:
> Hi,
>
> thanks for your fast answer.
> Right now I'm using lucene for 6M authors, but my whole dataset consists of
> nearly 25M authors.
> Can i use lucene there also, because I think this getting really slow to
> check if a user already exists.
> How can I change my heap memory settings and my memory-map settings, cause
> I'm using the transactional mode?
> Cause I think with 25M authors I will get a OutOfMemory Exception.
>
> Here is my code that I have already written so far:
>
> import java.io.BufferedReader;
> import java.io.FileReader;
> import java.io.IOException;
>
> import org.neo4j.graphdb.GraphDatabaseService;
> import org.neo4j.graphdb.Node;
> import org.neo4j.graphdb.Relationship;
> import org.neo4j.graphdb.Transaction;
> import org.neo4j.graphdb.index.Index;
> import org.neo4j.graphdb.index.IndexHits;
> import org.neo4j.graphdb.index.IndexManager;
> import org.neo4j.kernel.EmbeddedGraphDatabase;
>
> public class WikiGraphRegUser {
>
> /**
>* @param args
>*/
> public static void main(String[] args) throws IOException {
>
> BufferedReader bf = new BufferedReader(new FileReader(
> "E:/wiki0.csv"));
> WikiGraphRegUser wgru = new WikiGraphRegUser();
> wgru.createGraphDatabase(bf);
> }
>
> private String articleName = "";
> private GraphDatabaseService db;
> private IndexManager index;
> private Index authorList;
> private int transactionCounter = 0;
> private Node article;
> private boolean isFirstAuthor = false;
> private Node author;
> private Relationship relationship;
> private int node;
>
> private void createGraphDatabase(BufferedReader bf) {
> db = new EmbeddedGraphDatabase("target/db");
> index = db.index();
> authorList = index.forNodes("Author");
>
> String zeile;
> Transaction tx = db.beginTx();
>
> try {
> // reads lines of CSV-file
> while ((zeile = bf.readLine()) != null) {
> if (transactionCounter++ % 5 == 0) {
>
> tx.success();
> tx.finish();
> tx = db.beginTx();
> }
> // String[] looks like this: Article%;%
> Timestamp%;% Author
> String[] artikelinfo = zei
Re: [Neo4j] OSGi and Neo4j - superbundle approach
Nicolas, the best would be to be able to emulate this in a test, too. Maybe classloading issues? Maybe forking the PaxExam test will reproduce the issue, or running you Felix setup as a testcase to trigger this? Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/ - Öresund - Innovation happens HERE. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Sun, Jul 17, 2011 at 9:21 PM, Nicolas Jouanin wrote: > I've sent the CLA signup. > The OSGiTest is working fine for me too. Problems come when taking the felix > framework out of the box and deploying manually neo4j-osgi bundle with > another bundle containing the same activator class. > I made sure to deploy same bundles but I get > java.lang.IllegalArgumentException: No index provider 'lucene' found when > registring the IndexService. > I think i miss a bundle which is created automatically by pax during the test > case. May be lucene is missing. > What do you think > > Le 17 juil. 2011 à 17:33, Peter Neubauer a écrit : > >> Hi Nicolas, >> Have you been able to run >> https://github.com/njouanin/neo4j-osgi/blob/master/examples/src/test/java/org/neo4j/examples/osgi/OSGiTest.java >> properly? It actually is running on Equinox, so let me try it with >> Felix ... back in a moment. >> >> Also, in order to pull your changes, could you sign up as a committer, >> and send a mail according to >> http://wiki.neo4j.org/content/About_Contributor_License_Agreement ? >> >> >> Cheers, >> >> /peter neubauer >> >> GTalk: neubauer.peter >> Skype peter.neubauer >> Phone +46 704 106975 >> LinkedIn http://www.linkedin.com/in/neubauer >> Twitter http://twitter.com/peterneubauer >> >> http://www.neo4j.org - Your high performance graph database. >> http://startupbootcamp.org/ - Öresund - Innovation happens HERE. >> http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. >> >> >> >> On Sun, Jul 17, 2011 at 3:02 PM, Nicolas Jouanin >> wrote: >>> Hi Peter, >>> >>> Thanks for initiating this work on OSGi bundle. >>> I've forked you repository and tried to make is work using this sample >>> activator [1]. My goal is to make my activator work when deployed on Felix >>> with the neo4j superbundle. >>> First of all, I had troubles to mvn clean install the bundle module. >>> maven-bundle-plugin is complaining about bundle.namespace property not set. >>> I've fixed it. >>> Then when deploying it on Felix I had an error saying that >>> javax.transaction couldn't be imported. Because this package is already >>> included in superbundle, I've removed it explicitally from the >>> Import-Package directive. You can see the changes here [2]. >>> Then I tried to make my bundle activated using felix and the following >>> bundles installed : >>> g! lb >>> START LEVEL 1 >>> ID|State |Level|Name >>> 0|Active | 0|System Bundle (3.2.2) >>> 1|Active | 1|Neo4j OSGi default bundle (0.1.0.SNAPSHOT) >>> 2|Active | 1|Apache Felix Bundle Repository (1.6.2) >>> 3|Active | 1|Apache Felix Gogo Command (0.8.0) >>> 4|Active | 1|Apache Felix Gogo Runtime (0.8.0) >>> 5|Active | 1|Apache Felix Gogo Shell (0.8.0) >>> 6|Resolved | 1|EscapeK - Neo4j OSGi how-to - Embedded client >>> (0.0.1.SNAPSHOT) >>> There are only felix core, neo4j superbundle and my test bundle. >>> So when starting felix I get the following trace : >>> >>> Welcome to Apache Felix Gogo >>> >>> g! Opening database in embedded mode: 17 juil. 2011 14:57:08 >>> org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog >>> doInternalRecovery >>> INFO: Non clean shutdown detected on log [graphdb/nioneo_logical.log.1]. >>> Recovery started ... >>> OK >>> Populating it ... OK >>> Hello, brave Neo4j world! >>> ERROR: Bundle org.escapek.neo4j-osgi-howto.embedded-client [6] Error >>> starting >>> file:/Users/nico/Dev/felix-framework-3.2.2/bundle/org.escapek.neo4j-osgi-howto.embedded-client_0.0.1-SNAPSHOT.jar >>> (org.osgi.framework.BundleException: Activator start error in bundle >>> org.escapek.neo4j-osgi-howto.embedded-client [6].) >>> java.lang.IllegalArgumentException: No index provider 'lucene' found >>> at >>> org.neo4j.kernel.IndexManagerImpl.getIndexProvider(IndexManagerImpl.java:76) >>> at >>> org.neo4j.kernel.IndexManagerImpl.findIndexConfig(IndexManagerImpl.java:116) >>> at >>> org.neo4j.kernel.IndexManagerImpl.getOrCreateIndexConfig(IndexManagerImpl.java:178) >>> at >>> org.neo4j.kernel.IndexManagerImpl.getOrCreateNodeIndex(IndexManagerImpl.java:267) >>> at >>> org.neo4j.kernel.IndexManagerImpl.forNodes(IndexManagerImpl.java:255) >>> at >>> org.n
Re: [Neo4j] use one index per key or one index for all keys?
Hi Boris, If you store them both in one index you get the benefit of asking queries including both those keys... however the index will be bigger and performance may drop slightly compared to having those two in separate indices. But to get a more accurate answer you should really ask this in the Lucene forums. 2011/7/15 Boris Kizelshteyn > I have an index called UniqueIndex, here I am storing emails and ids which > you guessed it have to be unique. Is there any performance difference if I > store both keys on the same index vs. storing them on two different > indexes? > > Many thanks! > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] How can you have transactions with Lucene?
Exactly, that's what the neo4j integration layer adds to it. 2011/7/17 eialbur > Lucene itself says it doesn't properly support ACID or XA transaction > support. > (Bug 1149 - Major - New Feature - Reopened - Unresolved). > > -- > View this message in context: > http://neo4j-community-discussions.438527.n3.nabble.com/How-can-you-have-transactions-with-Lucene-tp3176918p3176981.html > Sent from the Neo4J Community Discussions mailing list archive at > Nabble.com. > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] OSGi and Neo4j - superbundle approach
I've sent the CLA signup. The OSGiTest is working fine for me too. Problems come when taking the felix framework out of the box and deploying manually neo4j-osgi bundle with another bundle containing the same activator class. I made sure to deploy same bundles but I get java.lang.IllegalArgumentException: No index provider 'lucene' found when registring the IndexService. I think i miss a bundle which is created automatically by pax during the test case. May be lucene is missing. What do you think Le 17 juil. 2011 à 17:33, Peter Neubauer a écrit : > Hi Nicolas, > Have you been able to run > https://github.com/njouanin/neo4j-osgi/blob/master/examples/src/test/java/org/neo4j/examples/osgi/OSGiTest.java > properly? It actually is running on Equinox, so let me try it with > Felix ... back in a moment. > > Also, in order to pull your changes, could you sign up as a committer, > and send a mail according to > http://wiki.neo4j.org/content/About_Contributor_License_Agreement ? > > > Cheers, > > /peter neubauer > > GTalk: neubauer.peter > Skype peter.neubauer > Phone +46 704 106975 > LinkedIn http://www.linkedin.com/in/neubauer > Twitter http://twitter.com/peterneubauer > > http://www.neo4j.org - Your high performance graph database. > http://startupbootcamp.org/- Öresund - Innovation happens HERE. > http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. > > > > On Sun, Jul 17, 2011 at 3:02 PM, Nicolas Jouanin > wrote: >> Hi Peter, >> >> Thanks for initiating this work on OSGi bundle. >> I've forked you repository and tried to make is work using this sample >> activator [1]. My goal is to make my activator work when deployed on Felix >> with the neo4j superbundle. >> First of all, I had troubles to mvn clean install the bundle module. >> maven-bundle-plugin is complaining about bundle.namespace property not set. >> I've fixed it. >> Then when deploying it on Felix I had an error saying that javax.transaction >> couldn't be imported. Because this package is already included in >> superbundle, I've removed it explicitally from the Import-Package directive. >> You can see the changes here [2]. >> Then I tried to make my bundle activated using felix and the following >> bundles installed : >> g! lb >> START LEVEL 1 >> ID|State |Level|Name >>0|Active |0|System Bundle (3.2.2) >>1|Active |1|Neo4j OSGi default bundle (0.1.0.SNAPSHOT) >>2|Active |1|Apache Felix Bundle Repository (1.6.2) >>3|Active |1|Apache Felix Gogo Command (0.8.0) >>4|Active |1|Apache Felix Gogo Runtime (0.8.0) >>5|Active |1|Apache Felix Gogo Shell (0.8.0) >>6|Resolved |1|EscapeK - Neo4j OSGi how-to - Embedded client >> (0.0.1.SNAPSHOT) >> There are only felix core, neo4j superbundle and my test bundle. >> So when starting felix I get the following trace : >> >> Welcome to Apache Felix Gogo >> >> g! Opening database in embedded mode: 17 juil. 2011 14:57:08 >> org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog doInternalRecovery >> INFO: Non clean shutdown detected on log [graphdb/nioneo_logical.log.1]. >> Recovery started ... >> OK >> Populating it ... OK >> Hello, brave Neo4j world! >> ERROR: Bundle org.escapek.neo4j-osgi-howto.embedded-client [6] Error >> starting >> file:/Users/nico/Dev/felix-framework-3.2.2/bundle/org.escapek.neo4j-osgi-howto.embedded-client_0.0.1-SNAPSHOT.jar >> (org.osgi.framework.BundleException: Activator start error in bundle >> org.escapek.neo4j-osgi-howto.embedded-client [6].) >> java.lang.IllegalArgumentException: No index provider 'lucene' found >>at >> org.neo4j.kernel.IndexManagerImpl.getIndexProvider(IndexManagerImpl.java:76) >>at >> org.neo4j.kernel.IndexManagerImpl.findIndexConfig(IndexManagerImpl.java:116) >>at >> org.neo4j.kernel.IndexManagerImpl.getOrCreateIndexConfig(IndexManagerImpl.java:178) >>at >> org.neo4j.kernel.IndexManagerImpl.getOrCreateNodeIndex(IndexManagerImpl.java:267) >>at >> org.neo4j.kernel.IndexManagerImpl.forNodes(IndexManagerImpl.java:255) >>at >> org.neo4j.kernel.IndexManagerImpl.forNodes(IndexManagerImpl.java:249) >>at >> org.escapek.neo4jOsgiHowto.embeddedClient.EmbeddedClientActivator.start(EmbeddedClientActivator.java:65) >>at >> org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:629) >>at org.apache.felix.framework.Felix.activateBundle(Felix.java:1904) >>at org.apache.felix.framework.Felix.startBundle(Felix.java:1822) >>at >> org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1192) >>at >> org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:266) >>at java.lang.Thread.run(Thread.java:680) >> >> So everything works fine (open database, populate it and register service) >> until the activator tries to register I
Re: [Neo4j] How to create a graph database out of a huge dataset?
Hi,
thanks for your fast answer.
Right now I'm using lucene for 6M authors, but my whole dataset consists of
nearly 25M authors.
Can i use lucene there also, because I think this getting really slow to
check if a user already exists.
How can I change my heap memory settings and my memory-map settings, cause
I'm using the transactional mode?
Cause I think with 25M authors I will get a OutOfMemory Exception.
Here is my code that I have already written so far:
import java.io.BufferedReader;
import java.io.FileReader;
import java.io.IOException;
import org.neo4j.graphdb.GraphDatabaseService;
import org.neo4j.graphdb.Node;
import org.neo4j.graphdb.Relationship;
import org.neo4j.graphdb.Transaction;
import org.neo4j.graphdb.index.Index;
import org.neo4j.graphdb.index.IndexHits;
import org.neo4j.graphdb.index.IndexManager;
import org.neo4j.kernel.EmbeddedGraphDatabase;
public class WikiGraphRegUser {
/**
* @param args
*/
public static void main(String[] args) throws IOException {
BufferedReader bf = new BufferedReader(new FileReader(
"E:/wiki0.csv"));
WikiGraphRegUser wgru = new WikiGraphRegUser();
wgru.createGraphDatabase(bf);
}
private String articleName = "";
private GraphDatabaseService db;
private IndexManager index;
private Index authorList;
private int transactionCounter = 0;
private Node article;
private boolean isFirstAuthor = false;
private Node author;
private Relationship relationship;
private int node;
private void createGraphDatabase(BufferedReader bf) {
db = new EmbeddedGraphDatabase("target/db");
index = db.index();
authorList = index.forNodes("Author");
String zeile;
Transaction tx = db.beginTx();
try {
// reads lines of CSV-file
while ((zeile = bf.readLine()) != null) {
if (transactionCounter++ % 5 == 0) {
tx.success();
tx.finish();
tx = db.beginTx();
}
// String[] looks like this: Article%;%
Timestamp%;% Author
String[] artikelinfo = zeile.split("%;% ");
if (artikelinfo.length != 3) {
System.out.println("ERROR: check CSV");
for (int i = 0; i < artikelinfo.length;
i++) {
System.out.println(artikelinfo[i]);
}
return;
}
if (articleName == "") {
// create Article and connect with
ReferenceNode
article = createArticle(artikelinfo[0],
db.getReferenceNode(),
MyRelationshipTypes.ARTICLE);
articleName = artikelinfo[0];
isFirstAuthor = true;
} else if (!articleName.equals(artikelinfo[0]))
{
// create Article and connect with
ReferenceNode
article = createArticle(artikelinfo[0],
db.getReferenceNode(),
MyRelationshipTypes.ARTICLE);
articleName = artikelinfo[0];
isFirstAuthor = true;
}
// checks if author already exists
IndexHits hits = authorList.get("Author",
artikelinfo[2]);
// if new author
if (hits.size() == 0) {
if (isFirstAuthor) {
// creates author and connects
him with an article
author =
createAndConnectNode(artikelinfo[2], article,
MyRelationshipTypes.WROTE, artikelinfo[1]);
isFirstAuthor = false;
} else {
author =
createAndConnectNode(artikelinfo[2], article,
MyRelationshipTypes.EDIT, artikelinfo[1]);
}
Re: [Neo4j] RDF data wrapper/triple store options for fast traversal
Jan, I would suggest to use the Tinkerpop RDF wrapper, then examine the resulting graph in neo4j, and se if raw traversals can find the relationships for you, After all, you can always drop down to the raw graph even while using the RDF implementation. That is one of the points of Neo4j dealing with raw graphs. Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/ - Öresund - Innovation happens HERE. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Sun, Jul 17, 2011 at 8:20 PM, Jan Bezget wrote: > Hello! > I'm new to Neo4j and need some advice regarding my project and how I should > proceed. > The purpose of the project is to find the shortest path between two things > (similar to RelFinder) as quickly as possible. > > So far I've loaded a few million dbpedia triples using NxParser and > neo4j-rdf component as well as tried neo4j's pathfinding capabilities. > Everything works fine! > However, I have noticed that neo4j-rdf's DenseTripleStore stores object > literals and their predicates within the subject node as a set of > properties. > > Since, above all, I'm after fast traversals I'm wondering, whether this > incurs any performance penalties to pathfinding algorithms? If I were to > store those literals as separate nodes, would _that_ incur penalties? > Is neo4j-rdf still in development or should I try out TinkerPop's Blueprints > (seems very popular lately!)? > Whilst I like the features these high-level wrappers provide, I'm really > only using pathfinding and indexing. Should I write my own lightweight > wrapper? But in the end is it really worth it, or should I just use existing > wrappers? > > Thank you! > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] RDF data wrapper/triple store options for fast traversal
Hello! I'm new to Neo4j and need some advice regarding my project and how I should proceed. The purpose of the project is to find the shortest path between two things (similar to RelFinder) as quickly as possible. So far I've loaded a few million dbpedia triples using NxParser and neo4j-rdf component as well as tried neo4j's pathfinding capabilities. Everything works fine! However, I have noticed that neo4j-rdf's DenseTripleStore stores object literals and their predicates within the subject node as a set of properties. Since, above all, I'm after fast traversals I'm wondering, whether this incurs any performance penalties to pathfinding algorithms? If I were to store those literals as separate nodes, would _that_ incur penalties? Is neo4j-rdf still in development or should I try out TinkerPop's Blueprints (seems very popular lately!)? Whilst I like the features these high-level wrappers provide, I'm really only using pathfinding and indexing. Should I write my own lightweight wrapper? But in the end is it really worth it, or should I just use existing wrappers? Thank you! ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] neo4j in a distributed system
Aliabbas, There is no general performance statement for large stores. The performance is dependend on the structure of your graph and the typical operations/use-cases you want to perform. Also underlying hardware (RAM, disk, IO) and JVM play important roles. The most sensible thing to do is to take the target hardware, a graph generator that creates a graph that resembles your domain and sample implementations for the most common use-cases. Then you can run a PoC with the required number of concurrent connections executing the use-cases in real-life environment, tune parameters so that your SLA's are met. We did that for several customers, if you are interested in this kind of information, please contact us for commercial support. Cheers Michael Am 17.07.2011 um 19:31 schrieb Aliabbas Petiwala: > Great Michael! > can you share with us the performance evaluation results with code of > the 32 billion nodes and relationships. This be a great proof for > neo4js high performance. > > > > On 7/17/11, Michael Hunger wrote: >> with "huge" database you mean that you want to shard the graph? >> >> And how large is your database? Please note that neo4j can handle 32 billion >> nodes and relationships in a single instance. >> >> If your database is larger than that you have to apply some kind of domain >> level sharding. >> (See Jims article: >> http://jim.webber.name/2011/02/23/abe72f61-27fb-4c1b-8ce1-d0db7583497b.aspx >> and >> http://jim.webber.name/2011/02/16/3b8f4b3d-c884-4fba-ae6b-7b75a191fa22.aspx) >> >> >> >> Am 17.07.2011 um 18:50 schrieb st3...@web.de: >> >>> Hi all, >>> >>> I would like to know if it is possible to use neo4j in a distributed >>> system. >>> If it is possible, are there any guides or tutorials how to realize that? >>> For my huge database it would be nice to use it in distributed system. >>> >>> Thanks for your help, >>> Stephan >>> ___ >>> Neo4j mailing list >>> User@lists.neo4j.org >>> https://lists.neo4j.org/mailman/listinfo/user >> >> ___ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user >> > > > -- > Aliabbas Petiwala > M.Tech CSE > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] neo4j in a distributed system
Great Michael! can you share with us the performance evaluation results with code of the 32 billion nodes and relationships. This be a great proof for neo4js high performance. On 7/17/11, Michael Hunger wrote: > with "huge" database you mean that you want to shard the graph? > > And how large is your database? Please note that neo4j can handle 32 billion > nodes and relationships in a single instance. > > If your database is larger than that you have to apply some kind of domain > level sharding. > (See Jims article: > http://jim.webber.name/2011/02/23/abe72f61-27fb-4c1b-8ce1-d0db7583497b.aspx > and > http://jim.webber.name/2011/02/16/3b8f4b3d-c884-4fba-ae6b-7b75a191fa22.aspx) > > > > Am 17.07.2011 um 18:50 schrieb st3...@web.de: > >> Hi all, >> >> I would like to know if it is possible to use neo4j in a distributed >> system. >> If it is possible, are there any guides or tutorials how to realize that? >> For my huge database it would be nice to use it in distributed system. >> >> Thanks for your help, >> Stephan >> ___ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user > > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Aliabbas Petiwala M.Tech CSE ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] neo4j in a distributed system
with "huge" database you mean that you want to shard the graph? And how large is your database? Please note that neo4j can handle 32 billion nodes and relationships in a single instance. If your database is larger than that you have to apply some kind of domain level sharding. (See Jims article: http://jim.webber.name/2011/02/23/abe72f61-27fb-4c1b-8ce1-d0db7583497b.aspx and http://jim.webber.name/2011/02/16/3b8f4b3d-c884-4fba-ae6b-7b75a191fa22.aspx) Am 17.07.2011 um 18:50 schrieb st3...@web.de: > Hi all, > > I would like to know if it is possible to use neo4j in a distributed system. > If it is possible, are there any guides or tutorials how to realize that? > For my huge database it would be nice to use it in distributed system. > > Thanks for your help, > Stephan > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] How to create a graph database out of a huge dataset?
Stephan, This is a common thing when inserting data. You should be able to use lucene in both settings (6M authors is not that much). Please have a look at your heap memory settings (and in transactional mode also your memory-map settings for neo4j). For batch inserter. You can query the index after you called flush on the index. That should return a node-id which you then can use to create relationship to that looked up node. If you could share the code that you've already written we can have a look at it. The simplest approach is to just take a HashMap for the authors and their name as key. (while indexing them for later use anyway). Just make sure your heap is large enough to hold the map and the objects created during the insert. #2 can you describe your distributed system? You can run a cluster of neo4j-HA instances that each of your distributed engines connects to. (HA instances can run embedded or as Neo4j-servers depending on your needs). (see here: http://docs.neo4j.org/chunked/stable/ha.html) Cheers Michael Am 17.07.2011 um 18:31 schrieb st3ven: > Hi all, > > I'm new to neo4j and graph databases. > To create my graph database I got two questions for you: > > 1. > I want to create a graph database out of a huge CSV file. > The problem is, that i need to index the nodes I have already created, so > that I don't create duplicate nodes. > > My CSV file looks like this: > > Article A, Timestamp, Author A > Article A, Timestamp, Author B > Article A, Timestamp, Author C > Article B, Timestamp, Author A > Article B, Timestamp, Author B > Article B, Timestamp, Author D > > As you can see I need to access nodes I have already created and connect > them to the next Article. > Right now I'm using the LuceneIndex, but with around 6M authors this is > getting really slow. > Is there any other possibility to access nodes that I have already created? > BatchInserter also doesn't work, because there you can't access nodes which > you have created before. > > 2. > Is it possible to use neo4j in a distributed system? > If it is possible, are there any guides or tutorials how to realize that? > > Thanks for your help, > Stephan > > > -- > View this message in context: > http://neo4j-community-discussions.438527.n3.nabble.com/How-to-create-a-graph-database-out-of-a-huge-dataset-tp3177076p3177076.html > Sent from the Neo4J Community Discussions mailing list archive at Nabble.com. > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Index returns all nodes for value="undefined"!
Perhaps it is also an issue of your REST-client? If the value when rendered as part of a string evaluates to the empty string the resulting URL would be the same as the one that retrieves all nodes. i.e. http://localhost:7474/db/data/index/node/nodes/username Just an idea, could you verify the REST url that is used? Am 17.07.2011 um 17:06 schrieb Mattias Persson: > No, there's no such "undefined" constant which will return everything. It > definitely sounds like a JavaScript issue, and that it turns nulls into > "undefined". And perhaps your code which tries to index null values. > > 2011/7/16 Aseem Kishore > >> We're seeing this crazy behavior where our exact index is returning all >> nodes if we pass a lookup value of "undefined"! E.g.: >> >> http://localhost:7474/db/data/index/node/nodes/username/undefined >> >> Is this a bug in Neo4j / REST API? Or is this a symptom of some bug we have >> in our own code? >> >> It could very well be our code, since we're using JS, which turns undefined >> values into the string "undefined" if you try to add it to a string >> (definitely a wtfjs candidate), but I wanted to see if others are seeing >> this in Neo4j too! >> >> Thanks, >> >> Aseem >> ___ >> Neo4j mailing list >> User@lists.neo4j.org >> https://lists.neo4j.org/mailman/listinfo/user >> > > > > -- > Mattias Persson, [matt...@neotechnology.com] > Hacker, Neo Technology > www.neotechnology.com > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Officially supported mechanism for J2EE
Hi, do you mean a javax.sql.DataSource or an JCA adaptor ? Could you please explain your architecture and usage scenario where this would be necessary. You can use Neo4j in embedded mode inside any J2EE container without issues. If the container comes with its own XA-TM then you can integrate with that. Cheers Michael Am 17.07.2011 um 17:03 schrieb eialbur: > I really like Neo, but am concerned that there is so little official company > support for J2EE. I want to use Neo as a standard data source in J2EE, with > container managed transactions, etc. > > I don't want to use the REST interface - I want a standard data source > interface. There does not appear to be an officially provided mechanism to > use Neo as a data source in J2EE. I've seen a couple of people post their > code to help someone get started, but shouldn't Neo provide a real official > sanctioned mechanism? > > P.S. I don't want this thread to turn into a religious war over REST. If > you like REST and want to defend it then please start a thread for that > purpose. Please just accept that I don't want to use the REST interface - I > want a standard DB interface for my DB. > > -- > View this message in context: > http://neo4j-community-discussions.438527.n3.nabble.com/Officially-supported-mechanism-for-J2EE-tp3176938p3176938.html > Sent from the Neo4J Community Discussions mailing list archive at Nabble.com. > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Fwd: RDF Examples not working - parent-pom
Anfang der weitergeleiteten E-Mail: > Von: datablend > Datum: 17. Juli 2011 17:40:39 MESZ > An: Michael Hunger > Betreff: Re: [Neo4j] RDF Examples not working - parent-pom > > Hi Michael, > > Thanks for the information. Wrote a new article on the use of Tinkerpop for > importing and querying RDF data: http://datablend.be/?p=554 > > Greetings, > > Davy > > On 14 Jul 2011, at 17:50, Michael Hunger wrote: > >> Davy, >> >> you used the neo4j-sail project for your examples, tools and demos. >> >> Unfortunately that one is no longer under active development, so things like >> the parent pom issue pop up. >> >> Would you mind having a look at the much more complete and up to date sail >> implementation from tinkerpop to run on top of neo4j? >> >> Thanks so much and great work ! >> >> >> Michael >> >> Anfang der weitergeleiteten E-Mail: >> >>> Von: Thomas Fritz >>> Datum: 14. Juli 2011 17:33:06 MESZ >>> An: Neo4j user discussions >>> Betreff: Re: [Neo4j] RDF Examples not working - parent-pom >>> Antwort an: Neo4j user discussions >>> >>> Hi, >>> >>> No, it is of course not in my own pom.xml - the projects pom.xml - it >>> is in the Neo4J dependencies (rdf, sparql, sail). I think it has to be >>> fixed from Neo4J and has to be updated on maven central? >>> >>> Kind regards >>> >>> >>> >>> >>> >>> --- >>> Thomas FRITZ >>> web http://fritzthomas.com >>> twitter http://twitter.com/thomasf >>> >>> >>> >>> 2011/7/14 Michael Hunger : It should be directly in your pom.xml at the top, there is an xml section with the parent element. Cheers Michael If that is not in your local pom, try to pull that project from svn remove that section and build it locally. Am 14.07.2011 um 14:49 schrieb Thomas Fritz: > Thanks for answering! > How can i remove the parent section from the pom.xml? It is in the > dependency project maven tries to resolve, right? So how and where can > i edit it? > > Kind regards > > > --- > Thomas FRITZ > web http://fritzthomas.com > twitter http://twitter.com/thomasf > > > > 2011/7/14 Michael Hunger : >> You can just try to remove the >> >> >> ... >> >> >> block, that should work. Those are just meant for being built integrated >> in a larger system build. >> >> Cheers >> >> Michael >> >> Am 14.07.2011 um 13:23 schrieb Thomas Fritz: >> >>> Hi, >>> >>> Who can fix this? As i can see one of the neo4j-rdf dependencys pulls >>> that dependency in. >>> Or can i bypass and fix it locally until it is fixed? >>> >>> Thanks >>> Kind regards >>> >>> >>> >>> --- >>> Thomas FRITZ >>> web http://fritzthomas.com >>> twitter http://twitter.com/thomasf >>> >>> >>> >>> 2011/7/14 Peter Neubauer : Not sure but parent - pom should maybe be parent - central? /peter Sent from my phone. On Jul 14, 2011 12:23 PM, "Thomas Fritz" wrote: > I have tried the code example from here: http://datablend.be/?p=411 . > I downloaded the source from github - > https://github.com/datablend/neo4j-sail-test . I have also tried this > example: https://github.com/aduric/Neo-SPARQL-Test > They both can not be build with maven. The "parent-pom" can not be > found by maven. I am using netbeans 7 with bundled maven 3.0.3: > > > > Building Unnamed - neo4j-sail-test:neo4j-sail-test:jar:1.0 > task-segment: [clean, install] > > [clean:clean] > Deleting directory /home/thomas/Projekte/Neo4J-Tests/neo4j-sail-test/target > [resources:resources] > Using platform encoding (UTF-8 actually) to copy filtered resources, > i.e. build is platform dependent! > Copying 1 resource > snapshot org.neo4j:neo4j-rdf-sail:0.6-SNAPSHOT: checking for updates > from central > snapshot org.neo4j:neo4j-rdf-sail:0.6-SNAPSHOT: checking for updates > from aduna-repo > snapshot org.neo4j:neo4j-kernel:1.1-SNAPSHOT: checking for updates > from aduna-repo > snapshot org.neo4j:neo4j-kernel:1.1-SNAPSHOT: checking for updates > from neo4j-snapshot-repository > snapshot org.neo4j:neo4j-kernel:1.1-SNAPSHOT: checking for updates > from sonatype-nexus-snapshots > snapshot org.neo4j:neo4j-rdf:0.7-SNAPSHOT: checking for updates from aduna-repo > snapshot org.neo4j:neo4j-rdf:0.7-SNAPSHOT: checking for updates from > sonatype-nexus-snapshots > snapshot org.neo4j:neo4j-meta-model:0.9-SNAPSHOT: checking f
[Neo4j] neo4j in a distributed system
Hi all, I would like to know if it is possible to use neo4j in a distributed system. If it is possible, are there any guides or tutorials how to realize that? For my huge database it would be nice to use it in distributed system. Thanks for your help, Stephan ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Django + Neo4j seamless installation error: No module named CallbackFilter
On Fri, Jul 15, 2011 at 2:33 AM, Richard Schwab wrote:
> Hello,
> I'm having difficulty integrating Django and Neo4j. I've tried it on
> several
> different systems including Mac OSX 10.6.8 and Ubuntu 11.04. Perhaps
> someone
> can give me some feedback or suggestions? I'm good with Django but haven't
> played with VirtualEnv, Jython or Neo4j before.
>
> Here is the error I'm getting trying the steps at thobe.org's seamless
> neo4j
> integration in django:
>
> == Error ==
> ValueError: Unable to configure filter 'require_debug_false': Cannot
> resolve
> 'django.utils.log.CallbackFilter': No module named CallbackFilter
>
> == Steps ==
> Mainly using
> http://journal.thobe.org/2009/12/seamless-neo4j-integration-in-django.html
> ,
> I'm following these steps and I apologize if these are unnecessary, I'm
> just
> trying to give more info to get to an answer quickly.
>
> 1. VirtualEnv
> 1. Install virtualenv ($ sudo apt-get install virtualenv)
> 2. Set up a virtualenv ($ virtualenv TestEnv)
> 3. Activate the virtualenv ($ source TestEnv/bin/activate)
> 2. Jython - 2.5.2 (I've also tried 2.53b and no luck)
> 1. Download jython (sourceforge.net) & uncompress
> 2. Copy jython to /usr/lib, make symbolic link ($ ln -s
> /usr/lib/jython2.5.2/lib/jython /usr/bin/jython)
> 3. Neo4j - 1.4
> 1. Download Neo4j ($ svn co
> https://svn.neo4j.org/components/neo4j.py/trunk neo4j-python)
> 2. Install Neo4j ($ sudo jython setup.py install)
> 3. Tested: jython: import neo4j works
> 4. Django - 1.3.0
> 1. Download & extract Django (https://www.djangoproject.com/download/
> )
> 2. Install Django ($ sudo jython setup.py install)
> 3. Tested: jython: import django works
> 5. Django-Jython - 1.2.0
> 1. Download & extract Django-Jython (
> http://code.google.com/p/django-jython/downloads/list)
> 2. Install Django-Jython ($ sudo jython setup.py install)
> 6. Created a Django-Admin-Jy ln ($ alias django-admin-jy="jython
> /usr/lib/jython2.5.2/Lib/site-packages/django/bin/django-admin.py")
> 7. Created a Django Project ($ django-admin-jy startproject neo4django)
> 8. Created a Django App ($ jython neo4django/manage.py startapp business)
> 9. Edited settings.py
> 1.
>
> Edit settings.py
>
> NEO4J_RESOURCE_URI = '/var/neo4j/neo4django'
>
> # NEO4J_RESOURCE_URI should be the path to where
>
> #you want to store the Neo4j database.
>
>
> NEO4J_OPTIONS = {
>
> # this is optional and can be used to specify
>
> # extra startup parameters for Neo4j, such as
>
> # the classpath to load Neo4j from.
>
> }
> 10. Edited business/models.py
> 1.
>
> from neo4j.model import django_model as model
>
>
> class Person(model.NodeModel):
>
> first_name = model.Property()
>
> last_name = model.Property()
>
> def __unicode__(self):
>
> return u"%s %s" % (self.first_name, self.last_name)
>
>
> class Company(model.NodeModel):
>
> name = model.Property(indexed=True)
>
> owners = model.Relationship(Person,
>
> type=model.Outgoing.OWNED_BY,
>
> related_name="owns",
>
> )
>
> employees = model.Relationship(Person,
>
> type=model.Incoming.WORKS_AT,
>
> related_name="employer",
>
> related_single=True, # Only allow Persons to work at one
> Company
>
> )
>
> def __unicode__(self):
>
> return self.name
> 11. Tested it out: $ jython neo4django/manage.py shell results in the
> error (same as written above)
> - ValueError: Unable to configure filter 'require_debug_false': Cannot
> resolve 'django.utils.log.CallbackFilter': No module named
> CallbackFilter
>
> So all that said if anyone has any ideas, I'd appreciate it immensely. I
> was
> wondering also if my versioning was correct, or if I should drop to some
> older versions of django perhaps? It doesn't seem like there's been much
> work on this since 2009.
>
That is a very good suspicion. That Django has changed since I wrote that
code, and that article.
Trying an older version of Django might do the trick. Sadly I can't find
which version of Django I used back then.
However, there is no use of CallbackFilter in my code, which seems to imply
that this would be a problem in Django.
Looking at the Django documentation:
https://docs.djangoproject.com/en/dev/topics/logging/#id3 the CallbackFilter
feature seems to not be released yet. How it got into your settings.py (I'm
guessing that is where 'require_debug_false' is configures) I don't know. If
I had not had your complete description of everything you did, I would have
guessed that you either:
a) Followed the documentation of the wrong version of Django from their
website (the dev version is easier to find then the ones for released
versions). Or
b) Used a newer version of Django to auto-ge
[Neo4j] How to create a graph database out of a huge dataset?
Hi all, I'm new to neo4j and graph databases. To create my graph database I got two questions for you: 1. I want to create a graph database out of a huge CSV file. The problem is, that i need to index the nodes I have already created, so that I don't create duplicate nodes. My CSV file looks like this: Article A, Timestamp, Author A Article A, Timestamp, Author B Article A, Timestamp, Author C Article B, Timestamp, Author A Article B, Timestamp, Author B Article B, Timestamp, Author D As you can see I need to access nodes I have already created and connect them to the next Article. Right now I'm using the LuceneIndex, but with around 6M authors this is getting really slow. Is there any other possibility to access nodes that I have already created? BatchInserter also doesn't work, because there you can't access nodes which you have created before. 2. Is it possible to use neo4j in a distributed system? If it is possible, are there any guides or tutorials how to realize that? Thanks for your help, Stephan -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/How-to-create-a-graph-database-out-of-a-huge-dataset-tp3177076p3177076.html Sent from the Neo4J Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] OSGi and Neo4j - superbundle approach
Nicolas,
just pushed some more initialization along your lines, and have the
test running with the output:
[~/code/neo/neo4j-osgi/examples] $mvn clean test
[INFO] Scanning for projects...
[INFO]
[INFO] Building Neo4j OSGi examples
[INFO]task-segment: [clean, test]
[INFO]
[INFO] [clean:clean {execution: default-clean}]
[INFO] Deleting /Users/peterneubauer/code/neo/neo4j-osgi/examples/target
[WARNING] POM for 'biz.aQute:bndlib:pom:1.43.0:test' is invalid.
Its dependencies (if any) will NOT be available to the current build.
[WARNING] POM for 'biz.aQute:bndlib:pom:1.15.0:test' is invalid.
Its dependencies (if any) will NOT be available to the current build.
[INFO] [enforcer:enforce {execution: enforce-maven}]
[INFO] [license:check {execution: check-licenses}]
[INFO] Checking licenses...
[INFO] [resources:resources {execution: default-resources}]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/Users/peterneubauer/code/neo/neo4j-osgi/examples/src/main/resources
[INFO] skip non existing resourceDirectory
/Users/peterneubauer/code/neo/neo4j-osgi/examples/src/main/resources/META-INF
[INFO] Copying 0 resource to META-INF
[INFO] [compiler:compile {execution: default-compile}]
[INFO] No sources to compile
[INFO] [bundle:manifest {execution: bundle-manifest}]
[WARNING] Warning in manifest for
org.neo4j.examples:neo4j-osgi-examples:jar:0.1-SNAPSHOT : Superfluous
export-package instructions: [null.*]
[WARNING] Warning in manifest for
org.neo4j.examples:neo4j-osgi-examples:jar:0.1-SNAPSHOT : Did not find
matching referal for *
[INFO] [resources:testResources {execution: default-testResources}]
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory
/Users/peterneubauer/code/neo/neo4j-osgi/examples/src/test/resources
[INFO] Copying 0 resource to META-INF
[INFO] [compiler:testCompile {execution: default-testCompile}]
[INFO] Compiling 3 source files to
/Users/peterneubauer/code/neo/neo4j-osgi/examples/target/test-classes
[INFO] [surefire:test {execution: default-test}]
[INFO] Surefire report directory:
/Users/peterneubauer/code/neo/neo4j-osgi/examples/target/surefire-reports
---
T E S T S
---
Running org.neo4j.examples.osgi.OSGiTest
300 [main] INFO org.ops4j.pax.exam.spi.DefaultExamSystem - Pax Exam
System (Version: 2.1.0) created.
Jul 17, 2011 5:52:07 PM
org.ops4j.pax.url.mvn.internal.AetherBasedResolver resolve
INFO: Resolved (org.ops4j.pax.exam:pax-exam-container-rbc:jar:2.1.0)
as
/Users/peterneubauer/.m2/repository/org/ops4j/pax/exam/pax-exam-container-rbc/2.1.0/pax-exam-container-rbc-2.1.0.jar
Jul 17, 2011 5:52:07 PM
org.ops4j.pax.url.mvn.internal.AetherBasedResolver resolve
INFO: Resolved (org.ops4j.pax.exam:pax-exam-extender-service:jar:2.1.0)
as
/Users/peterneubauer/.m2/repository/org/ops4j/pax/exam/pax-exam-extender-service/2.1.0/pax-exam-extender-service-2.1.0.jar
Jul 17, 2011 5:52:07 PM
org.ops4j.pax.url.mvn.internal.AetherBasedResolver resolve
INFO: Resolved (org.osgi:org.osgi.compendium:jar:4.2.0) as
/Users/peterneubauer/.m2/repository/org/osgi/org.osgi.compendium/4.2.0/org.osgi.compendium-4.2.0.jar
Jul 17, 2011 5:52:07 PM
org.ops4j.pax.url.mvn.internal.AetherBasedResolver resolve
INFO: Resolved (org.ops4j.pax.logging:pax-logging-api:jar:1.6.2) as
/Users/peterneubauer/.m2/repository/org/ops4j/pax/logging/pax-logging-api/1.6.2/pax-logging-api-1.6.2.jar
Jul 17, 2011 5:52:07 PM
org.ops4j.pax.url.mvn.internal.AetherBasedResolver resolve
INFO: Resolved (org.ops4j.pax.logging:pax-logging-service:jar:1.6.2)
as
/Users/peterneubauer/.m2/repository/org/ops4j/pax/logging/pax-logging-service/1.6.2/pax-logging-service-1.6.2.jar
Jul 17, 2011 5:52:07 PM
org.ops4j.pax.url.mvn.internal.AetherBasedResolver resolve
INFO: Resolved (org.neo4j:neo4j-osgi-bundle:jar:0.1-SNAPSHOT) as
/Users/peterneubauer/.m2/repository/org/neo4j/neo4j-osgi-bundle/0.1-SNAPSHOT/neo4j-osgi-bundle-0.1-SNAPSHOT.jar
[org.ops4j.pax.exam.rbc.internal.Activator] : Name, port or host is
null. So this RBC remains inactive.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Enabling SLF4J API support.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Enabling Jakarta Commons Logging API support.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Enabling Log4J API support.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Enabling Avalon Logger API support.
org.ops4j.pax.logging.pax-logging-api[org.ops4j.pax.logging.internal.Activator]
: Enabling JULI Logger API support.
[Framework Event Dispatcher] DEBUG
org.ops4j.pax.logging.pax-logging-service - BundleEvent STARTED
[Framework Event
Re: [Neo4j] OSGi and Neo4j - superbundle approach
Hi Nicolas, Have you been able to run https://github.com/njouanin/neo4j-osgi/blob/master/examples/src/test/java/org/neo4j/examples/osgi/OSGiTest.java properly? It actually is running on Equinox, so let me try it with Felix ... back in a moment. Also, in order to pull your changes, could you sign up as a committer, and send a mail according to http://wiki.neo4j.org/content/About_Contributor_License_Agreement ? Cheers, /peter neubauer GTalk: neubauer.peter Skype peter.neubauer Phone +46 704 106975 LinkedIn http://www.linkedin.com/in/neubauer Twitter http://twitter.com/peterneubauer http://www.neo4j.org - Your high performance graph database. http://startupbootcamp.org/ - Öresund - Innovation happens HERE. http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party. On Sun, Jul 17, 2011 at 3:02 PM, Nicolas Jouanin wrote: > Hi Peter, > > Thanks for initiating this work on OSGi bundle. > I've forked you repository and tried to make is work using this sample > activator [1]. My goal is to make my activator work when deployed on Felix > with the neo4j superbundle. > First of all, I had troubles to mvn clean install the bundle module. > maven-bundle-plugin is complaining about bundle.namespace property not set. > I've fixed it. > Then when deploying it on Felix I had an error saying that javax.transaction > couldn't be imported. Because this package is already included in > superbundle, I've removed it explicitally from the Import-Package directive. > You can see the changes here [2]. > Then I tried to make my bundle activated using felix and the following > bundles installed : > g! lb > START LEVEL 1 > ID|State |Level|Name > 0|Active | 0|System Bundle (3.2.2) > 1|Active | 1|Neo4j OSGi default bundle (0.1.0.SNAPSHOT) > 2|Active | 1|Apache Felix Bundle Repository (1.6.2) > 3|Active | 1|Apache Felix Gogo Command (0.8.0) > 4|Active | 1|Apache Felix Gogo Runtime (0.8.0) > 5|Active | 1|Apache Felix Gogo Shell (0.8.0) > 6|Resolved | 1|EscapeK - Neo4j OSGi how-to - Embedded client > (0.0.1.SNAPSHOT) > There are only felix core, neo4j superbundle and my test bundle. > So when starting felix I get the following trace : > > Welcome to Apache Felix Gogo > > g! Opening database in embedded mode: 17 juil. 2011 14:57:08 > org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog doInternalRecovery > INFO: Non clean shutdown detected on log [graphdb/nioneo_logical.log.1]. > Recovery started ... > OK > Populating it ... OK > Hello, brave Neo4j world! > ERROR: Bundle org.escapek.neo4j-osgi-howto.embedded-client [6] Error starting > file:/Users/nico/Dev/felix-framework-3.2.2/bundle/org.escapek.neo4j-osgi-howto.embedded-client_0.0.1-SNAPSHOT.jar > (org.osgi.framework.BundleException: Activator start error in bundle > org.escapek.neo4j-osgi-howto.embedded-client [6].) > java.lang.IllegalArgumentException: No index provider 'lucene' found > at > org.neo4j.kernel.IndexManagerImpl.getIndexProvider(IndexManagerImpl.java:76) > at > org.neo4j.kernel.IndexManagerImpl.findIndexConfig(IndexManagerImpl.java:116) > at > org.neo4j.kernel.IndexManagerImpl.getOrCreateIndexConfig(IndexManagerImpl.java:178) > at > org.neo4j.kernel.IndexManagerImpl.getOrCreateNodeIndex(IndexManagerImpl.java:267) > at > org.neo4j.kernel.IndexManagerImpl.forNodes(IndexManagerImpl.java:255) > at > org.neo4j.kernel.IndexManagerImpl.forNodes(IndexManagerImpl.java:249) > at > org.escapek.neo4jOsgiHowto.embeddedClient.EmbeddedClientActivator.start(EmbeddedClientActivator.java:65) > at > org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:629) > at org.apache.felix.framework.Felix.activateBundle(Felix.java:1904) > at org.apache.felix.framework.Felix.startBundle(Felix.java:1822) > at > org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1192) > at > org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:266) > at java.lang.Thread.run(Thread.java:680) > > So everything works fine (open database, populate it and register service) > until the activator tries to register IndexProvider service. I then get the > previous exception which reminds me of something ... > > May be you can have a look at the stack trace to look for the error cause. > > NJ > > > [1] - > https://github.com/njouanin/neo4j-osgi-howto/blob/master/embedded-client/src/main/java/org/escapek/neo4jOsgiHowto/embeddedClient/EmbeddedClientActivator.java > [2] - > https://github.com/njouanin/neo4j-osgi/commit/4ec3eaf58b514d806fde3e48dd942c00fdfddf70 > > Le 5 juil. 2011 à 22:41, Peter Neubauer a écrit : > >> Hi all, >> I just pushed an initial working bundle-build to >> https://github.com/neo4j/neo4j-osgi. You can see the working test in >> https://github.com/neo4j/neo4j-osgi/blob/master/exa
Re: [Neo4j] How can you have transactions with Lucene?
Lucene itself says it doesn't properly support ACID or XA transaction support. (Bug 1149 - Major - New Feature - Reopened - Unresolved). -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/How-can-you-have-transactions-with-Lucene-tp3176918p3176981.html Sent from the Neo4J Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] How can you have transactions with Lucene?
It just waits to write stuff until the transaction committs. Application crashes in between are guarded by the logical log, because all operations are written in the prepare-phase to a logical log (before the commit starts). Neo4j doesn't do any magical tweaking of lucene at all because it isn't necessary... it's fully ACID and transactional with the graph as it is. 2011/7/17 eialbur > Sorry, but how does this thin layer control how Lucene interacts with the > disk (since Lucene isn't transactional). Does it somehow buffer the disk > I/O? Does it coerce Lucene to write to a different file and then update > the > real file at the end of the transaction? > > I'm not trying to be difficult, but I am interested in Lucene for my > project > as well, and need it to behave properly regarding transactions. > > -- > View this message in context: > http://neo4j-community-discussions.438527.n3.nabble.com/How-can-you-have-transactions-with-Lucene-tp3176918p3176951.html > Sent from the Neo4J Community Discussions mailing list archive at > Nabble.com. > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] NoSuchElementException
get/query methods are only guaranteed to return stuff added before the latest call to flush() (it's all there in the javadocs). I think that's why you probably get stale results from get/query. flush(), or at least the first get/query after a flush() is expensive so try to minimize the number of calls to flush(). Do this by having different steps in your batch insertion, f.ex. one where you insert all nodes and index them... call flush() and then insert relationships. 2011/7/15 sulabh choudhury > Well while inserting the nodes I keep a > check batchInserter.nodeExists(node1) so I would guess a node would not be > duplicated. > Otherwise during a traversal I guess duplicate nodes. IIs there a way I can > look into the BtachInsert data before I flush so that I can make sure no > duplicate data has been inserted ? > > On Sat, Jun 25, 2011 at 10:49 AM, sulabh choudhury wrote: > >> Thank you Jim. >> I will wait for 1.5 and hope it resolves the issue :) >> >> >> On Fri, Jun 24, 2011 at 7:11 PM, sulabh choudhury wrote: >> >>> Hi, >>> >>> I just downloaded the neo4j-community-1.4.M04. I stumbled into the >>> java.util.NoSuchElementException: More than one element in >>> org.neo4j.index.impl.lucene.LuceneIndex$1@396cbd97. First element is >>> 'Node[3]' and the second element is 'Node[2]' >>> at >>> org.neo4j.helpers.collection.IteratorUtil.singleOrNull(IteratorUtil.java:114) >>> ~[working_graphGen.jar:na] >>> at >>> org.neo4j.index.impl.lucene.IdToEntityIterator.getSingle(IdToEntityIterator.java:88) >>> ~[working_graphGen.jar:na] >>> at >>> org.neo4j.index.impl.lucene.IdToEntityIterator.getSingle(IdToEntityIterator.java:32) >>> ~[working_graphGen.jar:na] >>> >>> I looked up and found that this is a bug and has been fixed. I was >>> wondering if the fix has been incorporated in the latest Milestone or not? >>> >> >> >> >> -- >> >> -- >> Thanks and Regards, >> Sulabh Choudhury >> >> > > > -- > > -- > > Thanks and Regards, > Sulabh Choudhury > > > -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] How can you have transactions with Lucene?
Sorry, but how does this thin layer control how Lucene interacts with the disk (since Lucene isn't transactional). Does it somehow buffer the disk I/O? Does it coerce Lucene to write to a different file and then update the real file at the end of the transaction? I'm not trying to be difficult, but I am interested in Lucene for my project as well, and need it to behave properly regarding transactions. -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/How-can-you-have-transactions-with-Lucene-tp3176918p3176951.html Sent from the Neo4J Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Index returns all nodes for value="undefined"!
No, there's no such "undefined" constant which will return everything. It definitely sounds like a JavaScript issue, and that it turns nulls into "undefined". And perhaps your code which tries to index null values. 2011/7/16 Aseem Kishore > We're seeing this crazy behavior where our exact index is returning all > nodes if we pass a lookup value of "undefined"! E.g.: > > http://localhost:7474/db/data/index/node/nodes/username/undefined > > Is this a bug in Neo4j / REST API? Or is this a symptom of some bug we have > in our own code? > > It could very well be our code, since we're using JS, which turns undefined > values into the string "undefined" if you try to add it to a string > (definitely a wtfjs candidate), but I wanted to see if others are seeing > this in Neo4j too! > > Thanks, > > Aseem > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] How can you have transactions with Lucene?
Neo4j wraps Lucene in a thin layer which makes it transactional. It's fully ACID and can participate in the same transactions as neo4j operations. It's modeled as its own data source in an XA environment (the neo4j graph is another XA data source) with a logical log for commands so that it's safe for crashes and will participate in the recovery process just like any other XA data source, f.ex. the neo4j graph. 2011/7/17 eialbur > I like Neo4j and am interested in using it in a number of projects (once I > figure out a good way to make Neo play nice in Glassfish) ... but I am > confused about something. I apologize if this is a stupid question. > > My understanding is that Lucene does *not* support transactions. > Supposedly > that lack of transaction support was a prime motivator for the Compass > project, and is a selling point for the Lucene competitor, Sphinx. > > So, how can Neo4j provide real transaction support if it relies on Lucene? > I am confused. > > Thanks > > -- > View this message in context: > http://neo4j-community-discussions.438527.n3.nabble.com/How-can-you-have-transactions-with-Lucene-tp3176918p3176918.html > Sent from the Neo4J Community Discussions mailing list archive at > Nabble.com. > ___ > Neo4j mailing list > User@lists.neo4j.org > https://lists.neo4j.org/mailman/listinfo/user > -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] Officially supported mechanism for J2EE
I really like Neo, but am concerned that there is so little official company support for J2EE. I want to use Neo as a standard data source in J2EE, with container managed transactions, etc. I don't want to use the REST interface - I want a standard data source interface. There does not appear to be an officially provided mechanism to use Neo as a data source in J2EE. I've seen a couple of people post their code to help someone get started, but shouldn't Neo provide a real official sanctioned mechanism? P.S. I don't want this thread to turn into a religious war over REST. If you like REST and want to defend it then please start a thread for that purpose. Please just accept that I don't want to use the REST interface - I want a standard DB interface for my DB. -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/Officially-supported-mechanism-for-J2EE-tp3176938p3176938.html Sent from the Neo4J Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo4j] How can you have transactions with Lucene?
I like Neo4j and am interested in using it in a number of projects (once I figure out a good way to make Neo play nice in Glassfish) ... but I am confused about something. I apologize if this is a stupid question. My understanding is that Lucene does *not* support transactions. Supposedly that lack of transaction support was a prime motivator for the Compass project, and is a selling point for the Lucene competitor, Sphinx. So, how can Neo4j provide real transaction support if it relies on Lucene? I am confused. Thanks -- View this message in context: http://neo4j-community-discussions.438527.n3.nabble.com/How-can-you-have-transactions-with-Lucene-tp3176918p3176918.html Sent from the Neo4J Community Discussions mailing list archive at Nabble.com. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] OSGi and Neo4j - superbundle approach
Hi Peter, Thanks for initiating this work on OSGi bundle. I've forked you repository and tried to make is work using this sample activator [1]. My goal is to make my activator work when deployed on Felix with the neo4j superbundle. First of all, I had troubles to mvn clean install the bundle module. maven-bundle-plugin is complaining about bundle.namespace property not set. I've fixed it. Then when deploying it on Felix I had an error saying that javax.transaction couldn't be imported. Because this package is already included in superbundle, I've removed it explicitally from the Import-Package directive. You can see the changes here [2]. Then I tried to make my bundle activated using felix and the following bundles installed : g! lb START LEVEL 1 ID|State |Level|Name 0|Active |0|System Bundle (3.2.2) 1|Active |1|Neo4j OSGi default bundle (0.1.0.SNAPSHOT) 2|Active |1|Apache Felix Bundle Repository (1.6.2) 3|Active |1|Apache Felix Gogo Command (0.8.0) 4|Active |1|Apache Felix Gogo Runtime (0.8.0) 5|Active |1|Apache Felix Gogo Shell (0.8.0) 6|Resolved |1|EscapeK - Neo4j OSGi how-to - Embedded client (0.0.1.SNAPSHOT) There are only felix core, neo4j superbundle and my test bundle. So when starting felix I get the following trace : Welcome to Apache Felix Gogo g! Opening database in embedded mode: 17 juil. 2011 14:57:08 org.neo4j.kernel.impl.transaction.xaframework.XaLogicalLog doInternalRecovery INFO: Non clean shutdown detected on log [graphdb/nioneo_logical.log.1]. Recovery started ... OK Populating it ... OK Hello, brave Neo4j world! ERROR: Bundle org.escapek.neo4j-osgi-howto.embedded-client [6] Error starting file:/Users/nico/Dev/felix-framework-3.2.2/bundle/org.escapek.neo4j-osgi-howto.embedded-client_0.0.1-SNAPSHOT.jar (org.osgi.framework.BundleException: Activator start error in bundle org.escapek.neo4j-osgi-howto.embedded-client [6].) java.lang.IllegalArgumentException: No index provider 'lucene' found at org.neo4j.kernel.IndexManagerImpl.getIndexProvider(IndexManagerImpl.java:76) at org.neo4j.kernel.IndexManagerImpl.findIndexConfig(IndexManagerImpl.java:116) at org.neo4j.kernel.IndexManagerImpl.getOrCreateIndexConfig(IndexManagerImpl.java:178) at org.neo4j.kernel.IndexManagerImpl.getOrCreateNodeIndex(IndexManagerImpl.java:267) at org.neo4j.kernel.IndexManagerImpl.forNodes(IndexManagerImpl.java:255) at org.neo4j.kernel.IndexManagerImpl.forNodes(IndexManagerImpl.java:249) at org.escapek.neo4jOsgiHowto.embeddedClient.EmbeddedClientActivator.start(EmbeddedClientActivator.java:65) at org.apache.felix.framework.util.SecureAction.startActivator(SecureAction.java:629) at org.apache.felix.framework.Felix.activateBundle(Felix.java:1904) at org.apache.felix.framework.Felix.startBundle(Felix.java:1822) at org.apache.felix.framework.Felix.setActiveStartLevel(Felix.java:1192) at org.apache.felix.framework.StartLevelImpl.run(StartLevelImpl.java:266) at java.lang.Thread.run(Thread.java:680) So everything works fine (open database, populate it and register service) until the activator tries to register IndexProvider service. I then get the previous exception which reminds me of something ... May be you can have a look at the stack trace to look for the error cause. NJ [1] - https://github.com/njouanin/neo4j-osgi-howto/blob/master/embedded-client/src/main/java/org/escapek/neo4jOsgiHowto/embeddedClient/EmbeddedClientActivator.java [2] - https://github.com/njouanin/neo4j-osgi/commit/4ec3eaf58b514d806fde3e48dd942c00fdfddf70 Le 5 juil. 2011 à 22:41, Peter Neubauer a écrit : > Hi all, > I just pushed an initial working bundle-build to > https://github.com/neo4j/neo4j-osgi. You can see the working test in > https://github.com/neo4j/neo4j-osgi/blob/master/examples/src/test/java/org/neo4j/examples/osgi/OSGiTest.java > that waits for the Activator to instantiate and register a > GraphDatabaseService and a LuceneIndex. > > This should be tweaked, but maybe you can use the bundle project > https://github.com/neo4j/neo4j-osgi/blob/master/bundle/pom.xml , which > is just one pom.xml, as a blueprint for your OSGi environment Neo4j > bundle. > > Right now JTA and others are drawn into the bundle, which might be too > much, but I think you get the picture. > > This works even with the Java Service Loader architecture. Feel free > to fork and tune the bundles exports, and add more tests to it. > > WDYT? > > Cheers, > > /peter neubauer > > GTalk: neubauer.peter > Skype peter.neubauer > Phone +46 704 106975 > LinkedIn http://www.linkedin.com/in/neubauer > Twitter http://twitter.com/peterneubauer > > http://www.neo4j.org - Your high performance graph database. > http://startupbootcamp.org/- Öresund - Innovation happens HERE. > http://w
