[Neo] Jruby on rails
Hi all, Has anyone tried embedding neo4j with jruby on rails? Any suggestions? Regards -- Ajay Sharma ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Looking at DB contents
I don't follow entirely... you mean that the shell held locks on nodes which made another traverser (executed via script or something) wait? At the moment every command in the shell is wrapped in its own transaction. 2009/9/7 Todd Stavish toddstav...@gmail.com: Hi Martin, Anders, The standalone version of neoclipse works like a charm, thanks. I was surprised that the shell locked out a non-updating traverse application that I wrote. Where do I explicitly declare locking? My natural assumption was in beginTx( kindOfLock ). Thanks, Todd On Mon, Sep 7, 2009 at 1:17 AM, Martin Filipczyk i...@mupitu.de wrote: Hi Todd, have you already tried the neoclipse standalone version? -Martin Todd Stavish schrieb: Thanks I'll give the shell a try. Was it crashing on opening the database or what happened? It shows the reference node and then just hangs. I am using Galileo though and maybe the wrong version of GEF. So no big deal. -Todd /anders -- Anders Nawroth [and...@neotechnology.com] GTalk, Skype: anders.nawroth Phone: +46 737 894 163 http://twitter.com/nawroth http://blog.nawroth.com/ ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Neo Technology, www.neotechnology.com ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Jruby on rails
Hi Ajay, You can find that info over at the JRuby project, github.com/andreasronge/neo4j/tree/master /peter On Sep 9, 2009 11:19 AM, Ajay Sharma sharmaa...@gmail.com wrote: Hi all, Has anyone tried embedding neo4j with jruby on rails? Any suggestions? Regards -- Ajay Sharma ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Jruby on rails
hi! Peter Neubauer wrote: You can find that info over at the JRuby project, github.com/andreasronge/neo4j/tree/master And they also have a google group: http://groups.google.com/group/neo4jrb/ /anders ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Looking at DB contents
Hi Mattias, Yes, that is correct. I think fundamentally I do not understand how your locking works. I assumed that both the shell and traverser would have read only locks. However, I left the shell open and the traverser was blocked. I did not do anything in the shell other than read-only commands. -Todd On Wed, Sep 9, 2009 at 5:47 AM, Mattias Persson matt...@neotechnology.comwrote: I don't follow entirely... you mean that the shell held locks on nodes which made another traverser (executed via script or something) wait? At the moment every command in the shell is wrapped in its own transaction. 2009/9/7 Todd Stavish toddstav...@gmail.com: Hi Martin, Anders, The standalone version of neoclipse works like a charm, thanks. I was surprised that the shell locked out a non-updating traverse application that I wrote. Where do I explicitly declare locking? My natural assumption was in beginTx( kindOfLock ). Thanks, Todd On Mon, Sep 7, 2009 at 1:17 AM, Martin Filipczyk i...@mupitu.de wrote: Hi Todd, have you already tried the neoclipse standalone version? -Martin Todd Stavish schrieb: Thanks I'll give the shell a try. Was it crashing on opening the database or what happened? It shows the reference node and then just hangs. I am using Galileo though and maybe the wrong version of GEF. So no big deal. -Todd /anders -- Anders Nawroth [and...@neotechnology.com] GTalk, Skype: anders.nawroth Phone: +46 737 894 163 http://twitter.com/nawroth http://blog.nawroth.com/ ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Neo Technology, www.neotechnology.com ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
[Neo] All your node are belong to us
Hello,
I was running some tests with 1.0-b10-SNAPSHOT (trunk) and if the JVM is not
closed properly (ex: server crashes) the database cannot be recovered.
The test will run ok with 1.0-b8 but the recover procedure is very slow.
public class NeoShutdownTest
{
public static void main(String[] args)
{
NeoService neo = new EmbeddedNeo( /var/neo );
Transaction tx = neo.beginTx();
try
{
Node referenceNode = neo.getReferenceNode();
for ( int i = 0; i 10; i++ )
referenceNode.createRelationshipTo(
neo.createNode(),
DynamicRelationshipType.withName( TEST ) );
tx.success();
System.out.println( success );
}
catch ( Exception e )
{
tx.failure();
e.printStackTrace();
}
finally
{
tx.finish();
}
//neo.shutdown();
}
}
___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Looking at DB contents
That's strange since the shell only wraps commands in transactions. This means that leaving the shell open, just standing on a node doesn't create/hold a lock or transaction whatsoever. You could debug this kind of problem (when it appears) by making a complete thread dump and you'll see which threads are holding neo4j locks, to see if it's really the shell which has locks or not. 2009/9/9 Todd Stavish toddstav...@gmail.com: Hi Mattias, Yes, that is correct. I think fundamentally I do not understand how your locking works. I assumed that both the shell and traverser would have read only locks. However, I left the shell open and the traverser was blocked. I did not do anything in the shell other than read-only commands. -Todd On Wed, Sep 9, 2009 at 5:47 AM, Mattias Persson matt...@neotechnology.comwrote: I don't follow entirely... you mean that the shell held locks on nodes which made another traverser (executed via script or something) wait? At the moment every command in the shell is wrapped in its own transaction. 2009/9/7 Todd Stavish toddstav...@gmail.com: Hi Martin, Anders, The standalone version of neoclipse works like a charm, thanks. I was surprised that the shell locked out a non-updating traverse application that I wrote. Where do I explicitly declare locking? My natural assumption was in beginTx( kindOfLock ). Thanks, Todd On Mon, Sep 7, 2009 at 1:17 AM, Martin Filipczyk i...@mupitu.de wrote: Hi Todd, have you already tried the neoclipse standalone version? -Martin Todd Stavish schrieb: Thanks I'll give the shell a try. Was it crashing on opening the database or what happened? It shows the reference node and then just hangs. I am using Galileo though and maybe the wrong version of GEF. So no big deal. -Todd /anders -- Anders Nawroth [and...@neotechnology.com] GTalk, Skype: anders.nawroth Phone: +46 737 894 163 http://twitter.com/nawroth http://blog.nawroth.com/ ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Neo Technology, www.neotechnology.com ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Neo Technology, www.neotechnology.com ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Looking at DB contents
Btw. are you using the shell to connect remotely to a running NeoService or or you using it to start a local one? If you're starting a local (inside the shell JVM) you might have the problem that the neo4j files are locked so that you cannot start up your traverser application... is that the case? 2009/9/9 Mattias Persson matt...@neotechnology.com: That's strange since the shell only wraps commands in transactions. This means that leaving the shell open, just standing on a node doesn't create/hold a lock or transaction whatsoever. You could debug this kind of problem (when it appears) by making a complete thread dump and you'll see which threads are holding neo4j locks, to see if it's really the shell which has locks or not. 2009/9/9 Todd Stavish toddstav...@gmail.com: Hi Mattias, Yes, that is correct. I think fundamentally I do not understand how your locking works. I assumed that both the shell and traverser would have read only locks. However, I left the shell open and the traverser was blocked. I did not do anything in the shell other than read-only commands. -Todd On Wed, Sep 9, 2009 at 5:47 AM, Mattias Persson matt...@neotechnology.comwrote: I don't follow entirely... you mean that the shell held locks on nodes which made another traverser (executed via script or something) wait? At the moment every command in the shell is wrapped in its own transaction. 2009/9/7 Todd Stavish toddstav...@gmail.com: Hi Martin, Anders, The standalone version of neoclipse works like a charm, thanks. I was surprised that the shell locked out a non-updating traverse application that I wrote. Where do I explicitly declare locking? My natural assumption was in beginTx( kindOfLock ). Thanks, Todd On Mon, Sep 7, 2009 at 1:17 AM, Martin Filipczyk i...@mupitu.de wrote: Hi Todd, have you already tried the neoclipse standalone version? -Martin Todd Stavish schrieb: Thanks I'll give the shell a try. Was it crashing on opening the database or what happened? It shows the reference node and then just hangs. I am using Galileo though and maybe the wrong version of GEF. So no big deal. -Todd /anders -- Anders Nawroth [and...@neotechnology.com] GTalk, Skype: anders.nawroth Phone: +46 737 894 163 http://twitter.com/nawroth http://blog.nawroth.com/ ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Neo Technology, www.neotechnology.com ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Neo Technology, www.neotechnology.com -- Mattias Persson, [matt...@neotechnology.com] Neo Technology, www.neotechnology.com ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo] Neo4J Scalability
Todd,
I successfully ran the code resulting in roughly 190M nodes/properties
and 390M relationships injected in 35min (the condition in one of your
for loops probably has a typo, should have been personnumberOfPeople
and then more relationships would have been created). This was on a
Linux machine with 8G RAM running with a 5G heap and the following
configuration:
neostore.nodestore.db.mapped_memory=1700M
neostore.relationshipstore.db.mapped_memory=2000M
neostore.propertystore.db.mapped_memory=100M
neostore.propertystore.db.index.mapped_memory=1M
neostore.propertystore.db.index.keys.mapped_memory=1M
neostore.propertystore.db.strings.mapped_memory=250M
neostore.propertystore.db.arrays.mapped_memory=130M
I first tried on my laptop but ran out of disk space (with a similar
exception you got) since the full load requires 41GB of free space.
Could you please try the injection again (using similar configuration
and verifying you have enough disk space)?
Also (maybe not important since this is just a test but) relationships
can be treated as bi-directional/non directed using the enum constant
Direction.BOTH, so the two relationships a-KNOWS-b and b-KNOWS-a
can be represented with one a-KNOWS-b.
Regards,
-Johan
On Wed, Sep 9, 2009 at 3:11 AM, Todd Stavishtoddstav...@gmail.com wrote:
He Johan,
What OS do you run this on? The FileChannel.write fail with an
exception saying buffer is too large is new to me.
OSX. Snow Leopard.
java -version
java version 1.6.0_15
Java(TM) SE Runtime Environment (build 1.6.0_15-b03-219)
Java HotSpot(TM) 64-Bit Server VM (build 14.1-b02-90, mixed mode)
I can try it on centos too. I was going to try to run in it in EC2,
but lost momentum and ideas to try.
Try decrease the amount of memory you give the store files (if you
have modified default configuration) and make sure you at least have
disk space to write out 1/10th of the dedicated memory for each store
file.
Plenty of disk space. Do you mean pre-allocate the database file? How
do you do that?
I will run some tests on this code first thing in the morning to see
if I can reproduce it.
Thanks. I will send you the input files. And other code.
Regards,
Todd
-Johan
On Tue, Sep 8, 2009 at 5:45 PM, Todd Stavishtoddstav...@gmail.com wrote:
Hello All,
I hope that I haven't been too demanding of this list. I am pushing a
tight deadline. I can't seem to get Neo4J to scale above 1.5 million
nodes. The code is simple, just create a node, give it one property.
Then go back and randomly create relationships. I moved to the
BatchInserter too. Is there a way to use more than one DB? Is it a
file size limitation? Can I commit incrementally somehow? I looked at
the BatchInterter source, and there doesn't seem to be anything that I
can do.
Exceptions and demo code below.
Thanks,
Todd
Created 1516633 nodes in 33 seconds
Exception in thread main
org.neo4j.impl.nioneo.store.StoreFailureException: Unable to write
record[1419120] @[188742960]
at
org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:111)
at
org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.force(AbstractPersistenceWindow.java:123)
at
org.neo4j.impl.nioneo.store.PersistenceWindowPool.flushAll(PersistenceWindowPool.java:229)
at
org.neo4j.impl.nioneo.store.PersistenceWindowPool.close(PersistenceWindowPool.java:203)
at
org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:503)
at
org.neo4j.impl.nioneo.store.PropertyStore.closeStorage(PropertyStore.java:106)
at
org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500)
at
org.neo4j.impl.nioneo.store.NeoStore.closeStorage(NeoStore.java:81)
at
org.neo4j.impl.nioneo.store.CommonAbstractStore.close(CommonAbstractStore.java:500)
at
org.neo4j.impl.batchinsert.BatchInserterImpl.shutdown(BatchInserterImpl.java:311)
at
com.palantir.pathfind.GraphIngestBatch.main(GraphIngestBatch.java:78)
Caused by: java.io.IOException: Result too large
at sun.nio.ch.FileDispatcher.pwrite0(Native Method)
at sun.nio.ch.FileDispatcher.pwrite(FileDispatcher.java:45)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:100)
at sun.nio.ch.IOUtil.write(IOUtil.java:75)
at sun.nio.ch.FileChannelImpl.write(FileChannelImpl.java:651)
at
org.neo4j.impl.nioneo.store.AbstractPersistenceWindow.writeOut(AbstractPersistenceWindow.java:105)
public class GraphIngestBatch
{
public static void main(String[] args)
{
try
{
// Create graph database service
BatchInserter neo = new BatchInserterImpl(graphDB/,
BatchInserterImpl.loadProperties(neo.props));
// Read raw text for name data
FileReader namesFile = new
Re: [Neo] All your node are belong to us (Johan Svensson)
If I kill the JVM the stackTrace of the exception thrown by recovery process looks like this Sep 9, 2009 3:24:09 PM org.neo4j.impl.transaction.xaframework.XaLogicalLog doInternalRecovery INFO: Non clean shutdown detected on log [\var\neo\nioneo_logical.log.1]. Recovery started ... org.neo4j.impl.nioneo.store.StoreFailureException: Position[9] requested for operation is high id[2] or store is flagged as dirty[true] at org.neo4j.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:442) at org.neo4j.impl.nioneo.store.RelationshipStore.getRecord(RelationshipStore.java:89) at org.neo4j.impl.nioneo.xa.NeoTransaction.connectRelationship(NeoTransaction.java:1363) at org.neo4j.impl.nioneo.xa.NeoTransaction.relationshipCreate(NeoTransaction.java:1344) at org.neo4j.impl.nioneo.xa.NeoStoreXaConnection$RelationshipEventConsumerImpl.createRelationship(NeoStoreXaConnection.java:252) at org.neo4j.impl.nioneo.xa.NioNeoDbPersistenceSource$NioNeoDbResourceConnection.relationshipCreate(NioNeoDbPersistenceSource.java:349) at org.neo4j.impl.persistence.PersistenceManager.relationshipCreate(PersistenceManager.java:147) at org.neo4j.impl.core.NodeManager.createRelationship(NodeManager.java:306) at org.neo4j.impl.core.NodeImpl.createRelationshipTo(NodeImpl.java:357) at org.neo4j.impl.core.NodeProxy.createRelationshipTo(NodeProxy.java:177) ___ Neo mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
