[Neo] I/O load in Neo during traversals

2009-12-09 Thread Rick Bullotta
When doing some large traversal testing (no writes/updates), I noticed that the neostore.propertystore.db.strings file was seeing a lot of read I/O (as expected) but also a huge amount of write I/O (almost 5X the read I/O rate). Out of curiosity, what is the write activity that needs to occur when

Re: [Neo] Troubleshooting performance/memory issues

2009-12-09 Thread Rick Bullotta
FYI, we experimented with different heap size (1GB), along with different "chunk sizes", and were able to eliminate the heap error and get about a 10X improvement in insert speed. It would be helpful to better understand the interactions of the various Neo startup parameters, transaction buffers,

[Neo] Troubleshooting performance/memory issues

2009-12-09 Thread Rick Bullotta
Hi, all. When trying to load a few hundred thousand nodes & relationships (chunking it in groups of 1000 nodes or so), we are getting an out of memory heap error after 15-20 minutes or so. No big deal, we expanded the heap settings for the JVM. But then we also noticed that the nioneo_logical

[Neo] Noob questions/comments

2009-12-09 Thread Rick Bullotta
Hi, all. Here are a few questions and comments that I'd welcome feedback on : Questions: - If you delete the reference node (id = 0), how can you recreate it? - If you have a number of "loose" or disjoint graphs structured as trees with a single root node, is there a

Re: [Neo] Lucene Index Corruption?

2009-12-09 Thread Adam Rabung
Hi, Lots of questions :) 1. This iterator will just prevent duplicates from being returned from the iterator? If there's a condition (bug in my code) that causes shutdown w/ open transactions, will the Lucene indexes continue to double until they're huge? 2. Would it be possible to detect this s

Re: [Neo] Type metadata in properties/nodes

2009-12-09 Thread Rick Bullotta
Hi, Tobias. Actually, I think we'll use your approach for the "known relationships" and "known types" (there are quite a few in our domain model) in addition to the dynamic approach. Thanks for the help! Rick -Original Message- From: user-boun...@lists.neo4j.org [mailto:user-boun...@lis

Re: [Neo] Lucene Index Corruption?

2009-12-09 Thread Mattias Persson
Hi Adam, We're aware of such problems and I just now committed a fix which basically is a cover-up until those bugs are fixed... the iterable from getNodes() now runs through a filter (lazily before each next()) so your problem should go away. 2009/12/8 Adam Rabung : > Hi, > I've recently run int

Re: [Neo] Type metadata in properties/nodes

2009-12-09 Thread Tobias Ivarsson
I see. I realized that this was what you were after. What I was proposing was that you would know the types for the properties given the type of the node. The types for the nodes in your case would be more abstract, perhaps just defined by the set of properties. I used concrete types in my explanat

Re: [Neo] Type metadata in properties/nodes

2009-12-09 Thread rick . bullotta
Hi, Tobias. Thanks for your thoughts and ideas. My requirement is not only to know the "type" of something, but also to store metadata for "types" so that I can catalog the "property type" of each individual property in a node for a given "type". It's a bit complicated, but

Re: [Neo] Type metadata in properties/nodes

2009-12-09 Thread Tobias Ivarsson
Associating nodes with a type node is a good approach, especially if you want to be able to do queries like "give me all nodes of type X". But for knowing the semantic type of a node when found through a general traversal I prefer to use the navigational context of the node. For example if I have a

Re: [Neo] LuceneIndexBatchInserter doubt

2009-12-09 Thread Núria Trench
Hi Mattias, I have already done it 10 minutes ago. If you need an example to see the format of the 4 csv files, I can send it to you. Thanks again, Núria. 2009/12/9 Mattias Persson > Oh ok, It could be our attachments filter / security or something... > could you try to mail them to me directl

Re: [Neo] LuceneIndexBatchInserter doubt

2009-12-09 Thread Mattias Persson
Oh ok, It could be our attachments filter / security or something... could you try to mail them to me directly at matt...@neotechnology.com ? 2009/12/9 Núria Trench : > Hi Mattias, > > In my last e-mail I have attached the sample code, haven't you received it? > I will try to attach it again. > >

Re: [Neo] LuceneIndexBatchInserter doubt

2009-12-09 Thread Núria Trench
Hi Mattias, In my last e-mail I have attached the sample code, haven't you received it? I will try to attach it again. Núria. 2009/12/9 Mattias Persson > Hi again, Núria (it was I, Mattias who asked for the sample code). > Well... the fact that you parse 4 csv files doesn't really help me > se

Re: [Neo] LuceneIndexBatchInserter doubt

2009-12-09 Thread Mattias Persson
Hi again, Núria (it was I, Mattias who asked for the sample code). Well... the fact that you parse 4 csv files doesn't really help me setup a test for this... I mean how can I know that my test will be similar to yours? Would it be ok to attach your code/csv files as well? / Mattias 2009/12/9 Núr

Re: [Neo] LuceneIndexBatchInserter doubt

2009-12-09 Thread Núria Trench
Hi Todd, The sample code creates nodes and relationships by parsing 4 csv files. Thank you for trying to trigger this behaviour with this sample. Núria 2009/12/9 Mattias Persson > Could you provide me with some sample code which can trigger this > behaviour with the latest index-util-0.9-SNAPS

Re: [Neo] LuceneIndexBatchInserter doubt

2009-12-09 Thread Mattias Persson
Could you provide me with some sample code which can trigger this behaviour with the latest index-util-0.9-SNAPSHOT Núria? 2009/12/9 Núria Trench : > Todd, > > I haven't the same problem. In my case, after indexing all the > attributes/properties of each node, the application creates all the edges

Re: [Neo] LuceneIndexBatchInserter doubt

2009-12-09 Thread Núria Trench
Todd, I haven't the same problem. In my case, after indexing all the attributes/properties of each node, the application creates all the edges by looking up the tail node and the head node. So, it calls the method "org.neo4j.util.index. LuceneIndexBatchInserterImpl.getSingleNode" which returns -1