Re: [Neo] Neo4j RDF component download path missing?

[francis adediran]

Yeah, that helps. Thanks a lot

Francis,
we have not released the surrounding components that are not contained
in the APOC as compliant snapshots. That means, you should check out
trunk from svn for the interesting components with e.g.
svn co https://svn.neo4j.org/components/neo-rdf/trunk/ neo-rdf
mvn install
And then use that deployed local SNAPSHOT version (0.6-SNAPSHOT).
Is that ok for you?
Cheers,
/peter neubauer
COO and Sales, Neo Technology
GTalk:  neubauer.peter
Skype   peter.neubauer
Phone   +46 704 106975
LinkedIn   http://www.linkedin.com/in/neubauer
Twitter  http://twitter.com/peterneubauer
http://www.neo4j.org- Your high performance graph database.
http://gremlin.tinkerpop.com- PageRank in 2 lines of code.




On Wed, Jan 27, 2010 at 3:28 PM, francis adediran wrote:

> Hi,
>  I'm trying to use neo4j rdf component. I have downloaded the zipped
> archive from the download page http://neo4j.org/download/. included them
> in my class path. and every thing is working fine
>
> now i would like to work with rdf data
>
> but i'm unable to use the Rdf store and VerboseQuadStore as
> explained here  http://components.neo4j.org/neo4j-rdf/
> *
>
> import
> *org.neo4j.api.core.*;*
>
> import
> *org.neo4j.util.index.IndexService;*
>
> import
> *org.neo4j.util.index.LuceneIndexService;**
>
> import *org.neo4j.rdf*.*;
>
> IndexService indexService = *new* LuceneIndexService( neo );
>
> *RdfStore* store = *new* *VerboseQuadStore*( neo, indexService );
>
> (underlined words are errors)
>
> Is there another jar file or archive that i need to download? or is my
> imports wrong?
>
> any suggestion would be much appreciated
>
> --
>
> Francis
>
___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo] Relation properties

[Rick Bullotta]

Why do you need to set your own ID's?  You can use the relationship's
built-in as your index key (it is guaranteed to be unique) and
getRelationshipByID to retrieve the relationship.  You'd need to very
slightly extend the LuceneIndex stuff to deal with relationships, but it
shouldn't be difficult.

-Original Message-
From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On
Behalf Of Anton Popov
Sent: Friday, January 29, 2010 9:31 AM
To: Neo user discussions
Subject: Re: [Neo] Relation properties

And as far as I can see there is no way to set my own id'es for Relations?
Only Nodes has such functionality - am I right?

-- Forwarded message --
From: Anton Popov 
Date: 2010/1/29
Subject: Re: [Neo] Relation properties
To: Neo user discussions 


Hello Mattias,

Got it. Thanks.
Is there any RoadMap available?

2010/1/29 Mattias Persson 

Unfortunately you cannot index Relationships, but it's a (frquently
> requested) feature we're planning to add.
>
> 2010/1/29 Anton Popov :
> > Hello everybody,
> >
> > A related question is can I index both Nodes & Relations?
> > As I can see LuceneIndexBatchInserter.index( long node, String key,
> Object
> > value ) applies for Nodes only.
> >
> > 2010/1/28 Mattias Persson 
> >
> >> Not at all, you're not the first one to overlook that inheritance!
> >>
> >> 2010/1/28, Anton Popov :
> >> > Thanks for a quick answer, Mattias.
> >> >
> >> > Sorry for a stupid questions, just haven't noticed the inheritance of
> >> > Relationship interface.
> >> >
> >> > On 28 January 2010 19:43, Mattias Persson 
> >> wrote:
> >> >
> >> >> Relationship has all the property methods, just like Node... they're
> >> >> actually on PropertyContainer (a common super interface to Node and
> >> >> Relationship) see
> >> >> http://api.neo4j.org/current/org/neo4j/graphdb/Relationship.html and
> >> >>
> http://api.neo4j.org/current/org/neo4j/graphdb/PropertyContainer.html
> >> >> for more information.
> >> >>
> >> >> So to manage the cost in a dijkstra algorithm you can use whatever
> >> >> property key, f.ex. "weight", like this:
> >> >>
> >> >>Relationship rel = ... // any relationship
> >> >>rel.setProperty( "weight", 4 );
> >> >>
> >> >> And if you're using the Dijkstra algorithms in the
"neo4j-graph-algo"
> >> >> component, http://components.neo4j.org/neo4j-graph-algo/ , you can
> >> >> instantiate it with a CostEvaluator instance something like this:
> >> >>
> >> >>new CostEvaluator()
> >> >>{
> >> >>CostType getCost( Relationship relationship, boolean
backwards
> )
> >> >>{
> >> >>/ TODO Handle backwards flag?
> >> >>return ( Integer ) relationship.getProperty( "weight", 1
> );
> >> >>}
> >> >>}
> >> >>
> >> >> ...and pass it to your Dijkstra algo instance.
> >> >>
> >> >> Is this what you were looking for?
> >> >>
> >> >> 2010/1/28 Anton Popov :
> >> >> > Hello everyone,
> >> >> >
> >> >> > Is there any possibility to assign properties to Relations?
> >> >> > The matter is I want to try Dijkstra algorithm performance on some
> >> >> weighted
> >> >> > directed graph and don't know where to put weight actually...
> >> >> >
> >> >> > --
> >> >> > Best regards,
> >> >> > Popov Anton
> >> >> > ___
> >> >> > Neo mailing list
> >> >> > User@lists.neo4j.org
> >> >> > https://lists.neo4j.org/mailman/listinfo/user
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Mattias Persson, [matt...@neotechnology.com]
> >> >> Neo Technology, www.neotechnology.com
> >> >> ___
> >> >> Neo mailing list
> >> >> User@lists.neo4j.org
> >> >> https://lists.neo4j.org/mailman/listinfo/user
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Best regards,
> >> > Popov Anton
> >> > ___
> >> > Neo mailing list
> >> > User@lists.neo4j.org
> >> > https://lists.neo4j.org/mailman/listinfo/user
> >> >
> >>
> >>
> >> --
> >> Mattias Persson, [matt...@neotechnology.com]
> >> Neo Technology, www.neotechnology.com
> >> ___
> >> Neo mailing list
> >> User@lists.neo4j.org
> >> https://lists.neo4j.org/mailman/listinfo/user
> >>
> >
> >
> >
> > --
> > Best regards,
> > Popov Anton
> > ___
> > Neo mailing list
> > User@lists.neo4j.org
> > https://lists.neo4j.org/mailman/listinfo/user
> >
>
>
>
> --
> Mattias Persson, [matt...@neotechnology.com]
> Neo Technology, www.neotechnology.com
> ___
> Neo mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>



-- 
Best regards,
Popov Anton



-- 
Best regards,
Popov Anton
___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4

Re: [Neo] Relation properties

[Mattias Persson]

2010/1/29 Anton Popov :
> And as far as I can see there is no way to set my own id'es for Relations?
> Only Nodes has such functionality - am I right?
No you can't and "setting your own ids" shouldn't be misused since the
id of a node is actually relative to where in the file (on disc) it is
located. So by spreading ids and "misusing" the id space you could end
up having very large store files where most of it is wasted space.

What is your particular use case for indexing relationships? Maybe you
could get some tips on modelling it in some other way here on the
mailing list if you asked!
>
> -- Forwarded message --
> From: Anton Popov 
> Date: 2010/1/29
> Subject: Re: [Neo] Relation properties
> To: Neo user discussions 
>
>
> Hello Mattias,
>
> Got it. Thanks.
> Is there any RoadMap available?
>
> 2010/1/29 Mattias Persson 
>
> Unfortunately you cannot index Relationships, but it's a (frquently
>> requested) feature we're planning to add.
>>
>> 2010/1/29 Anton Popov :
>> > Hello everybody,
>> >
>> > A related question is can I index both Nodes & Relations?
>> > As I can see LuceneIndexBatchInserter.index( long node, String key,
>> Object
>> > value ) applies for Nodes only.
>> >
>> > 2010/1/28 Mattias Persson 
>> >
>> >> Not at all, you're not the first one to overlook that inheritance!
>> >>
>> >> 2010/1/28, Anton Popov :
>> >> > Thanks for a quick answer, Mattias.
>> >> >
>> >> > Sorry for a stupid questions, just haven't noticed the inheritance of
>> >> > Relationship interface.
>> >> >
>> >> > On 28 January 2010 19:43, Mattias Persson 
>> >> wrote:
>> >> >
>> >> >> Relationship has all the property methods, just like Node... they're
>> >> >> actually on PropertyContainer (a common super interface to Node and
>> >> >> Relationship) see
>> >> >> http://api.neo4j.org/current/org/neo4j/graphdb/Relationship.html and
>> >> >>
>> http://api.neo4j.org/current/org/neo4j/graphdb/PropertyContainer.html
>> >> >> for more information.
>> >> >>
>> >> >> So to manage the cost in a dijkstra algorithm you can use whatever
>> >> >> property key, f.ex. "weight", like this:
>> >> >>
>> >> >>    Relationship rel = ... // any relationship
>> >> >>    rel.setProperty( "weight", 4 );
>> >> >>
>> >> >> And if you're using the Dijkstra algorithms in the "neo4j-graph-algo"
>> >> >> component, http://components.neo4j.org/neo4j-graph-algo/ , you can
>> >> >> instantiate it with a CostEvaluator instance something like this:
>> >> >>
>> >> >>    new CostEvaluator()
>> >> >>    {
>> >> >>        CostType getCost( Relationship relationship, boolean backwards
>> )
>> >> >>        {
>> >> >>            / TODO Handle backwards flag?
>> >> >>            return ( Integer ) relationship.getProperty( "weight", 1
>> );
>> >> >>        }
>> >> >>    }
>> >> >>
>> >> >> ...and pass it to your Dijkstra algo instance.
>> >> >>
>> >> >> Is this what you were looking for?
>> >> >>
>> >> >> 2010/1/28 Anton Popov :
>> >> >> > Hello everyone,
>> >> >> >
>> >> >> > Is there any possibility to assign properties to Relations?
>> >> >> > The matter is I want to try Dijkstra algorithm performance on some
>> >> >> weighted
>> >> >> > directed graph and don't know where to put weight actually...
>> >> >> >
>> >> >> > --
>> >> >> > Best regards,
>> >> >> > Popov Anton
>> >> >> > ___
>> >> >> > Neo mailing list
>> >> >> > User@lists.neo4j.org
>> >> >> > https://lists.neo4j.org/mailman/listinfo/user
>> >> >> >
>> >> >>
>> >> >>
>> >> >>
>> >> >> --
>> >> >> Mattias Persson, [matt...@neotechnology.com]
>> >> >> Neo Technology, www.neotechnology.com
>> >> >> ___
>> >> >> Neo mailing list
>> >> >> User@lists.neo4j.org
>> >> >> https://lists.neo4j.org/mailman/listinfo/user
>> >> >>
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Best regards,
>> >> > Popov Anton
>> >> > ___
>> >> > Neo mailing list
>> >> > User@lists.neo4j.org
>> >> > https://lists.neo4j.org/mailman/listinfo/user
>> >> >
>> >>
>> >>
>> >> --
>> >> Mattias Persson, [matt...@neotechnology.com]
>> >> Neo Technology, www.neotechnology.com
>> >> ___
>> >> Neo mailing list
>> >> User@lists.neo4j.org
>> >> https://lists.neo4j.org/mailman/listinfo/user
>> >>
>> >
>> >
>> >
>> > --
>> > Best regards,
>> > Popov Anton
>> > ___
>> > Neo mailing list
>> > User@lists.neo4j.org
>> > https://lists.neo4j.org/mailman/listinfo/user
>> >
>>
>>
>>
>> --
>> Mattias Persson, [matt...@neotechnology.com]
>> Neo Technology, www.neotechnology.com
>> ___
>> Neo mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
>
>
>
> --
> Best regards,
> Popov Anton
>
>
>
> --
> Best regards,
> Popov Anton
> ___
> Neo mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/

Re: [Neo] Performance/storage considerations - best practice question

[Craig Taverner]

> > Is there any recommendation on when to use additional node/relationship
> > connections versus additional node properties?
>
> It depends of use case but go with relationships if it "enriches" the
> graph with information that makes sense.
>

And this related to another aspect, based on the balance between two
factors:

   - Total data size (the graph approach increases the primitives count
   which can be an issue for scalability and performance)
   - Need to perform graph-like operations on the data

I'm also thinking about exactly this issue because it is related to the idea
of storing map data in the graph. Most databases store geographic objects as
blobs that do not benefit from the database itself, and include internal
opaque facilities for operating on that data. But with neo4j the option
exists to store this as a sub-graph, exposing the structure and operations
to the developer (for better and worse :-). See 'Neo4j
Spatial'
for some initial discussions on this.

So, in my opinion the fact that you said the data would be written
infrequently seems to imply that you will not be making much use of the
graph nature of the data, and then you might as well store it as properties
and benefit from fewer primitives. I see Johan also suggested a blob /
array. I guess it could even be a byte[] from serializing the entire hash.
___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo] Possible causes for: org.apache.lucene.index.MergePolicy$MergeException

[Mattias Persson]

Now it's available at
http://m2.neo4j.org/org/neo4j/neo4j-index/1.0-SNAPSHOT/neo4j-index-1.0-20100129.143758-4.jar

2010/1/29 Mattias Persson :
> 2010/1/29 Symeon (Akis) Papadopoulos :
>>
>>> I just fixed the bug and your benchmarks run fine w/o any modification
>>> on my machine, so they should run fine on your end too!
>>>
>> Thank you Mattias!
>> So, in the end it was a neo4j-index bug? Thus I assume we don't need to
>> modify our code (of course we'll take into consideration your previous
>> recommendations for using the BatchInserter, but for the sake of it, we
>> would also like to run the benchmark in transactional mode as well).
> Yep, there shouldn't be any need to modify your code at all. Yep, I
> understand if you'd like to benchmark the transactional mode as well!
>>
>>> The bug is just committed so the latest build will be available within
>>> an hour. I see that you aren't using maven, am I right? In that case
>>> you can download it from
>>> http://m2.neo4j.org/org/neo4j/neo4j-index/1.0-SNAPSHOT/ (look for .jar
>>> files around this date/time + 1 hour or something). Or you could check
>>> out the code and build it yourself:
>>>
>>>   svn co https://svn.neo4j.org/components/index/trunk
>>>
>> Out of the two built jars of 29th Jan 2010, I guess we should use the
>> most recent one (neo4j-index-1.0-20100129.113202-3.jar), right?
> Hmm, I don't think it have been deployed yet... so wait a little
> longer for this change to be deployed. We have a little "lag" in
> deployed jar files, which is kind of annoying in situations like this
> :)
>>
>>
>>> You could update to the latest neo4j-kernel as well, but maybe that's
>>> not necessary though.
>>>
>> We'll first try out to update just the neo-index.
>> If we face more problems we'll proceed with the kernel update as well.
>>
>> Thanks again for the prompt response!
> It should be me thanking you, you gave me a complete test case
> triggering a bug... very nice
>>
>>> 2010/1/29 Mattias Persson :
>>>
 2010/1/29 Symeon (Akis) Papadopoulos :

> Mattias Persson wrote:
>
>> Another problem I see is that you're having too granular transactions
>> which will slow down the insertion process quite a bit. Try grouping a
>> couple of thousands operations in one transaction and you'll see a
>> performance boost!
>>
>> FYI: I can trigger the problem you were having with lucene "too many
>> open files" issue. And I'm almost 100% sure that it will be resolved
>> if you increase the span of your transactions.
>>
>>
> That's what we are already doing through the class SimpleBatchTxManager
> (in package org.neo4j.util). We group read and write transactions in
> batches of some thousands. Isn't this correct?
>
 You're right, sorry for missing that.

 However I just found the bug which causes this... hang on, I'll fix it
 in an hour or two!

> ___
> Neo mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
>

 --
 Mattias Persson, [matt...@neotechnology.com]
 Neo Technology, www.neotechnology.com


>>>
>>>
>>>
>>>
>>
>> ___
>> Neo mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
>
>
>
> --
> Mattias Persson, [matt...@neotechnology.com]
> Neo Technology, www.neotechnology.com
>



-- 
Mattias Persson, [matt...@neotechnology.com]
Neo Technology, www.neotechnology.com
___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo] Relation properties

[Anton Popov]

And as far as I can see there is no way to set my own id'es for Relations?
Only Nodes has such functionality - am I right?

-- Forwarded message --
From: Anton Popov 
Date: 2010/1/29
Subject: Re: [Neo] Relation properties
To: Neo user discussions 


Hello Mattias,

Got it. Thanks.
Is there any RoadMap available?

2010/1/29 Mattias Persson 

Unfortunately you cannot index Relationships, but it's a (frquently
> requested) feature we're planning to add.
>
> 2010/1/29 Anton Popov :
> > Hello everybody,
> >
> > A related question is can I index both Nodes & Relations?
> > As I can see LuceneIndexBatchInserter.index( long node, String key,
> Object
> > value ) applies for Nodes only.
> >
> > 2010/1/28 Mattias Persson 
> >
> >> Not at all, you're not the first one to overlook that inheritance!
> >>
> >> 2010/1/28, Anton Popov :
> >> > Thanks for a quick answer, Mattias.
> >> >
> >> > Sorry for a stupid questions, just haven't noticed the inheritance of
> >> > Relationship interface.
> >> >
> >> > On 28 January 2010 19:43, Mattias Persson 
> >> wrote:
> >> >
> >> >> Relationship has all the property methods, just like Node... they're
> >> >> actually on PropertyContainer (a common super interface to Node and
> >> >> Relationship) see
> >> >> http://api.neo4j.org/current/org/neo4j/graphdb/Relationship.html and
> >> >>
> http://api.neo4j.org/current/org/neo4j/graphdb/PropertyContainer.html
> >> >> for more information.
> >> >>
> >> >> So to manage the cost in a dijkstra algorithm you can use whatever
> >> >> property key, f.ex. "weight", like this:
> >> >>
> >> >>Relationship rel = ... // any relationship
> >> >>rel.setProperty( "weight", 4 );
> >> >>
> >> >> And if you're using the Dijkstra algorithms in the "neo4j-graph-algo"
> >> >> component, http://components.neo4j.org/neo4j-graph-algo/ , you can
> >> >> instantiate it with a CostEvaluator instance something like this:
> >> >>
> >> >>new CostEvaluator()
> >> >>{
> >> >>CostType getCost( Relationship relationship, boolean backwards
> )
> >> >>{
> >> >>/ TODO Handle backwards flag?
> >> >>return ( Integer ) relationship.getProperty( "weight", 1
> );
> >> >>}
> >> >>}
> >> >>
> >> >> ...and pass it to your Dijkstra algo instance.
> >> >>
> >> >> Is this what you were looking for?
> >> >>
> >> >> 2010/1/28 Anton Popov :
> >> >> > Hello everyone,
> >> >> >
> >> >> > Is there any possibility to assign properties to Relations?
> >> >> > The matter is I want to try Dijkstra algorithm performance on some
> >> >> weighted
> >> >> > directed graph and don't know where to put weight actually...
> >> >> >
> >> >> > --
> >> >> > Best regards,
> >> >> > Popov Anton
> >> >> > ___
> >> >> > Neo mailing list
> >> >> > User@lists.neo4j.org
> >> >> > https://lists.neo4j.org/mailman/listinfo/user
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Mattias Persson, [matt...@neotechnology.com]
> >> >> Neo Technology, www.neotechnology.com
> >> >> ___
> >> >> Neo mailing list
> >> >> User@lists.neo4j.org
> >> >> https://lists.neo4j.org/mailman/listinfo/user
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Best regards,
> >> > Popov Anton
> >> > ___
> >> > Neo mailing list
> >> > User@lists.neo4j.org
> >> > https://lists.neo4j.org/mailman/listinfo/user
> >> >
> >>
> >>
> >> --
> >> Mattias Persson, [matt...@neotechnology.com]
> >> Neo Technology, www.neotechnology.com
> >> ___
> >> Neo mailing list
> >> User@lists.neo4j.org
> >> https://lists.neo4j.org/mailman/listinfo/user
> >>
> >
> >
> >
> > --
> > Best regards,
> > Popov Anton
> > ___
> > Neo mailing list
> > User@lists.neo4j.org
> > https://lists.neo4j.org/mailman/listinfo/user
> >
>
>
>
> --
> Mattias Persson, [matt...@neotechnology.com]
> Neo Technology, www.neotechnology.com
> ___
> Neo mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>



-- 
Best regards,
Popov Anton



-- 
Best regards,
Popov Anton
___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo] Possible causes for: org.apache.lucene.index.MergePolicy$MergeException

[Mattias Persson]

2010/1/29 Symeon (Akis) Papadopoulos :
>
>> I just fixed the bug and your benchmarks run fine w/o any modification
>> on my machine, so they should run fine on your end too!
>>
> Thank you Mattias!
> So, in the end it was a neo4j-index bug? Thus I assume we don't need to
> modify our code (of course we'll take into consideration your previous
> recommendations for using the BatchInserter, but for the sake of it, we
> would also like to run the benchmark in transactional mode as well).
Yep, there shouldn't be any need to modify your code at all. Yep, I
understand if you'd like to benchmark the transactional mode as well!
>
>> The bug is just committed so the latest build will be available within
>> an hour. I see that you aren't using maven, am I right? In that case
>> you can download it from
>> http://m2.neo4j.org/org/neo4j/neo4j-index/1.0-SNAPSHOT/ (look for .jar
>> files around this date/time + 1 hour or something). Or you could check
>> out the code and build it yourself:
>>
>>   svn co https://svn.neo4j.org/components/index/trunk
>>
> Out of the two built jars of 29th Jan 2010, I guess we should use the
> most recent one (neo4j-index-1.0-20100129.113202-3.jar), right?
Hmm, I don't think it have been deployed yet... so wait a little
longer for this change to be deployed. We have a little "lag" in
deployed jar files, which is kind of annoying in situations like this
:)
>
>
>> You could update to the latest neo4j-kernel as well, but maybe that's
>> not necessary though.
>>
> We'll first try out to update just the neo-index.
> If we face more problems we'll proceed with the kernel update as well.
>
> Thanks again for the prompt response!
It should be me thanking you, you gave me a complete test case
triggering a bug... very nice
>
>> 2010/1/29 Mattias Persson :
>>
>>> 2010/1/29 Symeon (Akis) Papadopoulos :
>>>
 Mattias Persson wrote:

> Another problem I see is that you're having too granular transactions
> which will slow down the insertion process quite a bit. Try grouping a
> couple of thousands operations in one transaction and you'll see a
> performance boost!
>
> FYI: I can trigger the problem you were having with lucene "too many
> open files" issue. And I'm almost 100% sure that it will be resolved
> if you increase the span of your transactions.
>
>
 That's what we are already doing through the class SimpleBatchTxManager
 (in package org.neo4j.util). We group read and write transactions in
 batches of some thousands. Isn't this correct?

>>> You're right, sorry for missing that.
>>>
>>> However I just found the bug which causes this... hang on, I'll fix it
>>> in an hour or two!
>>>
 ___
 Neo mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user


>>>
>>> --
>>> Mattias Persson, [matt...@neotechnology.com]
>>> Neo Technology, www.neotechnology.com
>>>
>>>
>>
>>
>>
>>
>
> ___
> Neo mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>



-- 
Mattias Persson, [matt...@neotechnology.com]
Neo Technology, www.neotechnology.com
___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo] Neo transactions

[Peter Neubauer]

Tim,
Jetty is run frequently together with Jersey and Neo4j. There is no
sign of any of these Exceptions coming from Neo4j code. However, could
you please provide (maybe off-list) some example code so we can take a
look at it?

Cheers,

/peter neubauer

COO and Sales, Neo Technology

GTalk:  neubauer.peter
Skype   peter.neubauer
Phone   +46 704 106975
LinkedIn   http://www.linkedin.com/in/neubauer
Twitter  http://twitter.com/peterneubauer

http://www.neo4j.org- Your high performance graph database.
http://gremlin.tinkerpop.com- The terminal to the Giant Global Graph.



On Wed, Jan 27, 2010 at 9:02 AM, Tim Langley  wrote:
> hi Johan
>
> thank you for the reply - i think you're right :) (hope so)
> I'm currently using jetty as the web-server and these exceptions are
> all being thrown by jetty
>
> example (and sorry for the dump)
> com.sun.jersey.api.container.MappableContainerException:
> java.lang.Exception: Unable to return Neo - This is not in a transaction
>        at
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch
> (ResourceJavaMethodDispatcher.java:74)
>        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept
> (HttpMethodRule.java:166)
>        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept
> (RightHandPathRule.java:114)
>        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept
> (ResourceClassRule.java:74)
>        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept
> (RightHandPathRule.java:114)
>        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept
> (RootResourceClassesRule.java:66)
>        at
> com.sun.jersey.server.impl.application.WebApplicationImpl
> ._handleRequest(WebApplicationImpl.java:658)
>        at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest
> (WebApplicationImpl.java:616)
>        at
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest
> (WebApplicationImpl.java:607)
>        at com.sun.jersey.spi.container.servlet.WebComponent.service
> (WebComponent.java:309)
>        at com.sun.jersey.spi.container.servlet.ServletContainer.service
> (ServletContainer.java:425)
>        at com.sun.jersey.spi.container.servlet.ServletContainer.service
> (ServletContainer.java:590)
>        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
>        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:
> 511)
>        at org.mortbay.jetty.servlet.ServletHandler.handle
> (ServletHandler.java:390)
>        at org.mortbay.jetty.security.SecurityHandler.handle
> (SecurityHandler.java:216)
>        at org.mortbay.jetty.servlet.SessionHandler.handle
> (SessionHandler.java:182)
>        at org.mortbay.jetty.handler.ContextHandler.handle
> (ContextHandler.java:765)
>        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:
> 418)
>        at org.mortbay.jetty.handler.ContextHandlerCollection.handle
> (ContextHandlerCollection.java:230)
>        at org.mortbay.jetty.handler.HandlerCollection.handle
> (HandlerCollection.java:114)
>        at org.mortbay.jetty.handler.HandlerWrapper.handle
> (HandlerWrapper.java:152)
>        at org.mortbay.jetty.Server.handle(Server.java:326)
>        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:
> 542)
>        at org.mortbay.jetty.HttpConnection$RequestHandler.content
> (HttpConnection.java:938)
>        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:755)
>        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:218)
>        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
>        at org.mortbay.io.nio.SelectChannelEndPoint.run
> (SelectChannelEndPoint.java:409)
>        at org.mortbay.thread.QueuedThreadPool$PoolThread.run
> (QueuedThreadPool.java:582)
>
> When Jetty start's up it reports:
> 2010-01-26 13:23:18.650:INFO::No Transaction manager found - if your
> webapp requires one, please configure one.
>
> which is what got me starting thinking about this
>
> I'd really appreciate any guidance on:
> a) do I need to configure neo or jetty better
> b) should I take jetty out of the equation and use a different web-
> server (Tomcat?)
>
> I'm wandering round in the dark here - would appreciate any thoughts :)
>
> T
>
>
> On 26 Jan 2010, at 20:34, Johan Svensson wrote:
>
>> Hi,
>>
>> This sounds like a scala wrapper issue.
>>
>> Short lived write transactions together with long lived reads should
>> be no problem. By default no locks are taken during reads instead each
>> read operation behaves as if it took a snapshot of the graph
>> (committed state) at point of invocation. So to answer your questions:
>>
>> 1. Neo4j is by default configured to handle concurrent access.
>>
>> 2. Neo4j read operations will not block a write operation and vice
>> versa since by default no read lock is taken.
>>
>> Could you provide some more detailed error message 

Re: [Neo] Possible causes for: org.apache.lucene.index.MergePolicy$MergeException

[Symeon (Akis) Papadopoulos]

> I just fixed the bug and your benchmarks run fine w/o any modification
> on my machine, so they should run fine on your end too!
>   
Thank you Mattias!
So, in the end it was a neo4j-index bug? Thus I assume we don't need to 
modify our code (of course we'll take into consideration your previous 
recommendations for using the BatchInserter, but for the sake of it, we 
would also like to run the benchmark in transactional mode as well).

> The bug is just committed so the latest build will be available within
> an hour. I see that you aren't using maven, am I right? In that case
> you can download it from
> http://m2.neo4j.org/org/neo4j/neo4j-index/1.0-SNAPSHOT/ (look for .jar
> files around this date/time + 1 hour or something). Or you could check
> out the code and build it yourself:
>
>   svn co https://svn.neo4j.org/components/index/trunk
>   
Out of the two built jars of 29th Jan 2010, I guess we should use the 
most recent one (neo4j-index-1.0-20100129.113202-3.jar), right?


> You could update to the latest neo4j-kernel as well, but maybe that's
> not necessary though.
>   
We'll first try out to update just the neo-index.
If we face more problems we'll proceed with the kernel update as well.

Thanks again for the prompt response!

> 2010/1/29 Mattias Persson :
>   
>> 2010/1/29 Symeon (Akis) Papadopoulos :
>> 
>>> Mattias Persson wrote:
>>>   
 Another problem I see is that you're having too granular transactions
 which will slow down the insertion process quite a bit. Try grouping a
 couple of thousands operations in one transaction and you'll see a
 performance boost!

 FYI: I can trigger the problem you were having with lucene "too many
 open files" issue. And I'm almost 100% sure that it will be resolved
 if you increase the span of your transactions.

 
>>> That's what we are already doing through the class SimpleBatchTxManager
>>> (in package org.neo4j.util). We group read and write transactions in
>>> batches of some thousands. Isn't this correct?
>>>   
>> You're right, sorry for missing that.
>>
>> However I just found the bug which causes this... hang on, I'll fix it
>> in an hour or two!
>> 
>>> ___
>>> Neo mailing list
>>> User@lists.neo4j.org
>>> https://lists.neo4j.org/mailman/listinfo/user
>>>
>>>   
>>
>> --
>> Mattias Persson, [matt...@neotechnology.com]
>> Neo Technology, www.neotechnology.com
>>
>> 
>
>
>
>   

___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo] Performance/storage considerations - best practice question

[Johan Svensson]

Hi,

On Wed, Jan 27, 2010 at 11:15 PM, Rick Bullotta
 wrote:
> Is there any recommendation on when to use additional node/relationship
> connections versus additional node properties?

It depends of use case but go with relationships if it "enriches" the
graph with information that makes sense.

In your case it sounds like you will not modify the hashmap data and
only read it once so the best way here would probably be to store it
as a blob (array property).  If your use case included modifying the
hashmap often and performing traversals (or matching graph patterns)
filtering out nodes depending on the properties of the hashmap you
should use relationships.

The cost of a relationship operation vs a property operation will
depend on the property but for primitive types (that can fit in 8
bytes) a property add is a little faster than a relationship create.
Adding a string or an array will however be slower than a single
relationship create.

-Johan

>
>
>
> Specifically, I have a need to serialize/save a well-defined data structure
> to Neo.
>
>
>
> Imagine I have a Java class.  That class has some properties, A, B, C.  A
> and B are primitives, and C is a String-keyed hashmap of complex types that
> each have five properties, all of which are primitives, that we'll call
> P1,P2,P3,P4 and P5.  Assume there are 10 items in this collection, with keys
> X1,X2.X10.  These values will generally be read only once at application
> startup, and written very infrequently.
>
>
>
> Option #1: Create a node, store A and B as properties on that node.  Create
> a node for each object in the hashmap, with a relationship to the previously
> created node, and set properties on this node corresponding to the key and
> each of the properties.
>
>
>
> Option #2:  Create a node, store A and B as properties on the node, and
> store the rest of the data as properties using a naming pattern such as:
> C.X1.P1, C.X1.P2...C.X10.P5
>
>
>
> I guess the core question is, what is the relative cost of storing
> additional properties and some light parsing versus storing additional
> relationships and nodes?
>
>
>
> Option #1 is obviously much more "pure", but at what cost?
>
>
>
> Thanks,
>
>
>
> Rick
___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo] Relation properties

[Anton Popov]

Hello Mattias,

Got it. Thanks.
Is there any RoadMap available?

2010/1/29 Mattias Persson 

> Unfortunately you cannot index Relationships, but it's a (frquently
> requested) feature we're planning to add.
>
> 2010/1/29 Anton Popov :
> > Hello everybody,
> >
> > A related question is can I index both Nodes & Relations?
> > As I can see LuceneIndexBatchInserter.index( long node, String key,
> Object
> > value ) applies for Nodes only.
> >
> > 2010/1/28 Mattias Persson 
> >
> >> Not at all, you're not the first one to overlook that inheritance!
> >>
> >> 2010/1/28, Anton Popov :
> >> > Thanks for a quick answer, Mattias.
> >> >
> >> > Sorry for a stupid questions, just haven't noticed the inheritance of
> >> > Relationship interface.
> >> >
> >> > On 28 January 2010 19:43, Mattias Persson 
> >> wrote:
> >> >
> >> >> Relationship has all the property methods, just like Node... they're
> >> >> actually on PropertyContainer (a common super interface to Node and
> >> >> Relationship) see
> >> >> http://api.neo4j.org/current/org/neo4j/graphdb/Relationship.html and
> >> >>
> http://api.neo4j.org/current/org/neo4j/graphdb/PropertyContainer.html
> >> >> for more information.
> >> >>
> >> >> So to manage the cost in a dijkstra algorithm you can use whatever
> >> >> property key, f.ex. "weight", like this:
> >> >>
> >> >>Relationship rel = ... // any relationship
> >> >>rel.setProperty( "weight", 4 );
> >> >>
> >> >> And if you're using the Dijkstra algorithms in the "neo4j-graph-algo"
> >> >> component, http://components.neo4j.org/neo4j-graph-algo/ , you can
> >> >> instantiate it with a CostEvaluator instance something like this:
> >> >>
> >> >>new CostEvaluator()
> >> >>{
> >> >>CostType getCost( Relationship relationship, boolean backwards
> )
> >> >>{
> >> >>/ TODO Handle backwards flag?
> >> >>return ( Integer ) relationship.getProperty( "weight", 1
> );
> >> >>}
> >> >>}
> >> >>
> >> >> ...and pass it to your Dijkstra algo instance.
> >> >>
> >> >> Is this what you were looking for?
> >> >>
> >> >> 2010/1/28 Anton Popov :
> >> >> > Hello everyone,
> >> >> >
> >> >> > Is there any possibility to assign properties to Relations?
> >> >> > The matter is I want to try Dijkstra algorithm performance on some
> >> >> weighted
> >> >> > directed graph and don't know where to put weight actually...
> >> >> >
> >> >> > --
> >> >> > Best regards,
> >> >> > Popov Anton
> >> >> > ___
> >> >> > Neo mailing list
> >> >> > User@lists.neo4j.org
> >> >> > https://lists.neo4j.org/mailman/listinfo/user
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Mattias Persson, [matt...@neotechnology.com]
> >> >> Neo Technology, www.neotechnology.com
> >> >> ___
> >> >> Neo mailing list
> >> >> User@lists.neo4j.org
> >> >> https://lists.neo4j.org/mailman/listinfo/user
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Best regards,
> >> > Popov Anton
> >> > ___
> >> > Neo mailing list
> >> > User@lists.neo4j.org
> >> > https://lists.neo4j.org/mailman/listinfo/user
> >> >
> >>
> >>
> >> --
> >> Mattias Persson, [matt...@neotechnology.com]
> >> Neo Technology, www.neotechnology.com
> >> ___
> >> Neo mailing list
> >> User@lists.neo4j.org
> >> https://lists.neo4j.org/mailman/listinfo/user
> >>
> >
> >
> >
> > --
> > Best regards,
> > Popov Anton
> > ___
> > Neo mailing list
> > User@lists.neo4j.org
> > https://lists.neo4j.org/mailman/listinfo/user
> >
>
>
>
> --
> Mattias Persson, [matt...@neotechnology.com]
> Neo Technology, www.neotechnology.com
> ___
> Neo mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>



-- 
Best regards,
Popov Anton
___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo] Possible causes for: org.apache.lucene.index.MergePolicy$MergeException

[Mattias Persson]

I just fixed the bug and your benchmarks run fine w/o any modification
on my machine, so they should run fine on your end too!

The bug is just committed so the latest build will be available within
an hour. I see that you aren't using maven, am I right? In that case
you can download it from
http://m2.neo4j.org/org/neo4j/neo4j-index/1.0-SNAPSHOT/ (look for .jar
files around this date/time + 1 hour or something). Or you could check
out the code and build it yourself:

  svn co https://svn.neo4j.org/components/index/trunk

You could update to the latest neo4j-kernel as well, but maybe that's
not necessary though.

2010/1/29 Mattias Persson :
> 2010/1/29 Symeon (Akis) Papadopoulos :
>> Mattias Persson wrote:
>>> Another problem I see is that you're having too granular transactions
>>> which will slow down the insertion process quite a bit. Try grouping a
>>> couple of thousands operations in one transaction and you'll see a
>>> performance boost!
>>>
>>> FYI: I can trigger the problem you were having with lucene "too many
>>> open files" issue. And I'm almost 100% sure that it will be resolved
>>> if you increase the span of your transactions.
>>>
>> That's what we are already doing through the class SimpleBatchTxManager
>> (in package org.neo4j.util). We group read and write transactions in
>> batches of some thousands. Isn't this correct?
> You're right, sorry for missing that.
>
> However I just found the bug which causes this... hang on, I'll fix it
> in an hour or two!
>>
>>
>> ___
>> Neo mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
>
>
>
> --
> Mattias Persson, [matt...@neotechnology.com]
> Neo Technology, www.neotechnology.com
>



-- 
Mattias Persson, [matt...@neotechnology.com]
Neo Technology, www.neotechnology.com
___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo] Relation properties

[Mattias Persson]

Unfortunately you cannot index Relationships, but it's a (frquently
requested) feature we're planning to add.

2010/1/29 Anton Popov :
> Hello everybody,
>
> A related question is can I index both Nodes & Relations?
> As I can see LuceneIndexBatchInserter.index( long node, String key, Object
> value ) applies for Nodes only.
>
> 2010/1/28 Mattias Persson 
>
>> Not at all, you're not the first one to overlook that inheritance!
>>
>> 2010/1/28, Anton Popov :
>> > Thanks for a quick answer, Mattias.
>> >
>> > Sorry for a stupid questions, just haven't noticed the inheritance of
>> > Relationship interface.
>> >
>> > On 28 January 2010 19:43, Mattias Persson 
>> wrote:
>> >
>> >> Relationship has all the property methods, just like Node... they're
>> >> actually on PropertyContainer (a common super interface to Node and
>> >> Relationship) see
>> >> http://api.neo4j.org/current/org/neo4j/graphdb/Relationship.html and
>> >> http://api.neo4j.org/current/org/neo4j/graphdb/PropertyContainer.html
>> >> for more information.
>> >>
>> >> So to manage the cost in a dijkstra algorithm you can use whatever
>> >> property key, f.ex. "weight", like this:
>> >>
>> >>    Relationship rel = ... // any relationship
>> >>    rel.setProperty( "weight", 4 );
>> >>
>> >> And if you're using the Dijkstra algorithms in the "neo4j-graph-algo"
>> >> component, http://components.neo4j.org/neo4j-graph-algo/ , you can
>> >> instantiate it with a CostEvaluator instance something like this:
>> >>
>> >>    new CostEvaluator()
>> >>    {
>> >>        CostType getCost( Relationship relationship, boolean backwards )
>> >>        {
>> >>            / TODO Handle backwards flag?
>> >>            return ( Integer ) relationship.getProperty( "weight", 1 );
>> >>        }
>> >>    }
>> >>
>> >> ...and pass it to your Dijkstra algo instance.
>> >>
>> >> Is this what you were looking for?
>> >>
>> >> 2010/1/28 Anton Popov :
>> >> > Hello everyone,
>> >> >
>> >> > Is there any possibility to assign properties to Relations?
>> >> > The matter is I want to try Dijkstra algorithm performance on some
>> >> weighted
>> >> > directed graph and don't know where to put weight actually...
>> >> >
>> >> > --
>> >> > Best regards,
>> >> > Popov Anton
>> >> > ___
>> >> > Neo mailing list
>> >> > User@lists.neo4j.org
>> >> > https://lists.neo4j.org/mailman/listinfo/user
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Mattias Persson, [matt...@neotechnology.com]
>> >> Neo Technology, www.neotechnology.com
>> >> ___
>> >> Neo mailing list
>> >> User@lists.neo4j.org
>> >> https://lists.neo4j.org/mailman/listinfo/user
>> >>
>> >
>> >
>> >
>> > --
>> > Best regards,
>> > Popov Anton
>> > ___
>> > Neo mailing list
>> > User@lists.neo4j.org
>> > https://lists.neo4j.org/mailman/listinfo/user
>> >
>>
>>
>> --
>> Mattias Persson, [matt...@neotechnology.com]
>> Neo Technology, www.neotechnology.com
>> ___
>> Neo mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
>
>
>
> --
> Best regards,
> Popov Anton
> ___
> Neo mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>



-- 
Mattias Persson, [matt...@neotechnology.com]
Neo Technology, www.neotechnology.com
___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo] Relation properties

[Anton Popov]

Hello everybody,

A related question is can I index both Nodes & Relations?
As I can see LuceneIndexBatchInserter.index( long node, String key, Object
value ) applies for Nodes only.

2010/1/28 Mattias Persson 

> Not at all, you're not the first one to overlook that inheritance!
>
> 2010/1/28, Anton Popov :
> > Thanks for a quick answer, Mattias.
> >
> > Sorry for a stupid questions, just haven't noticed the inheritance of
> > Relationship interface.
> >
> > On 28 January 2010 19:43, Mattias Persson 
> wrote:
> >
> >> Relationship has all the property methods, just like Node... they're
> >> actually on PropertyContainer (a common super interface to Node and
> >> Relationship) see
> >> http://api.neo4j.org/current/org/neo4j/graphdb/Relationship.html and
> >> http://api.neo4j.org/current/org/neo4j/graphdb/PropertyContainer.html
> >> for more information.
> >>
> >> So to manage the cost in a dijkstra algorithm you can use whatever
> >> property key, f.ex. "weight", like this:
> >>
> >>Relationship rel = ... // any relationship
> >>rel.setProperty( "weight", 4 );
> >>
> >> And if you're using the Dijkstra algorithms in the "neo4j-graph-algo"
> >> component, http://components.neo4j.org/neo4j-graph-algo/ , you can
> >> instantiate it with a CostEvaluator instance something like this:
> >>
> >>new CostEvaluator()
> >>{
> >>CostType getCost( Relationship relationship, boolean backwards )
> >>{
> >>/ TODO Handle backwards flag?
> >>return ( Integer ) relationship.getProperty( "weight", 1 );
> >>}
> >>}
> >>
> >> ...and pass it to your Dijkstra algo instance.
> >>
> >> Is this what you were looking for?
> >>
> >> 2010/1/28 Anton Popov :
> >> > Hello everyone,
> >> >
> >> > Is there any possibility to assign properties to Relations?
> >> > The matter is I want to try Dijkstra algorithm performance on some
> >> weighted
> >> > directed graph and don't know where to put weight actually...
> >> >
> >> > --
> >> > Best regards,
> >> > Popov Anton
> >> > ___
> >> > Neo mailing list
> >> > User@lists.neo4j.org
> >> > https://lists.neo4j.org/mailman/listinfo/user
> >> >
> >>
> >>
> >>
> >> --
> >> Mattias Persson, [matt...@neotechnology.com]
> >> Neo Technology, www.neotechnology.com
> >> ___
> >> Neo mailing list
> >> User@lists.neo4j.org
> >> https://lists.neo4j.org/mailman/listinfo/user
> >>
> >
> >
> >
> > --
> > Best regards,
> > Popov Anton
> > ___
> > Neo mailing list
> > User@lists.neo4j.org
> > https://lists.neo4j.org/mailman/listinfo/user
> >
>
>
> --
> Mattias Persson, [matt...@neotechnology.com]
> Neo Technology, www.neotechnology.com
> ___
> Neo mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>



-- 
Best regards,
Popov Anton
___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo] Possible causes for: org.apache.lucene.index.MergePolicy$MergeException

[Mattias Persson]

2010/1/29 Symeon (Akis) Papadopoulos :
> Mattias Persson wrote:
>> Another problem I see is that you're having too granular transactions
>> which will slow down the insertion process quite a bit. Try grouping a
>> couple of thousands operations in one transaction and you'll see a
>> performance boost!
>>
>> FYI: I can trigger the problem you were having with lucene "too many
>> open files" issue. And I'm almost 100% sure that it will be resolved
>> if you increase the span of your transactions.
>>
> That's what we are already doing through the class SimpleBatchTxManager
> (in package org.neo4j.util). We group read and write transactions in
> batches of some thousands. Isn't this correct?
You're right, sorry for missing that.

However I just found the bug which causes this... hang on, I'll fix it
in an hour or two!
>
>
> ___
> Neo mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>



-- 
Mattias Persson, [matt...@neotechnology.com]
Neo Technology, www.neotechnology.com
___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo] Possible causes for: org.apache.lucene.index.MergePolicy$MergeException

[Symeon (Akis) Papadopoulos]

Mattias Persson wrote:
> Another problem I see is that you're having too granular transactions
> which will slow down the insertion process quite a bit. Try grouping a
> couple of thousands operations in one transaction and you'll see a
> performance boost!
>
> FYI: I can trigger the problem you were having with lucene "too many
> open files" issue. And I'm almost 100% sure that it will be resolved
> if you increase the span of your transactions.
>   
That's what we are already doing through the class SimpleBatchTxManager 
(in package org.neo4j.util). We group read and write transactions in 
batches of some thousands. Isn't this correct?


___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo] Possible causes for: org.apache.lucene.index.MergePolicy$MergeException

[Mattias Persson]

(Sorry for spamming you guys)

See http://wiki.neo4j.org/content/Transactions for general information
about how to manage transactions in Neo4j.

2010/1/29 Mattias Persson :
> Another problem I see is that you're having too granular transactions
> which will slow down the insertion process quite a bit. Try grouping a
> couple of thousands operations in one transaction and you'll see a
> performance boost!
>
> FYI: I can trigger the problem you were having with lucene "too many
> open files" issue. And I'm almost 100% sure that it will be resolved
> if you increase the span of your transactions.
>
> 2010/1/29 Mattias Persson :
>> I see that you're not using the Neo4j BatchInserter. Your use case
>> could definately benefit from it. Take a look at
>> http://wiki.neo4j.org/content/Batch_Insert on how to use it. And since
>> you're using index lookups you'll have to keep most of the index in
>> your own HashMap or other cache (instead of going down to index) for
>> maximum performance. (the LuceneIndexBatchInserterImpl doesn't have
>> cache built-in, like the LuceneIndexService does).
>>
>> You typically gain 5 times the speed or more when using the
>> BatchInserter instead of GraphDatabaseService for inserting big data
>> sets.
>>
>> 2010/1/28 Mattias Persson :
>>> Im sorry but I think your attachment got caught in our mail filters.
>>>
>>> Could you perhaps send me your project (including the entire data file
>>> via some file sending service, f.ex http://sprend.com) and send it to
>>> me directly, matt...@neotechnology.com?
>>>
>>> 2010/1/28, Symeon (Akis) Papadopoulos :
 
> Great, lucene handles merging in the background automatically if the
> files are too sharded. So this error shouldn't occur unless there are
> some corner case where some IndexReader/IndexWriter isn't closed
> properly... so that's why I'm suspecting a bug here :)
>
> Great that you found a work-around, but I suspect it could happen even
> with a higher ulimit though.
>
>> The project I'm running is rather large, but at some point I will try to
>> prepare a script in order to replicate the error.
>>
> That would be great indeed!
>
 I attach an eclipse project which is an extract from the larger
 benchmark project I have been referring to in my previous emails. You
 can run the class graph.load.LoadGraphBenchmark as a java application to
 see what happens. However, this will not replicate the error in your
 machine, because the test file included in the project is small (the one
 I used when I ran into the error was rather large to send by mail).
 Perhaps you will be able to see what's wrong with the code just by
 reading through it. I suspect that I misuse Neo, but I can't really
 pinpoint the problem.

 Thank you in advance for your help!
 Best regards,
 Symeon







>>
>>
> 2010/1/26 Symeon (Akis) Papadopoulos :
>
>
>
>> Hi all
>>
>> While populating a Neo graph, I got the attached exception
>> (org.apache.lucene.index.MergePolicy$MergeException. See file for
>> details: I
>> replaced a local path with [some-local-path]).
>> My setup is like this: I want to benchmark Neo4j for some operations,
>> amongst which is graph loading.
>> So, I try to load graphs of various sizes to Neo4j. Up to size 1M
>> edges the
>> graphs are loaded without any problem, but then this exception is
>> thrown. I
>> suspect this has to do with the transaction management in my program,
>> which
>> is handled by a BatchTxManager (a class written by me, thus highly
>> likely to
>> be the source of trouble). Does the exception ring any bells? What
>> could I
>> try out in order to identify the problem?
>>
>> Best regards,
>> Akis
>>
>>
>>
>> Exception in thread "Lucene Merge Thread #0"
>> org.apache.lucene.index.MergePolicy$MergeException:
>> java.io.FileNotFoundException: /[some-local-path]/_4k.cfs (Too many
>> open
>> files)
>>        at
>> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
>>        at
>> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
>> Caused by: java.io.FileNotFoundException: /[some-local-path]/_4k.cfs
>> (Too
>> many open files)
>>        at java.io.RandomAccessFile.open(Native Method)
>>        at java.io.RandomAccessFile.(RandomAccessFile.java:233)
>>        at
>> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:78)
>>        at
>> org.apache.lucene.store.SimpleFSDirector

Re: [Neo] Possible causes for: org.apache.lucene.index.MergePolicy$MergeException

[Mattias Persson]

Another problem I see is that you're having too granular transactions
which will slow down the insertion process quite a bit. Try grouping a
couple of thousands operations in one transaction and you'll see a
performance boost!

FYI: I can trigger the problem you were having with lucene "too many
open files" issue. And I'm almost 100% sure that it will be resolved
if you increase the span of your transactions.

2010/1/29 Mattias Persson :
> I see that you're not using the Neo4j BatchInserter. Your use case
> could definately benefit from it. Take a look at
> http://wiki.neo4j.org/content/Batch_Insert on how to use it. And since
> you're using index lookups you'll have to keep most of the index in
> your own HashMap or other cache (instead of going down to index) for
> maximum performance. (the LuceneIndexBatchInserterImpl doesn't have
> cache built-in, like the LuceneIndexService does).
>
> You typically gain 5 times the speed or more when using the
> BatchInserter instead of GraphDatabaseService for inserting big data
> sets.
>
> 2010/1/28 Mattias Persson :
>> Im sorry but I think your attachment got caught in our mail filters.
>>
>> Could you perhaps send me your project (including the entire data file
>> via some file sending service, f.ex http://sprend.com) and send it to
>> me directly, matt...@neotechnology.com?
>>
>> 2010/1/28, Symeon (Akis) Papadopoulos :
>>> 
 Great, lucene handles merging in the background automatically if the
 files are too sharded. So this error shouldn't occur unless there are
 some corner case where some IndexReader/IndexWriter isn't closed
 properly... so that's why I'm suspecting a bug here :)

 Great that you found a work-around, but I suspect it could happen even
 with a higher ulimit though.

> The project I'm running is rather large, but at some point I will try to
> prepare a script in order to replicate the error.
>
 That would be great indeed!

>>> I attach an eclipse project which is an extract from the larger
>>> benchmark project I have been referring to in my previous emails. You
>>> can run the class graph.load.LoadGraphBenchmark as a java application to
>>> see what happens. However, this will not replicate the error in your
>>> machine, because the test file included in the project is small (the one
>>> I used when I ran into the error was rather large to send by mail).
>>> Perhaps you will be able to see what's wrong with the code just by
>>> reading through it. I suspect that I misuse Neo, but I can't really
>>> pinpoint the problem.
>>>
>>> Thank you in advance for your help!
>>> Best regards,
>>> Symeon
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>
>
 2010/1/26 Symeon (Akis) Papadopoulos :



> Hi all
>
> While populating a Neo graph, I got the attached exception
> (org.apache.lucene.index.MergePolicy$MergeException. See file for
> details: I
> replaced a local path with [some-local-path]).
> My setup is like this: I want to benchmark Neo4j for some operations,
> amongst which is graph loading.
> So, I try to load graphs of various sizes to Neo4j. Up to size 1M
> edges the
> graphs are loaded without any problem, but then this exception is
> thrown. I
> suspect this has to do with the transaction management in my program,
> which
> is handled by a BatchTxManager (a class written by me, thus highly
> likely to
> be the source of trouble). Does the exception ring any bells? What
> could I
> try out in order to identify the problem?
>
> Best regards,
> Akis
>
>
>
> Exception in thread "Lucene Merge Thread #0"
> org.apache.lucene.index.MergePolicy$MergeException:
> java.io.FileNotFoundException: /[some-local-path]/_4k.cfs (Too many
> open
> files)
>        at
> org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
>        at
> org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
> Caused by: java.io.FileNotFoundException: /[some-local-path]/_4k.cfs
> (Too
> many open files)
>        at java.io.RandomAccessFile.open(Native Method)
>        at java.io.RandomAccessFile.(RandomAccessFile.java:233)
>        at
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:78)
>        at
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:108)
>        at
> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:94)
>        at
> org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:70)
>        at
> or

Re: [Neo] Possible causes for: org.apache.lucene.index.MergePolicy$MergeException

[Mattias Persson]

I see that you're not using the Neo4j BatchInserter. Your use case
could definately benefit from it. Take a look at
http://wiki.neo4j.org/content/Batch_Insert on how to use it. And since
you're using index lookups you'll have to keep most of the index in
your own HashMap or other cache (instead of going down to index) for
maximum performance. (the LuceneIndexBatchInserterImpl doesn't have
cache built-in, like the LuceneIndexService does).

You typically gain 5 times the speed or more when using the
BatchInserter instead of GraphDatabaseService for inserting big data
sets.

2010/1/28 Mattias Persson :
> Im sorry but I think your attachment got caught in our mail filters.
>
> Could you perhaps send me your project (including the entire data file
> via some file sending service, f.ex http://sprend.com) and send it to
> me directly, matt...@neotechnology.com?
>
> 2010/1/28, Symeon (Akis) Papadopoulos :
>> 
>>> Great, lucene handles merging in the background automatically if the
>>> files are too sharded. So this error shouldn't occur unless there are
>>> some corner case where some IndexReader/IndexWriter isn't closed
>>> properly... so that's why I'm suspecting a bug here :)
>>>
>>> Great that you found a work-around, but I suspect it could happen even
>>> with a higher ulimit though.
>>>
 The project I'm running is rather large, but at some point I will try to
 prepare a script in order to replicate the error.

>>> That would be great indeed!
>>>
>> I attach an eclipse project which is an extract from the larger
>> benchmark project I have been referring to in my previous emails. You
>> can run the class graph.load.LoadGraphBenchmark as a java application to
>> see what happens. However, this will not replicate the error in your
>> machine, because the test file included in the project is small (the one
>> I used when I ran into the error was rather large to send by mail).
>> Perhaps you will be able to see what's wrong with the code just by
>> reading through it. I suspect that I misuse Neo, but I can't really
>> pinpoint the problem.
>>
>> Thank you in advance for your help!
>> Best regards,
>> Symeon
>>
>>
>>
>>
>>
>>
>>


>>> 2010/1/26 Symeon (Akis) Papadopoulos :
>>>
>>>
>>>
 Hi all

 While populating a Neo graph, I got the attached exception
 (org.apache.lucene.index.MergePolicy$MergeException. See file for
 details: I
 replaced a local path with [some-local-path]).
 My setup is like this: I want to benchmark Neo4j for some operations,
 amongst which is graph loading.
 So, I try to load graphs of various sizes to Neo4j. Up to size 1M
 edges the
 graphs are loaded without any problem, but then this exception is
 thrown. I
 suspect this has to do with the transaction management in my program,
 which
 is handled by a BatchTxManager (a class written by me, thus highly
 likely to
 be the source of trouble). Does the exception ring any bells? What
 could I
 try out in order to identify the problem?

 Best regards,
 Akis



 Exception in thread "Lucene Merge Thread #0"
 org.apache.lucene.index.MergePolicy$MergeException:
 java.io.FileNotFoundException: /[some-local-path]/_4k.cfs (Too many
 open
 files)
        at
 org.apache.lucene.index.ConcurrentMergeScheduler.handleMergeException(ConcurrentMergeScheduler.java:351)
        at
 org.apache.lucene.index.ConcurrentMergeScheduler$MergeThread.run(ConcurrentMergeScheduler.java:315)
 Caused by: java.io.FileNotFoundException: /[some-local-path]/_4k.cfs
 (Too
 many open files)
        at java.io.RandomAccessFile.open(Native Method)
        at java.io.RandomAccessFile.(RandomAccessFile.java:233)
        at
 org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor.(SimpleFSDirectory.java:78)
        at
 org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput.(SimpleFSDirectory.java:108)
        at
 org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput.(NIOFSDirectory.java:94)
        at
 org.apache.lucene.store.NIOFSDirectory.openInput(NIOFSDirectory.java:70)
        at
 org.apache.lucene.index.CompoundFileReader.(CompoundFileReader.java:70)
        at
 org.apache.lucene.index.SegmentReader$CoreReaders.(SegmentReader.java:111)
        at
 org.apache.lucene.index.SegmentReader.get(SegmentReader.java:638)
        at
 org.apache.lucene.index.SegmentReader.get(SegmentReader.java:608)
        at
 org.apache.lucene.index.IndexWriter$ReaderPool.get(IndexWriter.java:686)
        at
 org.apache.lucene.index.IndexWriter.mergeMiddle(IndexWriter.java:4979)
>

[Neo] Question before I use neo4j, how to synchronize databases

[philip andrew]

Hi,

I think neo4j seems to be a good fit for what I am trying to do but I am not
sure about a few things.

I have a requirement that I need to move data from one database to another.
There will be many client databases and they need to synchronize with a
central database, so SOME (not all data) data updates are needed to be sent
to the server, and some back down, but not all records, just selected
records. How can I do this with neo4j?
How could I package up this data and move it to another database over the
wire, eg via http post of a file to the server? or other way? In batch mode,
one big file?

Thanks, Philip
___
Neo mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user