Re: [Neo4j] Webadmin browser dependencies?

[Anders Nawroth]

Hi!

> Hi, Anders.
>
> Just wanted to let you know that the 1.4 version of Neoclipse seems to always 
> show all properties as "graph decorations", regardless of what is 
> configured/specified.

It's because I made the "Expanded node mode" on by default. I thought 
that it would be a better default for most people, but didn't realize it 
would be confusing as well :-)

Not sure how to present the different options the best way in the UI. Or 
if I should remove some options to make things more straight-forward. 
Suggestions are very welcome!

/anders

>
> Thanks,
>
> Rick
>
>
>
> -Original Message-
> From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On 
> Behalf Of Anders Nawroth
> Sent: Thursday, June 23, 2011 8:15 PM
> To: Neo4j user discussions
> Subject: Re: [Neo4j] Webadmin browser dependencies?
>
>
> 2011-06-23 19:54, Paul Bandler skrev:
>> Tried Neoclipse but after waiting for it to connect for over half an hour I 
>> concluded that maybe the latest 1.3 is incompatible with ths 1.4m04 neo that 
>> I'm using. Couldn't find a later neoclipse build is there one?
>
> I just created a 1.4.M04 Neoclipse:
>
> http://neo4j.org/get?file=neoclipse-1.4.M04-win32.win32.x86.zip
> http://neo4j.org/get?file=neoclipse-1.4.M04-win32.win32.x86_64.zip
> http://neo4j.org/get?file=neoclipse-1.4.M04-macosx.carbon.x86.tar.gz
> http://neo4j.org/get?file=neoclipse-1.4.M04-linux.gtk.x86.tar.gz
> http://neo4j.org/get?file=neoclipse-1.4.M04-linux.gtk.x86_64.tar.gz
>
> It comes with a new experimental feature: in read-only mode there's a
> "Sync database" button enabled (next to the start/stop buttons), which
> will restart the read-only instance and set the same node as starting
> point again (if it still exists, otherwise there will be a warning).
>
> The download page is a bit outdated at the moment ... will get fixed soon.
>
>
> /anders
>
>>
>> Sent from my iPhone
>>
>> On 23 Jun 2011, at 18:34, Michael Hunger   
>> wrote:
>>
>>> Which parts of webadmin are absolutely critical?
>>>
>>> What about neoclipse readonly?
>>>
>>> Sent from my iBrick4
>>>
>>>
>>> Am 23.06.2011 um 19:00 schrieb Rick Bullotta:
>>>
 Welcome to my world, Paul...

 Web applications that don't recognize the realities of the "enterprise 
 desktop" won't ultimately be able to leverage a given technology if it 
 doesn't fit with their corporate "standards".  As painful as it may be at 
 times, broad-based browser support is a must-have for applications that 
 want to make a big push into the enterprise (behind the firewall) world.

 My experience, however, is that the "corporate standards" can usually be 
 bypassed by developers/admins but not by the general user.  Exceptions 
 tend to be in defense, public safety/security or other highly sensitive 
 industries.  Perhaps your client would agree to violate the "standard" for 
 admins?




 -Original Message-
 From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] 
 On Behalf Of Paul Bandler
 Sent: Thursday, June 23, 2011 11:03 AM
 To: Neo4j user discussions
 Cc: Neo4j user discussions
 Subject: Re: [Neo4j] Webadmin browser dependencies?

 Thanks but none of those browsers are allowed to be downloaded within my 
 clients corporate network...

 Again, does anyone know what IE level works?

 Sent from my iPhone

 On 23 Jun 2011, at 13:53, Tatham Oddie   wrote:

> Data browser works in Chome / Firefox / Safari.
>
> It should give you a message to this effect on unsupported browsers.
>
>
> -- Tatham
>
>
> -Original Message-
> From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] 
> On Behalf Of Paul Bandler
> Sent: Thursday, 23 June 2011 10:30 PM
> To: Neo4j user discussions
> Subject: [Neo4j] Webadmin browser dependencies?
>
> Unable to access the data tab using IE version 7. Tried using the Eclipse 
> built in browser and it momentarily switches to the data tab but then it 
> disappears again.
>
> What are the supported browser platforms?
>
> Sent from my iPhone
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user
>>> ___
>>> Neo4j mailing list
>>> User@lists.neo4j.org
>>> https://lists.neo4j.org/mailman/

[Neo4j] no remote thread-safe API

[Aliabbas Petiwala]

is it true that in neo4j there is no remote thread-safe API. This
means that when making a multi-threaded solution you run into problems
when updating relationships between nodes. Under stress you are likely
to want to update a relationship while another thread has a lock and
as such you run into problems.
Will this problem be solved in future?

-- 
Aliabbas Petiwala
M.Tech CSE
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

[Neo4j] PHP REST Client for Neo4j 1.3 available

[Todd Chaffee]

I just wanted to remind folks that a PHP REST client based on Neo4j V1.3 
has been available for some time at

http://github.com/tchaffee/Neo4J-REST-PHP-API-client 


I am working on finishing up traversals, and other than that it is a 
complete implementation of the REST API.  It's fully unit tested 
(including exceptions) with 100% code coverage.

Would it be possible to have these pages on the wiki updated to point to 
the repo above (page is locked for editing)?

http://wiki.neo4j.org/content/PHP
http://wiki.neo4j.org/content/Main_Page#Language_and_framework_bindings

The version by onewheelgood seems to be no longer maintained and does 
not work for V1.3 of Neo4j.  I am committed to keeping my fork up to 
date and working with the latest stable version of Neo4j.

I also have some PHP code examples I could add to the wiki if someone 
could let me know the best place to put them.

Would also be great if anyone else wants to join in and contribute to 
improving the PHP REST client.

Thanks,
Todd Chaffee
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

[Neo4j] Unable to memory map [was java.io.IOException: Resource temporarily unavailable in MappedPersistenceWindow.java?

[Paul Bandler]

Further to an earlier posting I also notice this warning from neo4j which 
sounds relevant..? Note that the application does continue and eventually 
complete successfully having taken orders of magnitude longer than when run on 
windows. This is running on solaris  -  any suggestions about what could be 
causing this behaviour most welcome . Are there  issues with using neo4j on 
Solaris?

Jun 30, 2011 10:58:59 AM 
org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool logWarn
WARNING: [./neo4j-advanced-1.4.M05/data/graph.db/neostore.relationshipstore.db] 
Unable to memory map
org.neo4j.kernel.impl.nioneo.store.MappedMemException: Unable to map pos=28593 
recordSize=33 totalSize=104841
at 
org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow.(MappedPersistenceWindow.java:61)
at 
org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.allocateNewWindow(PersistenceWindowPool.java:603)
at 
org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.refreshBricks(PersistenceWindowPool.java:501)
at 
org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:128)
at 
org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:526)
at 
org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getChainRecord(RelationshipStore.java:327)
at 
org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:114)
at 
org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:97)
at 
org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:108)
at 
org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:604)
at 
org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:403)
at 
org.neo4j.kernel.impl.core.IntArrayIterator.hasNext(IntArrayIterator.java:98)
at 
com.nomura.smo.vcs.rdm.neo4j.AbstractBaseNeo4j.getSectorRelations(AbstractBaseNeo4j.java:42)

Sent from my iPhone

On 30 Jun 2011, at 07:40, Paul Bandler  wrote:

> When running a test neo4j application on Solaris that I have previously run 
> successfully on Windows I'm encountering the following exception:
> 
> Caused by: java.io.IOException: Resource temporarily unavailable
>at sun.nio.ch.FileChannelImpl.map0(Native Method)
>at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:747)
>at 
> org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow.(MappedPersistenceWindow.java:53)
> 
> Indeed this in  turn is causing a huge knock-on effect but I've not included 
> that stack trace for clarity.  The program appears to attempt to continue, 
> albeit at a snails pace.
> 
> I'm running using the latest 1.4 milestone release, with a maximum heap of 
> 2G, and defaulting all other store parameters.
> 
> Any suggestions as to what is happening would be most welcome.
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] traversing densely populated nodes

[Aliabbas Petiwala]

can you please mail me the code, dataset and results as we too need to
evaluate neo4j and your code will come handy
thanx

On 6/29/11, Agelos Pikoulas  wrote:
> My problem pattern is exactly the same as Niels's :
>
> A dense-node has millions of relations of a certain direction & type,
> and only a few (sparse) relations of a different direction and type.
> The traversing is usually following only those sparse relationships on those
> dense-nodes.
>
> Now, even when traversing on these sparse relations, neo4j becomes extremely
> slow
> on a certainly non linear Order (the big cs O).
>
> Some tests I run (email me if u want the code) reveal that even the number
> of those dense-nodes in the database greatly influences the results.
>
> I just reported to Michael the runs with the latest M05 snapshot, which are
> not very positive...
> I have suggested an (auto) indexing of relationship types / direction that
> is used by traversing frameworks,
> but I ain't no graphdb-engine expert :-(
>
> A'
>
>
> Message: 5
>> Date: Wed, 29 Jun 2011 18:19:10 +0200
>> From: Niels Hoogeveen 
>> Subject: Re: [Neo4j] traversing densely populated nodes
>> To: 
>> Message-ID: 
>> Content-Type: text/plain; charset="iso-8859-1"
>>
>>
>> Michael,
>>
>>
>>
>> The issue I am refering to does not pertain to traversing many relations
>> at
>> once
>>
>> but the impact many relationship of one type have on relationships
>>
>> of another type on the same node.
>>
>>
>>
>> Example:
>>
>>
>>
>> A topic class has 2 million outgoing relationships of type "HAS_INSTANCE"
>> and
>>
>> has 3 outgoing relationships of type "SUB_CLASS_OF".
>>
>>
>>
>> Fetching the 3 relations of type "SUB_CLASS_OF" takes very long,
>>
>> I presume due to the presence of the 2 million other relationships.
>>
>>
>>
>> I have no need to ever fetch the "HAS_INSTANCE" relationships from
>>
>> the topic node. That relation is always traversed from the other
>> direction.
>>
>>
>>
>> I do want to know the class of a topic instance, leading to he topic
>> class,
>>
>> but have no real interest ever to traverse all topic instance from  the
>> topic
>>
>> class (at least not directly.. i do want to know the most recent addition,
>>
>> and that's what I use the timeline index for).
>>
>>
>>
>> Niels
>>
>>
>> > From: michael.hun...@neotechnology.com
>> > Date: Wed, 29 Jun 2011 17:50:08 +0200
>> > To: user@lists.neo4j.org
>> > Subject: Re: [Neo4j] traversing densely populated nodes
>> >
>> > I think this is the same problem that Angelos is facing, we are
>> > currently
>> evaluating options to improve the performance on those highly connected
>> supernodes.
>> >
>> > A traditional option is really to split them into group or even kind of
>> shard their relationships to a second layer.
>> >
>> > We're looking into storage improvement options as well as modifications
>> to retrieval of that many relationships at once.
>> >
>> > Cheers
>> >
>> > Michael
>>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>


-- 
Aliabbas Petiwala
M.Tech CSE
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] PHP REST Client for Neo4j 1.3 available

[Anders Nawroth]

Hi!

Thanks for the reminder!

Regarding the wiki: did you just sign up? There's a 24h delay in place 
at the moment, before you can edit pages. You also have to confirm your 
email address to get edit permission.

/anders

On 06/30/2011 11:03 AM, Todd Chaffee wrote:
> I just wanted to remind folks that a PHP REST client based on Neo4j V1.3
> has been available for some time at
>
> http://github.com/tchaffee/Neo4J-REST-PHP-API-client
> 
>
> I am working on finishing up traversals, and other than that it is a
> complete implementation of the REST API.  It's fully unit tested
> (including exceptions) with 100% code coverage.
>
> Would it be possible to have these pages on the wiki updated to point to
> the repo above (page is locked for editing)?
>
> http://wiki.neo4j.org/content/PHP
> http://wiki.neo4j.org/content/Main_Page#Language_and_framework_bindings
>
> The version by onewheelgood seems to be no longer maintained and does
> not work for V1.3 of Neo4j.  I am committed to keeping my fork up to
> date and working with the latest stable version of Neo4j.
>
> I also have some PHP code examples I could add to the wiki if someone
> could let me know the best place to put them.
>
> Would also be great if anyone else wants to join in and contribute to
> improving the PHP REST client.
>
> Thanks,
> Todd Chaffee
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] no remote thread-safe API

[Jim Webber]

Hi Aliabbas,

> is it true that in neo4j there is no remote thread-safe API.

No. That is a false assertion. The REST API is thread safe since it wraps the 
same thread-safe code as the Java APIs.

There can still be *contention* but there is always thread safety (and ACIDity).

Jim
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] no remote thread-safe API

[Michael Hunger]

The issue there is that your client requests each have their own transaction so 
there.
If you want to execution multiple requests transactionally in one block you 
should use the new batch-rest API.

http://docs.neo4j.org/chunked/milestone/rest-api-batch-ops.html

Cheers

Michael

Am 30.06.2011 um 13:04 schrieb Jim Webber:

> Hi Aliabbas,
> 
>> is it true that in neo4j there is no remote thread-safe API.
> 
> No. That is a false assertion. The REST API is thread safe since it wraps the 
> same thread-safe code as the Java APIs.
> 
> There can still be *contention* but there is always thread safety (and 
> ACIDity).
> 
> Jim
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user

___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Unable to memory map [was java.io.IOException: Resource temporarily unavailable in MappedPersistenceWindow.java?

[Paul Bandler]

A colleague has speculated that it maybe related to permissions. Im  on a 
shared solaris box with no setuid access - can anyone elaborate on whether some 
specific access rights are required to use memory mapped IO?

Sent from my iPhone

On 30 Jun 2011, at 11:20, Paul Bandler  wrote:

> Further to an earlier posting I also notice this warning from neo4j which 
> sounds relevant..? Note that the application does continue and eventually 
> complete successfully having taken orders of magnitude longer than when run 
> on windows. This is running on solaris  -  any suggestions about what could 
> be causing this behaviour most welcome . Are there  issues with using neo4j 
> on Solaris?
> 
> Jun 30, 2011 10:58:59 AM 
> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool logWarn
> WARNING: 
> [./neo4j-advanced-1.4.M05/data/graph.db/neostore.relationshipstore.db] Unable 
> to memory map
> org.neo4j.kernel.impl.nioneo.store.MappedMemException: Unable to map 
> pos=28593 recordSize=33 totalSize=104841
>at 
> org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow.(MappedPersistenceWindow.java:61)
>at 
> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.allocateNewWindow(PersistenceWindowPool.java:603)
>at 
> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.refreshBricks(PersistenceWindowPool.java:501)
>at 
> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:128)
>at 
> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:526)
>at 
> org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getChainRecord(RelationshipStore.java:327)
>at 
> org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:114)
>at 
> org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:97)
>at 
> org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:108)
>at 
> org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:604)
>at 
> org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:403)
>at 
> org.neo4j.kernel.impl.core.IntArrayIterator.hasNext(IntArrayIterator.java:98)
>at 
> com.nomura.smo.vcs.rdm.neo4j.AbstractBaseNeo4j.getSectorRelations(AbstractBaseNeo4j.java:42)
> 
> Sent from my iPhone
> 
> On 30 Jun 2011, at 07:40, Paul Bandler  wrote:
> 
>> When running a test neo4j application on Solaris that I have previously run 
>> successfully on Windows I'm encountering the following exception:
>> 
>> Caused by: java.io.IOException: Resource temporarily unavailable
>>   at sun.nio.ch.FileChannelImpl.map0(Native Method)
>>   at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:747)
>>   at 
>> org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow.(MappedPersistenceWindow.java:53)
>> 
>> Indeed this in  turn is causing a huge knock-on effect but I've not included 
>> that stack trace for clarity.  The program appears to attempt to continue, 
>> albeit at a snails pace.
>> 
>> I'm running using the latest 1.4 milestone release, with a maximum heap of 
>> 2G, and defaulting all other store parameters.
>> 
>> Any suggestions as to what is happening would be most welcome.
>> ___
>> Neo4j mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Unable to memory map [was java.io.IOException: Resource temporarily unavailable in MappedPersistenceWindow.java?

[Michael Hunger]

Paul,

what version of Solaris are you running that on? We don't have Solaris as part 
of our build-qa workflow (yet). So I would try to see if there is an ec2 
instance that I could just use for that.

Cheers

Michael

Am 30.06.2011 um 13:26 schrieb Paul Bandler:

> A colleague has speculated that it maybe related to permissions. Im  on a 
> shared solaris box with no setuid access - can anyone elaborate on whether 
> some specific access rights are required to use memory mapped IO?
> 
> Sent from my iPhone
> 
> On 30 Jun 2011, at 11:20, Paul Bandler  wrote:
> 
>> Further to an earlier posting I also notice this warning from neo4j which 
>> sounds relevant..? Note that the application does continue and eventually 
>> complete successfully having taken orders of magnitude longer than when run 
>> on windows. This is running on solaris  -  any suggestions about what could 
>> be causing this behaviour most welcome . Are there  issues with using neo4j 
>> on Solaris?
>> 
>> Jun 30, 2011 10:58:59 AM 
>> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool logWarn
>> WARNING: 
>> [./neo4j-advanced-1.4.M05/data/graph.db/neostore.relationshipstore.db] 
>> Unable to memory map
>> org.neo4j.kernel.impl.nioneo.store.MappedMemException: Unable to map 
>> pos=28593 recordSize=33 totalSize=104841
>>   at 
>> org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow.(MappedPersistenceWindow.java:61)
>>   at 
>> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.allocateNewWindow(PersistenceWindowPool.java:603)
>>   at 
>> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.refreshBricks(PersistenceWindowPool.java:501)
>>   at 
>> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:128)
>>   at 
>> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:526)
>>   at 
>> org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getChainRecord(RelationshipStore.java:327)
>>   at 
>> org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:114)
>>   at 
>> org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:97)
>>   at 
>> org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:108)
>>   at 
>> org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:604)
>>   at 
>> org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:403)
>>   at 
>> org.neo4j.kernel.impl.core.IntArrayIterator.hasNext(IntArrayIterator.java:98)
>>   at 
>> com.nomura.smo.vcs.rdm.neo4j.AbstractBaseNeo4j.getSectorRelations(AbstractBaseNeo4j.java:42)
>> 
>> Sent from my iPhone
>> 
>> On 30 Jun 2011, at 07:40, Paul Bandler  wrote:
>> 
>>> When running a test neo4j application on Solaris that I have previously run 
>>> successfully on Windows I'm encountering the following exception:
>>> 
>>> Caused by: java.io.IOException: Resource temporarily unavailable
>>>  at sun.nio.ch.FileChannelImpl.map0(Native Method)
>>>  at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:747)
>>>  at 
>>> org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow.(MappedPersistenceWindow.java:53)
>>> 
>>> Indeed this in  turn is causing a huge knock-on effect but I've not 
>>> included that stack trace for clarity.  The program appears to attempt to 
>>> continue, albeit at a snails pace.
>>> 
>>> I'm running using the latest 1.4 milestone release, with a maximum heap of 
>>> 2G, and defaulting all other store parameters.
>>> 
>>> Any suggestions as to what is happening would be most welcome.
>>> ___
>>> Neo4j mailing list
>>> User@lists.neo4j.org
>>> https://lists.neo4j.org/mailman/listinfo/user
>> ___
>> Neo4j mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user

___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] traversing densely populated nodes

[Craig Taverner]

This topics has come up before, and the domain level solutions are usually
very similar, like Norbert's category/proxy nodes (to group by
type/direction) and Niels' TimeLineIndex (BTree). I wonder whether we can
build a generic user-level solution that can also be wrapped to appear as an
internal database solution?

For example, consider Niels's solution of the TimeLine index. In this case
we group all the nodes based on a consistent hash. Usually the timeline
would use a timestamp, but really any reasonably variable property can do,
even the node-id itself. Then we have a BTree between the dense nodes and
the root node (node with too many relationships). How about this crazy idea,
create an API that mimics the normal node.getRelationship*() API, but
internally traverses the entire tree? And also for creating the
relationships? So for most cod we just do the usual
node.createRelationshipTo(node,type,direction) and node.traverse(...), but
internally we actually traverse the b-tree.

This would solve the performance bottleneck being observed while keeping the
'illusion' of directly connected relationships. The solution would be
implemented mostly in the application space, so will not need any changes to
the core database. I see this as being of the same kind of solution as the
auto-indexing. We setup some initial configuration that results in certain
structures being created on demand. With auto-indexing we are talking about
mostly automatically adding lucene indexes. With this idea we are talking
about automatically replacing direct relationships with b-trees to resolve a
specific performance issue.

And when the relationship density is very low, if the b-tree is
auto-balancing, it could just be a direct relationship anyway.

On Wed, Jun 29, 2011 at 6:56 PM, Agelos Pikoulas
wrote:

> My problem pattern is exactly the same as Niels's :
>
> A dense-node has millions of relations of a certain direction & type,
> and only a few (sparse) relations of a different direction and type.
> The traversing is usually following only those sparse relationships on
> those
> dense-nodes.
>
> Now, even when traversing on these sparse relations, neo4j becomes
> extremely
> slow
> on a certainly non linear Order (the big cs O).
>
> Some tests I run (email me if u want the code) reveal that even the number
> of those dense-nodes in the database greatly influences the results.
>
> I just reported to Michael the runs with the latest M05 snapshot, which are
> not very positive...
> I have suggested an (auto) indexing of relationship types / direction that
> is used by traversing frameworks,
> but I ain't no graphdb-engine expert :-(
>
> A'
>
>
> Message: 5
> > Date: Wed, 29 Jun 2011 18:19:10 +0200
> > From: Niels Hoogeveen 
> > Subject: Re: [Neo4j] traversing densely populated nodes
> > To: 
> > Message-ID: 
> > Content-Type: text/plain; charset="iso-8859-1"
> >
> >
> > Michael,
> >
> >
> >
> > The issue I am refering to does not pertain to traversing many relations
> at
> > once
> >
> > but the impact many relationship of one type have on relationships
> >
> > of another type on the same node.
> >
> >
> >
> > Example:
> >
> >
> >
> > A topic class has 2 million outgoing relationships of type "HAS_INSTANCE"
> > and
> >
> > has 3 outgoing relationships of type "SUB_CLASS_OF".
> >
> >
> >
> > Fetching the 3 relations of type "SUB_CLASS_OF" takes very long,
> >
> > I presume due to the presence of the 2 million other relationships.
> >
> >
> >
> > I have no need to ever fetch the "HAS_INSTANCE" relationships from
> >
> > the topic node. That relation is always traversed from the other
> direction.
> >
> >
> >
> > I do want to know the class of a topic instance, leading to he topic
> class,
> >
> > but have no real interest ever to traverse all topic instance from  the
> > topic
> >
> > class (at least not directly.. i do want to know the most recent
> addition,
> >
> > and that's what I use the timeline index for).
> >
> >
> >
> > Niels
> >
> >
> > > From: michael.hun...@neotechnology.com
> > > Date: Wed, 29 Jun 2011 17:50:08 +0200
> > > To: user@lists.neo4j.org
> > > Subject: Re: [Neo4j] traversing densely populated nodes
> > >
> > > I think this is the same problem that Angelos is facing, we are
> currently
> > evaluating options to improve the performance on those highly connected
> > supernodes.
> > >
> > > A traditional option is really to split them into group or even kind of
> > shard their relationships to a second layer.
> > >
> > > We're looking into storage improvement options as well as modifications
> > to retrieval of that many relationships at once.
> > >
> > > Cheers
> > >
> > > Michael
> >
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] traversing densely populated nodes

[Michael Hunger]

The issue being that relationships on disk are not ordered, so that, even when 
just accessing the few relationships of the one
type you still have to scan all rels.

For supporting different approaches you either have to change the store-format 
to handle the storage and loading of relationships of supernodes differently.

Or: 
- if you'd rather want to avoid supernodes you can have an Expander in your 
traversers that delays the traversals of supernodes (perhaps
the other nodes give you already enough results for the domain to consume). 
(You could also limit/timebox that).
- You can also have Expanders that use an B-Tree or Relationship index 
internally (for the "few"-rels). With autoindexing you could actually put a 
property on those few "rels" that makes them automatically indexed. And then in 
the Expander use
relIndex.get(startNode,null,null) to get all of them for expansion.

It can be useful to get something like this implemented once and then offered 
to all users either as part of the product or perhaps in a component like 
neo4j-collections ? (these are just ideas)

Cheers

Michael

Am 30.06.2011 um 13:57 schrieb Craig Taverner:

> This topics has come up before, and the domain level solutions are usually
> very similar, like Norbert's category/proxy nodes (to group by
> type/direction) and Niels' TimeLineIndex (BTree). I wonder whether we can
> build a generic user-level solution that can also be wrapped to appear as an
> internal database solution?
> 
> For example, consider Niels's solution of the TimeLine index. In this case
> we group all the nodes based on a consistent hash. Usually the timeline
> would use a timestamp, but really any reasonably variable property can do,
> even the node-id itself. Then we have a BTree between the dense nodes and
> the root node (node with too many relationships). How about this crazy idea,
> create an API that mimics the normal node.getRelationship*() API, but
> internally traverses the entire tree? And also for creating the
> relationships? So for most cod we just do the usual
> node.createRelationshipTo(node,type,direction) and node.traverse(...), but
> internally we actually traverse the b-tree.
> 
> This would solve the performance bottleneck being observed while keeping the
> 'illusion' of directly connected relationships. The solution would be
> implemented mostly in the application space, so will not need any changes to
> the core database. I see this as being of the same kind of solution as the
> auto-indexing. We setup some initial configuration that results in certain
> structures being created on demand. With auto-indexing we are talking about
> mostly automatically adding lucene indexes. With this idea we are talking
> about automatically replacing direct relationships with b-trees to resolve a
> specific performance issue.
> 
> And when the relationship density is very low, if the b-tree is
> auto-balancing, it could just be a direct relationship anyway.
> 
> On Wed, Jun 29, 2011 at 6:56 PM, Agelos Pikoulas
> wrote:
> 
>> My problem pattern is exactly the same as Niels's :
>> 
>> A dense-node has millions of relations of a certain direction & type,
>> and only a few (sparse) relations of a different direction and type.
>> The traversing is usually following only those sparse relationships on
>> those
>> dense-nodes.
>> 
>> Now, even when traversing on these sparse relations, neo4j becomes
>> extremely
>> slow
>> on a certainly non linear Order (the big cs O).
>> 
>> Some tests I run (email me if u want the code) reveal that even the number
>> of those dense-nodes in the database greatly influences the results.
>> 
>> I just reported to Michael the runs with the latest M05 snapshot, which are
>> not very positive...
>> I have suggested an (auto) indexing of relationship types / direction that
>> is used by traversing frameworks,
>> but I ain't no graphdb-engine expert :-(
>> 
>> A'
>> 
>> 
>> Message: 5
>>> Date: Wed, 29 Jun 2011 18:19:10 +0200
>>> From: Niels Hoogeveen 
>>> Subject: Re: [Neo4j] traversing densely populated nodes
>>> To: 
>>> Message-ID: 
>>> Content-Type: text/plain; charset="iso-8859-1"
>>> 
>>> 
>>> Michael,
>>> 
>>> 
>>> 
>>> The issue I am refering to does not pertain to traversing many relations
>> at
>>> once
>>> 
>>> but the impact many relationship of one type have on relationships
>>> 
>>> of another type on the same node.
>>> 
>>> 
>>> 
>>> Example:
>>> 
>>> 
>>> 
>>> A topic class has 2 million outgoing relationships of type "HAS_INSTANCE"
>>> and
>>> 
>>> has 3 outgoing relationships of type "SUB_CLASS_OF".
>>> 
>>> 
>>> 
>>> Fetching the 3 relations of type "SUB_CLASS_OF" takes very long,
>>> 
>>> I presume due to the presence of the 2 million other relationships.
>>> 
>>> 
>>> 
>>> I have no need to ever fetch the "HAS_INSTANCE" relationships from
>>> 
>>> the topic node. That relation is always traversed from the other
>> direction.
>>> 
>>> 
>>> 
>>> I do want to know the class of a topic instance, leading t

Re: [Neo4j] traversing densely populated nodes

[Niels Hoogeveen]

Michael,
Is it possible/feasible to change the store-format so relationships are stored 
partitioned per relationshiptype per direction? What would be the impact of 
such change?
Niels

> From: michael.hun...@neotechnology.com
> Date: Thu, 30 Jun 2011 14:10:38 +0200
> To: user@lists.neo4j.org
> Subject: Re: [Neo4j] traversing densely populated nodes
> 
> The issue being that relationships on disk are not ordered, so that, even 
> when just accessing the few relationships of the one
> type you still have to scan all rels.
> 
> For supporting different approaches you either have to change the 
> store-format to handle the storage and loading of relationships of supernodes 
> differently.
> 
> Or: 
> - if you'd rather want to avoid supernodes you can have an Expander in your 
> traversers that delays the traversals of supernodes (perhaps
> the other nodes give you already enough results for the domain to consume). 
> (You could also limit/timebox that).
> - You can also have Expanders that use an B-Tree or Relationship index 
> internally (for the "few"-rels). With autoindexing you could actually put a 
> property on those few "rels" that makes them automatically indexed. And then 
> in the Expander use
> relIndex.get(startNode,null,null) to get all of them for expansion.
> 
> It can be useful to get something like this implemented once and then offered 
> to all users either as part of the product or perhaps in a component like 
> neo4j-collections ? (these are just ideas)
> 
> Cheers
> 
> Michael
> 
> Am 30.06.2011 um 13:57 schrieb Craig Taverner:
> 
> > This topics has come up before, and the domain level solutions are usually
> > very similar, like Norbert's category/proxy nodes (to group by
> > type/direction) and Niels' TimeLineIndex (BTree). I wonder whether we can
> > build a generic user-level solution that can also be wrapped to appear as an
> > internal database solution?
> > 
> > For example, consider Niels's solution of the TimeLine index. In this case
> > we group all the nodes based on a consistent hash. Usually the timeline
> > would use a timestamp, but really any reasonably variable property can do,
> > even the node-id itself. Then we have a BTree between the dense nodes and
> > the root node (node with too many relationships). How about this crazy idea,
> > create an API that mimics the normal node.getRelationship*() API, but
> > internally traverses the entire tree? And also for creating the
> > relationships? So for most cod we just do the usual
> > node.createRelationshipTo(node,type,direction) and node.traverse(...), but
> > internally we actually traverse the b-tree.
> > 
> > This would solve the performance bottleneck being observed while keeping the
> > 'illusion' of directly connected relationships. The solution would be
> > implemented mostly in the application space, so will not need any changes to
> > the core database. I see this as being of the same kind of solution as the
> > auto-indexing. We setup some initial configuration that results in certain
> > structures being created on demand. With auto-indexing we are talking about
> > mostly automatically adding lucene indexes. With this idea we are talking
> > about automatically replacing direct relationships with b-trees to resolve a
> > specific performance issue.
> > 
> > And when the relationship density is very low, if the b-tree is
> > auto-balancing, it could just be a direct relationship anyway.
> > 
> > On Wed, Jun 29, 2011 at 6:56 PM, Agelos Pikoulas
> > wrote:
> > 
> >> My problem pattern is exactly the same as Niels's :
> >> 
> >> A dense-node has millions of relations of a certain direction & type,
> >> and only a few (sparse) relations of a different direction and type.
> >> The traversing is usually following only those sparse relationships on
> >> those
> >> dense-nodes.
> >> 
> >> Now, even when traversing on these sparse relations, neo4j becomes
> >> extremely
> >> slow
> >> on a certainly non linear Order (the big cs O).
> >> 
> >> Some tests I run (email me if u want the code) reveal that even the number
> >> of those dense-nodes in the database greatly influences the results.
> >> 
> >> I just reported to Michael the runs with the latest M05 snapshot, which are
> >> not very positive...
> >> I have suggested an (auto) indexing of relationship types / direction that
> >> is used by traversing frameworks,
> >> but I ain't no graphdb-engine expert :-(
> >> 
> >> A'
> >> 
> >> 
> >> Message: 5
> >>> Date: Wed, 29 Jun 2011 18:19:10 +0200
> >>> From: Niels Hoogeveen 
> >>> Subject: Re: [Neo4j] traversing densely populated nodes
> >>> To: 
> >>> Message-ID: 
> >>> Content-Type: text/plain; charset="iso-8859-1"
> >>> 
> >>> 
> >>> Michael,
> >>> 
> >>> 
> >>> 
> >>> The issue I am refering to does not pertain to traversing many relations
> >> at
> >>> once
> >>> 
> >>> but the impact many relationship of one type have on relationships
> >>> 
> >>> of another type on the same node.
> >>> 
> >>> 
> >>> 
> >>> Examp

Re: [Neo4j] Database engine using Neo4j

[Craig Taverner]

Hi Kriti,

I can comment on a few things, especially neo4j-spatial:

   - Neo4j is certainly good for social networks, and people have used it
   for that, but I personally do not have experience with that so I will not
   comment further (others can chip in where necessary).
   - Neo4j-Spatial is good for performing some spatial queries on your
   domain data. So you start by modeling your domain however you want, and then
   when you want to start using neo4j-spatial, just add all nodes that have
   spatial components (eg. location) to the spatial index and they will be
   available for querying. The SimplePointLayer class has support for querying
   by proximity, which sounds like what you want. You can also query with a
   filter on properties (so only nearby objects matching some other criteria).
   - I do my neo4j-spatial development in eclipse, so there should be no
   issues for you using eclipse. Just use m2eclipse, and add the dependency to
   your pom.xml. The current version o neo4j-spatial requires neo4j1.4, so if
   you are using older neo4j, you might need to make minor changes.
   - Neo4j is not optimized for storing BLOBs, so while it can store images
   as byte[], it is advisable to rather store a reference to the image (eg.
   URI), and store the image in another way (filesystem, other database, etc.)

Regards, Craig

On Wed, Jun 29, 2011 at 2:06 PM, kriti sharma  wrote:

> Dear Users,
>
> I am developing a time capsule DB engine using Neo4j as a database.
> I intend to develop three scales (temporal , geo/spatial and
> egocentric/personal relationships) in the db structure.
> for the geolocation part, i would like to be able to query upon a location
> keyword and also some nearby places/photos/people that i have in my DB.
>
> Do you think neo4j spatial will be a good choice for such a spatial scheme?
> I have developed a timeline in the usual neo4j using timeline feature. Can
> I
> simply integrate neo4j spatial in my existing code for neo4j in eclipse?
>
> i am retrieving data from twitter, flickr, facebook etc. so the format of
> data may not be uniform. Therefore i found Neo4j to be an excellent option.
> Has some work been done in modelling a user's Facebook data(friends and
> networks) relationships in Neo4j?
>
> How should I go about storing images in the DB?
>
> Thanks
> Kriti
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

[Neo4j] PHP REST Client for Neo4j 1.3 available

[Todd Chaffee]

Hi Anders,

I did just sign up to the wiki.  I'll wait 24 hours and give it another try.

Thanks,
Todd

On 30/06/11 12:57, user-requ...@lists.neo4j.org wrote:
> Message: 3 Date: Thu, 30 Jun 2011 12:51:42 +0200 From: Anders Nawroth 
>  Subject: Re: [Neo4j] PHP REST Client for 
> Neo4j 1.3 available To: Neo4j user discussions  
> Message-ID: <4e0c553e.1030...@neotechnology.com> Content-Type: 
> text/plain; charset=ISO-8859-1; format=flowed Hi! Thanks for the 
> reminder! Regarding the wiki: did you just sign up? There's a 24h 
> delay in place at the moment, before you can edit pages. You also have 
> to confirm your email address to get edit permission. /anders On 
> 06/30/2011 11:03 AM, Todd Chaffee wrote:
>> >  I just wanted to remind folks that a PHP REST client based on Neo4j V1.3
>> >  has been available for some time at
>> >
>> >  http://github.com/tchaffee/Neo4J-REST-PHP-API-client
>> >  
>> >
>> >  I am working on finishing up traversals, and other than that it is a
>> >  complete implementation of the REST API.  It's fully unit tested
>> >  (including exceptions) with 100% code coverage.
>> >
>> >  Would it be possible to have these pages on the wiki updated to point to
>> >  the repo above (page is locked for editing)?
>> >
>> >  http://wiki.neo4j.org/content/PHP
>> >  http://wiki.neo4j.org/content/Main_Page#Language_and_framework_bindings
>> >
>> >  The version by onewheelgood seems to be no longer maintained and does
>> >  not work for V1.3 of Neo4j.  I am committed to keeping my fork up to
>> >  date and working with the latest stable version of Neo4j.
>> >
>> >  I also have some PHP code examples I could add to the wiki if someone
>> >  could let me know the best place to put them.
>> >
>> >  Would also be great if anyone else wants to join in and contribute to
>> >  improving the PHP REST client.
>> >
>> >  Thanks,
>> >  Todd Chaffee
>> >  ___
>> >  Neo4j mailing list
>> >  User@lists.neo4j.org
>> >  https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Can I model a Graph data structure without persisting it in DB using Neo4j ?

[V]

Any suggestions on this please ?
It is really acting as a blocker for me. I have tried multiple alternatives,
but I come up with one issue or the other.


On Wed, Jun 22, 2011 at 8:51 PM, V  wrote:

> I am confused, elements is defined as
>
> @RelatedTo(type = "ELEMENT", elementClass = Element.class, direction =
> OUTGOING)
> private Set elements;
>
> if I call getElements() without persisting the node first, will it not
> return null ?
>
> It is throwing null pointer exception.
>
> When I persist the node before calling getElements, then it returns empty
> set. This is in line with my original email.
>
> Do you want me to do something else? Please let me know
>
>
>
>
> On Wed, Jun 22, 2011 at 8:39 PM, Michael Hunger <
> michael.hun...@neotechnology.com> wrote:
>
>> getElements() returns a set of your elements. If you do an add(element) on
>> those then you build up an in-memory structure.
>>
>> So if you use just the getters and setters of your instances you can
>> navigate this structure, but you can't use any of the graph operations
>> supplied by neo4j (remember it is still just in memory)
>>
>> Cheers
>>
>> Michael
>>
>> Am 22.06.2011 um 17:06 schrieb V:
>>
>> Michael -
>>
>> In my quest for a cleaner solution , I am coming back to your original
>> reply
>>
>> What do you mean by *"Right now this works only with the
>> getElements().add method."*
>> Please elaborate .
>>
>> Thanks for your help .
>>
>>
>> On Sun, Jun 19, 2011 at 2:34 AM, Michael Hunger <
>> michael.hun...@neotechnology.com> wrote:
>>
>>> Right now this works only with the getElements().add method.
>>>
>>> But then you don't get all the graph methods (traversals and such).
>>>
>>> We discussed some time ago writing a virtual graph layer on top of the
>>> real nodes and relationships for SDG.
>>> (That should be then used to have a simpler handling of attached/detached
>>> nodes and keeping back-references for cluster-persistance).
>>>
>>>
>>> But that hasn't happend yet and I'm not convinced it will make it in the
>>> 1.1. timeframe.
>>>
>>> Cheers
>>>
>>> Michael
>>>
>>> P.S. You can still persist your graph and remove the nodes later (either
>>> index them on a "transient" index or keep their node-id's somewhere).
>>>
>>>
>>> Am 18.06.2011 um 03:40 schrieb V:
>>>
>>> Any suggestions on this please ?
>>>
>>>
>>> On Fri, Jun 17, 2011 at 11:03 PM, V  wrote:
>>>
 I created a graph model with 2 classes Node and Element as follows:

 public class Node {

  @RelatedTo(type = "ELEMENT", elementClass = Element.class, direction =
 OUTGOING)
 private Set Element;


 public void addElement(Element e) {
 relateTo(e, Relationships.ELEMENT.toString());
 }

 }

 public class Element{

public String name;

 }
 I want to create an in memory graph structure without persisting the
 nodes as follows :

 Node n = new Node();
 n.addElement(new Element());

 *However it throws an exception as the Node n has not been persisted so
 the call to relateTo(..) fails. *

 If instead I do
 n.persist()
 and then call addElement(..) it works fine as the aspect kicks in.

 Any workaround for this ?  That is, is there a way I can still use the
 above style without persisting the Node object ?


 My application needs this as first I create a structure and persist it,
 and then I create another structure to pass around some values to the
 persisted structure when doing some computations.

 -Karan




>>>
>>>
>>
>>
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

[Neo4j] Unique index nodes

[V]

I have an element of my node marked with
@Indexed
String name

I want to ensure that when I persist nodes, only one node with that name
gets persisted.  (I want to throw an error here and ask the user to give a
different name)
What is the best way to ensure this. Any suggestions  ?

- Karan
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Unable to memory map [was java.io.IOException: Resource temporarily unavailable in MappedPersistenceWindow.java?

[Paul Bandler]

TEST:bandlerp@us2187$ uname -X
System = SunOS
Node = us2187
Release = 5.10
KernelID = Generic_13-03
Machine = sun4u
BusType = 
Serial = 
Users = 
OEM# = 0
Origin# = 1
NumCPU = 16

Sent from my iPhone

On 30 Jun 2011, at 12:53, Michael Hunger  
wrote:

> Paul,
> 
> what version of Solaris are you running that on? We don't have Solaris as 
> part of our build-qa workflow (yet). So I would try to see if there is an ec2 
> instance that I could just use for that.
> 
> Cheers
> 
> Michael
> 
> Am 30.06.2011 um 13:26 schrieb Paul Bandler:
> 
>> A colleague has speculated that it maybe related to permissions. Im  on a 
>> shared solaris box with no setuid access - can anyone elaborate on whether 
>> some specific access rights are required to use memory mapped IO?
>> 
>> Sent from my iPhone
>> 
>> On 30 Jun 2011, at 11:20, Paul Bandler  wrote:
>> 
>>> Further to an earlier posting I also notice this warning from neo4j which 
>>> sounds relevant..? Note that the application does continue and eventually 
>>> complete successfully having taken orders of magnitude longer than when run 
>>> on windows. This is running on solaris  -  any suggestions about what could 
>>> be causing this behaviour most welcome . Are there  issues with using neo4j 
>>> on Solaris?
>>> 
>>> Jun 30, 2011 10:58:59 AM 
>>> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool logWarn
>>> WARNING: 
>>> [./neo4j-advanced-1.4.M05/data/graph.db/neostore.relationshipstore.db] 
>>> Unable to memory map
>>> org.neo4j.kernel.impl.nioneo.store.MappedMemException: Unable to map 
>>> pos=28593 recordSize=33 totalSize=104841
>>>  at 
>>> org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow.(MappedPersistenceWindow.java:61)
>>>  at 
>>> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.allocateNewWindow(PersistenceWindowPool.java:603)
>>>  at 
>>> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.refreshBricks(PersistenceWindowPool.java:501)
>>>  at 
>>> org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.acquire(PersistenceWindowPool.java:128)
>>>  at 
>>> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.acquireWindow(CommonAbstractStore.java:526)
>>>  at 
>>> org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getChainRecord(RelationshipStore.java:327)
>>>  at 
>>> org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:114)
>>>  at 
>>> org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:97)
>>>  at 
>>> org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:108)
>>>  at 
>>> org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:604)
>>>  at 
>>> org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:403)
>>>  at 
>>> org.neo4j.kernel.impl.core.IntArrayIterator.hasNext(IntArrayIterator.java:98)
>>>  at 
>>> com.nomura.smo.vcs.rdm.neo4j.AbstractBaseNeo4j.getSectorRelations(AbstractBaseNeo4j.java:42)
>>> 
>>> Sent from my iPhone
>>> 
>>> On 30 Jun 2011, at 07:40, Paul Bandler  wrote:
>>> 
 When running a test neo4j application on Solaris that I have previously 
 run successfully on Windows I'm encountering the following exception:
 
 Caused by: java.io.IOException: Resource temporarily unavailable
 at sun.nio.ch.FileChannelImpl.map0(Native Method)
 at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:747)
 at 
 org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow.(MappedPersistenceWindow.java:53)
 
 Indeed this in  turn is causing a huge knock-on effect but I've not 
 included that stack trace for clarity.  The program appears to attempt to 
 continue, albeit at a snails pace.
 
 I'm running using the latest 1.4 milestone release, with a maximum heap of 
 2G, and defaulting all other store parameters.
 
 Any suggestions as to what is happening would be most welcome.
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user
>>> ___
>>> Neo4j mailing list
>>> User@lists.neo4j.org
>>> https://lists.neo4j.org/mailman/listinfo/user
>> ___
>> Neo4j mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
> 
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Unique index nodes

[Matt Chambers]

I asked this same question a little while ago, the answer is that its up to
you to guarantee uniqueness.  There are a few different ways to to this, for
now I took the easy way out and do all writes with a single thread.  I'm
basically just using a ThreadPoolExecutor with a single thread and different
command objects implementing Callable.

-Matt

On Thu, Jun 30, 2011 at 6:03 AM, V  wrote:

> I have an element of my node marked with
> @Indexed
> String name
>
> I want to ensure that when I persist nodes, only one node with that name
> gets persisted.  (I want to throw an error here and ask the user to give a
> different name)
> What is the best way to ensure this. Any suggestions  ?
>
> - Karan
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] BatchInserter with Lucene Index

[Johan Svensson]

Hi Dario,

Could you post the error message and stacktrace?

Did the error happen after initial import but still running in batch
inserter mode or normal server/embedded transactional mode?

Regards,
Johan

On Wed, Jun 29, 2011 at 4:30 PM, Dario Rexin  wrote:
> Hi all,
>
> Recently i tried import a huge dataset into neo using the BatchInserter. I 
> also used the BatchInserterIndex. The import itself went very well and i was 
> able to read from the graph, but when i tried to insert something i always 
> got an error saying that no new transaction could be created. Here’s how I 
> used the index:
>
> // provider is a LuceneBatchInserterIndexProvider
> BatchInserterIndex urnIndex = provider.nodeIndex("urn", MapUtil.stringMap( 
> "type", "exact" ));
>
> for (Map.Entry entry : nodes.entrySet()) {
>      Map props = MapUtil.map("urn", entry.getKey());
>      urnIndex.add(entry.getValue(), props);
> }
>
> I also called shutdown() on the provider and the BatchInserter afterwards. Is 
> there anything i am missing?
>
>
> Cheers,
>
> Dario
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

[Neo4j] graph-collections List class

[Niels Hoogeveen]

Today I added a List class to the graph-collections repo:
https://github.com/peterneubauer/graph-collections/tree/master/src/main/java/org/neo4j/collections/list
The List class implements the java.util.List interface. 
Niels 
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

[Neo4j] problem with neography and index.

[Laurent Laborde]

Friendly greetings !
i'm on the same problem since many days (an hour per day) and i can't
find a solution
i have 2 index (see source doe below)
No problem with the "parsed" index, but the "url" index never return any result.
I don't if it's because the url isn't indexed or because the query on
the index is wrong.
Or something else ?

Could you please take a look and see what's wrong ?
thank you

(you can try to run the script, it works)

require 'nokogiri'
require 'open-uri'
require 'neography'

#init neography
@neo = Neography::Rest.new
neo_root = @neo.get_root

domaine = 'http://www.over-blog.com/'
parsed_idx = "ob_parsed_idx"
url_idx = "ob_url_idx"

#FIRST RUN
#ob_root_node = @neo.create_node("domaine" => domaine, "parsed" =>
"false", "url" => domaine)
#@neo.create_relationship("obgraph", neo_root, ob_root_node)
#pidx = @neo.create_node_index(parsed_idx)
#uidx = @neo.create_node_index(url_idx)
#@neo.add_node_to_index(parsed_idx, "parsed", "false", ob_root_node)
##@neo.add_node_to_index(url_idx, "url", domaine, ob_root_node)
#node_to_parse = @neo.get_node_index(parsed_idx, "parsed", "false")

ob_root_node = @neo.traverse(neo_root, "nodes", { "relationships" =>
[{"type"=> "obgraph", "direction" => "out" }], "depth" => 1})
#node_to_parse = @neo.traverse(ob_root_node, "nodes", {
"relationships" => [{"type"=> "link", "direction" => "out" }] })
node_to_parse = @neo.get_node_index(parsed_idx, "parsed", "false")

#print @neo.list_node_indexes

node_to_parse.each do |node|

url_to_parse = @neo.get_node_properties(node)["url"]
printf("exploring : %s\n", url_to_parse)

doc = Nokogiri::HTML(open(url_to_parse))
@neo.set_node_properties(node, {"parsed" => "true"})
@neo.remove_node_from_index(parsed_idx, node)
@neo.add_node_to_index(parsed_idx, "parsed", "true", node)

doc.xpath('//a').each do |link|

link_text = link.content.strip()
link_url = link['href'].to_s().strip()
link_title = link['title'].to_s().strip()

link_url = link_url.sub(/#.*$/, "")

if(link_url =~ /^\/.*/)
link_url = link_url.sub(/^\//, '')
link_url = domaine + link_url
end

if(link_text == '')
link_text = link_title
end


#skiping empty stuff
next if link_url.empty?
next if link_text.empty?

node_found = @neo.find_node_index(url_idx, "url", link_url)
#node_found = @neo.traverse(ob_root_node, "nodes", {
"relationships" => [{"direction" => "out" }], "prune evaluator" =>
{"language" => "javascript", "body" =>
"position.endNode().getProperty(url) == #{link_url};"}, "return
filter" => {"language" => "builtin",  "name" => "all but start
node"}})
print "\nsearching url #{link_url}\n"
printf("node_found : %s \n", node_found)
if(node_found.nil?)
printf("create node %s\n", link_url)
nnode = @neo.create_node("parsed" => "false", "url" => link_url)
@neo.add_node_to_index(url_idx, "url", link_url, nnode)
@neo.add_node_to_index(parsed_idx, "parsed", "false", nnode)
else
printf("node_found : %s \n", node_found)
end


nrel = @neo.create_relationship("link", node, nnode)
@neo.set_relationship_properties(nrel, {"text" => link_text})

#printf("%s => %s\n", link_text, link_url)

end

sleep(1.0)


end


-- 
Laurent "ker2x" Laborde
Sysadmin & DBA at http://www.over-blog.com/
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] graph-collections List class

[Peter Neubauer]

Guys,
if you can write some illustrative test code for the different
collections, I could undertake to put some documentation skeleton into
place that can be merged into the main manual and look like
http://docs.neo4j.org/chunked/snapshot/. To start with, we could have
it separately though.

WDYT?

Cheers,

/peter neubauer

GTalk:      neubauer.peter
Skype       peter.neubauer
Phone       +46 704 106975
LinkedIn   http://www.linkedin.com/in/neubauer
Twitter      http://twitter.com/peterneubauer

http://www.neo4j.org               - Your high performance graph database.
http://startupbootcamp.org/    - Öresund - Innovation happens HERE.
http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party.



On Thu, Jun 30, 2011 at 5:28 PM, Niels Hoogeveen
 wrote:
>
> Today I added a List class to the graph-collections repo:
> https://github.com/peterneubauer/graph-collections/tree/master/src/main/java/org/neo4j/collections/list
> The List class implements the java.util.List interface.
> Niels
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] problem with neography and index.

[Michael Hunger]

you have to escape the url index value
otherwise the jersey rest framework consumes it silently. I had this problem 
when working on the birdies demo app. Took me a while to work that out.

see http://github.com/jexp/birdies
and http://birdies.heroku.com

Michael

Sent from my iBrick4


Am 30.06.2011 um 17:43 schrieb Laurent Laborde :

> Friendly greetings !
> i'm on the same problem since many days (an hour per day) and i can't
> find a solution
> i have 2 index (see source doe below)
> No problem with the "parsed" index, but the "url" index never return any 
> result.
> I don't if it's because the url isn't indexed or because the query on
> the index is wrong.
> Or something else ?
> 
> Could you please take a look and see what's wrong ?
> thank you
> 
> (you can try to run the script, it works)
> 
> require 'nokogiri'
> require 'open-uri'
> require 'neography'
> 
> #init neography
> @neo = Neography::Rest.new
> neo_root = @neo.get_root
> 
> domaine = 'http://www.over-blog.com/'
> parsed_idx = "ob_parsed_idx"
> url_idx = "ob_url_idx"
> 
> #FIRST RUN
> #ob_root_node = @neo.create_node("domaine" => domaine, "parsed" =>
> "false", "url" => domaine)
> #@neo.create_relationship("obgraph", neo_root, ob_root_node)
> #pidx = @neo.create_node_index(parsed_idx)
> #uidx = @neo.create_node_index(url_idx)
> #@neo.add_node_to_index(parsed_idx, "parsed", "false", ob_root_node)
> ##@neo.add_node_to_index(url_idx, "url", domaine, ob_root_node)
> #node_to_parse = @neo.get_node_index(parsed_idx, "parsed", "false")
> 
> ob_root_node = @neo.traverse(neo_root, "nodes", { "relationships" =>
> [{"type"=> "obgraph", "direction" => "out" }], "depth" => 1})
> #node_to_parse = @neo.traverse(ob_root_node, "nodes", {
> "relationships" => [{"type"=> "link", "direction" => "out" }] })
> node_to_parse = @neo.get_node_index(parsed_idx, "parsed", "false")
> 
> #print @neo.list_node_indexes
> 
> node_to_parse.each do |node|
> 
>url_to_parse = @neo.get_node_properties(node)["url"]
>printf("exploring : %s\n", url_to_parse)
> 
>doc = Nokogiri::HTML(open(url_to_parse))
>@neo.set_node_properties(node, {"parsed" => "true"})
>@neo.remove_node_from_index(parsed_idx, node)
>@neo.add_node_to_index(parsed_idx, "parsed", "true", node)
> 
>doc.xpath('//a').each do |link|
> 
>link_text = link.content.strip()
>link_url = link['href'].to_s().strip()
>link_title = link['title'].to_s().strip()
> 
>link_url = link_url.sub(/#.*$/, "")
> 
>if(link_url =~ /^\/.*/)
>link_url = link_url.sub(/^\//, '')
>link_url = domaine + link_url
>end
> 
>if(link_text == '')
>link_text = link_title
>end
> 
> 
>#skiping empty stuff
>next if link_url.empty?
>next if link_text.empty?
> 
>node_found = @neo.find_node_index(url_idx, "url", link_url)
>#node_found = @neo.traverse(ob_root_node, "nodes", {
> "relationships" => [{"direction" => "out" }], "prune evaluator" =>
> {"language" => "javascript", "body" =>
> "position.endNode().getProperty(url) == #{link_url};"}, "return
> filter" => {"language" => "builtin",  "name" => "all but start
> node"}})
>print "\nsearching url #{link_url}\n"
>printf("node_found : %s \n", node_found)
>if(node_found.nil?)
>printf("create node %s\n", link_url)
>nnode = @neo.create_node("parsed" => "false", "url" => link_url)
>@neo.add_node_to_index(url_idx, "url", link_url, nnode)
>@neo.add_node_to_index(parsed_idx, "parsed", "false", nnode)
>else
>printf("node_found : %s \n", node_found)
>end
> 
> 
>nrel = @neo.create_relationship("link", node, nnode)
>@neo.set_relationship_properties(nrel, {"text" => link_text})
> 
>#printf("%s => %s\n", link_text, link_url)
> 
>end
> 
>sleep(1.0)
> 
> 
> end
> 
> 
> -- 
> Laurent "ker2x" Laborde
> Sysadmin & DBA at http://www.over-blog.com/
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Unique index nodes

[Michael Hunger]

I'll look into that in the next few days.

Michael

Sent from my iBrick4


Am 30.06.2011 um 16:25 schrieb Matt Chambers :

> I asked this same question a little while ago, the answer is that its up to
> you to guarantee uniqueness.  There are a few different ways to to this, for
> now I took the easy way out and do all writes with a single thread.  I'm
> basically just using a ThreadPoolExecutor with a single thread and different
> command objects implementing Callable.
> 
> -Matt
> 
> On Thu, Jun 30, 2011 at 6:03 AM, V  wrote:
> 
>> I have an element of my node marked with
>> @Indexed
>> String name
>> 
>> I want to ensure that when I persist nodes, only one node with that name
>> gets persisted.  (I want to throw an error here and ask the user to give a
>> different name)
>> What is the best way to ensure this. Any suggestions  ?
>> 
>> - Karan
>> ___
>> Neo4j mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>> 
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] graph-collections List class

[Niels Hoogeveen]

For neo4j.graph-collections.list.List I suggest the following.
The graph collections List class is an implementation of the java.util.List 
class, parameterized with Node, supporting all methods, except subList(int 
start, int end). 
Whenever nodes need to be stored as a list, the List class can be used, 
guaranteeing the List contract is maintained.
package org.neo4j.collections.list;

import org.neo4j.collections.list.List;
import org.neo4j.graphdb.Node;
import org.neo4j.kernel.EmbeddedGraphDatabase;
import java.util.ArrayList;
import org.neo4j.graphdb.Transaction;

public class Demo {

public static void main(String[] args){



//create/load database
EmbeddedGraphDatabase graphDb = new EmbeddedGraphDatabase( 
"var/db" );

Transaction tx = graphDb.beginTx();

//create node representing the list
Node indexNode = graphDb.createNode();

//create list
List list = new List( indexNode, graphDb );

//create nodes to be added to the list
Node node1 = graphDb.createNode();
Node node2 = graphDb.createNode();
Node node3 = graphDb.createNode();
Node node4 = graphDb.createNode();

//add nodes to the list one by one
list.add(node1);
list.add(node2);
list.add(node3);
list.add(node4);

//remove entries by index
list.remove(3);
list.remove(2);

//remove entries by node
list.remove(node2);
list.remove(node1);

//add a collection of nodes to the list
ArrayList al1 = new ArrayList();
al1.add(node1);
al1.add(node2);
al1.add(node3);
al1.add(node4);

list.addAll(al1);

//use the list in a for loop
for(Node n: list){
  System.out.println(n);
}

//find first entry in the list containing a specific node
list.indexOf(node2);

//find last entry in the list containing a specific node
list.lastIndexOf(node2);

//retain a collection of nodes from the list
ArrayList al2 = new ArrayList();
al2.add(node1);
al2.add(node3);
al2.add(node4);

list.retainAll(al2);

//remove a collection of nodes from the list
ArrayList al3 = new ArrayList();
al2.add(node3);
al2.add(node4);

list.removeAll(al3);

//delete the list
list.delete();

tx.success();
tx.finish();
}
}


> From: peter.neuba...@neotechnology.com
> Date: Thu, 30 Jun 2011 17:49:46 +0200
> To: user@lists.neo4j.org
> Subject: Re: [Neo4j] graph-collections List class
> 
> Guys,
> if you can write some illustrative test code for the different
> collections, I could undertake to put some documentation skeleton into
> place that can be merged into the main manual and look like
> http://docs.neo4j.org/chunked/snapshot/. To start with, we could have
> it separately though.
> 
> WDYT?
> 
> Cheers,
> 
> /peter neubauer
> 
> GTalk:  neubauer.peter
> Skype   peter.neubauer
> Phone   +46 704 106975
> LinkedIn   http://www.linkedin.com/in/neubauer
> Twitter  http://twitter.com/peterneubauer
> 
> http://www.neo4j.org   - Your high performance graph database.
> http://startupbootcamp.org/- Öresund - Innovation happens HERE.
> http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party.
> 
> 
> 
> On Thu, Jun 30, 2011 at 5:28 PM, Niels Hoogeveen
>  wrote:
> >
> > Today I added a List class to the graph-collections repo:
> > https://github.com/peterneubauer/graph-collections/tree/master/src/main/java/org/neo4j/collections/list
> > The List class implements the java.util.List interface.
> > Niels
> > ___
> > Neo4j mailing list
> > User@lists.neo4j.org
> > https://lists.neo4j.org/mailman/listinfo/user
> >
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
  
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] graph-collections List class

[Mattias Persson]

I have done a lot of these structures already, so before (re)doing a lot of
these structures take a look at:
https://svn.neo4j.org/components/utils/trunk/src/main/java/org/neo4j/util/(the
old subversion repository, but the utils are quite up2date). Please
take a look there and feel free to pull good implementations in to
graph-collections. There are f.ex. queues (with workers picking items from
them), linked lists, limited-sized lists and more.

2011/6/30 Niels Hoogeveen 

>
> For neo4j.graph-collections.list.List I suggest the following.
> The graph collections List class is an implementation of the java.util.List
> class, parameterized with Node, supporting all methods, except subList(int
> start, int end).
> Whenever nodes need to be stored as a list, the List class can be used,
> guaranteeing the List contract is maintained.
> package org.neo4j.collections.list;
>
> import org.neo4j.collections.list.List;
> import org.neo4j.graphdb.Node;
> import org.neo4j.kernel.EmbeddedGraphDatabase;
> import java.util.ArrayList;
> import org.neo4j.graphdb.Transaction;
>
> public class Demo {
>
>public static void main(String[] args){
>
>
>
>//create/load database
>EmbeddedGraphDatabase graphDb = new EmbeddedGraphDatabase(
> "var/db" );
>
>Transaction tx = graphDb.beginTx();
>
>//create node representing the list
>Node indexNode = graphDb.createNode();
>
>//create list
>List list = new List( indexNode, graphDb );
>
>//create nodes to be added to the list
>Node node1 = graphDb.createNode();
>Node node2 = graphDb.createNode();
>Node node3 = graphDb.createNode();
>Node node4 = graphDb.createNode();
>
>//add nodes to the list one by one
>list.add(node1);
>list.add(node2);
>list.add(node3);
>list.add(node4);
>
>//remove entries by index
>list.remove(3);
>list.remove(2);
>
>//remove entries by node
>list.remove(node2);
>list.remove(node1);
>
>//add a collection of nodes to the list
>ArrayList al1 = new ArrayList();
>al1.add(node1);
>al1.add(node2);
>al1.add(node3);
>al1.add(node4);
>
>list.addAll(al1);
>
>//use the list in a for loop
>for(Node n: list){
>  System.out.println(n);
>}
>
>//find first entry in the list containing a specific node
>list.indexOf(node2);
>
>//find last entry in the list containing a specific node
>list.lastIndexOf(node2);
>
>//retain a collection of nodes from the list
>ArrayList al2 = new ArrayList();
>al2.add(node1);
>al2.add(node3);
>al2.add(node4);
>
>list.retainAll(al2);
>
>//remove a collection of nodes from the list
>ArrayList al3 = new ArrayList();
>al2.add(node3);
>al2.add(node4);
>
>list.removeAll(al3);
>
>//delete the list
>list.delete();
>
>tx.success();
>tx.finish();
>}
> }
>
>
> > From: peter.neuba...@neotechnology.com
> > Date: Thu, 30 Jun 2011 17:49:46 +0200
> > To: user@lists.neo4j.org
> > Subject: Re: [Neo4j] graph-collections List class
> >
> > Guys,
> > if you can write some illustrative test code for the different
> > collections, I could undertake to put some documentation skeleton into
> > place that can be merged into the main manual and look like
> > http://docs.neo4j.org/chunked/snapshot/. To start with, we could have
> > it separately though.
> >
> > WDYT?
> >
> > Cheers,
> >
> > /peter neubauer
> >
> > GTalk:  neubauer.peter
> > Skype   peter.neubauer
> > Phone   +46 704 106975
> > LinkedIn   http://www.linkedin.com/in/neubauer
> > Twitter  http://twitter.com/peterneubauer
> >
> > http://www.neo4j.org   - Your high performance graph
> database.
> > http://startupbootcamp.org/- Öresund - Innovation happens HERE.
> > http://www.thoughtmade.com - Scandinavia's coolest Bring-a-Thing party.
> >
> >
> >
> > On Thu, Jun 30, 2011 at 5:28 PM, Niels Hoogeveen
> >  wrote:
> > >
> > > Today I added a List class to the graph-collections repo:
> > >
> https://github.com/peterneubauer/graph-collections/tree/master/src/main/java/org/neo4j/collections/list
> > > The List class implements the java.util.List interface.
> > > Niels
> > > ___
> > > Neo4j mailing list
> > > User@lists.neo4j.org
> > > https://lists.neo4j.org/mailman/listinfo/user
> > >
> > __

[Neo4j] Does anyone use Solaris for neo4j app? [was Unable to memory map [was java.io.IOException: Resource temporarily unavailable in MappedPersistenceWindow.java?

[Paul Bandler]

Further to the problems reported below from running a test application on a 
server class Solaris box (M4000)  I re-ran the test but with the setting 
'use_memory_mapped_buffers' set to false.  This had the +ve effect of avoiding 
the warnings and nio exceptions mentioned before so clearly there's some 
additional magic required to run under Solaris using memory mapped buffers.  I 
have enquired as to whether one needs setuid rights to use memory mapped files 
on Solaris and am told not btw...

The performance was woeful however - like walking thru' molasses in (a Swedish) 
mid-winter!  The test comprises a relatively complex navigation of a domain 
model to derive some information.  This is a re-implementation of a function  
that  already exists in a production C++ application that uses an object 
database (based on a product called ObjectStore) -  so I have something to 
compare the results against.  I also have the results of running the same neo4j 
test on a low-end Windows PC - here are the comparative timings.  Each test 
contains 2 executions of exactly the same function with the idea that the first 
execution will be the worst case scenario as the caches are stone-cold, but 
then the second execution, being exactly the same traversals, should be the 
best case scenario with the cache's red-hot...  In all cases I'm using an 
Embedded neo4j database approach.

Windows (3.2Ghz PC with 1Gb of heap, neo4j database on local disk):-
- execution 1 ~ 41seconds
- execution 21.6 seconds

Solaris (Sun SPARC Enterprise M4000 Server System clock frequency: 1012 MHz 
Memory size: 32768 Megabytes with 3.5Gb heap; NEO4J database on nfs mounted 
filesystem...)
- execution 1 ~ 27+ _minutes_
- execution 2 ~ 5.5 seconds

Re-run with neo4j database on a local file system
- execution 1  ~20 seconds
- execution 2   1.56 seconds


Existing C++/ObjectStore implementation on same Solaris box as above:-
- execution 1 ~ 3.2 seconds
- execution 2 ~ 0.1 seconds

These results are disappointing.  It appears that the first execution is 
extremely I/O bound with a local enterprise server disk only improving on a 
low-end Windows PC by a factor of 2, and nfs reducing performance by a factor 
of about x80.  Once warmed, the low-end Windows PC is almost on par withe 
enterprise Solaris performance.  The fastest neo4j execution 2 is 16x slower 
than the existing C++/Objectstore implementation.   

Any suggestions on how it might be possible to narrow the gap between the 
performance of C++/Objectstore implementation and the neo4j one would be most 
welcome, as otherwise this will be the end of the line for this approach  I 
can accept the poor comparison of the first run if once the cache's are hot the 
second run comes somewhere reasonably comparable to what might expect from a 
comparison between a C++ and Java implementation of a CPU intensive application 
operation (say 2-3 times slower in Java...)?

cheers,

Paul
On 30 Jun 2011, at 14:06, Paul Bandler wrote:

> TEST:bandlerp@us2187$ uname -X
> System = SunOS
> Node = us2187
> Release = 5.10
> KernelID = Generic_13-03
> Machine = sun4u
> BusType = 
> Serial = 
> Users = 
> OEM# = 0
> Origin# = 1
> NumCPU = 16
> 
> Sent from my iPhone
> 
> On 30 Jun 2011, at 12:53, Michael Hunger  
> wrote:
> 
>> Paul,
>> 
>> what version of Solaris are you running that on? We don't have Solaris as 
>> part of our build-qa workflow (yet). So I would try to see if there is an 
>> ec2 instance that I could just use for that.
>> 
>> Cheers
>> 
>> Michael
>> 
>> Am 30.06.2011 um 13:26 schrieb Paul Bandler:
>> 
>>> A colleague has speculated that it maybe related to permissions. Im  on a 
>>> shared solaris box with no setuid access - can anyone elaborate on whether 
>>> some specific access rights are required to use memory mapped IO?
>>> 
>>> Sent from my iPhone
>>> 
>>> On 30 Jun 2011, at 11:20, Paul Bandler  wrote:
>>> 
 Further to an earlier posting I also notice this warning from neo4j which 
 sounds relevant..? Note that the application does continue and eventually 
 complete successfully having taken orders of magnitude longer than when 
 run on windows. This is running on solaris  -  any suggestions about what 
 could be causing this behaviour most welcome . Are there  issues with 
 using neo4j on Solaris?
 
 Jun 30, 2011 10:58:59 AM 
 org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool logWarn
 WARNING: 
 [./neo4j-advanced-1.4.M05/data/graph.db/neostore.relationshipstore.db] 
 Unable to memory map
 org.neo4j.kernel.impl.nioneo.store.MappedMemException: Unable to map 
 pos=28593 recordSize=33 totalSize=104841
 at 
 org.neo4j.kernel.impl.nioneo.store.MappedPersistenceWindow.(MappedPersistenceWindow.java:61)
 at 
 org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.allocateNewWindow(PersistenceWindowPool.java:603)
 at 
 org.neo4j.kernel.impl.nioneo.store.PersistenceWindowPool.refreshB

Re: [Neo4j] graph-collections List class

[Niels Hoogeveen]

Hi Mattias,
I just looked through the code in neo4j-util and will try to port relevant 
classes to graph-collections. I have one question, though. Several methods make 
a call to "GraphDatabaseUtil.acquireWriteLock", which performs one operation: 
the removal of the property "___dummy_property_for_locking___". I don't see any 
place where this property is set. Can you explain this usage?
Niels

> Date: Thu, 30 Jun 2011 20:50:42 +0200
> From: matt...@neotechnology.com
> To: user@lists.neo4j.org
> Subject: Re: [Neo4j] graph-collections List class
> 
> I have done a lot of these structures already, so before (re)doing a lot of
> these structures take a look at:
> https://svn.neo4j.org/components/utils/trunk/src/main/java/org/neo4j/util/(the
> old subversion repository, but the utils are quite up2date). Please
> take a look there and feel free to pull good implementations in to
> graph-collections. There are f.ex. queues (with workers picking items from
> them), linked lists, limited-sized lists and more.
> 
> 2011/6/30 Niels Hoogeveen 
> 
> >
> > For neo4j.graph-collections.list.List I suggest the following.
> > The graph collections List class is an implementation of the java.util.List
> > class, parameterized with Node, supporting all methods, except subList(int
> > start, int end).
> > Whenever nodes need to be stored as a list, the List class can be used,
> > guaranteeing the List contract is maintained.
> > package org.neo4j.collections.list;
> >
> > import org.neo4j.collections.list.List;
> > import org.neo4j.graphdb.Node;
> > import org.neo4j.kernel.EmbeddedGraphDatabase;
> > import java.util.ArrayList;
> > import org.neo4j.graphdb.Transaction;
> >
> > public class Demo {
> >
> >public static void main(String[] args){
> >
> >
> >
> >//create/load database
> >EmbeddedGraphDatabase graphDb = new EmbeddedGraphDatabase(
> > "var/db" );
> >
> >Transaction tx = graphDb.beginTx();
> >
> >//create node representing the list
> >Node indexNode = graphDb.createNode();
> >
> >//create list
> >List list = new List( indexNode, graphDb );
> >
> >//create nodes to be added to the list
> >Node node1 = graphDb.createNode();
> >Node node2 = graphDb.createNode();
> >Node node3 = graphDb.createNode();
> >Node node4 = graphDb.createNode();
> >
> >//add nodes to the list one by one
> >list.add(node1);
> >list.add(node2);
> >list.add(node3);
> >list.add(node4);
> >
> >//remove entries by index
> >list.remove(3);
> >list.remove(2);
> >
> >//remove entries by node
> >list.remove(node2);
> >list.remove(node1);
> >
> >//add a collection of nodes to the list
> >ArrayList al1 = new ArrayList();
> >al1.add(node1);
> >al1.add(node2);
> >al1.add(node3);
> >al1.add(node4);
> >
> >list.addAll(al1);
> >
> >//use the list in a for loop
> >for(Node n: list){
> >  System.out.println(n);
> >}
> >
> >//find first entry in the list containing a specific node
> >list.indexOf(node2);
> >
> >//find last entry in the list containing a specific node
> >list.lastIndexOf(node2);
> >
> >//retain a collection of nodes from the list
> >ArrayList al2 = new ArrayList();
> >al2.add(node1);
> >al2.add(node3);
> >al2.add(node4);
> >
> >list.retainAll(al2);
> >
> >//remove a collection of nodes from the list
> >ArrayList al3 = new ArrayList();
> >al2.add(node3);
> >al2.add(node4);
> >
> >list.removeAll(al3);
> >
> >//delete the list
> >list.delete();
> >
> >tx.success();
> >tx.finish();
> >}
> > }
> >
> >
> > > From: peter.neuba...@neotechnology.com
> > > Date: Thu, 30 Jun 2011 17:49:46 +0200
> > > To: user@lists.neo4j.org
> > > Subject: Re: [Neo4j] graph-collections List class
> > >
> > > Guys,
> > > if you can write some illustrative test code for the different
> > > collections, I could undertake to put some documentation skeleton into
> > > place that can be merged into the main manual and look like
> > > http://docs.neo4j.org/chunked/snapshot/. To start with, we could have
> > > it separately though.
> > >
> > > WDYT?
> > >
> > > Cheers,
> > >
> > > /peter neubauer
> > >
> > > GTalk:  neubauer.peter
> > > Skype   peter.neubauer
> > > Phone   +46 704 106975
> > > LinkedIn   http://www.linkedin.c

Re: [Neo4j] graph-collections List class

[Mattias Persson]

Oh, that. Well, doing an operation on a Node/Relationship grabs a lock on
that entity for that transaction. One operation which doesn't really do
anything is to remove a non-existent property. It's just to make sure that a
write lock is taken before entering a block of
read-then-write-based-on-the-read-value. Functionality-wise it's required
for those implementations to be thread-safe.

In other words: it's because there's no Transaction#acquireWriteLock(Node),
which I'd really like to see pretty soon in there.

2011/6/30 Niels Hoogeveen 

>
> Hi Mattias,
> I just looked through the code in neo4j-util and will try to port relevant
> classes to graph-collections. I have one question, though. Several methods
> make a call to "GraphDatabaseUtil.acquireWriteLock", which performs one
> operation: the removal of the property "___dummy_property_for_locking___". I
> don't see any place where this property is set. Can you explain this usage?
> Niels
>
> > Date: Thu, 30 Jun 2011 20:50:42 +0200
> > From: matt...@neotechnology.com
> > To: user@lists.neo4j.org
> > Subject: Re: [Neo4j] graph-collections List class
> >
> > I have done a lot of these structures already, so before (re)doing a lot
> of
> > these structures take a look at:
> >
> https://svn.neo4j.org/components/utils/trunk/src/main/java/org/neo4j/util/(the
> > old subversion repository, but the utils are quite up2date). Please
> > take a look there and feel free to pull good implementations in to
> > graph-collections. There are f.ex. queues (with workers picking items
> from
> > them), linked lists, limited-sized lists and more.
> >
> > 2011/6/30 Niels Hoogeveen 
> >
> > >
> > > For neo4j.graph-collections.list.List I suggest the following.
> > > The graph collections List class is an implementation of the
> java.util.List
> > > class, parameterized with Node, supporting all methods, except
> subList(int
> > > start, int end).
> > > Whenever nodes need to be stored as a list, the List class can be used,
> > > guaranteeing the List contract is maintained.
> > > package org.neo4j.collections.list;
> > >
> > > import org.neo4j.collections.list.List;
> > > import org.neo4j.graphdb.Node;
> > > import org.neo4j.kernel.EmbeddedGraphDatabase;
> > > import java.util.ArrayList;
> > > import org.neo4j.graphdb.Transaction;
> > >
> > > public class Demo {
> > >
> > >public static void main(String[] args){
> > >
> > >
> > >
> > >//create/load database
> > >EmbeddedGraphDatabase graphDb = new
> EmbeddedGraphDatabase(
> > > "var/db" );
> > >
> > >Transaction tx = graphDb.beginTx();
> > >
> > >//create node representing the list
> > >Node indexNode = graphDb.createNode();
> > >
> > >//create list
> > >List list = new List( indexNode, graphDb );
> > >
> > >//create nodes to be added to the list
> > >Node node1 = graphDb.createNode();
> > >Node node2 = graphDb.createNode();
> > >Node node3 = graphDb.createNode();
> > >Node node4 = graphDb.createNode();
> > >
> > >//add nodes to the list one by one
> > >list.add(node1);
> > >list.add(node2);
> > >list.add(node3);
> > >list.add(node4);
> > >
> > >//remove entries by index
> > >list.remove(3);
> > >list.remove(2);
> > >
> > >//remove entries by node
> > >list.remove(node2);
> > >list.remove(node1);
> > >
> > >//add a collection of nodes to the list
> > >ArrayList al1 = new ArrayList();
> > >al1.add(node1);
> > >al1.add(node2);
> > >al1.add(node3);
> > >al1.add(node4);
> > >
> > >list.addAll(al1);
> > >
> > >//use the list in a for loop
> > >for(Node n: list){
> > >  System.out.println(n);
> > >}
> > >
> > >//find first entry in the list containing a specific
> node
> > >list.indexOf(node2);
> > >
> > >//find last entry in the list containing a specific node
> > >list.lastIndexOf(node2);
> > >
> > >//retain a collection of nodes from the list
> > >ArrayList al2 = new ArrayList();
> > >al2.add(node1);
> > >al2.add(node3);
> > >al2.add(node4);
> > >
> > >list.retainAll(al2);
> > >
> > >//remove a collection of nodes from the list
> > >ArrayList al3 = new ArrayList();
> > >al2.add(node3);
> > >al2.add(node4);
> > >
> > >list.removeAll(al3);
> > >
> > >//delete the list
> > >list.delete();
> > >
> > >tx.succes

Re: [Neo4j] User Digest, Vol 51, Issue 164

[Agelos Pikoulas]

Dear Graphistas,

Im glad this topic is getting traction, cause I think this issue is *MAJOR*.


I firmly believe that if neo4j wants to become the Web 2/3 ready and Social
1/2 ready graphdb, it has to be able to accommodate fast Traversals among
hundreds of thousands or millions of "artifact" nodes (eg Books, Movies,
songs) that have perhaps thousands or millions of relationships of "what"
(likes, purchases, categories etc) from  perhaps some among hundred millions
of "person" nodes (If your graph app makes it, that is the point, right ?).
There's no way avoiding having no super/dense nodes.

To the topic,

Aliabbas, I ve sent you the code and some info.

>
Craig Taverne, I think your idea sounds wonderful. Perhaps it could be even
simpler than this : each node should be able to access *only* its outgoing
or incoming relations, as well as each relation-types using a separate
inherent "collection" for each. Perhaps this could be a simple
Hashmap or something. But maybe I am being too naive here, if it were
so trivial the wonderful people @ neotechnology would have done it, right?

Best Regards,

Agelos Pikoulas
--

>
> Message: 8
> Date: Thu, 30 Jun 2011 13:57:20 +0200
> From: Craig Taverner 
> Subject: Re: [Neo4j] traversing densely populated nodes
> To: Neo4j user discussions 
> Message-ID: 
> Content-Type: text/plain; charset=ISO-8859-1
>
> This topics has come up before, and the domain level solutions are usually
> very similar, like Norbert's category/proxy nodes (to group by
> type/direction) and Niels' TimeLineIndex (BTree). I wonder whether we can
> build a generic user-level solution that can also be wrapped to appear as
> an
> internal database solution?
>
> For example, consider Niels's solution of the TimeLine index. In this case
> we group all the nodes based on a consistent hash. Usually the timeline
> would use a timestamp, but really any reasonably variable property can do,
> even the node-id itself. Then we have a BTree between the dense nodes and
> the root node (node with too many relationships). How about this crazy
> idea,
> create an API that mimics the normal node.getRelationship*() API, but
> internally traverses the entire tree? And also for creating the
> relationships? So for most cod we just do the usual
> node.createRelationshipTo(node,type,direction) and node.traverse(...), but
> internally we actually traverse the b-tree.
>
> This would solve the performance bottleneck being observed while keeping
> the
> 'illusion' of directly connected relationships. The solution would be
> implemented mostly in the application space, so will not need any changes
> to
> the core database. I see this as being of the same kind of solution as the
> auto-indexing. We setup some initial configuration that results in certain
> structures being created on demand. With auto-indexing we are talking about
> mostly automatically adding lucene indexes. With this idea we are talking
> about automatically replacing direct relationships with b-trees to resolve
> a
> specific performance issue.
>
> And when the relationship density is very low, if the b-tree is
> auto-balancing, it could just be a direct relationship anyway.
>
> On Wed, Jun 29, 2011 at 6:56 PM, Agelos Pikoulas
> wrote:
>
> > My problem pattern is exactly the same as Niels's :
> >
> > A dense-node has millions of relations of a certain direction & type,
> > and only a few (sparse) relations of a different direction and type.
> > The traversing is usually following only those sparse relationships on
> > those
> > dense-nodes.
> >
> > Now, even when traversing on these sparse relations, neo4j becomes
> > extremely
> > slow
> > on a certainly non linear Order (the big cs O).
> >
> > Some tests I run (email me if u want the code) reveal that even the
> number
> > of those dense-nodes in the database greatly influences the results.
> >
> > I just reported to Michael the runs with the latest M05 snapshot, which
> are
> > not very positive...
> > I have suggested an (auto) indexing of relationship types / direction
> that
> > is used by traversing frameworks,
> > but I ain't no graphdb-engine expert :-(
> >
> > A'
> >
> >
> > Message: 5
> > > Date: Wed, 29 Jun 2011 18:19:10 +0200
> > > From: Niels Hoogeveen 
> > > Subject: Re: [Neo4j] traversing densely populated nodes
> > > To: 
> > > Message-ID: 
> > > Content-Type: text/plain; charset="iso-8859-1"
> > >
> > >
> > > Michael,
> > >
> > >
> > >
> > > The issue I am refering to does not pertain to traversing many
> relations
> > at
> > > once
> > >
> > > but the impact many relationship of one type have on relationships
> > >
> > > of another type on the same node.
> > >
> > >
> > >
> > > Example:
> > >
> > >
> > >
> > > A topic class has 2 million outgoing relationships of type
> "HAS_INSTANCE"
> > > and
> > >
> > > has 3 outgoing relationships of type "SUB_CLASS_OF".
> > >
> > >
> > >
> > > Fetching the 3 relations of type "SUB_CLASS_OF" takes very long,
> > >
> > > I 

Re: [Neo4j] traversing densely populated nodes

[Michael Hunger]

Just putting Angelos answer in this thread where it belongs :)

Dear Graphistas,

Im glad this topic is getting traction, cause I think this issue is *MAJOR*.

I firmly believe that if neo4j wants to become the Web 2/3 ready and Social
1/2 ready graphdb, it has to be able to accommodate fast Traversals among
hundreds of thousands or millions of "artifact" nodes (eg Books, Movies,
songs) that have perhaps thousands or millions of relationships of "what"
(likes, purchases, categories etc) from  perhaps some among hundred millions
of "person" nodes (If your graph app makes it, that is the point, right ?).
There's no way avoiding having no super/dense nodes.

To the topic,

Aliabbas, I ve sent you the code and some info.

> 
Craig Taverne, I think your idea sounds wonderful. Perhaps it could be even
simpler than this : each node should be able to access *only* its outgoing
or incoming relations, as well as each relation-types using a separate
inherent "collection" for each. Perhaps this could be a simple
Hashmap or something. But maybe I am being too naive here, if it were
so trivial the wonderful people @ neotechnology would have done it, right?

Best Regards,

Agelos Pikoulas
--

Am 30.06.2011 um 14:47 schrieb Niels Hoogeveen:

> 
> Michael,
> Is it possible/feasible to change the store-format so relationships are 
> stored partitioned per relationshiptype per direction? What would be the 
> impact of such change?
> Niels
> 
>> From: michael.hun...@neotechnology.com
>> Date: Thu, 30 Jun 2011 14:10:38 +0200
>> To: user@lists.neo4j.org
>> Subject: Re: [Neo4j] traversing densely populated nodes
>> 
>> The issue being that relationships on disk are not ordered, so that, even 
>> when just accessing the few relationships of the one
>> type you still have to scan all rels.
>> 
>> For supporting different approaches you either have to change the 
>> store-format to handle the storage and loading of relationships of 
>> supernodes differently.
>> 
>> Or: 
>> - if you'd rather want to avoid supernodes you can have an Expander in your 
>> traversers that delays the traversals of supernodes (perhaps
>> the other nodes give you already enough results for the domain to consume). 
>> (You could also limit/timebox that).
>> - You can also have Expanders that use an B-Tree or Relationship index 
>> internally (for the "few"-rels). With autoindexing you could actually put a 
>> property on those few "rels" that makes them automatically indexed. And then 
>> in the Expander use
>> relIndex.get(startNode,null,null) to get all of them for expansion.
>> 
>> It can be useful to get something like this implemented once and then 
>> offered to all users either as part of the product or perhaps in a component 
>> like neo4j-collections ? (these are just ideas)
>> 
>> Cheers
>> 
>> Michael
>> 
>> Am 30.06.2011 um 13:57 schrieb Craig Taverner:
>> 
>>> This topics has come up before, and the domain level solutions are usually
>>> very similar, like Norbert's category/proxy nodes (to group by
>>> type/direction) and Niels' TimeLineIndex (BTree). I wonder whether we can
>>> build a generic user-level solution that can also be wrapped to appear as an
>>> internal database solution?
>>> 
>>> For example, consider Niels's solution of the TimeLine index. In this case
>>> we group all the nodes based on a consistent hash. Usually the timeline
>>> would use a timestamp, but really any reasonably variable property can do,
>>> even the node-id itself. Then we have a BTree between the dense nodes and
>>> the root node (node with too many relationships). How about this crazy idea,
>>> create an API that mimics the normal node.getRelationship*() API, but
>>> internally traverses the entire tree? And also for creating the
>>> relationships? So for most cod we just do the usual
>>> node.createRelationshipTo(node,type,direction) and node.traverse(...), but
>>> internally we actually traverse the b-tree.
>>> 
>>> This would solve the performance bottleneck being observed while keeping the
>>> 'illusion' of directly connected relationships. The solution would be
>>> implemented mostly in the application space, so will not need any changes to
>>> the core database. I see this as being of the same kind of solution as the
>>> auto-indexing. We setup some initial configuration that results in certain
>>> structures being created on demand. With auto-indexing we are talking about
>>> mostly automatically adding lucene indexes. With this idea we are talking
>>> about automatically replacing direct relationships with b-trees to resolve a
>>> specific performance issue.
>>> 
>>> And when the relationship density is very low, if the b-tree is
>>> auto-balancing, it could just be a direct relationship anyway.
>>> 
>>> On Wed, Jun 29, 2011 at 6:56 PM, Agelos Pikoulas
>>> wrote:
>>> 
 My problem pattern is exactly the same as Niels's :
 
 A dense-node has millions of relations of a certain direction & type,
 and only a fe

Re: [Neo4j] traversing densely populated nodes

[Michael Hunger]

If you want to go with the "indexed-relationship" solution for the supernode 
with massive relationships and just a _few_ relationships that I want to 
traverse, I tried the following solution that performs better for the 
cold-cache but a bit worse for the warm-cache (2nd run) test.

Please make sure that you only use that approach for supernodes and for cold 
caches otherwise it will impact the store-performance (as lucene is 
definitively slower than graph traversal on cached nodes). And please make
also sure that you only index the "few" important relationships otherwise the 
index lookup will be slower.

Cheers

Michael

private static class IndexedRelationshipExpander implements 
RelationshipExpander {
private final Direction direction;
private RelType relType;
private static RelationshipIndex index = 
gr.index().forRelationships("myindex");

public IndexedRelationshipExpander(Direction direction, RelType 
relType) {
this.direction = direction;
this.relType = relType;
}

@Override
public Iterable expand(Node node) {
if (direction==Direction.OUTGOING)
return index.get("type", relType.name(), node, null);
else
return index.get("type", relType.name(), null, node);
}

public static Relationship index(Relationship relationship) {
index.add(relationship,"type",relationship.getType().name());
return relationship;
}

@Override
public RelationshipExpander reversed() {
   return new IndexedRelationshipExpander(direction.reverse(), relType);
}
}


static void traverseAndPrintIndexedTraversal(Node mainPartNode) {
TraversalDescription td = Traversal.description().expand(new 
IndexedRelationshipExpander(Direction.OUTGOING, RelType.HAS_PART)).depthFirst();
long startMillis = System.currentTimeMillis();
int totalCount = 0;
for (Node part : td.traverse(mainPartNode).nodes()) {
totalCount++;
System.out.println(part.getProperty("name"));
}
long delta = (System.currentTimeMillis() - startMillis);
System.out.printf("traversing %d relationships took %d ms. %.2f 
rels/ms%n", totalCount, delta, (float)(delta>0 ? totalCount / delta : 
totalCount));
}

private static Node createPart(Node part, int nr) {
Node newPart = gr.createNode();
final Relationship rel = part.createRelationshipTo(newPart, 
RelType.HAS_PART);
newPart.setProperty("name", "Part" + nr);
IndexedRelationshipExpander.index(rel);
return newPart;
}


static void createNodeData() {
Transaction tx = gr.beginTx();
try {
mainPart = gr.createNode();
mainPart.setProperty("name", "MainPart");
gr.getReferenceNode().createRelationshipTo(mainPart, 
RelType.HAS_PART);
List nodes = new ArrayList(10);
nodes.add(createPart(mainPart, 1));
Node part2 = createPart(mainPart,2);
nodes.add(part2);

nodes.addAll(asList(createPart(part2,3),createPart(part2,4),createPart(part2,5),createPart(part2,6)));

int count = 0;
for (int i = 1; i <= containerCount; i++) {
for (Node part : nodes) {
Node container = gr.createNode();
container.setProperty("name", "Container-" + i);
relateToContainer(container, part);

// inform us every 1000 nodes
count++;
if (count >= 1) {
tx.success();
tx.finish();
tx = gr.beginTx();
System.out.println("Written " + i + " nodes...");
count = 0;
}
}}

tx.success();
System.out.println("Nodes created successfully!");
} finally {
tx.finish();
}
}




Am 30.06.2011 um 14:47 schrieb Niels Hoogeveen:

> 
> Michael,
> Is it possible/feasible to change the store-format so relationships are 
> stored partitioned per relationshiptype per direction? What would be the 
> impact of such change?
> Niels
> 
>> From: michael.hun...@neotechnology.com
>> Date: Thu, 30 Jun 2011 14:10:38 +0200
>> To: user@lists.neo4j.org
>> Subject: Re: [Neo4j] traversing densely populated nodes
>> 
>> The issue being that relationships on disk are not ordered, so that, even 
>> when just accessing the few relationships of the one
>> type you still have to scan all rels.
>> 
>> For supporting different approaches you either have to change the 
>> store-format to handle the storage and loading of relationships of 
>> supernodes differently.
>> 
>> Or: 
>> - if you'd rather want to avoid supernodes you can have an Expander in your 
>> traversers that delays the traversals of supernodes (perhaps
>> the other nodes give y

Re: [Neo4j] Can I model a Graph data structure without persisting it in DB using Neo4j ?

[Michael Hunger]

Neo4j is not implementing storing and traversing memory only structures. So it 
would be really advisable to do that in your domain layer only.

I'll look into the elements returning null. I thought I already solved that 
some time ago. Which version of SDG are you using?

Cheers

Michael

Am 30.06.2011 um 15:00 schrieb V:

> Any suggestions on this please ? 
> It is really acting as a blocker for me. I have tried multiple alternatives, 
> but I come up with one issue or the other. 
> 
> 
> On Wed, Jun 22, 2011 at 8:51 PM, V  wrote:
> I am confused, elements is defined as 
> 
> @RelatedTo(type = "ELEMENT", elementClass = Element.class, direction = 
> OUTGOING)
> private Set elements;
> 
> if I call getElements() without persisting the node first, will it not return 
> null ? 
> 
> It is throwing null pointer exception. 
> 
> When I persist the node before calling getElements, then it returns empty 
> set. This is in line with my original email.
> 
> Do you want me to do something else? Please let me know 
> 
> 
> 
> 
> On Wed, Jun 22, 2011 at 8:39 PM, Michael Hunger 
>  wrote:
> getElements() returns a set of your elements. If you do an add(element) on 
> those then you build up an in-memory structure.
> 
> So if you use just the getters and setters of your instances you can navigate 
> this structure, but you can't use any of the graph operations supplied by 
> neo4j (remember it is still just in memory)
> 
> Cheers
> 
> Michael
> 
> Am 22.06.2011 um 17:06 schrieb V:
> 
>> Michael - 
>> 
>> In my quest for a cleaner solution , I am coming back to your original reply
>> 
>> What do you mean by "Right now this works only with the getElements().add 
>> method."
>> Please elaborate . 
>> 
>> Thanks for your help .
>> 
>> 
>> On Sun, Jun 19, 2011 at 2:34 AM, Michael Hunger 
>>  wrote:
>> Right now this works only with the getElements().add method.
>> 
>> But then you don't get all the graph methods (traversals and such).
>> 
>> We discussed some time ago writing a virtual graph layer on top of the real 
>> nodes and relationships for SDG.
>> (That should be then used to have a simpler handling of attached/detached 
>> nodes and keeping back-references for cluster-persistance).
>> 
>> 
>> But that hasn't happend yet and I'm not convinced it will make it in the 
>> 1.1. timeframe.
>> 
>> Cheers
>> 
>> Michael
>> 
>> P.S. You can still persist your graph and remove the nodes later (either 
>> index them on a "transient" index or keep their node-id's somewhere).
>> 
>> 
>> Am 18.06.2011 um 03:40 schrieb V:
>> 
>>> Any suggestions on this please ? 
>>> 
>>> 
>>> On Fri, Jun 17, 2011 at 11:03 PM, V  wrote:
>>> I created a graph model with 2 classes Node and Element as follows: 
>>> 
>>> public class Node {
>>> 
>>>  @RelatedTo(type = "ELEMENT", elementClass = Element.class, direction = 
>>> OUTGOING)
>>> private Set Element;
>>> 
>>> 
>>> public void addElement(Element e) {
>>> relateTo(e, Relationships.ELEMENT.toString());
>>> }
>>> 
>>> }
>>> 
>>> public class Element{
>>> 
>>>public String name;
>>> 
>>> }
>>> I want to create an in memory graph structure without persisting the nodes 
>>> as follows : 
>>> 
>>> Node n = new Node();
>>> n.addElement(new Element());
>>> 
>>> However it throws an exception as the Node n has not been persisted so the 
>>> call to relateTo(..) fails. 
>>> 
>>> If instead I do 
>>> n.persist()
>>> and then call addElement(..) it works fine as the aspect kicks in. 
>>> 
>>> Any workaround for this ?  That is, is there a way I can still use the 
>>> above style without persisting the Node object ? 
>>> 
>>> 
>>> My application needs this as first I create a structure and persist it, and 
>>> then I create another structure to pass around some values to the persisted 
>>> structure when doing some computations. 
>>> 
>>> -Karan 
>>> 
>>> 
>>> 
>>> 
>> 
>> 
> 
> 
> 

___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Problems building neo4j-rdf module

[Marko Rodriguez]

Hi,

You may want to have references to different graph representations. For example:

GraphDatabaseService neo4j = new EmbeddedGraphDatabase("/tmp/neo4j");
Graph graph = new Neo4jGraph(neo4j);
Sail sail = new GraphSail(graph);

With 'neo4j' you have a reference to the GraphDatabaseService and can use 
Neo4j's ecosystem of tools and technologies.
With 'graph' you have a reference to the Blueprints Graph and can use 
TinkerPop's ecosystem of tools and technologies.
With 'sail' you have a reference to a Sail and can use OpenRDF's ecosystem of 
tools and technologies.

Hope that helps,
Marko.

http://markorodriguez.com

On Jun 30, 2011, at 12:23 AM, Thomas Fritz wrote:

> Hello
> 
> I am also interesed in testing Neo4J as a RDF Triple Store.
> Thanks Marco for pointing to this.
> 
> If we use the Ouplementation of Tinkerpop for Sail and Neo4J, are there any
> features Neo4J offers or performance optimizations or other things we can
> not use in such a setup?
> I think i have read somewhere, also i could not find it again that indexing
> is not supported?
> 
> Is it possible to use also cypher besides sparql in such a setup?
> 
> 
> 
> Kind regards
> 
> 
> 
> ---
> 
> *Thomas FRITZ*
> *web* http://fritzthomas.com
> *twitter* http://twitter.com/thomasf
> 
> 
> 
> 2011/6/29 Marko Rodriguez 
> 
>> Hi,
>> 
>> This is the link:
>>   https://github.com/tinkerpop/blueprints/wiki/Sail-Ouplementation
>> 
>> The other one was for making an RDF triple/quad store look like a property
>> graph database.
>> 
>> If you are familiar with OpenRDF's Sail, then the take home point is:
>> 
>> Sail sail = new GraphSail(new Neo4jGraph('/tmp/mygraph'));
>> 
>> At which point, sail provides the RDF layer for inferencing, SPARQL, RQL,
>> loading/saving RDF, etc.
>> 
>> See ya,
>> Marko.
>> 
>> http://markorodriguez.com
>> 
>> On Jun 29, 2011, at 1:03 PM, Nikola Milikic wrote:
>> 
>>> Hi,
>>> 
>>> Thanks for the reply!
>>> 
>>> I'm not familiar with Thinkerpop, but I'll give it a try. My intention
>> was
>>> just to check out the performance neo4j would give when working with RDF
>>> data, but to be honest, on a first glimpse, this looks over too
>> complicated
>>> for just reading some RDF data into a store.
>>> 
>>> Best,
>>> Nikola
>>> 
>>> 
>>> On Tue, Jun 28, 2011 at 3:13 PM, Mattias Persson
>>> wrote:
>>> 
 Another option could be to run it through the Tinkerpop stack and get
 access
 to a Sail with neo4j underneath that way. I think it's more up to date.
 
 https://github.com/tinkerpop/blueprints/wiki/Sail-Implementation
 
 2011/6/28 Nikola Milikic 
 
> Hi all,
> 
> I wanted to try out neo4j-rdf library and found somewhere that there
>> are
 no
> official releases, but the library should be built from the source (
> https://svn.neo4j.org/components/rdf/trunk). But when I checked out
>> the
> project, it reported dependency problems as some of the dependencies
>> are
> not
> available. After I changed them to the ones available from the Maven
> Central, then I got compilation errors.
> 
> Did anyone face with the same problem and knows how to resolve this?
>> Or,
> better would be if somebody could provide me with builded jar.
> 
> Thanks a lot for the help!
> 
> Best,
> Nikola Milikic
> 
> 
> Nikola Milikic, BSc
> Teaching Assistant
> University of Belgrade, Serbia
> 
> email: nikola.mili...@gmail.com
> URL:   nikola.milikic.info
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
> 
 
 
 
 --
 Mattias Persson, [matt...@neotechnology.com]
 Hacker, Neo Technology
 www.neotechnology.com
 ___
 Neo4j mailing list
 User@lists.neo4j.org
 https://lists.neo4j.org/mailman/listinfo/user
 
>>> ___
>>> Neo4j mailing list
>>> User@lists.neo4j.org
>>> https://lists.neo4j.org/mailman/listinfo/user
>> 
>> ___
>> Neo4j mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>> 
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user

___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] traversing densely populated nodes

[Niels Hoogeveen]

In theory the approach I described earlier could work, though there are some 
pitfalls to the current implementation that need ironing out before this can 
become a recommended approach.
The choice of Timeline instead of Btree may actually be the wrong choice after 
all. I chose Timeline because of my familiarity with this particular class, but 
its implementation may actually not be all that suitable for this particular 
use case. This has to do with the fact that Timeline is not just  a tree, but a 
list where entries with an interval of max. 1000 are stored in a Btree index. 
This works reasonably well for a Timeline, but makes the approach less ideal 
for storing dense relationships.
The problem with the Timeline implementation is the ability to lookup the tree 
root from a particular leave. In an ordinary Btree is would simply be a 
traversal from the leave through the layers of block nodes to the tree root. In 
Timeline the traversal will be different. It first has to move through the 
Timeline list until it finds an entry that is stored in the Btree (which worst 
case takes 1000 hops), and then it has to traverse the Btree up to the tree 
root. To avoid this complicated traversal I ended up doing a lookup through 
Lucene of the timeline URI (which is stored in all timeline list entries). In 
fact I might as well have added the URI of the dense node as a property and do 
the lookup through Lucene without the Timeline, it just happens that I like the 
sort order of Timeline, making it a useful approach anyway.
I will experiment using Btree directly (without Timeline) and see if that leads 
to a simpler and faster traversal from leave to root node. 
There is one more issue before this can become production ready. Btree as it is 
implemented now is not thread safe (per the implementations Javadocs), so it 
need some love and attention to make it work properly.
Niels

> Date: Thu, 30 Jun 2011 13:57:20 +0200
> From: cr...@amanzi.com
> To: user@lists.neo4j.org
> Subject: Re: [Neo4j] traversing densely populated nodes
> 
> This topics has come up before, and the domain level solutions are usually
> very similar, like Norbert's category/proxy nodes (to group by
> type/direction) and Niels' TimeLineIndex (BTree). I wonder whether we can
> build a generic user-level solution that can also be wrapped to appear as an
> internal database solution?
> 
> For example, consider Niels's solution of the TimeLine index. In this case
> we group all the nodes based on a consistent hash. Usually the timeline
> would use a timestamp, but really any reasonably variable property can do,
> even the node-id itself. Then we have a BTree between the dense nodes and
> the root node (node with too many relationships). How about this crazy idea,
> create an API that mimics the normal node.getRelationship*() API, but
> internally traverses the entire tree? And also for creating the
> relationships? So for most cod we just do the usual
> node.createRelationshipTo(node,type,direction) and node.traverse(...), but
> internally we actually traverse the b-tree.
> 
> This would solve the performance bottleneck being observed while keeping the
> 'illusion' of directly connected relationships. The solution would be
> implemented mostly in the application space, so will not need any changes to
> the core database. I see this as being of the same kind of solution as the
> auto-indexing. We setup some initial configuration that results in certain
> structures being created on demand. With auto-indexing we are talking about
> mostly automatically adding lucene indexes. With this idea we are talking
> about automatically replacing direct relationships with b-trees to resolve a
> specific performance issue.
> 
> And when the relationship density is very low, if the b-tree is
> auto-balancing, it could just be a direct relationship anyway.
> 
> On Wed, Jun 29, 2011 at 6:56 PM, Agelos Pikoulas
> wrote:
> 
> > My problem pattern is exactly the same as Niels's :
> >
> > A dense-node has millions of relations of a certain direction & type,
> > and only a few (sparse) relations of a different direction and type.
> > The traversing is usually following only those sparse relationships on
> > those
> > dense-nodes.
> >
> > Now, even when traversing on these sparse relations, neo4j becomes
> > extremely
> > slow
> > on a certainly non linear Order (the big cs O).
> >
> > Some tests I run (email me if u want the code) reveal that even the number
> > of those dense-nodes in the database greatly influences the results.
> >
> > I just reported to Michael the runs with the latest M05 snapshot, which are
> > not very positive...
> > I have suggested an (auto) indexing of relationship types / direction that
> > is used by traversing frameworks,
> > but I ain't no graphdb-engine expert :-(
> >
> > A'
> >
> >
> > Message: 5
> > > Date: Wed, 29 Jun 2011 18:19:10 +0200
> > > From: Niels Hoogeveen 
> > > Subject: Re: [Neo4j] traversing densely populated nodes
> > > To:

Re: [Neo4j] traversing densely populated nodes

[Craig Taverner]

In the amanzi-index I link all indexed nodes into the index tree, so
traversals are straight up the tree. Of course this also means that there
are at least as many relationships as indexed nodes.

I was reviewing Michaels code for the relationship expander, and think that
is a great idea, tranparently using an index instead of the normal
relationships API, and can imagine using the relationship expander to
instead traverse the BTree to the final relationship to the leaf nodes.

So if we imagine a BTree with perhaps 10 or 20 hops from the root to the
leaf node, the relationship expander Michael described would complete all
hops and return only the last relationship, giving the illusion of direct
connections from root to leaf. This would certainly perform well, especially
for cases where there are factors limiting the number of relationships we
want returned. I think the request for type and direction is the first
obvious case, but we could be even more explicit than that, if we pass
constraints based on the BTree's consistent hash.

On Thu, Jun 30, 2011 at 11:36 PM, Niels Hoogeveen  wrote:

>
> In theory the approach I described earlier could work, though there are
> some pitfalls to the current implementation that need ironing out before
> this can become a recommended approach.
> The choice of Timeline instead of Btree may actually be the wrong choice
> after all. I chose Timeline because of my familiarity with this particular
> class, but its implementation may actually not be all that suitable for this
> particular use case. This has to do with the fact that Timeline is not just
>  a tree, but a list where entries with an interval of max. 1000 are stored
> in a Btree index. This works reasonably well for a Timeline, but makes the
> approach less ideal for storing dense relationships.
> The problem with the Timeline implementation is the ability to lookup the
> tree root from a particular leave. In an ordinary Btree is would simply be a
> traversal from the leave through the layers of block nodes to the tree root.
> In Timeline the traversal will be different. It first has to move through
> the Timeline list until it finds an entry that is stored in the Btree (which
> worst case takes 1000 hops), and then it has to traverse the Btree up to the
> tree root. To avoid this complicated traversal I ended up doing a lookup
> through Lucene of the timeline URI (which is stored in all timeline list
> entries). In fact I might as well have added the URI of the dense node as a
> property and do the lookup through Lucene without the Timeline, it just
> happens that I like the sort order of Timeline, making it a useful approach
> anyway.
> I will experiment using Btree directly (without Timeline) and see if that
> leads to a simpler and faster traversal from leave to root node.
> There is one more issue before this can become production ready. Btree as
> it is implemented now is not thread safe (per the implementations Javadocs),
> so it need some love and attention to make it work properly.
> Niels
>
> > Date: Thu, 30 Jun 2011 13:57:20 +0200
> > From: cr...@amanzi.com
> > To: user@lists.neo4j.org
> > Subject: Re: [Neo4j] traversing densely populated nodes
> >
> > This topics has come up before, and the domain level solutions are
> usually
> > very similar, like Norbert's category/proxy nodes (to group by
> > type/direction) and Niels' TimeLineIndex (BTree). I wonder whether we can
> > build a generic user-level solution that can also be wrapped to appear as
> an
> > internal database solution?
> >
> > For example, consider Niels's solution of the TimeLine index. In this
> case
> > we group all the nodes based on a consistent hash. Usually the timeline
> > would use a timestamp, but really any reasonably variable property can
> do,
> > even the node-id itself. Then we have a BTree between the dense nodes and
> > the root node (node with too many relationships). How about this crazy
> idea,
> > create an API that mimics the normal node.getRelationship*() API, but
> > internally traverses the entire tree? And also for creating the
> > relationships? So for most cod we just do the usual
> > node.createRelationshipTo(node,type,direction) and node.traverse(...),
> but
> > internally we actually traverse the b-tree.
> >
> > This would solve the performance bottleneck being observed while keeping
> the
> > 'illusion' of directly connected relationships. The solution would be
> > implemented mostly in the application space, so will not need any changes
> to
> > the core database. I see this as being of the same kind of solution as
> the
> > auto-indexing. We setup some initial configuration that results in
> certain
> > structures being created on demand. With auto-indexing we are talking
> about
> > mostly automatically adding lucene indexes. With this idea we are talking
> > about automatically replacing direct relationships with b-trees to
> resolve a
> > specific performance issue.
> >
> > And when the relationship dens

Re: [Neo4j] Does anyone use Solaris for neo4j app? [was Unable to memory map [was java.io.IOException: Resource temporarily unavailable in MappedPersistenceWindow.java?

[Paul Bandler]

Oops - it seems it might not be neo4j that's chewing up the CPUby making my 
log4j calls disabled with 'if(_log.isDebugEnabled)' pre-conditions to avoid the 
expense of constructing the strings that might or might get logged, I was able 
to improve the times for the 'hot-cache' tests described below by about x10, 
bringing it within a factor of 1.5-2 of the C++ performance which is within 
acceptable limits.

I'm still interested to know whether others are using neo4j on Solaris, whether 
it's possible to use memory mapped i/o for buffers on Solaris, and indeed 
whether that would further help performance, or does it just reduce the heap 
demand?

On 30 Jun 2011, at 21:02, Paul Bandler wrote:

> Further to the problems reported below from running a test application on a 
> server class Solaris box (M4000)  I re-ran the test but with the setting 
> 'use_memory_mapped_buffers' set to false.  This had the +ve effect of 
> avoiding the warnings and nio exceptions mentioned before so clearly there's 
> some additional magic required to run under Solaris using memory mapped 
> buffers.  I have enquired as to whether one needs setuid rights to use memory 
> mapped files on Solaris and am told not btw...
> 
> The performance was woeful however - like walking thru' molasses in (a 
> Swedish) mid-winter!  The test comprises a relatively complex navigation of a 
> domain model to derive some information.  This is a re-implementation of a 
> function  that  already exists in a production C++ application that uses an 
> object database (based on a product called ObjectStore) -  so I have 
> something to compare the results against.  I also have the results of running 
> the same neo4j test on a low-end Windows PC - here are the comparative 
> timings.  Each test contains 2 executions of exactly the same function with 
> the idea that the first execution will be the worst case scenario as the 
> caches are stone-cold, but then the second execution, being exactly the same 
> traversals, should be the best case scenario with the cache's red-hot...  In 
> all cases I'm using an Embedded neo4j database approach.
> 
> Windows (3.2Ghz PC with 1Gb of heap, neo4j database on local disk):-
> - execution 1 ~ 41seconds
> - execution 21.6 seconds
> 
> Solaris (Sun SPARC Enterprise M4000 Server System clock frequency: 1012 MHz 
> Memory size: 32768 Megabytes with 3.5Gb heap; NEO4J database on nfs mounted 
> filesystem...)
> - execution 1 ~ 27+ _minutes_
> - execution 2 ~ 5.5 seconds
> 
> Re-run with neo4j database on a local file system
> - execution 1  ~20 seconds
> - execution 2   1.56 seconds
> 
> 
> Existing C++/ObjectStore implementation on same Solaris box as above:-
> - execution 1 ~ 3.2 seconds
> - execution 2 ~ 0.1 seconds
> 
> These results are disappointing.  It appears that the first execution is 
> extremely I/O bound with a local enterprise server disk only improving on a 
> low-end Windows PC by a factor of 2, and nfs reducing performance by a factor 
> of about x80.  Once warmed, the low-end Windows PC is almost on par withe 
> enterprise Solaris performance.  The fastest neo4j execution 2 is 16x slower 
> than the existing C++/Objectstore implementation.   
> 
> Any suggestions on how it might be possible to narrow the gap between the 
> performance of C++/Objectstore implementation and the neo4j one would be most 
> welcome, as otherwise this will be the end of the line for this approach  
> I can accept the poor comparison of the first run if once the cache's are hot 
> the second run comes somewhere reasonably comparable to what might expect 
> from a comparison between a C++ and Java implementation of a CPU intensive 
> application operation (say 2-3 times slower in Java...)?
> 
> cheers,
> 
> Paul
> On 30 Jun 2011, at 14:06, Paul Bandler wrote:
> 
>> TEST:bandlerp@us2187$ uname -X
>> System = SunOS
>> Node = us2187
>> Release = 5.10
>> KernelID = Generic_13-03
>> Machine = sun4u
>> BusType = 
>> Serial = 
>> Users = 
>> OEM# = 0
>> Origin# = 1
>> NumCPU = 16
>> 
>> Sent from my iPhone
>> 
>> On 30 Jun 2011, at 12:53, Michael Hunger  
>> wrote:
>> 
>>> Paul,
>>> 
>>> what version of Solaris are you running that on? We don't have Solaris as 
>>> part of our build-qa workflow (yet). So I would try to see if there is an 
>>> ec2 instance that I could just use for that.
>>> 
>>> Cheers
>>> 
>>> Michael
>>> 
>>> Am 30.06.2011 um 13:26 schrieb Paul Bandler:
>>> 
 A colleague has speculated that it maybe related to permissions. Im  on a 
 shared solaris box with no setuid access - can anyone elaborate on whether 
 some specific access rights are required to use memory mapped IO?
 
 Sent from my iPhone
 
 On 30 Jun 2011, at 11:20, Paul Bandler  wrote:
 
> Further to an earlier posting I also notice this warning from neo4j which 
> sounds relevant..? Note that the application does continue and eventually 
> complete successfully having taken orders of magnitude 

Re: [Neo4j] Can I model a Graph data structure without persisting it in DB using Neo4j ?

[V]

I am using the latest version 1.1.0.M1
Please have a look and let me know

Karan


On Fri, Jul 1, 2011 at 2:29 AM, Michael Hunger <
michael.hun...@neotechnology.com> wrote:

> Neo4j is not implementing storing and traversing memory only structures. So
> it would be really advisable to do that in your domain layer only.
>
> I'll look into the elements returning null. I thought I already solved that
> some time ago. Which version of SDG are you using?
>
> Cheers
>
> Michael
>
> Am 30.06.2011 um 15:00 schrieb V:
>
> Any suggestions on this please ?
> It is really acting as a blocker for me. I have tried multiple
> alternatives, but I come up with one issue or the other.
>
>
> On Wed, Jun 22, 2011 at 8:51 PM, V  wrote:
>
>> I am confused, elements is defined as
>>
>> @RelatedTo(type = "ELEMENT", elementClass = Element.class, direction =
>> OUTGOING)
>> private Set elements;
>>
>> if I call getElements() without persisting the node first, will it not
>> return null ?
>>
>> It is throwing null pointer exception.
>>
>> When I persist the node before calling getElements, then it returns empty
>> set. This is in line with my original email.
>>
>> Do you want me to do something else? Please let me know
>>
>>
>>
>>
>> On Wed, Jun 22, 2011 at 8:39 PM, Michael Hunger <
>> michael.hun...@neotechnology.com> wrote:
>>
>>> getElements() returns a set of your elements. If you do an add(element)
>>> on those then you build up an in-memory structure.
>>>
>>> So if you use just the getters and setters of your instances you can
>>> navigate this structure, but you can't use any of the graph operations
>>> supplied by neo4j (remember it is still just in memory)
>>>
>>> Cheers
>>>
>>> Michael
>>>
>>> Am 22.06.2011 um 17:06 schrieb V:
>>>
>>> Michael -
>>>
>>> In my quest for a cleaner solution , I am coming back to your original
>>> reply
>>>
>>> What do you mean by *"Right now this works only with the
>>> getElements().add method."*
>>> Please elaborate .
>>>
>>> Thanks for your help .
>>>
>>>
>>> On Sun, Jun 19, 2011 at 2:34 AM, Michael Hunger <
>>> michael.hun...@neotechnology.com> wrote:
>>>
 Right now this works only with the getElements().add method.

 But then you don't get all the graph methods (traversals and such).

 We discussed some time ago writing a virtual graph layer on top of the
 real nodes and relationships for SDG.
 (That should be then used to have a simpler handling of
 attached/detached nodes and keeping back-references for
 cluster-persistance).


 But that hasn't happend yet and I'm not convinced it will make it in the
 1.1. timeframe.

 Cheers

 Michael

 P.S. You can still persist your graph and remove the nodes later (either
 index them on a "transient" index or keep their node-id's somewhere).


 Am 18.06.2011 um 03:40 schrieb V:

 Any suggestions on this please ?


 On Fri, Jun 17, 2011 at 11:03 PM, V  wrote:

> I created a graph model with 2 classes Node and Element as follows:
>
> public class Node {
>
>  @RelatedTo(type = "ELEMENT", elementClass = Element.class, direction =
> OUTGOING)
> private Set Element;
>
>
> public void addElement(Element e) {
> relateTo(e, Relationships.ELEMENT.toString());
> }
>
> }
>
> public class Element{
>
>public String name;
>
> }
> I want to create an in memory graph structure without persisting the
> nodes as follows :
>
> Node n = new Node();
> n.addElement(new Element());
>
> *However it throws an exception as the Node n has not been persisted
> so the call to relateTo(..) fails. *
>
> If instead I do
> n.persist()
> and then call addElement(..) it works fine as the aspect kicks in.
>
> Any workaround for this ?  That is, is there a way I can still use the
> above style without persisting the Node object ?
>
>
> My application needs this as first I create a structure and persist it,
> and then I create another structure to pass around some values to the
> persisted structure when doing some computations.
>
> -Karan
>
>
>
>


>>>
>>>
>>
>
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] traversing densely populated nodes

[Niels Hoogeveen]

The relationship expander approach could certainly work for me. I already have 
a meta layer containing information about relationship types, so it would be a 
minor upgrade to store the type of relationship expander per relationship type 
and to access relationships through the expander instead of the direct API. 
What we need now is a solid balanced tree to partition those dense 
relationships. The Btree implementation in graph-collections is the probably 
our best option right now, though as I said earlier, it needs to attention to 
make it production ready. We need to make sure the implementation is thread 
safe and there are some other issues we need to look into. For example, right 
now the order of the Btree is fixed at 9. We could lift this value into the 
constructor of the Btree, so we can create Btrees of various different orders 
and then do performance tests, looking for the optimal order. The larger the 
order, the fewer the hops, but too large an order may in fact create dense 
nodes, something we try to avoid in the first place. 
There will certainly be more issues we need to look into, but the combination 
of the use of relationship expanders and a balanced tree index looks very 
promising to me.
As to the value for the Btree hash, how about trying the inverse of the node 
id. Using the node id itself or a time stamp is actually disadvantageous, 
because those values are in practice monotone (except for the reuse of node 
ids) leading to lots of balancing acts. The inverse of the node id is equally 
unique, but has a nicer distribution, making inserts into the Btree probably a 
bit faster.
Niels


> Date: Thu, 30 Jun 2011 23:50:24 +0200
> From: cr...@amanzi.com
> To: user@lists.neo4j.org
> Subject: Re: [Neo4j] traversing densely populated nodes
> 
> In the amanzi-index I link all indexed nodes into the index tree, so
> traversals are straight up the tree. Of course this also means that there
> are at least as many relationships as indexed nodes.
> 
> I was reviewing Michaels code for the relationship expander, and think that
> is a great idea, tranparently using an index instead of the normal
> relationships API, and can imagine using the relationship expander to
> instead traverse the BTree to the final relationship to the leaf nodes.
> 
> So if we imagine a BTree with perhaps 10 or 20 hops from the root to the
> leaf node, the relationship expander Michael described would complete all
> hops and return only the last relationship, giving the illusion of direct
> connections from root to leaf. This would certainly perform well, especially
> for cases where there are factors limiting the number of relationships we
> want returned. I think the request for type and direction is the first
> obvious case, but we could be even more explicit than that, if we pass
> constraints based on the BTree's consistent hash.
> 
> On Thu, Jun 30, 2011 at 11:36 PM, Niels Hoogeveen  > wrote:
> 
> >
> > In theory the approach I described earlier could work, though there are
> > some pitfalls to the current implementation that need ironing out before
> > this can become a recommended approach.
> > The choice of Timeline instead of Btree may actually be the wrong choice
> > after all. I chose Timeline because of my familiarity with this particular
> > class, but its implementation may actually not be all that suitable for this
> > particular use case. This has to do with the fact that Timeline is not just
> >  a tree, but a list where entries with an interval of max. 1000 are stored
> > in a Btree index. This works reasonably well for a Timeline, but makes the
> > approach less ideal for storing dense relationships.
> > The problem with the Timeline implementation is the ability to lookup the
> > tree root from a particular leave. In an ordinary Btree is would simply be a
> > traversal from the leave through the layers of block nodes to the tree root.
> > In Timeline the traversal will be different. It first has to move through
> > the Timeline list until it finds an entry that is stored in the Btree (which
> > worst case takes 1000 hops), and then it has to traverse the Btree up to the
> > tree root. To avoid this complicated traversal I ended up doing a lookup
> > through Lucene of the timeline URI (which is stored in all timeline list
> > entries). In fact I might as well have added the URI of the dense node as a
> > property and do the lookup through Lucene without the Timeline, it just
> > happens that I like the sort order of Timeline, making it a useful approach
> > anyway.
> > I will experiment using Btree directly (without Timeline) and see if that
> > leads to a simpler and faster traversal from leave to root node.
> > There is one more issue before this can become production ready. Btree as
> > it is implemented now is not thread safe (per the implementations Javadocs),
> > so it need some love and attention to make it work properly.
> > Niels
> >
> > > Date: Thu, 30 Jun 2011 13:57:20 +0200
> > > From

Re: [Neo4j] Does anyone use Solaris for neo4j app? [was Unable to memory map [was java.io.IOException: Resource temporarily unavailable in MappedPersistenceWindow.java?

[Michael Hunger]

Paul,

that's great news. I already wanted to ask you to profile it.

Could you please re-post the new numbers ?

Unfortunately we have no Solaris Box right now and on amazon ec2 there are just 
some OpenSolaris AMI images from 2009 available (don't know how compatible they 
are with Solaris 5.10 (10) that you use.

The higher latency of the NFS drive certainly impacts performance. 

I would love to look into the memory mapping issues on Solaris.

For profiling it would be great to have an application resembling your 
test-suite to evaluate/profile it.


Could you please enable gc-logs for your jvm and look at those?

Also could you run the write-test suite on solaris and report back the results: 
http://wiki.neo4j.org/content/Linux_Performance_Guide

Cheers

Michael

And the smallish windows PC is probably not the best PoC system for your 
production evaluation?

Am 01.07.2011 um 00:02 schrieb Paul Bandler:

> Oops - it seems it might not be neo4j that's chewing up the CPUby making 
> my log4j calls disabled with 'if(_log.isDebugEnabled)' pre-conditions to 
> avoid the expense of constructing the strings that might or might get logged, 
> I was able to improve the times for the 'hot-cache' tests described below by 
> about x10, bringing it within a factor of 1.5-2 of the C++ performance which 
> is within acceptable limits.
> 
> I'm still interested to know whether others are using neo4j on Solaris, 
> whether it's possible to use memory mapped i/o for buffers on Solaris, and 
> indeed whether that would further help performance, or does it just reduce 
> the heap demand?
> 
> On 30 Jun 2011, at 21:02, Paul Bandler wrote:
> 
>> Further to the problems reported below from running a test application on a 
>> server class Solaris box (M4000)  I re-ran the test but with the setting 
>> 'use_memory_mapped_buffers' set to false.  This had the +ve effect of 
>> avoiding the warnings and nio exceptions mentioned before so clearly there's 
>> some additional magic required to run under Solaris using memory mapped 
>> buffers.  I have enquired as to whether one needs setuid rights to use 
>> memory mapped files on Solaris and am told not btw...
>> 
>> The performance was woeful however - like walking thru' molasses in (a 
>> Swedish) mid-winter!  The test comprises a relatively complex navigation of 
>> a domain model to derive some information.  This is a re-implementation of a 
>> function  that  already exists in a production C++ application that uses an 
>> object database (based on a product called ObjectStore) -  so I have 
>> something to compare the results against.  I also have the results of 
>> running the same neo4j test on a low-end Windows PC - here are the 
>> comparative timings.  Each test contains 2 executions of exactly the same 
>> function with the idea that the first execution will be the worst case 
>> scenario as the caches are stone-cold, but then the second execution, being 
>> exactly the same traversals, should be the best case scenario with the 
>> cache's red-hot...  In all cases I'm using an Embedded neo4j database 
>> approach.
>> 
>> Windows (3.2Ghz PC with 1Gb of heap, neo4j database on local disk):-
>> - execution 1 ~ 41seconds
>> - execution 21.6 seconds
>> 
>> Solaris (Sun SPARC Enterprise M4000 Server System clock frequency: 1012 MHz 
>> Memory size: 32768 Megabytes with 3.5Gb heap; NEO4J database on nfs mounted 
>> filesystem...)
>> - execution 1 ~ 27+ _minutes_
>> - execution 2 ~ 5.5 seconds
>> 
>> Re-run with neo4j database on a local file system
>> - execution 1  ~20 seconds
>> - execution 2   1.56 seconds
>> 
>> 
>> Existing C++/ObjectStore implementation on same Solaris box as above:-
>> - execution 1 ~ 3.2 seconds
>> - execution 2 ~ 0.1 seconds
>> 
>> These results are disappointing.  It appears that the first execution is 
>> extremely I/O bound with a local enterprise server disk only improving on a 
>> low-end Windows PC by a factor of 2, and nfs reducing performance by a 
>> factor of about x80.  Once warmed, the low-end Windows PC is almost on par 
>> withe enterprise Solaris performance.  The fastest neo4j execution 2 is 16x 
>> slower than the existing C++/Objectstore implementation.   
>> 
>> Any suggestions on how it might be possible to narrow the gap between the 
>> performance of C++/Objectstore implementation and the neo4j one would be 
>> most welcome, as otherwise this will be the end of the line for this 
>> approach  I can accept the poor comparison of the first run if once the 
>> cache's are hot the second run comes somewhere reasonably comparable to what 
>> might expect from a comparison between a C++ and Java implementation of a 
>> CPU intensive application operation (say 2-3 times slower in Java...)?
>> 
>> cheers,
>> 
>> Paul
>> On 30 Jun 2011, at 14:06, Paul Bandler wrote:
>> 
>>> TEST:bandlerp@us2187$ uname -X
>>> System = SunOS
>>> Node = us2187
>>> Release = 5.10
>>> KernelID = Generic_13-03
>>> Machine = sun4u
>>> BusType = 

Re: [Neo4j] traversing densely populated nodes

[Niels Hoogeveen]

So far we have only considered the situation where one side of the relationship 
leads to a densely populated node, while the other side of the relationship 
leads to a non-densely populated node. This is likely to be the most prevalent 
situation (eg. category or class nodes), but it is conceivable that both sides 
of the relationship lead to densely populated nodes (eg. social networks with 
hugely popular users).
A single Btree per relationship type per direction may not suffice in such 
situation, but instead a double Btree should be used.
In the dense to non-dense scenario, the dense side will have a relationship to 
the root of the Btree for a particular combination of relationship type and 
direction, and the leave nodes of the Btree will have relations to the 
non-dense nodes.
A traversal listing all entries in the tree will look 
like:node-(TREE_ROOT)->node-(SUB_TREE)*->node-(KEY_ENTRY)->node
A traversal from an entry to to the related node is the 
reverse:node<-(KEY_ENTRY)-node<-(SUB_TREE)*-node<-(TREE_ROOT)-node
In the dense to dense scenario, both sides will have a relationship to the root 
of a Btree for a particular combination of relationship type and direction, and 
the leave nodes of both Btrees will have relations to one another.
The traversal in a dense to dense scenario 
is:node-(TREE_ROOT)->node-(SUB_TREE)*->node-(KEY_ENTRY)->node<-(KEY_ENTRY)-node<-(SUB_TREE)*-node<-(TREE_ROOT)-node
Note: Relationships that need to be traversed transitively are marked with a *
Niels

> From: pd_aficion...@hotmail.com
> To: user@lists.neo4j.org
> Date: Fri, 1 Jul 2011 00:36:32 +0200
> Subject: Re: [Neo4j] traversing densely populated nodes
> 
> 
> The relationship expander approach could certainly work for me. I already 
> have a meta layer containing information about relationship types, so it 
> would be a minor upgrade to store the type of relationship expander per 
> relationship type and to access relationships through the expander instead of 
> the direct API. 
> What we need now is a solid balanced tree to partition those dense 
> relationships. The Btree implementation in graph-collections is the probably 
> our best option right now, though as I said earlier, it needs to attention to 
> make it production ready. We need to make sure the implementation is thread 
> safe and there are some other issues we need to look into. For example, right 
> now the order of the Btree is fixed at 9. We could lift this value into the 
> constructor of the Btree, so we can create Btrees of various different orders 
> and then do performance tests, looking for the optimal order. The larger the 
> order, the fewer the hops, but too large an order may in fact create dense 
> nodes, something we try to avoid in the first place. 
> There will certainly be more issues we need to look into, but the combination 
> of the use of relationship expanders and a balanced tree index looks very 
> promising to me.
> As to the value for the Btree hash, how about trying the inverse of the node 
> id. Using the node id itself or a time stamp is actually disadvantageous, 
> because those values are in practice monotone (except for the reuse of node 
> ids) leading to lots of balancing acts. The inverse of the node id is equally 
> unique, but has a nicer distribution, making inserts into the Btree probably 
> a bit faster.
> Niels
> 
> 
> > Date: Thu, 30 Jun 2011 23:50:24 +0200
> > From: cr...@amanzi.com
> > To: user@lists.neo4j.org
> > Subject: Re: [Neo4j] traversing densely populated nodes
> > 
> > In the amanzi-index I link all indexed nodes into the index tree, so
> > traversals are straight up the tree. Of course this also means that there
> > are at least as many relationships as indexed nodes.
> > 
> > I was reviewing Michaels code for the relationship expander, and think that
> > is a great idea, tranparently using an index instead of the normal
> > relationships API, and can imagine using the relationship expander to
> > instead traverse the BTree to the final relationship to the leaf nodes.
> > 
> > So if we imagine a BTree with perhaps 10 or 20 hops from the root to the
> > leaf node, the relationship expander Michael described would complete all
> > hops and return only the last relationship, giving the illusion of direct
> > connections from root to leaf. This would certainly perform well, especially
> > for cases where there are factors limiting the number of relationships we
> > want returned. I think the request for type and direction is the first
> > obvious case, but we could be even more explicit than that, if we pass
> > constraints based on the BTree's consistent hash.
> > 
> > On Thu, Jun 30, 2011 at 11:36 PM, Niels Hoogeveen  > > wrote:
> > 
> > >
> > > In theory the approach I described earlier could work, though there are
> > > some pitfalls to the current implementation that need ironing out before
> > > this can become a recommended approach.
> > > The choice of Timeline instead of Btree may actually b