Re: [Neo4j] There's already an HA cluster managed by this ZooKeeper cluster

[Chris Gioran]

Hi,

Two questions, two requests:

1) Does this happen only on one machine or any member of the cluster?
Is it reproducible at will?
1) Could you please provide the messages.log for the other two machines as well?
2) Can you please tell me the result of this scenario:
- Create the cluster from scratch
- Restart one cluster member
  - It will fail as you have described
- Wipe that member's database, start it
  - It will successfully join the cluster, as you have described.
- What happens if you restart it again?

thank you,
CG

On Mon, Dec 5, 2011 at 2:26 PM, anirudha  wrote:
> Link for the messages.log file : https://gist.github.com/1433340
>
> --
> View this message in context: 
> http://neo4j-community-discussions.438527.n3.nabble.com/There-s-already-an-HA-cluster-managed-by-this-ZooKeeper-cluster-tp3554731p3561268.html
> Sent from the Neo4j Community Discussions mailing list archive at Nabble.com.
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Problem Starting the Server from Java

[Chris Gioran]

Ok, a shot in the dark here.

There seems to be a bug related with this error message, see

http://java.net/jira/browse/JERSEY-601

Does the code you use to startup the Neo4j server work nicely outside
of GF? If yes, then please try the workaround in the link above and
also see

http://jersey.java.net/nonav/documentation/latest/glassfish.html

which might be of use for forcing the neo4j provided jersey version to be used.

If startup fails outside of GF as well then please provide the new
stacktrace so we can have a closer look.

Hope that helps.

cheers,
CG

On Sun, Nov 27, 2011 at 8:18 PM,   wrote:
>
> I start the Server (latest stable ha-version) in a EJB-Application via a
> Singleton Bean with Glassfish 3.1.
>
> I do exactly what is described here:
> http://docs.neo4j.org/chunked/stable/server-embedded.html#_starting_the_server_from_java
>
> when i try to access 127.0.0.1:7474 i get a 503 error (see stacktrace
> below)
>
> any ideas what i can do to get it work ?
>
> thanks,
> ¨reinhard
>
>
>
> HTTP ERROR 503
>
> Problem accessing /. Reason:
>
>    java.lang.RuntimeException: javax.naming.NamingException: Lookup failed 
> for 'com/sun/jersey/config/CDIExtension' in 
> SerialContext[myEnv={java.naming.factory.initial=com.sun.enterprise.naming.impl.SerialInitContextFactory,
>  
> java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl,
>  java.naming.factory.url.pkgs=com.sun.enterprise.naming} [Root exception is 
> javax.naming.NameNotFoundException: CDIExtension not found]
>
> Caused by:
>
> javax.servlet.UnavailableException: java.lang.RuntimeException: 
> javax.naming.NamingException: Lookup failed for 
> 'com/sun/jersey/config/CDIExtension' in 
> SerialContext[myEnv={java.naming.factory.initial=com.sun.enterprise.naming.impl.SerialInitContextFactory,
>  
> java.naming.factory.state=com.sun.corba.ee.impl.presentation.rmi.JNDIStateFactoryImpl,
>  java.naming.factory.url.pkgs=com.sun.enterprise.naming} [Root exception is 
> javax.naming.NameNotFoundException: CDIExtension not found]
>        at 
> org.mortbay.jetty.servlet.ServletHolder.makeUnavailable(ServletHolder.java:415)
>        at 
> org.mortbay.jetty.servlet.ServletHolder.initServlet(ServletHolder.java:458)
>        at 
> org.mortbay.jetty.servlet.ServletHolder.doStart(ServletHolder.java:263)
>        at 
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>        at 
> org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:685)
>        at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
>        at 
> org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:517)
>        at 
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>        at 
> org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
>        at 
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>        at 
> org.mortbay.jetty.handler.HandlerWrapper.doStart(HandlerWrapper.java:130)
>        at org.mortbay.jetty.Server.doStart(Server.java:224)
>        at 
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
>        at 
> org.neo4j.server.web.Jetty6WebServer.startJetty(Jetty6WebServer.java:168)
>        at org.neo4j.server.web.Jetty6WebServer.start(Jetty6WebServer.java:105)
>        at 
> org.neo4j.server.NeoServerWithEmbeddedWebServer.startWebServer(NeoServerWithEmbeddedWebServer.java:252)
>        at 
> org.neo4j.server.NeoServerWithEmbeddedWebServer.start(NeoServerWithEmbeddedWebServer.java:106)
>        at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:87)
>        at org.neo4j.server.Bootstrapper.start(Bootstrapper.java:76)
>        at com.callooh.recommender.db.Database.start(Database.java:48)
>        at com.callooh.recommender.db.ConfigDB.(ConfigDB.java:56)
>        at 
> com.callooh.recommender.ejb.StatusMapBean.startup(StatusMapBean.java:69)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>        at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>        at java.lang.reflect.Method.invoke(Method.java:601)
>        at 
> com.sun.ejb.containers.interceptors.BeanCallbackInterceptor.intercept(InterceptorManager.java:1009)
>        at 
> com.sun.ejb.containers.interceptors.CallbackChainImpl.invokeNext(CallbackChainImpl.java:65)
>        at 
> com.sun.ejb.containers.interceptors.CallbackInvocationContext.proceed(CallbackInvocationContext.java:113)
>        at 
> com.sun.ejb.containers.interceptors.SystemInterceptorProxy.doCallback(SystemInterceptorProxy.java:138)
>        at 
> com.sun.ejb.containers.interceptors.SystemInterceptorProxy.init(SystemInterceptorProxy.java:120)
>        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>        at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorIm

Re: [Neo4j] Neo4j Data Capacities

[Chris Gioran]

Yes, this is true, with a few notes:

ID reuse complicates things a bit, meaning that if you delete nodes
and relationships some ids will remain unused until you restart the
database. Unclean shutdowns also may require scanning of the store
files to determine unused records - the
Config.REBUILD_IDGENERATORS_FAST parameter. So, the 35 bit address
space is an upper limit. Normally this number of "lost" records is
minuscule and easily recoverable so not a big deal.

The 36 bit address space for properties is a low limit - the id reuse
issue is practically non-existent for properties and since 1.5 there
is no 1-1 correspondence between property id and property entry (the
smallest ratio is 1:4). It all depends on the type of property - if it
classifies as a short string or short array and how big
(http://docs.neo4j.org/chunked/milestone/short-strings.html).

So you could have a db with around 34 billion nodes with one OUTGOING
relationship per node (so up to two per node, one INCOMING and one
OUTGOING, since every relationship connects two nodes) and at least 68
billion properties, with a max of 68*4=272 billion properties.

That's a lot of stuff! For reference, the smallest of those files will
be the node store with a size of (9 bytes/record * 2^35 records)/(2^30
bytes/gigabyte)  = 288 gigabytes. So you will start hitting machine
restrictions before you run out of id space.

cheers,
CG

On Thu, Nov 24, 2011 at 2:55 PM, bm3780  wrote:
> I've read that Neo4j has data capacity limitations
> (http://docs.neo4j.org/chunked/milestone/capabilities-capacity.html).  I
> would like to confirm my understandings that the node, properties, and
> relationships limitations are for each type (e.g. AND condition), not an
> either/or  (e.g. OR condition).
>
> Neo4j can hold:
>   * ~34 billions nodes, AND
>   * ~34 billion relationships, AND
>   * ~68 billion properties
>
> So I could theoretically have a single graph with 34 billion nodes, where
> each node had two properties and a single relationship.
>
> --
> View this message in context: 
> http://neo4j-community-discussions.438527.n3.nabble.com/Neo4j-Data-Capacities-tp3533552p3533552.html
> Sent from the Neo4j Community Discussions mailing list archive at Nabble.com.
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Sampling a Neo4j instance?

[Chris Gioran]

No, GraphDatabaseService wisely hides those things away. I would
suggest using instanceof and casting to EmbeddedGraphDatabase.

cheers,
CG

2011/11/16 Anders Lindström :
>
> Chris, thanks again for your replies.
> I realize now that I don't have the 'getConfig' method -- I'm writing a 
> server plugin and I only get the GraphDatabaseService interface passed to my 
> method, not a EmbeddedGraphDatabase. Is there an equivalent way of getting 
> the highest node index through the interface?
> Thanks.
>
>> Date: Thu, 10 Nov 2011 12:01:31 +0200
>> From: chris.gio...@neotechnology.com
>> To: user@lists.neo4j.org
>> Subject: Re: [Neo4j] Sampling a Neo4j instance?
>>
>> Answers inline.
>>
>> 2011/11/9 Anders Lindström :
>> >
>> > Thanks to the both of you. I am very grateful that you took your time to 
>> > put this into code -- how's that for community!
>> > I presume this way of getting 'highId' is constant in time? It looks 
>> > rather messy though -- is it really the most straightforward way to do it?
>>
>> This is the safest way to do it, that takes into consideration crashes
>> and HA cluster membership.
>>
>> Another way to do it is
>>
>> long highId = db.getConfig().getIdGeneratorFactory().get( IdType.NODE
>> ).getHighId();
>>
>> which can return the same value with the first, if some conditions are
>> met. It is shorter and cast-free but i'd still use the first way.
>>
>> getHighId() is a constant time operation for both ways described - it
>> is just a field access, with an additional long comparison for the
>> first case.
>>
>> > I am thinking about how efficient this will be. As I understand it, the 
>> > "sampling misses" come from deleted nodes that once was there. But if I 
>> > remember correctly, Neo4j tries to reuse these unused node indices when 
>> > new nodes are added. But is an unused node index _guaranteed_ to be used 
>> > given that there is one, or could inserting another node result in 
>> > increasing 'highId' even though some indices below it are not used?
>>
>> During the lifetime of a Neo4j instance there is no id reuse for Nodes
>> and Relationships - deleted ids are saved however and will be reused
>> the next time Neo4j starts. This means that if during run A you
>> deleted nodes 3 and 5, the first two nodes returned by createNode() on
>> the next run will have ids 3 and 5 - so highId will not change.
>> Additionally, during run A, after deleting nodes 3 and 5, no new nodes
>> would have the id 3 or 5. A crash (or improper shutdown) of the
>> database will break this however, since the ids-to-recycle will
>> probably not make it to disk.
>>
>> So, in short, it is guaranteed that ids *won't* be reused in the same
>> run but not guaranteed to be reused between runs.
>>
>> > My conclusion is that the "sampling misses" will increase with index usage 
>> > sparseness and that we will have a high rate of "sampling misses" when we 
>> > had many deletes and few insertions recently. Would you agree?
>>
>> Yes, that is true, especially given the cost of the "wasted" I/O and
>> of handling the exception. However, this cost can go down
>> significantly if you keep a hash set for the ids of nodes you have
>> deleted and check that before asking for the node by id, instead of
>> catching an exception. Persisting that between runs would move you
>> away from encapsulated Neo4j constructs and would also be more
>> efficient.
>>
>> > Thanks again.
>> > Regards,Anders
>> >
>> >> Date: Wed, 9 Nov 2011 19:30:36 +0200
>> >> From: chris.gio...@neotechnology.com
>> >> To: user@lists.neo4j.org
>> >> Subject: Re: [Neo4j] Sampling a Neo4j instance?
>> >>
>> >> Hi,
>> >>
>> >> Backing Jim's algorithm with some code:
>> >>
>> >>     public static void main( String[] args )
>> >>     {
>> >>         long SAMPLE_SIZE = 1;
>> >>         EmbeddedGraphDatabase db = new EmbeddedGraphDatabase(
>> >>                 "path/to/db/" );
>> >>         // Determine the highest possible id for the node store
>> >>         long highId = ( (NeoStoreXaDataSource)
>> >> db.getConfig().getTxModule().getXaDataSourceManager().getXaDataSource(
>> >>                 Config.DEFAULT_DATA_SOURCE_NAME )
>> >> ).getNeoStore().getNodeStore().getHighId();
>> >>         System.out.println( highId + " is the highest id" );
>> >>         long i = 0;
>> >>         long nextId;
>> >>
>> >>         // Do the sampling
>> >>         Random random = new Random();
>> >>         while ( i < SAMPLE_SIZE )
>> >>         {
>> >>             nextId = Math.abs( random.nextLong() ) % highId;
>> >>             try
>> >>             {
>> >>                 db.getNodeById( nextId );
>> >>                 i++;
>> >>                 System.out.println( "id " + nextId + " is there" );
>> >>             }
>> >>             catch ( NotFoundException e )
>> >>             {
>> >>                 // NotFoundException is thrown when the node asked is not 
>> >> in use
>> >>                 System.out.println( "id " + nextId + " not in use" );
>> >>          

Re: [Neo4j] Sampling a Neo4j instance?

[Chris Gioran]

Answers inline.

2011/11/9 Anders Lindström :
>
> Thanks to the both of you. I am very grateful that you took your time to put 
> this into code -- how's that for community!
> I presume this way of getting 'highId' is constant in time? It looks rather 
> messy though -- is it really the most straightforward way to do it?

This is the safest way to do it, that takes into consideration crashes
and HA cluster membership.

Another way to do it is

long highId = db.getConfig().getIdGeneratorFactory().get( IdType.NODE
).getHighId();

which can return the same value with the first, if some conditions are
met. It is shorter and cast-free but i'd still use the first way.

getHighId() is a constant time operation for both ways described - it
is just a field access, with an additional long comparison for the
first case.

> I am thinking about how efficient this will be. As I understand it, the 
> "sampling misses" come from deleted nodes that once was there. But if I 
> remember correctly, Neo4j tries to reuse these unused node indices when new 
> nodes are added. But is an unused node index _guaranteed_ to be used given 
> that there is one, or could inserting another node result in increasing 
> 'highId' even though some indices below it are not used?

During the lifetime of a Neo4j instance there is no id reuse for Nodes
and Relationships - deleted ids are saved however and will be reused
the next time Neo4j starts. This means that if during run A you
deleted nodes 3 and 5, the first two nodes returned by createNode() on
the next run will have ids 3 and 5 - so highId will not change.
Additionally, during run A, after deleting nodes 3 and 5, no new nodes
would have the id 3 or 5. A crash (or improper shutdown) of the
database will break this however, since the ids-to-recycle will
probably not make it to disk.

So, in short, it is guaranteed that ids *won't* be reused in the same
run but not guaranteed to be reused between runs.

> My conclusion is that the "sampling misses" will increase with index usage 
> sparseness and that we will have a high rate of "sampling misses" when we had 
> many deletes and few insertions recently. Would you agree?

Yes, that is true, especially given the cost of the "wasted" I/O and
of handling the exception. However, this cost can go down
significantly if you keep a hash set for the ids of nodes you have
deleted and check that before asking for the node by id, instead of
catching an exception. Persisting that between runs would move you
away from encapsulated Neo4j constructs and would also be more
efficient.

> Thanks again.
> Regards,Anders
>
>> Date: Wed, 9 Nov 2011 19:30:36 +0200
>> From: chris.gio...@neotechnology.com
>> To: user@lists.neo4j.org
>> Subject: Re: [Neo4j] Sampling a Neo4j instance?
>>
>> Hi,
>>
>> Backing Jim's algorithm with some code:
>>
>>     public static void main( String[] args )
>>     {
>>         long SAMPLE_SIZE = 1;
>>         EmbeddedGraphDatabase db = new EmbeddedGraphDatabase(
>>                 "path/to/db/" );
>>         // Determine the highest possible id for the node store
>>         long highId = ( (NeoStoreXaDataSource)
>> db.getConfig().getTxModule().getXaDataSourceManager().getXaDataSource(
>>                 Config.DEFAULT_DATA_SOURCE_NAME )
>> ).getNeoStore().getNodeStore().getHighId();
>>         System.out.println( highId + " is the highest id" );
>>         long i = 0;
>>         long nextId;
>>
>>         // Do the sampling
>>         Random random = new Random();
>>         while ( i < SAMPLE_SIZE )
>>         {
>>             nextId = Math.abs( random.nextLong() ) % highId;
>>             try
>>             {
>>                 db.getNodeById( nextId );
>>                 i++;
>>                 System.out.println( "id " + nextId + " is there" );
>>             }
>>             catch ( NotFoundException e )
>>             {
>>                 // NotFoundException is thrown when the node asked is not in 
>> use
>>                 System.out.println( "id " + nextId + " not in use" );
>>             }
>>         }
>>         db.shutdown();
>>     }
>>
>> Like already mentioned, this will be slow. Random jumps around the
>> graph are not something caches can keep up with - unless your whole db
>> fits in memory. But accessing random pieces of an on-disk file cannot
>> be done much faster.
>>
>> cheers,
>> CG
>>
>> On Wed, Nov 9, 2011 at 6:08 PM, Jim Webber  wrote:
>> > Hi Anders,
>> >
>> > When you do getAllNodes, you're getting back an iterable so as you point 
>> > out the sample isn't random (unless it was written randomly to disk). If 
>> > you're prepared to take a scattergun approach and tolerate being 
>> > disk-bound, then you can ask for getNodeById using a made-up ID and deal 
>> > with the times when your ID's don't resolve.
>> >
>> > It'll be slow (since the chances of having the nodes in cache are low) but 
>> > as random as your random ID generator.
>> >
>> > Jim
>> > ___
>> >

Re: [Neo4j] Sampling a Neo4j instance?

[Chris Gioran]

Hi,

Backing Jim's algorithm with some code:

public static void main( String[] args )
{
long SAMPLE_SIZE = 1;
EmbeddedGraphDatabase db = new EmbeddedGraphDatabase(
"path/to/db/" );
// Determine the highest possible id for the node store
long highId = ( (NeoStoreXaDataSource)
db.getConfig().getTxModule().getXaDataSourceManager().getXaDataSource(
Config.DEFAULT_DATA_SOURCE_NAME )
).getNeoStore().getNodeStore().getHighId();
System.out.println( highId + " is the highest id" );
long i = 0;
long nextId;

// Do the sampling
Random random = new Random();
while ( i < SAMPLE_SIZE )
{
nextId = Math.abs( random.nextLong() ) % highId;
try
{
db.getNodeById( nextId );
i++;
System.out.println( "id " + nextId + " is there" );
}
catch ( NotFoundException e )
{
// NotFoundException is thrown when the node asked is not in use
System.out.println( "id " + nextId + " not in use" );
}
}
db.shutdown();
}

Like already mentioned, this will be slow. Random jumps around the
graph are not something caches can keep up with - unless your whole db
fits in memory. But accessing random pieces of an on-disk file cannot
be done much faster.

cheers,
CG

On Wed, Nov 9, 2011 at 6:08 PM, Jim Webber  wrote:
> Hi Anders,
>
> When you do getAllNodes, you're getting back an iterable so as you point out 
> the sample isn't random (unless it was written randomly to disk). If you're 
> prepared to take a scattergun approach and tolerate being disk-bound, then 
> you can ask for getNodeById using a made-up ID and deal with the times when 
> your ID's don't resolve.
>
> It'll be slow (since the chances of having the nodes in cache are low) but as 
> random as your random ID generator.
>
> Jim
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] 1.5.M02 - JVM BootStrapper Issues

[Chris Gioran]

This issue was unintentionally introduced along with some changes
needed for better control and output from the windows wrapper. It was
a regression that has now been fixed. I would suggest either

- waiting for the next release, or

- if you are feeling adventurous, download a snapshot windows build
(http://neo4j.org/download/), extract
bin/windows-service-wrapper-3-SNAPSHOT.jar, rename that to
windows-service-wrapper-2.jar and replace the existing jar under bin/
in your current installation - the important thing is for the new jar
to have the same filename as the old one. That should fix all issues
that you have.

As a side note, i would really appreciate if you confirmed that your
configuration runs with a SNAPSHOT build.

thanks,
CG

On Sat, Nov 5, 2011 at 8:33 AM, Peter Neubauer
 wrote:
> Romiko,
> this is probably linked to the same issue,
> https://github.com/neo4j/packaging/issues/2
>
> Cheers,
>
> /peter neubauer
>
> GTalk:      neubauer.peter
> Skype       peter.neubauer
> Phone       +46 704 106975
> LinkedIn   http://www.linkedin.com/in/neubauer
> Twitter      http://twitter.com/peterneubauer
>
> http://www.neo4j.org              - NOSQL for the Enterprise.
> http://startupbootcamp.org/    - Öresund - Innovation happens HERE.
>
>
>
> On Thu, Nov 3, 2011 at 4:07 PM, Romiko Derbynew
>  wrote:
>> Hi Guys,
>>
>> I have downloaded the latest Community edition and notice these three custom 
>> lines of code causing the Neo4J server not to start up:
>>
>> wrapper.java.additional.1=-d64
>> wrapper.java.additional.1=-server
>> wrapper.java.additional.1=-Xss2048k
>>
>> With the previous version, they worked fine. They live at the bottom of the 
>> neo4j-wrapper.conf
>>
>> Error I get is.
>> java.lang.IllegalArgumentException
>>        at java.lang.ProcessImpl.(Unknown Source)
>>        at java.lang.ProcessImpl.start(Unknown Source)
>>        at java.lang.ProcessBuilder.start(Unknown Source)
>>        at java.lang.Runtime.exec(Unknown Source)
>>        at 
>> org.neo4j.wrapper.ServerProcessConsole.doStart(ServerProcessConsole.j
>> ava:39)
>>        at org.neo4j.wrapper.ServerProcess.(ServerProcess.java:116)
>>        at 
>> org.neo4j.wrapper.ServerProcessConsole.(ServerProcessConsole.ja
>> va:29)
>>        at 
>> org.neo4j.wrapper.NeoServiceWrapper.launchAsConsoleApp(NeoServiceWrap
>> per.java:48)
>>        at org.neo4j.wrapper.NeoServiceWrapper.main(NeoServiceWrapper.java:35)
>>
>> Is this expected, since I am sure the JVM should support adding these 
>> additional parameters?
>>
>> I want these in here, so I can always ensure it is 64bit, server mode with a 
>> stack size of 2048k
>>
>>
>> ___
>> Neo4j mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] 1.5M02 failed to start on reboot

[Chris Gioran]

Hi,

Are the three servers members of an HA cluster? If yes, can you
provide details of your setup (the conf/ directory contents would do
nicely)?
Could you provide us with the messages.log files of the servers?
Contact me off list if you have concerns about sharing it publicly.
If you could provide the corrupt database, that would be very nice.
Same off-list contact suggestion as above applies.
Finally, what prompted you to delete the data directory?

cheers,
CG

On Wed, Oct 26, 2011 at 8:58 PM, KanTube  wrote:
> I encountered a odd error.  I installed several windows updates (it was
> Tuesday :) ) and then proceeded to reboot my servers (three in total).  on
> one server the Neo4j service did not restart.  I then deleted the data
> directory and the service started correctly.  I then moved the data from
> another server to this server and it started correctly again.  Seems like
> original data file became corrupted during the shutdown.  I have the data
> files if you think this is something you want to investigate (it is a very
> small data set)
>
> setup win7 x64 (vm running in Fusion)
> Neo4j 1.5M02
>
>
>
> --
> View this message in context: 
> http://neo4j-community-discussions.438527.n3.nabble.com/1-5M02-failed-to-start-on-reboot-tp3455296p3455296.html
> Sent from the Neo4j Community Discussions mailing list archive at Nabble.com.
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] indexquery 12times faster than cypher --> neostore-upgrade-question

[Chris Gioran]

Hi,

it is hard to make predictions on how long it will take to migrate a
store to 1.5. What the migration time actually depends on is node and
relationship population, number of properties and number of dynamic
records in use. This correlates to file size but one cannot determine
the time based on just that, since not in use records still take up
space but do not contribute to the migration time. Also indices are
not touched, so they have nothing to do with upgrade time.

Based on our runs, a store of the size you mention should take in the
order of 1-2 hours to upgrade, depending of course on OS, filesystem,
hardware etc.

As far as the safety of your data goes: It is highly recommended to do
a backup of your data in a safe location before starting the upgrade
process and after shutting down cleanly the neo4j database. However,
we have taken all the measures we could to make sure that the process
is as safe as possible and we also keep a copy of the existing
database files under /upgrade_backup just to be extra
cautious. In the unlikely case anything goes south, you can always fix
the problem, restore the old database and try the upgrade again.

hope that helps,
CG

On Wed, Oct 12, 2011 at 9:30 AM, st.pa  wrote:
>
> Hi, this is great, I was still holding my breath and at first didnt even
> notice, then I got the snapshot and changed classpaths in my project to
> use 1.5.m, when I noticed the following error
>
> org.neo4j.graphdb.TransactionFailureException: Could not create data
> source [nioneodb], see nested exception for cause of error
> org.neo4j.kernel.impl.transaction.TxModule.registerDataSource(TxModule.java:153)
> org.neo4j.kernel.GraphDbInstance.start(GraphDbInstance.java:112)
>        at
> org.neo4j.kernel.EmbeddedGraphDbImpl.(EmbeddedGraphDbImpl.java:190)
>        at
> org.neo4j.kernel.EmbeddedGraphDatabase.(EmbeddedGraphDatabase.java:80)
> ...
> Caused by:
> org.neo4j.kernel.impl.storemigration.UpgradeNotAllowedByConfigurationException:
> To enable automatic upgrade, please set allow_store_upgrade in
> configuration properties
>        at
> org.neo4j.kernel.impl.storemigration.ConfigMapUpgradeConfiguration.checkConfigurationAllowsAutomaticUpgrade(ConfigMapUpgradeConfiguration.java:41)
>        at
> org.neo4j.kernel.impl.storemigration.StoreUpgrader.attemptUpgrade(StoreUpgrader.java:50)
>        at
> org.neo4j.kernel.impl.nioneo.store.NeoStore.tryToUpgradeStores(NeoStore.java:127)
>        at
> org.neo4j.kernel.impl.nioneo.store.NeoStore.verifyCorrectTypeDescriptorAndVersion(NeoStore.java:98)
>        at
> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.loadStorage(CommonAbstractStore.java:196)
>        at
> org.neo4j.kernel.impl.nioneo.store.CommonAbstractStore.(CommonAbstractStore.java:104)
>        at
> org.neo4j.kernel.impl.nioneo.store.AbstractStore.(AbstractStore.java:120)
>        at org.neo4j.kernel.impl.nioneo.store.NeoStore.(NeoStore.java:72)
>        at
> org.neo4j.kernel.impl.nioneo.xa.NeoStoreXaDataSource.(NeoStoreXaDataSource.java:137)
>        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
>        at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
>        at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
>        at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
>        at
> org.neo4j.kernel.impl.transaction.XaDataSourceManager.create(XaDataSourceManager.java:75)
>        at
> org.neo4j.kernel.impl.transaction.TxModule.registerDataSource(TxModule.java:147)
>        ... 24 more
>
>
> which tells me to allow the neo-store to be updated. That spooked me so
> much that I didnt yet try it in earnest, focusing instead on rewriting
> my app to work without thread-interrupts because that was apparently
> messing with lucene, and trying to find the reason why the
> worker-threads' transactions always keep deadlocking... as much as I
> desired that cypher-feature with the RelationshipIndexLookup, there are
> work-arounds, but I'll get around to testing it sometime soon.
>
> Before I do upgrade the store,
>
>   39 MB  neostore.nodestore.db
>  412 MB  neostore.propertystore.db
> 4315 MB  neostore.propertystore.db.arrays
> 1857 MB  neostore.propertystore.db.strings
> 1090 MB  neostore.relationshipstore.db
> plus 1.6GB of indices,
>
> any idea how long such a neostore-upgrade might take from 1.4.1. to that
> 1.5.snapshot? like seconds, minutes, hours, or worse?
>
>
> Am 02.10.2011 19:25, schrieb Andres Taylor:
>> And now it's ready.
>>
>> If you are using SNAPSHOT, please update your dependencies, and check it
>> out.
>> http://docs.neo4j.org/chunked/snapshot/query-start.html#start-relationship-by-index-lookup
>>
>> Thanks again for providing feedback. It's much more fun coding when I know
>> someone is using it.
>>
>> Andrés
>>
>> On Fri, Sep 30, 2011 at 10:37 AM, Andres Taylor<
>> andres.tay...@neotechnology.com>  wrote:
>>
>>> Hey!
>

Re: [Neo4j] Trying to use Neo4J with Atomikos transaction manager, issues with Lucene index

[Chris Gioran]

Hi Thibaut,

we currently have a working solution that changes the commitCows()
call from a during commit call to an afterCompletion synchronization.
It seems to work fine and your test case passes. Essentially it roots
out the implicit assumption in that part of the kernel that he
transaction should be available during the commit.
There is a code freeze in place however in preparation for the
upcoming milestone release, so you could either wait for the release
after that or clone the source shortly after, probably late this week.

hope that helps,
CG

On Tue, Oct 11, 2011 at 8:01 AM, tcolar  wrote:
> Hi Chris.
> I don't mean to pressure, I write OSS myself and I know time is
> available whenever it is, so i just have some quick questions about this
> issue.
>
> Basically I just need to decided whether to shelf this feature of our
> project for now or if i can get any idea on how much work / how long it
> would take to get it working.
>
> 1) Do you think it's an issue with atomikos itself and I should I bring
> the issue to them ?
> 2) Is that something that would be fixed in neo4j itself, if so do you
> have any rough estimates ?
> 3) Is there some sort of workaround you could think of that I could
> implement ?
>
> Any other suggestions or ammount of work/time needed would be appreciated.
>
> Thanks.
>
>
> --
> View this message in context: 
> http://neo4j-community-discussions.438527.n3.nabble.com/Trying-to-use-Neo4J-with-Atomikos-transaction-manager-issues-with-Lucene-index-tp3400319p3411656.html
> Sent from the Neo4j Community Discussions mailing list archive at Nabble.com.
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Trying to use Neo4J with Atomikos transaction manager, issues with Lucene index

[Chris Gioran]

This appears to be a different issue, having to do with the
implementation of Atomikos. From some preliminary testing it seems
that during commit the transaction returned by Atomikos'
TransactionManager#getTransaction() is null, which makes commiting of
the property COW maps to fail, making impossible to see the changes in
the properties of the node. The native Neo4j TransactionManager
returns the Transaction object and so the LockReleaser#commitCows()
call works correctly. I will get back to you on this.

cheers,
CG

On Sun, Oct 9, 2011 at 9:12 AM, Thibaut Colar  wrote:
> OK, I've modified the unit test and it shows something strange now ...
> assuming I'm not just too tired and messed up again :)
>
> So the previous  test, corrected, passes fine, but now after the first
> transaction is commited, i start a second one where i do the same exact
> check (node by prop)
>
> see:
> https://bitbucket.org/tcolar/stuff/src/0c4116d5b6ba/AtomikosNeo4j/src/test/java/net/colar/atomikosNeo4j/AtomikosNeo4jTest.java#cl-41
>
> Now in this second transaction the test fails 
>
> org.neo4j.graphdb.NotFoundException: testProp property not found for
> NodeImpl#1.
>    at
> org.neo4j.kernel.impl.core.Primitive.newPropertyNotFoundException(Primitive.java:173)
>    at org.neo4j.kernel.impl.core.Primitive.getProperty(Primitive.java:168)
>    at org.neo4j.kernel.impl.core.NodeProxy.getProperty(NodeProxy.java:145)
>    at
> net.colar.atomikosNeo4j.AtomikosNeo4jTest.test(AtomikosNeo4jTest.java:59)
>
> I assume that should work no, am I doing something wrong ?
>
> Thanks again.
>
> On 10/8/11 10:34 PM, Thibaut Colar wrote:
>>
>> You are right, was just using that in this (poor) unit test I made
>> quickly late last week, it's not used in the real code.
>>
>> So the unit test does work and does not have the issue I have in the
>> real and much more complex code.
>>
>> I guess I'll try to bring more of it in this unit test until i break it
>> too.
>>
>> One of the thing we use, is the neo4j events mechanism(to sync data to
>> another system), so i wonder if maybe using that in the context of
>> transactions might cause the issue, or maybe some other sort of
>> threading issue(what it looked like when i was debugging).
>>
>> Anyway will try to see if i can give you a failing test and/or better
>> info tomorrow or Monday.
>>
>> Thanks
>>
>>
>> On 10/8/11 1:51 PM, Mattias Persson wrote:
>>>
>>> The implicit key is "_id_" but that's an implementation detail and
>>> shouldn't
>>> be used to assert correctness of the index in unit tests.
>>>
>>> 2011/10/8 Thibaut Colar
>>>
>>>> Actually you are right, seems like "id" is the only one that fails ... i
>>>> really thought that was implicitly created, could it be possible than
>>>> somehow it should be but doesn't when using the transaction manager ?
>>>>
>>>> On 10/8/11 11:41 AM, Thibaut Colar wrote:
>>>>>
>>>>> Right, id is an implicit field ... works when not in transaction.
>>>>> If i remove that test, then I fail on the later index.get.
>>>>>
>>>>> As far as deleting the db, it's not in my test, but i removed the
>>>>> folder
>>>>> in y real code and manually before running the test.
>>>>>
>>>>> On 10/8/11 11:10 AM, Rick Bullotta wrote:
>>>>>>
>>>>>> If I remember correctly, neo has an implicit field on each index, and
>>>>
>>>> the name is either "id" or "_id".
>>>>>>
>>>>>> On Oct 8, 2011, at 1:58 PM, "Chris Gioran"<
>>>>
>>>> chris.gio...@neotechnology.com>     wrote:
>>>>>>>
>>>>>>> Hi Thibaut,
>>>>>>>
>>>>>>> I noticed the following snippet in your test case code:
>>>>>>>
>>>>>>>           long id = node.getId();
>>>>>>>           node.setProperty("testProp", "test");
>>>>>>>           index.add(node, "testProp", "test");
>>>>>>>
>>>>>>>           Assert.assertEquals(node.getProperty("testProp"), "test");
>>>>>>>
>>>>>>>           // Lookup By id
>>>>>>>           Assert.assertNotNull(graphDb.getNodeById(id));
>>>>>>>
>>>>>>>          

Re: [Neo4j] Trying to use Neo4J with Atomikos transaction manager, issues with Lucene index

[Chris Gioran]

Hi Thibaut,

I noticed the following snippet in your test case code:

long id = node.getId();
node.setProperty("testProp", "test");
index.add(node, "testProp", "test");

Assert.assertEquals(node.getProperty("testProp"), "test");

// Lookup By id
Assert.assertNotNull(graphDb.getNodeById(id));

// Now via Index
Node found = index.get("id", id).getSingle();
Assert.assertNotNull(found);
Assert.assertEquals(found.getId(), id);

which leads to the first test failure with a null result returned.
Here you are indexing a node with key "testProp" and value "test" but
you go on to ask it from the index with key "id" and value its id.
Since you haven't added that key/value pair in the index, rightfully
you don't get any results - the test fails in my setup even with the
native Neo4j transaction manager.

I changed that to ask the node from the index with the proper
key/value pair and the test passes - the same thing happens a bit
further down where you ask the index for a node with a key/value of
"testProp"/"prop" (instead of "testProp"/"test"). Moreover, it passes
with both the native Transaction Manager and the Atomikos
implementation.

Also, you are not deleting the db for the test - this means that since
you add to the index nodes with the same key/value the test will pass
only on the first run as getSingle() on the index hits will find the
previously added nodes and it will fail.

Could you try that out and see if it solves your issues?

I am glad that this functionality is being used in production - please
provide any feedback on any problems and what you think of this
feature.

hope that helps,
CG

On Thu, Oct 6, 2011 at 7:50 PM, tcolar  wrote:
> I've been trying to make this work for a few days but while it mostly works,
> the Lucene index just won't.
>
> We have a fairly complex setup including neo4j, mysql, mongo and JMS ad
> trying to do transactions accross all that with Atomikos.
>
> Since that is quite complex, I've made a smaller unit test just using
> Atomikos and Neo4j to demonstrate the issue:
>
> https://bitbucket.org/tcolar/stuff/src/ddd17191e9a4/AtomikosNeo4j
>
> The main test is here:
> https://bitbucket.org/tcolar/stuff/src/ddd17191e9a4/AtomikosNeo4j/src/test/java/net/colar/atomikosNeo4j/AtomikosNeo4jTest.java
>
> and results:
> https://bitbucket.org/tcolar/stuff/src/ddd17191e9a4/AtomikosNeo4j/test.log
>
> I used
> http://digitalstain.blogspot.com/2010/11/using-jotm-as-transactionmanager-in.html
> to get me going.
>
> And my code for that part is here:
> https://bitbucket.org/tcolar/stuff/src/ddd17191e9a4/AtomikosNeo4j/src/main/java/net/colar/atomikosNeo4j
>
> I'm thinking that's probably where I'm doing something wrong maybe ?
>
> I'm probably not doing something right but i can't seem to find what's going
> on with Lucene.
>
> The test works if not using Atomikos but just plain Neo4j Transactions.
>
> Anybody as an idea what is wrong?
>
> Thanks.
>
> --
> View this message in context: 
> http://neo4j-community-discussions.438527.n3.nabble.com/Trying-to-use-Neo4J-with-Atomikos-transaction-manager-issues-with-Lucene-index-tp3400319p3400319.html
> Sent from the Neo4j Community Discussions mailing list archive at Nabble.com.
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Neo4j Internals

[Chris Gioran]

On Thu, Oct 6, 2011 at 4:27 PM, Alexandre de Assis Bento Lima
 wrote:
> Thank you, Chris!
> BTY, do you intend to update your blog with the new information?

It seems harder to do for the older posts - it comes easier to just
write up changes as they are worked on and integrated.
This means by the way that the upcoming property stuff will probably
get their own writeup. Stay tuned.

Also, keep in mind that as Neo4j evolves, all team members write up
interesting stuff they work on. Look at http://planet.neo4j.org/ for a
list of posts covering various aspects of Neo4j, including, but not
limited to, some very interesting walk-throughs of kernel parts.

have fun,
CG

> Best,
> Alexandre.
>
> On Thu, 6 Oct 2011 11:02:18 +0300, Chris Gioran wrote
>> Hi Alexandre,
>>
>> first off, the overall structure of the database has not changed in
>> any drastic way. This means that most of the code paths are the same,
>> maybe with some classes/methods renamed, so with a bit of patience you
>> can follow through the new code.
>>
>> The most dramatic changes have been in the store files layout. Bits
>> have had their semantics changed to support a bigger id space and,
>> moreover, 1.5 will bring a complete rewrite of the PropertyStore.
>> However the code is again localized, so you should be able to find
>> your way around pretty quick - in-code comments should be helpful
>> there.
>>
>> If you have any trouble walking through the code I am sure you will
>> find this list very helpful.
>>
>> thank you,
>> CG
>>
>> On Wed, Oct 5, 2011 at 6:50 PM, Alexandre de Assis Bento Lima
>>  wrote:
>> > Hi,
>> >
>> > I couldn't find any information about Neo4j storage system and other 
>> > internal details on
>> > its website. Searching the web, I found:
>> >
>> > http://nosql.mypopescu.com/post/1396301953/neo4j-internals
>> >
>> > which gives many details and seems to be very good material. My question 
>> > is: as the
>> > articles were published on October, 2010, are they still up-to-date? Neo4j 
>> > 1.5 has the
>> > same implementation?
>> >
>> > Thanks a lot and special thanks to Chris Gioran, who wrote the articles.
>> >
>> > Alexandre.
>> >
>> > ___
>> > Neo4j mailing list
>> > User@lists.neo4j.org
>> > https://lists.neo4j.org/mailman/listinfo/user
>> >
>> ___
>> Neo4j mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>
>
> Alexandre A. B. Lima.
> ---
> COPPE / Computer Science
> Federal University of Rio de Janeiro
> Rio de Janeiro, Brazil.
> +55-21-2562-8690
>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Neo4j Internals

[Chris Gioran]

Hi Alexandre,

first off, the overall structure of the database has not changed in
any drastic way. This means that most of the code paths are the same,
maybe with some classes/methods renamed, so with a bit of patience you
can follow through the new code.

The most dramatic changes have been in the store files layout. Bits
have had their semantics changed to support a bigger id space and,
moreover, 1.5 will bring a complete rewrite of the PropertyStore.
However the code is again localized, so you should be able to find
your way around pretty quick - in-code comments should be helpful
there.

If you have any trouble walking through the code I am sure you will
find this list very helpful.

thank you,
CG

On Wed, Oct 5, 2011 at 6:50 PM, Alexandre de Assis Bento Lima
 wrote:
> Hi,
>
> I couldn't find any information about Neo4j storage system and other internal 
> details on
> its website. Searching the web, I found:
>
> http://nosql.mypopescu.com/post/1396301953/neo4j-internals
>
> which gives many details and seems to be very good material. My question is: 
> as the
> articles were published on October, 2010, are they still up-to-date? Neo4j 
> 1.5 has the
> same implementation?
>
> Thanks a lot and special thanks to Chris Gioran, who wrote the articles.
>
> Alexandre.
>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Neo4J Backup / Security

[Chris Gioran]

Hi. Check for answers inline

> I am currently evaluating Neo4J and browsing the docs (particularly 
> http://docs.neo4j.org/chunked/milestone/operations-backup.html). I couldn't 
> find anwers to the following questions:
> 1) Let's assume I have a windows development machine and linux production 
> server. Can I just copy the neo4j database folder from one system to the 
> other to get a working database? Are neo4j databases created with the windows 
> and the linux version compatible to each other?

Yes, the database files are created in an OS-agnostic way so moving
store files among machines and operating systems should present no
problems. Moreover, the online backup tool that is included in the
enterprise distribution works between platforms - you can backup a
Neo4j instance running on Linux from a Windows machine and the
resulting database will be functional on all platforms. Note however
that the online backup tool does not copy files - it replays database
logs and for that reason it works against a live Neo4j instance.
Simply copying store files must never be done while the database is
running - just cleanly shutdown the database, copy over the files and
restart.

> 2) Let's assume I have a linux production server with a neo4j instance 
> (embedded or server) running and want to make a backup to my local machine. 
> How do I proceed in terms of authentication? Actually, I couldn't find 
> anything in the docs about authentication at all. How can I connect to a 
> neo4j server from my local machine while preventing the rest of the world 
> from doing so? Also, I couldn't find anything about user management. Doesn't 
> neo4j have any user management system included?

Currently Neo4j does not come with any means of authentication or
permission control. All security concerns are left with the host
system and its facilities, like file permissions, firewall settings
and similar. Additionally, there is no concept of user in Neo4j. For
the time being, you can follow and build on some host-specific
guidelines available at:

http://docs.neo4j.org/chunked/snapshot/operations-security.html

Integration of security features is something we are considering
however for inclusion in upcoming versions.

Hope that helps,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] InvalidRecordException: Record[1983624] not in useRecord[1983624] not in use

[Chris Gioran]

Hi Rene,

According to git log, 1,4.M04 was tagged 10/06/2011, while the fix for
a bug in the BatchInserterImpl that caused corrupted stores which gave
the message you are seeing is

https://github.com/neo4j/community/commit/71fbe4ac1174fad16a7bb154c9f78d3f1b377248

which has a date of 04/07/2011.

This makes 1.4.M06 the earliest release to use that does not have this
issue or, even better, go for 1.4.1 which is the latest stable
release. Since you are comfortable using milestones, 1.5.M01 might be
of interest to you.

http://neo4j.org/download has the links that you need.

hope that helped,
CG

2011/9/28 René Pickhardt :
> Neo1.4.M04 community edition no self build.
>
> 2011/9/28 Mattias Persson 
>
>> Which version of neo4j did you use to import your data, was it a
>> downloadable package, from maven or building yourself from a branch?
>>
>> 2011/9/28 René Pickhardt 
>>
>> > hey everyone
>> >
>> > I got a strange error message that a record is not in use if I call
>> > hasRelationship() function even though I checked against null before! My
>> > real purpose was to start the traverser but it exited with the same error
>> > message. That is why I put the n.hasRelationship which also would not
>> work.
>> >
>> > I imported a graph with 50 mio. nodes and about the same size of
>> > relationships. (though I am about to import more relationships later on)
>> I
>> > put 2 mio. nodes in a lucence index called article.
>> >
>> > the counting variable t has the value 2385 (so the first 2385 calls work
>> > fine) if I skip this node the same problem accoures again at t = 3239.
>> >
>> > maybe it is usefull to say that I inserted the graph using the batch
>> > inserter.
>> >
>> > IndexHits res = article.get("key", values[0]);
>> > if (res!=null){
>> > Node n = res.getSingle();
>> >  if (n!=null){
>> > if (n.hasRelationship(DynamicRelationshipType.withName( "UPDATE" ),
>> > Direction.BOTH)){
>> >  //n.traverse(Order.BREADTH_FIRST,StopEvaluator.DEPTH_ONE,
>> > ReturnableEvaluator.ALL_BUT_START_NODE,DynamicRelationshipType.withName(
>> > "UPDATE" ) );
>> >  t++;
>> > }
>> > }
>> >
>> > callstack:
>> >
>> > org.neo4j.kernel.impl.nioneo.store.InvalidRecordException:
>> Record[1983624]
>> > not in useRecord[1983624] not in use
>> >
>> > at
>> >
>> >
>> org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getRecord(RelationshipStore.java:230)
>> >  at
>> >
>> >
>> org.neo4j.kernel.impl.nioneo.store.RelationshipStore.getChainRecord(RelationshipStore.java:337)
>> > at
>> >
>> >
>> org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:114)
>> >  at
>> >
>> >
>> org.neo4j.kernel.impl.nioneo.xa.ReadTransaction.getMoreRelationships(ReadTransaction.java:97)
>> > at
>> >
>> >
>> org.neo4j.kernel.impl.persistence.PersistenceManager.getMoreRelationships(PersistenceManager.java:108)
>> >  at
>> >
>> >
>> org.neo4j.kernel.impl.core.NodeManager.getMoreRelationships(NodeManager.java:603)
>> > at
>> >
>> org.neo4j.kernel.impl.core.NodeImpl.getMoreRelationships(NodeImpl.java:351)
>> >  at
>> >
>> >
>> org.neo4j.kernel.impl.core.NodeImpl.loadInitialRelationships(NodeImpl.java:318)
>> > at
>> >
>> >
>> org.neo4j.kernel.impl.core.NodeImpl.ensureRelationshipMapNotNull(NodeImpl.java:305)
>> >  at
>> >
>> >
>> org.neo4j.kernel.impl.core.NodeImpl.getAllRelationshipsOfType(NodeImpl.java:156)
>> > at
>> org.neo4j.kernel.impl.core.NodeImpl.getRelationships(NodeImpl.java:236)
>> >  at
>> org.neo4j.kernel.impl.core.NodeImpl.hasRelationship(NodeImpl.java:576)
>> > at
>> org.neo4j.kernel.impl.core.NodeProxy.hasRelationship(NodeProxy.java:109)
>> >  at IndexEvaluation.GenerateStreams(IndexEvaluation.java:124)
>> > at IndexEvaluation.(IndexEvaluation.java:98)
>> >  at EntryPoint.main(EntryPoint.java:20)
>> >
>> > --
>> > --
>> > mobile: +49 (0)176 6433 2481
>> >
>> > Skype: +49 (0)6131 / 4958926
>> >
>> > Skype: rene.pickhardt
>> >
>> > www.rene-pickhardt.de
>> >  
>> > ___
>> > Neo4j mailing list
>> > User@lists.neo4j.org
>> > https://lists.neo4j.org/mailman/listinfo/user
>> >
>>
>>
>>
>> --
>> Mattias Persson, [matt...@neotechnology.com]
>> Hacker, Neo Technology
>> www.neotechnology.com
>> ___
>> Neo4j mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
>
>
>
> --
> --
> mobile: +49 (0)176 6433 2481
>
> Skype: +49 (0)6131 / 4958926
>
> Skype: rene.pickhardt
>
> www.rene-pickhardt.de
>  
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Garbage Collection not in wrapper config

[Chris Gioran]

Hi,

if i understand you correctly, you are looking at the documentation
for snapshot (currently 1.5) but looking at the configuration file
from the 1st milestone of 1.4 (1.4.M01).

>From what git history shows, these lines was added on May 30, 2011,
which places them in the distribution around 1.4M04.

Anyway, i just downloaded a community snapshot (1.5-SNAPSHOT) from
neo4j.org and conf/neo4j-wrapper.conf has the expected contents, lines
8-9.

Is there something that i am missing here?

Note that adding manually this line in the configuration in versions
before 1.4M04 should work as expected.

cheers,
CG

On Mon, Sep 19, 2011 at 8:16 AM, Romiko Derbynew
 wrote:
> Hi Guys,
>
> I notice the docs 
> (http://docs.neo4j.org/chunked/snapshot/server-configuration.html) mention 
> that this line of config exists commented out.
>
> However, I cannot find it in the neo4j-wrapper.conf
> wrapper.java.additional.3=-Xloggc:data/log/neo4j-gc.log (1.4.M01 (2011-04-28)
>
> Is the docs out of date, or should this be added back, I see it as a very 
> useful tool to leave commented out, as I can use this with gcviewer?
>
> Romiko
>
>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Neo4j low-level data storage

[Chris Gioran]

Hi,


You can find a decent explanation for pre-1.4 storage layout here:

http://digitalstain.blogspot.com/2010/10/neo4j-internals-file-storage.html

It has not been updated however to explain extended addressing for 4B+
entities in a db (which do not change the layout, just the semantics
of some bits). Nevertheless, the basic structures (fixed size records,
doubly linked list etc) are still there.

For the way I/O is performed (and minimized/improved upon) you can
look at the way memory mapping is used here:

http://digitalstain.blogspot.com/2010/10/neo4j-internals-persistence-and-memory.html

And of course, there is object caching explained here:

http://digitalstain.blogspot.com/2010/10/neo4j-internals-caching.html



The organization in general is greatly favored by SSDs which are more
friendly to linked lists on disk. However, there is no optimization in
place yet targeting a specific storage technology - as in most
applications, the better the I/O performance, the better Neo4j does.

I'd be happy to help dive into more specific questions as they come to you.

cheers,
CG

On Wed, Sep 14, 2011 at 8:09 PM, danielb  wrote:
> Hello everybody,
>
> I have some questions regarding data storage in neo4j. How does neo4j store
> the data on the physical level? Are elements which have a close relationship
> in the graph stored in an adjacent way on disk? Any special binary format?
> How do you treat SSDs compared to common harddisks? What are your measures
> to improve I/O? Is there a technical documention which describes this?
>
> regards,
> Daniel
>
> --
> View this message in context: 
> http://neo4j-community-discussions.438527.n3.nabble.com/Neo4j-low-level-data-storage-tp3336483p3336483.html
> Sent from the Neo4j Community Discussions mailing list archive at Nabble.com.
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Size on disk, and number of properties

[Chris Gioran]

Hi Assem

The logical log file is where all changes to be performed to the store
are written out before they are actually applied - also referred to as
the Write Ahead Log. The file that hosts this is bound to a specific
size above which a rotation happens: a new file is created to host the
WAL, all pending transactions are moved over, the store files are
flushed and the old file marked as unused.
Setting the configuration option "keep_logical_logs" to "false"
deletes the old log files instead of keeping them around when that
size limit is hit and a rotation is triggered.

Hope that clear things up.

cheers,
CG

On Wed, Sep 7, 2011 at 8:58 PM, Aseem Kishore  wrote:
> Thanks Johan!
>
> Configuring Neo4j with "keep_logical_logs=false" the logs will
>> automatically be deleted upon rotation.
>
>
> What does "upon rotation" mean here?
>
> Aseem
>
> On Wed, Sep 7, 2011 at 5:56 AM, Johan Svensson wrote:
>
>> Removing the log files ending with ".v" at runtime is
>> perfectly safe to do but will turn off the ability to do incremental
>> backups. You can however still perform live full backups.
>>
>> Configuring Neo4j with "keep_logical_logs=false" the logs will
>> automatically be deleted upon rotation.
>>
>> -Johan
>>
>> On Sat, Sep 3, 2011 at 1:49 AM, Aseem Kishore 
>> wrote:
>> > Thanks for the insights Johan!
>> >
>> > Regarding the existing disk space then, by far the bulk of it is from the
>> > logs. Is there a way to prune or garbage collect them? Is simply deleting
>> > the files safe? Should the db be off if I do that? Etc.
>> >
>> > Thanks much!
>> >
>> > Aseem
>> >
>> > On Tue, Aug 30, 2011 at 2:47 AM, Johan Svensson > >wrote:
>> >
>> >> Hi Aseem,
>> >>
>> >> This is actually expected behavior when performing file copy of
>> >> running db and starting up with default configuration. If you remove
>> >> the files ending with .id in the db directory on the local snapshot
>> >> and start up setting "rebuild_idgenerators_fast=false" you should see
>> >> the accurate amount of nodes, relationships and properties.
>> >>
>> >> Regarding the amount of properties not matching this could be due to a
>> >> non clean shutdown on the production system. We are planing on
>> >> improving this in the near future by allowing for more aggressive
>> >> reuse of ids for properties. This will specifically improve things for
>> >> workloads that perform a lot of property updates.
>> >>
>> >> -Johan
>> >>
>> >> On Tue, Aug 30, 2011 at 10:05 AM, Aseem Kishore <
>> aseem.kish...@gmail.com>
>> >> wrote:
>> >> > Hey guys,
>> >> >
>> >> > We do offline backups of our db on a semi-regular basis (every few
>> days),
>> >> > where we (1) stop the running db, (2) copy its data directory and (3)
>> >> > restart the db.
>> >> >
>> >> > A few times early on, we did running backups -- but not the proper
>> >> "online"
>> >> > way -- where we simply copied the data directory while the db was
>> still
>> >> > running. (We did this during times where we were confident no requests
>> >> were
>> >> > hitting the db.)
>> >> >
>> >> > We noticed that every time we did the running backup, the number of
>> >> > properties the web admin reported -- and the space on disk of the db
>> --
>> >> > would jump up quite a bit. We stopped doing that recently.
>> >> >
>> >> > But even now, both these numbers have gotten quite a bit higher than
>> we
>> >> > expect to, and strangely, they seem to differ highly between the
>> running
>> >> db
>> >> > and the copies.
>> >> >
>> >> > What could be causing all of this?
>> >> >
>> >> > Here are our current numbers:
>> >> >
>> >> > *Production*
>> >> > - 2,338 nodes
>> >> > - 4,473 rels
>> >> > - 114,231 props (higher than we would expect it to be, but not by an
>> >> order
>> >> > of magnitude)
>> >> > - *1.39 GB!* <-- this is way unexpected, particularly since our db
>> used
>> >> to
>> >> > be in the ~10 KB ballpark, and we certainly haven't experienced hockey
>> >> stick
>> >> > growth yet ;) The logical log only takes up 57 KB (0%) btw.
>> >> >
>> >> >
>> >> > *Local snapshot*
>> >> > - 2,338 nodes
>> >> > - 4,473 rels
>> >> > - *2,607,892 props!!!* <-- ???
>> >> > - *1.37 GB!* <-- equally surprisingly high, but also interesting that
>> >> it's
>> >> > less than the production db's size. 0 KB logical logs.
>> >> >
>> >> >
>> >> > I looked around the wiki and searched this mailing list but didn't
>> find
>> >> much
>> >> > clues here. But as requested on another thread, here's the output of
>> `ls
>> >> -lh
>> >> > data/graph.db/`:
>> >> >
>> >> > total 1474520
>> >> > -rw-r--r--   1 aseemk  staff    11B Aug 30 00:46 active_tx_log
>> >> > drwxr-xr-x  52 aseemk  staff   1.7K Aug 30 00:46 index/
>> >> > -rw-r--r--   1 aseemk  staff   343B Aug 30 00:46 index.db
>> >> > -rw-r--r--   1 aseemk  staff   854K Aug 30 00:46 messages.log
>> >> > -rw-r--r--   1 aseemk  staff    36B Aug 30 00:46 neostore
>> >> > -rw-r--r--   1 aseemk  staff     9B Aug 30 00:46 neostore.id
>> >> > -rw-r--r--   1 aseemk  staff    26K Aug 30 0

Re: [Neo4j] Online-backup error

[Chris Gioran]

Hi,

though the command syntax seems ok, i find it strange that you try to
backup from the webadmin port (7474). Have you actually changed the
configuration to enable backup from port 7474? If not, and provided
you have not explicitly set the backup port to something different,
try as

bin/neo4j-backup -full -from single://192.168.11.88 -to
/home/sometime/backup

since the client defaults to 6362, which is where the backup server
listens to by default.

If, for some reason, you need to change the backup port, change in
conf/neo4j.properties the line

enable_online_backup=true

to

enable_online_backup=port=

and the command to

bin/neo4j-backup -full -from single://192.168.11.88: -to
/home/sometime/backup

and you should be fine.

hope that helps,
CG

On Wed, Aug 31, 2011 at 3:43 PM, sometime  wrote:
> I use
> bin/neo4j-backup -full -from single://192.168.11.88:7474 -to
> /home/sometime/backup
>
> output from console:
> Performing full backup from 'single://192.168.11.88:7474'
> Couldn't connect to 'single://192.168.11.88:7474',
> org.neo4j.com.ComException: Channel has been closed
>
> strange...
> any idea?
>
> --
> View this message in context: 
> http://neo4j-community-discussions.438527.n3.nabble.com/Online-backup-error-tp3298211p3298211.html
> Sent from the Neo4j Community Discussions mailing list archive at Nabble.com.
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] bug in neo4j-community-1.4.1-unix.tar.gz neo4j script ?

[Chris Gioran]

The only circumstances under which i can reproduce a *similar* failure
involve a linux client, specifically an Ubuntu 11.04 installation. In
more detail:

I have a neo4j shell server running remotely (on a windows machine,
fwiw). Then, from a recently unpacked installation of neo4j (1.4.1) i
run

bin/neo4j-shell -host 192.168.0.8

I manage to connect (meaning i get back a prompt) but for every
entered command i get (in this case for pwd)

neo4j-sh (0)$ pwd
Connection refused to host: 127.0.1.1; nested exception is:
java.net.ConnectException: Connection refused: connect

This is tracked back to an entry in /etc/hosts file:

127.0.1.1   

After removing that line everything works.

If after trying the above you still face problems, would you mind
sharing some more details about your setup?

- Client/server details (at least OS versions, , java -version), the
more the merrier.
- Network topology
- A sample console session with the error

thanks,
CG

On Fri, Aug 26, 2011 at 7:19 PM, Francesco Ruffino  wrote:
> Thank you Chris!
>
> I'm playing with the neo4j-shell.
> I'm using the one I have found in the tar.gz
>
> I have tried to run this command:
> [myclient]$ ./neo4j-community-1.4.1/bin/neo4j-shell -host myserver -port 1337
>
> but nothing happens: the shell remains hanging...
>
> I have checked on the server-side
> [myserver] netstat -an | grep 1337
>
> I can see the port in LISTEN and then in ESTABLISHED:
> tcp        0      0 :::1337                     :::*
>     LISTEN
> tcp        0      0 :::10.226.222.155:1337
> :::10.48.14.57:58329    ESTABLISHED
>  but nothing happens on the client side...
>
> If I use the neo4j-shell directly on the server it works perfectly.
>
> what is wrong?
>
> thank you,
> franz
>
>
>
>
> 2011/8/26 Chris Gioran :
>> Hi,
>>
>> this is a known issue. Please follow the documentation and run from
>> the installation root (/home/ec2-user/neo4j-community-1.4.1/ in your
>> example) as
>>
>> bin/neo4j 
>>
>> cheers,
>> CG
>>
>> On Fri, Aug 26, 2011 at 1:41 PM, Francesco Ruffino  
>> wrote:
>>> [ec2-user@xxx bin]$ pwd
>>> /home/ec2-user/neo4j-community-1.4.1/bin
>>>
>>> [ec2-user@ bin]$ sudo ./neo4j start
>>> Starting Neo4j Server...Error: missing Neo4j Library, expected at
>>> /home/ec2-user/lib
>>>
>>>
>>> doesn't resolve correctly the "$NEO4J_HOME" variable
>>> ___
>>> Neo4j mailing list
>>> User@lists.neo4j.org
>>> https://lists.neo4j.org/mailman/listinfo/user
>>>
>> ___
>> Neo4j mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] bug in neo4j-community-1.4.1-unix.tar.gz neo4j script ?

[Chris Gioran]

Hi,

this is a known issue. Please follow the documentation and run from
the installation root (/home/ec2-user/neo4j-community-1.4.1/ in your
example) as

bin/neo4j 

cheers,
CG

On Fri, Aug 26, 2011 at 1:41 PM, Francesco Ruffino  wrote:
> [ec2-user@xxx bin]$ pwd
> /home/ec2-user/neo4j-community-1.4.1/bin
>
> [ec2-user@ bin]$ sudo ./neo4j start
> Starting Neo4j Server...Error: missing Neo4j Library, expected at
> /home/ec2-user/lib
>
>
> doesn't resolve correctly the "$NEO4J_HOME" variable
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] setProperty Exception

[Chris Gioran]

Hi John,

maybe you are trying to add a property that has a value of String[]
with at least one of the array members being null?. That would cause
the first exception. The second exception is most probably because you
try to success() and finish() your transaction even after the
exception.

For the first issue, always make sure that you are not adding null
values, not even as members of an array.
For the second issue, make sure that you have a try/catch block around
your transactional code that does tx.success() only when everything is
in place, with the catch block doing the tx.failure(). At the finally
block, just do tx.finish() - that should do it.

In code:

Transaction tx = db.beginTx();
try {
  // do tx stuff here
  tx.success();
}
catch (Exception e) {
  tx.failure();
}
finally {
  tx.finish();
}

Hope the above help. If your code does not conform to my assumptions
please share some more details.

cheers,
CG

On Thu, Aug 25, 2011 at 5:43 AM, John Howard  wrote:
> I am using embedded 1.4.1
>
> I get these 2 strange exceptions almost simultaneously when try to create a
> new property.
>
> java.lang.NullPointerException
> at
> org.neo4j.kernel.impl.nioneo.store.DynamicArrayStore.allocateFromString(DynamicArrayStore.java:142)
> at
> org.neo4j.kernel.impl.nioneo.store.DynamicArrayStore.allocateRecords(DynamicArrayStore.java:369)
> at
> org.neo4j.kernel.impl.nioneo.store.PropertyStore.allocateArrayRecords(PropertyStore.java:466)
> at
> org.neo4j.kernel.impl.nioneo.store.PropertyStore.encodeValue(PropertyStore.java:535)
> at
> org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.nodeAddProperty(WriteTransaction.java:1314)
> at
> org.neo4j.kernel.impl.nioneo.xa.WriteTransaction.nodeAddProperty(WriteTransaction.java:1741)
> at
> org.neo4j.kernel.impl.persistence.PersistenceManager.nodeAddProperty(PersistenceManager.java:140)
> at
> org.neo4j.kernel.impl.core.NodeManager.nodeAddProperty(NodeManager.java:849)
> at org.neo4j.kernel.impl.core.NodeImpl.addProperty(NodeImpl.java:101)
> at org.neo4j.kernel.impl.core.Primitive.setProperty(Primitive.java:411)
> at org.neo4j.kernel.impl.core.NodeProxy.setProperty(NodeProxy.java:120)
> at graph.createProperty(graph.java:232)
>
> and then
> org.neo4j.graphdb.TransactionFailureException: Unable to commit transaction
> at org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:98)
> Caused by: javax.transaction.RollbackException: Failed to commit,
> transaction rolledback
> at
> org.neo4j.kernel.impl.transaction.TxManager.rollbackCommit(TxManager.java:811)
> at org.neo4j.kernel.impl.transaction.TxManager.commit(TxManager.java:645)
> at
> org.neo4j.kernel.impl.transaction.TransactionImpl.commit(TransactionImpl.java:109)
> at org.neo4j.kernel.TopLevelTransaction.finish(TopLevelTransaction.java:85)
> ... 40 more
>
>
> TIA
>
> John
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Neo4j Rest Server memory configuration

[Chris Gioran]

Hi there,

see inline for my take to answering your questions. If someone has
more info please share and/or correct.

On Sun, Aug 21, 2011 at 10:39 AM, mike_t  wrote:
> I installed neo4j as a service on linux. Then I configured in the
> neo4j-community-1.4.1/conf/neo4j-wrapper.conf file following init and max
> memory:
>
> # Initial Java Heap Size (in MB)
> wrapper.java.initmemory=512
>
> # Maximum Java Heap Size (in MB)
> wrapper.java.maxmemory=768
>
> The "top" output shows following:
> PID    USER   PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
> 17408 neo4j  18   0 1188m 155m  11m S    0  5.1   2:12.76 java
>
>
> Should neo4j not reserverd 512m memory? What did I wrong?

Asking for any amount of memory does not mean that the memory pages
are actually allocated - they are reserved. The JVM marks a range of
addresses as reserved and the OS obliges, but through the way virtual
memory works, the RAM is not actually used. This means that the JVM
"thinks" that it has 512M but the OS knows better. The upside of not
needing to do re-sizing calculations on heap growth is still there -
when the JVM decides to write to a reserved page that is not still
allocated, a page fault will be raised, the kernel will intervene etc,
as per usual.

> How can I discover If neo4j is running in java server mode? Result of "java
> -version" is:
> OpenJDK 64-Bit Server VM (build 19.0-b09, mixed mode).

Interesting question. It seems there is no vendor independent way -
the client vs server thing is vendor dependent to begin with. The
closest thing that seems to work is parsing the VM name string,
reachable via

System.getProperty( "java.vm.name" )

which on my machine returns

Java HotSpot(TM) 64-Bit Server VM

with java -version printing

java version "1.6.0_24"
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)

The same is reported of course via jconsole. Interestingly, i could
not start a client JVM. The JRE's help text talks about a -server
option only and that the default is server. No -client, although the
JVM starts with it in the command line. I guess this has to do with
all my machines being 64-bit multicore.

Hope the above helped.

cheers,
CG

> --
> View this message in context: 
> http://neo4j-community-discussions.438527.n3.nabble.com/Neo4j-Rest-Server-memory-configuration-tp3272360p3272360.html
> Sent from the Neo4j Community Discussions mailing list archive at Nabble.com.
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Neo4j startup script syntax errors

[Chris Gioran]

Hi,

yes, the lsof utility must be present (an assumption that holds for
most modern distributions). If you have a workaround for Solaris,
please share.

Could please specify which version of neo4j gives you this error?

thanks,
CG

On Tue, Aug 16, 2011 at 8:42 PM, Vyoma  wrote:
> Hi Andreas,
>
> I think that worked:
>
>> diff utils utils.bak
> 30c30
> <       if [[ ${line} =~ "^([^#\s][^=]+)=(.+)$" ]]; then
> ---
>>       if [[ ${line} =~ ^([^#\s][^=]+)=(.+)$ ]]; then
> 33c33
> <         if [[ ${key} =~ "^(.*)_([0-9]+)$" ]]; then
> ---
>>         if [[ ${key} =~ ^(.*)_([0-9]+)$ ]]; then
>
> It is not giving me those syntax errors now.
>
> It does give me an "command not found" error now:
>
>> ./bin/neo4j start
> id: illegal option -- u
> Usage: id [-ap] [user]
> /home/00/ntmop/neo4j
> Starting Neo4j Server...WARNING: not changing user
>  process [8975]... waiting for server to be ready../bin/neo4j: line 172:
> lsof: command not found
> grep: can't open -B1
> ../bin/neo4j: line 172: lsof: command not found
> grep: can't open -B1
>  BAD.
> Neo4j Server may have failed to start, please check the logs.
>
> But this must be due to "lsof" not being available. I think I need to get
> that installed.
>
> Thanks!
>
> -
> KMaheshBhat.com
> --
> View this message in context: 
> http://neo4j-community-discussions.438527.n3.nabble.com/Neo4j-Neo4j-startup-script-syntax-errors-tp3136965p3259350.html
> Sent from the Neo4j Community Discussions mailing list archive at Nabble.com.
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] cant use index after commit

[Chris Gioran]

Hi Ahmed,

if you are still having trouble, could you please provide a bare bones
test case that reliably reproduces the problem?

cheers,
CG

On Sun, Aug 7, 2011 at 6:56 PM, ahmed.elsharkasy
 wrote:
> Are you sure no other reason?
>
> --
> View this message in context: 
> http://neo4j-community-discussions.438527.n3.nabble.com/cant-use-index-after-commit-tp3233038p3233302.html
> Sent from the Neo4j Community Discussions mailing list archive at Nabble.com.
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] auto indexes in REST in 1.4?

[Chris Gioran]

Hi,

just enabling auto indexing via ???_auto_indexing=true is not enough
to actually include content in the auto index. You will also have to
explicitly state which are the properties to monitor for each
primitive (node or relationship) in the ???_keys_indexable list. In
other words, nothing is auto indexed by default.

The current implementation of auto indexing does not automatically add
or remove auto index contents from the whole graph following changes
in the monitored properties list. This is a known shortcoming and it
is the next improvement in the list for auto indexing.

hope this helps,
CG

On Sat, Jul 30, 2011 at 4:04 PM, dhsieh  wrote:
> Does one need to specify the following list of node_keys_indexable &
> relationship_keys_indexable in neo4j.property to enable auto indexing for
> those node/relationship properties? Also, will they work if those properties
> were created prior to adding the list in neo4j.property file?
>
> node_auto_indexing=true
> *node_keys_indexable=name,age*
>
> relationship_auto_indexing=true
> *relationship_keys_indexable=ROOT,KNOWS,CODED_BY *
>
>
> --
> View this message in context: 
> http://neo4j-community-discussions.438527.n3.nabble.com/Neo4j-auto-indexes-in-REST-in-1-4-tp3159205p3211749.html
> Sent from the Neo4j Community Discussions mailing list archive at Nabble.com.
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] neo4j.bat - java path

[Chris Gioran]

Hi,

that is a valid complaint. The main idea was to use the java
executable that is on the Path - however that can be still the case
and allow for custom executable locations with just one string
substitution.
Consider this added into the backlog.

cheers,
CG

On Fri, Jul 29, 2011 at 2:04 PM, Romiko Derbynew
 wrote:
> Hi Guys,
>
> Is it possible to update the neo4j.bat file for future releases to have a 
> global variable for "java "
>
> Currently it is located in two places in the file neo4j.bat. We using  Neo4j 
> in the cloud and currently we have to do string replacements at two locations:
>
> e.g. - these are my configs that work on dev fabric as I do not rely on 
> environment paths since java and neo4j are boot strapped from blob storage 
> and the db is on a cloud drive.
> INSTALL SECTION
> set mainclass="-DserverMainClass=org.neo4j.server.Bootstrapper"
> set binPath="java "-DworkingDir...
>
> CONSOLE SECTION
> set mainclass="-DserverMainClass=org.neo4j.server.Bootstrapper"
> java "-DworkingDir=%~dp0.."
>
> Perhaps it would be easier to have one location to store this value, to make 
> it easier to use absolute paths for java. We use absolute paths due to Azure 
> VM's.
>
> e.g.
> $Java_Path  
> ="e:\AzureTemp\s0\deployment(337)\res\deployment(337).Neo4j.Azure.Server.Neo4jServerHost.0\directory\Neo4jInst\jre6\bin\java.exe
>  "
>
> And then
> set mainclass="-DserverMainClass=org.neo4j.server.Bootstrapper"
> set binPath="$Java_Path "-DworkingDir...
>
> CONSOLE SECTION
> set mainclass="-DserverMainClass=org.neo4j.server.Bootstrapper"
> "$Java_Path " "-DworkingDir=%~dp0.."
>
> We use auto configuration in the cloud, so we always customize the configs on 
> the fly when vm's spin up. Easier way to search and replace strings would be 
> much appreciated. This will make it much easier for us to customize the 
> config on the fly by just replacing $Java_Path.
>
> e.g. after boot strap in the cloud.
> :install
> set 
> classpath="-DserverClasspath=lib/*.jar;system/lib/*.jar;plugins/*.jar;system/coordinator/lib/*.jar"
> set mainclass="-DserverMainClass=org.neo4j.server.Bootstrapper"
> set 
> binPath="e:\AzureTemp\s0\deployment(337)\res\deployment(337).Neo4j.Azure.Server.Neo4jServerHost.0\directory\Neo4jInst\jre6\bin\java.exe
>  "-DworkingDir=%~dps0.." -DconfigFile=conf\neo4j-wrapper.conf %classpath% 
> %mainclass% -jar "%~dps0windows-service-wrapper-1.jar" %serviceName%"
> and
> :console
> set 
> classpath="-DserverClasspath=lib/*.jar;system/lib/*.jar;plugins/*.jar;system/coordinator/lib/*.jar"
> set mainclass="-DserverMainClass=org.neo4j.server.Bootstrapper"
> "e:\AzureTemp\s0\deployment(337)\res\deployment(337).Neo4j.Azure.Server.Neo4jServerHost.0\directory\Neo4jInst\jre6\bin\java.exe"
>  "-DworkingDir=%~dp0.." -DconfigFile=conf\neo4j-wrapper.conf %classpath% 
> %mainclass% -jar "%~dp0windows-service-wrapper-1.jar"
>
>
>
> Thanks.
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] auto indexes in REST in 1.4?

[Chris Gioran]

In the upper left part where you can select edition and platform, the
"Select Release" drop down list has a "Current stable version (1.4)"
option. Select that instead of the default "Current Milestone version"
option and you should be fine.

cheers,
CG

On Thu, Jul 28, 2011 at 5:32 PM, dhsieh  wrote:
> Where is the GA version download link? I can't find it in
> http://neo4j.org/download.
>
> --
> View this message in context: 
> http://neo4j-community-discussions.438527.n3.nabble.com/Neo4j-auto-indexes-in-REST-in-1-4-tp3159205p3206845.html
> Sent from the Neo4j Community Discussions mailing list archive at Nabble.com.
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] strange problem while getting a node property

[Chris Gioran]

Hi Jean-Sébastien,

could you please tell us:

0. Are you sure that Node#0 is the one reporting that it has the
property key "package"?
1. How is the graph created? Specifically, how are the properties
added? The "package" property is being reported as existing, so it has
to get there somehow.
2. What version of Neo4j are you using?
3. Are the operations you described part of the same or different
transactions? The same instance of the database?

It would be very helpful if you provided a small piece of code that
reproduces what you see.

cheers,
CG

On Thu, Jul 28, 2011 at 2:18 PM, Jean-Sébastien Stoffen  wrote:
> Hi,
> I've this strange problem when I try to collect data from the graph with
> the Java API in Groovy :
>
> db.allNodes.each {node ->
>     cpt=0
>     node.getRelationships().each {rel ->
>             cpt++
>         }
>     println ("${node} ${cpt}")
>     println node.getPropertyKeys()
> }
>
> The iteration on each node is right working.
> The iteration to count the relationships on each node is working too.
>
> The call node.getPropertyKeys() gives me the list of the properties like
> this :
> [nbrel, version, maintainer, section, architecture, package, priority,
> dataset, installedSize]
>
> But,
>
> If a call node.getProperty("package")
> I receive this error :
> Caught: org.neo4j.graphdb.NotFoundException: package property not found
> for NodeImpl#0
>
> And, If I set the value just before, for test like this :
> node.setProperty("package", "test")
> println node.getProperty("package")
>
> I get the value.
>
> So I can't get property which was not set by the node.setProperty method.
> The initial data are copied into the graph with a perl script using the
> Neo4j REST interface.
>
> Maybe I do something wrong,
> I'm a newbie in both Neo4j and Groovy
>
> Regards,
> Jean-Sébastien Stoffen
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Deletion From Neo4j

[Chris Gioran]

Hi Ahmad,

Deleting a Node (or a Relationship or a Property for that matter) does
not actually remove the corresponding record - it marks it as unused.
Also the store is *never* compacted. The id of the deleted nodes are
stored in the corresponding .id file so that they can be reused later
on (after a restart). This of course means that deleting a Node does
not reduce space usage but actually increases it by writing on disk
its id.

RelationshipTypes are not deleted, ever, neither explicitly or when
you delete all Relationships that have that type.

A way to remedy this is to copy over your database when you are done
with your manipulations - do your additions/deletions in a "scratch"
db and then create a new database instance and copy over your graph -
this way no "holes" will be created and the store will be compacted,
effectively defragmenting your database. The downside of this is that
your ids will be not the same any more - if, for example, you deleted
node with id 3 in the initial db, then you will have a node with that
id in the defragged database.

cheers,
CG

On Tue, Jul 26, 2011 at 12:53 PM, Ahmad Bakr  wrote:
> Hi All,
>
> I have a question please and may you want to help me to figure it out, i
> inserted about 240 nodes, 717 properties and 418 relationships with 5
> relationship types. the disk space usage was 77.3 MB (including all
> components of Neo4j), the problem is i have to keep a specific disk space
> for Neo4j thus i need to delete some nodes, the strange thing is when i
> deleted the exisiting nodes and their relationships and properties, the disk
> space usage is increased to 77.4 MB and this is weird, although the admin
> interface of Neo4j shows that all nodes and relationship were deleted
> however the relationship types still 5  :)
>
> Any help ?
>
> Thanks :)
>
>
> --
> Ahmad Bakr
>
> Software Engineer
> eSpace Technologies
> www.espace.com.eg
>
> Mob: (+2) 010-410-2280
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Auto-index fulltext?

[Chris Gioran]

On Mon, Jul 18, 2011 at 1:50 PM, Aseem Kishore  wrote:
> Okay, we look forward to the multiple auto-indexes feature then!
>
>> There will still be one auto index for
>> each primitive category.
>
> Can you explain what a "primitive category" is then

I apologize for the confusion. Primitives (or primitive categories) in
Neo4j are Nodes and Relationships.
Named so because their in-kernel classes, NodeImpl and
RelationshipImpl extend org.neo4j.kernel.impl.core.Primitive

> Aseem
>
> On Mon, Jul 18, 2011 at 6:15 AM, Chris Gioran <
> chris.gio...@neotechnology.com> wrote:
>
>> No, that is not what i meant. The main idea is to provide the means to
>> configure at least some aspects of the auto index instead of relying
>> on the default settings only. There will still be one auto index for
>> each primitive category.
>> However, one feature under consideration is to actually allow an
>> arbitrary number of auto indexes, each of which will allow for
>> individual configuration. So, when that comes along you will have what
>> you described (and more, actually).
>>
>> thanks,
>> CG
>>
>> On Mon, Jul 18, 2011 at 12:29 PM, Aseem Kishore 
>> wrote:
>> > Awesome to hear, Chris, thanks. Just to clarify/confirm then: in the
>> future,
>> > we will be able to have *both* an exact auto-index and a fulltext
>> auto-index
>> > side-by-side?
>> >
>> > Cheers,
>> > Aseem
>> >
>> > On Mon, Jul 18, 2011 at 3:18 AM, Chris Gioran <
>> > chris.gio...@neotechnology.com> wrote:
>> >
>> >> Hi Aseem,
>> >>
>> >> On Sat, Jul 16, 2011 at 9:46 AM, Aseem Kishore > >
>> >> wrote:
>> >> > Is the 1.4 auto-index only "exact"? Or can it be configured to be a
>> >> > "fulltext" index?
>> >>
>> >> Yes, currently the auto-indexes are only exact, there is no
>> >> straightforward way to configure them explicitly. This is a known
>> >> shortcoming and will be remedied pretty soon.
>> >> Another addition that is coming, by the way, is the visibility of
>> >> changes of the auto index within the transaction, instead of waiting
>> >> for commit time as is now.
>> >>
>> >> > (Btw, it would be awesome if we could have two auto-indexes: one
>> exact,
>> >> one
>> >> > full-text. It would be great in general if all indexing could be auto.
>> >> Not
>> >> > sure when you would ever want/need manual indexing.)
>> >>
>> >> Well, a lot of use cases call for manual indexing, when for example it
>> >> is conditional or the use of multiple indexes is required. If,
>> >> however, auto indexing covers all your needs then by all means, do
>> >> just that.
>> >>
>> >> cheers,
>> >> CG
>> >>
>> >> > Aseem
>> >> > ___
>> >> > Neo4j mailing list
>> >> > User@lists.neo4j.org
>> >> > https://lists.neo4j.org/mailman/listinfo/user
>> >> >
>> >> ___
>> >> Neo4j mailing list
>> >> User@lists.neo4j.org
>> >> https://lists.neo4j.org/mailman/listinfo/user
>> >>
>> > ___
>> > Neo4j mailing list
>> > User@lists.neo4j.org
>> > https://lists.neo4j.org/mailman/listinfo/user
>> >
>> ___
>> Neo4j mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Auto-index fulltext?

[Chris Gioran]

No, that is not what i meant. The main idea is to provide the means to
configure at least some aspects of the auto index instead of relying
on the default settings only. There will still be one auto index for
each primitive category.
However, one feature under consideration is to actually allow an
arbitrary number of auto indexes, each of which will allow for
individual configuration. So, when that comes along you will have what
you described (and more, actually).

thanks,
CG

On Mon, Jul 18, 2011 at 12:29 PM, Aseem Kishore  wrote:
> Awesome to hear, Chris, thanks. Just to clarify/confirm then: in the future,
> we will be able to have *both* an exact auto-index and a fulltext auto-index
> side-by-side?
>
> Cheers,
> Aseem
>
> On Mon, Jul 18, 2011 at 3:18 AM, Chris Gioran <
> chris.gio...@neotechnology.com> wrote:
>
>> Hi Aseem,
>>
>> On Sat, Jul 16, 2011 at 9:46 AM, Aseem Kishore 
>> wrote:
>> > Is the 1.4 auto-index only "exact"? Or can it be configured to be a
>> > "fulltext" index?
>>
>> Yes, currently the auto-indexes are only exact, there is no
>> straightforward way to configure them explicitly. This is a known
>> shortcoming and will be remedied pretty soon.
>> Another addition that is coming, by the way, is the visibility of
>> changes of the auto index within the transaction, instead of waiting
>> for commit time as is now.
>>
>> > (Btw, it would be awesome if we could have two auto-indexes: one exact,
>> one
>> > full-text. It would be great in general if all indexing could be auto.
>> Not
>> > sure when you would ever want/need manual indexing.)
>>
>> Well, a lot of use cases call for manual indexing, when for example it
>> is conditional or the use of multiple indexes is required. If,
>> however, auto indexing covers all your needs then by all means, do
>> just that.
>>
>> cheers,
>> CG
>>
>> > Aseem
>> > ___
>> > Neo4j mailing list
>> > User@lists.neo4j.org
>> > https://lists.neo4j.org/mailman/listinfo/user
>> >
>> ___
>> Neo4j mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Auto-indexing feature request: transform

[Chris Gioran]

Hi Aseem,

This is a use case that auto indexing may not be the best fit. The
purpose of auto indexing is taking care of the common use case in the
simplest possible manner, which is a property added is a property
indexed. Indexing a property with a different value that is present on
the primitive (that is what I understood from your description,
correct if wrong) is a case that is best suited for custom indexing
operations. Additionally, it is a desirable quality to have an index
which follows the contents of the actual database with guaranteed
consistency. I would be interested to know if there are others that do
similar transformations and in what manner - maybe there is room for
providing a customizing API.

cheers,
CG

On Sat, Jul 16, 2011 at 9:48 AM, Aseem Kishore  wrote:
> We transform some of our properties currently before (manually) indexing
> them. Simple examples are lowercase ("fulltext" indexes supports a
> "lowercase" config that defaults to true, but "exact" ones don't, I was
> disappointed to find), but more complex ones are e.g. stripping special
> characters.
>
> It would be great if the auto-indexer could support this notion of
> transforming property values (and queries) through user-defined function,
> just like traversal accepts a user-defined filter function.
>
> (And, fwiw, we're users of the REST API, so ideally this functionality would
> be exposed over REST too.)
>
> Cheers,
> Aseem
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Auto-index fulltext?

[Chris Gioran]

Hi Aseem,

On Sat, Jul 16, 2011 at 9:46 AM, Aseem Kishore  wrote:
> Is the 1.4 auto-index only "exact"? Or can it be configured to be a
> "fulltext" index?

Yes, currently the auto-indexes are only exact, there is no
straightforward way to configure them explicitly. This is a known
shortcoming and will be remedied pretty soon.
Another addition that is coming, by the way, is the visibility of
changes of the auto index within the transaction, instead of waiting
for commit time as is now.

> (Btw, it would be awesome if we could have two auto-indexes: one exact, one
> full-text. It would be great in general if all indexing could be auto. Not
> sure when you would ever want/need manual indexing.)

Well, a lot of use cases call for manual indexing, when for example it
is conditional or the use of multiple indexes is required. If,
however, auto indexing covers all your needs then by all means, do
just that.

cheers,
CG

> Aseem
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Neo4j with Tomcat...need a transaction manager

[Chris Gioran]

Hey Jaf,

On Sat, Jul 16, 2011 at 4:23 AM, etc3  wrote:
> Michael,
>
> If I use the default trx manager in neo4j, what connection pool driver
> should be used in Tomcat?

The internal implementation of Neo4j's transaction manager is not
currently exposed for use by 3rd party resources. This means that,
while capable of coordinating a 2PC you cannot use it to do so as a
standalone TM like JOTM. The work we have done is actually the reverse
of that - it substitutes Neo4's TM for an externally provided one,
since both our transactional resources (neo store and lucene indexes)
are XA compatible.

For you this means that you have to use an external TM (like Atomikos
and JBossTM that you mentioned - we have also tested SpringSource's
implementation and JOTM and found all four of them to work) to
coordinate your XA datasources. Doing so for Tomcat however is
something we have not tried yet - the blog posts Michael mentioned
should cover a lot of ground nevertheless, the only thing not covered
is integration with the container, the rest should work.

I will try and replicate your scenario locally and see what trouble
may rise but I am really interested to see how it went for you.

Hope that helped a bit,

CG

> -Original Message-
> From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On
> Behalf Of Michael Hunger
> Sent: Friday, July 15, 2011 9:05 PM
> To: Neo4j user discussions
> Subject: Re: [Neo4j] Neo4j with Tomcat...need a transaction manager
>
> Yes they were added.
>
> The XA-TX-Manager is configurable in Neo4j (defaults to our own but if you
> have an external one like JOTM or Atomikos) then you can configure those to
> be used.
>
> Example implementation for JOTM:
> https://github.com/digitalstain/JOTMServiceProvider
>
> Implementation in neo4j is here:
> https://github.com/neo4j/community/tree/master/kernel/src/main/java/org/neo4
> j/kernel/impl/transaction
>
> Cheers
>
> Michael
>
> Am 16.07.2011 um 02:43 schrieb etc3:
>
>> Thanks, Micheal.
>>
>> The blog post from Chris was from last year; were those changes for
>> JOTM ever added to neo4j?
>>
>> -Original Message-
>> From: user-boun...@lists.neo4j.org
>> [mailto:user-boun...@lists.neo4j.org] On Behalf Of Michael Hunger
>> Sent: Friday, July 15, 2011 8:00 PM
>> To: Neo4j user discussions
>> Subject: Re: [Neo4j] Neo4j with Tomcat...need a transaction manager
>>
>> Springsource also provides/is working on their own tomcat based XA TM.
>> Perhaps you'd like to check that out.
>> (http://forum.springsource.org/showthread.php?76843-JTA-for-Tc-Server&;
>> p=2583
>> 83#post258383)
>>
>> Otherwise there is only JOTM left.
>>
>> Neo4j can integrate with those TM.
>> See also these blog posts by our team member Chris Gioran:
>> http://digitalstain.blogspot.com/2010/11/using-jotm-as-transactionmana
>> ger-in
>> .html
>> http://digitalstain.blogspot.com/2011/02/springy-ouside-graphy-inside.
>> html
>> http://digitalstain.blogspot.com/2010/11/neo4j-internals-transactions-
>> part-3
>> -as.html
>>
>> Internally it uses also 2PC with the nioneo store and lucene.
>>
>> Cheers
>>
>> Michael
>>
>> Am 16.07.2011 um 01:37 schrieb etc3:
>>
>>> We plan on using neo4j as the database for our website, which will
>>> run on Tomcat. Also, we'll be using ActiveMQ for sending messages
>>> within neo4j transactions, so that leaves the question of what to use
>>> for the transaction manager. Atomikos, JBossTM comes to mind, are we
>>> missing
>> anything obvious?
>>> Does neo4j provide anything out-of-box for transactions management?
>>>
>>>
>>>
>>> Thanks
>>>
>>> Jaf
>>>
>>> ___
>>> Neo4j mailing list
>>> User@lists.neo4j.org
>>> https://lists.neo4j.org/mailman/listinfo/user
>>
>> ___
>> Neo4j mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
>> ___
>> Neo4j mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] missing required jta.jar for debian?

[Chris Gioran]

Hi Eddy,

what you are missing is the JTA specification jar. You can find a copy here:

http://repo1.maven.org/maven2/org/apache/geronimo/specs/geronimo-jta_1.1_spec/1.1/geronimo-jta_1.1_spec-1.1.jar

Neo4j uses maven to manage its dependencies so that such necessary
libraries are located and downloaded automatically. This has the
additional advantage that if you have a dependency management system
as well all this will be automatically taken care for you.

To get back to your questions, no, JavaEE is nowhere near needed for
running neo4j. For the kernel you will need, apart from JTA above,

http://repo1.maven.org/maven2/org/osgi/osgi_R4_core/1.0/osgi_R4_core-1.0.jar
http://repo1.maven.org/maven2/org/osgi/osgi_R4_compendium/1.0/osgi_R4_compendium-1.0.jar

as well and you should be good to go. For the tests, junit is also needed.
I don't know of a way to have these dependencies managed from the
package managers that come with linux distros, so either a dependency
manager or manual downloads are your options here.

In general, the pom.xml in every project has a listing of all the
dependencies you will need.

cheers,
CG

On Tue, Jul 12, 2011 at 4:18 PM, Eddy Respondek
 wrote:
> I'm currently getting the following error message in Debian Lenny when
> trying to run a class file.
>
> Exception in thread "main" java.lang.NoClassDefFoundError:
> javax/transaction/TransactionManager
>
> As far as I can tell Debian is missing a jta.jar file which is apparently
> apart Java EE 6 but this isn't one of Debian packages.
>
> Do I need to manually install Java EE 6 or can I get JTA.jar somewhere
> separately?
>
> More info
> javac -cp /usr/share/java/neo4j-kernel.jar:/usr/share/java/jta.jar
> /tmp/gpneo4jimport.java
> java -cp /tmp:/usr/share/java/neo4j-kernel.jar:/usr/share/java/jta.jar
> gpneo4jimport
>
> java version "1.6.0_22"
> Java(TM) SE Runtime Environment (build 1.6.0_22-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Unique Constaint on Index

[Chris Gioran]

Hi,

the ability to acquire locks cluster-wide exists, albeit in an ad hoc
fashion. Grabbing a write lock on the node you want to ensure is
uniquely indexed will ensure that the operations are serialized across
all cluster members.
The most simple way to get that lock currently is the (somewhat
hackish but entirely correct) removal of a non-existing property.

cheers,
CG

On Thu, Jul 7, 2011 at 5:53 PM, etc3  wrote:
> How do I ensure another request is not performing the same operation on
> another node in the cluster?
>
>
> -Original Message-
> From: user-boun...@lists.neo4j.org [mailto:user-boun...@lists.neo4j.org] On
> Behalf Of Marko Rodriguez
> Sent: Thursday, July 07, 2011 10:35 AM
> To: Neo4j user discussions
> Subject: Re: [Neo4j] Unique Constaint on Index
>
> Hi,
>
>> We are testing Neo4J and need to support unique emails across all
>> users. Is this possible with the current API?
>
> You can add such a constraint when updating the indices:
>
> if(index.get('email', address).hasNext()) {
>  throw new RuntimeException("There are two nodes that share the same email
> address."); } else {
>  index.put('email', address, node);
> }
>
> Marko.
>
> http://markorodriguez.com
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Announcing the latest milestone (1.4.M05)

[Chris Gioran]

Hi Aseem,

there is currently no ability to configure the auto indexer from the
REST API. It is definite that this functionality will be added but it
is unclear at this time when this will happen. We would be happy as
well though if this was included in 1.4 GA.

cheers,
CG

On Wed, Jun 29, 2011 at 8:46 AM, Aseem Kishore  wrote:
> Awesome stuff!
>
> One question: is there any ability yet to use (i.e. configure) auto-indexing
> from the REST API? If not, will that be a part of 1.4 final?
>
> Cheers,
> Aseem
>
> On Tue, Jun 28, 2011 at 7:03 AM, Andres Taylor <
> andres.tay...@neotechnology.com> wrote:
>
>> Hello graphistas,
>>
>> Today we’re releasing the fifth and final milestone in our 1.4 “Kiruna
>> Stol”
>> family. We’ve expanded our feature set to include Auto Indexing and paged
>> traversers for the REST API, the Cypher query language has seen some major
>> improvements and as always, we’ve squeezed more performance out of our
>> database engine.
>>
>> Auto indexing
>> --
>> In this milestone we have included an improved version of the auto indexing
>> functionality that was announced shortly after 1.4.M04 and has been
>> available in development snapshots since then. You’ll now find new
>> configuration options for separately enabling auto indexing for node and
>> relationship properties and for defining which properties will be indexed.
>> For example, if you want to auto-index all node properties called “name”
>> and
>> all relationship properties called “since” the proper configuration is:
>>
>> Map config = new HashMap();
>> config.put(Config.NODE_AUTO_INDEXING, “true”);
>> config.put(Config.NODE_KEYS_INDEXABLE, “name”);
>> config.put(Config.RELATIONSHIP_AUTO_INDEXING, “true”);
>> config.put(Config.RELATIONSHIP_KEYS_INDEXABLE, “since”);
>>
>> Passing the above to the GraphDatabaseService constructor will enable auto
>> indexing for matching nodes and relationships, and on every commit the auto
>> index will be updated automatically. For more details, check out the
>> javadocs and the examples in the manual available at:
>>
>> http://docs.neo4j.org/chunked/1.4.M05/auto-indexing.html
>>
>> Paged REST Traversers
>> 
>> After including batch operations in the previous milestone, now we also
>> provide the option to page results of traversals via a special URI
>> contained
>> in Node representations and that is similar to the existing Traverser REST
>> API. You can define the page size and lease time, having this way finer
>> control the retrieval of traverser results from the client (and also help
>> improve performance on the server).
>>
>> Cypher improvements
>> -
>> The newly introduced Cypher query language is growing more sophisticated
>> and
>> in this milestone we’ve added aggregates, ordering and limits. This allows
>> for more expressive queries, making the following a valid statement against
>> the cineasts database:
>>
>> START user=(User,login,'micha')
>> MATCH (user)-[:FRIEND]-(friend)-[r,:RATED]->(movie)
>> RETURN movie.title, AVG(r.stars), COUNT(*)
>>       ORDER BY AVG(r.stars) DESC, COUNT(*) DESC limit 7
>>
>> To see Cypher in action, check out the
>> screencastby
>> our own Michael Hunger.
>>
>> Almost there
>> 
>> Of course that is not all. As always, we have bug fixes, performance
>> improvements and usability enhancements, the most prevalent of which is the
>> removal of YAJSW as the wrapper and service installer, which should bring
>> joy to the mac users among us.
>>
>> With the GA release to follow shortly, we’d like you to download, try out
>> and provide us with your valuable feedback on these new features so that we
>> can deliver the best possible 1.4 to you, our community. For more details
>> and links go to the release announcement
>> here> >
>> .
>>
>> Andrés Taylor
>> ___
>> Neo4j mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Neo4J server - java

[Chris Gioran]

Of course, on the other hand, if the access method per se is the
problem, then Michael Hunger has this to offer:

https://github.com/jexp/neo4j-java-rest-binding

which is a wrapper that exposes the REST server endpoint as a
GraphDatabaseService. Everything works through REST so performance
remains the same but you use Java APIs.

cheers,
CG

On Fri, Jun 17, 2011 at 7:56 PM, Chris Gioran
 wrote:
> Glad to help!
>
> The main overhead associated with the REST interface is the request
> processing. The new batch request functionality available as of the
> latest milestone can reduce significantly that cost, so if performance
> against a remote instance is your main goal and you are not hesitant
> about using milestones, i urge you to try it out wherever you can - it
> will give you a significant boost and we always welcome feedback.
>
> cheers,
> CG
>
> On Fri, Jun 17, 2011 at 7:36 PM, Antriani Stylianou
>  wrote:
>> Hi,
>> That answered my question!
>> I was just aiming for performance and thought that java api would be
>> better than REST.
>>
>> a.
>>
>> On 6/17/11 6:34 PM, Chris Gioran wrote:
>>> Hi Antriani,
>>>
>>> no, besides the two ways you mention, such functionality is not
>>> currently available, with a question mark over the HA way, depending
>>> on your use case.
>>>
>>> I assume what you need is to program against a remotely running
>>> instance of Neo4j through the same Java API, kind of like what is
>>> feasible through JDBC drivers for relational stores.
>>> The brand new Cypher query language is a step in this direction,
>>> providing a way to serialize 'job descriptions" that can lead to a
>>> binary protocol and finally a driver that will provide what you ask
>>> for.
>>>
>>> However, the REST API is currently the proper way to talk to a Neo4j
>>> server, recently improved by batch functionality.
>>>
>>> Does your use case have a particular reason for which the REST
>>> interface is not sufficient?
>>>
>>> cheers,
>>> CG
>>>
>>> On Fri, Jun 17, 2011 at 7:00 PM, Antriani Stylianou
>>>   wrote:
>>>> Hi,
>>>>
>>>> A startup question here!
>>>> Can I connect from java to a neo4j server running the database without
>>>> using the REST API or the Remote Server.?
>>>>
>>>> Something like :
>>>>
>>>> GraphDatabaseService graphDb = new 
>>>> EmbeddedGraphDatabase("localhost:7474/");
>>>>
>>>>
>>>> Thanks,
>>>> A.
>>>>
>>>> ___
>>>> Neo4j mailing list
>>>> User@lists.neo4j.org
>>>> https://lists.neo4j.org/mailman/listinfo/user
>>>>
>>> ___
>>> Neo4j mailing list
>>> User@lists.neo4j.org
>>> https://lists.neo4j.org/mailman/listinfo/user
>>
>> ___
>> Neo4j mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Neo4J server - java

[Chris Gioran]

Glad to help!

The main overhead associated with the REST interface is the request
processing. The new batch request functionality available as of the
latest milestone can reduce significantly that cost, so if performance
against a remote instance is your main goal and you are not hesitant
about using milestones, i urge you to try it out wherever you can - it
will give you a significant boost and we always welcome feedback.

cheers,
CG

On Fri, Jun 17, 2011 at 7:36 PM, Antriani Stylianou
 wrote:
> Hi,
> That answered my question!
> I was just aiming for performance and thought that java api would be
> better than REST.
>
> a.
>
> On 6/17/11 6:34 PM, Chris Gioran wrote:
>> Hi Antriani,
>>
>> no, besides the two ways you mention, such functionality is not
>> currently available, with a question mark over the HA way, depending
>> on your use case.
>>
>> I assume what you need is to program against a remotely running
>> instance of Neo4j through the same Java API, kind of like what is
>> feasible through JDBC drivers for relational stores.
>> The brand new Cypher query language is a step in this direction,
>> providing a way to serialize 'job descriptions" that can lead to a
>> binary protocol and finally a driver that will provide what you ask
>> for.
>>
>> However, the REST API is currently the proper way to talk to a Neo4j
>> server, recently improved by batch functionality.
>>
>> Does your use case have a particular reason for which the REST
>> interface is not sufficient?
>>
>> cheers,
>> CG
>>
>> On Fri, Jun 17, 2011 at 7:00 PM, Antriani Stylianou
>>   wrote:
>>> Hi,
>>>
>>> A startup question here!
>>> Can I connect from java to a neo4j server running the database without
>>> using the REST API or the Remote Server.?
>>>
>>> Something like :
>>>
>>> GraphDatabaseService graphDb = new EmbeddedGraphDatabase("localhost:7474/");
>>>
>>>
>>> Thanks,
>>> A.
>>>
>>> ___
>>> Neo4j mailing list
>>> User@lists.neo4j.org
>>> https://lists.neo4j.org/mailman/listinfo/user
>>>
>> ___
>> Neo4j mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Neo4J server - java

[Chris Gioran]

Hi Antriani,

no, besides the two ways you mention, such functionality is not
currently available, with a question mark over the HA way, depending
on your use case.

I assume what you need is to program against a remotely running
instance of Neo4j through the same Java API, kind of like what is
feasible through JDBC drivers for relational stores.
The brand new Cypher query language is a step in this direction,
providing a way to serialize 'job descriptions" that can lead to a
binary protocol and finally a driver that will provide what you ask
for.

However, the REST API is currently the proper way to talk to a Neo4j
server, recently improved by batch functionality.

Does your use case have a particular reason for which the REST
interface is not sufficient?

cheers,
CG

On Fri, Jun 17, 2011 at 7:00 PM, Antriani Stylianou
 wrote:
> Hi,
>
> A startup question here!
> Can I connect from java to a neo4j server running the database without
> using the REST API or the Remote Server.?
>
> Something like :
>
> GraphDatabaseService graphDb = new EmbeddedGraphDatabase("localhost:7474/");
>
>
> Thanks,
> A.
>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Parallelism

[Chris Gioran]

Hi,

Keep in mind that Neo4j is a fully ACID database. As such, it is safe
to use in the same manner as any other ACID database, since it makes
use of locking over primitives, deadlock detection, isolation between
transactions etc. Having said that:

Yes, the API Neo4j exposes is suitable for manipulation by
concurrently running threads.
Each thread will run in its own transaction - the XA standard that
Neo4j implements enforces a 1-1 correspondence between transaction
context and thread.
Yes, the API is thread safe. Of course, objects that maintain state,
such as Iterators, are still not safe to publish outside the thread.

The advantage of having multiple threads is dependent on your use case.
If you have lots of I/O blocking operations, read-only threads that
work with the memory resident set will progress faster.
Or if you have multiple cores to dedicate, it probably makes sense to
have one thread per such.
Or simply if you want the transaction isolation offered by working in
discreet threads such as when serving client requests.
Or whatever other scenario you might have where multiple threads make sense.

For example, building a web site over Neo4j is a good use case for
having multiple threads, one per request. Creating a bulk data import
tool on the other hand is not a good case to apply parallelism - since
it will be constantly writing to disk, a single thread is fast enough,
adding more does not increase throughput.

The reasoning behind multithreaded access to Neo4j is pretty much the
same as for any other use case. Whatever you choose however is well
supported.

cheers,
CG

On Fri, Jun 17, 2011 at 3:23 PM, Norbert Tausch  wrote:
> Hi,
>
> is it possible to traverse on a Neo4J graph DB using the Java API in
> parallel threads? Is this possible within one transaction or does every
> thread has to use its own transaction? Is the API thread-safe concerning
> read-only access?
> Is there any advantage of concerning parallelism when using Neo4j as an
> embedded DB?
>
> Best regards
>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

[Neo4j] Auto Indexing for Neo4j

[Chris Gioran]

Good news everyone,

A request that's often come up on the mailing list is a mechanism for
automatically indexing properties of nodes and relationships.

As of today's SNAPSHOT, auto-indexing is part of Neo4j which means nodes
and relationships can now be indexed based on convention, requiring
far less effort and code from the developer's point of view.

Getting hold of an automatic index is straightforward:

AutoIndexer nodeAutoIndexer = graphDb.index().getNodeAutoIndexer();
AutoIndex nodeAutoIndex = nodeAutoIndexer.getAutoIndex();

Once you've got an instance of AutoIndex, you can use it as a read-only
Index.

The AutoIndexer interface also supports runtime changes and
enabling/disabling the auto indexing functionality.

To support the new features, there are new Config
options you can pass to the startup configuration map in
EmbeddedGraphDatabase, the most important of which are:

Config.NODE_AUTO_INDEXING (defaults to "false")
Config.RELATIONSHIP_AUTO_INDEXING (defaults to "false")

If set to "true" (independently of each other) these properties will
enable auto indexing functionality and at the successful finish() of
each transaction, all newly added properties on the primitives for which
auto indexing is enabled will be added to a special AutoIndex (and
deleted or changed properties will be updated accordingly too).

There are options for fine grained control to determine
properties are indexed, default behaviors and so forth. For example, by
default all properties are indexed. If you want only properties "name" and
"age" for Nodes and "since" and "until" for Relationships
to be auto indexed, simply set the initial configuration as follows:

Config.NODE_KEYS_INDEXABLE = "name, age";
Config.RELATIONSHIP_KEYS_INDEXABLE="since, until";

For the semantics of the auto-indexing operations, constraints and more
detailed examples, see the documentation available  at

http://docs.neo4j.org/chunked/1.4-SNAPSHOT/auto-indexing.html

We're pretty excited about this feature since we think it'll make your lives
as developers much more productive in a range of use-cases. If you're
comfortable with using SNAPSHOT versions of Neo4j, please try it out
and let us know what you think - we'd really value your feedback.

If you're happier with using packaged milestones then this feature
will be available from 1.4 M05 in a couple of weeks from now.
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Error(s) found in manifest configuration when trying to import community/pom.xml

[Chris Gioran]

Hi there,

the problem is the maven version you are using - Neo4j does not work
well with mvn 3. The exact same steps are successful with my setup
which is the same but for maven 2.2.1
I suggest you downgrade your maven install or create an environment
for Neo4j that uses mvn 2, until we become maven 3 compatible.

cheers,
CG

On Sun, Jun 12, 2011 at 4:55 AM, Super Wang  wrote:
> Hi folks,
>
> I found some warnings and errors when trying to import neo4j maven project
> to my eclipse, here's my steps:
>
> 1. Clone git://github.com/neo4j/community.git to
> local E:\git\neo4j.community
> 2. Go to eclipse and using Import -> Exist Maven Projects and point
> to E:\git\neo4j.community\pom.xml
> 3. Let it import.
> 4. After done, I got two projects with "Error(s) found in manifest
> configuration" errors: neo4j-server and neo4j-sunshine, below is same part
> of  the eclipse console log
>
> 11-6-12 上午09时29分17秒: [WARN]
> 11-6-12 上午09时29分17秒: [WARN] Some problems were encountered while building
> the effective model for org.neo4j:neo4j-sunshine:jar:1.4-SNAPSHOT
> 11-6-12 上午09时29分17秒: [WARN] 'parent.relativePath' points at
> org.neo4j.build:community-build instead of org.neo4j:parent-central, please
> verify your project structure @ line 3, column 11
> 11-6-12 上午09时29分17秒: [WARN]
> 11-6-12 上午09时29分17秒: [WARN] It is highly recommended to fix these problems
> because they threaten the stability of your build.
> 11-6-12 上午09时29分17秒: [WARN]
> 11-6-12 上午09时29分17秒: [WARN] For this reason, future Maven versions might no
> longer support building such malformed projects.
> 11-6-12 上午09时29分17秒: [WARN]
> 11-6-12 上午09时29分17秒: [INFO] Checking licenses...
> 11-6-12 上午09时29分17秒: [INFO] Using 'UTF-8' encoding to copy filtered
> resources.
> 11-6-12 上午09时29分17秒: [INFO] skip non existing resourceDirectory
> E:\git\neo4j.community\sunshine\src\main\resources
> 11-6-12 上午09时29分17秒: [INFO] skip non existing resourceDirectory
> E:\git\neo4j.community\sunshine\src\main\resources\META-INF
> 11-6-12 上午09时29分17秒: [INFO] Copying 0 resource to META-INF
> 11-6-12 上午09时29分17秒: [INFO] Checking for multiple versions of scala
> 11-6-12 上午09时29分17秒: [INFO] includes = [**/*.scala,**/*.java,]
> 11-6-12 上午09时29分17秒: [INFO] excludes = []
> 11-6-12 上午09时29分17秒: [INFO] Nothing to compile - all classes are up to date
> 11-6-12 上午09时29分18秒: [INFO] Nothing to compile - all classes are up to date
> 11-6-12 上午09时29分18秒: [INFO] Checking for multiple versions of scala
> 11-6-12 上午09时29分18秒: [INFO] includes = [**/*.scala,**/*.java,]
> 11-6-12 上午09时29分18秒: [INFO] excludes = []
> 11-6-12 上午09时29分18秒: [INFO] Nothing to compile - all classes are up to date
> 11-6-12 上午09时29分18秒: [ERROR] Error in manifest for
> org.neo4j:neo4j-sunshine:jar:1.4-SNAPSHOT : The default package '.' is not
> permitted by the Import-Package syntax.
>  This can be caused by compile errors in Eclipse because Eclipse creates
> valid class files regardless of compile errors.
> The following package(s) import from the default package
> [org.neo4j.sunshine.javacompat]
> 11-6-12 上午09时29分18秒: [ERROR] Error(s) found in manifest configuration
> 11-6-12 上午09时29分18秒: Build errors for neo4j-sunshine;
> org.apache.maven.lifecycle.LifecycleExecutionException: Failed to execute
> goal org.apache.felix:maven-bundle-plugin:2.1.0:manifest (bundle-manifest)
> on project neo4j-sunshine: Error(s) found in manifest configuration
> 11-6-12 上午09时29分18秒: [ERROR] Error in manifest for
> org.neo4j:neo4j-sunshine:jar:1.4-SNAPSHOT : The default package '.' is not
> permitted by the Import-Package syntax.
>  This can be caused by compile errors in Eclipse because Eclipse creates
> valid class files regardless of compile errors.
> The following package(s) import from the default package
> [org.neo4j.sunshine.javacompat]
> 11-6-12 上午09时29分18秒: [ERROR] Error(s) found in manifest configuration
>
> Any idea about this?
>
> BTW, I'm using eclipse 3.6.2 with maven 3.0.3.
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Problems binding to a specific ip

[Chris Gioran]

Hi Max,

there are two distinct problems here

One is that there is no IP access control that can be configured
through Neo4j. Setting the
listen URIs to a specific address in the configuration does not bind
the server there exclusively. This
behavior has to be imposed with external means, such as a firewall.

The other is a benign bug that has to do with logging the listen
address - currently it uses always the
java.net.InetAddress methods to find the hostname, ignoring the
configuration and leading to the
behavior you describe, since the JVM does not prefer the localhost
interface. This will be of course
addressed in subsequent SNAPSHOTs and milestone releases but it will
remain the default behavior.

Hope that clarifies things.

cheers,
CG

On Wed, Jun 8, 2011 at 2:59 PM, Maximilian Schulz  wrote:
> Hi everyone,
>
> after weeks of experimentation, we finally migrated one of our apps features
> to neo4j. But now that we are about to deploy the app to our staging server,
> we discovered a small problem with the server configuration. We have checked
> the manual at
> http://docs.neo4j.org/chunked/1.4.M03/server-configuration.html and followed
> the suggested configuration, but we are not getting the expected results.
>
> What we are trying to achieve is to bind the server to localhost, so it is
> no longer accessible from outside. We restricted access to IPv4 in the
> wrapper config and set the webadmin data and manage uri to
> http://localhost:7474. But when starting the server it still binds to the
> hostname.
>
> Here is the output of the start process. Check the last line, where it
> states that the server is started on http://OUR_DOMAIN:7474/
>
> 6/8/11 1:44:46 PM org.neo4j.server.database.Database INFO: Using database at
> /tmp/neo4j-community-1.4.M03/data/graph.db
> 6/8/11 1:44:47 PM org.neo4j.server.modules.DiscoveryModule INFO: Mounted
> discovery module at [/]
> 6/8/11 1:44:47 PM org.neo4j.server.modules.RESTApiModule INFO: Mounted REST
> API at [http://localhost:7474/db/data/]
> 6/8/11 1:44:47 PM org.neo4j.server.modules.ManagementApiModule INFO: Mounted
> management API at [http://localhost:7474/db/manage/]
> 6/8/11 1:44:47 PM org.neo4j.server.modules.WebAdminModule INFO: Mounted
> webadmin at [/webadmin]
> 6/8/11 1:44:47 PM org.neo4j.server.NeoServerWithEmbeddedWebServer INFO:
> Starting Neo Server on port [7474]
> 6/8/11 1:44:47 PM org.neo4j.server.web.Jetty6WebServer INFO: Mounting static
> content at [/webadmin] from [webadmin-html]
> 6/8/11 1:44:48 PM org.neo4j.server.NeoServerWithEmbeddedWebServer INFO:
> Server started on [http://OUR_DOMAIN:7474/]
>
>
> I am pretty sure that I have missed something, but I cannot figure out what.
> Any help is greatly appreciated!
>
> Max
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Speeding up initial import of graph

[Chris Gioran]

Hi Daniel,

I am working currently on a tool for importing big data sets into Neo4j graphs.
The main problem in such operations is that the usual index
implementations are just too
slow for retrieving the mapping from keys to created node ids, so a
custom solution is
needed, that is dependent to a varying degree on the distribution of
values of the input set.

While your dataset is smaller than the data sizes i deal with, i would
like to use it as a test case. If you could
provide somehow the actual data or something that emulates them, I
would be grateful.

If you want to see my approach, it is available here

https://github.com/digitalstain/BigDataImport

The core algorithm is an XJoin style two-level-hashing scheme with
adaptable eviction strategies but it is not production ready yet,
mainly from an API perspective.

You can contact me directly for any details regarding this issue.

cheers,
CG

On Thu, Jun 9, 2011 at 12:59 PM, Daniel Hepper  wrote:
> Hi all,
>
> I'm struggling with importing a graph with about 10m nodes and 20m
> relationships, with nodes having 0 to 10 relationships. Creating the
> nodes takes about 10 minutes, but creating the relationships is slower
> by several orders of magnitude. I'm using a 2.4 GHz i7 MacBookPro with
> 4GB RAM and conventional HDD.
>
> The graph is stored as adjacency list in a text file where each line
> has this form:
>
> Foo|Bar|Baz
> (Node Foo has relations to Bar and Baz)
>
> My current approach is to iterate over the whole file twice. In the
> first run, I create a node with the property "name" for the first
> entry in the line (Foo in this case) and add it to an index.
> In the second run, I get the start node and the end nodes from the
> index by name and create the relationships.
>
> My code can be found here: http://pastie.org/2041801
>
> With my approach, the best I can achieve is 100 created relationships
> per second.
> I experimented with mapped memory settings, but without much effect.
> Is this the speed I can expect?
> Any advice on how to speed up this process?
>
> Best regards,
> Daniel Hepper
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] neo4j jee connector

[Chris Gioran]

On Fri, May 20, 2011 at 8:14 AM, Alexander Smirnov  wrote:
> Seems what I did miss it in preliminary research :-)
> For my application ( prototype for something that I hope to convert
> into startup company ), I need exact the same thing that you
> demonstrate in your blog, add graph data to the JPA entities. I need
> to join SQL and neo4j transactions together, and it's possible with
> JCA because server can manage XA-compatible resources together.

If 2PC with Neo4j and JDBC is what you need, then maybe go with an
external transaction manager,
a service implementation for it for Neo4j and a connection pooler for
your JDBC driver of choice. The
JCA overhead is worth the trouble mainly if you need the J2EE server
environment. But purely
from a functional perspective they are the equivalent.

> Another case was to decouple application and database instance. With
> connector they only have one single interface in common.
> My plan was to use single in-memory instance for development and
> initial launch. In the future, It can be converted into HA neo4j
> instances running in the cluster server nodes, and, when Neo4j server
> got binary remote connection ( Emil Eifrem told me at East Bay meetup
> what Neo4j working on it ) convert it into set of AS nodes connected
> to the one or more Neo4j servers. Pretty easy for scale, isn't it?

The binary protocol will not cater specifically for this need, but
will make remote
connections to the server efficient. This, in turn will open the way
for a remote
driver that can handle transactions and make embedding into an AS
feasible and eventually the scenario you described above a reality.

> Your solutions seems more robust when mine, where I use simple hack,
> similar to the one used by Spring Data Graph:
> get system transaction manager and enlist Neo4j XAResources in the
> current system transaction.
> Also, I'm confused how you can manage all Neo4j XAResources together ?
> As I understood, there is at least one more for Lucene indexes.

This is actually a problem - by the spec, one ResourceAdapter is responsible
for one and only one XAConnection. This means that you can either create two
adapters, one for nioneo connections (the graph database store) and
one for the index
or cheat and use the same ResourceAdapter doing internally some
shuffling with the XAResources.
In my case I opted for the third option: ignore index operations,
mainly because I was planning on supporting
the more correct first approach but eventually never did.

cheers,
CG

> On Thu, May 19, 2011 at 1:02 AM, Chris Gioran
>  wrote:
>>
>> Hi Alexander,
>>
>> It is encouraging that you came up with a solution similar JCA module
>> that I wrote some time ago:
>>
>> https://svn.neo4j.org/laboratory/users/cgioran/NeoJCA/
>>
>> with a description here:
>>
>> http://digitalstain.blogspot.com/2010/12/towards-neo4j-connector-implementation.html
>>
>> (probably outdated now, of course)
>>
>> While a very interesting topic, it is currently little more than spec
>> gymnastics since the only way this can work is in embedded mode,
>> something that is not very flexible when running in an app server. It
>> will show its usefulness however when Neo4j exposes remote XAResources
>> making participation of the server into 2PC a reality. Other than
>> that, really impressive work! I hope others will use this to start
>> getting feedback on what we want from an J2EE embeddable Neo4j
>> database.
>>
>> cheers,
>> CG
>>
>> On Thu, May 19, 2011 at 9:23 AM, Alexander Smirnov  
>> wrote:
>> > I've created java connector for Neo4j database, that can be used in any 
>> > Java
>> > EE 6 compatible server ( currently, it has been tested on Glassfish 3.1, 
>> > any
>> > feedback about other servers welcome ). The project hosted on github
>> > https://github.com/alexsmirnov/neo4j-connector , and there is a short
>> > description:
>> > http://alexsmirnov.wordpress.com/2011/05/18/neo4j-java-ee-connector/
>> > It supports both Local and XA transactions, so you can use it with data
>> > divided between SQL and Nao4j stores. Probably, it would also correctly
>> > support nested transactions.
>> >
>> > Future plans:
>> > HA version support. I hope to start both zookeeper and HA database in the
>> > same adapter. In the perfect case, it should use cluster configuration from
>> > server.
>> > Remote server support.
>> >
>> > Enjoy
>> > --
>> > _
>> > entia non sunt multiplicanda praeter necessitatem,
>> > (entities should not be multiplied 

Re: [Neo4j] neo4j jee connector

[Chris Gioran]

Hi Alexander,

It is encouraging that you came up with a solution similar JCA module
that I wrote some time ago:

https://svn.neo4j.org/laboratory/users/cgioran/NeoJCA/

with a description here:

http://digitalstain.blogspot.com/2010/12/towards-neo4j-connector-implementation.html

(probably outdated now, of course)

While a very interesting topic, it is currently little more than spec
gymnastics since the only way this can work is in embedded mode,
something that is not very flexible when running in an app server. It
will show its usefulness however when Neo4j exposes remote XAResources
making participation of the server into 2PC a reality. Other than
that, really impressive work! I hope others will use this to start
getting feedback on what we want from an J2EE embeddable Neo4j
database.

cheers,
CG

On Thu, May 19, 2011 at 9:23 AM, Alexander Smirnov  wrote:
> I've created java connector for Neo4j database, that can be used in any Java
> EE 6 compatible server ( currently, it has been tested on Glassfish 3.1, any
> feedback about other servers welcome ). The project hosted on github
> https://github.com/alexsmirnov/neo4j-connector , and there is a short
> description:
> http://alexsmirnov.wordpress.com/2011/05/18/neo4j-java-ee-connector/
> It supports both Local and XA transactions, so you can use it with data
> divided between SQL and Nao4j stores. Probably, it would also correctly
> support nested transactions.
>
> Future plans:
> HA version support. I hope to start both zookeeper and HA database in the
> same adapter. In the perfect case, it should use cluster configuration from
> server.
> Remote server support.
>
> Enjoy
> --
> _
> entia non sunt multiplicanda praeter necessitatem,
> (entities should not be multiplied beyond necessity.)
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

[Neo4j] Neo4j 1.4 M01 - headed to Kiruna Stol

[Chris Gioran]

Hi fellow graphistas.

Keeping up with our schedule, two weeks after our 1.3 GA release we
present to you the first milestone towards Neo4j 1.4 “Kiruna Stol”.

This iteration does not introduce any major features. Instead it takes
care of some annoyances in the existing codebase, incorporating your
feedback to iron out issues with ops and user interaction. So if you
are comfortable with 1.3, you should be right at home with this
milestone, only happier!

This release also paves the way for the cool new features we are
considering for Kiruna Stol. Many improvements are in our sights,
including paging and streaming of REST results, investigation of a
query language, revisiting the API and introduce self referencing
relationships and more. This won’t stop usability and performance
improvements from coming though.

So, go ahead and download our latest release and read more about it at

http://blog.neo4j.org/2011/04/neo4j-14-m01-announcing-kiruna-stol.html

and keep those suggestions coming.
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Regarding Sub grouping In Graph DB

[Chris Gioran]

If I understand you correctly, you want to create groups of your nodes
+ relationships that exist in your graph. The way you do that depends
on some things.

1. What is the degree of logical separation you desire? If two nodes,
A and B have a relationship in common, will they always end up in the
same category? If not, should the relationship remain?*
2. How do you plan to access those nodes? If you want to find out the
category of nodes and relationships as you iterate over your data,
then maybe a tagging property would work. If you want direct access to
the nodes that belong to a category, an option is a "supernode" which
represents that category and have it relate to every node that belongs
to it.

Creating a tagging property has the advantage of being indexable and
works both on nodes and relationships. A "supernode" (the quotes are
justified since we are talking about a simple node that just receives
special semantics, not some inherently different node kind) is a more
graphy way of doing things and probably faster too.

I am sure there are other ways and I would also like to see how people
categorize entities in their graphs.

*What currently cannot be done is role-based "hard" separation of
nodes. Suppose there are two nodes, a and b with a relation a-->b. Say
a ends up in category A and b in category B and the relationship
stays. If someone with access to A accesses a, then she will be able
to access b, no matter what.

cheers,
CG

On Mon, Apr 25, 2011 at 9:53 PM, pooja naik  wrote:
> Hi Chris,
> Thanks for a prompt response.
> I am trying to create a Network Graph Infrastructure for SP 's , in such a
> way that they can lease out the parts of this network infrastructure to
> other small companies.
> This requirement make it necessary to create a logical boundary on the main
> Network graph , in other words to create a overlay graphs on the underlying
> physical graph.
> Does "categories" would still help me here?
> Is there any another way to do it?
> Let me know
> Thanks n Regards
> Pooja
> 
> From: Chris Gioran 
> To: Neo4j user discussions 
> Sent: Monday, April 25, 2011 11:34 AM
> Subject: Re: [Neo4j] Regarding Sub grouping In Graph DB
>
> Hi Pooja,
>
> what would qualify as a subcategory in your use case?
>
> The most straightforward thing I can come up with is a property in
> each node named "category" or something similar, that has an
> enumerated value (not a Java Enumeration but something treated as
> such) that would place the node in its category. This is one of many
> ways to go ahead, depending on what degrees of separation you want,
> flexibility etc.
> Could you elaborate a little on what you are trying to do?
>
> cheers,
> CG
>
> On Mon, Apr 25, 2011 at 9:27 PM, pooja naik  wrote:
>> Hi all,
>>
>>
>> I am using ne04j for a IP network resource graph in my project.
>> I would like to know whether there is a way to divide the graphical
>> network into sub categories in neo4j?
>>
>> Any help or pointers is appreciated.
>>
>> Thanks
>> Pooja
>> ___
>> Neo4j mailing list
>> User@lists.neo4j.org
>> https://lists.neo4j.org/mailman/listinfo/user
>>
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
>
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Regarding Sub grouping In Graph DB

[Chris Gioran]

Hi Pooja,

what would qualify as a subcategory in your use case?

The most straightforward thing I can come up with is a property in
each node named "category" or something similar, that has an
enumerated value (not a Java Enumeration but something treated as
such) that would place the node in its category. This is one of many
ways to go ahead, depending on what degrees of separation you want,
flexibility etc.
Could you elaborate a little on what you are trying to do?

cheers,
CG

On Mon, Apr 25, 2011 at 9:27 PM, pooja naik  wrote:
> Hi all,
>
>
> I am using ne04j for a IP network resource graph in my project.
> I would like to know whether there is a way to divide the graphical network 
> into sub categories in neo4j?
>
> Any help or pointers is appreciated.
>
> Thanks
> Pooja
> ___
> Neo4j mailing list
> User@lists.neo4j.org
> https://lists.neo4j.org/mailman/listinfo/user
>
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Starting neo4j Server doesn't return to promt

[Chris Gioran]

Hi Stephan,

please see inline

On Wed, Apr 20, 2011 at 12:58 AM, Stephan Hagemann
 wrote:
> Hi Mattias, hi List,
>
> I am running this on Linux... and I solved the problem (with a workaround).
>
> What confused me the whole time was that there were three files for the same
> purpose in the conf directory: log4j.properties, logback.xml,
> logging.properties (the first two aren't used - I am using 1.3 advanced -
> could they go away?). To make matters worse, even neo4j-wrapper.conf has
> some logging related settings. And then there are 'settings' in the neo4j
> executable about whether the startup should wait for success messages from
> the server.

The 3 files that pertain to the logging frameworks have not been
removed since there are lingering libraries that could use them -
log4j is a hard requirement for zookeeper and some dependencies of the
wrapper, for instance. This doesn't mean that they are necessary, it
means we are in the process of fixing this.

The neo4j-wrapper.conf does indeed contain a logging settings section,
since it prefers to do the logging setup on its own.
The reason things seem complicated is that YAJSW pipes stdout/stderr
of the wrapped process into itself and everything that is received
from there is printed to the console. This means that the actual
control of what is logged from the neo4j server (at the latest
snapshots) is the logging.properties file in the usual j.u.l manner -
the neo4j-wrapper.conf logging settings control the logging for the
wrapper *only*, with the exception of the log statement format.

> I am pretty sure this line from logging.properties wasn't there in 1.3.M5,
> because I didn't have a whole lot of console logging before:
>  handlers=java.util.logging.FileHandler, java.util.logging.ConsoleHandler

This is because we want, by default, some interesting messages, such
as the server URI and the webadmin location to be printed to the
console during startup. Essentially, we have set the console handler
to print statements of INFO level or greater. Removing the Console
handler will completely stop the logging from neo4j-server, but the
wrapper will continue outputting its statements.

> When I switched it to
>  handlers=java.util.logging.FileHandler
> most of the logging ouput in the console went away, but the process for some
> reason still held on to the console. E.g., when I do a 'neo4j dump' the
> result is shown in the original console. I have not yet been able to find
> out, where I can make the server fully detach.
>
> The workaround for me is to explicitly pipe the neo4j command's output to a
> file.

This is actually a common behavioral pattern in all *nix applications
- holding on to the pts it was started at. It can be annoying though,
so at some point it may have to be fixed.
Finally, the same applies to "not getting back to the prompt". The
shell script returns as soon as the neo4j-server process is forked,
but there is still some log statements to be written. This clutters up
the screen, as output from any background process does, making it seem
like the prompt isn't there. If you just press enter, you will get it
back. As a workaround, you can increase the WAIT_AFTER_STARTUP
variable in the neo4j script (measured in seconds, set to 5 at the
latest SNAPSHOT) making sure this way that there remains no more
output that could clutter up your console.

> Cheers
> Stephan

Hope the above clarify things. Again, check out the latest builds,
most of this stuff is fixed/made sane.

cheers,
CG

>
> PS: here is my logging.properties:
>
> # Global logging properties.
> # --
> # The set of handlers to be loaded upon startup.
> # Comma-separated list of class names.
> # (? LogManager docs say no comma here, but JDK example has comma.)
> handlers=java.util.logging.FileHandler
>
> # Default global logging level.
> # Loggers and Handlers may override this level
> .level=INFO
>
> # Loggers
> # --
> # Loggers are usually attached to packages.
> # Here, the level for each package is specified.
> # The global level is used by default, so levels
> # specified here simply act as an override.
> org.neo4j.server.level=INFO
>
> # Handlers
> # -
>
> # --- ConsoleHandler ---
> # Override of global logging level
> java.util.logging.ConsoleHandler.level=INFO
> java.util.logging.ConsoleHandler.formatter=java.util.logging.SimpleFormatter
> java.util.logging.ConsoleHandler.filter=org.neo4j.server.logging.NeoLogFilter
>
> # --- FileHandler ---
> # Override of global logging level
> java.util.logging.FileHandler.level=ALL
>
> # Naming style for the output file:
> # (The output file is placed in the directory
> # defined by the "user.home" System property.)
> java.util.logging.FileHandler.pattern=data/log/neo4j.%u.log
>
> # Limiting size of output file in bytes:
> java.util.logging.FileHandler.limit=5
>
> # Number of output files to cycle through, by appendin

Re: [Neo4j] Node safe in a Set

[Chris Gioran]

On Tue, Mar 15, 2011 at 5:17 PM, Massimo Lusetti  wrote:
> It's safe to use Node in a Set... I don't see it implements
> equals() nor hashCode()

Of course it does! Node is the interface implemented by NodeProxy -
the objects you get back and use as Nodes. NodeProxy implement
equals() and hashCode() based on the id of the Node. Using them in a
Collection is safe.

> Cheers
> --
> Massimo
> http://meridio.blogspot.com

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

[Neo4j] Announcing Neo4j 1.3 “Abisko Lampa” M04

[Chris Gioran]

Greeting fellow graphistas,

Keeping up with the release schedule, today we present the fourth
milestone towards the 1.3 “Abisko Lampa” version of Neo4j. The most
important feature of this release is of course the expansion of the
storage layer to allow for even more entities, but also includes
improvements on the testing methodology used for Windows builds and
the REST interface for Indexing operations. As always, extended
details are available in the announcement post, reachable at

http://blog.neo4j.org/2011/03/neo4j-13-abisko-lampa-m04-size-really.html

There you can read a comprehensive description of the new features,
however a special mention must be made to the new store format.

This milestone of Neo4j brings with it an updated store version that
allows for an 8x increase in the possible number of Nodes and
Relationships and 16x increase for the number of Properties. All this
happens without increasing the size of the database so you can go
right ahead and upgrade, certain in the knowledge that everything will
be done in place. A few words of caution are in order, though.

An upgrade will not happen unless you explicitly request it - just
pass in the configuration parameter “allow_store_upgrade” as “true” -
otherwise an exception will be thrown that advises you to do just that
and your store will be left untouched. There are some additional
requirements for the upgrade to succeed, such as less than 65536
RelationshipTypes stored, but if you do not satisfy those the upgrade
procedure will inform you of what the exact problem is and leave your
store untouched.
Finally, note that after a successful upgrade your database will be
unreadable by previous versions. So keep in mind that while safe, this
is a one way procedure.
You can find more details on the subject at

http://docs.neo4j.org/chunked/milestone/deployment-upgrading.html

We invite you to try out these new features and provide us with your
ever valuable feedback.

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] IllegalStoreVersionException - Store version is older than expected...

[Chris Gioran]

On Wed, Mar 9, 2011 at 5:26 PM, José Devezas  wrote:
> Hello,
>
> I'm using Blueprints 0.5 to read a Neo4j graph. I've just come back to the
> project today, maven updated to the latest JARs and now I get this error.
> I've tried searching for documentation on how to set this option, but I
> couldn't find it yet. How do I upgrade the NeoStore to the latest version?
>
> Caused by: org.neo4j.kernel.impl.nioneo.store.IllegalStoreVersionException:
> Store version [NeoStore v0.9.6] is older than expected, but could be
> upgraded automatically if 'allow_store_upgrade' configuration parameter was
> set to 'true'.

The changes, as Peter notes, are because of the upgrade of the store
format to accommodate
an even larger number of entities in a Neo4j database. This of course
requires an upgrade
procedure.

To start the latest SNAPSHOT in upgrade mode, you should pass in the
configuration parameter
"allow_store_upgrade" as "true", as the exception hints. A sample way
to do this is

Map properties = new HashMap();
properties.put(Config.ALLOW_STORE_UPGRADE, "true");

new EmbeddedGraphDatabase("db", properties);

After upgrading the store (meaning successful startup with the above
or equivalent code)
then you *cannot* go back to previous versions - including 1.3M03.
More details on this
procedure are at

http://docs.neo4j.org/chunked/snapshot/deployment-upgrading.html

Note however that your store will be touched only if you satisfy all
the requirements the
documentation outlines AND you explicitly ask for an upgrade, making
the procedure safe
for your data. Reversely, to use Neo4j version 1.3 you MUST upgrade to
the new format.

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

[Neo4j] Neo4j, Spring Framework and 2PC

[Chris Gioran]

Hello graphistas of the world,

trying to create a more seamless coupling between Neo4j and Spring
Framework, we have now built a solution for getting these two to work
with an external transaction manager. This work is based on some
groundwork for pluggable TMs in the Neo4j kernel and is a step in the
direction of bringing support for graph operations from a managed
environment in conjunction with other XA compliant data sources, the
most prominent example being JDBC connections. So, start enjoying the
magic through this blog post:

http://digitalstain.blogspot.com/2011/02/springy-ouside-graphy-inside.html

For the impatient, the code is available at

https://github.com/digitalstain/Neo4j-Spring-Integration

As always, feedback and suggestions are welcome and greatly appreciated.

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Does the index commit with the transaction?

[Chris Gioran]

On 02/13/2011 07:18 PM, Massimo Lusetti wrote:
> Hi all,
>Does the inner Lucene index commit toghether with the Transaction
> success()/finish() cycle?
>
> I mean if I start a Transaction and do like
> http://wiki.neo4j.org/content/Transactions#Big_transactions I suppose
> I'm guarantee the Lucene index is synched to disk as soon as the
> Transaction finish(), right?

Hi Massimo,

your understanding is correct. The lucene index is exposed as a 
XAResource that participates in the global transaction that is started 
with graphDb.beginTx() and committed on successful tx.finish(). The 
commit (or rollback) happens through a 2PC which ensures that it is 
treated as an atomic operation.

The info on Big Transactions is essentially just a workaround for 
avoiding running out of memory, so the principle as I outlined it above 
still holds.

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Fatty Stack Trace to Lift your Spirits

[Chris Gioran]

On Wed, Dec 22, 2010 at 6:11 PM, Marko Rodriguez  wrote:

> So I bumped Blueprints 0 .4-SNAPSHOT to use 1.2M06 and during the test suite 
> evaluation, I was getting fatty stack traces being thrown all up in my face 
> left, right, and yes, center.

As long as they are not coming from below, there is no reason for
alarm. However...

> My random thought on the matter---Luca and I have been having *potentially* 
> similar problems with OrientDB and shutting down, opening, and deleting the 
> DB really fast. Don't > know if this is the same issue now showing up in 
> Neo4j? The TestSuite will, in general, create a graph (directory), use it, 
> close it, open it, delete the directory, create a > new graph (directory), 
> etc. Is the OS not having happy good time? Why happy good time with 1.2M05?

I slapped up a loop doing pretty much that - create a db, store+index
a few things, close, [open, read, close] (in a loop), delete, do over,
for around 50 iterations. Using straight up (no tinkerpop stack)
vanilla 1.2M06 on Ubuntu 10.10. No problems so far. I will leave it to
run for a while, to teach my HDD some manners and I will check it
again later. That lock file left behind, however, smells like
crash/improper shutdown (or overlapping instances?). That really fast
open/shutdown cycle you speak of, how do you do it, shell script or
from main()? Does the problem appear in every cycle or intermittently?

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] How to delete a node when it's already deleted ?

[Chris Gioran]

On Tue, Dec 7, 2010 at 3:09 PM, Andreas Ronge  wrote:
> An Node#isDeleted() method would also be fine.

The way I see it, there are two concerns here.
The first is focused at the lower levels, where the
WriteTransaction/LockReleaser discover an illegal operation - deletion
of an already deleted primitive. This is a hard error and at their
level it should throw an exception and of course set the tx to
rollback only. This is mainly an engineering decision.
The second is the user level where either the same logic should apply
or a check should be made first to make sure things don't go downhill.
Obviously the current approach is the former.
Having just a isDeleted() method is kind of awkward because it would
litter the code with if statements and things would be even worse with
(checked) exceptions. Maybe stealing a bit off the id would be a
better solution and have the NodeImpl/NodeProxy objects do the check
internally. BTW, I think that from a user perspective with the current
kernel such a wrapper object (with a boolean field possibly) would be
the best approach, minimizing the bookkeeping in "business logic"
code.

What I find more interesting to discuss are the semantics of
operations on primitives. At the moment there is no standard to adhere
to and in that respect there is a decision to be made. What I mean is:
what is the proper thing to do, conceptually, when doing basic
primitive manipulations. Since there is an effort to standardize a
graph traversal algebra, a similar thing should be done on a data
definition level, with rationalization and detailed description of
what is the Right Thing (TM) to do when, for instance, one deletes a
Node from a graph, regardless of implementation. Obviously my thinking
is influenced from the relational model, where there are hard
constraints on different things - primary keys are an obvious example
here. In that case, the proper thing to do was to make it propagate a
hard error all the way up and all implementations do exactly that. In
this way, behavior is standardized for all common operations. Should
graph databases, beginning with Neo, undergo a similar process? Such
an effort would give definite answers to most such problems, for
example the "forced"/cascading deletion issue mentioned before.

On the other hand, maybe I am overthinking this.

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Cats and Dogs, living together

[Chris Gioran]

> I will now go and write a post explaining in more detail how to build,
> install and use the above.

And it is available at

http://digitalstain.blogspot.com/2010/12/towards-neo4j-connector-implementation.html

be gentle.

thank you for your time,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Cats and Dogs, living together

[Chris Gioran]

> Would anybody be willing to share experiences with trying to introduce Neo4j 
> into a system with another relational (or other NoSQL) database?

Let me contribute my 2 cents here.

I have developed a rough demo that shows what my thoughts are for
integration of Neo4j instances in a managed environment. Using this
JCA 1.6 compliant connector module I have successfully

- Injected a ConnectionFactory (akin to a DataSource from JDBC) that
returns NeoConnections (similar to javax.sql.Connection).
- Performed operations on it (node creations/property settings) in
tandem with operations on a mySQL instance. That is, in the same EJB
method, with a container managed transaction. Both resources were
enlisted in the top level tx and were successfully committed or
rolledback (when I violated constraints in either db, for instance),
in proper 2PC manner.

So far, based on my experience working with JDBC, the feeling is the
same, as far as usability goes. The GraphDatabaseService interface is
not yet completely available.

The code is available at the repository:

The transaction manager to use (based on previous work):
https://svn.neo4j.org/laboratory/users/cgioran/ContainerProvidedTxManager/

The connector module:
https://svn.neo4j.org/laboratory/users/cgioran/NeoJCA/

The modified kernel (nothing major, just exposed retrieval of
XAResources and enforced the new txm)
https://svn.neo4j.org/laboratory/users/cgioran/neo4j-kernel-jta/

The idea is that you build/install all three above, add the jta kernel
and the txm service in your ear's libraries and the connector module
as an ear connector module (in application.xml). Create a pool and
bind it to the JNDI and retrieve it from there in your EJB/Servlet.

DISCLAIMER: The above code is a toy and completely temporary. I have
not verified its operation in any significant way, I do not even know
if (in fact I doubt that) the pooling mechanism works properly. Do not
use it in any way that might be remotely important.

The connector will work only in glassfish. Other app servers (JonAS
and Geronimo for certain) require additional descriptor files apart
from ra.xml If you want to deploy in another container, please consult
its manual on the proper procedure.

I will now go and write a post explaining in more detail how to build,
install and use the above. If, in the meantime, someone feels
adventurous and decides to look at the code, feedback is always
welcome. The important thing for me is not the correctness of the code
- there is not much there anyway. What I am interested in is its
dynamics and the possible interactions in an app server.

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] JTA support for Neo

[Chris Gioran]

Ok, I have written a small post describing how this thing comes
together. It is available at

http://digitalstain.blogspot.com/2010/11/using-jotm-as-transactionmanager-in.html

I hope it is clear enough. Questions asked here will end up as
corrections to this.
However, let me point out that here:

https://svn.neo4j.org/laboratory/users/cgioran/recovery-robustness/

is a branched project that contains everything that is needed to run
the JOTM enabled Neo, provided you have installed the jta kernel and
the jotm service implementation. All configuration and dependencies
are taken care of (the JOTM configuration is in the resources/
directory and the JVM parameters in the run-one script).

Thank you for your time.

CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] JTA support for Neo

[Chris Gioran]

On Mon, Nov 22, 2010 at 4:10 PM, Johan Svensson  wrote:
> This looks great, very good work!
>
> I would like to get this merged into trunk after we releases 1.2 (not
> after this iteration but after the next one).
>
> The changes looks minimal to me and hopefully there are no problems
> going forward with the current design. Looking forward to the guide so
> me and others can try this out.
>
> Would it be possible to continue investigate this and see how these
> changes would work in a Spring environment? The goal would be to run
> on an external TM, in Spring (using annotated transactions) and have
> both Neo4j and MySQL (or some other resource) participate in the same
> transaction.
>
> Regards,
> Johan

Currently I am working on a branch of the recovery robustness suite
that runs the jta kernel with a jotm instance. Works like a charm,
after some more successful cycles I will commit it at my laboratory
space, probably within the hour. This will contain a completely setup
environment for running an example and the guide comes next.

If this test is considered enough, I could delay integrating my
JDI-based test suite and continue work on the JCA module. Personally,
I feel confident with the robustness suite, so if there are no
objections until I am done with the guide, I will begin with the
connector.

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] JTA support for Neo

[Chris Gioran]

> IMHO you should start a branch
> in the SVN so others can look at the code.

So, at

https://svn.neo4j.org/laboratory/users/cgioran/neo4j-kernel-jta/

you can find the kernel component with my changes incorporated. The
classes added are

org.neo4j.kernel.impl.transaction.TransactionManagerImpl
org.neo4j.kernel.impl.transaction.TransactionManagerService

that are the hooks for providing custom transaction managers.
The first is an extension of javax.transaction.TransactionManager
adding support for
startup and shutdown, an operation present in all tx managers but not
part of their API. This
provides the ability to plugin custom implementations in the TxModule.
The second is a convenience class that is extended by tx managers that
are to be provided as a service.

Also, changes are present in

org.neo4j.kernel.impl.transaction.TxModule

for using this new way of doing things,

org.neo4j.kernel.impl.transaction.TxManager
org.neo4j.kernel.impl.transaction.ReadOnlyTxManager

for them to fit in this and

org.neo4j.kernel.EmbeddedGraphDbImpl
org.neo4j.kernel.Config

to bind them.

This fork is (or should be) completely compatible with the official
kernel, so it can be used as a drop in replacement. Any deviation is a
bug and if reported it will be fixed.

The second project is at

https://svn.neo4j.org/laboratory/users/cgioran/JOTMService/

and is a sample implementation of a tx manager service for JOTM. To
use this, build it, add the resulting jar to your classpath and, if
you are using the new jta fork of the kernel, you can pass a
configuration parameter of "tx_manager_impl"="jotm" to your
EmbeddedGraphDatabase and presto!, if all is well you will be using a
JOTM TxManager to do your thing.
Of course, the jotm libraries must be also in your classpath, version 2.1.9

If this way of doing things is met with approval, I will write a
complete guide to using the above, implementation and design details
and as a result a how to for adding more external tx managers.

There is more to come.

CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] JTA support for Neo

[Chris Gioran]

> Chris,
> This sounds like some major progress!

Meh. It was some work for me, mainly making the vm crash at will and getting
JOTM to enlist/recover the resource managers. Things worked out of the box from
the side of neo.

> IMHO you should start a branch
> in the SVN so others can look at the code.

That would be the ideal way, yes. Would you set up a laboratory space for me?

> Having the compatibility
> tests is great - there are other breaking test that Mattias and Johan
> are using to purposely break recovery on both the normal server and HA
> setups. They are run on a number of amazon instances and it would be
> great if your tests could be integrated and run there, too, so we can
> find failing recovery scenarios.

I assume you are referring to qa/recovery-robustness. I haven't studied its
mechanics yet but I could put it there, I suppose. From a cursory look
I understood that it runs from a script so my tests could fit there.

> any suggestions on making the use of external TX managers easier or
> document the setup process somewhere?

For my part I could post an article on my blag(TM) on how I have setup JOTM
as a Neo service and how to enable it. I would prefer to do that however after
the code has been seen from eyeballs other than mine and actually certified not
to be a big, fat, slobbering mess. This way the API will be
standardized and people
will have something concrete to read about.

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] JTA support for Neo

[Chris Gioran]

> If you can get the above test case working using other TMs together
> with Neo4j would be great!

OK, it took some time, since I messed with some things for the first time.
I have created the following:
- I have added two classes in the kernel module that provide support
for registration
of TransactionManager implementations as services. Like the lucene
indexer, if the jar
is in the classpath it will be picked up and made available. The configuration
passed to the GraphDatabase must define the name of the TxManager to
use, defaulting
of course to the native one. These changes required some modifications
in TxManager, ReadOnlyTxManager,
TxModule, EmbeddedGraphDbImpl and Config.
- A sample implementation of the above for providing a JOTM instance as a
TransactionManager.
- A test "platform" that, through JDI crashes the system at specific points,
up to a resolution of line of code. This way at any point in the transaction
the system can be brought down in flames. This aspect is still a
little awkward to
integrate since a new JVM process must be started and terminated. At the time
being I use it as a standalone test (launched at the console).

Using the above, after crashing a neo instance that uses a lucene
XaResource and (of course)
a Neo XaResource at any point during commit or rollback, using JOTM
2.1.9, restating the system brings it in
a consistent state, committing all pending resources or rolling back,
whatever is needed.
There are many test cases still to be implemented to
make sure that the setup is production ready but what I have tested so
far passes. This includes
of course the stock tests for the kernel component, ensuring (to
whatever possible extent) that my
changes do not mess up with normal operations.

>> - Find the least intrusive way of making neo fit in the picture, in
>> terms of configuration/code changes etc, approve and commit those.

See above.

What I would like as feedback is ways the community would integrate
these capabilities in their
environment. This way I can provide integration methods that will make
the procedure less risky
and more transparent, as well as develop test cases that do not at the
moment occur to me.

As this feature requires changes in the neo kernel, I am still
searching for the best way to make
the code available. I will inform you when that changes, hopefully soon.
Also, if there is interest, I could write an article on how this is implemented.

What I will test next is how this thing plays with JDBC, seeing as
this would be a common case. Also,
I have read up on JCA and there is some preliminary code to produce a
.rar archive that will provide neo
functionality within a container, although this still very premature.

Thank you for your time,

CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] JTA support for Neo

[Chris Gioran]

OK, I have a question.

The XA/Open spec in paragraph 2.3 states that the TM can discard its
knowledge of the current global tx after the RMs have committed or
rolled back. Neo takes advantage of this fact in several places, where
during the completion of the current tx either way (commit or
rollback) it asks the TM for the current tx to retrieve related
resources. In fact, the TxManager class that is provided with Neo as a
top level TM "forgets" (i.e. unmaps the tx from the thread) *after*
calling afterCompletion() hooks, so everything works fine out of the
box. JOTM on the other hand is not consistent (and, eventually, spec
compliant), at least this is what is seems to me. Let me explain.

JOTM (its Current class, that is the TM implementation) has a commit()
method that first asks RMs to commit() and then dissociates the global
tx from the thread. But on rollback, first it dissociates the tx from
the thread and then, from a method local copy it asks the RMs to
rollback(). This leaves any RM to get a null reference when asking the
JOTM TM for the current tx, which leads to failures in the process. If
the rollback() code in Current is changed to first ask the RMs to
rollback() and then forget the global tx, everything works as it
should.

My point is that, from a first view, this is a bug in JOTM, not in
Neo. Does anyone have any experience with this? Am I interpreting this
wrong or should it be taken upstream?

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] JTA support for Neo

[Chris Gioran]

> Chris,
> Awesome! I think the next step would be to start testing things when
> neo4j needs to recover, rollback etc, I think this is where the
> problems arise :)
>
> Also, any chance of making a maven project out of it and having the
> project as a test component somewhere in the Svn repo, so it can be
> run as part of the QA process for releases etc?

OK, so the plan I have in my mind is this:

- Do runs to see what problems exist when rolling back/starting in
recovery mode with the only resource being neo.
- See how the whole thing works when another XA compatible resource is
thrown in the mix, probably a RDBMS, checking that 2PC works.
- Find the least intrusive way of making neo fit in the picture, in
terms of configuration/code changes etc, approve and commit those.
- Write test cases and a maven project so that it can be integrated in
the release cycle to be checked for correct functionality.
- After that probably I would like to fill in the gaps so that from an
app server I can do a container managed tx over a jdbc connection and
a neo connection. After all, this is the ultimate purpose of this
exercise.

I will fill you in as I go through each of the above. Thanks for your time.

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

[Neo4j] JTA support for Neo

[Chris Gioran]

Hi people,

I have started work on providing support for integrating Neo in a JTA
environment. This is the first time I have done something like this so
some feedback would be welcome.

My setup is this: I have an ApacheDS instance running to provide an
LDAP service to store objects. I have tweaked (very slightly) a
standalone JOTM instance to provide its TransactionManager via JNDI to
the directory (I prefer that over RMI). I then start an
EmbeddedGraphDatabase and replace the native neo TxManager with the
JOTM provided implementation as I retrieve it from the directory, both
in the PersistenceModule and the TxModule. From there, instead of
asking the graphDb to beginTx() I ask the JOTM TransactionManager to
begin(). I perform some Node creations and they are persisted on
commit(). Also, via a debugger, I can see that the XAResources of Neo
are indeed enlisted on the remove Transaction.

I know that this is very premature and for the time being the changes
are very rough but I think I have a proof of concept. Comments? Does
anyone have any experience in a similar setting? Next steps?

This has a very high coolness factor, btw :)

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] An attempt at documenting the internals and architecture of Neo

[Chris Gioran]

Hi people,

the last part of my series:

Neo4j Internals: Transactions (Part 3) as a complete run and a conclusion

is available here:

http://digitalstain.blogspot.com/2010/11/neo4j-internals-transactions-part-3-as.html

I try, by following a very small example, to demonstrate the
interactions that lead to a transactional operation and at the same
time show how the various parts I have described previously work
together to store a node.

I think I am done writing on Neo internals for some time, I now know
enough to start meddling with the code. I hope you found my work
useful and that you enjoyed it as much as I did writing it. If there
are any questions or subjects that you feel are underexposed or wrong,
or if you would like to discuss something, this mailing list or the
#neo4j channel on freenode is where I encourage you to do it, so that
others can benefit as well.

Thank you for your time.

CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] An attempt at documenting the internals and architecture of Neo

[Chris Gioran]

The second-to-last post in the core series:

Neo4j Internals: Interlude - Xa roundup and consistency

available here:

http://digitalstain.blogspot.com/2010/11/neo4j-internals-interlude-xa-roundup.html

A needed coverage of some classes that are a core part of the Neo
mechanism. It should have been posted before I started the tx
discussion but I had not understood their details yet. Anyway, this is
kind of boring text, dealing mostly with engineering details and
hardly with any algorithms, discussing mainly the Xa related classes.
The second part is the LockReleaser and how it achieves a consistent
view of Primitives within a tx. I hope you will enjoy it and find it
helpful.

The next post is under construction and delivers what this series was
supposed to: A walkthrough from db boot-up to shutdown.

As always, all comments welcome.

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Delete a RelationshipType

[Chris Gioran]

> My understanding is this:

And it is wrong. Consider the following snippet:

GraphDatabaseService graphDb = new EmbeddedGraphDatabase("var/base");

Transaction tx = graphDb.beginTx();

Node firstNode = graphDb.createNode();
Node secondNode = graphDb.createNode();
System.out.println(firstNode.getId());
System.out.println(secondNode.getId());
secondNode.delete();
Node thirdNode = graphDb.createNode();
System.out.println(thirdNode.getId());
tx.success();
tx.finish();

tx = graphDb.beginTx();
Node fourthNode = graphDb.createNode();
System.out.println(fourthNode.getId());
tx.success();
tx.finish();
graphDb.shutdown();

If the id pool was reused after a successful commit() (success() in
the above code) as I described then, on a new db, this should print
1
2
3
2

but instead it prints
1
2
3
4

The id 2 is reused upon restart of the db.
That will teach me to speak without checking first. What I did wrong
is left as an exercise for tomorrow.

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Delete a RelationshipType

[Chris Gioran]

> I am curious, are there any 'developer tricks' for influencing when old
> slots are reused? For example, instead of adding the new relationships
> immediately after deleting old ones, but keeping a cache of 'relationships
> to add' and adding them a little later. Will that increase the chance of old
> slots being reused?

My understanding is this:
An id for every record and, correspondingly, a number of bytes in the
file, are reserved the moment that, during a tx, a new entity is
created. Conversely, the id of an entity is freed upon successful
commit() of the tx that deleted it, although the space is still
occupied but marked as free and available to be returned at the next
request for a new entity. Note that all these operations have entity
type scope, meaning that a freed Node does not in any way influence
Relationship operations.
The above lead to a strategy of performing and committing any deletes
first and then, in a new tx, create all that is needed. If, during the
same tx, a delete is made and then a create, the deleted id is not
reused.

> I think this is relevant to cases where the developer wants to keep the
> database reasonably compact but does large amounts of deletion and creation.

I think there was a similar discussion here some time ago and I seem
to recall that the proposed solution for compacting a db after a long
streak of deletes/inserts was to recreate the db in a new location.

Hope that helps somewhat.

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] Indirection

[Chris Gioran]

On Sat, Oct 30, 2010 at 12:51 PM, Andres Taylor
 wrote:
> Guys,
>
> I'm trying to grokk the code. In a couple of places, there are indirections
> that I don't understand. Like EmbeddedGraphDatabase, which is, as far as I
> can see, mostly a thin wrapper around EmbeddedGraphDatabaseImpl. Why is
> this?
>
> Andrés

I second that. It isn't like there is a factory in
EmbeddedGraphDatabase that will provide different implementations of
something based on the Config (besides, the read-only vs the full r/w
db issue is handled differently). The EmbeddedGraphDatabaseImpl field
is hardwired there. I can only assume this was supposed to (or will?)
augment the GraphDatabaseService interface with operations other that
the EmbeddedGraphDatabaseImpl ones, given that the latter does not
implement that.

I also have difficulty grasping the EventConsumers concept. They are
created in the XaConnection only to be stored in the
NioNeoDbPersistenceSource.NioNeoDbResourceConnection and then used to
forward calls about primitives. Why is that level of abstraction
necessary? I think it would be cleaner and without loss of
extensibility for the ResourceConnection to forward directly to the
XaConnection. But I admit there are many things happening there, I may
be missing something.

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] An attempt at documenting the internals and architecture of Neo

[Chris Gioran]

Another post in the same series

Neo4j Internals: Transactions (Part 2) - XaResources, Transactions and
TransactionManagers

available at

http://digitalstain.blogspot.com/2010/10/neo4j-internals-transactions-part-2.html

As the posts go by, I tend to have an intuitive feel about the code,
so I may presume some things as obvious while they are not. Please,
inform me of such shortcomings along with any errors you find. Also,
if you have any comments on the wording, the format or the content, it
would be a great help.

Thank you for your time.

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] An attempt at documenting the internals and architecture of Neo

[Chris Gioran]

Hi people,

here is the next part of my series on Neo internals

Neo4j Internals: Transactions (Part 1) - Write Ahead Log and Deadlock Detection

available at 
http://digitalstain.blogspot.com/2010/10/neo4j-internals-transactions-part-1.html

Since the tx handling code is much more complex than the previously
described components, I have omitted many details, trying to show only
how the basic things are done. Even so, it was proved that there are
many things to write about, so I had to break down the tx description
to many parts. In this first one, the write ahead log and the deadlock
detection mechanisms are described, somewhat coarsely.

As always, any mistakes you find, please take some time to point them
out so I can have them fixed.

Thank you for your time and your positive comments.

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] An attempt at documenting the internals and architecture of Neo

[Chris Gioran]

On Mon, Oct 18, 2010 at 11:57 PM, Peter Neubauer
 wrote:
> Chris,
> great reading your blogs on this! Did you get on with the transaction
> piece yet? In case you need info, don't hesitate to ask here on the
> list, there are a number of folks that have disected the area before
> :)
>
> Also, when it comes to Neo4j working with other TX managers in a JTA
> environment like JOTM and Atomikos, I hope we will be able at least to
> start with failing tests in these environments in the next iteration
> and beginning with that, start working around the issues that JTA is
> leaving for non-relational stores like Neo4j. There might be some
> details that you are interested in when digging there :)
>
> Cheers,
>
> /peter neubauer

Peter, thanks for your input.
The transaction handling code is the aspect of a db that I am least
familiar with from an implementation standpoint, so it took a bit more
time than I had planned. I have come through, however, wiser and ready
to write about it in what will in all probability be more than one
post. Soon :)

As for JTA support, it is one of the reasons I started my
documentation efforts and I would like to see neo working in a JTA/JTS
environment.

But first I have to write about XALogicalLog and its friends. Stay tuned.

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] [SPAM] Re: [SPAM] Re: EmbeddedGraphDatabase shutdown leaves a Timer thread running

[Chris Gioran]

On Wed, Oct 13, 2010 at 5:48 PM, Rick Bullotta
 wrote:
> Lucky guess. ;-)

It seems that the load() method of the UdcExtensionImpl creates a
java.util.Timer to do its thing but I cannot find anywhere a call to
the timer's cancel(). In that case (as per the javadoc) the Timer will
take arbitrarily long to terminate. I think this explains the observed
behavior.

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] [SPAM] Re: EmbeddedGraphDatabase shutdown leaves a Timer thread running

[Chris Gioran]

On Wed, Oct 13, 2010 at 5:34 PM, Rick Bullotta
 wrote:
> Probably the recently added "phone home" functionality (UDC).  Just a guess,
> though.

in that case removing the dependency as per

http://wiki.neo4j.org/content/UDC

should make it work. My setup is the same bar a _21 JVM and 2 cores.
All my tests terminate normally.
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] EmbeddedGraphDatabase shutdown leaves a Timer thread running

[Chris Gioran]

On Wed, Oct 13, 2010 at 4:11 PM, Adam Lehenbauer  wrote:
> I think I have the Getting Started example working, but after it creates the
> nodes and shuts down, my JVM won't exit because there is a non-daemon Timer
> thread running.
>
> I have an example class with a main() that is copied near-verbatim from
> http://wiki.neo4j.org/content/Getting_Started_Guide except for the db path
> and a few extra System.outs.
>
> Everything seems to work normally, but after shutting down
> the EmbeddedGraphDatabase the JVM will not exit. A jstack shows a running
> java.util.TimerThread, which I assume is started by the database.
>
> Its stack is:
>
> ---
> "Timer-0" prio=10 tid=0x7f59c0056800 nid=0xa29 in Object.wait()
> [0x7f59bf6d5000]
>   java.lang.Thread.State: TIMED_WAITING (on object monitor)
>  at java.lang.Object.wait(Native Method)
> - waiting on <0x7f5a1ed37270> (a java.util.TaskQueue)
>  at java.util.TimerThread.mainLoop(Timer.java:509)
> - locked <0x7f5a1ed37270> (a java.util.TaskQueue)
>  at java.util.TimerThread.run(Timer.java:462)
> ---
>
> I'm using the neo4j maven dependency: org.neo4j.neo4j, version=1.2.M01,
> type=pom. When I changed this to neo4j-kernel, version=1.0 the JVM exited as
> expected (no code change on my end).
>
> Does anyone have any idea why this is happening?
>
> Adam

Hi there,

Could you run it with a configuration setting of "cache_type"="soft"
and inform on the result?
Also, describe a bit your environment (CPU, OS, JVM)

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] An attempt at documenting the internals and architecture of Neo

[Chris Gioran]

Hi list,

I have another couple of posts on the same series.

The first can be read at

http://digitalstain.blogspot.com/2010/10/neo4j-internals-file-storage.html

and it is an exposition of my understanding of the format of the files
that neo keeps its data in. It is a fairly big post that deals with
small details, so should anyone read it do not be surprised if it
proves to be reality incompatible. Corrections will be incorporated
ASAP.

The second is about the next level in the persistence subsystem of neo
and is here:

http://digitalstain.blogspot.com/2010/10/neo4j-internals-persistence-and-memory.html

I try to explain how the buckets of bits are translated to entities in
the domain of neo, how the memory mapping works and I dwell for a bit
over the light vs heavy issue.

Note that both posts where written in one stretch each after I
finished going through the code. I expect that at some points I may
have left out details that are not so clear for someone going through
the implementation for the first time. If you note something like that
(or any other errors, of course) please comment so that I can remedy
them.

Hopefully within the week I will have a piece about the transaction
handling and how that interfaces with the store and the NodeManager.
But to get there I have some reading to do.

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

Re: [Neo4j] An attempt at documenting the internals and architecture of Neo

[Chris Gioran]

On Thu, Oct 7, 2010 at 2:59 PM, Andres Taylor
 wrote:
> Hi Chris,
>
> Absolutely awesome stuff you have written. As a new hire in Neo
> Technologies, I'm reading your blog posts with great interest. I like how
> you take code and make a story out of it - it's much easier to understand
> the code after reading your posts on it. Thank you!

I am happy to know that it proved useful to someone. Thank you.

> I have two wishes that would make your posts even more awesomer (for me at
> least).
>
> 1. I'd love it if you could point to the code. In this post, you write about
> getNodeById() - why not make the method name a link to the class file in
> https://svn.neo4org?

Fair enough. I had decided against it because I don't like to litter
the text with links and,
given the expectation that there would be many references, I feared it
would be distracting.
Moreover, if someone is reading my posts I assumed she has already
downloaded the
code and set it up in her IDE. However, I think I should add links to
at least some of the
referenced code, to save some trouble and link back to the project website.

> 2. I think that it would be interesting to have a post about how backups are
> done. Personally, I have always been interested in how you make a backup of
> a live database without using huge locks. But maybe that's just me... :)

Nice idea. Added in the TODO list.

> You are doing a great job Chris. If you have any questions, ask away here on
> the list.

Appreciate it.

> Andrés

CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user

[Neo4j] An attempt at documenting the internals and architecture of Neo

[Chris Gioran]

Hi list,

I have started a series of small articles that comprise an attempt to
provide an exposition of the internals of Neo, explaining some of the
inner workings and the functions of some core classes. So far I have
taken a shot at the startup/base class creation and the caching
subsystem. You can find them here:

http://digitalstain.blogspot.com/2010/10/inside-neo4j-hello-world.html
http://digitalstain.blogspot.com/2010/10/neo4j-internals-caching.html

This effort begun as a personal documentation project to keep notes as
I was going through the code, but judging the result as potentially
useful to a broader audience I thought it might be a good idea to
share. I am, however, not convinced that the format is explanatory
enough or deep enough (or correct, for that matter :), so I was hoping
for some community feedback for improvement.

I have still to do the persistence layer, the traversal framework and,
of course, transactions. Is there anything else I should cover (or is
this a complete waste of time)?

Thank you for your time.

cheers,
CG
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user