Re: [Neo4j] Node creation limit
The 1.2 release is scheduled to be released in Q4 (most likely in November). Regarding implementations running on large graphs using Neo4j there have been several mentions of that on the list so you could try search the user archives (http://www.mail-archive.com/user@lists.neo4j.org/). For example: http://lists.neo4j.org/pipermail/user/2010-April/003493.html Regarding sharding, if your domain is shardable you can use the same domain-specific sharding scheme with a graph database as you would using some other solution (RDBMS, document store etc). Traversals over shards would then have to be managed by the application that knows about the domain and sharding scheme in place. -Johan On Wed, Jun 9, 2010 at 3:13 AM, Biren Gandhi biren.gan...@gmail.com wrote: Any timeline guidance on release 1.2? We would like to learn about any implementation supporting following claim on the main neo4j page. Does anyone know about sharding schemes and how would traverser work with distributed graph? - massive scalability. Neo4j can handle graphs of several *billion*nodes/relationships/properties on a single machine and can be sharded to scale out across multiple machines. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Node creation limit
I just added code in trunk so block size for string and array store can be configured when the store is created. This will be available in the 1.1 release but if you want to try it out now use 1.1-SNAPSHOT and create a new store like this: MapString,String config = new HashMapString, String(); config.put( string_block_size, 60 ); config.put( array_block_size, 300 ); // create a new store with string block size 60 and array block size 300 new EmbeddedGraphDatabase( path-to-db-that-do-not-exist, config ).shutdown(); The default value (120 bytes) was picked to fit common/avg size string/array properties in one block since it will be slower to load a property that is spread out on many blocks. Since datasets vary a lot in string size / array size and the like I think it is better to have it configurable at creation time. When tweaking these values remember that strings will consume twice the string length in bytes so a string block size of 60 will be able to fit a string of length 30 in a single block. Regarding scaling 1.0 and 1.1 releases have a limit of 4 billion records / store file so if you need to store 4 billion strings you have to make sure every string fits in a single block. This limit will be increased to 32 billion or more in the 1.2 release. -Johan On Mon, Jun 7, 2010 at 4:27 PM, Biren Gandhi biren.gan...@gmail.com wrote: Similar issue on my side as well. Test data is ok, but production data (100 million+ objects, 200 relationships per object and 10 properties per object, with multi-million queries per day about search and traversal) would need clear disk sizing calculations due to iops and other hardware limits in a monolithic storage model. Has anyone been able to use neo4j succeessfully in scaling needs similar to mentioned avove? -b On Jun 7, 2010, at 4:45 AM, Craig Taverner cr...@amanzi.com wrote: Is there a specific constrain on disk space? Normally disk space isn't a problem... it's cheap and there's usually loads of it. Actually for most of my use cases the disk space has been fine. Except for one data source, that surprised me by expanding from less than a gig of original binary data, to over 20GB database. While this too can be managed, it was just a sample, and so I have yet to see what the customers 'real data' will do to the database (several hundred times larger, I'm expecting). When we get to that point we will need to decide how to deal with it. Currently we 'solve' the issue by allowing the user to filter out data on import, so we don't store everything. This will not satisfy all users, however. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Node creation limit
Hi, Johan.
In the scenario you describe below, if one attempts to open an
existing Neo DB that has been created with a non-standard block size,
will it be able to get that information from the DB itself or must you
provide that information when you attempt via config parameters to open
it?
Thanks,
Rick
Original Message
Subject: Re: [Neo4j] Node creation limit
From: Johan Svensson [1]jo...@neotechnology.com
Date: Tue, June 08, 2010 5:15 am
To: Neo4j user discussions [2]u...@lists.neo4j.org
I just added code in trunk so block size for string and array store
can be configured when the store is created. This will be available in
the 1.1 release but if you want to try it out now use 1.1-SNAPSHOT and
create a new store like this:
MapString,String config = new HashMapString, String();
config.put( string_block_size, 60 );
config.put( array_block_size, 300 );
// create a new store with string block size 60 and array block size
300
new EmbeddedGraphDatabase( path-to-db-that-do-not-exist, config
).shutdown();
The default value (120 bytes) was picked to fit common/avg size
string/array properties in one block since it will be slower to load a
property that is spread out on many blocks. Since datasets vary a lot
in string size / array size and the like I think it is better to have
it configurable at creation time. When tweaking these values remember
that strings will consume twice the string length in bytes so a string
block size of 60 will be able to fit a string of length 30 in a single
block.
Regarding scaling 1.0 and 1.1 releases have a limit of 4 billion
records / store file so if you need to store 4 billion strings you
have to make sure every string fits in a single block. This limit will
be increased to 32 billion or more in the 1.2 release.
-Johan
On Mon, Jun 7, 2010 at 4:27 PM, Biren Gandhi
[3]biren.gan...@gmail.com wrote:
Similar issue on my side as well. Test data is ok, but production
data
(100 million+ objects, 200 relationships per object and 10 properties
per object, with multi-million queries per day about search and
traversal) would need clear disk sizing calculations due to iops and
other hardware limits in a monolithic storage model.
Has anyone been able to use neo4j succeessfully in scaling needs
similar to mentioned avove?
-b
On Jun 7, 2010, at 4:45 AM, Craig Taverner [4]cr...@amanzi.com
wrote:
Is there a specific constrain on disk space? Normally disk space
isn't
a problem... it's cheap and there's usually loads of it.
Actually for most of my use cases the disk space has been fine.
Except for
one data source, that surprised me by expanding from less than a gig
of
original binary data, to over 20GB database. While this too can be
managed,
it was just a sample, and so I have yet to see what the customers
'real
data' will do to the database (several hundred times larger, I'm
expecting).
When we get to that point we will need to decide how to deal with
it.
Currently we 'solve' the issue by allowing the user to filter out
data on
import, so we don't store everything. This will not satisfy all
users,
however.
___
Neo4j mailing list
[5]u...@lists.neo4j.org
[6]https://lists.neo4j.org/mailman/listinfo/user
References
1. mailto://jo...@neotechnology.com/
2. mailto://user@lists.neo4j.org/
3. mailto://biren.gan...@gmail.com/
4. mailto://cr...@amanzi.com/
5. mailto://User@lists.neo4j.org/
6. https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Node creation limit
2010/6/8 rick.bullo...@burningskysoftware.com: Hi, Johan. In the scenario you describe below, if one attempts to open an existing Neo DB that has been created with a non-standard block size, will it be able to get that information from the DB itself or must you provide that information when you attempt via config parameters to open it? It will read the config parameter the first time (when the neo4j database is created) and store those block size config values in the store itself. This means that if you open the database the next time those config parameters will be ignored and the values used at creation time will be used. Thanks, Rick Original Message Subject: Re: [Neo4j] Node creation limit From: Johan Svensson [1]jo...@neotechnology.com Date: Tue, June 08, 2010 5:15 am To: Neo4j user discussions [2]u...@lists.neo4j.org I just added code in trunk so block size for string and array store can be configured when the store is created. This will be available in the 1.1 release but if you want to try it out now use 1.1-SNAPSHOT and create a new store like this: MapString,String config = new HashMapString, String(); config.put( string_block_size, 60 ); config.put( array_block_size, 300 ); // create a new store with string block size 60 and array block size 300 new EmbeddedGraphDatabase( path-to-db-that-do-not-exist, config ).shutdown(); The default value (120 bytes) was picked to fit common/avg size string/array properties in one block since it will be slower to load a property that is spread out on many blocks. Since datasets vary a lot in string size / array size and the like I think it is better to have it configurable at creation time. When tweaking these values remember that strings will consume twice the string length in bytes so a string block size of 60 will be able to fit a string of length 30 in a single block. Regarding scaling 1.0 and 1.1 releases have a limit of 4 billion records / store file so if you need to store 4 billion strings you have to make sure every string fits in a single block. This limit will be increased to 32 billion or more in the 1.2 release. -Johan On Mon, Jun 7, 2010 at 4:27 PM, Biren Gandhi [3]biren.gan...@gmail.com wrote: Similar issue on my side as well. Test data is ok, but production data (100 million+ objects, 200 relationships per object and 10 properties per object, with multi-million queries per day about search and traversal) would need clear disk sizing calculations due to iops and other hardware limits in a monolithic storage model. Has anyone been able to use neo4j succeessfully in scaling needs similar to mentioned avove? -b On Jun 7, 2010, at 4:45 AM, Craig Taverner [4]cr...@amanzi.com wrote: Is there a specific constrain on disk space? Normally disk space isn't a problem... it's cheap and there's usually loads of it. Actually for most of my use cases the disk space has been fine. Except for one data source, that surprised me by expanding from less than a gig of original binary data, to over 20GB database. While this too can be managed, it was just a sample, and so I have yet to see what the customers 'real data' will do to the database (several hundred times larger, I'm expecting). When we get to that point we will need to decide how to deal with it. Currently we 'solve' the issue by allowing the user to filter out data on import, so we don't store everything. This will not satisfy all users, however. ___ Neo4j mailing list [5]u...@lists.neo4j.org [6]https://lists.neo4j.org/mailman/listinfo/user References 1. mailto://jo...@neotechnology.com/ 2. mailto://user@lists.neo4j.org/ 3. mailto://biren.gan...@gmail.com/ 4. mailto://cr...@amanzi.com/ 5. mailto://User@lists.neo4j.org/ 6. https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Node creation limit
Thanks. Regarding scaling 1.0 and 1.1 releases have a limit of 4 billion records / store file so if you need to store 4 billion strings you have to make sure every string fits in a single block. This limit will be increased to 32 billion or more in the 1.2 release. Any timeline guidance on release 1.2? We would like to learn about any implementation supporting following claim on the main neo4j page. Does anyone know about sharding schemes and how would traverser work with distributed graph? - massive scalability. Neo4j can handle graphs of several *billion*nodes/relationships/properties on a single machine and can be sharded to scale out across multiple machines. -Johan On Mon, Jun 7, 2010 at 4:27 PM, Biren Gandhi biren.gan...@gmail.com wrote: Similar issue on my side as well. Test data is ok, but production data (100 million+ objects, 200 relationships per object and 10 properties per object, with multi-million queries per day about search and traversal) would need clear disk sizing calculations due to iops and other hardware limits in a monolithic storage model. Has anyone been able to use neo4j succeessfully in scaling needs similar to mentioned avove? -b On Jun 7, 2010, at 4:45 AM, Craig Taverner cr...@amanzi.com wrote: Is there a specific constrain on disk space? Normally disk space isn't a problem... it's cheap and there's usually loads of it. Actually for most of my use cases the disk space has been fine. Except for one data source, that surprised me by expanding from less than a gig of original binary data, to over 20GB database. While this too can be managed, it was just a sample, and so I have yet to see what the customers 'real data' will do to the database (several hundred times larger, I'm expecting). When we get to that point we will need to decide how to deal with it. Currently we 'solve' the issue by allowing the user to filter out data on import, so we don't store everything. This will not satisfy all users, however. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Node creation limit
Hi, These are the current record sizes in bytes that can be used to calculate the actual store size: nodestore: 9 relationshipstore: 33 propertystore: 25 stringstore: 133 arraystore: 133 All properties except strings and arrays will take a single propertystore record (25 bytes). A string or array property will use one record from the propertystore then as many blocks needed from the string/array store file (each block of 133 bytes can store 120 bytes of data). This means if all your strings are in 120 bytes multiples in size you will make very efficient use of the store file while if they are empty you will not make very good use of the space (exactly like a normal filesystem taking up space for empty files). -Johan On Fri, Jun 4, 2010 at 9:15 AM, Mattias Persson matt...@neotechnology.com wrote: That formula is correct regarding nodes and relationships, yes. When properties comes into play another formula would, of course, have to be applied. Depending on property types and length of keys/string values it is different. It could be good though with a formula/tool to calculate that. 2010/6/4, Biren Gandhi biren.gan...@gmail.com: In that case, what are the ways to estimate storage capacity numbers? Basic formula of nodes*9 + edges*33 doesn't seem like a practical one. On Wed, Jun 2, 2010 at 11:26 PM, Mattias Persson matt...@neotechnology.comwrote: String properties are stored in blocks so even if you have tiny string values each property value will occupy a full block (30 or 60 bytes, can someone correct me here?). That's what taking most of your space IMHO ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Node creation limit
Seems that the string store is not optimal for the 'common' usage of properties for names or labels, which are typically 5 to 20 characters long, leading to about 5x (or more) space utilization than needed. By 'names or labels' I mean things like username, tags, categorizations, product names, etc. These seem like very common scenarios. I guess there is no simple way of optimizing for this? At an app level it is possible to pack a bunch of strings into a single property using, for example, JSON. But that only makes sense if the usage is always to work with them all as a unit. On Mon, Jun 7, 2010 at 10:32 AM, Johan Svensson jo...@neotechnology.comwrote: Hi, These are the current record sizes in bytes that can be used to calculate the actual store size: nodestore: 9 relationshipstore: 33 propertystore: 25 stringstore: 133 arraystore: 133 All properties except strings and arrays will take a single propertystore record (25 bytes). A string or array property will use one record from the propertystore then as many blocks needed from the string/array store file (each block of 133 bytes can store 120 bytes of data). This means if all your strings are in 120 bytes multiples in size you will make very efficient use of the store file while if they are empty you will not make very good use of the space (exactly like a normal filesystem taking up space for empty files). -Johan On Fri, Jun 4, 2010 at 9:15 AM, Mattias Persson matt...@neotechnology.com wrote: That formula is correct regarding nodes and relationships, yes. When properties comes into play another formula would, of course, have to be applied. Depending on property types and length of keys/string values it is different. It could be good though with a formula/tool to calculate that. 2010/6/4, Biren Gandhi biren.gan...@gmail.com: In that case, what are the ways to estimate storage capacity numbers? Basic formula of nodes*9 + edges*33 doesn't seem like a practical one. On Wed, Jun 2, 2010 at 11:26 PM, Mattias Persson matt...@neotechnology.comwrote: String properties are stored in blocks so even if you have tiny string values each property value will occupy a full block (30 or 60 bytes, can someone correct me here?). That's what taking most of your space IMHO ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Node creation limit
2010/6/7 Craig Taverner cr...@amanzi.com: Seems that the string store is not optimal for the 'common' usage of properties for names or labels, which are typically 5 to 20 characters long, leading to about 5x (or more) space utilization than needed. By 'names or labels' I mean things like username, tags, categorizations, product names, etc. These seem like very common scenarios. You're quite right about that, however: Is there a specific constrain on disk space? Normally disk space isn't a problem... it's cheap and there's usually loads of it. I guess there is no simple way of optimizing for this? At an app level it is possible to pack a bunch of strings into a single property using, for example, JSON. But that only makes sense if the usage is always to work with them all as a unit. On Mon, Jun 7, 2010 at 10:32 AM, Johan Svensson jo...@neotechnology.comwrote: Hi, These are the current record sizes in bytes that can be used to calculate the actual store size: nodestore: 9 relationshipstore: 33 propertystore: 25 stringstore: 133 arraystore: 133 All properties except strings and arrays will take a single propertystore record (25 bytes). A string or array property will use one record from the propertystore then as many blocks needed from the string/array store file (each block of 133 bytes can store 120 bytes of data). This means if all your strings are in 120 bytes multiples in size you will make very efficient use of the store file while if they are empty you will not make very good use of the space (exactly like a normal filesystem taking up space for empty files). -Johan On Fri, Jun 4, 2010 at 9:15 AM, Mattias Persson matt...@neotechnology.com wrote: That formula is correct regarding nodes and relationships, yes. When properties comes into play another formula would, of course, have to be applied. Depending on property types and length of keys/string values it is different. It could be good though with a formula/tool to calculate that. 2010/6/4, Biren Gandhi biren.gan...@gmail.com: In that case, what are the ways to estimate storage capacity numbers? Basic formula of nodes*9 + edges*33 doesn't seem like a practical one. On Wed, Jun 2, 2010 at 11:26 PM, Mattias Persson matt...@neotechnology.comwrote: String properties are stored in blocks so even if you have tiny string values each property value will occupy a full block (30 or 60 bytes, can someone correct me here?). That's what taking most of your space IMHO ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Node creation limit
Is there a specific constrain on disk space? Normally disk space isn't a problem... it's cheap and there's usually loads of it. Actually for most of my use cases the disk space has been fine. Except for one data source, that surprised me by expanding from less than a gig of original binary data, to over 20GB database. While this too can be managed, it was just a sample, and so I have yet to see what the customers 'real data' will do to the database (several hundred times larger, I'm expecting). When we get to that point we will need to decide how to deal with it. Currently we 'solve' the issue by allowing the user to filter out data on import, so we don't store everything. This will not satisfy all users, however. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Node creation limit
Disk space is cheap, yes, but of course there is a performance cost to
read from it. More importantly, do the same record sizes apply to the
in-memory representation of properties? If so, that might be an issue,
since memory is a more precious commodity. If not, then doing a
pre-load of much of the graph would help mitigate the disk space and
I/O concerns.
Can you confirm whether or not the in-memory structures use a
fixed-record size model also?
Original Message
Subject: Re: [Neo4j] Node creation limit
From: Mattias Persson [1]matt...@neotechnology.com
Date: Mon, June 07, 2010 7:38 am
To: Neo4j user discussions [2]u...@lists.neo4j.org
2010/6/7 Craig Taverner [3]cr...@amanzi.com:
Seems that the string store is not optimal for the 'common' usage of
properties for names or labels, which are typically 5 to 20
characters long,
leading to about 5x (or more) space utilization than needed. By
'names or
labels' I mean things like username, tags, categorizations, product
names,
etc. These seem like very common scenarios.
You're quite right about that, however:
Is there a specific constrain on disk space? Normally disk space isn't
a problem... it's cheap and there's usually loads of it.
I guess there is no simple way of optimizing for this? At an app
level it is
possible to pack a bunch of strings into a single property using, for
example, JSON. But that only makes sense if the usage is always to
work with
them all as a unit.
On Mon, Jun 7, 2010 at 10:32 AM, Johan Svensson
[4]jo...@neotechnology.comwrote:
Hi,
These are the current record sizes in bytes that can be used to
calculate the actual store size:
nodestore: 9
relationshipstore: 33
propertystore: 25
stringstore: 133
arraystore: 133
All properties except strings and arrays will take a single
propertystore record (25 bytes). A string or array property will use
one record from the propertystore then as many blocks needed from
the
string/array store file (each block of 133 bytes can store 120 bytes
of data). This means if all your strings are in 120 bytes multiples
in
size you will make very efficient use of the store file while if
they
are empty you will not make very good use of the space (exactly like
a
normal filesystem taking up space for empty files).
-Johan
On Fri, Jun 4, 2010 at 9:15 AM, Mattias Persson
[5]matt...@neotechnology.com wrote:
That formula is correct regarding nodes and relationships, yes.
When
properties comes into play another formula would, of course, have
to
be applied. Depending on property types and length of keys/string
values it is different. It could be good though with a
formula/tool to
calculate that.
2010/6/4, Biren Gandhi [6]biren.gan...@gmail.com:
In that case, what are the ways to estimate storage capacity
numbers?
Basic
formula of nodes*9 + edges*33 doesn't seem like a practical one.
On Wed, Jun 2, 2010 at 11:26 PM, Mattias Persson
[7]matt...@neotechnology.comwrote:
String properties are stored in blocks so even if you have tiny
string
values each property value will occupy a full block (30 or 60
bytes,
can someone correct me here?). That's what taking most of your
space
IMHO
___
Neo4j mailing list
[8]u...@lists.neo4j.org
[9]https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
[10]u...@lists.neo4j.org
[11]https://lists.neo4j.org/mailman/listinfo/user
--
Mattias Persson, [[12]matt...@neotechnology.com]
Hacker, Neo Technology
[13]www.neotechnology.com
___
Neo4j mailing list
[14]u...@lists.neo4j.org
[15]https://lists.neo4j.org/mailman/listinfo/user
References
1. mailto://matt...@neotechnology.com/
2. mailto://user@lists.neo4j.org/
3. mailto://cr...@amanzi.com/
4. mailto://jo...@neotechnology.com/
5. mailto://matt...@neotechnology.com/
6. mailto://biren.gan...@gmail.com/
7. mailto://matt...@neotechnology.com/
8. mailto://User@lists.neo4j.org/
9. https://lists.neo4j.org/mailman/listinfo/user
10. mailto://User@lists.neo4j.org/
11. https://lists.neo4j.org/mailman/listinfo/user
12. mailto://matt...@neotechnology.com/
13. http://www.neotechnology.com/
14. mailto://User@lists.neo4j.org/
15. https://lists.neo4j.org/mailman/listinfo/user
___
Neo4j mailing list
User@lists.neo4j.org
https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Node creation limit
Similar issue on my side as well. Test data is ok, but production data (100 million+ objects, 200 relationships per object and 10 properties per object, with multi-million queries per day about search and traversal) would need clear disk sizing calculations due to iops and other hardware limits in a monolithic storage model. Has anyone been able to use neo4j succeessfully in scaling needs similar to mentioned avove? -b On Jun 7, 2010, at 4:45 AM, Craig Taverner cr...@amanzi.com wrote: Is there a specific constrain on disk space? Normally disk space isn't a problem... it's cheap and there's usually loads of it. Actually for most of my use cases the disk space has been fine. Except for one data source, that surprised me by expanding from less than a gig of original binary data, to over 20GB database. While this too can be managed, it was just a sample, and so I have yet to see what the customers 'real data' will do to the database (several hundred times larger, I'm expecting). When we get to that point we will need to decide how to deal with it. Currently we 'solve' the issue by allowing the user to filter out data on import, so we don't store everything. This will not satisfy all users, however. ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Node creation limit
That formula is correct regarding nodes and relationships, yes. When properties comes into play another formula would, of course, have to be applied. Depending on property types and length of keys/string values it is different. It could be good though with a formula/tool to calculate that. 2010/6/4, Biren Gandhi biren.gan...@gmail.com: In that case, what are the ways to estimate storage capacity numbers? Basic formula of nodes*9 + edges*33 doesn't seem like a practical one. On Wed, Jun 2, 2010 at 11:26 PM, Mattias Persson matt...@neotechnology.comwrote: String properties are stored in blocks so even if you have tiny string values each property value will occupy a full block (30 or 60 bytes, can someone correct me here?). That's what taking most of your space IMHO 2010/6/3, Biren Gandhi biren.gan...@gmail.com: Here is some content from neostore.propertystore.db.strings - another huge file. What are the max number of nodes/relationships that people have tried with Neo4j so far? Can someone share disk space usage characteristics? od -N 1000 -x -c neostore.propertystore.db.strings 000 8500 \0 \0 \0 205 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 020 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 200 0100 0c00 \0 \0 \0 \0 \0 001 377 377 377 377 \0 \0 \0 \f 377 377 220 4e00 6f00 6400 6500 2d00 3000 377 377 \0 N \0 o \0 d \0 e \0 - \0 0 \0 \0 240 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 400 ff01 00ff \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 377 377 377 377 \0 420 ff0c 00ff 004e 006f 0064 0065 \0 \0 \f 377 377 377 377 \0 N \0 o \0 d \0 e \0 440 002d 0031 - \0 1 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 460 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 600 0100 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 620 0c00 4e00 6f00 377 377 377 377 \0 \0 \0 \f 377 377 377 377 \0 N \0 o 640 6400 6500 2d00 3200 \0 d \0 e \0 - \0 2 \0 \0 \0 \0 \0 \0 \0 \0 660 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0001020 ff01 00ff ff0c \0 \0 \0 \0 001 377 377 377 377 \0 \0 \0 \f 377 377 377 0001040 00ff 004e 006f 0064 0065 002d 0033 377 \0 N \0 o \0 d \0 e \0 - \0 3 \0 \0 \0 0001060 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0001220 0100 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 377 377 377 377 \0 \0 0001240 0c00 4e00 6f00 6400 6500 2d00 \0 \f 377 377 377 377 \0 N \0 o \0 d \0 e \0 - 0001260 3400 \0 4 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0001300 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0001420 ff01 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 377 0001440 00ff ff0c 00ff 004e 006f 377 377 377 \0 \0 \0 \f 377 377 377 377 \0 N \0 o \0 0001460 0064 0065 002d 0035 d \0 e \0 - \0 5 \0 \0 \0 \0 \0 \0 \0 \0 \0 0001500 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * On Wed, Jun 2, 2010 at 3:50 PM, Biren Gandhi biren.gan...@gmail.com wrote: There is only 1 property - n (to store name of the node) - used as follows: Node node = graphDb.createNode(); node.setProperty( NAME_KEY, username ); And the values of username are Node-1, Node-2 etc. On Wed, Jun 2, 2010 at 3:14 PM, Mattias Persson matt...@neotechnology.com wrote: Only 4,4mb out of those 80 is consumed by nodes so you must be storing some properties somewhere. Would you mind sharing your code so that it would be easier to get a better insight into your problem? 2010/6/2, Biren Gandhi biren.gan...@gmail.com: Thanks. Big transactions were indeed problematic. Splitting them down into smaller chunks did the trick. I'm still disappointed by the on-disk size of a minimal node without any
Re: [Neo4j] Node creation limit
String properties are stored in blocks so even if you have tiny string values each property value will occupy a full block (30 or 60 bytes, can someone correct me here?). That's what taking most of your space IMHO 2010/6/3, Biren Gandhi biren.gan...@gmail.com: Here is some content from neostore.propertystore.db.strings - another huge file. What are the max number of nodes/relationships that people have tried with Neo4j so far? Can someone share disk space usage characteristics? od -N 1000 -x -c neostore.propertystore.db.strings 000 8500 \0 \0 \0 205 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 020 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 200 0100 0c00 \0 \0 \0 \0 \0 001 377 377 377 377 \0 \0 \0 \f 377 377 220 4e00 6f00 6400 6500 2d00 3000 377 377 \0 N \0 o \0 d \0 e \0 - \0 0 \0 \0 240 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 400 ff01 00ff \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 377 377 377 377 \0 420 ff0c 00ff 004e 006f 0064 0065 \0 \0 \f 377 377 377 377 \0 N \0 o \0 d \0 e \0 440 002d 0031 - \0 1 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 460 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 600 0100 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 620 0c00 4e00 6f00 377 377 377 377 \0 \0 \0 \f 377 377 377 377 \0 N \0 o 640 6400 6500 2d00 3200 \0 d \0 e \0 - \0 2 \0 \0 \0 \0 \0 \0 \0 \0 660 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0001020 ff01 00ff ff0c \0 \0 \0 \0 001 377 377 377 377 \0 \0 \0 \f 377 377 377 0001040 00ff 004e 006f 0064 0065 002d 0033 377 \0 N \0 o \0 d \0 e \0 - \0 3 \0 \0 \0 0001060 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0001220 0100 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 377 377 377 377 \0 \0 0001240 0c00 4e00 6f00 6400 6500 2d00 \0 \f 377 377 377 377 \0 N \0 o \0 d \0 e \0 - 0001260 3400 \0 4 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0001300 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0001420 ff01 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 377 0001440 00ff ff0c 00ff 004e 006f 377 377 377 \0 \0 \0 \f 377 377 377 377 \0 N \0 o \0 0001460 0064 0065 002d 0035 d \0 e \0 - \0 5 \0 \0 \0 \0 \0 \0 \0 \0 \0 0001500 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * On Wed, Jun 2, 2010 at 3:50 PM, Biren Gandhi biren.gan...@gmail.com wrote: There is only 1 property - n (to store name of the node) - used as follows: Node node = graphDb.createNode(); node.setProperty( NAME_KEY, username ); And the values of username are Node-1, Node-2 etc. On Wed, Jun 2, 2010 at 3:14 PM, Mattias Persson matt...@neotechnology.com wrote: Only 4,4mb out of those 80 is consumed by nodes so you must be storing some properties somewhere. Would you mind sharing your code so that it would be easier to get a better insight into your problem? 2010/6/2, Biren Gandhi biren.gan...@gmail.com: Thanks. Big transactions were indeed problematic. Splitting them down into smaller chunks did the trick. I'm still disappointed by the on-disk size of a minimal node without any relationships or attributes. For 500K nodes, it is taking 80MB space (160 byes/node) and for 1M objects it is consuming 160MB (again 160 byes/node). Is this normal? 4.0Kactive_tx_log 12K lucene 12K lucene-fulltext 4.0Kneostore 4.0Kneostore.id 4.4Mneostore.nodestore.db 4.0Kneostore.nodestore.db.id 12M neostore.propertystore.db 4.0Kneostore.propertystore.db.arrays 4.0Kneostore.propertystore.db.arrays.id 4.0Kneostore.propertystore.db.id 4.0Kneostore.propertystore.db.index 4.0Kneostore.propertystore.db.index.id 4.0Kneostore.propertystore.db.index.keys 4.0K
Re: [Neo4j] Node creation limit
Exactly, the problem is most likely that you try to insert all your stuff in one transaction. All data for a transaction is kept in memory until committed so for really big transactions it can fill your entire heap. Try to group 10k operations or so for big insertions or use the batch inserter. Links: http://wiki.neo4j.org/content/Transactions#Big_transactions http://wiki.neo4j.org/content/Batch_Insert 2010/6/2, Laurent Laborde kerdez...@gmail.com: On Wed, Jun 2, 2010 at 3:50 AM, Biren Gandhi biren.gan...@gmail.com wrote: Is there any limit on number of nodes that can be created in a neo4j instance? Any other tips? I created hundreds of millions of nodes without problems, but it was splitted into many transaction. -- Laurent ker2x Laborde Sysadmin DBA at http://www.over-blog.com/ ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Node creation limit
Exactly, the problem is most likely that you try to insert all your stuff in one transaction. All data for a transaction is kept in memory until committed so for really big transactions it can fill your entire heap. Try to group 10k operations or so for big insertions or use the batch inserter. Links: http://wiki.neo4j.org/content/Transactions#Big_transactions http://wiki.neo4j.org/content/Batch_Insert 2010/6/2, Laurent Laborde kerdez...@gmail.com: On Wed, Jun 2, 2010 at 3:50 AM, Biren Gandhi biren.gan...@gmail.com wrote: Is there any limit on number of nodes that can be created in a neo4j instance? Any other tips? I created hundreds of millions of nodes without problems, but it was splitted into many transaction. -- Laurent ker2x Laborde Sysadmin DBA at http://www.over-blog.com/ ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Node creation limit
Thanks. Big transactions were indeed problematic. Splitting them down into smaller chunks did the trick. I'm still disappointed by the on-disk size of a minimal node without any relationships or attributes. For 500K nodes, it is taking 80MB space (160 byes/node) and for 1M objects it is consuming 160MB (again 160 byes/node). Is this normal? 4.0Kactive_tx_log 12K lucene 12K lucene-fulltext 4.0Kneostore 4.0Kneostore.id 4.4Mneostore.nodestore.db 4.0Kneostore.nodestore.db.id 12M neostore.propertystore.db 4.0Kneostore.propertystore.db.arrays 4.0Kneostore.propertystore.db.arrays.id 4.0Kneostore.propertystore.db.id 4.0Kneostore.propertystore.db.index 4.0Kneostore.propertystore.db.index.id 4.0Kneostore.propertystore.db.index.keys 4.0Kneostore.propertystore.db.index.keys.id 64M neostore.propertystore.db.strings 4.0Kneostore.propertystore.db.strings.id 4.0Kneostore.relationshipstore.db 4.0Kneostore.relationshipstore.db.id 4.0Kneostore.relationshiptypestore.db 4.0Kneostore.relationshiptypestore.db.id 4.0Kneostore.relationshiptypestore.db.names 4.0Kneostore.relationshiptypestore.db.names.id 4.0Knioneo_logical.log.active 4.0Ktm_tx_log.1 80M total On Wed, Jun 2, 2010 at 12:17 AM, Mattias Persson matt...@neotechnology.comwrote: Exactly, the problem is most likely that you try to insert all your stuff in one transaction. All data for a transaction is kept in memory until committed so for really big transactions it can fill your entire heap. Try to group 10k operations or so for big insertions or use the batch inserter. Links: http://wiki.neo4j.org/content/Transactions#Big_transactions http://wiki.neo4j.org/content/Batch_Insert 2010/6/2, Laurent Laborde kerdez...@gmail.com: On Wed, Jun 2, 2010 at 3:50 AM, Biren Gandhi biren.gan...@gmail.com wrote: Is there any limit on number of nodes that can be created in a neo4j instance? Any other tips? I created hundreds of millions of nodes without problems, but it was splitted into many transaction. -- Laurent ker2x Laborde Sysadmin DBA at http://www.over-blog.com/ ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Node creation limit
Only 4,4mb out of those 80 is consumed by nodes so you must be storing some properties somewhere. Would you mind sharing your code so that it would be easier to get a better insight into your problem? 2010/6/2, Biren Gandhi biren.gan...@gmail.com: Thanks. Big transactions were indeed problematic. Splitting them down into smaller chunks did the trick. I'm still disappointed by the on-disk size of a minimal node without any relationships or attributes. For 500K nodes, it is taking 80MB space (160 byes/node) and for 1M objects it is consuming 160MB (again 160 byes/node). Is this normal? 4.0Kactive_tx_log 12K lucene 12K lucene-fulltext 4.0Kneostore 4.0Kneostore.id 4.4Mneostore.nodestore.db 4.0Kneostore.nodestore.db.id 12M neostore.propertystore.db 4.0Kneostore.propertystore.db.arrays 4.0Kneostore.propertystore.db.arrays.id 4.0Kneostore.propertystore.db.id 4.0Kneostore.propertystore.db.index 4.0Kneostore.propertystore.db.index.id 4.0Kneostore.propertystore.db.index.keys 4.0Kneostore.propertystore.db.index.keys.id 64M neostore.propertystore.db.strings 4.0Kneostore.propertystore.db.strings.id 4.0Kneostore.relationshipstore.db 4.0Kneostore.relationshipstore.db.id 4.0Kneostore.relationshiptypestore.db 4.0Kneostore.relationshiptypestore.db.id 4.0Kneostore.relationshiptypestore.db.names 4.0Kneostore.relationshiptypestore.db.names.id 4.0Knioneo_logical.log.active 4.0Ktm_tx_log.1 80M total On Wed, Jun 2, 2010 at 12:17 AM, Mattias Persson matt...@neotechnology.comwrote: Exactly, the problem is most likely that you try to insert all your stuff in one transaction. All data for a transaction is kept in memory until committed so for really big transactions it can fill your entire heap. Try to group 10k operations or so for big insertions or use the batch inserter. Links: http://wiki.neo4j.org/content/Transactions#Big_transactions http://wiki.neo4j.org/content/Batch_Insert 2010/6/2, Laurent Laborde kerdez...@gmail.com: On Wed, Jun 2, 2010 at 3:50 AM, Biren Gandhi biren.gan...@gmail.com wrote: Is there any limit on number of nodes that can be created in a neo4j instance? Any other tips? I created hundreds of millions of nodes without problems, but it was splitted into many transaction. -- Laurent ker2x Laborde Sysadmin DBA at http://www.over-blog.com/ ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Node creation limit
There is only 1 property - n (to store name of the node) - used as follows: Node node = graphDb.createNode(); node.setProperty( NAME_KEY, username ); And the values of username are Node-1, Node-2 etc. On Wed, Jun 2, 2010 at 3:14 PM, Mattias Persson matt...@neotechnology.comwrote: Only 4,4mb out of those 80 is consumed by nodes so you must be storing some properties somewhere. Would you mind sharing your code so that it would be easier to get a better insight into your problem? 2010/6/2, Biren Gandhi biren.gan...@gmail.com: Thanks. Big transactions were indeed problematic. Splitting them down into smaller chunks did the trick. I'm still disappointed by the on-disk size of a minimal node without any relationships or attributes. For 500K nodes, it is taking 80MB space (160 byes/node) and for 1M objects it is consuming 160MB (again 160 byes/node). Is this normal? 4.0Kactive_tx_log 12K lucene 12K lucene-fulltext 4.0Kneostore 4.0Kneostore.id 4.4Mneostore.nodestore.db 4.0Kneostore.nodestore.db.id 12M neostore.propertystore.db 4.0Kneostore.propertystore.db.arrays 4.0Kneostore.propertystore.db.arrays.id 4.0Kneostore.propertystore.db.id 4.0Kneostore.propertystore.db.index 4.0Kneostore.propertystore.db.index.id 4.0Kneostore.propertystore.db.index.keys 4.0Kneostore.propertystore.db.index.keys.id 64M neostore.propertystore.db.strings 4.0Kneostore.propertystore.db.strings.id 4.0Kneostore.relationshipstore.db 4.0Kneostore.relationshipstore.db.id 4.0Kneostore.relationshiptypestore.db 4.0Kneostore.relationshiptypestore.db.id 4.0Kneostore.relationshiptypestore.db.names 4.0Kneostore.relationshiptypestore.db.names.id 4.0Knioneo_logical.log.active 4.0Ktm_tx_log.1 80M total On Wed, Jun 2, 2010 at 12:17 AM, Mattias Persson matt...@neotechnology.comwrote: Exactly, the problem is most likely that you try to insert all your stuff in one transaction. All data for a transaction is kept in memory until committed so for really big transactions it can fill your entire heap. Try to group 10k operations or so for big insertions or use the batch inserter. Links: http://wiki.neo4j.org/content/Transactions#Big_transactions http://wiki.neo4j.org/content/Batch_Insert 2010/6/2, Laurent Laborde kerdez...@gmail.com: On Wed, Jun 2, 2010 at 3:50 AM, Biren Gandhi biren.gan...@gmail.com wrote: Is there any limit on number of nodes that can be created in a neo4j instance? Any other tips? I created hundreds of millions of nodes without problems, but it was splitted into many transaction. -- Laurent ker2x Laborde Sysadmin DBA at http://www.over-blog.com/ ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user -- Mattias Persson, [matt...@neotechnology.com] Hacker, Neo Technology www.neotechnology.com ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Node creation limit
Here is some content from neostore.propertystore.db.strings - another huge file. What are the max number of nodes/relationships that people have tried with Neo4j so far? Can someone share disk space usage characteristics? od -N 1000 -x -c neostore.propertystore.db.strings 000 8500 \0 \0 \0 205 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 020 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 200 0100 0c00 \0 \0 \0 \0 \0 001 377 377 377 377 \0 \0 \0 \f 377 377 220 4e00 6f00 6400 6500 2d00 3000 377 377 \0 N \0 o \0 d \0 e \0 - \0 0 \0 \0 240 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 400 ff01 00ff \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 377 377 377 377 \0 420 ff0c 00ff 004e 006f 0064 0065 \0 \0 \f 377 377 377 377 \0 N \0 o \0 d \0 e \0 440 002d 0031 - \0 1 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 460 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 600 0100 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 620 0c00 4e00 6f00 377 377 377 377 \0 \0 \0 \f 377 377 377 377 \0 N \0 o 640 6400 6500 2d00 3200 \0 d \0 e \0 - \0 2 \0 \0 \0 \0 \0 \0 \0 \0 660 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0001020 ff01 00ff ff0c \0 \0 \0 \0 001 377 377 377 377 \0 \0 \0 \f 377 377 377 0001040 00ff 004e 006f 0064 0065 002d 0033 377 \0 N \0 o \0 d \0 e \0 - \0 3 \0 \0 \0 0001060 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0001220 0100 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 377 377 377 377 \0 \0 0001240 0c00 4e00 6f00 6400 6500 2d00 \0 \f 377 377 377 377 \0 N \0 o \0 d \0 e \0 - 0001260 3400 \0 4 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 0001300 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * 0001420 ff01 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 001 377 0001440 00ff ff0c 00ff 004e 006f 377 377 377 \0 \0 \0 \f 377 377 377 377 \0 N \0 o \0 0001460 0064 0065 002d 0035 d \0 e \0 - \0 5 \0 \0 \0 \0 \0 \0 \0 \0 \0 0001500 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 \0 * On Wed, Jun 2, 2010 at 3:50 PM, Biren Gandhi biren.gan...@gmail.com wrote: There is only 1 property - n (to store name of the node) - used as follows: Node node = graphDb.createNode(); node.setProperty( NAME_KEY, username ); And the values of username are Node-1, Node-2 etc. On Wed, Jun 2, 2010 at 3:14 PM, Mattias Persson matt...@neotechnology.com wrote: Only 4,4mb out of those 80 is consumed by nodes so you must be storing some properties somewhere. Would you mind sharing your code so that it would be easier to get a better insight into your problem? 2010/6/2, Biren Gandhi biren.gan...@gmail.com: Thanks. Big transactions were indeed problematic. Splitting them down into smaller chunks did the trick. I'm still disappointed by the on-disk size of a minimal node without any relationships or attributes. For 500K nodes, it is taking 80MB space (160 byes/node) and for 1M objects it is consuming 160MB (again 160 byes/node). Is this normal? 4.0Kactive_tx_log 12K lucene 12K lucene-fulltext 4.0Kneostore 4.0Kneostore.id 4.4Mneostore.nodestore.db 4.0Kneostore.nodestore.db.id 12M neostore.propertystore.db 4.0Kneostore.propertystore.db.arrays 4.0Kneostore.propertystore.db.arrays.id 4.0Kneostore.propertystore.db.id 4.0Kneostore.propertystore.db.index 4.0Kneostore.propertystore.db.index.id 4.0Kneostore.propertystore.db.index.keys 4.0Kneostore.propertystore.db.index.keys.id 64M neostore.propertystore.db.strings 4.0Kneostore.propertystore.db.strings.id 4.0Kneostore.relationshipstore.db 4.0Kneostore.relationshipstore.db.id 4.0Kneostore.relationshiptypestore.db 4.0Kneostore.relationshiptypestore.db.id 4.0K
[Neo4j] Node creation limit
While trying to perform a create-only stress test for nodes, i'm constantly getting Out of Memory error even while running with these params (with default config - as no searching/optimizations are being exercised just yet): EXTRA_JVM_ARGUMENTS=-d64 -server -Xms256m -Xmx1024m Able to create 200K nodes (without any attributes - bare bone nodes only) successfully, but anything above that Java gives up. One strange observation is that the disk size gets capped at 116K in all of the cases while tying to create 100K, 200K, 300K nodes on a fresh DB. Is there any limit on number of nodes that can be created in a neo4j instance? Any other tips? ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
Re: [Neo4j] Node creation limit
Correction - disk size 116K is applicable only in failure cases. Here are the numbers for 100K node inserts (takes up 17MB): 4.0Kactive_tx_log 12K lucene 12K lucene-fulltext 4.0Kneostore 4.0Kneostore.id 884Kneostore.nodestore.db 4.0Kneostore.nodestore.db.id 2.4Mneostore.propertystore.db 4.0Kneostore.propertystore.db.arrays 4.0Kneostore.propertystore.db.arrays.id 4.0Kneostore.propertystore.db.id 4.0Kneostore.propertystore.db.index 4.0Kneostore.propertystore.db.index.id 4.0Kneostore.propertystore.db.index.keys 4.0Kneostore.propertystore.db.index.keys.id 13M neostore.propertystore.db.strings 4.0Kneostore.propertystore.db.strings.id 4.0Kneostore.relationshipstore.db 4.0Kneostore.relationshipstore.db.id 4.0Kneostore.relationshiptypestore.db 4.0Kneostore.relationshiptypestore.db.id 4.0Kneostore.relationshiptypestore.db.names 4.0Kneostore.relationshiptypestore.db.names.id 4.0Knioneo_logical.log.active 4.0Ktm_tx_log.1 17M total On Tue, Jun 1, 2010 at 6:50 PM, Biren Gandhi biren.gan...@gmail.com wrote: While trying to perform a create-only stress test for nodes, i'm constantly getting Out of Memory error even while running with these params (with default config - as no searching/optimizations are being exercised just yet): EXTRA_JVM_ARGUMENTS=-d64 -server -Xms256m -Xmx1024m Able to create 200K nodes (without any attributes - bare bone nodes only) successfully, but anything above that Java gives up. One strange observation is that the disk size gets capped at 116K in all of the cases while tying to create 100K, 200K, 300K nodes on a fresh DB. Is there any limit on number of nodes that can be created in a neo4j instance? Any other tips? ___ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
