Re: [Neo4j] Neo4j Property Store

['Michael Hunger' via Neo4j]

Is it the actual store that's keeping the space or tx logs?

neo4j marks records as unused to reuse them but doesn't currently free
space on disk.

with my store-utils you can compact it but that's an offline tool.

Join our new Community Site & Forum 


On Mon, Jan 7, 2019 at 3:05 PM Vaibhav Aiyar  wrote:

> Hi,
>
> Is there anyway to compact the store without having to shutdown NEO4J. Or
> is there a way to configure NEO4J to actually free up space after a given
> time period
>
> On Thursday, May 14, 2015 at 5:05:00 AM UTC-5, Michael Hunger wrote:
>>
>> You can run a consistency check on your graph:
>>
>>
>> http://www.markhneedham.com/blog/2014/01/22/neo4j-backup-store-copy-and-consistency-check/
>>
>> You can use my StoryCopy tool to migrate/compact your store:
>>
>> https://github.com/jexp/store-utils
>>
>> HTH
>>
>> Michael
>>
>> Am 14.05.2015 um 08:42 schrieb Vaibhav Aiyar :
>>
>> Would that not be a waste of space? Why would it need 19G if all it needs
>> is to mark the blocks unused if there is no data?
>> We are concerned that the size may have increased due to some corruption
>> in the files.
>> We wanted to load the entire graph into memory to improve performance and
>> 19G is too large.
>>
>> On Wednesday, May 13, 2015 at 6:36:32 PM UTC-5, Michael Hunger wrote:
>>>
>>> I don't think disk space is freed, just the blocks marked as unused.
>>>
>>> And the restart will start using those blocks (for nodes and rels)
>>> For properties etc. the blocks will be reused directly.
>>>
>>> M
>>>
>>> Am 13.05.2015 um 16:28 schrieb Vaibhav Aiyar :
>>>
>>> Hi,
>>>
>>> When does NEO4j cleanup neostore.propertystore.db.strings store file.
>>>
>>> We see huge variations in the size of the files in our three
>>> environments even though the node count and the data is similar.
>>> Below are the sizes of this files in our three environments along with
>>> node counts
>>>
>>> DEV: 19GB - 55699 nodes
>>> TEST: 7.4GB - 184023 nodes
>>> PROD: 700 MB - 258845 nodes
>>>
>>> I deleted around 3000 Nodes is DEV which have the most number of
>>> properties and restarted NEO4j.
>>> I saw a reduction in the storage space for
>>> neostore.propertystore.db.strings.id but no change in
>>> neostore.propertystore.db.strings.
>>>
>>> --
>>> You received this message because you are subscribed to the Google
>>> Groups "Neo4j" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to neo4j+un...@googlegroups.com.
>>> For more options, visit https://groups.google.com/d/optout.
>>>
>>>
>>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "Neo4j" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to neo4j+un...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>> --
> You received this message because you are subscribed to the Google Groups
> "Neo4j" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to neo4j+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

-- 
You received this message because you are subscribed to the Google Groups 
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to neo4j+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Neo4j] Neo4j Property Store

[Vaibhav Aiyar]

Hi,

Is there anyway to compact the store without having to shutdown NEO4J. Or 
is there a way to configure NEO4J to actually free up space after a given 
time period

On Thursday, May 14, 2015 at 5:05:00 AM UTC-5, Michael Hunger wrote:
>
> You can run a consistency check on your graph:
>
>
> http://www.markhneedham.com/blog/2014/01/22/neo4j-backup-store-copy-and-consistency-check/
>
> You can use my StoryCopy tool to migrate/compact your store:
>
> https://github.com/jexp/store-utils
>
> HTH
>
> Michael
>
> Am 14.05.2015 um 08:42 schrieb Vaibhav Aiyar  >:
>
> Would that not be a waste of space? Why would it need 19G if all it needs 
> is to mark the blocks unused if there is no data?
> We are concerned that the size may have increased due to some corruption 
> in the files. 
> We wanted to load the entire graph into memory to improve performance and 
> 19G is too large.
>
> On Wednesday, May 13, 2015 at 6:36:32 PM UTC-5, Michael Hunger wrote:
>>
>> I don't think disk space is freed, just the blocks marked as unused.
>>
>> And the restart will start using those blocks (for nodes and rels)
>> For properties etc. the blocks will be reused directly.
>>
>> M
>>
>> Am 13.05.2015 um 16:28 schrieb Vaibhav Aiyar :
>>
>> Hi,
>>
>> When does NEO4j cleanup neostore.propertystore.db.strings store file.
>>
>> We see huge variations in the size of the files in our three environments 
>> even though the node count and the data is similar.
>> Below are the sizes of this files in our three environments along with 
>> node counts
>>
>> DEV: 19GB - 55699 nodes
>> TEST: 7.4GB - 184023 nodes
>> PROD: 700 MB - 258845 nodes
>>
>> I deleted around 3000 Nodes is DEV which have the most number of 
>> properties and restarted NEO4j. 
>> I saw a reduction in the storage space for 
>> neostore.propertystore.db.strings.id but no change in 
>> neostore.propertystore.db.strings.
>>
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Neo4j" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to neo4j+un...@googlegroups.com.
>> For more options, visit https://groups.google.com/d/optout.
>>
>>
>>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Neo4j" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to neo4j+un...@googlegroups.com .
> For more options, visit https://groups.google.com/d/optout.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to neo4j+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Neo4j] neo4j property store

[Michael Hunger]

Can you try to compact your store first on a copy? If you delete a lot of 
nodes, rels, properties and recreate them again, you'll have empty space in 
your store files.

Try this: https://github.com/jexp/store-utils/tree/23#store-copy 


Michael

> Am 28.12.2015 um 10:22 schrieb Rakesh Kumar :
> 
> Hi,
> When we run our application for months together, we would be ending up with 
> propertystore and nodestore of huge size.
> For ex: neostore.propertystore.db -> around 50GB , 
> neostore.relationshipstore.db -> around 15GB, neostore.nodestore.db -> around 
> 2-3 GB.
> Is there a way we can split these files (for ex: rollover) or would these be 
> single large files growing forever ?
> 
> Thanks.
> 
> On Friday, May 2, 2014 at 7:43:25 PM UTC+5:30, Michael Hunger wrote:
> Look at the file size on disk
> 
> Propertystore, stringstore and arraystore
> 
> Michael
> 
> Sent from mobile device
> 
> Am 02.05.2014 um 15:28 schrieb Alex Frieden >:
> 
>> Is there a way to check how big the neo4j propertystore is?  Thanks!
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Neo4j" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to neo4j+un...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout 
>> .
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Neo4j" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to neo4j+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to neo4j+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Neo4j] neo4j property store

[Rakesh Kumar]

Hi,
When we run our application for months together, we would be ending up with 
propertystore and nodestore of huge size.
For ex: neostore.propertystore.db -> around 50GB 
, neostore.relationshipstore.db -> around 15GB, neostore.nodestore.db -> 
around 2-3 GB.
Is there a way we can split these files (for ex: rollover) or would these 
be single large files growing forever ?

Thanks.

On Friday, May 2, 2014 at 7:43:25 PM UTC+5:30, Michael Hunger wrote:
>
> Look at the file size on disk
>
> Propertystore, stringstore and arraystore
>
> Michael
>
> Sent from mobile device
>
> Am 02.05.2014 um 15:28 schrieb Alex Frieden  >:
>
> Is there a way to check how big the neo4j propertystore is?  Thanks!
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Neo4j" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to neo4j+un...@googlegroups.com .
> For more options, visit https://groups.google.com/d/optout.
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to neo4j+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Neo4j] Neo4j Property Store

[Michael Hunger]

You can run a consistency check on your graph:

http://www.markhneedham.com/blog/2014/01/22/neo4j-backup-store-copy-and-consistency-check/
 


You can use my StoryCopy tool to migrate/compact your store:

https://github.com/jexp/store-utils 

HTH

Michael

> Am 14.05.2015 um 08:42 schrieb Vaibhav Aiyar :
> 
> Would that not be a waste of space? Why would it need 19G if all it needs is 
> to mark the blocks unused if there is no data?
> We are concerned that the size may have increased due to some corruption in 
> the files. 
> We wanted to load the entire graph into memory to improve performance and 19G 
> is too large.
> 
> On Wednesday, May 13, 2015 at 6:36:32 PM UTC-5, Michael Hunger wrote:
> I don't think disk space is freed, just the blocks marked as unused.
> 
> And the restart will start using those blocks (for nodes and rels)
> For properties etc. the blocks will be reused directly.
> 
> M
> 
>> Am 13.05.2015 um 16:28 schrieb Vaibhav Aiyar > >:
>> 
>> Hi,
>> 
>> When does NEO4j cleanup neostore.propertystore.db.strings store file.
>> 
>> We see huge variations in the size of the files in our three environments 
>> even though the node count and the data is similar.
>> Below are the sizes of this files in our three environments along with node 
>> counts
>> 
>> DEV: 19GB - 55699 nodes
>> TEST: 7.4GB - 184023 nodes
>> PROD: 700 MB - 258845 nodes
>> 
>> I deleted around 3000 Nodes is DEV which have the most number of properties 
>> and restarted NEO4j. 
>> I saw a reduction in the storage space for 
>> neostore.propertystore.db.strings.id 
>>  but no change in 
>> neostore.propertystore.db.strings.
>> 
>> -- 
>> You received this message because you are subscribed to the Google Groups 
>> "Neo4j" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to neo4j+un...@googlegroups.com .
>> For more options, visit https://groups.google.com/d/optout 
>> .
> 
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Neo4j" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to neo4j+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to neo4j+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Neo4j] Neo4j Property Store

[Vaibhav Aiyar]

Would that not be a waste of space? Why would it need 19G if all it needs 
is to mark the blocks unused if there is no data?
We are concerned that the size may have increased due to some corruption in 
the files. 
We wanted to load the entire graph into memory to improve performance and 
19G is too large.

On Wednesday, May 13, 2015 at 6:36:32 PM UTC-5, Michael Hunger wrote:
>
> I don't think disk space is freed, just the blocks marked as unused.
>
> And the restart will start using those blocks (for nodes and rels)
> For properties etc. the blocks will be reused directly.
>
> M
>
> Am 13.05.2015 um 16:28 schrieb Vaibhav Aiyar  >:
>
> Hi,
>
> When does NEO4j cleanup neostore.propertystore.db.strings store file.
>
> We see huge variations in the size of the files in our three environments 
> even though the node count and the data is similar.
> Below are the sizes of this files in our three environments along with 
> node counts
>
> DEV: 19GB - 55699 nodes
> TEST: 7.4GB - 184023 nodes
> PROD: 700 MB - 258845 nodes
>
> I deleted around 3000 Nodes is DEV which have the most number of 
> properties and restarted NEO4j. 
> I saw a reduction in the storage space for 
> neostore.propertystore.db.strings.id but no change in 
> neostore.propertystore.db.strings.
>
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Neo4j" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to neo4j+un...@googlegroups.com .
> For more options, visit https://groups.google.com/d/optout.
>
>
>

-- 
You received this message because you are subscribed to the Google Groups 
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to neo4j+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Neo4j] Neo4j Property Store

[Michael Hunger]

I don't think disk space is freed, just the blocks marked as unused.

And the restart will start using those blocks (for nodes and rels)
For properties etc. the blocks will be reused directly.

M

> Am 13.05.2015 um 16:28 schrieb Vaibhav Aiyar :
> 
> Hi,
> 
> When does NEO4j cleanup neostore.propertystore.db.strings store file.
> 
> We see huge variations in the size of the files in our three environments 
> even though the node count and the data is similar.
> Below are the sizes of this files in our three environments along with node 
> counts
> 
> DEV: 19GB - 55699 nodes
> TEST: 7.4GB - 184023 nodes
> PROD: 700 MB - 258845 nodes
> 
> I deleted around 3000 Nodes is DEV which have the most number of properties 
> and restarted NEO4j. 
> I saw a reduction in the storage space for 
> neostore.propertystore.db.strings.id but no change in 
> neostore.propertystore.db.strings.
> 
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Neo4j" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to neo4j+unsubscr...@googlegroups.com 
> .
> For more options, visit https://groups.google.com/d/optout 
> .

-- 
You received this message because you are subscribed to the Google Groups 
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to neo4j+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Neo4j] neo4j property store

[Stefan Armbruster]

In enterprise edition, the size of store files is exposed by JMX, see
http://docs.neo4j.org/chunked/stable/jmx-mxbeans.html#jmx-store-file-sizes.

You can use neo4j-shell to query them:
echo 'dbinfo -g "Store file sizes" ArrayStoreSize PropertyStoreSize
StringStoreSize' | bin/neo4j-shell

Cheers,
Stefan

2014-05-02 16:13 GMT+02:00 Michael Hunger :
> Look at the file size on disk
>
> Propertystore, stringstore and arraystore
>
> Michael
>
> Sent from mobile device
>
> Am 02.05.2014 um 15:28 schrieb Alex Frieden :
>
> Is there a way to check how big the neo4j propertystore is?  Thanks!
>
> --
> You received this message because you are subscribed to the Google Groups
> "Neo4j" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to neo4j+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>
> --
> You received this message because you are subscribed to the Google Groups
> "Neo4j" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to neo4j+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to neo4j+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Re: [Neo4j] neo4j property store

[Michael Hunger]

Look at the file size on disk

Propertystore, stringstore and arraystore

Michael

Sent from mobile device

Am 02.05.2014 um 15:28 schrieb Alex Frieden :

> Is there a way to check how big the neo4j propertystore is?  Thanks!
> -- 
> You received this message because you are subscribed to the Google Groups 
> "Neo4j" group.
> To unsubscribe from this group and stop receiving emails from it, send an 
> email to neo4j+unsubscr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

-- 
You received this message because you are subscribed to the Google Groups 
"Neo4j" group.
To unsubscribe from this group and stop receiving emails from it, send an email 
to neo4j+unsubscr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.