Hey guys, We do offline backups of our db on a semi-regular basis (every few days), where we (1) stop the running db, (2) copy its data directory and (3) restart the db.
A few times early on, we did running backups -- but not the proper "online" way -- where we simply copied the data directory while the db was still running. (We did this during times where we were confident no requests were hitting the db.) We noticed that every time we did the running backup, the number of properties the web admin reported -- and the space on disk of the db -- would jump up quite a bit. We stopped doing that recently. But even now, both these numbers have gotten quite a bit higher than we expect to, and strangely, they seem to differ highly between the running db and the copies. What could be causing all of this? Here are our current numbers: *Production* - 2,338 nodes - 4,473 rels - 114,231 props (higher than we would expect it to be, but not by an order of magnitude) - *1.39 GB!* <-- this is way unexpected, particularly since our db used to be in the ~10 KB ballpark, and we certainly haven't experienced hockey stick growth yet ;) The logical log only takes up 57 KB (0%) btw. *Local snapshot* - 2,338 nodes - 4,473 rels - *2,607,892 props!!!* <-- ??? - *1.37 GB!* <-- equally surprisingly high, but also interesting that it's less than the production db's size. 0 KB logical logs. I looked around the wiki and searched this mailing list but didn't find much clues here. But as requested on another thread, here's the output of `ls -lh data/graph.db/`: total 1474520 -rw-r--r-- 1 aseemk staff 11B Aug 30 00:46 active_tx_log drwxr-xr-x 52 aseemk staff 1.7K Aug 30 00:46 index/ -rw-r--r-- 1 aseemk staff 343B Aug 30 00:46 index.db -rw-r--r-- 1 aseemk staff 854K Aug 30 00:46 messages.log -rw-r--r-- 1 aseemk staff 36B Aug 30 00:46 neostore -rw-r--r-- 1 aseemk staff 9B Aug 30 00:46 neostore.id -rw-r--r-- 1 aseemk staff 26K Aug 30 00:46 neostore.nodestore.db -rw-r--r-- 1 aseemk staff 9B Aug 30 00:46 neostore.nodestore.db.id -rw-r--r-- 1 aseemk staff 62M Aug 30 00:46 neostore.propertystore.db -rw-r--r-- 1 aseemk staff 133B Aug 30 00:46 neostore.propertystore.db.arrays -rw-r--r-- 1 aseemk staff 9B Aug 30 00:46 neostore.propertystore.db.arrays.id -rw-r--r-- 1 aseemk staff 9B Aug 30 00:46 neostore.propertystore.db.id -rw-r--r-- 1 aseemk staff 1.0K Aug 30 00:46 neostore.propertystore.db.index -rw-r--r-- 1 aseemk staff 9B Aug 30 00:46 neostore.propertystore.db.index.id -rw-r--r-- 1 aseemk staff 4.0K Aug 30 00:46 neostore.propertystore.db.index.keys -rw-r--r-- 1 aseemk staff 9B Aug 30 00:46 neostore.propertystore.db.index.keys.id -rw-r--r-- 1 aseemk staff 69M Aug 30 00:46 neostore.propertystore.db.strings -rw-r--r-- 1 aseemk staff 9B Aug 30 00:46 neostore.propertystore.db.strings.id -rw-r--r-- 1 aseemk staff 144K Aug 30 00:46 neostore.relationshipstore.db -rw-r--r-- 1 aseemk staff 9B Aug 30 00:46 neostore.relationshipstore.db.id -rw-r--r-- 1 aseemk staff 55B Aug 30 00:46 neostore.relationshiptypestore.db -rw-r--r-- 1 aseemk staff 9B Aug 30 00:46 neostore.relationshiptypestore.db.id -rw-r--r-- 1 aseemk staff 602B Aug 30 00:46 neostore.relationshiptypestore.db.names -rw-r--r-- 1 aseemk staff 9B Aug 30 00:46 neostore.relationshiptypestore.db.names.id -rw-r--r-- 1 aseemk staff 16B Aug 30 00:46 nioneo_logical.log.1 -rw-r--r-- 1 aseemk staff 4B Aug 30 00:46 nioneo_logical.log.active -rw-r--r-- 1 aseemk staff 945K Aug 30 00:46 nioneo_logical.log.v0 -rw-r--r-- 1 aseemk staff 16B Aug 30 00:46 nioneo_logical.log.v1 -rw-r--r-- 1 aseemk staff 33K Aug 30 00:46 nioneo_logical.log.v10 -rw-r--r-- 1 aseemk staff 11K Aug 30 00:46 nioneo_logical.log.v11 -rw-r--r-- 1 aseemk staff 32K Aug 30 00:46 nioneo_logical.log.v12 -rw-r--r-- 1 aseemk staff 16B Aug 30 00:46 nioneo_logical.log.v13 -rw-r--r-- 1 aseemk staff 12M Aug 30 00:46 nioneo_logical.log.v14 -rw-r--r-- 1 aseemk staff 1.4M Aug 30 00:46 nioneo_logical.log.v15 -rw-r--r-- 1 aseemk staff 6.8M Aug 30 00:46 nioneo_logical.log.v16 -rw-r--r-- 1 aseemk staff 25M Aug 30 00:46 nioneo_logical.log.v17 -rw-r--r-- 1 aseemk staff 25M Aug 30 00:46 nioneo_logical.log.v18 -rw-r--r-- 1 aseemk staff 25M Aug 30 00:46 nioneo_logical.log.v19 -rw-r--r-- 1 aseemk staff 1.3M Aug 30 00:46 nioneo_logical.log.v2 -rw-r--r-- 1 aseemk staff 25M Aug 30 00:46 nioneo_logical.log.v20 -rw-r--r-- 1 aseemk staff 25M Aug 30 00:46 nioneo_logical.log.v21 -rw-r--r-- 1 aseemk staff 25M Aug 30 00:46 nioneo_logical.log.v22 -rw-r--r-- 1 aseemk staff 25M Aug 30 00:46 nioneo_logical.log.v23 -rw-r--r-- 1 aseemk staff 25M Aug 30 00:46 nioneo_logical.log.v24 -rw-r--r-- 1 aseemk staff 25M Aug 30 00:46 nioneo_logical.log.v25 -rw-r--r-- 1 aseemk staff 25M Aug 30 00:46 nioneo_logical.log.v26 -rw-r--r-- 1 aseemk staff 14M Aug 30 00:46 nioneo_logical.log.v27 -rw-r--r-- 1 aseemk staff 25M Aug 30 00:46 nioneo_logical.log.v28 -rw-r--r-- 1 aseemk staff 7.8M Aug 30 00:46 nioneo_logical.log.v29 -rw-r--r-- 1 aseemk staff 800K Aug 30 00:46 nioneo_logical.log.v3 -rw-r--r-- 1 aseemk staff 25M Aug 30 00:46 nioneo_logical.log.v30 -rw-r--r-- 1 aseemk staff 25M Aug 30 00:46 nioneo_logical.log.v31 -rw-r--r-- 1 aseemk staff 25M Aug 30 00:46 nioneo_logical.log.v32 -rw-r--r-- 1 aseemk staff 25M Aug 30 00:46 nioneo_logical.log.v33 -rw-r--r-- 1 aseemk staff 25M Aug 30 00:46 nioneo_logical.log.v34 -rw-r--r-- 1 aseemk staff 25M Aug 30 00:46 nioneo_logical.log.v35 -rw-r--r-- 1 aseemk staff 4.5M Aug 30 00:46 nioneo_logical.log.v36 -rw-r--r-- 1 aseemk staff 16B Aug 30 00:46 nioneo_logical.log.v37 -rw-r--r-- 1 aseemk staff 25M Aug 30 00:46 nioneo_logical.log.v38 -rw-r--r-- 1 aseemk staff 25M Aug 30 00:46 nioneo_logical.log.v39 -rw-r--r-- 1 aseemk staff 67K Aug 30 00:46 nioneo_logical.log.v4 -rw-r--r-- 1 aseemk staff 25M Aug 30 00:46 nioneo_logical.log.v40 -rw-r--r-- 1 aseemk staff 16M Aug 30 00:46 nioneo_logical.log.v41 -rw-r--r-- 1 aseemk staff 14M Aug 30 00:46 nioneo_logical.log.v42 -rw-r--r-- 1 aseemk staff 1.0M Aug 30 00:46 nioneo_logical.log.v43 -rw-r--r-- 1 aseemk staff 5.7M Aug 30 00:46 nioneo_logical.log.v44 -rw-r--r-- 1 aseemk staff 16B Aug 30 00:46 nioneo_logical.log.v5 -rw-r--r-- 1 aseemk staff 16B Aug 30 00:46 nioneo_logical.log.v6 -rw-r--r-- 1 aseemk staff 16B Aug 30 00:46 nioneo_logical.log.v7 -rw-r--r-- 1 aseemk staff 16B Aug 30 00:46 nioneo_logical.log.v8 -rw-r--r-- 1 aseemk staff 16B Aug 30 00:46 nioneo_logical.log.v9 -rw-r--r-- 1 aseemk staff 29K Aug 30 00:46 tm_tx_log.1 -rw-r--r-- 1 aseemk staff 0B Aug 30 00:46 tm_tx_log.2 Looking at these numbers, I suppose the logs do add up -- is there a way to prune/garbage collect old logs? -- but I'm also surprised at the size of property stores. The latter depends on the number of properties though, which I'm not sure is right either, even in production. (We would see the property count jump by a factor of 2-3 after each running backup.) Thanks in advance for any pointers! Aseem _______________________________________________ Neo4j mailing list User@lists.neo4j.org https://lists.neo4j.org/mailman/listinfo/user
