I've posted about this earlier, see http://lists.basho.com/pipermail/riak-users_lists.basho.com/2013-September/013270.html
I would like to provide some more background information as I still see no
decrease in storage space used at all so far. I currently have a 6 node cluster
where I am using the multi-backend, the relevant part of app.config is below.
Smaller K/V items are stored in the default backend on a small but fast disk
(also leveldb), larger items ranging 1-6Mb per value are stored in a separate
LevelDB backend that is mounted on a 2TB SATA drive (side note: I used the
"filesystem" backend during development in the early Riak versions, hence the
<<"fs">> name). Using the bucket properties the backend is chosen and also for
the large items the n_val is set to 2. See the curl output below.
%% Riak KV config
{riak_kv, [
%% Storage_backend specifies the Erlang module defining the storage
%% mechanism that will be used on this node.
{storage_backend, riak_kv_multi_backend},
{multi_backend_default, <<"eleveldb">>},
{multi_backend, [
{<<"eleveldb">>, riak_kv_eleveldb_backend, [
{data_root, "/data/riak/leveldb"}
]},
{<<"fs">>, riak_kv_eleveldb_backend, [
{data_root, "/bigdata/riak/bigleveldb"}
]},
{<<"cache">>, riak_kv_memory_backend, [
{max_memory, 16}, %% 16Mb
{ttl, 86400} %% 1 Day in seconds
]}
]},
# curl http://localhost:8098/riak/avis
{"props":{"allow_mult":false,"backend":"fs","big_vclock":50,"chash_keyfun":{"mod":"riak_core_util","fun":"chash_std_keyfun"},"dw":"quorum","last_write_wins":false,"linkfun":{"mod":"riak_kv_wm_link_walker","fun":"mapreduce_linkfun"},"n_val":2,"name":"avis","old_vclock":86400,"postcommit":[],"precommit":[],"r":1,"rw":"quorum","small_vclock":10,"w":"quorum","young_vclock":20}}
In the last two weeks I have deleted over 400,000 of large values (stored in
/bigdata/riak/bigleveldb), placing only small meta-information in there (in
Ruby: robject.raw_data = '', robject.meta['info']='info'). So far I have seen
absolutely no decrease in data usage at all. Since it is a production system
new data is also coming in every day and I've deleted many months of older (>1
year) data in the last two weeks. I do see a few "Compacting" lines in the
LevelDB log files (see logging below) but the storage usage is only going up.
I got some pointers from Matthew (from Basho) and Jens, however I'm still
stuck. Maybe I should wait even longer but my disks are filling up so I'll have
to expand the cluster or install larger 4TB drives in all the nodes (one by
one, copying the data of course).
Can anyone, maybe especially Matthew, tell me if I need to be more patient or
that there is something I can actively do about this situation?
Thanks,
Timo
p.s. tail of LevelDB logs: # tail /bigdata/riak/bigleveldb/*/LOG
==>
/bigdata/riak/bigleveldb/1027618338748291114361965898003636498195577569280/LOG
<==
2013/09/24-23:36:05.257431 2b0980a4a940 Delete type=2 #103231
2013/09/24-23:36:05.323310 2b0980a4a940 Delete type=2 #103233
2013/09/24-23:36:05.504968 2b0980a4a940 Delete type=2 #103821
2013/09/24-23:36:05.949831 2b0980a4a940 Compacting 1@3 + 1@4 files
2013/09/24-23:36:16.909783 2b0980a4a940 Generated table #104017: 155 keys,
420484068 bytes
2013/09/24-23:36:21.798106 2b0980a4a940 Generated table #104018: 59 keys,
111461278 bytes
2013/09/24-23:36:21.798138 2b0980a4a940 Compacted 1@3 + 1@4 files => 531945346
bytes
2013/09/24-23:36:21.799208 2b0980a4a940 compacted to: files[ 0 5 47 293 1253
14319 0 ]
2013/09/24-23:36:21.820027 2b0980a4a940 Delete type=2 #103929
2013/09/24-23:36:22.846278 2b0980a4a940 Delete type=2 #103183
==>
/bigdata/riak/bigleveldb/1141798154164767904846628775559596109106197299200/LOG
<==
2013/09/25-07:06:51.920881 2b097acfe940 Delete type=2 #112230
2013/09/25-07:06:52.013155 2b097acfe940 Compacting 1@3 + 3@4 files
2013/09/25-07:07:06.047538 2b097acfe940 Generated table #112277: 197 keys,
422634227 bytes
2013/09/25-07:07:11.122818 2b097acfe940 Generated table #112278: 40 keys,
121542577 bytes
2013/09/25-07:07:11.122847 2b097acfe940 Compacted 1@3 + 3@4 files => 544176804
bytes
2013/09/25-07:07:11.123944 2b097acfe940 compacted to: files[ 0 7 51 282 1336
13911 0 ]
2013/09/25-07:07:11.148538 2b097acfe940 Delete type=2 #110275
2013/09/25-07:07:13.149601 2b097acfe940 Delete type=2 #107642
2013/09/25-07:07:13.622819 2b097acfe940 Delete type=2 #107609
2013/09/25-07:07:14.179580 2b097acfe940 Delete type=2 #107611
==>
/bigdata/riak/bigleveldb/114179815416476790484662877555959610910619729920/LOG
<==
2013/09/24-18:32:50.114531 2b0980a4a940 Delete type=2 #93671
2013/09/24-18:32:50.303141 2b0980a4a940 Delete type=2 #101508
2013/09/24-18:32:50.487194 2b0980a4a940 Delete type=2 #89442
2013/09/24-18:32:50.915524 2b0980a4a940 Delete type=2 #89773
2013/09/24-23:33:32.698558 2b098532c940 Level-0 table #108357: started
2013/09/24-23:33:33.064892 2b098532c940 Level-0 table #108357: 57458722 bytes,
331 keys OK
2013/09/24-23:33:33.657453 2b098532c940 Delete type=0 #108326
2013/09/25-06:37:42.526121 2b098532c940 Level-0 table #108359: started
2013/09/25-06:37:42.935994 2b098532c940 Level-0 table #108359: 55931165 bytes,
412 keys OK
2013/09/25-06:37:43.447568 2b098532c940 Delete type=0 #108356
==>
/bigdata/riak/bigleveldb/1255977969581244695331291653115555720016817029120/LOG
<==
2013/09/24-19:25:02.361168 2b0982b28940 Delete type=2 #102679
2013/09/24-21:57:44.215287 2b098492b940 Level-0 table #109548: started
2013/09/24-21:57:44.560724 2b098492b940 Level-0 table #109548: 44681848 bytes,
224 keys OK
2013/09/24-21:57:45.232747 2b098492b940 Delete type=0 #109512
2013/09/25-00:14:08.894855 2b098492b940 Level-0 table #109550: started
2013/09/25-00:14:09.114181 2b098492b940 Level-0 table #109550: 44591929 bytes,
203 keys OK
2013/09/25-00:14:09.714129 2b098492b940 Delete type=0 #109547
2013/09/25-05:56:00.394497 2b098492b940 Level-0 table #109552: started
2013/09/25-05:56:00.750631 2b098492b940 Level-0 table #109552: 46178500 bytes,
368 keys OK
2013/09/25-05:56:01.198618 2b098492b940 Delete type=0 #109549
==>
/bigdata/riak/bigleveldb/1370157784997721485815954530671515330927436759040/LOG
<==
2013/09/24-19:06:38.592678 2b0991618940 Generated table #109843: 197 keys,
248701376 bytes
2013/09/24-19:06:38.592698 2b0991618940 Compacted 1@2 + 3@3 files => 395861477
bytes
2013/09/24-19:06:38.593401 2b0991618940 compacted to: files[ 0 7 50 285 1396
14138 0 ]
2013/09/24-23:56:59.835858 2b099341b940 Level-0 table #109845: started
2013/09/24-23:57:00.120522 2b099341b940 Level-0 table #109845: 48290387 bytes,
253 keys OK
2013/09/24-23:57:00.773145 2b099341b940 Delete type=2 #109294
2013/09/24-23:57:01.322884 2b099341b940 Delete type=2 #109777
2013/09/24-23:57:01.407742 2b099341b940 Delete type=2 #109295
2013/09/24-23:57:02.185318 2b099341b940 Delete type=0 #109809
2013/09/24-23:57:02.592206 2b099341b940 Delete type=2 #109293
==>
/bigdata/riak/bigleveldb/228359630832953580969325755111919221821239459840/LOG
<==
2013/09/24-19:57:54.841673 2b0991618940 Compacted 1@3 + 1@4 files => 379994890
bytes
2013/09/24-19:57:54.842748 2b0991618940 compacted to: files[ 0 8 51 270 1221
13846 0 ]
2013/09/24-19:57:54.866534 2b0991618940 Delete type=2 #101969
2013/09/24-19:57:58.056087 2b0991618940 Delete type=2 #106525
2013/09/24-23:14:37.772560 2b099341b940 Level-0 table #106769: started
2013/09/24-23:14:38.158647 2b099341b940 Level-0 table #106769: 49340840 bytes,
164 keys OK
2013/09/24-23:14:38.563483 2b099341b940 Delete type=0 #106741
2013/09/25-06:44:41.739209 2b099341b940 Level-0 table #106771: started
2013/09/25-06:44:41.976788 2b099341b940 Level-0 table #106771: 51705699 bytes,
291 keys OK
2013/09/25-06:44:42.441905 2b099341b940 Delete type=0 #106768
==>
/bigdata/riak/bigleveldb/342539446249430371453988632667878832731859189760/LOG
<==
2013/09/25-00:15:07.250766 2b0980a4a940 Generated table #119499: 254 keys,
248480802 bytes
2013/09/25-00:15:07.250808 2b0980a4a940 Compacted 1@2 + 3@3 files => 269219364
bytes
2013/09/25-00:15:07.251918 2b0980a4a940 compacted to: files[ 0 7 43 270 1381
14278 0 ]
2013/09/25-07:00:41.323223 2b098532c940 Level-0 table #119501: started
2013/09/25-07:00:41.591647 2b098532c940 Level-0 table #119501: 46957115 bytes,
497 keys OK
2013/09/25-07:00:42.052639 2b098532c940 Delete type=2 #119141
2013/09/25-07:00:42.943180 2b098532c940 Delete type=2 #119139
2013/09/25-07:00:43.000400 2b098532c940 Delete type=0 #119449
2013/09/25-07:00:43.331502 2b098532c940 Delete type=2 #119414
2013/09/25-07:00:43.412299 2b098532c940 Delete type=2 #119140
==>
/bigdata/riak/bigleveldb/570899077082383952423314387779798054553098649600/LOG
<==
2013/09/24-18:12:10.641006 2b0991618940 Delete type=2 #92945
2013/09/24-19:46:46.133549 2b099341b940 Level-0 table #110509: started
2013/09/24-19:46:46.334122 2b099341b940 Level-0 table #110509: 34692200 bytes,
120 keys OK
2013/09/24-19:46:46.703185 2b099341b940 Delete type=0 #110491
2013/09/24-23:14:05.269672 2b099341b940 Level-0 table #110511: started
2013/09/24-23:14:05.523896 2b099341b940 Level-0 table #110511: 36141859 bytes,
256 keys OK
2013/09/24-23:14:05.871651 2b099341b940 Delete type=0 #110508
2013/09/25-05:59:12.975178 2b099341b940 Level-0 table #110513: started
2013/09/25-05:59:13.220227 2b099341b940 Level-0 table #110513: 34597843 bytes,
442 keys OK
2013/09/25-05:59:13.652574 2b099341b940 Delete type=0 #110510
==>
/bigdata/riak/bigleveldb/685078892498860742907977265335757665463718379520/LOG
<==
2013/09/24-22:00:19.553978 2b097d7da940 Compacted 1@2 + 4@3 files => 472157173
bytes
2013/09/24-22:00:19.554663 2b097d7da940 compacted to: files[ 0 8 47 261 1152
13977 0 ]
2013/09/24-22:00:19.565408 2b097d7da940 Delete type=2 #117373
2013/09/24-22:00:19.731409 2b097d7da940 Delete type=2 #121117
2013/09/24-22:00:20.466842 2b097d7da940 Delete type=2 #121787
2013/09/24-22:00:20.623605 2b097d7da940 Delete type=2 #121119
2013/09/24-22:00:21.318316 2b097d7da940 Delete type=2 #121495
2013/09/25-03:08:09.339865 2b098de02940 Level-0 table #121950: started
2013/09/25-03:08:09.661629 2b098de02940 Level-0 table #121950: 51046687 bytes,
333 keys OK
2013/09/25-03:08:10.297044 2b098de02940 Delete type=0 #121920
==>
/bigdata/riak/bigleveldb/799258707915337533392640142891717276374338109440/LOG
<==
2013/09/24-20:19:02.314812 2b097acfe940 Delete type=2 #111932
2013/09/24-20:19:02.740509 2b097acfe940 Delete type=2 #112083
2013/09/24-20:19:02.763467 2b097acfe940 Delete type=2 #112022
2013/09/24-20:19:03.012092 2b097acfe940 Delete type=2 #112021
2013/09/25-00:07:09.219094 2b097cd98940 Level-0 table #112092: started
2013/09/25-00:07:09.550132 2b097cd98940 Level-0 table #112092: 60145463 bytes,
350 keys OK
2013/09/25-00:07:10.245902 2b097cd98940 Delete type=0 #112073
2013/09/25-05:53:47.104078 2b097cd98940 Level-0 table #112094: started
2013/09/25-05:53:47.556040 2b097cd98940 Level-0 table #112094: 60746639 bytes,
451 keys OK
2013/09/25-05:53:48.120232 2b097cd98940 Delete type=0 #112091
==>
/bigdata/riak/bigleveldb/913438523331814323877303020447676887284957839360/LOG
<==
2013/09/25-06:16:03.591159 2b097d7da940 Delete type=2 #114611
2013/09/25-06:16:03.611863 2b097d7da940 Compacting 1@3 + 2@4 files
2013/09/25-06:16:11.030322 2b097d7da940 Generated table #114622: 73 keys,
217984826 bytes
2013/09/25-06:16:11.042952 2b097d7da940 Generated table #114623: 98 keys, 33748
bytes
2013/09/25-06:16:18.991480 2b097d7da940 Generated table #114624: 147 keys,
251249017 bytes
2013/09/25-06:16:18.991508 2b097d7da940 Compacted 1@3 + 2@4 files => 469267591
bytes
2013/09/25-06:16:18.993797 2b097d7da940 compacted to: files[ 0 9 38 266 1161
16339 0 ]
2013/09/25-06:16:19.019274 2b097d7da940 Delete type=2 #92924
2013/09/25-06:16:20.779619 2b097d7da940 Delete type=2 #109438
2013/09/25-06:16:21.368453 2b097d7da940 Delete type=2 #114425
smime.p7s
Description: S/MIME cryptographic signature
_______________________________________________ riak-users mailing list riak-users@lists.basho.com http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
