Hi, [ this is a possible duplicate, but it seems that messages sent by my other email account ([email protected] don't reach list recipients - if I'm wrong, let me know - so I'm sending it again with this address ]
I'm seeing strange things with bitcask expiration: When a new node joins a Ring, it gets data from the other nodes to rebalance. It seems that the starting date to count expiration of this data is the time when they were copied to the new node, *not* the date they were originally created in the Ring For instance, on a Ring with 2 weeks expiration: I write 1 key at T=0 in a Ring with 3 nodes. At T+7days I add a new node 4. This node gets the ownership of the key. It seems that this key will expire on node 4 only at T+21 instead of T+14 That's annoying because we have a lot of data that is constantly being written and that *must* be expired at proper time. Here is a graph (warning, it shows the free disk space, not the disk space used) of data expiration when adding new nodes. https://www.dropbox.com/s/1od5xo5wxsvk11j/render_1.png?dl=0 Is it a wrong configuration ? Here is my bitcask expiration configuration: multi_backend.bitcask_mult_events.storage_backend = bitcask multi_backend.bitcask_mult_events.bitcask.data_root = /var/lib/riak/bitcask_mult_events multi_backend.bitcask_mult_events.bitcask.expiry = 8d multi_backend.bitcask_mult_events.bitcask.expiry.grace_time = 1d multi_backend.bitcask_mult_events.bitcask.io_mode = erlang multi_backend.bitcask_mult_events.bitcask.max_file_size = 4GB multi_backend.bitcask_mult_events.bitcask.merge.policy = always multi_backend.bitcask_mult_events.bitcask.merge.thresholds.dead_bytes = 4GB multi_backend.bitcask_mult_events.bitcask.merge.thresholds.fragmentation = 100 multi_backend.bitcask_mult_events.bitcask.merge.triggers.dead_bytes = 4GB multi_backend.bitcask_mult_events.bitcask.merge.triggers.fragmentation = 100 Maybe the configuration is not right, let me know if you think so. What I try to do here is to limit disk usage by compacting a file only if it contains only expired keys. Our data is pure readonly. EDIT: on IRC I got this reply from EvilEngel: EvilEngel> Hey dams. Unfortunately that is how Bitcask expiry works. the timestamp is internal and not part of the object, so re-writing it restarts the TTL. This is worrying: it implies that any new node added to a cluster should have enough disk space to be able to store *twice* the amount of expirable data it is supposed to store. This is ( in my opinion ) an important fact that should be stressed in the documentation at least, and if possible, fixed. One way would be to have handoffs transfer transmit the original timestamp in addition to the data, so that it's properly set on the target node. Thanks, dams. _______________________________________________ riak-users mailing list [email protected] http://lists.basho.com/mailman/listinfo/riak-users_lists.basho.com
