On Mon, May 14, 2012 at 9:50 PM, Liu Yuan <[email protected]> wrote: > On 05/15/2012 01:31 AM, Shawn Moore wrote: > >> If I had to list them in order of importance for us, it's: >> >> 1. reclaim unused objects automatically > I want to mention that Farm doesn't have this issue. > > Kazum and I have tried to address this issue for Simple store, but > unfortunately no a simple yet efficient method coming out. This problem > is more difficult than it looks, due to independent node recovering and > it is very hard to reach agreement for a distributed system without > simple mechanism. (need paxos?) > > Thanks, > Yuan
Below is my testing about the reclaiming of space in farm. This was tested with latest source pulled today. Feel free to rename the thread if you want to. # collie cluster format -b farm -c 1 -H # collie node info Id Size Used Use% 0 63 GB 0.0 MB 0% 1 63 GB 0.0 MB 0% ... 42 63 GB 0.0 MB 0% 43 63 GB 0.0 MB 0% Total 2.7 TB 0.2 MB 0% Total virtual image size 0.0 MB # collie vdi create -P test01 128M # collie vdi list Name Id Size Used Shared Creation time VDI id Tag test01 1 128 MB 128 MB 0.0 MB 2012-05-16 14:30 40687a # - LIST ALL OBJECTS - [1] 14:32:57 [SUCCESS] 192.168.1.101 [2] 14:32:57 [SUCCESS] 192.168.1.102 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000018 [3] 14:32:57 [SUCCESS] 192.168.1.103 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000019 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000001d [4] 14:32:57 [SUCCESS] 192.168.1.104 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000000 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000016 -rw-r----- 1 root root 4198968 May 16 14:30 /sheep/obj/8040687a00000000 [5] 14:32:57 [SUCCESS] 192.168.1.105 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000012 [6] 14:32:57 [SUCCESS] 192.168.1.106 [7] 14:32:57 [SUCCESS] 192.168.1.107 [8] 14:32:57 [SUCCESS] 192.168.1.108 [9] 14:32:57 [SUCCESS] 192.168.1.109 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000006 [10] 14:32:57 [SUCCESS] 192.168.1.110 [11] 14:32:57 [SUCCESS] 192.168.1.111 [12] 14:32:57 [SUCCESS] 192.168.1.112 [13] 14:32:57 [SUCCESS] 192.168.1.113 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000007 [14] 14:32:57 [SUCCESS] 192.168.1.114 [15] 14:32:57 [SUCCESS] 192.168.1.115 [16] 14:32:57 [SUCCESS] 192.168.1.116 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000011 [17] 14:32:57 [SUCCESS] 192.168.1.117 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000002 [18] 14:32:57 [SUCCESS] 192.168.1.118 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000000c [19] 14:32:57 [SUCCESS] 192.168.1.119 [20] 14:32:57 [SUCCESS] 192.168.1.120 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000013 [21] 14:32:57 [SUCCESS] 192.168.1.121 [22] 14:32:57 [SUCCESS] 192.168.1.122 [23] 14:32:57 [SUCCESS] 192.168.1.124 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000000e [24] 14:32:57 [SUCCESS] 192.168.1.125 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000001c [25] 14:32:57 [SUCCESS] 192.168.1.126 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000001b [26] 14:32:57 [SUCCESS] 192.168.1.127 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000009 [27] 14:32:57 [SUCCESS] 192.168.1.128 [28] 14:32:57 [SUCCESS] 192.168.1.129 [29] 14:32:57 [SUCCESS] 192.168.1.131 [30] 14:32:57 [SUCCESS] 192.168.1.134 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000000d [31] 14:32:57 [SUCCESS] 192.168.1.133 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000000f [32] 14:32:57 [SUCCESS] 192.168.1.135 [33] 14:32:57 [SUCCESS] 192.168.1.136 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000005 [34] 14:32:58 [SUCCESS] 192.168.1.123 [35] 14:32:58 [SUCCESS] 192.168.1.132 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000004 [36] 14:32:58 [SUCCESS] 192.168.1.130 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000008 [37] 14:32:58 [SUCCESS] 192.168.1.138 [38] 14:32:58 [SUCCESS] 192.168.1.139 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000000a -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000000b -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000017 [39] 14:32:58 [SUCCESS] 192.168.1.137 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000001 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000001f [40] 14:32:58 [SUCCESS] 192.168.1.140 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000003 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000015 [41] 14:32:58 [SUCCESS] 192.168.1.141 [42] 14:32:58 [SUCCESS] 192.168.1.142 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000010 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000001a -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a0000001e [43] 14:32:58 [SUCCESS] 192.168.1.143 [44] 14:32:58 [SUCCESS] 192.168.1.144 -rw-r----- 1 root root 4194304 May 16 14:30 /sheep/obj/0040687a00000014 Add up all these sizes and you get 132 MB, which is 128MB (data) plus 4MB for 8040687a00000000 (not sure what is? Index?) file. Now if I delete the VDI I'm left with the 8040687a00000000 file, but all else is gone. [1] 14:36:18 [SUCCESS] 192.168.1.101 [2] 14:36:18 [SUCCESS] 192.168.1.102 [3] 14:36:18 [SUCCESS] 192.168.1.103 [4] 14:36:18 [SUCCESS] 192.168.1.104 -rw-r----- 1 root root 4198968 May 16 14:36 /sheep/obj/8040687a00000000 [5] 14:36:18 [SUCCESS] 192.168.1.105 ... [42] 14:36:18 [SUCCESS] 192.168.1.143 [43] 14:36:18 [SUCCESS] 192.168.1.144 [44] 14:36:23 [SUCCESS] 192.168.1.137 We can live with this, just wasted 4MB per deleted VDI unless it is re-created later with the same VDI name, then it's re-used. But when you involve snapshots, you can end up with a lot of left over "80" files that add up at 4MB a piece per deleted snapshot. If you create test01 again you get: Name Id Size Used Shared Creation time VDI id Tag test01 1 128 MB 128 MB 0.0 MB 2012-05-16 14:38 40687a Now create 100 snapshots: s test01 1 134217728 134217728 0 1337193484 40687a SNAP_0 s test01 2 134217728 0 134217728 1337193559 40687b SNAP_1 ... s test01 99 134217728 0 134217728 1337193613 4068dc SNAP_98 s test01 100 134217728 0 134217728 1337193613 4068dd SNAP_99 All the VDI id's increment by hex 1. When you delete these, you are left with: = test01 101 134217728 0 134217728 1337193614 4068de Even after you delete the VDI: # collie vdi list Name Id Size Used Shared Creation time VDI id Tag You are left with: [1] 14:45:57 [SUCCESS] 192.168.1.101 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688500000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068a600000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068c300000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068cb00000000 [2] 14:45:57 [SUCCESS] 192.168.1.102 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688e00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068ba00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068d400000000 [3] 14:45:57 [SUCCESS] 192.168.1.103 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068ab00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068c800000000 [4] 14:45:57 [SUCCESS] 192.168.1.104 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040687a00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689a00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068a500000000 [5] 14:45:57 [SUCCESS] 192.168.1.105 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068a200000000 [6] 14:45:57 [SUCCESS] 192.168.1.106 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689b00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068c700000000 -rw-r----- 1 root root 4198968 May 16 14:45 /sheep/obj/804068de00000000 [7] 14:45:57 [SUCCESS] 192.168.1.107 [8] 14:45:57 [SUCCESS] 192.168.1.108 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688300000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068a900000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068af00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068b600000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068d300000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068db00000000 [9] 14:45:57 [SUCCESS] 192.168.1.109 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040687e00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068b400000000 [10] 14:45:57 [SUCCESS] 192.168.1.110 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068cf00000000 [11] 14:45:57 [SUCCESS] 192.168.1.111 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040687d00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688f00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068bc00000000 [12] 14:45:57 [SUCCESS] 192.168.1.112 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689400000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068ce00000000 [13] 14:45:57 [SUCCESS] 192.168.1.113 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068a100000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068b200000000 [14] 14:45:57 [SUCCESS] 192.168.1.114 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040687f00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688000000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689f00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068a800000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068d100000000 [15] 14:45:57 [SUCCESS] 192.168.1.115 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688200000000 [16] 14:45:57 [SUCCESS] 192.168.1.116 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688900000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068b900000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068bf00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068c600000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068c900000000 [17] 14:45:57 [SUCCESS] 192.168.1.117 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040687c00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689700000000 [18] 14:45:57 [SUCCESS] 192.168.1.118 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068c200000000 [19] 14:45:57 [SUCCESS] 192.168.1.119 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068a000000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068a300000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068c500000000 [20] 14:45:57 [SUCCESS] 192.168.1.120 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688100000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688b00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689500000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689900000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068a400000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068b700000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068d500000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068da00000000 [21] 14:45:57 [SUCCESS] 192.168.1.121 [22] 14:45:57 [SUCCESS] 192.168.1.122 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068c000000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068c400000000 [23] 14:45:57 [SUCCESS] 192.168.1.123 [24] 14:45:57 [SUCCESS] 192.168.1.124 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040687b00000000 [25] 14:45:57 [SUCCESS] 192.168.1.125 [26] 14:45:57 [SUCCESS] 192.168.1.126 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689200000000 [27] 14:45:57 [SUCCESS] 192.168.1.127 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068aa00000000 [28] 14:45:57 [SUCCESS] 192.168.1.128 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689e00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068bd00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068dc00000000 [29] 14:45:57 [SUCCESS] 192.168.1.129 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689600000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068b500000000 [30] 14:45:57 [SUCCESS] 192.168.1.130 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068b000000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068ca00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068d800000000 [31] 14:45:57 [SUCCESS] 192.168.1.131 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688400000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688d00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068ae00000000 [32] 14:45:57 [SUCCESS] 192.168.1.133 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689000000000 [33] 14:45:57 [SUCCESS] 192.168.1.132 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688600000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688800000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688a00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689800000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068ad00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068be00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068c100000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068d600000000 [34] 14:45:57 [SUCCESS] 192.168.1.134 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689300000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068d000000000 [35] 14:45:57 [SUCCESS] 192.168.1.135 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688700000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068b300000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068cc00000000 [36] 14:45:57 [SUCCESS] 192.168.1.138 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068d900000000 [37] 14:45:57 [SUCCESS] 192.168.1.137 [38] 14:45:57 [SUCCESS] 192.168.1.136 [39] 14:45:57 [SUCCESS] 192.168.1.139 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040688c00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068a700000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068ac00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068b800000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068dd00000000 [40] 14:45:57 [SUCCESS] 192.168.1.140 [41] 14:45:57 [SUCCESS] 192.168.1.141 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689100000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689d00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068d200000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068d700000000 [42] 14:45:57 [SUCCESS] 192.168.1.142 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/8040689c00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068b100000000 [43] 14:45:57 [SUCCESS] 192.168.1.143 [44] 14:45:57 [SUCCESS] 192.168.1.144 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068bb00000000 -rw-r----- 1 root root 4198968 May 16 14:42 /sheep/obj/804068cd00000000 Which is ~404MB left over. Now this is with just one VDI and copies set to 1. In our real usage, we want 3 copies. Currently we take 4 snapshots per hour (*/15 min) per VDI with ZFS. With 4 snaps per hour per day per year and copies set to 3 this equals "(((4*24)*365)*3)/1024" ~ 102.6 GB per VDI per year wasted. We currently have what will be 140 VDI's. Even if you delete all snapshots, and then the main VDI, when you re-create the VDI it re-uses the initial 80 file, but new snapshots do not re-use snapshot 80 files. So I create 1 VDI and 100 snapshots. After deleting all snapshots and the VDI itself, I was left with 101 80 files. Then I re-create the VDI and keep the same number of 80 files (101). Now I create 20 new snapshots and the number of 80 files goes to 121. I then delete all snapshots and VDI and I am still left with 121 total files. I have not found a way to reclaim this space without formatting the cluster. If I just delete a "80" file I get: # collie vdi list Name Id Size Used Shared Creation time VDI id Tag Failed to read object 8040687a00000000 No object found Failed to read inode header Regards, Shawn -- sheepdog mailing list [email protected] http://lists.wpkg.org/mailman/listinfo/sheepdog
