On Thu, Jan 30, 2014 at 11:33:27AM +0900, Hitoshi Mitake wrote: > At Thu, 30 Jan 2014 10:17:16 +0800, > Liu Yuan wrote: > > > > On Wed, Jan 29, 2014 at 05:19:09PM +0900, Hitoshi Mitake wrote: > > > At Wed, 29 Jan 2014 16:14:56 +0800, > > > Liu Yuan wrote: > > > > > > > > On Wed, Jan 29, 2014 at 05:01:52PM +0900, Hitoshi Mitake wrote: > > > > > At Wed, 29 Jan 2014 15:53:57 +0800, > > > > > Liu Yuan wrote: > > > > > > > > > > > > On Wed, Jan 29, 2014 at 03:32:34PM +0800, Liu Yuan wrote: > > > > > > > On Wed, Jan 29, 2014 at 04:28:35PM +0900, Hitoshi Mitake wrote: > > > > > > > > At Tue, 28 Jan 2014 18:01:42 +0800, > > > > > > > > Liu Yuan wrote: > > > > > > > > > > > > > > > > > > Rationale for multi-threaded recovery: > > > > > > > > > > > > > > > > > > 1. If one node is added, we find that all the VMs on other > > > > > > > > > nodes will get > > > > > > > > > noticeably affected until 50% data is transferred to the > > > > > > > > > new node. > > > > > > > > > > > > > > > > > > 2. For node failure, we might not have problems of running VM > > > > > > > > > but the > > > > > > > > > recovery process boost will benefit IO operation of VM > > > > > > > > > with less > > > > > > > > > chances to be blocked for write and also improve > > > > > > > > > reliability. > > > > > > > > > > > > > > > > > > 3. For disk failure in node, this is similar to adding a > > > > > > > > > node. All > > > > > > > > > the data on the broken disk will be recovered on other > > > > > > > > > disks in > > > > > > > > > this node. Speedy recoery not only improve data > > > > > > > > > reliability but > > > > > > > > > also cause less writing blocking on the lost data. > > > > > > > > > > > > > > > > > > Our oid scheduling algorithm is intact and simply add > > > > > > > > > multi-threading onto top > > > > > > > > > of current recovery algorithm with minimal changes. > > > > > > > > > > > > > > > > > > - we still have ->oids array to denote oids to be recovered > > > > > > > > > - we start up 2 * nr_disks threads for recovery > > > > > > > > > - the tricky part is that we need to wait all the running > > > > > > > > > threads to > > > > > > > > > completion before start next recovery events for multiple > > > > > > > > > nodes/disks events > > > > > > > > > > > > > > > > > > This patch passes "./check -g md -md" on my local box > > > > > > > > > > > > > > > > On my box, at least 32 and 33 failed. I'm seeking the root > > > > > > > > cause now > > > > > > > > but this patch seems to be a little bit dangerous. > > > > > > > > > > > > > > > > > > > > > > Yes, this shouldn't go to stable-0.8, but master is okay and at > > > > > > > least we need to > > > > > > > pass all the tests before it can goto master. > > > > > > > > > > > > 32 and 33 isn't md-ready tests. Please use > > > > > > > > > > > > ./check -g md -md > > > > > > > > > > > > to test this patch. > > > > > > > > > > 32 and 33 are tests for recovery. So I think we shouldn't exclude them > > > > > for testing your patch, no? > > > > > > > > I run 'sudo tests/functional/check 32 33' several times and no failure. > > > > > > On my environment, 32 and 33 fail every time I run (sudo DRIVER=local > > > ./check 32 33). > > > > > > Below is 32.out.bad and 33.out.bad: > > > > > > 32.out.bad: > > > QA output created by 032 > > > using backend plain store > > > 9c7766570b3be3aff2724f587c2f4107 - > > > STORE/1/obj/807c2b2500000000 > > > STORE/2/obj/807c2b2500000000 > > > STORE/4/obj/807c2b2500000000 > > > STORE/5/obj/807c2b2500000000 > > > STORE/6/obj/807c2b2500000000 > > > STORE/7/obj/807c2b2500000000 > > > STORE/0/obj/007c2b2500000000 > > > STORE/1/obj/007c2b2500000000 > > > STORE/4/obj/007c2b2500000000 > > > STORE/5/obj/007c2b2500000000 > > > STORE/6/obj/007c2b2500000000 > > > STORE/7/obj/007c2b2500000000 > > > STORE/0/obj/007c2b2500000001 > > > STORE/1/obj/007c2b2500000001 > > > STORE/2/obj/007c2b2500000001 > > > STORE/5/obj/007c2b2500000001 > > > STORE/6/obj/007c2b2500000001 > > > STORE/7/obj/007c2b2500000001 > > > STORE/0/obj/007c2b2500000002 > > > STORE/2/obj/007c2b2500000002 > > > STORE/4/obj/007c2b2500000002 > > > STORE/5/obj/007c2b2500000002 > > > STORE/6/obj/007c2b2500000002 > > > STORE/7/obj/007c2b2500000002 > > > STORE/0/obj/007c2b2500000003 > > > STORE/1/obj/007c2b2500000003 > > > STORE/2/obj/007c2b2500000003 > > > STORE/5/obj/007c2b2500000003 > > > STORE/6/obj/007c2b2500000003 > > > STORE/7/obj/007c2b2500000003 > > > STORE/0/obj/007c2b2500000004 > > > STORE/1/obj/007c2b2500000004 > > > STORE/2/obj/007c2b2500000004 > > > STORE/4/obj/007c2b2500000004 > > > STORE/5/obj/007c2b2500000004 > > > STORE/6/obj/007c2b2500000004 > > > STORE/1/obj/007c2b2500000005 > > > STORE/2/obj/007c2b2500000005 > > > STORE/4/obj/007c2b2500000005 > > > STORE/5/obj/007c2b2500000005 > > > STORE/6/obj/007c2b2500000005 > > > STORE/7/obj/007c2b2500000005 > > > STORE/0/obj/007c2b2500000006 > > > STORE/1/obj/007c2b2500000006 > > > STORE/4/obj/007c2b2500000006 > > > STORE/5/obj/007c2b2500000006 > > > STORE/6/obj/007c2b2500000006 > > > STORE/7/obj/007c2b2500000006 > > > STORE/0/obj/007c2b2500000007 > > > STORE/1/obj/007c2b2500000007 > > > STORE/2/obj/007c2b2500000007 > > > STORE/4/obj/007c2b2500000007 > > > STORE/5/obj/007c2b2500000007 > > > STORE/6/obj/007c2b2500000007 > > > STORE/0/obj/007c2b2500000008 > > > STORE/1/obj/007c2b2500000008 > > > STORE/2/obj/007c2b2500000008 > > > STORE/4/obj/007c2b2500000008 > > > STORE/5/obj/007c2b2500000008 > > > STORE/6/obj/007c2b2500000008 > > > STORE/1/obj/007c2b2500000009 > > > STORE/2/obj/007c2b2500000009 > > > STORE/4/obj/007c2b2500000009 > > > STORE/5/obj/007c2b2500000009 > > > STORE/6/obj/007c2b2500000009 > > > STORE/7/obj/007c2b2500000009 > > > STORE/0/obj/007c2b250000000a > > > STORE/1/obj/007c2b250000000a > > > STORE/2/obj/007c2b250000000a > > > STORE/4/obj/007c2b250000000a > > > STORE/5/obj/007c2b250000000a > > > STORE/6/obj/007c2b250000000a > > > STORE/0/obj/007c2b250000000b > > > STORE/1/obj/007c2b250000000b > > > STORE/2/obj/007c2b250000000b > > > STORE/4/obj/007c2b250000000b > > > STORE/5/obj/007c2b250000000b > > > STORE/6/obj/007c2b250000000b > > > STORE/0/obj/007c2b250000000c > > > STORE/1/obj/007c2b250000000c > > > STORE/4/obj/007c2b250000000c > > > STORE/5/obj/007c2b250000000c > > > STORE/6/obj/007c2b250000000c > > > STORE/7/obj/007c2b250000000c > > > STORE/1/obj/007c2b250000000d > > > STORE/2/obj/007c2b250000000d > > > STORE/3/obj/007c2b250000000d > > > STORE/4/obj/007c2b250000000d > > > STORE/5/obj/007c2b250000000d > > > STORE/6/obj/007c2b250000000d > > > STORE/7/obj/007c2b250000000d > > > STORE/1/obj/007c2b250000000e > > > STORE/2/obj/007c2b250000000e > > > STORE/4/obj/007c2b250000000e > > > STORE/5/obj/007c2b250000000e > > > STORE/6/obj/007c2b250000000e > > > STORE/7/obj/007c2b250000000e > > > STORE/1/obj/007c2b250000000f > > > STORE/2/obj/007c2b250000000f > > > STORE/4/obj/007c2b250000000f > > > STORE/5/obj/007c2b250000000f > > > STORE/6/obj/007c2b250000000f > > > STORE/7/obj/007c2b250000000f > > > STORE/0/obj/007c2b2500000010 > > > STORE/2/obj/007c2b2500000010 > > > STORE/4/obj/007c2b2500000010 > > > STORE/5/obj/007c2b2500000010 > > > STORE/6/obj/007c2b2500000010 > > > STORE/7/obj/007c2b2500000010 > > > STORE/1/obj/007c2b2500000011 > > > STORE/2/obj/007c2b2500000011 > > > STORE/4/obj/007c2b2500000011 > > > STORE/5/obj/007c2b2500000011 > > > STORE/6/obj/007c2b2500000011 > > > STORE/7/obj/007c2b2500000011 > > > STORE/1/obj/007c2b2500000012 > > > STORE/2/obj/007c2b2500000012 > > > STORE/4/obj/007c2b2500000012 > > > STORE/5/obj/007c2b2500000012 > > > STORE/6/obj/007c2b2500000012 > > > STORE/7/obj/007c2b2500000012 > > > STORE/0/obj/007c2b2500000013 > > > STORE/1/obj/007c2b2500000013 > > > STORE/2/obj/007c2b2500000013 > > > STORE/4/obj/007c2b2500000013 > > > STORE/5/obj/007c2b2500000013 > > > STORE/7/obj/007c2b2500000013 > > > STORE/0/obj/007c2b2500000014 > > > STORE/1/obj/007c2b2500000014 > > > STORE/2/obj/007c2b2500000014 > > > STORE/4/obj/007c2b2500000014 > > > STORE/5/obj/007c2b2500000014 > > > STORE/6/obj/007c2b2500000014 > > > STORE/0/obj/007c2b2500000015 > > > STORE/1/obj/007c2b2500000015 > > > STORE/2/obj/007c2b2500000015 > > > STORE/4/obj/007c2b2500000015 > > > STORE/6/obj/007c2b2500000015 > > > STORE/7/obj/007c2b2500000015 > > > STORE/0/obj/007c2b2500000016 > > > STORE/1/obj/007c2b2500000016 > > > STORE/3/obj/007c2b2500000016 > > > STORE/4/obj/007c2b2500000016 > > > STORE/5/obj/007c2b2500000016 > > > STORE/6/obj/007c2b2500000016 > > > STORE/7/obj/007c2b2500000016 > > > STORE/0/obj/007c2b2500000017 > > > STORE/1/obj/007c2b2500000017 > > > STORE/2/obj/007c2b2500000017 > > > STORE/4/obj/007c2b2500000017 > > > STORE/5/obj/007c2b2500000017 > > > STORE/6/obj/007c2b2500000017 > > > STORE/1/obj/007c2b2500000018 > > > STORE/2/obj/007c2b2500000018 > > > STORE/4/obj/007c2b2500000018 > > > STORE/5/obj/007c2b2500000018 > > > STORE/6/obj/007c2b2500000018 > > > STORE/7/obj/007c2b2500000018 > > > > > > > > > > > > > > > > > > > > > > > > 007c2b2500000000.5 > > > 007c2b2500000001.5 > > > 007c2b2500000002.5 > > > 007c2b2500000003.5 > > > 007c2b2500000004.5 > > > 007c2b2500000005.5 > > > 007c2b2500000006.5 > > > 007c2b2500000007.5 > > > 007c2b2500000008.5 > > > 007c2b2500000009.5 > > > 007c2b250000000a.5 > > > 007c2b250000000b.5 > > > 007c2b250000000c.5 > > > 007c2b250000000d.5 > > > 007c2b250000000e.5 > > > 007c2b250000000f.5 > > > 007c2b2500000010.5 > > > 007c2b2500000011.5 > > > 007c2b2500000012.5 > > > 007c2b2500000013.5 > > > 007c2b2500000014.5 > > > 007c2b2500000015.5 > > > 007c2b2500000016.5 > > > 007c2b2500000017.5 > > > 007c2b2500000018.5 > > > 807c2b2500000000.5 > > > > For 032, I think this is timing issue because all the other output is good. > > I have no idea why you have so many xxxx.5. There is no directory for it? > > it should be in .stale directory. > > I executed 032 and the bad output differed from previous one: > > QA output created by 032 > using backend plain store > 9c7766570b3be3aff2724f587c2f4107 - > STORE/0/obj/807c2b2500000000 > STORE/2/obj/807c2b2500000000 > STORE/4/obj/807c2b2500000000 > STORE/5/obj/807c2b2500000000 > STORE/6/obj/807c2b2500000000 > STORE/7/obj/807c2b2500000000 > STORE/0/obj/007c2b2500000000 > STORE/2/obj/007c2b2500000000 > STORE/4/obj/007c2b2500000000 > STORE/5/obj/007c2b2500000000 > STORE/6/obj/007c2b2500000000 > STORE/7/obj/007c2b2500000000 > STORE/0/obj/007c2b2500000001 > STORE/2/obj/007c2b2500000001 > STORE/4/obj/007c2b2500000001 > STORE/5/obj/007c2b2500000001 > STORE/6/obj/007c2b2500000001 > STORE/7/obj/007c2b2500000001 > STORE/0/obj/007c2b2500000002 > STORE/2/obj/007c2b2500000002 > STORE/4/obj/007c2b2500000002 > STORE/5/obj/007c2b2500000002 > STORE/6/obj/007c2b2500000002 > STORE/7/obj/007c2b2500000002 > STORE/0/obj/007c2b2500000003 > STORE/1/obj/007c2b2500000003 > STORE/2/obj/007c2b2500000003 > STORE/4/obj/007c2b2500000003 > STORE/5/obj/007c2b2500000003 > STORE/6/obj/007c2b2500000003 > STORE/7/obj/007c2b2500000003 > STORE/0/obj/007c2b2500000004 > STORE/2/obj/007c2b2500000004 > STORE/4/obj/007c2b2500000004 > STORE/5/obj/007c2b2500000004 > STORE/6/obj/007c2b2500000004 > STORE/7/obj/007c2b2500000004 > STORE/0/obj/007c2b2500000005 > STORE/2/obj/007c2b2500000005 > STORE/4/obj/007c2b2500000005 > STORE/5/obj/007c2b2500000005 > STORE/6/obj/007c2b2500000005 > STORE/7/obj/007c2b2500000005 > STORE/0/obj/007c2b2500000006 > STORE/2/obj/007c2b2500000006 > STORE/4/obj/007c2b2500000006 > STORE/5/obj/007c2b2500000006 > STORE/6/obj/007c2b2500000006 > STORE/7/obj/007c2b2500000006 > STORE/0/obj/007c2b2500000007 > STORE/2/obj/007c2b2500000007 > STORE/4/obj/007c2b2500000007 > STORE/5/obj/007c2b2500000007 > STORE/6/obj/007c2b2500000007 > STORE/7/obj/007c2b2500000007 > STORE/0/obj/007c2b2500000008 > STORE/2/obj/007c2b2500000008 > STORE/4/obj/007c2b2500000008 > STORE/5/obj/007c2b2500000008 > STORE/6/obj/007c2b2500000008 > STORE/7/obj/007c2b2500000008 > STORE/0/obj/007c2b2500000009 > STORE/2/obj/007c2b2500000009 > STORE/4/obj/007c2b2500000009 > STORE/5/obj/007c2b2500000009 > STORE/6/obj/007c2b2500000009 > STORE/7/obj/007c2b2500000009 > STORE/0/obj/007c2b250000000a > STORE/1/obj/007c2b250000000a > STORE/2/obj/007c2b250000000a > STORE/4/obj/007c2b250000000a > STORE/5/obj/007c2b250000000a > STORE/6/obj/007c2b250000000a > STORE/7/obj/007c2b250000000a > STORE/0/obj/007c2b250000000b > STORE/2/obj/007c2b250000000b > STORE/4/obj/007c2b250000000b > STORE/5/obj/007c2b250000000b > STORE/6/obj/007c2b250000000b > STORE/7/obj/007c2b250000000b > STORE/0/obj/007c2b250000000c > STORE/2/obj/007c2b250000000c > STORE/4/obj/007c2b250000000c > STORE/5/obj/007c2b250000000c > STORE/6/obj/007c2b250000000c > STORE/7/obj/007c2b250000000c > STORE/0/obj/007c2b250000000d > STORE/1/obj/007c2b250000000d > STORE/2/obj/007c2b250000000d > STORE/3/obj/007c2b250000000d > STORE/4/obj/007c2b250000000d > STORE/5/obj/007c2b250000000d > STORE/6/obj/007c2b250000000d > STORE/7/obj/007c2b250000000d > STORE/0/obj/007c2b250000000e > STORE/2/obj/007c2b250000000e > STORE/4/obj/007c2b250000000e > STORE/5/obj/007c2b250000000e > STORE/6/obj/007c2b250000000e > STORE/7/obj/007c2b250000000e > STORE/0/obj/007c2b250000000f > STORE/2/obj/007c2b250000000f > STORE/4/obj/007c2b250000000f > STORE/5/obj/007c2b250000000f > STORE/6/obj/007c2b250000000f > STORE/7/obj/007c2b250000000f > STORE/0/obj/007c2b2500000010 > STORE/2/obj/007c2b2500000010 > STORE/4/obj/007c2b2500000010 > STORE/5/obj/007c2b2500000010 > STORE/6/obj/007c2b2500000010 > STORE/7/obj/007c2b2500000010 > STORE/0/obj/007c2b2500000011 > STORE/2/obj/007c2b2500000011 > STORE/4/obj/007c2b2500000011 > STORE/5/obj/007c2b2500000011 > STORE/6/obj/007c2b2500000011 > STORE/7/obj/007c2b2500000011 > STORE/0/obj/007c2b2500000012 > STORE/2/obj/007c2b2500000012 > STORE/4/obj/007c2b2500000012 > STORE/5/obj/007c2b2500000012 > STORE/6/obj/007c2b2500000012 > STORE/7/obj/007c2b2500000012 > STORE/0/obj/007c2b2500000013 > STORE/2/obj/007c2b2500000013 > STORE/4/obj/007c2b2500000013 > STORE/5/obj/007c2b2500000013 > STORE/6/obj/007c2b2500000013 > STORE/7/obj/007c2b2500000013 > STORE/0/obj/007c2b2500000014 > STORE/2/obj/007c2b2500000014 > STORE/4/obj/007c2b2500000014 > STORE/5/obj/007c2b2500000014 > STORE/6/obj/007c2b2500000014 > STORE/7/obj/007c2b2500000014 > STORE/0/obj/007c2b2500000015 > STORE/2/obj/007c2b2500000015 > STORE/4/obj/007c2b2500000015 > STORE/5/obj/007c2b2500000015 > STORE/6/obj/007c2b2500000015 > STORE/7/obj/007c2b2500000015 > STORE/0/obj/007c2b2500000016 > STORE/1/obj/007c2b2500000016 > STORE/2/obj/007c2b2500000016 > STORE/3/obj/007c2b2500000016 > STORE/4/obj/007c2b2500000016 > STORE/5/obj/007c2b2500000016 > STORE/6/obj/007c2b2500000016 > STORE/7/obj/007c2b2500000016 > STORE/0/obj/007c2b2500000017 > STORE/2/obj/007c2b2500000017 > STORE/4/obj/007c2b2500000017 > STORE/5/obj/007c2b2500000017 > STORE/6/obj/007c2b2500000017 > STORE/7/obj/007c2b2500000017 > STORE/0/obj/007c2b2500000018 > STORE/2/obj/007c2b2500000018 > STORE/4/obj/007c2b2500000018 > STORE/5/obj/007c2b2500000018 > STORE/6/obj/007c2b2500000018 > STORE/7/obj/007c2b2500000018 > > > > > > > > 007c2b2500000000.5 > 007c2b2500000000.7 > 007c2b2500000001.5 > 007c2b2500000001.7 > 007c2b2500000002.5 > 007c2b2500000002.7 > 007c2b2500000003.5 > 007c2b2500000003.7 > 007c2b2500000004.5 > 007c2b2500000004.7 > 007c2b2500000005.5 > 007c2b2500000005.7 > 007c2b2500000006.5 > 007c2b2500000006.7 > 007c2b2500000007.5 > 007c2b2500000007.7 > 007c2b2500000008.5 > 007c2b2500000008.7 > 007c2b2500000009.5 > 007c2b2500000009.7 > 007c2b250000000a.5 > 007c2b250000000a.7 > 007c2b250000000b.5 > 007c2b250000000b.7 > 007c2b250000000c.5 > 007c2b250000000c.7 > 007c2b250000000d.5 > 007c2b250000000d.7 > 007c2b250000000e.5 > 007c2b250000000e.7 > 007c2b250000000f.5 > 007c2b250000000f.7 > 007c2b2500000010.5 > 007c2b2500000010.7 > 007c2b2500000011.5 > 007c2b2500000011.7 > 007c2b2500000012.5 > 007c2b2500000012.7 > 007c2b2500000013.5 > 007c2b2500000013.7 > 007c2b2500000014.5 > 007c2b2500000014.7 > 007c2b2500000015.5 > 007c2b2500000015.7 > 007c2b2500000016.5 > 007c2b2500000016.7 > 007c2b2500000017.5 > 007c2b2500000017.7 > 007c2b2500000018.5 > 007c2b2500000018.7 > 807c2b2500000000.5 > 807c2b2500000000.7 > STORE/0/obj/.stale: > STORE/1/obj/.stale: > STORE/2/obj/.stale: > STORE/3/obj/.stale: > STORE/4/obj/.stale: > STORE/5/obj/.stale: > STORE/6/obj/.stale: > STORE/7/obj/.stale: > 9c7766570b3be3aff2724f587c2f4107 - > > This is an output of "tree -a /tmp/sheepdog/032" > > /tmp/sheepdog/032 > ├── 0 > │ ├── config > │ ├── epoch > │ │ ├── 00000001 > │ │ ├── 00000002 > │ │ ├── 00000003 > │ │ ├── 00000004 > │ │ ├── 00000005 > │ │ ├── 00000006 > │ │ ├── 00000007 > │ │ ├── 00000008 > │ │ ├── 00000009 > │ │ ├── 00000010 > │ │ ├── 00000011 > │ │ ├── 00000012 > │ │ ├── 00000013 > │ │ ├── 00000014 > │ │ ├── 00000015 > │ │ ├── 00000016 > │ │ └── 00000017 > │ ├── lock > │ ├── obj > │ │ ├── .stale > │ │ ├── 007c2b2500000000 > │ │ ├── 007c2b2500000001 > │ │ ├── 007c2b2500000002 > │ │ ├── 007c2b2500000003 > │ │ ├── 007c2b2500000004 > │ │ ├── 007c2b2500000005 > │ │ ├── 007c2b2500000006 > │ │ ├── 007c2b2500000007 > │ │ ├── 007c2b2500000008 > │ │ ├── 007c2b2500000009 > │ │ ├── 007c2b250000000a > │ │ ├── 007c2b250000000b > │ │ ├── 007c2b250000000c > │ │ ├── 007c2b250000000d > │ │ ├── 007c2b250000000e > │ │ ├── 007c2b250000000f > │ │ ├── 007c2b2500000010 > │ │ ├── 007c2b2500000011 > │ │ ├── 007c2b2500000012 > │ │ ├── 007c2b2500000013 > │ │ ├── 007c2b2500000014 > │ │ ├── 007c2b2500000015 > │ │ ├── 007c2b2500000016 > │ │ ├── 007c2b2500000017 > │ │ ├── 007c2b2500000018 > │ │ └── 807c2b2500000000 > │ ├── sheep.log > │ └── sock > ├── 1 > │ ├── config > │ ├── core > │ ├── epoch > │ │ ├── 00000001 > │ │ ├── 00000002 > │ │ ├── 00000003 > │ │ ├── 00000004 > │ │ ├── 00000005 > │ │ ├── 00000006 > │ │ ├── 00000007 > │ │ ├── 00000009 > │ │ ├── 00000010 > │ │ ├── 00000011 > │ │ ├── 00000012 > │ │ └── 00000013 > │ ├── gdb.txt > │ ├── lock > │ ├── obj > │ │ ├── .stale > │ │ │ ├── 007c2b2500000000.7 > │ │ │ ├── 007c2b2500000001.7 > │ │ │ ├── 007c2b2500000002.7 > │ │ │ ├── 007c2b2500000003.7 > │ │ │ ├── 007c2b2500000004.7 > │ │ │ ├── 007c2b2500000005.7 > │ │ │ ├── 007c2b2500000006.7 > │ │ │ ├── 007c2b2500000007.7 > │ │ │ ├── 007c2b2500000008.7 > │ │ │ ├── 007c2b2500000009.7 > │ │ │ ├── 007c2b250000000a.7 > │ │ │ ├── 007c2b250000000b.7 > │ │ │ ├── 007c2b250000000c.7 > │ │ │ ├── 007c2b250000000d.7 > │ │ │ ├── 007c2b250000000e.7 > │ │ │ ├── 007c2b250000000f.7 > │ │ │ ├── 007c2b2500000010.7 > │ │ │ ├── 007c2b2500000011.7 > │ │ │ ├── 007c2b2500000012.7 > │ │ │ ├── 007c2b2500000013.7 > │ │ │ ├── 007c2b2500000014.7 > │ │ │ ├── 007c2b2500000015.7 > │ │ │ ├── 007c2b2500000016.7 > │ │ │ ├── 007c2b2500000017.7 > │ │ │ ├── 007c2b2500000018.7 > │ │ │ └── 807c2b2500000000.7 > │ │ ├── 007c2b2500000003 > │ │ ├── 007c2b250000000a > │ │ ├── 007c2b250000000d > │ │ └── 007c2b2500000016 > │ ├── sheep.log > │ └── sock > ├── 2 > │ ├── config > │ ├── epoch > │ │ ├── 00000001 > │ │ ├── 00000002 > │ │ ├── 00000003 > │ │ ├── 00000004 > │ │ ├── 00000005 > │ │ ├── 00000006 > │ │ ├── 00000010 > │ │ ├── 00000011 > │ │ ├── 00000012 > │ │ ├── 00000013 > │ │ ├── 00000014 > │ │ ├── 00000015 > │ │ ├── 00000016 > │ │ └── 00000017 > │ ├── lock > │ ├── obj > │ │ ├── .stale > │ │ ├── 007c2b2500000000 > │ │ ├── 007c2b2500000001 > │ │ ├── 007c2b2500000002 > │ │ ├── 007c2b2500000003 > │ │ ├── 007c2b2500000004 > │ │ ├── 007c2b2500000005 > │ │ ├── 007c2b2500000006 > │ │ ├── 007c2b2500000007 > │ │ ├── 007c2b2500000008 > │ │ ├── 007c2b2500000009 > │ │ ├── 007c2b250000000a > │ │ ├── 007c2b250000000b > │ │ ├── 007c2b250000000c > │ │ ├── 007c2b250000000d > │ │ ├── 007c2b250000000e > │ │ ├── 007c2b250000000f > │ │ ├── 007c2b2500000010 > │ │ ├── 007c2b2500000011 > │ │ ├── 007c2b2500000012 > │ │ ├── 007c2b2500000013 > │ │ ├── 007c2b2500000014 > │ │ ├── 007c2b2500000015 > │ │ ├── 007c2b2500000016 > │ │ ├── 007c2b2500000017 > │ │ ├── 007c2b2500000018 > │ │ └── 807c2b2500000000 > │ ├── sheep.log > │ └── sock > ├── 3 > │ ├── config > │ ├── core > │ ├── epoch > │ │ ├── 00000001 > │ │ ├── 00000002 > │ │ ├── 00000003 > │ │ ├── 00000004 > │ │ ├── 00000005 > │ │ ├── 00000012 > │ │ ├── 00000013 > │ │ ├── 00000014 > │ │ └── 00000015 > │ ├── gdb.txt > │ ├── lock > │ ├── obj > │ │ ├── .stale > │ │ │ ├── 007c2b2500000000.5 > │ │ │ ├── 007c2b2500000001.5 > │ │ │ ├── 007c2b2500000002.5 > │ │ │ ├── 007c2b2500000003.5 > │ │ │ ├── 007c2b2500000004.5 > │ │ │ ├── 007c2b2500000005.5 > │ │ │ ├── 007c2b2500000006.5 > │ │ │ ├── 007c2b2500000007.5 > │ │ │ ├── 007c2b2500000008.5 > │ │ │ ├── 007c2b2500000009.5 > │ │ │ ├── 007c2b250000000a.5 > │ │ │ ├── 007c2b250000000b.5 > │ │ │ ├── 007c2b250000000c.5 > │ │ │ ├── 007c2b250000000d.5 > │ │ │ ├── 007c2b250000000e.5 > │ │ │ ├── 007c2b250000000f.5 > │ │ │ ├── 007c2b2500000010.5 > │ │ │ ├── 007c2b2500000011.5 > │ │ │ ├── 007c2b2500000012.5 > │ │ │ ├── 007c2b2500000013.5 > │ │ │ ├── 007c2b2500000014.5 > │ │ │ ├── 007c2b2500000015.5 > │ │ │ ├── 007c2b2500000016.5 > │ │ │ ├── 007c2b2500000017.5 > │ │ │ ├── 007c2b2500000018.5 > │ │ │ └── 807c2b2500000000.5 > │ │ ├── 007c2b250000000d > │ │ └── 007c2b2500000016 > │ ├── sheep.log > │ └── sock > ├── 4 > │ ├── config > │ ├── epoch > │ │ ├── 00000001 > │ │ ├── 00000002 > │ │ ├── 00000003 > │ │ ├── 00000004 > │ │ ├── 00000011 > │ │ ├── 00000012 > │ │ ├── 00000013 > │ │ ├── 00000014 > │ │ ├── 00000015 > │ │ ├── 00000016 > │ │ └── 00000017 > │ ├── lock > │ ├── obj > │ │ ├── .stale > │ │ ├── 007c2b2500000000 > │ │ ├── 007c2b2500000001 > │ │ ├── 007c2b2500000002 > │ │ ├── 007c2b2500000003 > │ │ ├── 007c2b2500000004 > │ │ ├── 007c2b2500000005 > │ │ ├── 007c2b2500000006 > │ │ ├── 007c2b2500000007 > │ │ ├── 007c2b2500000008 > │ │ ├── 007c2b2500000009 > │ │ ├── 007c2b250000000a > │ │ ├── 007c2b250000000b > │ │ ├── 007c2b250000000c > │ │ ├── 007c2b250000000d > │ │ ├── 007c2b250000000e > │ │ ├── 007c2b250000000f > │ │ ├── 007c2b2500000010 > │ │ ├── 007c2b2500000011 > │ │ ├── 007c2b2500000012 > │ │ ├── 007c2b2500000013 > │ │ ├── 007c2b2500000014 > │ │ ├── 007c2b2500000015 > │ │ ├── 007c2b2500000016 > │ │ ├── 007c2b2500000017 > │ │ ├── 007c2b2500000018 > │ │ └── 807c2b2500000000 > │ ├── sheep.log > │ └── sock > ├── 5 > │ ├── config > │ ├── epoch > │ │ ├── 00000001 > │ │ ├── 00000002 > │ │ ├── 00000003 > │ │ ├── 00000013 > │ │ ├── 00000014 > │ │ ├── 00000015 > │ │ ├── 00000016 > │ │ └── 00000017 > │ ├── lock > │ ├── obj > │ │ ├── .stale > │ │ ├── 007c2b2500000000 > │ │ ├── 007c2b2500000001 > │ │ ├── 007c2b2500000002 > │ │ ├── 007c2b2500000003 > │ │ ├── 007c2b2500000004 > │ │ ├── 007c2b2500000005 > │ │ ├── 007c2b2500000006 > │ │ ├── 007c2b2500000007 > │ │ ├── 007c2b2500000008 > │ │ ├── 007c2b2500000009 > │ │ ├── 007c2b250000000a > │ │ ├── 007c2b250000000b > │ │ ├── 007c2b250000000c > │ │ ├── 007c2b250000000d > │ │ ├── 007c2b250000000e > │ │ ├── 007c2b250000000f > │ │ ├── 007c2b2500000010 > │ │ ├── 007c2b2500000011 > │ │ ├── 007c2b2500000012 > │ │ ├── 007c2b2500000013 > │ │ ├── 007c2b2500000014 > │ │ ├── 007c2b2500000015 > │ │ ├── 007c2b2500000016 > │ │ ├── 007c2b2500000017 > │ │ ├── 007c2b2500000018 > │ │ └── 807c2b2500000000 > │ ├── sheep.log > │ └── sock > ├── 6 > │ ├── config > │ ├── epoch > │ │ ├── 00000001 > │ │ ├── 00000002 > │ │ ├── 00000014 > │ │ ├── 00000015 > │ │ ├── 00000016 > │ │ └── 00000017 > │ ├── lock > │ ├── obj > │ │ ├── .stale > │ │ ├── 007c2b2500000000 > │ │ ├── 007c2b2500000001 > │ │ ├── 007c2b2500000002 > │ │ ├── 007c2b2500000003 > │ │ ├── 007c2b2500000004 > │ │ ├── 007c2b2500000005 > │ │ ├── 007c2b2500000006 > │ │ ├── 007c2b2500000007 > │ │ ├── 007c2b2500000008 > │ │ ├── 007c2b2500000009 > │ │ ├── 007c2b250000000a > │ │ ├── 007c2b250000000b > │ │ ├── 007c2b250000000c > │ │ ├── 007c2b250000000d > │ │ ├── 007c2b250000000e > │ │ ├── 007c2b250000000f > │ │ ├── 007c2b2500000010 > │ │ ├── 007c2b2500000011 > │ │ ├── 007c2b2500000012 > │ │ ├── 007c2b2500000013 > │ │ ├── 007c2b2500000014 > │ │ ├── 007c2b2500000015 > │ │ ├── 007c2b2500000016 > │ │ ├── 007c2b2500000017 > │ │ ├── 007c2b2500000018 > │ │ └── 807c2b2500000000 > │ ├── sheep.log > │ └── sock > └── 7 > ├── config > ├── epoch > │ ├── 00000001 > │ ├── 00000015 > │ ├── 00000016 > │ └── 00000017 > ├── lock > ├── obj > │ ├── .stale > │ ├── 007c2b2500000000 > │ ├── 007c2b2500000001 > │ ├── 007c2b2500000002 > │ ├── 007c2b2500000003 > │ ├── 007c2b2500000004 > │ ├── 007c2b2500000005 > │ ├── 007c2b2500000006 > │ ├── 007c2b2500000007 > │ ├── 007c2b2500000008 > │ ├── 007c2b2500000009 > │ ├── 007c2b250000000a > │ ├── 007c2b250000000b > │ ├── 007c2b250000000c > │ ├── 007c2b250000000d > │ ├── 007c2b250000000e > │ ├── 007c2b250000000f > │ ├── 007c2b2500000010 > │ ├── 007c2b2500000011 > │ ├── 007c2b2500000012 > │ ├── 007c2b2500000013 > │ ├── 007c2b2500000014 > │ ├── 007c2b2500000015 > │ ├── 007c2b2500000016 > │ ├── 007c2b2500000017 > │ ├── 007c2b2500000018 > │ └── 807c2b2500000000 > ├── sheep.log > └── sock > > 32 directories, 331 files
This bad output means, our recvoery completion cleaning patch has something wrong because it failed to clean up the .stale (very likely a timing issue because .stale need time to clean up and we should 'ls .stale' after clean up work is done) Thanks Yuan -- sheepdog mailing list sheepdog@lists.wpkg.org http://lists.wpkg.org/mailman/listinfo/sheepdog