At Thu, 30 Jan 2014 13:51:02 +0800, Liu Yuan wrote: > > On Thu, Jan 30, 2014 at 02:42:56PM +0900, Hitoshi Mitake wrote: > > At Thu, 30 Jan 2014 10:44:06 +0800, > > Liu Yuan wrote: > > > > > > On Thu, Jan 30, 2014 at 11:33:27AM +0900, Hitoshi Mitake wrote: > > > > At Thu, 30 Jan 2014 10:17:16 +0800, > > > > Liu Yuan wrote: > > > > > > > > > > On Wed, Jan 29, 2014 at 05:19:09PM +0900, Hitoshi Mitake wrote: > > > > > > At Wed, 29 Jan 2014 16:14:56 +0800, > > > > > > Liu Yuan wrote: > > > > > > > > > > > > > > On Wed, Jan 29, 2014 at 05:01:52PM +0900, Hitoshi Mitake wrote: > > > > > > > > At Wed, 29 Jan 2014 15:53:57 +0800, > > > > > > > > Liu Yuan wrote: > > > > > > > > > > > > > > > > > > On Wed, Jan 29, 2014 at 03:32:34PM +0800, Liu Yuan wrote: > > > > > > > > > > On Wed, Jan 29, 2014 at 04:28:35PM +0900, Hitoshi Mitake > > > > > > > > > > wrote: > > > > > > > > > > > At Tue, 28 Jan 2014 18:01:42 +0800, > > > > > > > > > > > Liu Yuan wrote: > > > > > > > > > > > > > > > > > > > > > > > > Rationale for multi-threaded recovery: > > > > > > > > > > > > > > > > > > > > > > > > 1. If one node is added, we find that all the VMs on > > > > > > > > > > > > other nodes will get > > > > > > > > > > > > noticeably affected until 50% data is transferred to > > > > > > > > > > > > the new node. > > > > > > > > > > > > > > > > > > > > > > > > 2. For node failure, we might not have problems of > > > > > > > > > > > > running VM but the > > > > > > > > > > > > recovery process boost will benefit IO operation of > > > > > > > > > > > > VM with less > > > > > > > > > > > > chances to be blocked for write and also improve > > > > > > > > > > > > reliability. > > > > > > > > > > > > > > > > > > > > > > > > 3. For disk failure in node, this is similar to adding > > > > > > > > > > > > a node. All > > > > > > > > > > > > the data on the broken disk will be recovered on > > > > > > > > > > > > other disks in > > > > > > > > > > > > this node. Speedy recoery not only improve data > > > > > > > > > > > > reliability but > > > > > > > > > > > > also cause less writing blocking on the lost data. > > > > > > > > > > > > > > > > > > > > > > > > Our oid scheduling algorithm is intact and simply add > > > > > > > > > > > > multi-threading onto top > > > > > > > > > > > > of current recovery algorithm with minimal changes. > > > > > > > > > > > > > > > > > > > > > > > > - we still have ->oids array to denote oids to be > > > > > > > > > > > > recovered > > > > > > > > > > > > - we start up 2 * nr_disks threads for recovery > > > > > > > > > > > > - the tricky part is that we need to wait all the > > > > > > > > > > > > running threads to > > > > > > > > > > > > completion before start next recovery events for > > > > > > > > > > > > multiple nodes/disks events > > > > > > > > > > > > > > > > > > > > > > > > This patch passes "./check -g md -md" on my local box > > > > > > > > > > > > > > > > > > > > > > On my box, at least 32 and 33 failed. I'm seeking the > > > > > > > > > > > root cause now > > > > > > > > > > > but this patch seems to be a little bit dangerous. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Yes, this shouldn't go to stable-0.8, but master is okay > > > > > > > > > > and at least we need to > > > > > > > > > > pass all the tests before it can goto master. > > > > > > > > > > > > > > > > > > 32 and 33 isn't md-ready tests. Please use > > > > > > > > > > > > > > > > > > ./check -g md -md > > > > > > > > > > > > > > > > > > to test this patch. > > > > > > > > > > > > > > > > 32 and 33 are tests for recovery. So I think we shouldn't > > > > > > > > exclude them > > > > > > > > for testing your patch, no? > > > > > > > > > > > > > > I run 'sudo tests/functional/check 32 33' several times and no > > > > > > > failure. > > > > > > > > > > > > On my environment, 32 and 33 fail every time I run (sudo > > > > > > DRIVER=local > > > > > > ./check 32 33). > > > > > > > > > > > > Below is 32.out.bad and 33.out.bad: > > > > > > > > > > > > 32.out.bad: > > > > > > QA output created by 032 > > > > > > using backend plain store > > > > > > 9c7766570b3be3aff2724f587c2f4107 - > > > > > > STORE/1/obj/807c2b2500000000 > > > > > > STORE/2/obj/807c2b2500000000 > > > > > > STORE/4/obj/807c2b2500000000 > > > > > > STORE/5/obj/807c2b2500000000 > > > > > > STORE/6/obj/807c2b2500000000 > > > > > > STORE/7/obj/807c2b2500000000 > > > > > > STORE/0/obj/007c2b2500000000 > > > > > > STORE/1/obj/007c2b2500000000 > > > > > > STORE/4/obj/007c2b2500000000 > > > > > > STORE/5/obj/007c2b2500000000 > > > > > > STORE/6/obj/007c2b2500000000 > > > > > > STORE/7/obj/007c2b2500000000 > > > > > > STORE/0/obj/007c2b2500000001 > > > > > > STORE/1/obj/007c2b2500000001 > > > > > > STORE/2/obj/007c2b2500000001 > > > > > > STORE/5/obj/007c2b2500000001 > > > > > > STORE/6/obj/007c2b2500000001 > > > > > > STORE/7/obj/007c2b2500000001 > > > > > > STORE/0/obj/007c2b2500000002 > > > > > > STORE/2/obj/007c2b2500000002 > > > > > > STORE/4/obj/007c2b2500000002 > > > > > > STORE/5/obj/007c2b2500000002 > > > > > > STORE/6/obj/007c2b2500000002 > > > > > > STORE/7/obj/007c2b2500000002 > > > > > > STORE/0/obj/007c2b2500000003 > > > > > > STORE/1/obj/007c2b2500000003 > > > > > > STORE/2/obj/007c2b2500000003 > > > > > > STORE/5/obj/007c2b2500000003 > > > > > > STORE/6/obj/007c2b2500000003 > > > > > > STORE/7/obj/007c2b2500000003 > > > > > > STORE/0/obj/007c2b2500000004 > > > > > > STORE/1/obj/007c2b2500000004 > > > > > > STORE/2/obj/007c2b2500000004 > > > > > > STORE/4/obj/007c2b2500000004 > > > > > > STORE/5/obj/007c2b2500000004 > > > > > > STORE/6/obj/007c2b2500000004 > > > > > > STORE/1/obj/007c2b2500000005 > > > > > > STORE/2/obj/007c2b2500000005 > > > > > > STORE/4/obj/007c2b2500000005 > > > > > > STORE/5/obj/007c2b2500000005 > > > > > > STORE/6/obj/007c2b2500000005 > > > > > > STORE/7/obj/007c2b2500000005 > > > > > > STORE/0/obj/007c2b2500000006 > > > > > > STORE/1/obj/007c2b2500000006 > > > > > > STORE/4/obj/007c2b2500000006 > > > > > > STORE/5/obj/007c2b2500000006 > > > > > > STORE/6/obj/007c2b2500000006 > > > > > > STORE/7/obj/007c2b2500000006 > > > > > > STORE/0/obj/007c2b2500000007 > > > > > > STORE/1/obj/007c2b2500000007 > > > > > > STORE/2/obj/007c2b2500000007 > > > > > > STORE/4/obj/007c2b2500000007 > > > > > > STORE/5/obj/007c2b2500000007 > > > > > > STORE/6/obj/007c2b2500000007 > > > > > > STORE/0/obj/007c2b2500000008 > > > > > > STORE/1/obj/007c2b2500000008 > > > > > > STORE/2/obj/007c2b2500000008 > > > > > > STORE/4/obj/007c2b2500000008 > > > > > > STORE/5/obj/007c2b2500000008 > > > > > > STORE/6/obj/007c2b2500000008 > > > > > > STORE/1/obj/007c2b2500000009 > > > > > > STORE/2/obj/007c2b2500000009 > > > > > > STORE/4/obj/007c2b2500000009 > > > > > > STORE/5/obj/007c2b2500000009 > > > > > > STORE/6/obj/007c2b2500000009 > > > > > > STORE/7/obj/007c2b2500000009 > > > > > > STORE/0/obj/007c2b250000000a > > > > > > STORE/1/obj/007c2b250000000a > > > > > > STORE/2/obj/007c2b250000000a > > > > > > STORE/4/obj/007c2b250000000a > > > > > > STORE/5/obj/007c2b250000000a > > > > > > STORE/6/obj/007c2b250000000a > > > > > > STORE/0/obj/007c2b250000000b > > > > > > STORE/1/obj/007c2b250000000b > > > > > > STORE/2/obj/007c2b250000000b > > > > > > STORE/4/obj/007c2b250000000b > > > > > > STORE/5/obj/007c2b250000000b > > > > > > STORE/6/obj/007c2b250000000b > > > > > > STORE/0/obj/007c2b250000000c > > > > > > STORE/1/obj/007c2b250000000c > > > > > > STORE/4/obj/007c2b250000000c > > > > > > STORE/5/obj/007c2b250000000c > > > > > > STORE/6/obj/007c2b250000000c > > > > > > STORE/7/obj/007c2b250000000c > > > > > > STORE/1/obj/007c2b250000000d > > > > > > STORE/2/obj/007c2b250000000d > > > > > > STORE/3/obj/007c2b250000000d > > > > > > STORE/4/obj/007c2b250000000d > > > > > > STORE/5/obj/007c2b250000000d > > > > > > STORE/6/obj/007c2b250000000d > > > > > > STORE/7/obj/007c2b250000000d > > > > > > STORE/1/obj/007c2b250000000e > > > > > > STORE/2/obj/007c2b250000000e > > > > > > STORE/4/obj/007c2b250000000e > > > > > > STORE/5/obj/007c2b250000000e > > > > > > STORE/6/obj/007c2b250000000e > > > > > > STORE/7/obj/007c2b250000000e > > > > > > STORE/1/obj/007c2b250000000f > > > > > > STORE/2/obj/007c2b250000000f > > > > > > STORE/4/obj/007c2b250000000f > > > > > > STORE/5/obj/007c2b250000000f > > > > > > STORE/6/obj/007c2b250000000f > > > > > > STORE/7/obj/007c2b250000000f > > > > > > STORE/0/obj/007c2b2500000010 > > > > > > STORE/2/obj/007c2b2500000010 > > > > > > STORE/4/obj/007c2b2500000010 > > > > > > STORE/5/obj/007c2b2500000010 > > > > > > STORE/6/obj/007c2b2500000010 > > > > > > STORE/7/obj/007c2b2500000010 > > > > > > STORE/1/obj/007c2b2500000011 > > > > > > STORE/2/obj/007c2b2500000011 > > > > > > STORE/4/obj/007c2b2500000011 > > > > > > STORE/5/obj/007c2b2500000011 > > > > > > STORE/6/obj/007c2b2500000011 > > > > > > STORE/7/obj/007c2b2500000011 > > > > > > STORE/1/obj/007c2b2500000012 > > > > > > STORE/2/obj/007c2b2500000012 > > > > > > STORE/4/obj/007c2b2500000012 > > > > > > STORE/5/obj/007c2b2500000012 > > > > > > STORE/6/obj/007c2b2500000012 > > > > > > STORE/7/obj/007c2b2500000012 > > > > > > STORE/0/obj/007c2b2500000013 > > > > > > STORE/1/obj/007c2b2500000013 > > > > > > STORE/2/obj/007c2b2500000013 > > > > > > STORE/4/obj/007c2b2500000013 > > > > > > STORE/5/obj/007c2b2500000013 > > > > > > STORE/7/obj/007c2b2500000013 > > > > > > STORE/0/obj/007c2b2500000014 > > > > > > STORE/1/obj/007c2b2500000014 > > > > > > STORE/2/obj/007c2b2500000014 > > > > > > STORE/4/obj/007c2b2500000014 > > > > > > STORE/5/obj/007c2b2500000014 > > > > > > STORE/6/obj/007c2b2500000014 > > > > > > STORE/0/obj/007c2b2500000015 > > > > > > STORE/1/obj/007c2b2500000015 > > > > > > STORE/2/obj/007c2b2500000015 > > > > > > STORE/4/obj/007c2b2500000015 > > > > > > STORE/6/obj/007c2b2500000015 > > > > > > STORE/7/obj/007c2b2500000015 > > > > > > STORE/0/obj/007c2b2500000016 > > > > > > STORE/1/obj/007c2b2500000016 > > > > > > STORE/3/obj/007c2b2500000016 > > > > > > STORE/4/obj/007c2b2500000016 > > > > > > STORE/5/obj/007c2b2500000016 > > > > > > STORE/6/obj/007c2b2500000016 > > > > > > STORE/7/obj/007c2b2500000016 > > > > > > STORE/0/obj/007c2b2500000017 > > > > > > STORE/1/obj/007c2b2500000017 > > > > > > STORE/2/obj/007c2b2500000017 > > > > > > STORE/4/obj/007c2b2500000017 > > > > > > STORE/5/obj/007c2b2500000017 > > > > > > STORE/6/obj/007c2b2500000017 > > > > > > STORE/1/obj/007c2b2500000018 > > > > > > STORE/2/obj/007c2b2500000018 > > > > > > STORE/4/obj/007c2b2500000018 > > > > > > STORE/5/obj/007c2b2500000018 > > > > > > STORE/6/obj/007c2b2500000018 > > > > > > STORE/7/obj/007c2b2500000018 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 007c2b2500000000.5 > > > > > > 007c2b2500000001.5 > > > > > > 007c2b2500000002.5 > > > > > > 007c2b2500000003.5 > > > > > > 007c2b2500000004.5 > > > > > > 007c2b2500000005.5 > > > > > > 007c2b2500000006.5 > > > > > > 007c2b2500000007.5 > > > > > > 007c2b2500000008.5 > > > > > > 007c2b2500000009.5 > > > > > > 007c2b250000000a.5 > > > > > > 007c2b250000000b.5 > > > > > > 007c2b250000000c.5 > > > > > > 007c2b250000000d.5 > > > > > > 007c2b250000000e.5 > > > > > > 007c2b250000000f.5 > > > > > > 007c2b2500000010.5 > > > > > > 007c2b2500000011.5 > > > > > > 007c2b2500000012.5 > > > > > > 007c2b2500000013.5 > > > > > > 007c2b2500000014.5 > > > > > > 007c2b2500000015.5 > > > > > > 007c2b2500000016.5 > > > > > > 007c2b2500000017.5 > > > > > > 007c2b2500000018.5 > > > > > > 807c2b2500000000.5 > > > > > > > > > > For 032, I think this is timing issue because all the other output is > > > > > good. > > > > > I have no idea why you have so many xxxx.5. There is no directory for > > > > > it? > > > > > it should be in .stale directory. > > > > > > > > I executed 032 and the bad output differed from previous one: > > > > > > > > QA output created by 032 > > > > using backend plain store > > > > 9c7766570b3be3aff2724f587c2f4107 - > > > > STORE/0/obj/807c2b2500000000 > > > > STORE/2/obj/807c2b2500000000 > > > > STORE/4/obj/807c2b2500000000 > > > > STORE/5/obj/807c2b2500000000 > > > > STORE/6/obj/807c2b2500000000 > > > > STORE/7/obj/807c2b2500000000 > > > > STORE/0/obj/007c2b2500000000 > > > > STORE/2/obj/007c2b2500000000 > > > > STORE/4/obj/007c2b2500000000 > > > > STORE/5/obj/007c2b2500000000 > > > > STORE/6/obj/007c2b2500000000 > > > > STORE/7/obj/007c2b2500000000 > > > > STORE/0/obj/007c2b2500000001 > > > > STORE/2/obj/007c2b2500000001 > > > > STORE/4/obj/007c2b2500000001 > > > > STORE/5/obj/007c2b2500000001 > > > > STORE/6/obj/007c2b2500000001 > > > > STORE/7/obj/007c2b2500000001 > > > > STORE/0/obj/007c2b2500000002 > > > > STORE/2/obj/007c2b2500000002 > > > > STORE/4/obj/007c2b2500000002 > > > > STORE/5/obj/007c2b2500000002 > > > > STORE/6/obj/007c2b2500000002 > > > > STORE/7/obj/007c2b2500000002 > > > > STORE/0/obj/007c2b2500000003 > > > > STORE/1/obj/007c2b2500000003 > > > > STORE/2/obj/007c2b2500000003 > > > > STORE/4/obj/007c2b2500000003 > > > > STORE/5/obj/007c2b2500000003 > > > > STORE/6/obj/007c2b2500000003 > > > > STORE/7/obj/007c2b2500000003 > > > > STORE/0/obj/007c2b2500000004 > > > > STORE/2/obj/007c2b2500000004 > > > > STORE/4/obj/007c2b2500000004 > > > > STORE/5/obj/007c2b2500000004 > > > > STORE/6/obj/007c2b2500000004 > > > > STORE/7/obj/007c2b2500000004 > > > > STORE/0/obj/007c2b2500000005 > > > > STORE/2/obj/007c2b2500000005 > > > > STORE/4/obj/007c2b2500000005 > > > > STORE/5/obj/007c2b2500000005 > > > > STORE/6/obj/007c2b2500000005 > > > > STORE/7/obj/007c2b2500000005 > > > > STORE/0/obj/007c2b2500000006 > > > > STORE/2/obj/007c2b2500000006 > > > > STORE/4/obj/007c2b2500000006 > > > > STORE/5/obj/007c2b2500000006 > > > > STORE/6/obj/007c2b2500000006 > > > > STORE/7/obj/007c2b2500000006 > > > > STORE/0/obj/007c2b2500000007 > > > > STORE/2/obj/007c2b2500000007 > > > > STORE/4/obj/007c2b2500000007 > > > > STORE/5/obj/007c2b2500000007 > > > > STORE/6/obj/007c2b2500000007 > > > > STORE/7/obj/007c2b2500000007 > > > > STORE/0/obj/007c2b2500000008 > > > > STORE/2/obj/007c2b2500000008 > > > > STORE/4/obj/007c2b2500000008 > > > > STORE/5/obj/007c2b2500000008 > > > > STORE/6/obj/007c2b2500000008 > > > > STORE/7/obj/007c2b2500000008 > > > > STORE/0/obj/007c2b2500000009 > > > > STORE/2/obj/007c2b2500000009 > > > > STORE/4/obj/007c2b2500000009 > > > > STORE/5/obj/007c2b2500000009 > > > > STORE/6/obj/007c2b2500000009 > > > > STORE/7/obj/007c2b2500000009 > > > > STORE/0/obj/007c2b250000000a > > > > STORE/1/obj/007c2b250000000a > > > > STORE/2/obj/007c2b250000000a > > > > STORE/4/obj/007c2b250000000a > > > > STORE/5/obj/007c2b250000000a > > > > STORE/6/obj/007c2b250000000a > > > > STORE/7/obj/007c2b250000000a > > > > STORE/0/obj/007c2b250000000b > > > > STORE/2/obj/007c2b250000000b > > > > STORE/4/obj/007c2b250000000b > > > > STORE/5/obj/007c2b250000000b > > > > STORE/6/obj/007c2b250000000b > > > > STORE/7/obj/007c2b250000000b > > > > STORE/0/obj/007c2b250000000c > > > > STORE/2/obj/007c2b250000000c > > > > STORE/4/obj/007c2b250000000c > > > > STORE/5/obj/007c2b250000000c > > > > STORE/6/obj/007c2b250000000c > > > > STORE/7/obj/007c2b250000000c > > > > STORE/0/obj/007c2b250000000d > > > > STORE/1/obj/007c2b250000000d > > > > STORE/2/obj/007c2b250000000d > > > > STORE/3/obj/007c2b250000000d > > > > STORE/4/obj/007c2b250000000d > > > > STORE/5/obj/007c2b250000000d > > > > STORE/6/obj/007c2b250000000d > > > > STORE/7/obj/007c2b250000000d > > > > STORE/0/obj/007c2b250000000e > > > > STORE/2/obj/007c2b250000000e > > > > STORE/4/obj/007c2b250000000e > > > > STORE/5/obj/007c2b250000000e > > > > STORE/6/obj/007c2b250000000e > > > > STORE/7/obj/007c2b250000000e > > > > STORE/0/obj/007c2b250000000f > > > > STORE/2/obj/007c2b250000000f > > > > STORE/4/obj/007c2b250000000f > > > > STORE/5/obj/007c2b250000000f > > > > STORE/6/obj/007c2b250000000f > > > > STORE/7/obj/007c2b250000000f > > > > STORE/0/obj/007c2b2500000010 > > > > STORE/2/obj/007c2b2500000010 > > > > STORE/4/obj/007c2b2500000010 > > > > STORE/5/obj/007c2b2500000010 > > > > STORE/6/obj/007c2b2500000010 > > > > STORE/7/obj/007c2b2500000010 > > > > STORE/0/obj/007c2b2500000011 > > > > STORE/2/obj/007c2b2500000011 > > > > STORE/4/obj/007c2b2500000011 > > > > STORE/5/obj/007c2b2500000011 > > > > STORE/6/obj/007c2b2500000011 > > > > STORE/7/obj/007c2b2500000011 > > > > STORE/0/obj/007c2b2500000012 > > > > STORE/2/obj/007c2b2500000012 > > > > STORE/4/obj/007c2b2500000012 > > > > STORE/5/obj/007c2b2500000012 > > > > STORE/6/obj/007c2b2500000012 > > > > STORE/7/obj/007c2b2500000012 > > > > STORE/0/obj/007c2b2500000013 > > > > STORE/2/obj/007c2b2500000013 > > > > STORE/4/obj/007c2b2500000013 > > > > STORE/5/obj/007c2b2500000013 > > > > STORE/6/obj/007c2b2500000013 > > > > STORE/7/obj/007c2b2500000013 > > > > STORE/0/obj/007c2b2500000014 > > > > STORE/2/obj/007c2b2500000014 > > > > STORE/4/obj/007c2b2500000014 > > > > STORE/5/obj/007c2b2500000014 > > > > STORE/6/obj/007c2b2500000014 > > > > STORE/7/obj/007c2b2500000014 > > > > STORE/0/obj/007c2b2500000015 > > > > STORE/2/obj/007c2b2500000015 > > > > STORE/4/obj/007c2b2500000015 > > > > STORE/5/obj/007c2b2500000015 > > > > STORE/6/obj/007c2b2500000015 > > > > STORE/7/obj/007c2b2500000015 > > > > STORE/0/obj/007c2b2500000016 > > > > STORE/1/obj/007c2b2500000016 > > > > STORE/2/obj/007c2b2500000016 > > > > STORE/3/obj/007c2b2500000016 > > > > STORE/4/obj/007c2b2500000016 > > > > STORE/5/obj/007c2b2500000016 > > > > STORE/6/obj/007c2b2500000016 > > > > STORE/7/obj/007c2b2500000016 > > > > STORE/0/obj/007c2b2500000017 > > > > STORE/2/obj/007c2b2500000017 > > > > STORE/4/obj/007c2b2500000017 > > > > STORE/5/obj/007c2b2500000017 > > > > STORE/6/obj/007c2b2500000017 > > > > STORE/7/obj/007c2b2500000017 > > > > STORE/0/obj/007c2b2500000018 > > > > STORE/2/obj/007c2b2500000018 > > > > STORE/4/obj/007c2b2500000018 > > > > STORE/5/obj/007c2b2500000018 > > > > STORE/6/obj/007c2b2500000018 > > > > STORE/7/obj/007c2b2500000018 > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 007c2b2500000000.5 > > > > 007c2b2500000000.7 > > > > 007c2b2500000001.5 > > > > 007c2b2500000001.7 > > > > 007c2b2500000002.5 > > > > 007c2b2500000002.7 > > > > 007c2b2500000003.5 > > > > 007c2b2500000003.7 > > > > 007c2b2500000004.5 > > > > 007c2b2500000004.7 > > > > 007c2b2500000005.5 > > > > 007c2b2500000005.7 > > > > 007c2b2500000006.5 > > > > 007c2b2500000006.7 > > > > 007c2b2500000007.5 > > > > 007c2b2500000007.7 > > > > 007c2b2500000008.5 > > > > 007c2b2500000008.7 > > > > 007c2b2500000009.5 > > > > 007c2b2500000009.7 > > > > 007c2b250000000a.5 > > > > 007c2b250000000a.7 > > > > 007c2b250000000b.5 > > > > 007c2b250000000b.7 > > > > 007c2b250000000c.5 > > > > 007c2b250000000c.7 > > > > 007c2b250000000d.5 > > > > 007c2b250000000d.7 > > > > 007c2b250000000e.5 > > > > 007c2b250000000e.7 > > > > 007c2b250000000f.5 > > > > 007c2b250000000f.7 > > > > 007c2b2500000010.5 > > > > 007c2b2500000010.7 > > > > 007c2b2500000011.5 > > > > 007c2b2500000011.7 > > > > 007c2b2500000012.5 > > > > 007c2b2500000012.7 > > > > 007c2b2500000013.5 > > > > 007c2b2500000013.7 > > > > 007c2b2500000014.5 > > > > 007c2b2500000014.7 > > > > 007c2b2500000015.5 > > > > 007c2b2500000015.7 > > > > 007c2b2500000016.5 > > > > 007c2b2500000016.7 > > > > 007c2b2500000017.5 > > > > 007c2b2500000017.7 > > > > 007c2b2500000018.5 > > > > 007c2b2500000018.7 > > > > 807c2b2500000000.5 > > > > 807c2b2500000000.7 > > > > STORE/0/obj/.stale: > > > > STORE/1/obj/.stale: > > > > STORE/2/obj/.stale: > > > > STORE/3/obj/.stale: > > > > STORE/4/obj/.stale: > > > > STORE/5/obj/.stale: > > > > STORE/6/obj/.stale: > > > > STORE/7/obj/.stale: > > > > 9c7766570b3be3aff2724f587c2f4107 - > > > > > > > > This is an output of "tree -a /tmp/sheepdog/032" > > > > > > > > /tmp/sheepdog/032 > > > > ├── 0 > > > > │ ├── config > > > > │ ├── epoch > > > > │ │ ├── 00000001 > > > > │ │ ├── 00000002 > > > > │ │ ├── 00000003 > > > > │ │ ├── 00000004 > > > > │ │ ├── 00000005 > > > > │ │ ├── 00000006 > > > > │ │ ├── 00000007 > > > > │ │ ├── 00000008 > > > > │ │ ├── 00000009 > > > > │ │ ├── 00000010 > > > > │ │ ├── 00000011 > > > > │ │ ├── 00000012 > > > > │ │ ├── 00000013 > > > > │ │ ├── 00000014 > > > > │ │ ├── 00000015 > > > > │ │ ├── 00000016 > > > > │ │ └── 00000017 > > > > │ ├── lock > > > > │ ├── obj > > > > │ │ ├── .stale > > > > │ │ ├── 007c2b2500000000 > > > > │ │ ├── 007c2b2500000001 > > > > │ │ ├── 007c2b2500000002 > > > > │ │ ├── 007c2b2500000003 > > > > │ │ ├── 007c2b2500000004 > > > > │ │ ├── 007c2b2500000005 > > > > │ │ ├── 007c2b2500000006 > > > > │ │ ├── 007c2b2500000007 > > > > │ │ ├── 007c2b2500000008 > > > > │ │ ├── 007c2b2500000009 > > > > │ │ ├── 007c2b250000000a > > > > │ │ ├── 007c2b250000000b > > > > │ │ ├── 007c2b250000000c > > > > │ │ ├── 007c2b250000000d > > > > │ │ ├── 007c2b250000000e > > > > │ │ ├── 007c2b250000000f > > > > │ │ ├── 007c2b2500000010 > > > > │ │ ├── 007c2b2500000011 > > > > │ │ ├── 007c2b2500000012 > > > > │ │ ├── 007c2b2500000013 > > > > │ │ ├── 007c2b2500000014 > > > > │ │ ├── 007c2b2500000015 > > > > │ │ ├── 007c2b2500000016 > > > > │ │ ├── 007c2b2500000017 > > > > │ │ ├── 007c2b2500000018 > > > > │ │ └── 807c2b2500000000 > > > > │ ├── sheep.log > > > > │ └── sock > > > > ├── 1 > > > > │ ├── config > > > > │ ├── core > > > > │ ├── epoch > > > > │ │ ├── 00000001 > > > > │ │ ├── 00000002 > > > > │ │ ├── 00000003 > > > > │ │ ├── 00000004 > > > > │ │ ├── 00000005 > > > > │ │ ├── 00000006 > > > > │ │ ├── 00000007 > > > > │ │ ├── 00000009 > > > > │ │ ├── 00000010 > > > > │ │ ├── 00000011 > > > > │ │ ├── 00000012 > > > > │ │ └── 00000013 > > > > │ ├── gdb.txt > > > > │ ├── lock > > > > │ ├── obj > > > > │ │ ├── .stale > > > > │ │ │ ├── 007c2b2500000000.7 > > > > │ │ │ ├── 007c2b2500000001.7 > > > > │ │ │ ├── 007c2b2500000002.7 > > > > │ │ │ ├── 007c2b2500000003.7 > > > > │ │ │ ├── 007c2b2500000004.7 > > > > │ │ │ ├── 007c2b2500000005.7 > > > > │ │ │ ├── 007c2b2500000006.7 > > > > │ │ │ ├── 007c2b2500000007.7 > > > > │ │ │ ├── 007c2b2500000008.7 > > > > │ │ │ ├── 007c2b2500000009.7 > > > > │ │ │ ├── 007c2b250000000a.7 > > > > │ │ │ ├── 007c2b250000000b.7 > > > > │ │ │ ├── 007c2b250000000c.7 > > > > │ │ │ ├── 007c2b250000000d.7 > > > > │ │ │ ├── 007c2b250000000e.7 > > > > │ │ │ ├── 007c2b250000000f.7 > > > > │ │ │ ├── 007c2b2500000010.7 > > > > │ │ │ ├── 007c2b2500000011.7 > > > > │ │ │ ├── 007c2b2500000012.7 > > > > │ │ │ ├── 007c2b2500000013.7 > > > > │ │ │ ├── 007c2b2500000014.7 > > > > │ │ │ ├── 007c2b2500000015.7 > > > > │ │ │ ├── 007c2b2500000016.7 > > > > │ │ │ ├── 007c2b2500000017.7 > > > > │ │ │ ├── 007c2b2500000018.7 > > > > │ │ │ └── 807c2b2500000000.7 > > > > │ │ ├── 007c2b2500000003 > > > > │ │ ├── 007c2b250000000a > > > > │ │ ├── 007c2b250000000d > > > > │ │ └── 007c2b2500000016 > > > > │ ├── sheep.log > > > > │ └── sock > > > > ├── 2 > > > > │ ├── config > > > > │ ├── epoch > > > > │ │ ├── 00000001 > > > > │ │ ├── 00000002 > > > > │ │ ├── 00000003 > > > > │ │ ├── 00000004 > > > > │ │ ├── 00000005 > > > > │ │ ├── 00000006 > > > > │ │ ├── 00000010 > > > > │ │ ├── 00000011 > > > > │ │ ├── 00000012 > > > > │ │ ├── 00000013 > > > > │ │ ├── 00000014 > > > > │ │ ├── 00000015 > > > > │ │ ├── 00000016 > > > > │ │ └── 00000017 > > > > │ ├── lock > > > > │ ├── obj > > > > │ │ ├── .stale > > > > │ │ ├── 007c2b2500000000 > > > > │ │ ├── 007c2b2500000001 > > > > │ │ ├── 007c2b2500000002 > > > > │ │ ├── 007c2b2500000003 > > > > │ │ ├── 007c2b2500000004 > > > > │ │ ├── 007c2b2500000005 > > > > │ │ ├── 007c2b2500000006 > > > > │ │ ├── 007c2b2500000007 > > > > │ │ ├── 007c2b2500000008 > > > > │ │ ├── 007c2b2500000009 > > > > │ │ ├── 007c2b250000000a > > > > │ │ ├── 007c2b250000000b > > > > │ │ ├── 007c2b250000000c > > > > │ │ ├── 007c2b250000000d > > > > │ │ ├── 007c2b250000000e > > > > │ │ ├── 007c2b250000000f > > > > │ │ ├── 007c2b2500000010 > > > > │ │ ├── 007c2b2500000011 > > > > │ │ ├── 007c2b2500000012 > > > > │ │ ├── 007c2b2500000013 > > > > │ │ ├── 007c2b2500000014 > > > > │ │ ├── 007c2b2500000015 > > > > │ │ ├── 007c2b2500000016 > > > > │ │ ├── 007c2b2500000017 > > > > │ │ ├── 007c2b2500000018 > > > > │ │ └── 807c2b2500000000 > > > > │ ├── sheep.log > > > > │ └── sock > > > > ├── 3 > > > > │ ├── config > > > > │ ├── core > > > > │ ├── epoch > > > > │ │ ├── 00000001 > > > > │ │ ├── 00000002 > > > > │ │ ├── 00000003 > > > > │ │ ├── 00000004 > > > > │ │ ├── 00000005 > > > > │ │ ├── 00000012 > > > > │ │ ├── 00000013 > > > > │ │ ├── 00000014 > > > > │ │ └── 00000015 > > > > │ ├── gdb.txt > > > > │ ├── lock > > > > │ ├── obj > > > > │ │ ├── .stale > > > > │ │ │ ├── 007c2b2500000000.5 > > > > │ │ │ ├── 007c2b2500000001.5 > > > > │ │ │ ├── 007c2b2500000002.5 > > > > │ │ │ ├── 007c2b2500000003.5 > > > > │ │ │ ├── 007c2b2500000004.5 > > > > │ │ │ ├── 007c2b2500000005.5 > > > > │ │ │ ├── 007c2b2500000006.5 > > > > │ │ │ ├── 007c2b2500000007.5 > > > > │ │ │ ├── 007c2b2500000008.5 > > > > │ │ │ ├── 007c2b2500000009.5 > > > > │ │ │ ├── 007c2b250000000a.5 > > > > │ │ │ ├── 007c2b250000000b.5 > > > > │ │ │ ├── 007c2b250000000c.5 > > > > │ │ │ ├── 007c2b250000000d.5 > > > > │ │ │ ├── 007c2b250000000e.5 > > > > │ │ │ ├── 007c2b250000000f.5 > > > > │ │ │ ├── 007c2b2500000010.5 > > > > │ │ │ ├── 007c2b2500000011.5 > > > > │ │ │ ├── 007c2b2500000012.5 > > > > │ │ │ ├── 007c2b2500000013.5 > > > > │ │ │ ├── 007c2b2500000014.5 > > > > │ │ │ ├── 007c2b2500000015.5 > > > > │ │ │ ├── 007c2b2500000016.5 > > > > │ │ │ ├── 007c2b2500000017.5 > > > > │ │ │ ├── 007c2b2500000018.5 > > > > │ │ │ └── 807c2b2500000000.5 > > > > │ │ ├── 007c2b250000000d > > > > │ │ └── 007c2b2500000016 > > > > │ ├── sheep.log > > > > │ └── sock > > > > ├── 4 > > > > │ ├── config > > > > │ ├── epoch > > > > │ │ ├── 00000001 > > > > │ │ ├── 00000002 > > > > │ │ ├── 00000003 > > > > │ │ ├── 00000004 > > > > │ │ ├── 00000011 > > > > │ │ ├── 00000012 > > > > │ │ ├── 00000013 > > > > │ │ ├── 00000014 > > > > │ │ ├── 00000015 > > > > │ │ ├── 00000016 > > > > │ │ └── 00000017 > > > > │ ├── lock > > > > │ ├── obj > > > > │ │ ├── .stale > > > > │ │ ├── 007c2b2500000000 > > > > │ │ ├── 007c2b2500000001 > > > > │ │ ├── 007c2b2500000002 > > > > │ │ ├── 007c2b2500000003 > > > > │ │ ├── 007c2b2500000004 > > > > │ │ ├── 007c2b2500000005 > > > > │ │ ├── 007c2b2500000006 > > > > │ │ ├── 007c2b2500000007 > > > > │ │ ├── 007c2b2500000008 > > > > │ │ ├── 007c2b2500000009 > > > > │ │ ├── 007c2b250000000a > > > > │ │ ├── 007c2b250000000b > > > > │ │ ├── 007c2b250000000c > > > > │ │ ├── 007c2b250000000d > > > > │ │ ├── 007c2b250000000e > > > > │ │ ├── 007c2b250000000f > > > > │ │ ├── 007c2b2500000010 > > > > │ │ ├── 007c2b2500000011 > > > > │ │ ├── 007c2b2500000012 > > > > │ │ ├── 007c2b2500000013 > > > > │ │ ├── 007c2b2500000014 > > > > │ │ ├── 007c2b2500000015 > > > > │ │ ├── 007c2b2500000016 > > > > │ │ ├── 007c2b2500000017 > > > > │ │ ├── 007c2b2500000018 > > > > │ │ └── 807c2b2500000000 > > > > │ ├── sheep.log > > > > │ └── sock > > > > ├── 5 > > > > │ ├── config > > > > │ ├── epoch > > > > │ │ ├── 00000001 > > > > │ │ ├── 00000002 > > > > │ │ ├── 00000003 > > > > │ │ ├── 00000013 > > > > │ │ ├── 00000014 > > > > │ │ ├── 00000015 > > > > │ │ ├── 00000016 > > > > │ │ └── 00000017 > > > > │ ├── lock > > > > │ ├── obj > > > > │ │ ├── .stale > > > > │ │ ├── 007c2b2500000000 > > > > │ │ ├── 007c2b2500000001 > > > > │ │ ├── 007c2b2500000002 > > > > │ │ ├── 007c2b2500000003 > > > > │ │ ├── 007c2b2500000004 > > > > │ │ ├── 007c2b2500000005 > > > > │ │ ├── 007c2b2500000006 > > > > │ │ ├── 007c2b2500000007 > > > > │ │ ├── 007c2b2500000008 > > > > │ │ ├── 007c2b2500000009 > > > > │ │ ├── 007c2b250000000a > > > > │ │ ├── 007c2b250000000b > > > > │ │ ├── 007c2b250000000c > > > > │ │ ├── 007c2b250000000d > > > > │ │ ├── 007c2b250000000e > > > > │ │ ├── 007c2b250000000f > > > > │ │ ├── 007c2b2500000010 > > > > │ │ ├── 007c2b2500000011 > > > > │ │ ├── 007c2b2500000012 > > > > │ │ ├── 007c2b2500000013 > > > > │ │ ├── 007c2b2500000014 > > > > │ │ ├── 007c2b2500000015 > > > > │ │ ├── 007c2b2500000016 > > > > │ │ ├── 007c2b2500000017 > > > > │ │ ├── 007c2b2500000018 > > > > │ │ └── 807c2b2500000000 > > > > │ ├── sheep.log > > > > │ └── sock > > > > ├── 6 > > > > │ ├── config > > > > │ ├── epoch > > > > │ │ ├── 00000001 > > > > │ │ ├── 00000002 > > > > │ │ ├── 00000014 > > > > │ │ ├── 00000015 > > > > │ │ ├── 00000016 > > > > │ │ └── 00000017 > > > > │ ├── lock > > > > │ ├── obj > > > > │ │ ├── .stale > > > > │ │ ├── 007c2b2500000000 > > > > │ │ ├── 007c2b2500000001 > > > > │ │ ├── 007c2b2500000002 > > > > │ │ ├── 007c2b2500000003 > > > > │ │ ├── 007c2b2500000004 > > > > │ │ ├── 007c2b2500000005 > > > > │ │ ├── 007c2b2500000006 > > > > │ │ ├── 007c2b2500000007 > > > > │ │ ├── 007c2b2500000008 > > > > │ │ ├── 007c2b2500000009 > > > > │ │ ├── 007c2b250000000a > > > > │ │ ├── 007c2b250000000b > > > > │ │ ├── 007c2b250000000c > > > > │ │ ├── 007c2b250000000d > > > > │ │ ├── 007c2b250000000e > > > > │ │ ├── 007c2b250000000f > > > > │ │ ├── 007c2b2500000010 > > > > │ │ ├── 007c2b2500000011 > > > > │ │ ├── 007c2b2500000012 > > > > │ │ ├── 007c2b2500000013 > > > > │ │ ├── 007c2b2500000014 > > > > │ │ ├── 007c2b2500000015 > > > > │ │ ├── 007c2b2500000016 > > > > │ │ ├── 007c2b2500000017 > > > > │ │ ├── 007c2b2500000018 > > > > │ │ └── 807c2b2500000000 > > > > │ ├── sheep.log > > > > │ └── sock > > > > └── 7 > > > > ├── config > > > > ├── epoch > > > > │ ├── 00000001 > > > > │ ├── 00000015 > > > > │ ├── 00000016 > > > > │ └── 00000017 > > > > ├── lock > > > > ├── obj > > > > │ ├── .stale > > > > │ ├── 007c2b2500000000 > > > > │ ├── 007c2b2500000001 > > > > │ ├── 007c2b2500000002 > > > > │ ├── 007c2b2500000003 > > > > │ ├── 007c2b2500000004 > > > > │ ├── 007c2b2500000005 > > > > │ ├── 007c2b2500000006 > > > > │ ├── 007c2b2500000007 > > > > │ ├── 007c2b2500000008 > > > > │ ├── 007c2b2500000009 > > > > │ ├── 007c2b250000000a > > > > │ ├── 007c2b250000000b > > > > │ ├── 007c2b250000000c > > > > │ ├── 007c2b250000000d > > > > │ ├── 007c2b250000000e > > > > │ ├── 007c2b250000000f > > > > │ ├── 007c2b2500000010 > > > > │ ├── 007c2b2500000011 > > > > │ ├── 007c2b2500000012 > > > > │ ├── 007c2b2500000013 > > > > │ ├── 007c2b2500000014 > > > > │ ├── 007c2b2500000015 > > > > │ ├── 007c2b2500000016 > > > > │ ├── 007c2b2500000017 > > > > │ ├── 007c2b2500000018 > > > > │ └── 807c2b2500000000 > > > > ├── sheep.log > > > > └── sock > > > > > > > > 32 directories, 331 files > > > > > > Can you try this patch for 032 and rerun test? > > > > > > diff --git a/tests/functional/032 b/tests/functional/032 > > > index d902a30..4b65c5d 100755 > > > --- a/tests/functional/032 > > > +++ b/tests/functional/032 > > > @@ -34,6 +34,7 @@ for i in `seq 1 7`; do > > > _start_sheep $i > > > done > > > > > > +_wait_for_sheep 8 > > > _wait_for_sheep_recovery 0 > > > > > > ls $STORE/*/obj/807c2b2500000000 | _filter_store | sort > > > > > > > I tested it and 032 can be passed now. > > > > But the problem of 033 isn't solved yet. > > This problem isn't related to my patch, my patch just trigger the bug of > lib/work.c
Even if the root cause is bugs in work.c, the patch hits them and has a possibility of breaking the recovery process. Could you apply Yuan's patch and do "./check 33" when you have time, Robin and Kazutaka-san? If the problem happens only on my environment, I don't disagree with applying this patch. Thanks, Hitoshi -- sheepdog mailing list sheepdog@lists.wpkg.org http://lists.wpkg.org/mailman/listinfo/sheepdog