Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request
Am 06.11.2012 10:07, schrieb Alexandre DERUMIER: Most mem get used while doing a lot of small reads. I see same problem with my test and wip-rbd-read. I got also strange fio results, from 5k to 100K iops (increase and increase in time during the bench). Update to latest wip-rbd-read frmo yesterday. Sage fixed the bug but didn't report back. Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request
>>Most mem get used while doing a lot of small reads. I see same problem with my test and wip-rbd-read. I got also strange fio results, from 5k to 100K iops (increase and increase in time during the bench). - Mail original - De: "Stefan Priebe" À: "Sage Weil" Cc: ceph-devel@vger.kernel.org Envoyé: Lundi 5 Novembre 2012 21:43:05 Objet: Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request Am 05.11.2012 20:33, schrieb Stefan Priebe: > Am 04.11.2012 15:12, schrieb Sage Weil: >> On Sun, 4 Nov 2012, Stefan Priebe wrote: >>> Can i merge wip-rbd-read into master? >> >> Yeah. I'm going to do a bit more testing first before I do it, but it >> should apply cleanly. Hopefully later today. > > Thanks - seems to be fixed with wip-rbd-read but i have a memory leak now. > > kvm process raises and raises and raises which each new test around 5GB > of memory. Should i write a new mail? Most mem get used while doing a lot of small reads. Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request
Am 05.11.2012 20:33, schrieb Stefan Priebe: Am 04.11.2012 15:12, schrieb Sage Weil: On Sun, 4 Nov 2012, Stefan Priebe wrote: Can i merge wip-rbd-read into master? Yeah. I'm going to do a bit more testing first before I do it, but it should apply cleanly. Hopefully later today. Thanks - seems to be fixed with wip-rbd-read but i have a memory leak now. kvm process raises and raises and raises which each new test around 5GB of memory. Should i write a new mail? Most mem get used while doing a lot of small reads. Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request
Am 05.11.2012 21:20, schrieb Andrey Korolyov: On Mon, Nov 5, 2012 at 11:33 PM, Stefan Priebe wrote: Am 04.11.2012 15:12, schrieb Sage Weil: On Sun, 4 Nov 2012, Stefan Priebe wrote: Can i merge wip-rbd-read into master? Yeah. I'm going to do a bit more testing first before I do it, but it should apply cleanly. Hopefully later today. Thanks - seems to be fixed with wip-rbd-read but i have a memory leak now. kvm process raises and raises and raises which each new test around 5GB of memory. Should i write a new mail? Do you use qemu 1.2.0? It has a memory leak, at least with rbd backend, but I didn`t profiled it yet, so it may be something generic :) Yes i'm using kvm-qemu 1.2. But i dought that in my case it is a qemu problem. Memory usage is constant using ceph git master. When i merge ceph wip-rbd-read into master kvm process raises massivly while reading from disk. Greets, Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request
On Mon, Nov 5, 2012 at 11:33 PM, Stefan Priebe wrote: > Am 04.11.2012 15:12, schrieb Sage Weil: >> >> On Sun, 4 Nov 2012, Stefan Priebe wrote: >>> >>> Can i merge wip-rbd-read into master? >> >> >> Yeah. I'm going to do a bit more testing first before I do it, but it >> should apply cleanly. Hopefully later today. > > > Thanks - seems to be fixed with wip-rbd-read but i have a memory leak now. > > kvm process raises and raises and raises which each new test around 5GB of > memory. Should i write a new mail? Do you use qemu 1.2.0? It has a memory leak, at least with rbd backend, but I didn`t profiled it yet, so it may be something generic :) > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request
Am 05.11.2012 20:33, schrieb Stefan Priebe: Am 04.11.2012 15:12, schrieb Sage Weil: On Sun, 4 Nov 2012, Stefan Priebe wrote: Can i merge wip-rbd-read into master? Yeah. I'm going to do a bit more testing first before I do it, but it should apply cleanly. Hopefully later today. Thanks - seems to be fixed with wip-rbd-read but i have a memory leak now. kvm process raises and raises and raises which each new test around 5GB of memory. Should i write a new mail? Works fine with vanilla origin/master. So it must be in wip-rbd-read. Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request
Am 04.11.2012 15:12, schrieb Sage Weil: On Sun, 4 Nov 2012, Stefan Priebe wrote: Can i merge wip-rbd-read into master? Yeah. I'm going to do a bit more testing first before I do it, but it should apply cleanly. Hopefully later today. Thanks - seems to be fixed with wip-rbd-read but i have a memory leak now. kvm process raises and raises and raises which each new test around 5GB of memory. Should i write a new mail? Greets, Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request
On Sun, 4 Nov 2012, Stefan Priebe wrote: > Am 04.11.2012 20:19, schrieb Sage Weil: > > On Sun, 4 Nov 2012, Stefan Priebe wrote: > > > Am 04.11.2012 15:12, schrieb Sage Weil: > > > > On Sun, 4 Nov 2012, Stefan Priebe wrote: > > > > > Can i merge wip-rbd-read into master? > > > > > > > > Yeah. I'm going to do a bit more testing first before I do it, but it > > > > should apply cleanly. Hopefully later today. > > > > > > Any other branches to merge locally while benchmarking / testing? > > > > Nope, although you get the latest wip-rbd-read (I fixed an issue with > > glibc's random_r sucking). > > Did i get you wrong? There's no commit from sage today? > https://github.com/ceph/ceph/commits/wip-rbd-read That's the right one. (There was briefly some other stuff in wip-rbd-read earlier today that was broken.) sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request
Am 04.11.2012 20:19, schrieb Sage Weil: On Sun, 4 Nov 2012, Stefan Priebe wrote: Am 04.11.2012 15:12, schrieb Sage Weil: On Sun, 4 Nov 2012, Stefan Priebe wrote: Can i merge wip-rbd-read into master? Yeah. I'm going to do a bit more testing first before I do it, but it should apply cleanly. Hopefully later today. Any other branches to merge locally while benchmarking / testing? Nope, although you get the latest wip-rbd-read (I fixed an issue with glibc's random_r sucking). Did i get you wrong? There's no commit from sage today? https://github.com/ceph/ceph/commits/wip-rbd-read Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request
On Sun, 4 Nov 2012, Stefan Priebe wrote: > Am 04.11.2012 15:12, schrieb Sage Weil: > > On Sun, 4 Nov 2012, Stefan Priebe wrote: > > > Can i merge wip-rbd-read into master? > > > > Yeah. I'm going to do a bit more testing first before I do it, but it > > should apply cleanly. Hopefully later today. > > Any other branches to merge locally while benchmarking / testing? Nope, although you get the latest wip-rbd-read (I fixed an issue with glibc's random_r sucking). sage -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request
Am 04.11.2012 15:12, schrieb Sage Weil: On Sun, 4 Nov 2012, Stefan Priebe wrote: Can i merge wip-rbd-read into master? Yeah. I'm going to do a bit more testing first before I do it, but it should apply cleanly. Hopefully later today. Any other branches to merge locally while benchmarking / testing? Greets, Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request
On Sun, 4 Nov 2012, Stefan Priebe wrote: > Can i merge wip-rbd-read into master? Yeah. I'm going to do a bit more testing first before I do it, but it should apply cleanly. Hopefully later today. sage > > Stefan > Am 04.11.2012 15:06, schrieb Sage Weil: > > On Sun, 4 Nov 2012, Stefan Priebe wrote: > > > Hello, > > > > > > while benchmarking the new v2 rbd format i saw several crashes of KVM. > > > First i > > > thought this was kernel related but i was just happening faster with one > > > kernel than the other. > > > > Can you try the wip-rbd-read branch? It has a bunch of fixes that aren't > > in master yet. I haven't seen this specific crash (I don't think), but I > > was reliable hitting other issues with master that wip-rbd-read reslves. > > > > Alternatively, you could cherry-pick > > b1d866efbdf7e8a7bc083b1f3ea56f24e10c8fcd, which I'm guessing is your > > problem. > > > > Thanks! > > sage > > > > > > > > > > The KVM crash / exit looks like this: > > > librbd/AioCompletion.cc: In function 'void > > > librbd::AioCompletion::complete_request(CephContext*, ssize_t)' thread > > > 7f73db67c780 time 2012-11-04 14:59:44.997009 > > > librbd/AioCompletion.cc: 43: FAILED assert(bl.length() == read_buf_len) > > > ceph version 0.53-585-ga01b112 > > > (a01b112d71a0b6a1bb206d53867d13536d17bbf6) > > > 1: (librbd::AioCompletion::complete_request(CephContext*, long)+0xa89) > > > [0x7f73d9337319] > > > 2: (librbd::C_AioRead::finish(int)+0x174) [0x7f73d93374e4] > > > 3: (Context::complete(int)+0xa) [0x7f73d933796a] > > > 4: (librbd::C_CacheRead::finish(int)+0xe) [0x7f73d933683e] > > > 5: (Context::complete(int)+0xa) [0x7f73d933796a] > > > 6: (librbd::ImageCtx::aio_read_from_cache(object_t, ceph::buffer::list*, > > > unsigned long, unsigned long, Context*)+0x3a0) [0x7f73d9347bd0] > > > 7: (librbd::aio_read(librbd::ImageCtx*, std::vector > > long, > > > unsigned long>, std::allocator > > > > > const&, char*, ceph::buffer::list*, librbd::AioCompletion*)+0x765) > > > [0x7f73d9354995] > > > 8: (librbd::aio_read(librbd::ImageCtx*, unsigned long, unsigned long, > > > char*, > > > ceph::buffer::list*, librbd::AioCompletion*)+0xa2) [0x7f73d9354f92] > > > 9: (()+0x9f2de) [0x7f73db73d2de] > > > 10: (()+0x9f490) [0x7f73db73d490] > > > 11: (()+0x83c45) [0x7f73db721c45] > > > 12: (()+0x850ea) [0x7f73db7230ea] > > > NOTE: a copy of the executable, or `objdump -rdS ` is needed > > > to > > > interpret this. > > > terminate called after throwing an instance of 'ceph::FailedAssertion' > > > > > > Stefan > > > -- > > > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > > > the body of a message to majord...@vger.kernel.org > > > More majordomo info at http://vger.kernel.org/majordomo-info.html > > > > > > > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request
Can i merge wip-rbd-read into master? Stefan Am 04.11.2012 15:06, schrieb Sage Weil: On Sun, 4 Nov 2012, Stefan Priebe wrote: Hello, while benchmarking the new v2 rbd format i saw several crashes of KVM. First i thought this was kernel related but i was just happening faster with one kernel than the other. Can you try the wip-rbd-read branch? It has a bunch of fixes that aren't in master yet. I haven't seen this specific crash (I don't think), but I was reliable hitting other issues with master that wip-rbd-read reslves. Alternatively, you could cherry-pick b1d866efbdf7e8a7bc083b1f3ea56f24e10c8fcd, which I'm guessing is your problem. Thanks! sage The KVM crash / exit looks like this: librbd/AioCompletion.cc: In function 'void librbd::AioCompletion::complete_request(CephContext*, ssize_t)' thread 7f73db67c780 time 2012-11-04 14:59:44.997009 librbd/AioCompletion.cc: 43: FAILED assert(bl.length() == read_buf_len) ceph version 0.53-585-ga01b112 (a01b112d71a0b6a1bb206d53867d13536d17bbf6) 1: (librbd::AioCompletion::complete_request(CephContext*, long)+0xa89) [0x7f73d9337319] 2: (librbd::C_AioRead::finish(int)+0x174) [0x7f73d93374e4] 3: (Context::complete(int)+0xa) [0x7f73d933796a] 4: (librbd::C_CacheRead::finish(int)+0xe) [0x7f73d933683e] 5: (Context::complete(int)+0xa) [0x7f73d933796a] 6: (librbd::ImageCtx::aio_read_from_cache(object_t, ceph::buffer::list*, unsigned long, unsigned long, Context*)+0x3a0) [0x7f73d9347bd0] 7: (librbd::aio_read(librbd::ImageCtx*, std::vector, std::allocator > > const&, char*, ceph::buffer::list*, librbd::AioCompletion*)+0x765) [0x7f73d9354995] 8: (librbd::aio_read(librbd::ImageCtx*, unsigned long, unsigned long, char*, ceph::buffer::list*, librbd::AioCompletion*)+0xa2) [0x7f73d9354f92] 9: (()+0x9f2de) [0x7f73db73d2de] 10: (()+0x9f490) [0x7f73db73d490] 11: (()+0x83c45) [0x7f73db721c45] 12: (()+0x850ea) [0x7f73db7230ea] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. terminate called after throwing an instance of 'ceph::FailedAssertion' Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
Re: BUG: kvm crashing in void librbd::AioCompletion::complete_request
On Sun, 4 Nov 2012, Stefan Priebe wrote: > Hello, > > while benchmarking the new v2 rbd format i saw several crashes of KVM. First i > thought this was kernel related but i was just happening faster with one > kernel than the other. Can you try the wip-rbd-read branch? It has a bunch of fixes that aren't in master yet. I haven't seen this specific crash (I don't think), but I was reliable hitting other issues with master that wip-rbd-read reslves. Alternatively, you could cherry-pick b1d866efbdf7e8a7bc083b1f3ea56f24e10c8fcd, which I'm guessing is your problem. Thanks! sage > > The KVM crash / exit looks like this: > librbd/AioCompletion.cc: In function 'void > librbd::AioCompletion::complete_request(CephContext*, ssize_t)' thread > 7f73db67c780 time 2012-11-04 14:59:44.997009 > librbd/AioCompletion.cc: 43: FAILED assert(bl.length() == read_buf_len) > ceph version 0.53-585-ga01b112 (a01b112d71a0b6a1bb206d53867d13536d17bbf6) > 1: (librbd::AioCompletion::complete_request(CephContext*, long)+0xa89) > [0x7f73d9337319] > 2: (librbd::C_AioRead::finish(int)+0x174) [0x7f73d93374e4] > 3: (Context::complete(int)+0xa) [0x7f73d933796a] > 4: (librbd::C_CacheRead::finish(int)+0xe) [0x7f73d933683e] > 5: (Context::complete(int)+0xa) [0x7f73d933796a] > 6: (librbd::ImageCtx::aio_read_from_cache(object_t, ceph::buffer::list*, > unsigned long, unsigned long, Context*)+0x3a0) [0x7f73d9347bd0] > 7: (librbd::aio_read(librbd::ImageCtx*, std::vector unsigned long>, std::allocator > > > const&, char*, ceph::buffer::list*, librbd::AioCompletion*)+0x765) > [0x7f73d9354995] > 8: (librbd::aio_read(librbd::ImageCtx*, unsigned long, unsigned long, char*, > ceph::buffer::list*, librbd::AioCompletion*)+0xa2) [0x7f73d9354f92] > 9: (()+0x9f2de) [0x7f73db73d2de] > 10: (()+0x9f490) [0x7f73db73d490] > 11: (()+0x83c45) [0x7f73db721c45] > 12: (()+0x850ea) [0x7f73db7230ea] > NOTE: a copy of the executable, or `objdump -rdS ` is needed to > interpret this. > terminate called after throwing an instance of 'ceph::FailedAssertion' > > Stefan > -- > To unsubscribe from this list: send the line "unsubscribe ceph-devel" in > the body of a message to majord...@vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > > -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
BUG: kvm crashing in void librbd::AioCompletion::complete_request
Hello, while benchmarking the new v2 rbd format i saw several crashes of KVM. First i thought this was kernel related but i was just happening faster with one kernel than the other. The KVM crash / exit looks like this: librbd/AioCompletion.cc: In function 'void librbd::AioCompletion::complete_request(CephContext*, ssize_t)' thread 7f73db67c780 time 2012-11-04 14:59:44.997009 librbd/AioCompletion.cc: 43: FAILED assert(bl.length() == read_buf_len) ceph version 0.53-585-ga01b112 (a01b112d71a0b6a1bb206d53867d13536d17bbf6) 1: (librbd::AioCompletion::complete_request(CephContext*, long)+0xa89) [0x7f73d9337319] 2: (librbd::C_AioRead::finish(int)+0x174) [0x7f73d93374e4] 3: (Context::complete(int)+0xa) [0x7f73d933796a] 4: (librbd::C_CacheRead::finish(int)+0xe) [0x7f73d933683e] 5: (Context::complete(int)+0xa) [0x7f73d933796a] 6: (librbd::ImageCtx::aio_read_from_cache(object_t, ceph::buffer::list*, unsigned long, unsigned long, Context*)+0x3a0) [0x7f73d9347bd0] 7: (librbd::aio_read(librbd::ImageCtx*, std::vectorlong, unsigned long>, std::allocatorlong> > > const&, char*, ceph::buffer::list*, librbd::AioCompletion*)+0x765) [0x7f73d9354995] 8: (librbd::aio_read(librbd::ImageCtx*, unsigned long, unsigned long, char*, ceph::buffer::list*, librbd::AioCompletion*)+0xa2) [0x7f73d9354f92] 9: (()+0x9f2de) [0x7f73db73d2de] 10: (()+0x9f490) [0x7f73db73d490] 11: (()+0x83c45) [0x7f73db721c45] 12: (()+0x850ea) [0x7f73db7230ea] NOTE: a copy of the executable, or `objdump -rdS ` is needed to interpret this. terminate called after throwing an instance of 'ceph::FailedAssertion' Stefan -- To unsubscribe from this list: send the line "unsubscribe ceph-devel" in the body of a message to majord...@vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html
