Re: [ceph-users] Lost access when removing cache pool overlay
-BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Do the client key have access to the base pool? Something similar bit us when adding a caching tier. Since the cache tier may be proxying all the I/O, the client may not have had access to the base pool and it still worked ok. Once you removed the cache tier, it could no longer access the pool. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Fri, Jan 29, 2016 at 8:47 AM, Gerd Jakobovitsch wrote: > Dear all, > > I had to move .rgw.buckets.index pool to another structure; therefore, I > created a new pool .rgw.buckets.index.new ; added the old pool as cache > pool, and flushed the data. > > Up to this moment everything was ok. With radosgw -p df, I saw the > objects moving to the new pool; the moved objects where ok, I could list > omap keys and so on. > > When everything got moved, I removed the overlay cache pool. But at this > moment, the objects became unresponsive: > > [(13:39:20) ceph@spchaog1 ~]$ rados -p .rgw.buckets.index listomapkeys > .dir.default.198764998.1 > error getting omap key set .rgw.buckets.index/.dir.default.198764998.1: (5) > Input/output error > > That happens to all objects. When trying the access to the bucket through > radosgw, I also get problems: > > [(13:16:01) root@spcogp1 ~]# radosgw-admin bucket stats --bucket="mybucket" > error getting bucket stats ret=-2 > > Looking at the disk, data seems to be there: > > [(13:47:10) root@spcsnp1 ~]# ls > /var/lib/ceph/osd/ceph-23/current/34.1f_head/|grep 198764998.1 > \.dir.default.198764998.1__head_8A7482FF__22 > > Does anyone have a hint? Could I have lost ownership of the objects? > > Regards. > > > > > > > > > > > > > > -- > > As informações contidas nesta mensagem são CONFIDENCIAIS, protegidas pelo > sigilo legal e por direitos autorais. A divulgação, distribuição, reprodução > ou qualquer forma de utilização do teor deste documento depende de > autorização do emissor, sujeitando-se o infrator às sanções legais. Caso > esta comunicação tenha sido recebida por engano, favor avisar imediatamente, > respondendo esta mensagem. > > > > ___ > ceph-users mailing list > ceph-users@lists.ceph.com > http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com > -BEGIN PGP SIGNATURE- Version: Mailvelope v1.3.4 Comment: https://www.mailvelope.com wsFcBAEBCAAQBQJWq5YjCRDmVDuy+mK58QAAzUMQALqON8Ux5KPaotbyOMcr SzWVigfIa9Go1on8snKmehVzkwC25XaJxYQNU2OwsUUhHa1cy7v+rG6DTbDQ UDUK4IJ0O6ItGz4IeoyL06KyqmRy06OnuLRyzpQQD+nbIN+/82CVhMRMaKN+ U/GM+avDArN1JjjuQXFMgX/bS6ZoqJOBGqZKt3QWpJnkob1wgxP1tZA7MjZt p6Sfm/ci0dhveRhzylpEoxYKXwR6hN1hy/wiH2P5yeQBYYmpOALLDJDSTvln VZ/MbxPL5c0U/RRAkVMic1CvteeQ2nil2wEPFlu7cDjERvoBCMoyQeDXlep4 l+sAJbkKoOEKqE9xDo6CPnPNTePZsEaeSWvupkaypKL2bocBuZcwK6/c4IKE ITrhT2WTMxDiV5+h29f1ph5TQOHN72nEebggHtPnvoFI9nU50AaWb+QMr8oP ImerkQpLtvTwO3riLOY5arHXljf5X5IPtj+yDCD03QUoFLqELV+nnL8+v85v x3C0cL0n0TKm0zQpqvSoB1cXkZ1pCKATq8l7GFclR46P7a5PrDcVzsl+/p3X lqX94IoI+IIWqm7jVmOuMI2Pgo9c6FuprnG+bT997ivmucka4h/2ORNPbVt+ lz8hB1jU6dClgiaN1IdmzHDNFYniDFgnBWgfSN/N0qNZ2a84S1aTka+fr0ac MU8o =laAp -END PGP SIGNATURE- ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] Lost access when removing cache pool overlay
Dear all, I had to move .rgw.buckets.index pool to another structure; therefore, I created a new pool .rgw.buckets.index.new ; added the old pool as cache pool, and flushed the data. Up to this moment everything was ok. With radosgw -p df, I saw the objects moving to the new pool; the moved objects where ok, I could list omap keys and so on. When everything got moved, I removed the overlay cache pool. But at this moment, the objects became unresponsive: [(13:39:20) ceph@spchaog1 ~]$ rados -p .rgw.buckets.index listomapkeys .dir.default.198764998.1 error getting omap key set .rgw.buckets.index/.dir.default.198764998.1: (5) Input/output error That happens to all objects. When trying the access to the bucket through radosgw, I also get problems: [(13:16:01) root@spcogp1 ~]# radosgw-admin bucket stats --bucket="mybucket" error getting bucket stats ret=-2 Looking at the disk, data seems to be there: [(13:47:10) root@spcsnp1 ~]# ls /var/lib/ceph/osd/ceph-23/current/34.1f_head/|grep 198764998.1 \.dir.default.198764998.1__head_8A7482FF__22 Does anyone have a hint? Could I have lost ownership of the objects? Regards. -- As informações contidas nesta mensagem são CONFIDENCIAIS, protegidas pelo sigilo legal e por direitos autorais. A divulgação, distribuição, reprodução ou qualquer forma de utilização do teor deste documento depende de autorização do emissor, sujeitando-se o infrator às sanções legais. Caso esta comunicação tenha sido recebida por engano, favor avisar imediatamente, respondendo esta mensagem.___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] Lost access when removing cache pool overlay
Thank you for the response. It seems to me it is a transient situation. At this moment, I regained access to most, but not all buckets/index objects. But the overall performance dropped once again - I already have huge performance issues. Regards. Em 29-01-2016 14:41, Robert LeBlanc escreveu: -BEGIN PGP SIGNED MESSAGE- Hash: SHA256 Do the client key have access to the base pool? Something similar bit us when adding a caching tier. Since the cache tier may be proxying all the I/O, the client may not have had access to the base pool and it still worked ok. Once you removed the cache tier, it could no longer access the pool. - Robert LeBlanc PGP Fingerprint 79A2 9CA4 6CC4 45DD A904 C70E E654 3BB2 FA62 B9F1 On Fri, Jan 29, 2016 at 8:47 AM, Gerd Jakobovitsch wrote: Dear all, I had to move .rgw.buckets.index pool to another structure; therefore, I created a new pool .rgw.buckets.index.new ; added the old pool as cache pool, and flushed the data. Up to this moment everything was ok. With radosgw -p df, I saw the objects moving to the new pool; the moved objects where ok, I could list omap keys and so on. When everything got moved, I removed the overlay cache pool. But at this moment, the objects became unresponsive: [(13:39:20) ceph@spchaog1 ~]$ rados -p .rgw.buckets.index listomapkeys .dir.default.198764998.1 error getting omap key set .rgw.buckets.index/.dir.default.198764998.1: (5) Input/output error That happens to all objects. When trying the access to the bucket through radosgw, I also get problems: [(13:16:01) root@spcogp1 ~]# radosgw-admin bucket stats --bucket="mybucket" error getting bucket stats ret=-2 Looking at the disk, data seems to be there: [(13:47:10) root@spcsnp1 ~]# ls /var/lib/ceph/osd/ceph-23/current/34.1f_head/|grep 198764998.1 \.dir.default.198764998.1__head_8A7482FF__22 Does anyone have a hint? Could I have lost ownership of the objects? Regards. -- As informações contidas nesta mensagem são CONFIDENCIAIS, protegidas pelo sigilo legal e por direitos autorais. A divulgação, distribuição, reprodução ou qualquer forma de utilização do teor deste documento depende de autorização do emissor, sujeitando-se o infrator às sanções legais. Caso esta comunicação tenha sido recebida por engano, favor avisar imediatamente, respondendo esta mensagem. ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com -BEGIN PGP SIGNATURE- Version: Mailvelope v1.3.4 Comment: https://www.mailvelope.com wsFcBAEBCAAQBQJWq5YjCRDmVDuy+mK58QAAzUMQALqON8Ux5KPaotbyOMcr SzWVigfIa9Go1on8snKmehVzkwC25XaJxYQNU2OwsUUhHa1cy7v+rG6DTbDQ UDUK4IJ0O6ItGz4IeoyL06KyqmRy06OnuLRyzpQQD+nbIN+/82CVhMRMaKN+ U/GM+avDArN1JjjuQXFMgX/bS6ZoqJOBGqZKt3QWpJnkob1wgxP1tZA7MjZt p6Sfm/ci0dhveRhzylpEoxYKXwR6hN1hy/wiH2P5yeQBYYmpOALLDJDSTvln VZ/MbxPL5c0U/RRAkVMic1CvteeQ2nil2wEPFlu7cDjERvoBCMoyQeDXlep4 l+sAJbkKoOEKqE9xDo6CPnPNTePZsEaeSWvupkaypKL2bocBuZcwK6/c4IKE ITrhT2WTMxDiV5+h29f1ph5TQOHN72nEebggHtPnvoFI9nU50AaWb+QMr8oP ImerkQpLtvTwO3riLOY5arHXljf5X5IPtj+yDCD03QUoFLqELV+nnL8+v85v x3C0cL0n0TKm0zQpqvSoB1cXkZ1pCKATq8l7GFclR46P7a5PrDcVzsl+/p3X lqX94IoI+IIWqm7jVmOuMI2Pgo9c6FuprnG+bT997ivmucka4h/2ORNPbVt+ lz8hB1jU6dClgiaN1IdmzHDNFYniDFgnBWgfSN/N0qNZ2a84S1aTka+fr0ac MU8o =laAp -END PGP SIGNATURE- -- -- As informa��es contidas nesta mensagem s�o CONFIDENCIAIS, protegidas pelo sigilo legal e por direitos autorais. A divulga��o, distribui��o, reprodu��o ou qualquer forma de utiliza��o do teor deste documento depende de autoriza��o do emissor, sujeitando-se o infrator �s san��es legais. Caso esta comunica��o tenha sido recebida por engano, favor avisar imediatamente, respondendo esta mensagem.___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com