Re: [ceph-users] radosgw can still get the object even if this object's physical file is removed on OSDs
Thanks, Yehuda, for your reply. I think you may be right because I did another experiment which seems there might be a reference to an object. I am just curious to do that for some corner case. On Tue, Oct 15, 2013 at 12:39 AM, Yehuda Sadeh yeh...@inktank.com wrote: On Mon, Oct 14, 2013 at 4:04 AM, david zhang zhang.david2...@gmail.com wrote: Hi ceph-users, I uploaded an object successfully to radosgw with 3 replicas. And I located all the physical paths of 3 replicas on different OSDs. i.e, one of the 3 physical paths is /var/lib/ceph/osd/ceph-2/current/3.5_head/DIR_D/default.4896.65\\u20131014\\u1__head_0646563D__3 Then I manually deleted all the 3 replica files on OSDs, but this object can still get from radosgw with http code 200 even I cleaned all the caches on both radosgw and OSDs by 'echo 3 /proc/sys/vm/drop_caches'. Only after I restarted the 3 OSDs, get request will return 404. What did I miss? Is it not right to clean cache in that way? I'm not too sure what you're trying to achieve. You should never ever access the osd objects directly like that. The reason you're still able to read the objects is probably because the osd keeps open fds for recently opened files and it still holds a reference to them. If you need to remove objects off the rados backend you should use the rados tool to do that. However, since you created the objects via radosgw, you're going to have some radosgw consistency issues, so in that case the way to go would be by going through radosgw-admin (or through the radosgw RESTful api). Yehuda -- Regards, Zhi ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
[ceph-users] radosgw can still get the object even if this object's physical file is removed on OSDs
Hi ceph-users, I uploaded an object successfully to radosgw with 3 replicas. And I located all the physical paths of 3 replicas on different OSDs. i.e, one of the 3 physical paths is /var/lib/ceph/osd/ceph-2/current/3.5_head/DIR_D/default.4896.65\\u20131014\\u1__head_0646563D__3 Then I manually deleted all the 3 replica files on OSDs, but this object can still get from radosgw with http code 200 even I cleaned all the caches on both radosgw and OSDs by 'echo 3 /proc/sys/vm/drop_caches'. Only after I restarted the 3 OSDs, get request will return 404. What did I miss? Is it not right to clean cache in that way? Thanks. -- Regards, Zhi ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
Re: [ceph-users] radosgw can still get the object even if this object's physical file is removed on OSDs
On Mon, Oct 14, 2013 at 4:04 AM, david zhang zhang.david2...@gmail.com wrote: Hi ceph-users, I uploaded an object successfully to radosgw with 3 replicas. And I located all the physical paths of 3 replicas on different OSDs. i.e, one of the 3 physical paths is /var/lib/ceph/osd/ceph-2/current/3.5_head/DIR_D/default.4896.65\\u20131014\\u1__head_0646563D__3 Then I manually deleted all the 3 replica files on OSDs, but this object can still get from radosgw with http code 200 even I cleaned all the caches on both radosgw and OSDs by 'echo 3 /proc/sys/vm/drop_caches'. Only after I restarted the 3 OSDs, get request will return 404. What did I miss? Is it not right to clean cache in that way? I'm not too sure what you're trying to achieve. You should never ever access the osd objects directly like that. The reason you're still able to read the objects is probably because the osd keeps open fds for recently opened files and it still holds a reference to them. If you need to remove objects off the rados backend you should use the rados tool to do that. However, since you created the objects via radosgw, you're going to have some radosgw consistency issues, so in that case the way to go would be by going through radosgw-admin (or through the radosgw RESTful api). Yehuda ___ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
