On Wed, Oct 17, 2018 at 1:14 AM Yang Yang <inksin...@gmail.com> wrote:
>
> Hi,
> A few weeks ago I found radosgw index has been inconsistent with reality.
> Some object I can not list, but I can get them by key. Please see the details
> below:
>
> BACKGROUND：
> Ceph version 12.2.4 (52085d5249a80c5f5121a76d6288429f35e4e77b) luminous
> (stable)
> Index pool is on ssd.
> There is a very big bucket with more than 10 million object and 500TB
> data.
> Ceph health is OK.
> I use s3 api on radosgw.
>
> DESCRIBE:
> When use s3 list_object() to list, some uploaded object can not be listed
> and some uploaded object have an old lastModified time.
> But at the same time, we can get this object by an exact key. And if I
> put a new object into this bucket, it can be listed.
> It seems that some indexes during a period of time have been lost.
>
> I try to run "radosgw-admin bucket check --bucket <bucketname> --fix
> --check-objects" and I get nothing at all.
>
> SOME ELSE：
> I found that one bucket will have many indexes, and we can use
> "radosgw-admin metadata list bucket.instance | grep "{bucket name}" to show
> them. But I can not found a doc to describe this feature. And we can use
> "radosgw-admin bucket stats --bucket {bucket_name}" to get id as the active
> instance id.
> I use "rados listomapkeys" at active(or latest) index to get all object
> in a index, it is really lost. But when I use "rados listomapkeys" at another
> index which is not active as mentioned above, I found the lost object index.
>
> Resharding is within my consideration. Listomapkeys means do this action
> on all shards(more than 300).
> In my understanding, a big bucket has one latest index and many old
> indexes. Every index has many shards. So listomapkeys on a index means
> listomapkeys on many shards.
>
> QUESTION:
> Why my index lost?
> How to recover？
I don't really know what happened, haven't seen this exact issue
before. You can try copying objects into themselves. That should
recreate their bucket index entry.
> Why radosgw has many index instances, how do radosgw use them and how to
> change active index?
Could be related to an existing bug. You can unlink the bucket and
then link a specific bucket instance version (to the user), however,
I'm not sure I recommend going this path if it isn't necessary.
Regards,
Yehuda
>
>
> Thanks,
>
> Inksink
>
> _______________________________________________
> ceph-users mailing list
> ceph-users@lists.ceph.com
> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________
ceph-users mailing list
ceph-users@lists.ceph.com
http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
