Hello folks.
I got a failed cluster recovery job and the cluster is: 2PB EC 8+2 pool
used as RGW S3 storage. (14.2.16 nautilus)
I fixed most of the issues and rescued a lot of failed OSD's but some OSD's
are not recoverable and the cluster size is huge, recovery takes time.
Fixing all the issues is impossible because missing OSD's and running PG
repair take weeks. I did my best and the current state is enough for
starting data export.
I only need to export the healthy data first and then I'm gonna
destroy-rebuild the cluster.
I tried to export data by using rclone and it runs good at the beginning
but when the radosgw service hits an unfound object it slows down and it
freezes.
Because of this issue I can not export data by using rclone+radosgw
I'm using these parameters for the export:
> rclone copy --files-from=object-list.txt --progress --checkers=8
> --transfers=8 --no-traverse \
> --no-update-modtime --ignore-checksum --ignore-size --no-gzip-encoding
> --s3-no-check-bucket \
> --buffer-size=1G --low-level-retries=1 --retries=1 --retries-sleep=100ms
> --timeout=30s \
> --contimeout=30s --ignore-errors --ignore-existing
> --log-file=export-rclone.log --log-level=INFO
Current state is:
> # ceph health detail (14.2.16 nautilus)
> HEALTH_ERR 3303 large omap objects; 1460729/1238267531 objects unfound
> (0.118%); Possible data damage: 2 pgs recovery_unfound, 1 pg
> backfill_unfound; Degraded data redundancy: 5487055/12315635595 objects
> degraded (0.045%), 7 pgs degraded, 7 pgs undersized; 1987 pgs not
> deep-scrubbed in time; 1824 pgs not scrubbed in time
> LARGE_OMAP_OBJECTS 3303 large omap objects
> 165 large objects found in pool 'xyz.rgw.log'
> 3138 large objects found in pool 'xyz.rgw.buckets.index'
> Search the cluster log for 'Large omap object found' for more details.
> OBJECT_UNFOUND 1460729/1238267531 objects unfound (0.118%)
> pg 56.73a has 99125 unfound objects
> pg 56.470 has 572028 unfound objects
> pg 56.3a9 has 564371 unfound objects
> pg 56.346 has 7871 unfound objects
> pg 56.36 has 1 unfound objects
> pg 56.1e2 has 108269 unfound objects
> pg 56.204 has 109064 unfound objects
> PG_DAMAGED Possible data damage: 2 pgs recovery_unfound, 1 pg
> backfill_unfound
> pg 56.36 is active+backfill_unfound+undersized+degraded+remapped,
> acting [166,36,7,46,152,193,128,223,80,2147483647], 1 unfound
> pg 56.3a9 is active+recovery_unfound+undersized+degraded+remapped,
> acting [153,69,177,156,49,167,2,2147483647,2147483647,31], 564371 unfound
> pg 56.470 is active+recovery_unfound+undersized+degraded+remapped,
> acting [135,2147483647,2147483647,176,37,85,65,48,183,224], 572028 unfound
> PG_DEGRADED Degraded data redundancy: 5487055/12315635595 objects degraded
> (0.045%), 7 pgs degraded, 7 pgs undersized
> pg 56.36 is stuck undersized for 47661.320358, current state
> active+backfill_unfound+undersized+degraded+remapped, last acting
> [166,36,7,46,152,193,128,223,80,2147483647]
> pg 56.1e2 is stuck undersized for 47660.312165, current state
> active+recovering+undersized+degraded+remapped, last acting
> [2147483647,127,77,140,167,195,220,131,113,31]
> pg 56.204 is stuck undersized for 47660.325926, current state
> active+recovering+undersized+degraded+remapped, last acting
> [2147483647,42,173,17,64,189,22,113,136,158]
> pg 56.346 is stuck undersized for 47660.335077, current state
> active+recovering+undersized+degraded+remapped, last acting
> [63,108,53,140,132,2147483647,183,171,0,98]
> pg 56.3a9 is stuck undersized for 4403604.202577, current state
> active+recovery_unfound+undersized+degraded+remapped, last acting
> [153,69,177,156,49,167,2,2147483647,2147483647,31]
> pg 56.470 is stuck undersized for 47661.762140, current state
> active+recovery_unfound+undersized+degraded+remapped, last acting
> [135,2147483647,2147483647,176,37,85,65,48,183,224]
> pg 56.73a is stuck undersized for 47656.126741, current state
> active+recovering+undersized+degraded+remapped, last acting
> [2147483647,189,85,155,170,106,40,25,222,121]
> PG_NOT_DEEP_SCRUBBED 1987 pgs not deep-scrubbed in time
I considered "bucket check" but each bucket has 10M objects and 100TB of
data. It will take days-weeks and it may fail because of the cluster state.
> "rgw.main": {
> "size": 106443540756257,
> "size_actual": 106463698989056,
> "size_utilized": 106443540756257,
> "size_kb": 103948770270,
> "size_kb_actual": 103968456044,
> "size_kb_utilized": 103948770270,
> "num_objects": 9855872
> },
I'm thinking of writing a rados exporter script and I need to do this:
1. Read bucket index and have a list of objects.
2. Check the bucket data by each object and eliminate the unfound
objects and generate a clean list with healthy objects
3. Export the data directly from rados to an NFS share by combining
multipart data.
My questions are:
1. Are there any other ways to use radosgw+rclone for fast export from
ERROR state cluster?
2. If not, then do you know any rados export tool I can use?
I really need your advice. What is the best approach to this?
Thank you.
_______________________________________________
ceph-users mailing list -- [email protected]
To unsubscribe send an email to [email protected]
