That interesting information, unfortunately our failing server just failed completely, so copying the disks may be the only option at this point. It’s a JBOD setup, so we could pull the data off directly from the drives. It would have been nice to restore from the tool nice and clean. I had to use the tool to mark the PGs complete just so we could use the radosgw’s again.
-Brent From: Brady Deetz [mailto:bde...@gmail.com] Sent: Sunday, January 14, 2018 2:35 AM To: Brent Kennedy <bkenn...@cfl.rr.com> Cc: ceph-users <ceph-users@lists.ceph.com> Subject: Re: [ceph-users] Ceph-objectstore-tool import failure It's not simply a zip. I recently went through an incomplete pg incident as well. I'm not sure why your import is failing, but I do know that much. Here's a note in slack from our effort to reverse the export. I'm hoping to explore this a bit more in the next week. Data frames appear to have the following format: ceff DTDT SIZE PAYLOAD Size is probably in bytes? DTDT is frame type, 8-bits, repeated (so BEGIN=1 is encoded as 0101). File format looks like: Superblock - starts with: ceff ceff 0200 PG_BEGIN ceff 0101 <64 bit little-Endian size> PG_METADATA ceff 0909 OBJECT_BEGIN ceff 0303 <64 bit little-Endian size> (represents first object of a file?) TYPE_DATA ceff 0505 (represents an object?) (repeat TYPE_DATA frames until file completed) TYPE_ATTRS ceff 0606 TYPE_OMAP ceff 0808 OBJECT_END ceff 0404 (repeat above block for all files?) enum { TYPE_NONE = 0, TYPE_PG_BEGIN, TYPE_PG_END, TYPE_OBJECT_BEGIN, TYPE_OBJECT_END, TYPE_DATA, TYPE_ATTRS, TYPE_OMAP_HDR, TYPE_OMAP, TYPE_PG_METADATA, TYPE_POOL_BEGIN, TYPE_POOL_END, END_OF_TYPES, //Keep at the end }; On Jan 14, 2018 1:27 AM, "Brent Kennedy" <bkenn...@cfl.rr.com <mailto:bkenn...@cfl.rr.com> > wrote: I was able to bring a server back online for a short time and perform an export of the incomplete PGs I originally posted about last week. The export showed the files it was exporting and then dropped them all to a PGID.export file. I then SCP’ed the four PGID.export files to a server where I had an empty OSD weighted to 0. I stopped that OSD and then tried to import all four PGs. I then got the following messages for all four I tried: finish_remove_pgs 11.720_head removing 11.720 Importing pgid 11.c13 do_import threw exception error buffer::malformed_input: void object_stat_sum_t::decode(ceph::buffer::list::iterator&) decode past end of struct encoding Corrupt input for import Command I ran: ceph-objectstore-tool --op import --data-path /var/lib/ceph/osd/ceph-13 --journal-path /var/lib/ceph/osd/ceph-13/block --file 11.c13.export The files match the space used by PGs on the disk. As noted above, I saw it copy the PG to the export file successfully. Both servers are running Ubuntu 14 with the newest ceph-objectstore-tool installed via the package from here: http://download.ceph.com/debian-luminous/pool/main/c/ceph/ceph-test_12.2.2-1trusty_amd64.deb ( cluster is Luminous 12.2.2 . Its possible the PGs in question are on the jewel version as I wasn’t able to complete the upgrade to luminous on them. Am I missing something? Can I just copy the files off the failing server via a zip operation locally and then a unzip operation at the destination server? -Brent _______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com <mailto:ceph-users@lists.ceph.com> http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com
_______________________________________________ ceph-users mailing list ceph-users@lists.ceph.com http://lists.ceph.com/listinfo.cgi/ceph-users-ceph.com