[
https://issues.apache.org/jira/browse/HBASE-12978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
stack updated HBASE-12978:
--------------------------
Attachment: e7cadabc6e5e46c7bf6b3d445f0c53cf
Here is the last hfile that had the missing Cell in it. The missing cell is:
{code}
754 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:regioninfo/1423438436466/Put/vlen=82/seqid=126045
{code}
If I try to get this row from this file, I get nothing back, though the file
has a bunch of entries on this row. Here is dump of the entries that are in
this file for this row:
{code}
754 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:regioninfo/1423438436466/Put/vlen=82/seqid=126045
755 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:seqnumDuringOpen/1423442164769/Put/vlen=8/seqid=130685
756 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:seqnumDuringOpen/1423442143845/Put/vlen=8/seqid=130562
757 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:seqnumDuringOpen/1423442046094/Put/vlen=8/seqid=130346
758 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:seqnumDuringOpen/1423441959921/Put/vlen=8/seqid=130285
759 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:seqnumDuringOpen/1423441807390/Put/vlen=8/seqid=129999
760 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:seqnumDuringOpen/1423441726587/Put/vlen=8/seqid=129821
761 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:seqnumDuringOpen/1423441661740/Put/vlen=8/seqid=129550
762 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:seqnumDuringOpen/1423441646296/Put/vlen=8/seqid=129459
763 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:seqnumDuringOpen/1423441601265/Put/vlen=8/seqid=129275
764 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:seqnumDuringOpen/1423441340085/Put/vlen=8/seqid=129031
765 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:server/1423442164769/Put/vlen=30/seqid=130685
766 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:server/1423442143845/Put/vlen=30/seqid=130562
767 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:server/1423442046094/Put/vlen=30/seqid=130346
768 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:server/1423441959921/Put/vlen=30/seqid=130285
769 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:server/1423441807390/Put/vlen=30/seqid=129999
770 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:server/1423441726587/Put/vlen=30/seqid=129821
771 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:server/1423441661740/Put/vlen=30/seqid=129550
772 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:server/1423441646296/Put/vlen=30/seqid=129459
773 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:server/1423441601265/Put/vlen=30/seqid=129275
774 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:server/1423441340085/Put/vlen=30/seqid=129031
775 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:serverstartcode/1423442164769/Put/vlen=8/seqid=130685
776 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:serverstartcode/1423442143845/Put/vlen=8/seqid=130562
777 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:serverstartcode/1423442046094/Put/vlen=8/seqid=130346
778 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:serverstartcode/1423441959921/Put/vlen=8/seqid=130285
779 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:serverstartcode/1423441807390/Put/vlen=8/seqid=129999
780 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:serverstartcode/1423441726587/Put/vlen=8/seqid=129821
781 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:serverstartcode/1423441661740/Put/vlen=8/seqid=129550
782 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:serverstartcode/1423441646296/Put/vlen=8/seqid=129459
783 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:serverstartcode/1423441601265/Put/vlen=8/seqid=129275
784 K:
IntegrationTestBigLinkedList,+\x84\xFF\xFC\xE4%\xF2\x11\xDE\x97t\xF0(\xF1$\xE8,1423438433508.014990fd6eb13141c04018f19c8910c8./info:serverstartcode/1423441340085/Put/vlen=8/seqid=129031
{code}
You can see this using the HFile tool.
The get that is failing is out of a Region#get (I have faked the region locally
but mimiciing the directory structure for my hbase:meta up on my cluster
locally and then hacking up a RegionTool to open the region and do gets.
So, the content is there but I can't get it. If I study which hfile block is
loaded, it is always the one AFTER the block that has the content I want.
Looking at the hfile index, it is actually damaged:
{code}
...
key=IntegrationTestBigLinkedList,(\xF5\xC2\x8F\x5C(\xF5\xC0,1423438433508.af43ed5cea6702e26970278287c4a06a./info:seqnumDuringOpen/1423441725896/Put
offset=107856, dataSize=8245
key=IntegrationTestBigLinkedList,,//LATEST_TIMESTAMP/Maximum
offset=116101, dataSize=8298
key=IntegrationTestBigLinkedList,.\x14W3,1423438609857.8e26a4c1da76844cacf51ddd568efb57./info:serverstartcode/1423442050289/Put
offset=124399, dataSize=8260
...
{code}
The key in the index for our block is:
key=IntegrationTestBigLinkedList,,//LATEST_TIMESTAMP/Maximum
Let me see if I can figure how that'd happen.
> hbase:meta has a row missing hregioninfo and it causes my long-running job to
> fail
> ----------------------------------------------------------------------------------
>
> Key: HBASE-12978
> URL: https://issues.apache.org/jira/browse/HBASE-12978
> Project: HBase
> Issue Type: Bug
> Reporter: stack
> Fix For: 1.0.1
>
> Attachments: e7cadabc6e5e46c7bf6b3d445f0c53cf
>
>
> Testing 1.0.0 trying long-running tests.
> A row in hbase:meta was missing its HRI entry. It caused the job to fail.
> Around the time of the first task failure, there are balances of the
> hbase:meta region and it was on a server that crashed. I tried to look at
> what happened around time of our writing hbase:meta and I ran into another
> issue; 20 logs of 256MBs filled with WrongRegionException written over a
> minute or two. The actual update of hbase:meta was not in the logs, it'd been
> rotated off.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
