Have you examined the wals for writes to the impacted cells to verify an update wasn't written with the change to the value?
On Wed, Mar 20, 2019, 17:47 Austin Heyne <[email protected]> wrote: > Hey all, > > We're running HBase 1.4.8 on EMR 5.20 backed by S3 and we're seeing a > bit get flipped in some record values. > > We've preformed a bulk ingest and bulk load of a large chunk of data and > then pointed a live ingest feed to that table. After a period of time we > found that a few records in the table had been corrupted and were one > bit different from their original value. Since we saved the output of > the bulk ingest we re-loaded those files and verified that at the time > of bulk load the record was correct. This seems to us to indicate that > at some point during the live ingest writes the record was corrupted. > > I've verified that the region that the record is in has never been split > but it has received over 2 million write requests so there very likely > could have been some minor compactions there. > > Has anyone seen anything like this before? > > Thanks, > Austin > > -- > Austin L. Heyne > >
