Hi jms,How are you? Hope everything is ok:)We have encountered a strange
behavior in mob..some of our values contain only the reference cells..instead
of leading us to the mob we get hex value+ mob file name suffix..Have you seen
such behavior?We suspect the compact mob doesn't update the reference..Will
appreciate your thoughts on the issue..ThanksSent from my Samsung Galaxy
smartphone.
-------- Original message --------From: Omri Cohen <omri_co...@outlook.com>
Date: 8/17/19 16:42 (GMT+02:00) To: hbase-u...@hadoop.apache.org Subject: Re:
Mob reference tags go missing Hello,We recently encountered a problem in our
production hbase cluster (CDH deployment, version 5.13.1).We have a cluster
with a high mob percentage ( > 50% of objects are MOB, the threshold is the
default 102400 bytes). The cluster has been active for about a year. Recently,
our clients started to receive the mob reference instead of the mob data when
trying to access certain rows (in most tables about 10% of the rows are bad, in
one table it is about 50% of rows).When we investigated the HFiles, we saw that
the "bad" cells are missing two tags that exist in the "good" rows, it looks
something like that:K: {rowkey of a "good" row} {Column family, column
qualifier}... vlen=76/seqid=... V: \x00\x04\xFB.{MOB file name} T[0]: T[1]:
{table name}K: {rowkey of a "bad" row} {Column family, column qualifier}...
vlen=76/seqid=... V: \x00\x13\x1A{MOB file name}The "good" row returns the
expected data when queried, while the "bad" row returned the mob file reference
instead of the data. This happened when we queried from the REST server, the
native java client, and the hbase shell.When we examined the mob files, we saw
that all the data was there. * Has anyone encountered a similar situation?
* Is it possible to manually add the missing mob reference tags to the "bad"
rows?No changes where made to the table in recently. We never encountered this
issue before.We would appreciate any help on this
issue.________________________________From: Omri CohenSent: Saturday, August
17, 2019 4:28 PMTo: hbase-u...@hadoop.apache.org
<hbase-u...@hadoop.apache.org>Subject: Mob reference tags go missingHello,We
recently encountered a problem in our production hbase cluster.We have a
cluster with a high mob percentage ( > 50% of objects are MOB, the threshold is
the default 102400 bytes). The cluster has been active for about a year.
Recently, our clients started to receive the mob reference instead of the mob
data when trying to access certain rows (in most tables about 10% of the rows
are bad, in one table it is about 50% of rows).When we investigated the HFiles,
we saw that the "bad" cells are missing two tags that exist in the "good" rows,
it looks something like that:
