On Fri, May 27, 2016 at 3:17 PM, Tianying Chang <[email protected]> wrote:
> Yes, it is 94.26. By a quick glance, I didn't see any put that is older
> than the delete marker's TS, which could go as far as about couple weeks
> ago since major compaction on it for long time seems.
>
Also it is really strange that if the region is split, then seems
> everything is working as expected. Also we noticed, the same region
> replicated at the slave side is totally normal, i.e. at 20+G....
>
>
If you move the region to another server, does that work?
Looking in 0.94 codebase, I see this in Compactor#compact
// For major compactions calculate the earliest put timestamp
// of all involved storefiles. This is used to remove
// family delete marker during the compaction.
if (majorCompaction) {
tmp = fileInfo.get(StoreFile.EARLIEST_PUT_TS);
if (tmp == null) {
// There's a file with no information, must be an old one
// assume we have very old puts
earliestPutTs = HConstants.OLDEST_TIMESTAMP;
} else {
earliestPutTs = Math.min(earliestPutTs, Bytes.toLong(tmp));
}
}
The above is followed by this log line:
if (LOG.isDebugEnabled()) {
LOG.debug("Compacting " + file +
", keycount=" + keyCount +
", bloomtype=" + r.getBloomFilterType().toString() +
", size=" + StringUtils.humanReadableInt(r.length()) +
", encoding=" + r.getHFileReader().getEncodingOnDisk() +
(majorCompaction? ", earliestPutTs=" + earliestPutTs: ""));
}
This prints out earliestPutTs. You see that in the logs? You running with
DEBUG? Does the earliest put ts preclude our dropping delete family?
Looking more in code, we retain deletes in following circumstances:
this.retainDeletesInOutput = scanType == ScanType.MINOR_COMPACT || scan
.isRaw();
So, for sure we are running major compaction?
Otherwise, have to dig in a bit more here.. This stuff is a little involved.
St.Ack
> On Fri, May 27, 2016 at 3:13 PM, Stack <[email protected]> wrote:
>
> > On Fri, May 27, 2016 at 2:32 PM, Tianying Chang <[email protected]>
> wrote:
> >
> > > Hi,
> > >
> > > We saw a very strange case in one of our production cluster. A couple
> > > regions cannot get their deleted rows or delete marker removed even
> after
> > > major compaction. However when the region triggered split (we set 100G
> > for
> > > auto split), the deletion worked. The 100G region becomes two 10G
> > daughter
> > > regions, and all the delete marker are gone.
> > >
> > > Also, the same region in the slave cluster (through replication) have
> > > normal size at about 20+G.
> > >
> > > BTW, the delete marker in the regions are mostly deleteFamily if it
> > > matters.
> > >
> > > This is really weird. Anyone has any clue for this strange behavior?
> > >
> > > Thanks
> > > Tian-Ying
> > >
> > > These 0.94 Tian-Ying?
> >
> > It looks like the DeleteFamily is retained only; do you see incidence
> where
> > there may have been versions older than the DeleteFamily that are also
> > retained post-major-compaction?
> >
> > St.Ack
> >
> >
> >
> > > A snippet of the HFile generated by the major compaction:
> > >
> > > : \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > \x10\x00?PF/d:/1459808114380/DeleteFamily/vlen=0/ts=2292870047
> > > V:
> > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > \x10\x00?PF/d:/1459808114011/DeleteFamily/vlen=0/ts=2292869794
> > > V:
> > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > \x10\x00?PF/d:/1459805381104/DeleteFamily/vlen=0/ts=2291072240
> > > V:
> > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > \x10\x00?PF/d:/1459805380673/DeleteFamily/vlen=0/ts=2291071997
> > > V:
> > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > \x10\x00?PF/d:/1459802643449/DeleteFamily/vlen=0/ts=2290248886
> > > V:
> > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > \x10\x00?PF/d:/1459802643246/DeleteFamily/vlen=0/ts=2290248786
> > > V:
> > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > \x10\x00?PF/d:/1459799913003/DeleteFamily/vlen=0/ts=2289446916
> > > V:
> > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > \x10\x00?PF/d:/1459797181831/DeleteFamily/vlen=0/ts=2288670451
> > > V:
> > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > \x10\x00?PF/d:/1459794447388/DeleteFamily/vlen=0/ts=2287911443
> > > V:
> > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > \x10\x00?PF/d:/1459791708803/DeleteFamily/vlen=0/ts=2287213792
> > > V:
> > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > \x10\x00?PF/d:/1459788978387/DeleteFamily/vlen=0/ts=2286488738
> > > V:
> > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@
> > > \x10\x00?PF/d:/1459786243642/DeleteFamily/vlen=0/ts=2285778927
> > > V:
> > >
> >
>
