Hi, Stack Based on the log, the major compaction was run, and it took 5+ hours. And I also manually run major_compact from hbase shell explicitly to verify.
I just moved the region to a different RS and issued a major_compact on that region again, let me see if the major compaction can succeed and will report back. Thanks Tian-Ying On Sun, May 29, 2016 at 4:35 PM, Stack <[email protected]> wrote: > On Fri, May 27, 2016 at 3:17 PM, Tianying Chang <[email protected]> wrote: > > > Yes, it is 94.26. By a quick glance, I didn't see any put that is older > > than the delete marker's TS, which could go as far as about couple weeks > > ago since major compaction on it for long time seems. > > > Also it is really strange that if the region is split, then seems > > everything is working as expected. Also we noticed, the same region > > replicated at the slave side is totally normal, i.e. at 20+G.... > > > > > If you move the region to another server, does that work? > > Looking in 0.94 codebase, I see this in Compactor#compact > > > // For major compactions calculate the earliest put timestamp > > // of all involved storefiles. This is used to remove > > // family delete marker during the compaction. > > if (majorCompaction) { > > tmp = fileInfo.get(StoreFile.EARLIEST_PUT_TS); > > if (tmp == null) { > > // There's a file with no information, must be an old one > > // assume we have very old puts > > earliestPutTs = HConstants.OLDEST_TIMESTAMP; > > } else { > > earliestPutTs = Math.min(earliestPutTs, Bytes.toLong(tmp)); > > } > > } > > > The above is followed by this log line: > > > if (LOG.isDebugEnabled()) { > > LOG.debug("Compacting " + file + > > ", keycount=" + keyCount + > > ", bloomtype=" + r.getBloomFilterType().toString() + > > ", size=" + StringUtils.humanReadableInt(r.length()) + > > ", encoding=" + r.getHFileReader().getEncodingOnDisk() + > > (majorCompaction? ", earliestPutTs=" + earliestPutTs: "")); > > } > > This prints out earliestPutTs. You see that in the logs? You running with > DEBUG? Does the earliest put ts preclude our dropping delete family? > > > Looking more in code, we retain deletes in following circumstances: > > > this.retainDeletesInOutput = scanType == ScanType.MINOR_COMPACT || scan > .isRaw(); > > > So, for sure we are running major compaction? > > Otherwise, have to dig in a bit more here.. This stuff is a little > involved. > St.Ack > > > > > > On Fri, May 27, 2016 at 3:13 PM, Stack <[email protected]> wrote: > > > > > On Fri, May 27, 2016 at 2:32 PM, Tianying Chang <[email protected]> > > wrote: > > > > > > > Hi, > > > > > > > > We saw a very strange case in one of our production cluster. A couple > > > > regions cannot get their deleted rows or delete marker removed even > > after > > > > major compaction. However when the region triggered split (we set > 100G > > > for > > > > auto split), the deletion worked. The 100G region becomes two 10G > > > daughter > > > > regions, and all the delete marker are gone. > > > > > > > > Also, the same region in the slave cluster (through replication) have > > > > normal size at about 20+G. > > > > > > > > BTW, the delete marker in the regions are mostly deleteFamily if it > > > > matters. > > > > > > > > This is really weird. Anyone has any clue for this strange behavior? > > > > > > > > Thanks > > > > Tian-Ying > > > > > > > > These 0.94 Tian-Ying? > > > > > > It looks like the DeleteFamily is retained only; do you see incidence > > where > > > there may have been versions older than the DeleteFamily that are also > > > retained post-major-compaction? > > > > > > St.Ack > > > > > > > > > > > > > A snippet of the HFile generated by the major compaction: > > > > > > > > : \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > > > > \x10\x00?PF/d:/1459808114380/DeleteFamily/vlen=0/ts=2292870047 > > > > V: > > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > > > > \x10\x00?PF/d:/1459808114011/DeleteFamily/vlen=0/ts=2292869794 > > > > V: > > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > > > > \x10\x00?PF/d:/1459805381104/DeleteFamily/vlen=0/ts=2291072240 > > > > V: > > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > > > > \x10\x00?PF/d:/1459805380673/DeleteFamily/vlen=0/ts=2291071997 > > > > V: > > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > > > > \x10\x00?PF/d:/1459802643449/DeleteFamily/vlen=0/ts=2290248886 > > > > V: > > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > > > > \x10\x00?PF/d:/1459802643246/DeleteFamily/vlen=0/ts=2290248786 > > > > V: > > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > > > > \x10\x00?PF/d:/1459799913003/DeleteFamily/vlen=0/ts=2289446916 > > > > V: > > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > > > > \x10\x00?PF/d:/1459797181831/DeleteFamily/vlen=0/ts=2288670451 > > > > V: > > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > > > > \x10\x00?PF/d:/1459794447388/DeleteFamily/vlen=0/ts=2287911443 > > > > V: > > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > > > > \x10\x00?PF/d:/1459791708803/DeleteFamily/vlen=0/ts=2287213792 > > > > V: > > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > > > > \x10\x00?PF/d:/1459788978387/DeleteFamily/vlen=0/ts=2286488738 > > > > V: > > > > K: \xA0\x00\x00L\x1A@\x1CBe\x00\x00\x08m\x03\x1A@ > > > > \x10\x00?PF/d:/1459786243642/DeleteFamily/vlen=0/ts=2285778927 > > > > V: > > > > > > > > > >
