[
https://issues.apache.org/jira/browse/HBASE-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12736225#action_12736225
]
Jonathan Gray commented on HBASE-1715:
--------------------------------------
It seems really unlikely to me that it's in the normal Put codepath. It's
fairly straightforward, and things are divided up by family from the
client-side on... families are never mixed in a List or anything, ever, on the
Put codepath... Hard to see where they'd switch. It seems more likely to me
that this is happening during compactions, splits, assignment/reassignment, log
replays, etc... Though no stone should be left unturned.
We should debug on the way in though, so we can see when it's actually
happening. I was writing a patch to check HFile.Writer to make sure it never
wrote out a different family, but then saw this in the append() code:
{code}
if (this.lastKeyBuffer != null) {
if (this.comparator.compare(this.lastKeyBuffer, this.lastKeyOffset,
this.lastKeyLength, key, offset, length) > 0) {
throw new IOException("Added a key not lexically larger than" +
" previous key=" + Bytes.toString(key, offset, length) +
", lastkey=" + Bytes.toString(this.lastKeyBuffer,
this.lastKeyOffset,
this.lastKeyLength));
}
}
{code}
That is being called for every KV that is appended to the HFile.Writer. It
seems that this would throw an exception if we ever had the wrong family
intermixed with another one, however would not work at the edges if the wrong
family was first or last (and appropriately ordered as to not be triggered
here).
> compaction failure in ScanWildcardColumnTracker.checkColumn
> -----------------------------------------------------------
>
> Key: HBASE-1715
> URL: https://issues.apache.org/jira/browse/HBASE-1715
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: Andrew Purtell
> Priority: Blocker
> Fix For: 0.20.0
>
> Attachments: HBASE-1715-v2.patch, HBASE-1715-v3.patch,
> HBASE-1715.patch
>
>
> I have at least one region that won't compact.
> {noformat}
> hbase> status 'detailed'
> [...]
> content,0b41cdd2ee9b36b0674ad423089800ba,1248257816633
> stores=3, storefiles=113, storefileSizeMB=1213, memstoreSizeMB=0,
> storefileIndexSizeMB=0
> [...]
> hbase> major_compact 'content'
> [...]
> hbase> status 'detailed'
> [...]
> content,0b41cdd2ee9b36b0674ad423089800ba,1248257816633
> stores=3, storefiles=113, storefileSizeMB=1213, memstoreSizeMB=0,
> storefileIndexSizeMB=0
> [...]
> hbase> major_compact 'content,0b41cdd2ee9b36b0674ad423089800ba,1248257816633'
> [...]
> hbase> status 'detailed'
> [...]
> content,0b41cdd2ee9b36b0674ad423089800ba,1248257816633
> stores=3, storefiles=113, storefileSizeMB=1213, memstoreSizeMB=0,
> storefileIndexSizeMB=0
> [...]
> {noformat}
> {noformat}
> $ hadoop fs -ls /hbase/content/1226419153/content
> Found 37 items
> -rw-r--r-- 2 hadoop supergroup 86422089 2009-07-24 16:11
> /hbase/content/1226419153/content/1045181559873943545
> -rw-r--r-- 2 hadoop supergroup 7167852 2009-07-26 05:13
> /hbase/content/1226419153/content/1225211885117827793
> -rw-r--r-- 2 hadoop supergroup 2678 2009-07-24 20:24
> /hbase/content/1226419153/content/1275251824404920815
> -rw-r--r-- 2 hadoop supergroup 10288489 2009-07-26 02:16
> /hbase/content/1226419153/content/1386224161790061095
> -rw-r--r-- 2 hadoop supergroup 16111798 2009-07-24 23:43
> /hbase/content/1226419153/content/1454452327579297030
> -rw-r--r-- 2 hadoop supergroup 12650208 2009-07-26 23:31
> /hbase/content/1226419153/content/1534514745508157864
> -rw-r--r-- 2 hadoop supergroup 6994590 2009-07-27 10:45
> /hbase/content/1226419153/content/1706519919143970421
> -rw-r--r-- 2 hadoop supergroup 13449537 2009-07-26 20:53
> /hbase/content/1226419153/content/2186668356269910871
> -rw-r--r-- 2 hadoop supergroup 10397457 2009-07-26 22:15
> /hbase/content/1226419153/content/2292706332643873473
> -rw-r--r-- 2 hadoop supergroup 6481576 2009-07-27 05:59
> /hbase/content/1226419153/content/2438489739664745000
> -rw-r--r-- 2 hadoop supergroup 10810191 2009-07-25 19:48
> /hbase/content/1226419153/content/2538008511360230014
> -rw-r--r-- 2 hadoop supergroup 13877559 2009-07-27 18:26
> /hbase/content/1226419153/content/2772019170563217117
> -rw-r--r-- 2 hadoop supergroup 12796040 2009-07-26 10:14
> /hbase/content/1226419153/content/2902101782772083009
> -rw-r--r-- 2 hadoop supergroup 6541657 2009-07-27 01:06
> /hbase/content/1226419153/content/3113808018684114931
> -rw-r--r-- 2 hadoop supergroup 8655428 2009-07-25 18:17
> /hbase/content/1226419153/content/3817932211925236778
> -rw-r--r-- 2 hadoop supergroup 1445535 2009-07-27 16:10
> /hbase/content/1226419153/content/4286233593585189878
> -rw-r--r-- 2 hadoop supergroup 9139508 2009-07-27 02:11
> /hbase/content/1226419153/content/495340788313226264
> -rw-r--r-- 2 hadoop supergroup 3313459 2009-07-26 05:57
> /hbase/content/1226419153/content/5167064731599803595
> -rw-r--r-- 2 hadoop supergroup 9473393 2009-07-26 14:44
> /hbase/content/1226419153/content/5490426319631514899
> -rw-r--r-- 2 hadoop supergroup 9321224 2009-07-26 19:40
> /hbase/content/1226419153/content/5790825797519034907
> -rw-r--r-- 2 hadoop supergroup 12171283 2009-07-27 12:20
> /hbase/content/1226419153/content/6036401533248383324
> -rw-r--r-- 2 hadoop supergroup 824790136 2009-07-23 22:28
> /hbase/content/1226419153/content/6211942192349190964
> -rw-r--r-- 2 hadoop supergroup 9905606 2009-07-27 07:45
> /hbase/content/1226419153/content/6295275445036553977
> -rw-r--r-- 2 hadoop supergroup 23857510 2009-07-27 17:02
> /hbase/content/1226419153/content/6535266251812885635
> -rw-r--r-- 2 hadoop supergroup 9177439 2009-07-26 11:21
> /hbase/content/1226419153/content/6625735185665629662
> -rw-r--r-- 2 hadoop supergroup 6916543 2009-07-27 20:15
> /hbase/content/1226419153/content/6934569497672884872
> -rw-r--r-- 2 hadoop supergroup 8818427 2009-07-27 13:48
> /hbase/content/1226419153/content/7162767181372457089
> -rw-r--r-- 2 hadoop supergroup 12028925 2009-07-25 08:48
> /hbase/content/1226419153/content/7254368961746328584
> -rw-r--r-- 2 hadoop supergroup 14695089 2009-07-27 09:16
> /hbase/content/1226419153/content/728058506789102941
> -rw-r--r-- 2 hadoop supergroup 9154325 2009-07-27 03:43
> /hbase/content/1226419153/content/7462549804082617977
> -rw-r--r-- 2 hadoop supergroup 22295283 2009-07-27 21:46
> /hbase/content/1226419153/content/7498645200757573769
> -rw-r--r-- 2 hadoop supergroup 5414671 2009-07-27 15:27
> /hbase/content/1226419153/content/758918234053985701
> -rw-r--r-- 2 hadoop supergroup 11260015 2009-07-26 08:16
> /hbase/content/1226419153/content/7626050291010778604
> -rw-r--r-- 2 hadoop supergroup 11309116 2009-07-26 12:57
> /hbase/content/1226419153/content/7704085331915902813
> -rw-r--r-- 2 hadoop supergroup 6326623 2009-07-26 17:58
> /hbase/content/1226419153/content/7735737241739599956
> -rw-r--r-- 2 hadoop supergroup 832 2009-07-24 20:15
> /hbase/content/1226419153/content/8170327402351444879
> -rw-r--r-- 2 hadoop supergroup 13480887 2009-07-26 16:32
> /hbase/content/1226419153/content/890729481029340216
> {noformat}
> Relevant nonstandard options:
> {noformat}
> <property>
> <name>hbase.hregion.max.filesize</name>
> <value>1073741824</value>
> </property>
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
