[ https://issues.apache.org/jira/browse/HBASE-23950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kai Xie updated HBASE-23950: ---------------------------- Description: My HBase 2.0/2.1 cluster experienced severe performance drop after running reversed scanning for days. When debug into regionserver logs, it appears the bucket cache is full for every ~100 ms and its eviction can't work properly because ref count has become negative: {code:java} (every ~100 ms) Free started because "Full!" This block AAA is still referred by -1 readers. Cannot be freed now This block BBB is still referred by -2 readers. Cannot be freed now This block CCC is still referred by -3 readers. Cannot be freed now ... This block ... is still referred by (negative number) readers. Cannot be freed now {code} There have been patch HBASE-22480 addressing this ref count problem in 2.2+. I'd like to backport it to 2.1 (understand that 2.0 is EOL) was: My HBase 2.0/2.1 cluster experienced severe performance drop after running reversed scanning for days. When debug into regionserver logs, it appears the bucket cache is full for every ~100 ms and its eviction can't work properly because ref count has become negative: {code:java} (every ~100 ms) Free started because "Full!" This block AAA is still referred by -1 readers. Cannot be freed now This block BBB is still referred by -2 readers. Cannot be freed now This block CCC is still referred by -3 readers. Cannot be freed now ... This block ... is still referred by (negative number) readers. Cannot be freed now {code} There have been patches HBASE-22480 addressing this ref count problem in 2.2+. I'd like to backport it to 2.1 (understand that 2.0 is EOL) > Backport HBASE-22480 to branch-2.1 > ---------------------------------- > > Key: HBASE-23950 > URL: https://issues.apache.org/jira/browse/HBASE-23950 > Project: HBase > Issue Type: Bug > Components: BucketCache > Affects Versions: 2.1.9 > Reporter: Kai Xie > Priority: Major > > My HBase 2.0/2.1 cluster experienced severe performance drop after running > reversed scanning for days. When debug into regionserver logs, it appears the > bucket cache is full for every ~100 ms and its eviction can't work properly > because ref count has become negative: > {code:java} > (every ~100 ms) > Free started because "Full!" > This block AAA is still referred by -1 readers. Cannot be freed now > This block BBB is still referred by -2 readers. Cannot be freed now > This block CCC is still referred by -3 readers. Cannot be freed now > ... > This block ... is still referred by (negative number) readers. Cannot be > freed now > {code} > There have been patch HBASE-22480 addressing this ref count problem in 2.2+. > I'd like to backport it to 2.1 (understand that 2.0 is EOL) > -- This message was sent by Atlassian Jira (v8.3.4#803005)