[jira] [Commented] (HBASE-22705) IllegalArgumentException exception occured during MobFileCache eviction
[
https://issues.apache.org/jira/browse/HBASE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17019672#comment-17019672
]
Michael Stack commented on HBASE-22705:
---
Unscheduling. No update. Looks like real issue.
> IllegalArgumentException exception occured during MobFileCache eviction
> ---
>
> Key: HBASE-22705
> URL: https://issues.apache.org/jira/browse/HBASE-22705
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.5
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Attachments: HBASE-22705.branch-2.patch
>
>
> IllegalArgumentException occured during scan operation,
> {noformat}
> 2019-07-08 01:46:57,764 | ERROR |
> RpcServer.FifoWFPBQ.default.handler=129,queue=9,port=21302 | Unexpected
> throwable object |
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2502)
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.ComparableTimSort.mergeHi(ComparableTimSort.java:866)
> at java.util.ComparableTimSort.mergeAt(ComparableTimSort.java:483)
> at
> java.util.ComparableTimSort.mergeForceCollapse(ComparableTimSort.java:422)
> at java.util.ComparableTimSort.sort(ComparableTimSort.java:222)
> at java.util.Arrays.sort(Arrays.java:1312)
> at java.util.Arrays.sort(Arrays.java:1506)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:141)
> at org.apache.hadoop.hbase.mob.MobFileCache.evict(MobFileCache.java:144)
> at
> org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:214)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:397)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:358)
> at
> org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:74)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:150)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22705) IllegalArgumentException exception occured during MobFileCache eviction
[
https://issues.apache.org/jira/browse/HBASE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16968912#comment-16968912
]
Junhong Xu commented on HBASE-22705:
There is no multi-thread access here, priority queue will be enough?
> IllegalArgumentException exception occured during MobFileCache eviction
> ---
>
> Key: HBASE-22705
> URL: https://issues.apache.org/jira/browse/HBASE-22705
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.5
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.3.0
>
> Attachments: HBASE-22705.branch-2.patch
>
>
> IllegalArgumentException occured during scan operation,
> {noformat}
> 2019-07-08 01:46:57,764 | ERROR |
> RpcServer.FifoWFPBQ.default.handler=129,queue=9,port=21302 | Unexpected
> throwable object |
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2502)
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.ComparableTimSort.mergeHi(ComparableTimSort.java:866)
> at java.util.ComparableTimSort.mergeAt(ComparableTimSort.java:483)
> at
> java.util.ComparableTimSort.mergeForceCollapse(ComparableTimSort.java:422)
> at java.util.ComparableTimSort.sort(ComparableTimSort.java:222)
> at java.util.Arrays.sort(Arrays.java:1312)
> at java.util.Arrays.sort(Arrays.java:1506)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:141)
> at org.apache.hadoop.hbase.mob.MobFileCache.evict(MobFileCache.java:144)
> at
> org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:214)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:397)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:358)
> at
> org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:74)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:150)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22705) IllegalArgumentException exception occured during MobFileCache eviction
[
https://issues.apache.org/jira/browse/HBASE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16968877#comment-16968877
]
Junhong Xu commented on HBASE-22705:
The code is something like that
{code:java}
PriorityBlockingQueue files = new
PriorityBlockingQueue<>(map.values());
int start = (int) (mobFileMaxCacheSize * evictRemainRatio);
int i = 0;
while((start >= 0) && !files.isEmpty()) {
CachedMobFile cachedMobFile = files.poll();
if (i++ >= start) {
String name = cachedMobFile.getFileName();
CachedMobFile evictedFile = map.remove(name);
if (evictedFile != null) {
evictedFiles.add(evictedFile);
}
}
}
{code}
> IllegalArgumentException exception occured during MobFileCache eviction
> ---
>
> Key: HBASE-22705
> URL: https://issues.apache.org/jira/browse/HBASE-22705
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.5
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.3.0
>
> Attachments: HBASE-22705.branch-2.patch
>
>
> IllegalArgumentException occured during scan operation,
> {noformat}
> 2019-07-08 01:46:57,764 | ERROR |
> RpcServer.FifoWFPBQ.default.handler=129,queue=9,port=21302 | Unexpected
> throwable object |
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2502)
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.ComparableTimSort.mergeHi(ComparableTimSort.java:866)
> at java.util.ComparableTimSort.mergeAt(ComparableTimSort.java:483)
> at
> java.util.ComparableTimSort.mergeForceCollapse(ComparableTimSort.java:422)
> at java.util.ComparableTimSort.sort(ComparableTimSort.java:222)
> at java.util.Arrays.sort(Arrays.java:1312)
> at java.util.Arrays.sort(Arrays.java:1506)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:141)
> at org.apache.hadoop.hbase.mob.MobFileCache.evict(MobFileCache.java:144)
> at
> org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:214)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:397)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:358)
> at
> org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:74)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:150)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22705) IllegalArgumentException exception occured during MobFileCache eviction
[
https://issues.apache.org/jira/browse/HBASE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16968427#comment-16968427
]
Junhong Xu commented on HBASE-22705:
I encountered the failure too, is it possible to use thread safe priority queue?
> IllegalArgumentException exception occured during MobFileCache eviction
> ---
>
> Key: HBASE-22705
> URL: https://issues.apache.org/jira/browse/HBASE-22705
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.5
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.3.0
>
> Attachments: HBASE-22705.branch-2.patch
>
>
> IllegalArgumentException occured during scan operation,
> {noformat}
> 2019-07-08 01:46:57,764 | ERROR |
> RpcServer.FifoWFPBQ.default.handler=129,queue=9,port=21302 | Unexpected
> throwable object |
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2502)
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.ComparableTimSort.mergeHi(ComparableTimSort.java:866)
> at java.util.ComparableTimSort.mergeAt(ComparableTimSort.java:483)
> at
> java.util.ComparableTimSort.mergeForceCollapse(ComparableTimSort.java:422)
> at java.util.ComparableTimSort.sort(ComparableTimSort.java:222)
> at java.util.Arrays.sort(Arrays.java:1312)
> at java.util.Arrays.sort(Arrays.java:1506)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:141)
> at org.apache.hadoop.hbase.mob.MobFileCache.evict(MobFileCache.java:144)
> at
> org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:214)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:397)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:358)
> at
> org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:74)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:150)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-22705) IllegalArgumentException exception occured during MobFileCache eviction
[
https://issues.apache.org/jira/browse/HBASE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16913004#comment-16913004
]
Pankaj Kumar commented on HBASE-22705:
--
I thought of lockless solution but in a heavy read scenario that wont work, So
added a global lock to update the access count and sort the cached file.
> IllegalArgumentException exception occured during MobFileCache eviction
> ---
>
> Key: HBASE-22705
> URL: https://issues.apache.org/jira/browse/HBASE-22705
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.5
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.3.0
>
> Attachments: HBASE-22705.branch-2.patch
>
>
> IllegalArgumentException occured during scan operation,
> {noformat}
> 2019-07-08 01:46:57,764 | ERROR |
> RpcServer.FifoWFPBQ.default.handler=129,queue=9,port=21302 | Unexpected
> throwable object |
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2502)
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.ComparableTimSort.mergeHi(ComparableTimSort.java:866)
> at java.util.ComparableTimSort.mergeAt(ComparableTimSort.java:483)
> at
> java.util.ComparableTimSort.mergeForceCollapse(ComparableTimSort.java:422)
> at java.util.ComparableTimSort.sort(ComparableTimSort.java:222)
> at java.util.Arrays.sort(Arrays.java:1312)
> at java.util.Arrays.sort(Arrays.java:1506)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:141)
> at org.apache.hadoop.hbase.mob.MobFileCache.evict(MobFileCache.java:144)
> at
> org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:214)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:397)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:358)
> at
> org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:74)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:150)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22705) IllegalArgumentException exception occured during MobFileCache eviction
[
https://issues.apache.org/jira/browse/HBASE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16912881#comment-16912881
]
Vladimir Rodionov commented on HBASE-22705:
---
Apologize for delay, [~pankaj2461]. Additional global lock on MOB file cache is
the last resort approach - do not use it until you explore other (lockless)
options.
> IllegalArgumentException exception occured during MobFileCache eviction
> ---
>
> Key: HBASE-22705
> URL: https://issues.apache.org/jira/browse/HBASE-22705
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.5
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.3.0
>
> Attachments: HBASE-22705.branch-2.patch
>
>
> IllegalArgumentException occured during scan operation,
> {noformat}
> 2019-07-08 01:46:57,764 | ERROR |
> RpcServer.FifoWFPBQ.default.handler=129,queue=9,port=21302 | Unexpected
> throwable object |
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2502)
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.ComparableTimSort.mergeHi(ComparableTimSort.java:866)
> at java.util.ComparableTimSort.mergeAt(ComparableTimSort.java:483)
> at
> java.util.ComparableTimSort.mergeForceCollapse(ComparableTimSort.java:422)
> at java.util.ComparableTimSort.sort(ComparableTimSort.java:222)
> at java.util.Arrays.sort(Arrays.java:1312)
> at java.util.Arrays.sort(Arrays.java:1506)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:141)
> at org.apache.hadoop.hbase.mob.MobFileCache.evict(MobFileCache.java:144)
> at
> org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:214)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:397)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:358)
> at
> org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:74)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:150)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
[jira] [Commented] (HBASE-22705) IllegalArgumentException exception occured during MobFileCache eviction
[
https://issues.apache.org/jira/browse/HBASE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892422#comment-16892422
]
Pankaj Kumar commented on HBASE-22705:
--
Thanks [~vrodionov]...
> IllegalArgumentException exception occured during MobFileCache eviction
> ---
>
> Key: HBASE-22705
> URL: https://issues.apache.org/jira/browse/HBASE-22705
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.5
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.3.0
>
> Attachments: HBASE-22705.branch-2.patch
>
>
> IllegalArgumentException occured during scan operation,
> {noformat}
> 2019-07-08 01:46:57,764 | ERROR |
> RpcServer.FifoWFPBQ.default.handler=129,queue=9,port=21302 | Unexpected
> throwable object |
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2502)
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.ComparableTimSort.mergeHi(ComparableTimSort.java:866)
> at java.util.ComparableTimSort.mergeAt(ComparableTimSort.java:483)
> at
> java.util.ComparableTimSort.mergeForceCollapse(ComparableTimSort.java:422)
> at java.util.ComparableTimSort.sort(ComparableTimSort.java:222)
> at java.util.Arrays.sort(Arrays.java:1312)
> at java.util.Arrays.sort(Arrays.java:1506)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:141)
> at org.apache.hadoop.hbase.mob.MobFileCache.evict(MobFileCache.java:144)
> at
> org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:214)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:397)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:358)
> at
> org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:74)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:150)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22705) IllegalArgumentException exception occured during MobFileCache eviction
[
https://issues.apache.org/jira/browse/HBASE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16892018#comment-16892018
]
Vladimir Rodionov commented on HBASE-22705:
---
On my list today.
> IllegalArgumentException exception occured during MobFileCache eviction
> ---
>
> Key: HBASE-22705
> URL: https://issues.apache.org/jira/browse/HBASE-22705
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.5
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.3.0
>
> Attachments: HBASE-22705.branch-2.patch
>
>
> IllegalArgumentException occured during scan operation,
> {noformat}
> 2019-07-08 01:46:57,764 | ERROR |
> RpcServer.FifoWFPBQ.default.handler=129,queue=9,port=21302 | Unexpected
> throwable object |
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2502)
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.ComparableTimSort.mergeHi(ComparableTimSort.java:866)
> at java.util.ComparableTimSort.mergeAt(ComparableTimSort.java:483)
> at
> java.util.ComparableTimSort.mergeForceCollapse(ComparableTimSort.java:422)
> at java.util.ComparableTimSort.sort(ComparableTimSort.java:222)
> at java.util.Arrays.sort(Arrays.java:1312)
> at java.util.Arrays.sort(Arrays.java:1506)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:141)
> at org.apache.hadoop.hbase.mob.MobFileCache.evict(MobFileCache.java:144)
> at
> org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:214)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:397)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:358)
> at
> org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:74)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:150)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22705) IllegalArgumentException exception occured during MobFileCache eviction
[
https://issues.apache.org/jira/browse/HBASE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16890303#comment-16890303
]
Vladimir Rodionov commented on HBASE-22705:
---
OK, let look today [~pankaj2461]
> IllegalArgumentException exception occured during MobFileCache eviction
> ---
>
> Key: HBASE-22705
> URL: https://issues.apache.org/jira/browse/HBASE-22705
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.5
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.3.0
>
> Attachments: HBASE-22705.branch-2.patch
>
>
> IllegalArgumentException occured during scan operation,
> {noformat}
> 2019-07-08 01:46:57,764 | ERROR |
> RpcServer.FifoWFPBQ.default.handler=129,queue=9,port=21302 | Unexpected
> throwable object |
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2502)
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.ComparableTimSort.mergeHi(ComparableTimSort.java:866)
> at java.util.ComparableTimSort.mergeAt(ComparableTimSort.java:483)
> at
> java.util.ComparableTimSort.mergeForceCollapse(ComparableTimSort.java:422)
> at java.util.ComparableTimSort.sort(ComparableTimSort.java:222)
> at java.util.Arrays.sort(Arrays.java:1312)
> at java.util.Arrays.sort(Arrays.java:1506)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:141)
> at org.apache.hadoop.hbase.mob.MobFileCache.evict(MobFileCache.java:144)
> at
> org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:214)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:397)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:358)
> at
> org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:74)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:150)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22705) IllegalArgumentException exception occured during MobFileCache eviction
[
https://issues.apache.org/jira/browse/HBASE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16889960#comment-16889960
]
Pankaj Kumar commented on HBASE-22705:
--
Ping [~vrodionov] [~anoop.hbase]
> IllegalArgumentException exception occured during MobFileCache eviction
> ---
>
> Key: HBASE-22705
> URL: https://issues.apache.org/jira/browse/HBASE-22705
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.5
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
> Fix For: 2.3.0
>
> Attachments: HBASE-22705.branch-2.patch
>
>
> IllegalArgumentException occured during scan operation,
> {noformat}
> 2019-07-08 01:46:57,764 | ERROR |
> RpcServer.FifoWFPBQ.default.handler=129,queue=9,port=21302 | Unexpected
> throwable object |
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2502)
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.ComparableTimSort.mergeHi(ComparableTimSort.java:866)
> at java.util.ComparableTimSort.mergeAt(ComparableTimSort.java:483)
> at
> java.util.ComparableTimSort.mergeForceCollapse(ComparableTimSort.java:422)
> at java.util.ComparableTimSort.sort(ComparableTimSort.java:222)
> at java.util.Arrays.sort(Arrays.java:1312)
> at java.util.Arrays.sort(Arrays.java:1506)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:141)
> at org.apache.hadoop.hbase.mob.MobFileCache.evict(MobFileCache.java:144)
> at
> org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:214)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:397)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:358)
> at
> org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:74)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:150)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (HBASE-22705) IllegalArgumentException exception occured during MobFileCache eviction
[
https://issues.apache.org/jira/browse/HBASE-22705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16887147#comment-16887147
]
Pankaj Kumar commented on HBASE-22705:
--
Problem is, we sort the MOB files based on the accessCount in CachedMobFile
during MobFileCache eviction.
{code}
@Override
public int compareTo(CachedMobFile that) {
if (this.accessCount == that.accessCount) return 0;
return this.accessCount < that.accessCount ? 1 : -1;
}
{code}
So in a concurrent scenario if accessCount is modified then this probelm will
happen as per the "Comparison method general contract". We should update the
accessCount in sync way.
> IllegalArgumentException exception occured during MobFileCache eviction
> ---
>
> Key: HBASE-22705
> URL: https://issues.apache.org/jira/browse/HBASE-22705
> Project: HBase
> Issue Type: Bug
> Components: mob
>Affects Versions: 2.0.5
>Reporter: Pankaj Kumar
>Assignee: Pankaj Kumar
>Priority: Critical
>
> IllegalArgumentException occured during scan operation,
> {noformat}
> 2019-07-08 01:46:57,764 | ERROR |
> RpcServer.FifoWFPBQ.default.handler=129,queue=9,port=21302 | Unexpected
> throwable object |
> org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2502)
> java.lang.IllegalArgumentException: Comparison method violates its general
> contract!
> at java.util.ComparableTimSort.mergeHi(ComparableTimSort.java:866)
> at java.util.ComparableTimSort.mergeAt(ComparableTimSort.java:483)
> at
> java.util.ComparableTimSort.mergeForceCollapse(ComparableTimSort.java:422)
> at java.util.ComparableTimSort.sort(ComparableTimSort.java:222)
> at java.util.Arrays.sort(Arrays.java:1312)
> at java.util.Arrays.sort(Arrays.java:1506)
> at java.util.ArrayList.sort(ArrayList.java:1462)
> at java.util.Collections.sort(Collections.java:141)
> at org.apache.hadoop.hbase.mob.MobFileCache.evict(MobFileCache.java:144)
> at
> org.apache.hadoop.hbase.mob.MobFileCache.openFile(MobFileCache.java:214)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.readCell(HMobStore.java:397)
> at
> org.apache.hadoop.hbase.regionserver.HMobStore.resolve(HMobStore.java:358)
> at
> org.apache.hadoop.hbase.regionserver.MobStoreScanner.next(MobStoreScanner.java:74)
> at
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:150)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
