[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16039721#comment-16039721
]
Hudson commented on PHOENIX-3797:
-
FAILURE: Integrated in Jenkins build Phoenix-master #1644 (See
[https://builds.apache.org/job/Phoenix-master/1644/])
PHOENIX-3797 Local Index - Compaction fails on table with local index
(ankitsinghal59: rev fbe6ee22d8a6bafa325a56ee7282fa77253b7e07)
* (edit)
phoenix-core/src/main/java/org/apache/phoenix/coprocessor/UngroupedAggregateRegionObserver.java
* (edit)
phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/IndexHalfStoreFileReaderGenerator.java
* (edit)
phoenix-core/src/it/java/org/apache/phoenix/end2end/index/LocalIndexIT.java
* (edit)
phoenix-core/src/main/java/org/apache/hadoop/hbase/regionserver/DataTableLocalIndexRegionScanner.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/util/RepairUtil.java
* (edit) phoenix-core/src/main/java/org/apache/phoenix/util/ServerUtil.java
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch, PHOENIX-3797_v2.patch,
> PHOENIX-3797_v3.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16039580#comment-16039580
]
Ankit Singhal commented on PHOENIX-3797:
Thanks [~lhofhansl] and [~jamestaylor] for the review and ideas. created
PHOENIX-3916 for the follow-on work.
Committed to master and 4.x branches.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch, PHOENIX-3797_v2.patch,
> PHOENIX-3797_v3.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16039051#comment-16039051
]
James Taylor commented on PHOENIX-3797:
---
+1. Please commit this, [~an...@apache.org].
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch, PHOENIX-3797_v2.patch,
> PHOENIX-3797_v3.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16037418#comment-16037418
]
Lars Hofhansl commented on PHOENIX-3797:
+1 on commit to 4.11 and then file a follow-on for 4.12.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch, PHOENIX-3797_v2.patch,
> PHOENIX-3797_v3.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16037323#comment-16037323
]
Ankit Singhal commented on PHOENIX-3797:
bq. Can we optionally disable the index? I'd be somewhat concerned about
performance during compactions.
We can but if the local index spans in multiple column family/stores then I'll
not be sure when I can enable the indexes as compaction runs independently for
each store.
bq. You'd also want to truncate all the column families for the local index.
Column families will be truncated(with an empty file) and re-written(from
region path) as a part of compaction itself. so, we may not need to do it
explicitly.
bq. Please let us know, Ankit Singhal, if you think the above is feasible for
4.11.0. If not, please commit your v3 patch.
As said above, without state management or client executing the repair
operation, it will be difficult to change the state of index accurately. Let me
commit this for 4.11.0 and we can work on adding "CHECK/REPAIR TABLE"
functionality in a repair tool(ping [~sergey.soldatov]) which can be run after
hbck to fix this.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch, PHOENIX-3797_v2.patch,
> PHOENIX-3797_v3.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035680#comment-16035680
]
James Taylor commented on PHOENIX-3797:
---
The disable index could be plugged in as the else block here:
{code}
+if
(c.getEnvironment().getConfiguration().getBoolean(LOCAL_INDEX_AUTOMATIC_REPAIR,
true)) {
+LOG.info("Starting automatic repair of local Index for
region:"
++ c.getEnvironment().getRegion().getRegionInfo());
+repairScanner = getRepairScanner(c.getEnvironment(),
store);
+}
{code}
You'd need to run a scan against SYSTEM.CATALOG and find all tables with a link
to the local index physical table and invoke the endpoint coprocessor API to
disable the index. You'd also want to truncate all the column families for the
local index. Not sure if there's any trickiness in the disable to prevent the
partial index rebuild to kick in (as in this case you'd want to ensure that the
index is rebuilt from scratch). See PHOENIX-3909 for some potential future work.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch, PHOENIX-3797_v2.patch,
> PHOENIX-3797_v3.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035148#comment-16035148
]
Lars Hofhansl commented on PHOENIX-3797:
I think -v3 is fine.
Can we optionally disable the index? I'd be somewhat concerned about
performance during compactions.
If that's too much work, let's commit this.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch, PHOENIX-3797_v2.patch,
> PHOENIX-3797_v3.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034641#comment-16034641
]
Ankit Singhal commented on PHOENIX-3797:
[~jamestaylor], I'm not sure if we agreed on the approach yet so uploading
revised patch(v3) to fix local index during compaction by writing it through
the region. (as we can't attach LocalIndexStoreFileScanner to files as there is
no way to identify the start key of the second region merged during hbck and
this will even become more complicated if multiple regions get merged).
bq.Would also be nice if custom logic could be hooked into HBCK to address
issues like this right there.
Yes [~lhofhansl], that would be great if it is possible([~devaraj] was also
suggesting the same). Let me know if we can add a support in hbck to support
custom hooks, I'll be happy update the patch.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch, PHOENIX-3797_v2.patch,
> PHOENIX-3797_v3.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034116#comment-16034116
]
Lars Hofhansl commented on PHOENIX-3797:
[~jamestaylor] that's certainly a valid option. We'd have to reliable detect
that.
Would also be nice if custom logic could be hooked into HBCK to address issues
like this right there.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch, PHOENIX-3797_v2.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033689#comment-16033689
]
James Taylor commented on PHOENIX-3797:
---
[~an...@apache.org] - appreciate your efforts here. What's your ETA for a
revised patch? Another alternative would be to disable the local index if this
situation is detected and require a manual rebuild after hbck is run.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch, PHOENIX-3797_v2.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16032685#comment-16032685
]
Ankit Singhal commented on PHOENIX-3797:
Another problem of doing repair during compaction only, is that the data will
be inconsistent for the queries until we find and fix them during compaction.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch, PHOENIX-3797_v2.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16032682#comment-16032682
]
Ankit Singhal commented on PHOENIX-3797:
bq. Actually... Can the same thing happen with just a regular merge operation
from HBaseAdmin?
No, this will not happen with the regular merge operation from HBaseAdmin as
there will be reference files created with split row as start key of 2nd
region, so it is easy to detect start key of the store file which can be used
to parse the rows of daughter region during the scan and re-write the complete
data with new start key during compaction using LocalIndexStoreFileScanner
(IndexHalfStoreFileReader)
bq. Here's yet another idea: Can we hook a scanner right above the HFiles? That
scanner would rewrite the keys based on the new region startkey. So now the
store scanner for the index would do the right thing (merge sort between the
values from the HFile scanners).
yes, If we can identify the start key of the 2nd region, we can make a use of
LocalIndexStoreFileScanner. But with new region start key , we can't parse the
local index data from store files of second region.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch, PHOENIX-3797_v2.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16032445#comment-16032445
]
Lars Hofhansl commented on PHOENIX-3797:
Sorry for the many comments. It turns out the repair scanner is trying to
rebuild the local index during the compaction while scanning along the main
column families, and that's why the index writes can be out of order.
The idea is pretty clever! [~an...@apache.org]
And hence your new idea of writing the index update through the region, so that
they will get sorted.
So that can work. For large regions that would lead to a lot of HFiles, though
(for a 10g region with 256mb flush size it would lead to 40 files after the
major compaction).
I think my idea of rewriting the index data from the merged index data itself
is pretty tricky, too.
Here's yet another idea: Can we hook a scanner right above the HFiles? That
scanner would rewrite the keys based on the new region startkey. So now the
store scanner for the index would do the right thing (merge sort between the
values from the HFile scanners).
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch, PHOENIX-3797_v2.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16032338#comment-16032338
]
Lars Hofhansl commented on PHOENIX-3797:
Actually... Can the same thing happen with just a regular merge operation from
HBaseAdmin?
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch, PHOENIX-3797_v2.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16031966#comment-16031966
]
Lars Hofhansl commented on PHOENIX-3797:
Some more ideas that [~jamestaylor] and I were brainstorming. Essentially we
need to find the two region start keys for the local index rows that got merged
together into one.
One solution is to find the first key, extract its region start key, then
create a second scanner that is guaranteed to start after all keys that start
with that region key (so that scanner will find the rows from the 2nd former
region that got merged).
Or we can use a reverse scanner to find the last key, and then extract the 2nd
region start key from there.
Perhaps there are easier ways to find the start keys of the previous regions
that got merged?
In either cases we'd have a scanner that wraps these two scanners and does a
mergesort between them.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch, PHOENIX-3797_v2.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16030388#comment-16030388
]
Lars Hofhansl commented on PHOENIX-3797:
Thanks [~an...@apache.org].
So if understand this correctly the issue happens when HBCK merges two regions.
Then we'll two sets of local index rows in the L#0 column family right, so as
we read and rewrite we might find a previous key following the later start key
of the 2nd region and hence get the out of order problem. Your solution is to
not have the compaction write write the local index stuff, but instead write it
in smaller batches that themselves will not overlap, directly through the
region.
Is that about right?
If would certainly be nice if we could somehow leave the compaction writer do
the work, but I do not see how we can do the scanning then. Perhaps one
solution is to wrap two scanner inside a scanner wrapper and scan from both
region start keys. In that way we could get the re-written keys in the correct
order and leave the writing to the compation writer - just thinking aloud here.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch, PHOENIX-3797_v2.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16030139#comment-16030139
]
Ankit Singhal commented on PHOENIX-3797:
bq. It also seems fishy to apply the mutations in a method that is called
get..., the passed localIndexResults is not used
Yes [~lhofhansl] , v2 patch was created at the last minute by just updating the
required pieces to give an idea that there is an alternate approach to fix the
index writes in repair scanner.
Refactoring and comments will follow in subsequent patch.
bq.Lastly it seems now that each time next is called we execute the batchmutate?
yes, I was planning to re-use the same batch size configuration.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch, PHOENIX-3797_v2.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16029880#comment-16029880
]
Lars Hofhansl commented on PHOENIX-3797:
Thanks [~an...@apache.org], I'm having a hard time understanding understanding
the change. Can you put some comments in the code? (And also explain here what
this is doing)
It also seems fishy to apply the mutations in a method that is called
{{get...}}, the passed {{localIndexResults}} is not used. Lastly it seems now
that each time next is called we execute the batchmutate?
So that does not seem quite right.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch, PHOENIX-3797_v2.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16029628#comment-16029628
]
James Taylor commented on PHOENIX-3797:
---
Thanks, [~an...@apache.org]. Sounds like your v2 patch is the way to go so that
the RepairScanner works as expected. WDYT, [~rajeshbabu]?
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch, PHOENIX-3797_v2.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16029614#comment-16029614
]
Ankit Singhal commented on PHOENIX-3797:
[~jamestaylor], Yes Local Index data is in right sorted order.
Compaction was failing because :-
We have a RepairScanner to handle the cases when two regions are merged during
the hbck run(to repair overlaps or something) ,as these merges will be fine for
data regions but can corrupt the data for local index(as we use start key of
the region as suffix for local index to maintain the data within the region)
Due to the bug(basically the typo), the local index files are indentified
always inconsistent with respect to region boundaries resulting in this repair
scanner to run everytime and which is failing with above exception.(because we
are creating index mutation from data store and writing directly to local index
hfiles).
Attaching the fix for the same too, in case we want to pursue with this repair
as well.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16029551#comment-16029551
]
James Taylor commented on PHOENIX-3797:
---
Is that just masking the issue, [~an...@apache.org]? We require the local index
rows to be sorted in the store or we'll get incorrect query results.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
> Attachments: PHOENIX-3797.patch
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16029290#comment-16029290
]
Ankit Singhal commented on PHOENIX-3797:
Let's disable the repair scanner for now as it may take a time to identify why
index mutations from the data store are not built correctly during repair.
Attaching the patch for the same.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Ankit Singhal
>Priority: Blocker
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16029268#comment-16029268
]
Rajeshbabu Chintaguntla commented on PHOENIX-3797:
--
[~ankit.singhal] This is coming with the RepairTool added as part of
PHOENIX-3609. Can you please check it.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Rajeshbabu Chintaguntla
>Priority: Blocker
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16026994#comment-16026994
]
Lars Hofhansl commented on PHOENIX-3797:
[~mujtabachohan] do you only see this when at least one split happens?
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Rajeshbabu Chintaguntla
>Priority: Blocker
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16026992#comment-16026992
]
Lars Hofhansl commented on PHOENIX-3797:
A flush would have indicated the same exception. So this is likely following a
split.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Rajeshbabu Chintaguntla
>Priority: Blocker
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16026981#comment-16026981
]
Lars Hofhansl commented on PHOENIX-3797:
The exception is only a symptom of what happens earlier, I think.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Rajeshbabu Chintaguntla
>Priority: Blocker
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025078#comment-16025078
]
Mujtaba Chohan commented on PHOENIX-3797:
-
[~lhofhansl] This is on HBase 1.3.1 running in local mode.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Rajeshbabu Chintaguntla
>Priority: Blocker
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024957#comment-16024957
]
Lars Hofhansl commented on PHOENIX-3797:
[~mujtabachohan], you have assertions enabled? (i.e. this is a unit test or
something?)
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Rajeshbabu Chintaguntla
>Priority: Blocker
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024819#comment-16024819
]
James Taylor commented on PHOENIX-3797:
---
[~rajeshbabu] - do you have everything you need for this one? I think we're
good for an RC once this is fixed. Any ETA?
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Rajeshbabu Chintaguntla
>Priority: Blocker
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16023355#comment-16023355
]
Mujtaba Chohan commented on PHOENIX-3797:
-
[~rajeshbabu]
{noformat}
ERROR [RS:0;localhost:44384-longCompactions-1495575134596]
regionserver.CompactSplitThread: Compaction failed Request =
regionName=T,00Dxx001gES005001xx03DGPc\x7F\xFF\xFE\xB6\xE8Fi\xA701736008Zudx
,1495578683299.bce91e7e78ef24d1f475f49faa48cfce., storeName=L#0,
fileCount=2, fileSize=29.8 M (26.6 M, 3.2 M), priority=1, time=946101537224889
java.io.IOException: Added a key not lexically larger than previous. Current
cell = 00Dxx001gES005001xx03DGPc\x7F\xFF\xFE\xB6\xE8Fi\xA701736008Zudx
\x00\x00\xDxx001gES005001xx03DGPc\x7F\xFF\xFE\xB6\xEF\x80\xF0\x070171153586Z9
/L#0:_0/1495578782112/Put/vlen=2/seqid=0, lastCell =
00Dxx001gES005001xx03DGPc\x7F\xFF\xFE\xB6\xE8Fi\xA701736008Zudx
\x00\x00\xC0//ZY\x1ER\x12ZG\xDxx001gES005001xx03DGPc\x7F\xFF\xFE\xB6\xEFU\xBDo017894704R85
/L#0:_0/1495578741485/Put/vlen=2/seqid=0
at
org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:204)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:267)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterV3.append(HFileWriterV3.java:87)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:1047)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:465)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:320)
at
org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:66)
at
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:115)
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1286)
at
org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1938)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:525)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:564)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{noformat}
This is the log from master/RS running as a single process.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Assignee: Rajeshbabu Chintaguntla
>Priority: Blocker
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
>
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022334#comment-16022334
]
Rajeshbabu Chintaguntla commented on PHOENIX-3797:
--
[~mujtabachohan] can you please share the logs of both master and RS reproduced
this issue.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Priority: Blocker
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022331#comment-16022331
]
Rajeshbabu Chintaguntla commented on PHOENIX-3797:
--
Will take a look into it [~giacomotaylor].
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Priority: Blocker
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16022037#comment-16022037
]
Mujtaba Chohan commented on PHOENIX-3797:
-
[~jamestaylor], [~rajeshbabu] Still saw compaction failing due to
non-increasing keys with the default bloom filters turned off:
{noformat}
ERROR [RS:0;RS:44384-longCompactions-1495575134596]
regionserver.CompactSplitThread: Compaction failed Request =
regionName=T,,1495578683299.14310c7fd34e45336d884244231368a9., storeName=L#0,
fileCount=1, fileSize=29.2 M (29.2 M), priority=1, time=946101536307883
java.io.IOException: Added a key not lexically larger than previous. Current
cell =
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xDxx001gER001001xx03DGPa\x7F\xFF\xFE\xB6\xD3\xD0`'01730689402d
/L#0:_0/1495579097807/Put/vlen=2/seqid=0, lastCell =
\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\xC0\x15D\x08\x19d\x08^7\x1F\xDxx001gER001001xx03DGPa\x7F\xFF\xFE\xB6\xD3\xC3\xF4w017899970MFJ
/L#0:_0/1495578741485/Put/vlen=2/seqid=0
at
org.apache.hadoop.hbase.io.hfile.AbstractHFileWriter.checkKey(AbstractHFileWriter.java:204)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterV2.append(HFileWriterV2.java:267)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterV3.append(HFileWriterV3.java:87)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:1047)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:465)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:320)
at
org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:66)
at
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:115)
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1286)
at
org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1938)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:525)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:564)
{noformat}
FYI manually compacted table and build is with
[PHOENIX-3827_v5.patch|https://issues.apache.org/jira/secure/attachment/12869519/PHOENIX-3827_v5.patch]
running on HBase 1.3.1.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Priority: Blocker
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserv
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16021903#comment-16021903
]
James Taylor commented on PHOENIX-3797:
---
Had an offline discussion with [~lhofhansl] who enlightened me on this issue.
The non-increasing bloom key exception is happening to catch that rows are
being written in the wrong order during compaction. Understandably, HBase can't
detect this without a severe performance hit. Though we're not creating bloom
filters any longer, this issue would still occur. We'd only detect it when a
query serviced by the index happens to return incorrect results.
[~rajeshbabu] - I think we need to get to the bottom of this one. WDYT?
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16007057#comment-16007057
]
Mujtaba Chohan commented on PHOENIX-3797:
-
[~jamestaylor], [~rajeshbabu] Repros consistently with Bloom filter turned back
on. Tested with HBase 0.98.23 and head of Phoenix 4.x.
{noformat}
ERROR [RS:0;RS-smallCompactions-1494529773221] regionserver.CompactSplitThread:
Compaction failed Request =
regionName=T,00Dxx001gES005001xx03DGQW\x80\x00\x01H\xF7\xAA,1494529828163.b505fb3b8b97119ff0a0da18b6e11171.,
storeName=L#0, fileCount=9, fileSize=126.4 M (15.7 M, 23.5 M, 23.5 M, 23.5 M,
8.0 M, 8.1 M, 8.1 M, 8.0 M, 8.1 M), priority=2, time=9328384816705648
java.io.IOException: Non-increasing Bloom keys:
00Dxx001gES005001xx03DGQW\x80\x00\x01H\xF7\xAA\x00\xDxx001gER\x80\x00\x01H*\xEA\x86p001001xx03DGPb0171499340na
after
00Dxx001gES005001xx03DGQW\x80\x00\x01H\xF7\xAA\x00\xDxx001gER\x80\x00\x01I,T\xFE@001001xx03DGPa01776146431b
at
org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
at
org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
at
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
{noformat}
Schema:
{noformat}
CREATE TABLE IF NOT EXISTS T (
OD CHAR(15) NOT NULL,
PX CHAR(3) NOT NULL,
PD CHAR(15) NOT NULL,
CRD DATE NOT NULL,
ED CHAR(15) NOT NULL,
FHD CHAR(15),
CONSTRAINT PK PRIMARY KEY
(
OD,
PX,
PD,
CRD,
ED
)
) MULTI_TENANT=true,IMMUTABLE_ROWS=true,VERSIONS=1
CREATE LOCAL INDEX IF NOT EXISTS T_IDX ON FHA (CRD)
{noformat}
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16006827#comment-16006827
]
Andrew Purtell commented on PHOENIX-3797:
-
bq. (might be an HBase issue).
I'd be inclined to think something else about the local index implementation is
off.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16006825#comment-16006825
]
Andrew Purtell commented on PHOENIX-3797:
-
bq. Filed PHOENIX-3842 and posted a patch there.
I committed PHOENIX-3842 so the underlying issue should manifest in a more
direct way
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16006822#comment-16006822
]
James Taylor commented on PHOENIX-3797:
---
Since bloom filters will be off now, we'll skirt this issue, hence lower the
priority. I agree, though, it's still an issue (might be an HBase issue). Can
you repro this consistently, [~mujtabachohan]? You'll likely need to turn bloom
filters back on to repro it.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003881#comment-16003881
]
Lars Hofhansl commented on PHOENIX-3797:
Filed PHOENIX-3842 and posted a patch there.
*This* is still a problem, though. Shouldn't get keys in the wrong order.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Priority: Blocker
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003838#comment-16003838
]
Lars Hofhansl commented on PHOENIX-3797:
[~jamestaylor], [~apurtell], looks like ROW BFs are turned on by default!
Phoenix doesn't ever do GETs, so that is completely wasted. (See
HColumnDescription.DEFAULT_BLOOMFILTER)
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Priority: Blocker
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003832#comment-16003832
]
Lars Hofhansl commented on PHOENIX-3797:
So it seem the write really gets the keys in the wrong order.
{{00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf}}
is strictly <
{{00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM}}
lexicographically
So then it is probably the scanner that reads the current files/memstore that
gets keys in the wrong order. Or perhaps an earlier flush got the order wrong?
- but then I'd expect the BF there would have failed. Somewhat scary.
And, lastly, why is it doing a BF in the first place?!
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Priority: Blocker
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16003695#comment-16003695
]
Lars Hofhansl commented on PHOENIX-3797:
Agree on the blocker prio.
The region split under load seems unrelated.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Priority: Blocker
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16001695#comment-16001695
]
Geoffrey Jacoby commented on PHOENIX-3797:
--
I created PHOENIX-3838 for the region split bug investigation to keep it
separate from the compaction issue, and assigned it to myself.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Priority: Blocker
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995867#comment-15995867
]
Andrew Purtell commented on PHOENIX-3797:
-
This seems like maybe more than one related issue, but in any case a blocker
priority problem given basic issues with the functionality. To get attention to
them, hopefully. Feel to change the priority back if you disagree.
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>Priority: Blocker
> Fix For: 4.11.0
>
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15995774#comment-15995774
]
Mujtaba Chohan commented on PHOENIX-3797:
-
[~rajeshbabu] The above was with HBase 0.98 and I'll try to get a clean repro.
Meanwhile with HBase 1.3.1 if I try to split a table while data load is in
progress, the table state remains in SPLITTING_NEW and index writer blocked.
Table splits fine if there is no active writes happening to the table when
split is requested.
{noformat}
Thread 163 (RpcServer.FifoWFPBQ.priority.handler=19,queue=1,port=48109):
State: WAITING
Blocked count: 100
Waited count: 463
Waiting on com.google.common.util.concurrent.AbstractFuture$Sync@16703eda
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:275)
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:111)
org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submit(BaseTaskRunner.java:66)
org.apache.phoenix.hbase.index.parallel.BaseTaskRunner.submitUninterruptible(BaseTaskRunner.java:99)
org.apache.phoenix.hbase.index.write.ParallelWriterIndexCommitter.write(ParallelWriterIndexCommitter.java:197)
org.apache.phoenix.hbase.index.write.IndexWriter.write(IndexWriter.java:185)
org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:146)
org.apache.phoenix.hbase.index.write.IndexWriter.writeAndKillYourselfOnFailure(IndexWriter.java:135)
org.apache.phoenix.hbase.index.Indexer.doPostWithExceptions(Indexer.java:474)
org.apache.phoenix.hbase.index.Indexer.doPost(Indexer.java:407)
org.apache.phoenix.hbase.index.Indexer.postPut(Indexer.java:375)
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$32.call(RegionCoprocessorHost.java:956)
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost$RegionOperation.call(RegionCoprocessorHost.java:1673)
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1749)
org.apache.hadoop.hbase.regionserver.RegionCoprocessorHost.execOperation(RegionCoprocessorHost.java:1705)
{noformat}
Following schema was used with batch size of 1000 inserting data in background:
{noformat}
CREATE TABLE IF NOT EXISTS T (PKA CHAR(15) NOT NULL, PKF CHAR(3) NOT NULL,
PKP CHAR(15) NOT NULL, CRD DATE NOT NULL, EHI CHAR(15) NOT NULL, FID CHAR(15),
CREATED_BY_ID VARCHAR,
FH VARCHAR, DT VARCHAR, OS VARCHAR, NS VARCHAR, OFN VARCHAR CONSTRAINT PK
PRIMARY KEY ( PKA, PKF, PKP, CRD DESC, EHI ))
VERSIONS=1,MULTI_TENANT=true,IMMUTABLE_ROWS=true;
CREATE LOCAL INDEX IF NOT EXISTS TIDX ON T (PKF, CRD, PKP, EHI)
INCLUDE (FID, CREATED_BY_ID, FH, DT, OS, NS, OFN);
{noformat}
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionse
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994596#comment-15994596
]
Rajeshbabu Chintaguntla commented on PHOENIX-3797:
--
[~lhofhansl] Not sure about this. Any chance to get the steps to reproduce this
so that I can check why it's coming? is it easy to reproduce?
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (PHOENIX-3797) Local Index - Compaction fails on table with local index due to non-increasing bloom keys
[
https://issues.apache.org/jira/browse/PHOENIX-3797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15994297#comment-15994297
]
Lars Hofhansl commented on PHOENIX-3797:
Whoa. [~rajeshbabu], any hints?
> Local Index - Compaction fails on table with local index due to
> non-increasing bloom keys
> -
>
> Key: PHOENIX-3797
> URL: https://issues.apache.org/jira/browse/PHOENIX-3797
> Project: Phoenix
> Issue Type: Bug
> Environment: Head of 4.x-HBase-0.98 with PHOENIX-3796 patch applied.
> HBase 0.98.23-hadoop2
>Reporter: Mujtaba Chohan
>
> Compaction fails on table with local index.
> {noformat}
> 2017-04-19 16:37:56,521 ERROR
> [RS:0;host:59455-smallCompactions-1492644947594]
> regionserver.CompactSplitThread: Compaction failed Request =
> regionName=FHA,00Dxx001gES005001xx03DGPd,1492644985470.92ec6436984981cdc8ef02388005a957.,
> storeName=L#0, fileCount=3, fileSize=44.4 M (23.0 M, 10.7 M, 10.8 M),
> priority=7, time=7442973347247614
> java.io.IOException: Non-increasing Bloom keys:
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01H+&\xA1(00Dxx001gER001001xx03DGPb01739544DCtf
> after
> 00Dxx001gES005001xx03DGPd\x00\x00\x80\x00\x01I+\xF4\x9Ax00Dxx001gER001001xx03DGPa017115434KTM
>
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.appendGeneralBloomfilter(StoreFile.java:960)
> at
> org.apache.hadoop.hbase.regionserver.StoreFile$Writer.append(StoreFile.java:996)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:428)
> at
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:276)
> at
> org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:64)
> at
> org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:121)
> at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1154)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:1559)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.doCompaction(CompactSplitThread.java:502)
> at
> org.apache.hadoop.hbase.regionserver.CompactSplitThread$CompactionRunner.run(CompactSplitThread.java:540)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
