Apache-Phoenix | master | HBase 2.5 | Build #580 SUCCESS
master branch HBase 2.5 build #580 status SUCCESS Build #580 https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-mulitbranch/job/master/580/
Apache-Phoenix | master | HBase 2.4 | Build #580 FAILURE
master branch HBase 2.4 build #580 status FAILURE Build #580 https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-mulitbranch/job/master/580/
Apache-Phoenix | 5.1 | HBase 2.4 | Build #268 SUCCESS
5.1 branch HBase 2.4 build #268 status SUCCESS Build #268 https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-mulitbranch/job/5.1/268/
Apache-Phoenix | 5.1 | HBase 2.3 | Build #268 FAILURE
5.1 branch HBase 2.3 build #268 status FAILURE Build #268 https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-mulitbranch/job/5.1/268/
Apache-Phoenix | 5.1 | HBase 2.5 | Build #268 FAILURE
5.1 branch HBase 2.5 build #268 status FAILURE Build #268 https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-mulitbranch/job/5.1/268/
Apache-Phoenix | 5.1 | HBase 2.2 | Build #268 FAILURE
5.1 branch HBase 2.2 build #268 status FAILURE Build #268 https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-mulitbranch/job/5.1/268/
Apache-Phoenix | 5.1 | HBase 2.1 | Build #268 FAILURE
5.1 branch HBase 2.1 build #268 status FAILURE Build #268 https://ci-hadoop.apache.org/job/Phoenix/job/Phoenix-mulitbranch/job/5.1/268/
[phoenix] 01/01: Merge branch 'master' into PHOENIX-7001-feature
This is an automated email from the ASF dual-hosted git repository. vjasani pushed a commit to branch PHOENIX-7001-feature in repository https://gitbox.apache.org/repos/asf/phoenix.git commit a0501c4e68532376f30dbd89c9e4b8c929520005 Merge: 581e61365c f5a425e3ef Author: Viraj Jasani AuthorDate: Sat Sep 30 11:58:48 2023 -0700 Merge branch 'master' into PHOENIX-7001-feature .../apache/phoenix/iterate/SnapshotScanner.java| 44 +- 1 file changed, 43 insertions(+), 1 deletion(-)
[phoenix] branch PHOENIX-7001-feature updated (581e61365c -> a0501c4e68)
This is an automated email from the ASF dual-hosted git repository. vjasani pushed a change to branch PHOENIX-7001-feature in repository https://gitbox.apache.org/repos/asf/phoenix.git from 581e61365c PHOENIX-7008 Implementation for CREATE CDC (#1681) add f5a425e3ef PHOENIX-7039 Snapshot scanner should skip replay WAL and update seqid while opening region (#1677) new a0501c4e68 Merge branch 'master' into PHOENIX-7001-feature The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/phoenix/iterate/SnapshotScanner.java| 44 +- 1 file changed, 43 insertions(+), 1 deletion(-)
[phoenix] branch PHOENIX-6978-feature updated (3993b314db -> ab3aae8dc2)
This is an automated email from the ASF dual-hosted git repository. vjasani pushed a change to branch PHOENIX-6978-feature in repository https://gitbox.apache.org/repos/asf/phoenix.git from 3993b314db Merge branch 'master' into PHOENIX-6978-feature add f5a425e3ef PHOENIX-7039 Snapshot scanner should skip replay WAL and update seqid while opening region (#1677) new ab3aae8dc2 Merge branch 'master' into PHOENIX-6978-feature The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/phoenix/iterate/SnapshotScanner.java| 44 +- 1 file changed, 43 insertions(+), 1 deletion(-)
[phoenix] 01/01: Merge branch 'master' into PHOENIX-6883-feature
This is an automated email from the ASF dual-hosted git repository. vjasani pushed a commit to branch PHOENIX-6883-feature in repository https://gitbox.apache.org/repos/asf/phoenix.git commit eaba11623313b091b1a2c9c2fe7f5c1269d25dde Merge: 8746c42353 f5a425e3ef Author: Viraj Jasani AuthorDate: Sat Sep 30 11:57:11 2023 -0700 Merge branch 'master' into PHOENIX-6883-feature .../apache/phoenix/iterate/SnapshotScanner.java| 44 +- 1 file changed, 43 insertions(+), 1 deletion(-)
[phoenix] branch PHOENIX-6883-feature updated (8746c42353 -> eaba116233)
This is an automated email from the ASF dual-hosted git repository. vjasani pushed a change to branch PHOENIX-6883-feature in repository https://gitbox.apache.org/repos/asf/phoenix.git from 8746c42353 Merge branch 'master' into PHOENIX-6883-feature add f5a425e3ef PHOENIX-7039 Snapshot scanner should skip replay WAL and update seqid while opening region (#1677) new eaba116233 Merge branch 'master' into PHOENIX-6883-feature The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/phoenix/iterate/SnapshotScanner.java| 44 +- 1 file changed, 43 insertions(+), 1 deletion(-)
[phoenix] 01/01: Merge branch 'master' into PHOENIX-6978-feature
This is an automated email from the ASF dual-hosted git repository. vjasani pushed a commit to branch PHOENIX-6978-feature in repository https://gitbox.apache.org/repos/asf/phoenix.git commit ab3aae8dc27fd5575daa84f0332f9725713be1b2 Merge: 3993b314db f5a425e3ef Author: Viraj Jasani AuthorDate: Sat Sep 30 11:58:23 2023 -0700 Merge branch 'master' into PHOENIX-6978-feature .../apache/phoenix/iterate/SnapshotScanner.java| 44 +- 1 file changed, 43 insertions(+), 1 deletion(-)
[phoenix] 01/01: Merge branch 'master' into PHOENIX-628-feature
This is an automated email from the ASF dual-hosted git repository. vjasani pushed a commit to branch PHOENIX-628-feature in repository https://gitbox.apache.org/repos/asf/phoenix.git commit 8c9170f73e2103a8e0f1430b6e1be048c41e1d5a Merge: 5bcebebd8d f5a425e3ef Author: Viraj Jasani AuthorDate: Sat Sep 30 11:56:41 2023 -0700 Merge branch 'master' into PHOENIX-628-feature .../apache/phoenix/iterate/SnapshotScanner.java| 44 +- 1 file changed, 43 insertions(+), 1 deletion(-)
[phoenix] branch 5.1 updated: PHOENIX-7039 Snapshot scanner should skip replay WAL and update seqid while opening region (#1677)
This is an automated email from the ASF dual-hosted git repository.
vjasani pushed a commit to branch 5.1
in repository https://gitbox.apache.org/repos/asf/phoenix.git
The following commit(s) were added to refs/heads/5.1 by this push:
new d556b45c32 PHOENIX-7039 Snapshot scanner should skip replay WAL and
update seqid while opening region (#1677)
d556b45c32 is described below
commit d556b45c32fe2d3ed9a0ff99c72345525f38050f
Author: Viraj Jasani
AuthorDate: Sat Sep 30 10:55:05 2023 -0800
PHOENIX-7039 Snapshot scanner should skip replay WAL and update seqid while
opening region (#1677)
---
.../apache/phoenix/iterate/SnapshotScanner.java| 44 +-
1 file changed, 43 insertions(+), 1 deletion(-)
diff --git
a/phoenix-core/src/main/java/org/apache/phoenix/iterate/SnapshotScanner.java
b/phoenix-core/src/main/java/org/apache/phoenix/iterate/SnapshotScanner.java
index f15c5705ab..2cefbece00 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/iterate/SnapshotScanner.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/iterate/SnapshotScanner.java
@@ -37,11 +37,14 @@ import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.client.TableDescriptor;
import org.apache.hadoop.hbase.coprocessor.RegionCoprocessor;
import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheFactory;
import org.apache.hadoop.hbase.metrics.MetricRegistry;
+import org.apache.hadoop.hbase.mob.MobFileCache;
import org.apache.hadoop.hbase.regionserver.HRegion;
import org.apache.hadoop.hbase.regionserver.OnlineRegions;
import org.apache.hadoop.hbase.regionserver.Region;
import org.apache.hadoop.hbase.regionserver.RegionScanner;
+import org.apache.hadoop.hbase.util.CommonFSUtils;
import org.apache.phoenix.coprocessor.BaseScannerRegionObserver;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.TableName;
@@ -79,7 +82,11 @@ public class SnapshotScanner extends AbstractClientScanner {
scan.setIsolationLevel(IsolationLevel.READ_UNCOMMITTED);
values = new ArrayList<>();
-this.region = HRegion.openHRegion(conf, fs, rootDir, hri, htd, null, null,
null);
+
+// TODO : Use hbase provided snapshot scanner (make it IA.LimitedPrivate)
+// region init should follow the same pattern as hbase
ClientSideRegionScanner.
+initRegionForSnapshotScanner(conf, fs, rootDir, htd, hri);
+
this.scan = scan;
RegionCoprocessorEnvironment snapshotEnv =
getSnapshotContextEnvironment(conf);
@@ -107,6 +114,41 @@ public class SnapshotScanner extends AbstractClientScanner
{
region.startRegionOperation();
}
+ /**
+ * Initialize region for snapshot scanner utility. This is client side
region initialization and
+ * hence it should follow the same region init pattern as the one used by
hbase
+ * ClientSideRegionScanner.
+ *
+ * @param conf The configuration.
+ * @param fs The filesystem instance.
+ * @param rootDir Restored region root dir.
+ * @param htd The table descriptor instance used to retrieve the region root
dir.
+ * @param hri The region info.
+ * @throws IOException If region init throws IOException.
+ */
+ private void initRegionForSnapshotScanner(Configuration conf, FileSystem fs,
Path rootDir,
+TableDescriptor htd,
+RegionInfo hri) throws IOException
{
+region = HRegion.newHRegion(CommonFSUtils.getTableDir(rootDir,
htd.getTableName()), null, fs,
+conf, hri, htd, null);
+region.setRestoredRegion(true);
+// non RS process does not have a block cache, and this a client side
scanner,
+// create one for MapReduce jobs to cache the INDEX block by setting to use
+// IndexOnlyLruBlockCache and set a value to
HBASE_CLIENT_SCANNER_BLOCK_CACHE_SIZE_KEY
+conf.set(BlockCacheFactory.BLOCKCACHE_POLICY_KEY, "IndexOnlyLRU");
+conf.setIfUnset(HConstants.HFILE_ONHEAP_BLOCK_CACHE_FIXED_SIZE_KEY,
+
String.valueOf(HConstants.HBASE_CLIENT_SCANNER_ONHEAP_BLOCK_CACHE_FIXED_SIZE_DEFAULT));
+// don't allow L2 bucket cache for non RS process to avoid unexpected disk
usage.
+conf.unset(HConstants.BUCKET_CACHE_IOENGINE_KEY);
+region.setBlockCache(BlockCacheFactory.createBlockCache(conf));
+// we won't initialize the MobFileCache when not running in RS process. so
provided an
+// initialized cache. Consider the case: an CF was set from an mob to
non-mob. if we only
+// initialize cache for MOB region, NPE from HMobStore will still happen.
So Initialize the
+// cache for every region although it may hasn't any mob CF, BTW the cache
is very light-weight.
+region.setMobFileCache(new MobFileCache(conf));
+region.initialize();
+ }
+
@Override
public Result next() throws IOException {
[phoenix] branch PHOENIX-628-feature updated (5bcebebd8d -> 8c9170f73e)
This is an automated email from the ASF dual-hosted git repository. vjasani pushed a change to branch PHOENIX-628-feature in repository https://gitbox.apache.org/repos/asf/phoenix.git from 5bcebebd8d Merge branch 'master' into PHOENIX-628-feature add f5a425e3ef PHOENIX-7039 Snapshot scanner should skip replay WAL and update seqid while opening region (#1677) new 8c9170f73e Merge branch 'master' into PHOENIX-628-feature The 1 revisions listed above as "new" are entirely new to this repository and will be described in separate emails. The revisions listed as "add" were already present in the repository and have only been added to this reference. Summary of changes: .../apache/phoenix/iterate/SnapshotScanner.java| 44 +- 1 file changed, 43 insertions(+), 1 deletion(-)
[phoenix] branch master updated: PHOENIX-7039 Snapshot scanner should skip replay WAL and update seqid while opening region (#1677)
This is an automated email from the ASF dual-hosted git repository.
vjasani pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/phoenix.git
The following commit(s) were added to refs/heads/master by this push:
new f5a425e3ef PHOENIX-7039 Snapshot scanner should skip replay WAL and
update seqid while opening region (#1677)
f5a425e3ef is described below
commit f5a425e3ef9cf86ed4de61d602c826671342a667
Author: Viraj Jasani
AuthorDate: Sat Sep 30 10:55:05 2023 -0800
PHOENIX-7039 Snapshot scanner should skip replay WAL and update seqid while
opening region (#1677)
---
.../apache/phoenix/iterate/SnapshotScanner.java| 44 +-
1 file changed, 43 insertions(+), 1 deletion(-)
diff --git
a/phoenix-core/src/main/java/org/apache/phoenix/iterate/SnapshotScanner.java
b/phoenix-core/src/main/java/org/apache/phoenix/iterate/SnapshotScanner.java
index f15c5705ab..2cefbece00 100644
--- a/phoenix-core/src/main/java/org/apache/phoenix/iterate/SnapshotScanner.java
+++ b/phoenix-core/src/main/java/org/apache/phoenix/iterate/SnapshotScanner.java
@@ -37,11 +37,14 @@ import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.client.TableDescriptor;
import org.apache.hadoop.hbase.coprocessor.RegionCoprocessor;
import org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment;
+import org.apache.hadoop.hbase.io.hfile.BlockCacheFactory;
import org.apache.hadoop.hbase.metrics.MetricRegistry;
+import org.apache.hadoop.hbase.mob.MobFileCache;
import org.apache.hadoop.hbase.regionserver.HRegion;
import org.apache.hadoop.hbase.regionserver.OnlineRegions;
import org.apache.hadoop.hbase.regionserver.Region;
import org.apache.hadoop.hbase.regionserver.RegionScanner;
+import org.apache.hadoop.hbase.util.CommonFSUtils;
import org.apache.phoenix.coprocessor.BaseScannerRegionObserver;
import org.apache.hadoop.hbase.HConstants;
import org.apache.hadoop.hbase.TableName;
@@ -79,7 +82,11 @@ public class SnapshotScanner extends AbstractClientScanner {
scan.setIsolationLevel(IsolationLevel.READ_UNCOMMITTED);
values = new ArrayList<>();
-this.region = HRegion.openHRegion(conf, fs, rootDir, hri, htd, null, null,
null);
+
+// TODO : Use hbase provided snapshot scanner (make it IA.LimitedPrivate)
+// region init should follow the same pattern as hbase
ClientSideRegionScanner.
+initRegionForSnapshotScanner(conf, fs, rootDir, htd, hri);
+
this.scan = scan;
RegionCoprocessorEnvironment snapshotEnv =
getSnapshotContextEnvironment(conf);
@@ -107,6 +114,41 @@ public class SnapshotScanner extends AbstractClientScanner
{
region.startRegionOperation();
}
+ /**
+ * Initialize region for snapshot scanner utility. This is client side
region initialization and
+ * hence it should follow the same region init pattern as the one used by
hbase
+ * ClientSideRegionScanner.
+ *
+ * @param conf The configuration.
+ * @param fs The filesystem instance.
+ * @param rootDir Restored region root dir.
+ * @param htd The table descriptor instance used to retrieve the region root
dir.
+ * @param hri The region info.
+ * @throws IOException If region init throws IOException.
+ */
+ private void initRegionForSnapshotScanner(Configuration conf, FileSystem fs,
Path rootDir,
+TableDescriptor htd,
+RegionInfo hri) throws IOException
{
+region = HRegion.newHRegion(CommonFSUtils.getTableDir(rootDir,
htd.getTableName()), null, fs,
+conf, hri, htd, null);
+region.setRestoredRegion(true);
+// non RS process does not have a block cache, and this a client side
scanner,
+// create one for MapReduce jobs to cache the INDEX block by setting to use
+// IndexOnlyLruBlockCache and set a value to
HBASE_CLIENT_SCANNER_BLOCK_CACHE_SIZE_KEY
+conf.set(BlockCacheFactory.BLOCKCACHE_POLICY_KEY, "IndexOnlyLRU");
+conf.setIfUnset(HConstants.HFILE_ONHEAP_BLOCK_CACHE_FIXED_SIZE_KEY,
+
String.valueOf(HConstants.HBASE_CLIENT_SCANNER_ONHEAP_BLOCK_CACHE_FIXED_SIZE_DEFAULT));
+// don't allow L2 bucket cache for non RS process to avoid unexpected disk
usage.
+conf.unset(HConstants.BUCKET_CACHE_IOENGINE_KEY);
+region.setBlockCache(BlockCacheFactory.createBlockCache(conf));
+// we won't initialize the MobFileCache when not running in RS process. so
provided an
+// initialized cache. Consider the case: an CF was set from an mob to
non-mob. if we only
+// initialize cache for MOB region, NPE from HMobStore will still happen.
So Initialize the
+// cache for every region although it may hasn't any mob CF, BTW the cache
is very light-weight.
+region.setMobFileCache(new MobFileCache(conf));
+region.initialize();
+ }
+
@Override
public Result next() throws IOException {
