[
https://issues.apache.org/jira/browse/HBASE-26021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17368796#comment-17368796
]
Viraj Jasani edited comment on HBASE-26021 at 6/24/21, 12:04 PM:
-----------------------------------------------------------------
Here is the patch that resolves this (de)serialization issue, I have confirmed
this. The only thing I am worried about is what else could be compromised with
Proto incompatibilities with missing Proto message.
{code:java}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
index c275f00c72..a2a9b0df0f 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
@@ -96,7 +96,8 @@ public class TableDescriptor {
* @see #parseFrom(byte[])
*/
public byte [] toByteArray() {
- return ProtobufUtil.prependPBMagic(convert().toByteArray());
+ return ProtobufUtil
+ .prependPBMagic(hTableDescriptor.convert().toByteArray());
}
{code}
This patch works for HBase 1.7 as well, because as I mentioned above, hbase-1
takes care of backward compatibility by catching DeserializationException and
trying to deserialize to old TD i.e HTableDescriptor. Confirmed this as well.
What I believe we should consider path forward for this issue:
# Apply this patch and consider rolling out 1.8 release for anyone looking
forward to upgrade from HBase 1 to 2.
# Thoughts on catching any missing edge cases of HBASE-7767 backport. I think
as long as backport helps MasterRegistry's work to land on branch-1, we are
good and we might not want to consider rework on it. I am already running PE as
of now in my local testing to generate more traffic and add millions of rows
and scanning them, but this might not be enough and we could plan for rigorous
testing on big cluster.
Edit: From the code, I can see HBaseProtos.TableDescriptor is not used anywhere
else other than the usecase we are focusing on. Other than introducing
TableDescriptor and TableState in HBase proto, I don't see any changes in proto
of big concern. The patch also has changes in Master and Zookeeper protos but
they don't seem problematic from their usage.
I think we should be good with the above patch once we have enough +1 and we
can soon plan for new release from branch-1.
was (Author: vjasani):
Here is the patch that resolves this (de)serialization issue, I have confirmed
this. The only thing I am worried about is what else could be compromised with
Proto incompatibilities with missing Proto message.
{code:java}
diff --git
a/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
index c275f00c72..a2a9b0df0f 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptor.java
@@ -96,7 +96,8 @@ public class TableDescriptor {
* @see #parseFrom(byte[])
*/
public byte [] toByteArray() {
- return ProtobufUtil.prependPBMagic(convert().toByteArray());
+ return ProtobufUtil
+ .prependPBMagic(hTableDescriptor.convert().toByteArray());
}
{code}
This patch works for HBase 1.7 as well, because as I mentioned above, hbase-1
takes care of backward compatibility by catching DeserializationException and
trying to deserialize to old TD i.e HTableDescriptor. Confirmed this as well.
What I believe we should consider path forward for this issue:
# Apply this patch and consider rolling out 1.8 release for anyone looking
forward to upgrade from HBase 1 to 2.
# Thoughts on catching any missing edge cases of HBASE-7767 backport. I think
as long as backport helps MasterRegistry's work to land on branch-1, we are
good but looks like we requires rigorous testing with this patch. I am already
running pe as of now in my local testing to generate more traffic and add
millions of rows and scanning them, but this might not be enough.
> HBase 1.7 to 2.4 upgrade issue due to incompatible deserialization
> ------------------------------------------------------------------
>
> Key: HBASE-26021
> URL: https://issues.apache.org/jira/browse/HBASE-26021
> Project: HBase
> Issue Type: Bug
> Affects Versions: 1.7.0, 2.4.4
> Reporter: Viraj Jasani
> Priority: Major
> Attachments: Screenshot 2021-06-22 at 12.54.21 PM.png, Screenshot
> 2021-06-22 at 12.54.30 PM.png
>
>
> As of today, if we bring up HBase cluster using branch-1 and upgrade to
> branch-2.4, we are facing issue while parsing namespace from HDFS fileinfo.
> Instead of "*hbase:meta*" and "*hbase:namespace*", parsing using ProtobufUtil
> seems to be producing "*\n hbase meta*" and "*\n hbase namespace*"
> {code:java}
> 2021-06-22 00:05:56,611 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
> regionserver.RSRpcServices: Open hbase:meta,,1.1588230740
> 2021-06-22 00:05:56,648 INFO
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> regionserver.RSRpcServices: Open
> hbase:namespace,,1624297762817.396cb6cc00cd4334cb1ea3a792d7529a.
> 2021-06-22 00:05:56,759 ERROR
> [RpcServer.priority.RWQ.Fifo.read.handler=5,queue=1,port=16025]
> ipc.RpcServer: Unexpected throwable object
> java.lang.IllegalArgumentException: Illegal character <
> > at 0. Namespaces may only contain 'alphanumeric characters' from any
> > language and digits:
> ^Ehbase^R namespace
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
> at org.apache.hadoop.hbase.TableName.<init>(TableName.java:348)
> at
> org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:385)
> at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:508)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2292)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableDescriptor(ProtobufUtil.java:2937)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.parseFrom(TableDescriptorBuilder.java:1625)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.access$200(TableDescriptorBuilder.java:597)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder.parseFrom(TableDescriptorBuilder.java:320)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:511)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:496)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:482)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:210)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:2112)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:35218)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:395)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
> 2021-06-22 00:05:56,759 ERROR
> [RpcServer.priority.RWQ.Fifo.read.handler=3,queue=1,port=16025]
> ipc.RpcServer: Unexpected throwable object
> java.lang.IllegalArgumentException: Illegal character <
> > at 0. Namespaces may only contain 'alphanumeric characters' from any
> > language and digits:
> ^Ehbase^R^Dmeta
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:246)
> at
> org.apache.hadoop.hbase.TableName.isLegalNamespaceName(TableName.java:220)
> at org.apache.hadoop.hbase.TableName.<init>(TableName.java:348)
> at
> org.apache.hadoop.hbase.TableName.createTableNameIfNecessary(TableName.java:385)
> at org.apache.hadoop.hbase.TableName.valueOf(TableName.java:508)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableName(ProtobufUtil.java:2292)
> at
> org.apache.hadoop.hbase.shaded.protobuf.ProtobufUtil.toTableDescriptor(ProtobufUtil.java:2937)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.parseFrom(TableDescriptorBuilder.java:1625)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder$ModifyableTableDescriptor.access$200(TableDescriptorBuilder.java:597)
> at
> org.apache.hadoop.hbase.client.TableDescriptorBuilder.parseFrom(TableDescriptorBuilder.java:320)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.readTableDescriptor(FSTableDescriptors.java:511)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:496)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.getTableDescriptorFromFs(FSTableDescriptors.java:482)
> at
> org.apache.hadoop.hbase.util.FSTableDescriptors.get(FSTableDescriptors.java:210)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.openRegion(RSRpcServices.java:2112)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:35218)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:395)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
