[jira] [Commented] (HBASE-25340) Protobuf Mesage Incompatibility Detector
[
https://issues.apache.org/jira/browse/HBASE-25340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339121#comment-17339121
]
Nick Dimiduk commented on HBASE-25340:
--
[~jwjwyoung] do you have a link to the newer version? Does the Java
implementation replace the Python implementation going forward? I think you'll
find good interest in this tool over in the [Apache
Yetus|https://yetus.apache.org] community. If you can get the tool added as a
new Yetus check, then it can easily be integrated into HBase and Hadoop CI
testing. If you want to target the Java ecosystem, you'll have much better
adoption if you offer a plugin for popular build tools like Maven, Gradle.
Thanks for the effort!
> Protobuf Mesage Incompatibility Detector
>
>
> Key: HBASE-25340
> URL: https://issues.apache.org/jira/browse/HBASE-25340
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.3, 2.2.6
>Reporter: junwen yang
>Priority: Major
> Attachments: protoCompatibility.txt
>
>
> Regarding the issue HBASE-25238. caused by the incompatibility of protobuf
> message, we have created a static checker which keeps track of the proto file
> change, and detect potential incompatibility:
> # Add/delete required field, which is exactly the case reported in
> HBASE-25238.
> # The tag number of a field has been changed, as described in HDFS-9788.
> Also, the [protobuf guidelines
> |https://developers.google.com/protocol-buffers/docs/proto]suggests _each
> field in the message definition has a *unique number*. These numbers are used
> to identify your fields in the [message binary
> format|https://developers.google.com/protocol-buffers/docs/encoding], and
> should not be changed once your message type is in use_.
> # A required field has been changed to optional, or an optional field has
> been changed to required. According to the guidelines in [protobuf official
> website|https://developers.google.com/protocol-buffers/docs/proto],
> _*Required Is Forever* You should be very careful about marking fields as
> {{required}}. If at some point you wish to stop writing or sending a required
> field, it will be problematic to change the field to an optional field - old
> readers will consider messages without this field to be incomplete and may
> reject or drop them unintentionally. You should consider writing
> application-specific custom validation routines for your buffers instead._
> We have applied our checker on the frequently maintained HBASE versions:
> rel/1.4.3, rel/1.6.0, rel/2.2.6, rel/2.3.3, and master, and we found 37
> potential problems as attached, which includes the one found in HBASE-25238.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25340) Protobuf Mesage Incompatibility Detector
[
https://issues.apache.org/jira/browse/HBASE-25340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17339036#comment-17339036
]
junwen yang commented on HBASE-25340:
-
Hi, [~pankajkumar], just a quick update that we have rewritten the tool in
Java. Let me know if you need any help to integrate this into your system. We
are more than willing to help!
> Protobuf Mesage Incompatibility Detector
>
>
> Key: HBASE-25340
> URL: https://issues.apache.org/jira/browse/HBASE-25340
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.3, 2.2.6
>Reporter: junwen yang
>Priority: Major
> Attachments: protoCompatibility.txt
>
>
> Regarding the issue HBASE-25238. caused by the incompatibility of protobuf
> message, we have created a static checker which keeps track of the proto file
> change, and detect potential incompatibility:
> # Add/delete required field, which is exactly the case reported in
> HBASE-25238.
> # The tag number of a field has been changed, as described in HDFS-9788.
> Also, the [protobuf guidelines
> |https://developers.google.com/protocol-buffers/docs/proto]suggests _each
> field in the message definition has a *unique number*. These numbers are used
> to identify your fields in the [message binary
> format|https://developers.google.com/protocol-buffers/docs/encoding], and
> should not be changed once your message type is in use_.
> # A required field has been changed to optional, or an optional field has
> been changed to required. According to the guidelines in [protobuf official
> website|https://developers.google.com/protocol-buffers/docs/proto],
> _*Required Is Forever* You should be very careful about marking fields as
> {{required}}. If at some point you wish to stop writing or sending a required
> field, it will be problematic to change the field to an optional field - old
> readers will consider messages without this field to be incomplete and may
> reject or drop them unintentionally. You should consider writing
> application-specific custom validation routines for your buffers instead._
> We have applied our checker on the frequently maintained HBASE versions:
> rel/1.4.3, rel/1.6.0, rel/2.2.6, rel/2.3.3, and master, and we found 37
> potential problems as attached, which includes the one found in HBASE-25238.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25340) Protobuf Mesage Incompatibility Detector
[
https://issues.apache.org/jira/browse/HBASE-25340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241226#comment-17241226
]
junwen yang commented on HBASE-25340:
-
Hi, [~pankajkumar], this is the repo to our static checker:
[https://github.com/jwjwyoung/SLCChecker.]
To run it, you can check the README file, you simply need to specify the
application path, the version tags, and the type of file you want to check.
Currently it's still a prototype and we are keeping enhancing it with more
functionalities such as supporting other message serialization compatibility
library. If you have any requests, feel free to tell us and we'll improve
based on them.
Also, if you feel this tool is valuable to be integrated into HBASE, we are
more than willing to offer any help.
> Protobuf Mesage Incompatibility Detector
>
>
> Key: HBASE-25340
> URL: https://issues.apache.org/jira/browse/HBASE-25340
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.3, 2.2.6
>Reporter: junwen yang
>Priority: Major
> Attachments: protoCompatibility.txt
>
>
> Regarding the issue HBASE-25238. caused by the incompatibility of protobuf
> message, we have created a static checker which keeps track of the proto file
> change, and detect potential incompatibility:
> # Add/delete required field, which is exactly the case reported in
> HBASE-25238.
> # The tag number of a field has been changed, as described in HDFS-9788.
> Also, the [protobuf guidelines
> |https://developers.google.com/protocol-buffers/docs/proto]suggests _each
> field in the message definition has a *unique number*. These numbers are used
> to identify your fields in the [message binary
> format|https://developers.google.com/protocol-buffers/docs/encoding], and
> should not be changed once your message type is in use_.
> # A required field has been changed to optional, or an optional field has
> been changed to required. According to the guidelines in [protobuf official
> website|https://developers.google.com/protocol-buffers/docs/proto],
> _*Required Is Forever* You should be very careful about marking fields as
> {{required}}. If at some point you wish to stop writing or sending a required
> field, it will be problematic to change the field to an optional field - old
> readers will consider messages without this field to be incomplete and may
> reject or drop them unintentionally. You should consider writing
> application-specific custom validation routines for your buffers instead._
> We have applied our checker on the frequently maintained HBASE versions:
> rel/1.4.3, rel/1.6.0, rel/2.2.6, rel/2.3.3, and master, and we found 37
> potential problems as attached, which includes the one found in HBASE-25238.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25340) Protobuf Mesage Incompatibility Detector
[
https://issues.apache.org/jira/browse/HBASE-25340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241208#comment-17241208
]
Pankaj Kumar commented on HBASE-25340:
--
Yes [~zagol], I'm expecting a tool to detect incompatibiilities based on proto
files. It will be helpful to identify incompatibiilities during HBase upgrade
and need to support old version clients as well.
> Protobuf Mesage Incompatibility Detector
>
>
> Key: HBASE-25340
> URL: https://issues.apache.org/jira/browse/HBASE-25340
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.3, 2.2.6
>Reporter: junwen yang
>Priority: Major
> Attachments: protoCompatibility.txt
>
>
> Regarding the issue HBASE-25238. caused by the incompatibility of protobuf
> message, we have created a static checker which keeps track of the proto file
> change, and detect potential incompatibility:
> # Add/delete required field, which is exactly the case reported in
> HBASE-25238.
> # The tag number of a field has been changed, as described in HDFS-9788.
> Also, the [protobuf guidelines
> |https://developers.google.com/protocol-buffers/docs/proto]suggests _each
> field in the message definition has a *unique number*. These numbers are used
> to identify your fields in the [message binary
> format|https://developers.google.com/protocol-buffers/docs/encoding], and
> should not be changed once your message type is in use_.
> # A required field has been changed to optional, or an optional field has
> been changed to required. According to the guidelines in [protobuf official
> website|https://developers.google.com/protocol-buffers/docs/proto],
> _*Required Is Forever* You should be very careful about marking fields as
> {{required}}. If at some point you wish to stop writing or sending a required
> field, it will be problematic to change the field to an optional field - old
> readers will consider messages without this field to be incomplete and may
> reject or drop them unintentionally. You should consider writing
> application-specific custom validation routines for your buffers instead._
> We have applied our checker on the frequently maintained HBASE versions:
> rel/1.4.3, rel/1.6.0, rel/2.2.6, rel/2.3.3, and master, and we found 37
> potential problems as attached, which includes the one found in HBASE-25238.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25340) Protobuf Mesage Incompatibility Detector
[
https://issues.apache.org/jira/browse/HBASE-25340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241080#comment-17241080
]
Wei-Chiu Chuang commented on HBASE-25340:
-
If this jira is about adding a checker, rather than making an incompatible
change, please remove the "Incompatible change" flag from the jira.
> Protobuf Mesage Incompatibility Detector
>
>
> Key: HBASE-25340
> URL: https://issues.apache.org/jira/browse/HBASE-25340
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.3, 2.2.6
>Reporter: junwen yang
>Priority: Major
> Attachments: protoCompatibility.txt
>
>
> Regarding the issue HBASE-25238. caused by the incompatibility of protobuf
> message, we have created a static checker which keeps track of the proto file
> change, and detect potential incompatibility:
> # Add/delete required field, which is exactly the case reported in
> HBASE-25238.
> # The tag number of a field has been changed, as described in HDFS-9788.
> Also, the [protobuf guidelines
> |https://developers.google.com/protocol-buffers/docs/proto]suggests _each
> field in the message definition has a *unique number*. These numbers are used
> to identify your fields in the [message binary
> format|https://developers.google.com/protocol-buffers/docs/encoding], and
> should not be changed once your message type is in use_.
> # A required field has been changed to optional, or an optional field has
> been changed to required. According to the guidelines in [protobuf official
> website|https://developers.google.com/protocol-buffers/docs/proto],
> _*Required Is Forever* You should be very careful about marking fields as
> {{required}}. If at some point you wish to stop writing or sending a required
> field, it will be problematic to change the field to an optional field - old
> readers will consider messages without this field to be incomplete and may
> reject or drop them unintentionally. You should consider writing
> application-specific custom validation routines for your buffers instead._
> We have applied our checker on the frequently maintained HBASE versions:
> rel/1.4.3, rel/1.6.0, rel/2.2.6, rel/2.3.3, and master, and we found 37
> potential problems as attached, which includes the one found in HBASE-25238.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25340) Protobuf Mesage Incompatibility Detector
[
https://issues.apache.org/jira/browse/HBASE-25340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17241005#comment-17241005
]
Yongle Zhang commented on HBASE-25340:
--
Hi [~pankajkumar], do you mean a PR for the static checker?
> Protobuf Mesage Incompatibility Detector
>
>
> Key: HBASE-25340
> URL: https://issues.apache.org/jira/browse/HBASE-25340
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.3, 2.2.6
>Reporter: junwen yang
>Priority: Major
> Attachments: protoCompatibility.txt
>
>
> Regarding the issue HBASE-25238. caused by the incompatibility of protobuf
> message, we have created a static checker which keeps track of the proto file
> change, and detect potential incompatibility:
> # Add/delete required field, which is exactly the case reported in
> HBASE-25238.
> # The tag number of a field has been changed, as described in HDFS-9788.
> Also, the [protobuf guidelines
> |https://developers.google.com/protocol-buffers/docs/proto]suggests _each
> field in the message definition has a *unique number*. These numbers are used
> to identify your fields in the [message binary
> format|https://developers.google.com/protocol-buffers/docs/encoding], and
> should not be changed once your message type is in use_.
> # A required field has been changed to optional, or an optional field has
> been changed to required. According to the guidelines in [protobuf official
> website|https://developers.google.com/protocol-buffers/docs/proto],
> _*Required Is Forever* You should be very careful about marking fields as
> {{required}}. If at some point you wish to stop writing or sending a required
> field, it will be problematic to change the field to an optional field - old
> readers will consider messages without this field to be incomplete and may
> reject or drop them unintentionally. You should consider writing
> application-specific custom validation routines for your buffers instead._
> We have applied our checker on the frequently maintained HBASE versions:
> rel/1.4.3, rel/1.6.0, rel/2.2.6, rel/2.3.3, and master, and we found 37
> potential problems as attached, which includes the one found in HBASE-25238.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-25340) Protobuf Mesage Incompatibility Detector
[
https://issues.apache.org/jira/browse/HBASE-25340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240480#comment-17240480
]
Pankaj Kumar commented on HBASE-25340:
--
Nice one [~jwjwyoung], do you mind raising PR?
> Protobuf Mesage Incompatibility Detector
>
>
> Key: HBASE-25340
> URL: https://issues.apache.org/jira/browse/HBASE-25340
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.3, 2.2.6
>Reporter: junwen yang
>Priority: Major
> Attachments: protoCompatibility.txt
>
>
> Regarding the issue HBASE-25238. caused by the incompatibility of protobuf
> message, we have created a static checker which keeps track of the proto file
> change, and detect potential incompatibility:
> # Add/delete required field, which is exactly the case reported in
> HBASE-25238.
> # The tag number of a field has been changed, as described in HDFS-9788.
> Also, the [protobuf guidelines
> |https://developers.google.com/protocol-buffers/docs/proto]suggests _each
> field in the message definition has a *unique number*. These numbers are used
> to identify your fields in the [message binary
> format|https://developers.google.com/protocol-buffers/docs/encoding], and
> should not be changed once your message type is in use_.
> # A required field has been changed to optional, or an optional field has
> been changed to required. According to the guidelines in [protobuf official
> website|https://developers.google.com/protocol-buffers/docs/proto],
> _*Required Is Forever* You should be very careful about marking fields as
> {{required}}. If at some point you wish to stop writing or sending a required
> field, it will be problematic to change the field to an optional field - old
> readers will consider messages without this field to be incomplete and may
> reject or drop them unintentionally. You should consider writing
> application-specific custom validation routines for your buffers instead._
> We have applied our checker on the frequently maintained HBASE versions:
> rel/1.4.3, rel/1.6.0, rel/2.2.6, rel/2.3.3, and master, and we found 37
> potential problems as attached, which includes the one found in HBASE-25238.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
