Re: RIP23: Support gRPC protocol
这里的一个思考是，当下多语言大家真的觉得痛吗？可以参考Dapr社区目前的多语言问题。社区的小伙伴可以发表一下自己的意见。另外，我非常担心新协议的兼容性，当下社区发展还是非常快的，全世界范围内使用的人也非常多，希望能够看到平滑过度的更细致的方案。 HTTP3是未来的方向，也是目前尖端企业在研究跟进包括落地的方向，我也希望在proposal里看到这块的想法。 p.s 这个问题可以更开放一些，所以这里我用中文来回复。非常希望听到更多的来自社区朋友的反馈。 炼龙 <1936978...@qq.com> 于2021年6月10日周四 上午6:13写道： > 能否像dubbo那样，除了实现一些常用的序列化，也用SPI，让用户可以自定义序列化？ > > > > 发自我的iPhone > > > -- Original -- > From: yukon Date: Wed,Jun 9,2021 9:48 PM > To: dev Subject: Re: RIP23: Support gRPC protocol > > > > +1 for this proposal. > > Obviously, supporting gRPC could make it easier for RocketMQ contributors > to write multi-Language SDKs. Looking forward to more details of this > proposal. > > Regards, > yukon > > On Wed, Jun 9, 2021 at 11:10 AM Zhanhui Li wrote: > > Hi, > This proposal, in general, is in the right direction that helps > RocketMQ > provide full-fledged SDK for popular languages and platforms. Taking > full > advantage of gRPC does save a lot of effort in terms of serialization > and > RPC tiers. Obviously, this proposal also brings complexities and > potential > compatibility issues. > > One of the potential issues is that gRPC does not expose Channel in > the > implementation while RocketMQ processors make heavy use of it, even > if both > of them are built on top of Netty 4.x. Will this an issue when > reuse > existing code? > > Zhanhui Li > > On Tue, Jun 8, 2021 at 8:28 PM i yangkun wrote: > > Background Motivation > What do we need to do > > > * Will we add a new module? > maybe. > * Will we add new APIs? > Yes. > > * Will we add new feature? > Yes. > > > Why should we do that > > > * Are there any problems of our current > project? > > a. Remoting module is too complicated to maintain, gRPC makes it > easier > to > establish a robust communication layer, the current remoting > module would > be simplified radically. > > b. gRPC has been the de-facto standard in CloudNative, service > mesh would > be easily applied if gRPC is enabled. > > c. The private protocol of RocketMQ depends on the FastJson, it > is > difficult to adapt for other language. > > On the other side, since the pop consumer has been merged, we > could > implement new SDK based on gRPC and pop, which is easier to > develop and > maintain. > > Chinese Version: > > a. Remoting 模块对于长期的维护而言过于复杂了，我们可以使用 gRPC 更轻松地建立起一个健壮的通信层，这会使得现有的 > remoting > 模块从根本上得到简化。 > > b. gRPC 目前已经是云原生时代的事实标准，使用 gRPC 可以使得我们天然获取一些云原生的能力，比如 Service > Mesh。 > > c. 目前 RocketMQ 的私有协议强烈依赖 FastJson，多语言的适配将会变得困难。 > > > 从另外一个角度来说，鉴于 pop 消费者已经被合并，我们可以基于 gRPC 和 pop 实现新的 SDK，新的 SDK > 将会更加易于开发和维护。 > > Goals > > > * What problem is this proposal designed > to solve? > > Support gRPC's protocol, simplify current communication layer oof > RocketMQ, make it easier to adapt for other language, which is > not > limited > to CPP/GO/C#/GO。 > > Chinese Version: > > 支持 gRPC 协议，简化 RocketMQ 现有的通信层，复用 gRPC 的能力，简化多语言适配成本，不限于 > CPP/GO/C#/GO。 > > * To what degree should we solve the > problem? > This RIP must guarantee below point: > > 1. Compatibility: Both of gRPC and > RemotingCommand should be > supported. > 2. High performance: This implementation does > not affects latency and > throughput. > > > Chinese Version: > > 新方案需要保证两点： > > 1. 兼容性：同时支持 gRPC 和 RemotingCommand 协议，不影响现有功能。 > 2. 高性能：基于 gRPC 的实现不影响整理的延时和吞吐量。 > > > Non-Goals > > > * What problem is this proposal NOT > designed to solve? > Nothing specific. > * Are there any limits of this proposal? > Nothing specific. > > > Changes > Architecture > > > Current broker processor and client. > > [ > > > https://intranetproxy.alipay.com/skylark/lark/0/2021/png/200096/1623142547507-128b85f5-98f4-4568-85f8-28ef32982b7c.png > ] > > Proposed gRPC processor and client. > > [ > > > https://intranetproxy.alipay.com/skylark/lark/0/2021/png/200096/1623142552491-a7f58ac0-cd7d-4ddd-936e-fb296b667196.png > ] > > Broker would provide a protocol negotiate procedure to > distinguish > RemotingCommand from gRPC protocol. two kinds or processor in > broker > would > re-use the same port to serve for RPC from different SDK. > > > Chinese Version: > > broker 本身提供协议协商机制用于区分 RemotingCommnad 和 gRPC 协议，broker 针对 gRPC 和 > RemotingCommand 提供不同的 processor 为各自的 SDK 服务。 > > Interfac
Re: RIP23: Support gRPC protocol
能否像dubbo那样，除了实现一些常用的序列化，也用SPI，让用户可以自定义序列化？ 发自我的iPhone -- Original -- From: yukon https://intranetproxy.alipay.com/skylark/lark/0/2021/png/200096/1623142547507-128b85f5-98f4-4568-85f8-28ef32982b7c.png ] Proposed gRPC processor and client. [ https://intranetproxy.alipay.com/skylark/lark/0/2021/png/200096/1623142552491-a7f58ac0-cd7d-4ddd-936e-fb296b667196.png ] Broker would provide a protocol negotiate procedure to distinguish RemotingCommand from gRPC protocol. two kinds or processor in broker would re-use the same port to serve for RPC from different SDK. Chinese Version: broker 本身提供协议协商机制用于区分 RemotingCommnad 和 gRPC 协议，broker 针对 gRPC 和 RemotingCommand 提供不同的 processor 为各自的 SDK 服务。 Interface Design/Change * Method signature changes Nothing specific. * Method behavior changes Nothing specific. * CLI command changes Nothing specific. * Log format or content changes Nothing specific. Compatibility, Deprecation, and Migration Plan * Are backward and forward compatibility taken into consideration? Broker support processor of RemotingCommand and gRPC simultaneously, so there are one compatibility situations: If user migrates from original SDK to gRPC SDK in push mode, the re-balance policy should make sure that it would not cause repeated consumption for a lot of messages. * Are there deprecated APIs? Nothing specific. * How do we do migration? Nothing specific. Implementation Outline We will implement the proposed changes by 4 phases. Phase 1 1. Provides gRPC protocol definition(IDL) Phase 2 1. Implement gRPC processor of broker. 2. Implement protocol negotiation of two kinds of protocol(gRPC and RemotingCommand) Phase 3 1. Implement new JAVA and CPP native SDK based on gRPC Phase 4 1. Implement native SDK base on gRPC for other language. Rejected Alternatives How does alternatives solve the issue you proposed? Thrift? not so much impact as gRPC in community. Pros and Cons of alternatives Nothing specific. Why should we reject above alternatives
Re: RIP23: Support gRPC protocol
+1 for this proposal. Obviously, supporting gRPC could make it easier for RocketMQ contributors to write multi-Language SDKs. Looking forward to more details of this proposal. Regards, yukon On Wed, Jun 9, 2021 at 11:10 AM Zhanhui Li wrote: > Hi, > This proposal, in general, is in the right direction that helps RocketMQ > provide full-fledged SDK for popular languages and platforms. Taking full > advantage of gRPC does save a lot of effort in terms of serialization and > RPC tiers. Obviously, this proposal also brings complexities and potential > compatibility issues. > > One of the potential issues is that gRPC does not expose Channel in the > implementation while RocketMQ processors make heavy use of it, even if both > of them are built on top of Netty 4.x. Will this an issue when reuse > existing code? > > Zhanhui Li > > On Tue, Jun 8, 2021 at 8:28 PM i yangkun wrote: > > > Background & Motivation > > What do we need to do > > > > > > * Will we add a new module? > > maybe. > > * Will we add new APIs? > > Yes. > > > > * Will we add new feature? > > Yes. > > > > > > Why should we do that > > > > > > * Are there any problems of our current project? > > > > a. Remoting module is too complicated to maintain, gRPC makes it easier > to > > establish a robust communication layer, the current remoting module would > > be simplified radically. > > > > b. gRPC has been the de-facto standard in CloudNative, service mesh would > > be easily applied if gRPC is enabled. > > > > c. The private protocol of RocketMQ depends on the FastJson, it is > > difficult to adapt for other language. > > > > On the other side, since the pop consumer has been merged, we could > > implement new SDK based on gRPC and pop, which is easier to develop and > > maintain. > > > > Chinese Version: > > > > a. Remoting 模块对于长期的维护而言过于复杂了，我们可以使用 gRPC 更轻松地建立起一个健壮的通信层，这会使得现有的 remoting > > 模块从根本上得到简化。 > > > > b. gRPC 目前已经是云原生时代的事实标准，使用 gRPC 可以使得我们天然获取一些云原生的能力，比如 Service Mesh。 > > > > c. 目前 RocketMQ 的私有协议强烈依赖 FastJson，多语言的适配将会变得困难。 > > > > > > 从另外一个角度来说，鉴于 pop 消费者已经被合并，我们可以基于 gRPC 和 pop 实现新的 SDK，新的 SDK 将会更加易于开发和维护。 > > > > Goals > > > > > > * What problem is this proposal designed to solve? > > > > Support gRPC's protocol, simplify current communication layer oof > > RocketMQ, make it easier to adapt for other language, which is not > limited > > to CPP/GO/C#/GO。 > > > > Chinese Version: > > > > 支持 gRPC 协议，简化 RocketMQ 现有的通信层，复用 gRPC 的能力，简化多语言适配成本，不限于 CPP/GO/C#/GO。 > > > > * To what degree should we solve the problem? > > This RIP must guarantee below point: > > > > 1. Compatibility: Both of gRPC and RemotingCommand should be > supported. > > 2. High performance: This implementation does not affects latency and > > throughput. > > > > > > Chinese Version: > > > > 新方案需要保证两点： > > > > 1. 兼容性：同时支持 gRPC 和 RemotingCommand 协议，不影响现有功能。 > > 2. 高性能：基于 gRPC 的实现不影响整理的延时和吞吐量。 > > > > > > Non-Goals > > > > > > * What problem is this proposal NOT designed to solve? > > Nothing specific. > > * Are there any limits of this proposal? > > Nothing specific. > > > > > > Changes > > Architecture > > > > > > Current broker processor and client. > > > > [ > > > https://intranetproxy.alipay.com/skylark/lark/0/2021/png/200096/1623142547507-128b85f5-98f4-4568-85f8-28ef32982b7c.png > > ] > > > > Proposed gRPC processor and client. > > > > [ > > > https://intranetproxy.alipay.com/skylark/lark/0/2021/png/200096/1623142552491-a7f58ac0-cd7d-4ddd-936e-fb296b667196.png > > ] > > > > Broker would provide a protocol negotiate procedure to distinguish > > RemotingCommand from gRPC protocol. two kinds or processor in broker > would > > re-use the same port to serve for RPC from different SDK. > > > > > > Chinese Version: > > > > broker 本身提供协议协商机制用于区分 RemotingCommnad 和 gRPC 协议，broker 针对 gRPC 和 > > RemotingCommand 提供不同的 processor 为各自的 SDK 服务。 > > > > Interface Design/Change > > > > > > * Method signature changes > > Nothing specific. > > * Method behavior changes > > Nothing specific. > > > > * CLI command changes > > Nothing specific. > > * Log format or content changes > > Nothing specific. > > > > > > Compatibility, Deprecation, and Migration Plan > > > > > > * Are backward and forward compatibility taken into consideration? > > > > Broker support processor of RemotingCommand and gRPC simultaneously, so > > there are one compatibility situations: > > > > If user migrates from original SDK to gRPC SDK in push mode, the > > re-balance policy should make sure that it would not cause repeated > > consumption for a lot of messages. > > > > * Are there deprecated APIs? > > Nothing specific. > > * How do we do migration? > > Nothing specific. > > > > > > Implementation Outline > > > > > > We will implement the proposed changes by 4 phases. > > > > > > Phase 1 > > > > 1. Provides gRPC protocol definition(IDL) > > > > Phase 2 > > > > 1. Implement gRPC processor of broker. > > 2. Implement protocol negotiation
Re: RIP23: Support gRPC protocol
Hi, This proposal, in general, is in the right direction that helps RocketMQ provide full-fledged SDK for popular languages and platforms. Taking full advantage of gRPC does save a lot of effort in terms of serialization and RPC tiers. Obviously, this proposal also brings complexities and potential compatibility issues. One of the potential issues is that gRPC does not expose Channel in the implementation while RocketMQ processors make heavy use of it, even if both of them are built on top of Netty 4.x. Will this an issue when reuse existing code? Zhanhui Li On Tue, Jun 8, 2021 at 8:28 PM i yangkun wrote: > Background & Motivation > What do we need to do > > > * Will we add a new module? > maybe. > * Will we add new APIs? > Yes. > > * Will we add new feature? > Yes. > > > Why should we do that > > > * Are there any problems of our current project? > > a. Remoting module is too complicated to maintain, gRPC makes it easier to > establish a robust communication layer, the current remoting module would > be simplified radically. > > b. gRPC has been the de-facto standard in CloudNative, service mesh would > be easily applied if gRPC is enabled. > > c. The private protocol of RocketMQ depends on the FastJson, it is > difficult to adapt for other language. > > On the other side, since the pop consumer has been merged, we could > implement new SDK based on gRPC and pop, which is easier to develop and > maintain. > > Chinese Version: > > a. Remoting 模块对于长期的维护而言过于复杂了，我们可以使用 gRPC 更轻松地建立起一个健壮的通信层，这会使得现有的 remoting > 模块从根本上得到简化。 > > b. gRPC 目前已经是云原生时代的事实标准，使用 gRPC 可以使得我们天然获取一些云原生的能力，比如 Service Mesh。 > > c. 目前 RocketMQ 的私有协议强烈依赖 FastJson，多语言的适配将会变得困难。 > > > 从另外一个角度来说，鉴于 pop 消费者已经被合并，我们可以基于 gRPC 和 pop 实现新的 SDK，新的 SDK 将会更加易于开发和维护。 > > Goals > > > * What problem is this proposal designed to solve? > > Support gRPC's protocol, simplify current communication layer oof > RocketMQ, make it easier to adapt for other language, which is not limited > to CPP/GO/C#/GO。 > > Chinese Version: > > 支持 gRPC 协议，简化 RocketMQ 现有的通信层，复用 gRPC 的能力，简化多语言适配成本，不限于 CPP/GO/C#/GO。 > > * To what degree should we solve the problem? > This RIP must guarantee below point: > > 1. Compatibility: Both of gRPC and RemotingCommand should be supported. > 2. High performance: This implementation does not affects latency and > throughput. > > > Chinese Version: > > 新方案需要保证两点： > > 1. 兼容性：同时支持 gRPC 和 RemotingCommand 协议，不影响现有功能。 > 2. 高性能：基于 gRPC 的实现不影响整理的延时和吞吐量。 > > > Non-Goals > > > * What problem is this proposal NOT designed to solve? > Nothing specific. > * Are there any limits of this proposal? > Nothing specific. > > > Changes > Architecture > > > Current broker processor and client. > > [ > https://intranetproxy.alipay.com/skylark/lark/0/2021/png/200096/1623142547507-128b85f5-98f4-4568-85f8-28ef32982b7c.png > ] > > Proposed gRPC processor and client. > > [ > https://intranetproxy.alipay.com/skylark/lark/0/2021/png/200096/1623142552491-a7f58ac0-cd7d-4ddd-936e-fb296b667196.png > ] > > Broker would provide a protocol negotiate procedure to distinguish > RemotingCommand from gRPC protocol. two kinds or processor in broker would > re-use the same port to serve for RPC from different SDK. > > > Chinese Version: > > broker 本身提供协议协商机制用于区分 RemotingCommnad 和 gRPC 协议，broker 针对 gRPC 和 > RemotingCommand 提供不同的 processor 为各自的 SDK 服务。 > > Interface Design/Change > > > * Method signature changes > Nothing specific. > * Method behavior changes > Nothing specific. > > * CLI command changes > Nothing specific. > * Log format or content changes > Nothing specific. > > > Compatibility, Deprecation, and Migration Plan > > > * Are backward and forward compatibility taken into consideration? > > Broker support processor of RemotingCommand and gRPC simultaneously, so > there are one compatibility situations: > > If user migrates from original SDK to gRPC SDK in push mode, the > re-balance policy should make sure that it would not cause repeated > consumption for a lot of messages. > > * Are there deprecated APIs? > Nothing specific. > * How do we do migration? > Nothing specific. > > > Implementation Outline > > > We will implement the proposed changes by 4 phases. > > > Phase 1 > > 1. Provides gRPC protocol definition(IDL) > > Phase 2 > > 1. Implement gRPC processor of broker. > 2. Implement protocol negotiation of two kinds of protocol(gRPC and > RemotingCommand) > > Phase 3 > > 1. Implement new JAVA and CPP native SDK based on gRPC > > Phase 4 > > 1. Implement native SDK base on gRPC for other language. > > > Rejected Alternatives > > > How does alternatives solve the issue you proposed? > > > Thrift? not so much impact as gRPC in community. > > > Pros and Cons of alternatives > > > Nothing specific. > > Why should we reject above alternatives > >
