OK, Andrii clarified that we need KAFKA-1927 before KIP-4 for the infrastructure for using the common request/response classes in core.
Jun, when you got a moment, please confirm if the approach I'm taking is acceptable, or if you see issues that I'm missing. On Wed, Mar 18, 2015 at 11:33 AM, Gwen Shapira <gshap...@cloudera.com> wrote: > Looking at the SimpleConsumer, we need to leave in place: > > ConsumerMetadataResponse > FetchRequest / FetchResponse > OffsetFetchRequest / OffsetFetchResponse > OffsetCommitRequest / OffsetCommitResponse > OffsetRequest / OffsetResponse > TopicMetadata > TopicMetadataRequest / TopicMetadataResponse > > Specifically, TopicMetadata request and response will remain > duplicated after KAFKA-1927. > So I am no longer sure why this is a blocker for KIP-4. > > Gwen > > > > On Wed, Mar 18, 2015 at 11:11 AM, Gwen Shapira <gshap...@cloudera.com> wrote: >> Hi Jun, >> >> I was taking a slightly different approach. Let me know if it makes >> sense to you: >> >> 1. Get the bytes from network (kinda unavoidable...) >> 2. Modify RequestChannel.Request to contain header and body (instead >> of a single object) >> 3. Create the head and body from bytes as follow: >> val header: RequestHeader = RequestHeader.parse(buffer) >> val apiKey: Int = header.apiKey >> val body: Struct = >> ProtoUtils.currentRequestSchema(apiKey).read(buffer).asInstanceOf[Struct] >> 4. KafkaAPIs will continue getting RequestChannel.Request, but will >> now have access to body and header separately. >> >> I agree that I need a Request/Response objects that contain only the >> body for all requests objects. >> I'm thinking of implementing them in o.a.k.Common.Requests in Java for >> consistency. >> >> When we are discussing the requests/responses used in SimpleConsumer, >> we mean everything used in javaapi, right? >> >> Gwen >> >> >> >> On Wed, Mar 18, 2015 at 9:55 AM, Jun Rao <j...@confluent.io> wrote: >>> Hi, Gwen, >>> >>> I was thinking that we will be doing the following in KAFKA-1927. >>> >>> 1. Get the bytes from network. >>> 2. Use a new generic approach to convert bytes into request objects. >>> 2.1 Read the fixed request header (using the util in client). >>> 2.2 Based on the request id in the header, deserialize the rest of the >>> bytes into a request specific object (using the new java objects). >>> 3. We will then be passing a header and an AbstractRequestResponse to >>> KafkaApis. >>> >>> In order to do that, we will need to create similar request/response >>> objects for internal requests such as StopReplica, LeaderAndIsr, >>> UpdateMetadata, ControlledShutdown. Not sure whether they should be written >>> in java or scala, but perhaps they should be only in the core project. >>> >>> Also note, there are some scala requests/responses used directly in >>> SimpleConsumer. Since that's our public api, we can't remove those scala >>> objects until the old consumer is phased out. We can remove the rest of the >>> scala request objects. >>> >>> Thanks, >>> >>> Jun >>> >>> >>> On Tue, Mar 17, 2015 at 6:08 PM, Gwen Shapira <gshap...@cloudera.com> wrote: >>> >>>> Hi, >>>> >>>> I'm starting this thread for the various questions I run into while >>>> refactoring the server to use client requests and responses. >>>> >>>> Help is appreciated :) >>>> >>>> First question: LEADER_AND_ISR request and STOP_REPLICA request are >>>> unimplemented in the client. >>>> >>>> Do we want to implement them as part of this refactoring? >>>> Or should we continue using the scala implementation for those? >>>> >>>> Gwen >>>>
