dajac commented on pull request #9689: URL: https://github.com/apache/kafka/pull/9689#issuecomment-747363671
@hachikuji I totally agree with your comment. I have been pursuing this migration for this API to better understand the issues that we would face if we really want to directly use the auto-generated requests and responses in the near future. If we want to proceed on that path, I think that we should evolved the auto-generated protocol. I envision the following improvements: 1. The first area is about the `Map`. We made the choice to not use `Map` in the auto-generated classes in favour of using our internal map-like data structure which is more efficient. The down side of this is that it always requires to convert our internal `Map` to that data structure. It would be great if we could just use the internal `Map` directly. That would reduce the boiler plate code to do the conversion. 2. Similarly, it seems that could make `TopicPartition` a first class citizen in the auto-generated protocol. It would be so nice if we would get `Map` with `TopicPartition` as the key when a request or a response represents topics/partitions. I have put some details here: https://issues.apache.org/jira/browse/KAFKA-10795. This would remove the awkwardness that you are talking about as, in our particular case, it would remove the partition index from `EpochData`. 3. I believe that we should add support for `Optional` in the auto-generated protocol. The idea is to get an empty option when the value is equals to the sentinel value. I prototyped this a while ago. The PR is a bit outdated but the overall idea remains. It would be great it you could take a look. https://github.com/apache/kafka/pull/9085 4. Another area would be to have the ability to define common structs in the auto-generated protocol. That would allow to share them between requests/responses. This is not strictly necessary but that would be nice to have. Overall, I believe that we could make it happen if we improve the auto-generated protocol. The substitution reads pretty well when the auto-generated data structure is a one-to-one replacement. This PR is a good example: https://github.com/apache/kafka/pull/9746. Assuming that we can it make feel more natural with the suggested improvement, do you believe that we should try to continue on trying to use the auto-generate requests and responses directly? ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
