Re: Kafka protocol ApiVersions request/response

2023-09-29 Thread Neeraj Vaidya
 I think I have found the answer to my question about how many ApiKey elements 
are in the response.
The array type for ApiKey is probably a COMPACT_ARRAY but the BNF grammar does 
not say so in the Documentation.

Is this a documentation defect/gap ?

ApiVersions Response (Version: 3) => error_code [api_keys] throttle_time_ms 
TAG_BUFFER 
 error_code => INT16
 api_keys => api_key min_version max_version TAG_BUFFER 
 api_key => INT16
 min_version => INT16
 max_version => INT16
 throttle_time_ms => INT32

Regards,
Neeraj On Friday, 29 September, 2023 at 01:04:07 pm GMT+10, Neeraj Vaidya 
 wrote:  
 
 Hi,

I am trying to create a library in Go, to use the Kafka protocol binary format. 
As a first step, I am trying the ApiVersions request.
The request is processed successfully by the Kafka broker and it responds with 
an ApiVersions response as well. And it conforms to the following spec as per 
the protocol :

ApiVersions Response (Version: 0) => error_code [api_keys] 
  error_code => INT16
  api_keys => api_key min_version max_version 
    api_key => INT16
    min_version => INT16
    max_version => INT16

When I look at the response, how do I know how many api_keys list items I need 
to process ?

I did manage to arrive at a way of doing this as below, but not sure if that is 
the way to do it :
If I look at the Wireshark capture, I can see that just before the first 
api_keys item in the response, there is a single byte/octet which contains a 
value of let's say 0x61. This evaluates to 97. 
Whereas I get 96 api_keys items in the response.
This is the case everytime I send an ApiVersions request to different versions 
of Kafka. 
Meaning that the value in this byte is always 1 more than the number of items 
returned.

Is it a fair understanding that this byte will always indicate a number one 
more than the number of ApiKey items returned in the response ?

Regards,
Neeraj
  

Kafka protocol ApiVersions request/response

2023-09-28 Thread Neeraj Vaidya
Hi,

I am trying to create a library in Go, to use the Kafka protocol binary format. 
As a first step, I am trying the ApiVersions request.
The request is processed successfully by the Kafka broker and it responds with 
an ApiVersions response as well. And it conforms to the following spec as per 
the protocol :

ApiVersions Response (Version: 0) => error_code [api_keys] 
  error_code => INT16
  api_keys => api_key min_version max_version 
api_key => INT16
min_version => INT16
max_version => INT16

When I look at the response, how do I know how many api_keys list items I need 
to process ?

I did manage to arrive at a way of doing this as below, but not sure if that is 
the way to do it :
If I look at the Wireshark capture, I can see that just before the first 
api_keys item in the response, there is a single byte/octet which contains a 
value of let's say 0x61. This evaluates to 97. 
Whereas I get 96 api_keys items in the response.
This is the case everytime I send an ApiVersions request to different versions 
of Kafka. 
Meaning that the value in this byte is always 1 more than the number of items 
returned.

Is it a fair understanding that this byte will always indicate a number one 
more than the number of ApiKey items returned in the response ?

Regards,
Neeraj