Re: [VOTE] KIP-799: Align behaviour for producer callbacks with documented behaviour

[Luke Chen]

Hi Séamus,

Thanks for the update.
Looks better now!

Thank you.
Luke

On Sat, Dec 4, 2021 at 12:57 AM Séamus Ó Ceanainn <
seamus.oceana...@zalando.ie> wrote:

> Hey Luke,
>
> Thanks for the feedback. I've updated the relevant section to hopefully
> make it more clear from the KIP itself what placeholder value would be
> returned.
>
> Regards,
> Séamus.
>
> On Tue, 30 Nov 2021 at 09:52, Luke Chen  wrote:
>
> > Hi Séamus,
> > Thanks for the KIP!
> > We definitely want to keep the producer callback consistent for all types
> > of errors.
> >
> > Just one comment for the KIP:
> > In the "Proposed Changes" section, could you please "explicitly" describe
> > what placeholder you'll return, in addition to adding a hyperlink to
> other
> > places, to make it clear.
> >
> > +1 (non-binding)
> >
> > Thank you.
> > Luke
> >
> > On Tue, Nov 30, 2021 at 1:17 PM John Roesler 
> wrote:
> >
> > > Thanks, Séamus!
> > >
> > > I'm +1 (binding).
> > >
> > > On Mon, 2021-11-29 at 16:14 +, Séamus Ó Ceanainn wrote:
> > > > Hi everyone,
> > > >
> > > > I'd like to start a vote for KIP-799: Align behaviour for producer
> > > > callbacks with documented behaviour
> > > > <
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-799%3A+Align+behaviour+for+producer+callbacks+with+documented+behaviour
> > > >
> > > > .
> > > >
> > > > The KIP proposes a breaking change in the behaviour of producer
> client
> > > > callbacks. The breaking change would align the behaviour of callbacks
> > > with
> > > > the documented behaviour for the method, and makes it consistent with
> > > > similar methods for producer client interceptors.
> > > >
> > > > Regards,
> > > > Séamus.
> > >
> > >
> >
>

Re: [VOTE] KIP-792: Add "generation" field into consumer protocol

[Luke Chen]

Hi Colin,
Thanks for your comment.

> How are we going to avoid the situation where the broker restarts, and
the same generation number is reused?

Actually, this KIP doesn't have anything to do with the brokers. The
"generation" field I added, is in the subscription metadata, which will not
be deserialized by brokers. The metadata is only deserialized by consumer
lead. And for the consumer lead, the only thing the lead cared about, is
the highest generation of the ownedPartitions among all the consumers. With
the highest generation of the ownedPartitions, the consumer lead can
distribute the partitions as sticky as possible, and most importantly,
without errors.

That is, after this KIP, if the broker restarts, and the same generation
number is reused, it won't break current rebalance behavior. But it'll help
the consumer lead do the sticky assignments correctly.

Thank you.
Luke

On Fri, Dec 3, 2021 at 6:30 AM Colin McCabe  wrote:

> How are we going to avoid the situation where the broker restarts, and the
> same generation number is reused?
>
> best,
> Colin
>
> On Tue, Nov 30, 2021, at 16:36, Luke Chen wrote:
> > Hi all,
> >
> > I'd like to start the vote for KIP-792: Add "generation" field into
> > consumer protocol.
> >
> > The goal of this KIP is to allow the assignor/consumer coordinator to
> have
> > a way to identify the out-of-date members/assignments, to avoid rebalance
> > stuck issues in current protocol.
> >
> > Detailed description can be found here:
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=191336614
> >
> > Any feedback is welcome.
> >
> > Thank you.
> > Luke
>

[jira] [Created] (KAFKA-13507) GlobalProcessor ignored user specified names

[Matthias J. Sax (Jira)]

Matthias J. Sax created KAFKA-13507:
---

 Summary: GlobalProcessor ignored user specified names
 Key: KAFKA-13507
 URL: https://issues.apache.org/jira/browse/KAFKA-13507
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Matthias J. Sax


Using `StreamsBuilder.addGlobalStore` users can specify a name via `Consumed` 
parameter. However, the specified name is ignored and the created source and 
global processor get generated names assigned.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Randall Hauch]

One more thing, inline below:

Randall

On Fri, Dec 3, 2021 at 10:38 AM Mickael Maison  wrote:
>
> Thanks Randall for the feedback!
>
> 1. Done
>
> 2. Yes that makes sense. I've updated the KIP
>
> 3.a I've updated to explicitly mention the plugin can be provided
> using the fully qualified class or the alias.
> 3.b Only concrete classes will be returned. I made the mistake when I
> crafted the examples by hand. Fixed

Can we also enumerate all of the values that can be included in the
"type" field in the response from the "GET
/connector-plugins?connectorsOnly=false"?

>
> 4. If a plugin does not override config(), the response is an empty
> array. I've updated the KIP
>
> 5. I have updated the compatibility section with more details
>
> 6. Can you clarify what you mean? KIP-507 was for securing internal
> REST endpoints.
>
> 7. I don't think it is necessary. I think the use case Chris mentioned
> could be useful for admins. I'm interested in exposing the
> connector-level plugins and their configs. I thought it was easy to
> add an API for worker plugins while working in this area but if there
> isn't a strong consensus, I'm happy to drop it.
>
> Mickael
>
>
>
>
>
>
> On Wed, Dec 1, 2021 at 5:39 PM Randall Hauch  wrote:
> >
> > Thanks for the KIP, Mickael. And thanks to everyone else for the good
> > discussion so far.
> >
> > Overall this is pretty good, but I do have a few questions/comments:
> > 1. Can we include the HTTP method/verb in each of the endpoints in the
> > "Public Interfaces" section?
> > 2. The sample output for `GET /connector-plugins?connectorsOnly=false`
> > includes a `version` field for the connector plugin, but not for the
> > transform or converter plugins. Neither the `Transformation` or
> > `Converter` interfaces have a version nor implement the `Versioned`
> > interface, so should we add the `Versioned` interface to the
> > `Transformation` and `Converter` interfaces and add a default method
> > that returns "unknown"? Yes, we could do that in a separate KIP, but I
> > think it's appropriate to do it in this KIP for a few reasons. First,
> > we're already modifying the `Converter` interface in this KIP and it
> > would be quite straightforward to also extend the `Versioned`
> > interface and add a default method; then doing that for
> > `Transformation` would also be relatively straightforward. Second,
> > this KIP provides the justification for adding a version to the
> > `Transformation` and `Converter` interfaces.
> > 3. In the "Public Interfaces" section, there are two inconsistencies
> > between the `GET /connector-plugins` and `GET
> > /connector-plugins//config` methods. Both will make it
> > difficult to use the second method, since the input to the second
> > method is not directly found in the response from the first method:
> >  a. The example for `GET /connector-plugins/Cast$Value/config` uses
> > short name, or what Connect often refers to as the alias, "Cast$Value"
> > for the plugin, rather than the fully-qualified name
> > "org.apache.kafka.connect.transforms.Cast$Value". The KIP should
> > specify whether it will support both the shortened name and the
> > fully-qualified name, or whether only one of these will be supported.
> > Not supporting the fully-qualified name will make it impossible to use
> > the response from the first method as an input to the second.
> >  a. The example for the list method returns the `Transformation` base
> > and abstract classes (e.g.,
> > "org.apache.kafka.connect.transforms.Cast") rather than the concrete
> > classes (e.g., "org.apache.kafka.connect.transforms.Cast$Value" and
> > "org.apache.kafka.connect.transforms.Cast$Key"), while the `GET
> > /connector-plugins//config` will require the concrete class.
> > 4. The "Public Interfaces" section's description of the `GET
> > /connector-plugins//config` method should probably specify the
> > expected behavior for `Converter` implementations that don't override
> > the new `config()` method.
> > 5. The "Compatibility, Deprecation, and Migration Plan" section should
> > describe whether older `Converter` implementations (compiled with
> > older versions of the Connect API without Converter extending
> > Configurable) will be able to be installed into newer Connect
> > runtimes, and what the behavior will be for the `GET
> > /connector-plugins//config` method. (Although my suggestion
> > for #5 will partially address the latter, we still need to talk about
> > it in terms of compatibility.)
> > 6. Can we add a small section addressing the security implications of
> > these new APIs? KIP-507 [1] applied to APIs that existed at the time,
> > and this KIP should at least mention how it aligns with that
> > mechanism.
> > 7. Do we really think adding `GET /worker-plugins` is necessary? While
> > I understand the reasoning, I'm less convinced that it's worth adding
> > to the public API. First, an administrator still has to modify the
> > worker config for all workers to take advantage 

[jira] [Created] (KAFKA-13506) IQv2: Transmit position to standbys

[John Roesler (Jira)]

John Roesler created KAFKA-13506:


 Summary: IQv2: Transmit position to standbys
 Key: KAFKA-13506
 URL: https://issues.apache.org/jira/browse/KAFKA-13506
 Project: Kafka
  Issue Type: Sub-task
Reporter: John Roesler






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Randall Hauch]

Thanks for the quick response, Mickael. Responses inline below.

Randall

On Fri, Dec 3, 2021 at 10:38 AM Mickael Maison  wrote:
>
> Thanks Randall for the feedback!
>
> 1. Done
>
> 2. Yes that makes sense. I've updated the KIP
>
> 3.a I've updated to explicitly mention the plugin can be provided
> using the fully qualified class or the alias.
> 3.b Only concrete classes will be returned. I made the mistake when I
> crafted the examples by hand. Fixed
>
> 4. If a plugin does not override config(), the response is an empty
> array. I've updated the KIP
>
> 5. I have updated the compatibility section with more details
>
> 6. Can you clarify what you mean? KIP-507 was for securing internal
> REST endpoints.

My bad. I referenced the wrong KIP. I was simply asking that this KIP
mention the security implication of adding new endpoints. IOW, if a
client can access the existing (non-internal) endpoints, they will be
able to access these new endpoints.

>
> 7. I don't think it is necessary. I think the use case Chris mentioned
> could be useful for admins. I'm interested in exposing the
> connector-level plugins and their configs. I thought it was easy to
> add an API for worker plugins while working in this area but if there
> isn't a strong consensus, I'm happy to drop it.

I see Chris has also responded, so I'll respond to that.

>
> Mickael
>
>
>
>
>
>
> On Wed, Dec 1, 2021 at 5:39 PM Randall Hauch  wrote:
> >
> > Thanks for the KIP, Mickael. And thanks to everyone else for the good
> > discussion so far.
> >
> > Overall this is pretty good, but I do have a few questions/comments:
> > 1. Can we include the HTTP method/verb in each of the endpoints in the
> > "Public Interfaces" section?
> > 2. The sample output for `GET /connector-plugins?connectorsOnly=false`
> > includes a `version` field for the connector plugin, but not for the
> > transform or converter plugins. Neither the `Transformation` or
> > `Converter` interfaces have a version nor implement the `Versioned`
> > interface, so should we add the `Versioned` interface to the
> > `Transformation` and `Converter` interfaces and add a default method
> > that returns "unknown"? Yes, we could do that in a separate KIP, but I
> > think it's appropriate to do it in this KIP for a few reasons. First,
> > we're already modifying the `Converter` interface in this KIP and it
> > would be quite straightforward to also extend the `Versioned`
> > interface and add a default method; then doing that for
> > `Transformation` would also be relatively straightforward. Second,
> > this KIP provides the justification for adding a version to the
> > `Transformation` and `Converter` interfaces.
> > 3. In the "Public Interfaces" section, there are two inconsistencies
> > between the `GET /connector-plugins` and `GET
> > /connector-plugins//config` methods. Both will make it
> > difficult to use the second method, since the input to the second
> > method is not directly found in the response from the first method:
> >  a. The example for `GET /connector-plugins/Cast$Value/config` uses
> > short name, or what Connect often refers to as the alias, "Cast$Value"
> > for the plugin, rather than the fully-qualified name
> > "org.apache.kafka.connect.transforms.Cast$Value". The KIP should
> > specify whether it will support both the shortened name and the
> > fully-qualified name, or whether only one of these will be supported.
> > Not supporting the fully-qualified name will make it impossible to use
> > the response from the first method as an input to the second.
> >  a. The example for the list method returns the `Transformation` base
> > and abstract classes (e.g.,
> > "org.apache.kafka.connect.transforms.Cast") rather than the concrete
> > classes (e.g., "org.apache.kafka.connect.transforms.Cast$Value" and
> > "org.apache.kafka.connect.transforms.Cast$Key"), while the `GET
> > /connector-plugins//config` will require the concrete class.
> > 4. The "Public Interfaces" section's description of the `GET
> > /connector-plugins//config` method should probably specify the
> > expected behavior for `Converter` implementations that don't override
> > the new `config()` method.
> > 5. The "Compatibility, Deprecation, and Migration Plan" section should
> > describe whether older `Converter` implementations (compiled with
> > older versions of the Connect API without Converter extending
> > Configurable) will be able to be installed into newer Connect
> > runtimes, and what the behavior will be for the `GET
> > /connector-plugins//config` method. (Although my suggestion
> > for #5 will partially address the latter, we still need to talk about
> > it in terms of compatibility.)
> > 6. Can we add a small section addressing the security implications of
> > these new APIs? KIP-507 [1] applied to APIs that existed at the time,
> > and this KIP should at least mention how it aligns with that
> > mechanism.
> > 7. Do we really think adding `GET /worker-plugins` is necessary? While
> > I 

[DISCUSS] KIP-807: Refactor KafkaStreams exposed metadata hierarchy

[Josep Prat]

Hi all,

I've created KIP-807 to refactor KafaStreams exposed metadata hierarchy.
Here you can find the document: https://cwiki.apache.org/confluence/x/TpiqCw

Please take a look at the proposal and let me know what you think,

Thanks in advance,

-- 

Josep Prat

*Aiven Deutschland GmbH*

Immanuelkirchstraße 26, 10405 Berlin

Amtsgericht Charlottenburg, HRB 209739 B

Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen

*m:* +491715557497

*w:* aiven.io

*e:* josep.p...@aiven.io

Re: [VOTE] KIP-799: Align behaviour for producer callbacks with documented behaviour

[Séamus Ó Ceanainn]

Hey Luke,

Thanks for the feedback. I've updated the relevant section to hopefully
make it more clear from the KIP itself what placeholder value would be
returned.

Regards,
Séamus.

On Tue, 30 Nov 2021 at 09:52, Luke Chen  wrote:

> Hi Séamus,
> Thanks for the KIP!
> We definitely want to keep the producer callback consistent for all types
> of errors.
>
> Just one comment for the KIP:
> In the "Proposed Changes" section, could you please "explicitly" describe
> what placeholder you'll return, in addition to adding a hyperlink to other
> places, to make it clear.
>
> +1 (non-binding)
>
> Thank you.
> Luke
>
> On Tue, Nov 30, 2021 at 1:17 PM John Roesler  wrote:
>
> > Thanks, Séamus!
> >
> > I'm +1 (binding).
> >
> > On Mon, 2021-11-29 at 16:14 +, Séamus Ó Ceanainn wrote:
> > > Hi everyone,
> > >
> > > I'd like to start a vote for KIP-799: Align behaviour for producer
> > > callbacks with documented behaviour
> > > <
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-799%3A+Align+behaviour+for+producer+callbacks+with+documented+behaviour
> > >
> > > .
> > >
> > > The KIP proposes a breaking change in the behaviour of producer client
> > > callbacks. The breaking change would align the behaviour of callbacks
> > with
> > > the documented behaviour for the method, and makes it consistent with
> > > similar methods for producer client interceptors.
> > >
> > > Regards,
> > > Séamus.
> >
> >
>

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Chris Egerton]

Hi Mickael and Randall,

I think there's a decent use case for exposing worker plugins. If a cluster
administrator wants to enable a worker plugin on their cluster, it may make
things easier if they can verify ahead of time that that plugin is
available on every worker in the cluster (and possibly that the same
version is present on every worker). The current alternative to this would
require sifting through worker logs.

I don't think the additional workload to expose worker plugins is
significant (it should largely mirror the same logic we'll use to expose
connector plugins), but if there's serious objection to including that
feature in this KIP, we could probably live without it now that we've made
sure that the current proposal doesn't block us from adding it in a
reasonable way later on down the road.

Cheers,

Chris

On Fri, Dec 3, 2021 at 11:38 AM Mickael Maison 
wrote:

> Thanks Randall for the feedback!
>
> 1. Done
>
> 2. Yes that makes sense. I've updated the KIP
>
> 3.a I've updated to explicitly mention the plugin can be provided
> using the fully qualified class or the alias.
> 3.b Only concrete classes will be returned. I made the mistake when I
> crafted the examples by hand. Fixed
>
> 4. If a plugin does not override config(), the response is an empty
> array. I've updated the KIP
>
> 5. I have updated the compatibility section with more details
>
> 6. Can you clarify what you mean? KIP-507 was for securing internal
> REST endpoints.
>
> 7. I don't think it is necessary. I think the use case Chris mentioned
> could be useful for admins. I'm interested in exposing the
> connector-level plugins and their configs. I thought it was easy to
> add an API for worker plugins while working in this area but if there
> isn't a strong consensus, I'm happy to drop it.
>
> Mickael
>
>
>
>
>
>
> On Wed, Dec 1, 2021 at 5:39 PM Randall Hauch  wrote:
> >
> > Thanks for the KIP, Mickael. And thanks to everyone else for the good
> > discussion so far.
> >
> > Overall this is pretty good, but I do have a few questions/comments:
> > 1. Can we include the HTTP method/verb in each of the endpoints in the
> > "Public Interfaces" section?
> > 2. The sample output for `GET /connector-plugins?connectorsOnly=false`
> > includes a `version` field for the connector plugin, but not for the
> > transform or converter plugins. Neither the `Transformation` or
> > `Converter` interfaces have a version nor implement the `Versioned`
> > interface, so should we add the `Versioned` interface to the
> > `Transformation` and `Converter` interfaces and add a default method
> > that returns "unknown"? Yes, we could do that in a separate KIP, but I
> > think it's appropriate to do it in this KIP for a few reasons. First,
> > we're already modifying the `Converter` interface in this KIP and it
> > would be quite straightforward to also extend the `Versioned`
> > interface and add a default method; then doing that for
> > `Transformation` would also be relatively straightforward. Second,
> > this KIP provides the justification for adding a version to the
> > `Transformation` and `Converter` interfaces.
> > 3. In the "Public Interfaces" section, there are two inconsistencies
> > between the `GET /connector-plugins` and `GET
> > /connector-plugins//config` methods. Both will make it
> > difficult to use the second method, since the input to the second
> > method is not directly found in the response from the first method:
> >  a. The example for `GET /connector-plugins/Cast$Value/config` uses
> > short name, or what Connect often refers to as the alias, "Cast$Value"
> > for the plugin, rather than the fully-qualified name
> > "org.apache.kafka.connect.transforms.Cast$Value". The KIP should
> > specify whether it will support both the shortened name and the
> > fully-qualified name, or whether only one of these will be supported.
> > Not supporting the fully-qualified name will make it impossible to use
> > the response from the first method as an input to the second.
> >  a. The example for the list method returns the `Transformation` base
> > and abstract classes (e.g.,
> > "org.apache.kafka.connect.transforms.Cast") rather than the concrete
> > classes (e.g., "org.apache.kafka.connect.transforms.Cast$Value" and
> > "org.apache.kafka.connect.transforms.Cast$Key"), while the `GET
> > /connector-plugins//config` will require the concrete class.
> > 4. The "Public Interfaces" section's description of the `GET
> > /connector-plugins//config` method should probably specify the
> > expected behavior for `Converter` implementations that don't override
> > the new `config()` method.
> > 5. The "Compatibility, Deprecation, and Migration Plan" section should
> > describe whether older `Converter` implementations (compiled with
> > older versions of the Connect API without Converter extending
> > Configurable) will be able to be installed into newer Connect
> > runtimes, and what the behavior will be for the `GET
> > /connector-plugins//config` 

Re: KIP-769: Connect API to retrieve connector configuration definitions

[Mickael Maison]

Thanks Randall for the feedback!

1. Done

2. Yes that makes sense. I've updated the KIP

3.a I've updated to explicitly mention the plugin can be provided
using the fully qualified class or the alias.
3.b Only concrete classes will be returned. I made the mistake when I
crafted the examples by hand. Fixed

4. If a plugin does not override config(), the response is an empty
array. I've updated the KIP

5. I have updated the compatibility section with more details

6. Can you clarify what you mean? KIP-507 was for securing internal
REST endpoints.

7. I don't think it is necessary. I think the use case Chris mentioned
could be useful for admins. I'm interested in exposing the
connector-level plugins and their configs. I thought it was easy to
add an API for worker plugins while working in this area but if there
isn't a strong consensus, I'm happy to drop it.

Mickael






On Wed, Dec 1, 2021 at 5:39 PM Randall Hauch  wrote:
>
> Thanks for the KIP, Mickael. And thanks to everyone else for the good
> discussion so far.
>
> Overall this is pretty good, but I do have a few questions/comments:
> 1. Can we include the HTTP method/verb in each of the endpoints in the
> "Public Interfaces" section?
> 2. The sample output for `GET /connector-plugins?connectorsOnly=false`
> includes a `version` field for the connector plugin, but not for the
> transform or converter plugins. Neither the `Transformation` or
> `Converter` interfaces have a version nor implement the `Versioned`
> interface, so should we add the `Versioned` interface to the
> `Transformation` and `Converter` interfaces and add a default method
> that returns "unknown"? Yes, we could do that in a separate KIP, but I
> think it's appropriate to do it in this KIP for a few reasons. First,
> we're already modifying the `Converter` interface in this KIP and it
> would be quite straightforward to also extend the `Versioned`
> interface and add a default method; then doing that for
> `Transformation` would also be relatively straightforward. Second,
> this KIP provides the justification for adding a version to the
> `Transformation` and `Converter` interfaces.
> 3. In the "Public Interfaces" section, there are two inconsistencies
> between the `GET /connector-plugins` and `GET
> /connector-plugins//config` methods. Both will make it
> difficult to use the second method, since the input to the second
> method is not directly found in the response from the first method:
>  a. The example for `GET /connector-plugins/Cast$Value/config` uses
> short name, or what Connect often refers to as the alias, "Cast$Value"
> for the plugin, rather than the fully-qualified name
> "org.apache.kafka.connect.transforms.Cast$Value". The KIP should
> specify whether it will support both the shortened name and the
> fully-qualified name, or whether only one of these will be supported.
> Not supporting the fully-qualified name will make it impossible to use
> the response from the first method as an input to the second.
>  a. The example for the list method returns the `Transformation` base
> and abstract classes (e.g.,
> "org.apache.kafka.connect.transforms.Cast") rather than the concrete
> classes (e.g., "org.apache.kafka.connect.transforms.Cast$Value" and
> "org.apache.kafka.connect.transforms.Cast$Key"), while the `GET
> /connector-plugins//config` will require the concrete class.
> 4. The "Public Interfaces" section's description of the `GET
> /connector-plugins//config` method should probably specify the
> expected behavior for `Converter` implementations that don't override
> the new `config()` method.
> 5. The "Compatibility, Deprecation, and Migration Plan" section should
> describe whether older `Converter` implementations (compiled with
> older versions of the Connect API without Converter extending
> Configurable) will be able to be installed into newer Connect
> runtimes, and what the behavior will be for the `GET
> /connector-plugins//config` method. (Although my suggestion
> for #5 will partially address the latter, we still need to talk about
> it in terms of compatibility.)
> 6. Can we add a small section addressing the security implications of
> these new APIs? KIP-507 [1] applied to APIs that existed at the time,
> and this KIP should at least mention how it aligns with that
> mechanism.
> 7. Do we really think adding `GET /worker-plugins` is necessary? While
> I understand the reasoning, I'm less convinced that it's worth adding
> to the public API. First, an administrator still has to modify the
> worker config for all workers to take advantage of that. Second,
> adding this method increases the surface area and therefore risk of
> additional attack vectors, despite not really being able to use the
> resulting information in any other part of the API.
>
> Best regards,
>
> Randall
>
> [1] 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints
>
>
> On Wed, Dec 1, 2021 at 7:52 AM Mickael Maison  
> wrote:
> >
> > 

Re: requesting contributor access

[Bill Bejeck]

You're all set now.

-Bill


On Fri, Dec 3, 2021 at 10:13 AM Andras Katona 
wrote:

> Hey folks,
>
> Could you grant me contributor access in wiki, please?
> wiki id: akatona
> (jira id: akatona is already okay, just I forgot to create wiki too back
> then)
>
> Thanks,
> Andras
>

requesting contributor access

[Andras Katona]

Hey folks,

Could you grant me contributor access in wiki, please?
wiki id: akatona
(jira id: akatona is already okay, just I forgot to create wiki too back
then)

Thanks,
Andras

[jira] [Created] (KAFKA-13505) Kafka Connect should respect Avro 1.10.X enum defaults spec

[Guus De Graeve (Jira)]

Guus De Graeve created KAFKA-13505:
--

 Summary: Kafka Connect should respect Avro 1.10.X enum defaults 
spec
 Key: KAFKA-13505
 URL: https://issues.apache.org/jira/browse/KAFKA-13505
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: Guus De Graeve


We are using Kafka Connect to pipe data from Kafka topics into +parquet files 
on+ S3. Our Kafka data is serialised using Avro (schema registry). We use the 
Amazon S3 Sink Connector for this.

Up until recently we would set "schema.compatibility" to "NONE" in our 
connectors, but this had the pain-full side-effect that during deploys of our 
application we got huge file explosions (lots of very small files in HDFS / 
S3). This happens because kafka connect will create a new file every time the 
schema id of a log changes compared to the previous log. During deploys of our 
applications (which can take up to 20 minutes) multiple logs of mixed schema 
ids are inevitable and given the huge amounts of logs file explosions of up to 
a million files weren't uncommon.

To solve this problem we switched all our connectors "schema.compatibility" to 
"BACKWARD", which should only create a new file when a higher schema id is 
detected and deserialise all logs with the latest known schema id. Which should 
only create one new file during deploys.

An example connector config:
{code:java}
{
"name": "hdfs-Project_Test_Subject",
"config": {
"connector.class": "io.confluent.connect.hdfs.HdfsSinkConnector",
"partition.duration.ms": "8640",
"topics.dir": "/user/kafka/Project",
"hadoop.conf.dir": "/opt/hadoop/conf",
"flush.size": "100",
"schema.compatibility": "BACKWARD",
"topics": "Project_Test_Subject",
"timezone": "UTC",
"hdfs.url": "hdfs://hadoophost:9000",
"value.converter.value.subject.name.strategy": 
"io.confluent.kafka.serializers.subject.TopicNameStrategy",
"rotate.interval.ms": "720",
"locale": "C",
"hadoop.home": "/opt/hadoop",
"logs.dir": "/user/kafka/_logs",
"format.class": "io.confluent.connect.hdfs.parquet.ParquetFormat",
"partitioner.class": 
"io.confluent.connect.storage.partitioner.TimeBasedPartitioner",
"name": "hdfs-Project_Test_Subject",
"errors.tolerance": "all",
"storage.class": "io.confluent.connect.hdfs.storage.HdfsStorage",
"path.format": "/MM/dd"
}
}{code}
However, we have lots of enum fields in our data records (avro schemas) to 
which subjects get added frequently, and this is causing issues with our Kafka 
Connect connectors FAILING with these kinds of errors:
{code:java}
Schema parameters not equal. source parameters: 
{io.confluent.connect.avro.enum.default.testfield=null, 
io.confluent.connect.avro.Enum=Ablo.testfield, 
io.confluent.connect.avro.Enum.null=null, 
io.confluent.connect.avro.Enum.value1=value1, 
io.confluent.connect.avro.Enum.value2=value2} and target parameters: 
{io.confluent.connect.avro.enum.default.testfield=null, 
io.confluent.connect.avro.Enum=Ablo.testfield, 
io.confluent.connect.avro.Enum.null=null, 
io.confluent.connect.avro.Enum.value1=value1, 
io.confluent.connect.avro.Enum.value2=value2, 
io.confluent.connect.avro.Enum.value3=value3}{code}
Since Avro 1.10.X specification, enum values support defaults, which makes 
schema evolution possible even when adding subjects (values) to an enum. When 
testing our schemas for compatibility using the Schema Registry api we always 
get "is_compatible" => true. So schema evolution should in theory not be a 
problem.

The error above is thrown in the `SchemaProjector` class which is part of Kafka 
Connect, more specifically in the function `checkMaybeCompatible()`. It seems 
like this function is not respecting the Avro 1.10.X specification for enum 
schema evolution, and I'm not sure if it is meant to respect it? As we 
currently don't have any other routes to fix this issue and returning to the 
"NONE" schema compatibility is no options considering the file explosions, 
we're kinda stuck here.

This issue was discussed more in detail on the Confluent forum in this thread:

[https://forum.confluent.io/t/should-will-kafka-connect-support-schema-evolution-using-avro-1-10-x-enum-defaults/3076/8]

Adem from Confluent is quite confident this is a bug and asked me to file a bug 
report here.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (KAFKA-13504) Retry connect internal topics' creation in case of InvalidReplicationFactorException

[Andras Katona (Jira)]

Andras Katona created KAFKA-13504:
-

 Summary: Retry connect internal topics' creation in case of 
InvalidReplicationFactorException
 Key: KAFKA-13504
 URL: https://issues.apache.org/jira/browse/KAFKA-13504
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Reporter: Andras Katona






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] Apache Kafka 3.1.0 release

[David Jacot]

Hi Rajini,

Interesting bug. The patch seems to be low risk so I suppose that
it is fine to keep it in 3.1.0.

Thanks,
David

On Fri, Dec 3, 2021 at 2:26 PM David Jacot  wrote:
>
> Hi Colin,
>
> Thanks for the heads up. It makes sense to include it in order
> to keep the KRaft inline with ZK behavior.
>
> Thanks,
> David
>
> On Fri, Dec 3, 2021 at 9:44 AM Rajini Sivaram  wrote:
> >
> > Hi David,
> >
> > Sorry, I had completely forgotten about code freeze and merged
> > https://issues.apache.org/jira/browse/KAFKA-13461 to 3.1 branch yesterday.
> > Can you take a look and see if we want it in 3.1.0? It is not a regression
> > in 3.1, but we see this issue in tests and when it happens, the controller
> > no longer operates as a controller.
> >
> > Thank you,
> >
> > Rajini
> >
> > On Thu, Dec 2, 2021 at 10:56 PM Colin McCabe  wrote:
> >
> > > Hi David,
> > >
> > > We'd like to include "KAFKA-13490: Fix createTopics and
> > > incrementalAlterConfigs for KRaft mode #11416" in the upcoming release.
> > > This fixes some bugs in how createTopics and incrementalAlterConfigs are
> > > handled by the controller. It is specific to KRaft, so will not affect ZK
> > > mode.
> > >
> > > best,
> > > Colin
> > >
> > > On Wed, Nov 24, 2021, at 01:20, David Jacot wrote:
> > > > Hi Mickael,
> > > >
> > > > Thanks for reporting it. It makes sense to include it in the 3.1 release
> > > > as well as it is a regression.
> > > >
> > > > Thanks,
> > > > David
> > > >
> > > > On Tue, Nov 23, 2021 at 6:52 PM Mickael Maison 
> > > > 
> > > wrote:
> > > >>
> > > >> Hi David,
> > > >>
> > > >> Can we also consider https://issues.apache.org/jira/browse/KAFKA-13397?
> > > >> It's essentially a regression but in a very specific case. To hit it,
> > > >> you must be running MirrorMaker in dedicated mode and have changed the
> > > >> separator of the default replication policy.
> > > >>
> > > >> Thanks,
> > > >> Mickael
> > > >>
> > > >> On Tue, Nov 23, 2021 at 4:58 PM David Jacot 
> > > >> 
> > > wrote:
> > > >> >
> > > >> > Hi Ron,
> > > >> >
> > > >> > Thank you for reaching out about this. While this is clearly not a
> > > >> > regression, I agree with including it in 3.1 in order to have proper
> > > >> > and correct configuration constraints for KRaft. You can proceed.
> > > >> >
> > > >> > Cheers,
> > > >> > David
> > > >> >
> > > >> > On Tue, Nov 23, 2021 at 2:55 PM Ron Dagostino 
> > > wrote:
> > > >> > >
> > > >> > > Hi David.  I would like to nominate
> > > >> > > https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13456
> > > >> > > "Tighten KRaft config checks/constraints" as a 3.1.0 blocker.  The
> > > >> > > existing configuration constraints/checks related to KRaft 
> > > >> > > currently
> > > >> > > do not eliminate certain illegal configuration combinations. The
> > > >> > > illegal combinations do not cause harm at the moment, but we would
> > > >> > > like to implement constraints in 3.1.0 to catch them while KRaft is
> > > >> > > still in Preview.  We could add these additional checks later in
> > > 3.2.x
> > > >> > > instead, but we would like to add these as early as possible: we
> > > >> > > expect more people to begin trying KRaft with each subsequent
> > > release,
> > > >> > > and it would be best to eliminate as quickly as we can the
> > > possibility
> > > >> > > of people using configurations that would need fixing later.
> > > >> > >
> > > >> > > A patch is available at https://github.com/apache/kafka/pull/11503/
> > > .
> > > >> > >
> > > >> > > Ron
> > > >> > >
> > > >> > >
> > > >> > > On Tue, Nov 23, 2021 at 3:19 AM David Jacot
> > >  wrote:
> > > >> > > >
> > > >> > > > Hi Chris,
> > > >> > > >
> > > >> > > > Thanks for reporting both issues. As both are regressions, I do
> > > agree that
> > > >> > > > they are blockers and that we would fix them for 3.1.
> > > >> > > >
> > > >> > > > Cheers,
> > > >> > > > David
> > > >> > > >
> > > >> > > > On Mon, Nov 22, 2021 at 10:50 PM Chris Egerton
> > > >> > > >  wrote:
> > > >> > > > >
> > > >> > > > > Hi David,
> > > >> > > > >
> > > >> > > > > I have another blocker to propose. KAFKA-13472 (
> > > >> > > > > https://issues.apache.org/jira/browse/KAFKA-13472) is another
> > > regression in
> > > >> > > > > Connect caused by recently-merged changes for KAFKA-12487 (
> > > >> > > > > https://issues.apache.org/jira/browse/KAFKA-12487) which can
> > > lead to data
> > > >> > > > > loss in sink connectors in some rare edge cases. I've opened a
> > > fix PR (
> > > >> > > > > https://github.com/apache/kafka/pull/11526) already, and have
> > > also opened a
> > > >> > > > > fix PR (https://github.com/apache/kafka/pull/11524) for the
> > > aforementioned
> > > >> > > > > KAFKA-13469.
> > > >> > > > >
> > > >> > > > > Please let me know if we can merge a fix for this in time for
> > > the 3.1.0
> > > >> > > > > release; if not, as with KAFKA-13469, we may want to revert the
> > > changes for
> > > >> > > > > the PR that cause this issue (in this case, that'd be 

Re: [DISCUSS] Apache Kafka 3.1.0 release

[David Jacot]

Hi Colin,

Thanks for the heads up. It makes sense to include it in order
to keep the KRaft inline with ZK behavior.

Thanks,
David

On Fri, Dec 3, 2021 at 9:44 AM Rajini Sivaram  wrote:
>
> Hi David,
>
> Sorry, I had completely forgotten about code freeze and merged
> https://issues.apache.org/jira/browse/KAFKA-13461 to 3.1 branch yesterday.
> Can you take a look and see if we want it in 3.1.0? It is not a regression
> in 3.1, but we see this issue in tests and when it happens, the controller
> no longer operates as a controller.
>
> Thank you,
>
> Rajini
>
> On Thu, Dec 2, 2021 at 10:56 PM Colin McCabe  wrote:
>
> > Hi David,
> >
> > We'd like to include "KAFKA-13490: Fix createTopics and
> > incrementalAlterConfigs for KRaft mode #11416" in the upcoming release.
> > This fixes some bugs in how createTopics and incrementalAlterConfigs are
> > handled by the controller. It is specific to KRaft, so will not affect ZK
> > mode.
> >
> > best,
> > Colin
> >
> > On Wed, Nov 24, 2021, at 01:20, David Jacot wrote:
> > > Hi Mickael,
> > >
> > > Thanks for reporting it. It makes sense to include it in the 3.1 release
> > > as well as it is a regression.
> > >
> > > Thanks,
> > > David
> > >
> > > On Tue, Nov 23, 2021 at 6:52 PM Mickael Maison 
> > wrote:
> > >>
> > >> Hi David,
> > >>
> > >> Can we also consider https://issues.apache.org/jira/browse/KAFKA-13397?
> > >> It's essentially a regression but in a very specific case. To hit it,
> > >> you must be running MirrorMaker in dedicated mode and have changed the
> > >> separator of the default replication policy.
> > >>
> > >> Thanks,
> > >> Mickael
> > >>
> > >> On Tue, Nov 23, 2021 at 4:58 PM David Jacot 
> > wrote:
> > >> >
> > >> > Hi Ron,
> > >> >
> > >> > Thank you for reaching out about this. While this is clearly not a
> > >> > regression, I agree with including it in 3.1 in order to have proper
> > >> > and correct configuration constraints for KRaft. You can proceed.
> > >> >
> > >> > Cheers,
> > >> > David
> > >> >
> > >> > On Tue, Nov 23, 2021 at 2:55 PM Ron Dagostino 
> > wrote:
> > >> > >
> > >> > > Hi David.  I would like to nominate
> > >> > > https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13456
> > >> > > "Tighten KRaft config checks/constraints" as a 3.1.0 blocker.  The
> > >> > > existing configuration constraints/checks related to KRaft currently
> > >> > > do not eliminate certain illegal configuration combinations. The
> > >> > > illegal combinations do not cause harm at the moment, but we would
> > >> > > like to implement constraints in 3.1.0 to catch them while KRaft is
> > >> > > still in Preview.  We could add these additional checks later in
> > 3.2.x
> > >> > > instead, but we would like to add these as early as possible: we
> > >> > > expect more people to begin trying KRaft with each subsequent
> > release,
> > >> > > and it would be best to eliminate as quickly as we can the
> > possibility
> > >> > > of people using configurations that would need fixing later.
> > >> > >
> > >> > > A patch is available at https://github.com/apache/kafka/pull/11503/
> > .
> > >> > >
> > >> > > Ron
> > >> > >
> > >> > >
> > >> > > On Tue, Nov 23, 2021 at 3:19 AM David Jacot
> >  wrote:
> > >> > > >
> > >> > > > Hi Chris,
> > >> > > >
> > >> > > > Thanks for reporting both issues. As both are regressions, I do
> > agree that
> > >> > > > they are blockers and that we would fix them for 3.1.
> > >> > > >
> > >> > > > Cheers,
> > >> > > > David
> > >> > > >
> > >> > > > On Mon, Nov 22, 2021 at 10:50 PM Chris Egerton
> > >> > > >  wrote:
> > >> > > > >
> > >> > > > > Hi David,
> > >> > > > >
> > >> > > > > I have another blocker to propose. KAFKA-13472 (
> > >> > > > > https://issues.apache.org/jira/browse/KAFKA-13472) is another
> > regression in
> > >> > > > > Connect caused by recently-merged changes for KAFKA-12487 (
> > >> > > > > https://issues.apache.org/jira/browse/KAFKA-12487) which can
> > lead to data
> > >> > > > > loss in sink connectors in some rare edge cases. I've opened a
> > fix PR (
> > >> > > > > https://github.com/apache/kafka/pull/11526) already, and have
> > also opened a
> > >> > > > > fix PR (https://github.com/apache/kafka/pull/11524) for the
> > aforementioned
> > >> > > > > KAFKA-13469.
> > >> > > > >
> > >> > > > > Please let me know if we can merge a fix for this in time for
> > the 3.1.0
> > >> > > > > release; if not, as with KAFKA-13469, we may want to revert the
> > changes for
> > >> > > > > the PR that cause this issue (in this case, that'd be the PR for
> > >> > > > > KAFKA-12487).
> > >> > > > >
> > >> > > > > Cheers,
> > >> > > > >
> > >> > > > > Chris
> > >> > > > >
> > >> > > > > On Mon, Nov 22, 2021 at 11:42 AM Chris Egerton <
> > chr...@confluent.io> wrote:
> > >> > > > >
> > >> > > > > > Hi David,
> > >> > > > > >
> > >> > > > > > I'd like to propose KAFKA-13469 (
> > >> > > > > > https://issues.apache.org/jira/browse/KAFKA-13469) as a
> > blocker. It is a
> > >> > > > > > regression in Connect 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.1 #29

[Apache Jenkins Server]

See 

Re: [DISCUSS] Apache Kafka 3.1.0 release

[Rajini Sivaram]

Hi David,

Sorry, I had completely forgotten about code freeze and merged
https://issues.apache.org/jira/browse/KAFKA-13461 to 3.1 branch yesterday.
Can you take a look and see if we want it in 3.1.0? It is not a regression
in 3.1, but we see this issue in tests and when it happens, the controller
no longer operates as a controller.

Thank you,

Rajini

On Thu, Dec 2, 2021 at 10:56 PM Colin McCabe  wrote:

> Hi David,
>
> We'd like to include "KAFKA-13490: Fix createTopics and
> incrementalAlterConfigs for KRaft mode #11416" in the upcoming release.
> This fixes some bugs in how createTopics and incrementalAlterConfigs are
> handled by the controller. It is specific to KRaft, so will not affect ZK
> mode.
>
> best,
> Colin
>
> On Wed, Nov 24, 2021, at 01:20, David Jacot wrote:
> > Hi Mickael,
> >
> > Thanks for reporting it. It makes sense to include it in the 3.1 release
> > as well as it is a regression.
> >
> > Thanks,
> > David
> >
> > On Tue, Nov 23, 2021 at 6:52 PM Mickael Maison 
> wrote:
> >>
> >> Hi David,
> >>
> >> Can we also consider https://issues.apache.org/jira/browse/KAFKA-13397?
> >> It's essentially a regression but in a very specific case. To hit it,
> >> you must be running MirrorMaker in dedicated mode and have changed the
> >> separator of the default replication policy.
> >>
> >> Thanks,
> >> Mickael
> >>
> >> On Tue, Nov 23, 2021 at 4:58 PM David Jacot 
> wrote:
> >> >
> >> > Hi Ron,
> >> >
> >> > Thank you for reaching out about this. While this is clearly not a
> >> > regression, I agree with including it in 3.1 in order to have proper
> >> > and correct configuration constraints for KRaft. You can proceed.
> >> >
> >> > Cheers,
> >> > David
> >> >
> >> > On Tue, Nov 23, 2021 at 2:55 PM Ron Dagostino 
> wrote:
> >> > >
> >> > > Hi David.  I would like to nominate
> >> > > https://issues.apache.org/jira/projects/KAFKA/issues/KAFKA-13456
> >> > > "Tighten KRaft config checks/constraints" as a 3.1.0 blocker.  The
> >> > > existing configuration constraints/checks related to KRaft currently
> >> > > do not eliminate certain illegal configuration combinations. The
> >> > > illegal combinations do not cause harm at the moment, but we would
> >> > > like to implement constraints in 3.1.0 to catch them while KRaft is
> >> > > still in Preview.  We could add these additional checks later in
> 3.2.x
> >> > > instead, but we would like to add these as early as possible: we
> >> > > expect more people to begin trying KRaft with each subsequent
> release,
> >> > > and it would be best to eliminate as quickly as we can the
> possibility
> >> > > of people using configurations that would need fixing later.
> >> > >
> >> > > A patch is available at https://github.com/apache/kafka/pull/11503/
> .
> >> > >
> >> > > Ron
> >> > >
> >> > >
> >> > > On Tue, Nov 23, 2021 at 3:19 AM David Jacot
>  wrote:
> >> > > >
> >> > > > Hi Chris,
> >> > > >
> >> > > > Thanks for reporting both issues. As both are regressions, I do
> agree that
> >> > > > they are blockers and that we would fix them for 3.1.
> >> > > >
> >> > > > Cheers,
> >> > > > David
> >> > > >
> >> > > > On Mon, Nov 22, 2021 at 10:50 PM Chris Egerton
> >> > > >  wrote:
> >> > > > >
> >> > > > > Hi David,
> >> > > > >
> >> > > > > I have another blocker to propose. KAFKA-13472 (
> >> > > > > https://issues.apache.org/jira/browse/KAFKA-13472) is another
> regression in
> >> > > > > Connect caused by recently-merged changes for KAFKA-12487 (
> >> > > > > https://issues.apache.org/jira/browse/KAFKA-12487) which can
> lead to data
> >> > > > > loss in sink connectors in some rare edge cases. I've opened a
> fix PR (
> >> > > > > https://github.com/apache/kafka/pull/11526) already, and have
> also opened a
> >> > > > > fix PR (https://github.com/apache/kafka/pull/11524) for the
> aforementioned
> >> > > > > KAFKA-13469.
> >> > > > >
> >> > > > > Please let me know if we can merge a fix for this in time for
> the 3.1.0
> >> > > > > release; if not, as with KAFKA-13469, we may want to revert the
> changes for
> >> > > > > the PR that cause this issue (in this case, that'd be the PR for
> >> > > > > KAFKA-12487).
> >> > > > >
> >> > > > > Cheers,
> >> > > > >
> >> > > > > Chris
> >> > > > >
> >> > > > > On Mon, Nov 22, 2021 at 11:42 AM Chris Egerton <
> chr...@confluent.io> wrote:
> >> > > > >
> >> > > > > > Hi David,
> >> > > > > >
> >> > > > > > I'd like to propose KAFKA-13469 (
> >> > > > > > https://issues.apache.org/jira/browse/KAFKA-13469) as a
> blocker. It is a
> >> > > > > > regression in Connect caused by recently-merged changes for
> KAFKA-12226 (
> >> > > > > > https://issues.apache.org/jira/browse/KAFKA-12226) which
> leads to
> >> > > > > > duplicate records for source tasks. I plan to have a fix PR
> opened by the
> >> > > > > > end of the day.
> >> > > > > >
> >> > > > > > Please let me know if we can merge a fix for this in time for
> the 3.1.0
> >> > > > > > release; if not, we may want to revert the changes for
> KAFKA-12226.
> >> >