Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins

2024-05-22 Thread Chris Egerton
very contrived scenario, but I think similar arguments are > used for attributing validation errors to endpoints/urls/hostnames > when the associated credentials are unable to log into the remote > system. Did the user provide the correct testing credentials, but > accidentally typed the p

Re: [VOTE] KIP-1040: Improve handling of nullable values in InsertField, ExtractField, and other transformations

2024-05-20 Thread Chris Egerton
Thanks for the KIP! +1 (binding) On Mon, May 20, 2024 at 4:22 AM Mario Fiore Vitale wrote: > Hi everyone, > > I'd like to call a vote on KIP-1040 which aims to improve handling of > nullable values in InsertField, ExtractField, and other transformations > > KIP - >

[jira] [Resolved] (KAFKA-16603) Data loss when kafka connect sending data to Kafka

2024-05-20 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-16603. --- Resolution: Not A Bug > Data loss when kafka connect sending data to Ka

[jira] [Resolved] (KAFKA-16656) Using a custom replication.policy.separator with DefaultReplicationPolicy

2024-05-20 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-16656. --- Resolution: Not A Bug > Using a custom replication.policy.separa

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-05-15 Thread Chris Egerton
eption for that. > > Ismael > > On Wed, May 15, 2024 at 9:20 AM Chris Egerton > wrote: > > > FWIW I think that the low blast radius for KIP-1028 should allow it to > > proceed without adhering to the usual KIP and feature freeze dates. Code > > freeze is probably wo

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-05-15 Thread Chris Egerton
ide any additional information or > updates > > >> regarding KIPs that target this release version (3.8). > > >> If you have authored any KIPs that have an inaccurate status in the > > list, > > >> or are not in the list and should be, or are in the li

Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins

2024-05-15 Thread Chris Egerton
gt; contains a hard requirement, select the latest installed version which > satisfies the requirement." and "This configuration is re-evaluated > each time the connector or task are assigned to a new worker". I would > call this "eager" upgrade behavior, rather tha

Re: [VOTE] KIP-1028: Docker Official Image for Apache Kafka

2024-05-14 Thread Chris Egerton
+1 (binding), thanks for the KIP! On Tue, May 14, 2024, 12:13 Vedarth Sharma wrote: > Hi everyone, > > I'd like to call a vote on KIP-1028 which aims to introduce a JVM based > Docker Official Image (DOI) for Apache Kafka. > > KIP - > >

Re: [DISCUSS] KIP-891: Running multiple versions of Connector plugins

2024-05-14 Thread Chris Egerton
Hi all, Thanks Greg for updating the KIP, and thanks Snehashis for starting the work on this originally. The motivation section makes a pretty convincing case for this kind of feature. My thoughts are mostly about specific details: 1) I like the support for version ranges (the example

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

2024-05-14 Thread Chris Egerton
t; during discussions. I am happy to have a separate email thread on it to > iron things out. > > I hope this addresses all of your concerns! > > Thanks and regards, > Vedarth > > On Mon, May 13, 2024 at 10:55 PM Chris Egerton > wrote: > > > Thanks both for your responses

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

2024-05-14 Thread Chris Egerton
SWALLOW, there will be no retry, and if the > handler > > > instructs to RETRY, that will be the default behavior, which follows > the > > > values in already existing config parameters such as `max.block.ms`. > > Does > > > that make sense? > > > >

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

2024-05-13 Thread Chris Egerton
th images within the same community facilitates ease > of > >management and fosters a singular source of truth. > >- While Apache may not retain ownership of the hosted Docker Official > >Image, we are, in essence, providing Docker with a foundation that > > a

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

2024-05-13 Thread Chris Egerton
y 13, 2024 at 8:40 AM Chris Egerton > wrote: > > > Hi Alieh, > > > > Thanks for the updates! I just have a few more thoughts: > > > > - I don't think a boolean property is sufficient to dictate retries for > > unknown topic partitions, though. These errors

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

2024-05-13 Thread Chris Egerton
n Wed, May 8, 2024 at 9:47 AM Justine Olshan > >>> > >> wrote: > >> > >>> My concern with respect to it being fragile: the code that ensures the > >>> error type is internal to the producer. Someone may see it and say, I > >> want > >&

Re: KIP-1040: Improve handling of nullable values in InsertField/ExtractField transformations

2024-05-10 Thread Chris Egerton
 Done On Fri, May 10, 2024, 10:55 Mario Fiore Vitale wrote: > Thanks a lot! I have just a minor comment, should we also update the title > to be more generic since now it's also related to other SMTs? > > On Fri, May 10, 2024 at 4:44 PM Chris Egerton > wrote: > > >

Re: KIP-1040: Improve handling of nullable values in InsertField/ExtractField transformations

2024-05-10 Thread Chris Egerton
I've finished updating the KIP; @Mario, please let me know what you think. On Fri, May 10, 2024 at 10:26 AM Chris Egerton wrote: > I can do it :) > > On Fri, May 10, 2024 at 10:02 AM Mario Fiore Vitale > wrote: > >> Yes, I agree. Unfortunately due to the issue of

Re: KIP-1040: Improve handling of nullable values in InsertField/ExtractField transformations

2024-05-10 Thread Chris Egerton
alternatives? > > Thanks, > Mario. > > On Fri, May 10, 2024 at 3:40 PM Chris Egerton > wrote: > > > Yes, I think we should just do one KIP for all the SMTs. You don't have > to > > implement everything all at once or by yourself, but I don't see why we > > should

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

2024-05-10 Thread Chris Egerton
ant to introduce another image, since it > >> adds > >> > to the cognitive burden of people who will inevitably have to answer > the > >> > question of "What are the differences between all of the available > >> images > >> > and

Re: KIP-1040: Improve handling of nullable values in InsertField/ExtractField transformations

2024-05-10 Thread Chris Egerton
Fiore Vitale wrote: > Hi Chris, > > Thanks for the survey. Do you think I need to update the KIP to put all of > these? > > On Thu, May 9, 2024 at 4:14 PM Chris Egerton > wrote: > > > After doing a brief survey of the SMTs that ship with Connect, it seems >

Re: KIP-1040: Improve handling of nullable values in InsertField/ExtractField transformations

2024-05-09 Thread Chris Egerton
On Thu, May 9, 2024 at 3:28 AM Mario Fiore Vitale wrote: > Hi Chris, > > > Wouldn't ValueToKey [1] be applicable as well, for example? > Yes, also that one can be affected. > > On Wed, May 8, 2024 at 5:59 PM Chris Egerton > wrote: > > > Wait, just one mor

Re: KIP-1040: Improve handling of nullable values in InsertField/ExtractField transformations

2024-05-08 Thread Chris Egerton
/ValueToKey.java#L106 On Wed, May 8, 2024 at 11:46 AM Chris Egerton wrote: > Hi Mario, > > I think we could have something like `copy` and `copyWithoutDefaults` to > get around that, but now that you bring up compatibility, I think it's best > to hold off on this. I'm forced to recal

Re: KIP-1040: Improve handling of nullable values in InsertField/ExtractField transformations

2024-05-08 Thread Chris Egerton
ult values. If we replace the "copy" > logic with a method in the Struct we remove this behavior. > > Isn't it? > > Mario. > > On Wed, May 8, 2024 at 2:14 PM Chris Egerton > wrote: > > > Hi Mario, > > > > Thanks for the KIP! Looks good overall. One quic

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

2024-05-08 Thread Chris Egerton
may implement handleRetriable() for > >> RecordTooLargeException > >>> and define SWALLOW for the exception, but in practice, he sees no > change > >> in > >>> default behaviour since he implemented the wrong method. I think we can > >>> n

[jira] [Resolved] (KAFKA-16108) Backport fix for KAFKA-16093 to 3.7

2024-05-08 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-16108. --- Resolution: Done > Backport fix for KAFKA-16093 to

Re: KIP-1040: Improve handling of nullable values in InsertField/ExtractField transformations

2024-05-08 Thread Chris Egerton
Hi Mario, Thanks for the KIP! Looks good overall. One quick thought--it wasn't immediately obvious to me why we had to touch on InsertField for this. It looks like the reason is that we use Struct::get [1] to create a clone of the input value [2], instead of Struct::getWithoutDefault [3]. It

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

2024-05-07 Thread Chris Egerton
> >>>>> but we added similar handlers to Kafka Streams, and have good > > > > >> experience > > > > >>>>> with it. Hence, I understand, but don't share the concern > raised. > > > > >>>>> > > > >

[jira] [Resolved] (KAFKA-15018) Potential tombstone offsets corruption for exactly-once source connectors

2024-05-07 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-15018. --- Fix Version/s: 3.8.0 Resolution: Fixed > Potential tombstone offsets corrupt

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

2024-05-07 Thread Chris Egerton
;>>> always do follow up KIP to also apply the handler to other errors > to > > >>>>> expand the scope of the handler. The KIP does mention examples, but > > it > > >>>>> might be good to explicitly state for what cases the handler gets >

[jira] [Resolved] (KAFKA-13329) Connect does not perform preflight validation for per-connector key and value converters

2024-05-07 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13329?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-13329. --- Fix Version/s: 3.8.0 Resolution: Fixed > Connect does not perform prefli

[jira] [Resolved] (KAFKA-13328) Connect does not perform preflight validation for per-connector header converters

2024-05-07 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-13328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-13328. --- Fix Version/s: 3.8.0 Resolution: Fixed > Connect does not perform prefli

Re: lambda not returning?

2024-04-30 Thread Chris Egerton
Hi John, Do you have logs demonstrating that the connector was able to start up successfully? If not, it's possible that instead of the callback being dropped, it was properly retained but never invoked because the connector was blocked. Cheers, Chris On Tue, Apr 30, 2024 at 1:57 PM John

Re: [DISCUSS] KIP-1028: Docker Official Image for Apache Kafka

2024-04-30 Thread Chris Egerton
Hi Vedarth and Krish, Thanks for the KIP! I have to admit I'm a little skeptical; hopefully you can help me understand the need for these additional images. 1) In the motivation section it's stated that "Several other Apache projects, like Flink, Spark, Solr, have already released Docker

Re: [DISCUSS] KIP-1039: Disable automatic topic creation for MirrorMaker2 consumers

2024-04-30 Thread Chris Egerton
Hi Aaron, Thanks for publishing this KIP after the feedback on your PR. I'm generally in favor but have a few questions: 1) Is the consumer mentioned in the KIP the one constructed by the MirrorSourceConnector for polling replicated records from the source cluster? If yes, are there any other

Re: [DISCUSS] KIP-1038: Add Custom Error Handler to Producer

2024-04-17 Thread Chris Egerton
Hi Alieh, Thanks for the KIP! The issue with writing to non-existent topics is particularly frustrating for users of Kafka Connect and has been the source of a handful of Jira tickets over the years. My thoughts: 1. An additional detail we can add to the motivation (or possibly rejected

Re: [VOTE] KIP-899: Allow producer and consumer clients to rebootstrap

2024-04-15 Thread Chris Egerton
Hi Ivan, Thanks for the KIP. After the recent changes, this LGTM. +1 (binding) Cheers, Chris On Wed, Aug 2, 2023 at 12:15 AM Ivan Yurchenko wrote: > Hello, > > The discussion [1] for KIP-899 [2] has been open for quite some time. I'd > like to put the KIP up for a vote. > > Best, > Ivan > >

[ANNOUNCE] New Kafka PMC Member: Greg Harris

2024-04-13 Thread Chris Egerton
Hi all, Greg Harris has been a Kafka committer since July 2023. He has remained very active and instructive in the community since becoming a committer. It's my pleasure to announce that Greg is now a member of Kafka PMC. Congratulations, Greg! Chris, on behalf of the Apache Kafka PMC

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

2024-04-12 Thread Chris Egerton
ki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=240881396=14=12 > > On Thu, Apr 11, 2024, at 10:49, Chris Egerton wrote: > > Hi Ivan, > > > > I agree with Andrew that we can save cluster ID checking for later. This > > feature is opt-in and if

Re: [VOTE] KIP-1031: Control offset translation in MirrorSourceConnector

2024-04-12 Thread Chris Egerton
+1 (binding), thanks Omnia! On Fri, Apr 12, 2024, 03:46 Mickael Maison wrote: > Hi Omnia, > > +1 (binding), thanks for the KIP! > > Mickael > > On Fri, Apr 12, 2024 at 9:01 AM Omnia Ibrahim > wrote: > > > > Hi everyone, I would like to start a voting thread for KIP-1031: Control > offset

Re: [DISCUSS] KIP-899: Allow clients to rebootstrap

2024-04-11 Thread Chris Egerton
t; shouldn't we still attempt a re-bootstrap at some point (if the > user has > >>>>>> enabled this feature)? > >>>>> > >>>>> Yes, you're right. Particularly `canConnect` here [1] can always be > >>>>> returning `true` if `reco

Re: [VOTE] KIP-477: Add PATCH method for connector config in Connect REST API

2024-04-08 Thread Chris Egerton
Thanks Ivan! +1 (binding) from me. On Mon, Apr 8, 2024, 06:59 Ivan Yurchenko wrote: > Hello! > > I'd like to put the subj KIP[1] to a vote. Thank you. > > Best regards, > Ivan > > [1] >

Re: [DISCUSS] KIP-1031: Control offset translation in MirrorSourceConnector

2024-04-07 Thread Chris Egerton
or should fail to start if > emit.offset-syncs.enabled is disabled while emit.checkpoints.enabled and/or > sync.group.offsets.enabled are enabled as no point of creating this > connector if the main part is disabled. WDYT? > > Thanks > Omnia > > > On 3 Apr 2024, at 12:45, Chris Egerton

Re: [DISCUSS] KIP-1031: Control offset translation in MirrorSourceConnector

2024-04-03 Thread Chris Egerton
Hi Omnia, Thanks for the KIP! Two small things come to mind: 1. It'd be nice to mention that increasing the max offset lag to INT_MAX could work as a partial workaround for users on existing versions (though of course this wouldn't prevent creation of the syncs topic). 2. Will it be illegal to

Re: [DISCUSS] KIP-477: Add PATCH method for connector config in Connect REST API

2024-03-28 Thread Chris Egerton
them. > > Best, > Ivan > > On Wed, Mar 27, 2024, at 17:08, Chris Egerton wrote: > > Hi Ivan, > > > > Thanks for the updates. LGTM! > > > > RE atomicity: I think it should be possible and not _too_ invasive to > > detect and handle these kinds of ra

Re: [DISCUSS] KIP-477: Add PATCH method for connector config in Connect REST API

2024-03-27 Thread Chris Egerton
; > > > > > On Tue, Mar 26, 2024, at 14:19, Ivan Yurchenko wrote: > > > Hi all, > > > > > > This KIP is a bit old now :) but I think its context hasn't changed > much since then and the KIP is still valid. I would like to finally bring > it to some

[jira] [Resolved] (KAFKA-16423) Ignoring offset partition key with an unexpected format for the first element in the partition key list. Expected type: java.lang.String, actual type: null"

2024-03-26 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-16423. --- Fix Version/s: (was: 3.6.2) (was: 3.8.0

[jira] [Resolved] (KAFKA-16423) Ignoring offset partition key with an unexpected format for the first element in the partition key list. Expected type: java.lang.String, actual type: null"

2024-03-26 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-16423. --- Resolution: Duplicate > Ignoring offset partition key with an unexpected for

[jira] [Reopened] (KAFKA-16423) Ignoring offset partition key with an unexpected format for the first element in the partition key list. Expected type: java.lang.String, actual type: null"

2024-03-26 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton reopened KAFKA-16423: --- Assignee: (was: johndoe) > Ignoring offset partition key with an unexpected for

[jira] [Created] (KAFKA-16392) Spurious log warnings: "Ignoring offset partition key with an unexpected format for the second element in the partition key list. Expected type: java.util.Map, actual ty

2024-03-20 Thread Chris Egerton (Jira)
Chris Egerton created KAFKA-16392: - Summary: Spurious log warnings: "Ignoring offset partition key with an unexpected format for the second element in the partition key list. Expected type: java.util.Map, actual type: null"

Re: [VOTE] KIP-995: Allow users to specify initial offsets while creating connectors

2024-03-06 Thread Chris Egerton
ts_status" might be more appropriate, what do you think? I don't > > think the current name is terrible though, so I'm +1 (binding) if > everyone > > else agrees that it's suitable. > > > > Thanks, > > Yash > > > > On Tue, Mar 5, 2024 at 9:

Re: [VOTE] KIP-995: Allow users to specify initial offsets while creating connectors

2024-03-05 Thread Chris Egerton
ion to users here as well. Thoughts? > > > > Thanks, > > Yash > > > > On Wed, Jan 17, 2024 at 3:31 PM Ashwin > > wrote: > > > > > Hi All , > > > > > > Can I please get one more binding vote, so that the KIP is approved ? > > >

Re: Shortened URLs for KIPs?

2024-02-28 Thread Chris Egerton
Hi Kirk, Interesting proposal! I gave it a shot with one of my own prior KIPs and was able to generate https://s.apache.org/kip-618 for it. It looks like uppercase characters aren't permitted for URL IDs (even though the regex listed in that text box does claim to allow them). I can't commit to

Re: [ANNOUNCE] Apache Kafka 3.7.0

2024-02-27 Thread Chris Egerton
a, Alyssa Huang, Aman Singh, Andras Katona, Andrew > Schofield, Anna Sophie Blee-Goldman, Anton Agestam, Apoorv Mittal, > Arnout Engelen, Arpit Goyal, Artem Livshits, Ashwin Pankaj, > ashwinpankaj, atu-sharm, bachmanity1, Bob Barrett, Bruno Cadonna, > Calvin Liu, Cerchie, chern, Chris Egerton, Chris

Re: [DISCUSS] Apache Kafka 3.8.0 release

2024-02-26 Thread Chris Egerton
Thanks Josep, I'm +1 as well. On Mon, Feb 26, 2024 at 12:32 PM Justine Olshan wrote: > Thanks Joesp. +1 from me. > > On Mon, Feb 26, 2024 at 3:37 AM Josep Prat > wrote: > > > Hi all, > > > > I'd like to volunteer as release manager for the Apache Kafka 3.8.0 > > release. > > If there are no

Re: Github build queue

2024-02-09 Thread Chris Egerton
+1, would love this. On Fri, Feb 9, 2024, 10:04 David Arthur wrote: > Hey folks, > > I recently learned about Github's Merge Queue feature, and I think it could > help us out. > > Essentially, when you hit the Merge button on a PR, it will add the PR to a > queue and let you run a CI job before

[jira] [Resolved] (KAFKA-15575) Prevent Connectors from exceeding tasks.max configuration

2024-02-01 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15575?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-15575. --- Fix Version/s: 3.8.0 Resolution: Fixed > Prevent Connectors from exceeding tasks.

[DISCUSS] KIP-1017: A health check endpoint for Kafka Connect

2024-01-26 Thread Chris Egerton
Hi all, Happy Friday! I'd like to kick off discussion for KIP-1017, which (as the title suggests) proposes adding a health check endpoint for Kafka Connect: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1017%3A+Health+check+endpoint+for+Kafka+Connect This is one of the longest-standing

Re: [VOTE] KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

2024-01-24 Thread Chris Egerton
Thanks Ziming! +1 (binding) On Wed, Jan 24, 2024, 03:23 ziming deng wrote: > Thank you for reminding this, David, > > I have mentioned this in the [Compatibility] section as a following work. > > --, > Best, > Ziming > > > On Jan 23, 2024, at 15:17, David Jacot > wrote: > > > > Hi Chris,

Re: [DISCUSS] KIP-1016 Make MM2 heartbeats topic name configurable

2024-01-22 Thread Chris Egerton
Hi Berci, Thanks for the KIP! IMO we don't need the "default." prefix for the new property, and it deviates a bit from the precedent set by properties like "replication.policy.internal.topic.separator.enabled". I think we can just call it "replication.policy.heartbeats.topic", or if we really

Re: [VOTE] KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

2024-01-22 Thread Chris Egerton
Hi David, I've only briefly skimmed KAFKA-10140 but it seems that it may only apply to append/subtract operations on list-type properties. If my understanding is correct then this shouldn't be a problem for the KIP since we only use the set/delete operations in the kafka-configs.sh script. If the

Re: DISCUSS KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

2024-01-18 Thread Chris Egerton
gt;>> don't appear to support appending/subtracting from list properties via > >> the > >>> CLI for any other entity type right now, > >> You are right about this, I tried and found that we can’t subtract or > >> append configs, I will change the KI

Re: [VOTE] 3.7.0 RC2

2024-01-12 Thread Chris Egerton
ache/kafka/pull/15186). I will use the quickstart > approach as a second means to reproduce/verify while I wait for the PR's > Jenkins job to finish. > > Thanks, > Kirk > > On Fri, Jan 12, 2024, at 11:31 AM, Chris Egerton wrote: > > Hi Stanislav, > > > > >

Re: [VOTE] 3.7.0 RC2

2024-01-12 Thread Chris Egerton
Hi Stanislav, Thanks for running this release! To verify, I: - Built from source using Java 11 with both: - - the 3.7.0-rc2 tag on GitHub - - the kafka-3.7.0-src.tgz artifact from https://home.apache.org/~stanislavkozlovski/kafka-3.7.0-rc2/ - Checked signatures and checksums - Ran the

Re: [PROPOSAL] Add commercial support page on website

2024-01-11 Thread Chris Egerton
Hi François, Is it an official policy of the ASF that projects provide a listing of commercial support options for themselves? I understand that other projects have chosen to provide one, but this doesn't necessarily imply that all projects should do the same, and I can't say I find this point

Re: [VOTE] KIP-1004: Enforce tasks.max property in Kafka Connect

2024-01-11 Thread Chris Egerton
ing it in 4.0, leaning towards > keeping it, for the same reasons you listed above. > > Thanks! > Greg > > On Fri, Jan 5, 2024 at 9:40 AM Chris Egerton > wrote: > > > > Hi Yash, > > > > Thanks for raising the possibility of a more aggressive removal sched

[jira] [Created] (KAFKA-16108) Backport fix for KAFKA-16093 to 3.7

2024-01-10 Thread Chris Egerton (Jira)
Chris Egerton created KAFKA-16108: - Summary: Backport fix for KAFKA-16093 to 3.7 Key: KAFKA-16108 URL: https://issues.apache.org/jira/browse/KAFKA-16108 Project: Kafka Issue Type

Re: [VOTE] KIP-995: Allow users to specify initial offsets while creating connectors

2024-01-09 Thread Chris Egerton
Thanks for the KIP! +1 (binding) On Mon, Jan 8, 2024 at 9:35 AM Ashwin wrote: > Hi All, > > I would like to start a vote on KIP-995. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-995%3A+Allow+users+to+specify+initial+offsets+while+creating+connectors > > Discussion thread - >

Re: [VOTE] KIP-1012: The need for a Kafka 3.8.x release

2024-01-08 Thread Chris Egerton
3.9, 3.10, etc.). But perhaps I'm missing something? Cheers, Chris On Mon, Jan 8, 2024 at 2:38 PM Colin McCabe wrote: > On Mon, Jan 8, 2024, at 09:05, Chris Egerton wrote: > > Hi Josep, > > > > Thanks for the KIP! +1 (binding). > > > > One small nit: I don't

[jira] [Created] (KAFKA-16093) Spurious warnings logged to stderr about empty path annotations and providers not implementing provider interfaces

2024-01-08 Thread Chris Egerton (Jira)
Chris Egerton created KAFKA-16093: - Summary: Spurious warnings logged to stderr about empty path annotations and providers not implementing provider interfaces Key: KAFKA-16093 URL: https://issues.apache.org/jira

Re: [VOTE] KIP-1012: The need for a Kafka 3.8.x release

2024-01-08 Thread Chris Egerton
Hi Josep, Thanks for the KIP! +1 (binding). One small nit: I don't think we necessarily have to hold ourselves to releasing 4.0.0 "3 to 4 months after 3.8 branch is created" (quoting the timeline section of the KIP). IMO it's fine to leave some wiggle room for the 4.0.0 release without codifying

Re: [VOTE] KIP-1004: Enforce tasks.max property in Kafka Connect

2024-01-05 Thread Chris Egerton
; > > > > > From: dev@kafka.apache.org At: 01/02/24 11:49:18 UTC-5:00To: > > > dev@kafka.apache.org > > > > > Subject: Re: [VOTE] KIP-1004: Enforce tasks.max property in Kafka > > > Connect > > > > > > > > > > Hi all,

Re: DISCUSS KIP-1011: Use incrementalAlterConfigs when updating broker configs by kafka-configs.sh

2024-01-05 Thread Chris Egerton
Hi all, Can we clarify any changes in the user-facing semantics for the CLI tool that would come about as a result of this KIP? I think the debate over the necessity of an opt-in flag, or waiting for 4.0.0, ultimately boils down to this. My understanding is that the only changes in semantics are

Re: [VOTE] KIP-1004: Enforce tasks.max property in Kafka Connect

2024-01-02 Thread Chris Egerton
Hi all, Happy New Year! Wanted to give this a bump now that the holidays are over for a lot of us. Looking forward to people's thoughts! Cheers, Chris On Mon, Dec 4, 2023 at 10:36 AM Chris Egerton wrote: > Hi all, > > I'd like to call for a vote on KIP-1004, which adds en

Re: [VOTE] KIP-993: Allow restricting files accessed by File and Directory ConfigProviders

2023-12-13 Thread Chris Egerton
Hi Tina, Thanks for the KIP! +1 (binding) Cheers, Chris On Wed, Dec 13, 2023 at 5:50 AM Gantigmaa Selenge wrote: > Hi > > I'd like to call for a vote on KIP-993, which allows restricting files > accessed by File and Directory ConfigProviders when using Connect. > > The KIP: > >

[DISCUSS] Kafka Connect source task interruption semantics

2023-12-12 Thread Chris Egerton
Hi all, I'd like to solicit input from users and maintainers on a problem we've been dealing with for source task cleanup logic. If you'd like to pore over some Jira history, here's the primary link: https://issues.apache.org/jira/browse/KAFKA-15090 To summarize, we accidentally introduced a

Re: KIP-993: Allow restricting files accessed by File and Directory ConfigProviders

2023-12-12 Thread Chris Egerton
Interfaces section but I also added > another sentence to make it clearer. > > > 4. Yes, I have just tested this to confirm and it looks like "/" gives > access to the entire file system. > > > Thanks. > Regards, > Tina > > > > > On Thu,

Re: [DISCUSS] KIP-995: Allow users to specify initial offsets while creating connectors

2023-12-12 Thread Chris Egerton
for pointing this out Chris. > > I have updated the KIP with the correct sequence of steps. > > Thanks, > Ashwin > > On Wed, Dec 6, 2023 at 11:48 PM Chris Egerton > wrote: > > > Hi Ashwin, > > > > Regarding point 4--I think this is still a bit unwi

[jira] [Resolved] (KAFKA-15563) Provide informative error messages when Connect REST requests time out

2023-12-11 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-15563. --- Fix Version/s: 3.7.0 Resolution: Fixed > Provide informative error messages w

[jira] [Created] (KAFKA-15988) Kafka Connect OffsetsApiIntegrationTest takes too long

2023-12-07 Thread Chris Egerton (Jira)
Chris Egerton created KAFKA-15988: - Summary: Kafka Connect OffsetsApiIntegrationTest takes too long Key: KAFKA-15988 URL: https://issues.apache.org/jira/browse/KAFKA-15988 Project: Kafka

Re: KIP-993: Allow restricting files accessed by File and Directory ConfigProviders

2023-12-07 Thread Chris Egerton
Hi Tina, Thanks for the KIP! Looks good overall. A few minor thoughts: 1. We can remove the "This page is meant as a template for writing a KIP" section from the beginning. 2. The type of the allowed.paths property is string in the KIP, but the description mentions it'll contain multiple

Re: [DISCUSS] KIP-995: Allow users to specify initial offsets while creating connectors

2023-12-06 Thread Chris Egerton
ting config to the config topic. Have updated > the wiki to make it clearer. > > > Nits: > > Thanks for pointing these - I have made the necessary changes. > > > Thanks again, > > Ashwin > > > On Mon, Dec 4, 2023 at 9:05 PM Chris Egerton > wrote: >

Re: [DISCUSS] KIP-967: Support custom SSL configuration for Kafka Connect RestServer

2023-12-04 Thread Chris Egerton
Hi Taras, Regarding slimming down the interface: IMO, we should do this right the first time, and that includes not requiring unnecessary methods from users. I think BaseSslEngineFactory is good enough as a superinterface. Regarding the parsing logic: I think the KIP needs to be more explicit.

Re: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka Connect

2023-12-04 Thread Chris Egerton
Hi all, It seems like there are no objections to this KIP, so I've kicked off a vote thread: https://lists.apache.org/thread/dgq332o5j25rwddbvfydf7ttrclldw17 Cheers, Chris On Fri, Nov 24, 2023 at 10:39 PM Chris Egerton wrote: > Hi Yash, > > Thanks for taking a look! Yeah, it looks l

[VOTE] KIP-1004: Enforce tasks.max property in Kafka Connect

2023-12-04 Thread Chris Egerton
Hi all, I'd like to call for a vote on KIP-1004, which adds enforcement for the tasks.max connector property in Kafka Connect. The KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1004%3A+Enforce+tasks.max+property+in+Kafka+Connect The discussion thread:

Re: [DISCUSS] KIP-995: Allow users to specify initial offsets while creating connectors

2023-12-04 Thread Chris Egerton
Oh, one more thing--can we add the KIP to the list of KIPs? https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-KIPsunderdiscussion On Mon, Dec 4, 2023 at 10:33 AM Chris Egerton wrote: > Hi Ashwin, > > Thanks for the KIP! This would

Re: [DISCUSS] KIP-995: Allow users to specify initial offsets while creating connectors

2023-12-04 Thread Chris Egerton
Hi Ashwin, Thanks for the KIP! This would be a nice simplification to the process for migrating connectors enabled by KIP-980, and would also add global support for a feature I've seen implemented by hand in at least a couple connectors over the years. High-level thoughts: 1. If there are

Re: [DISCUSS] Should we continue to merge without a green build? No!

2023-11-28 Thread Chris Egerton
or has not completed. > > Best, > David > > On Sat, Nov 25, 2023 at 5:25 AM Chris Egerton > wrote: > > > Hi all, > > > > There's a lot to catch up on here but I wanted to clarify something. > > Regarding this comment from Sophie: > > > > > > &g

Re: [DISCUSS] Should we continue to merge without a green build? No!

2023-11-24 Thread Chris Egerton
Hi all, There's a lot to catch up on here but I wanted to clarify something. Regarding this comment from Sophie: > Yet multiple people in this thread so far have voiced support for "gating merges on the successful completion of all parts of the build except tests". Just to be totally clear, I

Re: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka Connect

2023-11-24 Thread Chris Egerton
be a 3.8.0 release before the 4.0.0 release, would we then be > forced to punt the removal of "tasks.max.enforce" to a future 5.0.0 > release? I don't have any other comments, and the proposed changes make > sense to me. > > Thanks, > Yash > > On Mon, Nov 20, 2023

Re: [DISCUSS] KIP-1004: Enforce tasks.max property in Kafka Connect

2023-11-20 Thread Chris Egerton
Hi Hector, Thanks for taking a look! I think the key difference between the proposed behavior and the rejected alternative is that the set of tasks that will be running with the former is still a complete set of tasks, whereas the set of tasks in the latter is a subset of tasks. Also noteworthy

Re: [DISCUSS] Should we continue to merge without a green build? No!

2023-11-14 Thread Chris Egerton
One potential forcing function could be to allow an unconditional revert of any commit (including testing and non-testing changes) that can be shown to have introduced test flakiness. This could at least prevent us from accruing more flaky tests, though it would obviously not be viable for

[jira] [Created] (KAFKA-15821) Active topics for deleted connectors are not reset in standalone mode

2023-11-13 Thread Chris Egerton (Jira)
Chris Egerton created KAFKA-15821: - Summary: Active topics for deleted connectors are not reset in standalone mode Key: KAFKA-15821 URL: https://issues.apache.org/jira/browse/KAFKA-15821 Project

[DISCUSS] KIP-1004: Enforce tasks.max property in Kafka Connect

2023-11-10 Thread Chris Egerton
Hi all, I'd like to open up KIP-1004 for discussion: https://cwiki.apache.org/confluence/display/KAFKA/KIP-1004%3A+Enforce+tasks.max+property+in+Kafka+Connect As a brief summary: this KIP proposes that the Kafka Connect runtime start failing connectors that generate a greater number of tasks

[jira] [Resolved] (KAFKA-15680) Partition-Count is not getting updated Correctly in the Incremental Co-operative Rebalancing(ICR) Mode of Rebalancing

2023-11-06 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-15680. --- Fix Version/s: 3.7.0 (was: 3.6.1) Resolution: Fixed

[jira] [Resolved] (KAFKA-15787) Investigate new test case failure - testReplicateSourceDefault - MirrorConnectorsIntegrationBaseTest

2023-11-06 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-15787. --- Resolution: Duplicate > Investigate new test case failure - testReplicateSourceDefa

Re: [VOTE] KIP-998: Give ProducerConfig(props, doLog) constructor protected access

2023-11-03 Thread Chris Egerton
No objections, I'm +1 ether way. On Fri, Nov 3, 2023, 18:50 Sophie Blee-Goldman wrote: > I am fine with making them public. Of course in that case we should also > change the corresponding constructors in ConsumerConfig, AdminConfig, and > StreamsConfig from protected to public as well, to be

Re: [VOTE] KIP-998: Give ProducerConfig(props, doLog) constructor protected access

2023-11-03 Thread Chris Egerton
+1 (binding) FWIW, I agree that this change should require a KIP. Gating upgrades of visibility from private or package-private to protected may be cumbersome, but it comes with the expectation that downgrades in visibility for those same classes/methods will also be gated behind a KIP, which is

Re: [DISCUSS] KIP-967: Support custom SSL configuration for Kafka Connect RestServer

2023-11-02 Thread Chris Egerton
broker just prefers to reuse the code as is. > > [1]. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-967%3A+Support+custom+SSL+configuration+for+Kafka+Connect+RestServer#KIP967:SupportcustomSSLconfigurationforKafkaConnectRestServer-Background > > -- > With best regards,

[jira] [Resolved] (KAFKA-15761) ConnectorRestartApiIntegrationTest.testMultiWorkerRestartOnlyConnector is flaky

2023-10-31 Thread Chris Egerton (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-15761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-15761. --- Resolution: Duplicate

Re: [DISCUSS] KIP-967: Support custom SSL configuration for Kafka Connect RestServer

2023-10-30 Thread Chris Egerton
Hi Taras, Thanks for the KIP! I have some feedback but ultimately I like this proposal: 1. The "ssl.engine.factory.class" property was originally added for Kafka brokers in KIP-519 [1]. It'd be nice to link to that KIP (possibly in a "Background" section?) so that reviewers who don't have that

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

2023-10-27 Thread Chris Egerton
Congrats, Satish! On Fri, Oct 27, 2023, 11:04 Jun Rao wrote: > Hi, Everyone, > > Satish Duggana has been a Kafka committer since 2022. He has been very > instrumental to the community since becoming a committer. It's my pleasure > to announce that Satish is now a member of Kafka PMC. > >

  1   2   3   4   5   6   7   >