[jira] [Updated] (KAFKA-14873) Pluggable storage for Kafka Connect internal topics
[ https://issues.apache.org/jira/browse/KAFKA-14873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-14873: -- Labels: needs-kip (was: ) > Pluggable storage for Kafka Connect internal topics > --- > > Key: KAFKA-14873 > URL: https://issues.apache.org/jira/browse/KAFKA-14873 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Malthe Borch >Priority: Major > Labels: needs-kip > > The Kafka Connect framework relies on compacted topics to store config, > offset and status information for each connector. > This conflates two kinds of data, control and content, which some people > disagree with. Notably, [Azure Event > Hub|https://learn.microsoft.com/en-us/azure/event-hubs/log-compaction] does > not (or _did not_, because there's currently a preview release out which does > have support for compacted topics albeit only at the more expensive premium > tiers). > In some deployments, it may be desirable to use a different backend for these > control settings (which essentially take a key/value form), for example > [Azure Table > Storage|https://learn.microsoft.com/en-us/rest/api/storageservices/table-service-rest-api] > – basically any key/value store that provides the Write-If-Matches primitive > to update a key only if the current value matches a known value. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14873) Pluggable storage for Kafka Connect internal topics
[ https://issues.apache.org/jira/browse/KAFKA-14873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-14873: -- Component/s: KafkaConnect > Pluggable storage for Kafka Connect internal topics > --- > > Key: KAFKA-14873 > URL: https://issues.apache.org/jira/browse/KAFKA-14873 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Malthe Borch >Priority: Major > > The Kafka Connect framework relies on compacted topics to store config, > offset and status information for each connector. > This conflates two kinds of data, control and content, which some people > disagree with. Notably, [Azure Event > Hub|https://learn.microsoft.com/en-us/azure/event-hubs/log-compaction] does > not (or _did not_, because there's currently a preview release out which does > have support for compacted topics albeit only at the more expensive premium > tiers). > In some deployments, it may be desirable to use a different backend for these > control settings (which essentially take a key/value form), for example > [Azure Table > Storage|https://learn.microsoft.com/en-us/rest/api/storageservices/table-service-rest-api] > – basically any key/value store that provides the Write-If-Matches primitive > to update a key only if the current value matches a known value. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14420) MirrorMaker should not clear filtered configs on target topics
[ https://issues.apache.org/jira/browse/KAFKA-14420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710197#comment-17710197 ] Chris Egerton commented on KAFKA-14420: --- Addressed with [KIP-894|https://cwiki.apache.org/confluence/display/KAFKA/KIP-894%3A+Use+incrementalAlterConfigs+API+for+syncing+topic+configurations]. > MirrorMaker should not clear filtered configs on target topics > -- > > Key: KAFKA-14420 > URL: https://issues.apache.org/jira/browse/KAFKA-14420 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 3.3.1 >Reporter: Mickael Maison >Assignee: Gantigmaa Selenge >Priority: Major > Fix For: 3.5.0 > > > If you set additional configurations on a remote topic, MirrorMaker will > clear them when it syncs topic configurations. > The issue is that it also clears topic configurations that are filtered. For > example this prevents running Cruise Control on the target cluster as it may > set follower.replication.throttled.replicas and > leader.replication.throttled.replicas. > MirrorMaker should not clear topic configurations that are filtered on the > target cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14420) MirrorMaker should not clear filtered configs on target topics
[ https://issues.apache.org/jira/browse/KAFKA-14420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-14420: -- Fix Version/s: 3.5.0 > MirrorMaker should not clear filtered configs on target topics > -- > > Key: KAFKA-14420 > URL: https://issues.apache.org/jira/browse/KAFKA-14420 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 3.3.1 >Reporter: Mickael Maison >Assignee: Gantigmaa Selenge >Priority: Major > Fix For: 3.5.0 > > > If you set additional configurations on a remote topic, MirrorMaker will > clear them when it syncs topic configurations. > The issue is that it also clears topic configurations that are filtered. For > example this prevents running Cruise Control on the target cluster as it may > set follower.replication.throttled.replicas and > leader.replication.throttled.replicas. > MirrorMaker should not clear topic configurations that are filtered on the > target cluster. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-14783) Implement new STOPPED state for connectors
[
https://issues.apache.org/jira/browse/KAFKA-14783?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton resolved KAFKA-14783.
---
Fix Version/s: 3.5.0
Resolution: Done
> Implement new STOPPED state for connectors
> --
>
> Key: KAFKA-14783
> URL: https://issues.apache.org/jira/browse/KAFKA-14783
> Project: Kafka
> Issue Type: Sub-task
> Components: KafkaConnect
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Major
> Fix For: 3.5.0
>
>
> Implement the {{STOPPED}} state [described in
> KIP-875|https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect#KIP875:FirstclassoffsetssupportinKafkaConnect-Newtargetstate:STOPPED].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-14746) Throwing in Connector.taskConfigs in distributed mode generates a lot of logs
[
https://issues.apache.org/jira/browse/KAFKA-14746?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17710972#comment-17710972
]
Chris Egerton commented on KAFKA-14746:
---
Hi Yash,
{quote}are you suggesting a KIP to modify the existing retry mechanism to not
cover exceptions thrown from connectors' taskConfigs method?
{quote}
Yep, exactly. But it's not high-priority and given Mickael's thoughts I don't
think we should move forward with it for now. The suggestion to at least
document this behavior is plenty (although we'll have to think about how to
handle the discrepancy in behavior between distributed and standalone modes).
{quote}Why would this require a KIP if this retry mechanism isn't part of the
public API?
{quote}
Some developers may have written connectors around this logic to get automatic
retries for free. Changing that behavior has the potential to break those
connectors. I also think that, even though we don't document this behavior in,
e.g., the {{taskConfigs}} Javadoc, it is still part of public API since it
directly affects how we interact with connectors.
> Throwing in Connector.taskConfigs in distributed mode generates a lot of logs
> -
>
> Key: KAFKA-14746
> URL: https://issues.apache.org/jira/browse/KAFKA-14746
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Reporter: Mickael Maison
>Priority: Major
>
> If a Connector throws in its taskConfigs() method, the runtime ends up
> retrying using DistributedHerder.RECONFIGURE_CONNECTOR_TASKS_BACKOFF_MS which
> is a fixed value (250ms). For each retry, the runtime prints the connector
> configuration and the enriched configuration so this can quickly generate a
> lot of logs.
> There is some value in throwing in taskConfigs() as it allows to fail fast in
> case the connector is given bad credentials. For example this is what some of
> the Debezium connectors do:
> https://github.com/debezium/debezium/blob/main/debezium-connector-sqlserver/src/main/java/io/debezium/connector/sqlserver/SqlServerConnector.java#L56-L69
> The way Connectors are expected to work today is to instead always create
> tasks and let each task fail in case the configuration is wrong. We should
> document that and make it clear in the javadoc that throwing in taskConfigs
> is not recommended.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-14666) MM2 should translate consumer group offsets behind replication flow
[ https://issues.apache.org/jira/browse/KAFKA-14666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17716280#comment-17716280 ] Chris Egerton commented on KAFKA-14666: --- [~mimaison] I believe this should be a release blocker. We don't necessarily have to merge the fix associated with this issue for 3.5.0, but the alternative would be to revert several other improvements and fixes we've made to MM2 that, while useful, exacerbated the impact of this issue. I've been reviewing the PR more closely over the past few days with the goal of merging either today or tomorrow (day of the 3.5.0 code freeze deadline). I've just approved it and am waiting on the CI build to complete. Are you okay with backporting this to the 3.5 branch if CI goes well? > MM2 should translate consumer group offsets behind replication flow > --- > > Key: KAFKA-14666 > URL: https://issues.apache.org/jira/browse/KAFKA-14666 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Reporter: Greg Harris >Assignee: Greg Harris >Priority: Major > Fix For: 3.5.0 > > > MirrorMaker2 includes an offset translation feature which can translate the > offsets for an upstream consumer group to a corresponding downstream consumer > group. It does this by keeping a topic of offset-syncs to correlate upstream > and downstream offsets, and translates any source offsets which are ahead of > the replication flow. > However, if a replication flow is closer to the end of a topic than the > consumer group, then the offset translation feature will refuse to translate > the offset for correctness reasons. This is because the MirrorCheckpointTask > only keeps the latest offset correlation between source and target, it does > not have sufficient information to translate older offsets. > The workarounds for this issue are to: > 1. Pause the replication flow occasionally to allow the source to get ahead > of MM2 > 2. Increase the offset.lag.max to delay offset syncs, increasing the window > for translation to happen. With the fix for KAFKA-12468, this will also > increase the lag of applications that are ahead of the replication flow, so > this is a tradeoff. > Instead, the MirrorCheckpointTask should provide correct and best-effort > translation for consumer groups behind the replication flow by keeping > additional state, or re-reading the offset-syncs topic. This should be a > substantial improvement for use-cases where applications have a higher > latency to commit than the replication flow, or where applications are > reading from the earliest offset. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14666) MM2 should translate consumer group offsets behind replication flow
[ https://issues.apache.org/jira/browse/KAFKA-14666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-14666: -- Priority: Blocker (was: Major) > MM2 should translate consumer group offsets behind replication flow > --- > > Key: KAFKA-14666 > URL: https://issues.apache.org/jira/browse/KAFKA-14666 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Reporter: Greg Harris >Assignee: Greg Harris >Priority: Blocker > Fix For: 3.5.0 > > > MirrorMaker2 includes an offset translation feature which can translate the > offsets for an upstream consumer group to a corresponding downstream consumer > group. It does this by keeping a topic of offset-syncs to correlate upstream > and downstream offsets, and translates any source offsets which are ahead of > the replication flow. > However, if a replication flow is closer to the end of a topic than the > consumer group, then the offset translation feature will refuse to translate > the offset for correctness reasons. This is because the MirrorCheckpointTask > only keeps the latest offset correlation between source and target, it does > not have sufficient information to translate older offsets. > The workarounds for this issue are to: > 1. Pause the replication flow occasionally to allow the source to get ahead > of MM2 > 2. Increase the offset.lag.max to delay offset syncs, increasing the window > for translation to happen. With the fix for KAFKA-12468, this will also > increase the lag of applications that are ahead of the replication flow, so > this is a tradeoff. > Instead, the MirrorCheckpointTask should provide correct and best-effort > translation for consumer groups behind the replication flow by keeping > additional state, or re-reading the offset-syncs topic. This should be a > substantial improvement for use-cases where applications have a higher > latency to commit than the replication flow, or where applications are > reading from the earliest offset. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14666) MM2 should translate consumer group offsets behind replication flow
[ https://issues.apache.org/jira/browse/KAFKA-14666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17716579#comment-17716579 ] Chris Egerton commented on KAFKA-14666: --- Merged to trunk and backported to 3.5. I'll backport further to other affected branches sometime this week. > MM2 should translate consumer group offsets behind replication flow > --- > > Key: KAFKA-14666 > URL: https://issues.apache.org/jira/browse/KAFKA-14666 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Reporter: Greg Harris >Assignee: Greg Harris >Priority: Blocker > Fix For: 3.5.0 > > > MirrorMaker2 includes an offset translation feature which can translate the > offsets for an upstream consumer group to a corresponding downstream consumer > group. It does this by keeping a topic of offset-syncs to correlate upstream > and downstream offsets, and translates any source offsets which are ahead of > the replication flow. > However, if a replication flow is closer to the end of a topic than the > consumer group, then the offset translation feature will refuse to translate > the offset for correctness reasons. This is because the MirrorCheckpointTask > only keeps the latest offset correlation between source and target, it does > not have sufficient information to translate older offsets. > The workarounds for this issue are to: > 1. Pause the replication flow occasionally to allow the source to get ahead > of MM2 > 2. Increase the offset.lag.max to delay offset syncs, increasing the window > for translation to happen. With the fix for KAFKA-12468, this will also > increase the lag of applications that are ahead of the replication flow, so > this is a tradeoff. > Instead, the MirrorCheckpointTask should provide correct and best-effort > translation for consumer groups behind the replication flow by keeping > additional state, or re-reading the offset-syncs topic. This should be a > substantial improvement for use-cases where applications have a higher > latency to commit than the replication flow, or where applications are > reading from the earliest offset. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14876) Public documentation for new Kafka Connect offset management REST APIs in 3.5
[
https://issues.apache.org/jira/browse/KAFKA-14876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17718258#comment-17718258
]
Chris Egerton commented on KAFKA-14876:
---
[~yash.mayya] I think it's worth documenting the stop API now, both to give
users more time to learn about it and since it may be useful as a more
efficient long-term variant of the pause API.
> Public documentation for new Kafka Connect offset management REST APIs in 3.5
> -
>
> Key: KAFKA-14876
> URL: https://issues.apache.org/jira/browse/KAFKA-14876
> Project: Kafka
> Issue Type: Sub-task
>Reporter: Yash Mayya
>Assignee: Yash Mayya
>Priority: Major
> Fix For: 3.5.0
>
>
> Add public documentation for the new Kafka Connect offset management REST API
> being introduced in
> [KIP-875|https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect]
> in 3.5:
> * *GET* /connectors/\{connector}/offsets
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-14666) MM2 should translate consumer group offsets behind replication flow
[ https://issues.apache.org/jira/browse/KAFKA-14666?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17718669#comment-17718669 ] Chris Egerton commented on KAFKA-14666: --- Backported to 3.4 and 3.3. > MM2 should translate consumer group offsets behind replication flow > --- > > Key: KAFKA-14666 > URL: https://issues.apache.org/jira/browse/KAFKA-14666 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Reporter: Greg Harris >Assignee: Greg Harris >Priority: Blocker > Fix For: 3.5.0 > > > MirrorMaker2 includes an offset translation feature which can translate the > offsets for an upstream consumer group to a corresponding downstream consumer > group. It does this by keeping a topic of offset-syncs to correlate upstream > and downstream offsets, and translates any source offsets which are ahead of > the replication flow. > However, if a replication flow is closer to the end of a topic than the > consumer group, then the offset translation feature will refuse to translate > the offset for correctness reasons. This is because the MirrorCheckpointTask > only keeps the latest offset correlation between source and target, it does > not have sufficient information to translate older offsets. > The workarounds for this issue are to: > 1. Pause the replication flow occasionally to allow the source to get ahead > of MM2 > 2. Increase the offset.lag.max to delay offset syncs, increasing the window > for translation to happen. With the fix for KAFKA-12468, this will also > increase the lag of applications that are ahead of the replication flow, so > this is a tradeoff. > Instead, the MirrorCheckpointTask should provide correct and best-effort > translation for consumer groups behind the replication flow by keeping > additional state, or re-reading the offset-syncs topic. This should be a > substantial improvement for use-cases where applications have a higher > latency to commit than the replication flow, or where applications are > reading from the earliest offset. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14666) MM2 should translate consumer group offsets behind replication flow
[ https://issues.apache.org/jira/browse/KAFKA-14666?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-14666: -- Fix Version/s: 3.4.1 3.3.3 > MM2 should translate consumer group offsets behind replication flow > --- > > Key: KAFKA-14666 > URL: https://issues.apache.org/jira/browse/KAFKA-14666 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Reporter: Greg Harris >Assignee: Greg Harris >Priority: Blocker > Fix For: 3.5.0, 3.4.1, 3.3.3 > > > MirrorMaker2 includes an offset translation feature which can translate the > offsets for an upstream consumer group to a corresponding downstream consumer > group. It does this by keeping a topic of offset-syncs to correlate upstream > and downstream offsets, and translates any source offsets which are ahead of > the replication flow. > However, if a replication flow is closer to the end of a topic than the > consumer group, then the offset translation feature will refuse to translate > the offset for correctness reasons. This is because the MirrorCheckpointTask > only keeps the latest offset correlation between source and target, it does > not have sufficient information to translate older offsets. > The workarounds for this issue are to: > 1. Pause the replication flow occasionally to allow the source to get ahead > of MM2 > 2. Increase the offset.lag.max to delay offset syncs, increasing the window > for translation to happen. With the fix for KAFKA-12468, this will also > increase the lag of applications that are ahead of the replication flow, so > this is a tradeoff. > Instead, the MirrorCheckpointTask should provide correct and best-effort > translation for consumer groups behind the replication flow by keeping > additional state, or re-reading the offset-syncs topic. This should be a > substantial improvement for use-cases where applications have a higher > latency to commit than the replication flow, or where applications are > reading from the earliest offset. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14837) The MirrorCheckPointConnector of MM2 will rebalance frequently, when the source cluster group is many more and changes frequently (but the list of configured synchronous
[ https://issues.apache.org/jira/browse/KAFKA-14837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-14837: -- Fix Version/s: 3.4.1 3.3.3 > The MirrorCheckPointConnector of MM2 will rebalance frequently, when the > source cluster group is many more and changes frequently (but the list of > configured synchronous group does not change) > > > Key: KAFKA-14837 > URL: https://issues.apache.org/jira/browse/KAFKA-14837 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect, mirrormaker >Affects Versions: 3.3.2 >Reporter: hudeqi >Assignee: hudeqi >Priority: Major > Fix For: 3.5.0, 3.4.1, 3.3.3 > > > In practice, I found that when I configure a mirror checkpoint connector, > because the source cluster has a large number of group or the number of group > under a topic changes frequently, the connector will frequently rebalance > between its tasks, although there is no change in the synchronized group list > of the configuration. > I don't think connector should rebalance frequently in this case to affect > group synchronization tasks without any group changes. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14842) MirrorCheckpointTask can reduce the rpc calls of "listConsumerGroupOffsets(group)" of irrelevant groups at each poll
[ https://issues.apache.org/jira/browse/KAFKA-14842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-14842: -- Fix Version/s: 3.4.1 3.3.3 > MirrorCheckpointTask can reduce the rpc calls of > "listConsumerGroupOffsets(group)" of irrelevant groups at each poll > > > Key: KAFKA-14842 > URL: https://issues.apache.org/jira/browse/KAFKA-14842 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect, mirrormaker >Affects Versions: 3.3.2 >Reporter: hudeqi >Assignee: hudeqi >Priority: Major > Fix For: 3.5.0, 3.4.1, 3.3.3 > > > sorry, wrong related. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14978) ExactlyOnceWorkerSourceTask does not remove parent metrics
[ https://issues.apache.org/jira/browse/KAFKA-14978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-14978: -- Fix Version/s: 3.4.1 > ExactlyOnceWorkerSourceTask does not remove parent metrics > -- > > Key: KAFKA-14978 > URL: https://issues.apache.org/jira/browse/KAFKA-14978 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Daniel Urban >Assignee: Daniel Urban >Priority: Major > Fix For: 3.4.1, 3.6.0 > > > ExactlyOnceWorkerSourceTask removeMetrics does not invoke > super.removeMetrics, meaning that only the transactional metrics are removed, > and common source task metrics are not. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-14980) MirrorMaker consumers don't get configs prefixed with source.cluster
[
https://issues.apache.org/jira/browse/KAFKA-14980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton reassigned KAFKA-14980:
-
Assignee: Chris Egerton
> MirrorMaker consumers don't get configs prefixed with source.cluster
>
>
> Key: KAFKA-14980
> URL: https://issues.apache.org/jira/browse/KAFKA-14980
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.5.0
>Reporter: Mickael Maison
>Assignee: Chris Egerton
>Priority: Blocker
> Fix For: 3.5.0
>
>
> As part of KAFKA-14021, we made a change to
> MirrorConnectorConfig.sourceConsumerConfig() to grab all configs that start
> with "source.". Previously it was grabbing configs prefixed with
> "source.cluster.".
> This means existing connector configuration stop working, as configurations
> such as bootstrap.servers are not passed to source consumers.
> For example, the following connector configuration was valid in 3.4 and now
> makes the connector tasks fail:
> {code:json}
> {
> "connector.class":
> "org.apache.kafka.connect.mirror.MirrorSourceConnector",
> "name": "source",
> "topics": "test",
> "tasks.max": "30",
> "source.cluster.alias": "one",
> "target.cluster.alias": "two",
> "source.cluster.bootstrap.servers": "localhost:9092",
>"target.cluster.bootstrap.servers": "localhost:29092"
> }
> {code}
> The connector attempts to start source consumers with bootstrap.servers = []
> and the task crash with
> {noformat}
> org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:837)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:671)
> at
> org.apache.kafka.connect.mirror.MirrorUtils.newConsumer(MirrorUtils.java:59)
> at
> org.apache.kafka.connect.mirror.MirrorSourceTask.start(MirrorSourceTask.java:103)
> at
> org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.initializeAndStart(AbstractWorkerSourceTask.java:274)
> at
> org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:202)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259)
> at
> org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:75)
> at
> org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:181)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: org.apache.kafka.common.config.ConfigException: No resolvable
> bootstrap urls given in bootstrap.servers
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-15018) Potential tombstone offsets corruption for exactly-once source connectors
Chris Egerton created KAFKA-15018: - Summary: Potential tombstone offsets corruption for exactly-once source connectors Key: KAFKA-15018 URL: https://issues.apache.org/jira/browse/KAFKA-15018 Project: Kafka Issue Type: Bug Components: KafkaConnect Affects Versions: 3.3.2, 3.3.1, 3.4.0, 3.3.0, 3.5.0, 3.4.1 Reporter: Chris Egerton When exactly-once support is enabled for source connectors, source offsets can potentially be written to two different offsets topics: a topic specific to the connector, and the global offsets topic (which was used for all connectors prior to KIP-618 / version 3.3.0). Precedence is given to offsets in the per-connector offsets topic, but if none are found for a given partition, then the global offsets topic is used as a fallback. When committing offsets, a transaction is used to ensure that source records and source offsets are written to the Kafka cluster targeted by the source connector. This transaction only includes the connector-specific offsets topic. Writes to the global offsets topic take place after writes to the connector-specific offsets topic have completed successfully, and if they fail, a warning message is logged, but no other action is taken. Normally, this ensures that, for offsets committed by exactly-once-supported source connectors, the per-connector offsets topic is at least as up-to-date as the global offsets topic, and sometimes even ahead. However, for tombstone offsets, we lose that guarantee. If a tombstone offset is successfully written to the per-connector offsets topic, but cannot be written to the global offsets topic, then the global offsets topic will still contain that source offset, but the per-connector topic will not. Due to the fallback-on-global logic used by the worker, if a task requests offsets for one of the tombstoned partitions, the worker will provide it with the offsets present in the global offsets topic, instead of indicating to the task that no offsets can be found. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15018) Potential tombstone offsets corruption for exactly-once source connectors
[ https://issues.apache.org/jira/browse/KAFKA-15018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17725502#comment-17725502 ] Chris Egerton commented on KAFKA-15018: --- One possible fix for this could be to preemptively write tombstone offsets to the global offsets topic before writing any offsets to the per-connector offsets topic, and preserve the existing write logic for all non-tombstone offsets. Tombstone offsets should be fairly rare and so in the common case, this will have no impact on connector performance or availability. However, when this case is hit, the proposed fix would require two synchronous writes to topics that are potentially hosted on different clusters. This is not ideal, but it's unclear whether a better alternative exists. > Potential tombstone offsets corruption for exactly-once source connectors > - > > Key: KAFKA-15018 > URL: https://issues.apache.org/jira/browse/KAFKA-15018 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 3.3.0, 3.4.0, 3.3.1, 3.3.2, 3.5.0, 3.4.1 >Reporter: Chris Egerton >Priority: Major > > When exactly-once support is enabled for source connectors, source offsets > can potentially be written to two different offsets topics: a topic specific > to the connector, and the global offsets topic (which was used for all > connectors prior to KIP-618 / version 3.3.0). > Precedence is given to offsets in the per-connector offsets topic, but if > none are found for a given partition, then the global offsets topic is used > as a fallback. > When committing offsets, a transaction is used to ensure that source records > and source offsets are written to the Kafka cluster targeted by the source > connector. This transaction only includes the connector-specific offsets > topic. Writes to the global offsets topic take place after writes to the > connector-specific offsets topic have completed successfully, and if they > fail, a warning message is logged, but no other action is taken. > Normally, this ensures that, for offsets committed by exactly-once-supported > source connectors, the per-connector offsets topic is at least as up-to-date > as the global offsets topic, and sometimes even ahead. > However, for tombstone offsets, we lose that guarantee. If a tombstone offset > is successfully written to the per-connector offsets topic, but cannot be > written to the global offsets topic, then the global offsets topic will still > contain that source offset, but the per-connector topic will not. Due to the > fallback-on-global logic used by the worker, if a task requests offsets for > one of the tombstoned partitions, the worker will provide it with the offsets > present in the global offsets topic, instead of indicating to the task that > no offsets can be found. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15012) JsonConverter fails when there are leading Zeros in a field
[
https://issues.apache.org/jira/browse/KAFKA-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17727631#comment-17727631
]
Chris Egerton commented on KAFKA-15012:
---
[~yash.mayya] I'm wondering if compatibility is necessarily a concern here. The
logic this ticket is concerned with is deserialization of data already present
in Kafka, and converting it to the Kafka Connect format for use by a sink
connector. There will be nothing different about the data given to those
connectors depending on whether the JSON in the upstream Kafka record's
key/value contains a leading zero or not.
As a result, it's hard to think of a reason for those records in Kafka to
qualify as "invalid" and for there to be any reason for them to land in the DLQ
besides there being a bug in the converter.
If we agree that this is a bug in the converter, then even if it causes records
to be sent to the DLQ, that is a result of the DLQ mechanism being a catch-all
for errors–expected or unexpected–that occur in the connector's data pipeline,
and fixing those errors should not be considered a breaking change as long as
they do not lead to unexpected behavior in the connector (which, in this case,
should be fine).
[~ranjanrao] Thanks for filing this ticket. We don't accept patch requests over
Jira, but if you'd like to submit a pull request on GitHub (preferably with a
unit test added to verify behavior and prevent regression), I'd be happy to
review (as long as the discussion on compatibility brought up by Yash can be
addressed).
I'll also note that the idea of a general-purpose KIP to allow users to
configure arbitrary features for the JsonConverter's underlying (de)serializers
is fascinating and may be worth pursuing if there are other valuable use cases
(perhaps skipping over comments could be useful, for example?).
> JsonConverter fails when there are leading Zeros in a field
> ---
>
> Key: KAFKA-15012
> URL: https://issues.apache.org/jira/browse/KAFKA-15012
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 3.4.0, 3.3.2
>Reporter: Ranjan Rao
>Priority: Major
> Attachments:
> enable_ALLOW_LEADING_ZEROS_FOR_NUMBERS_in_jackson_object_mapper_.patch
>
>
> When there are leading zeros in a field in the Kakfa Record, a sink connector
> using JsonConverter fails with the below exception
>
> {code:java}
> org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error
> handler
> at
> org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:206)
> at
> org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:132)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertAndTransformRecord(WorkerSinkTask.java:494)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:474)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:329)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:232)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:201)
> at
> org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:188)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:237)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: org.apache.kafka.connect.errors.DataException: Converting byte[]
> to Kafka Connect data failed due to serialization error:
> at
> org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:324)
> at
> org.apache.kafka.connect.storage.Converter.toConnectData(Converter.java:87)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertKey(WorkerSinkTask.java:531)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.lambda$convertAndTransformRecord$1(WorkerSinkTask.java:494)
> at
> org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:156)
> at
> org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:190)
> ... 13 more
> Caused by: org.apache.kafka.common.errors.SerializationException:
> com.fasterxml.jackson
[jira] [Resolved] (KAFKA-14863) Plugins which do not have a valid no-args constructor are visible in the REST API
[ https://issues.apache.org/jira/browse/KAFKA-14863?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-14863. --- Fix Version/s: 3.6.0 Resolution: Fixed > Plugins which do not have a valid no-args constructor are visible in the REST > API > - > > Key: KAFKA-14863 > URL: https://issues.apache.org/jira/browse/KAFKA-14863 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Greg Harris >Assignee: Greg Harris >Priority: Minor > Fix For: 3.6.0 > > > Currently, the Connect plugin discovery mechanisms only assert that a no-args > constructor is present when necessary. In particular, this assertion happens > for Connectors when the framework needs to evaluate the connector's version > method. > It also happens for ConnectorConfigOverridePolicy, ConnectRestExtension, and > ConfigProvider plugins, which are loaded via the ServiceLoader. The > ServiceLoader constructs instances of plugins with their no-args constructor > during discovery, so these plugins are discovered even if they are not > Versioned. > This has the effect that these unusable plugins which are missing a default > constructor appear in the REST API, but are not able to be instantiated or > used. To make the ServiceLoader and Reflections discovery mechanisms behave > more similar, this assertion should be applied to all plugins, and a log > message emitted when plugins do not follow the constructor requirements. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15012) JsonConverter fails when there are leading Zeros in a field
[
https://issues.apache.org/jira/browse/KAFKA-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton resolved KAFKA-15012.
---
Fix Version/s: 3.6.0
Resolution: Fixed
> JsonConverter fails when there are leading Zeros in a field
> ---
>
> Key: KAFKA-15012
> URL: https://issues.apache.org/jira/browse/KAFKA-15012
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 3.4.0, 3.3.2
>Reporter: Ranjan Rao
>Assignee: Yash Mayya
>Priority: Major
> Fix For: 3.6.0
>
> Attachments:
> enable_ALLOW_LEADING_ZEROS_FOR_NUMBERS_in_jackson_object_mapper_.patch
>
>
> When there are leading zeros in a field in the Kakfa Record, a sink connector
> using JsonConverter fails with the below exception
>
> {code:java}
> org.apache.kafka.connect.errors.ConnectException: Tolerance exceeded in error
> handler
> at
> org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:206)
> at
> org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execute(RetryWithToleranceOperator.java:132)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertAndTransformRecord(WorkerSinkTask.java:494)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertMessages(WorkerSinkTask.java:474)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:329)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:232)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:201)
> at
> org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:188)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:237)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: org.apache.kafka.connect.errors.DataException: Converting byte[]
> to Kafka Connect data failed due to serialization error:
> at
> org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:324)
> at
> org.apache.kafka.connect.storage.Converter.toConnectData(Converter.java:87)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.convertKey(WorkerSinkTask.java:531)
> at
> org.apache.kafka.connect.runtime.WorkerSinkTask.lambda$convertAndTransformRecord$1(WorkerSinkTask.java:494)
> at
> org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndRetry(RetryWithToleranceOperator.java:156)
> at
> org.apache.kafka.connect.runtime.errors.RetryWithToleranceOperator.execAndHandleError(RetryWithToleranceOperator.java:190)
> ... 13 more
> Caused by: org.apache.kafka.common.errors.SerializationException:
> com.fasterxml.jackson.core.JsonParseException: Invalid numeric value: Leading
> zeroes not allowed
> at [Source: (byte[])"00080153032837"; line: 1, column: 2]
> Caused by: com.fasterxml.jackson.core.JsonParseException: Invalid numeric
> value: Leading zeroes not allowed
> at [Source: (byte[])"00080153032837"; line: 1, column: 2]
> at
> com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1840)
> at
> com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:712)
> at
> com.fasterxml.jackson.core.base.ParserMinimalBase.reportInvalidNumber(ParserMinimalBase.java:551)
> at
> com.fasterxml.jackson.core.json.UTF8StreamJsonParser._verifyNoLeadingZeroes(UTF8StreamJsonParser.java:1520)
> at
> com.fasterxml.jackson.core.json.UTF8StreamJsonParser._parsePosNumber(UTF8StreamJsonParser.java:1372)
> at
> com.fasterxml.jackson.core.json.UTF8StreamJsonParser._nextTokenNotInObject(UTF8StreamJsonParser.java:855)
> at
> com.fasterxml.jackson.core.json.UTF8StreamJsonParser.nextToken(UTF8StreamJsonParser.java:754)
> at
> com.fasterxml.jackson.databind.ObjectMapper._readTreeAndClose(ObjectMapper.java:4247)
> at
> com.fasterxml.jackson.databind.ObjectMapper.readTree(ObjectMapper.java:2734)
> at
> org.apache.kafka.connect.json.JsonDeserializer.deserialize(JsonDeserializer.java:64)
> at
> org.apache.kafka.connect.json.JsonConverter.toConnectData(JsonConverter.java:322)
> at
> org.apache.kafka.connect.storage.Converter.toConnectData(Converter.java:87)
> at
> org.apache.kafka.connect.runtime.WorkerSin
[jira] [Commented] (KAFKA-15053) Regression for security.protocol validation starting from 3.3.0
[
https://issues.apache.org/jira/browse/KAFKA-15053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17729323#comment-17729323
]
Chris Egerton commented on KAFKA-15053:
---
Thanks [~dlgaobo]. I agree that this is a regression and we should re-introduce
case-insensitive support for this this property.
Are you interested in providing a patch for this?
> Regression for security.protocol validation starting from 3.3.0
> ---
>
> Key: KAFKA-15053
> URL: https://issues.apache.org/jira/browse/KAFKA-15053
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 3.3.0
>Reporter: Bo Gao
>Priority: Major
>
> [This|https://issues.apache.org/jira/browse/KAFKA-13793] Jira issue
> introduced validations on multiple configs. As a consequence, config
> {{security.protocol}} now only allows upper case values such as PLAINTEXT,
> SSL, SASL_PLAINTEXT, SASL_SSL. Before this change, lower case values like
> sasl_ssl, ssl are also supported, there's even a case insensitive logic
> inside
> [SecurityProtocol|https://github.com/apache/kafka/blob/146a6976aed0d9f90c70b6f21dca8b887cc34e71/clients/src/main/java/org/apache/kafka/common/security/auth/SecurityProtocol.java#L70-L73]
> to handle the lower case values.
> I think we should treat this as a regression bug since we don't support lower
> case values anymore since 3.3.0. For versions later than 3.3.0, we are
> getting error like this when using lower case value sasl_ssl
> {{Invalid value sasl_ssl for configuration security.protocol: String must be
> one of: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-15051) docs: add missing connector plugin endpoint to documentation
[ https://issues.apache.org/jira/browse/KAFKA-15051?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton reassigned KAFKA-15051: - Assignee: Jorge Esteban Quilcate Otoya > docs: add missing connector plugin endpoint to documentation > > > Key: KAFKA-15051 > URL: https://issues.apache.org/jira/browse/KAFKA-15051 > Project: Kafka > Issue Type: Task > Components: docs, documentation >Reporter: Jorge Esteban Quilcate Otoya >Assignee: Jorge Esteban Quilcate Otoya >Priority: Minor > > GET /plugin/config endpoint added in > [KIP-769|https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions] > is not included in the connect documentation page: > https://kafka.apache.org/documentation/#connect_rest -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15059) Exactly-once source tasks fail to start during pending rebalances
Chris Egerton created KAFKA-15059: - Summary: Exactly-once source tasks fail to start during pending rebalances Key: KAFKA-15059 URL: https://issues.apache.org/jira/browse/KAFKA-15059 Project: Kafka Issue Type: Bug Components: KafkaConnect, mirrormaker Affects Versions: 3.3.2, 3.3.1, 3.4.0, 3.3.0, 3.5.0, 3.4.1 Reporter: Chris Egerton Assignee: Chris Egerton When asked to perform a round of zombie fencing, the distributed herder will [reject the request|https://github.com/apache/kafka/blob/17fd30e6b457f097f6a524b516eca1a6a74a9144/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1249-L1250] if a rebalance is pending, which can happen if (among other things) a config for a new connector or a new set of task configs has been recently read from the config topic. Normally this can be alleviated with a simple task restart, which isn't great but isn't terrible. However, when running MirrorMaker 2 in dedicated mode, there is no API to restart failed tasks, and it can be more common to see this kind of failure on a fresh cluster because three connector configurations are written in rapid succession to the config topic. In order to provide a better experience for users of both vanilla Kafka Connect and dedicated MirrorMaker 2 clusters, we can retry (likely with the same exponential backoff introduced with KAFKA-14732) zombie fencing attempts that fail due to a pending rebalance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14718) Flaky DedicatedMirrorIntegrationTest test suite
[
https://issues.apache.org/jira/browse/KAFKA-14718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17729486#comment-17729486
]
Chris Egerton commented on KAFKA-14718:
---
I think a 30-second startup timeout is a bit too conservative for our CI, so
bumping the timeout to 60 or even 120 seconds may help.
On top of that, I've also found an actual issue that may be causing some of
these flaky test failures. Details in KAFKA-15059.
> Flaky DedicatedMirrorIntegrationTest test suite
> ---
>
> Key: KAFKA-14718
> URL: https://issues.apache.org/jira/browse/KAFKA-14718
> Project: Kafka
> Issue Type: Test
> Components: mirrormaker
>Reporter: Chris Egerton
>Assignee: Divij Vaidya
>Priority: Major
> Labels: flaky-test
> Fix For: 3.6.0
>
>
> These tests were added recently in
> [https://github.com/apache/kafka/pull/13137] and have been failing
> occasionally on Jenkins. For example, in
> [https://github.com/apache/kafka/pull/13163|https://github.com/apache/kafka/pull/13163:],
> both test cases (testSingleNodeCluster and testMultiNodeCluster) failed on a
> [single Jenkins
> node|https://ci-builds.apache.org/blue/organizations/jenkins/Kafka%2Fkafka-pr/detail/PR-13163/4/pipeline/11/]
> with timeout errors:
> {quote}[2023-02-14T16:43:19.054Z] Gradle Test Run
> :connect:mirror:integrationTest > Gradle Test Executor 155 >
> DedicatedMirrorIntegrationTest > testSingleNodeCluster() FAILED
> [2023-02-14T16:43:19.054Z] org.opentest4j.AssertionFailedError: Condition not
> met within timeout 3. topic A.test-topic-1 was not created on cluster B
> in time ==> expected: but was: [2023-02-14T16:43:19.054Z] at
> org.junit.jupiter.api.AssertionFailureBuilder.build(AssertionFailureBuilder.java:151)
> [2023-02-14T16:43:19.054Z] at
> org.junit.jupiter.api.AssertionFailureBuilder.buildAndThrow(AssertionFailureBuilder.java:132)
> [2023-02-14T16:43:19.054Z] at
> org.junit.jupiter.api.AssertTrue.failNotTrue(AssertTrue.java:63)
> [2023-02-14T16:43:19.054Z] at
> org.junit.jupiter.api.AssertTrue.assertTrue(AssertTrue.java:36)
> [2023-02-14T16:43:19.054Z] at
> org.junit.jupiter.api.Assertions.assertTrue(Assertions.java:211)
> [2023-02-14T16:43:19.054Z] at
> org.apache.kafka.test.TestUtils.lambda$waitForCondition$4(TestUtils.java:337)
> [2023-02-14T16:43:19.054Z] at
> org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:385)
> [2023-02-14T16:43:19.054Z] at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:334)
> [2023-02-14T16:43:19.054Z] at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:318)
> [2023-02-14T16:43:19.054Z] at
> org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:308)
> [2023-02-14T16:43:19.054Z] at
> org.apache.kafka.connect.mirror.integration.DedicatedMirrorIntegrationTest.awaitTopicCreation(DedicatedMirrorIntegrationTest.java:255)
> [2023-02-14T16:43:19.054Z] at
> org.apache.kafka.connect.mirror.integration.DedicatedMirrorIntegrationTest.testSingleNodeCluster(DedicatedMirrorIntegrationTest.java:153){quote}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15059) Exactly-once source tasks fail to start during pending rebalances
[ https://issues.apache.org/jira/browse/KAFKA-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17729709#comment-17729709 ] Chris Egerton commented on KAFKA-15059: --- On second thought, it may be unnecessary to check for a pending rebalance at all. If the worker that we forward the zombie fencing request to is a zombie leader (i.e., a worker that believes it is the leader but in reality is not), it will fail to finish the round of zombie fencing because it won't be able to write to the config topic with a transactional producer. If the connector has just been deleted, we'll still fail the request since we force a read-to-end of the config topic and refresh our snapshot of its contents before checking to see if the connector exists. And regardless, the worker that owns the task will still do a read-to-end of the config topic and verify that (1) no new task configs have been generated for the connector and (2) the worker is still assigned the connector, before allowing the task to process any data. > Exactly-once source tasks fail to start during pending rebalances > - > > Key: KAFKA-15059 > URL: https://issues.apache.org/jira/browse/KAFKA-15059 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect, mirrormaker >Affects Versions: 3.3.0, 3.4.0, 3.3.1, 3.3.2, 3.5.0, 3.4.1 >Reporter: Chris Egerton >Assignee: Chris Egerton >Priority: Major > > When asked to perform a round of zombie fencing, the distributed herder will > [reject the > request|https://github.com/apache/kafka/blob/17fd30e6b457f097f6a524b516eca1a6a74a9144/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1249-L1250] > if a rebalance is pending, which can happen if (among other things) a config > for a new connector or a new set of task configs has been recently read from > the config topic. > Normally this can be alleviated with a simple task restart, which isn't great > but isn't terrible. > However, when running MirrorMaker 2 in dedicated mode, there is no API to > restart failed tasks, and it can be more common to see this kind of failure > on a fresh cluster because three connector configurations are written in > rapid succession to the config topic. > > In order to provide a better experience for users of both vanilla Kafka > Connect and dedicated MirrorMaker 2 clusters, we can retry (likely with the > same exponential backoff introduced with KAFKA-14732) zombie fencing attempts > that fail due to a pending rebalance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-12857) Using Connect Sink with CooperativeStickyAssignor results in commit offsets failure
[
https://issues.apache.org/jira/browse/KAFKA-12857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton resolved KAFKA-12857.
---
Assignee: (was: dgd_contributor)
Resolution: Duplicate
> Using Connect Sink with CooperativeStickyAssignor results in commit offsets
> failure
> ---
>
> Key: KAFKA-12857
> URL: https://issues.apache.org/jira/browse/KAFKA-12857
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.7.1
> Environment: Linux
>Reporter: Oliver Hsu
>Priority: Major
>
> We are attempting to use a Kafka Connect Sink Connector with
> {{CooperativeStickyAssignor}} assignment strategy. When we use
> {{CooperativeStickyAssignor}} offset commits sometimes fail with
> {{[2021-05-26 22:03:36,435] WARN WorkerSinkTask\{id=sink-connector-7}
> Ignoring invalid task provided offset
> mytopic-0/OffsetAndMetadata\{offset=16305575, leaderEpoch=null, metadata=''}
> – partition not assigned, assignment=[mytopic-0]
> (org.apache.kafka.connect.runtime.WorkerSinkTask:434)}}
> Note that the invalid partition in the warning message matches the partition
> assignment.
> *Config changes*
> {{consumer.partition.assignment.strategy=org.apache.kafka.clients.consumer.CooperativeStickyAssignor}}
> *Cooperative vs Eager Assignment Strategy background*
>
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-429%3A+Kafka+Consumer+Incremental+Rebalance+Protocol#KIP429:KafkaConsumerIncrementalRebalanceProtocol-ConsumerRebalanceListenerandConsumerPartitionAssignorSemantics]
> With eager assignment:
> {quote}Listener#onPartitionsAssigned: called on the full set of assigned
> partitions (may have overlap with the partitions passed to
> #onPartitionsRevoked
> {quote}
> With cooperative assignment:
> {quote}Listener#onPartitionsAssigned: called on the subset of assigned
> partitions that were not previously owned before this rebalance. There should
> be no overlap with the revoked partitions (if any). This will always be
> called, even if there are no new partitions being assigned to a given member.
> {quote}
> This means with cooperative assignment, `onPartitionsAssigned` may be called
> with a partial assignment or an empty collection.
> However, the
> [WorkerSinkTask.HandleRebalance|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L669-L680]
> class makes the assumption that `onPartitionsAssigned` is called with the
> full set of assigned partitions which is true for eager but not coooperative.
> {code:java|title=WorkerSinkTask.HandleRebalance.java|borderStyle=solid}
> public void onPartitionsAssigned(Collection
> partitions) {
> log.debug("{} Partitions assigned {}", WorkerSinkTask.this,
> partitions);
> lastCommittedOffsets = new HashMap<>();
> currentOffsets = new HashMap<>();
> for (TopicPartition tp : partitions) {
> long pos = consumer.position(tp);
> lastCommittedOffsets.put(tp, new OffsetAndMetadata(pos));
> currentOffsets.put(tp, new OffsetAndMetadata(pos));
> log.debug("{} Assigned topic partition {} with offset {}",
> WorkerSinkTask.this, tp, pos);
> }
> {code}
> The {{onPartitionsAssigned}} creates a new empty {{HashMap}} and puts the
> offsets of the {{partitions}} in that {{HashMap}}.
> In the logs we see
> {{[2021-05-26 22:02:09,785] DEBUG WorkerSinkTask\{id=sink-connector-7}
> Partitions assigned [myTopic-0]
> (org.apache.kafka.connect.runtime.WorkerSinkTask:677)}}
> {{[2021-05-26 22:02:13,063] DEBUG WorkerSinkTask\{id=sink-connector-7}
> Partitions assigned [] (org.apache.kafka.connect.runtime.WorkerSinkTask:677)}}
> {{[2021-05-26 22:02:16,074] DEBUG WorkerSinkTask\{id=sink-connector-7} }}
> Partitions assigned [] (org.apache.kafka.connect.runtime.WorkerSinkTask:677)}}
> These logs show that the {{CooperativeStickyAssignor}} calls
> {{onPartitionsAssigned}} first with the partition assigned to it followed by
> additional calls with an empty {{partitions}} collection.
> When {{HandleRebalance.onPartitionsAssigned}} is called first with the
> assigned partition followed by empty collections, the result will be
> {{lastCommittedOffsets}} initialized to an empty {{HashMap}}.
> Inside
> [WorkerSinkTask.commitOffsets|https://github.com/apache/kafka/blob/trunk/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerSinkTask.java#L415-L419],
> the current {{committableOffsets}} are based on the
> {{lastCommittedOffsets}}, which is an empty {{HashMap}}:
> {code:java|title=WorkerSinkTask.java|borderStyle=solid}
> private void commitOffsets(long now, boolean closing) {
> ...
>
[jira] [Updated] (KAFKA-12226) High-throughput source tasks fail to commit offsets
[
https://issues.apache.org/jira/browse/KAFKA-12226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton updated KAFKA-12226:
--
Description:
The current source task thread has the following workflow:
# Poll messages from the source task
# Queue these messages to the producer and send them to Kafka asynchronously.
# Add the message to outstandingMessages, or if a flush is currently active,
outstandingMessagesBacklog
# When the producer completes the send of a record, remove it from
outstandingMessages
The commit offsets thread has the following workflow:
# Wait a flat timeout for outstandingMessages to flush completely
# If this times out, add all of the outstandingMessagesBacklog to the
outstandingMessages and reset
# If it succeeds, commit the source task offsets to the backing store.
# Retry the above on a fixed schedule
If the source task is producing records quickly (faster than the producer can
send), then the producer will throttle the task thread by blocking in its
{{send}} method, waiting at most {{max.block.ms}} for space in the
{{buffer.memory}} to be available. This means that the number of records in
{{outstandingMessages}} + {{outstandingMessagesBacklog}} is proportional to the
size of the producer memory buffer.
This amount of data might take more than {{offset.flush.timeout.ms}} to flush,
and thus the flush will never succeed while the source task is rate-limited by
the producer memory. This means that we may write multiple hours of data to
Kafka and not ever commit source offsets for the connector. When the task is
lost due to a worker failure, hours of data will be re-processed that otherwise
were successfully written to Kafka.
was:
The current source task thread has the following workflow:
# Poll messages from the source task
# Queue these messages to the producer and send them to Kafka asynchronously.
# Add the message to outstandingMessages, or if a flush is currently active,
outstandingMessagesBacklog
# When the producer completes the send of a record, remove it from
outstandingMessages
The commit offsets thread has the following workflow:
# Wait a flat timeout for outstandingMessages to flush completely
# If this times out, add all of the outstandingMessagesBacklog to the
outstandingMessages and reset
# If it succeeds, commit the source task offsets to the backing store.
# Retry the above on a fixed schedule
If the source task is producing records quickly (faster than the producer can
send), then the producer will throttle the task thread by blocking in its
{{send}} method, waiting at most {{max.block.ms}} for space in the
{{buffer.memory}} to be available. This means that the number of records in
{{outstandingMessages}} + {{outstandingMessagesBacklog}} is proportional to the
size of the producer memory buffer.
This amount of data might take more than {{offset.flush.timeout.ms}} to flush,
and thus the flush will never succeed while the source task is rate-limited by
the producer memory. This means that we may write multiple hours of data to
Kafka and not ever commit source offsets for the connector. When the task is
lost due to a worker failure, hours of data will be re-processed that otherwise
were successfully written to Kafka.
> High-throughput source tasks fail to commit offsets
> ---
>
> Key: KAFKA-12226
> URL: https://issues.apache.org/jira/browse/KAFKA-12226
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Major
>
> The current source task thread has the following workflow:
> # Poll messages from the source task
> # Queue these messages to the producer and send them to Kafka asynchronously.
> # Add the message to outstandingMessages, or if a flush is currently active,
> outstandingMessagesBacklog
> # When the producer completes the send of a record, remove it from
> outstandingMessages
> The commit offsets thread has the following workflow:
> # Wait a flat timeout for outstandingMessages to flush completely
> # If this times out, add all of the outstandingMessagesBacklog to the
> outstandingMessages and reset
> # If it succeeds, commit the source task offsets to the backing store.
> # Retry the above on a fixed schedule
> If the source task is producing records quickly (faster than the producer can
> send), then the producer will throttle the task thread by blocking in its
> {{send}} method, waiting at most {{max.block.ms}} for space in the
> {{buffer.memory}} to be available. This means that the number of records in
> {{outstandingMessages}} + {{outstandingMessagesBacklog}} is proportional to
> the size of the producer memory buffer.
> This amount of data might take more than {{offset.flush.timeout.ms}} to
> fl
[jira] [Assigned] (KAFKA-9228) Reconfigured converters and clients may not be propagated to connector tasks
[
https://issues.apache.org/jira/browse/KAFKA-9228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton reassigned KAFKA-9228:
Assignee: (was: Chris Egerton)
> Reconfigured converters and clients may not be propagated to connector tasks
>
>
> Key: KAFKA-9228
> URL: https://issues.apache.org/jira/browse/KAFKA-9228
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Affects Versions: 2.3.0, 2.4.0, 2.3.1, 2.3.2
>Reporter: Chris Egerton
>Priority: Major
>
> If an existing connector is reconfigured but the only changes are to its
> converters and/or Kafka clients (enabled as of
> [KIP-458|https://cwiki.apache.org/confluence/display/KAFKA/KIP-458%3A+Connector+Client+Config+Override+Policy]),
> the changes will not propagate to its tasks unless the connector also
> generates task configs that differ from the existing task configs. Even after
> this point, if the connector tasks are reconfigured, they will still not pick
> up on the new converter and/or Kafka client configs.
> This is because the {{DistributedHerder}} only writes new task configurations
> to the connect config topic [if the connector-provided task configs differ
> from the task configs already in the config
> topic|https://github.com/apache/kafka/blob/e499c960e4f9cfc462f1a05a110d79ffa1c5b322/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1285-L1332],
> and neither of those contain converter or Kafka client configs.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-13327) Preflight validations of connectors leads to 500 responses
Chris Egerton created KAFKA-13327:
-
Summary: Preflight validations of connectors leads to 500 responses
Key: KAFKA-13327
URL: https://issues.apache.org/jira/browse/KAFKA-13327
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Reporter: Chris Egerton
Assignee: Chris Egerton
The Connect framework performs some preflight validations for all connectors
that are created in addition to allowing connectors to define their own custom
validation logic by providing a {{ConfigDef}} object in
[Connector::config|https://kafka.apache.org/30/javadoc/org/apache/kafka/connect/connector/Connector.html#config()]
and performing multi-property validation in
[Connector::validate|https://kafka.apache.org/30/javadoc/org/apache/kafka/connect/connector/Connector.html#validate(java.util.Map)].
When performed correctly, this validation information is surfaced to the user
in the form of a
[ConfigInfos|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/entities/ConfigInfos.java]
object containing a list of [config
objects|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/entities/ConfigInfo.java#L42-L45]
whose
[values|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/entities/ConfigInfo.java#L42-L45]
contain one or more [error
messages|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/entities/ConfigValueInfo.java#L61-L64].
This can be used as the response for a REST request to PUT
/connector-plugins/\{connectorType}/config/validate and allows programmatic UIs
to render error messages for every invalid property to the user.
However, some validations performed by the Connect framework do not follow this
pattern and instead result in a 500 response being returned to the user. For
example, logic specific to sink connectors (see
[AbstractHerder|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractHerder.java#L436]
and
[SinkConnectorConfig|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/SinkConnectorConfig.java#L88-L125])
simply throws an exception instead of documenting the error with the offending
property and returning it in a standard response.
We should correct this logic wherever possible so that configurations that are
not fatally invalid (i.e., may have invalid properties but can still be
translated into a meaningful {{ConfigInfos}} response object) do not cause a
500 response to be returned to the user.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-13328) Preflight validation of header converters is not performed by Connect
Chris Egerton created KAFKA-13328:
-
Summary: Preflight validation of header converters is not
performed by Connect
Key: KAFKA-13328
URL: https://issues.apache.org/jira/browse/KAFKA-13328
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Reporter: Chris Egerton
Assignee: Chris Egerton
{{HeaderConverter}} implementations are required to provide a valid
{{ConfigDef}} to the Connect framework via
[HeaderConverter::config|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/api/src/main/java/org/apache/kafka/connect/storage/HeaderConverter.java#L48-L52],
but this object isn't actually leveraged anywhere by Connect.
Connect should make use of this config object during preflight validation for
connectors to fail faster when their header converters are misconfigured.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-13328) Connect does not perform preflight validation for per-connector header converters
[
https://issues.apache.org/jira/browse/KAFKA-13328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton updated KAFKA-13328:
--
Summary: Connect does not perform preflight validation for per-connector
header converters (was: Preflight validation of header converters is not
performed by Connect)
> Connect does not perform preflight validation for per-connector header
> converters
> -
>
> Key: KAFKA-13328
> URL: https://issues.apache.org/jira/browse/KAFKA-13328
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Major
>
> {{HeaderConverter}} implementations are required to provide a valid
> {{ConfigDef}} to the Connect framework via
> [HeaderConverter::config|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/api/src/main/java/org/apache/kafka/connect/storage/HeaderConverter.java#L48-L52],
> but this object isn't actually leveraged anywhere by Connect.
> Connect should make use of this config object during preflight validation for
> connectors to fail faster when their header converters are misconfigured.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-13329) Connect does not perform preflight validation for per-connector key and value converters
Chris Egerton created KAFKA-13329:
-
Summary: Connect does not perform preflight validation for
per-connector key and value converters
Key: KAFKA-13329
URL: https://issues.apache.org/jira/browse/KAFKA-13329
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Reporter: Chris Egerton
Assignee: Chris Egerton
Users may specify a key and/or value converter class for their connector
directly in the configuration for that connector. If this occurs, no preflight
validation is performed to ensure that the specified converter is valid.
Unfortunately, the [Converter
interface|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/api/src/main/java/org/apache/kafka/connect/storage/Converter.java]
does not require converters to expose a {{ConfigDef}} (unlike the
[HeaderConverter
interface|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/api/src/main/java/org/apache/kafka/connect/storage/HeaderConverter.java#L48-L52],
which does have that requirement), so it's unlikely that the configuration
properties of the converter itself can be validated.
However, we can and should still validate that the converter class exists, can
be instantiated (i.e., has a public, no-args constructor and is a concrete,
non-abstract class), and implements the {{Converter}} interface.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KAFKA-13328) Connect does not perform preflight validation for per-connector header converters
[
https://issues.apache.org/jira/browse/KAFKA-13328?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton updated KAFKA-13328:
--
Description:
Users may specify a header converter class for their connector directly in the
configuration for that connector. If this occurs, no preflight validation is
performed to ensure that the specified converter is valid.
{{HeaderConverter}} implementations are required to provide a valid
{{ConfigDef}} to the Connect framework via
[HeaderConverter::config|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/api/src/main/java/org/apache/kafka/connect/storage/HeaderConverter.java#L48-L52],
but this object isn't actually leveraged anywhere by Connect.
Connect should make use of this config object during preflight validation for
connectors to fail faster when their header converters are misconfigured.
was:
{{HeaderConverter}} implementations are required to provide a valid
{{ConfigDef}} to the Connect framework via
[HeaderConverter::config|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/api/src/main/java/org/apache/kafka/connect/storage/HeaderConverter.java#L48-L52],
but this object isn't actually leveraged anywhere by Connect.
Connect should make use of this config object during preflight validation for
connectors to fail faster when their header converters are misconfigured.
> Connect does not perform preflight validation for per-connector header
> converters
> -
>
> Key: KAFKA-13328
> URL: https://issues.apache.org/jira/browse/KAFKA-13328
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Major
>
> Users may specify a header converter class for their connector directly in
> the configuration for that connector. If this occurs, no preflight validation
> is performed to ensure that the specified converter is valid.
> {{HeaderConverter}} implementations are required to provide a valid
> {{ConfigDef}} to the Connect framework via
> [HeaderConverter::config|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/api/src/main/java/org/apache/kafka/connect/storage/HeaderConverter.java#L48-L52],
> but this object isn't actually leveraged anywhere by Connect.
> Connect should make use of this config object during preflight validation for
> connectors to fail faster when their header converters are misconfigured.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-13329) Connect does not perform preflight validation for per-connector key and value converters
[
https://issues.apache.org/jira/browse/KAFKA-13329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17421507#comment-17421507
]
Chris Egerton commented on KAFKA-13329:
---
I agree that that'd be beneficial, although I think it'd require a KIP.
Thinking about Confluent's Avro converter, there are some useful preflight
validations that would be very useful but would require access to multiple
properties at a time (such as ensuring that the URL and credentials for Schema
Registry are valid), which is beyond the single-property-at-a-time scope of a
{{ConfigDef}} object.
If we're at the point of writing a KIP for this already, I wonder if we might
follow the same pattern for key/value converters as we do for connectors: a
[validate
method|https://github.com/apache/kafka/blob/5a5c05807ddc82b183f4489e81b9ad02fe83a0df/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146]
that, by default, simply delegates to a {{ConfigDef}} provided in a [config
method|https://github.com/apache/kafka/blob/5a5c05807ddc82b183f4489e81b9ad02fe83a0df/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L148-L152].
Obviously we'd have to make some small tweaks (default implementation for
{{config}} that returns {{null}}, permitting null {{ConfigDef}} objects in
{{validate}}), but overall it's been a very friendly API to use when writing
connectors (just took advantage of it in [a recent
PR|https://github.com/confluentinc/kafka-connect-bigquery/pull/153], in fact)
and I don't see why we wouldn't want to extend key converters, value
converters, and possibly even header converters, transformations, and
predicates in the same way.
> Connect does not perform preflight validation for per-connector key and value
> converters
>
>
> Key: KAFKA-13329
> URL: https://issues.apache.org/jira/browse/KAFKA-13329
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Major
>
> Users may specify a key and/or value converter class for their connector
> directly in the configuration for that connector. If this occurs, no
> preflight validation is performed to ensure that the specified converter is
> valid.
> Unfortunately, the [Converter
> interface|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/api/src/main/java/org/apache/kafka/connect/storage/Converter.java]
> does not require converters to expose a {{ConfigDef}} (unlike the
> [HeaderConverter
> interface|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/api/src/main/java/org/apache/kafka/connect/storage/HeaderConverter.java#L48-L52],
> which does have that requirement), so it's unlikely that the configuration
> properties of the converter itself can be validated.
> However, we can and should still validate that the converter class exists,
> can be instantiated (i.e., has a public, no-args constructor and is a
> concrete, non-abstract class), and implements the {{Converter}} interface.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KAFKA-13329) Connect does not perform preflight validation for per-connector key and value converters
[
https://issues.apache.org/jira/browse/KAFKA-13329?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17421507#comment-17421507
]
Chris Egerton edited comment on KAFKA-13329 at 10/1/21, 4:02 PM:
-
I agree that that'd be beneficial, although I think it'd require a KIP.
Thinking about Confluent's Avro converter, there are some preflight validations
that would be very useful but would require access to multiple properties at a
time (such as ensuring that the URL and credentials for Schema Registry are
valid), which is beyond the single-property-at-a-time scope of a {{ConfigDef}}
object.
If we're at the point of writing a KIP for this already, I wonder if we might
follow the same pattern for key/value converters as we do for connectors: a
[validate
method|https://github.com/apache/kafka/blob/5a5c05807ddc82b183f4489e81b9ad02fe83a0df/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146]
that, by default, simply delegates to a {{ConfigDef}} provided in a [config
method|https://github.com/apache/kafka/blob/5a5c05807ddc82b183f4489e81b9ad02fe83a0df/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L148-L152].
Obviously we'd have to make some small tweaks (default implementation for
{{config}} that returns {{null}}, permitting null {{ConfigDef}} objects in
{{validate}}), but overall it's been a very friendly API to use when writing
connectors (just took advantage of it in [a recent
PR|https://github.com/confluentinc/kafka-connect-bigquery/pull/153], in fact)
and I don't see why we wouldn't want to extend key converters, value
converters, and possibly even header converters, transformations, and
predicates in the same way.
was (Author: chrisegerton):
I agree that that'd be beneficial, although I think it'd require a KIP.
Thinking about Confluent's Avro converter, there are some useful preflight
validations that would be very useful but would require access to multiple
properties at a time (such as ensuring that the URL and credentials for Schema
Registry are valid), which is beyond the single-property-at-a-time scope of a
{{ConfigDef}} object.
If we're at the point of writing a KIP for this already, I wonder if we might
follow the same pattern for key/value converters as we do for connectors: a
[validate
method|https://github.com/apache/kafka/blob/5a5c05807ddc82b183f4489e81b9ad02fe83a0df/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L131-L146]
that, by default, simply delegates to a {{ConfigDef}} provided in a [config
method|https://github.com/apache/kafka/blob/5a5c05807ddc82b183f4489e81b9ad02fe83a0df/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java#L148-L152].
Obviously we'd have to make some small tweaks (default implementation for
{{config}} that returns {{null}}, permitting null {{ConfigDef}} objects in
{{validate}}), but overall it's been a very friendly API to use when writing
connectors (just took advantage of it in [a recent
PR|https://github.com/confluentinc/kafka-connect-bigquery/pull/153], in fact)
and I don't see why we wouldn't want to extend key converters, value
converters, and possibly even header converters, transformations, and
predicates in the same way.
> Connect does not perform preflight validation for per-connector key and value
> converters
>
>
> Key: KAFKA-13329
> URL: https://issues.apache.org/jira/browse/KAFKA-13329
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Major
>
> Users may specify a key and/or value converter class for their connector
> directly in the configuration for that connector. If this occurs, no
> preflight validation is performed to ensure that the specified converter is
> valid.
> Unfortunately, the [Converter
> interface|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/api/src/main/java/org/apache/kafka/connect/storage/Converter.java]
> does not require converters to expose a {{ConfigDef}} (unlike the
> [HeaderConverter
> interface|https://github.com/apache/kafka/blob/4eb386f6e060e12e1940c0d780987e3a7c438d74/connect/api/src/main/java/org/apache/kafka/connect/storage/HeaderConverter.java#L48-L52],
> which does have that requirement), so it's unlikely that the configuration
> properties of the converter itself can be validated.
> However, we can and should still validate that the converter class exists,
> can be instantiated (i.e., has a public, no-args constructor and is a
> concrete, non-abstract class), and implements the {{Converter}} interface.
--
This message was sent by Atlassian Jira
(v8.3.4#
[jira] [Commented] (KAFKA-10334) Transactions not working properly
[
https://issues.apache.org/jira/browse/KAFKA-10334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17444589#comment-17444589
]
Chris Egerton commented on KAFKA-10334:
---
It seems like this may be a duplicate of
https://issues.apache.org/jira/browse/KAFKA-9279, which also reports successful
calls to {{Producer::commitTransaction}} even though one or more records have
failed to send due to being too large.
> Transactions not working properly
> -
>
> Key: KAFKA-10334
> URL: https://issues.apache.org/jira/browse/KAFKA-10334
> Project: Kafka
> Issue Type: Bug
> Components: clients, producer
>Affects Versions: 2.1.0, 2.3.0
>Reporter: Luis Araujo
>Priority: Major
>
> I'm using transactions provided by Kafka Producer API in a Scala project
> built with SBT. The dependency used in the project is:
> {code:java}
> "org.apache.kafka" % "kafka-clients" % "2.1.0" {code}
> I followed the documentation and I was expecting that transactions fail when
> I call *.commitTransaction* if some problem is raised when sending a message
> like it's described in the
> [documentation|https://kafka.apache.org/10/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html#send-org.apache.kafka.clients.producer.ProducerRecord-org.apache.kafka.clients.producer.Callback-].
> Unfortunately, when testing this behaviour using a message larger than the
> size accepted by the Kafka broker/cluster, the transactions are not working
> properly.
> I tested with a 3 Kafka broker cluster with 1MB message max size (default
> value):
> - when the message has 1MB, the transaction is aborted and an exception is
> raised when calling *commitTransaction()*
> - when the message is bigger than 1MB, the transaction is completed
> successfully *without* the message being written. No exception is thrown.
> As an example, this means that when I produce 9 messages with 1 KB and 1
> message with 1.1MB in the same transaction, the transaction is completed but
> only 9 messages are written to the Kafka cluster.
> I tested this behaviour with Kafka version 2.1.0 and 2.3.0 in both Kafka
> cluster and Kafka Producer API.
> The configs that I'm using to create the KafkaProducer in order to use
> transactions:
> {code:java}
> new Properties() {
> {
> put(BOOTSTRAP_SERVERS_CONFIG,
> "localhost:29092,localhost:29093,localhost:29094")
> put(ACKS_CONFIG, "-1")
> put(MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION, "1")
> put(KEY_SERIALIZER_CLASS_CONFIG,
> Class.forName(classOf[StringSerializer].getName))
> put(VALUE_SERIALIZER_CLASS_CONFIG,
> Class.forName(classOf[ByteArraySerializer].getName))
> put(CLIENT_ID_CONFIG, "app")
> put(TRANSACTIONAL_ID_CONFIG, "app")
> put(ENABLE_IDEMPOTENCE_CONFIG, "true")
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (KAFKA-15059) Exactly-once source tasks fail to start during pending rebalances
[ https://issues.apache.org/jira/browse/KAFKA-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-15059: -- Affects Version/s: 3.6.0 (was: 3.3.0) (was: 3.4.0) (was: 3.3.1) (was: 3.3.2) (was: 3.5.0) (was: 3.4.1) > Exactly-once source tasks fail to start during pending rebalances > - > > Key: KAFKA-15059 > URL: https://issues.apache.org/jira/browse/KAFKA-15059 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect, mirrormaker >Affects Versions: 3.6.0 >Reporter: Chris Egerton >Assignee: Chris Egerton >Priority: Major > > When asked to perform a round of zombie fencing, the distributed herder will > [reject the > request|https://github.com/apache/kafka/blob/17fd30e6b457f097f6a524b516eca1a6a74a9144/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1249-L1250] > if a rebalance is pending, which can happen if (among other things) a config > for a new connector or a new set of task configs has been recently read from > the config topic. > Normally this can be alleviated with a simple task restart, which isn't great > but isn't terrible. > However, when running MirrorMaker 2 in dedicated mode, there is no API to > restart failed tasks, and it can be more common to see this kind of failure > on a fresh cluster because three connector configurations are written in > rapid succession to the config topic. > > In order to provide a better experience for users of both vanilla Kafka > Connect and dedicated MirrorMaker 2 clusters, we can retry (likely with the > same exponential backoff introduced with KAFKA-14732) zombie fencing attempts > that fail due to a pending rebalance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15059) Exactly-once source tasks fail to start during pending rebalances
[ https://issues.apache.org/jira/browse/KAFKA-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-15059: -- Priority: Blocker (was: Major) > Exactly-once source tasks fail to start during pending rebalances > - > > Key: KAFKA-15059 > URL: https://issues.apache.org/jira/browse/KAFKA-15059 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect, mirrormaker >Affects Versions: 3.6.0 >Reporter: Chris Egerton >Assignee: Chris Egerton >Priority: Blocker > > When asked to perform a round of zombie fencing, the distributed herder will > [reject the > request|https://github.com/apache/kafka/blob/17fd30e6b457f097f6a524b516eca1a6a74a9144/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1249-L1250] > if a rebalance is pending, which can happen if (among other things) a config > for a new connector or a new set of task configs has been recently read from > the config topic. > Normally this can be alleviated with a simple task restart, which isn't great > but isn't terrible. > However, when running MirrorMaker 2 in dedicated mode, there is no API to > restart failed tasks, and it can be more common to see this kind of failure > on a fresh cluster because three connector configurations are written in > rapid succession to the config topic. > > In order to provide a better experience for users of both vanilla Kafka > Connect and dedicated MirrorMaker 2 clusters, we can retry (likely with the > same exponential backoff introduced with KAFKA-14732) zombie fencing attempts > that fail due to a pending rebalance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15059) Exactly-once source tasks fail to start during pending rebalances
[ https://issues.apache.org/jira/browse/KAFKA-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-15059: -- Fix Version/s: 3.6.0 > Exactly-once source tasks fail to start during pending rebalances > - > > Key: KAFKA-15059 > URL: https://issues.apache.org/jira/browse/KAFKA-15059 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect, mirrormaker >Affects Versions: 3.6.0 >Reporter: Chris Egerton >Assignee: Chris Egerton >Priority: Blocker > Fix For: 3.6.0 > > > When asked to perform a round of zombie fencing, the distributed herder will > [reject the > request|https://github.com/apache/kafka/blob/17fd30e6b457f097f6a524b516eca1a6a74a9144/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1249-L1250] > if a rebalance is pending, which can happen if (among other things) a config > for a new connector or a new set of task configs has been recently read from > the config topic. > Normally this can be alleviated with a simple task restart, which isn't great > but isn't terrible. > However, when running MirrorMaker 2 in dedicated mode, there is no API to > restart failed tasks, and it can be more common to see this kind of failure > on a fresh cluster because three connector configurations are written in > rapid succession to the config topic. > > In order to provide a better experience for users of both vanilla Kafka > Connect and dedicated MirrorMaker 2 clusters, we can retry (likely with the > same exponential backoff introduced with KAFKA-14732) zombie fencing attempts > that fail due to a pending rebalance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (KAFKA-15090) Source tasks are no longer stopped on a separate thread
Chris Egerton created KAFKA-15090:
-
Summary: Source tasks are no longer stopped on a separate thread
Key: KAFKA-15090
URL: https://issues.apache.org/jira/browse/KAFKA-15090
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Affects Versions: 3.3.2, 3.3.1, 3.2.3, 3.2.2, 3.4.0, 3.2.1, 3.1.2, 3.0.2,
3.3.0, 3.1.1, 3.2.0, 3.0.1, 3.0.0, 3.1.0, 3.2.4, 3.1.3, 3.0.3, 3.5.0, 3.4.1,
3.3.3, 3.6.0, 3.5.1
Reporter: Chris Egerton
Assignee: Chris Egerton
Before [https://github.com/apache/kafka/pull/9669,] in distributed mode, the
{{SourceTask::stop}} method would be invoked on the herder tick thread, which
is a separate thread from the dedicated thread which was responsible for
polling data from the task and producing it to Kafka.
This aligned with the Javadocs for {{{}SourceTask:poll{}}}, which state:
{quote}The task will be stopped on a separate thread, and when that happens
this method is expected to unblock, quickly finish up any remaining processing,
and return.
{quote}
However, it came with the downside that the herder's tick thread would be
blocked until the invocation of {{SourceTask::stop}} completed, which could
result in major parts of the worker's REST API becoming unavailable and even
the worker falling out of the cluster.
As a result, in [https://github.com/apache/kafka/pull/9669,] we changed the
logic for task shutdown to cause {{SourceTask::stop}} to be invoked on the
dedicated thread for the task (i.e., the one responsible for polling data from
it and producing that data to Kafka).
This altered the semantics for {{SourceTask:poll}} and {{SourceTask::stop}} and
may have broken connectors that block during {{poll}} with the expectation that
{{stop}} can and will be invoked concurrently as a signal that any ongoing
polls should be interrupted immediately.
Although reverting the fix is likely not a viable option (blocking the herder
thread on interactions with user-written plugins is high-risk and we have tried
to eliminate all instances of this where feasible), we may try to restore the
expected contract by spinning up a separate thread exclusively for invoking
{{SourceTask::stop}} separately from the dedicated thread for the task and the
herder's thread.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-15091) Javadocs for SourceTask::commit are incorrect
Chris Egerton created KAFKA-15091:
-
Summary: Javadocs for SourceTask::commit are incorrect
Key: KAFKA-15091
URL: https://issues.apache.org/jira/browse/KAFKA-15091
Project: Kafka
Issue Type: Bug
Components: KafkaConnect
Reporter: Chris Egerton
The Javadocs for {{SourceTask::commit}} state that the method should:
{quote}Commit the offsets, up to the offsets that have been returned by
[{{poll()}}|https://kafka.apache.org/34/javadoc/org/apache/kafka/connect/source/SourceTask.html#poll()].
{quote}
However, this is obviously incorrect given how the Connect runtime (when not
configured with exactly-once support for source connectors) performs polling
and offset commits on separate threads. There's also some extensive discussion
on the semantics of that method in KAFKA-5716 where it's made clear that
altering the behavior of the runtime to align with the documented semantics of
that method is not a viable option.
We should update the Javadocs for this method to state that it does not have
anything to do with the offsets returned from {{SourceTask:poll}} and is
instead just a general, periodically-invoked hook to let the task know that an
offset commit has taken place (but with no guarantees as to which offsets have
been committed and which ones correspond to still-in-flight records).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (KAFKA-15053) Regression for security.protocol validation starting from 3.3.0
[
https://issues.apache.org/jira/browse/KAFKA-15053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton reassigned KAFKA-15053:
-
Assignee: Bo Gao
> Regression for security.protocol validation starting from 3.3.0
> ---
>
> Key: KAFKA-15053
> URL: https://issues.apache.org/jira/browse/KAFKA-15053
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 3.3.0
>Reporter: Bo Gao
>Assignee: Bo Gao
>Priority: Major
>
> [This|https://issues.apache.org/jira/browse/KAFKA-13793] Jira issue
> introduced validations on multiple configs. As a consequence, config
> {{security.protocol}} now only allows upper case values such as PLAINTEXT,
> SSL, SASL_PLAINTEXT, SASL_SSL. Before this change, lower case values like
> sasl_ssl, ssl are also supported, there's even a case insensitive logic
> inside
> [SecurityProtocol|https://github.com/apache/kafka/blob/146a6976aed0d9f90c70b6f21dca8b887cc34e71/clients/src/main/java/org/apache/kafka/common/security/auth/SecurityProtocol.java#L70-L73]
> to handle the lower case values.
> I think we should treat this as a regression bug since we don't support lower
> case values anymore since 3.3.0. For versions later than 3.3.0, we are
> getting error like this when using lower case value sasl_ssl
> {{Invalid value sasl_ssl for configuration security.protocol: String must be
> one of: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15059) Exactly-once source tasks fail to start during pending rebalances
[ https://issues.apache.org/jira/browse/KAFKA-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17735711#comment-17735711 ] Chris Egerton commented on KAFKA-15059: --- Ah, thanks Viktor! > Exactly-once source tasks fail to start during pending rebalances > - > > Key: KAFKA-15059 > URL: https://issues.apache.org/jira/browse/KAFKA-15059 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect, mirrormaker >Affects Versions: 3.6.0 >Reporter: Chris Egerton >Assignee: Chris Egerton >Priority: Blocker > Fix For: 3.6.0 > > > When asked to perform a round of zombie fencing, the distributed herder will > [reject the > request|https://github.com/apache/kafka/blob/17fd30e6b457f097f6a524b516eca1a6a74a9144/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1249-L1250] > if a rebalance is pending, which can happen if (among other things) a config > for a new connector or a new set of task configs has been recently read from > the config topic. > Normally this can be alleviated with a simple task restart, which isn't great > but isn't terrible. > However, when running MirrorMaker 2 in dedicated mode, there is no API to > restart failed tasks, and it can be more common to see this kind of failure > on a fresh cluster because three connector configurations are written in > rapid succession to the config topic. > > In order to provide a better experience for users of both vanilla Kafka > Connect and dedicated MirrorMaker 2 clusters, we can retry (likely with the > same exponential backoff introduced with KAFKA-14732) zombie fencing attempts > that fail due to a pending rebalance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15113) Gracefully handle cases where a sink connector's admin and consumer client config overrides target different Kafka clusters
[
https://issues.apache.org/jira/browse/KAFKA-15113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton updated KAFKA-15113:
--
Priority: Minor (was: Major)
> Gracefully handle cases where a sink connector's admin and consumer client
> config overrides target different Kafka clusters
> ---
>
> Key: KAFKA-15113
> URL: https://issues.apache.org/jira/browse/KAFKA-15113
> Project: Kafka
> Issue Type: Task
> Components: KafkaConnect
>Reporter: Yash Mayya
>Priority: Minor
>
> Background reading -
> *
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-458%3A+Connector+Client+Config+Override+Policy]
>
> *
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect]
>
>
> From [https://github.com/apache/kafka/pull/13434#discussion_r1144415671] -
> {quote}Currently, admin clients are only instantiated for sink connectors to
> create the DLQ topic if required. So it seems like it could be technically
> possible for a sink connector's consumer client overrides to target a
> different Kafka cluster from its producer and admin client overrides. Such a
> setup won't work with this implementation of the get offsets API as it is
> using an admin client to get a sink connector's consumer group offsets.
> However, I'm not sure we want to use a consumer client to retrieve the
> offsets either as we shouldn't be disrupting the existing sink tasks'
> consumer group just to fetch offsets. Leveraging a sink task's consumer also
> isn't an option because fetching offsets for a stopped sink connector (where
> all the tasks will be stopped) should be allowed. I'm wondering if we should
> document that a connector's various client config override policies shouldn't
> target different Kafka clusters (side note - looks like we don't [currently
> document|https://kafka.apache.org/documentation/#connect] client config
> overrides for Connect beyond just the worker property
> {{{}connector.client.config.override.policy{}}}).
> {quote}
>
> {quote}I don't think we need to worry too much about this. I cannot imagine a
> sane use case that involves overriding a connector's Kafka clients with
> different Kafka clusters (not just bootstrap servers, but actually different
> clusters) for producer/consumer/admin. I'd be fine with adding a note to our
> docs that that kind of setup isn't supported but I really, really hope that
> it's not necessary and nobody's trying to do that in the first place. I also
> suspect that there are other places where this might cause issues, like with
> exactly-once source support or automatic topic creation for source connectors.
> That said, there is a different case we may want to consider: someone may
> have configured consumer overrides for a sink connector, but not admin
> overrides. This may happen if they don't use a DLQ topic. I don't know if we
> absolutely need to handle this now and we may consider filing a follow-up
> ticket to look into this, but one quick-and-dirty thought I've had is to
> configure the admin client used here with a combination of the configurations
> for the connector's admin client and its consumer, giving precedent to the
> latter.
> {quote}
>
> Also from [https://github.com/apache/kafka/pull/13818#discussion_r1224138055]
> -
> {quote}We will have undesirable behavior if the connector is targeting a
> Kafka cluster different from the Connect cluster's backing Kafka cluster and
> the user has configured the consumer overrides appropriately for their
> connector, but not the admin overrides (something we also discussed
> previously
> [here|https://github.com/apache/kafka/pull/13434#discussion_r1144415671]).
> In the above case, if a user attempts to reset their sink connector's offsets
> via the {{DELETE /connectors/\{connector}/offsets}} endpoint, the following
> will occur:
> # We list the consumer group offsets via {{Admin::listConsumerGroupOffsets}}
> which returns an empty partition offsets map for the sink connector's
> consumer group ID (it exists on a different Kafka cluster to the one that the
> admin client is connecting to).
> # We call {{SinkConnector::alterOffsets}} with an empty offsets map which
> could cause the sink connector to propagate the offsets reset related changes
> to the sink system.
> # We attempt to delete the consumer group via
> {{Admin::deleteConsumerGroups}} which returns {{GroupIdNotFoundException}}
> which we essentially swallow in order to keep offsets reset operations
> idempotent and return a success message to the user (even though the real
> consumer group for the sink connector on the other Kafka cluster hasn't been
> deleted).
> This will occur i
[jira] [Commented] (KAFKA-15113) Gracefully handle cases where a sink connector's admin and consumer client config overrides target different Kafka clusters
[
https://issues.apache.org/jira/browse/KAFKA-15113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17736177#comment-17736177
]
Chris Egerton commented on KAFKA-15113:
---
[~yash.mayya] Thanks for filing this!
One thing I'm struggling with is that, as far as I've been able to tell,
there's no realistic use case for setting different bootstrap servers for,
e.g., a sink connector's consumer and admin client, or a source connector's
producer and admin client, or a sink connector's consumer and producer (which
can be set up if the connector uses a DLQ topic).
If we don't want to support this use case, I think our effort might be better
spent trying to make it easier for users to do the right thing instead of
trying to gracefully recover when they've done the wrong thing.
One idea I've been toying with is adding support for declaring single
properties in connector configurations that affect all Kafka clients spun up by
the Connect runtime. For example, instead of specifying
{{{}consumer.override.bootstrap.servers{}}},
{{{}producer.override.bootstrap.servers{}}}, and
{{admin.override.bootstrap.servers}} in a connector config, we could allow
users to simply declare {{{}kafka.clients.override.bootstrap.servers{}}}.
If we wanted to get fancier about this and avoid some of the compatibility
constraints of adding framework-level properties to connector configurations
(which always run the risk of conflicting with connector-defined properties),
we might even expand the structure of connector configurations by separating
configs that apply to the connector from the ones that apply to its
runtime-constructed Kafka clients, its key/value/header converters, etc. That
could look something like this (assuming the request is issued against the
{{POST /connectors}} endpoint):
{{{}}
{{ "name": "reddit-source",}}
{{ "connector.config": {}}
{{ "connector.class": "RedditSource",}}
{{ "tasks.max": "1",}}
{{ "posts.subreddits": "CatsStandingUp",}}
{{ "posts.topic": "reddit"}}
{{ },}}
{{ "kafka.clients.config": {}}
{{ "bootstrap.servers": "localhost:9093",}}
{{ "security.protocol": "PLAINTEXT"}}
{{ },}}
{{ "producer.config": {}}
{{ "buffer.memory": "4194304"}}
{{ }}}
{{}}}
Both of these would come with the advantage that, if users start actually using
the feature, it'd be harder to screw up connector configurations. Of course, we
would still have to decide if/how to handle misconfiguration by the user, but
it might allow us to pursue more opinionated options, like failing requests
(and even rejecting connector configurations), which IMO is a fine option as
long as we provide a clear error message with easy-to-follow instructions on
how to correct the connector configuration.
TL;DR: We should start rejecting connector configurations (and failing offset
modification requests for connectors) that have mismatched bootstrap servers
across Kafka clients, but we should also make it easier for users to correctly
configure a connector with overridden client bootstrap servers, which will
almost certainly require a KIP.
> Gracefully handle cases where a sink connector's admin and consumer client
> config overrides target different Kafka clusters
> ---
>
> Key: KAFKA-15113
> URL: https://issues.apache.org/jira/browse/KAFKA-15113
> Project: Kafka
> Issue Type: Task
> Components: KafkaConnect
>Reporter: Yash Mayya
>Priority: Minor
>
> Background reading -
> *
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-458%3A+Connector+Client+Config+Override+Policy]
>
> *
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect]
>
>
> From [https://github.com/apache/kafka/pull/13434#discussion_r1144415671] -
> {quote}Currently, admin clients are only instantiated for sink connectors to
> create the DLQ topic if required. So it seems like it could be technically
> possible for a sink connector's consumer client overrides to target a
> different Kafka cluster from its producer and admin client overrides. Such a
> setup won't work with this implementation of the get offsets API as it is
> using an admin client to get a sink connector's consumer group offsets.
> However, I'm not sure we want to use a consumer client to retrieve the
> offsets either as we shouldn't be disrupting the existing sink tasks'
> consumer group just to fetch offsets. Leveraging a sink task's consumer also
> isn't an option because fetching offsets for a stopped sink connector (where
> all the tasks will be stopped) should be allowed. I'm wondering if we should
> document that a connector's various
[jira] [Comment Edited] (KAFKA-15113) Gracefully handle cases where a sink connector's admin and consumer client config overrides target different Kafka clusters
[
https://issues.apache.org/jira/browse/KAFKA-15113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17736177#comment-17736177
]
Chris Egerton edited comment on KAFKA-15113 at 6/22/23 3:35 PM:
[~yash.mayya] Thanks for filing this!
One thing I'm struggling with is that, as far as I've been able to tell,
there's no realistic use case for setting different bootstrap servers for,
e.g., a sink connector's consumer and admin client, or a source connector's
producer and admin client, or a sink connector's consumer and producer (which
can be set up if the connector uses a DLQ topic).
If we don't want to support this use case, I think our effort might be better
spent trying to make it easier for users to do the right thing instead of
trying to gracefully recover when they've done the wrong thing.
One idea I've been toying with is adding support for declaring single
properties in connector configurations that affect all Kafka clients spun up by
the Connect runtime. For example, instead of specifying
{{{}consumer.override.bootstrap.servers{}}},
{{{}producer.override.bootstrap.servers{}}}, and
{{admin.override.bootstrap.servers}} in a connector config, we could allow
users to simply declare {{{}kafka.clients.override.bootstrap.servers{}}}.
If we wanted to get fancier about this and avoid some of the compatibility
constraints of adding framework-level properties to connector configurations
(which always run the risk of conflicting with connector-defined properties),
we might even expand the structure of connector configurations by separating
configs that apply to the connector from the ones that apply to its
runtime-constructed Kafka clients, its key/value/header converters, etc. That
could look something like this (assuming the request is issued against the
{{POST /connectors}} endpoint):
{
{{ "name": "reddit-source",}}
{{ "connector.config": {}}
{{ "connector.class": "RedditSource",}}
{{ "tasks.max": "1",}}
{{ "posts.subreddits": "CatsStandingUp",}}
{{ "posts.topic": "reddit"}}
{{ },}}
{{ "kafka.clients.config": {}}
{{ "bootstrap.servers": "localhost:9093",}}
{{ "security.protocol": "PLAINTEXT"}}
{{ },}}
{{ "producer.config": {}}
{{ "buffer.memory": "4194304"}}
}
}
Both of these would come with the advantage that, if users start actually using
the feature, it'd be harder to screw up connector configurations. Of course, we
would still have to decide if/how to handle misconfiguration by the user, but
it might allow us to pursue more opinionated options, like failing requests
(and even rejecting connector configurations), which IMO is a fine option as
long as we provide a clear error message with easy-to-follow instructions on
how to correct the connector configuration.
TL;DR: We should start rejecting connector configurations (and failing offset
modification requests for connectors) that have mismatched bootstrap servers
across Kafka clients, but we should also make it easier for users to correctly
configure a connector with overridden client bootstrap servers, which will
almost certainly require a KIP.
was (Author: chrisegerton):
[~yash.mayya] Thanks for filing this!
One thing I'm struggling with is that, as far as I've been able to tell,
there's no realistic use case for setting different bootstrap servers for,
e.g., a sink connector's consumer and admin client, or a source connector's
producer and admin client, or a sink connector's consumer and producer (which
can be set up if the connector uses a DLQ topic).
If we don't want to support this use case, I think our effort might be better
spent trying to make it easier for users to do the right thing instead of
trying to gracefully recover when they've done the wrong thing.
One idea I've been toying with is adding support for declaring single
properties in connector configurations that affect all Kafka clients spun up by
the Connect runtime. For example, instead of specifying
{{{}consumer.override.bootstrap.servers{}}},
{{{}producer.override.bootstrap.servers{}}}, and
{{admin.override.bootstrap.servers}} in a connector config, we could allow
users to simply declare {{{}kafka.clients.override.bootstrap.servers{}}}.
If we wanted to get fancier about this and avoid some of the compatibility
constraints of adding framework-level properties to connector configurations
(which always run the risk of conflicting with connector-defined properties),
we might even expand the structure of connector configurations by separating
configs that apply to the connector from the ones that apply to its
runtime-constructed Kafka clients, its key/value/header converters, etc. That
could look something like this (assuming the request is issued against the
{{POST /connectors}} endpoint):
{{{}}
{{ "name": "reddit-source",}}
[jira] [Commented] (KAFKA-15053) Regression for security.protocol validation starting from 3.3.0
[
https://issues.apache.org/jira/browse/KAFKA-15053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17736529#comment-17736529
]
Chris Egerton commented on KAFKA-15053:
---
[~dlgaobo] I think that may have been a typo, but just to be extra clear–we
will never backport this to 3.3.0, which has already been released. We may
backport it to the 3.3 branch, which will then cause the change to be included
if we do another 3.3.x release (right now, that would be 3.3.3).
Assuming you're asking about 3.3.x in general and not 3.3.0 specifically, then
yes, I believe that once they've upgraded to the new version (e.g., 3.3.3),
there's nothing else users will have to do to benefit from this fix. However,
again, I'm not certain there will be another 3.3.x release, so they may have to
upgrade to a 3.4.x version to benefit from this fix.
> Regression for security.protocol validation starting from 3.3.0
> ---
>
> Key: KAFKA-15053
> URL: https://issues.apache.org/jira/browse/KAFKA-15053
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 3.3.0
>Reporter: Bo Gao
>Assignee: Bo Gao
>Priority: Major
>
> [This|https://issues.apache.org/jira/browse/KAFKA-13793] Jira issue
> introduced validations on multiple configs. As a consequence, config
> {{security.protocol}} now only allows upper case values such as PLAINTEXT,
> SSL, SASL_PLAINTEXT, SASL_SSL. Before this change, lower case values like
> sasl_ssl, ssl are also supported, there's even a case insensitive logic
> inside
> [SecurityProtocol|https://github.com/apache/kafka/blob/146a6976aed0d9f90c70b6f21dca8b887cc34e71/clients/src/main/java/org/apache/kafka/common/security/auth/SecurityProtocol.java#L70-L73]
> to handle the lower case values.
> I think we should treat this as a regression bug since we don't support lower
> case values anymore since 3.3.0. For versions later than 3.3.0, we are
> getting error like this when using lower case value sasl_ssl
> {{Invalid value sasl_ssl for configuration security.protocol: String must be
> one of: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15053) Regression for security.protocol validation starting from 3.3.0
[
https://issues.apache.org/jira/browse/KAFKA-15053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17736529#comment-17736529
]
Chris Egerton edited comment on KAFKA-15053 at 6/23/23 3:31 PM:
[~dlgaobo] I think that may have been a typo, but just to be extra clear–we
will never backport this to 3.3.0, which has already been released. We may
backport it to the 3.3 branch, which will then cause the change to be included
if we do another 3.3.x release (right now, that would be 3.3.3).
Assuming you're asking about 3.3.x in general and not 3.3.0 specifically, then
yes, I believe that once they've upgraded to the new version (e.g., 3.3.3),
there's nothing else users will have to do to benefit from this fix. However,
again, I'm not certain there will be another 3.3.x release, so they may have to
upgrade to a 3.4.x version to get this fix.
was (Author: chrisegerton):
[~dlgaobo] I think that may have been a typo, but just to be extra clear–we
will never backport this to 3.3.0, which has already been released. We may
backport it to the 3.3 branch, which will then cause the change to be included
if we do another 3.3.x release (right now, that would be 3.3.3).
Assuming you're asking about 3.3.x in general and not 3.3.0 specifically, then
yes, I believe that once they've upgraded to the new version (e.g., 3.3.3),
there's nothing else users will have to do to benefit from this fix. However,
again, I'm not certain there will be another 3.3.x release, so they may have to
upgrade to a 3.4.x version to benefit from this fix.
> Regression for security.protocol validation starting from 3.3.0
> ---
>
> Key: KAFKA-15053
> URL: https://issues.apache.org/jira/browse/KAFKA-15053
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 3.3.0
>Reporter: Bo Gao
>Assignee: Bo Gao
>Priority: Major
>
> [This|https://issues.apache.org/jira/browse/KAFKA-13793] Jira issue
> introduced validations on multiple configs. As a consequence, config
> {{security.protocol}} now only allows upper case values such as PLAINTEXT,
> SSL, SASL_PLAINTEXT, SASL_SSL. Before this change, lower case values like
> sasl_ssl, ssl are also supported, there's even a case insensitive logic
> inside
> [SecurityProtocol|https://github.com/apache/kafka/blob/146a6976aed0d9f90c70b6f21dca8b887cc34e71/clients/src/main/java/org/apache/kafka/common/security/auth/SecurityProtocol.java#L70-L73]
> to handle the lower case values.
> I think we should treat this as a regression bug since we don't support lower
> case values anymore since 3.3.0. For versions later than 3.3.0, we are
> getting error like this when using lower case value sasl_ssl
> {{Invalid value sasl_ssl for configuration security.protocol: String must be
> one of: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14784) Implement connector offset reset REST API
[
https://issues.apache.org/jira/browse/KAFKA-14784?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton updated KAFKA-14784:
--
Fix Version/s: 3.6.0
> Implement connector offset reset REST API
> -
>
> Key: KAFKA-14784
> URL: https://issues.apache.org/jira/browse/KAFKA-14784
> Project: Kafka
> Issue Type: Sub-task
> Components: KafkaConnect
>Reporter: Chris Egerton
>Assignee: Yash Mayya
>Priority: Major
> Fix For: 3.6.0
>
>
> Implement the {{DELETE /connectors/name/offsets}} endpoint [described in
> KIP-875|https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect#KIP875:FirstclassoffsetssupportinKafkaConnect-Resettingoffsets].
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-4107) Support offset reset capability in Kafka Connect
[ https://issues.apache.org/jira/browse/KAFKA-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-4107. -- Resolution: Done > Support offset reset capability in Kafka Connect > > > Key: KAFKA-4107 > URL: https://issues.apache.org/jira/browse/KAFKA-4107 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Jason Gustafson >Assignee: Chris Egerton >Priority: Major > Labels: kip > Fix For: 3.6.0 > > > It would be useful in some cases to be able to reset connector offsets. For > example, if a topic in Kafka corresponding to a source database is > accidentally deleted (or deleted because of corrupt data), an administrator > may want to reset offsets and reproduce the log from the beginning. It may > also be useful to have support for overriding offsets, but that seems like a > less likely use case. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-4107) Support offset reset capability in Kafka Connect
[ https://issues.apache.org/jira/browse/KAFKA-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-4107: - Fix Version/s: 3.6.0 > Support offset reset capability in Kafka Connect > > > Key: KAFKA-4107 > URL: https://issues.apache.org/jira/browse/KAFKA-4107 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Jason Gustafson >Assignee: Chris Egerton >Priority: Major > Labels: kip > Fix For: 3.6.0 > > > It would be useful in some cases to be able to reset connector offsets. For > example, if a topic in Kafka corresponding to a source database is > accidentally deleted (or deleted because of corrupt data), an administrator > may want to reset offsets and reproduce the log from the beginning. It may > also be useful to have support for overriding offsets, but that seems like a > less likely use case. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-4107) Support offset reset capability in Kafka Connect
[ https://issues.apache.org/jira/browse/KAFKA-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17736570#comment-17736570 ] Chris Egerton commented on KAFKA-4107: -- Wanted to update everyone here and let you all know that we've finished work on this feature and it will be available in 3.6.0, though some connectors that manage offsets externally may require additional changes. > Support offset reset capability in Kafka Connect > > > Key: KAFKA-4107 > URL: https://issues.apache.org/jira/browse/KAFKA-4107 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Jason Gustafson >Assignee: Chris Egerton >Priority: Major > Labels: kip > Fix For: 3.6.0 > > > It would be useful in some cases to be able to reset connector offsets. For > example, if a topic in Kafka corresponding to a source database is > accidentally deleted (or deleted because of corrupt data), an administrator > may want to reset offsets and reproduce the log from the beginning. It may > also be useful to have support for overriding offsets, but that seems like a > less likely use case. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-14972) Make KafkaConsumer usable in async runtimes
[
https://issues.apache.org/jira/browse/KAFKA-14972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton reassigned KAFKA-14972:
-
Assignee: Erik van Oosten
> Make KafkaConsumer usable in async runtimes
> ---
>
> Key: KAFKA-14972
> URL: https://issues.apache.org/jira/browse/KAFKA-14972
> Project: Kafka
> Issue Type: Wish
> Components: consumer
>Reporter: Erik van Oosten
>Assignee: Erik van Oosten
>Priority: Major
>
> KafkaConsumer contains a check that rejects nested invocations from different
> threads (method {{{}acquire{}}}). For users that use an async runtime, this
> is an almost impossible requirement. Examples of async runtimes that are
> affected are Kotlin co-routines (see KAFKA-7143) and Zio.
> We propose to replace the thread-id check with an access-id that is stored on
> a thread-local variable. Existing programs will not be affected. Developers
> that work in an async runtime can pick up the access-id and set it on the
> thread-local variable in a thread of their choosing.
> Every time a callback is invoked a new access-id is generated. When the
> callback completes, the previous access-id is restored.
> This proposal does not make it impossible to use the client incorrectly.
> However, we think it strikes a good balance between making correct usage from
> an async runtime possible while making incorrect usage difficult.
> Alternatives considered:
> # Configuration that switches off the check completely.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15127) Allow offsets to be reset at the same time a connector is deleted.
[ https://issues.apache.org/jira/browse/KAFKA-15127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738578#comment-17738578 ] Chris Egerton commented on KAFKA-15127: --- I think it's a little premature to start working on this. It'd be nice if we could give users some time to try out the V1 offsets API before we decide if this feature is necessary or not, and if it is, what the user-facing API and the underlying semantics should be. > Allow offsets to be reset at the same time a connector is deleted. > -- > > Key: KAFKA-15127 > URL: https://issues.apache.org/jira/browse/KAFKA-15127 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Sagar Rao >Assignee: Sagar Rao >Priority: Major > Labels: needs-kip > > This has been listed as [Future > Work|https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect#KIP875:FirstclassoffsetssupportinKafkaConnect-Automaticallydeleteoffsetswithconnectors] > in KIP-875. Now that the delete offsets mechanism is also in place, we can > take this up which will allow connector names to be reused after connector > deletion. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15068) Incorrect replication Latency may be calculated when the timestamp of the record is of type CREATE_TIME
[ https://issues.apache.org/jira/browse/KAFKA-15068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738591#comment-17738591 ] Chris Egerton commented on KAFKA-15068: --- Thanks for filing this [~hudeqi], good catch! I agree with your position that "replication latency reflects the performance of replication and should not be affected by this abnormal situation". Let me know if you have any other questions about this; happy to discuss further here or, if you feel ready, review a PR. > Incorrect replication Latency may be calculated when the timestamp of the > record is of type CREATE_TIME > --- > > Key: KAFKA-15068 > URL: https://issues.apache.org/jira/browse/KAFKA-15068 > Project: Kafka > Issue Type: Improvement > Components: mirrormaker >Reporter: hudeqi >Assignee: hudeqi >Priority: Major > > When MM2 is used to replicate topics between Kafka clusters, if the timestamp > of the record of the source cluster is of type CREATE_TIME, then the value of > timestamp may not be the actual "creation time (that is, append time)", for > example, the value of timestamp is less than the current time for a long time > (this is determined by the producer, and often occurs in the online > environment), at this time, the calculated replication latency will be too > large. > I understand that replication latency reflects the performance of replication > and should not be affected by this abnormal situation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[
https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738733#comment-17738733
]
Chris Egerton commented on KAFKA-15102:
---
[~ddufour1a] Thanks for raising this. I believe if we wanted to preserve
backward compatibility perfectly, we would have had to ignore the custom
separator when creating topics affected by KIP-690, and possibly introduced
separate opt-in configuration logic to disable that behavior (i.e., resume
taking custom separators into account, even for topics that were not affected
prior to KIP-690).
However, since 3.1.0 came out a year and a half ago, things are less
cut-and-dry: there may be more negative fallout for users who are accustomed to
the current behavior than what's currently being experienced by those who are
used to the previous behavior.
One possibility could be to revert to older behavior for the remainder of our
3.x.y releases with a single configuration property such as
{{replication.policy.internal.topic.separator.enabled}} that defaults to
{{false}} for now and, come 4.0, defaults to {{{}true{}}}.
[~omnia_h_ibrahim] [~mimaison] do either of you have thoughts on how to
approach this accidental break in compatibility?
> Mirror Maker 2 - KIP690 backward compatibility
> --
>
> Key: KAFKA-15102
> URL: https://issues.apache.org/jira/browse/KAFKA-15102
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.1.0
>Reporter: David Dufour
>Priority: Major
>
> According to KIP690, "When users upgrade an existing MM2 cluster they don’t
> need to change any of their current configuration as this proposal maintains
> the default behaviour for MM2."
> Now, the separator is subject to customization.
> As a consequence, when an MM2 upgrade is performed, if the separator was
> customized with replication.policy.separator, the name of this internal topic
> changes. It then generates issues like:
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.InvalidTopicException: Topic
> 'mm2-offset-syncs_bkts28_internal' collides with existing topics:
> mm2-offset-syncs.bkts28.internal
> It has been observed that the replication can then be broken sometimes
> several days after the upgrade (reason not identified). By deleting the old
> topic name, it recovers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[
https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738733#comment-17738733
]
Chris Egerton edited comment on KAFKA-15102 at 6/29/23 6:43 PM:
[~ddufour1a] Thanks for raising this. I believe if we wanted to preserve
backward compatibility perfectly, we would have had to ignore the custom
separator when creating topics affected by KIP-690, and possibly introduced
separate opt-in configuration logic to disable that behavior (i.e., resume
taking custom separators into account, even for topics that were not affected
prior to KIP-690).
However, since 3.1.0 came out a year and a half ago, things are less
cut-and-dry: there may be more negative fallout for users who are accustomed to
the current behavior than what's currently being experienced by those who are
used to the previous behavior.
One possibility could be to revert to older behavior for the remainder of our
3.x.y releases with a single configuration property such as
{{replication.policy.internal.topic.separator.enabled}} that defaults to
{{false}} for now and, come 4.0, defaults to {{{}true{}}}.
[~omnia_h_ibrahim] [~mimaison] do either of you have thoughts on how to
approach this accidental break in compatibility? It's probably worth bringing
this up in a discussion thread on the user/dev mailing lists, but I'd rather
get some feedback on a potential fix before bringing this before a larger
audience.
was (Author: chrisegerton):
[~ddufour1a] Thanks for raising this. I believe if we wanted to preserve
backward compatibility perfectly, we would have had to ignore the custom
separator when creating topics affected by KIP-690, and possibly introduced
separate opt-in configuration logic to disable that behavior (i.e., resume
taking custom separators into account, even for topics that were not affected
prior to KIP-690).
However, since 3.1.0 came out a year and a half ago, things are less
cut-and-dry: there may be more negative fallout for users who are accustomed to
the current behavior than what's currently being experienced by those who are
used to the previous behavior.
One possibility could be to revert to older behavior for the remainder of our
3.x.y releases with a single configuration property such as
{{replication.policy.internal.topic.separator.enabled}} that defaults to
{{false}} for now and, come 4.0, defaults to {{{}true{}}}.
[~omnia_h_ibrahim] [~mimaison] do either of you have thoughts on how to
approach this accidental break in compatibility?
> Mirror Maker 2 - KIP690 backward compatibility
> --
>
> Key: KAFKA-15102
> URL: https://issues.apache.org/jira/browse/KAFKA-15102
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.1.0
>Reporter: David Dufour
>Priority: Major
>
> According to KIP690, "When users upgrade an existing MM2 cluster they don’t
> need to change any of their current configuration as this proposal maintains
> the default behaviour for MM2."
> Now, the separator is subject to customization.
> As a consequence, when an MM2 upgrade is performed, if the separator was
> customized with replication.policy.separator, the name of this internal topic
> changes. It then generates issues like:
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.InvalidTopicException: Topic
> 'mm2-offset-syncs_bkts28_internal' collides with existing topics:
> mm2-offset-syncs.bkts28.internal
> It has been observed that the replication can then be broken sometimes
> several days after the upgrade (reason not identified). By deleting the old
> topic name, it recovers.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15113) Gracefully handle cases where a sink connector's admin and consumer client config overrides target different Kafka clusters
[
https://issues.apache.org/jira/browse/KAFKA-15113?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17738740#comment-17738740
]
Chris Egerton commented on KAFKA-15113:
---
[~yash.mayya] although it seems a little unlikely, I agree that it's worth
considering the use case of a separate Kafka cluster for topics consumed by a
sink connector and used for a DLQ.
I'm tempted to dismiss it as unrealistic but we'd definitely need a KIP before
deciding not to support use cases like this. If we wanted to support them, we
could potentially introduce different config namespaces for the DLQ admin
client and the offset management admin client, which can be used to override
the existing general admin overrides for a sink connector. This could also be
left as "future work" for any KIP that makes that kind of use case impossible.
RE compatibility and a restructured request format for connector
configurations: I don't know if it would be too difficult; we could either
differentiate by request header (connect-config-version or something like
that), or even try to deduce the format automatically based on the shape of the
JSON request. It's the kind of thing that may get a little hairy under the
hood, but could be made pretty smooth for users.
> Gracefully handle cases where a sink connector's admin and consumer client
> config overrides target different Kafka clusters
> ---
>
> Key: KAFKA-15113
> URL: https://issues.apache.org/jira/browse/KAFKA-15113
> Project: Kafka
> Issue Type: Task
> Components: KafkaConnect
>Reporter: Yash Mayya
>Priority: Minor
>
> Background reading -
> *
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-458%3A+Connector+Client+Config+Override+Policy]
>
> *
> [https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect]
>
>
> From [https://github.com/apache/kafka/pull/13434#discussion_r1144415671] -
> {quote}Currently, admin clients are only instantiated for sink connectors to
> create the DLQ topic if required. So it seems like it could be technically
> possible for a sink connector's consumer client overrides to target a
> different Kafka cluster from its producer and admin client overrides. Such a
> setup won't work with this implementation of the get offsets API as it is
> using an admin client to get a sink connector's consumer group offsets.
> However, I'm not sure we want to use a consumer client to retrieve the
> offsets either as we shouldn't be disrupting the existing sink tasks'
> consumer group just to fetch offsets. Leveraging a sink task's consumer also
> isn't an option because fetching offsets for a stopped sink connector (where
> all the tasks will be stopped) should be allowed. I'm wondering if we should
> document that a connector's various client config override policies shouldn't
> target different Kafka clusters (side note - looks like we don't [currently
> document|https://kafka.apache.org/documentation/#connect] client config
> overrides for Connect beyond just the worker property
> {{{}connector.client.config.override.policy{}}}).
> {quote}
>
> {quote}I don't think we need to worry too much about this. I cannot imagine a
> sane use case that involves overriding a connector's Kafka clients with
> different Kafka clusters (not just bootstrap servers, but actually different
> clusters) for producer/consumer/admin. I'd be fine with adding a note to our
> docs that that kind of setup isn't supported but I really, really hope that
> it's not necessary and nobody's trying to do that in the first place. I also
> suspect that there are other places where this might cause issues, like with
> exactly-once source support or automatic topic creation for source connectors.
> That said, there is a different case we may want to consider: someone may
> have configured consumer overrides for a sink connector, but not admin
> overrides. This may happen if they don't use a DLQ topic. I don't know if we
> absolutely need to handle this now and we may consider filing a follow-up
> ticket to look into this, but one quick-and-dirty thought I've had is to
> configure the admin client used here with a combination of the configurations
> for the connector's admin client and its consumer, giving precedent to the
> latter.
> {quote}
>
> Also from [https://github.com/apache/kafka/pull/13818#discussion_r1224138055]
> -
> {quote}We will have undesirable behavior if the connector is targeting a
> Kafka cluster different from the Connect cluster's backing Kafka cluster and
> the user has configured the consumer overrides appropriately for their
> connector, but not the admin overrides (something we also discuss
[jira] [Commented] (KAFKA-15127) Allow offsets to be reset at the same time a connector is deleted.
[ https://issues.apache.org/jira/browse/KAFKA-15127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17739095#comment-17739095 ] Chris Egerton commented on KAFKA-15127: --- In that case, perhaps we can leave it unassigned and note in the description that we'd like to let the initial offset management API soak for a bit to gather user feedback before pursuing this? > Allow offsets to be reset at the same time a connector is deleted. > -- > > Key: KAFKA-15127 > URL: https://issues.apache.org/jira/browse/KAFKA-15127 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Sagar Rao >Assignee: Sagar Rao >Priority: Major > Labels: needs-kip > > This has been listed as [Future > Work|https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect#KIP875:FirstclassoffsetssupportinKafkaConnect-Automaticallydeleteoffsetswithconnectors] > in KIP-875. Now that the delete offsets mechanism is also in place, we can > take this up which will allow connector names to be reused after connector > deletion. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-13988) Mirrormaker 2 auto.offset.reset=latest not working
[
https://issues.apache.org/jira/browse/KAFKA-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton updated KAFKA-13988:
--
Fix Version/s: (was: 3.2.0)
> Mirrormaker 2 auto.offset.reset=latest not working
> --
>
> Key: KAFKA-13988
> URL: https://issues.apache.org/jira/browse/KAFKA-13988
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect, mirrormaker
>Affects Versions: 3.2.0
> Environment: Source Kafka cluster running on Ubuntu 20
> Source Kafka cluster Kafka v0.10
> Target Kafka cluster running in AWS MSK
> Target Kafka cluster Kafka v2.6.2
> Mirrormaker version 3.2.0 running on Ubuntu 20.
>Reporter: Daniel Florek
>Assignee: Ravindranath Kakarla
>Priority: Major
>
> Hi.
> I have problem setting up mirroring with MM2 from latest offset between 2
> clusters. In logs I can see that Consumer that is consuming topics has
> auto.offset.reset property set to latest. But still topics are read from
> offset 0. I am using following configuration:
>
> {code:java}
> clusters = A, B
> A.bootstrap.servers = broker-01A:9092
> B.bootstrap.servers = broker-01B:9092,broker-02B:9092,broker-03B:9092
> replication.policy.class =
> org.apache.kafka.connect.mirror.IdentityReplicationPolicy
> #Enable replication between clusters and define topics which should be
> replicated
> A->B.enabled = true
> A->B.topics = .*
> A->B.replication.factor=3
> A->B.emit.heartbeats.enabled = true
> A->B.emit.checkpoints.enabled = true
> auto.offset.reset=latest
> consumer.auto.offset.reset=latest
> A.consumer.auto.offset.reset=latest
> B.consumer.auto.offset.reset=latest
> refresh.topics.enabled=true
> heartbeats.topic.replication.factor=1
> checkpoints.topic.replication.factor=1
> offset-syncs.topic.replication.factor=1
> config.storage.replication.factor = 1
> offset.storage.replication.factor = 1
> status.storage.replication.factor = 1 {code}
> I am using Kafka 3.2.0 for Mirrormaker 2. Source kafka cluster is 1 broker
> running on EC2 instance in AWS (quite an old version I think 0.10). Target
> kafka cluster contains 3 brokers running in AWS MSK (version 2.6.2).
> Could you point me what I am doing wrong? Or is this possibly a bug?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-13988) Mirrormaker 2 auto.offset.reset=latest not working
[
https://issues.apache.org/jira/browse/KAFKA-13988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton updated KAFKA-13988:
--
Component/s: (was: KafkaConnect)
> Mirrormaker 2 auto.offset.reset=latest not working
> --
>
> Key: KAFKA-13988
> URL: https://issues.apache.org/jira/browse/KAFKA-13988
> Project: Kafka
> Issue Type: Bug
> Components: mirrormaker
>Affects Versions: 3.2.0
> Environment: Source Kafka cluster running on Ubuntu 20
> Source Kafka cluster Kafka v0.10
> Target Kafka cluster running in AWS MSK
> Target Kafka cluster Kafka v2.6.2
> Mirrormaker version 3.2.0 running on Ubuntu 20.
>Reporter: Daniel Florek
>Assignee: Ravindranath Kakarla
>Priority: Major
>
> Hi.
> I have problem setting up mirroring with MM2 from latest offset between 2
> clusters. In logs I can see that Consumer that is consuming topics has
> auto.offset.reset property set to latest. But still topics are read from
> offset 0. I am using following configuration:
>
> {code:java}
> clusters = A, B
> A.bootstrap.servers = broker-01A:9092
> B.bootstrap.servers = broker-01B:9092,broker-02B:9092,broker-03B:9092
> replication.policy.class =
> org.apache.kafka.connect.mirror.IdentityReplicationPolicy
> #Enable replication between clusters and define topics which should be
> replicated
> A->B.enabled = true
> A->B.topics = .*
> A->B.replication.factor=3
> A->B.emit.heartbeats.enabled = true
> A->B.emit.checkpoints.enabled = true
> auto.offset.reset=latest
> consumer.auto.offset.reset=latest
> A.consumer.auto.offset.reset=latest
> B.consumer.auto.offset.reset=latest
> refresh.topics.enabled=true
> heartbeats.topic.replication.factor=1
> checkpoints.topic.replication.factor=1
> offset-syncs.topic.replication.factor=1
> config.storage.replication.factor = 1
> offset.storage.replication.factor = 1
> status.storage.replication.factor = 1 {code}
> I am using Kafka 3.2.0 for Mirrormaker 2. Source kafka cluster is 1 broker
> running on EC2 instance in AWS (quite an old version I think 0.10). Target
> kafka cluster contains 3 brokers running in AWS MSK (version 2.6.2).
> Could you point me what I am doing wrong? Or is this possibly a bug?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[ https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17739146#comment-17739146 ] Chris Egerton commented on KAFKA-15102: --- [~omnia_h_ibrahim] Good call 👍 we should definitely update the compatibility section in the KIP to mention this. We may also want list the affected versions and link to this issue for further context. > Mirror Maker 2 - KIP690 backward compatibility > -- > > Key: KAFKA-15102 > URL: https://issues.apache.org/jira/browse/KAFKA-15102 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 3.1.0 >Reporter: David Dufour >Priority: Major > > According to KIP690, "When users upgrade an existing MM2 cluster they don’t > need to change any of their current configuration as this proposal maintains > the default behaviour for MM2." > Now, the separator is subject to customization. > As a consequence, when an MM2 upgrade is performed, if the separator was > customized with replication.policy.separator, the name of this internal topic > changes. It then generates issues like: > Caused by: java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.InvalidTopicException: Topic > 'mm2-offset-syncs_bkts28_internal' collides with existing topics: > mm2-offset-syncs.bkts28.internal > It has been observed that the replication can then be broken sometimes > several days after the upgrade (reason not identified). By deleting the old > topic name, it recovers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15091) Javadocs for SourceTask::commit are incorrect
[
https://issues.apache.org/jira/browse/KAFKA-15091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17739196#comment-17739196
]
Chris Egerton commented on KAFKA-15091:
---
I think that was also discussed with KAFKA-5716. I wouldn't necessarily be
opposed to deprecating {{{}SourceTask::commit{}}}, but given that we're several
years further along than when that ticket was last discussed, the likelihood of
connectors relying on that method have increased. We also currently make use of
this method in MirrorMaker 2 (see KAFKA-14610).
I think this ticket should focus on updating the docs for this method to be
correct for all releases of Kafka Connect that invoke it; if we want to take
more drastic action (which, again, I'm not currently opposed to), it probably
makes sense to tackle that in a separate ticket.
> Javadocs for SourceTask::commit are incorrect
> -
>
> Key: KAFKA-15091
> URL: https://issues.apache.org/jira/browse/KAFKA-15091
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Chris Egerton
>Priority: Major
>
> The Javadocs for {{SourceTask::commit}} state that the method should:
> {quote}Commit the offsets, up to the offsets that have been returned by
> [{{poll()}}|https://kafka.apache.org/34/javadoc/org/apache/kafka/connect/source/SourceTask.html#poll()].
> {quote}
> However, this is obviously incorrect given how the Connect runtime (when not
> configured with exactly-once support for source connectors) performs polling
> and offset commits on separate threads. There's also some extensive
> discussion on the semantics of that method in KAFKA-5716 where it's made
> clear that altering the behavior of the runtime to align with the documented
> semantics of that method is not a viable option.
> We should update the Javadocs for this method to state that it does not have
> anything to do with the offsets returned from {{SourceTask:poll}} and is
> instead just a general, periodically-invoked hook to let the task know that
> an offset commit has taken place (but with no guarantees as to which offsets
> have been committed and which ones correspond to still-in-flight records).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15127) Allow offsets to be reset at the same time a connector is deleted.
[ https://issues.apache.org/jira/browse/KAFKA-15127?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17739635#comment-17739635 ] Chris Egerton commented on KAFKA-15127: --- Thanks Sagar! > Allow offsets to be reset at the same time a connector is deleted. > -- > > Key: KAFKA-15127 > URL: https://issues.apache.org/jira/browse/KAFKA-15127 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Sagar Rao >Priority: Major > Labels: needs-kip > > This has been listed as [Future > Work|https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect#KIP875:FirstclassoffsetssupportinKafkaConnect-Automaticallydeleteoffsetswithconnectors] > in KIP-875. Now that the delete offsets mechanism is also in place, we can > take this up which will allow connector names to be reused after connector > deletion. > > Note that we can get started with this once the Offsets management API has > been adopted by a few users and we have got feedback around it's functioning. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[ https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17740248#comment-17740248 ] Chris Egerton commented on KAFKA-15102: --- Thanks [~omnia_h_ibrahim], I'd love it if you could take on that KIP! > Mirror Maker 2 - KIP690 backward compatibility > -- > > Key: KAFKA-15102 > URL: https://issues.apache.org/jira/browse/KAFKA-15102 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 3.1.0 >Reporter: David Dufour >Priority: Major > > According to KIP690, "When users upgrade an existing MM2 cluster they don’t > need to change any of their current configuration as this proposal maintains > the default behaviour for MM2." > Now, the separator is subject to customization. > As a consequence, when an MM2 upgrade is performed, if the separator was > customized with replication.policy.separator, the name of this internal topic > changes. It then generates issues like: > Caused by: java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.InvalidTopicException: Topic > 'mm2-offset-syncs_bkts28_internal' collides with existing topics: > mm2-offset-syncs.bkts28.internal > It has been observed that the replication can then be broken sometimes > several days after the upgrade (reason not identified). By deleting the old > topic name, it recovers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[ https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton reassigned KAFKA-15102: - Assignee: Omnia Ibrahim > Mirror Maker 2 - KIP690 backward compatibility > -- > > Key: KAFKA-15102 > URL: https://issues.apache.org/jira/browse/KAFKA-15102 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 3.1.0 >Reporter: David Dufour >Assignee: Omnia Ibrahim >Priority: Major > > According to KIP690, "When users upgrade an existing MM2 cluster they don’t > need to change any of their current configuration as this proposal maintains > the default behaviour for MM2." > Now, the separator is subject to customization. > As a consequence, when an MM2 upgrade is performed, if the separator was > customized with replication.policy.separator, the name of this internal topic > changes. It then generates issues like: > Caused by: java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.InvalidTopicException: Topic > 'mm2-offset-syncs_bkts28_internal' collides with existing topics: > mm2-offset-syncs.bkts28.internal > It has been observed that the replication can then be broken sometimes > several days after the upgrade (reason not identified). By deleting the old > topic name, it recovers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-15102) Mirror Maker 2 - KIP690 backward compatibility
[ https://issues.apache.org/jira/browse/KAFKA-15102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17740663#comment-17740663 ] Chris Egerton commented on KAFKA-15102: --- 💯 [~omnia_h_ibrahim] I've assigned you this ticket so nobody else picks it up by mistake; feel free to unassign if you no longer want to work on this. > Mirror Maker 2 - KIP690 backward compatibility > -- > > Key: KAFKA-15102 > URL: https://issues.apache.org/jira/browse/KAFKA-15102 > Project: Kafka > Issue Type: Bug > Components: mirrormaker >Affects Versions: 3.1.0 >Reporter: David Dufour >Priority: Major > > According to KIP690, "When users upgrade an existing MM2 cluster they don’t > need to change any of their current configuration as this proposal maintains > the default behaviour for MM2." > Now, the separator is subject to customization. > As a consequence, when an MM2 upgrade is performed, if the separator was > customized with replication.policy.separator, the name of this internal topic > changes. It then generates issues like: > Caused by: java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.InvalidTopicException: Topic > 'mm2-offset-syncs_bkts28_internal' collides with existing topics: > mm2-offset-syncs.bkts28.internal > It has been observed that the replication can then be broken sometimes > several days after the upgrade (reason not identified). By deleting the old > topic name, it recovers. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-14059) Replace EasyMock and PowerMock with Mockito in WorkerSourceTaskTest
[ https://issues.apache.org/jira/browse/KAFKA-14059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-14059. --- Resolution: Done > Replace EasyMock and PowerMock with Mockito in WorkerSourceTaskTest > --- > > Key: KAFKA-14059 > URL: https://issues.apache.org/jira/browse/KAFKA-14059 > Project: Kafka > Issue Type: Sub-task > Components: KafkaConnect >Reporter: Chris Egerton >Assignee: Hector Geraldino >Priority: Minor > Fix For: 3.6.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14059) Replace EasyMock and PowerMock with Mockito in WorkerSourceTaskTest
[ https://issues.apache.org/jira/browse/KAFKA-14059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-14059: -- Fix Version/s: 3.6.0 > Replace EasyMock and PowerMock with Mockito in WorkerSourceTaskTest > --- > > Key: KAFKA-14059 > URL: https://issues.apache.org/jira/browse/KAFKA-14059 > Project: Kafka > Issue Type: Sub-task > Components: KafkaConnect >Reporter: Chris Egerton >Assignee: Hector Geraldino >Priority: Minor > Fix For: 3.6.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15145) AbstractWorkerSourceTask re-processes records filtered out by SMTs on retriable exceptions
[ https://issues.apache.org/jira/browse/KAFKA-15145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-15145: -- Fix Version/s: 3.5.1 > AbstractWorkerSourceTask re-processes records filtered out by SMTs on > retriable exceptions > -- > > Key: KAFKA-15145 > URL: https://issues.apache.org/jira/browse/KAFKA-15145 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Affects Versions: 0.10.2.0, 0.10.2.1, 0.10.2.2, 0.11.0.0, 0.11.0.1, > 0.11.0.2, 0.11.0.3, 1.0.0, 1.0.1, 1.0.2, 1.1.0, 1.1.1, 2.0.0, 2.0.1, 2.1.0, > 2.2.0, 2.1.1, 2.3.0, 2.2.1, 2.2.2, 2.4.0, 2.3.1, 2.5.0, 2.4.1, 2.6.0, 2.5.1, > 2.7.0, 2.6.1, 2.8.0, 2.7.1, 2.6.2, 3.1.0, 2.6.3, 2.7.2, 2.8.1, 3.0.0, 3.0.1, > 2.8.2, 3.2.0, 3.1.1, 3.3.0, 3.0.2, 3.1.2, 3.2.1, 3.4.0, 3.2.2, 3.2.3, 3.3.1, > 3.3.2, 3.5.0, 3.4.1 >Reporter: Yash Mayya >Assignee: Yash Mayya >Priority: Minor > Fix For: 3.6.0, 3.5.1 > > > If a RetriableException is thrown from an admin client or producer client > operation in > [AbstractWorkerSourceTask::sendRecords|https://github.com/apache/kafka/blob/5c2492bca71200806ccf776ea31639a90290d43e/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/AbstractWorkerSourceTask.java#L388], > the send operation is retried for the remaining records in the batch. There > is a bug in the logic for computing the remaining records in a batch which > causes records that are filtered out by the task's transformation chain to be > re-processed. This will also result in the SourceTask::commitRecord method > being called twice for the same record, which can cause certain types of > source connectors to fail. This bug seems to exist since when SMTs were first > introduced in 0.10.2 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (KAFKA-15177) MirrorMaker 2 should implement the alterOffsets KIP-875 API
[
https://issues.apache.org/jira/browse/KAFKA-15177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton reassigned KAFKA-15177:
-
Assignee: Chris Egerton
> MirrorMaker 2 should implement the alterOffsets KIP-875 API
> ---
>
> Key: KAFKA-15177
> URL: https://issues.apache.org/jira/browse/KAFKA-15177
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect, mirrormaker
>Reporter: Yash Mayya
>Assignee: Chris Egerton
>Priority: Minor
>
> The {{MirrorSourceConnector}} class should implement the new alterOffsets API
> added in
> [KIP-875|https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect].
> We could also implement the API in
> {{MirrorCheckpointConnector}} and
> {{MirrorHeartbeatConnector}} to prevent external modification of offsets
> since the operation wouldn't really make sense in their case.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15059) Exactly-once source tasks fail to start during pending rebalances
[ https://issues.apache.org/jira/browse/KAFKA-15059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17742034#comment-17742034 ] Chris Egerton commented on KAFKA-15059: --- [~divijvaidya] no; this was introduced by [https://github.com/apache/kafka/pull/13465]. It did not affect anything except trunk and never made it into a release (see the affected/fix versions fields). > Exactly-once source tasks fail to start during pending rebalances > - > > Key: KAFKA-15059 > URL: https://issues.apache.org/jira/browse/KAFKA-15059 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect, mirrormaker >Affects Versions: 3.6.0 >Reporter: Chris Egerton >Assignee: Chris Egerton >Priority: Blocker > Fix For: 3.6.0 > > > When asked to perform a round of zombie fencing, the distributed herder will > [reject the > request|https://github.com/apache/kafka/blob/17fd30e6b457f097f6a524b516eca1a6a74a9144/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/distributed/DistributedHerder.java#L1249-L1250] > if a rebalance is pending, which can happen if (among other things) a config > for a new connector or a new set of task configs has been recently read from > the config topic. > Normally this can be alleviated with a simple task restart, which isn't great > but isn't terrible. > However, when running MirrorMaker 2 in dedicated mode, there is no API to > restart failed tasks, and it can be more common to see this kind of failure > on a fresh cluster because three connector configurations are written in > rapid succession to the config topic. > > In order to provide a better experience for users of both vanilla Kafka > Connect and dedicated MirrorMaker 2 clusters, we can retry (likely with the > same exponential backoff introduced with KAFKA-14732) zombie fencing attempts > that fail due to a pending rebalance. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15172) Allow exact mirroring of ACLs between clusters
[ https://issues.apache.org/jira/browse/KAFKA-15172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-15172: -- Labels: needs-kip (was: ) > Allow exact mirroring of ACLs between clusters > -- > > Key: KAFKA-15172 > URL: https://issues.apache.org/jira/browse/KAFKA-15172 > Project: Kafka > Issue Type: Task > Components: mirrormaker >Reporter: Mickael Maison >Priority: Major > Labels: needs-kip > > When mirroring ACLs, MirrorMaker downgrades allow ALL ACLs to allow READ. The > rationale to is prevent other clients to produce to remote topics. > However in disaster recovery scenarios, where the target cluster is not used > and just a "hot standby", it would be preferable to have exactly the same > ACLs on both clusters to speed up failover. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15179) Add integration tests for the FileStream Sink and Source connectors
[ https://issues.apache.org/jira/browse/KAFKA-15179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-15179: -- Issue Type: Test (was: Improvement) > Add integration tests for the FileStream Sink and Source connectors > --- > > Key: KAFKA-15179 > URL: https://issues.apache.org/jira/browse/KAFKA-15179 > Project: Kafka > Issue Type: Test >Reporter: Yash Mayya >Assignee: Yash Mayya >Priority: Minor > > Add integration tests for the FileStream Sink and Source connectors covering > various different common scenarios. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14938) Flaky test org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest#testConnectorBoundary
[ https://issues.apache.org/jira/browse/KAFKA-14938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-14938: -- Fix Version/s: 3.6.0 > Flaky test > org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest#testConnectorBoundary > -- > > Key: KAFKA-14938 > URL: https://issues.apache.org/jira/browse/KAFKA-14938 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Sagar Rao >Assignee: Sambhav Jain >Priority: Major > Fix For: 3.6.0 > > > Test seems to be failing with > ``` > ava.lang.AssertionError: Not enough records produced by source connector. > Expected at least: 100 + but got 72 > h4. Stacktrace > java.lang.AssertionError: Not enough records produced by source connector. > Expected at least: 100 + but got 72 > at org.junit.Assert.fail(Assert.java:89) > at org.junit.Assert.assertTrue(Assert.java:42) > at > org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest.testConnectorBoundary(ExactlyOnceSourceIntegrationTest.java:421) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > at > org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) > at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) > at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > at org.junit.runners.ParentRunner.run(ParentRunner.java:413) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40) > at > org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:60) > at > org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) > at > org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) > at > org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) > at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) > at > org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176) > at > org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129) > at > org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100) > at > org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60) > at > org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56) > at > org.gradle.process.internal.worker.chi
[jira] [Updated] (KAFKA-14938) Flaky test org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest#testConnectorBoundary
[ https://issues.apache.org/jira/browse/KAFKA-14938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-14938: -- Fix Version/s: 3.4.2 > Flaky test > org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest#testConnectorBoundary > -- > > Key: KAFKA-14938 > URL: https://issues.apache.org/jira/browse/KAFKA-14938 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Sagar Rao >Assignee: Sambhav Jain >Priority: Major > Fix For: 3.6.0, 3.4.2 > > > Test seems to be failing with > ``` > ava.lang.AssertionError: Not enough records produced by source connector. > Expected at least: 100 + but got 72 > h4. Stacktrace > java.lang.AssertionError: Not enough records produced by source connector. > Expected at least: 100 + but got 72 > at org.junit.Assert.fail(Assert.java:89) > at org.junit.Assert.assertTrue(Assert.java:42) > at > org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest.testConnectorBoundary(ExactlyOnceSourceIntegrationTest.java:421) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > at > org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) > at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) > at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > at org.junit.runners.ParentRunner.run(ParentRunner.java:413) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40) > at > org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:60) > at > org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) > at > org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) > at > org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) > at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) > at > org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176) > at > org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129) > at > org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100) > at > org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60) > at > org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56) > at > org.gradle.process.internal.wor
[jira] [Updated] (KAFKA-14938) Flaky test org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest#testConnectorBoundary
[ https://issues.apache.org/jira/browse/KAFKA-14938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-14938: -- Fix Version/s: 3.5.2 > Flaky test > org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest#testConnectorBoundary > -- > > Key: KAFKA-14938 > URL: https://issues.apache.org/jira/browse/KAFKA-14938 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Sagar Rao >Assignee: Sambhav Jain >Priority: Major > Fix For: 3.6.0, 3.4.2, 3.5.2 > > > Test seems to be failing with > ``` > ava.lang.AssertionError: Not enough records produced by source connector. > Expected at least: 100 + but got 72 > h4. Stacktrace > java.lang.AssertionError: Not enough records produced by source connector. > Expected at least: 100 + but got 72 > at org.junit.Assert.fail(Assert.java:89) > at org.junit.Assert.assertTrue(Assert.java:42) > at > org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest.testConnectorBoundary(ExactlyOnceSourceIntegrationTest.java:421) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > at > org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) > at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) > at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > at org.junit.runners.ParentRunner.run(ParentRunner.java:413) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:108) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58) > at > org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:40) > at > org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:60) > at > org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:52) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36) > at > org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24) > at > org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33) > at > org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94) > at com.sun.proxy.$Proxy2.processTestClass(Unknown Source) > at > org.gradle.api.internal.tasks.testing.worker.TestWorker$2.run(TestWorker.java:176) > at > org.gradle.api.internal.tasks.testing.worker.TestWorker.executeAndMaintainThreadName(TestWorker.java:129) > at > org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:100) > at > org.gradle.api.internal.tasks.testing.worker.TestWorker.execute(TestWorker.java:60) > at > org.gradle.process.internal.worker.child.ActionExecutionWorker.execute(ActionExecutionWorker.java:56) > at > org.gradle.process.inter
[jira] [Updated] (KAFKA-15182) Normalize offsets before invoking SourceConnector::alterOffsets
[
https://issues.apache.org/jira/browse/KAFKA-15182?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton updated KAFKA-15182:
--
Affects Version/s: 3.6.0
> Normalize offsets before invoking SourceConnector::alterOffsets
> ---
>
> Key: KAFKA-15182
> URL: https://issues.apache.org/jira/browse/KAFKA-15182
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 3.6.0
>Reporter: Yash Mayya
>Assignee: Yash Mayya
>Priority: Major
> Fix For: 3.6.0
>
>
> See discussion
> [here|https://github.com/apache/kafka/pull/13945#discussion_r1260946148]
>
> TLDR: When users attempt to externally modify source connector offsets via
> the {{PATCH /offsets}} endpoint (introduced in
> [KIP-875|https://cwiki.apache.org/confluence/display/KAFKA/KIP-875%3A+First-class+offsets+support+in+Kafka+Connect]),
> type mismatches can occur between offsets passed to
> {{SourceConnector::alterOffsets}} and the offsets that are retrieved by
> connectors / tasks via an instance of {{OffsetStorageReader }}after the
> offsets have been modified. In order to prevent this type mismatch that could
> lead to subtle bugs in connectors, we could serialize + deserialize the
> offsets using the worker's internal JSON converter before invoking
> {{{}SourceConnector::alterOffsets{}}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-14669) Include MirrorMaker connector configurations in docs
[ https://issues.apache.org/jira/browse/KAFKA-14669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-14669. --- Fix Version/s: 3.6.0 Resolution: Done > Include MirrorMaker connector configurations in docs > > > Key: KAFKA-14669 > URL: https://issues.apache.org/jira/browse/KAFKA-14669 > Project: Kafka > Issue Type: Improvement > Components: docs >Reporter: Mickael Maison >Assignee: Gantigmaa Selenge >Priority: Major > Fix For: 3.6.0 > > > In the https://kafka.apache.org/documentation/#georeplication-flow-configure > section we list some of the MirrorMaker connectors configurations. These are > hardcoded in the docs: > https://github.com/apache/kafka/blob/trunk/docs/ops.html#L768-L788 > Instead we should used the generated docs (added as part of > https://github.com/apache/kafka/commit/40af3a74507cce9155f4fb4fca317d3c68235d78) > like we do for the file connectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (KAFKA-14669) Include MirrorMaker connector configurations in docs
[ https://issues.apache.org/jira/browse/KAFKA-14669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton reopened KAFKA-14669: --- > Include MirrorMaker connector configurations in docs > > > Key: KAFKA-14669 > URL: https://issues.apache.org/jira/browse/KAFKA-14669 > Project: Kafka > Issue Type: Improvement > Components: docs >Reporter: Mickael Maison >Assignee: Gantigmaa Selenge >Priority: Major > Fix For: 3.6.0 > > > In the https://kafka.apache.org/documentation/#georeplication-flow-configure > section we list some of the MirrorMaker connectors configurations. These are > hardcoded in the docs: > https://github.com/apache/kafka/blob/trunk/docs/ops.html#L768-L788 > Instead we should used the generated docs (added as part of > https://github.com/apache/kafka/commit/40af3a74507cce9155f4fb4fca317d3c68235d78) > like we do for the file connectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (KAFKA-14669) Include MirrorMaker connector configurations in docs
[ https://issues.apache.org/jira/browse/KAFKA-14669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744337#comment-17744337 ] Chris Egerton commented on KAFKA-14669: --- Reopening and marking as a blocker with fix version 3.6.0 so that we can merge [https://github.com/apache/kafka/pull/14041] (or another fix for the same issue) before this makes it into a release. > Include MirrorMaker connector configurations in docs > > > Key: KAFKA-14669 > URL: https://issues.apache.org/jira/browse/KAFKA-14669 > Project: Kafka > Issue Type: Improvement > Components: docs >Reporter: Mickael Maison >Assignee: Gantigmaa Selenge >Priority: Blocker > Fix For: 3.6.0 > > > In the https://kafka.apache.org/documentation/#georeplication-flow-configure > section we list some of the MirrorMaker connectors configurations. These are > hardcoded in the docs: > https://github.com/apache/kafka/blob/trunk/docs/ops.html#L768-L788 > Instead we should used the generated docs (added as part of > https://github.com/apache/kafka/commit/40af3a74507cce9155f4fb4fca317d3c68235d78) > like we do for the file connectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-14669) Include MirrorMaker connector configurations in docs
[ https://issues.apache.org/jira/browse/KAFKA-14669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-14669: -- Priority: Blocker (was: Major) > Include MirrorMaker connector configurations in docs > > > Key: KAFKA-14669 > URL: https://issues.apache.org/jira/browse/KAFKA-14669 > Project: Kafka > Issue Type: Improvement > Components: docs >Reporter: Mickael Maison >Assignee: Gantigmaa Selenge >Priority: Blocker > Fix For: 3.6.0 > > > In the https://kafka.apache.org/documentation/#georeplication-flow-configure > section we list some of the MirrorMaker connectors configurations. These are > hardcoded in the docs: > https://github.com/apache/kafka/blob/trunk/docs/ops.html#L768-L788 > Instead we should used the generated docs (added as part of > https://github.com/apache/kafka/commit/40af3a74507cce9155f4fb4fca317d3c68235d78) > like we do for the file connectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15091) Javadocs for SourceTask::commit are incorrect
[
https://issues.apache.org/jira/browse/KAFKA-15091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton updated KAFKA-15091:
--
Fix Version/s: 3.5.2
> Javadocs for SourceTask::commit are incorrect
> -
>
> Key: KAFKA-15091
> URL: https://issues.apache.org/jira/browse/KAFKA-15091
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Chris Egerton
>Assignee: Yash Mayya
>Priority: Major
> Fix For: 3.6.0, 3.5.2
>
>
> The Javadocs for {{SourceTask::commit}} state that the method should:
> {quote}Commit the offsets, up to the offsets that have been returned by
> [{{poll()}}|https://kafka.apache.org/34/javadoc/org/apache/kafka/connect/source/SourceTask.html#poll()].
> {quote}
> However, this is obviously incorrect given how the Connect runtime (when not
> configured with exactly-once support for source connectors) performs polling
> and offset commits on separate threads. There's also some extensive
> discussion on the semantics of that method in KAFKA-5716 where it's made
> clear that altering the behavior of the runtime to align with the documented
> semantics of that method is not a viable option.
> We should update the Javadocs for this method to state that it does not have
> anything to do with the offsets returned from {{SourceTask:poll}} and is
> instead just a general, periodically-invoked hook to let the task know that
> an offset commit has taken place (but with no guarantees as to which offsets
> have been committed and which ones correspond to still-in-flight records).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15091) Javadocs for SourceTask::commit are incorrect
[
https://issues.apache.org/jira/browse/KAFKA-15091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton updated KAFKA-15091:
--
Fix Version/s: 3.4.2
> Javadocs for SourceTask::commit are incorrect
> -
>
> Key: KAFKA-15091
> URL: https://issues.apache.org/jira/browse/KAFKA-15091
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Chris Egerton
>Assignee: Yash Mayya
>Priority: Major
> Fix For: 3.6.0, 3.4.2, 3.5.2
>
>
> The Javadocs for {{SourceTask::commit}} state that the method should:
> {quote}Commit the offsets, up to the offsets that have been returned by
> [{{poll()}}|https://kafka.apache.org/34/javadoc/org/apache/kafka/connect/source/SourceTask.html#poll()].
> {quote}
> However, this is obviously incorrect given how the Connect runtime (when not
> configured with exactly-once support for source connectors) performs polling
> and offset commits on separate threads. There's also some extensive
> discussion on the semantics of that method in KAFKA-5716 where it's made
> clear that altering the behavior of the runtime to align with the documented
> semantics of that method is not a viable option.
> We should update the Javadocs for this method to state that it does not have
> anything to do with the offsets returned from {{SourceTask:poll}} and is
> instead just a general, periodically-invoked hook to let the task know that
> an offset commit has taken place (but with no guarantees as to which offsets
> have been committed and which ones correspond to still-in-flight records).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15091) Javadocs for SourceTask::commit are incorrect
[
https://issues.apache.org/jira/browse/KAFKA-15091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton updated KAFKA-15091:
--
Fix Version/s: 3.3.3
> Javadocs for SourceTask::commit are incorrect
> -
>
> Key: KAFKA-15091
> URL: https://issues.apache.org/jira/browse/KAFKA-15091
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Chris Egerton
>Assignee: Yash Mayya
>Priority: Major
> Fix For: 3.3.3, 3.6.0, 3.4.2, 3.5.2
>
>
> The Javadocs for {{SourceTask::commit}} state that the method should:
> {quote}Commit the offsets, up to the offsets that have been returned by
> [{{poll()}}|https://kafka.apache.org/34/javadoc/org/apache/kafka/connect/source/SourceTask.html#poll()].
> {quote}
> However, this is obviously incorrect given how the Connect runtime (when not
> configured with exactly-once support for source connectors) performs polling
> and offset commits on separate threads. There's also some extensive
> discussion on the semantics of that method in KAFKA-5716 where it's made
> clear that altering the behavior of the runtime to align with the documented
> semantics of that method is not a viable option.
> We should update the Javadocs for this method to state that it does not have
> anything to do with the offsets returned from {{SourceTask:poll}} and is
> instead just a general, periodically-invoked hook to let the task know that
> an offset commit has taken place (but with no guarantees as to which offsets
> have been committed and which ones correspond to still-in-flight records).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14669) Include MirrorMaker connector configurations in docs
[ https://issues.apache.org/jira/browse/KAFKA-14669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-14669: -- Priority: Major (was: Blocker) > Include MirrorMaker connector configurations in docs > > > Key: KAFKA-14669 > URL: https://issues.apache.org/jira/browse/KAFKA-14669 > Project: Kafka > Issue Type: Improvement > Components: docs >Reporter: Mickael Maison >Assignee: Gantigmaa Selenge >Priority: Major > Fix For: 3.6.0 > > > In the https://kafka.apache.org/documentation/#georeplication-flow-configure > section we list some of the MirrorMaker connectors configurations. These are > hardcoded in the docs: > https://github.com/apache/kafka/blob/trunk/docs/ops.html#L768-L788 > Instead we should used the generated docs (added as part of > https://github.com/apache/kafka/commit/40af3a74507cce9155f4fb4fca317d3c68235d78) > like we do for the file connectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-14669) Include MirrorMaker connector configurations in docs
[ https://issues.apache.org/jira/browse/KAFKA-14669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton resolved KAFKA-14669. --- Resolution: Done > Include MirrorMaker connector configurations in docs > > > Key: KAFKA-14669 > URL: https://issues.apache.org/jira/browse/KAFKA-14669 > Project: Kafka > Issue Type: Improvement > Components: docs >Reporter: Mickael Maison >Assignee: Gantigmaa Selenge >Priority: Major > Fix For: 3.6.0 > > > In the https://kafka.apache.org/documentation/#georeplication-flow-configure > section we list some of the MirrorMaker connectors configurations. These are > hardcoded in the docs: > https://github.com/apache/kafka/blob/trunk/docs/ops.html#L768-L788 > Instead we should used the generated docs (added as part of > https://github.com/apache/kafka/commit/40af3a74507cce9155f4fb4fca317d3c68235d78) > like we do for the file connectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15216) InternalSinkRecord::newRecord method ignores the headers argument
[
https://issues.apache.org/jira/browse/KAFKA-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton updated KAFKA-15216:
--
Fix Version/s: 3.5.2
> InternalSinkRecord::newRecord method ignores the headers argument
> -
>
> Key: KAFKA-15216
> URL: https://issues.apache.org/jira/browse/KAFKA-15216
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Yash Mayya
>Assignee: Yash Mayya
>Priority: Minor
> Fix For: 3.6.0, 3.5.2
>
>
> [https://github.com/apache/kafka/blob/a1f6ab69387deb10988461152a0087f0cd2827c4/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/InternalSinkRecord.java#L50-L56]
> - the headers argument passed to the {{InternalSinkRecord}} constructor is
> the instance field via the accessor {{headers()}} method instead of the
> {{newRecord}} method's {{headers}} argument value.
>
> Originally discovered
> [here.|https://github.com/apache/kafka/pull/14024#discussion_r1266917499]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15216) InternalSinkRecord::newRecord method ignores the headers argument
[
https://issues.apache.org/jira/browse/KAFKA-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton updated KAFKA-15216:
--
Fix Version/s: 3.4.2
> InternalSinkRecord::newRecord method ignores the headers argument
> -
>
> Key: KAFKA-15216
> URL: https://issues.apache.org/jira/browse/KAFKA-15216
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Yash Mayya
>Assignee: Yash Mayya
>Priority: Minor
> Fix For: 3.6.0, 3.4.2, 3.5.2
>
>
> [https://github.com/apache/kafka/blob/a1f6ab69387deb10988461152a0087f0cd2827c4/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/InternalSinkRecord.java#L50-L56]
> - the headers argument passed to the {{InternalSinkRecord}} constructor is
> the instance field via the accessor {{headers()}} method instead of the
> {{newRecord}} method's {{headers}} argument value.
>
> Originally discovered
> [here.|https://github.com/apache/kafka/pull/14024#discussion_r1266917499]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-15216) InternalSinkRecord::newRecord method ignores the headers argument
[
https://issues.apache.org/jira/browse/KAFKA-15216?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton updated KAFKA-15216:
--
Fix Version/s: 3.3.3
> InternalSinkRecord::newRecord method ignores the headers argument
> -
>
> Key: KAFKA-15216
> URL: https://issues.apache.org/jira/browse/KAFKA-15216
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect
>Reporter: Yash Mayya
>Assignee: Yash Mayya
>Priority: Minor
> Fix For: 3.3.3, 3.6.0, 3.4.2, 3.5.2
>
>
> [https://github.com/apache/kafka/blob/a1f6ab69387deb10988461152a0087f0cd2827c4/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/InternalSinkRecord.java#L50-L56]
> - the headers argument passed to the {{InternalSinkRecord}} constructor is
> the instance field via the accessor {{headers()}} method instead of the
> {{newRecord}} method's {{headers}} argument value.
>
> Originally discovered
> [here.|https://github.com/apache/kafka/pull/14024#discussion_r1266917499]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (KAFKA-13431) Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit users)
[
https://issues.apache.org/jira/browse/KAFKA-13431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton resolved KAFKA-13431.
---
Fix Version/s: 3.6.0
Resolution: Done
> Sink Connectors: Support topic-mutating SMTs for async connectors (preCommit
> users)
> ---
>
> Key: KAFKA-13431
> URL: https://issues.apache.org/jira/browse/KAFKA-13431
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Reporter: Diego Erdody
>Assignee: Yash Mayya
>Priority: Major
> Labels: needs-kip
> Fix For: 3.6.0
>
>
> There's currently an incompatibility between Sink connectors overriding the
> {{SinkTask.preCommit}} method (for asynchronous processing) and SMTs that
> mutate the topic field.
> The problem was present since the {{preCommit}} method inception and is
> rooted in a mismatch between the topic/partition that is passed to
> {{open/preCommit}} (the original topic and partition before applying any
> transformations) and the topic partition that is present in the SinkRecord
> that the {{SinkTask.put}} method receives (after transformations are
> applied). Since that's all the information the connector has to implement any
> kind of internal offset tracking, the topic/partitions it can return in
> preCommit will correspond to the transformed topic, when the framework
> actually expects it to be the original topic.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (KAFKA-14112) Expose replication-offset-lag Mirror metric
[
https://issues.apache.org/jira/browse/KAFKA-14112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton updated KAFKA-14112:
--
Labels: needs-kip (was: )
> Expose replication-offset-lag Mirror metric
> ---
>
> Key: KAFKA-14112
> URL: https://issues.apache.org/jira/browse/KAFKA-14112
> Project: Kafka
> Issue Type: Improvement
> Components: mirrormaker
>Reporter: Elkhan Eminov
>Assignee: Elkhan Eminov
>Priority: Minor
> Labels: needs-kip
>
> The replication offset lag is the difference of the {*}l{*}ast {*}e{*}nd
> {*}o{*}ffset of the source partition (LEO) the {*}l{*}ast {*}r{*}eplicated
> _source_ {*}o{*}ffset (LRO), +1 (to account for zero-based offset numbering)
> The offset lag is a difference (LEO-LRO), and its constituents are calculated
> at different points of time and place:
> * LEO shall be calculated during source task's poll loop (ready to get it
> from the consumer)
> * LRO shall be kept in an in-memory "cache", that is updated during the
> task's producer callback
> The difference shall be calculated when the freshest LEO acquired in the poll
> loop. The calculated amount shall be defined as a MirrorMaker metric.
> This would describe to amount of "to be replicated" number of records for a
> certain topic-partition.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-15249) Verify Connect test-plugins artifact is published to Maven Central
Chris Egerton created KAFKA-15249:
-
Summary: Verify Connect test-plugins artifact is published to
Maven Central
Key: KAFKA-15249
URL: https://issues.apache.org/jira/browse/KAFKA-15249
Project: Kafka
Issue Type: Task
Affects Versions: 3.6.0
Reporter: Chris Egerton
Assignee: Chris Egerton
Fix For: 3.6.0
In KAFKA-14759 we created a separate {{connect/test-plugins}} module to store
all testing-only Connect plugins and removed those plugins from existing
Connect modules.
These testing-only plugins are intentionally excluded from the project's
release file (which can be generated with {{{}./gradlew releaseTarGz{}}})
however, some users may still be relying on them for testing environments.
Although we should refrain from distributing these testing-only plugins with
our out-of-the-box distribution of Connect, we should still ensure that they're
available on an opt-in basis to users who would like to continue using them.
This can be accomplished by publishing them to [Maven
Central|https://search.maven.org/], like we do with our other modules.
This will probably happen automatically during the next release (3.6.0) with no
further action required. This ticket is just here as a reminder to verify that
the artifacts are present in the staging Maven repo when release candidates are
published for voting.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (KAFKA-15252) Task is not stopped until the poll interval passes in case of task restarting.
[ https://issues.apache.org/jira/browse/KAFKA-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17748778#comment-17748778 ] Chris Egerton commented on KAFKA-15252: --- I think this is caused by KAFKA-15090. [~nikita.krasnov] do you think we can close this as a duplicate, or is this a different issue? > Task is not stopped until the poll interval passes in case of task restarting. > -- > > Key: KAFKA-15252 > URL: https://issues.apache.org/jira/browse/KAFKA-15252 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Nikita >Priority: Major > > We face a problem with restarting of the tasks, sometimes it leads to > resource leak. > We used the jdbc source connector and noticed an increasing of count of > opened sessions on Vertica side. But this problem is applicable for all > databases and possibly for all source connectors. > Our case is the next: > 1) Run jdbc source connector (io.confluent.connect.jdbc.JdbcSourceConnector) > and set poll.interval.ms (8640) > task.shutdown.graceful.timeout.ms (it's > the property on Kafka-connect side, we set 1) > 2) Send POST /connectors//tasks//restart > ER: count of session is the same as before restart > AR: count of session increases > The main problem is when > org.apache.kafka.connect.runtime.Worker#stopAndAwaitTasks(java.util.Collection) > method is called it doesn't stop a source task itself. > The source task stops only if polling process stops on source task side. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (KAFKA-15252) Task is not stopped until the poll interval passes in case of task restarting.
[ https://issues.apache.org/jira/browse/KAFKA-15252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17748778#comment-17748778 ] Chris Egerton edited comment on KAFKA-15252 at 7/29/23 3:12 AM: I think this is caused by KAFKA-15090. [~nikita.krasnov] do you think we can close this as a duplicate, or is it a different issue? was (Author: chrisegerton): I think this is caused by KAFKA-15090. [~nikita.krasnov] do you think we can close this as a duplicate, or is this a different issue? > Task is not stopped until the poll interval passes in case of task restarting. > -- > > Key: KAFKA-15252 > URL: https://issues.apache.org/jira/browse/KAFKA-15252 > Project: Kafka > Issue Type: Bug > Components: KafkaConnect >Reporter: Nikita >Priority: Major > > We face a problem with restarting of the tasks, sometimes it leads to > resource leak. > We used the jdbc source connector and noticed an increasing of count of > opened sessions on Vertica side. But this problem is applicable for all > databases and possibly for all source connectors. > Our case is the next: > 1) Run jdbc source connector (io.confluent.connect.jdbc.JdbcSourceConnector) > and set poll.interval.ms (8640) > task.shutdown.graceful.timeout.ms (it's > the property on Kafka-connect side, we set 1) > 2) Send POST /connectors//tasks//restart > ER: count of session is the same as before restart > AR: count of session increases > The main problem is when > org.apache.kafka.connect.runtime.Worker#stopAndAwaitTasks(java.util.Collection) > method is called it doesn't stop a source task itself. > The source task stops only if polling process stops on source task side. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15249) Verify Connect test-plugins artifact is published to Maven Central
[
https://issues.apache.org/jira/browse/KAFKA-15249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton updated KAFKA-15249:
--
Component/s: KafkaConnect
> Verify Connect test-plugins artifact is published to Maven Central
> --
>
> Key: KAFKA-15249
> URL: https://issues.apache.org/jira/browse/KAFKA-15249
> Project: Kafka
> Issue Type: Task
> Components: KafkaConnect
>Affects Versions: 3.6.0
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Blocker
> Fix For: 3.6.0
>
>
> In KAFKA-14759 we created a separate {{connect/test-plugins}} module to store
> all testing-only Connect plugins and removed those plugins from existing
> Connect modules.
> These testing-only plugins are intentionally excluded from the project's
> release file (which can be generated with {{{}./gradlew releaseTarGz{}}})
> however, some users may still be relying on them for testing environments.
> Although we should refrain from distributing these testing-only plugins with
> our out-of-the-box distribution of Connect, we should still ensure that
> they're available on an opt-in basis to users who would like to continue
> using them. This can be accomplished by publishing them to [Maven
> Central|https://search.maven.org/], like we do with our other modules.
> This will probably happen automatically during the next release (3.6.0) with
> no further action required. This ticket is just here as a reminder to verify
> that the artifacts are present in the staging Maven repo when release
> candidates are published for voting.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (KAFKA-15563) Provide informative error messages when Connect REST requests time out
Chris Egerton created KAFKA-15563: - Summary: Provide informative error messages when Connect REST requests time out Key: KAFKA-15563 URL: https://issues.apache.org/jira/browse/KAFKA-15563 Project: Kafka Issue Type: Improvement Components: KafkaConnect Reporter: Chris Egerton Assignee: Chris Egerton The Kafka Connect REST API has a hardcoded timeout of 90 seconds. If any operations take longer than that, a 500 error response is returned with the message "Request timed out" (see [here|https://github.com/apache/kafka/blob/7e1c453af9533aba8c19da2d08ce6595c1441fc0/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/rest/HerderRequestHandler.java#L70]). This can be a source of frustration for users, who want to understand what is causing the request to time out. This can be specific to the request (for example, a connector's [custom multi-property validation logic|https://kafka.apache.org/35/javadoc/org/apache/kafka/connect/connector/Connector.html#validate(java.util.Map)] is taking too long), or applicable to any request that goes through the herder's tick thread (for which there are a variety of possible causes). We can give users better, immediate insight into what is causing requests to time out by including information about the last possibly-blocking operation the worker performed while servicing the request (or attempting to enter a state where all preconditions necessary to service the request have been satisfied), and when the worker began that operation. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (KAFKA-15428) Cluster-wide dynamic log adjustments for Connect
[ https://issues.apache.org/jira/browse/KAFKA-15428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Egerton updated KAFKA-15428: -- Labels: kip (was: needs-kip) > Cluster-wide dynamic log adjustments for Connect > > > Key: KAFKA-15428 > URL: https://issues.apache.org/jira/browse/KAFKA-15428 > Project: Kafka > Issue Type: New Feature > Components: KafkaConnect >Reporter: Chris Egerton >Assignee: Chris Egerton >Priority: Major > Labels: kip > > [KIP-495|https://cwiki.apache.org/confluence/display/KAFKA/KIP-495%3A+Dynamically+Adjust+Log+Levels+in+Connect] > added REST APIs to view and adjust the logging levels of Kafka Connect > workers at runtime. This has been tremendously valuable (thank you > [~wicknicks]!), but one frequently-observed area for improvement is that the > API requires a REST request to be issued to each to-be-adjusted worker. > If possible, we should add support for adjusting the logging level of all > workers in a cluster with a single REST request. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (KAFKA-15249) Verify Connect test-plugins artifact is published to Maven Central
[
https://issues.apache.org/jira/browse/KAFKA-15249?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Egerton resolved KAFKA-15249.
---
Resolution: Done
> Verify Connect test-plugins artifact is published to Maven Central
> --
>
> Key: KAFKA-15249
> URL: https://issues.apache.org/jira/browse/KAFKA-15249
> Project: Kafka
> Issue Type: Task
> Components: KafkaConnect
>Affects Versions: 3.6.0
>Reporter: Chris Egerton
>Assignee: Chris Egerton
>Priority: Blocker
>
> In KAFKA-14759 we created a separate {{connect/test-plugins}} module to store
> all testing-only Connect plugins and removed those plugins from existing
> Connect modules.
> These testing-only plugins are intentionally excluded from the project's
> release file (which can be generated with {{{}./gradlew releaseTarGz{}}})
> however, some users may still be relying on them for testing environments.
> Although we should refrain from distributing these testing-only plugins with
> our out-of-the-box distribution of Connect, we should still ensure that
> they're available on an opt-in basis to users who would like to continue
> using them. This can be accomplished by publishing them to [Maven
> Central|https://search.maven.org/], like we do with our other modules.
> This will probably happen automatically during the next release (3.6.0) with
> no further action required. This ticket is just here as a reminder to verify
> that the artifacts are present in the staging Maven repo when release
> candidates are published for voting.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
