[jira] [Created] (KAFKA-15010) KRaft Controller doesn't reconcile with Zookeeper metadata upon becoming new controller while in dual write mode.

[Akhilesh Chaganti (Jira)]

Akhilesh Chaganti created KAFKA-15010:
-

 Summary: KRaft Controller doesn't reconcile with Zookeeper 
metadata upon becoming new controller while in dual write mode.
 Key: KAFKA-15010
 URL: https://issues.apache.org/jira/browse/KAFKA-15010
 Project: Kafka
  Issue Type: Bug
  Components: kraft
Reporter: Akhilesh Chaganti


When a KRaft controller fails over, the existing migration driver (in dual 
write mode) can fail in between Zookeeper writes and may leave Zookeeper with 
incomplete and inconsistent data. So when a new controller becomes active (and 
by extension new migration driver becomes active), this first thing we should 
do is load the in-memory snapshot and use it to write metadata to Zookeeper to 
have a steady state. We currently do not do this and it may leave Zookeeper in 
inconsistent state.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (KAFKA-15009) ClientQuotas and ACLs are not correctly synchronized while handling snapshot during migration (dual write)

[Akhilesh Chaganti (Jira)]

Akhilesh Chaganti created KAFKA-15009:
-

 Summary: ClientQuotas and ACLs are not correctly synchronized 
while handling snapshot during migration (dual write)
 Key: KAFKA-15009
 URL: https://issues.apache.org/jira/browse/KAFKA-15009
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.5.0
Reporter: Akhilesh Chaganti
Assignee: Akhilesh Chaganti






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Akhilesh Chaganti]

Hi Mickael,

I raised the blockers for AK 3.5 and raised PRs for them. All of them are
in the Zk -> KRaft migration path and critical for the Zk -> KRaft
migration to be successful.
KAFKA-15003  -- Topic
metadata is not correctly synced with Zookeeper while handling snapshot
during migration (dual write). --
KAFKA-15004  -- Config
changes are not correctly synced with Zookeeper whale handling snapshot
during migration (dual write). --
KAFKA-15007  --
MigrationPropagator may have wrong IBP while sending UMR, LISR requests to
the Zk Brokers during migration. --


Thanks
Akhilesh


On Fri, May 19, 2023 at 9:28 AM Matthias J. Sax  wrote:

> Mickael,
>
> we included a bug-fix into 3.5, and just discovered a critical bug in
> the fix itself, that would introduce a regression into 3.5.
>
> We have already a PR to fix-forward:
> https://github.com/apache/kafka/pull/13734
>
> As we don't have an RC yet, I would like to cherry-pick this back to 3.5.
>
>
> -Matthias
>
> On 5/10/23 1:47 PM, Sophie Blee-Goldman wrote:
> > Thanks Mickael -- the fix has been merged to 3.5 now
> >
> > On Wed, May 10, 2023 at 1:12 AM Mickael Maison  >
> > wrote:
> >
> >> Hi Sophie,
> >>
> >> Yes that's fine, thanks for letting me know!
> >>
> >> Mickael
> >>
> >> On Tue, May 9, 2023 at 10:54 PM Sophie Blee-Goldman
> >>  wrote:
> >>>
> >>> Hey Mickael, I noticed a bug in the new versioned key-value byte store
> >>> where it's delegating to the wrong API
> >>> (copy/paste error I assume). I extracted this into its own PR which I
> >> think
> >>> should be included in the 3.5 release.
> >>>
> >>> The tests are still running, but it's just a one-liner so I'll merge it
> >>> when they're done, and cherrypick to 3.5 if
> >>> that's ok with you. See https://github.com/apache/kafka/pull/13695
> >>>
> >>> Thanks for running the release!
> >>>
> >>> On Tue, May 9, 2023 at 1:28 PM Randall Hauch  wrote:
> >>>
>  Thanks, Mickael.
> 
>  I've cherry-picked that commit to the `3.5` branch (
>  https://issues.apache.org/jira/browse/KAFKA-14974).
> 
>  Best regards,
>  Randall
> 
>  On Tue, May 9, 2023 at 2:13 PM Mickael Maison <
> >> mickael.mai...@gmail.com>
>  wrote:
> 
> > Hi Randall/Luke,
> >
> > Yes you can go ahead and merge these into 3.5. I've not started
> >> making
> > a release yet because:
> > - I found a regression today in MirrorMaker:
> > https://issues.apache.org/jira/browse/KAFKA-14980
> > - The 3.5 branch builder job in Jenkins has been disabled:
> > https://issues.apache.org/jira/browse/INFRA-24577
> >
> > Thanks,
> > Mickael
> >
> > On Tue, May 9, 2023 at 8:40 PM Luke Chen  wrote:
> >>
> >> Hi Mickael,
> >>
> >> Since we haven't had the CR created yet, I'm thinking we should
>  backport
> >> this doc improvement to v3.5.0 to make the doc complete.
> >> https://github.com/apache/kafka/pull/13660
> >>
> >> What do you think?
> >>
> >> Luke
> >>
> >> On Sat, May 6, 2023 at 11:31 PM David Arthur 
> >> wrote:
> >>
> >>> I resolved these three:
> >>> * KAFKA-14840 is merged to trunk and 3.5. I removed the 3.4.1 fix
> > version
> >>> * KAFKA-14805 is merged to trunk and 3.5
> >>> * KAFKA-14918 is merged to trunk and 3.5
> >>>
> >>> KAFKA-14692 (docs issue) is still a not done
> >>>
> >>> Looks like KAFKA-14084 is now resolved as well (it's in trunk and
>  3.5).
> >>>
> >>> I'll try to find out about KAFKA-14698, I think it's likely a
>  WONTFIX.
> >>>
> >>> -David
> >>>
> >>> On Fri, May 5, 2023 at 10:43 AM Mickael Maison <
> > mickael.mai...@gmail.com>
> >>> wrote:
> >>>
>  Hi David,
> 
>  Thanks for the update!
>  You still own 4 other tickets targeting 3.5: KAFKA-14840,
> > KAFKA-14805,
>  KAFKA-14918, KAFKA-14692. Should I move all of them to the next
>  release?
>  Also KAFKA-14698 and KAFKA-14084 are somewhat related to the
>  migration. Should I move them too?
> 
>  Thanks,
>  Mickael
> 
>  On Fri, May 5, 2023 at 4:27 PM David Arthur
>   wrote:
> >
> > Hey Mickael, my two ZK migration fixes are in 3.5 now.
> >
> > Cheers,
> > David
> >
> > On Fri, May 5, 2023 at 9:37 AM Mickael Maison <
> >>> mickael.mai...@gmail.com>
> > wrote:
> >
> >> Hi Divij,
> >>
> >> Some dependencies (ZooKeeper, Snappy, Swagger, zstd, etc)
> >> have
> > been
> >> updated since 3.4.
> >> Regarding your PR, I would have been in favor of bringing
> >> this
> > to 3.5
> >> a couple of weeks ago, but we're now a week past code
> >> freeze
>  

Re: [DISCUSS] Cluster Linking / Cross-Cluster Replication - Call for Interest & Co-Authors

[Thomas Scott]

Hi Greg,

  This sounds very interesting and I think it is a very valuable feature.
I'd like to volunteer in any way I can be useful and am more than happy to
help with co authoring KIPs and implementation.


Thanks

  Tom




On Fri, 19 May 2023, 23:15 Colt McNealy,  wrote:

> I'm highly interested in this feature. We are a startup right now so we
> can't commit resources to help *yet*, but once we are off the ground in 1-2
> years it would be cool to contribute to this.
>
> Colt McNealy
> *Founder, LittleHorse.dev*
>
>
> On Fri, May 19, 2023 at 10:14 AM hzh0425  wrote:
>
> > Hi, Grep Harris.
> > When I saw this discussion, I was very excited.
> > My team is a Kafka R team, currently planning research and development
> > on cluster linking. We believe that it is of great value in scenarios
> such
> > as multi region disaster recovery and cross cloud synchronization.
> > Therefore, I hope to join the subsequent design and development of this
> > KIP as a co-author for deep participation.
> >
> > Thanks!
> > hzh
> >
> >
> >
> >
> >
> >  回复的原邮件 
> > | 发件人 | Greg Harris |
> > | 日期 | 2023年05月20日 00:56 |
> > | 收件人 | dev@kafka.apache.org |
> > | 抄送至 | |
> > | 主题 | [DISCUSS] Cluster Linking / Cross-Cluster Replication - Call for
> > Interest & Co-Authors |
> > Hey all,
> >
> > I have heard some offline discussion around Cross-Cluster Replication,
> > a feature which would enable data to be shared between Kafka clusters
> > in a manner which preserves offsets.
> > Other names for this feature may include "Cluster Linking", "Remote
> > Topics", "Byte-for-Byte Mirroring", and similar.
> > I believe that there may be an interest in this feature being added to
> > Apache Kafka, but have not seen any mailing list activity to that
> > effect yet.
> >
> > Due to the complexity of the feature, I think it is appropriate to
> > gauge the interest in the feature before opening a KIP, and also
> > gather a set of contributors which are interested in collaborating on
> > and sponsoring its development.
> >
> > If you are a Kafka user, please:
> > 1. Let us know in this thread if you think Cross-Cluster Replication
> > is a valuable feature and would like to see it in Apache Kafka.
> >
> > If you are or want to be an individual contributor, please:
> > 1. Discuss with your users and/or product organization if
> > Cross-Cluster Replication is appealing, and summarize their level of
> > interest in the below thread.
> > 2. Consider volunteering in this thread as a reviewer or co-author of
> > the KIP and subsequent implementation. Co-authors would share the
> > design and implementation workload as their time and interest permits.
> > 3. If you have an employer, request that they sponsor this feature
> > through your full- or part-time contributions as a reviewer or
> > co-author. In exchange for their sponsorship, they can help ensure
> > that the feature is delivered with a more predictable timeline.
> >
> > Please save your technical expectations and details of this feature
> > for a subsequent KIP and DISCUSS thread, where that discussion is more
> > appropriate.
> >
> > Thanks Everyone!
> >
> > Greg Harris
> > Aiven, Inc
> >
>

Re: [DISCUSS] Cluster Linking / Cross-Cluster Replication - Call for Interest & Co-Authors

[Colt McNealy]

I'm highly interested in this feature. We are a startup right now so we
can't commit resources to help *yet*, but once we are off the ground in 1-2
years it would be cool to contribute to this.

Colt McNealy
*Founder, LittleHorse.dev*


On Fri, May 19, 2023 at 10:14 AM hzh0425  wrote:

> Hi, Grep Harris.
> When I saw this discussion, I was very excited.
> My team is a Kafka R team, currently planning research and development
> on cluster linking. We believe that it is of great value in scenarios such
> as multi region disaster recovery and cross cloud synchronization.
> Therefore, I hope to join the subsequent design and development of this
> KIP as a co-author for deep participation.
>
> Thanks!
> hzh
>
>
>
>
>
>  回复的原邮件 
> | 发件人 | Greg Harris |
> | 日期 | 2023年05月20日 00:56 |
> | 收件人 | dev@kafka.apache.org |
> | 抄送至 | |
> | 主题 | [DISCUSS] Cluster Linking / Cross-Cluster Replication - Call for
> Interest & Co-Authors |
> Hey all,
>
> I have heard some offline discussion around Cross-Cluster Replication,
> a feature which would enable data to be shared between Kafka clusters
> in a manner which preserves offsets.
> Other names for this feature may include "Cluster Linking", "Remote
> Topics", "Byte-for-Byte Mirroring", and similar.
> I believe that there may be an interest in this feature being added to
> Apache Kafka, but have not seen any mailing list activity to that
> effect yet.
>
> Due to the complexity of the feature, I think it is appropriate to
> gauge the interest in the feature before opening a KIP, and also
> gather a set of contributors which are interested in collaborating on
> and sponsoring its development.
>
> If you are a Kafka user, please:
> 1. Let us know in this thread if you think Cross-Cluster Replication
> is a valuable feature and would like to see it in Apache Kafka.
>
> If you are or want to be an individual contributor, please:
> 1. Discuss with your users and/or product organization if
> Cross-Cluster Replication is appealing, and summarize their level of
> interest in the below thread.
> 2. Consider volunteering in this thread as a reviewer or co-author of
> the KIP and subsequent implementation. Co-authors would share the
> design and implementation workload as their time and interest permits.
> 3. If you have an employer, request that they sponsor this feature
> through your full- or part-time contributions as a reviewer or
> co-author. In exchange for their sponsorship, they can help ensure
> that the feature is delivered with a more predictable timeline.
>
> Please save your technical expectations and details of this feature
> for a subsequent KIP and DISCUSS thread, where that discussion is more
> appropriate.
>
> Thanks Everyone!
>
> Greg Harris
> Aiven, Inc
>

Re: [DISCUSS] Cluster Linking / Cross-Cluster Replication - Call for Interest & Co-Authors

[hzh0425]

Hi, Grep Harris.
When I saw this discussion, I was very excited.
My team is a Kafka R team, currently planning research and development on 
cluster linking. We believe that it is of great value in scenarios such as 
multi region disaster recovery and cross cloud synchronization.
Therefore, I hope to join the subsequent design and development of this KIP as 
a co-author for deep participation.

Thanks!
hzh





 回复的原邮件 
| 发件人 | Greg Harris |
| 日期 | 2023年05月20日 00:56 |
| 收件人 | dev@kafka.apache.org |
| 抄送至 | |
| 主题 | [DISCUSS] Cluster Linking / Cross-Cluster Replication - Call for 
Interest & Co-Authors |
Hey all,

I have heard some offline discussion around Cross-Cluster Replication,
a feature which would enable data to be shared between Kafka clusters
in a manner which preserves offsets.
Other names for this feature may include "Cluster Linking", "Remote
Topics", "Byte-for-Byte Mirroring", and similar.
I believe that there may be an interest in this feature being added to
Apache Kafka, but have not seen any mailing list activity to that
effect yet.

Due to the complexity of the feature, I think it is appropriate to
gauge the interest in the feature before opening a KIP, and also
gather a set of contributors which are interested in collaborating on
and sponsoring its development.

If you are a Kafka user, please:
1. Let us know in this thread if you think Cross-Cluster Replication
is a valuable feature and would like to see it in Apache Kafka.

If you are or want to be an individual contributor, please:
1. Discuss with your users and/or product organization if
Cross-Cluster Replication is appealing, and summarize their level of
interest in the below thread.
2. Consider volunteering in this thread as a reviewer or co-author of
the KIP and subsequent implementation. Co-authors would share the
design and implementation workload as their time and interest permits.
3. If you have an employer, request that they sponsor this feature
through your full- or part-time contributions as a reviewer or
co-author. In exchange for their sponsorship, they can help ensure
that the feature is delivered with a more predictable timeline.

Please save your technical expectations and details of this feature
for a subsequent KIP and DISCUSS thread, where that discussion is more
appropriate.

Thanks Everyone!

Greg Harris
Aiven, Inc

[DISCUSS] Cluster Linking / Cross-Cluster Replication - Call for Interest & Co-Authors

[Greg Harris]

Hey all,

I have heard some offline discussion around Cross-Cluster Replication,
a feature which would enable data to be shared between Kafka clusters
in a manner which preserves offsets.
Other names for this feature may include "Cluster Linking", "Remote
Topics", "Byte-for-Byte Mirroring", and similar.
I believe that there may be an interest in this feature being added to
Apache Kafka, but have not seen any mailing list activity to that
effect yet.

Due to the complexity of the feature, I think it is appropriate to
gauge the interest in the feature before opening a KIP, and also
gather a set of contributors which are interested in collaborating on
and sponsoring its development.

If you are a Kafka user, please:
1. Let us know in this thread if you think Cross-Cluster Replication
is a valuable feature and would like to see it in Apache Kafka.

If you are or want to be an individual contributor, please:
1. Discuss with your users and/or product organization if
Cross-Cluster Replication is appealing, and summarize their level of
interest in the below thread.
2. Consider volunteering in this thread as a reviewer or co-author of
the KIP and subsequent implementation. Co-authors would share the
design and implementation workload as their time and interest permits.
3. If you have an employer, request that they sponsor this feature
through your full- or part-time contributions as a reviewer or
co-author. In exchange for their sponsorship, they can help ensure
that the feature is delivered with a more predictable timeline.

Please save your technical expectations and details of this feature
for a subsequent KIP and DISCUSS thread, where that discussion is more
appropriate.

Thanks Everyone!

Greg Harris
Aiven, Inc

Re: [DISCUSS] Apache Kafka 3.5.0 release

[Matthias J. Sax]

Mickael,

we included a bug-fix into 3.5, and just discovered a critical bug in 
the fix itself, that would introduce a regression into 3.5.


We have already a PR to fix-forward: 
https://github.com/apache/kafka/pull/13734


As we don't have an RC yet, I would like to cherry-pick this back to 3.5.


-Matthias

On 5/10/23 1:47 PM, Sophie Blee-Goldman wrote:

Thanks Mickael -- the fix has been merged to 3.5 now

On Wed, May 10, 2023 at 1:12 AM Mickael Maison 
wrote:


Hi Sophie,

Yes that's fine, thanks for letting me know!

Mickael

On Tue, May 9, 2023 at 10:54 PM Sophie Blee-Goldman
 wrote:


Hey Mickael, I noticed a bug in the new versioned key-value byte store
where it's delegating to the wrong API
(copy/paste error I assume). I extracted this into its own PR which I

think

should be included in the 3.5 release.

The tests are still running, but it's just a one-liner so I'll merge it
when they're done, and cherrypick to 3.5 if
that's ok with you. See https://github.com/apache/kafka/pull/13695

Thanks for running the release!

On Tue, May 9, 2023 at 1:28 PM Randall Hauch  wrote:


Thanks, Mickael.

I've cherry-picked that commit to the `3.5` branch (
https://issues.apache.org/jira/browse/KAFKA-14974).

Best regards,
Randall

On Tue, May 9, 2023 at 2:13 PM Mickael Maison <

mickael.mai...@gmail.com>

wrote:


Hi Randall/Luke,

Yes you can go ahead and merge these into 3.5. I've not started

making

a release yet because:
- I found a regression today in MirrorMaker:
https://issues.apache.org/jira/browse/KAFKA-14980
- The 3.5 branch builder job in Jenkins has been disabled:
https://issues.apache.org/jira/browse/INFRA-24577

Thanks,
Mickael

On Tue, May 9, 2023 at 8:40 PM Luke Chen  wrote:


Hi Mickael,

Since we haven't had the CR created yet, I'm thinking we should

backport

this doc improvement to v3.5.0 to make the doc complete.
https://github.com/apache/kafka/pull/13660

What do you think?

Luke

On Sat, May 6, 2023 at 11:31 PM David Arthur 

wrote:



I resolved these three:
* KAFKA-14840 is merged to trunk and 3.5. I removed the 3.4.1 fix

version

* KAFKA-14805 is merged to trunk and 3.5
* KAFKA-14918 is merged to trunk and 3.5

KAFKA-14692 (docs issue) is still a not done

Looks like KAFKA-14084 is now resolved as well (it's in trunk and

3.5).


I'll try to find out about KAFKA-14698, I think it's likely a

WONTFIX.


-David

On Fri, May 5, 2023 at 10:43 AM Mickael Maison <

mickael.mai...@gmail.com>

wrote:


Hi David,

Thanks for the update!
You still own 4 other tickets targeting 3.5: KAFKA-14840,

KAFKA-14805,

KAFKA-14918, KAFKA-14692. Should I move all of them to the next
release?
Also KAFKA-14698 and KAFKA-14084 are somewhat related to the
migration. Should I move them too?

Thanks,
Mickael

On Fri, May 5, 2023 at 4:27 PM David Arthur
 wrote:


Hey Mickael, my two ZK migration fixes are in 3.5 now.

Cheers,
David

On Fri, May 5, 2023 at 9:37 AM Mickael Maison <

mickael.mai...@gmail.com>

wrote:


Hi Divij,

Some dependencies (ZooKeeper, Snappy, Swagger, zstd, etc)

have

been

updated since 3.4.
Regarding your PR, I would have been in favor of bringing

this

to 3.5

a couple of weeks ago, but we're now a week past code

freeze

for

3.5.

Apart if this fixes CVEs, or significant bugs, I think we

should

only

merge it in trunk.

Thanks,
Mickael

On Fri, May 5, 2023 at 1:49 PM Divij Vaidya <

divijvaidy...@gmail.com



wrote:


Hey Mickael

Should we consider performing an update of the minor

versions

of

the

dependencies in 3.5 (per

https://github.com/apache/kafka/pull/13673

)?


--
Divij Vaidya



On Tue, May 2, 2023 at 5:48 PM Mickael Maison <

mickael.mai...@gmail.com>

wrote:


Hi Luke,

Yes I think it makes sense to backport both to 3.5.

Thanks,
Mickael

On Tue, May 2, 2023 at 11:38 AM Luke Chen <

show...@gmail.com



wrote:


Hi Mickael,

There are 1 bug and 1 improvement that I'd like to

backport to

3.5.

1. A small improvement for ZK migration based on

KAFKA-14840

(mentioned

above in David's mail). PR is already merged to

trunk.

https://issues.apache.org/jira/browse/KAFKA-14909

2. A bug will cause the KRaft controller node to shut

down

unexpectedly.

PR

is ready for review.
https://issues.apache.org/jira/browse/KAFKA-14946
https://github.com/apache/kafka/pull/13653

Thanks.
Luke



On Fri, Apr 28, 2023 at 4:18 PM Mickael Maison <

mickael.mai...@gmail.com


wrote:


Hi David,

Yes you can backport these to 3.5. Let me know

when you

are

done.


Thanks,
Mickael

On Thu, Apr 27, 2023 at 9:02 PM David Arthur
 wrote:


Hey Mickael,

I have one major ZK migration improvement

(KAFKA-14805)

that

landed

in

trunk this week that I'd like to merge to 3.5

(once

we

fix

some

test

failures it introduced). After that, I have

another

PR

for

KAFKA-14840

which is essentially a huge bug in the ZK

migration

logic

that

needs

to

land in 3.5.



https://issues.apache.org/jira/browse/KAFKA-14805

(done)



https://issues.apache.org/jira/browse/KAFKA-14840

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1855

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-14980) MirrorMaker consumers don't get configs prefixed with source.cluster

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-14980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-14980.

Resolution: Fixed

> MirrorMaker consumers don't get configs prefixed with source.cluster
> 
>
> Key: KAFKA-14980
> URL: https://issues.apache.org/jira/browse/KAFKA-14980
> Project: Kafka
>  Issue Type: Bug
>  Components: mirrormaker
>Affects Versions: 3.5.0
>Reporter: Mickael Maison
>Assignee: Chris Egerton
>Priority: Blocker
> Fix For: 3.5.0
>
>
> As part of KAFKA-14021, we made a change to 
> MirrorConnectorConfig.sourceConsumerConfig() to grab all configs that start 
> with "source.". Previously it was grabbing configs prefixed with 
> "source.cluster.". 
> This means existing connector configuration stop working, as configurations 
> such as bootstrap.servers are not passed to source consumers.
> For example, the following connector configuration was valid in 3.4 and now 
> makes the connector tasks fail:
> {code:json}
> {
> "connector.class": 
> "org.apache.kafka.connect.mirror.MirrorSourceConnector",
> "name": "source",
> "topics": "test",
> "tasks.max": "30",
> "source.cluster.alias": "one",
> "target.cluster.alias": "two",
> "source.cluster.bootstrap.servers": "localhost:9092",
>"target.cluster.bootstrap.servers": "localhost:29092"
> }
> {code}
> The connector attempts to start source consumers with bootstrap.servers = [] 
> and the task crash with 
> {noformat}
> org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:837)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.(KafkaConsumer.java:671)
>   at 
> org.apache.kafka.connect.mirror.MirrorUtils.newConsumer(MirrorUtils.java:59)
>   at 
> org.apache.kafka.connect.mirror.MirrorSourceTask.start(MirrorSourceTask.java:103)
>   at 
> org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.initializeAndStart(AbstractWorkerSourceTask.java:274)
>   at 
> org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:202)
>   at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:259)
>   at 
> org.apache.kafka.connect.runtime.AbstractWorkerSourceTask.run(AbstractWorkerSourceTask.java:75)
>   at 
> org.apache.kafka.connect.runtime.isolation.Plugins.lambda$withClassLoader$1(Plugins.java:181)
>   at 
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
>   at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
>   at 
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
>   at 
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
>   at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: org.apache.kafka.common.config.ConfigException: No resolvable 
> bootstrap urls given in bootstrap.servers
> {noformat}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] 3.4.1 RC0

[Josep Prat]

Hi Luke,
This gets a +1 from my end. I believe non-binding because if I understand
it correctly, binding votes for releases are only issued by PMCs (
https://cwiki.apache.org/confluence/display/KAFKA/Release+Process#ReleaseProcess-Afterthevotepasses
).

I did the following validations:
- Verified signatures and checksums for all the generated artifacts
- Built from source with Java 11 and Scala 2.13.10
- Run unit tests
- Run integration tests
- Run the quickstart with Zookeeper and KRaft

Best,

On Wed, May 17, 2023 at 2:11 PM Josep Prat  wrote:

> Hi Luke,
>
> I ran the tests from the source package you created and I didn't get any
> of the test failures you had on your CI build. I got other flaky tests
> though, that after being run in isolation ran successfully. I'll try to run
> signature validation, and some further testing later today or later this
> week.
>
> Best,
>
> On Wed, May 17, 2023 at 12:43 PM Federico Valeri 
> wrote:
>
>> Hi Luke, thanks for running the release.
>>
>> Looks like the Maven artifacts are not in staging:
>>
>> https://repository.apache.org/content/groups/staging/org/apache/kafka/kafka-clients/3.4.1/
>>
>> Documentation still has 3.4.0, instead of 3.4.1 (not sure if this will
>> be aligned later):
>> https://kafka.apache.org/34/documentation.html#producerapi
>>
>> Br
>> Fede
>>
>>
>> On Wed, May 17, 2023 at 5:24 AM Luke Chen  wrote:
>> >
>> > Hello Kafka users, developers and client-developers,
>> >
>> > This is the first candidate for release of Apache Kafka 3.4.1.
>> >
>> > This is a bugfix release with several fixes since the release of 3.4.0.
>> A
>> > few of the major issues include:
>> > - core
>> > KAFKA-14644  Process
>> > should stop after failure in raft IO thread
>> > KAFKA-14946  KRaft
>> > controller node shutting down while renouncing leadership
>> > KAFKA-14887  ZK
>> session
>> > timeout can cause broker to shutdown
>> > - client
>> > KAFKA-14639  Kafka
>> > CooperativeStickyAssignor revokes/assigns partition in one rebalance
>> cycle
>> > - connect
>> > KAFKA-12558  MM2
>> may not
>> > sync partition offsets correctly
>> > KAFKA-14666  MM2
>> should
>> > translate consumer group offsets behind replication flow
>> > - stream
>> > KAFKA-14172  bug:
>> State
>> > stores lose state when tasks are reassigned under EOS
>> >
>> > Release notes for the 3.4.1 release:
>> > https://home.apache.org/~showuon/kafka-3.4.1-rc0/RELEASE_NOTES.html
>> >
>> > *** Please download, test and vote by May 24, 2023
>> > Kafka's KEYS file containing PGP keys we use to sign the release:
>> > https://kafka.apache.org/KEYS
>> >
>> > * Release artifacts to be voted upon (source and binary):
>> > https://home.apache.org/~showuon/kafka-3.4.1-rc0/
>> >
>> > * Maven artifacts to be voted upon:
>> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
>> >
>> > * Javadoc:
>> > https://home.apache.org/~showuon/kafka-3.4.1-rc0/javadoc/
>> >
>> > * Tag to be voted upon (off 3.4 branch) is the 3.4.1 tag:
>> > https://github.com/apache/kafka/releases/tag/3.4.1-rc0
>> >
>> > * Documentation:
>> > https://kafka.apache.org/34/documentation.html
>> >
>> > * Protocol:
>> > https://kafka.apache.org/34/protocol.html
>> >
>> > The most recent build has had test failures. These all appear to be due
>> to
>> > flakiness, but it would be nice if someone more familiar with the failed
>> > tests could confirm this. I may update this thread with passing build
>> links
>> > if I can get one, or start a new release vote thread if test failures
>> must
>> > be addressed beyond re-running builds until they pass.
>> >
>> > Unit/integration tests:
>> > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.4/133/
>> >
>> > System tests:
>> > Will update the results later
>> >
>> > Thank you.
>> > Luke
>>
>
>
> --
> [image: Aiven] 
>
> *Josep Prat*
> Open Source Engineering Director, *Aiven*
> josep.p...@aiven.io   |   +491715557497
> aiven.io    |
> 
>    
> *Aiven Deutschland GmbH*
> Alexanderufer 3-7, 10117 Berlin
> Geschäftsführer: Oskari Saarenmaa & Hannu Valtonen
> Amtsgericht Charlottenburg, HRB 209739 B
>


-- 
[image: Aiven] 

*Josep Prat*
Open Source Engineering Director, *Aiven*
josep.p...@aiven.io   |   +491715557497
aiven.io    |   
     
*Aiven Deutschland GmbH*
Alexanderufer 3-7, 10117 Berlin
Geschäftsführer: Oskari Saarenmaa & Hannu 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.3 #176

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-15008) GCS Sink Connector to parse JSON having leading 0's for an integer field

[Mickael Maison (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-15008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mickael Maison resolved KAFKA-15008.

Resolution: Invalid

The issue you describe does not seem to be in Kafka Connect. This connector is 
not developed by the Apache Kafka project. I suggest you report this issue to 
whoever is providing you this connector.

> GCS Sink Connector to parse JSON having leading 0's for an integer field
> 
>
> Key: KAFKA-15008
> URL: https://issues.apache.org/jira/browse/KAFKA-15008
> Project: Kafka
>  Issue Type: Bug
>Reporter: Lubna Naqvi
>Priority: Major
>
> Our Kafka data which is in JSON format has an attribute (gtin) which is a 
> number starting with Zero and Kafka Connect GCS Sink Connector fails to parse 
> it. gtin is a very common attribute across any item related Kafka source and 
> it is causing a major(blocking) issue as we are not able to parse this 
> attribute.
>  
> We'd like to have the ability to parse the integer field in JSON message if 
> it is prefixed with zero. Can this be investigated for consideration into 
> release? 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #1854

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-15008) GCS Sink Connector to parse JSON having leading 0's for an integer field

[Lubna Naqvi (Jira)]

Lubna Naqvi created KAFKA-15008:
---

 Summary: GCS Sink Connector to parse JSON having leading 0's for 
an integer field
 Key: KAFKA-15008
 URL: https://issues.apache.org/jira/browse/KAFKA-15008
 Project: Kafka
  Issue Type: Bug
Reporter: Lubna Naqvi


Our Kafka data which is in JSON format has an attribute (gtin) which is a 
number starting with Zero and Kafka Connect GCS Sink Connector fails to parse 
it. gtin is a very common attribute across any item related Kafka source and it 
is causing a major(blocking) issue as we are not able to parse this attribute.

 

We'd like to have the ability to parse the integer field in JSON message if it 
is prefixed with zero. Can this be investigated for consideration into release? 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)