Re: Request for pull request review

[deng ziming]

Hello Igor,
Thank you for driving this, currently we have more discussion about this, the 
main concern is that we don't want to add an RPC which will be 
O(num_partitions_in_dir), please see 
https://issues.apache.org/jira/browse/KAFKA-9837 
 for more details.

—
Thanks,
Ziming

> On Mar 9, 2022, at 8:53 PM, Igor Soarez  wrote:
> 
> Can someone review this pull request please?
> 
> https://github.com/apache/kafka/pull/9577
> 
> It has been open for quite some time, and it seems pinging folks directly on 
> GitHub has not worked.
> 
> Thanks,
> 
> --
> Igor

Re: [DISCUSS] KIP-794: Strictly Uniform Sticky Partitioner

[Artem Livshits]

Thank you for feedback, I've discussed this offline with some of the folks
and updated the KIP.  The main change is that now instead of using
DefaultPartitioner and UniformStickyPartitioners as flags, in the new
proposal the default partitioner is null, so if no custom partitioner is
specified then the partitioning logic is implemented in KafkaProducer.
Compatibility section is updated as well.  Also the configuration options
are renamed to be more consistent.

-Artem

On Fri, Mar 4, 2022 at 10:38 PM Luke Chen  wrote:

> Hi Artem,
>
> Thanks for your explanation and update to the KIP.
> Some comments:
>
> 5. In the description for `enable.adaptive.partitioning`, the `false` case,
> you said:
> > the producer will try to distribute messages uniformly.
> I think we should describe the possible skewing distribution. Otherwise,
> user might be confused about why adaptive partitioning is important.
>
> 6. In the description for `partition.availability.timeout.ms`, I think we
> should mention in the last sentence about if `enable.adaptive.partitioning`
> is disabled this logic is also disabled.
>
> 7. Similar thoughts as Ismael, I think we should have a POC and test to
> prove that this adaptive partitioning algorithm can have better uniform
> partitioning, compared with original sticky one.
>
> Thank you.
> Luke
>
> On Fri, Mar 4, 2022 at 9:22 PM Ismael Juma  wrote:
>
> > Regarding `3`, we should only deprecate it if we're sure the new approach
> > handles all cases better. Are we confident about that for both of the
> > previous partitioners?
> >
> > Ismael
> >
> > On Fri, Mar 4, 2022 at 1:37 AM David Jacot 
> > wrote:
> >
> > > Hi Artem,
> > >
> > > Thanks for the KIP! I have a few comments:
> > >
> > > 1. In the preamble of the proposed change section, there is still a
> > > mention of the
> > > -1 approach. My understanding is that we have moved away from it now.
> > >
> > > 2. I am a bit concerned by the trick suggested about the
> > > DefaultPartitioner and
> > > the UniformStickyPartitioner. I do agree that implementing the logic in
> > the
> > > producer itself is a good thing. However, it is weird from a user
> > > perspective
> > > that he can set a class as partitioner that is not used in the end. I
> > > think that
> > > this will be confusing for our users. Have we considered changing the
> > > default
> > > value of partitioner.class to null to indicate that the new built-in
> > > partitioner
> > > must be used? By default, the built-in partitioner would be used unless
> > the
> > > user explicitly specify one. The downside is that the new default
> > behavior
> > > would not work if the user explicitly specify the partitioner but we
> > could
> > > mitigate this with my next point.
> > >
> > > 3. Related to the previous point, I think that we could deprecate both
> > the
> > > DefaultPartitioner and the UniformStickyPartitioner. I would also add a
> > > warning if one of them is explicitly provided by the user to inform
> them
> > > that they should switch to the new built-in one. I am pretty sure that
> > most
> > > of the folks use the default configuration anyway.
> > >
> > > 4. It would be great if we could explain why the -1 way was rejected.
> At
> > > the moment, the rejected alternative only explain the idea but does not
> > > say why we rejected it.
> > >
> > > Best,
> > > David
> > >
> > > On Fri, Mar 4, 2022 at 6:03 AM Artem Livshits
> > >  wrote:
> > > >
> > > > Hi Jun,
> > > >
> > > > 2. Removed the option from the KIP.  Now the sticky partitioning
> > > threshold
> > > > is hardcoded to batch.size.
> > > >
> > > > 20. Added the corresponding wording to the KIP.
> > > >
> > > > -Artem
> > > >
> > > > On Thu, Mar 3, 2022 at 10:52 AM Jun Rao 
> > > wrote:
> > > >
> > > > > Hi, Artem,
> > > > >
> > > > > Thanks for the reply.
> > > > >
> > > > > 1. Sounds good.
> > > > >
> > > > > 2. If we don't expect users to change it, we probably could just
> > leave
> > > out
> > > > > the new config. In general, it's easy to add a new config, but hard
> > to
> > > > > remove an existing config.
> > > > >
> > > > > 20. The two new configs enable.adaptive.partitioning and
> > > > > partition.availability.timeout.ms only apply to the two built-in
> > > > > partitioners DefaultPartitioner and UniformStickyPartitioner,
> right?
> > It
> > > > > would be useful to document that in the KIP.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Thu, Mar 3, 2022 at 9:47 AM Artem Livshits
> > > > >  wrote:
> > > > >
> > > > > > Hi Jun,
> > > > > >
> > > > > > Thank you for the suggestions.
> > > > > >
> > > > > > 1. As we discussed offline, we can hardcode the logic for
> > > > > > DefaultPartitioner and UniformStickyPartitioner in the
> > KafkaProducer
> > > > > (i.e.
> > > > > > the DefaultPartitioner.partition won't get called, instead
> > > KafkaProducer
> > > > > > would check if the partitioner is an instance of
> DefaultPartitioner
> > > and
> > > > > > then run the actual 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #755

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #754

[Apache Jenkins Server]

See 

[jira] [Resolved] (KAFKA-13717) KafkaConsumer.close throws authorization exception even when commit offsets is empty

[Luke Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Luke Chen resolved KAFKA-13717.
---
Fix Version/s: 3.2.0
 Assignee: Vincent Jiang
   Resolution: Fixed

> KafkaConsumer.close throws authorization exception even when commit offsets 
> is empty
> 
>
> Key: KAFKA-13717
> URL: https://issues.apache.org/jira/browse/KAFKA-13717
> Project: Kafka
>  Issue Type: Bug
>  Components: unit tests
>Reporter: Vincent Jiang
>Assignee: Vincent Jiang
>Priority: Major
> Fix For: 3.2.0
>
>
> When offsets is empty and coordinator is unknown, KafkaConsumer.close doesn't 
> throw exception before commit 
> [https://github.com/apache/kafka/commit/4b468a9d81f7380f7197a2a6b859c1b4dca84bd9|https://github.com/apache/kafka/commit/4b468a9d81f7380f7197a2a6b859c1b4dca84bd9,].
>   After this commit, Kafka.close may throw authorization exception.
>  
> Root cause is because in the commit, the logic is changed to call 
> lookupCoordinator even if offsets is empty. 
>  
> Even if a consumer doesn't have access to a group or a topic, it might be 
> better to not throw authorization exception in this case because close() call 
> doesn't touch actually access any resource.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Build failed in Jenkins: Kafka » Kafka Branch Builder » 2.8 #98

[Apache Jenkins Server]

See 


Changes:


--
[...truncated 467855 lines...]
[2022-03-10T01:38:18.424Z] [INFO] Parameter: package, Value: myapps
[2022-03-10T01:38:18.424Z] [INFO] Parameter: packageInPathFormat, Value: myapps
[2022-03-10T01:38:18.424Z] [INFO] Parameter: package, Value: myapps
[2022-03-10T01:38:18.424Z] [INFO] Parameter: version, Value: 0.1
[2022-03-10T01:38:18.424Z] [INFO] Parameter: groupId, Value: streams.examples
[2022-03-10T01:38:18.424Z] [INFO] Parameter: artifactId, Value: streams.examples
[2022-03-10T01:38:18.424Z] [INFO] Project created from Archetype in dir: 
/home/jenkins/workspace/Kafka_kafka_2.8/streams/quickstart/test-streams-archetype/streams.examples
[2022-03-10T01:38:18.424Z] [INFO] 

[2022-03-10T01:38:18.424Z] [INFO] BUILD SUCCESS
[2022-03-10T01:38:18.424Z] [INFO] 

[2022-03-10T01:38:18.424Z] [INFO] Total time:  1.308 s
[2022-03-10T01:38:18.424Z] [INFO] Finished at: 2022-03-10T01:38:16Z
[2022-03-10T01:38:18.424Z] [INFO] 

[2022-03-10T01:38:18.900Z] > Task :streams:compileTestJava
[Pipeline] dir
[2022-03-10T01:38:19.124Z] Running in 
/home/jenkins/workspace/Kafka_kafka_2.8/streams/quickstart/test-streams-archetype/streams.examples
[Pipeline] {
[Pipeline] sh
[2022-03-10T01:38:21.925Z] + mvn compile
[2022-03-10T01:38:22.974Z] [INFO] Scanning for projects...
[2022-03-10T01:38:22.974Z] [INFO] 
[2022-03-10T01:38:22.974Z] [INFO] -< 
streams.examples:streams.examples >--
[2022-03-10T01:38:22.974Z] [INFO] Building Kafka Streams Quickstart :: Java 0.1
[2022-03-10T01:38:22.974Z] [INFO] [ jar 
]-
[2022-03-10T01:38:22.974Z] [INFO] 
[2022-03-10T01:38:22.974Z] [INFO] --- maven-resources-plugin:2.6:resources 
(default-resources) @ streams.examples ---
[2022-03-10T01:38:22.974Z] [INFO] Using 'UTF-8' encoding to copy filtered 
resources.
[2022-03-10T01:38:22.974Z] [INFO] Copying 1 resource
[2022-03-10T01:38:22.974Z] [INFO] 
[2022-03-10T01:38:22.974Z] [INFO] --- maven-compiler-plugin:3.1:compile 
(default-compile) @ streams.examples ---
[2022-03-10T01:38:24.022Z] [INFO] Changes detected - recompiling the module!
[2022-03-10T01:38:24.022Z] [INFO] Compiling 3 source files to 
/home/jenkins/workspace/Kafka_kafka_2.8/streams/quickstart/test-streams-archetype/streams.examples/target/classes
[2022-03-10T01:38:25.071Z] [INFO] 

[2022-03-10T01:38:25.071Z] [INFO] BUILD SUCCESS
[2022-03-10T01:38:25.071Z] [INFO] 

[2022-03-10T01:38:25.071Z] [INFO] Total time:  2.062 s
[2022-03-10T01:38:25.071Z] [INFO] Finished at: 2022-03-10T01:38:23Z
[2022-03-10T01:38:25.071Z] [INFO] 

[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // dir
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // timestamps
[Pipeline] }
[Pipeline] // timeout
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[2022-03-10T01:40:23.917Z] > Task :streams:testClasses
[2022-03-10T01:40:23.917Z] > Task :streams:testJar
[2022-03-10T01:40:23.917Z] > Task :streams:testSrcJar
[2022-03-10T01:40:23.917Z] > Task :streams:signArchives SKIPPED
[2022-03-10T01:40:23.917Z] > Task :streams:install
[2022-03-10T01:40:23.917Z] 
[2022-03-10T01:40:23.917Z] Deprecated Gradle features were used in this build, 
making it incompatible with Gradle 7.0.
[2022-03-10T01:40:23.917Z] Use '--warning-mode all' to show the individual 
deprecation warnings.
[2022-03-10T01:40:23.917Z] See 
https://docs.gradle.org/6.8.1/userguide/command_line_interface.html#sec:command_line_warnings
[2022-03-10T01:40:23.917Z] 
[2022-03-10T01:40:23.917Z] BUILD SUCCESSFUL in 7m 18s
[2022-03-10T01:40:23.917Z] 69 actionable tasks: 34 executed, 35 up-to-date
[Pipeline] sh
[2022-03-10T01:40:26.601Z] + grep ^version= gradle.properties
[2022-03-10T01:40:26.601Z] + cut -d= -f 2
[Pipeline] dir
[2022-03-10T01:40:27.297Z] Running in 
/home/jenkins/jenkins-agent/workspace/Kafka_kafka_2.8/streams/quickstart
[Pipeline] {
[Pipeline] sh
[2022-03-10T01:40:29.462Z] + mvn clean install -Dgpg.skip
[2022-03-10T01:40:31.233Z] [INFO] Scanning for projects...
[2022-03-10T01:40:32.176Z] [INFO] 

[2022-03-10T01:40:32.176Z] [INFO] Reactor Build Order:
[2022-03-10T01:40:32.176Z] [INFO] 
[2022-03-10T01:40:32.176Z] [INFO] Kafka Streams :: Quickstart   
 [pom]

Jenkins build is still unstable: Kafka » Kafka Branch Builder » 3.0 #185

[Apache Jenkins Server]

See 

Jenkins build is unstable: Kafka » Kafka Branch Builder » 3.1 #85

[Apache Jenkins Server]

See 

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #753

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-714: Client metrics and observability

[Kirk True]

Hi Jun,

On Tue, Mar 8, 2022, at 5:47 PM, Jun Rao wrote:
> Hi, Magnus, Sarat and Xavier,
> 
> Thanks for the reply. A few more comments below.
> 
> 20. It seems that we are piggybacking the plugin on the
> existing MetricsReporter. So, this seems fine.
> 
> 21. That could work. Are we requiring any additional jar dependency on the
> client? Or, are you suggesting that we check the runtime dependency to pick
> the compression codec?

The Java client doesn't require any additional libraries for compression, no.

> 28. For the broker metrics, could you spell out the full metric name
> including groups, tags, etc? We typically don't add the broker_id label for
> broker metrics. Also, brokers use Yammer metrics, which doesn't have type
> Sum.
> 
> 29. There are several client metrics listed as histogram. However, the java
> client currently doesn't support histogram type.

There does appear to be some code related to histograms in the 
org.apache.kafka.common.metrics.stats package. But we're still looking into the 
implementation to see if there's anything needed for KIP-714.

> 30. Could you show an example of the metric payload in PushTelemetryRequest
> to help understand how we organize metrics at different levels (per
> instance, per topic, per partition, per broker, etc)?
> 
> 31. Could you add a bit more detail on which client thread sends the
> PushTelemetryRequest?

Yes, I will add that the KIP.

Thanks,
Kirk

> Thanks,
> 
> Jun
> 
> On Mon, Mar 7, 2022 at 11:48 AM Magnus Edenhill  wrote:
> 
> > Hi Jun,
> >
> > thanks for your initiated questions, see my answers below.
> > There's been a number of clarifications to the KIP.
> >
> >
> >
> > Den tors 27 jan. 2022 kl 20:08 skrev Jun Rao :
> >
> > > Hi, Magnus,
> > >
> > > Thanks for updating the KIP. The overall approach makes sense to me. A
> > few
> > > more detailed comments below.
> > >
> > > 20. ClientTelemetry: Should it be extending configurable and closable?
> > >
> >
> > I'll pass this question to Sarat and/or Xavier.
> >
> >
> >
> > > 21. Compression of the metrics on the client: what's the default?
> > >
> >
> > How about we specify a prioritized list: zstd, lz4, snappy, gzip?
> > But ultimately it is up to what the client supports.
> >
> >
> > 23. A client instance is considered a metric resource and the
> > > resource-level (thus client instance level) labels could include:
> > > client_software_name=confluent-kafka-python
> > > client_software_version=v2.1.3
> > > client_instance_id=B64CD139-3975-440A-91D4
> > > transactional_id=someTxnApp
> > > Are those labels added in PushTelemetryRequest? If so, are they per
> > metric
> > > or per request?
> > >
> >
> >
> > client_software* and client_instance_id are not added by the client, but
> > available to
> > the broker-side metrics plugin for adding as it see fits, remove them from
> > the KIP.
> >
> > As for transactional_id, group_id, etc, which I believe will be useful in
> > troubleshooting,
> > are included only once (per push) as resource-level attributes (the client
> > instance is a singular resource).
> >
> >
> > >
> > > 24.  "the broker will only send
> > > GetTelemetrySubscriptionsResponse.DeltaTemporality=True" :
> > > 24.1 If it's always true, does it need to be part of the protocol?
> > >
> >
> > We're anticipating that it will take a lot longer to upgrade the majority
> > of clients than the
> > broker/plugin side, which is why we want the client to support both
> > temporalities out-of-the-box
> > so that cumulative reporting can be turned on seamlessly in the future.
> >
> >
> >
> > > 24.2 Does delta only apply to Counter type?
> > >
> >
> >
> > And Histograms. More details in Xavier's OTLP link.
> >
> >
> >
> > > 24.3 In the delta representation, the first request needs to send the
> > full
> > > value, how does the broker plugin know whether a value is full or delta?
> > >
> >
> > The client may (should) send the start time for each metric sample,
> > indicating when
> > the metric began to be collected.
> > We've discussed whether this should be the client instance start time or
> > the time when a matching
> > metric subscription for that metric is received.
> > For completeness we recommend using the former, the client instance start
> > time.
> >
> >
> >
> > > 25. quota:
> > > 25.1 Since we are fitting PushTelemetryRequest into the existing request
> > > quota, it would be useful to document the impact, i.e. client metric
> > > throttling causes the data from the same client to be delayed.
> > > 25.2 Is PushTelemetryRequest subject to the write bandwidth quota like
> > the
> > > producer?
> > >
> >
> >
> > Yes, it should be, as to protect the cluster from rogue clients.
> > But, in practice the size of metrics will be quite low (e.g., 1-10kb per
> > 60s interval), so I don't think this will pose a problem.
> > The KIP has been updated with more details on quota/throttling behaviour,
> > see the
> > "Throttling and rate-limiting" section.
> >
> >
> > 25.3 

[jira] [Created] (KAFKA-13726) Fix Vulnerability CVE-2022-23181 -Upgrade org.apache.tomcat.embed_tomcat-embed-core

[Chris Sabelstrom (Jira)]

Chris Sabelstrom created KAFKA-13726:


 Summary: Fix Vulnerability CVE-2022-23181 -Upgrade 
org.apache.tomcat.embed_tomcat-embed-core
 Key: KAFKA-13726
 URL: https://issues.apache.org/jira/browse/KAFKA-13726
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.8.1
Reporter: Chris Sabelstrom


Our security scanner detected the following vulnerablity. Please upgrade to 
version noted in Fix Status column.
|CVE ID|Severity|Packages|Package Version|CVSS|Fix Status|
|CVE-2022-23181|high|org.apache.tomcat.embed_tomcat-embed-core|9.0.54|7|fixed 
in 10.0.0, 9.0.1|



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (KAFKA-13725) KIP-768 OAuth code mixes public and internal classes in same package

[Kirk True (Jira)]

Kirk True created KAFKA-13725:
-

 Summary: KIP-768 OAuth code mixes public and internal classes in 
same package
 Key: KAFKA-13725
 URL: https://issues.apache.org/jira/browse/KAFKA-13725
 Project: Kafka
  Issue Type: Bug
Affects Versions: 3.1.0, 3.2.0, 3.1.1
Reporter: Kirk True
Assignee: Kirk True






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (KAFKA-13724) Fix Vulnerability CVE-2021-43859 - Upgrade com.thoughtworks.xstream_xstream

[Chris Sabelstrom (Jira)]

Chris Sabelstrom created KAFKA-13724:


 Summary: Fix Vulnerability CVE-2021-43859 - Upgrade 
com.thoughtworks.xstream_xstream
 Key: KAFKA-13724
 URL: https://issues.apache.org/jira/browse/KAFKA-13724
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.8.1
Reporter: Chris Sabelstrom


Our security scanner detected the following vulnerablity. Please upgrade to 
version noted in Fix Status column.
|CVE ID|Severity|Packages|Package Version|CVSS|Fix Status|
|CVE-2021-43859|high|com.thoughtworks.xstream_xstream|1.4.18|7.5|fixed in 
1.4.19|



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #752

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-821: Connect Transforms support for nested structures

[Chris Egerton]

Hi Jorge,

Looking good! Got a few more thoughts.

1. Sorry to revisit this, but I think we may want to adopt a slightly
different escape syntax style. Backslashes are great, but since they're
already used by JSON, using them as an escape sequence in field notation
would also lead to some pretty ugly connector configs. Anyone who's had to
write regular expressions with backslashes in Java is probably already
familiar with this: "this.is.not.very.readable". What do
you think about using the dot character to escape itself? In other words,
to access a single field named "this.field", instead of using the syntax
"this\.field" (which in JSON would have to be expressed as "this\\.field"),
we could use "this..field", and for a single field named "this\field",
instead of using the syntax "this\\field" (or, in JSON, "thisfield"),
we could use "this\field" (or, in JSON, "this\\field").

2. Could you flesh out the details on the new "field.style" property,
including the type, default value, importance, and a preliminary docstring?
See
https://cwiki.apache.org/confluence/display/KAFKA/KIP-618%3A+Exactly-Once+Support+for+Source+Connectors#KIP618:ExactlyOnceSupportforSourceConnectors-Newproperties
for an example.

3. Is the "Compatibility, Deprecation, and Migration Plan" section still
accurate after the latest update? Seems like it's still written with the
assumption that nested field syntax will be hardcoded or opt-in, which IIUC
isn't the case anymore.

4. Nit: The "These SMTs do not require nested structure support" section
mentions a "Drop" SMT. I think this may be referring to the Confluent Drop
SMT, which isn't a part of Apache Kafka. Should we drop (heh) that SMT from
the list? Or perhaps just replace it with "DropHeaders", which is currently
missing from the list and shouldn't require any nested-field related
updates?

Cheers,

Chris

On Mon, Feb 28, 2022 at 2:12 PM Jorge Esteban Quilcate Otoya <
quilcate.jo...@gmail.com> wrote:

> Thank you, Chris! and sorry for the delayed response.
>
> Please, find my comments below:
>
> On Mon, 14 Feb 2022 at 17:34, Chris Egerton 
> wrote:
>
> > Hi Jorge,
> >
> > Thanks for the KIP! I'd love to see support for nested fields added to
> the
> > out-of-the-box SMTs provided with Connect. Here are my initial thoughts:
> >
> > 1. I agree that there's a case to be made for expanding HoistField with a
> > new config property for identifying a nested, to-be-hoisted field, but
> the
> > example in the KIP doesn't really demonstrate why this would be
> valuable. I
> > think it'd be helpful to expand the example to add other fields in order
> to
> > show how adding nested field support enables users to hoist a nested
> field
> > without dropping other fields from the value. Maybe something like this:
> >
> > source = nested.val
> > field = line
> >
> > value (before):
> > {
> > "nested": {
> > "val": 42,
> > "other val": 96
> > }
> > }
> >
> > value (after):
> > {
> > "nested": {
> > "line": {
> > "val": 42,
> > }
> > "other val": 96
> > }
> > }
> >
> > 2. Nit: I think "source" is a little strange for the new HoistField
> > property name. Maybe "hoisted" or "hoisted.field" would be more
> > descriptive?
> >
> >
> About 1. and 2.:
> Agree. The example for this SMT is updated and have added the `hoisted`
> configuration.
>
>
> > 3. Is there a reasonable use case for expanding Flatten to be able to
> > flatten specific fields? My understanding is that it's mostly useful for
> > writing to systems like databases that don't support nested values and
> > require everything to be a flat list of key-value pairs. Being able to
> > flatten a nested field wouldn't provide any advantage for that use case.
> > Are there other cases where it would?
> >
> > 4. I don't think we should unconditionally change the default delimiter
> for
> > Flatten. It's a backwards-incompatible, breaking change that could cause
> > headaches for users. It might be reasonable to change the default value
> > dynamically based on whether the user has specified a value for the
> "field"
> > property, but considering the motivation for changing the default is that
> > it creates conflicts with the to-be-introduced nested field syntax (which
> > could arise with downstream SMTs regardless of whether the user has
> > explicitly configured Flatten with the "field" property), I don't know
> that
> > this would be too useful either. I have some thoughts below on how to
> > handle possible conflicts between names with dots in their names and
> dotted
> > syntax for nested field references that should hopefully make either
> change
> > unnecessary.
> >
> >
> Fair enough. With the support for nested fields in other SMTs, Flatten
> could stay as it is.
> This removes the need for (4) changing Flatten config as well.
>

[jira] [Resolved] (KAFKA-13723) max.compaction.lag.ms implemented incorrectly

[Jun Rao (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-13723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-13723.
-
Resolution: Not A Problem

> max.compaction.lag.ms implemented incorrectly
> -
>
> Key: KAFKA-13723
> URL: https://issues.apache.org/jira/browse/KAFKA-13723
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 2.3.0
>Reporter: Jun Rao
>Priority: Major
>
> In https://issues.apache.org/jira/browse/KAFKA-7321, we introduced 
> max.compaction.lag.ms to guarantee that a record be cleaned before a certain 
> time. 
>  
> The implementation in LogCleanerManager has the following code. The path for 
> earliestDirtySegmentTimestamp < cleanUntilTime seems incorrect. In that case, 
> it seems that we should set the delay to 0 so that we could trigger cleaning 
> immediately since the segment has been dirty for longer than 
> max.compaction.lag.ms. 
>  
>  
> {code:java}
> def maxCompactionDelay(log: UnifiedLog, firstDirtyOffset: Long, now: Long) : 
> Long = {
> ...
> val maxCompactionLagMs = math.max(log.config.maxCompactionLagMs, 0L)
> val cleanUntilTime = now - maxCompactionLagMs
> if (earliestDirtySegmentTimestamp < cleanUntilTime)
> cleanUntilTime - earliestDirtySegmentTimestamp
> else
> 0L
> }{code}
>  



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (KAFKA-13723) max.compaction.lag.ms implemented incorrectly

[Jun Rao (Jira)]

Jun Rao created KAFKA-13723:
---

 Summary: max.compaction.lag.ms implemented incorrectly
 Key: KAFKA-13723
 URL: https://issues.apache.org/jira/browse/KAFKA-13723
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 2.3.0
Reporter: Jun Rao


In https://issues.apache.org/jira/browse/KAFKA-7321, we introduced 
max.compaction.lag.ms to guarantee that a record be cleaned before a certain 
time. 

 

The implementation in LogCleanerManager has the following code. The path for 
earliestDirtySegmentTimestamp < cleanUntilTime seems incorrect. In that case, 
it seems that we should set the delay to 0 so that we could trigger cleaning 
immediately since the segment has been dirty for longer than 
max.compaction.lag.ms. 

 

 
{code:java}
def maxCompactionDelay(log: UnifiedLog, firstDirtyOffset: Long, now: Long) : 
Long = {

...

val maxCompactionLagMs = math.max(log.config.maxCompactionLagMs, 0L)
val cleanUntilTime = now - maxCompactionLagMs

if (earliestDirtySegmentTimestamp < cleanUntilTime)
cleanUntilTime - earliestDirtySegmentTimestamp
else
0L
}{code}
 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] Apache Kafka 3.1.1

[David Jacot]

+1. Thanks Tom!

Le mer. 9 mars 2022 à 19:10, Bill Bejeck  a écrit :

> Thanks Tom!  It's a +1 for me.
>
> -Bill
>
> On Wed, Mar 9, 2022 at 12:00 PM Ismael Juma  wrote:
>
> > Thanks Tom. +1
> >
> > Ismael
> >
> >
> > On Wed, Mar 9, 2022 at 8:10 AM Tom Bentley  wrote:
> >
> > > Hi,
> > >
> > > I'd like to volunteer to be the release manager for the 3.1.1 bugfix
> > > release.
> > >
> > > Kind regards,
> > >
> > > Tom
> > >
> >
>

Re: [DISCUSS] Apache Kafka 3.1.1

[Bill Bejeck]

Thanks Tom!  It's a +1 for me.

-Bill

On Wed, Mar 9, 2022 at 12:00 PM Ismael Juma  wrote:

> Thanks Tom. +1
>
> Ismael
>
>
> On Wed, Mar 9, 2022 at 8:10 AM Tom Bentley  wrote:
>
> > Hi,
> >
> > I'd like to volunteer to be the release manager for the 3.1.1 bugfix
> > release.
> >
> > Kind regards,
> >
> > Tom
> >
>

[jira] [Created] (KAFKA-13722) Update internal interfaces that use ProcessorContext to use StateStoreContext instead

[Guozhang Wang (Jira)]

Guozhang Wang created KAFKA-13722:
-

 Summary: Update internal interfaces that use ProcessorContext to 
use StateStoreContext instead
 Key: KAFKA-13722
 URL: https://issues.apache.org/jira/browse/KAFKA-13722
 Project: Kafka
  Issue Type: Improvement
Reporter: Guozhang Wang


This is a remainder that when we remove the deprecated public APIs that uses 
the ProcessorContext, like `StateStore.init`, we should also consider updating 
the internal interfaces with the ProcessorContext as well. That includes:

1. Segments and related util classes which use ProcessorContext.
2. For state stores that leverage on ProcessorContext.getXXXTime, their logic 
should be moved out of the state store impl but to the processor node level 
that calls on these state stores.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] 3.0.1 RC0

[Bill Bejeck]

Hi Mickael,

Thanks for running the release.

I did the following steps to validate the release:

   1. Validated signatures and checksums
   2. Built from source and ran the unit tests
   3. I ran the quickstart steps for ZK and Kafka Streams
   4. Spot checked the docs and Javadocs

I notice the same issue as David regarding referencing the 3.0.0 releases.
I agree with him that we don't need to block the release and update the
docs separately.

+1(binding)

Regards,
Bill

On Wed, Mar 9, 2022 at 10:48 AM David Jacot 
wrote:

> Thanks for running the release, Mickael.
>
> I performed the following validations:
> * Verified all checksums and signatures.
> * Built from source and ran unit tests.
> * Ran the first quickstart steps for both ZK and KRaft.
> * Spotchecked the Javadocs.
>
> However, the document still references 3.0.0 in all places. It seems
> that it has not been updated for 3.0.1 yet. At least, I don't see any
> commits for that. I would not block the release on this because we
> can update the documentation independently.
>
> +1 (binding)
>
> Best,
> David
>
> On Wed, Mar 9, 2022 at 11:32 AM Tom Bentley  wrote:
> >
> > Hi Mickael,
> >
> > In addition to the results others have posted I've also validated the
> > checksums and keys, built from source and executed the unit and
> integration
> > tests.
> >
> > Overall I'm +1 (binding).
> >
> > Thanks,
> >
> > Tom
> >
> > On Wed, 9 Mar 2022 at 07:55, Michal Tóth  wrote:
> >
> > > Hello,
> > >
> > >  executed https://github.com/tombentley/kafka-verify-rc - with no
> issues.
> > > All checks have passed.
> > > * checksums, keys
> > > * unit tests
> > > * executed few system tests - all passed
> > >
> > > +1 (non-binding).
> > >
> > > Thank you
> > >
> > >
> > > ut 8. 3. 2022 o 8:33 Luke Chen  napísal(a):
> > >
> > > > Hi Mickael,
> > > >
> > > > Thanks for running the release!
> > > >
> > > > I did the following:
> > > >1. Validated the scala 2.13 checksums
> > > >2. Spot checked the java docs
> > > >3. Ran the quick start with scala 2.13 (found a minor bug
> KAFKA-13718
> > > > , won't block the
> > > > release)
> > > >
> > > > +1 (non-binding).
> > > >
> > > > Thank you.
> > > >
> > > > On Tue, Mar 8, 2022 at 1:33 AM Mickael Maison <
> mickael.mai...@gmail.com>
> > > > wrote:
> > > >
> > > > > Here is a successful Jenkins build for the 3.0 branch:
> > > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.0/183/
> > > > >
> > > > > On Mon, Mar 7, 2022 at 12:27 AM Jakub Scholz 
> wrote:
> > > > > >
> > > > > > +1 (non-binding). I used the staged Scala 2.13 binaries and the
> > > staging
> > > > > > Maven repository to run my tests. All seems to work fine, no
> issues
> > > > > found.
> > > > > >
> > > > > > Thanks
> > > > > > Jakub
> > > > > >
> > > > > > On Thu, Mar 3, 2022 at 7:05 PM Mickael Maison <
> mimai...@apache.org>
> > > > > wrote:
> > > > > >
> > > > > > > Hello Kafka users, developers and client-developers,
> > > > > > >
> > > > > > > This is the first candidate for release of Apache Kafka 3.0.1.
> > > > > > >
> > > > > > > Apache Kafka 3.0.1 is a bugfix release and 29 issues have been
> > > fixed
> > > > > > > since 3.0.0.
> > > > > > >
> > > > > > > Release notes for the 3.0.1 release:
> > > > > > >
> > > https://home.apache.org/~mimaison/kafka-3.0.1-rc0/RELEASE_NOTES.html
> > > > > > >
> > > > > > > *** Please download, test and vote by Thursday, March 10, 6pm
> GMT
> > > ***
> > > > > > >
> > > > > > > Kafka's KEYS file containing PGP keys we use to sign the
> release:
> > > > > > > https://kafka.apache.org/KEYS
> > > > > > >
> > > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > > https://home.apache.org/~mimaison/kafka-3.0.1-rc0/
> > > > > > >
> > > > > > > * Maven artifacts to be voted upon:
> > > > > > >
> > > >
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > > >
> > > > > > > * Javadoc:
> > > > > > > https://home.apache.org/~mimaison/kafka-3.0.1-rc0/javadoc/
> > > > > > >
> > > > > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.1 tag:
> > > > > > > https://github.com/apache/kafka/releases/tag/3.0.1-rc0
> > > > > > >
> > > > > > > * Documentation:
> > > > > > > https://kafka.apache.org/30/documentation.html
> > > > > > >
> > > > > > > * Protocol:
> > > > > > > https://kafka.apache.org/30/protocol.html
> > > > > > >
> > > > > > > * Successful Jenkins builds for the 3.0 branch:
> > > > > > > I'll share a link once the build complete
> > > > > > >
> > > > > > > /**
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Mickael
> > > > > > >
> > > > >
> > > >
> > >
> > >
> > > --
> > > S pozdravom
> > >  Michal Tóth
> > >
>

Re: [DISCUSS] Apache Kafka 3.1.1

[Ismael Juma]

Thanks Tom. +1

Ismael


On Wed, Mar 9, 2022 at 8:10 AM Tom Bentley  wrote:

> Hi,
>
> I'd like to volunteer to be the release manager for the 3.1.1 bugfix
> release.
>
> Kind regards,
>
> Tom
>

[jira] [Created] (KAFKA-13721) Left-join still emit spurious results in stream-stream joins in some cases

[Nollet (Jira)]

Nollet created KAFKA-13721:
--

 Summary: Left-join still emit spurious results in stream-stream 
joins in some cases
 Key: KAFKA-13721
 URL: https://issues.apache.org/jira/browse/KAFKA-13721
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 3.1.0
Reporter: Nollet


Stream-stream joins seems to still emit spurious results for some window 
configurations.

>From my tests, it happened when setting before to 0 and having a grace period 
>smaller than the window duration. More precisely it seems to happen when 
>setting before and 
window duration > grace period + before
h2. how to reproduce
{code:java}
import org.apache.kafka.common.serialization.Serdes;
import org.apache.kafka.streams.StreamsBuilder;
import org.apache.kafka.streams.StreamsConfig;
import org.apache.kafka.streams.TestInputTopic;
import org.apache.kafka.streams.TestOutputTopic;
import org.apache.kafka.streams.Topology;
import org.apache.kafka.streams.TopologyTestDriver;
import org.apache.kafka.streams.kstream.JoinWindows;
import org.apache.kafka.streams.kstream.KStream;
import org.junit.After;
import org.junit.Before;
import org.junit.Test;

import java.time.Duration;
import java.time.Instant;
import java.util.Properties;

public class SpuriousLeftJoinTest {

static final Duration WINDOW_DURATION = Duration.ofMinutes(10);
static final Duration GRACE = Duration.ofMinutes(6);
static final Duration BEFORE = Duration.ZERO;
static final String LEFT_TOPIC_NAME = "LEFT_TOPIC";
static final String RIGHT_TOPIC_NAME = "RIGHT_TOPIC";
static final String OUTPUT_TOPIC_NAME = "OUTPUT_TOPIC";


private static TopologyTestDriver testDriver;
private static TestInputTopic inputTopicLeft;
private static TestInputTopic inputTopicRight;
private static TestOutputTopic outputTopic;

public static Topology createTopology() {

StreamsBuilder builder = new StreamsBuilder();

KStream leftStream = builder.stream(LEFT_TOPIC_NAME);
KStream rightStream = builder.stream(RIGHT_TOPIC_NAME);

// return 1 if left join matched, otherwise 0
KStream joined = leftStream.leftJoin(
rightStream,
(value1, value2) -> {
if(value2 == null){
return 0;
}
return 1;
},
JoinWindows.ofTimeDifferenceAndGrace(WINDOW_DURATION, GRACE)
.before(BEFORE)
);

joined.to(OUTPUT_TOPIC_NAME);

return builder.build();
}


@Before
public void setup() {

Topology topology = createTopology();

Properties props = new Properties();
props.put(StreamsConfig.APPLICATION_ID_CONFIG, "test");
props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, "dummy:9092");
props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, 
Serdes.StringSerde.class);
props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, 
Serdes.IntegerSerde.class);

testDriver = new TopologyTestDriver(topology, props);

inputTopicLeft = testDriver.createInputTopic(LEFT_TOPIC_NAME, 
Serdes.String().serializer(), Serdes.Integer().serializer());
inputTopicRight = testDriver.createInputTopic(RIGHT_TOPIC_NAME, 
Serdes.String().serializer(), Serdes.Integer().serializer());

outputTopic = testDriver.createOutputTopic(OUTPUT_TOPIC_NAME, 
Serdes.String().deserializer(), Serdes.Integer().deserializer());

}

@After
public void tearDown() {
testDriver.close();
}

@Test
public void shouldEmitOnlyOneMessageForKey1(){
Instant now = Instant.now();
inputTopicLeft.pipeInput("key1", 12, now);
inputTopicRight.pipeInput("key1", 13, now.plus(WINDOW_DURATION));

// send later record to increase stream time & close the window
inputTopicLeft.pipeInput("other_key", 1212122, 
now.plus(WINDOW_DURATION).plus(GRACE).plusSeconds(10));

while (! outputTopic.isEmpty()){
System.out.println(outputTopic.readKeyValue());
}
}


}
{code}
Stdout of previous code is
{noformat}
KeyValue(key1, 0)
KeyValue(key1, 1)
{noformat}
However it should be
{noformat}
KeyValue(key1, 1)
{noformat}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[DISCUSS] Apache Kafka 3.1.1

[Tom Bentley]

Hi,

I'd like to volunteer to be the release manager for the 3.1.1 bugfix
release.

Kind regards,

Tom

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #751

[Apache Jenkins Server]

See 

Re: Newbie contributor

[Mickael Maison]

Hi Francisco,

Welcome! I've added you to the contributor list. Thanks for your
interest in Apache Kafka.

Thanks,
Mickael

On Wed, Mar 9, 2022 at 4:16 PM Francisco José Becerra López
 wrote:
>
> Hi,
>
> Hope you are well
>
> May I be added to the contribution list to lend a hand on this project,
> please?
>
> Jira username: fjbecerra
>
> Regards
> Francisco

Re: [VOTE] 3.0.1 RC0

[David Jacot]

Thanks for running the release, Mickael.

I performed the following validations:
* Verified all checksums and signatures.
* Built from source and ran unit tests.
* Ran the first quickstart steps for both ZK and KRaft.
* Spotchecked the Javadocs.

However, the document still references 3.0.0 in all places. It seems
that it has not been updated for 3.0.1 yet. At least, I don't see any
commits for that. I would not block the release on this because we
can update the documentation independently.

+1 (binding)

Best,
David

On Wed, Mar 9, 2022 at 11:32 AM Tom Bentley  wrote:
>
> Hi Mickael,
>
> In addition to the results others have posted I've also validated the
> checksums and keys, built from source and executed the unit and integration
> tests.
>
> Overall I'm +1 (binding).
>
> Thanks,
>
> Tom
>
> On Wed, 9 Mar 2022 at 07:55, Michal Tóth  wrote:
>
> > Hello,
> >
> >  executed https://github.com/tombentley/kafka-verify-rc - with no issues.
> > All checks have passed.
> > * checksums, keys
> > * unit tests
> > * executed few system tests - all passed
> >
> > +1 (non-binding).
> >
> > Thank you
> >
> >
> > ut 8. 3. 2022 o 8:33 Luke Chen  napísal(a):
> >
> > > Hi Mickael,
> > >
> > > Thanks for running the release!
> > >
> > > I did the following:
> > >1. Validated the scala 2.13 checksums
> > >2. Spot checked the java docs
> > >3. Ran the quick start with scala 2.13 (found a minor bug KAFKA-13718
> > > , won't block the
> > > release)
> > >
> > > +1 (non-binding).
> > >
> > > Thank you.
> > >
> > > On Tue, Mar 8, 2022 at 1:33 AM Mickael Maison 
> > > wrote:
> > >
> > > > Here is a successful Jenkins build for the 3.0 branch:
> > > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.0/183/
> > > >
> > > > On Mon, Mar 7, 2022 at 12:27 AM Jakub Scholz  wrote:
> > > > >
> > > > > +1 (non-binding). I used the staged Scala 2.13 binaries and the
> > staging
> > > > > Maven repository to run my tests. All seems to work fine, no issues
> > > > found.
> > > > >
> > > > > Thanks
> > > > > Jakub
> > > > >
> > > > > On Thu, Mar 3, 2022 at 7:05 PM Mickael Maison 
> > > > wrote:
> > > > >
> > > > > > Hello Kafka users, developers and client-developers,
> > > > > >
> > > > > > This is the first candidate for release of Apache Kafka 3.0.1.
> > > > > >
> > > > > > Apache Kafka 3.0.1 is a bugfix release and 29 issues have been
> > fixed
> > > > > > since 3.0.0.
> > > > > >
> > > > > > Release notes for the 3.0.1 release:
> > > > > >
> > https://home.apache.org/~mimaison/kafka-3.0.1-rc0/RELEASE_NOTES.html
> > > > > >
> > > > > > *** Please download, test and vote by Thursday, March 10, 6pm GMT
> > ***
> > > > > >
> > > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > > https://kafka.apache.org/KEYS
> > > > > >
> > > > > > * Release artifacts to be voted upon (source and binary):
> > > > > > https://home.apache.org/~mimaison/kafka-3.0.1-rc0/
> > > > > >
> > > > > > * Maven artifacts to be voted upon:
> > > > > >
> > > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > > >
> > > > > > * Javadoc:
> > > > > > https://home.apache.org/~mimaison/kafka-3.0.1-rc0/javadoc/
> > > > > >
> > > > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.1 tag:
> > > > > > https://github.com/apache/kafka/releases/tag/3.0.1-rc0
> > > > > >
> > > > > > * Documentation:
> > > > > > https://kafka.apache.org/30/documentation.html
> > > > > >
> > > > > > * Protocol:
> > > > > > https://kafka.apache.org/30/protocol.html
> > > > > >
> > > > > > * Successful Jenkins builds for the 3.0 branch:
> > > > > > I'll share a link once the build complete
> > > > > >
> > > > > > /**
> > > > > >
> > > > > > Thanks,
> > > > > > Mickael
> > > > > >
> > > >
> > >
> >
> >
> > --
> > S pozdravom
> >  Michal Tóth
> >

[DISCUSS] [KIP worthy?] Anyone with large clusters facing Produce Response Time degradation as in KAFKA-10690?

[マテュアルン]

Hello,
This is Arun from LINE Corporation.
We have a cluster with a large number of brokers (200+), node failures are 
bound to happen relatively often. Upon recovery of the machine, or upon 
reassignment of the replicas on failed node, we often have a large amount of 
lagging replica catch up. Multiple replicas (re-)assigned to a target broker 
could start fetching from the same source broker id holding the leader replica. 
This occasionally leads to Produce Response Time degradation as illustrated in 
https://issues.apache.org/jira/browse/KAFKA-10690 .
​
Wanted to check if this is faced by anyone else, and if a solution merits a KIP.
​
With Regards​
マテュアルン Mathew Arun
LINE Corporation

Newbie contributor

[Francisco José Becerra López]

Hi,

Hope you are well

May I be added to the contribution list to lend a hand on this project,
please?

Jira username: fjbecerra

Regards
Francisco

Request for pull request review

[Igor Soarez]

Can someone review this pull request please?

https://github.com/apache/kafka/pull/9577

It has been open for quite some time, and it seems pinging folks directly on 
GitHub has not worked.

Thanks,

--
Igor

[jira] [Created] (KAFKA-13720) Few topic partitions remain under replicated after broker lose connectivity to zookeeper

[Dhirendra Singh (Jira)]

Dhirendra Singh created KAFKA-13720:
---

 Summary: Few topic partitions remain under replicated after broker 
lose connectivity to zookeeper
 Key: KAFKA-13720
 URL: https://issues.apache.org/jira/browse/KAFKA-13720
 Project: Kafka
  Issue Type: Bug
  Components: controller
Affects Versions: 2.7.1
Reporter: Dhirendra Singh


Few topic partitions remain under replicated after broker lose connectivity to 
zookeeper.
It only happens when brokers lose connectivity to zookeeper and it results in 
change in active controller. Issue does not occur always but randomly.
Issue never occurs when there is no change in active controller when brokers 
lose connectivity to zookeeper.
Following error message i found in the log file.


[2022-02-28 04:01:20,217] WARN [Partition __consumer_offsets-4 broker=1] 
Controller failed to update ISR to PendingExpandIsr(isr=Set(1), 
newInSyncReplicaId=2) due to unexpected UNKNOWN_SERVER_ERROR. Retrying. 
(kafka.cluster.Partition)
[2022-02-28 04:01:20,217] ERROR [broker-1-to-controller] Uncaught error in 
request completion: (org.apache.kafka.clients.NetworkClient)
java.lang.IllegalStateException: Failed to enqueue `AlterIsr` request with 
state LeaderAndIsr(leader=1, leaderEpoch=2728, isr=List(1, 2), zkVersion=4719) 
for partition __consumer_offsets-4
at kafka.cluster.Partition.sendAlterIsrRequest(Partition.scala:1403)
at 
kafka.cluster.Partition.$anonfun$handleAlterIsrResponse$1(Partition.scala:1438)
at kafka.cluster.Partition.handleAlterIsrResponse(Partition.scala:1417)
at kafka.cluster.Partition.$anonfun$sendAlterIsrRequest$1(Partition.scala:1398)
at 
kafka.cluster.Partition.$anonfun$sendAlterIsrRequest$1$adapted(Partition.scala:1398)
at 
kafka.server.AlterIsrManagerImpl.$anonfun$handleAlterIsrResponse$8(AlterIsrManager.scala:166)
at 
kafka.server.AlterIsrManagerImpl.$anonfun$handleAlterIsrResponse$8$adapted(AlterIsrManager.scala:163)
at scala.collection.immutable.List.foreach(List.scala:333)
at 
kafka.server.AlterIsrManagerImpl.handleAlterIsrResponse(AlterIsrManager.scala:163)
at kafka.server.AlterIsrManagerImpl.responseHandler$1(AlterIsrManager.scala:94)
at 
kafka.server.AlterIsrManagerImpl.$anonfun$sendRequest$2(AlterIsrManager.scala:104)
at 
kafka.server.BrokerToControllerRequestThread.handleResponse(BrokerToControllerChannelManagerImpl.scala:175)
at 
kafka.server.BrokerToControllerRequestThread.$anonfun$generateRequests$1(BrokerToControllerChannelManagerImpl.scala:158)
at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
at 
org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:586)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:578)
at kafka.common.InterBrokerSendThread.doWork(InterBrokerSendThread.scala:71)
at 
kafka.server.BrokerToControllerRequestThread.doWork(BrokerToControllerChannelManagerImpl.scala:183)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:96)
 
under replication count goes to zero after the controller broker is restarted 
again. but this require manual intervention.
Expectation is that when broker reconnect with zookeeper cluster should come 
back to stable state with under replication count as zero by itself without any 
manual intervention.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Jenkins build is still unstable: Kafka » Kafka Branch Builder » trunk #750

[Apache Jenkins Server]

See 

[jira] [Created] (KAFKA-13719) connector restart cause duplicate tasks

[Shujie Sun (Jira)]

Shujie Sun created KAFKA-13719:
--

 Summary: connector restart cause duplicate tasks
 Key: KAFKA-13719
 URL: https://issues.apache.org/jira/browse/KAFKA-13719
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 3.0.0
Reporter: Shujie Sun
 Attachments: image-2022-03-09-18-57-09-467.png

Restart connector with parameter includeTasks=true=false cause 
duplicate tasks and duplicate message。



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] 3.0.1 RC0

[Tom Bentley]

Hi Mickael,

In addition to the results others have posted I've also validated the
checksums and keys, built from source and executed the unit and integration
tests.

Overall I'm +1 (binding).

Thanks,

Tom

On Wed, 9 Mar 2022 at 07:55, Michal Tóth  wrote:

> Hello,
>
>  executed https://github.com/tombentley/kafka-verify-rc - with no issues.
> All checks have passed.
> * checksums, keys
> * unit tests
> * executed few system tests - all passed
>
> +1 (non-binding).
>
> Thank you
>
>
> ut 8. 3. 2022 o 8:33 Luke Chen  napísal(a):
>
> > Hi Mickael,
> >
> > Thanks for running the release!
> >
> > I did the following:
> >1. Validated the scala 2.13 checksums
> >2. Spot checked the java docs
> >3. Ran the quick start with scala 2.13 (found a minor bug KAFKA-13718
> > , won't block the
> > release)
> >
> > +1 (non-binding).
> >
> > Thank you.
> >
> > On Tue, Mar 8, 2022 at 1:33 AM Mickael Maison 
> > wrote:
> >
> > > Here is a successful Jenkins build for the 3.0 branch:
> > > https://ci-builds.apache.org/job/Kafka/job/kafka/job/3.0/183/
> > >
> > > On Mon, Mar 7, 2022 at 12:27 AM Jakub Scholz  wrote:
> > > >
> > > > +1 (non-binding). I used the staged Scala 2.13 binaries and the
> staging
> > > > Maven repository to run my tests. All seems to work fine, no issues
> > > found.
> > > >
> > > > Thanks
> > > > Jakub
> > > >
> > > > On Thu, Mar 3, 2022 at 7:05 PM Mickael Maison 
> > > wrote:
> > > >
> > > > > Hello Kafka users, developers and client-developers,
> > > > >
> > > > > This is the first candidate for release of Apache Kafka 3.0.1.
> > > > >
> > > > > Apache Kafka 3.0.1 is a bugfix release and 29 issues have been
> fixed
> > > > > since 3.0.0.
> > > > >
> > > > > Release notes for the 3.0.1 release:
> > > > >
> https://home.apache.org/~mimaison/kafka-3.0.1-rc0/RELEASE_NOTES.html
> > > > >
> > > > > *** Please download, test and vote by Thursday, March 10, 6pm GMT
> ***
> > > > >
> > > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > > https://kafka.apache.org/KEYS
> > > > >
> > > > > * Release artifacts to be voted upon (source and binary):
> > > > > https://home.apache.org/~mimaison/kafka-3.0.1-rc0/
> > > > >
> > > > > * Maven artifacts to be voted upon:
> > > > >
> > https://repository.apache.org/content/groups/staging/org/apache/kafka/
> > > > >
> > > > > * Javadoc:
> > > > > https://home.apache.org/~mimaison/kafka-3.0.1-rc0/javadoc/
> > > > >
> > > > > * Tag to be voted upon (off 3.0 branch) is the 3.0.1 tag:
> > > > > https://github.com/apache/kafka/releases/tag/3.0.1-rc0
> > > > >
> > > > > * Documentation:
> > > > > https://kafka.apache.org/30/documentation.html
> > > > >
> > > > > * Protocol:
> > > > > https://kafka.apache.org/30/protocol.html
> > > > >
> > > > > * Successful Jenkins builds for the 3.0 branch:
> > > > > I'll share a link once the build complete
> > > > >
> > > > > /**
> > > > >
> > > > > Thanks,
> > > > > Mickael
> > > > >
> > >
> >
>
>
> --
> S pozdravom
>  Michal Tóth
>

Jenkins build is unstable: Kafka » Kafka Branch Builder » trunk #749

[Apache Jenkins Server]

See