Timestamp check in Kafka TransactionMetadata

[Guru C G]

We have come across an issue where in FATAL messages are logged in the broker.

FATAL kafka.coordinator.transaction.TransactionMetadata: 
TransactionMetadata(transactionalId=tx-id-1, producerId=96011, 
producerEpoch=51, txnTimeoutMs=6, state=CompleteCommit, 
pendingState=Some(Ongoing), topicPartitions=Set(), 
txnStartTimestamp=1580894482199, txnLastUpdateTimestamp=1580894482292)'s 
transition to TxnTransitMetadata(producerId=96011, producerEpoch=51, 
txnTimeoutMs=6, txnState=Ongoing, topicPartitions=Set(topic1-0), 
txnStartTimestamp=1580894480766, txnLastUpdateTimestamp=1580894480766) failed: 
this should not happen

On close inspection, we found the message is because the completed transaction 
has a newer timestamp(txnStartTimestamp=1580894482199) than the current 
timestamp of TxnTransitMetadata(txnStartTimestamp=1580894480766) and we also 
found the possibility of clocks in the broker being out of sync by a few 
seconds.

https://github.com/apache/kafka/blob/b526528cafe4142b73df8c930473b0cddc84ca9d/core/src/main/scala/kafka/coordinator/transaction/TransactionMetadata.scala#L382


The scenario in general is acknowledged and partially addressed below. However, 
it does not cover the case where the startTime of Ongoing transaction is older 
than start time of completed/aborted.
https://issues.apache.org/jira/browse/KAFKA-5415?focusedCommentId=16045170&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16045170

Is this deliberate? Do we need that check there?

Re: [DISCUSS] KIP-552: Add interface to handle unused config

[Matthias J. Sax]

Personally, I think that adding a specify prefix for user configs would
be the simplest fix. Kafka Streams and Kafka Connect could just use this
prefix to avoid the false positive WARN logs.

Hence, the KIP does not need to fix every false positive warning, it
just changes the AbstractConfig about when a WARN is logged, and we can
do follow up PRs to use this new feature in Connect and Streams.


-Matthias


On 3/11/20 7:55 AM, Artur Burtsev wrote:
> Long time no see, was on holiday 🇳🇿.
>
> Thread got an interesting twist into reporting and fixing every single
> warning individually, which is outside of my capacity. As I mentioned
> in the KIP and also as highlighted by Patric - you don't have to do
> anything extraordinary to get these warnings, you just use default
> Kafka components (Streams, Connect) or Confluent schema registry. So
> hopefully one day it will be fixed.
>
> I totally agree with Gwen, any misconfigs should be as visible as
> possible, however the more false positives you have, the easier it is
> to overlook misconfig.
>
> We don't worry about misconfigs as much as we do about warnings, so we
> would mute easily, unfortunately very useful configuration dump is on
> the same logger.
>
> Solution for as would be to wrap producer and consumer, log config
> dumps from wrappers and mute original config loggers.
>
> I don't mind KIP to be canceled.
>
> Thanks,
> Artur
>
> On Sat, Mar 7, 2020 at 10:11 PM Matthias J. Sax  wrote:
>>
> What is the status of this KIP?
> 
> 
> -Matthias
> 
> On 2/17/20 2:41 PM, John Roesler wrote:
 Thanks Matthias,

 I got the impression this was considered and rejected in
 KAFKA-7509, but I'm not sure why. Maybe it was never really
 considered at all, just proposed and not-noticed? Perhaps Randall
 or Sönke can comment. See:
> https://issues.apache.org/jira/browse/KAFKA-7509?focusedCommentId=166608
> 68&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpan
> el#comment-16660868

 It would be good to know why that proposal didn't move forward.

 Thanks, -John



 On Mon, Feb 17, 2020, at 12:17, Matthias J. Sax wrote: I am just
 getting aware of this KIP (not sure why I missed it).

 In Kafka Streams we have nested clients and need to "forward"
 configs from outer layer to inner layers -- hence, we prefix some
 configs to be able to know which inner nested clients needs this
 config.

 I think the simplest approach is, to add a prefix (like
 "userconfig."). All thus configs would be skipped in the
 validation step to avoid the WARN log.

 When forwarding configs to inner classed (like nested clients in
 KS, serializers etc) we would remove this prefix).

 Using a `RecordingMap` seem rather heavy weight and complex?

 Thoughts?

 -Matthias

 On 2/17/20 9:09 AM, John Roesler wrote:
>>> Thanks Patrik,
>>>
>>> This seems to be a long and wandering issue. It seems that
>>> KAFKA-7509 has followed a similar trajectory to
>>> KAFKA-6793/KIP-552 , and 7509 is just recently closed in
>>> favor of whatever we decide to do in KAFKA-6793.
>>>
>>> Considering (what I hope is) the whole history of this issue,
>>> a few things emerge:
>>>
>>> 1. It's useful to get warned when you pass an invalid
>>> configuration 2. It's not possible for the "top layer"
>>> (Streams, Connect, etc.) to know up front which
>>> configurations are applicable to pass down to the "second"
>>> layer (Clients, RocksDB) because those layers themselves are
>>> extensible (see below) 3. We should propose a change that
>>> fixes this issue for the whole Kafka ecosystem at once.
>>>
>>> Elaboration on point 2: Users of Kafka libraries need to
>>> register extra components like Processors, Interceptors,
>>> RocksDBConfigSetters, RebalanceListeners, etc. They need to
>>> pass configurations into these self-registered components.
>>> Therefore, the outermost component (the one that you directly
>>> pass a Properties to, and that instantiates other
>>> Configurable components) _cannot_ know which configurations
>>> are needed by the "extra" components inside the Configurable
>>> components. Therefore, no approach that involves filtering
>>> only the "needed" configurations up front, before
>>> constructing a Configurable component, could work.
>>>
>>> Randall made an aside in this comment:
>>> https://issues.apache.org/jira/browse/KAFKA-7509?focusedCommentId=1
> 667

>>>
> 3834&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpan
 el#comment-16673834
>>>
>>>
 which I think is the most promising path right now.
>>> Namely, to use RecordingMap (or a similar approach) when
>>> configuring internal components and finally warn when
>>> _everything_ has been wired up if some configuration val

Build failed in Jenkins: kafka-trunk-jdk11 #1229

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-9701: Add more debug log on client to reproduce the issue (#8272)


--
[...truncated 5.89 MB...]
org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0100:compileJava NO-SOURCE
> Task :streams:upgrade-system-tests-0100:processResources NO-SOURCE
> Task :streams:upgrade-system-tests-0100:classes UP-TO-DATE
> Task :streams:upgrade-system-tests-0100:checkstyleMain NO-SOURCE
> Task :streams:upgrade-system-tests-0100:compileTestJava
> Task :streams:upgrade-system-tests-0100:processTestResources NO-SOURCE
> Task :st

回复：回复：回复：[Vote] KIP-571: Add option to force remove members in StreamsResetter

[feyman2009]

Hi, Sophie
For 1) Sorry, I found that my expression is kind of misleading, what I 
actually mean is: "if --force not specified, an exception saying there are 
still active members on broker side will be thrown and suggesting using 
StreamsResetter with --force", I just updated the KIP page.

For 2)
I may also had some misleading expression previous, to clarify :

Also, it's more efficient to just send a single "clear the group" request vs 
sending a LeaveGroup
request for every single member. What do you think?
=> the comparison is to send a single "clear the group" request vs sending a 
"get members" + a "remove members" request since the 
adminClient.removeMembersFromConsumerGroup support batch removal. We don't need 
to send lots of leaveGroup requests for every single member.

   I can understand your point, but I think we could reuse the current 
adminClient.removeMembersFromConsumerGroup interface effectively with the KIP. 
What do you think?
 
Thanks!

Feyman


--
发件人：Sophie Blee-Goldman 
发送时间：2020年3月10日(星期二) 03:02
收件人：dev ; feyman2009 
主　题：Re: 回复：回复：[Vote] KIP-571: Add option to force remove members in 
StreamsResetter

Hey Feyman,

1) Regarding point 2 in your last email, if I understand correctly you propose 
to change
the current behavior of the reset tool when --force is not specified, and wait 
for (up to)
the session timeout for all members to be removed. I'm not sure we should 
change this,
especially now that we have a better way to handle the case when the group is 
not empty:
we should continue to throw an exception and fail fast, but can print a message 
suggesting
to use the new --force option to remove remaining group members. Why make users 
wait
for the session timeout when we've just added a new feature that means they 
don't have to?

2) Regarding Matthias' question:

> I am really wondering, if for a static group, we should allow users toremove 
> individual members? For a dynamic group this feature would not
make much sense IMHO, because the `memberId` is not know by the user.

I think his point is similar to what I was trying to get at earlier, with the 
proposal to add a new
#removeAllMembers API rather than an API to remove individual members according 
to their
memberId. As he explained, removing based on memberId is likely not that useful 
in general.
Also, it's not actually what we want to do here; maybe we should avoid adding a 
new API 
that we think may be useful in other contexts (remove individual member based 
on memberId),
and just add the API we actually need (remove all members from group) in this 
KIP? We can
always add the "remove individual member by memberId" API at a later point, if 
it turns out to
actually be requested for specific reasons?

Also, it's more efficient to just send a single "clear the group" request vs 
sending a LeaveGroup
request for every single member. What do you think?




On Sat, Mar 7, 2020 at 1:41 AM feyman2009  wrote:
Hi, Matthias
 Thanks, I updated the KIP to mention the deprecated and newly added 
methods.

 1. What happens is `groupInstanceId` is used for a dynamic group? What
 happens if both parameters are specified? What happens if `memberId`
 is specified for a static group?

 => my understanding is that the dynamic/static membership is member level 
other than group level, and I think above questions could be answered by the 
"Leave Group Logic Change" section in KIP-345: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-345%3A+Introduce+static+membership+protocol+to+reduce+consumer+rebalances,
 this KIP stays consistent with KIP-345.

 2. About the `--force` option. Currently, StreamsResetter fails with an
 error if the consumer group is not empty. You state in your KIP that:

 > without --force, we need to wait for session timeout.

 Is this an intended behavior change if `--force` is not used or is the
 KIP description incorrect?

 => This is the intended behavior. For this part, I think there are two ways to 
go:
 1) (the implicit way) Not introducing the new "--force" option, with this KIP, 
StreamsResetter will by default remove active members(with long session timeout 
configured) on broker side 
 2) (the explicit way) Introduce the new "--force" option, users need to 
explicitly specify --force to remove the active members. If --force not 
specified, the StreamsResetter behaviour is as previous versions'.

 I think the two alternatives above are both feasible, personally I prefer way 
2.

 3. For my own understanding: with the `--force` option, we intend to get
 all `memberIds` and send a "remove member" request for each with
 corresponding `memberId` to remove the member from the group
 (independent is the group is static or dynamic)?

 => Yeah, minor thing to mention is we will send the "remove member" request 
for each member(could be dynamic member or static member) to remove them from 
group
 for dynamic membe

Build failed in Jenkins: kafka-2.5-jdk8 #61

[Apache Jenkins Server]

See 


Changes:

[matthias] MINOR: Update Streams IQ JavaDocs to not point to a deprecated method


--
[...truncated 2.89 MB...]
org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicNameAndTimestamp P

[jira] [Created] (KAFKA-9709) add a confirm when kafka-server-stop.sh find multiple kafka instances to kill

[qiang Liu (Jira)]

qiang Liu created KAFKA-9709:


 Summary: add a confirm when kafka-server-stop.sh find multiple 
kafka instances to kill
 Key: KAFKA-9709
 URL: https://issues.apache.org/jira/browse/KAFKA-9709
 Project: Kafka
  Issue Type: Improvement
  Components: admin
Affects Versions: 2.4.1
Reporter: qiang Liu


currently kafka-server-stop.sh find all kafka instances on the machine and kill 
them all with out any confirm, when deploy multity instance in one machine, 
some may mistakenly kill all instance while only mean to kill single one.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-9605) EOS Producer could throw illegal state if trying to complete a failed batch after fatal error

[Jason Gustafson (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-9605.

Resolution: Fixed

> EOS Producer could throw illegal state if trying to complete a failed batch 
> after fatal error
> -
>
> Key: KAFKA-9605
> URL: https://issues.apache.org/jira/browse/KAFKA-9605
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 2.4.0, 2.5.0
>Reporter: Boyang Chen
>Assignee: Boyang Chen
>Priority: Major
> Fix For: 2.6.0
>
>
> In the Producer we could see network client hits fatal exception while trying 
> to complete the batches after Txn manager hits fatal fenced error:
> {code:java}
>  
> [2020-02-24T13:23:29-08:00] 
> (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 
> 21:23:28,673] ERROR [kafka-producer-network-thread | 
> stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer]
>  [Producer 
> clientId=stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer,
>  transactionalId=stream-soak-test-1_0] Aborting producer batches due to fatal 
> error (org.apache.kafka.clients.producer.internals.Sender)
> [2020-02-24T13:23:29-08:00] 
> (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) 
> org.apache.kafka.common.errors.ProducerFencedException: Producer attempted an 
> operation with an old epoch. Either there is a newer producer with the same 
> transactionalId, or the producer's transaction has been expired by the broker.
> [2020-02-24T13:23:29-08:00] 
> (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 
> 21:23:28,674] INFO 
> [stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3] 
> [Producer 
> clientId=stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-0_0-producer,
>  transactionalId=stream-soak-test-0_0] Closing the Kafka producer with 
> timeoutMillis = 9223372036854775807 ms. 
> (org.apache.kafka.clients.producer.KafkaProducer)
> [2020-02-24T13:23:29-08:00] 
> (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 
> 21:23:28,684] INFO [kafka-producer-network-thread | 
> stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer]
>  [Producer 
> clientId=stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer,
>  transactionalId=stream-soak-test-1_0] Resetting sequence number of batch 
> with current sequence 354277 for partition windowed-node-counts-0 to 354276 
> (org.apache.kafka.clients.producer.internals.TransactionManager)
> [2020-02-24T13:23:29-08:00] 
> (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 
> 21:23:28,684] INFO [kafka-producer-network-thread | 
> stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer]
>  Resetting sequence number of batch with current sequence 354277 for 
> partition windowed-node-counts-0 to 354276 
> (org.apache.kafka.clients.producer.internals.ProducerBatch)
> [2020-02-24T13:23:29-08:00] 
> (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) [2020-02-24 
> 21:23:28,685] ERROR [kafka-producer-network-thread | 
> stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer]
>  [Producer 
> clientId=stream-soak-test-5e16fa60-12a3-4c4f-9900-c75f7d10859f-StreamThread-3-1_0-producer,
>  transactionalId=stream-soak-test-1_0] Uncaught error in request completion: 
> (org.apache.kafka.clients.NetworkClient)
> [2020-02-24T13:23:29-08:00] 
> (streams-soak-trunk-eos_soak_i-02ea56d369c55eec2_streamslog) 
> java.lang.IllegalStateException: Should not reopen a batch which is already 
> aborted.
>         at 
> org.apache.kafka.common.record.MemoryRecordsBuilder.reopenAndRewriteProducerState(MemoryRecordsBuilder.java:295)
>         at 
> org.apache.kafka.clients.producer.internals.ProducerBatch.resetProducerState(ProducerBatch.java:395)
>         at 
> org.apache.kafka.clients.producer.internals.TransactionManager.lambda$adjustSequencesDueToFailedBatch$4(TransactionManager.java:770)
>         at 
> org.apache.kafka.clients.producer.internals.TransactionManager$TopicPartitionEntry.resetSequenceNumbers(TransactionManager.java:180)
>         at 
> org.apache.kafka.clients.producer.internals.TransactionManager.adjustSequencesDueToFailedBatch(TransactionManager.java:760)
>         at 
> org.apache.kafka.clients.producer.internals.TransactionManager.handleFailedBatch(TransactionManager.java:735)
>         at 
> org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:671)
>         at 
> org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:662)
>         at 
> org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:620)

[jira] [Created] (KAFKA-9708) Connector does not prefer to use packaged classes during configuration

[Greg Harris (Jira)]

Greg Harris created KAFKA-9708:
--

 Summary: Connector does not prefer to use packaged classes during 
configuration
 Key: KAFKA-9708
 URL: https://issues.apache.org/jira/browse/KAFKA-9708
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: Greg Harris
Assignee: Greg Harris


In connector tasks, classes loaded during configuration are preferentially 
loaded from the PluginClassLoader since KAFKA-8819 was implemented. This same 
prioritization is not currently respected in the connector itself, where the 
delegating classloader is used as the context classloader. This leads to the 
possibility for different versions of converters to be loaded, or different 
versions of dependencies to be found when executing code in the connector vs 
task.

Worker::startConnector should be changed to follow the startTask / KAFKA-8819 
prioritization scheme, by activating the PluginClassLoader earlier.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9707) InsertField.Key transformation does not apply to tombstone records

[Greg Harris (Jira)]

Greg Harris created KAFKA-9707:
--

 Summary: InsertField.Key transformation does not apply to 
tombstone records
 Key: KAFKA-9707
 URL: https://issues.apache.org/jira/browse/KAFKA-9707
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: Greg Harris
Assignee: Greg Harris


Reproduction steps:
 # Configure an InsertField.Key transformation
 # Pass a tombstone record (with non-null key, but null value) through the 
transform

Expected behavior:

The key field is inserted, and the value remains null

Observed behavior:

The key field is not inserted, and the value remains null



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9706) Flatten transformation fails when encountering tombstone event

[Greg Harris (Jira)]

Greg Harris created KAFKA-9706:
--

 Summary: Flatten transformation fails when encountering tombstone 
event
 Key: KAFKA-9706
 URL: https://issues.apache.org/jira/browse/KAFKA-9706
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: Greg Harris
Assignee: Greg Harris


When applying the {{Flatten}} transformation to a tombstone event, an exception 
is raised:
{code:java}
org.apache.kafka.connect.errors.DataException: Only Map objects supported in 
absence of schema for [flattening], found: null
{code}
Instead, the transform should pass the tombstone through the transform without 
throwing an exception.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jira Cleanup

[Sönke Liebau]

All,

I left a few comments on some old but still open jiras in an attempt to
clean up a little bit.

Since probably no one would notice these comments I thought I'd quickly
list them here to give people a chance to check on them:

KAFKA-1265  - SBT and
Gradle create jars without expected Maven files
KAFKA-1231  - Support
partition shrink (delete partition)
KAFKA-1336  - Create
partition compaction analyzer
KAFKA-1440  - Per-request
tracing
KAFKA-1518  -
KafkaMetricsReporter prevents Kafka from starting if the custom reporter
throws an exception


 I'll wait a few days for objections and then close these issues if none
are forthcoming.

Best regards,
Sönke

[jira] [Resolved] (KAFKA-5972) Flatten SMT does not work with null values

[Greg Harris (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5972?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Harris resolved KAFKA-5972.

Resolution: Fixed

> Flatten SMT does not work with null values
> --
>
> Key: KAFKA-5972
> URL: https://issues.apache.org/jira/browse/KAFKA-5972
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.11.0.0, 0.11.0.1
>Reporter: Tomas Zuklys
>Assignee: siva santhalingam
>Priority: Minor
>  Labels: easyfix, patch
> Fix For: 1.0.3, 1.1.2, 2.0.2, 2.1.2, 2.3.1, 2.4.0, 2.2.2
>
> Attachments: kafka-transforms.patch
>
>
> Hi,
> I noticed a bug in Flatten SMT while doing tests with different SMTs that are 
> provided out-of-box.
> Flatten SMT does not work as expected with schemaless JSON that has 
> properties with null values. 
> Example json:
> {code}
>   {A={D=dValue, B=null, C=cValue}}
> {code}
> The issue is in if statement that checks for null value.
> Current version:
> {code}
>   for (Map.Entry entry : originalRecord.entrySet()) {
> final String fieldName = fieldName(fieldNamePrefix, 
> entry.getKey());
> Object value = entry.getValue();
> if (value == null) {
> newRecord.put(fieldName(fieldNamePrefix, entry.getKey()), 
> null);
> return;
> }
> ...
> {code}
> should be
> {code}
>   for (Map.Entry entry : originalRecord.entrySet()) {
> final String fieldName = fieldName(fieldNamePrefix, 
> entry.getKey());
> Object value = entry.getValue();
> if (value == null) {
> newRecord.put(fieldName(fieldNamePrefix, entry.getKey()), 
> null);
> continue;
> }
> {code}
> I have attached a patch containing the fix for this issue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (KAFKA-1292) Command-line tools should print what they do as part of their usage command

[Jira]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1292?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sönke Liebau resolved KAFKA-1292.
-
Resolution: Fixed

This has been implemented by now, I do believe within the linked ticket.

> Command-line tools should print what they do as part of their usage command
> ---
>
> Key: KAFKA-1292
> URL: https://issues.apache.org/jira/browse/KAFKA-1292
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jay Kreps
>Priority: Major
>  Labels: usability
>
> It would be nice if you could get an explanation for each tool by running the 
> command with no arguments. Something like 
> bin/kafka-preferred-replica-election.sh is a little scary so it would be nice 
> to have it self-describe:
> > bin/kafka-preferred-replica-election.sh
> This command attempts to return leadership to the preferred replicas (if they 
> are alive) from whomever is currently the leader).
> Option  Description   
>  
> --  ---   
>  
> --alter Alter the configuration for the topic.
> ...



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Add a customized logo for Kafka Streams

[Sophie Blee-Goldman]

Just to throw another proposal out there and inspire some debate, here's a
similar-but-different
idea (inspired by John + some sketches I found on google):

*~~ See attachment, inlined image is too large for the mailing list ~~*

This one's definitely more fun, my only concern is that it doesn't really
scale well. At the lower end
of sizes the otters will be pretty difficult to see; and I had to stretch
out the Kafka circles even at
the larger end just to fit them through.

But maybe with a cleaner drawing and some color it'll still look good and
be recognizable + distinct
enough when small.

Any thoughts? Any binding and/or non-binding votes?

On Sun, Mar 8, 2020 at 1:00 AM Sophie Blee-Goldman 
wrote:

> Seems the mailing list may have filtered the inlined prototype logo,
> attaching it here instead
>
> On Sat, Mar 7, 2020 at 11:54 PM Sophie Blee-Goldman 
> wrote:
>
>> Matthias makes a good point about being careful not to position Streams as
>> outside of Apache Kafka. One obvious thing we could do it just include
>> the
>> Kafka logo as-is in the Streams logo, somehow.
>>
>> I have some unqualified opinions on what that might look like:
>> A good logo is simple and clean, so incorporating the Kafka logo as a
>> minor
>> detail within a more complicated image is probably not the best way to get
>> the quick and easy comprehension/recognition that we're going for.
>>
>> That said I'd throw out the idea of just attaching something to the Kafka
>> logo,
>> perhaps a stream-dwelling animal, perhaps a (river) otter? It could be
>> "swimming" left of the Kafka logo, with its head touching the upper
>> circle and
>> its tail touching the bottom one. Like Streams, it starts with Kafka and
>> ends
>> with Kafka (ie reading input topics and writing to output topics).
>>
>> Without further ado, here's my very rough prototype for the Kafka Streams
>> logo:
>>
>> [image: image.png]
>> Obviously the real thing would be colored and presumably done by someone
>> with actual artist talent/experience (or at least photoshop ability).
>>
>> Thoughts?
>>
>> On Sat, Mar 7, 2020, 1:08 PM Matthias J. Sax  wrote:
>>
>>> -BEGIN PGP SIGNED MESSAGE-
>>> Hash: SHA512
>>>
>>> Boyang,
>>>
>>> thanks for starting this discussion. I like the idea in general
>>> however we need to be a little careful IMHO -- as you mentioned Kafka
>>> is one project and thus we should avoid the impression that Kafka
>>> Streams is not part of Apache Kafka.
>>>
>>> Besides this, many projects use animals that are often very adorable.
>>> Maybe we could find a cute Streams related mascot? :)
>>>
>>> I would love to hear opinions especially from the PMC if having a logo
>>> for Kafka Streams is a viable thing to do.
>>>
>>>
>>> - -Matthias
>>>
>>> On 3/3/20 1:01 AM, Patrik Kleindl wrote:
>>> > Hi Boyang Great idea, that would help in some discussions. To throw
>>> > in a first idea: https://imgur.com/a/UowXaMk best regards Patrik
>>> >
>>> > On Mon, 2 Mar 2020 at 18:23, Boyang Chen
>>> >  wrote:
>>> >
>>> >> Hey Apache Kafka committers and community folks,
>>> >>
>>> >> over the years Kafka Streams has been widely adopted and tons of
>>> >> blog posts and tech talks have been trying to introduce it to
>>> >> people with need of stream processing. As it is part of Apache
>>> >> Kafka project, there is always an awkward situation where Kafka
>>> >> Streams could not be campaigned as a standalone streaming engine,
>>> >> and makes people confused about its relation to Kafka.
>>> >>
>>> >> So, do we want to introduce a customized logo just for Streams?
>>> >> The immediate benefit is when people are making technical
>>> >> decisions, we could list Streams as a logo just like Flink and
>>> >> Spark Streaming, instead of putting Kafka logo there as it is not
>>> >> literally a legitimate comparison between processing framework
>>> >> and messaging system. Should we do a KIP for this?
>>> >>
>>> >> Boyang
>>> >>
>>> >
>>> -BEGIN PGP SIGNATURE-
>>>
>>> iQIzBAEBCgAdFiEEI8mthP+5zxXZZdDSO4miYXKq/OgFAl5kDVQACgkQO4miYXKq
>>> /Og2Hg/+IOjjz3yrvHOz/p/Qbi9hXDK+GZT6Ixzk2HrrpMiLAAZs6WrNGQMwwI6a
>>> qrPdMRLKA45F53wwMXBqhLYe0O0vOIRSur3pef8UuTVEkbFstY6dNzduPjTXK4vV
>>> Ahzb+pu1lZsF+n2DbMuXWvkmvMAsSrKQXbH58rA7I4qx7Zr6g1a/KL2b2oOoo3kI
>>> 4x3PJfG6oLSnQHwkJxmM79ZjM7MLZh0d8cRqb7Oudy5MJiMzHY+Rm5aTu4nhPgRr
>>> cLxA8kz1PbGPboxjD9/ZGuZJMWfVnvY1wJcOp5UnOUs4kX5uYDyWw1sKIn3DcnW8
>>> YVzoto0syCHTAdbl89H2fxhJbtVp8JSxbBx9AW8mdgLOLsYRZGHZ1cbdJ3h4NkeE
>>> xTPTuTptQbhdcjbSVX6F0q+h1hiPCU5PKqcR12zGVTBI4rOGkhPIhDdnIti5Qp2e
>>> MQ1Urh/tWCfwiJZbOyjZ9Tz/2vbUBB4kzUI9DxwWpD15jgMuN3JfqFwcwSIm7NrY
>>> jHcM3UB2QlUt+uymU78xjy6er2AFiGrfL7UXeFHzzVIWBU3fdw4wqpuIuWjFhL3t
>>> NWnAvamXOc7kfE4VXQ8igoRUVDZ8xCCqNvVoaJZ1cWboY/Cfk+4sgd9QnDMZN9jU
>>> bzRpghQp3R93Vp5FH4L9z478uuh8DFM2/+6pqqapd9efydGtKrU=
>>> =OFBs
>>> -END PGP SIGNATURE-
>>>
>>

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Maulin Vasavada]

Thanks Rajini. Sounds good. I'll make changes and update this thread.

On Wed, Mar 11, 2020 at 6:41 AM Rajini Sivaram 
wrote:

> Hi Maulin,
>
> I have reviewed the PR and left some comments, can you turn it into a PR
> for https://github.com/apache/kafka? It looks good overall.
>
> We pass all configs to other plugins, so we can do the same here. That
> would be safer than assuming that all custom SSL-related configs start with
> `ssl.`. You can use principal builder as an example for what we do today.
>
> Regards,
>
> Rajini
>
> On Thu, Mar 5, 2020 at 12:03 AM Maulin Vasavada  >
> wrote:
>
> > Hi Rajini
> >
> > I made changes suggested by you on
> > https://github.com/maulin-vasavada/kafka/pull/4. Please check.
> >
> > In particular I had challenge in 'SslFactory#configure()' method the
> first
> > time to know which configs I have to add without having actual
> > SslEngineFactory impl. I think it is best to just copy all configs with
> > "ssl." prefix. Can you please review
> >
> >
> https://github.com/maulin-vasavada/kafka/pull/4/files#diff-1e3432211fdbb7b2e2b44b5d8838a40bR90
> >  particularly?
> >
> > Clement, I missed to see your point about Mode in previous post but even
> > after I realized what you are suggesting, my response would be the same
> as
> > before :)
> >
> > Thanks
> > Maulin
> >
> >
> > On Wed, Feb 5, 2020 at 8:40 PM Maulin Vasavada <
> maulin.vasav...@gmail.com>
> > wrote:
> >
> > > Hi Rajini
> > >
> > > Will accommodate your comments.
> > >
> > > Celement, while SSLContext factories are common, in this particular
> case,
> > > we need SSLEngine object because Kafka is using SSLEngine (as mentioned
> > > much before in this email thread, the SSLContext acts as factory for
> > > getting SSLEngine, SSLSocketFactory or SSLServerSocketFactory and Kafka
> > > chooses SSLEngine to be used for SSL Connections). The 'Mode' challenge
> > > doesn't go away easily since Kafka is using the "same" classes for
> Client
> > > side and Server side. Other stacks where you don't see this challenge
> is
> > > because either libraries are client centric or server centric and not
> > both
> > > at the same time. I would suggest you do a deeper dive into the sample
> > Pull
> > > request, build the code to get better idea about it. I don't find it
> > > strange to have 'Mode' argument in this context of Kafka. Kafka is not
> > > doing anything out of norm here since ultimately it is using JSSE spec
> > for
> > > SSL Connection.
> > >
> > > I will try to respond to code comments in couple of weeks since I am
> out
> > > for few weeks. Will keep you guys posted.
> > >
> > > Thanks
> > > Maulin
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Feb 5, 2020 at 9:50 PM Pellerin, Clement <
> > clement_pelle...@ibi.com>
> > > wrote:
> > >
> > >> Many of these points came up before.
> > >>
> > >> I had great hope when Maulin suggested the custom factory could
> > >> return an SSLContext instead of SSLEngine.  SSLContext factories are
> > >> common,
> > >> whereas I have never seen an SSLEngine factory being used before.
> > >> He must have hit the same problem I had with the Mode.
> > >>
> > >> If the Mode can be removed, can we find a way to return an SSLContext
> > now?
> > >> What is so special about Kafka that it needs to hardcode the Mode when
> > >> everyone
> > >> else works with the SSLContext and ignores the other mode they don't
> > use.
> > >>
> > >> -Original Message-
> > >> From: Rajini Sivaram [mailto:rajinisiva...@gmail.com]
> > >> Sent: Wednesday, February 5, 2020 10:03 AM
> > >> To: dev
> > >> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
> > >> extensible
> > >>
> > >> One more point:
> > >> 5) We should also add a method to SslEngineFactory that returns
> > >> `Set
> > >> reconfigurableConfigs()`
> > >>
> > >> On Wed, Feb 5, 2020 at 1:50 PM Rajini Sivaram <
> rajinisiva...@gmail.com>
> > >> wrote:
> > >>
> > >> > Hi Maulin,
> > >> >
> > >> > Thanks for the updates. A few comments below:
> > >> >
> > >> > 1) SslEngineFactory is currently in the internal package
> > >> > org.apache.kafka.common.security.ssl. We should move it to the
> public
> > >> > package org.apache.kafka.common.security.auth.
> > >> > 2) SslEngineFactory should extend Configurable and Closeable. We
> > should
> > >> > expect the implementation class to have a default constructor and
> > >> invoke configure()
> > >> > to be consistent with otSslher pluggable classes.
> > >> > 3) SslEngineFactory.createSslEngine uses the internal enum `Mode`.
> It
> > >> > would be better to add two methods instead:
> > >> >
> > >> >
> > >> >- SSLEngine createClientSslEngine(String peerHost, int peerPort,
> > >> String endpointIdentification);
> > >> >- SSLEngine createServerSslEngine(String peerHost, int peerPort);
> > >> >
> > >> > 4) Don't think we need a method on SslEngineFactory to return
> configs.
> > >> It seems like an odd thing to do in a pubic Configurable API and is
> > >> inconsist

[jira] [Created] (KAFKA-9705) (Incremental)AlterConfig should be propagated from Controller in bridge release

[Boyang Chen (Jira)]

Boyang Chen created KAFKA-9705:
--

 Summary: (Incremental)AlterConfig should be propagated from 
Controller in bridge release
 Key: KAFKA-9705
 URL: https://issues.apache.org/jira/browse/KAFKA-9705
 Project: Kafka
  Issue Type: Sub-task
Reporter: Boyang Chen
Assignee: Boyang Chen


In the bridge release, we need to restrict the direct access of ZK to 
controller only. This means the existing AlterConfig path should be migrated.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

[Guozhang Wang]

Hi Vijay,

The current progress is tracked under the ticket
https://issues.apache.org/jira/browse/KAFKA-9119 as sub-tasks. The first
milestone would be moving off all admin request handling out of ZK and let
them to talk to brokers.


Guozhang

On Wed, Mar 11, 2020 at 8:23 AM vijay gupta  wrote:

> Hi,
>
> What’s the latest status on this JIRA?
>
> Regards,
> Vijay
>
> On 2019/11/06 15:07:03 Viktor Somogyi-Vass wrote:
> > Hi Elmahdi,
> >
> > I've added the JIRA to the KIP (and also below) where you can track the
> > progress (but more subtask will come as the current ones don't represent
> > the full work to be done).
> > https://issues.apache.org/jira/browse/KAFKA-9119
> >
> > Viktor
> >
> > On Tue, Nov 5, 2019 at 5:15 PM Elmahdi FRID wrote:
> >
> > > Hello Folks any status abbout this kip and it's possible to test this
> use
> > > case ?
> > >
> > > On 2019/08/01 21:04:46, "Colin McCabe" wrote:
> > > > Hi all,
> > > >
> > > > I've written a KIP about removing ZooKeeper from Kafka. Please take a
> > > look and let me know what you think:
> > > >
> > > >
> > >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
> > > >
> > > > cheers,
> > > > Colin
> > > >
> > >
> >
>
>
> Sent from my iPhone



-- 
-- Guozhang

RE: Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

[vijay gupta]

Hi,

What’s the latest status on this JIRA?

Regards,
Vijay

On 2019/11/06 15:07:03 Viktor Somogyi-Vass wrote:
> Hi Elmahdi,
> 
> I've added the JIRA to the KIP (and also below) where you can track the
> progress (but more subtask will come as the current ones don't represent
> the full work to be done).
> https://issues.apache.org/jira/browse/KAFKA-9119
> 
> Viktor
> 
> On Tue, Nov 5, 2019 at 5:15 PM Elmahdi FRID wrote:
> 
> > Hello Folks any status abbout this kip and it's possible to test this use
> > case ?
> >
> > On 2019/08/01 21:04:46, "Colin McCabe" wrote:
> > > Hi all,
> > >
> > > I've written a KIP about removing ZooKeeper from Kafka. Please take a
> > look and let me know what you think:
> > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum
> > >
> > > cheers,
> > > Colin
> > >
> >
>


Sent from my iPhone

Re: [DISCUSS] KIP-552: Add interface to handle unused config

[Artur Burtsev]

Long time no see, was on holiday 🇳🇿.

Thread got an interesting twist into reporting and fixing every single
warning individually, which is outside of my capacity. As I mentioned
in the KIP and also as highlighted by Patric - you don't have to do
anything extraordinary to get these warnings, you just use default
Kafka components (Streams, Connect) or Confluent schema registry. So
hopefully one day it will be fixed.

I totally agree with Gwen, any misconfigs should be as visible as
possible, however the more false positives you have, the easier it is
to overlook misconfig.

We don't worry about misconfigs as much as we do about warnings, so we
would mute easily, unfortunately very useful configuration dump is on
the same logger.

Solution for as would be to wrap producer and consumer, log config
dumps from wrappers and mute original config loggers.

I don't mind KIP to be canceled.

Thanks,
Artur

On Sat, Mar 7, 2020 at 10:11 PM Matthias J. Sax  wrote:
>
> -BEGIN PGP SIGNED MESSAGE-
> Hash: SHA512
>
> What is the status of this KIP?
>
>
> - -Matthias
>
> On 2/17/20 2:41 PM, John Roesler wrote:
> > Thanks Matthias,
> >
> > I got the impression this was considered and rejected in
> > KAFKA-7509, but I'm not sure why. Maybe it was never really
> > considered at all, just proposed and not-noticed? Perhaps Randall
> > or Sönke can comment. See:
> https://issues.apache.org/jira/browse/KAFKA-7509?focusedCommentId=166608
> 68&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpan
> el#comment-16660868
> >
> > It would be good to know why that proposal didn't move forward.
> >
> > Thanks, -John
> >
> >
> >
> > On Mon, Feb 17, 2020, at 12:17, Matthias J. Sax wrote: I am just
> > getting aware of this KIP (not sure why I missed it).
> >
> > In Kafka Streams we have nested clients and need to "forward"
> > configs from outer layer to inner layers -- hence, we prefix some
> > configs to be able to know which inner nested clients needs this
> > config.
> >
> > I think the simplest approach is, to add a prefix (like
> > "userconfig."). All thus configs would be skipped in the
> > validation step to avoid the WARN log.
> >
> > When forwarding configs to inner classed (like nested clients in
> > KS, serializers etc) we would remove this prefix).
> >
> > Using a `RecordingMap` seem rather heavy weight and complex?
> >
> > Thoughts?
> >
> > -Matthias
> >
> > On 2/17/20 9:09 AM, John Roesler wrote:
>  Thanks Patrik,
> 
>  This seems to be a long and wandering issue. It seems that
>  KAFKA-7509 has followed a similar trajectory to
>  KAFKA-6793/KIP-552 , and 7509 is just recently closed in
>  favor of whatever we decide to do in KAFKA-6793.
> 
>  Considering (what I hope is) the whole history of this issue,
>  a few things emerge:
> 
>  1. It's useful to get warned when you pass an invalid
>  configuration 2. It's not possible for the "top layer"
>  (Streams, Connect, etc.) to know up front which
>  configurations are applicable to pass down to the "second"
>  layer (Clients, RocksDB) because those layers themselves are
>  extensible (see below) 3. We should propose a change that
>  fixes this issue for the whole Kafka ecosystem at once.
> 
>  Elaboration on point 2: Users of Kafka libraries need to
>  register extra components like Processors, Interceptors,
>  RocksDBConfigSetters, RebalanceListeners, etc. They need to
>  pass configurations into these self-registered components.
>  Therefore, the outermost component (the one that you directly
>  pass a Properties to, and that instantiates other
>  Configurable components) _cannot_ know which configurations
>  are needed by the "extra" components inside the Configurable
>  components. Therefore, no approach that involves filtering
>  only the "needed" configurations up front, before
>  constructing a Configurable component, could work.
> 
>  Randall made an aside in this comment:
>  https://issues.apache.org/jira/browse/KAFKA-7509?focusedCommentId=1
> 667
> >
> 
> 3834&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpan
> > el#comment-16673834
> 
> 
> > which I think is the most promising path right now.
>  Namely, to use RecordingMap (or a similar approach) when
>  configuring internal components and finally warn when
>  _everything_ has been wired up if some configuration value
>  wasn't used by _any_ component.
> 
>  It seems like this approach would satisfy all three of the
>  above points, but it needs some design/discovery work to see
>  what gaps exist in the current code base to achieve the goal.
>  It also might be a fair amount of work (which is why we
>  didn't follow that approach in KAFKA-7509), but I don't think
>  there have been any other suggestions that satisfy both point
>  1 and point 2.
> 
>  Thoughts? -John
> 
>  On 

Re: Feature request for 'prefixless' topic mirroring

[Ryanne Dolan]

Hello Marcin, actually KIP-382 includes LegacyReplicationPolicy (not yet
released) which is similar to your PrefixlessReplicationPolicy and the
MigrationReplicationPolicy mentioned. I don't think these are hacky, though
maybe the documentation could better address what to do when a method
doesn't necessarily apply cleanly.

Frankly, I anticipated needing to make changes to MM2 itself to accommodate
these sort of use-cases -- the fact that you got this working as-is is
encouraging :)

Re the proposal, I think we could ship one or more of these replication
policies with MM2 to support prefixless replication. There may be slightly
different implementations depending on the use-case, i.e. "migration" from
an old cluster vs active/passive w/o renaming topics vs mimicking MM1 with
LegacyReplicationPolicy. Maybe they are all identical and we just need one?

I like that use-cases are driving these requests rather than us try to
enumerate all possible ReplicationPolicy implementations. Given a
particular use-case and a successful application of a custom
ReplicationPolicy, I for one would support including it in Apache Kafka.

Ryanne

On Wed, Mar 11, 2020, 5:26 AM Marcin Wieloch 
wrote:

> Dear All,
>
> I would like to raise a feature request for ‘prefixless’ topic
> mirroring with MirrorMaker2 (see detailed description below).
>
> As after quite a few tries I am not 100% sure, could you please
> confirm that by now
> the ‘prefixless’ mirroring is not supported and that my feature
> request is valid? I would then file it in JIRA.
>
> Best regards,
>
> Marcin
>
>
> Feature Request
>
> We would like to use MirrorMaker2 as a DRP solution for a simple setup
> with two sites working in active-passive configuration
> where all topics are always mirrored in one direction only and the
> downstream topics keep their original names, i.e., without being
> prefixed by MirrorMaker2.
>
> After a few rounds of 'trial and error' we came up with a customized
> replication policy which, although hacky, seems to fulfill our need;
> it is presented in stackoverflow here:
> https://stackoverflow.com/a/60619233
> and source code is available here:
>
> https://github.com/mawileo/mm2-prefixless-policy/blob/feature/prefixless/src/main/java/ch/mawileo/kafka/mm2/PrefixlessReplicationPolicy.java
>
> This solution is 'hacky' in that it violates specification for the
> topicSource and upstreamTopic methods as expressed in javadoc of the
> ReplicationPolicy interface.
>
> That makes us suspect that MirrorMaker2 has been never meant to work
> in such a configuration and that our solution may stop
> working with future releases of Kafka.
>
> We would like then to ask for making the possibility of 'prefixless'
> topic mirroring in a simplified two-cluster setup a standard
> feature of Kafka Mirror Maker 2.
>

[RESULTS] [VOTE] Release Kafka version 2.4.1 UPDATE

[Bill Bejeck]

Previously I erroneously reported the vote with 4 PMC +1s. I thought one of
the voters was also a PMC member. But since there are still 3 PMC + 1s, the
passing vote for this release is still valid.


This vote passes with 10 +1 votes (3 bindings) and no 0 or -1 votes.

+1 votes
PMC Members (in voting order):
* Colin McCabe
* Vahid Hashemian
* Gwen Shapira

Committers:
* David Arthur
* Mickael Maison

Community (in voting order):
* Eric Lalonde
* Eno Thereska
* Tom Bentley
* Sean Glover
* Levani Kokhreidze

Vote thread:
https://www.mail-archive.com/dev@kafka.apache.org/msg105496.html

I'll continue with the release process, and the release announcement will
follow in the next few days.

Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Rajini Sivaram]

Hi Maulin,

I have reviewed the PR and left some comments, can you turn it into a PR
for https://github.com/apache/kafka? It looks good overall.

We pass all configs to other plugins, so we can do the same here. That
would be safer than assuming that all custom SSL-related configs start with
`ssl.`. You can use principal builder as an example for what we do today.

Regards,

Rajini

On Thu, Mar 5, 2020 at 12:03 AM Maulin Vasavada 
wrote:

> Hi Rajini
>
> I made changes suggested by you on
> https://github.com/maulin-vasavada/kafka/pull/4. Please check.
>
> In particular I had challenge in 'SslFactory#configure()' method the first
> time to know which configs I have to add without having actual
> SslEngineFactory impl. I think it is best to just copy all configs with
> "ssl." prefix. Can you please review
>
> https://github.com/maulin-vasavada/kafka/pull/4/files#diff-1e3432211fdbb7b2e2b44b5d8838a40bR90
>  particularly?
>
> Clement, I missed to see your point about Mode in previous post but even
> after I realized what you are suggesting, my response would be the same as
> before :)
>
> Thanks
> Maulin
>
>
> On Wed, Feb 5, 2020 at 8:40 PM Maulin Vasavada 
> wrote:
>
> > Hi Rajini
> >
> > Will accommodate your comments.
> >
> > Celement, while SSLContext factories are common, in this particular case,
> > we need SSLEngine object because Kafka is using SSLEngine (as mentioned
> > much before in this email thread, the SSLContext acts as factory for
> > getting SSLEngine, SSLSocketFactory or SSLServerSocketFactory and Kafka
> > chooses SSLEngine to be used for SSL Connections). The 'Mode' challenge
> > doesn't go away easily since Kafka is using the "same" classes for Client
> > side and Server side. Other stacks where you don't see this challenge is
> > because either libraries are client centric or server centric and not
> both
> > at the same time. I would suggest you do a deeper dive into the sample
> Pull
> > request, build the code to get better idea about it. I don't find it
> > strange to have 'Mode' argument in this context of Kafka. Kafka is not
> > doing anything out of norm here since ultimately it is using JSSE spec
> for
> > SSL Connection.
> >
> > I will try to respond to code comments in couple of weeks since I am out
> > for few weeks. Will keep you guys posted.
> >
> > Thanks
> > Maulin
> >
> >
> >
> >
> >
> >
> >
> >
> > On Wed, Feb 5, 2020 at 9:50 PM Pellerin, Clement <
> clement_pelle...@ibi.com>
> > wrote:
> >
> >> Many of these points came up before.
> >>
> >> I had great hope when Maulin suggested the custom factory could
> >> return an SSLContext instead of SSLEngine.  SSLContext factories are
> >> common,
> >> whereas I have never seen an SSLEngine factory being used before.
> >> He must have hit the same problem I had with the Mode.
> >>
> >> If the Mode can be removed, can we find a way to return an SSLContext
> now?
> >> What is so special about Kafka that it needs to hardcode the Mode when
> >> everyone
> >> else works with the SSLContext and ignores the other mode they don't
> use.
> >>
> >> -Original Message-
> >> From: Rajini Sivaram [mailto:rajinisiva...@gmail.com]
> >> Sent: Wednesday, February 5, 2020 10:03 AM
> >> To: dev
> >> Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration
> >> extensible
> >>
> >> One more point:
> >> 5) We should also add a method to SslEngineFactory that returns
> >> `Set
> >> reconfigurableConfigs()`
> >>
> >> On Wed, Feb 5, 2020 at 1:50 PM Rajini Sivaram 
> >> wrote:
> >>
> >> > Hi Maulin,
> >> >
> >> > Thanks for the updates. A few comments below:
> >> >
> >> > 1) SslEngineFactory is currently in the internal package
> >> > org.apache.kafka.common.security.ssl. We should move it to the public
> >> > package org.apache.kafka.common.security.auth.
> >> > 2) SslEngineFactory should extend Configurable and Closeable. We
> should
> >> > expect the implementation class to have a default constructor and
> >> invoke configure()
> >> > to be consistent with otSslher pluggable classes.
> >> > 3) SslEngineFactory.createSslEngine uses the internal enum `Mode`. It
> >> > would be better to add two methods instead:
> >> >
> >> >
> >> >- SSLEngine createClientSslEngine(String peerHost, int peerPort,
> >> String endpointIdentification);
> >> >- SSLEngine createServerSslEngine(String peerHost, int peerPort);
> >> >
> >> > 4) Don't think we need a method on SslEngineFactory to return configs.
> >> It seems like an odd thing to do in a pubic Configurable API and is
> >> inconsistent with other APIs. We can track configs in the internal
> >> SslFactory class instead.
> >>
> >
>

Has restarted broker caught up?

[Gokul Ramanan Subramanian]

Hi.

What is the idiomatic / correct way to know that a restarted broker in an
existing Kafka cluster has caught up to all the leaders for the partitions
it hosts? Is there a specific JMX metric published either by the leader /
follower that we can monitor?

Ideally, the metric would have the following characteristics:

*Metric:* IsInSync=Yes/No
*Dimensions:* LeaderId = L, FollowerId = F, Topic = T, Partition = P

If not, do you think this would be useful?

Feature request for 'prefixless' topic mirroring

[Marcin Wieloch]

Dear All,

I would like to raise a feature request for ‘prefixless’ topic
mirroring with MirrorMaker2 (see detailed description below).

As after quite a few tries I am not 100% sure, could you please
confirm that by now
the ‘prefixless’ mirroring is not supported and that my feature
request is valid? I would then file it in JIRA.

Best regards,

Marcin


Feature Request

We would like to use MirrorMaker2 as a DRP solution for a simple setup
with two sites working in active-passive configuration
where all topics are always mirrored in one direction only and the
downstream topics keep their original names, i.e., without being
prefixed by MirrorMaker2.

After a few rounds of 'trial and error' we came up with a customized
replication policy which, although hacky, seems to fulfill our need;
it is presented in stackoverflow here:
https://stackoverflow.com/a/60619233
and source code is available here:
https://github.com/mawileo/mm2-prefixless-policy/blob/feature/prefixless/src/main/java/ch/mawileo/kafka/mm2/PrefixlessReplicationPolicy.java

This solution is 'hacky' in that it violates specification for the
topicSource and upstreamTopic methods as expressed in javadoc of the
ReplicationPolicy interface.

That makes us suspect that MirrorMaker2 has been never meant to work
in such a configuration and that our solution may stop
working with future releases of Kafka.

We would like then to ask for making the possibility of 'prefixless'
topic mirroring in a simplified two-cluster setup a standard
feature of Kafka Mirror Maker 2.

[jira] [Created] (KAFKA-9704) z/OS won't let us resize file when mmap

[Shuo Zhang (Jira)]

Shuo Zhang created KAFKA-9704:
-

 Summary: z/OS won't let us resize file when mmap
 Key: KAFKA-9704
 URL: https://issues.apache.org/jira/browse/KAFKA-9704
 Project: Kafka
  Issue Type: Task
  Components: log
Affects Versions: 2.4.0
Reporter: Shuo Zhang
 Fix For: 2.5.0, 2.4.2


z/OS won't let us resize file when mmap, so we need to force unman like Windows.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9703) ProducerBatch.split takes up too many resources if the bigBatch is huge

[jiamei xie (Jira)]

jiamei xie created KAFKA-9703:
-

 Summary: ProducerBatch.split takes up too many resources if the 
bigBatch is huge
 Key: KAFKA-9703
 URL: https://issues.apache.org/jira/browse/KAFKA-9703
 Project: Kafka
  Issue Type: Bug
Reporter: jiamei xie


ProducerBatch.split takes up too many resources  and might cause outOfMemory 
error if the bigBatch is huge. About how I found this issue is in 
https://lists.apache.org/list.html?us...@kafka.apache.org:lte=1M:MESSAGE_TOO_LARGE

Following is the code which takes a lot of resources.

{code:java}
 for (Record record : recordBatch) {
assert thunkIter.hasNext();
Thunk thunk = thunkIter.next();
if (batch == null)
batch = createBatchOffAccumulatorForRecord(record, 
splitBatchSize);

// A newly created batch can always host the first message.
if (!batch.tryAppendForSplit(record.timestamp(), record.key(), 
record.value(), record.headers(), thunk)) {
batches.add(batch);
batch = createBatchOffAccumulatorForRecord(record, 
splitBatchSize);
batch.tryAppendForSplit(record.timestamp(), record.key(), 
record.value(), record.headers(), thunk);
}
{code}

Refer to RecordAccumulator#tryAppend, we can call closeForRecordAppends() after 
a batch is full.

{code:java}
private RecordAppendResult tryAppend(long timestamp, byte[] key, byte[] 
value, Header[] headers,
 Callback callback, 
Deque deque, long nowMs) {
ProducerBatch last = deque.peekLast();
if (last != null) {
FutureRecordMetadata future = last.tryAppend(timestamp, key, value, 
headers, callback, nowMs);
if (future == null)
last.closeForRecordAppends();
else
return new RecordAppendResult(future, deque.size() > 1 || 
last.isFull(), false, false);
}
return null;
}
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9702) Suspected memory leak

[Nishant Ranjan (Jira)]

Nishant Ranjan created KAFKA-9702:
-

 Summary: Suspected memory leak
 Key: KAFKA-9702
 URL: https://issues.apache.org/jira/browse/KAFKA-9702
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 2.4.0
Reporter: Nishant Ranjan


I am using Kafka consumer to fetch objects from Kafka topic and then persist 
them in DB.

When I ran, eclipse MAT for memory leaks, its giving following :
53 instances of *"java.util.zip.ZipFile$Source"*, loaded by *""* occupy *22,33,048 (31.01%)* bytes.

Also, my observation is that GC is not collecting objects.

Please let me know if more information is required.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)