Re: Embedded Ksql and Janino

[Jörn Franke]

Shade it into your application .

> Am 25.02.2020 um 07:16 schrieb Nishkam Ravi :
> 
> We are running into a curious problem with KSQL 5.3.0. When
> ksql-engine-5.3.0 is included as a dependency in ServiceX, the following
> error message shows up with "java -jar ServiceX.jar": "Could not find or
> load main class".
> We found that this is because of janino-3.0.7 (when excluded, the error
> message goes away).
> 
> Is it possible to build a Janino-free version of Ksql-5.3.0 or some other
> way to work around this problem? Any help would be appreciated.
> 
> Thanks,
> Nishkam

[jira] [Created] (KAFKA-9602) Incorrect close of producer instance during partition assignment

[Boyang Chen (Jira)]

Boyang Chen created KAFKA-9602:
--

 Summary: Incorrect close of producer instance during partition 
assignment
 Key: KAFKA-9602
 URL: https://issues.apache.org/jira/browse/KAFKA-9602
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Boyang Chen
Assignee: Boyang Chen


The new StreamProducer instance close doesn't distinguish between an 
EOS/non-EOS shutdown. The StreamProducer should take care of that.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-9601) Workers log raw connector configs, including values

[Chris Egerton (Jira)]

Chris Egerton created KAFKA-9601:


 Summary: Workers log raw connector configs, including values
 Key: KAFKA-9601
 URL: https://issues.apache.org/jira/browse/KAFKA-9601
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Reporter: Chris Egerton
Assignee: Chris Egerton


[This line right 
here|https://github.com/apache/kafka/blob/5359b2e3bc1cf13a301f32490a6630802afc4974/connect/runtime/src/main/java/org/apache/kafka/connect/runtime/WorkerConnector.java#L78]
 logs all configs (key and value) for a connector, which is bad, since it can 
lead to secrets (db credentials, cloud storage credentials, etc.) being logged 
in plaintext.

We can remove this line. Or change it to just log config keys. Or try to do 
some super-fancy parsing that masks sensitive values. Well, hopefully not that. 
That sounds like a lot of work.

Affects all versions of Connect back through 0.10.1.

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Embedded Ksql and Janino

[Nishkam Ravi]

We are running into a curious problem with KSQL 5.3.0. When
ksql-engine-5.3.0 is included as a dependency in ServiceX, the following
error message shows up with "java -jar ServiceX.jar": "Could not find or
load main class".
We found that this is because of janino-3.0.7 (when excluded, the error
message goes away).

Is it possible to build a Janino-free version of Ksql-5.3.0 or some other
way to work around this problem? Any help would be appreciated.

Thanks,
Nishkam

Re: [DISCUSS] KIP-573: Enable TLSv1.3 by default

[Nikolay Izhikov]

Hello.

Any feedback on this?

This change seems very simple, I can start vote right now if nothing to discuss 
here.

> 21 февр. 2020 г., в 15:18, Nikolay Izhikov  
> написал(а):
> 
> Hello, 
> 
> I'd like to start a discussion of KIP [1]
> This is follow-up for the KIP-553 [2]
> 
> Its goal is to enable TLSv1.3 by default.
> 
> Your comments and suggestions are welcome.
> 
> [1] 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-573%3A+Enable+TLSv1.3+by+default
> [2] https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=142641956

Build failed in Jenkins: kafka-2.4-jdk8 #153

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-9599 create unique sensor to record group rebalance (#8159)


--
[...truncated 2.75 MB...]
org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
STARTED

org.apache.kafka.streams.test.TestRecordTest > testPartialConstructorEquals 
PASSED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher STARTED

org.apache.kafka.streams.test.TestRecordTest > testMultiFieldMatcher PASSED

org.apache.kafka.streams.test.TestRecordTest > testFields STARTED

org.apache.kafka.streams.test.TestRecordTest > testFields PASSED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord STARTED

org.apache.kafka.streams.test.TestRecordTest > testProducerRecord PASSED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode STARTED

org.apache.kafka.streams.test.TestRecordTest > testEqualsAndHashCode PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordsFromKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKeyAndDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithDefaultTimestamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecordWithOtherTopicNameAndTimestampWithTimetamp 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullHeaders PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithDefaultTimestamp PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithNullKey PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateNullKeyConsumerRecord PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldCreateConsumerRecordWithOtherTopicName PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > shouldAdvanceTime 
PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldNotAllowToCreateTopicWithNullTopicNameWithKeyValuePairs PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 STARTED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 
shouldRequireCustomTopicNameIfNotDefaultFactoryTopicNameWithKeyValuePairsAndCustomTimestamps
 PASSED

org.apache.kafka.streams.test.ConsumerRecordFactoryTest > 

Build failed in Jenkins: kafka-2.4-jdk8 #152

[Apache Jenkins Server]

See 

Changes:


--
[...truncated 5.57 MB...]

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestampWithProducerRecord PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordWithExpectedRecordForCompareValueTimestamp 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullExpectedRecordForCompareValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldNotAllowNullProducerRecordForCompareKeyValue PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueAndTimestampIsEqualForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullReversForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfKeyAndValueIsEqualWithNullForCompareKeyValueWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfKeyIsDifferentWithNullForCompareKeyValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldPassIfValueAndTimestampIsEqualForCompareValueTimestampWithProducerRecord 
PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReversForCompareKeyValueTimestamp PASSED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 STARTED

org.apache.kafka.streams.test.OutputVerifierTest > 
shouldFailIfValueIsDifferentWithNullReverseForCompareValueTimestampWithProducerRecord
 PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > shouldReturnIsOpen 
PASSED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue STARTED

org.apache.kafka.streams.internals.KeyValueStoreFacadeTest > 
shouldDeleteAndReturnPlainValue PASSED

[jira] [Resolved] (KAFKA-9599) create unique sensor to record group rebalance

[Guozhang Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-9599.
--
Fix Version/s: 2.4.1
   2.5.0
   Resolution: Fixed

> create unique sensor to record group rebalance
> --
>
> Key: KAFKA-9599
> URL: https://issues.apache.org/jira/browse/KAFKA-9599
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Chia-Ping Tsai
>Assignee: Chia-Ping Tsai
>Priority: Major
> Fix For: 2.5.0, 2.4.1
>
>
> {code:scala}
>   val offsetDeletionSensor = metrics.sensor("OffsetDeletions")
>   ...
>   val groupCompletedRebalanceSensor = metrics.sensor("OffsetDeletions")
> {code}
> the "offset deletion" and "group rebalance" should not be recorded by the 
> same sensor since they are totally different.
> the code is introduced by KAFKA-8730



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Discussion thread for KIP-X

[Rangan Prabhakaran (BLOOMBERG/ 919 3RD A)]

Hi Boyang,
The Kafka clusters we manage are multi-tenant clusters hosting anywhere from 
hundreds to a few thousand different workloads on any given cluster. 

For our setup, we have noticed that the breaking limit wrt partition count is 
around 10k partitions per broker. Beyond this point, we start seeing 
significant replication slowness, election slowness, issues around too many 
files opened etc 

The type of workloads on our clusters that would benefit from the proposal 
outlined in this KIP are 

*Bursty workloads such as workloads that flood the topic once an hour and need 
to be processed quickly within a strict time window  
*Workloads that are using topics as simple queues (stateless and don’t care 
about ordering within a partition)
*Stream processing workloads where parallelism is driven by the number of input 
topic partitions 

Currently, we are over provisioning partitions to efficiently serve these 
workloads which results in significant under-utilization of the respective 
clusters. 

Additionally, we are also seeing quite a few workloads that are relying on the 
partition level ordering guarantees today and are filtering out the keys they 
don’t care about on the client side. These workloads would benefit from the key 
level ordering proposed in KIP-X and result in much simpler application logic 
for clients. 

Let me know if this helps and if you have any further questions
Rangan

From: dev@kafka.apache.org At: 02/21/20 15:45:49To:  dev@kafka.apache.org
Cc:  Rangan Prabhakaran (BLOOMBERG/ 919 3RD A ) ,  bche...@outlook.com
Subject: Re: Discussion thread for KIP-X

Hey Rangan,

thanks for the interest! In fact we are still in the design phase, and need
more supporting use cases that requires a higher scaling factor than number
of partitions. It would be good if you could share some of your needed use
case when the unit time of processing one record is the bottleneck, or some
cost wise concern of over-partitioning.

Boyang

On Fri, Feb 21, 2020 at 10:44 AM Guozhang Wang  wrote:

> cc @Boyang Chen  who authored this draft.
>
>
> Guozhang
>
> On Fri, Feb 21, 2020 at 10:29 AM Rangan Prabhakaran (BLOOMBERG/ 919 3RD A)
> <
> kprabhaka...@bloomberg.net> wrote:
>
> > Hi,
> > A few of us have been following KIP-X. We are interested in the roadmap /
> > plan there and would like to contribute towards the same.
> >
> > What are the next steps to discuss / iterate on this KIP ? Currently, its
> > in draft state and there does not seem to be a discussion thread attached
> > yet.
> >
> > KIP -
> >
> 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-X%3A+Introduce+a+cooperati
ve+consumer+processing+semantic
> >
> > Thanks
> > Rangan
>
>
>
> --
> -- Guozhang
>

[jira] [Created] (KAFKA-9600) EndTxn handler should check strict epoch equality

[Jason Gustafson (Jira)]

Jason Gustafson created KAFKA-9600:
--

 Summary: EndTxn handler should check strict epoch equality
 Key: KAFKA-9600
 URL: https://issues.apache.org/jira/browse/KAFKA-9600
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson


The EndTxn path in TransactionCoordinator is shared between direct calls to 
EndTxn from the client and internal transaction abort logic. To support the 
latter, the code is written to allow an epoch bump. However, if the client 
bumps the epoch unexpectedly (e.g. due to a buggy implementation), then we can 
be left with a hanging transaction. To fix this, we should ensure that an 
EndTxn from the client checks for strict epoch equality.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Request for adding wiki edit permission for creating KIP

[Xue LIU]

Thanks Matthias!

Cheers,
Xue

On Sat, Feb 22, 2020 at 5:00 AM Matthias J. Sax  wrote:

> Done.
>
> (Sorry for the delay)
>
>
> On 2/11/20 3:53 PM, Xue LIU wrote:
> > Hi guys,
> >
> > I am Xue, just joined this mailing list. Being new to Kafka developer
> > community, I am very glad to join and any help/guide is greatly
> appreciated!
> >
> > I am working on this JIRA:
> https://issues.apache.org/jira/browse/KAFKA-9440
> > I would need to create a new KIP for this, can the administrators give me
> > the permission to edit wiki?
> >
> > Wiki ID: *xuel1*
> >
> > Thanks!
> > Xue Liu
> >
>
>

Re: [DISCUSS] Apache Kafka 2.5.0 release

[David Arthur]

Thanks, Tu. I've moved KIP-467 out of the release plan.

-David

On Thu, Feb 20, 2020 at 6:00 PM Tu Tran  wrote:

> Hi David,
>
> Thanks for being the release main driver. Since the implementation for the
> last part of KIP-467 wasn't finalized prior to Feb 12th, could you remove
> KIP-467 from the list?
>
> Thanks,
> Tu
>
> On Thu, Feb 20, 2020 at 7:18 AM David Arthur  wrote:
>
> > Randall / Konstantine,
> >
> > Sorry for the late reply. Thanks for the fix and for the update! I see
> this
> > change on the 2.5 branch (@b403c66). Consider this a retroactive approval
> > for this bugfix :)
> >
> > -David
> >
> > On Fri, Feb 14, 2020 at 2:21 PM Konstantine Karantasis <
> > konstant...@confluent.io> wrote:
> >
> > > Hi David,
> > >
> > > I want to confirm what Randall mentions above. The code fixes for
> > > KAFKA-9556 were in place before code freeze on Wed, but we spent a bit
> > more
> > > time hardening the conditions of the integration tests and fixing some
> > > jenkins branch builders to run the test on repeat.
> > >
> > > Best,
> > > Konstantine
> > >
> > >
> > > On Fri, Feb 14, 2020 at 7:42 AM Randall Hauch 
> wrote:
> > >
> > > > Hi, David.
> > > >
> > > > I just filed https://issues.apache.org/jira/browse/KAFKA-9556 that
> > > > identifies two pretty minor issues with the new KIP-558 that adds new
> > > > Connect REST API endpoints to get the list of topics used by a
> > connector.
> > > > The impact is high: the feature cannot be fully disabled, and Connect
> > > does
> > > > not automatically reset the topic set when a connector is deleted.
> > > > https://github.com/apache/kafka/pull/8085 includes the two fixes,
> and
> > > also
> > > > adds more unit and integration tests for this feature. Although I
> just
> > > > created the blocker this AM, Konstantine has actually be working on
> the
> > > fix
> > > > for four days. Risk of merging this PR is low, since a) the new
> > > integration
> > > > tests add significant coverage and we've run the new tests numerous
> > > times,
> > > > and b) the fixes help gate the new feature even more and allow the
> > > feature
> > > > to be completely disabled.
> > > >
> > > > I'd like approve to merge https://github.com/apache/kafka/pull/8085
> > > >
> > > > Thanks!
> > > > Randall
> > > >
> > > > On Mon, Feb 10, 2020 at 11:31 AM David Arthur 
> > wrote:
> > > >
> > > > > Just a friendly reminder that this Wednesday, February 12th, is the
> > > code
> > > > > freeze for the 2.5.0 release. After this time we will only accept
> > > blocker
> > > > > bugs onto the release branch.
> > > > >
> > > > > Thanks!
> > > > > David
> > > > >
> > > > > On Fri, Jan 31, 2020 at 5:13 PM David Arthur 
> > wrote:
> > > > >
> > > > > > Thanks! I've updated the list.
> > > > > >
> > > > > > On Thu, Jan 30, 2020 at 5:48 PM Konstantine Karantasis <
> > > > > > konstant...@confluent.io> wrote:
> > > > > >
> > > > > >> Hi David,
> > > > > >>
> > > > > >> thanks for driving the release.
> > > > > >>
> > > > > >> Please also remove KIP-158 from the list of KIPs that you plan
> to
> > > > > include
> > > > > >> in 2.5
> > > > > >> KIP-158 has been accepted, but the implementation is not yet
> > final.
> > > It
> > > > > >> will be included in the release that follows 2.5.
> > > > > >>
> > > > > >> Regards,
> > > > > >> Konstantine
> > > > > >>
> > > > > >> On 1/30/20, Matthias J. Sax  wrote:
> > > > > >> > Hi David,
> > > > > >> >
> > > > > >> > the following KIP from the list did not make it:
> > > > > >> >
> > > > > >> >  - KIP-216 (no PR yet)
> > > > > >> >  - KIP-399 (no PR yet)
> > > > > >> >  - KIP-401 (PR not merged yet)
> > > > > >> >
> > > > > >> >
> > > > > >> > KIP-444 should be included as we did make progress, but it is
> > > still
> > > > > not
> > > > > >> > fully implement and we need to finish in in 2.6 release.
> > > > > >> >
> > > > > >> > KIP-447 is partially implemented in 2.5 (ie, broker and
> > > > > >> > consumer/producer changes -- the Kafka Streams parts slip)
> > > > > >> >
> > > > > >> >
> > > > > >> > -Matthias
> > > > > >> >
> > > > > >> >
> > > > > >> > On 1/29/20 9:05 AM, David Arthur wrote:
> > > > > >> >> Hey everyone, just a quick update on the 2.5 release.
> > > > > >> >>
> > > > > >> >> I have updated the list of planned KIPs on the release wiki
> > page
> > > > > >> >>
> > > > > >>
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=143428858
> > > > > >> .
> > > > > >> >> If I have missed anything, or there are KIPs included in this
> > > list
> > > > > >> which
> > > > > >> >> should *not* be included in 2.5, please let me know.
> > > > > >> >>
> > > > > >> >> Based on the release schedule, the feature freeze is today,
> Jan
> > > > 29th.
> > > > > >> Any
> > > > > >> >> major feature work that is not already complete will need to
> > push
> > > > out
> > > > > >> to
> > > > > >> >> 2.6. I will work on cutting the release branch during the day
> > > > > tomorrow
> > > > > >> >> (Jan
> > > > > >> >> 

[jira] [Resolved] (KAFKA-9581) Deprecate rebalanceException on StreamThread to avoid infinite loop

[Boyang Chen (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-9581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Boyang Chen resolved KAFKA-9581.

Resolution: Fixed

> Deprecate rebalanceException on StreamThread to avoid infinite loop
> ---
>
> Key: KAFKA-9581
> URL: https://issues.apache.org/jira/browse/KAFKA-9581
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Boyang Chen
>Assignee: Boyang Chen
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

KAFKA-8713 PR review

[Ruslan Gibaiev]

Hey guys,

Can someone please review a PR 
for KAFKA-8713  issue?
It is a quite critical issue and having it fixed soon would be awesome.
Thanks

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

[Satish Duggana]

Hi Jun,
Please look at the earlier reply and let us know your comments.

Thanks,
Satish.

On Wed, Feb 12, 2020 at 4:06 PM Satish Duggana  wrote:
>
> Hi Jun,
> Thanks for your comments on the separation of remote log metadata
> storage and remote log storage.
> We had a few discussions since early Jan on how to support eventually
> consistent stores like S3 by uncoupling remote log segment metadata
> and remote log storage. It is written with details in the doc here(1).
> Below is the brief summary of the discussion from that doc.
>
> The current approach consists of pulling the remote log segment
> metadata from remote log storage APIs. It worked fine for storages
> like HDFS. But one of the problems of relying on the remote storage to
> maintain metadata is that tiered-storage needs to be strongly
> consistent, with an impact not only on the metadata(e.g. LIST in S3)
> but also on the segment data(e.g. GET after a DELETE in S3). The cost
> of maintaining metadata in remote storage needs to be factored in.
> This is true in the case of S3, LIST APIs incur huge costs as you
> raised earlier.
> So, it is good to separate the remote storage from the remote log
> metadata store. We refactored the existing RemoteStorageManager and
> introduced RemoteLogMetadataManager. Remote log metadata store should
> give strong consistency semantics but remote log storage can be
> eventually consistent.
> We can have a default implementation for RemoteLogMetadataManager
> which uses an internal topic(as mentioned in one of our earlier
> emails) as storage. But users can always plugin their own
> RemoteLogMetadataManager implementation based on their environment.
>
> Please go through the updated KIP and let us know your comments. We
> have started refactoring for the changes mentioned in the KIP and
> there may be a few more updates to the APIs.
>
> [1] 
> https://docs.google.com/document/d/1qfkBCWL1e7ZWkHU7brxKDBebq4ie9yK20XJnKbgAlew/edit?ts=5e208ec7#
>
> On Fri, Dec 27, 2019 at 5:43 PM Ivan Yurchenko  
> wrote:
> >
> > Hi all,
> >
> >
> > Jun:
> > > (a) Cost: S3 list object requests cost $0.005 per 1000 requests. If you
> > > have 100,000 partitions and want to pull the metadata for each partition
> > at
> > > the rate of 1/sec. It can cost $0.5/sec, which is roughly $40K per day.
> >
> > I want to note here, that no reasonably durable storage will be cheap
> > at 100k RPS. For example, DynamoDB might give the same ballpark figures.
> > If we want to keep the pull-based approach, we can try to reduce this number
> > in several ways: doing listings less frequently (as Satish mentioned,
> > with the current defaults it's ~3.33k RPS for your example),
> > batching listing operations in some way (depending on the storage;
> > it might require the change of RSM's interface).
> >
> >
> > > There are different ways for doing push based metadata propagation. Some
> > > object stores may support that already. For example, S3 supports events
> > > notification
> > This sounds interesting. However, I see a couple of issues using it:
> >   1. As I understand the documentation, notification delivery is not
> > guaranteed
> > and it's recommended to periodically do LIST to fill the gaps.
> > Which brings us back to the same LIST consistency guarantees issue.
> >   2. The same goes for the broker start: to get the current state, we need
> > to LIST.
> >   3. The dynamic set of multiple consumers (RSMs): AFAIK SQS and SNS aren't
> > designed for such a case.
> >
> >
> > Alexandre:
> > > A.1 As commented on PR 7561, S3 consistency model [1][2] implies RSM
> > cannot
> > > relies solely on S3 APIs to guarantee the expected strong consistency. The
> > > proposed implementation [3] would need to be updated to take this into
> > > account. Let’s talk more about this.
> >
> > Thank you for the feedback. I clearly see the need for changing the S3
> > implementation
> > to provide stronger consistency guarantees. As it see from this thread,
> > there are
> > several possible approaches to this. Let's discuss RemoteLogManager's
> > contract and
> > behavior (like pull vs push model) further before picking one (or several -
> > ?) of them.
> > I'm going to do some evaluation of DynamoDB for the pull-based approach,
> > if it's possible to apply it paying a reasonable bill. Also, of the
> > push-based approach
> > with a Kafka topic as the medium.
> >
> >
> > > A.2.3 Atomicity – what does an implementation of RSM need to provide with
> > > respect to atomicity of the APIs copyLogSegment, cleanupLogUntil and
> > > deleteTopicPartition? If a partial failure happens in any of those (e.g.
> > in
> > > the S3 implementation, if one of the multiple uploads fails [4]),
> >
> > The S3 implementation is going to change, but it's worth clarifying anyway.
> > The segment log file is being uploaded after S3 has acked uploading of
> > all other files associated with the segment and only after this the whole
> > segment file set becomes visible 

Re: [DISCUSS] KIP-508: Make Suppression State Queriable - rebooted.

[John Roesler]

Hi Dongjin,

Ah, I think I may have been confused. I 100% agree that we need a materialized 
variant for suppress(). Then, you could do:
...suppress(..., Materialized.as(“final-count”))

If that’s your proposal, then we are on the same page. 

I was under the impression that you wanted to expand the scope of the KIP to 
additionally allow querying the internal buffer, not just the result. Can you 
clarify whether you are proposing to allow querying the state of the internal 
buffer, the result, or both?

Thanks,
John

On Thu, Feb 20, 2020, at 08:41, Dongjin Lee wrote:
> Hi John,
> Thanks for your kind explanation with an example.
> 
> > But it feels like you're saying you're trying to do something different
> than just query the windowed key and get back the current count?
> 
> Yes, for example, what if we need to retrieve the (all or range) keys with
> a closed window? In this example, let's imagine we need to retrieve only
> (key=A, window=10), not (key=A, window=20).
> 
> Of course, the value accompanied by a flushed key is exactly the same to
> the one in the upstream KTable; However, if our intention is not pointing
> out a specific key but retrieving a group of unspecified keys, we stuck in
> trouble - since we can't be sure which key is flushed out beforehand.
> 
> One workaround would be materializing it with `suppressed.filter(e -> true,
> Materialized.as("final-count"))`. But I think providing a materialized
> variant for suppress method is better than this workaround.
> 
> Thanks,
> Dongjin
> 
> On Thu, Feb 20, 2020 at 1:26 AM John Roesler  wrote:
> 
> > Thanks for the response, Dongjin,
> >
> > I'm sorry, but I'm still not following. It seems like the view you would
> > get on the "current state of the buffer" would always be equivalent to
> > the view of the upstream table.
> >
> > Let me try an example, and maybe you can point out the flaw in my
> > reasoning.
> >
> > Let's say we're doing 10 ms windows with a grace period of zero.
> > Let's also say we're computing a windowed count, and that we have
> > a "final results" suppression after the count. Let's  materialize the
> > count as "Count" and the suppressed result as "Final Count".
> >
> > Suppose we get an input event:
> > (time=10, key=A, value=...)
> >
> > Then, Count will look like:
> >
> > | window | key | value |
> > | 10 | A   | 1 |
> >
> > The (internal) suppression buffer will contain:
> >
> > | window | key | value |
> > | 10 | A   | 1 |
> >
> > The record is still buffered because the window isn't closed yet.
> > Final Count is an empty table:
> >
> > | window | key | value |
> >
> > ---
> >
> > Now, we get a second event:
> > (time=15, key=A, value=...)
> >
> > Then, Count will look like:
> >
> > | window | key | value |
> > | 10 | A   | 2 |
> >
> > The (internal) suppression buffer will contain:
> >
> > | window | key | value |
> > | 10 | A   | 2 |
> >
> > The record is still buffered because the window isn't closed yet.
> > Final Count is an empty table:
> >
> > | window | key | value |
> >
> >
> > ---
> >
> > Finally, we get a third event:
> > (time=20, key=A, value=...)
> >
> > Then, Count will look like:
> >
> > | window | key | value |
> > | 10 | A   | 2 |
> > | 20 | A   | 1 |
> >
> > The (internal) suppression buffer will contain:
> >
> > | window | key | value |
> > | 20 | A   | 1 |
> >
> > Note that window 10 has been flushed out, because it's now closed.
> > And window 20 is buffered because it isn't closed yet.
> > Final Count is now:
> >
> > | window | key | value |
> > | 10 | A   | 2 |
> >
> >
> > ---
> >
> > Reading your email, I can't figure out what value there is in querying the
> > internal suppression buffer, since it only contains exactly the same value
> > as
> > the upstream table, for each key that is still buffered. But it feels like
> > you're saying you're trying to do something different than just query the
> > windowed key and get back the current count?
> >
> > Thanks,
> > -John
> >
> >
> > On Wed, Feb 19, 2020, at 09:49, Dongjin Lee wrote:
> > > Hi John,
> > >
> > > 'The intermediate state of the suppression' in KIP does not mean the
> > state
> > > of upstream KTable - sure, the state of the upstream KTable can be
> > queried
> > > by materializing the operator immediately before the suppress as you
> > shown.
> > > What I meant in KIP was the final state of the buffer, which is not
> > emitted
> > > yet. (I agree, the current description may be confusing; it would be
> > better
> > > to change it with 'the current state of the suppression' or 'the results
> > of
> > > the suppression', like the Jira issue
> > >  states.)
> > >
> > > For a little bit more about the motivation, here is one of my
> > experience: I
> > > had to build a monitoring application which collects signals from IoT
> > > devices (say, a semiconductor production line.) If the 

回复：[Discuss] KIP-571: Add option to force remove members in StreamsResetter

[feyman2009]

Hi, Boyang
Thanks! I have updated the KIP :)
If Sophie also thinks it's ok, I will start a vote soon.

Thanks!
Feyman


--
发件人：Boyang Chen 
发送时间：2020年2月24日(星期一) 00:42
收件人：dev 
主　题：Re: [Discuss] KIP-571: Add option to force remove members in StreamsResetter

Hey Feyman,

thanks a lot for the update, the KIP LGTM now. Will let Sophie take a look
again, also a minor API change:
s/setGroupInstanceId/withGroupInstanceId, and similar to setMemberId, as
usually setters are not expected to return an actual object.

Boyang

On Sat, Feb 22, 2020 at 11:05 PM feyman2009  wrote:

> Hi, Boyang
> Thanks for your review, I have updated the KIP page :)
>
> Hi, Sophie
> Thanks for your suggestions!
> 1)  Did you consider an API that just removes *all* remaining members
> from a group?
> We plan to implement the batch removal in StreamsResetter as below:
> 1) adminClient#describeConsumerGroups to get members in each
> group, this part needs no change.
> 2) adminClient#removeMembersFromConsumerGroup to remove all the
> members got from the above call (This involves API change to support the
> dynamic member removal)
> I think your suggestion is feasible but maybe not necessary currently
> since it is a subset of the combination of the above two APIs. Looking at
> the APIs in KafkaAdminClient, the adminClient.deleteXXX always takes a
> collection as the input parameter and the caller does the "query and
> delete" if "delete all" is needed, this leaves more burden on the caller
> side but increases flexibility. Since the KafkaAdminClient's API is still
> evolving, I think it would be reasonable to follow the convention and not
> adding a "removal all members" API.
>
> 2) Thanks to Boyang's correction, broker version >= 2.4 is needed
> since batch members removal is introduced since then(please check KIP-345
> 
>  for
> details).
> If it is used upon the older clusters like 2.3, 
> *UnsupportedVersionException
> *will be thrown.
>
> Thanks!
> Haoran
>
> --
> 发件人：Boyang Chen 
> 发送时间：2020年2月19日(星期三) 11:57
> 收件人：dev 
> 主 题：Re: [Discuss] KIP-571: Add option to force remove members in
> StreamsResetter
>
> Also Feyman, there is one thing I forget which is that the leave group
> change was introduced in 2.4 broker instead of 2.3. Feel free to correct it
> on the KIP.
>
> On Tue, Feb 18, 2020 at 5:44 PM Sophie Blee-Goldman 
> wrote:
>
> > Hey Feyman,
> >
> > Thanks for the KIP! I had two high-level questions:
> >
>
> > It seems like, in the specific case motivating this KIP, we would only ever
> > want to remove *all* the members remaining in the group (and never just a
> > single member at a time). As you mention there is already an admin API to
>
> > remove static members, but we'd still need something new to handle dynamic
> > ones. Did you consider an API that just removes *all* remaining members
> > from a group, rather than requiring the caller to determine and then
> > specify the
> > group.id (static) or member.id (dynamic) for each one? This way we can
> > just
>
> > have a single API exposed that will handle what we need to do regardless of
> > whether static membership is used or not.
> >
>
> > My other question is, will this new option only work for clusters that are
> > on 2.3
> > or higher? Do you have any thoughts about whether it would be possible to
> > implement this feature for older clusters as well, or are we dependent on
> > changes only introduced in 2.3?
> >
> > If so, we should make it absolutely clear what will happen if this used
> > with
> > an older cluster. That is, will the reset tool exit with a clear error
> > message right
> > away, or will it potentially leave the app in a partially reset state?
> >
> > Thanks!
> > Sophie
> >
> > On Tue, Feb 18, 2020 at 4:30 PM Boyang Chen 
> > wrote:
> >
>
> > > Thanks for the update Feyman. The updates look great, except one thing I
> > > would like to be more specific is error cases display. In the "*2)* Add
>
> > > cmdline option" you mention throwing exception when request failed, does
> > > that suggest partial failure or a full failure? How do we deal with
> > > different scenarios?
> > >
> > > Also some minor syntax fix:
>
> > > 1. it only support remove static members -> it only supports the removal
> > of
> > > static members
>
> > > 2. "new constructor is added and the old constructor will be deprecated"
> > > you mean the `new helper` right? Should be `new helper is added`
> > > 3. users should make sure all the stream applications should be are
> > > shutdown
> > >
> > > Other than the above suggestions, I think the KIP is in pretty good
> > shape.
> > >
> > > Boyang
> > >
> > > On Fri, Feb 14, 2020 at 9:29 PM feyman2009 
> > wrote:
> > >
> > > > 

Re: KAFKA-9308: Request for Review of documentation

[Mickael Maison]

Thanks Sönke,

Your changes look good and greatly improved that section. I've
reviewed your PR and left a few comments

On Mon, Feb 24, 2020 at 9:22 AM Sönke Liebau
 wrote:
>
> Hi everybody,
>
> as there has not been any response to this request I figured maybe
> reviewing html changes in git changelogs is not that comfortable after all.
> So I uploaded a version of the page here:
> https://kafkadocs.liebau.biz/documentation/#security_ssl to make it easier
> to read.
>
> Again, any feedback would be appreciated!
>
> Best regards,
> Sönke
>
>
>
> On Tue, 11 Feb 2020 at 11:48, Sönke Liebau 
> wrote:
>
> > Hi everybody,
> >
> > I've reworked the SSL part of the documentation a little in order to fix
> > (among other things) KAFKA-9308[1] and would love some feedback if someone
> > can spare a few minutes.
> >
> > Pull request: https://github.com/apache/kafka/pull/8009
> >
> > [1] https://issues.apache.org/jira/browse/KAFKA-9308
> >
> > Best regards,
> > Sönke
> >
> >
> >
>
> --
> Sönke Liebau
> Partner
> Tel. +49 179 7940878
> OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

[VOTE] KIP-570: Add leader epoch in StopReplicaRequest

[David Jacot]

Hi all,

I would like to start a vote on KIP-570: Add leader epoch in
StopReplicaRequest

The KIP is here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-570%3A+Add+leader+epoch+in+StopReplicaRequest

Thanks,
David

[jira] [Created] (KAFKA-9599) create unique sensor to record group rebalance

[Chia-Ping Tsai (Jira)]

Chia-Ping Tsai created KAFKA-9599:
-

 Summary: create unique sensor to record group rebalance
 Key: KAFKA-9599
 URL: https://issues.apache.org/jira/browse/KAFKA-9599
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Chia-Ping Tsai
Assignee: Chia-Ping Tsai


{code:scala}
  val offsetDeletionSensor = metrics.sensor("OffsetDeletions")

  ...

  val groupCompletedRebalanceSensor = metrics.sensor("OffsetDeletions")
{code}

the "offset deletion" and "group rebalance" should not be recorded by the same 
sensor since they are totally different.

the code is introduced by KAFKA-8730



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: KAFKA-9308: Request for Review of documentation

[Sönke Liebau]

Hi everybody,

as there has not been any response to this request I figured maybe
reviewing html changes in git changelogs is not that comfortable after all.
So I uploaded a version of the page here:
https://kafkadocs.liebau.biz/documentation/#security_ssl to make it easier
to read.

Again, any feedback would be appreciated!

Best regards,
Sönke



On Tue, 11 Feb 2020 at 11:48, Sönke Liebau 
wrote:

> Hi everybody,
>
> I've reworked the SSL part of the documentation a little in order to fix
> (among other things) KAFKA-9308[1] and would love some feedback if someone
> can spare a few minutes.
>
> Pull request: https://github.com/apache/kafka/pull/8009
>
> [1] https://issues.apache.org/jira/browse/KAFKA-9308
>
> Best regards,
> Sönke
>
>
>

-- 
Sönke Liebau
Partner
Tel. +49 179 7940878
OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany