Jenkins build is back to normal : Kafka » kafka-trunk-jdk11 #191

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » kafka-2.6-jdk8 #44

[Apache Jenkins Server]

See 


Changes:

[github] KAFKA-10515: Properly initialize nullable Serdes with default values 
(#9467)

[A. Sophie Blee-Goldman] MINOR: Fix verification in 
StreamsUpgradeTest.test_version_probing_upgrade (#9530)


--
[...truncated 3.16 MB...]

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardClose 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardFlush 
PASSED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
STARTED

org.apache.kafka.streams.internals.WindowStoreFacadeTest > shouldForwardInit 
PASSED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonUsedOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testEmptyTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testStartTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testNegativeAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
STARTED

org.apache.kafka.streams.TestTopicsTest > shouldNotAllowToCreateWithNullDriver 
PASSED

org.apache.kafka.streams.TestTopicsTest > testDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testValue STARTED

org.apache.kafka.streams.TestTopicsTest > testValue PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestampAutoAdvance PASSED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testOutputWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputTopicWithNullTopicName PASSED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde STARTED

org.apache.kafka.streams.TestTopicsTest > testWrongSerde PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMapWithNull PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingOutputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics STARTED

org.apache.kafka.streams.TestTopicsTest > testMultipleTopics PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueList PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateOutputWithNullDriver PASSED

org.apache.kafka.streams.TestTopicsTest > testValueList STARTED

org.apache.kafka.streams.TestTopicsTest > testValueList PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordList PASSED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic STARTED

org.apache.kafka.streams.TestTopicsTest > testNonExistingInputTopic PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValuesToMap PASSED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList STARTED

org.apache.kafka.streams.TestTopicsTest > testRecordsToList PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValueListDuration PASSED

org.apache.kafka.streams.TestTopicsTest > testInputToString STARTED

org.apache.kafka.streams.TestTopicsTest > testInputToString PASSED

org.apache.kafka.streams.TestTopicsTest > testTimestamp STARTED

org.apache.kafka.streams.TestTopicsTest > testTimestamp PASSED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders STARTED

org.apache.kafka.streams.TestTopicsTest > testWithHeaders PASSED

org.apache.kafka.streams.TestTopicsTest > testKeyValue STARTED

org.apache.kafka.streams.TestTopicsTest > testKeyValue PASSED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName STARTED

org.apache.kafka.streams.TestTopicsTest > 
shouldNotAllowToCreateTopicWithNullTopicName PASSED

> Task :streams:upgrade-system-tests-0101:processTestResources NO-SOURCE
> Task :streams:upgrade-syste

Re: [DISCUSS] Apache Kafka 2.7.0 release

[Sophie Blee-Goldman]

Hey Bill,

We uncovered a bug that can cause Streams to continuously initialize
corrupted offsets from the checkpoint file and fall into an endless cycle
of hitting OffsetOutOfRangeException and reinitializing the task.

This was definitely a regression in 2.7, and the patch is ready already.

PR: https://github.com/apache/kafka/pull/9534



On Thu, Oct 29, 2020 at 11:07 AM Bill Bejeck  wrote:

> Hi Boyang,
>
> I've looked it over and I agree that this needs to go into 2.7.
>
> Thanks,
> Bill
>
> On Thu, Oct 29, 2020 at 1:57 PM Boyang Chen 
> wrote:
>
> > Hey Bill,
> >
> > I'm initiating a revert commit <
> https://github.com/apache/kafka/pull/9529>
> > to 2.7 to remove initial principal/client id from request header, to
> avoid
> > unexpected occupation for tagged fields in request header, since we
> changed
> > the design of KIP-590. Will merge after the build passes, let me know if
> > you have any questions.
> >
> > Best,
> > Boyang
> >
> > On Thu, Oct 29, 2020 at 9:00 AM Bill Bejeck  wrote:
> >
> > > Hi chia-ping,
> > >
> > > That sounds reasonable to me to include them in 2.7.
> > >
> > > Thanks,
> > > Bill
> > >
> > > On Thu, Oct 29, 2020 at 3:03 AM Chia-Ping Tsai 
> > > wrote:
> > >
> > > > hi Bill
> > > >
> > > > We noticed one bug that hw is added to incorrect log (when altering
> > > > replica folder is completing). The bug is not a blocker since it does
> > not
> > > > happen frequently. However, it is a regression of altering replica
> > folder
> > > > at runtime (i.e client producers should be able to keep sending data
> to
> > > > brokers)
> > > >
> > > > There is another small one (
> https://github.com/apache/kafka/pull/9525)
> > > > the async error report (in connector) does not work well (it is a new
> > > > feature in 2.6).
> > > >
> > > > Please take a look if you have free time. I'd hope 2.7.0 can include
> > them
> > > > :)
> > > >
> > > > --
> > > > chia-ping
> > > >
> > > > On 2020/10/27 23:17:59, Bill Bejeck  wrote:
> > > > > Hi Sophie,
> > > > >
> > > > > Thanks for the update.
> > > > >
> > > > > I've read the ticket and I agree this should go in the 2.7 release.
> > > > >
> > > > > -Bill
> > > > >
> > > > > On Tue, Oct 27, 2020 at 7:04 PM Sophie Blee-Goldman <
> > > sop...@confluent.io
> > > > >
> > > > > wrote:
> > > > >
> > > > > > Hey Bill,
> > > > > >
> > > > > > We found a bug that can cause the group to get stuck: KAFKA-10651
> > > > > > 
> > > > > >
> > > > > > While not strictly a blocker, in that it's not a regression in
> 2.7,
> > > it
> > > > > > impacts
> > > > > > potentially all Streams apps and breaks a pretty major feature
> that
> > > was
> > > > > > introduced in 2.6. I'm working on preparing a fix now and would
> > hope
> > > > > > to get this into 2.7.0
> > > > > >
> > > > > > -Sophie
> > > > > >
> > > > > > On Mon, Oct 26, 2020 at 1:48 PM David Jacot  >
> > > > wrote:
> > > > > >
> > > > > > > Hi Bill,
> > > > > > >
> > > > > > > We have found a small regression:
> > > > > > > https://issues.apache.org/jira/browse/KAFKA-10647.
> > > > > > > This was introduced while we migrated the consumer protocol to
> > > using
> > > > the
> > > > > > > auto-generated
> > > > > > > protocol. I have opened a PR to fix it (one line):
> > > > > > > https://github.com/apache/kafka/pull/9506.
> > > > > > >
> > > > > > > Best,
> > > > > > > David
> > > > > > >
> > > > > > > On Mon, Oct 26, 2020 at 4:23 PM Bill Bejeck  >
> > > > wrote:
> > > > > > >
> > > > > > > > Hi David,
> > > > > > > >
> > > > > > > > I agree that these small issues should be included in 2.7.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Bill
> > > > > > > >
> > > > > > > > On Mon, Oct 26, 2020 at 10:58 AM David Jacot <
> > > dja...@confluent.io>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Bill,
> > > > > > > > >
> > > > > > > > > We have found two small issues related to the newly
> > > > > > > > > introduced describeUserScramCredentials API:
> > > > > > > > > 1) https://github.com/apache/kafka/pull/9374
> > > > > > > > > 2) https://github.com/apache/kafka/pull/9504
> > > > > > > > >
> > > > > > > > > While not a regression, I'd like to get them in 2.7 if
> > possible
> > > > to
> > > > > > > avoid
> > > > > > > > > releasing a new API with known
> > > > > > > > > bugs.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > David
> > > > > > > > >
> > > > > > > > > On Thu, Oct 22, 2020 at 8:39 PM Bruno Cadonna <
> > > > br...@confluent.io>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > > > Hi Bill,
> > > > > > > > > >
> > > > > > > > > > I took a second look at the git history and now it
> actually
> > > > seems
> > > > > > to
> > > > > > > be
> > > > > > > > > > a regression. Probably, a change in August that extended
> > the
> > > > error
> > > > > > > > codes
> > > > > > > > > > introduced this bug.
> > > > > > > > > >
> > > > > > > > > > Best,
> > > > > > > > > > Bruno
> > > > > > > > > >
> > > > > > > > > > On 2

Build failed in Jenkins: Kafka » kafka-trunk-jdk15 #216

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Fix verification in 
StreamsUpgradeTest.test_version_probing_upgrade (#9530)

[github] MINOR: improve `null` checks for headers (#9513)


--
[...truncated 3.45 MB...]

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@20db65e5, 
timestamped = false, caching = false, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@3f347356, 
timestamped = false, caching = false, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@3f347356, 
timestamped = false, caching = false, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@6dff28ef, 
timestamped = false, caching = false, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@6dff28ef, 
timestamped = false, caching = false, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@3c3a8134, 
timestamped = false, caching = true, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@3c3a8134, 
timestamped = false, caching = true, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@47b53f1b, 
timestamped = false, caching = true, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@47b53f1b, 
timestamped = false, caching = true, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@5b875a78, 
timestamped = false, caching = true, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@5b875a78, 
timestamped = false, caching = true, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@4f2ba9b7, 
timestamped = false, caching = true, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@4f2ba9b7, 
timestamped = false, caching = true, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@27f0a247, 
timestamped = false, caching = false, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@27f0a247, 
timestamped = false, caching = false, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@49530c45, 
timestamped = false, caching = false, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@49530c45, 
timestamped = false, caching = false, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@28e6db8d, 
timestamped = false, caching = false, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@28e6db8d, 
timestamped = false, caching = false, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionS

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #190

[Apache Jenkins Server]

See 


Changes:

[github] MINOR: Fix verification in 
StreamsUpgradeTest.test_version_probing_upgrade (#9530)


--
[...truncated 3.45 MB...]

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@79000e69, 
timestamped = false, caching = false, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@46eb2c, timestamped 
= false, caching = false, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@46eb2c, timestamped 
= false, caching = false, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@6f39e96b, 
timestamped = false, caching = false, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@6f39e96b, 
timestamped = false, caching = false, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@4cdfed4a, 
timestamped = false, caching = true, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@4cdfed4a, 
timestamped = false, caching = true, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@35d5d1fe, 
timestamped = false, caching = true, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@35d5d1fe, 
timestamped = false, caching = true, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@2fd43e3, 
timestamped = false, caching = true, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@2fd43e3, 
timestamped = false, caching = true, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@22af8066, 
timestamped = false, caching = true, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@22af8066, 
timestamped = false, caching = true, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@11a98cc4, 
timestamped = false, caching = false, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@11a98cc4, 
timestamped = false, caching = false, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@a68e52e, 
timestamped = false, caching = false, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@a68e52e, 
timestamped = false, caching = false, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@62625ddb, 
timestamped = false, caching = false, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@62625ddb, 
timestamped = false, caching = false, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.SessionStoreBuilder@3c905d41, 
timestamped = false, caching = false, loggin

[jira] [Created] (KAFKA-10664) Streams fails to overwrite corrupted offsets leading to infinite OffsetOutOfRangeException loop

[A. Sophie Blee-Goldman (Jira)]

A. Sophie Blee-Goldman created KAFKA-10664:
--

 Summary: Streams fails to overwrite corrupted offsets leading to 
infinite OffsetOutOfRangeException loop
 Key: KAFKA-10664
 URL: https://issues.apache.org/jira/browse/KAFKA-10664
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 2.7.0
Reporter: A. Sophie Blee-Goldman
Assignee: A. Sophie Blee-Goldman
 Fix For: 2.7.0


In KAFKA-10391 we fixed an issue where Streams could get stuck in an infinite 
loop of  OffsetOutOfRangeException/TaskCorruptedException due to 
re-initializing the corrupted offsets from the checkpoint after each revival. 
The fix we applied was to remove the corrupted offsets from the state manager 
and then force it to write a new checkpoint file without those offsets during 
revival.

Unfortunately we missed that there's an optimization in OffsetCheckpoint#write 
to just return without writing anything when there's no offsets. So if a task 
doesn't have any offsets that _aren't_ corrupted, it will skip overwriting the 
corrupted checkpoint.

Probably we should just fix the optimization in OffsetCheckpoint so that it 
deletes the current checkpoint in the case there are no offsets to write



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Jenkins build is back to normal : Kafka » kafka-trunk-jdk8 #181

[Apache Jenkins Server]

See 

Jenkins build is back to normal : Kafka » kafka-trunk-jdk11 #189

[Apache Jenkins Server]

See 

Re: Permission for contribution

[Rohit Deshpande]

I don't know how to create wiki id. I am following this documentation:
https://kafka.apache.org/contributing

On Thu, Oct 29, 2020 at 1:40 PM Boyang Chen 
wrote:

> Added you to the JIRA. What's your id for wiki?
>
> On Thu, Oct 29, 2020 at 1:23 PM rohit deshpande  >
> wrote:
>
> > I forgot to provide details.
> >
> > My email id: rohitmdeshpa...@gmail.com
> > This request is regarding code contribution.
> > Thanks,
> > Rohit
> >
> > On Thu, Oct 29, 2020 at 11:58 AM rohit deshpande <
> > rohitmdeshpa...@gmail.com>
> > wrote:
> >
> > > Hi,
> > > I would like to get permission for the contribution.
> > > My jira id: rohitdeshaws
> > >
> > > Thanks,
> > > Rohit
> > >
> >
>

Re: Permission for contribution

[Boyang Chen]

Added you to the JIRA. What's your id for wiki?

On Thu, Oct 29, 2020 at 1:23 PM rohit deshpande 
wrote:

> I forgot to provide details.
>
> My email id: rohitmdeshpa...@gmail.com
> This request is regarding code contribution.
> Thanks,
> Rohit
>
> On Thu, Oct 29, 2020 at 11:58 AM rohit deshpande <
> rohitmdeshpa...@gmail.com>
> wrote:
>
> > Hi,
> > I would like to get permission for the contribution.
> > My jira id: rohitdeshaws
> >
> > Thanks,
> > Rohit
> >
>

Re: Permission for contribution

[rohit deshpande]

I forgot to provide details.

My email id: rohitmdeshpa...@gmail.com
This request is regarding code contribution.
Thanks,
Rohit

On Thu, Oct 29, 2020 at 11:58 AM rohit deshpande 
wrote:

> Hi,
> I would like to get permission for the contribution.
> My jira id: rohitdeshaws
>
> Thanks,
> Rohit
>

Permission for contribution

[rohit deshpande]

Hi,
I would like to get permission for the contribution.
My jira id: rohitdeshaws

Thanks,
Rohit

[jira] [Created] (KAFKA-10663) Flakey test ConsumerBounceTest#testSeekAndCommitWithBrokerFailures

[Boyang Chen (Jira)]

Boyang Chen created KAFKA-10663:
---

 Summary: Flakey test 
ConsumerBounceTest#testSeekAndCommitWithBrokerFailures
 Key: KAFKA-10663
 URL: https://issues.apache.org/jira/browse/KAFKA-10663
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 2.7.0
Reporter: Boyang Chen


org.apache.kafka.common.KafkaException: Socket server failed to bind to 
localhost:40823: Address already in use. at 
kafka.network.Acceptor.openServerSocket(SocketServer.scala:671) at 
kafka.network.Acceptor.(SocketServer.scala:539) at 
kafka.network.SocketServer.createAcceptor(SocketServer.scala:280) at 
kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1(SocketServer.scala:253)
 at 
kafka.network.SocketServer.$anonfun$createDataPlaneAcceptorsAndProcessors$1$adapted(SocketServer.scala:251)
 at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62) at 
scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55) at 
scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at 
kafka.network.SocketServer.createDataPlaneAcceptorsAndProcessors(SocketServer.scala:251)
 at kafka.network.SocketServer.startup(SocketServer.scala:125) at 
kafka.server.KafkaServer.startup(KafkaServer.scala:303) at 
kafka.utils.TestUtils$.createServer(TestUtils.scala:160) at 
kafka.utils.TestUtils$.createServer(TestUtils.scala:151) at 
kafka.integration.KafkaServerTestHarness.$anonfun$setUp$1(KafkaServerTestHarness.scala:102)
 at scala.collection.Iterator.foreach(Iterator.scala:943) at 
scala.collection.Iterator.foreach$(Iterator.scal

 

>From AK 2.7



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KAFKA-10662) Possible hanging test in 2.6 on JDK 11

[John Roesler (Jira)]

John Roesler created KAFKA-10662:


 Summary: Possible hanging test in 2.6 on JDK 11
 Key: KAFKA-10662
 URL: https://issues.apache.org/jira/browse/KAFKA-10662
 Project: Kafka
  Issue Type: Bug
Affects Versions: 2.6.1
Reporter: John Roesler
 Attachments: timeout-1.txt, timeout-2.txt, timeout-4.txt

While adding a Jenkinsfile to the 2.6 branch 
([https://github.com/apache/kafka/pull/9471),]

I observed the JDK 11 build specifically to hang, 3/5 times (and pass within a 
normal timeframe of 2.5 hours the other two times).

I haven't seen similar behavior on any other branch, so there may be something 
about the 2.6 codebase or the 2.6 tests themselves that interact poorly with 
Java 11.

 

I did some analysis on the failing results, and found that in all three hanging 
cases, all the tests that "STARTED" either "PASSED" or were "SKIPPED". So, I 
was not able to identify a specific culprit. I've attached the logs for these 
runs, in case they aid any investigation.

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Is AdminClient of Kafka thread-safe?

[Colin McCabe]

Yes, KafkaAdminClient is thread-safe.

best,
Colin

On Wed, Oct 7, 2020, at 12:38, Efe Gencer wrote:
> Hi All,
> 
> Other than a Stack Overflow comment (see 
> https://stackoverflow.com/a/61738065) by Colin Patrick McCabe (CC'd), 
> there is no source that verifies the thread-safety of KafkaAdminClient.
>  * In particular, JavaDocs of KafkaAdminClient class and Admin interface have 
> no discussion on thread-safety.
> I would appreciate information on the thread-safety of the AdminClient.
> 
> Best,
> Efe

[jira] [Created] (KAFKA-10661) Add resigned state to raft state machine to preserve leader/epoch information

[Jason Gustafson (Jira)]

Jason Gustafson created KAFKA-10661:
---

 Summary: Add resigned state to raft state machine to preserve 
leader/epoch information
 Key: KAFKA-10661
 URL: https://issues.apache.org/jira/browse/KAFKA-10661
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jason Gustafson
Assignee: Jason Gustafson


While working on KAFKA-10655, I realized we have a bug in the existing raft 
state initialization logic when the process shuts down as leader. After 
reinitializing, we retain the current epoch, but we discard the current leader 
status. This means that it is possible for the node to vote for another node in 
the same epoch that it was the leader of.

To fix this problem I think we should add a separate "resigned" state. When 
re-initializing after being shutdown as leader, we can enter the "resigned" 
state. This prevents us from voting for another candidate while still ensuring 
that a new election needs to be held.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] Apache Kafka 2.7.0 release

[Bill Bejeck]

Hi Boyang,

I've looked it over and I agree that this needs to go into 2.7.

Thanks,
Bill

On Thu, Oct 29, 2020 at 1:57 PM Boyang Chen 
wrote:

> Hey Bill,
>
> I'm initiating a revert commit 
> to 2.7 to remove initial principal/client id from request header, to avoid
> unexpected occupation for tagged fields in request header, since we changed
> the design of KIP-590. Will merge after the build passes, let me know if
> you have any questions.
>
> Best,
> Boyang
>
> On Thu, Oct 29, 2020 at 9:00 AM Bill Bejeck  wrote:
>
> > Hi chia-ping,
> >
> > That sounds reasonable to me to include them in 2.7.
> >
> > Thanks,
> > Bill
> >
> > On Thu, Oct 29, 2020 at 3:03 AM Chia-Ping Tsai 
> > wrote:
> >
> > > hi Bill
> > >
> > > We noticed one bug that hw is added to incorrect log (when altering
> > > replica folder is completing). The bug is not a blocker since it does
> not
> > > happen frequently. However, it is a regression of altering replica
> folder
> > > at runtime (i.e client producers should be able to keep sending data to
> > > brokers)
> > >
> > > There is another small one (https://github.com/apache/kafka/pull/9525)
> > > the async error report (in connector) does not work well (it is a new
> > > feature in 2.6).
> > >
> > > Please take a look if you have free time. I'd hope 2.7.0 can include
> them
> > > :)
> > >
> > > --
> > > chia-ping
> > >
> > > On 2020/10/27 23:17:59, Bill Bejeck  wrote:
> > > > Hi Sophie,
> > > >
> > > > Thanks for the update.
> > > >
> > > > I've read the ticket and I agree this should go in the 2.7 release.
> > > >
> > > > -Bill
> > > >
> > > > On Tue, Oct 27, 2020 at 7:04 PM Sophie Blee-Goldman <
> > sop...@confluent.io
> > > >
> > > > wrote:
> > > >
> > > > > Hey Bill,
> > > > >
> > > > > We found a bug that can cause the group to get stuck: KAFKA-10651
> > > > > 
> > > > >
> > > > > While not strictly a blocker, in that it's not a regression in 2.7,
> > it
> > > > > impacts
> > > > > potentially all Streams apps and breaks a pretty major feature that
> > was
> > > > > introduced in 2.6. I'm working on preparing a fix now and would
> hope
> > > > > to get this into 2.7.0
> > > > >
> > > > > -Sophie
> > > > >
> > > > > On Mon, Oct 26, 2020 at 1:48 PM David Jacot 
> > > wrote:
> > > > >
> > > > > > Hi Bill,
> > > > > >
> > > > > > We have found a small regression:
> > > > > > https://issues.apache.org/jira/browse/KAFKA-10647.
> > > > > > This was introduced while we migrated the consumer protocol to
> > using
> > > the
> > > > > > auto-generated
> > > > > > protocol. I have opened a PR to fix it (one line):
> > > > > > https://github.com/apache/kafka/pull/9506.
> > > > > >
> > > > > > Best,
> > > > > > David
> > > > > >
> > > > > > On Mon, Oct 26, 2020 at 4:23 PM Bill Bejeck 
> > > wrote:
> > > > > >
> > > > > > > Hi David,
> > > > > > >
> > > > > > > I agree that these small issues should be included in 2.7.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Bill
> > > > > > >
> > > > > > > On Mon, Oct 26, 2020 at 10:58 AM David Jacot <
> > dja...@confluent.io>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Bill,
> > > > > > > >
> > > > > > > > We have found two small issues related to the newly
> > > > > > > > introduced describeUserScramCredentials API:
> > > > > > > > 1) https://github.com/apache/kafka/pull/9374
> > > > > > > > 2) https://github.com/apache/kafka/pull/9504
> > > > > > > >
> > > > > > > > While not a regression, I'd like to get them in 2.7 if
> possible
> > > to
> > > > > > avoid
> > > > > > > > releasing a new API with known
> > > > > > > > bugs.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > David
> > > > > > > >
> > > > > > > > On Thu, Oct 22, 2020 at 8:39 PM Bruno Cadonna <
> > > br...@confluent.io>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > > Hi Bill,
> > > > > > > > >
> > > > > > > > > I took a second look at the git history and now it actually
> > > seems
> > > > > to
> > > > > > be
> > > > > > > > > a regression. Probably, a change in August that extended
> the
> > > error
> > > > > > > codes
> > > > > > > > > introduced this bug.
> > > > > > > > >
> > > > > > > > > Best,
> > > > > > > > > Bruno
> > > > > > > > >
> > > > > > > > > On 22.10.20 19:50, Bill Bejeck wrote:
> > > > > > > > > > Hi Bruno,
> > > > > > > > > >
> > > > > > > > > > While technically it's not a regression, I think this is
> an
> > > > > > important
> > > > > > > > fix
> > > > > > > > > > with a low-risk to include, so we can leave it as a
> > blocker.
> > > > > > > > > >
> > > > > > > > > > Thanks,
> > > > > > > > > > Bill
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu, Oct 22, 2020 at 1:25 PM Bruno Cadonna <
> > > > > br...@confluent.io>
> > > > > > > > > wrote:
> > > > > > > > > >
> > > > > > > > > >> Hi Bill,
> > > > > > > > > >>
> > > > > > > > > >> we encountered the following bug in our soak testing
> > > cluster.
> > > > > > > > > >>
> > > > > 

Re: [DISCUSS] Apache Kafka 2.7.0 release

[Boyang Chen]

Hey Bill,

I'm initiating a revert commit 
to 2.7 to remove initial principal/client id from request header, to avoid
unexpected occupation for tagged fields in request header, since we changed
the design of KIP-590. Will merge after the build passes, let me know if
you have any questions.

Best,
Boyang

On Thu, Oct 29, 2020 at 9:00 AM Bill Bejeck  wrote:

> Hi chia-ping,
>
> That sounds reasonable to me to include them in 2.7.
>
> Thanks,
> Bill
>
> On Thu, Oct 29, 2020 at 3:03 AM Chia-Ping Tsai 
> wrote:
>
> > hi Bill
> >
> > We noticed one bug that hw is added to incorrect log (when altering
> > replica folder is completing). The bug is not a blocker since it does not
> > happen frequently. However, it is a regression of altering replica folder
> > at runtime (i.e client producers should be able to keep sending data to
> > brokers)
> >
> > There is another small one (https://github.com/apache/kafka/pull/9525)
> > the async error report (in connector) does not work well (it is a new
> > feature in 2.6).
> >
> > Please take a look if you have free time. I'd hope 2.7.0 can include them
> > :)
> >
> > --
> > chia-ping
> >
> > On 2020/10/27 23:17:59, Bill Bejeck  wrote:
> > > Hi Sophie,
> > >
> > > Thanks for the update.
> > >
> > > I've read the ticket and I agree this should go in the 2.7 release.
> > >
> > > -Bill
> > >
> > > On Tue, Oct 27, 2020 at 7:04 PM Sophie Blee-Goldman <
> sop...@confluent.io
> > >
> > > wrote:
> > >
> > > > Hey Bill,
> > > >
> > > > We found a bug that can cause the group to get stuck: KAFKA-10651
> > > > 
> > > >
> > > > While not strictly a blocker, in that it's not a regression in 2.7,
> it
> > > > impacts
> > > > potentially all Streams apps and breaks a pretty major feature that
> was
> > > > introduced in 2.6. I'm working on preparing a fix now and would hope
> > > > to get this into 2.7.0
> > > >
> > > > -Sophie
> > > >
> > > > On Mon, Oct 26, 2020 at 1:48 PM David Jacot 
> > wrote:
> > > >
> > > > > Hi Bill,
> > > > >
> > > > > We have found a small regression:
> > > > > https://issues.apache.org/jira/browse/KAFKA-10647.
> > > > > This was introduced while we migrated the consumer protocol to
> using
> > the
> > > > > auto-generated
> > > > > protocol. I have opened a PR to fix it (one line):
> > > > > https://github.com/apache/kafka/pull/9506.
> > > > >
> > > > > Best,
> > > > > David
> > > > >
> > > > > On Mon, Oct 26, 2020 at 4:23 PM Bill Bejeck 
> > wrote:
> > > > >
> > > > > > Hi David,
> > > > > >
> > > > > > I agree that these small issues should be included in 2.7.
> > > > > >
> > > > > > Thanks,
> > > > > > Bill
> > > > > >
> > > > > > On Mon, Oct 26, 2020 at 10:58 AM David Jacot <
> dja...@confluent.io>
> > > > > wrote:
> > > > > >
> > > > > > > Hi Bill,
> > > > > > >
> > > > > > > We have found two small issues related to the newly
> > > > > > > introduced describeUserScramCredentials API:
> > > > > > > 1) https://github.com/apache/kafka/pull/9374
> > > > > > > 2) https://github.com/apache/kafka/pull/9504
> > > > > > >
> > > > > > > While not a regression, I'd like to get them in 2.7 if possible
> > to
> > > > > avoid
> > > > > > > releasing a new API with known
> > > > > > > bugs.
> > > > > > >
> > > > > > > Best,
> > > > > > > David
> > > > > > >
> > > > > > > On Thu, Oct 22, 2020 at 8:39 PM Bruno Cadonna <
> > br...@confluent.io>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Hi Bill,
> > > > > > > >
> > > > > > > > I took a second look at the git history and now it actually
> > seems
> > > > to
> > > > > be
> > > > > > > > a regression. Probably, a change in August that extended the
> > error
> > > > > > codes
> > > > > > > > introduced this bug.
> > > > > > > >
> > > > > > > > Best,
> > > > > > > > Bruno
> > > > > > > >
> > > > > > > > On 22.10.20 19:50, Bill Bejeck wrote:
> > > > > > > > > Hi Bruno,
> > > > > > > > >
> > > > > > > > > While technically it's not a regression, I think this is an
> > > > > important
> > > > > > > fix
> > > > > > > > > with a low-risk to include, so we can leave it as a
> blocker.
> > > > > > > > >
> > > > > > > > > Thanks,
> > > > > > > > > Bill
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Oct 22, 2020 at 1:25 PM Bruno Cadonna <
> > > > br...@confluent.io>
> > > > > > > > wrote:
> > > > > > > > >
> > > > > > > > >> Hi Bill,
> > > > > > > > >>
> > > > > > > > >> we encountered the following bug in our soak testing
> > cluster.
> > > > > > > > >>
> > > > > > > > >> https://issues.apache.org/jira/browse/KAFKA-10631
> > > > > > > > >>
> > > > > > > > >> I classified the bug as a blocker because it caused the
> > death
> > > > of a
> > > > > > > > >> stream thread. It does not seem to be a regression,
> though.
> > > > > > > > >>
> > > > > > > > >> I opened a PR to fix the bug here:
> > > > > > > > >>
> > > > > > > > >> https://github.com/apache/kafka/pull/9479
> > > > > > > > >>
> > > > > > > > >> Feel free to downgra

[jira] [Resolved] (KAFKA-10638) QueryableStateIntegrationTest fails due to stricter store checking

[John Roesler (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-10638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

John Roesler resolved KAFKA-10638.
--
Resolution: Fixed

> QueryableStateIntegrationTest fails due to stricter store checking
> --
>
> Key: KAFKA-10638
> URL: https://issues.apache.org/jira/browse/KAFKA-10638
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 2.7.0
>Reporter: John Roesler
>Assignee: John Roesler
>Priority: Major
> Fix For: 2.7.0
>
>
> Observed:
> {code:java}
> org.apache.kafka.streams.errors.InvalidStateStoreException: Cannot get state 
> store source-table because the stream thread is PARTITIONS_ASSIGNED, not 
> RUNNING
>   at 
> org.apache.kafka.streams.state.internals.StreamThreadStateStoreProvider.stores(StreamThreadStateStoreProvider.java:81)
>   at 
> org.apache.kafka.streams.state.internals.WrappingStoreProvider.stores(WrappingStoreProvider.java:50)
>   at 
> org.apache.kafka.streams.state.internals.CompositeReadOnlyKeyValueStore.get(CompositeReadOnlyKeyValueStore.java:52)
>   at 
> org.apache.kafka.streams.integration.StoreQueryIntegrationTest.shouldQuerySpecificActivePartitionStores(StoreQueryIntegrationTest.java:200)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
>   at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>   at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
>   at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:413)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
>   at 
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>   at sun.reflect.GeneratedMethodAccessor23.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:498)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
>   at 
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
>   at 
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
>   at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:119)
>   at sun.reflect.GeneratedMethodAccessor22.invoke

[jira] [Created] (KAFKA-10660) Poll time out logstash

[David (Jira)]

David created KAFKA-10660:
-

 Summary: Poll time out logstash
 Key: KAFKA-10660
 URL: https://issues.apache.org/jira/browse/KAFKA-10660
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 2.2.1
 Environment: Non Production
Reporter: David


I am getting below message (logstash log from kafka input which I believe I 
need increase max.poll.interval.ms (I think the default is 3)

 

This member will leave the group because consumer poll timeout has expired. 
This means the time between subsequent calls to poll() was longer than the 
configured max.poll.interval.ms, which typically implies that the poll loop is 
spending too much time processing messages. You can address this either by 
increasing max.poll.interval.ms or by reducing the maximum size of batches 
returned in poll() with max.poll.records



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] KIP-567: Kafka Cluster Audit (new discussion)

[Viktor Somogyi-Vass]

Hi Tom.

Sorry for the delay.
Answering your points:

> Why is it necessary to introduce this interface to produce the audit trail
> when there is logging that can already record a lot of the same
> information, albeit in less structured form? If logging isn't adequate it
> would be good to explain why not in the Motivation or Rejected
Alternatives
> section. It's worth pointing out that even the "less structured" part
would
> be helped by KIP-673, which proposes to change the RequestChannel's
logging
> to include a JSON representation of the request.

We will need authorization details as would an auditor normally have them
but a request logger doesn't as you correctly pointed out later in your
reply. They would also appear at different lifecycle points I imagine, like
the request logger is probably when the request enters Kafka and the
auditor catches them before sending the response, so it can obtain all
information (authorization, execution).
Furthermore this auditing API would specifically target other JVM based
components that depend on Kafka (like Ranger or Atlas) and from both side's
perspective it's much better to expose Java level classes rather than a
lower level (JSON) implementation. If a Java level object is exposed then
we need to create them once during request processing which is fairly
low-fat since we're parsing the request most of the time anyways as opposed
to JSON which would need to be serialized first and then deserialized for
the consumer of the API.

> I'm guessing what you gain from the proposed interface is the fact that
> it's called after the authorizer (perhaps after the request has been
> handled: I'm unclear about the purpose of AuditInfo.error), so you could
> generate a single record in the audit trail. That could still be achieved
> using logging, either by correlating existing log messages or by proposing
> some new logging just for this auditing purpose (perhaps with a logger per
> API key so people could avoid the performance hit on the produce and fetch
> paths if they weren't interested in auditing those things). Again, if this
> doesn't work it would be great for the KIP to explain why.

AuditInfo.error serves for capturing the possible errors that could happen
during the authorization and execution of the request. For instance a
partition creation request could be authorized and then rejected
with INVALID_TOPIC_EXCEPTION because the topic is queued for deletion. In
this case the AuditInfo.error would contain this API error thus emitting
information about the failure of the request. With normal auditing that
looks at only the authorization information we wouldn't detect it.
Regarding the produce and fetch performance: for these kinds of requests I
don't think we should enable parsing the batches themselves, just only pass
some meta information like which topics and partitions are affected. These
are parsed anyways for log reading and writing. Also similarly to the
authorizer we need to require implementations to run the auditing logic on
a different thread to minimize the performance impact.

> To me there were parallels with previous discussions about broker-side
> interceptors (
> https://www.mail-archive.com/dev@kafka.apache.org/msg103310.html if you've
> not seen it before), those seemed to founder on the unwillingness to make
> the request internal classes into a supported API. You've tried to address
> this by proposing a parallel set of classes implementing AuditEvent for
> exposing selective details about the request. It's not really clear that
> you really _need_ to access all that information about each request,
rather
> than simply recording it all, and it would also come with a significant
> implementation and maintenance cost. If it's simply about recording all
the
> information in the request, then it would likely be enough to pass a
> suitably formatted String rather than an AuditEvent, which basically
brings
> us back to point 1, but with some justification for not using logging.

Thanks for this email thread, I haven't seen it but now I see it's a much
bigger tree that I'm chopping :). But the point is that everyone basically
faces a similar issue, that we need server side interceptors and
observables. Indeed auditing can be part of such an interceptor chain and
I've been thinking of it like this too sometimes but as it has been
correctly assessed in the thread "we're doing the one-offs". I also admit
that maintaining all the implementation of AuditEvent could be cumbersome
and maybe this isn't a way. However I think we should expose more
structured forms. If we maintain suitably formatted Strings and if the
protocol changes for some requests it could be much harder to trace the
needed changes back to these Strings.
One idea that I had while reading the email is we could generate these
classes similarly to the *Data classes (like CreateTopicsData). There would
be a flag called "useForAuditing=true" in the JSON definition of the
protocol that would 

Re: [DISCUSS] Apache Kafka 2.7.0 release

[Bill Bejeck]

Hi chia-ping,

That sounds reasonable to me to include them in 2.7.

Thanks,
Bill

On Thu, Oct 29, 2020 at 3:03 AM Chia-Ping Tsai  wrote:

> hi Bill
>
> We noticed one bug that hw is added to incorrect log (when altering
> replica folder is completing). The bug is not a blocker since it does not
> happen frequently. However, it is a regression of altering replica folder
> at runtime (i.e client producers should be able to keep sending data to
> brokers)
>
> There is another small one (https://github.com/apache/kafka/pull/9525)
> the async error report (in connector) does not work well (it is a new
> feature in 2.6).
>
> Please take a look if you have free time. I'd hope 2.7.0 can include them
> :)
>
> --
> chia-ping
>
> On 2020/10/27 23:17:59, Bill Bejeck  wrote:
> > Hi Sophie,
> >
> > Thanks for the update.
> >
> > I've read the ticket and I agree this should go in the 2.7 release.
> >
> > -Bill
> >
> > On Tue, Oct 27, 2020 at 7:04 PM Sophie Blee-Goldman  >
> > wrote:
> >
> > > Hey Bill,
> > >
> > > We found a bug that can cause the group to get stuck: KAFKA-10651
> > > 
> > >
> > > While not strictly a blocker, in that it's not a regression in 2.7, it
> > > impacts
> > > potentially all Streams apps and breaks a pretty major feature that was
> > > introduced in 2.6. I'm working on preparing a fix now and would hope
> > > to get this into 2.7.0
> > >
> > > -Sophie
> > >
> > > On Mon, Oct 26, 2020 at 1:48 PM David Jacot 
> wrote:
> > >
> > > > Hi Bill,
> > > >
> > > > We have found a small regression:
> > > > https://issues.apache.org/jira/browse/KAFKA-10647.
> > > > This was introduced while we migrated the consumer protocol to using
> the
> > > > auto-generated
> > > > protocol. I have opened a PR to fix it (one line):
> > > > https://github.com/apache/kafka/pull/9506.
> > > >
> > > > Best,
> > > > David
> > > >
> > > > On Mon, Oct 26, 2020 at 4:23 PM Bill Bejeck 
> wrote:
> > > >
> > > > > Hi David,
> > > > >
> > > > > I agree that these small issues should be included in 2.7.
> > > > >
> > > > > Thanks,
> > > > > Bill
> > > > >
> > > > > On Mon, Oct 26, 2020 at 10:58 AM David Jacot 
> > > > wrote:
> > > > >
> > > > > > Hi Bill,
> > > > > >
> > > > > > We have found two small issues related to the newly
> > > > > > introduced describeUserScramCredentials API:
> > > > > > 1) https://github.com/apache/kafka/pull/9374
> > > > > > 2) https://github.com/apache/kafka/pull/9504
> > > > > >
> > > > > > While not a regression, I'd like to get them in 2.7 if possible
> to
> > > > avoid
> > > > > > releasing a new API with known
> > > > > > bugs.
> > > > > >
> > > > > > Best,
> > > > > > David
> > > > > >
> > > > > > On Thu, Oct 22, 2020 at 8:39 PM Bruno Cadonna <
> br...@confluent.io>
> > > > > wrote:
> > > > > >
> > > > > > > Hi Bill,
> > > > > > >
> > > > > > > I took a second look at the git history and now it actually
> seems
> > > to
> > > > be
> > > > > > > a regression. Probably, a change in August that extended the
> error
> > > > > codes
> > > > > > > introduced this bug.
> > > > > > >
> > > > > > > Best,
> > > > > > > Bruno
> > > > > > >
> > > > > > > On 22.10.20 19:50, Bill Bejeck wrote:
> > > > > > > > Hi Bruno,
> > > > > > > >
> > > > > > > > While technically it's not a regression, I think this is an
> > > > important
> > > > > > fix
> > > > > > > > with a low-risk to include, so we can leave it as a blocker.
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Bill
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Oct 22, 2020 at 1:25 PM Bruno Cadonna <
> > > br...@confluent.io>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi Bill,
> > > > > > > >>
> > > > > > > >> we encountered the following bug in our soak testing
> cluster.
> > > > > > > >>
> > > > > > > >> https://issues.apache.org/jira/browse/KAFKA-10631
> > > > > > > >>
> > > > > > > >> I classified the bug as a blocker because it caused the
> death
> > > of a
> > > > > > > >> stream thread. It does not seem to be a regression, though.
> > > > > > > >>
> > > > > > > >> I opened a PR to fix the bug here:
> > > > > > > >>
> > > > > > > >> https://github.com/apache/kafka/pull/9479
> > > > > > > >>
> > > > > > > >> Feel free to downgrade the priority to "Major" if you think
> it
> > > is
> > > > > not
> > > > > > a
> > > > > > > >> blocker.
> > > > > > > >>
> > > > > > > >> Best,
> > > > > > > >> Bruno
> > > > > > > >>
> > > > > > > >> On 22.10.20 17:49, Bill Bejeck wrote:
> > > > > > > >>> Hi All,
> > > > > > > >>>
> > > > > > > >>> We've hit code freeze.  The current status for cutting an
> RC is
> > > > > there
> > > > > > > is
> > > > > > > >>> one blocker issue.  It looks like there is a fix in the
> works,
> > > so
> > > > > > > >>> hopefully, it will get merged early next week.
> > > > > > > >>>
> > > > > > > >>> At that point, if there are no other blockers, I proceed
> with
> > > the
> > > > > RC
> > > > > > > >>> process.
> > > > > > > >>>
> > > > > > > >>> Thanks,
> 

Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema

[Lucas Bradstreet]

Hi David,

That is a good point that we should clarify. I think we should not commit
to guaranteeing the names or the format of the particular request and
response schemas themselves, though we should guarantee that they are
parseable as JSON. The pre-existing trace logging did not guarantee this
either and it was more difficult to parse as it was not well structured. I
believe for most structured logging use cases where we are analyzing
cluster behavior this should be sufficient.

Thanks,

Lucas

On Thu, Oct 29, 2020 at 3:47 AM David Jacot  wrote:

> Hi folks,
>
> I have looked at Anastasia's PR to implement this KIP and I was wondering
> how far we want to go with the backward compatibility of this in the
> future.
> Now that we rely on the auto-generated protocol, the outputted requests and
> responses use the name of the fields defined in the schema. Until now, we
> have been renaming fields rather easily in the schemas as they were purely
> internal. With this KIP, renaming a field will break the structured request
> log.
>
> Does this imply that we will now consider the schemas as part of our public
> API? We don't discuss this point in the KIP so it is subject to
> interpretation.
>
> I think that we should be clear on that point and either commit to only
> making
> the request log parsable while not guaranteeing the format of the requests
> and
> the responses; or commit to making the schemas part of our public API.
>
> Personally, I lean towards the former at the moment. That is
> probably sufficient
> for the targeted use cases. Breaking changes are annoying but as this is
> intended to be used for debugging purposes, that may be OK.
>
> What do you guys think?
>
> Best,
> David
>
>
> On Thu, Oct 8, 2020 at 7:19 PM Anastasia Vela  wrote:
>
> > Thanks everyone for the votes. In summary, the vote passed and the KIP
> was
> > accepted with 3 binding and 3 non-binding +1s.
> >
> > Best,
> > Anastasia
> >
> > On Mon, Oct 5, 2020 at 2:50 PM Jason Gustafson 
> wrote:
> >
> > > +1 Thanks for the KIP!
> > >
> > > On Fri, Oct 2, 2020 at 4:37 PM Colin McCabe 
> wrote:
> > >
> > > > Thanks, Anastasia!  This will be a lot easier to maintain.
> > > >
> > > > +1 (binding)
> > > >
> > > > best,
> > > > Colin
> > > >
> > > > On Wed, Sep 30, 2020, at 23:57, David Jacot wrote:
> > > > > Thanks for the KIP, Anastasia.
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > > On Thu, Oct 1, 2020 at 8:06 AM Tom Bentley 
> > > wrote:
> > > > >
> > > > > > Thanks Anastasia,
> > > > > >
> > > > > > +1 (non-binding)
> > > > > >
> > > > > >
> > > > > > On Thu, Oct 1, 2020 at 6:30 AM Gwen Shapira 
> > > wrote:
> > > > > >
> > > > > > > Thank you, this will be quite helpful.
> > > > > > >
> > > > > > > +1 (binding)
> > > > > > >
> > > > > > > On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela <
> > > av...@confluent.io>
> > > > > > > wrote:
> > > > > > > >
> > > > > > > > Hi everyone,
> > > > > > > >
> > > > > > > > Thanks again for the discussion. I'd like to start the vote
> for
> > > > this
> > > > > > KIP.
> > > > > > > >
> > > > > > >
> > > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema
> > > > > > > >
> > > > > > > > Thanks,
> > > > > > > > Anastasia
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Gwen Shapira
> > > > > > > Engineering Manager | Confluent
> > > > > > > 650.450.2760 | @gwenshap
> > > > > > > Follow us: Twitter | blog
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Jenkins build is back to normal : Kafka » kafka-trunk-jdk15 #214

[Apache Jenkins Server]

See 

Build failed in Jenkins: Kafka » kafka-trunk-jdk11 #188

[Apache Jenkins Server]

See 


Changes:

[github] Revert "KAFKA-9705 part 1: add KIP-590 request header fields (#9144)" 
(#9523)


--
[...truncated 6.90 MB...]

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.TimestampedWindowStoreBuilder@52b23835,
 timestamped = true, caching = true, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.TimestampedWindowStoreBuilder@73934268,
 timestamped = true, caching = true, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.TimestampedWindowStoreBuilder@73934268,
 timestamped = true, caching = true, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.TimestampedWindowStoreBuilder@690e467f,
 timestamped = true, caching = false, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.TimestampedWindowStoreBuilder@690e467f,
 timestamped = true, caching = false, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.TimestampedWindowStoreBuilder@5aeda1ce,
 timestamped = true, caching = false, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.TimestampedWindowStoreBuilder@5aeda1ce,
 timestamped = true, caching = false, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.TimestampedWindowStoreBuilder@195d4936,
 timestamped = true, caching = false, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.TimestampedWindowStoreBuilder@195d4936,
 timestamped = true, caching = false, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.TimestampedWindowStoreBuilder@7034d877,
 timestamped = true, caching = false, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.TimestampedWindowStoreBuilder@7034d877,
 timestamped = true, caching = false, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.TimestampedWindowStoreBuilder@6b9255b0,
 timestamped = true, caching = false, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.TimestampedWindowStoreBuilder@6b9255b0,
 timestamped = true, caching = false, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.TimestampedWindowStoreBuilder@5c467add,
 timestamped = true, caching = false, logging = false] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.TimestampedWindowStoreBuilder@5c467add,
 timestamped = true, caching = false, logging = false] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@2f218c9a, 
timestamped = false, caching = true, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@2f218c9a, 
timestamped = false, caching = true, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@79000e69, 
timestamped = false, caching = true, logging = true] STARTED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherInitOrThrow[builder = 
org.apache.kafka.streams.state.internals.WindowStoreBuilder@79000e69, 
timestamped = false, caching = true, logging = true] PASSED

org.apache.kafka.streams.test.MockProcessorContextStateStoreTest > 
shouldEitherI

Build failed in Jenkins: Kafka » kafka-trunk-jdk8 #180

[Apache Jenkins Server]

See 


Changes:

[github] Revert "KAFKA-9705 part 1: add KIP-590 request header fields (#9144)" 
(#9523)


--
[...truncated 6.84 MB...]
org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowPatternNotValidForTopicNameException[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldEnqueueLaterOutputsAfterEarlierOnes[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldEnqueueLaterOutputsAfterEarlierOnes[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializersDeprecated[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateIfEvenTimeAdvances[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowNoSuchElementExceptionForUnusedOutputTopicWithDynamicRouting[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowNoSuchElementExceptionForUnusedOutputTopicWithDynamicRouting[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldInitProcessor[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowForUnknownTopic[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldPunctuateOnStreamsTime[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldCaptureGlobalTopicNameIfWrittenInto[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfInMemoryBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldProcessFromSourcesThatMatchMultiplePattern[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldPopulateGlobalStore[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldThrowIfPersistentBuiltInStoreIsAccessedWithUntypedMethod[Eos enabled = 
false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldAllowPrePopulatingStatesStoresWithCachingEnabled[Eos enabled = false] 
PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldReturnCorrectPersistentStoreTypeOnly[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldRespectTaskIdling[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldUseSourceSpecificDeserializers[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > shouldReturnAllStores[Eos 
enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldNotCreateStateDirectoryForStatelessTopology[Eos enabled = false] PASSED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpdatesCorrectlyInRecursiveTopologies[Eos enabled = false] 
STARTED

org.apache.kafka.streams.TopologyTestDriverTest > 
shouldApplyGlobalUpd

Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema

[David Jacot]

Hi folks,

I have looked at Anastasia's PR to implement this KIP and I was wondering
how far we want to go with the backward compatibility of this in the future.
Now that we rely on the auto-generated protocol, the outputted requests and
responses use the name of the fields defined in the schema. Until now, we
have been renaming fields rather easily in the schemas as they were purely
internal. With this KIP, renaming a field will break the structured request
log.

Does this imply that we will now consider the schemas as part of our public
API? We don't discuss this point in the KIP so it is subject to
interpretation.

I think that we should be clear on that point and either commit to only
making
the request log parsable while not guaranteeing the format of the requests
and
the responses; or commit to making the schemas part of our public API.

Personally, I lean towards the former at the moment. That is
probably sufficient
for the targeted use cases. Breaking changes are annoying but as this is
intended to be used for debugging purposes, that may be OK.

What do you guys think?

Best,
David


On Thu, Oct 8, 2020 at 7:19 PM Anastasia Vela  wrote:

> Thanks everyone for the votes. In summary, the vote passed and the KIP was
> accepted with 3 binding and 3 non-binding +1s.
>
> Best,
> Anastasia
>
> On Mon, Oct 5, 2020 at 2:50 PM Jason Gustafson  wrote:
>
> > +1 Thanks for the KIP!
> >
> > On Fri, Oct 2, 2020 at 4:37 PM Colin McCabe  wrote:
> >
> > > Thanks, Anastasia!  This will be a lot easier to maintain.
> > >
> > > +1 (binding)
> > >
> > > best,
> > > Colin
> > >
> > > On Wed, Sep 30, 2020, at 23:57, David Jacot wrote:
> > > > Thanks for the KIP, Anastasia.
> > > >
> > > > +1 (non-binding)
> > > >
> > > > On Thu, Oct 1, 2020 at 8:06 AM Tom Bentley 
> > wrote:
> > > >
> > > > > Thanks Anastasia,
> > > > >
> > > > > +1 (non-binding)
> > > > >
> > > > >
> > > > > On Thu, Oct 1, 2020 at 6:30 AM Gwen Shapira 
> > wrote:
> > > > >
> > > > > > Thank you, this will be quite helpful.
> > > > > >
> > > > > > +1 (binding)
> > > > > >
> > > > > > On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela <
> > av...@confluent.io>
> > > > > > wrote:
> > > > > > >
> > > > > > > Hi everyone,
> > > > > > >
> > > > > > > Thanks again for the discussion. I'd like to start the vote for
> > > this
> > > > > KIP.
> > > > > > >
> > > > > >
> > > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Anastasia
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Gwen Shapira
> > > > > > Engineering Manager | Confluent
> > > > > > 650.450.2760 | @gwenshap
> > > > > > Follow us: Twitter | blog
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Re: [DISCUSS] Apache Kafka 2.7.0 release

[Chia-Ping Tsai]

hi Bill

We noticed one bug that hw is added to incorrect log (when altering replica 
folder is completing). The bug is not a blocker since it does not happen 
frequently. However, it is a regression of altering replica folder at runtime 
(i.e client producers should be able to keep sending data to brokers)

There is another small one (https://github.com/apache/kafka/pull/9525) the 
async error report (in connector) does not work well (it is a new feature in 
2.6).

Please take a look if you have free time. I'd hope 2.7.0 can include them :)

--
chia-ping

On 2020/10/27 23:17:59, Bill Bejeck  wrote: 
> Hi Sophie,
> 
> Thanks for the update.
> 
> I've read the ticket and I agree this should go in the 2.7 release.
> 
> -Bill
> 
> On Tue, Oct 27, 2020 at 7:04 PM Sophie Blee-Goldman 
> wrote:
> 
> > Hey Bill,
> >
> > We found a bug that can cause the group to get stuck: KAFKA-10651
> > 
> >
> > While not strictly a blocker, in that it's not a regression in 2.7, it
> > impacts
> > potentially all Streams apps and breaks a pretty major feature that was
> > introduced in 2.6. I'm working on preparing a fix now and would hope
> > to get this into 2.7.0
> >
> > -Sophie
> >
> > On Mon, Oct 26, 2020 at 1:48 PM David Jacot  wrote:
> >
> > > Hi Bill,
> > >
> > > We have found a small regression:
> > > https://issues.apache.org/jira/browse/KAFKA-10647.
> > > This was introduced while we migrated the consumer protocol to using the
> > > auto-generated
> > > protocol. I have opened a PR to fix it (one line):
> > > https://github.com/apache/kafka/pull/9506.
> > >
> > > Best,
> > > David
> > >
> > > On Mon, Oct 26, 2020 at 4:23 PM Bill Bejeck  wrote:
> > >
> > > > Hi David,
> > > >
> > > > I agree that these small issues should be included in 2.7.
> > > >
> > > > Thanks,
> > > > Bill
> > > >
> > > > On Mon, Oct 26, 2020 at 10:58 AM David Jacot 
> > > wrote:
> > > >
> > > > > Hi Bill,
> > > > >
> > > > > We have found two small issues related to the newly
> > > > > introduced describeUserScramCredentials API:
> > > > > 1) https://github.com/apache/kafka/pull/9374
> > > > > 2) https://github.com/apache/kafka/pull/9504
> > > > >
> > > > > While not a regression, I'd like to get them in 2.7 if possible to
> > > avoid
> > > > > releasing a new API with known
> > > > > bugs.
> > > > >
> > > > > Best,
> > > > > David
> > > > >
> > > > > On Thu, Oct 22, 2020 at 8:39 PM Bruno Cadonna 
> > > > wrote:
> > > > >
> > > > > > Hi Bill,
> > > > > >
> > > > > > I took a second look at the git history and now it actually seems
> > to
> > > be
> > > > > > a regression. Probably, a change in August that extended the error
> > > > codes
> > > > > > introduced this bug.
> > > > > >
> > > > > > Best,
> > > > > > Bruno
> > > > > >
> > > > > > On 22.10.20 19:50, Bill Bejeck wrote:
> > > > > > > Hi Bruno,
> > > > > > >
> > > > > > > While technically it's not a regression, I think this is an
> > > important
> > > > > fix
> > > > > > > with a low-risk to include, so we can leave it as a blocker.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Bill
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Oct 22, 2020 at 1:25 PM Bruno Cadonna <
> > br...@confluent.io>
> > > > > > wrote:
> > > > > > >
> > > > > > >> Hi Bill,
> > > > > > >>
> > > > > > >> we encountered the following bug in our soak testing cluster.
> > > > > > >>
> > > > > > >> https://issues.apache.org/jira/browse/KAFKA-10631
> > > > > > >>
> > > > > > >> I classified the bug as a blocker because it caused the death
> > of a
> > > > > > >> stream thread. It does not seem to be a regression, though.
> > > > > > >>
> > > > > > >> I opened a PR to fix the bug here:
> > > > > > >>
> > > > > > >> https://github.com/apache/kafka/pull/9479
> > > > > > >>
> > > > > > >> Feel free to downgrade the priority to "Major" if you think it
> > is
> > > > not
> > > > > a
> > > > > > >> blocker.
> > > > > > >>
> > > > > > >> Best,
> > > > > > >> Bruno
> > > > > > >>
> > > > > > >> On 22.10.20 17:49, Bill Bejeck wrote:
> > > > > > >>> Hi All,
> > > > > > >>>
> > > > > > >>> We've hit code freeze.  The current status for cutting an RC is
> > > > there
> > > > > > is
> > > > > > >>> one blocker issue.  It looks like there is a fix in the works,
> > so
> > > > > > >>> hopefully, it will get merged early next week.
> > > > > > >>>
> > > > > > >>> At that point, if there are no other blockers, I proceed with
> > the
> > > > RC
> > > > > > >>> process.
> > > > > > >>>
> > > > > > >>> Thanks,
> > > > > > >>> Bill
> > > > > > >>>
> > > > > > >>> On Wed, Oct 7, 2020 at 12:10 PM Bill Bejeck  > >
> > > > > wrote:
> > > > > > >>>
> > > > > >  Hi Anna,
> > > > > > 
> > > > > >  I've updated the table to only show KAFKA-10023 as going into
> > > 2.7
> > > > > > 
> > > > > >  Thanks,
> > > > > >  Bill
> > > > > > 
> > > > > >  On Tue, Oct 6, 2020 at 6:51 PM Anna Povzner <
> > a...@confluent.io>
> > > > > > wrote:
> > > > > > 
> > > > > >