Jenkins build is back to normal : kafka-2.1-jdk8 #230

[Apache Jenkins Server]

See 

Re: [DISCUSS] KIP-479: Add Materialized to Join

[John Roesler]

Hi Bill,

For the record, the current proposal looks good to me also.

Thanks,
-John

On Tue, Sep 17, 2019 at 5:06 PM Matthias J. Sax  wrote:
>
> > Just to clarify I'll update `as(StoreSupplier, StoreSupplier)` to
> > `with(..., ...)` and change the class name from `StreamJoined` to
> > `StreamJoin`
>
>
> Thanks Bill. SGTM.
>
>
>
> -Matthias
>
>
> On 9/17/19 4:52 PM, aishwarya kumar wrote:
> > Hi Bill,
> >
> > Thanks for clarifying, and the KIP looks great!!
> >
> > Best regards,
> > Aishwarya
> >
> > On Tue, Sep 17, 2019, 6:16 PM Bill Bejeck  wrote:
> >
> >> Hi Aishwarya,
> >>
> >> On Tue, Sep 17, 2019 at 1:41 PM aishwarya kumar 
> >> wrote:
> >>
> >>> Will this be applicable to Kstream-Ktable joins as well? Or do users
> >> always
> >>> materialize these joins explicitly?
> >>>
> >>
> >> No, this change applies to KStream-KStream joins only.  With KStream-KTable
> >> joins KafkaStreams has materialized the KTable already, and we don't need
> >> to do anything with the KStream instance in this case.
> >>
> >>
> >>> I'm not sure if its even necessary (or makes sense), just trying to
> >>> understand why the change is applicable to Kstream joins only?
> >>>
> >>
> >> The full details are in the KIP, but in a nutshell, we needed to break the
> >> naming of the StateStore from `Joined.withName` and provide users a way to
> >> name the StateStore explicitly.  While making the changes, we realized it
> >> would be beneficial to give users the ability to use different WindowStore
> >> implementations.  The most likely reason to use different WindowStores
> >> would be to enable in-memory joins.
> >>
> >>
> >>> Best,
> >>> Aishwarya
> >>>
> >>
> >> Regards,
> >> Bill
> >>
> >>
> >>>
> >>> On Tue, Sep 17, 2019 at 4:05 PM Bill Bejeck  wrote:
> >>>
>  Guozhang,
> 
>  Thanks for the comments.
> 
>  Yes, you are correct in your assessment regarding names, *if* the users
>  provide their own StoreSuppliers  When constructing as StoreSupplier,
> >> the
>  name can't be null, and additionally, there is no way to update the
> >> name.
>  Further downstream, the underlying StateStore instances use the
> >> provided
>  name for registration so they must be unique.  If users don't provide
>  distinct names for the StoreSuppliers, KafkaStreams will thow a
>  StreamsException when building the topology.
> 
>  Thanks,
>  Bill
> 
> 
> 
>  On Tue, Sep 17, 2019 at 9:39 AM Guozhang Wang 
> >>> wrote:
> 
> > Hello Bill,
> >
> > Thanks for the updated KIP. I made a pass on the StreamJoined
> >> section.
>  Just
> > a quick question from user's perspective: if a user wants to provide
> >> a
> > customized store-supplier, she is forced to also provide a name via
> >> the
> > store-supplier. So there's no way to say "I want to provide my own
> >>> store
> > engine but let the library decide its name", is that right?
> >
> >
> > Guozhang
> >
> >
> > On Tue, Sep 17, 2019 at 8:53 AM Bill Bejeck 
> >> wrote:
> >
> >> Bumping this discussion as we need to re-vote before the KIP
> >>> deadline.
> >>
> >> On Fri, Sep 13, 2019 at 10:29 AM Bill Bejeck 
>  wrote:
> >>
> >>> Hi All,
> >>>
> >>> While working on the implementation of KIP-479, some issues came
> >> to
> > light
> >>> that the KIP as written won't work.  I have updated the KIP with
> >> a
> >> solution
> >>> I believe will solve the original problem as well as address the
> >>> impediment to the initial approach.
> >>>
> >>> This update is a significant change, so please review the updated
> >>> KIP
> >>>
> >>
> >
> 
> >>>
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-479%3A+Add+StreamJoined+config+object+to+Join
> >> and
> >>> provide feedback.  After we conclude the discussion, there will
> >> be
> >>> a
> >>> re-vote.
> >>>
> >>> Thanks!
> >>> Bill
> >>>
> >>> On Wed, Jul 17, 2019 at 7:01 PM Guozhang Wang <
> >> wangg...@gmail.com>
> >> wrote:
> >>>
>  Hi Bill, thanks for your explanations. I'm on board with your
>  decision
>  too.
> 
> 
>  Guozhang
> 
>  On Wed, Jul 17, 2019 at 10:20 AM Bill Bejeck  >>>
> > wrote:
> 
> > Thanks for the response, John.
> >
> >> If I can offer my thoughts, it seems better to just document
> >>> on
> > the
> >> Stream join javadoc for the Materialized parameter that it
> >>> will
> > not
> >> make the join result queriable. I'm not opposed to the
> >>> queriable
> >> flag
> >> in general, but introducing it is a much larger
> >> consideration
>  that
> >> has
> >> previously derailed this KIP discussion. In the interest of
> >>> just
> >> closing the gap and keeping the API change small, it seems
>  better
> > to
> >> j

Build failed in Jenkins: kafka-trunk-jdk8 #3907

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: Default to 5 partitions of the __consumer_offsets topic in

[wangguoz] KAFKA-8839 : Improve streams debug logging (#7258)

--
[...truncated 8.71 MB...]
org.apache.kafka.streams.StreamsConfigTest > 
shouldAcceptBuiltInMetricsLatestVersion PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfValueSerdeConfigFails STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowStreamsExceptionIfValueSerdeConfigFails PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowIfBuiltInMetricsVersionInvalid STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowIfBuiltInMetricsVersionInvalid PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfConsumerAutoCommitIsOverridden STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldResetToDefaultIfConsumerAutoCommitIsOverridden PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingConsumerIsolationLevelIfEosDisabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAllowSettingConsumerIsolationLevelIfEosDisabled PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyOptimizationWhenNotExplicitlyAddedToConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSpecifyOptimizationWhenNotExplicitlyAddedToConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > testGetGlobalConsumerConfigs 
STARTED

org.apache.kafka.streams.StreamsConfigTest > testGetGlobalConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfRestoreConsumerConfig STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfRestoreConsumerConfig PASSED

org.apache.kafka.streams.StreamsConfigTest > 
testGetRestoreConsumerConfigsWithRestoreConsumerOverridenPrefix STARTED

org.apache.kafka.streams.StreamsConfigTest > 
testGetRestoreConsumerConfigsWithRestoreConsumerOverridenPrefix PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowConfigExceptionIfMaxInFlightRequestsPerConnectionIsInvalidStringIfEosEnabled
 STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowConfigExceptionIfMaxInFlightRequestsPerConnectionIsInvalidStringIfEosEnabled
 PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultProducerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultProducerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfProducerConfig STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfProducerConfig PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedGlobalConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedGlobalConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAcceptBuiltInMetricsVersion0100To23 STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldAcceptBuiltInMetricsVersion0100To23 PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedConsumerConfigs STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldBeSupportNonPrefixedConsumerConfigs PASSED

org.apache.kafka.streams.StreamsConfigTest > 
consumerConfigShouldContainAdminClientConfigsForRetriesAndRetryBackOffMsWithAdminPrefix
 STARTED

org.apache.kafka.streams.StreamsConfigTest > 
consumerConfigShouldContainAdminClientConfigsForRetriesAndRetryBackOffMsWithAdminPrefix
 PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConifgsOnRestoreConsumer STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldOverrideStreamsDefaultConsumerConifgsOnRestoreConsumer PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfGlobalConsumerConfig STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldSupportPrefixedPropertiesThatAreNotPartOfGlobalConsumerConfig PASSED

org.apache.kafka.streams.StreamsConfigTest > shouldAcceptExactlyOnce STARTED

org.apache.kafka.streams.StreamsConfigTest > shouldAcceptExactlyOnce PASSED

org.apache.kafka.streams.StreamsConfigTest > 
testGetGlobalConsumerConfigsWithGlobalConsumerOverridenPrefix STARTED

org.apache.kafka.streams.StreamsConfigTest > 
testGetGlobalConsumerConfigsWithGlobalConsumerOverridenPrefix PASSED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowExceptionIfMaxInFlightRequestsGreaterThanFiveIfEosEnabled STARTED

org.apache.kafka.streams.StreamsConfigTest > 
shouldThrowExceptionIfMaxInFlightRequestsGreaterThanFiveIfEosEnabled PASSED

org.apa

Re: [DISCUSS] KIP-517: Add consumer metric indicating time between poll calls

[Satish Duggana]

Hi Kevin,
Thanks for adding useful metrics with the KIP.

On Wed, 18 Sep, 2019, 1:49 AM Kevin Lu,  wrote:

> Hi Manikumar,
>
> Thanks for the support.
>
> Since we have added a couple additional metrics, I have renamed the KIP
> title to reflect the content better:  KIP-517: Add consumer metrics to
> observe user poll behavior
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-517%3A+Add+consumer+metrics+to+observe+user+poll+behavior
> >
>
> Regards,
> Kevin
>
> On Tue, Sep 17, 2019 at 11:07 AM Manikumar 
> wrote:
>
> > Hi Kevin,
> >
> > Thanks for the KIP.  LGTM. This will be useful.
> >
> > Thanks,
> >
> > On Mon, Sep 16, 2019 at 10:17 PM Harsha Chintalapani 
> > wrote:
> >
> > > Thanks. +1 LGTM.
> > >
> > >
> > > On Mon, Sep 16, 2019 at 9:19 AM, Kevin Lu 
> wrote:
> > >
> > > > Hi Harsha,
> > > >
> > > > Thanks for the feedback. I have added *last-poll-seconds-ago* to the
> > KIP
> > > > (being consistent with *last-heartbeat-seconds-ago*).
> > > >
> > > > Regards,
> > > > Kevin
> > > >
> > > > On Sat, Sep 14, 2019 at 9:44 AM Harsha Chintalapani  >
> > > > wrote:
> > > >
> > > > Thanks Kevin for the KIP. Overall LGTM.
> > > > On you second point, I think the metric will be really useful to
> > indicate
> > > > the perf bottlenecks on user code vs kakfa consumer/broker.
> > > >
> > > > Thanks,
> > > > Harsha
> > > >
> > > > On Fri, Sep 13, 2019 at 2:41 PM, Kevin Lu 
> > wrote:
> > > >
> > > > Hi Radai & Jason,
> > > >
> > > > Thanks for the support and suggestion.
> > > >
> > > > 1. I think ratio is a good additional metric since the current
> proposed
> > > > metrics are only absolute times which may not be useful in all
> > scenarios.
> > > >
> > > > I have added this to the KIP:
> > > > * - poll-idle-ratio*: The fraction of time the consumer spent waiting
> > for
> > > > the user to process records from poll.
> > > >
> > > > Thoughts on the metric name/description?
> > > >
> > > > 2. Would it be useful to include a metric measuring the time since
> poll
> > > > was last called? Similar to *heartbeat-last-seconds-ago*, it would be
> > > > *poll-last-ms-ago.
> > > > *This could be useful if (1) the user has a very high
> > *max.poll.interval.
> > > > ms
> > > > * configured and typically spends a
> long
> > > > time processing, or (2) comparing this metric with others such as
> > > > *heartbeat-last-seconds-ago* or something else for gathering data in
> > root
> > > > cause analyses (or identifying potential consumer bugs related to
> > poll).
> > > >
> > > > Regards,
> > > > Kevin
> > > >
> > > > On Fri, Sep 13, 2019 at 10:39 AM Jason Gustafson  >
> > > > wrote:
> > > >
> > > > Hi Kevin,
> > > >
> > > > This looks reasonable to me. I'd also +1 Radai's suggestion if you're
> > > > willing. Something like an idle ratio for the consumer would be
> > helpful.
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > > On Fri, Sep 13, 2019 at 10:08 AM radai 
> > > > wrote:
> > > >
> > > > while youre at it another metric that we have found to be useful is %
> > > >
> > > > time
> > > >
> > > > spent in user code vs time spent in poll() (so time between poll
> calls
> > /
> > > > time inside poll calls) - the higher the % value the more indicative
> of
> > > > user code being the cause of performance bottlenecks.
> > > >
> > > > On Fri, Sep 13, 2019 at 9:14 AM Kevin Lu 
> > wrote:
> > > >
> > > > Hi All,
> > > >
> > > > Happy Friday! Bumping this. Any thoughts?
> > > >
> > > > Thanks.
> > > >
> > > > Regards,
> > > > Kevin
> > > >
> > > > On Thu, Sep 5, 2019 at 9:35 AM Kevin Lu 
> wrote:
> > > >
> > > > Hi All,
> > > >
> > > > I'd like to propose a new consumer metric that measures the time
> > > >
> > > > between
> > > >
> > > > calls to poll() for use in issues related to hitting
> > > >
> > > > max.poll.interval.ms
> > > >
> > > > due to long processing time.
> > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > > KIP-517%3A+Add+consumer+metric+indicating+time+between+poll+calls
> > > >
> > > > Please give it a read and let me know what you think.
> > > >
> > > > Thanks!
> > > >
> > > > Regards,
> > > > Kevin
> > > >
> > > >
> > >
> >
>

[jira] [Resolved] (KAFKA-8919) Flaky Test kafka.api.AuthorizerIntegrationTest.testSimpleConsumeWithOffsetLookupAndNoGroupAccess

[Guozhang Wang (Jira)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-8919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-8919.
--
Resolution: Fixed

Realize it is fixed as part of polling (1L) in 
https://github.com/apache/kafka/pull/7312

> Flaky Test 
> kafka.api.AuthorizerIntegrationTest.testSimpleConsumeWithOffsetLookupAndNoGroupAccess
> 
>
> Key: KAFKA-8919
> URL: https://issues.apache.org/jira/browse/KAFKA-8919
> Project: Kafka
>  Issue Type: Bug
>  Components: unit tests
>Reporter: Guozhang Wang
>Priority: Major
>  Labels: flaky-test
>
> {code}
> Stacktrace
> org.scalatest.exceptions.TestFailedException: Consumed 0 records before 
> timeout instead of the expected 1 records
>   at 
> org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530)
>   at 
> org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529)
>   at 
> org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1389)
>   at org.scalatest.Assertions.fail(Assertions.scala:1091)
>   at org.scalatest.Assertions.fail$(Assertions.scala:1087)
>   at org.scalatest.Assertions$.fail(Assertions.scala:1389)
>   at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:830)
>   at kafka.utils.TestUtils$.pollRecordsUntilTrue(TestUtils.scala:795)
>   at 
> kafka.utils.TestUtils$.pollUntilAtLeastNumRecords(TestUtils.scala:1325)
>   at kafka.utils.TestUtils$.consumeRecords(TestUtils.scala:1333)
>   at 
> kafka.api.AuthorizerIntegrationTest.consumeRecords(AuthorizerIntegrationTest.scala:1772)
>   at 
> kafka.api.AuthorizerIntegrationTest.testSimpleConsumeWithOffsetLookupAndNoGroupAccess(AuthorizerIntegrationTest.scala:813)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
>   at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
>   at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
>   at 
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
>   at jdk.internal.reflect.GeneratedMethodAccessor110.invoke(Unknown 
> Source)
>   at 
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.base/java.lang.reflect.Method.invoke(Method.java:566)
>   at 
> 

Build failed in Jenkins: kafka-2.2-jdk8 #166

[Apache Jenkins Server]

See 


Changes:

[matthias] KAFKA-8913: Document topic based configs & ISR settings for Streams 
apps

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H49 (ubuntu bionic) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/2.2^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/2.2^{commit} # timeout=10
Checking out Revision bf0e10e5a79b4652bf630c48d488065837170f01 
(refs/remotes/origin/2.2)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f bf0e10e5a79b4652bf630c48d488065837170f01
Commit message: "KAFKA-8913: Document topic based configs & ISR settings for 
Streams apps (#7346)"
 > git rev-list --no-walk 7f58c25b7b1eea583738f0c2e5b59d45509039cb # timeout=10
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
[kafka-2.2-jdk8] $ /bin/bash -xe /tmp/jenkins7707367772026035443.sh
+ rm -rf 
+ /home/jenkins/tools/gradle/4.8.1/bin/gradle
/tmp/jenkins7707367772026035443.sh: line 4: 
/home/jenkins/tools/gradle/4.8.1/bin/gradle: No such file or directory
Build step 'Execute shell' marked build as failure
[FINDBUGS] Collecting findbugs analysis files...
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
[FINDBUGS] Searching for all files in 
 that match the pattern 
**/build/reports/findbugs/*.xml
[FINDBUGS] No files found. Configuration error?
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
No credentials specified
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
 Using GitBlamer to create author and commit information for all 
warnings.
 GIT_COMMIT=bf0e10e5a79b4652bf630c48d488065837170f01, 
workspace=
[FINDBUGS] Computing warning deltas based on reference build #163
Recording test results
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
Not sending mail to unregistered user wangg...@gmail.com

Build failed in Jenkins: kafka-2.0-jdk8 #294

[Apache Jenkins Server]

See 


Changes:

[matthias] KAFKA-8913: Document topic based configs & ISR settings for Streams 
apps

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H23 (ubuntu xenial) in workspace 

[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/apache/kafka.git
 > git init  # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # 
 > timeout=10
 > git config remote.origin.url https://github.com/apache/kafka.git # timeout=10
Fetching upstream changes from https://github.com/apache/kafka.git
 > git fetch --tags --progress https://github.com/apache/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/2.0^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/2.0^{commit} # timeout=10
Checking out Revision 440956a55c720c6a38072e5411127cb0cb8c46ed 
(refs/remotes/origin/2.0)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 440956a55c720c6a38072e5411127cb0cb8c46ed
Commit message: "KAFKA-8913: Document topic based configs & ISR settings for 
Streams apps (#7346)"
 > git rev-list --no-walk e30ae148035b0255c0b87502e26d72a2d9f62827 # timeout=10
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
[kafka-2.0-jdk8] $ /bin/bash -xe /tmp/jenkins3484324521604866259.sh
+ rm -rf 
+ /home/jenkins/tools/gradle/4.8.1/bin/gradle
/tmp/jenkins3484324521604866259.sh: line 4: 
/home/jenkins/tools/gradle/4.8.1/bin/gradle: No such file or directory
Build step 'Execute shell' marked build as failure
[FINDBUGS] Collecting findbugs analysis files...
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
[FINDBUGS] Searching for all files in 
 that match the pattern 
**/build/reports/findbugs/*.xml
[FINDBUGS] No files found. Configuration error?
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
No credentials specified
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
 Using GitBlamer to create author and commit information for all 
warnings.
 GIT_COMMIT=440956a55c720c6a38072e5411127cb0cb8c46ed, 
workspace=
[FINDBUGS] Computing warning deltas based on reference build #290
Recording test results
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
ERROR: Step ?Publish JUnit test result report? failed: No test report files 
were found. Configuration error?
Setting GRADLE_4_8_1_HOME=/home/jenkins/tools/gradle/4.8.1
Not sending mail to unregistered user b...@confluent.io
Not sending mail to unregistered user wangg...@gmail.com

Re: [DISCUSS] KIP-399: Extend ProductionExceptionHandler to cover serialization exceptions

[Matthias J. Sax]

Thanks for picking up this KIP.

My personal take about repartition topics is, that it seems to be ok to
apply the handler for those, too (in addition to output topics). It
seems to be more flexible and also simplifies the code. In the end, the
topic name is passed via `ProducerRecord` into the handler and thus
users can decide on a per-topic basis what to do.

About stores and changelogs: yes, serialization happens first. Hence,
when we put() into RocksDB and also send() to the changelog topic (in
that case we use `ByteArraySerializer`) no serialization error should
happen (if there would have been a problem, it would have happened earlier).

However, in KIP-210, we did not consider the case that a send() might
fail for a changelog topic while the put() into the store was already
successfully applies. Hence, it's possible atm, to skip a failed write
into the changelog topic, even if the put() into the store was
successful. This seems to be a bug to me and we might want to create a
separate Jira for it -- it's related to this KIP but should not be
mangled into the KIP IMHO.

For the KIP itself, what we _could_ do is, to apply the handler to the
serialization that happens before the data is put() into the store.
However, I am not sure if we should allow this -- atm, I tend to think
we should not allow it and exclude store serialization for the handler.



-Matthias

On 9/10/19 1:46 AM, Alaa Zbair wrote:
> Hi,
> 
> I have checked the KIP-399 and the discussion and also KIP-210.
> 
> So the question we need to answer is whether it's okay to also skip
> writing the record in the internal topics, the current implementation of
> 'ProductionExceptionHandler' is applied for all topics and if we decided
> to keep it that way, how to ensure that there will be no divergence in
> local store and changelog topic ?
> 
> I would like to get input from others on what they think about this.
> 
> There is a point that I don't understand which is: why in case of a
> serialization error do we have the choice of either skipping it or
> putting in the store ? shouldn't the record be correctly serialized
> before putting it into the store ?
> 
> On 13/12/2018 14:13, Matthias J. Sax wrote:
>> For store updates, records are first serialized and afterwards put into
>> the store and written into the changelog topic.
>>
>> In the current implementation, if the send() into the changelog topic
>> produces an error and the handler skips over it, the local store content
>> and the changelog topic diverge. This seems to be a correctness issue.
>>
>> For serialization error, it would not happen that store and changelog
>> diverge, because serialization happens before and put/send. Thus, with
>> this KIP we could skip both put() and send(). However, I am still
>> wondering, if it would be ok to skip a store update for this case? (Btw:
>> the current PR does not address this atm, and a serialization error for
>> a store write would not be covered but kill the instance).
>>
>> IIRC, the original idea of the KIP was to allow skipping over record for
>> output topics only. That's why I am wondering if it's ok to allow
>> skipper over record in repartitions topics, too.
>>
>> In the end, it's some data loss for all 3 cases, so maybe it's ok to
>> allow skipping for all 3 cases. However, we should not allow that local
>> store and changelog topic diverge IMHO (what might been an orthogonal
>> bug thought).
>>
>> I also don't have an answer or preference. Just think, it's important to
>> touch on those cases and get input how people think about it.
>>
>>
>> -Matthias
>>
>>
>>
>> On 12/11/18 11:43 AM, Kamal Chandraprakash wrote:
>>> Matthias,
>>>
>>> For changelog topics, I think it does not make sense to allow skipping
>>> records if serialization fails? For internal repartitions topics, I am
>>> not sure if we should allow it or not. Would you agree with this? We
>>> should discuss the implication to derive a sound design.
>>>
>>> Can you explain the issue that happens when records are skipped to
>>> changelog / internal-repartition topics ? So, that I can look into it.
>>>
>>> On Fri, Dec 7, 2018 at 12:07 AM Matthias J. Sax 
>>> wrote:
>>>
>> To accept different types of records from multiple topologies, I
>> have to
>> define the ProducerRecord without generics.
 Yes. It does make sense. My point was, that the KIP should
 mention/explain this explicitly to allow other not familiar with the
 code base to understand it more easily :)



 About `ClassCastException`: seems to be an implementation detail. No
 need to make it part of the KIP discussion.



 One more thing that came to my mind. We use the `RecordCollector` to
 write into all topics, ie, user output topics and internal repartition
 and changelog topics.

 For changelog topics, I think it does not make sense to allow skipping
 records if serialization fails? For internal repartitions topics, I am
 not su

Re: [DISCUSS] KIP-479: Add Materialized to Join

[Matthias J. Sax]

> Just to clarify I'll update `as(StoreSupplier, StoreSupplier)` to
> `with(..., ...)` and change the class name from `StreamJoined` to
> `StreamJoin`


Thanks Bill. SGTM.



-Matthias


On 9/17/19 4:52 PM, aishwarya kumar wrote:
> Hi Bill,
> 
> Thanks for clarifying, and the KIP looks great!!
> 
> Best regards,
> Aishwarya
> 
> On Tue, Sep 17, 2019, 6:16 PM Bill Bejeck  wrote:
> 
>> Hi Aishwarya,
>>
>> On Tue, Sep 17, 2019 at 1:41 PM aishwarya kumar 
>> wrote:
>>
>>> Will this be applicable to Kstream-Ktable joins as well? Or do users
>> always
>>> materialize these joins explicitly?
>>>
>>
>> No, this change applies to KStream-KStream joins only.  With KStream-KTable
>> joins KafkaStreams has materialized the KTable already, and we don't need
>> to do anything with the KStream instance in this case.
>>
>>
>>> I'm not sure if its even necessary (or makes sense), just trying to
>>> understand why the change is applicable to Kstream joins only?
>>>
>>
>> The full details are in the KIP, but in a nutshell, we needed to break the
>> naming of the StateStore from `Joined.withName` and provide users a way to
>> name the StateStore explicitly.  While making the changes, we realized it
>> would be beneficial to give users the ability to use different WindowStore
>> implementations.  The most likely reason to use different WindowStores
>> would be to enable in-memory joins.
>>
>>
>>> Best,
>>> Aishwarya
>>>
>>
>> Regards,
>> Bill
>>
>>
>>>
>>> On Tue, Sep 17, 2019 at 4:05 PM Bill Bejeck  wrote:
>>>
 Guozhang,

 Thanks for the comments.

 Yes, you are correct in your assessment regarding names, *if* the users
 provide their own StoreSuppliers  When constructing as StoreSupplier,
>> the
 name can't be null, and additionally, there is no way to update the
>> name.
 Further downstream, the underlying StateStore instances use the
>> provided
 name for registration so they must be unique.  If users don't provide
 distinct names for the StoreSuppliers, KafkaStreams will thow a
 StreamsException when building the topology.

 Thanks,
 Bill



 On Tue, Sep 17, 2019 at 9:39 AM Guozhang Wang 
>>> wrote:

> Hello Bill,
>
> Thanks for the updated KIP. I made a pass on the StreamJoined
>> section.
 Just
> a quick question from user's perspective: if a user wants to provide
>> a
> customized store-supplier, she is forced to also provide a name via
>> the
> store-supplier. So there's no way to say "I want to provide my own
>>> store
> engine but let the library decide its name", is that right?
>
>
> Guozhang
>
>
> On Tue, Sep 17, 2019 at 8:53 AM Bill Bejeck 
>> wrote:
>
>> Bumping this discussion as we need to re-vote before the KIP
>>> deadline.
>>
>> On Fri, Sep 13, 2019 at 10:29 AM Bill Bejeck 
 wrote:
>>
>>> Hi All,
>>>
>>> While working on the implementation of KIP-479, some issues came
>> to
> light
>>> that the KIP as written won't work.  I have updated the KIP with
>> a
>> solution
>>> I believe will solve the original problem as well as address the
>>> impediment to the initial approach.
>>>
>>> This update is a significant change, so please review the updated
>>> KIP
>>>
>>
>

>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-479%3A+Add+StreamJoined+config+object+to+Join
>> and
>>> provide feedback.  After we conclude the discussion, there will
>> be
>>> a
>>> re-vote.
>>>
>>> Thanks!
>>> Bill
>>>
>>> On Wed, Jul 17, 2019 at 7:01 PM Guozhang Wang <
>> wangg...@gmail.com>
>> wrote:
>>>
 Hi Bill, thanks for your explanations. I'm on board with your
 decision
 too.


 Guozhang

 On Wed, Jul 17, 2019 at 10:20 AM Bill Bejeck >>
> wrote:

> Thanks for the response, John.
>
>> If I can offer my thoughts, it seems better to just document
>>> on
> the
>> Stream join javadoc for the Materialized parameter that it
>>> will
> not
>> make the join result queriable. I'm not opposed to the
>>> queriable
>> flag
>> in general, but introducing it is a much larger
>> consideration
 that
>> has
>> previously derailed this KIP discussion. In the interest of
>>> just
>> closing the gap and keeping the API change small, it seems
 better
> to
>> just go with documentation for now.
>
> I agree with your statement here.  IMHO the most important
>> goal
>>> of
>> this
 KIP
> is to not breaking existing users and gain some consistency of
>>> the
>> API.
>
> I'll update the KIP accordingly.
>
> -Bill
>
> On Tue, Jul 16, 2019 at 11:55 AM John Roesler <
>>> j...@confluent.io>
 wrote:
>
>> Hi Bill,
>>
>

Re: [DISCUSS] KIP-479: Add Materialized to Join

[aishwarya kumar]

Hi Bill,

Thanks for clarifying, and the KIP looks great!!

Best regards,
Aishwarya

On Tue, Sep 17, 2019, 6:16 PM Bill Bejeck  wrote:

> Hi Aishwarya,
>
> On Tue, Sep 17, 2019 at 1:41 PM aishwarya kumar 
> wrote:
>
> > Will this be applicable to Kstream-Ktable joins as well? Or do users
> always
> > materialize these joins explicitly?
> >
>
> No, this change applies to KStream-KStream joins only.  With KStream-KTable
> joins KafkaStreams has materialized the KTable already, and we don't need
> to do anything with the KStream instance in this case.
>
>
> > I'm not sure if its even necessary (or makes sense), just trying to
> > understand why the change is applicable to Kstream joins only?
> >
>
> The full details are in the KIP, but in a nutshell, we needed to break the
> naming of the StateStore from `Joined.withName` and provide users a way to
> name the StateStore explicitly.  While making the changes, we realized it
> would be beneficial to give users the ability to use different WindowStore
> implementations.  The most likely reason to use different WindowStores
> would be to enable in-memory joins.
>
>
> > Best,
> > Aishwarya
> >
>
> Regards,
> Bill
>
>
> >
> > On Tue, Sep 17, 2019 at 4:05 PM Bill Bejeck  wrote:
> >
> > > Guozhang,
> > >
> > > Thanks for the comments.
> > >
> > > Yes, you are correct in your assessment regarding names, *if* the users
> > > provide their own StoreSuppliers  When constructing as StoreSupplier,
> the
> > > name can't be null, and additionally, there is no way to update the
> name.
> > > Further downstream, the underlying StateStore instances use the
> provided
> > > name for registration so they must be unique.  If users don't provide
> > > distinct names for the StoreSuppliers, KafkaStreams will thow a
> > > StreamsException when building the topology.
> > >
> > > Thanks,
> > > Bill
> > >
> > >
> > >
> > > On Tue, Sep 17, 2019 at 9:39 AM Guozhang Wang 
> > wrote:
> > >
> > > > Hello Bill,
> > > >
> > > > Thanks for the updated KIP. I made a pass on the StreamJoined
> section.
> > > Just
> > > > a quick question from user's perspective: if a user wants to provide
> a
> > > > customized store-supplier, she is forced to also provide a name via
> the
> > > > store-supplier. So there's no way to say "I want to provide my own
> > store
> > > > engine but let the library decide its name", is that right?
> > > >
> > > >
> > > > Guozhang
> > > >
> > > >
> > > > On Tue, Sep 17, 2019 at 8:53 AM Bill Bejeck 
> wrote:
> > > >
> > > > > Bumping this discussion as we need to re-vote before the KIP
> > deadline.
> > > > >
> > > > > On Fri, Sep 13, 2019 at 10:29 AM Bill Bejeck 
> > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > > While working on the implementation of KIP-479, some issues came
> to
> > > > light
> > > > > > that the KIP as written won't work.  I have updated the KIP with
> a
> > > > > solution
> > > > > > I believe will solve the original problem as well as address the
> > > > > > impediment to the initial approach.
> > > > > >
> > > > > > This update is a significant change, so please review the updated
> > KIP
> > > > > >
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-479%3A+Add+StreamJoined+config+object+to+Join
> > > > > and
> > > > > > provide feedback.  After we conclude the discussion, there will
> be
> > a
> > > > > > re-vote.
> > > > > >
> > > > > > Thanks!
> > > > > > Bill
> > > > > >
> > > > > > On Wed, Jul 17, 2019 at 7:01 PM Guozhang Wang <
> wangg...@gmail.com>
> > > > > wrote:
> > > > > >
> > > > > >> Hi Bill, thanks for your explanations. I'm on board with your
> > > decision
> > > > > >> too.
> > > > > >>
> > > > > >>
> > > > > >> Guozhang
> > > > > >>
> > > > > >> On Wed, Jul 17, 2019 at 10:20 AM Bill Bejeck  >
> > > > wrote:
> > > > > >>
> > > > > >> > Thanks for the response, John.
> > > > > >> >
> > > > > >> > > If I can offer my thoughts, it seems better to just document
> > on
> > > > the
> > > > > >> > > Stream join javadoc for the Materialized parameter that it
> > will
> > > > not
> > > > > >> > > make the join result queriable. I'm not opposed to the
> > queriable
> > > > > flag
> > > > > >> > > in general, but introducing it is a much larger
> consideration
> > > that
> > > > > has
> > > > > >> > > previously derailed this KIP discussion. In the interest of
> > just
> > > > > >> > > closing the gap and keeping the API change small, it seems
> > > better
> > > > to
> > > > > >> > > just go with documentation for now.
> > > > > >> >
> > > > > >> > I agree with your statement here.  IMHO the most important
> goal
> > of
> > > > > this
> > > > > >> KIP
> > > > > >> > is to not breaking existing users and gain some consistency of
> > the
> > > > > API.
> > > > > >> >
> > > > > >> > I'll update the KIP accordingly.
> > > > > >> >
> > > > > >> > -Bill
> > > > > >> >
> > > > > >> > On Tue, Jul 16, 2019 at 11:55 AM John Roesler <
> > j...@confluent.io>
> > > > > >> wrote:
> > > > > >> >
> > > > > >> 

Jenkins build is back to normal : kafka-trunk-jdk11 #816

[Apache Jenkins Server]

See 

Re: [VOTE] KIP-495: Dynamically Adjust Log Levels in Connect

[Arjun Satish]

Thanks so much everyone for your time and comments. Closing out this vote
with three binding votes (Jason, Gwen and Randall) and 3 non-binding
(Chris, Cyrus and Konstantine).

Much appreciated!



On Tue, Sep 17, 2019 at 10:41 AM Randall Hauch  wrote:

> +1 (binding).
>
> Thanks for adding this, Arjun!
>
> On Thu, Sep 12, 2019 at 11:58 AM Gwen Shapira  wrote:
>
> > +1 (binding) - Thank you. Love the new REST API for logs.
> >
> > On Tue, Aug 20, 2019 at 3:05 PM Konstantine Karantasis
> >  wrote:
> > >
> > > Thanks for the updates on the KIP Arjun!
> > >
> > > +1 by me (non-binding)
> > >
> > > Konstantine
> > >
> > >
> > > On Wed, Aug 14, 2019 at 6:46 AM Konstantine Karantasis <
> > > konstant...@confluent.io> wrote:
> > >
> > > >
> > > > Thanks for the KIP Arjun.
> > > >
> > > > FYI, I left a few comments on the discussion thread, but mentioning
> > here
> > > > too since I noticed that the vote started a few hours ago.
> > > >
> > > > Konstantine
> > > >
> > > >
> > > > On Wed, Aug 14, 2019 at 12:13 AM Cyrus Vafadari  >
> > > > wrote:
> > > >
> > > >> I am excited to see this implemented +1 nonbinding
> > > >>
> > > >> On Tue, Aug 13, 2019 at 2:01 PM Chris Egerton 
> > > >> wrote:
> > > >>
> > > >> > Nice stuff, Arjun! +1 (non-binding)
> > > >> >
> > > >> > On Tue, Aug 13, 2019 at 1:55 PM Arjun Satish <
> > arjun.sat...@gmail.com>
> > > >> > wrote:
> > > >> >
> > > >> > > Hey everyone,
> > > >> > >
> > > >> > > I'd like to start a vote for KIP-495 (
> > > >> > >
> > > >> > >
> > > >> >
> > > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-495%3A+Dynamically+Adjust+Log+Levels+in+Connect
> > > >> > > ).
> > > >> > > This change will make Connect easier to debug in production
> > > >> environment.
> > > >> > >
> > > >> > > Based on the discussion, I updated the KIP to reflect how
> Connect
> > will
> > > >> > use
> > > >> > > the changes to the log4j controller to initialize its mBean.
> > > >> > >
> > > >> > > For your reference, this is the discussion thread
> > > >> > > https://www.mail-archive.com/dev@kafka.apache.org/msg99656.html
> > > >> > >
> > > >> > > Thanks in advance,
> > > >> > > Arjun
> > > >> > >
> > > >> >
> > > >>
> > > >
> >
>

[jira] [Created] (KAFKA-8919) Flaky Test kafka.api.AuthorizerIntegrationTest.testSimpleConsumeWithOffsetLookupAndNoGroupAccess

[Guozhang Wang (Jira)]

Guozhang Wang created KAFKA-8919:


 Summary: Flaky Test 
kafka.api.AuthorizerIntegrationTest.testSimpleConsumeWithOffsetLookupAndNoGroupAccess
 Key: KAFKA-8919
 URL: https://issues.apache.org/jira/browse/KAFKA-8919
 Project: Kafka
  Issue Type: Bug
  Components: unit tests
Reporter: Guozhang Wang


{code}
Stacktrace
org.scalatest.exceptions.TestFailedException: Consumed 0 records before timeout 
instead of the expected 1 records
at 
org.scalatest.Assertions.newAssertionFailedException(Assertions.scala:530)
at 
org.scalatest.Assertions.newAssertionFailedException$(Assertions.scala:529)
at 
org.scalatest.Assertions$.newAssertionFailedException(Assertions.scala:1389)
at org.scalatest.Assertions.fail(Assertions.scala:1091)
at org.scalatest.Assertions.fail$(Assertions.scala:1087)
at org.scalatest.Assertions$.fail(Assertions.scala:1389)
at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:830)
at kafka.utils.TestUtils$.pollRecordsUntilTrue(TestUtils.scala:795)
at 
kafka.utils.TestUtils$.pollUntilAtLeastNumRecords(TestUtils.scala:1325)
at kafka.utils.TestUtils$.consumeRecords(TestUtils.scala:1333)
at 
kafka.api.AuthorizerIntegrationTest.consumeRecords(AuthorizerIntegrationTest.scala:1772)
at 
kafka.api.AuthorizerIntegrationTest.testSimpleConsumeWithOffsetLookupAndNoGroupAccess(AuthorizerIntegrationTest.scala:813)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
at 
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at 
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at 
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at jdk.internal.reflect.GeneratedMethodAccessor110.invoke(Unknown 
Source)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
at 
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
at com.sun.proxy.$Proxy2.p

Re: [DISCUSS] KIP-479: Add Materialized to Join

[Bill Bejeck]

Hi Aishwarya,

On Tue, Sep 17, 2019 at 1:41 PM aishwarya kumar  wrote:

> Will this be applicable to Kstream-Ktable joins as well? Or do users always
> materialize these joins explicitly?
>

No, this change applies to KStream-KStream joins only.  With KStream-KTable
joins KafkaStreams has materialized the KTable already, and we don't need
to do anything with the KStream instance in this case.


> I'm not sure if its even necessary (or makes sense), just trying to
> understand why the change is applicable to Kstream joins only?
>

The full details are in the KIP, but in a nutshell, we needed to break the
naming of the StateStore from `Joined.withName` and provide users a way to
name the StateStore explicitly.  While making the changes, we realized it
would be beneficial to give users the ability to use different WindowStore
implementations.  The most likely reason to use different WindowStores
would be to enable in-memory joins.


> Best,
> Aishwarya
>

Regards,
Bill


>
> On Tue, Sep 17, 2019 at 4:05 PM Bill Bejeck  wrote:
>
> > Guozhang,
> >
> > Thanks for the comments.
> >
> > Yes, you are correct in your assessment regarding names, *if* the users
> > provide their own StoreSuppliers  When constructing as StoreSupplier, the
> > name can't be null, and additionally, there is no way to update the name.
> > Further downstream, the underlying StateStore instances use the provided
> > name for registration so they must be unique.  If users don't provide
> > distinct names for the StoreSuppliers, KafkaStreams will thow a
> > StreamsException when building the topology.
> >
> > Thanks,
> > Bill
> >
> >
> >
> > On Tue, Sep 17, 2019 at 9:39 AM Guozhang Wang 
> wrote:
> >
> > > Hello Bill,
> > >
> > > Thanks for the updated KIP. I made a pass on the StreamJoined section.
> > Just
> > > a quick question from user's perspective: if a user wants to provide a
> > > customized store-supplier, she is forced to also provide a name via the
> > > store-supplier. So there's no way to say "I want to provide my own
> store
> > > engine but let the library decide its name", is that right?
> > >
> > >
> > > Guozhang
> > >
> > >
> > > On Tue, Sep 17, 2019 at 8:53 AM Bill Bejeck  wrote:
> > >
> > > > Bumping this discussion as we need to re-vote before the KIP
> deadline.
> > > >
> > > > On Fri, Sep 13, 2019 at 10:29 AM Bill Bejeck 
> > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > While working on the implementation of KIP-479, some issues came to
> > > light
> > > > > that the KIP as written won't work.  I have updated the KIP with a
> > > > solution
> > > > > I believe will solve the original problem as well as address the
> > > > > impediment to the initial approach.
> > > > >
> > > > > This update is a significant change, so please review the updated
> KIP
> > > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-479%3A+Add+StreamJoined+config+object+to+Join
> > > > and
> > > > > provide feedback.  After we conclude the discussion, there will be
> a
> > > > > re-vote.
> > > > >
> > > > > Thanks!
> > > > > Bill
> > > > >
> > > > > On Wed, Jul 17, 2019 at 7:01 PM Guozhang Wang 
> > > > wrote:
> > > > >
> > > > >> Hi Bill, thanks for your explanations. I'm on board with your
> > decision
> > > > >> too.
> > > > >>
> > > > >>
> > > > >> Guozhang
> > > > >>
> > > > >> On Wed, Jul 17, 2019 at 10:20 AM Bill Bejeck 
> > > wrote:
> > > > >>
> > > > >> > Thanks for the response, John.
> > > > >> >
> > > > >> > > If I can offer my thoughts, it seems better to just document
> on
> > > the
> > > > >> > > Stream join javadoc for the Materialized parameter that it
> will
> > > not
> > > > >> > > make the join result queriable. I'm not opposed to the
> queriable
> > > > flag
> > > > >> > > in general, but introducing it is a much larger consideration
> > that
> > > > has
> > > > >> > > previously derailed this KIP discussion. In the interest of
> just
> > > > >> > > closing the gap and keeping the API change small, it seems
> > better
> > > to
> > > > >> > > just go with documentation for now.
> > > > >> >
> > > > >> > I agree with your statement here.  IMHO the most important goal
> of
> > > > this
> > > > >> KIP
> > > > >> > is to not breaking existing users and gain some consistency of
> the
> > > > API.
> > > > >> >
> > > > >> > I'll update the KIP accordingly.
> > > > >> >
> > > > >> > -Bill
> > > > >> >
> > > > >> > On Tue, Jul 16, 2019 at 11:55 AM John Roesler <
> j...@confluent.io>
> > > > >> wrote:
> > > > >> >
> > > > >> > > Hi Bill,
> > > > >> > >
> > > > >> > > Thanks for driving this KIP toward a conclusion. I'm on board
> > with
> > > > >> > > your decision.
> > > > >> > >
> > > > >> > > You didn't mention whether you're still proposing to add the
> > > > >> > > "queriable" flag to the Materialized config object, or just
> > > document
> > > > >> > > that a Stream join is never queriable. Both options have come
> up
> > > > >> > > earlier in the discussion, so it would be good to pin this
> dow

Issues with partition assignments after Kafka Connect 2.3 Upgrade

[Matt Farmer]

Hello all,

After an upgrade to Kafka Connect 2.3.0, we've started seeing an abundance
of Exceptions pertaining to Kafka Connect Workers and their partition
assignments. The exception is:

java.lang.IllegalStateException: No current assignment for partition
[redacted]
at
org.apache.kafka.clients.consumer.internals.SubscriptionState.assignedState(SubscriptionState.java:323)
at
org.apache.kafka.clients.consumer.internals.SubscriptionState.seekUnvalidated(SubscriptionState.java:340)
at
org.apache.kafka.clients.consumer.KafkaConsumer.seek(KafkaConsumer.java:1550)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.rewind(WorkerSinkTask.java:574)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.access$1200(WorkerSinkTask.java:67)
at
org.apache.kafka.connect.runtime.WorkerSinkTask$HandleRebalance.onPartitionsAssigned(WorkerSinkTask.java:653)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:285)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:424)
at
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:358)
at
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:353)
at
org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1251)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)
at
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1201)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.pollConsumer(WorkerSinkTask.java:443)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.poll(WorkerSinkTask.java:316)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.iteration(WorkerSinkTask.java:224)
at
org.apache.kafka.connect.runtime.WorkerSinkTask.execute(WorkerSinkTask.java:192)
at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:177)
at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:227)
at
java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:834)

We have multiple workers that will fail with this. Most of them appear to
be running the WePay BigQuery Sink connector. Has anyone else been having
problems with this? Any idea as to what could be going on and causing us to
have no partitions in the onPartitionsAssigned handler?

Thanks,
Matt Farmer

Re: [DISCUSS] KIP-479: Add Materialized to Join

[Bill Bejeck]

Just to clarify I'll update `as(StoreSupplier, StoreSupplier)` to
`with(..., ...)` and change the class name from `StreamJoined` to
`StreamJoin`

-Bill

On Tue, Sep 17, 2019 at 2:09 PM Bill Bejeck  wrote:

> Thanks for the comments, Matthias.
>
> I like both suggestions regarding the names and I'll update the KIP
> accordingly.
>
> On Tue, Sep 17, 2019 at 11:22 AM Matthias J. Sax 
> wrote:
>
>> Thanks Bill!
>>
>> Using a new configuration object was suggested by John in the original
>> DISCUSS thread already. We rejected it because we wanted to avoid a new
>> configuration class. However, given your analysis, it seems it's
>> actually the right choice to introduce a new configuration class.
>>
>> Hence, overall I am +1 on the proposal.
>>
>>
>> Some nits about names (as always :))
>>
>> - `StreamJoined.as()` does not sound right for the `StoreSupplier`
>> overload. Maybe it's better to call that static method `with` (not 100%
>> sure)
>>
>> - Should we use plural instead of singular -> `Stream{s}Joined`
>>   -> or keep singular but call it `StreamJoin`
>>   `Joined` seems to refer to the input stream(s) while `Join` would
>> refer to the join-operator
>>
>> Thoughts?
>>
>>
>> You suggest to deprecate existing overload what I support -- can you
>> list deprecated method in the "Public Interface" section?
>>
>>
>>
>> -Matthias
>>
>>
>>
>> On 9/17/19 9:39 AM, Guozhang Wang wrote:
>> > Hello Bill,
>> >
>> > Thanks for the updated KIP. I made a pass on the StreamJoined section.
>> Just
>> > a quick question from user's perspective: if a user wants to provide a
>> > customized store-supplier, she is forced to also provide a name via the
>> > store-supplier. So there's no way to say "I want to provide my own store
>> > engine but let the library decide its name", is that right?
>> >
>> >
>> > Guozhang
>> >
>> >
>> > On Tue, Sep 17, 2019 at 8:53 AM Bill Bejeck  wrote:
>> >
>> >> Bumping this discussion as we need to re-vote before the KIP deadline.
>> >>
>> >> On Fri, Sep 13, 2019 at 10:29 AM Bill Bejeck 
>> wrote:
>> >>
>> >>> Hi All,
>> >>>
>> >>> While working on the implementation of KIP-479, some issues came to
>> light
>> >>> that the KIP as written won't work.  I have updated the KIP with a
>> >> solution
>> >>> I believe will solve the original problem as well as address the
>> >>> impediment to the initial approach.
>> >>>
>> >>> This update is a significant change, so please review the updated KIP
>> >>>
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-479%3A+Add+StreamJoined+config+object+to+Join
>> >> and
>> >>> provide feedback.  After we conclude the discussion, there will be a
>> >>> re-vote.
>> >>>
>> >>> Thanks!
>> >>> Bill
>> >>>
>> >>> On Wed, Jul 17, 2019 at 7:01 PM Guozhang Wang 
>> >> wrote:
>> >>>
>>  Hi Bill, thanks for your explanations. I'm on board with your
>> decision
>>  too.
>> 
>> 
>>  Guozhang
>> 
>>  On Wed, Jul 17, 2019 at 10:20 AM Bill Bejeck 
>> wrote:
>> 
>> > Thanks for the response, John.
>> >
>> >> If I can offer my thoughts, it seems better to just document on the
>> >> Stream join javadoc for the Materialized parameter that it will not
>> >> make the join result queriable. I'm not opposed to the queriable
>> >> flag
>> >> in general, but introducing it is a much larger consideration that
>> >> has
>> >> previously derailed this KIP discussion. In the interest of just
>> >> closing the gap and keeping the API change small, it seems better
>> to
>> >> just go with documentation for now.
>> >
>> > I agree with your statement here.  IMHO the most important goal of
>> >> this
>>  KIP
>> > is to not breaking existing users and gain some consistency of the
>> >> API.
>> >
>> > I'll update the KIP accordingly.
>> >
>> > -Bill
>> >
>> > On Tue, Jul 16, 2019 at 11:55 AM John Roesler 
>>  wrote:
>> >
>> >> Hi Bill,
>> >>
>> >> Thanks for driving this KIP toward a conclusion. I'm on board with
>> >> your decision.
>> >>
>> >> You didn't mention whether you're still proposing to add the
>> >> "queriable" flag to the Materialized config object, or just
>> document
>> >> that a Stream join is never queriable. Both options have come up
>> >> earlier in the discussion, so it would be good to pin this down.
>> >>
>> >> If I can offer my thoughts, it seems better to just document on the
>> >> Stream join javadoc for the Materialized parameter that it will not
>> >> make the join result queriable. I'm not opposed to the queriable
>> >> flag
>> >> in general, but introducing it is a much larger consideration that
>> >> has
>> >> previously derailed this KIP discussion. In the interest of just
>> >> closing the gap and keeping the API change small, it seems better
>> to
>> >> just go with documentation for now.
>> >>
>> >> Thanks again,
>> >> -John
>> >>
>> >> On Thu, Jul 11, 201

[jira] [Created] (KAFKA-8918) Flaky Test org.apache.kafka.trogdor.coordinator.CoordinatorTest.testTaskCancellation

[Guozhang Wang (Jira)]

Guozhang Wang created KAFKA-8918:


 Summary: Flaky Test 
org.apache.kafka.trogdor.coordinator.CoordinatorTest.testTaskCancellation
 Key: KAFKA-8918
 URL: https://issues.apache.org/jira/browse/KAFKA-8918
 Project: Kafka
  Issue Type: Bug
  Components: unit tests
Reporter: Guozhang Wang


{code}
Stacktrace
java.lang.AssertionError: Condition not met within timeout 15000. Timed out 
waiting for expected tasks 
{"foo":{"id":"foo","taskState":{"state":"DONE","spec":{"class":"org.apache.kafka.trogdor.task.NoOpTaskSpec","startMs":5,"durationMs":7},"startedMs":11,"doneMs":18,"cancelled":true,"status":{"node01":"done","node02":"done"}},"workerState":{"state":"DONE","taskId":"foo","spec":{"class":"org.apache.kafka.trogdor.task.NoOpTaskSpec","startMs":5,"durationMs":7},"startedMs":11,"doneMs":18,"status":"done"}}}
at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:376)
at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:336)
at 
org.apache.kafka.trogdor.common.ExpectedTasks.waitFor(ExpectedTasks.java:144)
at 
org.apache.kafka.trogdor.coordinator.CoordinatorTest.testTaskCancellation(CoordinatorTest.java:264)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
at 
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.lang.Thread.run(Thread.java:834)
{code}



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: [DISCUSS] KIP-479: Add Materialized to Join

[Bill Bejeck]

Thanks for the comments, Matthias.

I like both suggestions regarding the names and I'll update the KIP
accordingly.

On Tue, Sep 17, 2019 at 11:22 AM Matthias J. Sax 
wrote:

> Thanks Bill!
>
> Using a new configuration object was suggested by John in the original
> DISCUSS thread already. We rejected it because we wanted to avoid a new
> configuration class. However, given your analysis, it seems it's
> actually the right choice to introduce a new configuration class.
>
> Hence, overall I am +1 on the proposal.
>
>
> Some nits about names (as always :))
>
> - `StreamJoined.as()` does not sound right for the `StoreSupplier`
> overload. Maybe it's better to call that static method `with` (not 100%
> sure)
>
> - Should we use plural instead of singular -> `Stream{s}Joined`
>   -> or keep singular but call it `StreamJoin`
>   `Joined` seems to refer to the input stream(s) while `Join` would
> refer to the join-operator
>
> Thoughts?
>
>
> You suggest to deprecate existing overload what I support -- can you
> list deprecated method in the "Public Interface" section?
>
>
>
> -Matthias
>
>
>
> On 9/17/19 9:39 AM, Guozhang Wang wrote:
> > Hello Bill,
> >
> > Thanks for the updated KIP. I made a pass on the StreamJoined section.
> Just
> > a quick question from user's perspective: if a user wants to provide a
> > customized store-supplier, she is forced to also provide a name via the
> > store-supplier. So there's no way to say "I want to provide my own store
> > engine but let the library decide its name", is that right?
> >
> >
> > Guozhang
> >
> >
> > On Tue, Sep 17, 2019 at 8:53 AM Bill Bejeck  wrote:
> >
> >> Bumping this discussion as we need to re-vote before the KIP deadline.
> >>
> >> On Fri, Sep 13, 2019 at 10:29 AM Bill Bejeck  wrote:
> >>
> >>> Hi All,
> >>>
> >>> While working on the implementation of KIP-479, some issues came to
> light
> >>> that the KIP as written won't work.  I have updated the KIP with a
> >> solution
> >>> I believe will solve the original problem as well as address the
> >>> impediment to the initial approach.
> >>>
> >>> This update is a significant change, so please review the updated KIP
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-479%3A+Add+StreamJoined+config+object+to+Join
> >> and
> >>> provide feedback.  After we conclude the discussion, there will be a
> >>> re-vote.
> >>>
> >>> Thanks!
> >>> Bill
> >>>
> >>> On Wed, Jul 17, 2019 at 7:01 PM Guozhang Wang 
> >> wrote:
> >>>
>  Hi Bill, thanks for your explanations. I'm on board with your decision
>  too.
> 
> 
>  Guozhang
> 
>  On Wed, Jul 17, 2019 at 10:20 AM Bill Bejeck 
> wrote:
> 
> > Thanks for the response, John.
> >
> >> If I can offer my thoughts, it seems better to just document on the
> >> Stream join javadoc for the Materialized parameter that it will not
> >> make the join result queriable. I'm not opposed to the queriable
> >> flag
> >> in general, but introducing it is a much larger consideration that
> >> has
> >> previously derailed this KIP discussion. In the interest of just
> >> closing the gap and keeping the API change small, it seems better to
> >> just go with documentation for now.
> >
> > I agree with your statement here.  IMHO the most important goal of
> >> this
>  KIP
> > is to not breaking existing users and gain some consistency of the
> >> API.
> >
> > I'll update the KIP accordingly.
> >
> > -Bill
> >
> > On Tue, Jul 16, 2019 at 11:55 AM John Roesler 
>  wrote:
> >
> >> Hi Bill,
> >>
> >> Thanks for driving this KIP toward a conclusion. I'm on board with
> >> your decision.
> >>
> >> You didn't mention whether you're still proposing to add the
> >> "queriable" flag to the Materialized config object, or just document
> >> that a Stream join is never queriable. Both options have come up
> >> earlier in the discussion, so it would be good to pin this down.
> >>
> >> If I can offer my thoughts, it seems better to just document on the
> >> Stream join javadoc for the Materialized parameter that it will not
> >> make the join result queriable. I'm not opposed to the queriable
> >> flag
> >> in general, but introducing it is a much larger consideration that
> >> has
> >> previously derailed this KIP discussion. In the interest of just
> >> closing the gap and keeping the API change small, it seems better to
> >> just go with documentation for now.
> >>
> >> Thanks again,
> >> -John
> >>
> >> On Thu, Jul 11, 2019 at 2:45 PM Bill Bejeck 
>  wrote:
> >>>
> >>> Thanks all for the great discussion so far.
> >>>
> >>> Everyone has made excellent points, and I appreciate the detail
> > everyone
> >>> has put into their arguments.
> >>>
> >>> However, after carefully evaluating all the points made so far,
> > creating
> >> an
> >>> 

Re: [DISCUSS] KIP-479: Add Materialized to Join

[aishwarya kumar]

Will this be applicable to Kstream-Ktable joins as well? Or do users always
materialize these joins explicitly?
I'm not sure if its even necessary (or makes sense), just trying to
understand why the change is applicable to Kstream joins only?

Best,
Aishwarya

On Tue, Sep 17, 2019 at 4:05 PM Bill Bejeck  wrote:

> Guozhang,
>
> Thanks for the comments.
>
> Yes, you are correct in your assessment regarding names, *if* the users
> provide their own StoreSuppliers  When constructing as StoreSupplier, the
> name can't be null, and additionally, there is no way to update the name.
> Further downstream, the underlying StateStore instances use the provided
> name for registration so they must be unique.  If users don't provide
> distinct names for the StoreSuppliers, KafkaStreams will thow a
> StreamsException when building the topology.
>
> Thanks,
> Bill
>
>
>
> On Tue, Sep 17, 2019 at 9:39 AM Guozhang Wang  wrote:
>
> > Hello Bill,
> >
> > Thanks for the updated KIP. I made a pass on the StreamJoined section.
> Just
> > a quick question from user's perspective: if a user wants to provide a
> > customized store-supplier, she is forced to also provide a name via the
> > store-supplier. So there's no way to say "I want to provide my own store
> > engine but let the library decide its name", is that right?
> >
> >
> > Guozhang
> >
> >
> > On Tue, Sep 17, 2019 at 8:53 AM Bill Bejeck  wrote:
> >
> > > Bumping this discussion as we need to re-vote before the KIP deadline.
> > >
> > > On Fri, Sep 13, 2019 at 10:29 AM Bill Bejeck 
> wrote:
> > >
> > > > Hi All,
> > > >
> > > > While working on the implementation of KIP-479, some issues came to
> > light
> > > > that the KIP as written won't work.  I have updated the KIP with a
> > > solution
> > > > I believe will solve the original problem as well as address the
> > > > impediment to the initial approach.
> > > >
> > > > This update is a significant change, so please review the updated KIP
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-479%3A+Add+StreamJoined+config+object+to+Join
> > > and
> > > > provide feedback.  After we conclude the discussion, there will be a
> > > > re-vote.
> > > >
> > > > Thanks!
> > > > Bill
> > > >
> > > > On Wed, Jul 17, 2019 at 7:01 PM Guozhang Wang 
> > > wrote:
> > > >
> > > >> Hi Bill, thanks for your explanations. I'm on board with your
> decision
> > > >> too.
> > > >>
> > > >>
> > > >> Guozhang
> > > >>
> > > >> On Wed, Jul 17, 2019 at 10:20 AM Bill Bejeck 
> > wrote:
> > > >>
> > > >> > Thanks for the response, John.
> > > >> >
> > > >> > > If I can offer my thoughts, it seems better to just document on
> > the
> > > >> > > Stream join javadoc for the Materialized parameter that it will
> > not
> > > >> > > make the join result queriable. I'm not opposed to the queriable
> > > flag
> > > >> > > in general, but introducing it is a much larger consideration
> that
> > > has
> > > >> > > previously derailed this KIP discussion. In the interest of just
> > > >> > > closing the gap and keeping the API change small, it seems
> better
> > to
> > > >> > > just go with documentation for now.
> > > >> >
> > > >> > I agree with your statement here.  IMHO the most important goal of
> > > this
> > > >> KIP
> > > >> > is to not breaking existing users and gain some consistency of the
> > > API.
> > > >> >
> > > >> > I'll update the KIP accordingly.
> > > >> >
> > > >> > -Bill
> > > >> >
> > > >> > On Tue, Jul 16, 2019 at 11:55 AM John Roesler 
> > > >> wrote:
> > > >> >
> > > >> > > Hi Bill,
> > > >> > >
> > > >> > > Thanks for driving this KIP toward a conclusion. I'm on board
> with
> > > >> > > your decision.
> > > >> > >
> > > >> > > You didn't mention whether you're still proposing to add the
> > > >> > > "queriable" flag to the Materialized config object, or just
> > document
> > > >> > > that a Stream join is never queriable. Both options have come up
> > > >> > > earlier in the discussion, so it would be good to pin this down.
> > > >> > >
> > > >> > > If I can offer my thoughts, it seems better to just document on
> > the
> > > >> > > Stream join javadoc for the Materialized parameter that it will
> > not
> > > >> > > make the join result queriable. I'm not opposed to the queriable
> > > flag
> > > >> > > in general, but introducing it is a much larger consideration
> that
> > > has
> > > >> > > previously derailed this KIP discussion. In the interest of just
> > > >> > > closing the gap and keeping the API change small, it seems
> better
> > to
> > > >> > > just go with documentation for now.
> > > >> > >
> > > >> > > Thanks again,
> > > >> > > -John
> > > >> > >
> > > >> > > On Thu, Jul 11, 2019 at 2:45 PM Bill Bejeck 
> > > >> wrote:
> > > >> > > >
> > > >> > > > Thanks all for the great discussion so far.
> > > >> > > >
> > > >> > > > Everyone has made excellent points, and I appreciate the
> detail
> > > >> > everyone
> > > >> > > > has put into their arguments.
> > > >> > > >
> > > >> > > > However, aft

Re: [DISCUSS] KIP-517: Add consumer metric indicating time between poll calls

[Kevin Lu]

Hi Manikumar,

Thanks for the support.

Since we have added a couple additional metrics, I have renamed the KIP
title to reflect the content better:  KIP-517: Add consumer metrics to
observe user poll behavior


Regards,
Kevin

On Tue, Sep 17, 2019 at 11:07 AM Manikumar 
wrote:

> Hi Kevin,
>
> Thanks for the KIP.  LGTM. This will be useful.
>
> Thanks,
>
> On Mon, Sep 16, 2019 at 10:17 PM Harsha Chintalapani 
> wrote:
>
> > Thanks. +1 LGTM.
> >
> >
> > On Mon, Sep 16, 2019 at 9:19 AM, Kevin Lu  wrote:
> >
> > > Hi Harsha,
> > >
> > > Thanks for the feedback. I have added *last-poll-seconds-ago* to the
> KIP
> > > (being consistent with *last-heartbeat-seconds-ago*).
> > >
> > > Regards,
> > > Kevin
> > >
> > > On Sat, Sep 14, 2019 at 9:44 AM Harsha Chintalapani 
> > > wrote:
> > >
> > > Thanks Kevin for the KIP. Overall LGTM.
> > > On you second point, I think the metric will be really useful to
> indicate
> > > the perf bottlenecks on user code vs kakfa consumer/broker.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Fri, Sep 13, 2019 at 2:41 PM, Kevin Lu 
> wrote:
> > >
> > > Hi Radai & Jason,
> > >
> > > Thanks for the support and suggestion.
> > >
> > > 1. I think ratio is a good additional metric since the current proposed
> > > metrics are only absolute times which may not be useful in all
> scenarios.
> > >
> > > I have added this to the KIP:
> > > * - poll-idle-ratio*: The fraction of time the consumer spent waiting
> for
> > > the user to process records from poll.
> > >
> > > Thoughts on the metric name/description?
> > >
> > > 2. Would it be useful to include a metric measuring the time since poll
> > > was last called? Similar to *heartbeat-last-seconds-ago*, it would be
> > > *poll-last-ms-ago.
> > > *This could be useful if (1) the user has a very high
> *max.poll.interval.
> > > ms
> > > * configured and typically spends a long
> > > time processing, or (2) comparing this metric with others such as
> > > *heartbeat-last-seconds-ago* or something else for gathering data in
> root
> > > cause analyses (or identifying potential consumer bugs related to
> poll).
> > >
> > > Regards,
> > > Kevin
> > >
> > > On Fri, Sep 13, 2019 at 10:39 AM Jason Gustafson 
> > > wrote:
> > >
> > > Hi Kevin,
> > >
> > > This looks reasonable to me. I'd also +1 Radai's suggestion if you're
> > > willing. Something like an idle ratio for the consumer would be
> helpful.
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Fri, Sep 13, 2019 at 10:08 AM radai 
> > > wrote:
> > >
> > > while youre at it another metric that we have found to be useful is %
> > >
> > > time
> > >
> > > spent in user code vs time spent in poll() (so time between poll calls
> /
> > > time inside poll calls) - the higher the % value the more indicative of
> > > user code being the cause of performance bottlenecks.
> > >
> > > On Fri, Sep 13, 2019 at 9:14 AM Kevin Lu 
> wrote:
> > >
> > > Hi All,
> > >
> > > Happy Friday! Bumping this. Any thoughts?
> > >
> > > Thanks.
> > >
> > > Regards,
> > > Kevin
> > >
> > > On Thu, Sep 5, 2019 at 9:35 AM Kevin Lu  wrote:
> > >
> > > Hi All,
> > >
> > > I'd like to propose a new consumer metric that measures the time
> > >
> > > between
> > >
> > > calls to poll() for use in issues related to hitting
> > >
> > > max.poll.interval.ms
> > >
> > > due to long processing time.
> > >
> > > https://cwiki.apache.org/confluence/display/KAFKA/
> > > KIP-517%3A+Add+consumer+metric+indicating+time+between+poll+calls
> > >
> > > Please give it a read and let me know what you think.
> > >
> > > Thanks!
> > >
> > > Regards,
> > > Kevin
> > >
> > >
> >
>

Re: [DISCUSS] KIP-479: Add Materialized to Join

[Bill Bejeck]

Guozhang,

Thanks for the comments.

Yes, you are correct in your assessment regarding names, *if* the users
provide their own StoreSuppliers  When constructing as StoreSupplier, the
name can't be null, and additionally, there is no way to update the name.
Further downstream, the underlying StateStore instances use the provided
name for registration so they must be unique.  If users don't provide
distinct names for the StoreSuppliers, KafkaStreams will thow a
StreamsException when building the topology.

Thanks,
Bill



On Tue, Sep 17, 2019 at 9:39 AM Guozhang Wang  wrote:

> Hello Bill,
>
> Thanks for the updated KIP. I made a pass on the StreamJoined section. Just
> a quick question from user's perspective: if a user wants to provide a
> customized store-supplier, she is forced to also provide a name via the
> store-supplier. So there's no way to say "I want to provide my own store
> engine but let the library decide its name", is that right?
>
>
> Guozhang
>
>
> On Tue, Sep 17, 2019 at 8:53 AM Bill Bejeck  wrote:
>
> > Bumping this discussion as we need to re-vote before the KIP deadline.
> >
> > On Fri, Sep 13, 2019 at 10:29 AM Bill Bejeck  wrote:
> >
> > > Hi All,
> > >
> > > While working on the implementation of KIP-479, some issues came to
> light
> > > that the KIP as written won't work.  I have updated the KIP with a
> > solution
> > > I believe will solve the original problem as well as address the
> > > impediment to the initial approach.
> > >
> > > This update is a significant change, so please review the updated KIP
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-479%3A+Add+StreamJoined+config+object+to+Join
> > and
> > > provide feedback.  After we conclude the discussion, there will be a
> > > re-vote.
> > >
> > > Thanks!
> > > Bill
> > >
> > > On Wed, Jul 17, 2019 at 7:01 PM Guozhang Wang 
> > wrote:
> > >
> > >> Hi Bill, thanks for your explanations. I'm on board with your decision
> > >> too.
> > >>
> > >>
> > >> Guozhang
> > >>
> > >> On Wed, Jul 17, 2019 at 10:20 AM Bill Bejeck 
> wrote:
> > >>
> > >> > Thanks for the response, John.
> > >> >
> > >> > > If I can offer my thoughts, it seems better to just document on
> the
> > >> > > Stream join javadoc for the Materialized parameter that it will
> not
> > >> > > make the join result queriable. I'm not opposed to the queriable
> > flag
> > >> > > in general, but introducing it is a much larger consideration that
> > has
> > >> > > previously derailed this KIP discussion. In the interest of just
> > >> > > closing the gap and keeping the API change small, it seems better
> to
> > >> > > just go with documentation for now.
> > >> >
> > >> > I agree with your statement here.  IMHO the most important goal of
> > this
> > >> KIP
> > >> > is to not breaking existing users and gain some consistency of the
> > API.
> > >> >
> > >> > I'll update the KIP accordingly.
> > >> >
> > >> > -Bill
> > >> >
> > >> > On Tue, Jul 16, 2019 at 11:55 AM John Roesler 
> > >> wrote:
> > >> >
> > >> > > Hi Bill,
> > >> > >
> > >> > > Thanks for driving this KIP toward a conclusion. I'm on board with
> > >> > > your decision.
> > >> > >
> > >> > > You didn't mention whether you're still proposing to add the
> > >> > > "queriable" flag to the Materialized config object, or just
> document
> > >> > > that a Stream join is never queriable. Both options have come up
> > >> > > earlier in the discussion, so it would be good to pin this down.
> > >> > >
> > >> > > If I can offer my thoughts, it seems better to just document on
> the
> > >> > > Stream join javadoc for the Materialized parameter that it will
> not
> > >> > > make the join result queriable. I'm not opposed to the queriable
> > flag
> > >> > > in general, but introducing it is a much larger consideration that
> > has
> > >> > > previously derailed this KIP discussion. In the interest of just
> > >> > > closing the gap and keeping the API change small, it seems better
> to
> > >> > > just go with documentation for now.
> > >> > >
> > >> > > Thanks again,
> > >> > > -John
> > >> > >
> > >> > > On Thu, Jul 11, 2019 at 2:45 PM Bill Bejeck 
> > >> wrote:
> > >> > > >
> > >> > > > Thanks all for the great discussion so far.
> > >> > > >
> > >> > > > Everyone has made excellent points, and I appreciate the detail
> > >> > everyone
> > >> > > > has put into their arguments.
> > >> > > >
> > >> > > > However, after carefully evaluating all the points made so far,
> > >> > creating
> > >> > > an
> > >> > > > overload with Materialized is still my #1 option.
> > >> > > > My reasoning for saying so is two-fold:
> > >> > > >
> > >> > > >1. It's a small change, and IMHO since it's consistent with
> our
> > >> > > current
> > >> > > >API concerning state store usage, the cognitive load on users
> > >> will
> > >> > be
> > >> > > >minimal.
> > >> > > >2. It achieves the most important goal of this KIP, namely to
> > >> close
> > >> > > the
> > >> > > >gap of naming state stores indep

[DISCUSS] KIP-526: Reduce Producer Metadata Lookups for Large Number of Topics

[Brian Byrne]

Dev team,

Requesting discussion for improvement to the producer when dealing with a
large number of topics.

KIP:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-526%3A+Reduce+Producer+Metadata+Lookups+for+Large+Number+of+Topics

JIRA: https://issues.apache.org/jira/browse/KAFKA-8904

Thoughts and feedback would be appreciated.

Thanks,
Brian

Re: [VOTE] KIP-481: SerDe Improvements for Connect Decimal type in JSON

[Guozhang Wang]

+1 (binding).

Thanks Almog!

Guozhang

On Tue, Sep 17, 2019 at 11:56 AM Bill Bejeck  wrote:

> +1 (binding)
>
> On Tue, Sep 17, 2019 at 11:32 AM Matthias J. Sax 
> wrote:
>
> > +1 (binding)
> >
> > -Matthias
> >
> > On 9/17/19 12:20 AM, Arjun Satish wrote:
> > > Thanks for the KIP, Almog.
> > >
> > > +1 (non-binding)
> > >
> > > On Mon, Sep 16, 2019 at 4:45 PM Randall Hauch 
> wrote:
> > >
> > >> Thanks for the nice improvement, Almog!
> > >>
> > >> +1 (binding)
> > >>
> > >> Randall
> > >>
> > >> On Thu, Aug 15, 2019 at 11:59 AM Konstantine Karantasis <
> > >> konstant...@confluent.io> wrote:
> > >>
> > >>> Thanks Almog!
> > >>> Nicely designed and concise KIP.
> > >>>
> > >>> +1 non-binding
> > >>>
> > >>> Konstantine
> > >>>
> > >>> On Tue, Aug 6, 2019 at 11:44 PM Almog Gavra 
> > wrote:
> > >>>
> >  Hello Everyone,
> > 
> >  After discussions on
> > 
> > 
> > >>>
> > >>
> >
> https://lists.apache.org/thread.html/fa665a6dc59f73ca294a00bcbef2eaa3ad00cc69626e91c516fa4fca@%3Cdev.kafka.apache.org%3E
> >  I've opened this KIP up for formal voting.
> > 
> >  KIP:
> > 
> > 
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-481%3A+SerDe+Improvements+for+Connect+Decimal+type+in+JSON
> > 
> >  Please update the DISCUSS thread with any concerns/comments.
> > 
> >  Cheers!
> >  Almog
> > 
> > >>>
> > >>
> > >
> >
> >
>


-- 
-- Guozhang

About Testing Stream Applications Documentation

[uğur]

Hi,

I am not sure if it is the right email address to write about this topic,
please correct me if I am wrong.

As I read documentation of Testing Stream Application, I believe I noticed
wrong naming of method called shouldNotUpdateStoreForLargerValue. As I
understand it, either the name should change
to shouldUpdateStoreForLargerValue or assertion should be different.

Regards,
Ugur Yeter

Re: [VOTE] KIP-481: SerDe Improvements for Connect Decimal type in JSON

[Almog Gavra]

Thanks everyone for the comments and discussion! Closing the voting out for
this KIP and marking accepted:

* 3 Binding (Randall, Matthias, Bill)
* 2 Non-Binding (Arjun, Konstantine)

Cheers,
Almog

On Tue, Sep 17, 2019 at 11:56 AM Bill Bejeck  wrote:

> +1 (binding)
>
> On Tue, Sep 17, 2019 at 11:32 AM Matthias J. Sax 
> wrote:
>
> > +1 (binding)
> >
> > -Matthias
> >
> > On 9/17/19 12:20 AM, Arjun Satish wrote:
> > > Thanks for the KIP, Almog.
> > >
> > > +1 (non-binding)
> > >
> > > On Mon, Sep 16, 2019 at 4:45 PM Randall Hauch 
> wrote:
> > >
> > >> Thanks for the nice improvement, Almog!
> > >>
> > >> +1 (binding)
> > >>
> > >> Randall
> > >>
> > >> On Thu, Aug 15, 2019 at 11:59 AM Konstantine Karantasis <
> > >> konstant...@confluent.io> wrote:
> > >>
> > >>> Thanks Almog!
> > >>> Nicely designed and concise KIP.
> > >>>
> > >>> +1 non-binding
> > >>>
> > >>> Konstantine
> > >>>
> > >>> On Tue, Aug 6, 2019 at 11:44 PM Almog Gavra 
> > wrote:
> > >>>
> >  Hello Everyone,
> > 
> >  After discussions on
> > 
> > 
> > >>>
> > >>
> >
> https://lists.apache.org/thread.html/fa665a6dc59f73ca294a00bcbef2eaa3ad00cc69626e91c516fa4fca@%3Cdev.kafka.apache.org%3E
> >  I've opened this KIP up for formal voting.
> > 
> >  KIP:
> > 
> > 
> > >>>
> > >>
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-481%3A+SerDe+Improvements+for+Connect+Decimal+type+in+JSON
> > 
> >  Please update the DISCUSS thread with any concerns/comments.
> > 
> >  Cheers!
> >  Almog
> > 
> > >>>
> > >>
> > >
> >
> >
>

Re: [VOTE] KIP-481: SerDe Improvements for Connect Decimal type in JSON

[Bill Bejeck]

+1 (binding)

On Tue, Sep 17, 2019 at 11:32 AM Matthias J. Sax 
wrote:

> +1 (binding)
>
> -Matthias
>
> On 9/17/19 12:20 AM, Arjun Satish wrote:
> > Thanks for the KIP, Almog.
> >
> > +1 (non-binding)
> >
> > On Mon, Sep 16, 2019 at 4:45 PM Randall Hauch  wrote:
> >
> >> Thanks for the nice improvement, Almog!
> >>
> >> +1 (binding)
> >>
> >> Randall
> >>
> >> On Thu, Aug 15, 2019 at 11:59 AM Konstantine Karantasis <
> >> konstant...@confluent.io> wrote:
> >>
> >>> Thanks Almog!
> >>> Nicely designed and concise KIP.
> >>>
> >>> +1 non-binding
> >>>
> >>> Konstantine
> >>>
> >>> On Tue, Aug 6, 2019 at 11:44 PM Almog Gavra 
> wrote:
> >>>
>  Hello Everyone,
> 
>  After discussions on
> 
> 
> >>>
> >>
> https://lists.apache.org/thread.html/fa665a6dc59f73ca294a00bcbef2eaa3ad00cc69626e91c516fa4fca@%3Cdev.kafka.apache.org%3E
>  I've opened this KIP up for formal voting.
> 
>  KIP:
> 
> 
> >>>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-481%3A+SerDe+Improvements+for+Connect+Decimal+type+in+JSON
> 
>  Please update the DISCUSS thread with any concerns/comments.
> 
>  Cheers!
>  Almog
> 
> >>>
> >>
> >
>
>

Re: [Discuss] - KIP-532 - Add KStream#toTable to the Streams DSL

[Matthias J. Sax]

Aishwarya,

thanks for the KIP. Overall, I think it makes sense to allow converting
a KStream into a KTable.

From the KIP:

> materializing these KTables should only be allowed if the overloaded function 
> with Materialized is used (and if optimization is turned on it may still be 
> only logically materialized if the queryable name is not set).

Can you elaborate? I think the behavior we want should align with the
behavior of `StreamsBuilder#table()`.

From my understanding (correct me if I am wrong) it should be:

(1) If optimization is turned off, the KTable will always be
materialized, independent which method is used. The KTable will not be
queryable though.

(2) If optimization is turned on and if `toTable()` is used, the KTable
may or may not be materialized. For this case, even if the KTable is
materialized, the store would not be queryable.

(3) If `toTable(Materialized)` is use and a `storeName` or
`StoreSupplier` is specified, the store will always be materialized and
also be queryable. Otherwise, case (1) or (2) applies.



-Matthias


On 9/17/19 6:42 AM, aishwarya kumar wrote:
> Hi All,
> 
> Keeping this thread alive!!
> 
> The aim is to add two methods Kstream.toTable() &
> Kstream.toTable(Materialized), so users can choose to convert their
> event stream into a changelog stream at any stage.
> wiki link :
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-523:+Add+KStream%23toTable+to+the+Streams+DSL
> jira ticket : https://issues.apache.org/jira/browse/KAFKA-7658
> 
> Best,
> Aishwarya
> 
> On Fri, Sep 13, 2019 at 10:49 AM aishwarya kumar  wrote:
> 
>> Hello,
>>
>> Starting this thread to discuss KIP-532:
>> wiki link :
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-523:+Add+KStream%23toTable+to+the+Streams+DSL
>> jira ticket : https://issues.apache.org/jira/browse/KAFKA-7658
>>
>> There has been some discussion around the use-case of this KIP in the Jira
>> ticket.
>>
>> Regards,
>> Aishwarya
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [VOTE] KIP-481: SerDe Improvements for Connect Decimal type in JSON

[Matthias J. Sax]

+1 (binding)

-Matthias

On 9/17/19 12:20 AM, Arjun Satish wrote:
> Thanks for the KIP, Almog.
> 
> +1 (non-binding)
> 
> On Mon, Sep 16, 2019 at 4:45 PM Randall Hauch  wrote:
> 
>> Thanks for the nice improvement, Almog!
>>
>> +1 (binding)
>>
>> Randall
>>
>> On Thu, Aug 15, 2019 at 11:59 AM Konstantine Karantasis <
>> konstant...@confluent.io> wrote:
>>
>>> Thanks Almog!
>>> Nicely designed and concise KIP.
>>>
>>> +1 non-binding
>>>
>>> Konstantine
>>>
>>> On Tue, Aug 6, 2019 at 11:44 PM Almog Gavra  wrote:
>>>
 Hello Everyone,

 After discussions on


>>>
>> https://lists.apache.org/thread.html/fa665a6dc59f73ca294a00bcbef2eaa3ad00cc69626e91c516fa4fca@%3Cdev.kafka.apache.org%3E
 I've opened this KIP up for formal voting.

 KIP:


>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-481%3A+SerDe+Improvements+for+Connect+Decimal+type+in+JSON

 Please update the DISCUSS thread with any concerns/comments.

 Cheers!
 Almog

>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-479: Add Materialized to Join

[Matthias J. Sax]

Thanks Bill!

Using a new configuration object was suggested by John in the original
DISCUSS thread already. We rejected it because we wanted to avoid a new
configuration class. However, given your analysis, it seems it's
actually the right choice to introduce a new configuration class.

Hence, overall I am +1 on the proposal.


Some nits about names (as always :))

- `StreamJoined.as()` does not sound right for the `StoreSupplier`
overload. Maybe it's better to call that static method `with` (not 100%
sure)

- Should we use plural instead of singular -> `Stream{s}Joined`
  -> or keep singular but call it `StreamJoin`
  `Joined` seems to refer to the input stream(s) while `Join` would
refer to the join-operator

Thoughts?


You suggest to deprecate existing overload what I support -- can you
list deprecated method in the "Public Interface" section?



-Matthias



On 9/17/19 9:39 AM, Guozhang Wang wrote:
> Hello Bill,
> 
> Thanks for the updated KIP. I made a pass on the StreamJoined section. Just
> a quick question from user's perspective: if a user wants to provide a
> customized store-supplier, she is forced to also provide a name via the
> store-supplier. So there's no way to say "I want to provide my own store
> engine but let the library decide its name", is that right?
> 
> 
> Guozhang
> 
> 
> On Tue, Sep 17, 2019 at 8:53 AM Bill Bejeck  wrote:
> 
>> Bumping this discussion as we need to re-vote before the KIP deadline.
>>
>> On Fri, Sep 13, 2019 at 10:29 AM Bill Bejeck  wrote:
>>
>>> Hi All,
>>>
>>> While working on the implementation of KIP-479, some issues came to light
>>> that the KIP as written won't work.  I have updated the KIP with a
>> solution
>>> I believe will solve the original problem as well as address the
>>> impediment to the initial approach.
>>>
>>> This update is a significant change, so please review the updated KIP
>>>
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-479%3A+Add+StreamJoined+config+object+to+Join
>> and
>>> provide feedback.  After we conclude the discussion, there will be a
>>> re-vote.
>>>
>>> Thanks!
>>> Bill
>>>
>>> On Wed, Jul 17, 2019 at 7:01 PM Guozhang Wang 
>> wrote:
>>>
 Hi Bill, thanks for your explanations. I'm on board with your decision
 too.


 Guozhang

 On Wed, Jul 17, 2019 at 10:20 AM Bill Bejeck  wrote:

> Thanks for the response, John.
>
>> If I can offer my thoughts, it seems better to just document on the
>> Stream join javadoc for the Materialized parameter that it will not
>> make the join result queriable. I'm not opposed to the queriable
>> flag
>> in general, but introducing it is a much larger consideration that
>> has
>> previously derailed this KIP discussion. In the interest of just
>> closing the gap and keeping the API change small, it seems better to
>> just go with documentation for now.
>
> I agree with your statement here.  IMHO the most important goal of
>> this
 KIP
> is to not breaking existing users and gain some consistency of the
>> API.
>
> I'll update the KIP accordingly.
>
> -Bill
>
> On Tue, Jul 16, 2019 at 11:55 AM John Roesler 
 wrote:
>
>> Hi Bill,
>>
>> Thanks for driving this KIP toward a conclusion. I'm on board with
>> your decision.
>>
>> You didn't mention whether you're still proposing to add the
>> "queriable" flag to the Materialized config object, or just document
>> that a Stream join is never queriable. Both options have come up
>> earlier in the discussion, so it would be good to pin this down.
>>
>> If I can offer my thoughts, it seems better to just document on the
>> Stream join javadoc for the Materialized parameter that it will not
>> make the join result queriable. I'm not opposed to the queriable
>> flag
>> in general, but introducing it is a much larger consideration that
>> has
>> previously derailed this KIP discussion. In the interest of just
>> closing the gap and keeping the API change small, it seems better to
>> just go with documentation for now.
>>
>> Thanks again,
>> -John
>>
>> On Thu, Jul 11, 2019 at 2:45 PM Bill Bejeck 
 wrote:
>>>
>>> Thanks all for the great discussion so far.
>>>
>>> Everyone has made excellent points, and I appreciate the detail
> everyone
>>> has put into their arguments.
>>>
>>> However, after carefully evaluating all the points made so far,
> creating
>> an
>>> overload with Materialized is still my #1 option.
>>> My reasoning for saying so is two-fold:
>>>
>>>1. It's a small change, and IMHO since it's consistent with our
>> current
>>>API concerning state store usage, the cognitive load on users
 will
> be
>>>minimal.
>>>2. It achieves the most important goal of this KIP, namely to
 close
>> the
>>>gap of naming state stores indepe

Re: [DISCUSS] KIP-517: Add consumer metric indicating time between poll calls

[Manikumar]

Hi Kevin,

Thanks for the KIP.  LGTM. This will be useful.

Thanks,

On Mon, Sep 16, 2019 at 10:17 PM Harsha Chintalapani 
wrote:

> Thanks. +1 LGTM.
>
>
> On Mon, Sep 16, 2019 at 9:19 AM, Kevin Lu  wrote:
>
> > Hi Harsha,
> >
> > Thanks for the feedback. I have added *last-poll-seconds-ago* to the KIP
> > (being consistent with *last-heartbeat-seconds-ago*).
> >
> > Regards,
> > Kevin
> >
> > On Sat, Sep 14, 2019 at 9:44 AM Harsha Chintalapani 
> > wrote:
> >
> > Thanks Kevin for the KIP. Overall LGTM.
> > On you second point, I think the metric will be really useful to indicate
> > the perf bottlenecks on user code vs kakfa consumer/broker.
> >
> > Thanks,
> > Harsha
> >
> > On Fri, Sep 13, 2019 at 2:41 PM, Kevin Lu  wrote:
> >
> > Hi Radai & Jason,
> >
> > Thanks for the support and suggestion.
> >
> > 1. I think ratio is a good additional metric since the current proposed
> > metrics are only absolute times which may not be useful in all scenarios.
> >
> > I have added this to the KIP:
> > * - poll-idle-ratio*: The fraction of time the consumer spent waiting for
> > the user to process records from poll.
> >
> > Thoughts on the metric name/description?
> >
> > 2. Would it be useful to include a metric measuring the time since poll
> > was last called? Similar to *heartbeat-last-seconds-ago*, it would be
> > *poll-last-ms-ago.
> > *This could be useful if (1) the user has a very high *max.poll.interval.
> > ms
> > * configured and typically spends a long
> > time processing, or (2) comparing this metric with others such as
> > *heartbeat-last-seconds-ago* or something else for gathering data in root
> > cause analyses (or identifying potential consumer bugs related to poll).
> >
> > Regards,
> > Kevin
> >
> > On Fri, Sep 13, 2019 at 10:39 AM Jason Gustafson 
> > wrote:
> >
> > Hi Kevin,
> >
> > This looks reasonable to me. I'd also +1 Radai's suggestion if you're
> > willing. Something like an idle ratio for the consumer would be helpful.
> >
> > Thanks,
> > Jason
> >
> > On Fri, Sep 13, 2019 at 10:08 AM radai 
> > wrote:
> >
> > while youre at it another metric that we have found to be useful is %
> >
> > time
> >
> > spent in user code vs time spent in poll() (so time between poll calls /
> > time inside poll calls) - the higher the % value the more indicative of
> > user code being the cause of performance bottlenecks.
> >
> > On Fri, Sep 13, 2019 at 9:14 AM Kevin Lu  wrote:
> >
> > Hi All,
> >
> > Happy Friday! Bumping this. Any thoughts?
> >
> > Thanks.
> >
> > Regards,
> > Kevin
> >
> > On Thu, Sep 5, 2019 at 9:35 AM Kevin Lu  wrote:
> >
> > Hi All,
> >
> > I'd like to propose a new consumer metric that measures the time
> >
> > between
> >
> > calls to poll() for use in issues related to hitting
> >
> > max.poll.interval.ms
> >
> > due to long processing time.
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/
> > KIP-517%3A+Add+consumer+metric+indicating+time+between+poll+calls
> >
> > Please give it a read and let me know what you think.
> >
> > Thanks!
> >
> > Regards,
> > Kevin
> >
> >
>

Re: [VOTE] KIP-495: Dynamically Adjust Log Levels in Connect

[Randall Hauch]

+1 (binding).

Thanks for adding this, Arjun!

On Thu, Sep 12, 2019 at 11:58 AM Gwen Shapira  wrote:

> +1 (binding) - Thank you. Love the new REST API for logs.
>
> On Tue, Aug 20, 2019 at 3:05 PM Konstantine Karantasis
>  wrote:
> >
> > Thanks for the updates on the KIP Arjun!
> >
> > +1 by me (non-binding)
> >
> > Konstantine
> >
> >
> > On Wed, Aug 14, 2019 at 6:46 AM Konstantine Karantasis <
> > konstant...@confluent.io> wrote:
> >
> > >
> > > Thanks for the KIP Arjun.
> > >
> > > FYI, I left a few comments on the discussion thread, but mentioning
> here
> > > too since I noticed that the vote started a few hours ago.
> > >
> > > Konstantine
> > >
> > >
> > > On Wed, Aug 14, 2019 at 12:13 AM Cyrus Vafadari 
> > > wrote:
> > >
> > >> I am excited to see this implemented +1 nonbinding
> > >>
> > >> On Tue, Aug 13, 2019 at 2:01 PM Chris Egerton 
> > >> wrote:
> > >>
> > >> > Nice stuff, Arjun! +1 (non-binding)
> > >> >
> > >> > On Tue, Aug 13, 2019 at 1:55 PM Arjun Satish <
> arjun.sat...@gmail.com>
> > >> > wrote:
> > >> >
> > >> > > Hey everyone,
> > >> > >
> > >> > > I'd like to start a vote for KIP-495 (
> > >> > >
> > >> > >
> > >> >
> > >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-495%3A+Dynamically+Adjust+Log+Levels+in+Connect
> > >> > > ).
> > >> > > This change will make Connect easier to debug in production
> > >> environment.
> > >> > >
> > >> > > Based on the discussion, I updated the KIP to reflect how Connect
> will
> > >> > use
> > >> > > the changes to the log4j controller to initialize its mBean.
> > >> > >
> > >> > > For your reference, this is the discussion thread
> > >> > > https://www.mail-archive.com/dev@kafka.apache.org/msg99656.html
> > >> > >
> > >> > > Thanks in advance,
> > >> > > Arjun
> > >> > >
> > >> >
> > >>
> > >
>

Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

[Chris Egerton]

Hi Randall,

I've added your suggested paragraph to the KIP; it definitely clarifies the
intended behavior more than the content that it replaced. Thanks!

As far as I can tell, the two items remaining open for discussion are the
names for the new configs (which should be made less REST
request-specific), and whether separate configs should be enabled for the
key generation and MAC signing algorithms. As soon as I've gotten some
research done on the latter, I'll report back and then we can hopefully
also begin discussing the former.

Cheers,

Chris

On Mon, Sep 16, 2019 at 4:28 PM Randall Hauch  wrote:

> On Mon, Sep 16, 2019 at 3:06 PM Chris Egerton  wrote:
>
> > Hi Randall,
> >
> > The new default value for the key size configuration will be "null". I've
> > clarified this in the KIP. This will still preserve backwards
> compatibility
> > and should not be an issue.
> >
>
> Thanks for clarifying this in the KIP.
>
>
> >
> > I understand your point about key generation vs MAC signing algorithms;
> > like I said, I'll need to do more research.
> >
> > I respectfully disagree that a single algorithm list would be easier to
> > understand as opposed to a list of accepted algorithms and a signature
> > algorithm. Placing special priority on the first element in that list is
> > fairly implicit and leaves room for confusion where users might have the
> > same algorithms in their lists for that config but in different orders
> for
> > different workers. The group coordination protocol selection behavior
> isn't
> > really a great example since users don't configure that directly
> > themselves.
> >
> > RE the proposed set of new configs: like I stated in my previous
> response,
> > I object to the use of "cluster." as a configuration prefix for any
> worker
> > configs: "most configurations deal with the worker and, transitively, the
> > cluster; there doesn't seem to be good enough cause for including that
> > prefix for these configurations." I also think this discussion should
> > probably continue a little more before we start proposing new
> configuration
> > names, given that it's still undecided which configurations we're going
> to
> > expose.
> >
> > > I don't think we have any data about how how often a follower will be
> > fully caught up, and it's possible that a worker's consumer fails to keep
> > the worker caught up quickly enough with the configs topic and the new
> > session key. So can we really say that a follower making a request with
> an
> > expired key will be rare?
> > It would depend on the key rotation interval, but I can't imagine the
> > likelihood of this occurring with an interval as low as even 10 minutes
> > would be that high. The config topic is low-volume; the consumer for that
> > topic isn't going to be flooded with writes and it seems fine to expect
> > fairly low latency for the consumer of this topic.
> >
>
> My concern is that we're really *assuming* it's not a problem. All I'm
> asking for is more clarity on what happens when a worker doesn't know about
> the new session key when it makes a request to this REST resource? The KIP
> makes it clear that It will be retried and that the existing error message
> will be replaced with a debug message, at least for a time being. Perhaps
> the KIP paragraph that talks about this can be reworded to make this more
> clear, something akin to:
>
> "The leader will only accept requests signed with the most current key.
> However, Connect follower workers may routinely experience small delays
> when reading the new key. Rather than always logging such task
> configuration failure and retry attempts as errors (the current behavior),
> Connect's distributed herder will be modified slightly to handle such HTTP
> 403 responses for this task configuration request by quietly retrying them
> with the latest key for up to 1 minute. If failures persist for more than 1
> minute, they will be logged as errors."
>
>
> > > Can we not retry for longer than 1 second if the request fails with
> HTTP
> > 403? I'm concerned what the UX will be if/when this happens, and that the
> > user sees a very obtuse and seemingly unrelated error message they won't
> > know how to fix.
> > To be clear, the KIP doesn't propose any changes to the infinite retry
> > logic that's present in the worker today. All that the KIP proposes is
> that
> > an existing error-level message be replaced with a debug-level message if
> > it's suspected that a request has failed due to an out-of-date key. With
> > that out of the way, sure, we can bump the grace period before beginning
> to
> > emit error-level log messages. I think going as high as minute might be
> > acceptable; we should try to stay fairly low, however, in case the
> request
> > failures are due to some other reason that should be surfaced as soon as
> > possible and with some urgency.
> >
>
> Ack. See above .
>
> >
> > > The text in the "Failure to relay task configurations to leader due to
> > incorrect configu

Re: [DISCUSS] KIP-479: Add Materialized to Join

[Guozhang Wang]

Hello Bill,

Thanks for the updated KIP. I made a pass on the StreamJoined section. Just
a quick question from user's perspective: if a user wants to provide a
customized store-supplier, she is forced to also provide a name via the
store-supplier. So there's no way to say "I want to provide my own store
engine but let the library decide its name", is that right?


Guozhang


On Tue, Sep 17, 2019 at 8:53 AM Bill Bejeck  wrote:

> Bumping this discussion as we need to re-vote before the KIP deadline.
>
> On Fri, Sep 13, 2019 at 10:29 AM Bill Bejeck  wrote:
>
> > Hi All,
> >
> > While working on the implementation of KIP-479, some issues came to light
> > that the KIP as written won't work.  I have updated the KIP with a
> solution
> > I believe will solve the original problem as well as address the
> > impediment to the initial approach.
> >
> > This update is a significant change, so please review the updated KIP
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-479%3A+Add+StreamJoined+config+object+to+Join
> and
> > provide feedback.  After we conclude the discussion, there will be a
> > re-vote.
> >
> > Thanks!
> > Bill
> >
> > On Wed, Jul 17, 2019 at 7:01 PM Guozhang Wang 
> wrote:
> >
> >> Hi Bill, thanks for your explanations. I'm on board with your decision
> >> too.
> >>
> >>
> >> Guozhang
> >>
> >> On Wed, Jul 17, 2019 at 10:20 AM Bill Bejeck  wrote:
> >>
> >> > Thanks for the response, John.
> >> >
> >> > > If I can offer my thoughts, it seems better to just document on the
> >> > > Stream join javadoc for the Materialized parameter that it will not
> >> > > make the join result queriable. I'm not opposed to the queriable
> flag
> >> > > in general, but introducing it is a much larger consideration that
> has
> >> > > previously derailed this KIP discussion. In the interest of just
> >> > > closing the gap and keeping the API change small, it seems better to
> >> > > just go with documentation for now.
> >> >
> >> > I agree with your statement here.  IMHO the most important goal of
> this
> >> KIP
> >> > is to not breaking existing users and gain some consistency of the
> API.
> >> >
> >> > I'll update the KIP accordingly.
> >> >
> >> > -Bill
> >> >
> >> > On Tue, Jul 16, 2019 at 11:55 AM John Roesler 
> >> wrote:
> >> >
> >> > > Hi Bill,
> >> > >
> >> > > Thanks for driving this KIP toward a conclusion. I'm on board with
> >> > > your decision.
> >> > >
> >> > > You didn't mention whether you're still proposing to add the
> >> > > "queriable" flag to the Materialized config object, or just document
> >> > > that a Stream join is never queriable. Both options have come up
> >> > > earlier in the discussion, so it would be good to pin this down.
> >> > >
> >> > > If I can offer my thoughts, it seems better to just document on the
> >> > > Stream join javadoc for the Materialized parameter that it will not
> >> > > make the join result queriable. I'm not opposed to the queriable
> flag
> >> > > in general, but introducing it is a much larger consideration that
> has
> >> > > previously derailed this KIP discussion. In the interest of just
> >> > > closing the gap and keeping the API change small, it seems better to
> >> > > just go with documentation for now.
> >> > >
> >> > > Thanks again,
> >> > > -John
> >> > >
> >> > > On Thu, Jul 11, 2019 at 2:45 PM Bill Bejeck 
> >> wrote:
> >> > > >
> >> > > > Thanks all for the great discussion so far.
> >> > > >
> >> > > > Everyone has made excellent points, and I appreciate the detail
> >> > everyone
> >> > > > has put into their arguments.
> >> > > >
> >> > > > However, after carefully evaluating all the points made so far,
> >> > creating
> >> > > an
> >> > > > overload with Materialized is still my #1 option.
> >> > > > My reasoning for saying so is two-fold:
> >> > > >
> >> > > >1. It's a small change, and IMHO since it's consistent with our
> >> > > current
> >> > > >API concerning state store usage, the cognitive load on users
> >> will
> >> > be
> >> > > >minimal.
> >> > > >2. It achieves the most important goal of this KIP, namely to
> >> close
> >> > > the
> >> > > >gap of naming state stores independently of the join operator
> >> name.
> >> > > >
> >> > > > Additionally, I agree with the points made by Matthias earlier (I
> >> > realize
> >> > > > there is some overlap here).
> >> > > >
> >> > > > >  - the main purpose of this KIP is to close the naming gap what
> we
> >> > > achieve
> >> > > > >  - we can allow people to use the new in-memory store
> >> > > > >  - we allow people to enable/disable caching
> >> > > > >  - we unify the API
> >> > > > >  - we decouple querying from naming
> >> > > > >  - it's a small API change
> >> > > >
> >> > > > Although it's not a perfect solution,  IMHO the positives of using
> >> > > > Materialize far outweigh the negatives, and from what we've
> >> discussed
> >> > so
> >> > > > far, anything we implement seems to involve an additional change
> >> down
> >> > the
> >> > > > roa

[jira] [Created] (KAFKA-8917) When performing a Left/Right-Join, pick the headers of the same side

[Jira]

Pascal Büttiker created KAFKA-8917:
--

 Summary: When performing a Left/Right-Join, pick the headers of 
the same side
 Key: KAFKA-8917
 URL: https://issues.apache.org/jira/browse/KAFKA-8917
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Pascal Büttiker


As described in KAFKA-7718, headers are promoted by the "triggering" record in 
stateful operations such as Joins. This was very confusing and we spent quite 
some time debugging this.

While we ideally have full control over this behaviour as like the KAFKA-7718 
proposes, I hope we can solve some of the randomness before this:
 * Inner-Join: Keep as is (use the headers of the triggering record)
 * Full-Join: Keep as is (use the headers of the triggering record)
 * Left-Join: *Always pick the headers of the left record.*
 * Right-Join: *Always pick the headers of the right record.*

This behaviour would solve the most pressing issues when dealing with headers 
in Kafka Streams.

*Motivation*:

We depend on headers, especially when dealing with tombstone records. There is 
no other way to store additional information. If we do not use tombstone 
records, all default Kafka Features around compacted topics and KTabels are no 
longer useable. We are able to use custom Transformers to generate the headers 
(basically patch in the missing header support in Kafka Streams), but as soon 
that we use Join/Aggregate we loose control over the headers.

 

 

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: [VOTE] KIP-496: Administrative API to delete consumer offsets

[David Jacot]

Hi all,

We haven't included the changes in the command line tool to support the new
API. Therefore,
I would like to amend the current KIP to cover the changes in the
`kafka-consumer-groups`
command line tool. The change is rather small and it does not need to add
any new arguments
to the command line tool. so it doesn't make sense to create a new KIP for
it.

*Proposed API*
kafka-consumer-groups.sh --bootstrap-server  --group
 --topic :
ex: --bootstrap-server localhost:9092 --group my-group --topic topic1
--topic topic2:0,1,2

When partitions not provided, all partitions are used.

What do you think?

Best,
David


On Fri, Sep 13, 2019 at 6:42 PM Colin McCabe  wrote:

> Hi David,
>
> Sounds good.
>
> best,
> Colin
>
>
> On Fri, Sep 13, 2019, at 04:45, David Jacot wrote:
> > Hi all,
> >
> > I would like to do another modification to the proposal. In the proposal,
> > the OffsetDeleteResponse
> > doesn't have a top level error field so I would like to add one. Many
> > errors concern the whole
> > group (e.g. GROUP_ID_NOT_FOUND) so it would be great to have a way to
> > communicate them
> > back to the client without having to set such errors for all the
> requested
> > partitions. It makes the
> > error handling on the client easier and cleaner.
> >
> > *Proposed API with the ErrorCode:*
> > {
> >   "apiKey": 47,
> >   "type": "response",
> >   "name": "OffsetDeleteResponse",
> >   "validVersions": "0",
> >   "fields": [
> > { "name": "ErrorCode", "type": "int16", "versions": "0+",
> >   "about": "The top-level error code, or 0 if there was no error." },
> > { "name": "ThrottleTimeMs",  "type": "int32",  "versions": "0+",
> > "ignorable": true,
> >   "about": "The duration in milliseconds for which the request was
> > throttled due to a quota violation, or zero if the request did not
> violate
> > any quota." },
> > { "name": "Topics", "type": "[]OffsetDeleteResponseTopic",
> "versions":
> > "0+",
> >   "about": "The responses for each topic.", "fields": [
> > { "name": "Name", "type": "string", "versions": "0+", "mapKey":
> > true,
> >   "about": "The topic name." },
> > { "name": "Partitions", "type":
> "[]OffsetDeleteResponsePartition",
> > "versions": "0+",
> >   "about": "The responses for each partition in the topic.",
> > "fields": [
> > { "name": "PartitionIndex", "type": "int32", "versions":
> "0+",
> > "mapKey": true,
> >   "about": "The partition index." },
> > { "name": "ErrorCode", "type": "int16", "versions": "0+",
> >   "about": "The error code, or 0 if there was no error." }
> >   ]
> > }
> >   ]
> > }
> >   ]
> > }
> >
> > I would like to know if there are any concerns or objections regarding
> this
> > change before updating the KIP.
> >
> > Best,
> > David
> >
> > On Wed, Sep 4, 2019 at 9:24 AM David Jacot  wrote:
> >
> > > Hi all,
> > >
> > > While implementing the KIP, I have realized that a new error code and
> > > exception is required to notify the caller that offsets of a topic can
> not
> > > be deleted because the group is actively subscribed to the topic.
> > >
> > > I would like to know if there are any concerns with these changes
> before
> > > updating the KIP.
> > >
> > > *Proposed API:*
> > > GROUP_SUBSCRIBED_TO_TOPIC(86, "The consumer group is actively
> subscribed
> > > to the topic", GroupSubscribedToTopicException::new);
> > >
> > > public class GroupSubscribedToTopicException extends ApiException {
> > > public GroupSubscribedToTopicException(String message) {
> > > super(message);
> > > }
> > > }
> > >
> > > Best,
> > > David
> > >
> > > On Fri, Aug 16, 2019 at 10:58 AM Mickael Maison <
> mickael.mai...@gmail.com>
> > > wrote:
> > >
> > >> +1 (non binding)
> > >> Thanks!
> > >>
> > >> On Thu, Aug 15, 2019 at 11:53 PM Colin McCabe 
> wrote:
> > >> >
> > >> > On Thu, Aug 15, 2019, at 11:47, Jason Gustafson wrote:
> > >> > > Hey Colin, I think deleting all offsets is equivalent to deleting
> the
> > >> > > group, which can be done with the `deleteConsumerGroups` api. I
> > >> debated
> > >> > > whether there should be a way to delete partitions for all
> > >> unsubscribed
> > >> > > topics, but I decided to start with a simple API.
> > >> >
> > >> > That's a fair point-- deleting the group covers the main use-case
> for
> > >> deleting all offsets.  So we might as well keep it simple for now.
> > >> >
> > >> > cheers,
> > >> > Colin
> > >> >
> > >> > >
> > >> > > I'm going to close this vote. The final result is +3 with myself,
> > >> Guozhang,
> > >> > > and Colin voting.
> > >> > >
> > >> > > -Jason
> > >> > >
> > >> > > On Tue, Aug 13, 2019 at 9:21 AM Colin McCabe 
> > >> wrote:
> > >> > >
> > >> > > > Hi Jason,
> > >> > > >
> > >> > > > Thanks for the KIP.
> > >> > > >
> > >> > > > Is there ever a desire to delete all the offsets for a given
> group?
> > >> > > > Should the protocol and tools support this?
> > >> > > >
> > >>

Re: [DISCUSS] KIP-479: Add Materialized to Join

[Bill Bejeck]

Bumping this discussion as we need to re-vote before the KIP deadline.

On Fri, Sep 13, 2019 at 10:29 AM Bill Bejeck  wrote:

> Hi All,
>
> While working on the implementation of KIP-479, some issues came to light
> that the KIP as written won't work.  I have updated the KIP with a solution
> I believe will solve the original problem as well as address the
> impediment to the initial approach.
>
> This update is a significant change, so please review the updated KIP
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-479%3A+Add+StreamJoined+config+object+to+Join
>  and
> provide feedback.  After we conclude the discussion, there will be a
> re-vote.
>
> Thanks!
> Bill
>
> On Wed, Jul 17, 2019 at 7:01 PM Guozhang Wang  wrote:
>
>> Hi Bill, thanks for your explanations. I'm on board with your decision
>> too.
>>
>>
>> Guozhang
>>
>> On Wed, Jul 17, 2019 at 10:20 AM Bill Bejeck  wrote:
>>
>> > Thanks for the response, John.
>> >
>> > > If I can offer my thoughts, it seems better to just document on the
>> > > Stream join javadoc for the Materialized parameter that it will not
>> > > make the join result queriable. I'm not opposed to the queriable flag
>> > > in general, but introducing it is a much larger consideration that has
>> > > previously derailed this KIP discussion. In the interest of just
>> > > closing the gap and keeping the API change small, it seems better to
>> > > just go with documentation for now.
>> >
>> > I agree with your statement here.  IMHO the most important goal of this
>> KIP
>> > is to not breaking existing users and gain some consistency of the API.
>> >
>> > I'll update the KIP accordingly.
>> >
>> > -Bill
>> >
>> > On Tue, Jul 16, 2019 at 11:55 AM John Roesler 
>> wrote:
>> >
>> > > Hi Bill,
>> > >
>> > > Thanks for driving this KIP toward a conclusion. I'm on board with
>> > > your decision.
>> > >
>> > > You didn't mention whether you're still proposing to add the
>> > > "queriable" flag to the Materialized config object, or just document
>> > > that a Stream join is never queriable. Both options have come up
>> > > earlier in the discussion, so it would be good to pin this down.
>> > >
>> > > If I can offer my thoughts, it seems better to just document on the
>> > > Stream join javadoc for the Materialized parameter that it will not
>> > > make the join result queriable. I'm not opposed to the queriable flag
>> > > in general, but introducing it is a much larger consideration that has
>> > > previously derailed this KIP discussion. In the interest of just
>> > > closing the gap and keeping the API change small, it seems better to
>> > > just go with documentation for now.
>> > >
>> > > Thanks again,
>> > > -John
>> > >
>> > > On Thu, Jul 11, 2019 at 2:45 PM Bill Bejeck 
>> wrote:
>> > > >
>> > > > Thanks all for the great discussion so far.
>> > > >
>> > > > Everyone has made excellent points, and I appreciate the detail
>> > everyone
>> > > > has put into their arguments.
>> > > >
>> > > > However, after carefully evaluating all the points made so far,
>> > creating
>> > > an
>> > > > overload with Materialized is still my #1 option.
>> > > > My reasoning for saying so is two-fold:
>> > > >
>> > > >1. It's a small change, and IMHO since it's consistent with our
>> > > current
>> > > >API concerning state store usage, the cognitive load on users
>> will
>> > be
>> > > >minimal.
>> > > >2. It achieves the most important goal of this KIP, namely to
>> close
>> > > the
>> > > >gap of naming state stores independently of the join operator
>> name.
>> > > >
>> > > > Additionally, I agree with the points made by Matthias earlier (I
>> > realize
>> > > > there is some overlap here).
>> > > >
>> > > > >  - the main purpose of this KIP is to close the naming gap what we
>> > > achieve
>> > > > >  - we can allow people to use the new in-memory store
>> > > > >  - we allow people to enable/disable caching
>> > > > >  - we unify the API
>> > > > >  - we decouple querying from naming
>> > > > >  - it's a small API change
>> > > >
>> > > > Although it's not a perfect solution,  IMHO the positives of using
>> > > > Materialize far outweigh the negatives, and from what we've
>> discussed
>> > so
>> > > > far, anything we implement seems to involve an additional change
>> down
>> > the
>> > > > road.
>> > > >
>> > > > If others are still strongly opposed to using Materialized, my other
>> > > > preferences would be
>> > > >
>> > > >1. Add a "withStoreName" to Joined.  Although I agree with
>> Guozhang
>> > > that
>> > > >having a parameter that only applies to one use-case would be
>> > clumsy.
>> > > >2. Add a String overload for naming the store, but this would be
>> my
>> > > >least favorite option as IMHO this seems to be a step backward
>> from
>> > > why we
>> > > >introduced configuration objects in the first place.
>> > > >
>> > > > Thanks,
>> > > > Bill
>> > > >
>> > > > On Thu, Jun 27, 2019 at 4:45 PM Matthias J. Sax <
>> matth...@confluen

Re: Requesting Permission

[Jun Rao]

Hi, Brian,

Thanks for your interest. Just gave you the wiki permission.

Jun

On Mon, Sep 16, 2019 at 7:10 PM Brian Byrne  wrote:

> Hello,
>
> I'm requesting permission to the Kafka Wiki, specifically to create a KIP.
> Wiki ID is 'bbyrne'.
>
> Thanks,
> Brian
>

Re: [DISCUSS] KIP-491: Preferred Leader Deprioritized List (Temporary Blacklist)

[Stanislav Kozlovski]

Hey Harsha,

> If we want to go with making this an option and providing a tool which
abstracts moving the broker to end preferred leader list , it needs to do
it for all the partitions that broker is leader for. As said in the above
comment a broker i.e leader for 1000 partitions we have to this for all the
partitions.  Instead of having a blacklist will help simplify this process
and we can provide monitoring/alerts on such list.

Sorry, I thought that part of the reasoning for not using reassignment was
to optimize the process.

> Do you mind shedding some light what issue you are talking to propose a
KIP for?


The issue I was talking about is the one I quoted in my previous reply. I
understand that you want to have a way of running a "shallow" replica of
sorts - one that is lacking the historical data but has (and continues to
replicate) the latest data. That is the goal of setting the last offsets
for all partitions in replication-offset-checkpoint, right?

Thanks,
Stanislav

On Mon, Sep 16, 2019 at 3:39 PM Satish Duggana 
wrote:

> Hi George,
> Thanks for explaining the usecase for topic level preferred leader
> blacklist. As I mentioned earlier, I am fine with broker level config
> for now.
>
> ~Satish.
>
>
> On Sat, Sep 7, 2019 at 12:29 AM George Li
>  wrote:
> >
> >  Hi,
> >
> > Just want to ping and bubble up the discussion of KIP-491.
> >
> > On a large scale of Kafka clusters with thousands of brokers in many
> clusters.  Frequent hardware failures are common, although the
> reassignments to change the preferred leaders is a workaround, it incurs
> unnecessary additional work than the proposed preferred leader blacklist in
> KIP-491, and hard to scale.
> >
> > I am wondering whether others using Kafka in a big scale running into
> same problem.
> >
> >
> > Satish,
> >
> > Regarding your previous question about whether there is use-case for
> TopicLevel preferred leader "blacklist",  I thought about one use-case:  to
> improve rebalance/reassignment, the large partition will usually cause
> performance/stability issues, planning to change the say the New Replica
> will start with Leader's latest offset(this way the replica is almost
> instantly in the ISR and reassignment completed), and put this partition's
> NewReplica into Preferred Leader "Blacklist" at the Topic Level config for
> that partition. After sometime(retention time), this new replica has caught
> up and ready to serve traffic, update/remove the TopicConfig for this
> partition's preferred leader blacklist.
> >
> > I will update the KIP-491 later for this use case of Topic Level config
> for Preferred Leader Blacklist.
> >
> >
> > Thanks,
> > George
> >
> > On Wednesday, August 7, 2019, 07:43:55 PM PDT, George Li <
> sql_consult...@yahoo.com> wrote:
> >
> >   Hi Colin,
> >
> > > In your example, I think we're comparing apples and oranges.  You
> started by outlining a scenario where "an empty broker... comes up...
> [without] any > leadership[s]."  But then you criticize using reassignment
> to switch the order of preferred replicas because it "would not actually
> switch the leader > automatically."  If the empty broker doesn't have any
> leaderships, there is nothing to be switched, right?
> >
> > Let me explained in details of this particular use case example for
> comparing apples to apples.
> >
> > Let's say a healthy broker hosting 3000 partitions, and of which 1000
> are the preferred leaders (leader count is 1000). There is a hardware
> failure (disk/memory, etc.), and kafka process crashed. We swap this host
> with another host but keep the same broker.id, when this new broker
> coming up, it has no historical data, and we manage to have the current
> last offsets of all partitions set in the replication-offset-checkpoint (if
> we don't set them, it could cause crazy ReplicaFetcher pulling of
> historical data from other brokers and cause cluster high latency and other
> instabilities), so when Kafka is brought up, it is quickly catching up as
> followers in the ISR.  Note, we have auto.leader.rebalance.enable
> disabled, so it's not serving any traffic as leaders (leader count = 0),
> even there are 1000 partitions that this broker is the Preferred Leader.
> >
> > We need to make this broker not serving traffic for a few hours or days
> depending on the SLA of the topic retention requirement until after it's
> having enough historical data.
> >
> >
> > * The traditional way using the reassignments to move this broker in
> that 1000 partitions where it's the preferred leader to the end of
> assignment, this is O(N) operation. and from my experience, we can't submit
> all 1000 at the same time, otherwise cause higher latencies even the
> reassignment in this case can complete almost instantly.  After  a few
> hours/days whatever, this broker is ready to serve traffic,  we have to run
> reassignments again to restore that 1000 partitions preferred leaders for
> this broker: O(N) operation.  then run preferred leader el

[jira] [Created] (KAFKA-8916) Unreliable kafka-reassign-partitions.sh affecting performance

[VinayKumar (Jira)]

VinayKumar created KAFKA-8916:
-

 Summary: Unreliable kafka-reassign-partitions.sh affecting 
performance
 Key: KAFKA-8916
 URL: https://issues.apache.org/jira/browse/KAFKA-8916
 Project: Kafka
  Issue Type: Task
  Components: admin, config
Affects Versions: 2.1.1
 Environment: CentOS 7
Reporter: VinayKumar


Currently I have 3 node kafka cluster, and I want to add 2 more nodes to make 
it 5 node cluster.
 *After adding the nodes to cluster, I need all the topic partitions to be 
evenly distributed across all the 5 nodes.
 **In the past, when I ran kafka-reassign-partitions.sh & 
kafka-preferred-replica-election.sh, it ran for very long time, hung & made the 
cluster unstable. So I'm afraid to use this method.


Can you please suggest the best & foolproof way to assign partitions among all 
the cluster nodes.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: [DISCUSS] KIP-495: Dynamically Adjust Log Levels in Connect

[Randall Hauch]

Looks great. Thanks again, Arjun!

On Tue, Sep 17, 2019 at 1:17 AM Arjun Satish  wrote:

> Answers inline
>
> On Mon, Sep 16, 2019 at 5:06 PM Randall Hauch  wrote:
>
> > Thanks for the updates, Arjun. If possible, it'd be great to have the KIP
> > clarify a few things:
> >
> > 1) IIUC, the loggers returned by the GET methods are only those loggers
> > that have been activated/used/set in the JVM. If this is the case, this
> > should be specified.
> >
>
> The GET methods should return all initialized loggers (ancestors and the
> ones created by runtime classes).
>
>
> >
> > 2) It's possible to set a log level on an ancestor of other loggers, so
> we
> > should also specify whether or not ancestors are included in the GET
> > responses. Doing so would be helpful, but might not be feasible since two
> > different descendants might have different log levels.
> >
>
> The ancestors are also specified in the GET responses. Updated the KIP to
> highlight this.
>
>
> >
> > Otherwise this looks good!
> >
> > Best regards,
> >
> > Randall
> >
> > On Mon, Sep 16, 2019 at 4:15 AM Arjun Satish 
> > wrote:
> >
> > > Good catch, Randall. Yes, we should be able to set the level of any
> > logger
> > > given its name. If this is an ancestor, then the levels of all child
> > > classes are updated. I updated the KIP to be more explicit about what
> > > loggers we can set, and how they affect child classes, if any.
> > >
> > > Let me know what you think.
> > >
> > > Best,
> > >
> > > On Thu, Sep 12, 2019 at 5:02 PM Randall Hauch 
> wrote:
> > >
> > > > Thanks for the KIP, Arjun. It's going to be really nice to be able to
> > set
> > > > the log levels dynamically, especially through the REST API.
> > > >
> > > > However, I think it's not clear what behavior the KIP is actually
> > > proposing
> > > > with respect to which loggers are exposed (just those that are used,
> or
> > > > common ancestors) and the behavior when I change the log level on a
> > > > specific logger (is just that logger affected, or are descendants
> > > affected,
> > > > too).
> > > >
> > > > For example, in a Log4J configuration file we can set the logger for
> > > > packages (e.g., `org.apache.kafka`, `org.apache.kafka.connect`, etc.)
> > or
> > > > classes (e.g., `org.apache.kafka.connect.runtime.WorkerSinkTask`).
> > > Really,
> > > > those are just conventions, because if the code created a logger for
> > the
> > > > "kafka.connect.worker" context then we could set that, too. So by
> > > > convention, the loggers map to Kafka classes and packages.
> > > >
> > > > But it's unclear what behavior the KIP is proposing. Are the
> > intermediate
> > > > loggers such as all packages exposed as loggers? If so, if I set the
> > > logger
> > > > on `org.apache.kafka.connect.runtime`, will this set the log level
> for
> > > all
> > > > loggers below this?
> > > >
> > > > My concern is that if the behavior is (a) only concrete classes,
> and/or
> > > (b)
> > > > setting a log level for a specific logger sets only that logger, then
> > > this
> > > > deviates from what our users are familiar with when setting the log
> > > levels
> > > > in the Log4J configuration files, and would be a difficult user
> > > experience
> > > > if I have to set 30+ loggers rather than 1 or 2.
> > > >
> > > > On Thu, Sep 12, 2019 at 1:04 PM Jason Gustafson 
> > > > wrote:
> > > >
> > > > > Thanks Arjun. +1
> > > > >
> > > > > On Thu, Sep 12, 2019 at 9:58 AM Gwen Shapira 
> > > wrote:
> > > > >
> > > > > > The new REST API for logger management looks great to me.
> > > > > >
> > > > > >
> > > > > > On Thu, Sep 12, 2019 at 8:36 AM Arjun Satish <
> > arjun.sat...@gmail.com
> > > >
> > > > > > wrote:
> > > > > > >
> > > > > > > Bumping this thread.
> > > > > > >
> > > > > > > If there are no further comments, please add your votes here:
> > > > > > >
> https://www.mail-archive.com/dev@kafka.apache.org/msg100313.html
> > > > > > >
> > > > > > > Thanks in advance,
> > > > > > > Arjun
> > > > > > >
> > > > > > > On Fri, Sep 6, 2019 at 4:22 PM Arjun Satish <
> > > arjun.sat...@gmail.com>
> > > > > > wrote:
> > > > > > >
> > > > > > > > Thanks a lot, Jason! Answers inline. I'll also modify the kip
> > to
> > > > make
> > > > > > > > these clear.
> > > > > > > >
> > > > > > > > On Fri, Sep 6, 2019 at 4:01 PM Jason Gustafson <
> > > ja...@confluent.io
> > > > >
> > > > > > wrote:
> > > > > > > >
> > > > > > > >> Hi Arjun,
> > > > > > > >>
> > > > > > > >> The updated KIP looks good. Just a couple questions:
> > > > > > > >>
> > > > > > > >> 1. Is the /admin endpoint on the normal listener by default?
> > If
> > > > not,
> > > > > > is
> > > > > > > >> there a way to have it use the same listener?
> > > > > > > >>
> > > > > > > >
> > > > > > > > Uses the normal listener by default.
> > > > > > > >
> > > > > > > >
> > > > > > > >> 2. Changes to logging configuration are not intended to be
> > > > > > persistent, is
> > > > > > > >> that right? Also, I assume changes only apply to the worker
> > 

Re: [Discuss] - KIP-532 - Add KStream#toTable to the Streams DSL

[aishwarya kumar]

Hi All,

Keeping this thread alive!!

The aim is to add two methods Kstream.toTable() &
Kstream.toTable(Materialized), so users can choose to convert their
event stream into a changelog stream at any stage.
wiki link :
https://cwiki.apache.org/confluence/display/KAFKA/KIP-523:+Add+KStream%23toTable+to+the+Streams+DSL
jira ticket : https://issues.apache.org/jira/browse/KAFKA-7658

Best,
Aishwarya

On Fri, Sep 13, 2019 at 10:49 AM aishwarya kumar  wrote:

> Hello,
>
> Starting this thread to discuss KIP-532:
> wiki link :
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-523:+Add+KStream%23toTable+to+the+Streams+DSL
> jira ticket : https://issues.apache.org/jira/browse/KAFKA-7658
>
> There has been some discussion around the use-case of this KIP in the Jira
> ticket.
>
> Regards,
> Aishwarya
>

Build failed in Jenkins: kafka-trunk-jdk8 #3906

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-8859: Expose built-in streams metrics version in

--
[...truncated 6.07 MB...]

org.apache.kafka.trogdor.common.WorkerUtilsTest > 
testCreateTopicsFailsIfAtLeastOneTopicExists STARTED

org.apache.kafka.trogdor.common.WorkerUtilsTest > 
testCreateTopicsFailsIfAtLeastOneTopicExists PASSED

org.apache.kafka.trogdor.common.WorkerUtilsTest > 
testCreatesOneTopicVerifiesOneTopic STARTED

org.apache.kafka.trogdor.common.WorkerUtilsTest > 
testCreatesOneTopicVerifiesOneTopic PASSED

org.apache.kafka.trogdor.common.WorkerUtilsTest > 
testCommonConfigOverwritesDefaultProps STARTED

org.apache.kafka.trogdor.common.WorkerUtilsTest > 
testCommonConfigOverwritesDefaultProps PASSED

org.apache.kafka.trogdor.common.WorkerUtilsTest > 
testClientConfigOverwritesBothDefaultAndCommonConfigs STARTED

org.apache.kafka.trogdor.common.WorkerUtilsTest > 
testClientConfigOverwritesBothDefaultAndCommonConfigs PASSED

org.apache.kafka.trogdor.common.WorkerUtilsTest > testVerifyTopics STARTED

org.apache.kafka.trogdor.common.WorkerUtilsTest > testVerifyTopics PASSED

org.apache.kafka.trogdor.common.WorkerUtilsTest > testExistingTopicsNotCreated 
STARTED

org.apache.kafka.trogdor.common.WorkerUtilsTest > testExistingTopicsNotCreated 
PASSED

org.apache.kafka.trogdor.common.WorkerUtilsTest > 
testGetMatchingTopicPartitionsCorrectlyMatchesExactTopicName STARTED

org.apache.kafka.trogdor.common.WorkerUtilsTest > 
testGetMatchingTopicPartitionsCorrectlyMatchesExactTopicName PASSED

org.apache.kafka.trogdor.common.WorkerUtilsTest > testCreateRetriesOnTimeout 
STARTED

org.apache.kafka.trogdor.common.WorkerUtilsTest > testCreateRetriesOnTimeout 
PASSED

org.apache.kafka.trogdor.common.WorkerUtilsTest > 
testExistingTopicsMustHaveRequestedNumberOfPartitions STARTED

org.apache.kafka.trogdor.common.WorkerUtilsTest > 
testExistingTopicsMustHaveRequestedNumberOfPartitions PASSED

org.apache.kafka.trogdor.common.WorkerUtilsTest > 
testAddConfigsToPropertiesAddsAllConfigs STARTED

org.apache.kafka.trogdor.common.WorkerUtilsTest > 
testAddConfigsToPropertiesAddsAllConfigs PASSED

org.apache.kafka.trogdor.common.WorkerUtilsTest > 
testGetMatchingTopicPartitionsCorrectlyMatchesTopics STARTED

org.apache.kafka.trogdor.common.WorkerUtilsTest > 
testGetMatchingTopicPartitionsCorrectlyMatchesTopics PASSED

org.apache.kafka.trogdor.common.WorkerUtilsTest > testCreateOneTopic STARTED

org.apache.kafka.trogdor.common.WorkerUtilsTest > testCreateOneTopic PASSED

org.apache.kafka.trogdor.common.TopologyTest > testAgentNodeNames STARTED

org.apache.kafka.trogdor.common.TopologyTest > testAgentNodeNames PASSED

org.apache.kafka.trogdor.common.JsonUtilTest > 
testObjectFromCommandLineArgument STARTED

org.apache.kafka.trogdor.common.JsonUtilTest > 
testObjectFromCommandLineArgument PASSED

org.apache.kafka.trogdor.common.JsonUtilTest > testOpenBraceComesFirst STARTED

org.apache.kafka.trogdor.common.JsonUtilTest > testOpenBraceComesFirst PASSED

org.apache.kafka.trogdor.common.StringFormatterTest > testDurationString STARTED

org.apache.kafka.trogdor.common.StringFormatterTest > testDurationString PASSED

org.apache.kafka.trogdor.common.StringFormatterTest > testDateString STARTED

org.apache.kafka.trogdor.common.StringFormatterTest > testDateString PASSED

org.apache.kafka.trogdor.common.StringFormatterTest > testPrettyPrintGrid 
STARTED

org.apache.kafka.trogdor.common.StringFormatterTest > testPrettyPrintGrid PASSED

org.apache.kafka.trogdor.common.JsonSerializationTest > 
testDeserializationDoesNotProduceNulls STARTED

org.apache.kafka.trogdor.common.JsonSerializationTest > 
testDeserializationDoesNotProduceNulls PASSED

org.apache.kafka.trogdor.common.StringExpanderTest > testExpansions STARTED

org.apache.kafka.trogdor.common.StringExpanderTest > testExpansions PASSED

org.apache.kafka.trogdor.common.StringExpanderTest > testNoExpansionNeeded 
STARTED

org.apache.kafka.trogdor.common.StringExpanderTest > testNoExpansionNeeded 
PASSED

org.apache.kafka.trogdor.basic.BasicPlatformTest > testCreateBasicPlatform 
STARTED

org.apache.kafka.trogdor.basic.BasicPlatformTest > testCreateBasicPlatform 
PASSED

org.apache.kafka.trogdor.coordinator.CoordinatorTest > testTaskRequest STARTED

org.apache.kafka.trogdor.coordinator.CoordinatorTest > testTaskRequest PASSED

org.apache.kafka.trogdor.coordinator.CoordinatorTest > testTaskDestruction 
STARTED

org.apache.kafka.trogdor.coordinator.CoordinatorTest > testTaskDestruction 
PASSED

org.apache.kafka.trogdor.coordinator.CoordinatorTest > testTasksRequestMatches 
STARTED

org.apache.kafka.trogdor.coordinator.CoordinatorTest > testTasksRequestMatches 
PASSED

org.apache.kafka.trogdor.coordinator.CoordinatorTest > 
testWorkersExitingAtDifferentTimes STARTED

org.apache.kafka.trogdor.coordinator.CoordinatorTest > 
testWorkersExitingAtDifferentTimes PASSED

org.apache.

Build failed in Jenkins: kafka-trunk-jdk11 #815

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-8859: Expose built-in streams metrics version in

--
[...truncated 6.08 MB...]
org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimestampToString STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testWithSchemaTimestampToString PASSED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testConfigInvalidFormat STARTED

org.apache.kafka.connect.transforms.TimestampConverterTest > 
testConfigInvalidFormat PASSED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testNullList STARTED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testNullList PASSED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testEmptyList STARTED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testEmptyList PASSED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testValidList STARTED

org.apache.kafka.connect.transforms.util.NonEmptyListValidatorTest > 
testValidList PASSED

org.apache.kafka.connect.transforms.ReplaceFieldTest > withSchema STARTED

org.apache.kafka.connect.transforms.ReplaceFieldTest > withSchema PASSED

org.apache.kafka.connect.transforms.ReplaceFieldTest > schemaless STARTED

org.apache.kafka.connect.transforms.ReplaceFieldTest > schemaless PASSED

org.apache.kafka.connect.transforms.TimestampRouterTest > defaultConfiguration 
STARTED

org.apache.kafka.connect.transforms.TimestampRouterTest > defaultConfiguration 
PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeDateRecordValueWithSchemaString STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeDateRecordValueWithSchemaString PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessBooleanTrue STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessBooleanTrue PASSED

org.apache.kafka.connect.transforms.CastTest > testConfigInvalidTargetType 
STARTED

org.apache.kafka.connect.transforms.CastTest > testConfigInvalidTargetType 
PASSED

org.apache.kafka.connect.transforms.CastTest > 
testConfigMixWholeAndFieldTransformation STARTED

org.apache.kafka.connect.transforms.CastTest > 
testConfigMixWholeAndFieldTransformation PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessFloat32 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessFloat32 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessFloat64 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessFloat64 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessUnsupportedType STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessUnsupportedType PASSED

org.apache.kafka.connect.transforms.CastTest > castWholeRecordKeySchemaless 
STARTED

org.apache.kafka.connect.transforms.CastTest > castWholeRecordKeySchemaless 
PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaFloat32 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaFloat32 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaFloat64 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaFloat64 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaString STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaString PASSED

org.apache.kafka.connect.transforms.CastTest > testConfigEmpty STARTED

org.apache.kafka.connect.transforms.CastTest > testConfigEmpty PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt16 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt16 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt32 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt32 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt64 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueSchemalessInt64 PASSED

org.apache.kafka.connect.transforms.CastTest > castFieldsSchemaless STARTED

org.apache.kafka.connect.transforms.CastTest > castFieldsSchemaless PASSED

org.apache.kafka.connect.transforms.CastTest > testUnsupportedTargetType STARTED

org.apache.kafka.connect.transforms.CastTest > testUnsupportedTargetType PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaInt16 STARTED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValueWithSchemaInt16 PASSED

org.apache.kafka.connect.transforms.CastTest > 
castWholeRecordValu

RE: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

[Pellerin, Clement]

Good point about the two callers of SslFactory. We can move the SslEngine 
validation to a separate class and call it in both places. That SslEngine 
validation class would not be part of the public API and therefore we don't 
need to fuss about its API.
 
-Original Message-
From: Maulin Vasavada [mailto:maulin.vasav...@gmail.com] 
Sent: Tuesday, September 17, 2019 2:28 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-519: Make SSL context/engine configuration extensible

Hi Clement/Rajini

When I read your responses - I swing between both of your suggestions :) I
see both of your points. Let me ponder little bit more and give me take in
a day or so.

I tend to agree with Clement in a sense that we need to define clear
responsibilities of classes. Right now I feel it's not clear. Also, I tend
to agree to both of you about keystore/truststore validation - the conflict
I've to propose a clean agreeable solution to.

One clarification to Clement is - there are two classes using SslFactory
today - SslChannelBuilder and SaslChannelBuilder so we have to keep that in
mind. However, once we have clear responsibilities of classes, that should
automatically clear what goes where.

Thanks
Maulin

Re: [DISCUSS] KIP-525 - Return topic metadata and configs in CreateTopics response

[Rajini Sivaram]

Hi all,

Since this is minor KIP, I will start vote tomorrow if there are no
concerns.

Thank you,

Rajini

On Fri, Sep 13, 2019 at 10:17 PM Rajini Sivaram 
wrote:

> Hi all,
>
> I would like to start discussion on KIP-525 to return topic configs in
> CreateTopics response:
>
>-
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-525+-+Return+topic+metadata+and+configs+in+CreateTopics+response
>
>
> When validateOnly=false, this will be the actual configs of the created
> config. If validateOnly=true, this will be the configs with which the topic
> would have been created. This provides an alternative to
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-234%3A+add+support+for+getting+topic+defaults+from+AdminClient
> .
>
> Comments and suggestions welcome.
>
> Thank you,
>
> Rajini
>
>

[jira] [Created] (KAFKA-8915) 无法修改partition

[lingyi.zhong (Jira)]

lingyi.zhong created KAFKA-8915:
---

 Summary: 无法修改partition
 Key: KAFKA-8915
 URL: https://issues.apache.org/jira/browse/KAFKA-8915
 Project: Kafka
  Issue Type: Bug
Reporter: lingyi.zhong


[root@work1 kafka]# bin/kafka-topics.sh --create --zookeeper 10.20.30.77:2181 
--replication-factor 1 --partitions 1  --topic test_topic3[root@work1 kafka]# 
bin/kafka-topics.sh --create --zookeeper 10.20.30.77:2181 --replication-factor 
1 --partitions 1  --topic test_topic3

WARNING: Due to limitations in metric names, topics with a period ('.') or 
underscore ('_') could collide. To avoid issues it is best to use either, but 
not both.Created topic "test_topic3".[root@work1 kafka]# bin/kafka-topics.sh  
--alter --zookeeper 10.20.30.78:2181/chroot  --partition 2 --topic test_topic3

Exception in thread "main" joptsimple.UnrecognizedOptionException: partition is 
not a recognized option at 
joptsimple.OptionException.unrecognizedOption(OptionException.java:108) at 
joptsimple.OptionParser.handleLongOptionToken(OptionParser.java:510) at 
joptsimple.OptionParserState$2.handleArgument(OptionParserState.java:56) at 
joptsimple.OptionParser.parse(OptionParser.java:396) at 
kafka.admin.TopicCommand$TopicCommandOptions.(TopicCommand.scala:358) at 
kafka.admin.TopicCommand$.main(TopicCommand.scala:44) at 
kafka.admin.TopicCommand.main(TopicCommand.scala)



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: [DISCUSS] KIP-511: Collect and Expose Client's Name and Version in the Brokers

[David Jacot]

Hey Jose,

Yes, we have considered it. Check "Put clientName and clientVersion in the
RequestHeader" in the rejected alternatives.

Best,
David

On Tue, Sep 17, 2019 at 12:57 AM Jose Armando Garcia Sancio <
jsan...@confluent.io> wrote:

> On Mon, Sep 9, 2019 at 4:00 PM Colin McCabe  wrote:
>
> >
> > One solution to consider is adding the clientVersion and clientType to
> the
> > request header as optional (tagged) fields.  This would let us skip the
> > extra round trip.  I don't think it's that much more messy than having a
> > separate request type to set the client version and type.  In both cases,
> > you have to handle connections that set the version later than others, or
> > don't set the version at all (for compatibility).  So the version/type
> has
> > to be mutable and added after the TCP connection itself is established.
>
>
> Hey David,
>
> Did we consider Colin's suggestion of adding this information to every
> request using tagged field? If so can we add a section to the KIP
> documenting the decision?
>
> The HTTP protocol solves a similar problem by using a special header called
> User-Agent. In that field clients encode much more information than just
> client version and type. For example Mozilla uses this to include platform
> version and engine version. E.g.
>
> User-Agent: Mozilla/ () 
> () 
>
> https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/User-Agent
>
> Thanks!
>
> -Jose
>

Re: [VOTE] KIP-481: SerDe Improvements for Connect Decimal type in JSON

[Arjun Satish]

Thanks for the KIP, Almog.

+1 (non-binding)

On Mon, Sep 16, 2019 at 4:45 PM Randall Hauch  wrote:

> Thanks for the nice improvement, Almog!
>
> +1 (binding)
>
> Randall
>
> On Thu, Aug 15, 2019 at 11:59 AM Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
> > Thanks Almog!
> > Nicely designed and concise KIP.
> >
> > +1 non-binding
> >
> > Konstantine
> >
> > On Tue, Aug 6, 2019 at 11:44 PM Almog Gavra  wrote:
> >
> > > Hello Everyone,
> > >
> > > After discussions on
> > >
> > >
> >
> https://lists.apache.org/thread.html/fa665a6dc59f73ca294a00bcbef2eaa3ad00cc69626e91c516fa4fca@%3Cdev.kafka.apache.org%3E
> > > I've opened this KIP up for formal voting.
> > >
> > > KIP:
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-481%3A+SerDe+Improvements+for+Connect+Decimal+type+in+JSON
> > >
> > > Please update the DISCUSS thread with any concerns/comments.
> > >
> > > Cheers!
> > > Almog
> > >
> >
>