Re: Subject: [VOTE] 2.4.1 RC0

[Eric Lalonde]

Hi,

I ran:
$  https://github.com/elalonde/kafka/blob/master/bin/verify-kafka-rc.sh 
 2.4.1 
https://home.apache.org/~bbejeck/kafka-2.4.1-rc0 


All checksums and signatures are good and all unit and integration tests that 
were executed passed successfully.

- Eric

> On Mar 2, 2020, at 6:39 PM, Bill Bejeck  wrote:
> 
> Hello Kafka users, developers and client-developers,
> 
> This is the first candidate for release of Apache Kafka 2.4.1.
> 
> This is a bug fix release and it includes fixes and improvements from 38
> JIRAs, including a few critical bugs.
> 
> Release notes for the 2.4.1 release:
> https://home.apache.org/~bbejeck/kafka-2.4.1-rc0/RELEASE_NOTES.html
> 
> *Please download, test and vote by Thursday, March 5, 9 am PT*
> 
> Kafka's KEYS file containing PGP keys we use to sign the release:
> https://kafka.apache.org/KEYS
> 
> * Release artifacts to be voted upon (source and binary):
> https://home.apache.org/~bbejeck/kafka-2.4.1-rc0/
> 
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/org/apache/kafka/
> 
> * Javadoc:
> https://home.apache.org/~bbejeck/kafka-2.4.1-rc0/javadoc/
> 
> * Tag to be voted upon (off 2.4 branch) is the 2.4.1 tag:
> https://github.com/apache/kafka/releases/tag/2.4.1-rc0
> 
> * Documentation:
> https://kafka.apache.org/24/documentation.html
> 
> * Protocol:
> https://kafka.apache.org/24/protocol.html
> 
> * Successful Jenkins builds for the 2.4 branch:
> Unit/integration tests: Links to successful unit/integration test build to
> follow
> System tests:
> https://jenkins.confluent.io/job/system-test-kafka/job/2.4/152/
> 
> 
> Thanks,
> Bill Bejeck

Re: [VOTE] 2.4.0 RC2

[Eric Lalonde]

> I have also shared a pull request for the changes I have made to the test
> script
> 
> https://github.com/elalonde/kafka/pull/2


Thanks, this has been merged. 

Re: [VOTE] 2.4.0 RC2

[Eric Lalonde]

Hi all,

I ran the following:

$ git clone https://github.com/elalonde/kafka 
 
$ ./kafka/bin/verifykafka-rc.sh 2.4.0 
https://home.apache.org/~manikumar/kafka-2.4.0-rc2 


- All sha1, sha256 and md5 sums look good
- pgp signatures look good
- unit tests pass
- Integration tests pass with the following issue:

<=> 12% EXECUTING [30m 21s]
<=> 12% EXECUTING [30m 22s]mpleted, 16 skipped
> IDLE
<=> 12% EXECUTING [30m 27s]
> IDLE
> IDLE
> IDLE
> :core:integrationTest > Executing test kafka.api.AuthorizerIntegrationTest
> IDLE
> IDLE


After 30 minutes the above test had not finished, seemed stalled, so I 
cancelled it. LMK if I am overly impatient.

Re: [kafka-clients] [VOTE] 2.4.0 RC0

[Eric Lalonde]

Hi all,

Also seeing repeated failures of kafka.api.SaslPlainPlaintextConsumerTest > 
testCoordinatorFailover:

kafka.api.SaslPlainPlaintextConsumerTest > testCoordinatorFailover FAILED
java.lang.AssertionError: expected: but 
was:


Given the nature of the error, it may be a timing issue in the test itself, but 
it is repeatable, so I thought I’d raise it. 

To reproduce:

$ ./gradlew integrationTest



> On Nov 18, 2019, at 11:02 AM, Eric Lalonde  wrote:
> 
> This test has been failing when executed from the command line. I have not 
> run this test in an IDE. 
> 
>> On Nov 18, 2019, at 6:16 AM, Bruno Cadonna > <mailto:br...@confluent.io>> wrote:
>> 
>> Hi,
>> 
>> ClientMetricsTest.shouldAddCommitIdMetric should only fail if executed
>> from an IDE. The test fails because the test expects a file on the
>> class path which is not there when the test is executed from the IDE,
>> but is there when the test is executed from gradle. I will try to fix
>> the test so that it can also be executed from the IDE.
>> 
>> Best,
>> Bruno
>> 
>> On Mon, Nov 18, 2019 at 6:51 AM Vahid Hashemian
>> mailto:vahid.hashem...@gmail.com>> wrote:
>>> 
>>> Thanks Manikumar for managing this release. Looking forward to it.
>>> 
>>> I built binary from the source and was able to successfully run the 
>>> quickstarts.
>>> 
>>> However, this streams unit test also fails for me constantly:
>>> 
>>> ClientMetricsTest. shouldAddCommitIdMetric
>>> 
>>> java.lang.AssertionError:
>>>  Unexpected method call 
>>> StreamsMetricsImpl.addClientLevelImmutableMetric("commit-id", "The version 
>>> control commit ID of the Kafka Streams client", INFO, "unknown"):
>>>StreamsMetricsImpl.addClientLevelImmutableMetric("commit-id", "The 
>>> version control commit ID of the Kafka Streams client", INFO, 
>>> and(not("unknown"), notNull())): expected: 1, actual: 0
>>> at 
>>> org.easymock.internal.MockInvocationHandler.invoke(MockInvocationHandler.java:44)
>>> at 
>>> org.easymock.internal.ObjectMethodsFilter.invoke(ObjectMethodsFilter.java:101)
>>> at 
>>> org.easymock.internal.ClassProxyFactory$MockMethodInterceptor.intercept(ClassProxyFactory.java:97)
>>>...
>>> 
>>> Thanks,
>>> --Vahid
>>> 
>>> On Thu, Nov 14, 2019 at 10:21 AM Manikumar >> <mailto:manikumar.re...@gmail.com>> wrote:
>>>> 
>>>> Hello Kafka users, developers and client-developers,
>>>> 
>>>> This is the first candidate for release of Apache Kafka 2.4.0.
>>>> There is work in progress for couple blockers PRs. I am publishing RC0 to 
>>>> avoid further delays in testing the release.
>>>> 
>>>> This release includes many new features, including:
>>>> - Allow consumers to fetch from closest replica
>>>> - Support for incremental cooperative rebalancing to the consumer 
>>>> rebalance protocol
>>>> - MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter replication 
>>>> engine
>>>> - New Java authorizer Interface
>>>> - Support for  non-key joining in KTable
>>>> - Administrative API for replica reassignment
>>>> - Sticky partitioner
>>>> - Return topic metadata and configs in CreateTopics response
>>>> - Securing Internal connect REST endpoints
>>>> - API to delete consumer offsets and expose it via the AdminClient.
>>>> 
>>>> Release notes for the 2.4.0 release:
>>>> https://home.apache.org/~manikumar/kafka-2.4.0-rc0/RELEASE_NOTES.html 
>>>> <https://home.apache.org/~manikumar/kafka-2.4.0-rc0/RELEASE_NOTES.html>
>>>> 
>>>> *** Please download, test  by  Thursday, November 20, 9am PT
>>>> 
>>>> Kafka's KEYS file containing PGP keys we use to sign the release:
>>>> https://kafka.apache.org/KEYS
>>>> 
>>>> * Release artifacts to be voted upon (source and binary):
>>>> https://home.apache.org/~manikumar/kafka-2.4.0-rc0/
>>>> 
>>>> * Maven artifacts to be voted upon:
>>>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>>> 
>>>> * Javadoc:
>>>> https://home.apache.org/~manikumar/kafka-2.4.0-rc0/javadoc/
>>>> 
>>>> * Tag to be voted upon (off 2.4 branch) is the 2.4.0 tag:
>>>> https://

Re: [kafka-clients] [VOTE] 2.4.0 RC0

[Eric Lalonde]

This test has been failing when executed from the command line. I have not run 
this test in an IDE. 

> On Nov 18, 2019, at 6:16 AM, Bruno Cadonna  wrote:
> 
> Hi,
> 
> ClientMetricsTest.shouldAddCommitIdMetric should only fail if executed
> from an IDE. The test fails because the test expects a file on the
> class path which is not there when the test is executed from the IDE,
> but is there when the test is executed from gradle. I will try to fix
> the test so that it can also be executed from the IDE.
> 
> Best,
> Bruno
> 
> On Mon, Nov 18, 2019 at 6:51 AM Vahid Hashemian
> mailto:vahid.hashem...@gmail.com>> wrote:
>> 
>> Thanks Manikumar for managing this release. Looking forward to it.
>> 
>> I built binary from the source and was able to successfully run the 
>> quickstarts.
>> 
>> However, this streams unit test also fails for me constantly:
>> 
>> ClientMetricsTest. shouldAddCommitIdMetric
>> 
>> java.lang.AssertionError:
>>  Unexpected method call 
>> StreamsMetricsImpl.addClientLevelImmutableMetric("commit-id", "The version 
>> control commit ID of the Kafka Streams client", INFO, "unknown"):
>>StreamsMetricsImpl.addClientLevelImmutableMetric("commit-id", "The 
>> version control commit ID of the Kafka Streams client", INFO, 
>> and(not("unknown"), notNull())): expected: 1, actual: 0
>> at 
>> org.easymock.internal.MockInvocationHandler.invoke(MockInvocationHandler.java:44)
>> at 
>> org.easymock.internal.ObjectMethodsFilter.invoke(ObjectMethodsFilter.java:101)
>> at 
>> org.easymock.internal.ClassProxyFactory$MockMethodInterceptor.intercept(ClassProxyFactory.java:97)
>>...
>> 
>> Thanks,
>> --Vahid
>> 
>> On Thu, Nov 14, 2019 at 10:21 AM Manikumar  wrote:
>>> 
>>> Hello Kafka users, developers and client-developers,
>>> 
>>> This is the first candidate for release of Apache Kafka 2.4.0.
>>> There is work in progress for couple blockers PRs. I am publishing RC0 to 
>>> avoid further delays in testing the release.
>>> 
>>> This release includes many new features, including:
>>> - Allow consumers to fetch from closest replica
>>> - Support for incremental cooperative rebalancing to the consumer rebalance 
>>> protocol
>>> - MirrorMaker 2.0 (MM2), a new multi-cluster, cross-datacenter replication 
>>> engine
>>> - New Java authorizer Interface
>>> - Support for  non-key joining in KTable
>>> - Administrative API for replica reassignment
>>> - Sticky partitioner
>>> - Return topic metadata and configs in CreateTopics response
>>> - Securing Internal connect REST endpoints
>>> - API to delete consumer offsets and expose it via the AdminClient.
>>> 
>>> Release notes for the 2.4.0 release:
>>> https://home.apache.org/~manikumar/kafka-2.4.0-rc0/RELEASE_NOTES.html
>>> 
>>> *** Please download, test  by  Thursday, November 20, 9am PT
>>> 
>>> Kafka's KEYS file containing PGP keys we use to sign the release:
>>> https://kafka.apache.org/KEYS
>>> 
>>> * Release artifacts to be voted upon (source and binary):
>>> https://home.apache.org/~manikumar/kafka-2.4.0-rc0/
>>> 
>>> * Maven artifacts to be voted upon:
>>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>> 
>>> * Javadoc:
>>> https://home.apache.org/~manikumar/kafka-2.4.0-rc0/javadoc/
>>> 
>>> * Tag to be voted upon (off 2.4 branch) is the 2.4.0 tag:
>>> https://github.com/apache/kafka/releases/tag/2.4.0-rc0
>>> 
>>> * Documentation:
>>> https://kafka.apache.org/24/documentation.html
>>> 
>>> * Protocol:
>>> https://kafka.apache.org/24/protocol.html
>>> 
>>> Thanks,
>>> Manikumar
>>> 
>>> --
>>> You received this message because you are subscribed to the Google Groups 
>>> "kafka-clients" group.
>>> To unsubscribe from this group and stop receiving emails from it, send an 
>>> email to kafka-clients+unsubscr...@googlegroups.com.
>>> To view this discussion on the web visit 
>>> https://groups.google.com/d/msgid/kafka-clients/CAMVt_Aw945uqcpisFjZHAR5m8Sidw6hW4ia%2B7%3DjxEfadmBPzcw%40mail.gmail.com.
>> 
>> 
>> 
>> --
>> 
>> Thanks!
>> --Vahid
>> 
>> --
>> You received this message because you are subscribed to the Google Groups 
>> "kafka-clients" group.
>> To unsubscribe from this group and stop receiving emails from it, send an 
>> email to kafka-clients+unsubscr...@googlegroups.com 
>> .
>> To view this discussion on the web visit 
>> https://groups.google.com/d/msgid/kafka-clients/CAHR2v2mKJtHG6S9P%3Dmw08SxbWjQCowp8cpZNpzr9acW1EcdegQ%40mail.gmail.com
>>  
>> .

Re: [VOTE] 2.4.0 RC0

[Eric Lalonde]

Hello,

- all PGP signatures are good
- All md5, sha1sums and sha512sums pass

2648 unit test pass, 1 failure (ran twice)

ClientsMetricsTest.shouldAddCommitIdMetric Failed:

Java.lang.AssertionError: 
  Unexpected method call 
StreamsMetricsImpl.addClientLevelImmutableMetric("commit-id", "The version 
control commit ID of the Kafka Streams client", INFO, "unknown"):
StreamsMetricsImpl.addClientLevelImmutableMetric("commit-id", "The version 
control commit ID of the Kafka Streams client", INFO, and(not("unknown"), 
notNull())): expected: 1, actual: 0
at 
org.easymock.internal.MockInvocationHandler.invoke(MockInvocationHandler.java:44)
at 
org.easymock.internal.ObjectMethodsFilter.invoke(ObjectMethodsFilter.java:101)
at 
org.easymock.internal.ClassProxyFactory$MockMethodInterceptor.intercept(ClassProxyFactory.java:97)
at 
org.apache.kafka.streams.processor.internals.metrics.StreamsMetricsImpl$$EnhancerByCGLIB$$43750e14.addClientLevelImmutableMetric()
at 
org.apache.kafka.streams.internals.metrics.ClientMetrics.addCommitIdMetric(ClientMetrics.java:80)
at 
org.apache.kafka.streams.internals.metrics.ClientMetricsTest.setUpAndVerifyMetricOneParam(ClientMetricsTest.java:116)
at 
org.apache.kafka.streams.internals.metrics.ClientMetricsTest.shouldAddCommitIdMetric(ClientMetricsTest.java:65)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
at 
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at 
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
at 
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
at 
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:365)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at 
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:330)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:78)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:328)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:65)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:292)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:305)
at org.junit.runners.ParentRunner.run(ParentRunner.java:412)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:110)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
at 
org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
at 
org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:62)
at 
org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:33)
at 
org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:94)
at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
at 
org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:118)
at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:36)
at 
org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
at 
org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$D

Re: Subject: [VOTE] 2.2.2 RC2

[Eric Lalonde]

Hello,

In an effort to assist in the verification of release candidates, I have 
authored the following quick-and-dirty utility to help people verify release 
candidate artifacts: 
https://github.com/elalonde/kafka/blob/master/bin/verify-kafka-rc.sh 
 . I have 
executed this script for 2.2.2 rc2 and everything looks good:
- all checksums verify
- all executed gradle commands succeed
- all unit and integration tests pass.

Hope this helps in the release of 2.2.2.

- Eric

> On Nov 5, 2019, at 7:55 AM, Randall Hauch  wrote:
> 
> Thanks, Mickael!
> 
> Anyone else get a chance to validate the 2.2.2 RC2 build? It'd be great to
> get this out the door.
> 
> Randall
> 
> On Tue, Nov 5, 2019 at 6:34 AM Mickael Maison 
> wrote:
> 
>> +1 (non binding)
>> I verified signatures, built it from source, ran unit tests and quickstart
>> 
>> 
>> 
>> On Fri, Oct 25, 2019 at 3:10 PM Randall Hauch  wrote:
>>> 
>>> Hello all, we identified around three dozen bug fixes, including an
>> update
>>> of a third party dependency, and wanted to release a patch release for
>> the
>>> Apache Kafka 2.2.0 release.
>>> 
>>> This is the *second* candidate for release of Apache Kafka 2.2.2. (RC1
>> did
>>> not include a fix for https://issues.apache.org/jira/browse/KAFKA-9053,
>> but
>>> the fix appeared before RC1 was announced so it was easier to just create
>>> RC2.)
>>> 
>>> Check out the release notes for a complete list of the changes in this
>>> release candidate:
>>> https://home.apache.org/~rhauch/kafka-2.2.2-rc2/RELEASE_NOTES.html
>>> 
>>> *** Please download, test and vote by Wednesday, October 30, 9am PT>
>>> 
>>> Kafka's KEYS file containing PGP keys we use to sign the release:
>>> https://kafka.apache.org/KEYS
>>> 
>>> * Release artifacts to be voted upon (source and binary):
>>> https://home.apache.org/~rhauch/kafka-2.2.2-rc2/
>>> 
>>> * Maven artifacts to be voted upon:
>>> https://repository.apache.org/content/groups/staging/org/apache/kafka/
>>> 
>>> * Javadoc:
>>> https://home.apache.org/~rhauch/kafka-2.2.2-rc2/javadoc/
>>> 
>>> * Tag to be voted upon (off 2.2 branch) is the 2.2.2 tag:
>>> https://github.com/apache/kafka/releases/tag/2.2.2-rc2
>>> 
>>> * Documentation:
>>> https://kafka.apache.org/22/documentation.html
>>> 
>>> * Protocol:
>>> https://kafka.apache.org/22/protocol.html
>>> 
>>> * Successful Jenkins builds for the 2.2 branch:
>>> Unit/integration tests: https://builds.apache.org/job/kafka-2.2-jdk8/1/
>>> System tests:
>>> https://jenkins.confluent.io/job/system-test-kafka/job/2.2/216/
>>> 
>>> /**
>>> 
>>> Thanks,
>>> 
>>> Randall Hauch
>> 

Re: Why do I get an IllegalStateException when accessing record metadata?

[Eric Lalonde]

> On Jan 2, 2019, at 6:31 AM, Matthias J. Sax  wrote:
> 
> Thanks for reporting this. Feel free to edit the Wiki with the FAQ directly.
> 
> What is unclear to me: what do you mean by "the state store [...] was
> errantly scoped to the TransformerProvider, not the Transformer" ?
> 
> I would like to understand the actual issue.

See this gist: 
https://gist.github.com/elalonde/5b7be53973b5b847bac86754077b1a80#file-gistfile1-txt-L4
 
<https://gist.github.com/elalonde/5b7be53973b5b847bac86754077b1a80#file-gistfile1-txt-L4>

Because MyStore is declared in the parent supplier, it will be shared across 
tasks, even though the .get() function is instantiating a new MyTransformer() 
for each task. It should have been declared in the MyTransformer sub-class 
(say, around, line 15).


> -Matthias
> 
> On 12/31/18 2:36 AM, Eric Lalonde wrote:
>> Recently I encountered IllegalStateExceptions in my KafkaTransformer. My
>> application was exhibiting all of the behavior as discussed in the FAQ
>> (https://cwiki.apache.org/confluence/display/KAFKA/FAQ), under the section:
>> 
>> Why do I get an IllegalStateException when accessing record metadata? 
>> 
>> However, the root cause of my problem was not a due to the lack of
>> instantiating a new transformer for each task. The root cause of my
>> mistake was a bit more insidious: the state store that I instantiated
>> was errantly scoped to the TransformerProvider, not the Transformer. Of
>> course when I finally realized my error, the problem was obvious, but it
>> was not immediately obvious. May I suggest extending the FAQ to help
>> others as well? Perhaps it would be helpful to extend the aforementioned
>> FAQ section in the following way (highlighted text is my addition):
>> 
>> 
>>Why do I get an IllegalStateException when accessing record metadata? 
>> 
>>If you attach a new |Processor/Transformer/ValueTransformer| to your
>>topology using a corresponding supplier, you need to make sure that
>>the supplier returns a /new/ instance each time |get()| is called.
>>If you return the same object, a
>>single |Processor/Transformer/ValueTransformer| would be shared over
>>multiple tasks resulting in an |IllegalStateException| with error
>>message |"This should not happen as topic() should only be called
>>while a record is processed"| (depending on the method you are
>>calling it could also be |partition()|, |offset()|,
>>or |timestamp()| instead of |topic()|). Additionally, all
>>instantiated state stores must be scoped to the inner
>>Processor/Transformer/ValueTransformer class, and not to the parent
>>Provider class. Scoping state stores to the parent class will result
>>in state store re-use across tasks, which will also result
>>in IllegalStateExceptions.
>> 
>> 
>> Hope this saves someone else from making the same mistake :)
>> 
>> - Eric
> 



smime.p7s
Description: S/MIME Cryptographic Signature

Re: Why do I get an IllegalStateException when accessing record metadata?

[Eric Lalonde]

Recently I encountered IllegalStateExceptions in my KafkaTransformer. My 
application was exhibiting all of the behavior as discussed in the FAQ 
(https://cwiki.apache.org/confluence/display/KAFKA/FAQ 
), under the section:

Why do I get an IllegalStateException when accessing record metadata? 

However, the root cause of my problem was not a due to the lack of 
instantiating a new transformer for each task. The root cause of my mistake was 
a bit more insidious: the state store that I instantiated was errantly scoped 
to the TransformerProvider, not the Transformer. Of course when I finally 
realized my error, the problem was obvious, but it was not immediately obvious. 
May I suggest extending the FAQ to help others as well? Perhaps it would be 
helpful to extend the aforementioned FAQ section in the following way 
(highlighted text is my addition):

Why do I get an IllegalStateException when accessing record metadata? 

If you attach a new Processor/Transformer/ValueTransformer to your topology 
using a corresponding supplier, you need to make sure that the supplier returns 
a new instance each time get() is called. If you return the same object, a 
single Processor/Transformer/ValueTransformer would be shared over multiple 
tasks resulting in an IllegalStateException with error message "This should not 
happen as topic() should only be called while a record is processed" (depending 
on the method you are calling it could also be partition(), offset(), or 
timestamp() instead of topic()). Additionally, all instantiated state stores 
must be scoped to the inner Processor/Transformer/ValueTransformer class, and 
not to the parent Provider class. Scoping state stores to the parent class will 
result in state store re-use across tasks, which will also result in 
IllegalStateExceptions.

Hope this saves someone else from making the same mistake :)

- Eric

smime.p7s
Description: S/MIME Cryptographic Signature

Re: Kafka Streams 0.11.0.1 - Task Assignment Stickiness Behavior

[Eric Lalonde]

Hi Guozhang, I will talk to the team about prioritizing the recreation of this 
scenario and gathering logs.

- Eric

> On Nov 5, 2017, at 9:39 AM, Guozhang Wang  wrote:
> 
> Hello Laven / Eric,
> 
> I personally cannot think of any more bug fixes than KAFKA-5152 that is
> trying to solve a relative issue, and we would love to know what did you
> observe that in 0.11.0.1 this issue still exists: i.e. the first instance
> is stuck with the rebalance state, as we have moved the restoration process
> completely out of the rebalance phase in 0.11.0.1. So as Matthias
> mentioned, if you still saw this issue in 0.11.0.1 it may indicate another
> issue.
> 
> Again, we are super interested to learn your scenario, and see if it is
> indeed the case.
> 
> Guozhang
> 
> On Sat, Nov 4, 2017 at 7:26 AM, Matthias J. Sax 
> wrote:
> 
>> Streams is backward compatible from 0.10.1 on.
>> 
>> So 1.0.0 does not work with 0.10.0 but 0.10.1, 0.10.2, and 0.11.0.
>> 
>> Compare:
>> https://docs.confluent.io/current/streams/upgrade-guide.html#compatibility
>> 
>> 
>> -Matthias
>> 
>> On 10/25/17 7:54 AM, Eric Lalonde wrote:
>>> 
>>>> On Oct 24, 2017, at 8:53 PM, Matthias J. Sax 
>> wrote:
>>>> 
>>>> Might be worth a try with 1.0.0 RC3 -- even if I doubt that much
>> changes.
>>>> 
>>>> Can you provide debug logs for your Kafka streams applications as well
>>>> as brokers? This would help to dig into this.
>>>> 
>>> 
>>> I searched, but it wasn’t clear to me — would a Kstream 1.0.0 RC3 client
>> be backwards compatible with a broker from the 0.10.x line?
>>> 
>>> 
>> 
>> 
> 
> 
> -- 
> -- Guozhang

Re: Kafka Streams 0.11.0.1 - Task Assignment Stickiness Behavior

[Eric Lalonde]

> On Oct 24, 2017, at 8:53 PM, Matthias J. Sax  wrote:
> 
> Might be worth a try with 1.0.0 RC3 -- even if I doubt that much changes.
> 
> Can you provide debug logs for your Kafka streams applications as well
> as brokers? This would help to dig into this.
> 

I searched, but it wasn’t clear to me — would a Kstream 1.0.0 RC3 client be 
backwards compatible with a broker from the 0.10.x line?

Re: Kafka Streams 0.11.0.1 - Task Assignment Stickiness Behavior

[Eric Lalonde]

>>> Could it be, that the first KafkaStreams instance was still in status
>>> "rebalancing" when you started the second/third container? If yes, this
>>> might explain what you observed: if the first instance is in status
>>> "rebalancing" it would miss that new instanced are joining the group.
>>> (We fixed this in upcoming 1.0.0 release).


Hi Matthias,

I appreciate your time. Your characterization above of the rebalancing is 
exactly the behavior we are seeing. Unfortunately, the issue still occurs in 
0.11.0.1, so I suspect the behavior you are describing is not fixed by commits 
expressed in KAFKA-5152, or not completely. 

- Eric

Re: Kafka Streams 0.11.0.1 - Task Assignment Stickiness Behavior

[Eric Lalonde]

> 
> Could it be, that the first KafkaStreams instance was still in status
> "rebalancing" when you started the second/third container? If yes, this
> might explain what you observed: if the first instance is in status
> "rebalancing" it would miss that new instanced are joining the group.
> (We fixed this in upcoming 1.0.0 release).

Is there a JIRA which tracks this fix? 

Thanks,

- Eric

Re: Kafka Streams Application crashing on Rebalance

[Eric Lalonde]

> On Aug 1, 2017, at 10:01 AM, Damian Guy  wrote:
> 
> Hi, Yes the issue is in 0.10.2 also.

Hi,

Any chance of a backport to 0.10.2.x? We have lots and lots of state stores. :)

Re: Kafka Streams Application crashing on Rebalance

[Eric Lalonde]

> On Aug 1, 2017, at 8:00 AM, Damian Guy  wrote:
> 
> It is a bug in 0.10.2 or lower. It has been fixed in 0.11 by
> https://issues.apache.org/jira/browse/KAFKA-4494

Hi Damien, the Affects Version is set to 0.10.1.0 in KAFKA-4494. Is the issue 
in 0.10.2.0 as well?

Re: Kafka Streams 0.10.2.1 client crash - .checkpoint.tmp (No such file or directory)

[Eric Lalonde]

Hello, I am able to reproduce this. It occurs during rebalancing when the 
service is restarted. kafka-clients and kafka-streams are both at version 
0.10.2.1. 3 instances of the service, 4 threads per instance, 100 partitions.

log excerpt:

 Wed Jul 26 05:32:07 UTC 2017
 Streams state: REBALANCING
 Num Stream Threads: 4

 2017-07-26 05:32:20.497 ERROR 7 --- [ StreamThread-1] 
o.a.k.s.p.internals.StreamThread : stream-thread [StreamThread-1] 
Failed to remove suspended task 2_68

 org.apache.kafka.streams.errors.ProcessorStateException: Error while closing 
the state manager
 at 
org.apache.kafka.streams.processor.internals.AbstractTask.closeStateManager(AbstractTask.java:133)
 ~[kafka-streams-0.10.2.1.jar!/:na]
 at 
org.apache.kafka.streams.processor.internals.StreamThread.closeNonAssignedSuspendedTasks(StreamThread.java:898)
 [kafka-streams-0.10.2.1.jar!/:na]
 at 
org.apache.kafka.streams.processor.internals.StreamThread.access$500(StreamThread.java:69)
 [kafka-streams-0.10.2.1.jar!/:na]
 at 
org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:233)
 [kafka-streams-0.10.2.1.jar!/:na]
 at 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:259)
 [kafka-clients-0.10.2.1.jar!/:na]
 at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:352)
 [kafka-clients-0.10.2.1.jar!/:na]
 at 
org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:303)
 [kafka-clients-0.10.2.1.jar!/:na]
 at 
org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:290)
 [kafka-clients-0.10.2.1.jar!/:na]
 at 
org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:1029)
 [kafka-clients-0.10.2.1.jar!/:na]
 at 
org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:995) 
[kafka-clients-0.10.2.1.jar!/:na]
 at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:592)
 [kafka-streams-0.10.2.1.jar!/:na]
 at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:361)
 [kafka-streams-0.10.2.1.jar!/:na]
 Caused by: java.io.FileNotFoundException: 
/home/kafka-streams/data/2_68/.checkpoint.tmp (No such file or directory)
 at java.io.FileOutputStream.open0(Native Method) ~[na:1.8.0_102]
 at java.io.FileOutputStream.open(FileOutputStream.java:270) ~[na:1.8.0_102]
 at java.io.FileOutputStream.(FileOutputStream.java:213) ~[na:1.8.0_102]
 at java.io.FileOutputStream.(FileOutputStream.java:162) ~[na:1.8.0_102]
 at 
org.apache.kafka.streams.state.internals.OffsetCheckpoint.write(OffsetCheckpoint.java:71)
 ~[kafka-streams-0.10.2.1.jar!/:na]
 at 
org.apache.kafka.streams.processor.internals.ProcessorStateManager.close(ProcessorStateManager.java:386)
 ~[kafka-streams-0.10.2.1.jar!/:na]
 at 
org.apache.kafka.streams.processor.internals.AbstractTask.closeStateManager(AbstractTask.java:131)
 ~[kafka-streams-0.10.2.1.jar!/:na]
 ... 11 common frames omitted


Interestingly, the directory 2_68 does not appear to exist on the instance on 
which this  exception was thrown: 

$ ls /home/kafka-streams/data/

0_10  0_13  0_18  0_23  0_27  0_3   0_33  0_35  0_39  0_43  0_46  0_50  0_52  
0_62  0_66  0_7   0_76  0_81  0_83  0_87  0_89  0_97  1_0  1_11  1_2   1_26  
1_31  1_35  1_39  1_41  1_5   1_54  1_56  1_63  1_65  1_69  1_73  1_77  1_79  
1_88  1_92  1_94  1_96  2_0  2_12  2_16  2_2   2_24  2_33  2_40  2_46  2_54  
2_60  2_67  2_71  2_76  2_80  2_86  2_91 0_11  0_14  0_21  0_24   0_29  0_30  
0_34  0_37  0_42  0_44  0_5   0_51  0_57  0_65  0_67  0_71  0_80  0_82  0_84  
0_88  0_91  0_99  1_1  1_12  1_20  1_30  1_34  1_37  1_4   1_45  1_53  1_55  
1_62  1_64  1_68  1_70  1_75  1_78  1_80  1_89  1_93  1_95  1_99  2_1  2_13  
2_19  2_21  2_3   2_34  2_44  2_53  2_6   2_65  2_69  2_73  2_78  2_81  2_90  
2_99

> On Jul 6, 2017, at 7:50 AM, Ian Duffy  wrote:
> 
> Hi Damian,
> 
> Sorry for the delayed reply have been out of office.
> 
> I'm afraid I cannot check. We have alarms on our auto scaling groups for
> stream instances to kill them should the CPU utilization be < 1 for 30
> mins.
> 
> On Fri 30 Jun 2017 at 5:05 p.m., Damian Guy  wrote:
> 
>> Hi Ian,
>> 
>> Can you check if the file exists and it is indeed a file rather then a
>> directory?
>> 
>> Thanks,
>> Damian
>> 
>> On Fri, 30 Jun 2017 at 16:45 Damian Guy  wrote:
>> 
>>> Hi Ian,
>>> 
>>> We had another report of what looks like the same issue. Will look into
>> it.
>>> 
>>> Thanks,
>>> Damian
>>> 
>>> On Fri, 30 Jun 2017 at 16:38 Ian Duffy  wrote:
>>> 
 Hi All,
 
 I was wondering if any of those who know stream internals should shed
>> any
 light on the following exception:
 
 org.apache.kafka.streams.errors.ProcessorStateException: Error while
 closing the state manager at
 
 
>> org.apache.kafka.streams.processor.internals.AbstractT

WindowStore fetch() ordering

[Eric Lalonde]

Hello, I was reading:

https://kafka.apache.org/0102/javadoc/org/apache/kafka/streams/state/ReadOnlyWindowStore.html#fetch(K,%20long,%20long)
 


From both my experiments and the way I read the fetch() documentation, it 
appears the values in the iterator are returned in increasing temporal order. 
Is this ordering guaranteed? e.g. if I want the the most recent value in the 
window range passed to fetch(), will it *always* be the last value, or are 
there scenarios where this is relaxed? 

I've linked above to ReadOnlyWindowStore, but the question goes for any window 
store iterator.

Re: Enabling Top-k extraction module to Kafka

[Eric Lalonde]

> On Nov 21, 2016, at 6:41 PM, Janagan Sivagnanasundaram 
>  wrote:
> 
> Hi,
> 
> Is there any libraries or any other way to implement top-k quality
> extraction algorithm (top-k info per window/batch) to the existing Kafka,
> which enable the consumer to get the top-k quality information once the
> consumer subscribed to a particular topic.
> 
> Scenario: Producer free to produce message and once the subscriber
> subscribed to a particular topic, subscriber will be able to get top k
> quality information per batch/window as Kafka processed/
> 
> Thanks.

The top songs by genre example in the confluentinc/examples github repo solves 
the problem of ‘top 5 songs by genre’.

https://github.com/confluentinc/examples/blob/3.1.x/kafka-streams/src/main/java/io/confluent/examples/streams/interactivequeries/kafkamusic/KafkaMusicExample.java

I am not suggesting it is a solution, but a good start.

- Eric