[jira] [Created] (KAFKA-8003) Flaky Test TransactionsTest #testFencingOnTransactionExpiration
Matthias J. Sax created KAFKA-8003:
--
Summary: Flaky Test TransactionsTest
#testFencingOnTransactionExpiration
Key: KAFKA-8003
URL: https://issues.apache.org/jira/browse/KAFKA-8003
Project: Kafka
Issue Type: Bug
Components: core, unit tests
Affects Versions: 2.2.0
Reporter: Matthias J. Sax
Fix For: 2.2.1
To get stable nightly builds for `2.2` release, I create tickets for all
observed test failures.
[https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/34/]
{quote}java.lang.AssertionError: expected:<1> but was:<0> at
org.junit.Assert.fail(Assert.java:88) at
org.junit.Assert.failNotEquals(Assert.java:834) at
org.junit.Assert.assertEquals(Assert.java:645) at
org.junit.Assert.assertEquals(Assert.java:631) at
kafka.api.TransactionsTest.testFencingOnTransactionExpiration(TransactionsTest.scala:510){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7969) Flaky Test DescribeConsumerGroupTest#testDescribeOffsetsOfExistingGroupWithNoMembers
[
https://issues.apache.org/jira/browse/KAFKA-7969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1602#comment-1602
]
Matthias J. Sax commented on KAFKA-7969:
Failed again (same error):
[https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/33/]
> Flaky Test
> DescribeConsumerGroupTest#testDescribeOffsetsOfExistingGroupWithNoMembers
>
>
> Key: KAFKA-7969
> URL: https://issues.apache.org/jira/browse/KAFKA-7969
> Project: Kafka
> Issue Type: Bug
> Components: admin, clients, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/24/]
> {quote}java.lang.AssertionError: Expected no active member in describe group
> results, state: Some(Empty), assignments: Some(List()) at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> kafka.admin.DescribeConsumerGroupTest.testDescribeOffsetsOfExistingGroupWithNoMembers(DescribeConsumerGroupTest.scala:278{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7967) Kafka Streams: some values in statestore rollback to old value
[
https://issues.apache.org/jira/browse/KAFKA-7967?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1614#comment-1614
]
Ziming Dong commented on KAFKA-7967:
we implemented an in memory store, let's see if it's rocksdb store bug.
> Kafka Streams: some values in statestore rollback to old value
> --
>
> Key: KAFKA-7967
> URL: https://issues.apache.org/jira/browse/KAFKA-7967
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.1.0
>Reporter: Ziming Dong
>Priority: Critical
>
> We are using kafka streams 2.1.0, we use both persistentKeyValueStore
> statestore and persistentWindowStore statestore. We found sometimes both
> types of statestore could `fetch` old values instead of newly updated values.
> We didn't find any logs except INFO level logs, no instance restart in the
> period, also there is no rebalance log which indicates it's not a rebalance
> bug. The bug happened no more than one time each week, but many records were
> affected each time, and we didn't find a way to reproduce it manually.
> For example, the issue may happen like this, note the changelog contains all
> the `update`:
> # got value 1 from key 1
> # update value 2 to key 1
> # got value 2 from key 1
> # update value 3 to key 1
> # got value 1 from key 1(something wrong!!)
> # update value 2 to key 1
> there is only one type log as follow
>
> {code:java}
> 2019-02-19x14:20:00x xx INFO
> [org.apache.kafka.clients.FetchSessionHandler]
> [xxx-streams-xx-xxx--xxx-xx-StreamThread-1] [Consumer
> clientId=x--xxx-xxx--x-StreamThread-1-consumer,
> groupId=x] Node 2 was unable to process the fetch request with
> (sessionId=1998942517, epoch=4357): INVALID_FETCH_SESSION_EPOCH.
> {code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7982) ConcurrentModificationException and Continuous warnings "Attempting to send response via channel for which there is no open connection"
[
https://issues.apache.org/jira/browse/KAFKA-7982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1623#comment-1623
]
Abhi commented on KAFKA-7982:
-
*Jaas config file:*
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/local/apps/kafkatst-kafka/config/kafka_server.keytab"
principal="kafka/u-kafkatst-kafkadev-1.sd.deshaw@unix.deshaw.com";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true;
};
What logs do you want to see - server.logs, kafka-authorizer, state-change or
controller.log?
Thanks!
> ConcurrentModificationException and Continuous warnings "Attempting to send
> response via channel for which there is no open connection"
> ---
>
> Key: KAFKA-7982
> URL: https://issues.apache.org/jira/browse/KAFKA-7982
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.1.1
>Reporter: Abhi
>Priority: Major
>
> Hi,
> I am getting following warnings in server.log continuosly and due to this
> client consumer is not able to consumer messages.
> [2019-02-20 10:26:30,312] WARN Attempting to send response via channel for
> which there is no open connection, connection id
> 10.218.27.45:9092-10.219.25.239:35248-6259 (kafka.network.Processor)
> [2019-02-20 10:26:56,760] WARN Attempting to send response via channel for
> which there is no open connection, connection id
> 10.218.27.45:9092-10.219.25.239:35604-6261 (kafka.network.Processor)
> I also noticed that before these warnings started to appear, following
> concurrent modification exception for the same IP address:
> [2019-02-20 09:01:11,175] INFO Initiating logout for
> kafka/u-kafkatst-kafkadev-1.sd@unix.com
> (org.apache.kafka.common.security.kerberos.KerberosLogin)
> [2019-02-20 09:01:11,176] WARN [SocketServer brokerId=1] Unexpected error
> from /10.219.25.239; closing connection
> (org.apache.kafka.common.network.Selector)
> java.util.ConcurrentModificationException
> at
> java.base/java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:970)
> at java.base/java.util.LinkedList$ListItr.next(LinkedList.java:892)
> at java.base/javax.security.auth.Subject$SecureSet$1.next(Subject.java:1096)
> at java.base/javax.security.auth.Subject$ClassSet$1.run(Subject.java:1501)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at
> java.base/javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1499)
> at java.base/javax.security.auth.Subject$ClassSet.(Subject.java:1472)
> at
> java.base/javax.security.auth.Subject.getPrivateCredentials(Subject.java:764)
> at java.security.jgss/sun.security.jgss.GSSUtil$1.run(GSSUtil.java:336)
> at java.security.jgss/sun.security.jgss.GSSUtil$1.run(GSSUtil.java:328)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at
> java.security.jgss/sun.security.jgss.GSSUtil.searchSubject(GSSUtil.java:328)
> at
> java.security.jgss/sun.security.jgss.wrapper.NativeGSSFactory.getCredFromSubject(NativeGSSFactory.java:53)
> at
> java.security.jgss/sun.security.jgss.wrapper.NativeGSSFactory.getCredentialElement(NativeGSSFactory.java:116)
> at
> java.security.jgss/sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:187)
> at
> java.security.jgss/sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:439)
> at
> java.security.jgss/sun.security.jgss.GSSCredentialImpl.(GSSCredentialImpl.java:74)
> at
> java.security.jgss/sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:148)
> at
> jdk.security.jgss/com.sun.security.sasl.gsskerb.GssKrb5Server.(GssKrb5Server.java:108)
> at
> jdk.security.jgss/com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85)
> at
> java.security.sasl/javax.security.sasl.Sasl.createSaslServer(Sasl.java:537)
> at
> org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.lambda$createSaslKerberosServer$12(SaslServerAuthenticator.java:212)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.createSaslKerberosServer(SaslServerAuthenticator.java:211)
> at
> org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.createSaslServer(SaslServerAuthenticator.java:164)
> at
> org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.handleKafkaRequest(SaslServerAuthenticator.java:450)
> at
> org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.authenticate(SaslServerAuthenti
[jira] [Comment Edited] (KAFKA-7982) ConcurrentModificationException and Continuous warnings "Attempting to send response via channel for which there is no open connection"
[
https://issues.apache.org/jira/browse/KAFKA-7982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1623#comment-1623
]
Abhi edited comment on KAFKA-7982 at 2/26/19 8:30 AM:
--
*Jaas config file*
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/local/apps/kafkatst-kafka/config/kafka_server.keytab"
principal="kafka/u-kafkatst-kafkadev-1.sd.deshaw@unix.deshaw.com";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true;
};
What logs do you want to see - server.logs, kafka-authorizer, state-change or
controller.log?
Thanks!
was (Author: xabhi):
*Jaas config file: *
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/local/apps/kafkatst-kafka/config/kafka_server.keytab"
principal="kafka/u-kafkatst-kafkadev-1.sd.deshaw@unix.deshaw.com";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true;
};
What logs do you want to see - server.logs, kafka-authorizer, state-change or
controller.log?
Thanks!
> ConcurrentModificationException and Continuous warnings "Attempting to send
> response via channel for which there is no open connection"
> ---
>
> Key: KAFKA-7982
> URL: https://issues.apache.org/jira/browse/KAFKA-7982
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.1.1
>Reporter: Abhi
>Priority: Major
>
> Hi,
> I am getting following warnings in server.log continuosly and due to this
> client consumer is not able to consumer messages.
> [2019-02-20 10:26:30,312] WARN Attempting to send response via channel for
> which there is no open connection, connection id
> 10.218.27.45:9092-10.219.25.239:35248-6259 (kafka.network.Processor)
> [2019-02-20 10:26:56,760] WARN Attempting to send response via channel for
> which there is no open connection, connection id
> 10.218.27.45:9092-10.219.25.239:35604-6261 (kafka.network.Processor)
> I also noticed that before these warnings started to appear, following
> concurrent modification exception for the same IP address:
> [2019-02-20 09:01:11,175] INFO Initiating logout for
> kafka/u-kafkatst-kafkadev-1.sd@unix.com
> (org.apache.kafka.common.security.kerberos.KerberosLogin)
> [2019-02-20 09:01:11,176] WARN [SocketServer brokerId=1] Unexpected error
> from /10.219.25.239; closing connection
> (org.apache.kafka.common.network.Selector)
> java.util.ConcurrentModificationException
> at
> java.base/java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:970)
> at java.base/java.util.LinkedList$ListItr.next(LinkedList.java:892)
> at java.base/javax.security.auth.Subject$SecureSet$1.next(Subject.java:1096)
> at java.base/javax.security.auth.Subject$ClassSet$1.run(Subject.java:1501)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at
> java.base/javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1499)
> at java.base/javax.security.auth.Subject$ClassSet.(Subject.java:1472)
> at
> java.base/javax.security.auth.Subject.getPrivateCredentials(Subject.java:764)
> at java.security.jgss/sun.security.jgss.GSSUtil$1.run(GSSUtil.java:336)
> at java.security.jgss/sun.security.jgss.GSSUtil$1.run(GSSUtil.java:328)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at
> java.security.jgss/sun.security.jgss.GSSUtil.searchSubject(GSSUtil.java:328)
> at
> java.security.jgss/sun.security.jgss.wrapper.NativeGSSFactory.getCredFromSubject(NativeGSSFactory.java:53)
> at
> java.security.jgss/sun.security.jgss.wrapper.NativeGSSFactory.getCredentialElement(NativeGSSFactory.java:116)
> at
> java.security.jgss/sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:187)
> at
> java.security.jgss/sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:439)
> at
> java.security.jgss/sun.security.jgss.GSSCredentialImpl.(GSSCredentialImpl.java:74)
> at
> java.security.jgss/sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:148)
> at
> jdk.security.jgss/com.sun.security.sasl.gsskerb.GssKrb5Server.(GssKrb5Server.java:108)
> at
> jdk.security.jgss/com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85)
> at
> java.security.sasl/javax.security.sasl.Sasl.createSaslServer(Sasl.java:537)
> at
> org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.lambda$createSaslKerberosServer$12(SaslServerAuthenticator.java:212)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.aut
[jira] [Comment Edited] (KAFKA-7982) ConcurrentModificationException and Continuous warnings "Attempting to send response via channel for which there is no open connection"
[
https://issues.apache.org/jira/browse/KAFKA-7982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1623#comment-1623
]
Abhi edited comment on KAFKA-7982 at 2/26/19 8:29 AM:
--
*Jaas config file: *
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/local/apps/kafkatst-kafka/config/kafka_server.keytab"
principal="kafka/u-kafkatst-kafkadev-1.sd.deshaw@unix.deshaw.com";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true;
};
What logs do you want to see - server.logs, kafka-authorizer, state-change or
controller.log?
Thanks!
was (Author: xabhi):
*Jaas config file:*
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/local/apps/kafkatst-kafka/config/kafka_server.keytab"
principal="kafka/u-kafkatst-kafkadev-1.sd.deshaw@unix.deshaw.com";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true;
};
What logs do you want to see - server.logs, kafka-authorizer, state-change or
controller.log?
Thanks!
> ConcurrentModificationException and Continuous warnings "Attempting to send
> response via channel for which there is no open connection"
> ---
>
> Key: KAFKA-7982
> URL: https://issues.apache.org/jira/browse/KAFKA-7982
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.1.1
>Reporter: Abhi
>Priority: Major
>
> Hi,
> I am getting following warnings in server.log continuosly and due to this
> client consumer is not able to consumer messages.
> [2019-02-20 10:26:30,312] WARN Attempting to send response via channel for
> which there is no open connection, connection id
> 10.218.27.45:9092-10.219.25.239:35248-6259 (kafka.network.Processor)
> [2019-02-20 10:26:56,760] WARN Attempting to send response via channel for
> which there is no open connection, connection id
> 10.218.27.45:9092-10.219.25.239:35604-6261 (kafka.network.Processor)
> I also noticed that before these warnings started to appear, following
> concurrent modification exception for the same IP address:
> [2019-02-20 09:01:11,175] INFO Initiating logout for
> kafka/u-kafkatst-kafkadev-1.sd@unix.com
> (org.apache.kafka.common.security.kerberos.KerberosLogin)
> [2019-02-20 09:01:11,176] WARN [SocketServer brokerId=1] Unexpected error
> from /10.219.25.239; closing connection
> (org.apache.kafka.common.network.Selector)
> java.util.ConcurrentModificationException
> at
> java.base/java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:970)
> at java.base/java.util.LinkedList$ListItr.next(LinkedList.java:892)
> at java.base/javax.security.auth.Subject$SecureSet$1.next(Subject.java:1096)
> at java.base/javax.security.auth.Subject$ClassSet$1.run(Subject.java:1501)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at
> java.base/javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1499)
> at java.base/javax.security.auth.Subject$ClassSet.(Subject.java:1472)
> at
> java.base/javax.security.auth.Subject.getPrivateCredentials(Subject.java:764)
> at java.security.jgss/sun.security.jgss.GSSUtil$1.run(GSSUtil.java:336)
> at java.security.jgss/sun.security.jgss.GSSUtil$1.run(GSSUtil.java:328)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at
> java.security.jgss/sun.security.jgss.GSSUtil.searchSubject(GSSUtil.java:328)
> at
> java.security.jgss/sun.security.jgss.wrapper.NativeGSSFactory.getCredFromSubject(NativeGSSFactory.java:53)
> at
> java.security.jgss/sun.security.jgss.wrapper.NativeGSSFactory.getCredentialElement(NativeGSSFactory.java:116)
> at
> java.security.jgss/sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:187)
> at
> java.security.jgss/sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:439)
> at
> java.security.jgss/sun.security.jgss.GSSCredentialImpl.(GSSCredentialImpl.java:74)
> at
> java.security.jgss/sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:148)
> at
> jdk.security.jgss/com.sun.security.sasl.gsskerb.GssKrb5Server.(GssKrb5Server.java:108)
> at
> jdk.security.jgss/com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85)
> at
> java.security.sasl/javax.security.sasl.Sasl.createSaslServer(Sasl.java:537)
> at
> org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.lambda$createSaslKerberosServer$12(SaslServerAuthenticator.java:212)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.au
[jira] [Commented] (KAFKA-7970) Missing topic causes service shutdown without exception
[ https://issues.apache.org/jira/browse/KAFKA-7970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1638#comment-1638 ] Jonny Heavey commented on KAFKA-7970: - Thanks for your responses and confirmation both. I'll await a fix in an upcoming release, and look to mitigate in the meantime. Thanks again. > Missing topic causes service shutdown without exception > --- > > Key: KAFKA-7970 > URL: https://issues.apache.org/jira/browse/KAFKA-7970 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.1.0 >Reporter: Jonny Heavey >Priority: Minor > > When launching a KafkaStreams application that depends on a topic that > doesn't exist, the streams application correctly logs an error such as: > " is unknown yet during rebalance, please make sure they have > been pre-created before starting the Streams application." > The stream is then shutdown, however, no exception is thrown indicating that > an error has occurred. > In our circumstances, we run our streams app inside a container. The streams > service is shutdown, but the process is not exited, meaning that the > container does not crash (reducing visibility of the issue). > As no exception is thrown in the missing topic scenario described above, our > application code has no way to determine that something is wrong that would > then allow it to terminate the process. > > Could the onPartitionsAssigned method in StreamThread.java throw an exception > when it decides to shutdown the stream (somewhere around line 264)? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KAFKA-7982) ConcurrentModificationException and Continuous warnings "Attempting to send response via channel for which there is no open connection"
[
https://issues.apache.org/jira/browse/KAFKA-7982?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1623#comment-1623
]
Abhi edited comment on KAFKA-7982 at 2/26/19 9:38 AM:
--
The two different principals is a copy-paste mistake from my rough draft, kafka
server is using only one principal of type
kafka/u-kafkatst-kafkadev-1.sd.deshaw@unix.deshaw.com
*Jaas config file*
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/local/apps/kafkatst-kafka/config/kafka_server.keytab"
principal="kafka/u-kafkatst-kafkadev-1.sd.deshaw@unix.deshaw.com";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true;
};
What logs do you want to see - server.logs, kafka-authorizer, state-change or
controller.log?
Thanks!
was (Author: xabhi):
*Jaas config file*
KafkaServer {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
storeKey=true
keyTab="/local/apps/kafkatst-kafka/config/kafka_server.keytab"
principal="kafka/u-kafkatst-kafkadev-1.sd.deshaw@unix.deshaw.com";
};
Client {
com.sun.security.auth.module.Krb5LoginModule required
useTicketCache=true;
};
What logs do you want to see - server.logs, kafka-authorizer, state-change or
controller.log?
Thanks!
> ConcurrentModificationException and Continuous warnings "Attempting to send
> response via channel for which there is no open connection"
> ---
>
> Key: KAFKA-7982
> URL: https://issues.apache.org/jira/browse/KAFKA-7982
> Project: Kafka
> Issue Type: Bug
> Components: core
>Affects Versions: 2.1.1
>Reporter: Abhi
>Priority: Major
>
> Hi,
> I am getting following warnings in server.log continuosly and due to this
> client consumer is not able to consumer messages.
> [2019-02-20 10:26:30,312] WARN Attempting to send response via channel for
> which there is no open connection, connection id
> 10.218.27.45:9092-10.219.25.239:35248-6259 (kafka.network.Processor)
> [2019-02-20 10:26:56,760] WARN Attempting to send response via channel for
> which there is no open connection, connection id
> 10.218.27.45:9092-10.219.25.239:35604-6261 (kafka.network.Processor)
> I also noticed that before these warnings started to appear, following
> concurrent modification exception for the same IP address:
> [2019-02-20 09:01:11,175] INFO Initiating logout for
> kafka/u-kafkatst-kafkadev-1.sd@unix.com
> (org.apache.kafka.common.security.kerberos.KerberosLogin)
> [2019-02-20 09:01:11,176] WARN [SocketServer brokerId=1] Unexpected error
> from /10.219.25.239; closing connection
> (org.apache.kafka.common.network.Selector)
> java.util.ConcurrentModificationException
> at
> java.base/java.util.LinkedList$ListItr.checkForComodification(LinkedList.java:970)
> at java.base/java.util.LinkedList$ListItr.next(LinkedList.java:892)
> at java.base/javax.security.auth.Subject$SecureSet$1.next(Subject.java:1096)
> at java.base/javax.security.auth.Subject$ClassSet$1.run(Subject.java:1501)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at
> java.base/javax.security.auth.Subject$ClassSet.populateSet(Subject.java:1499)
> at java.base/javax.security.auth.Subject$ClassSet.(Subject.java:1472)
> at
> java.base/javax.security.auth.Subject.getPrivateCredentials(Subject.java:764)
> at java.security.jgss/sun.security.jgss.GSSUtil$1.run(GSSUtil.java:336)
> at java.security.jgss/sun.security.jgss.GSSUtil$1.run(GSSUtil.java:328)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at
> java.security.jgss/sun.security.jgss.GSSUtil.searchSubject(GSSUtil.java:328)
> at
> java.security.jgss/sun.security.jgss.wrapper.NativeGSSFactory.getCredFromSubject(NativeGSSFactory.java:53)
> at
> java.security.jgss/sun.security.jgss.wrapper.NativeGSSFactory.getCredentialElement(NativeGSSFactory.java:116)
> at
> java.security.jgss/sun.security.jgss.GSSManagerImpl.getCredentialElement(GSSManagerImpl.java:187)
> at
> java.security.jgss/sun.security.jgss.GSSCredentialImpl.add(GSSCredentialImpl.java:439)
> at
> java.security.jgss/sun.security.jgss.GSSCredentialImpl.(GSSCredentialImpl.java:74)
> at
> java.security.jgss/sun.security.jgss.GSSManagerImpl.createCredential(GSSManagerImpl.java:148)
> at
> jdk.security.jgss/com.sun.security.sasl.gsskerb.GssKrb5Server.(GssKrb5Server.java:108)
> at
> jdk.security.jgss/com.sun.security.sasl.gsskerb.FactoryImpl.createSaslServer(FactoryImpl.java:85)
> at
> java.security.sasl/javax.security.sasl.Sasl.createSaslServer(Sasl.java:537)
> at
> org.apache.kafka.common.security.authenticator.SaslServerAuthentic
[jira] [Commented] (KAFKA-4107) Support offset reset capability in Kafka Connect
[ https://issues.apache.org/jira/browse/KAFKA-4107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1682#comment-1682 ] Gueorgui commented on KAFKA-4107: - What is the status of this discussion? [KIP-199|https://cwiki.apache.org/confluence/display/KAFKA/KIP-199%3A+Add+Kafka+Connect+offset+tool] > Support offset reset capability in Kafka Connect > > > Key: KAFKA-4107 > URL: https://issues.apache.org/jira/browse/KAFKA-4107 > Project: Kafka > Issue Type: Improvement > Components: KafkaConnect >Reporter: Jason Gustafson >Priority: Major > Labels: needs-kip > > It would be useful in some cases to be able to reset connector offsets. For > example, if a topic in Kafka corresponding to a source database is > accidentally deleted (or deleted because of corrupt data), an administrator > may want to reset offsets and reproduce the log from the beginning. It may > also be useful to have support for overriding offsets, but that seems like a > less likely use case. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-7940) Flaky Test CustomQuotaCallbackTest#testCustomQuotaCallback
[
https://issues.apache.org/jira/browse/KAFKA-7940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stanislav Kozlovski reassigned KAFKA-7940:
--
Assignee: Stanislav Kozlovski
> Flaky Test CustomQuotaCallbackTest#testCustomQuotaCallback
> --
>
> Key: KAFKA-7940
> URL: https://issues.apache.org/jira/browse/KAFKA-7940
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Stanislav Kozlovski
>Priority: Critical
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/14/]
> {quote}java.lang.AssertionError: Too many quotaLimit calls Map(PRODUCE -> 1,
> FETCH -> 1, REQUEST -> 4) at org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> kafka.api.CustomQuotaCallbackTest.testCustomQuotaCallback(CustomQuotaCallbackTest.scala:105){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[ https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777910#comment-16777910 ] ASF GitHub Bot commented on KAFKA-1194: --- robertbraeutigam commented on pull request #6329: KAFKA-1194: Fix renaming open files on Windows URL: https://github.com/apache/kafka/pull/6329 Running Kafka on Windows results in errors relating to renaming files that are locked (still mapped) by running Kafka. This patch unmmap()s and mmap()s a file before and after the actual renameTo() respectively. Patch should have no effect on Linux. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > The kafka broker cannot delete the old log files after the configured time > -- > > Key: KAFKA-1194 > URL: https://issues.apache.org/jira/browse/KAFKA-1194 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.10.0.0, 0.11.0.0, 1.0.0 > Environment: window >Reporter: Tao Qin >Priority: Critical > Labels: features, patch, windows > Attachments: KAFKA-1194.patch, RetentionExpiredWindows.txt, > Untitled.jpg, image-2018-09-12-14-25-52-632.png, > image-2018-11-26-10-18-59-381.png, kafka-1194-v1.patch, kafka-1194-v2.patch, > kafka-bombarder.7z, screenshot-1.png > > Original Estimate: 72h > Remaining Estimate: 72h > > We tested it in windows environment, and set the log.retention.hours to 24 > hours. > # The minimum age of a log file to be eligible for deletion > log.retention.hours=24 > After several days, the kafka broker still cannot delete the old log file. > And we get the following exceptions: > [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task > 'kafka-log-retention' (kafka.utils.KafkaScheduler) > kafka.common.KafkaStorageException: Failed to change the log file suffix from > to .deleted for log segment 1516723 > at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:249) > at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:638) > at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:629) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418) > at > scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) > at scala.collection.immutable.List.foreach(List.scala:76) > at kafka.log.Log.deleteOldSegments(Log.scala:418) > at > kafka.log.LogManager.kafka$log$LogManager$$cleanupExpiredSegments(LogManager.scala:284) > at > kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:316) > at > kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:314) > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:743) > at scala.collection.Iterator$class.foreach(Iterator.scala:772) > at > scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:73) > at > scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615) > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:742) > at kafka.log.LogManager.cleanupLogs(LogManager.scala:314) > at > kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:143) > at kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > I think this error happens because kafka tries to rename the log file when it > is still opened. So we should close the file first before rename. > The index file uses a special data structure, the MappedByteBuffer. Javadoc > describes it as: > A mapped byte buffer
[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[ https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777911#comment-16777911 ] ASF GitHub Bot commented on KAFKA-1194: --- robertbraeutigam commented on pull request #6329: KAFKA-1194: Fix renaming open files on Windows URL: https://github.com/apache/kafka/pull/6329 Running Kafka on Windows results in errors relating to renaming files that are locked (still mapped) by running Kafka. This patch unmmap()s and mmap()s a file before and after the actual renameTo() respectively. Patch should have no effect on Linux. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > The kafka broker cannot delete the old log files after the configured time > -- > > Key: KAFKA-1194 > URL: https://issues.apache.org/jira/browse/KAFKA-1194 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.10.0.0, 0.11.0.0, 1.0.0 > Environment: window >Reporter: Tao Qin >Priority: Critical > Labels: features, patch, windows > Attachments: KAFKA-1194.patch, RetentionExpiredWindows.txt, > Untitled.jpg, image-2018-09-12-14-25-52-632.png, > image-2018-11-26-10-18-59-381.png, kafka-1194-v1.patch, kafka-1194-v2.patch, > kafka-bombarder.7z, screenshot-1.png > > Original Estimate: 72h > Remaining Estimate: 72h > > We tested it in windows environment, and set the log.retention.hours to 24 > hours. > # The minimum age of a log file to be eligible for deletion > log.retention.hours=24 > After several days, the kafka broker still cannot delete the old log file. > And we get the following exceptions: > [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task > 'kafka-log-retention' (kafka.utils.KafkaScheduler) > kafka.common.KafkaStorageException: Failed to change the log file suffix from > to .deleted for log segment 1516723 > at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:249) > at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:638) > at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:629) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418) > at > scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) > at scala.collection.immutable.List.foreach(List.scala:76) > at kafka.log.Log.deleteOldSegments(Log.scala:418) > at > kafka.log.LogManager.kafka$log$LogManager$$cleanupExpiredSegments(LogManager.scala:284) > at > kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:316) > at > kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:314) > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:743) > at scala.collection.Iterator$class.foreach(Iterator.scala:772) > at > scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:73) > at > scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615) > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:742) > at kafka.log.LogManager.cleanupLogs(LogManager.scala:314) > at > kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:143) > at kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > I think this error happens because kafka tries to rename the log file when it > is still opened. So we should close the file first before rename. > The index file uses a special data structure, the MappedByteBuffer. Javadoc > describes it as: > A mapped byte buffer
[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time
[ https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777914#comment-16777914 ] Robert Brautigam commented on KAFKA-1194: - We have to install Kafka on Windows for a client and this bug is blocking us at the moment (since Kafka crashes on first log deletion and then shortly after each restart). For this reason I submitted a PR that seems to solve this problem for us: PR: https://github.com/apache/kafka/pull/6329 > The kafka broker cannot delete the old log files after the configured time > -- > > Key: KAFKA-1194 > URL: https://issues.apache.org/jira/browse/KAFKA-1194 > Project: Kafka > Issue Type: Bug > Components: log >Affects Versions: 0.10.0.0, 0.11.0.0, 1.0.0 > Environment: window >Reporter: Tao Qin >Priority: Critical > Labels: features, patch, windows > Attachments: KAFKA-1194.patch, RetentionExpiredWindows.txt, > Untitled.jpg, image-2018-09-12-14-25-52-632.png, > image-2018-11-26-10-18-59-381.png, kafka-1194-v1.patch, kafka-1194-v2.patch, > kafka-bombarder.7z, screenshot-1.png > > Original Estimate: 72h > Remaining Estimate: 72h > > We tested it in windows environment, and set the log.retention.hours to 24 > hours. > # The minimum age of a log file to be eligible for deletion > log.retention.hours=24 > After several days, the kafka broker still cannot delete the old log file. > And we get the following exceptions: > [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task > 'kafka-log-retention' (kafka.utils.KafkaScheduler) > kafka.common.KafkaStorageException: Failed to change the log file suffix from > to .deleted for log segment 1516723 > at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:249) > at kafka.log.Log.kafka$log$Log$$asyncDeleteSegment(Log.scala:638) > at kafka.log.Log.kafka$log$Log$$deleteSegment(Log.scala:629) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418) > at kafka.log.Log$$anonfun$deleteOldSegments$1.apply(Log.scala:418) > at > scala.collection.LinearSeqOptimized$class.foreach(LinearSeqOptimized.scala:59) > at scala.collection.immutable.List.foreach(List.scala:76) > at kafka.log.Log.deleteOldSegments(Log.scala:418) > at > kafka.log.LogManager.kafka$log$LogManager$$cleanupExpiredSegments(LogManager.scala:284) > at > kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:316) > at > kafka.log.LogManager$$anonfun$cleanupLogs$3.apply(LogManager.scala:314) > at > scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:743) > at scala.collection.Iterator$class.foreach(Iterator.scala:772) > at > scala.collection.JavaConversions$JIteratorWrapper.foreach(JavaConversions.scala:573) > at scala.collection.IterableLike$class.foreach(IterableLike.scala:73) > at > scala.collection.JavaConversions$JListWrapper.foreach(JavaConversions.scala:615) > at > scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:742) > at kafka.log.LogManager.cleanupLogs(LogManager.scala:314) > at > kafka.log.LogManager$$anonfun$startup$1.apply$mcV$sp(LogManager.scala:143) > at kafka.utils.KafkaScheduler$$anon$1.run(KafkaScheduler.scala:100) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > I think this error happens because kafka tries to rename the log file when it > is still opened. So we should close the file first before rename. > The index file uses a special data structure, the MappedByteBuffer. Javadoc > describes it as: > A mapped byte buffer and the file mapping that it represents remain valid > until the buffer itself is garbage-collected. > Fortunately, I find a forceUnmap function in kafka code, and perhaps it can > be used to free the MappedByteBuffer. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-7969) Flaky Test DescribeConsumerGroupTest#testDescribeOffsetsOfExistingGroupWithNoMembers
[
https://issues.apache.org/jira/browse/KAFKA-7969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stanislav Kozlovski reassigned KAFKA-7969:
--
Assignee: Stanislav Kozlovski
> Flaky Test
> DescribeConsumerGroupTest#testDescribeOffsetsOfExistingGroupWithNoMembers
>
>
> Key: KAFKA-7969
> URL: https://issues.apache.org/jira/browse/KAFKA-7969
> Project: Kafka
> Issue Type: Bug
> Components: admin, clients, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Stanislav Kozlovski
>Priority: Critical
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/24/]
> {quote}java.lang.AssertionError: Expected no active member in describe group
> results, state: Some(Empty), assignments: Some(List()) at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> kafka.admin.DescribeConsumerGroupTest.testDescribeOffsetsOfExistingGroupWithNoMembers(DescribeConsumerGroupTest.scala:278{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7996) KafkaStreams does not pass timeout when closing Producer
[
https://issues.apache.org/jira/browse/KAFKA-7996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778198#comment-16778198
]
Guozhang Wang commented on KAFKA-7996:
--
I'd agree with [~mjsax]'s explanation here. More specifically the `timeout`
semantics in `KafkaStreams#close` function is not for "enforcing to complete
closing before this timeout", but "try to block on close for maximum of this
time, and if it still not complete then give up and return false". The javadocs
we wrote is intending for this purpose as well:
{code}
* Shutdown this {@code KafkaStreams} by signaling all the threads to stop,
and then wait up to the timeout for the
* threads to join.
{code}
If we change the semantics to the former, then like stated above we need to
inject this timeout all the way down to each modules being closed (similar to
sth we did on other client), and I'm not sure if it is worthwhile to do: if
lots of users found the current semantics inferior to the former, we can do
that (which would be quite intrusive as we did before).
So if the producer.close() is blocked which caused the close call to not be
able to complete within the timeout, the KafkaStreams#close() call should
return within timeout and users should get a "false" boolean. [~pkleindl] could
you check if this is not the case you observed?
> KafkaStreams does not pass timeout when closing Producer
>
>
> Key: KAFKA-7996
> URL: https://issues.apache.org/jira/browse/KAFKA-7996
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 2.1.0
>Reporter: Patrik Kleindl
>Assignee: Lee Dongjin
>Priority: Major
> Labels: needs-kip
>
> [https://confluentcommunity.slack.com/messages/C48AHTCUQ/convo/C48AHTCUQ-1550831721.026100/]
> We are running 2.1 and have a case where the shutdown of a streams
> application takes several minutes
> I noticed that although we call streams.close with a timeout of 30 seconds
> the log says
> [Producer
> clientId=…-8be49feb-8a2e-4088-bdd7-3c197f6107bb-StreamThread-1-producer]
> Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
> Matthias J Sax [vor 3 Tagen]
> I just checked the code, and yes, we don't provide a timeout for the producer
> on close()...
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7652) Kafka Streams Session store performance degradation from 0.10.2.2 to 0.11.0.0
[ https://issues.apache.org/jira/browse/KAFKA-7652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778224#comment-16778224 ] Guozhang Wang commented on KAFKA-7652: -- Oh that's bad news.. 1) when you profile on latest trunk did you see the same pattern as observed in https://i.imgur.com/IHxC2cZ.png as well as in the trace logging compared with 0.10.2.x? 2) practically the lookups in the caching layer is very cheap and hence even increased a lot it should not contribute to much overhead, whereas the fetches on the underlying store would be much more expensive. Could you confirm if the performance bottleneck is from the underlying rocksDB, or from the caching layer access? > Kafka Streams Session store performance degradation from 0.10.2.2 to 0.11.0.0 > - > > Key: KAFKA-7652 > URL: https://issues.apache.org/jira/browse/KAFKA-7652 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.11.0.0, 0.11.0.1, 0.11.0.2, 0.11.0.3, 1.1.1, 2.0.0, > 2.0.1 >Reporter: Jonathan Gordon >Assignee: Guozhang Wang >Priority: Major > Labels: kip > Fix For: 2.2.0 > > Attachments: kafka_10_2_1_flushes.txt, kafka_11_0_3_flushes.txt > > > I'm creating this issue in response to [~guozhang]'s request on the mailing > list: > [https://lists.apache.org/thread.html/97d620f4fd76be070ca4e2c70e2fda53cafe051e8fc4505dbcca0321@%3Cusers.kafka.apache.org%3E] > We are attempting to upgrade our Kafka Streams application from 0.10.2.1 but > experience a severe performance degradation. The highest amount of CPU time > seems spent in retrieving from the local cache. Here's an example thread > profile with 0.11.0.0: > [https://i.imgur.com/l5VEsC2.png] > When things are running smoothly we're gated by retrieving from the state > store with acceptable performance. Here's an example thread profile with > 0.10.2.1: > [https://i.imgur.com/IHxC2cZ.png] > Some investigation reveals that it appears we're performing about 3 orders > magnitude more lookups on the NamedCache over a comparable time period. I've > attached logs of the NamedCache flush logs for 0.10.2.1 and 0.11.0.3. > We're using session windows and have the app configured for > commit.interval.ms = 30 * 1000 and cache.max.bytes.buffering = 10485760 > I'm happy to share more details if they would be helpful. Also happy to run > tests on our data. > I also found this issue, which seems like it may be related: > https://issues.apache.org/jira/browse/KAFKA-4904 > > KIP-420: > [https://cwiki.apache.org/confluence/display/KAFKA/KIP-420%3A+Add+Single+Value+Fetch+in+Session+Stores] > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-7999) Flaky Test ExampleConnectIntegrationTest#testProduceConsumeConnector
[
https://issues.apache.org/jira/browse/KAFKA-7999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish reassigned KAFKA-7999:
---
Assignee: Arjun Satish
> Flaky Test ExampleConnectIntegrationTest#testProduceConsumeConnector
>
>
> Key: KAFKA-7999
> URL: https://issues.apache.org/jira/browse/KAFKA-7999
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Arjun Satish
>Priority: Critical
> Fix For: 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/30/]
> {quote}org.apache.kafka.common.KafkaException: Could not produce message to
> topic=test-topic at
> org.apache.kafka.connect.util.clusters.EmbeddedKafkaCluster.produce(EmbeddedKafkaCluster.java:257)
> at
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testProduceConsumeConnector(ExampleConnectIntegrationTest.java:129){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7999) Flaky Test ExampleConnectIntegrationTest#testProduceConsumeConnector
[
https://issues.apache.org/jira/browse/KAFKA-7999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778230#comment-16778230
]
Arjun Satish commented on KAFKA-7999:
-
https://github.com/apache/kafka/pull/6326 was merged to address this issue. We
were not able to reproduce it locally, and suspect that this failure happens
when many tests run in parallel on low resource VMs. Increasing the timeout (as
in the linked PR) should reduce the chances of random failure.
> Flaky Test ExampleConnectIntegrationTest#testProduceConsumeConnector
>
>
> Key: KAFKA-7999
> URL: https://issues.apache.org/jira/browse/KAFKA-7999
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Arjun Satish
>Priority: Critical
> Fix For: 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/30/]
> {quote}org.apache.kafka.common.KafkaException: Could not produce message to
> topic=test-topic at
> org.apache.kafka.connect.util.clusters.EmbeddedKafkaCluster.produce(EmbeddedKafkaCluster.java:257)
> at
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testProduceConsumeConnector(ExampleConnectIntegrationTest.java:129){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KAFKA-7999) Flaky Test ExampleConnectIntegrationTest#testProduceConsumeConnector
[
https://issues.apache.org/jira/browse/KAFKA-7999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arjun Satish resolved KAFKA-7999.
-
Resolution: Fixed
PR: https://github.com/apache/kafka/pull/6326
> Flaky Test ExampleConnectIntegrationTest#testProduceConsumeConnector
>
>
> Key: KAFKA-7999
> URL: https://issues.apache.org/jira/browse/KAFKA-7999
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Arjun Satish
>Priority: Critical
> Fix For: 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/30/]
> {quote}org.apache.kafka.common.KafkaException: Could not produce message to
> topic=test-topic at
> org.apache.kafka.connect.util.clusters.EmbeddedKafkaCluster.produce(EmbeddedKafkaCluster.java:257)
> at
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testProduceConsumeConnector(ExampleConnectIntegrationTest.java:129){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7956) Avoid blocking in ShutdownableThread.awaitShutdown if the thread has not been started
[ https://issues.apache.org/jira/browse/KAFKA-7956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778410#comment-16778410 ] ASF GitHub Bot commented on KAFKA-7956: --- junrao commented on pull request #6218: KAFKA-7956 In ShutdownableThread, immediately complete the shutdown if the thread has not been started URL: https://github.com/apache/kafka/pull/6218 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Avoid blocking in ShutdownableThread.awaitShutdown if the thread has not been > started > - > > Key: KAFKA-7956 > URL: https://issues.apache.org/jira/browse/KAFKA-7956 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Gardner Vickers >Priority: Minor > > Opening this Jira to track [https://github.com/apache/kafka/pull/6218], since > it's a rather subtle change. > In some test cases it's desirable to instantiate a subclass of > `ShutdownableThread` without starting it. Since most subclasses of > `ShutdownableThread` put cleanup logic in `ShutdownableThread.shutdown()`, > being able to call `shutdown()` on the non-running thread would be useful. > This change allows us to avoid blocking in `ShutdownableThread.shutdown()` if > the thread's `run()` method has not been called. We also add a check that > `initiateShutdown()` was called before `awaitShutdown()`, to protect against > the case where a user calls `awaitShutdown()` before the thread has been > started, and unexpectedly is not blocked on the thread shutting down. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-8002) Replica reassignment to new log dir may not complete if future and current replicas segment files have different base offsets
[ https://issues.apache.org/jira/browse/KAFKA-8002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Barrett reassigned KAFKA-8002: -- Assignee: Bob Barrett > Replica reassignment to new log dir may not complete if future and current > replicas segment files have different base offsets > - > > Key: KAFKA-8002 > URL: https://issues.apache.org/jira/browse/KAFKA-8002 > Project: Kafka > Issue Type: Bug > Components: core >Affects Versions: 2.1.1 >Reporter: Anna Povzner >Assignee: Bob Barrett >Priority: Critical > > Once future replica fetches log end offset, the intended logic is to finish > the move (and rename the future dir to current replica dir, etc). However, > the check in Partition.maybeReplaceCurrentWithFutureReplica compares the > whole LogOffsetMetadata vs. log end offset. The resulting behavior is that > the re-assignment will not finish for topic partitions that were cleaned/ > compacted such that base offset of the last segment is different for the > current and future replica. > The proposed fix is to compare only log end offsets of the current and future > replica. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KAFKA-7956) Avoid blocking in ShutdownableThread.awaitShutdown if the thread has not been started
[ https://issues.apache.org/jira/browse/KAFKA-7956?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-7956. Resolution: Fixed Assignee: Gardner Vickers Fix Version/s: 2.3.0 Merged the PR to trunk. > Avoid blocking in ShutdownableThread.awaitShutdown if the thread has not been > started > - > > Key: KAFKA-7956 > URL: https://issues.apache.org/jira/browse/KAFKA-7956 > Project: Kafka > Issue Type: Improvement > Components: core >Reporter: Gardner Vickers >Assignee: Gardner Vickers >Priority: Minor > Fix For: 2.3.0 > > > Opening this Jira to track [https://github.com/apache/kafka/pull/6218], since > it's a rather subtle change. > In some test cases it's desirable to instantiate a subclass of > `ShutdownableThread` without starting it. Since most subclasses of > `ShutdownableThread` put cleanup logic in `ShutdownableThread.shutdown()`, > being able to call `shutdown()` on the non-running thread would be useful. > This change allows us to avoid blocking in `ShutdownableThread.shutdown()` if > the thread's `run()` method has not been called. We also add a check that > `initiateShutdown()` was called before `awaitShutdown()`, to protect against > the case where a user calls `awaitShutdown()` before the thread has been > started, and unexpectedly is not blocked on the thread shutting down. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7994) Improve Stream-Time for rebalances and restarts
[
https://issues.apache.org/jira/browse/KAFKA-7994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778460#comment-16778460
]
Guozhang Wang commented on KAFKA-7994:
--
Thanks for brining this up! I've also thought about this lately, and another
semi-related issue is that the inferred partition/stream time is inconsistency
across sub-topologies today because each task independently infer that based on
its own incoming records' timestamp. As of today this is sorta by-design since
each task is executed independently, but still there are shortcomings like
inter-state-store consistency and global-consistent savepoints. Take your
example in the desciprtion, when we've processed up to r4 our current inferred
stream time is 11 already, but imagine this sub-topology will eventually have a
filter processor followed by a sink sending to an intermediate topic, and r2
and r3 are filtered, then in the down stream sub-topology we will only see the
the processed results of r1 and r4 (and their inherited record timestamp) and
then infer the stream time as 2.
So I'm feeling that if we can tackle those issues within a single solution,
then maybe we can consider propagating the stream time inferred at the "head"
sub-topologies, plus storing it at the offset metadata.
> Improve Stream-Time for rebalances and restarts
> ---
>
> Key: KAFKA-7994
> URL: https://issues.apache.org/jira/browse/KAFKA-7994
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Matthias J. Sax
>Priority: Major
>
> We compute a per-partition partition-time as the maximum timestamp over all
> records processed so far. Furthermore, we use partition-time to compute
> stream-time for each task as maximum over all partition-times (for all
> corresponding task partitions). This stream-time is used to make decisions
> about processing out-of-order records or drop them if they are late (ie,
> timestamp < stream-time - grace-period).
> During rebalances and restarts, stream-time is initialized as UNKNOWN (ie,
> -1) for tasks that are newly created (or migrated). In net effect, we forget
> current stream-time for this case what may lead to non-deterministic behavior
> if we stop processing right before a late record, that would be dropped if we
> continue processing, but is not dropped after rebalance/restart. Let's look
> at an examples with a grade period of 5ms for a tumbling windowed of 5ms, and
> the following records (timestamps in parenthesis):
>
> {code:java}
> r1(0) r2(5) r3(11) r4(2){code}
> In the example, stream-time advances as 0, 5, 11, 11 and thus record `r4` is
> dropped as late because 2 < 6 = 11 - 5. However, if we stop processing or
> rebalance after processing `r3` but before processing `r4`, we would
> reinitialize stream-time as -1, and thus would process `r4` on restart/after
> rebalance. The problem is, that stream-time does advance differently from a
> global point of view: 0, 5, 11, 2.
>
> Note, this is a corner case, because if we would stop processing one record
> earlier, ie, after processing `r2` but before processing `r3`, stream-time
> would be advance correctly from a global point of view.
> A potential fix would be, to store latest observed partition-time in the
> metadata of committed offsets. Thus way, on restart/rebalance we can
> re-initialize time correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7996) KafkaStreams does not pass timeout when closing Producer
[ https://issues.apache.org/jira/browse/KAFKA-7996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778486#comment-16778486 ] Patrik Kleindl commented on KAFKA-7996: --- [~mjsax] [~guozhang] Apologies for creating this stir, you are right that KafkaStreams.close is behaving as it should. I got too focused on the producer timeout and mixed things up. It seems we (and our code) take the "false" from the close method too strict as we interpreted this as a kind of fatal condition and don't try to start a new instance of the streams application as we can't be sure if the shutdown was successful. We are not running in the cloud so a proper shutdown seemed important. As it stands even with proper timeouts for the producer and adminClient the close can take up to several minutes in the worst case which is fine. So my hopefully last questions: If a KafkaStreams instance is in ERROR state, is it safe to start a new one or should one wait until the close() is completed? Can the close() change anything regarding topics or state stores which might prevent a new instance from starting or would it be better to call .cleanup() on the new instance? > KafkaStreams does not pass timeout when closing Producer > > > Key: KAFKA-7996 > URL: https://issues.apache.org/jira/browse/KAFKA-7996 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.1.0 >Reporter: Patrik Kleindl >Assignee: Lee Dongjin >Priority: Major > Labels: needs-kip > > [https://confluentcommunity.slack.com/messages/C48AHTCUQ/convo/C48AHTCUQ-1550831721.026100/] > We are running 2.1 and have a case where the shutdown of a streams > application takes several minutes > I noticed that although we call streams.close with a timeout of 30 seconds > the log says > [Producer > clientId=…-8be49feb-8a2e-4088-bdd7-3c197f6107bb-StreamThread-1-producer] > Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. > Matthias J Sax [vor 3 Tagen] > I just checked the code, and yes, we don't provide a timeout for the producer > on close()... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7989) Flaky Test RequestQuotaTest#testResponseThrottleTimeWhenBothFetchAndRequestQuotasViolated
[
https://issues.apache.org/jira/browse/KAFKA-7989?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7989:
---
Labels: flaky-test (was: )
> Flaky Test
> RequestQuotaTest#testResponseThrottleTimeWhenBothFetchAndRequestQuotasViolated
> -
>
> Key: KAFKA-7989
> URL: https://issues.apache.org/jira/browse/KAFKA-7989
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/27/]
> {quote}java.util.concurrent.ExecutionException: java.lang.AssertionError:
> Throttle time metrics for consumer quota not updated:
> small-quota-consumer-client at
> java.util.concurrent.FutureTask.report(FutureTask.java:122) at
> java.util.concurrent.FutureTask.get(FutureTask.java:206) at
> kafka.server.RequestQuotaTest.$anonfun$waitAndCheckResults$1(RequestQuotaTest.scala:415)
> at scala.collection.immutable.List.foreach(List.scala:392) at
> scala.collection.generic.TraversableForwarder.foreach(TraversableForwarder.scala:38)
> at
> scala.collection.generic.TraversableForwarder.foreach$(TraversableForwarder.scala:38)
> at scala.collection.mutable.ListBuffer.foreach(ListBuffer.scala:47) at
> kafka.server.RequestQuotaTest.waitAndCheckResults(RequestQuotaTest.scala:413)
> at
> kafka.server.RequestQuotaTest.testResponseThrottleTimeWhenBothFetchAndRequestQuotasViolated(RequestQuotaTest.scala:134){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7999) Flaky Test ExampleConnectIntegrationTest#testProduceConsumeConnector
[
https://issues.apache.org/jira/browse/KAFKA-7999?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7999:
---
Labels: flaky-test (was: )
> Flaky Test ExampleConnectIntegrationTest#testProduceConsumeConnector
>
>
> Key: KAFKA-7999
> URL: https://issues.apache.org/jira/browse/KAFKA-7999
> Project: Kafka
> Issue Type: Bug
> Components: KafkaConnect, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Arjun Satish
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/30/]
> {quote}org.apache.kafka.common.KafkaException: Could not produce message to
> topic=test-topic at
> org.apache.kafka.connect.util.clusters.EmbeddedKafkaCluster.produce(EmbeddedKafkaCluster.java:257)
> at
> org.apache.kafka.connect.integration.ExampleConnectIntegrationTest.testProduceConsumeConnector(ExampleConnectIntegrationTest.java:129){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7976) Flaky Test DynamicBrokerReconfigurationTest#testUncleanLeaderElectionEnable
[
https://issues.apache.org/jira/browse/KAFKA-7976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7976:
---
Labels: flaky-test (was: )
> Flaky Test DynamicBrokerReconfigurationTest#testUncleanLeaderElectionEnable
> ---
>
> Key: KAFKA-7976
> URL: https://issues.apache.org/jira/browse/KAFKA-7976
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Stanislav Kozlovski
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://builds.apache.org/blue/organizations/jenkins/kafka-2.2-jdk8/detail/kafka-2.2-jdk8/28/]
> {quote}java.lang.AssertionError: Unclean leader not elected
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> kafka.server.DynamicBrokerReconfigurationTest.testUncleanLeaderElectionEnable(DynamicBrokerReconfigurationTest.scala:488){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7969) Flaky Test DescribeConsumerGroupTest#testDescribeOffsetsOfExistingGroupWithNoMembers
[
https://issues.apache.org/jira/browse/KAFKA-7969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7969:
---
Labels: flaky-test (was: )
> Flaky Test
> DescribeConsumerGroupTest#testDescribeOffsetsOfExistingGroupWithNoMembers
>
>
> Key: KAFKA-7969
> URL: https://issues.apache.org/jira/browse/KAFKA-7969
> Project: Kafka
> Issue Type: Bug
> Components: admin, clients, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Stanislav Kozlovski
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/24/]
> {quote}java.lang.AssertionError: Expected no active member in describe group
> results, state: Some(Empty), assignments: Some(List()) at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> kafka.admin.DescribeConsumerGroupTest.testDescribeOffsetsOfExistingGroupWithNoMembers(DescribeConsumerGroupTest.scala:278{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7979) Flaky Test PartitionTest#testDelayedFetchAfterAppendRecords
[ https://issues.apache.org/jira/browse/KAFKA-7979?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-7979: --- Labels: flaky-test (was: ) > Flaky Test PartitionTest#testDelayedFetchAfterAppendRecords > --- > > Key: KAFKA-7979 > URL: https://issues.apache.org/jira/browse/KAFKA-7979 > Project: Kafka > Issue Type: Bug > Components: core, unit tests >Affects Versions: 2.2.0 >Reporter: Matthias J. Sax >Priority: Critical > Labels: flaky-test > Fix For: 2.3.0, 2.2.1 > > Attachments: error_2018_02_21.log > > > To get stable nightly builds for `2.2` release, I create tickets for all > observed test failures. > [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/25/] > Error log enclosed in file. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7978) Flaky Test SaslSslAdminClientIntegrationTest#testConsumerGroups
[
https://issues.apache.org/jira/browse/KAFKA-7978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7978:
---
Labels: flaky-test (was: )
> Flaky Test SaslSslAdminClientIntegrationTest#testConsumerGroups
> ---
>
> Key: KAFKA-7978
> URL: https://issues.apache.org/jira/browse/KAFKA-7978
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/25/]
> {quote}java.lang.AssertionError: expected:<2> but was:<0> at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:834) at
> org.junit.Assert.assertEquals(Assert.java:645) at
> org.junit.Assert.assertEquals(Assert.java:631) at
> kafka.api.AdminClientIntegrationTest.testConsumerGroups(AdminClientIntegrationTest.scala:1157)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7977) Flaky Test ReassignPartitionsClusterTest#shouldOnlyThrottleMovingReplicas
[
https://issues.apache.org/jira/browse/KAFKA-7977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7977:
---
Labels: flaky-test (was: )
> Flaky Test ReassignPartitionsClusterTest#shouldOnlyThrottleMovingReplicas
> -
>
> Key: KAFKA-7977
> URL: https://issues.apache.org/jira/browse/KAFKA-7977
> Project: Kafka
> Issue Type: Bug
> Components: admin, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/25/]
> {quote}org.apache.zookeeper.KeeperException$SessionExpiredException:
> KeeperErrorCode = Session expired for /brokers/topics/topic1 at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:130) at
> org.apache.zookeeper.KeeperException.create(KeeperException.java:54) at
> kafka.zookeeper.AsyncResponse.resultException(ZooKeeperClient.scala:534) at
> kafka.zk.KafkaZkClient.$anonfun$getReplicaAssignmentForTopics$2(KafkaZkClient.scala:579)
> at
> scala.collection.TraversableLike.$anonfun$flatMap$1(TraversableLike.scala:244)
> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at
> scala.collection.TraversableLike.flatMap(TraversableLike.scala:244) at
> scala.collection.TraversableLike.flatMap$(TraversableLike.scala:241) at
> scala.collection.AbstractTraversable.flatMap(Traversable.scala:108) at
> kafka.zk.KafkaZkClient.getReplicaAssignmentForTopics(KafkaZkClient.scala:574)
> at
> kafka.admin.ReassignPartitionsCommand$.parseAndValidate(ReassignPartitionsCommand.scala:338)
> at
> kafka.admin.ReassignPartitionsCommand$.executeAssignment(ReassignPartitionsCommand.scala:209)
> at
> kafka.admin.ReassignPartitionsClusterTest.shouldOnlyThrottleMovingReplicas(ReassignPartitionsClusterTest.scala:343){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7988) Flaky Test DynamicBrokerReconfigurationTest#testThreadPoolResize
[
https://issues.apache.org/jira/browse/KAFKA-7988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7988:
---
Labels: flaky-test (was: )
> Flaky Test DynamicBrokerReconfigurationTest#testThreadPoolResize
>
>
> Key: KAFKA-7988
> URL: https://issues.apache.org/jira/browse/KAFKA-7988
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://builds.apache.org/blue/organizations/jenkins/kafka-2.2-jdk8/detail/kafka-2.2-jdk8/30/]
> {quote}kafka.server.DynamicBrokerReconfigurationTest > testThreadPoolResize
> FAILED java.lang.AssertionError: Invalid threads: expected 6, got 5:
> List(ReplicaFetcherThread-0-0, ReplicaFetcherThread-0-1,
> ReplicaFetcherThread-0-0, ReplicaFetcherThread-0-2, ReplicaFetcherThread-0-1)
> at org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> kafka.server.DynamicBrokerReconfigurationTest.verifyThreads(DynamicBrokerReconfigurationTest.scala:1260)
> at
> kafka.server.DynamicBrokerReconfigurationTest.maybeVerifyThreadPoolSize$1(DynamicBrokerReconfigurationTest.scala:531)
> at
> kafka.server.DynamicBrokerReconfigurationTest.resizeThreadPool$1(DynamicBrokerReconfigurationTest.scala:550)
> at
> kafka.server.DynamicBrokerReconfigurationTest.reducePoolSize$1(DynamicBrokerReconfigurationTest.scala:536)
> at
> kafka.server.DynamicBrokerReconfigurationTest.$anonfun$testThreadPoolResize$3(DynamicBrokerReconfigurationTest.scala:559)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:158) at
> kafka.server.DynamicBrokerReconfigurationTest.verifyThreadPoolResize$1(DynamicBrokerReconfigurationTest.scala:558)
> at
> kafka.server.DynamicBrokerReconfigurationTest.testThreadPoolResize(DynamicBrokerReconfigurationTest.scala:572){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7957) Flaky Test DynamicBrokerReconfigurationTest#testMetricsReporterUpdate
[
https://issues.apache.org/jira/browse/KAFKA-7957?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7957:
---
Labels: flaky-test (was: )
> Flaky Test DynamicBrokerReconfigurationTest#testMetricsReporterUpdate
> -
>
> Key: KAFKA-7957
> URL: https://issues.apache.org/jira/browse/KAFKA-7957
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/18/]
> {quote}java.lang.AssertionError: Messages not sent at
> kafka.utils.TestUtils$.fail(TestUtils.scala:356) at
> kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:766) at
> kafka.server.DynamicBrokerReconfigurationTest.startProduceConsume(DynamicBrokerReconfigurationTest.scala:1270)
> at
> kafka.server.DynamicBrokerReconfigurationTest.testMetricsReporterUpdate(DynamicBrokerReconfigurationTest.scala:650){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7946) Flaky Test DeleteConsumerGroupsTest#testDeleteNonEmptyGroup
[
https://issues.apache.org/jira/browse/KAFKA-7946?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7946:
---
Labels: flaky-test (was: )
> Flaky Test DeleteConsumerGroupsTest#testDeleteNonEmptyGroup
> ---
>
> Key: KAFKA-7946
> URL: https://issues.apache.org/jira/browse/KAFKA-7946
> Project: Kafka
> Issue Type: Bug
> Components: admin, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Gwen Shapira
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/17/]
> {quote}java.lang.NullPointerException at
> kafka.admin.DeleteConsumerGroupsTest.testDeleteNonEmptyGroup(DeleteConsumerGroupsTest.scala:96){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7965) Flaky Test ConsumerBounceTest#testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
[
https://issues.apache.org/jira/browse/KAFKA-7965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7965:
---
Labels: flaky-test (was: )
> Flaky Test
> ConsumerBounceTest#testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup
>
>
> Key: KAFKA-7965
> URL: https://issues.apache.org/jira/browse/KAFKA-7965
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Stanislav Kozlovski
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/21/]
> {quote}java.lang.AssertionError: Received 0, expected at least 68 at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> kafka.api.ConsumerBounceTest.receiveAndCommit(ConsumerBounceTest.scala:557)
> at
> kafka.api.ConsumerBounceTest.$anonfun$testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup$1(ConsumerBounceTest.scala:320)
> at
> kafka.api.ConsumerBounceTest.$anonfun$testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup$1$adapted(ConsumerBounceTest.scala:319)
> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at
> kafka.api.ConsumerBounceTest.testRollingBrokerRestartsWithSmallerMaxGroupSizeConfigDisruptsBigGroup(ConsumerBounceTest.scala:319){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7940) Flaky Test CustomQuotaCallbackTest#testCustomQuotaCallback
[
https://issues.apache.org/jira/browse/KAFKA-7940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7940:
---
Labels: flaky-test (was: )
> Flaky Test CustomQuotaCallbackTest#testCustomQuotaCallback
> --
>
> Key: KAFKA-7940
> URL: https://issues.apache.org/jira/browse/KAFKA-7940
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Stanislav Kozlovski
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/14/]
> {quote}java.lang.AssertionError: Too many quotaLimit calls Map(PRODUCE -> 1,
> FETCH -> 1, REQUEST -> 4) at org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> kafka.api.CustomQuotaCallbackTest.testCustomQuotaCallback(CustomQuotaCallbackTest.scala:105){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7947) Flaky Test EpochDrivenReplicationProtocolAcceptanceTest #shouldFollowLeaderEpochBasicWorkflow
[
https://issues.apache.org/jira/browse/KAFKA-7947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7947:
---
Labels: flaky-test (was: )
> Flaky Test EpochDrivenReplicationProtocolAcceptanceTest
> #shouldFollowLeaderEpochBasicWorkflow
> -
>
> Key: KAFKA-7947
> URL: https://issues.apache.org/jira/browse/KAFKA-7947
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/17/]
> {quote}java.lang.AssertionError: expected: startOffset=0), EpochEntry(epoch=1, startOffset=1))> but
> was: startOffset=1))> at org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:834) at
> org.junit.Assert.assertEquals(Assert.java:118) at
> org.junit.Assert.assertEquals(Assert.java:144) at
> kafka.server.epoch.EpochDrivenReplicationProtocolAcceptanceTest.shouldFollowLeaderEpochBasicWorkflow(EpochDrivenReplicationProtocolAcceptanceTest.scala:101){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7540) Flaky Test ConsumerBounceTest#testClose
[
https://issues.apache.org/jira/browse/KAFKA-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7540:
---
Summary: Flaky Test ConsumerBounceTest#testClose (was: Flaky Test
ConsumerBounceTest.testClose)
> Flaky Test ConsumerBounceTest#testClose
> ---
>
> Key: KAFKA-7540
> URL: https://issues.apache.org/jira/browse/KAFKA-7540
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer, unit tests
>Affects Versions: 2.2.0
>Reporter: John Roesler
>Assignee: Jason Gustafson
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> Observed on Java 8:
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/17314/testReport/junit/kafka.api/ConsumerBounceTest/testClose/]
>
> Stacktrace:
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: -1
> at
> kafka.integration.KafkaServerTestHarness.killBroker(KafkaServerTestHarness.scala:146)
> at
> kafka.api.ConsumerBounceTest.checkCloseWithCoordinatorFailure(ConsumerBounceTest.scala:238)
> at kafka.api.ConsumerBounceTest.testClose(ConsumerBounceTest.scala:211)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:117)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispat
[jira] [Updated] (KAFKA-7939) Flaky Test KafkaAdminClientTest#testCreateTopicsRetryBackoff
[
https://issues.apache.org/jira/browse/KAFKA-7939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7939:
---
Labels: flaky-test (was: )
> Flaky Test KafkaAdminClientTest#testCreateTopicsRetryBackoff
>
>
> Key: KAFKA-7939
> URL: https://issues.apache.org/jira/browse/KAFKA-7939
> Project: Kafka
> Issue Type: Bug
> Components: admin, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/12/]
> {quote}org.junit.runners.model.TestTimedOutException: test timed out after
> 12 milliseconds at java.lang.Object.wait(Native Method) at
> java.lang.Object.wait(Object.java:502) at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:92)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
> at
> org.apache.kafka.clients.admin.KafkaAdminClientTest.testCreateTopicsRetryBackoff(KafkaAdminClientTest.java:347){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7288) Transient failure in SslSelectorTest.testCloseConnectionInClosingState
[
https://issues.apache.org/jira/browse/KAFKA-7288?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778667#comment-16778667
]
Matthias J. Sax commented on KAFKA-7288:
[~rsivaram] This happened again in `2.0` branch:
[https://builds.apache.org/blue/organizations/jenkins/kafka-2.0-jdk8/detail/kafka-2.0-jdk8/230/tests]
Can you back port the fix to `2.0` ?
> Transient failure in SslSelectorTest.testCloseConnectionInClosingState
> --
>
> Key: KAFKA-7288
> URL: https://issues.apache.org/jira/browse/KAFKA-7288
> Project: Kafka
> Issue Type: Bug
> Components: unit tests
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Major
> Fix For: 2.1.0
>
>
> Noticed this failure in SslSelectorTest.testCloseConnectionInClosingState a
> few times in unit tests in Jenkins:
> {quote}
> java.lang.AssertionError: Channel not expired expected null, but
> was: at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotNull(Assert.java:755) at
> org.junit.Assert.assertNull(Assert.java:737) at
> org.apache.kafka.common.network.SelectorTest.testCloseConnectionInClosingState(SelectorTest.java:341)
> {quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-8005) Flaky Test ServerShutdownTest#testCleanShutdownAfterFailedStartupDueToCorruptLogs
Matthias J. Sax created KAFKA-8005:
--
Summary: Flaky Test
ServerShutdownTest#testCleanShutdownAfterFailedStartupDueToCorruptLogs
Key: KAFKA-8005
URL: https://issues.apache.org/jira/browse/KAFKA-8005
Project: Kafka
Issue Type: Bug
Components: core, unit tests
Affects Versions: 2.3.0
Reporter: Matthias J. Sax
Fix For: 2.3.0
[https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk8/detail/kafka-trunk-jdk8/3418/tests]
{quote}java.lang.AssertionError: expected:<0> but was:<1>
at org.junit.Assert.fail(Assert.java:89)
at org.junit.Assert.failNotEquals(Assert.java:835)
at org.junit.Assert.assertEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:633)
at
kafka.server.ServerShutdownTest.verifyNonDaemonThreadsStatus(ServerShutdownTest.scala:190)
at
kafka.server.ServerShutdownTest.verifyCleanShutdownAfterFailedStartup(ServerShutdownTest.scala:180)
at
kafka.server.ServerShutdownTest.testCleanShutdownAfterFailedStartupDueToCorruptLogs(ServerShutdownTest.scala:157)
{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-6824) Flaky Test DynamicBrokerReconfigurationTest#testAddRemoveSslListener
[ https://issues.apache.org/jira/browse/KAFKA-6824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-6824: --- Labels: flaky-test (was: ) > Flaky Test DynamicBrokerReconfigurationTest#testAddRemoveSslListener > > > Key: KAFKA-6824 > URL: https://issues.apache.org/jira/browse/KAFKA-6824 > Project: Kafka > Issue Type: Bug > Components: core, unit tests >Affects Versions: 2.2.0 >Reporter: Anna Povzner >Assignee: Rajini Sivaram >Priority: Critical > Labels: flaky-test > Fix For: 2.3.0, 2.2.1 > > > Observed two failures of this test (both in PR builds) :( > > *Failure #1: (JDK 7 and Scala 2.11 )* > *17:20:49* kafka.server.DynamicBrokerReconfigurationTest > > testAddRemoveSslListener FAILED > *17:20:49* java.lang.AssertionError: expected:<10> but was:<12> > *17:20:49* at org.junit.Assert.fail(Assert.java:88) > *17:20:49* at org.junit.Assert.failNotEquals(Assert.java:834) > *17:20:49* at org.junit.Assert.assertEquals(Assert.java:645) > *17:20:49* at org.junit.Assert.assertEquals(Assert.java:631) > *17:20:49* at > kafka.server.DynamicBrokerReconfigurationTest.verifyProduceConsume(DynamicBrokerReconfigurationTest.scala:959) > *17:20:49* at > kafka.server.DynamicBrokerReconfigurationTest.verifyRemoveListener(DynamicBrokerReconfigurationTest.scala:784) > *17:20:49* at > kafka.server.DynamicBrokerReconfigurationTest.testAddRemoveSslListener(DynamicBrokerReconfigurationTest.scala:705) > > *Failure #2: (JDK 8)* > *18:46:23* kafka.server.DynamicBrokerReconfigurationTest > > testAddRemoveSslListener FAILED > *18:46:23* java.util.concurrent.ExecutionException: > org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is > not the leader for that topic-partition. > *18:46:23* at > org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:94) > *18:46:23* at > org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:77) > *18:46:23* at > org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:29) > *18:46:23* at > kafka.server.DynamicBrokerReconfigurationTest.$anonfun$verifyProduceConsume$3(DynamicBrokerReconfigurationTest.scala:953) > *18:46:23* at > scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:234) > *18:46:23* at scala.collection.Iterator.foreach(Iterator.scala:929) > *18:46:23* at scala.collection.Iterator.foreach$(Iterator.scala:929) > *18:46:23* at > scala.collection.AbstractIterator.foreach(Iterator.scala:1417) > *18:46:23* at > scala.collection.IterableLike.foreach(IterableLike.scala:71) > *18:46:23* at > scala.collection.IterableLike.foreach$(IterableLike.scala:70) > *18:46:23* at > scala.collection.AbstractIterable.foreach(Iterable.scala:54) > *18:46:23* at > scala.collection.TraversableLike.map(TraversableLike.scala:234) > *18:46:23* at > scala.collection.TraversableLike.map$(TraversableLike.scala:227) > *18:46:23* at > scala.collection.AbstractTraversable.map(Traversable.scala:104) > *18:46:23* at > kafka.server.DynamicBrokerReconfigurationTest.verifyProduceConsume(DynamicBrokerReconfigurationTest.scala:953) > *18:46:23* at > kafka.server.DynamicBrokerReconfigurationTest.verifyRemoveListener(DynamicBrokerReconfigurationTest.scala:816) > *18:46:23* at > kafka.server.DynamicBrokerReconfigurationTest.testAddRemoveSslListener(DynamicBrokerReconfigurationTest.scala:705) > *18:46:23* > *18:46:23* Caused by: > *18:46:23* > org.apache.kafka.common.errors.NotLeaderForPartitionException: This server is > not the leader for that topic-partition. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7964) Flaky Test ConsumerBounceTest#testConsumerReceivesFatalExceptionWhenGroupPassesMaxSize
[
https://issues.apache.org/jira/browse/KAFKA-7964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7964:
---
Labels: flaky-test (was: )
> Flaky Test
> ConsumerBounceTest#testConsumerReceivesFatalExceptionWhenGroupPassesMaxSize
> --
>
> Key: KAFKA-7964
> URL: https://issues.apache.org/jira/browse/KAFKA-7964
> Project: Kafka
> Issue Type: Bug
> Components: clients, consumer, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/21/]
> {quote}java.lang.AssertionError: expected:<100> but was:<0> at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:834) at
> org.junit.Assert.assertEquals(Assert.java:645) at
> org.junit.Assert.assertEquals(Assert.java:631) at
> kafka.api.ConsumerBounceTest.receiveExactRecords(ConsumerBounceTest.scala:551)
> at
> kafka.api.ConsumerBounceTest.$anonfun$testConsumerReceivesFatalExceptionWhenGroupPassesMaxSize$2(ConsumerBounceTest.scala:409)
> at
> kafka.api.ConsumerBounceTest.$anonfun$testConsumerReceivesFatalExceptionWhenGroupPassesMaxSize$2$adapted(ConsumerBounceTest.scala:408)
> at scala.collection.mutable.ResizableArray.foreach(ResizableArray.scala:62)
> at scala.collection.mutable.ResizableArray.foreach$(ResizableArray.scala:55)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:49) at
> kafka.api.ConsumerBounceTest.testConsumerReceivesFatalExceptionWhenGroupPassesMaxSize(ConsumerBounceTest.scala:408){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7980) Flaky Test SocketServerTest#testConnectionRateLimit
[
https://issues.apache.org/jira/browse/KAFKA-7980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7980:
---
Labels: flaky-test (was: )
> Flaky Test SocketServerTest#testConnectionRateLimit
> ---
>
> Key: KAFKA-7980
> URL: https://issues.apache.org/jira/browse/KAFKA-7980
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/25/]
> {quote}java.lang.AssertionError: Connections created too quickly: 4 at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.assertTrue(Assert.java:41) at
> kafka.network.SocketServerTest.testConnectionRateLimit(SocketServerTest.scala:1122){quote}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7966) Flaky Test DynamicBrokerReconfigurationTest#testLogCleanerConfig
[
https://issues.apache.org/jira/browse/KAFKA-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7966:
---
Labels: flaky-test (was: )
> Flaky Test DynamicBrokerReconfigurationTest#testLogCleanerConfig
>
>
> Key: KAFKA-7966
> URL: https://issues.apache.org/jira/browse/KAFKA-7966
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/22/]
> {quote}java.lang.AssertionError: Partition [__consumer_offsets,0] metadata
> not propagated after 15000 ms at
> kafka.utils.TestUtils$.fail(TestUtils.scala:356) at
> kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:766) at
> kafka.utils.TestUtils$.waitUntilMetadataIsPropagated(TestUtils.scala:855) at
> kafka.utils.TestUtils$.$anonfun$createTopic$1(TestUtils.scala:303) at
> kafka.utils.TestUtils$.$anonfun$createTopic$1$adapted(TestUtils.scala:302) at
> scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:237) at
> scala.collection.immutable.Range.foreach(Range.scala:158) at
> scala.collection.TraversableLike.map(TraversableLike.scala:237) at
> scala.collection.TraversableLike.map$(TraversableLike.scala:230) at
> scala.collection.AbstractTraversable.map(Traversable.scala:108) at
> kafka.utils.TestUtils$.createTopic(TestUtils.scala:302) at
> kafka.server.DynamicBrokerReconfigurationTest.setUp(DynamicBrokerReconfigurationTest.scala:137){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7312) Transient failure in kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts
[
https://issues.apache.org/jira/browse/KAFKA-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7312:
---
Affects Version/s: 2.3.0
> Transient failure in
> kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts
>
>
> Key: KAFKA-7312
> URL: https://issues.apache.org/jira/browse/KAFKA-7312
> Project: Kafka
> Issue Type: Bug
> Components: unit tests
>Affects Versions: 2.3.0
>Reporter: Guozhang Wang
>Priority: Critical
>
> {code}
> Error Message
> org.junit.runners.model.TestTimedOutException: test timed out after 12
> milliseconds
> Stacktrace
> org.junit.runners.model.TestTimedOutException: test timed out after 12
> milliseconds
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:92)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:262)
> at
> kafka.utils.TestUtils$.assertFutureExceptionTypeEquals(TestUtils.scala:1345)
> at
> kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts(AdminClientIntegrationTest.scala:1080)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-8003) Flaky Test TransactionsTest #testFencingOnTransactionExpiration
[
https://issues.apache.org/jira/browse/KAFKA-8003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-8003:
---
Labels: flaky-test (was: )
> Flaky Test TransactionsTest #testFencingOnTransactionExpiration
> ---
>
> Key: KAFKA-8003
> URL: https://issues.apache.org/jira/browse/KAFKA-8003
> Project: Kafka
> Issue Type: Bug
> Components: core, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> [https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/34/]
> {quote}java.lang.AssertionError: expected:<1> but was:<0> at
> org.junit.Assert.fail(Assert.java:88) at
> org.junit.Assert.failNotEquals(Assert.java:834) at
> org.junit.Assert.assertEquals(Assert.java:645) at
> org.junit.Assert.assertEquals(Assert.java:631) at
> kafka.api.TransactionsTest.testFencingOnTransactionExpiration(TransactionsTest.scala:510){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7312) Transient failure in kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts
[
https://issues.apache.org/jira/browse/KAFKA-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7312:
---
Fix Version/s: 2.3.0
> Transient failure in
> kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts
>
>
> Key: KAFKA-7312
> URL: https://issues.apache.org/jira/browse/KAFKA-7312
> Project: Kafka
> Issue Type: Bug
> Components: unit tests
>Affects Versions: 2.3.0
>Reporter: Guozhang Wang
>Priority: Critical
> Fix For: 2.3.0
>
>
> {code}
> Error Message
> org.junit.runners.model.TestTimedOutException: test timed out after 12
> milliseconds
> Stacktrace
> org.junit.runners.model.TestTimedOutException: test timed out after 12
> milliseconds
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:92)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:262)
> at
> kafka.utils.TestUtils$.assertFutureExceptionTypeEquals(TestUtils.scala:1345)
> at
> kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts(AdminClientIntegrationTest.scala:1080)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7312) Transient failure in kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts
[
https://issues.apache.org/jira/browse/KAFKA-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7312:
---
Component/s: admin
> Transient failure in
> kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts
>
>
> Key: KAFKA-7312
> URL: https://issues.apache.org/jira/browse/KAFKA-7312
> Project: Kafka
> Issue Type: Bug
> Components: admin, unit tests
>Affects Versions: 2.3.0
>Reporter: Guozhang Wang
>Priority: Critical
> Fix For: 2.3.0
>
>
> {code}
> Error Message
> org.junit.runners.model.TestTimedOutException: test timed out after 12
> milliseconds
> Stacktrace
> org.junit.runners.model.TestTimedOutException: test timed out after 12
> milliseconds
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:92)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:262)
> at
> kafka.utils.TestUtils$.assertFutureExceptionTypeEquals(TestUtils.scala:1345)
> at
> kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts(AdminClientIntegrationTest.scala:1080)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7312) Transient failure in kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts
[
https://issues.apache.org/jira/browse/KAFKA-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778673#comment-16778673
]
Matthias J. Sax commented on KAFKA-7312:
Happened again on `trunk`:
[https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk8/detail/kafka-trunk-jdk8/3418/tests]
{quote}org.junit.runners.model.TestTimedOutException: test timed out after
12 milliseconds
at java.lang.Object.wait(Native Method)
at java.lang.Object.wait(Object.java:502)
at
org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:92)
at
org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
at kafka.utils.TestUtils$.assertFutureExceptionTypeEquals(TestUtils.scala:1431)
at
kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts(AdminClientIntegrationTest.scala:1071){quote}
> Transient failure in
> kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts
>
>
> Key: KAFKA-7312
> URL: https://issues.apache.org/jira/browse/KAFKA-7312
> Project: Kafka
> Issue Type: Bug
> Components: admin, unit tests
>Affects Versions: 2.3.0
>Reporter: Guozhang Wang
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0
>
>
> {code}
> Error Message
> org.junit.runners.model.TestTimedOutException: test timed out after 12
> milliseconds
> Stacktrace
> org.junit.runners.model.TestTimedOutException: test timed out after 12
> milliseconds
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:92)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:262)
> at
> kafka.utils.TestUtils$.assertFutureExceptionTypeEquals(TestUtils.scala:1345)
> at
> kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts(AdminClientIntegrationTest.scala:1080)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (KAFKA-7937) Flaky Test ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup
[
https://issues.apache.org/jira/browse/KAFKA-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax reopened KAFKA-7937:
Happened again on `trunk`:
[https://builds.apache.org/blue/organizations/jenkins/kafka-trunk-jdk11/detail/kafka-trunk-jdk11/319/tests]
> Flaky Test ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup
>
>
> Key: KAFKA-7937
> URL: https://issues.apache.org/jira/browse/KAFKA-7937
> Project: Kafka
> Issue Type: Bug
> Components: admin, clients, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Gwen Shapira
>Priority: Critical
> Fix For: 2.2.0
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> https://builds.apache.org/blue/organizations/jenkins/kafka-2.2-jdk8/detail/kafka-2.2-jdk8/19/pipeline
> {quote}kafka.admin.ResetConsumerGroupOffsetTest >
> testResetOffsetsNotExistingGroup FAILED
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.CoordinatorNotAvailableException: The
> coordinator is not available. at
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
> at
> kafka.admin.ConsumerGroupCommand$ConsumerGroupService.resetOffsets(ConsumerGroupCommand.scala:306)
> at
> kafka.admin.ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup(ResetConsumerGroupOffsetTest.scala:89)
> Caused by: org.apache.kafka.common.errors.CoordinatorNotAvailableException:
> The coordinator is not available.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7937) Flaky Test ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup
[
https://issues.apache.org/jira/browse/KAFKA-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7937:
---
Fix Version/s: (was: 2.2.0)
2.2.1
2.3.0
> Flaky Test ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup
>
>
> Key: KAFKA-7937
> URL: https://issues.apache.org/jira/browse/KAFKA-7937
> Project: Kafka
> Issue Type: Bug
> Components: admin, clients, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Gwen Shapira
>Priority: Critical
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> https://builds.apache.org/blue/organizations/jenkins/kafka-2.2-jdk8/detail/kafka-2.2-jdk8/19/pipeline
> {quote}kafka.admin.ResetConsumerGroupOffsetTest >
> testResetOffsetsNotExistingGroup FAILED
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.CoordinatorNotAvailableException: The
> coordinator is not available. at
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
> at
> kafka.admin.ConsumerGroupCommand$ConsumerGroupService.resetOffsets(ConsumerGroupCommand.scala:306)
> at
> kafka.admin.ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup(ResetConsumerGroupOffsetTest.scala:89)
> Caused by: org.apache.kafka.common.errors.CoordinatorNotAvailableException:
> The coordinator is not available.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6755) MaskField SMT should optionally take a literal value to use instead of using null
[
https://issues.apache.org/jira/browse/KAFKA-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778679#comment-16778679
]
Randall Hauch commented on KAFKA-6755:
--
I've reviewed the PR, and it's in great shape. Time to create the KIP. :-D
First, I don't see an account in the Apache wiki, so create an account. See
https://cwiki.apache.org/confluence/signup.action
Once that's done, I'll give you normal edit privileges for the wiki.
Finally, you can then create a KIP following the procedure on
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals,
focusing on the public-facing API and behavioral changes. Happy to review it
before you publish it, or we can just discuss on the KIP discussion email
thread that you'll create.
Let me know if you have any questions, and once you have a wiki account.
> MaskField SMT should optionally take a literal value to use instead of using
> null
> -
>
> Key: KAFKA-6755
> URL: https://issues.apache.org/jira/browse/KAFKA-6755
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Assignee: Valeria Vasylieva
>Priority: Major
> Labels: needs-kip, newbie
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The existing {{org.apache.kafka.connect.transforms.MaskField}} SMT always
> uses the null value for the type of field. It'd be nice to *optionally* be
> able to specify a literal value for the type, where the SMT would convert the
> literal string value in the configuration to the desired type (using the new
> {{Values}} methods).
> Use cases: mask out the IP address, or SSN, or other personally identifiable
> information (PII).
> Since this changes the API, and thus will require a KIP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7312) Transient failure in kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts
[
https://issues.apache.org/jira/browse/KAFKA-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7312:
---
Labels: flaky-test (was: )
> Transient failure in
> kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts
>
>
> Key: KAFKA-7312
> URL: https://issues.apache.org/jira/browse/KAFKA-7312
> Project: Kafka
> Issue Type: Bug
> Components: admin, unit tests
>Affects Versions: 2.3.0
>Reporter: Guozhang Wang
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.3.0
>
>
> {code}
> Error Message
> org.junit.runners.model.TestTimedOutException: test timed out after 12
> milliseconds
> Stacktrace
> org.junit.runners.model.TestTimedOutException: test timed out after 12
> milliseconds
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:92)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:262)
> at
> kafka.utils.TestUtils$.assertFutureExceptionTypeEquals(TestUtils.scala:1345)
> at
> kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts(AdminClientIntegrationTest.scala:1080)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7312) Transient failure in kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts
[
https://issues.apache.org/jira/browse/KAFKA-7312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7312:
---
Priority: Critical (was: Major)
> Transient failure in
> kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts
>
>
> Key: KAFKA-7312
> URL: https://issues.apache.org/jira/browse/KAFKA-7312
> Project: Kafka
> Issue Type: Bug
> Components: unit tests
>Reporter: Guozhang Wang
>Priority: Critical
>
> {code}
> Error Message
> org.junit.runners.model.TestTimedOutException: test timed out after 12
> milliseconds
> Stacktrace
> org.junit.runners.model.TestTimedOutException: test timed out after 12
> milliseconds
> at java.lang.Object.wait(Native Method)
> at java.lang.Object.wait(Object.java:502)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:92)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:262)
> at
> kafka.utils.TestUtils$.assertFutureExceptionTypeEquals(TestUtils.scala:1345)
> at
> kafka.api.AdminClientIntegrationTest.testMinimumRequestTimeouts(AdminClientIntegrationTest.scala:1080)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7937) Flaky Test ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup
[
https://issues.apache.org/jira/browse/KAFKA-7937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7937:
---
Affects Version/s: 2.3.0
> Flaky Test ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup
>
>
> Key: KAFKA-7937
> URL: https://issues.apache.org/jira/browse/KAFKA-7937
> Project: Kafka
> Issue Type: Bug
> Components: admin, clients, unit tests
>Affects Versions: 2.2.0, 2.3.0
>Reporter: Matthias J. Sax
>Assignee: Gwen Shapira
>Priority: Critical
> Fix For: 2.3.0, 2.2.1
>
>
> To get stable nightly builds for `2.2` release, I create tickets for all
> observed test failures.
> https://builds.apache.org/blue/organizations/jenkins/kafka-2.2-jdk8/detail/kafka-2.2-jdk8/19/pipeline
> {quote}kafka.admin.ResetConsumerGroupOffsetTest >
> testResetOffsetsNotExistingGroup FAILED
> java.util.concurrent.ExecutionException:
> org.apache.kafka.common.errors.CoordinatorNotAvailableException: The
> coordinator is not available. at
> org.apache.kafka.common.internals.KafkaFutureImpl.wrapAndThrow(KafkaFutureImpl.java:45)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.access$000(KafkaFutureImpl.java:32)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl$SingleWaiter.await(KafkaFutureImpl.java:89)
> at
> org.apache.kafka.common.internals.KafkaFutureImpl.get(KafkaFutureImpl.java:260)
> at
> kafka.admin.ConsumerGroupCommand$ConsumerGroupService.resetOffsets(ConsumerGroupCommand.scala:306)
> at
> kafka.admin.ResetConsumerGroupOffsetTest.testResetOffsetsNotExistingGroup(ResetConsumerGroupOffsetTest.scala:89)
> Caused by: org.apache.kafka.common.errors.CoordinatorNotAvailableException:
> The coordinator is not available.{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7990) Flaky Test KafkaStreamsTest#shouldCleanupOldStateDirs
[
https://issues.apache.org/jira/browse/KAFKA-7990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778778#comment-16778778
]
ASF GitHub Bot commented on KAFKA-7990:
---
guozhangwang commented on pull request #6334: KAFKA-7990: Close streams at the
end in KafkaStreamsTest
URL: https://github.com/apache/kafka/pull/6334
This fix is already in 2.1+ branches, but did not get into older branches.
Should be cherry-picked all the way to 0.10.2.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Flaky Test KafkaStreamsTest#shouldCleanupOldStateDirs
> -
>
> Key: KAFKA-7990
> URL: https://issues.apache.org/jira/browse/KAFKA-7990
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Priority: Major
>
> [https://builds.apache.org/blue/organizations/jenkins/kafka-2.0-jdk8/detail/kafka-2.0-jdk8/229/tests]
>
> {quote}Exception in thread
> "appId-78a5ef7e-0f4d-47bd-af2e-54f4606fb19e-StreamThread-189"
> java.lang.IllegalArgumentException: Assigned partition input-0 for
> non-subscribed topic regex pattern; subscription pattern is topic
> at
> org.apache.kafka.clients.consumer.internals.SubscriptionState.assignFromSubscribed(SubscriptionState.java:187)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:244)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:422)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:352)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:337)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:343)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1218)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1175)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1154)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:861)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:810)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:767)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:736){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KAFKA-7990) Flaky Test KafkaStreamsTest#shouldCleanupOldStateDirs
[
https://issues.apache.org/jira/browse/KAFKA-7990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guozhang Wang reassigned KAFKA-7990:
Assignee: Guozhang Wang
> Flaky Test KafkaStreamsTest#shouldCleanupOldStateDirs
> -
>
> Key: KAFKA-7990
> URL: https://issues.apache.org/jira/browse/KAFKA-7990
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Guozhang Wang
>Priority: Major
>
> [https://builds.apache.org/blue/organizations/jenkins/kafka-2.0-jdk8/detail/kafka-2.0-jdk8/229/tests]
>
> {quote}Exception in thread
> "appId-78a5ef7e-0f4d-47bd-af2e-54f4606fb19e-StreamThread-189"
> java.lang.IllegalArgumentException: Assigned partition input-0 for
> non-subscribed topic regex pattern; subscription pattern is topic
> at
> org.apache.kafka.clients.consumer.internals.SubscriptionState.assignFromSubscribed(SubscriptionState.java:187)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:244)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:422)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:352)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:337)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:343)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1218)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1175)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1154)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:861)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:810)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:767)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:736){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7996) KafkaStreams does not pass timeout when closing Producer
[ https://issues.apache.org/jira/browse/KAFKA-7996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778780#comment-16778780 ] Guozhang Wang commented on KAFKA-7996: -- If a KafkaStreams instance is in ERROR state, one should call close() on it to make sure all the corresponding resources (state directory, threads, locks, etc) are reclaimed before starting a new one. The cleanup() call will just wipe out the local state stores, and is usually used if you do not want to have another instance on this host and just want to free-up its state store footprint on disks. > KafkaStreams does not pass timeout when closing Producer > > > Key: KAFKA-7996 > URL: https://issues.apache.org/jira/browse/KAFKA-7996 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.1.0 >Reporter: Patrik Kleindl >Assignee: Lee Dongjin >Priority: Major > Labels: needs-kip > > [https://confluentcommunity.slack.com/messages/C48AHTCUQ/convo/C48AHTCUQ-1550831721.026100/] > We are running 2.1 and have a case where the shutdown of a streams > application takes several minutes > I noticed that although we call streams.close with a timeout of 30 seconds > the log says > [Producer > clientId=…-8be49feb-8a2e-4088-bdd7-3c197f6107bb-StreamThread-1-producer] > Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. > Matthias J Sax [vor 3 Tagen] > I just checked the code, and yes, we don't provide a timeout for the producer > on close()... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7996) KafkaStreams does not pass timeout when closing Producer
[ https://issues.apache.org/jira/browse/KAFKA-7996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778805#comment-16778805 ] Matthias J. Sax commented on KAFKA-7996: I still think this ticket is valuable, as we should allow to specify a timeout for all `close()` calls of internal client. Thus, do we want to add new configs as suggested on the PR and go with a KIP? This way, we can keep the current `KafkaStreams#close()` semantics and address the blocking issue for internal clients. Thoughts? > KafkaStreams does not pass timeout when closing Producer > > > Key: KAFKA-7996 > URL: https://issues.apache.org/jira/browse/KAFKA-7996 > Project: Kafka > Issue Type: Bug > Components: streams >Affects Versions: 2.1.0 >Reporter: Patrik Kleindl >Assignee: Lee Dongjin >Priority: Major > Labels: needs-kip > > [https://confluentcommunity.slack.com/messages/C48AHTCUQ/convo/C48AHTCUQ-1550831721.026100/] > We are running 2.1 and have a case where the shutdown of a streams > application takes several minutes > I noticed that although we call streams.close with a timeout of 30 seconds > the log says > [Producer > clientId=…-8be49feb-8a2e-4088-bdd7-3c197f6107bb-StreamThread-1-producer] > Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. > Matthias J Sax [vor 3 Tagen] > I just checked the code, and yes, we don't provide a timeout for the producer > on close()... -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7994) Improve Stream-Time for rebalances and restarts
[
https://issues.apache.org/jira/browse/KAFKA-7994?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778810#comment-16778810
]
Matthias J. Sax commented on KAFKA-7994:
{quote}So I'm feeling that if we can tackle those issues within a single
solution, then maybe we can consider propagating the stream time inferred at
the "head" sub-topologies, plus storing it at the offset metadata.
{quote}
This might be helpful. I actually believe, that `suppress()` might face a
similar issue atm. \cc [~vvcephei] What is the impact on `suppress()` on
restarts? Could it be that we emit incorrectly for this case?
> Improve Stream-Time for rebalances and restarts
> ---
>
> Key: KAFKA-7994
> URL: https://issues.apache.org/jira/browse/KAFKA-7994
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Reporter: Matthias J. Sax
>Priority: Major
>
> We compute a per-partition partition-time as the maximum timestamp over all
> records processed so far. Furthermore, we use partition-time to compute
> stream-time for each task as maximum over all partition-times (for all
> corresponding task partitions). This stream-time is used to make decisions
> about processing out-of-order records or drop them if they are late (ie,
> timestamp < stream-time - grace-period).
> During rebalances and restarts, stream-time is initialized as UNKNOWN (ie,
> -1) for tasks that are newly created (or migrated). In net effect, we forget
> current stream-time for this case what may lead to non-deterministic behavior
> if we stop processing right before a late record, that would be dropped if we
> continue processing, but is not dropped after rebalance/restart. Let's look
> at an examples with a grade period of 5ms for a tumbling windowed of 5ms, and
> the following records (timestamps in parenthesis):
>
> {code:java}
> r1(0) r2(5) r3(11) r4(2){code}
> In the example, stream-time advances as 0, 5, 11, 11 and thus record `r4` is
> dropped as late because 2 < 6 = 11 - 5. However, if we stop processing or
> rebalance after processing `r3` but before processing `r4`, we would
> reinitialize stream-time as -1, and thus would process `r4` on restart/after
> rebalance. The problem is, that stream-time does advance differently from a
> global point of view: 0, 5, 11, 2.
>
> Note, this is a corner case, because if we would stop processing one record
> earlier, ie, after processing `r2` but before processing `r3`, stream-time
> would be advance correctly from a global point of view.
> A potential fix would be, to store latest observed partition-time in the
> metadata of committed offsets. Thus way, on restart/rebalance we can
> re-initialize time correctly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-8006) Guard calls to init() and close() within user code
Matthias J. Sax created KAFKA-8006: -- Summary: Guard calls to init() and close() within user code Key: KAFKA-8006 URL: https://issues.apache.org/jira/browse/KAFKA-8006 Project: Kafka Issue Type: Improvement Components: streams Reporter: Matthias J. Sax When using a global state store within the provided Processor, the user is able to call init() and close() on the state stores. Those APIs should only be called by Kafka Streams runtime but not by the user. We should guard those APIs so that the user cannot call them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8006) Guard calls to init() and close() within user code
[ https://issues.apache.org/jira/browse/KAFKA-8006?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778820#comment-16778820 ] Matthias J. Sax commented on KAFKA-8006: This ticket is a follow up to KAFKA-6970 that added guards for regular stores. > Guard calls to init() and close() within user code > -- > > Key: KAFKA-8006 > URL: https://issues.apache.org/jira/browse/KAFKA-8006 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Matthias J. Sax >Priority: Major > Labels: beginner, newbie > > When using a global state store within the provided Processor, the user is > able to call init() and close() on the state stores. Those APIs should only > be called by Kafka Streams runtime but not by the user. > We should guard those APIs so that the user cannot call them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-6970) Kafka streams lets the user call init() and close() on a state store, when inside Processors
[ https://issues.apache.org/jira/browse/KAFKA-6970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778821#comment-16778821 ] Matthias J. Sax commented on KAFKA-6970: Created https://issues.apache.org/jira/browse/KAFKA-8006 as follow up because it will get a new "fix version" number. > Kafka streams lets the user call init() and close() on a state store, when > inside Processors > > > Key: KAFKA-6970 > URL: https://issues.apache.org/jira/browse/KAFKA-6970 > Project: Kafka > Issue Type: Bug > Components: streams >Reporter: James Cheng >Assignee: Nikolay Izhikov >Priority: Major > Fix For: 2.2.0 > > > When using a state store within Transform (and Processor and > TransformValues), the user is able to call init() and close() on the state > stores. Those APIs should only be called by kafka streams itself. > If possible, it would be good to guard those APIs so that the user cannot > call them. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-8007) In-memory window store copies on fetch
Sophie Blee-Goldman created KAFKA-8007: -- Summary: In-memory window store copies on fetch Key: KAFKA-8007 URL: https://issues.apache.org/jira/browse/KAFKA-8007 Project: Kafka Issue Type: Improvement Reporter: Sophie Blee-Goldman Assignee: Sophie Blee-Goldman The current implementation of InMemoryWindowStore copies all record data into a new list and returns an iterator over that list when fetching. That is inefficient in terms of both memory and time. The fetching logic should be moved to the iterators, which could be cleaned up as well (currently they rely on ListIterator to provide peekNextKey() functionality, which is unnecessary). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-8007) In-memory window store copies on fetch
[ https://issues.apache.org/jira/browse/KAFKA-8007?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778960#comment-16778960 ] ASF GitHub Bot commented on KAFKA-8007: --- ableegoldman commented on pull request #6335: KAFKA-8007: Avoid copying on fetch in InMemoryWindowStore URL: https://github.com/apache/kafka/pull/6335 Rewrote the InMemoryWindowStore implementation by moving the work of a fetch to the iterator, and cleaned up the iterators as well. Blocked by [KAFKA-7918](https://github.com/apache/kafka/pull/6328) ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > In-memory window store copies on fetch > -- > > Key: KAFKA-8007 > URL: https://issues.apache.org/jira/browse/KAFKA-8007 > Project: Kafka > Issue Type: Improvement >Reporter: Sophie Blee-Goldman >Assignee: Sophie Blee-Goldman >Priority: Major > > The current implementation of InMemoryWindowStore copies all record data into > a new list and returns an iterator over that list when fetching. That is > inefficient in terms of both memory and time. The fetching logic should be > moved to the iterators, which could be cleaned up as well (currently they > rely on ListIterator to provide peekNextKey() functionality, which is > unnecessary). -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7990) Flaky Test KafkaStreamsTest#shouldCleanupOldStateDirs
[
https://issues.apache.org/jira/browse/KAFKA-7990?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16778953#comment-16778953
]
ASF GitHub Bot commented on KAFKA-7990:
---
guozhangwang commented on pull request #6334: KAFKA-7990: Close streams at the
end in KafkaStreamsTest
URL: https://github.com/apache/kafka/pull/6334
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Flaky Test KafkaStreamsTest#shouldCleanupOldStateDirs
> -
>
> Key: KAFKA-7990
> URL: https://issues.apache.org/jira/browse/KAFKA-7990
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Affects Versions: 2.2.0
>Reporter: Matthias J. Sax
>Assignee: Guozhang Wang
>Priority: Major
>
> [https://builds.apache.org/blue/organizations/jenkins/kafka-2.0-jdk8/detail/kafka-2.0-jdk8/229/tests]
>
> {quote}Exception in thread
> "appId-78a5ef7e-0f4d-47bd-af2e-54f4606fb19e-StreamThread-189"
> java.lang.IllegalArgumentException: Assigned partition input-0 for
> non-subscribed topic regex pattern; subscription pattern is topic
> at
> org.apache.kafka.clients.consumer.internals.SubscriptionState.assignFromSubscribed(SubscriptionState.java:187)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:244)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.joinGroupIfNeeded(AbstractCoordinator.java:422)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:352)
> at
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:337)
> at
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:343)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1218)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1175)
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1154)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.pollRequests(StreamThread.java:861)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runOnce(StreamThread.java:810)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:767)
> at
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:736){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7912) In-memory key-value store does not support concurrent access
[ https://issues.apache.org/jira/browse/KAFKA-7912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779003#comment-16779003 ] ASF GitHub Bot commented on KAFKA-7912: --- ableegoldman commented on pull request #6336: KAFKA-7912: Support concurrent access in InMemoryKeyValueStore URL: https://github.com/apache/kafka/pull/6336 Previously the InMemoryKeyValue store would throw a ConcurrentModificationException if the store was modified beneath an open iterator. The TreeMap implementation was swapped with a ConcurrentSkipListMap for similar performance while supporting concurrent access. Added one test to AbstractKeyValueStoreTest, no existing tests caught this. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > In-memory key-value store does not support concurrent access > - > > Key: KAFKA-7912 > URL: https://issues.apache.org/jira/browse/KAFKA-7912 > Project: Kafka > Issue Type: Bug >Reporter: Sophie Blee-Goldman >Assignee: Sophie Blee-Goldman >Priority: Major > > Currently, the in-memory key-value store uses a Map to store key-value pairs > and fetches them by calling subMap and returning an iterator to this submap. > This is unsafe as the submap is just a view of the original map and there is > risk of concurrent access. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
