[jira] [Created] (KAFKA-7958) Transactions are broken with kubernetes hosted brokers
Thomas Dickinson created KAFKA-7958: --- Summary: Transactions are broken with kubernetes hosted brokers Key: KAFKA-7958 URL: https://issues.apache.org/jira/browse/KAFKA-7958 Project: Kafka Issue Type: Bug Affects Versions: 2.1.1 Environment: cp-kakfka 2.1.1-1, kafka-streams 2.1.1 Reporter: Thomas Dickinson After a rolling re-start in a kubernetes-like environment, brokers may change IP address. From our logs it seems that the transaction manager in the brokers never re-resolves the DNS name of other brokers, keeping stale pod IPs. Thus transactions stop working. ??[2019-02-20 02:20:20,085] WARN [TransactionCoordinator id=1001] Connection to node 0 (khaki-joey-kafka-0.khaki-joey-kafka-headless.hyperspace-dev/[10.233.124.181:9092|http://10.233.124.181:9092/]) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)?? ??[2019-02-20 02:20:57,205] WARN [TransactionCoordinator id=1001] Connection to node 1 (khaki-joey-kafka-1.khaki-joey-kafka-headless.hyperspace-dev/[10.233.122.67:9092|http://10.233.122.67:9092/]) could not be established. Broker may not be available. (org.apache.kafka.clients.NetworkClient)?? This is from the log from broker 1001 which was restarted first, followed by 1 and then 0. The log entries are from the day after the rolling restart. I note a similar issue was fixed for clients 2.1.1 https://issues.apache.org/jira/browse/KAFKA-7755. We are using streams lib 2.1.1 We have turned off EOS in our stream applications to work-around this issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7834) Extend collected logs in system test services to include heap dumps
[
https://issues.apache.org/jira/browse/KAFKA-7834?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ewen Cheslack-Postava updated KAFKA-7834:
-
Fix Version/s: 2.1.2
> Extend collected logs in system test services to include heap dumps
> ---
>
> Key: KAFKA-7834
> URL: https://issues.apache.org/jira/browse/KAFKA-7834
> Project: Kafka
> Issue Type: Improvement
> Components: system tests
>Reporter: Konstantine Karantasis
>Assignee: Konstantine Karantasis
>Priority: Major
> Fix For: 2.2.0, 2.0.2, 2.3.0, 2.1.2
>
>
> Overall I'd suggest enabling by default:
> {\{-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="}}
> in the major system test services, so that a heap dump is captured on OOM.
> Given these flags, we should also extend the set of collected logs in each
> service to include the predetermined filename for the heap dump.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7934) Optimize restore for windowed and session stores
[ https://issues.apache.org/jira/browse/KAFKA-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772544#comment-16772544 ] Matthias J. Sax commented on KAFKA-7934: It's not impossible. However, Kafka does not allow us to read a topic backwards. The consumer logic would be quite unnatural if we try to do this. > Optimize restore for windowed and session stores > > > Key: KAFKA-7934 > URL: https://issues.apache.org/jira/browse/KAFKA-7934 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Matthias J. Sax >Priority: Major > > During state restore of window/session stores, the changelog topic is scanned > from the oldest entries to the newest entry. This happen on a > record-per-record basis or in record batches. > During this process, new segments are created while time advances (base on > the record timestamp of the record that are restored). However, depending on > the retention time, we might expire segments during restore process later > again. This is wasteful. Because retention time is based on the largest > timestamp per partition, it is possible to compute a bound for live and > expired segment upfront (assuming that we know the largest timestamp). This > way, during restore, we could avoid creating segments that are expired later > anyway and skip over all corresponding records. > The problem is, that we don't know the largest timestamp per partition. Maybe > the broker timestamp index could help to provide an approximation for this > value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7133) DisconnectException every 5 minutes in single restore consumer thread
[
https://issues.apache.org/jira/browse/KAFKA-7133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772532#comment-16772532
]
sd1700092 commented on KAFKA-7133:
--
I have met the similar problem. Mine is in the Kafka Connect when I use my sink
connectors, I encoutered "INFO [Consumer clientId=consumer-4,
groupId=connect-KuduSink_0218_2] Error sending fetch request
(sessionId=INVALID, epoch=INITIAL) to node 51:
org.apache.kafka.common.errors.DisconnectException.
(org.apache.kafka.clients.FetchSessionHandler:440) "。。。
Any help would be deeply appreciated.
> DisconnectException every 5 minutes in single restore consumer thread
> -
>
> Key: KAFKA-7133
> URL: https://issues.apache.org/jira/browse/KAFKA-7133
> Project: Kafka
> Issue Type: Bug
> Components: streams
>Affects Versions: 1.1.0
> Environment: Kafka Streams application in Kubernetes.
> Kafka Server in Docker on machine in host mode
>Reporter: Chris Schwarzfischer
>Priority: Major
>
> One of our streams applications (and only this one) gets a
> {{org.apache.kafka.common.errors.DisconnectException}} almost exactly every 5
> minutes.
> The application has two of
> KStream -> KGroupedStream -> KTable -> KGroupedTable -> KTable
> aggregations.
> Relevant config is in Streams:
> {code:java}
> this.properties.put(StreamsConfig.PROCESSING_GUARANTEE_CONFIG,
> StreamsConfig.AT_LEAST_ONCE);
> //...
> this.properties.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 2);
> this.properties.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 1024 *
> 1024 * 500 /* 500 MB */ );
> this.properties.put(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG, 1024 *
> 1024 * 100 /* 100 MB */);
> this.properties.put(ConsumerConfig.FETCH_MAX_BYTES_CONFIG, 1024 * 1024 * 50
> /* 50 MB */);
> {code}
> On the broker:
> {noformat}
> KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 3
> KAFKA_OFFSETS_RETENTION_MINUTES: 108000
> KAFKA_MIN_INSYNC_REPLICAS: 2
> KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 3
> KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 2
> KAFKA_TRANSACTIONAL_ID_EXPIRATION_MS: 2147483000
> KAFKA_LOG_RETENTION_HOURS: 2688
> KAFKA_OFFSETS_RETENTION_CHECK_INTERVAL_MS: 120
> KAFKA_ZOOKEEPER_SESSION_TIMEOUT_MS: 12000
> {noformat}
> Logging gives us a single restore consumer thread that throws exceptions
> every 5 mins:
>
> {noformat}
> July 4th 2018, 15:38:51.560 dockertest032018-07-04T13:38:51,559Z INFO
> :
> [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
> FetchSessionHandler::handleError:440 - [Consumer
> clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
> groupId=] Error sending fetch request (sessionId=317141939, epoch=INITIAL)
> to node 1: org.apache.kafka.common.errors.DisconnectException.
> July 4th 2018, 15:37:54.833 dockertest032018-07-04T13:37:54,832Z INFO
> :
> [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
> FetchSessionHandler::handleError:440 - [Consumer
> clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
> groupId=] Error sending fetch request (sessionId=2064325970, epoch=INITIAL)
> to node 3: org.apache.kafka.common.errors.DisconnectException.
> July 4th 2018, 15:37:54.833 dockertest032018-07-04T13:37:54,832Z INFO
> :
> [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
> FetchSessionHandler::handleError:440 - [Consumer
> clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
> groupId=] Error sending fetch request (sessionId=1735432619, epoch=INITIAL)
> to node 2: org.apache.kafka.common.errors.DisconnectException.
> July 4th 2018, 15:32:26.379 dockertest032018-07-04T13:32:26,378Z INFO
> :
> [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
> FetchSessionHandler::handleError:440 - [Consumer
> clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
> groupId=] Error sending fetch request (sessionId=317141939, epoch=INITIAL)
> to node 1: org.apache.kafka.common.errors.DisconnectException.
> July 4th 2018, 15:32:01.926 dockertest032018-07-04T13:32:01,925Z INFO
> :
> [testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2][]:
> FetchSessionHandler::handleError:440 - [Consumer
> clientId=testdev-cs9-test-aggregate-udrs-e39ef2d4-452b-4697-b031-26fc1bac8831-StreamThread-2-restore-consumer,
> groupId=] Error sending fetch request (sessionId=1735432619, epoch=INITIAL)
> to node 2: org.apache.kafka.common.errors.DisconnectException.
> July 4th 201
[jira] [Commented] (KAFKA-7934) Optimize restore for windowed and session stores
[ https://issues.apache.org/jira/browse/KAFKA-7934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772534#comment-16772534 ] Sophie Blee-Goldman commented on KAFKA-7934: Is there some reason we can't just work backwards from the last message using a putIfAbsent method (would need to be implemented for these stores, I believe..) That would definitely minimize the number of expired records we insert then delete > Optimize restore for windowed and session stores > > > Key: KAFKA-7934 > URL: https://issues.apache.org/jira/browse/KAFKA-7934 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Matthias J. Sax >Priority: Major > > During state restore of window/session stores, the changelog topic is scanned > from the oldest entries to the newest entry. This happen on a > record-per-record basis or in record batches. > During this process, new segments are created while time advances (base on > the record timestamp of the record that are restored). However, depending on > the retention time, we might expire segments during restore process later > again. This is wasteful. Because retention time is based on the largest > timestamp per partition, it is possible to compute a bound for live and > expired segment upfront (assuming that we know the largest timestamp). This > way, during restore, we could avoid creating segments that are expired later > anyway and skip over all corresponding records. > The problem is, that we don't know the largest timestamp per partition. Maybe > the broker timestamp index could help to provide an approximation for this > value. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7927) Read committed receives aborted events
[
https://issues.apache.org/jira/browse/KAFKA-7927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772533#comment-16772533
]
Guozhang Wang commented on KAFKA-7927:
--
[~huxi_2b] Thanks for the quick response!
[~gsomogyi] would you like to update the title / description of the JIRA ticket
as an cmd tool improvement for older versions?
> Read committed receives aborted events
> --
>
> Key: KAFKA-7927
> URL: https://issues.apache.org/jira/browse/KAFKA-7927
> Project: Kafka
> Issue Type: Bug
> Components: consumer, core, producer
>Affects Versions: 1.0.0
>Reporter: Gabor Somogyi
>Priority: Blocker
> Attachments: KafkaProducer.scala, consumer.log, producer.log.gz
>
>
> When a kafka client produces ~30k events and at the end it aborts the
> transaction a consumer can read part of the aborted messages when
> "isolation.level" set to "READ_COMMITTED".
> Kafka client version: 2.0.0
> Kafka broker version: 1.0.0
> Producer:
> {code:java}
> java -jar
> kafka-producer/target/kafka-producer-0.0.1-SNAPSHOT-jar-with-dependencies.jar
> gsomogyi-cdh5144-220cloudera2-1.gce.cloudera.com:9092 src-topic
> {code}
> See attached code.
> Consumer:
> {code:java}
> kafka-console-consumer --zookeeper localhost:2181 --topic src-topic
> --from-beginning --isolation-level read_committed
> {code}
> Same behavior seen when re-implemented the consumer in scala.
> The whole application can be found here:
> https://github.com/gaborgsomogyi/kafka-semantics-tester
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-7957) Flaky Test DynamicBrokerReconfigurationTest #testMetricsReporterUpdate
Matthias J. Sax created KAFKA-7957:
--
Summary: Flaky Test DynamicBrokerReconfigurationTest
#testMetricsReporterUpdate
Key: KAFKA-7957
URL: https://issues.apache.org/jira/browse/KAFKA-7957
Project: Kafka
Issue Type: Bug
Components: core, unit tests
Affects Versions: 2.2.0
Reporter: Matthias J. Sax
Fix For: 2.2.0
To get stable nightly builds for `2.2` release, I create tickets for all
observed test failures.
[https://jenkins.confluent.io/job/apache-kafka-test/job/2.2/18/]
{quote}java.lang.AssertionError: Messages not sent at
kafka.utils.TestUtils$.fail(TestUtils.scala:356) at
kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:766) at
kafka.server.DynamicBrokerReconfigurationTest.startProduceConsume(DynamicBrokerReconfigurationTest.scala:1270)
at
kafka.server.DynamicBrokerReconfigurationTest.testMetricsReporterUpdate(DynamicBrokerReconfigurationTest.scala:650){quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7927) Read committed receives aborted events
[
https://issues.apache.org/jira/browse/KAFKA-7927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772477#comment-16772477
]
huxihx commented on KAFKA-7927:
---
[~gsomogyi] Currently, the old consumer had already been removed from the
codebase. That's to say, you cannot specify `--zookeeper` for ConsoleConsumer
anymore.
> Read committed receives aborted events
> --
>
> Key: KAFKA-7927
> URL: https://issues.apache.org/jira/browse/KAFKA-7927
> Project: Kafka
> Issue Type: Bug
> Components: consumer, core, producer
>Affects Versions: 1.0.0
>Reporter: Gabor Somogyi
>Priority: Blocker
> Attachments: KafkaProducer.scala, consumer.log, producer.log.gz
>
>
> When a kafka client produces ~30k events and at the end it aborts the
> transaction a consumer can read part of the aborted messages when
> "isolation.level" set to "READ_COMMITTED".
> Kafka client version: 2.0.0
> Kafka broker version: 1.0.0
> Producer:
> {code:java}
> java -jar
> kafka-producer/target/kafka-producer-0.0.1-SNAPSHOT-jar-with-dependencies.jar
> gsomogyi-cdh5144-220cloudera2-1.gce.cloudera.com:9092 src-topic
> {code}
> See attached code.
> Consumer:
> {code:java}
> kafka-console-consumer --zookeeper localhost:2181 --topic src-topic
> --from-beginning --isolation-level read_committed
> {code}
> Same behavior seen when re-implemented the consumer in scala.
> The whole application can be found here:
> https://github.com/gaborgsomogyi/kafka-semantics-tester
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7909) Ensure timely rebalance completion after pending members rejoin or fail
[ https://issues.apache.org/jira/browse/KAFKA-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772458#comment-16772458 ] ASF GitHub Bot commented on KAFKA-7909: --- hachikuji commented on pull request #6251: KAFKA-7909: Ensure timely rebalance completion after pending members rejoin or fail URL: https://github.com/apache/kafka/pull/6251 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Ensure timely rebalance completion after pending members rejoin or fail > --- > > Key: KAFKA-7909 > URL: https://issues.apache.org/jira/browse/KAFKA-7909 > Project: Kafka > Issue Type: Bug > Components: consumer, core >Reporter: Arjun Satish >Assignee: Arjun Satish >Priority: Blocker > Fix For: 2.2.0 > > > We recently introduced integration tests in Connect. This test spins up one > or more Connect workers along with a Kafka broker and Zk in a single process > and attempts to move records using a Connector. In the [Example Integration > Test|https://github.com/apache/kafka/blob/3c73633/connect/runtime/src/test/java/org/apache/kafka/connect/integration/ExampleConnectIntegrationTest.java#L105], > we spin up three workers each hosting a Connector task that consumes records > from a Kafka topic. When the connector starts up, it may go through multiple > rounds of rebalancing. We notice the following two problems in the last few > days: > # After members join a group, there are no pendingMembers remaining, but the > join group method does not complete, and send these members a signal that > they are not ready to start consuming from their respective partitions. > # Because of quick rebalances, a consumer might have started a group, but > Connect starts a rebalance, after we which we create three new instances of > the consumer (one from each worker/task). But the group coordinator seems to > have 4 members in the group. This causes the JoinGroup to indefinitely stall. > Even though this ticket is described in the connect of Connect, it may be > applicable to general consumers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7909) Ensure timely rebalance completion after pending members rejoin or fail
[ https://issues.apache.org/jira/browse/KAFKA-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson updated KAFKA-7909: --- Affects Version/s: (was: 2.2.0) > Ensure timely rebalance completion after pending members rejoin or fail > --- > > Key: KAFKA-7909 > URL: https://issues.apache.org/jira/browse/KAFKA-7909 > Project: Kafka > Issue Type: Bug > Components: consumer, core >Reporter: Arjun Satish >Assignee: Arjun Satish >Priority: Blocker > Fix For: 2.2.0 > > > We recently introduced integration tests in Connect. This test spins up one > or more Connect workers along with a Kafka broker and Zk in a single process > and attempts to move records using a Connector. In the [Example Integration > Test|https://github.com/apache/kafka/blob/3c73633/connect/runtime/src/test/java/org/apache/kafka/connect/integration/ExampleConnectIntegrationTest.java#L105], > we spin up three workers each hosting a Connector task that consumes records > from a Kafka topic. When the connector starts up, it may go through multiple > rounds of rebalancing. We notice the following two problems in the last few > days: > # After members join a group, there are no pendingMembers remaining, but the > join group method does not complete, and send these members a signal that > they are not ready to start consuming from their respective partitions. > # Because of quick rebalances, a consumer might have started a group, but > Connect starts a rebalance, after we which we create three new instances of > the consumer (one from each worker/task). But the group coordinator seems to > have 4 members in the group. This causes the JoinGroup to indefinitely stall. > Even though this ticket is described in the connect of Connect, it may be > applicable to general consumers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7909) Ensure timely rebalance completion after pending members rejoin or fail
[ https://issues.apache.org/jira/browse/KAFKA-7909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arjun Satish updated KAFKA-7909: Summary: Ensure timely rebalance completion after pending members rejoin or fail (was: Coordinator changes cause Connect integration test to fail) > Ensure timely rebalance completion after pending members rejoin or fail > --- > > Key: KAFKA-7909 > URL: https://issues.apache.org/jira/browse/KAFKA-7909 > Project: Kafka > Issue Type: Bug > Components: consumer, core >Affects Versions: 2.2.0 >Reporter: Arjun Satish >Assignee: Arjun Satish >Priority: Blocker > Fix For: 2.2.0 > > > We recently introduced integration tests in Connect. This test spins up one > or more Connect workers along with a Kafka broker and Zk in a single process > and attempts to move records using a Connector. In the [Example Integration > Test|https://github.com/apache/kafka/blob/3c73633/connect/runtime/src/test/java/org/apache/kafka/connect/integration/ExampleConnectIntegrationTest.java#L105], > we spin up three workers each hosting a Connector task that consumes records > from a Kafka topic. When the connector starts up, it may go through multiple > rounds of rebalancing. We notice the following two problems in the last few > days: > # After members join a group, there are no pendingMembers remaining, but the > join group method does not complete, and send these members a signal that > they are not ready to start consuming from their respective partitions. > # Because of quick rebalances, a consumer might have started a group, but > Connect starts a rebalance, after we which we create three new instances of > the consumer (one from each worker/task). But the group coordinator seems to > have 4 members in the group. This causes the JoinGroup to indefinitely stall. > Even though this ticket is described in the connect of Connect, it may be > applicable to general consumers. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7956) Avoid blocking in ShutdownableThread.awaitShutdown if the thread has not been started
Gardner Vickers created KAFKA-7956: -- Summary: Avoid blocking in ShutdownableThread.awaitShutdown if the thread has not been started Key: KAFKA-7956 URL: https://issues.apache.org/jira/browse/KAFKA-7956 Project: Kafka Issue Type: Improvement Components: core Reporter: Gardner Vickers Opening this Jira to track [https://github.com/apache/kafka/pull/6218], since it's a rather subtle change. In some test cases it's desirable to instantiate a subclass of `ShutdownableThread` without starting it. Since most subclasses of `ShutdownableThread` put cleanup logic in `ShutdownableThread.shutdown()`, being able to call `shutdown()` on the non-running thread would be useful. This change allows us to avoid blocking in `ShutdownableThread.shutdown()` if the thread's `run()` method has not been called. We also add a check that `initiateShutdown()` was called before `awaitShutdown()`, to protect against the case where a user calls `awaitShutdown()` before the thread has been started, and unexpectedly is not blocked on the thread shutting down. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7955) Provide a BOM for EmbeddedConnectCluster and EmbeddedCluster
Jeremy Custenborder created KAFKA-7955:
--
Summary: Provide a BOM for EmbeddedConnectCluster and
EmbeddedCluster
Key: KAFKA-7955
URL: https://issues.apache.org/jira/browse/KAFKA-7955
Project: Kafka
Issue Type: Improvement
Components: KafkaConnect
Affects Versions: 2.1.1
Reporter: Jeremy Custenborder
Using EmbeddedConnectCluster for testing connectors is a little difficult given
the number of dependencies that are required. Providing a
[BOM|https://maven.apache.org/guides/introduction/introduction-to-dependency-mechanism.html]
will make it easier for connector developers. For example here are the
dependencies that are required.
{code:xml}
org.apache.kafka
connect-api
${kafka.version}
org.apache.kafka
connect-runtime
${kafka.version}
test
test-jar
org.apache.kafka
connect-runtime
${kafka.version}
org.apache.kafka
kafka-clients
${kafka.version}
junit
junit
4.12
org.apache.kafka
kafka-clients
${kafka.version}
test
test-jar
org.apache.kafka
kafka_2.11
${kafka.version}
org.apache.kafka
kafka_2.11
test-jar
test
${kafka.version}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7954) When broker IP addresses change client never resolves new addresses to fetch metadata
[ https://issues.apache.org/jira/browse/KAFKA-7954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772329#comment-16772329 ] Ismael Juma commented on KAFKA-7954: Have you looked at the code in trunk? There have been a few fixes after 2.1.0 was released. > When broker IP addresses change client never resolves new addresses to fetch > metadata > - > > Key: KAFKA-7954 > URL: https://issues.apache.org/jira/browse/KAFKA-7954 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 2.1.0 >Reporter: Robert Pofuk >Priority: Major > > I'm running Kafka on AWS autoscaling group. Currently we are having > infrastructure immutable and we have no fixed IP addresses. > When we recreate out Kafka cluster all brokers get new IP addresses, even in > case when all nodes die horrible death simultaneously. > Looking at the code IP addresses are resolved and added > ClusterConnectionStates class line 119: > > nodeState.put(id, new NodeConnectionState(ConnectionState.CONNECTING, now, > this.reconnectBackoffInitMs, ClientUtils.resolve(host, clientDnsLookup))); > At this point ClientUtils.resolve resolves IP addreses. If brokers receive > new IP addresses each subsequent metadata fetch will fail because none of the > nodes will be addressable by IP addresses that where resolved on previous > metadata fetch. > > Since addresses list will never be evicted old IP addresses will stay there > forever. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7954) When broker IP addresses change client never resolves new addresses to fetch metadata
[ https://issues.apache.org/jira/browse/KAFKA-7954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772328#comment-16772328 ] Robert Pofuk commented on KAFKA-7954: - Also I would expect client to fall back to bootstrap servers list after certain amount of metadata failures > When broker IP addresses change client never resolves new addresses to fetch > metadata > - > > Key: KAFKA-7954 > URL: https://issues.apache.org/jira/browse/KAFKA-7954 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 2.1.0 >Reporter: Robert Pofuk >Priority: Major > > I'm running Kafka on AWS autoscaling group. Currently we are having > infrastructure immutable and we have no fixed IP addresses. > When we recreate out Kafka cluster all brokers get new IP addresses, even in > case when all nodes die horrible death simultaneously. > Looking at the code IP addresses are resolved and added > ClusterConnectionStates class line 119: > > nodeState.put(id, new NodeConnectionState(ConnectionState.CONNECTING, now, > this.reconnectBackoffInitMs, ClientUtils.resolve(host, clientDnsLookup))); > At this point ClientUtils.resolve resolves IP addreses. If brokers receive > new IP addresses each subsequent metadata fetch will fail because none of the > nodes will be addressable by IP addresses that where resolved on previous > metadata fetch. > > Since addresses list will never be evicted old IP addresses will stay there > forever. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7954) When broker IP addresses change client never resolves new addresses to fetch metadata
Robert Pofuk created KAFKA-7954: --- Summary: When broker IP addresses change client never resolves new addresses to fetch metadata Key: KAFKA-7954 URL: https://issues.apache.org/jira/browse/KAFKA-7954 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 2.1.0 Reporter: Robert Pofuk I'm running Kafka on AWS autoscaling group. Currently we are having infrastructure immutable and we have no fixed IP addresses. When we recreate out Kafka cluster all brokers get new IP addresses, even in case when all nodes die horrible death simultaneously. Looking at the code IP addresses are resolved and added ClusterConnectionStates class line 119: nodeState.put(id, new NodeConnectionState(ConnectionState.CONNECTING, now, this.reconnectBackoffInitMs, ClientUtils.resolve(host, clientDnsLookup))); At this point ClientUtils.resolve resolves IP addreses. If brokers receive new IP addresses each subsequent metadata fetch will fail because none of the nodes will be addressable by IP addresses that where resolved on previous metadata fetch. Since addresses list will never be evicted old IP addresses will stay there forever. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7953) Kafka Producer buffer-full policy should be configurable
[ https://issues.apache.org/jira/browse/KAFKA-7953?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772301#comment-16772301 ] Roman Leventov commented on KAFKA-7953: --- For context, this issue originates from this discussion: https://github.com/apache/incubator-druid/issues/7075 > Kafka Producer buffer-full policy should be configurable > > > Key: KAFKA-7953 > URL: https://issues.apache.org/jira/browse/KAFKA-7953 > Project: Kafka > Issue Type: Improvement > Components: producer >Reporter: Justin Borromeo >Priority: Major > Labels: features > > Currently, any attempt to write a message to a full KafkaProducer queue will > block until the message is written or `max.block.ms` is reached. For some > use cases, it would be preferable to treat the new message as higher-priority > and expire the oldest message from the queue. If there isn't a way to do > that with the existing implementation, an additional configuration value for > expiration policy (either expire the newest message or the oldest message) > would be useful. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7799) Fix flaky test RestServerTest.testCORSEnabled
[
https://issues.apache.org/jira/browse/KAFKA-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksandr Diachenko updated KAFKA-7799:
---
Fix Version/s: 2.1.2
2.0.2
2.2.0
> Fix flaky test RestServerTest.testCORSEnabled
> -
>
> Key: KAFKA-7799
> URL: https://issues.apache.org/jira/browse/KAFKA-7799
> Project: Kafka
> Issue Type: Test
> Components: KafkaConnect
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Major
> Fix For: 2.2.0, 2.0.2, 2.3.0, 2.1.2
>
>
> Starting to see this failure quite a lot, locally and on jenkins:
> {code}
> org.apache.kafka.connect.runtime.rest.RestServerTest.testCORSEnabled
> Failing for the past 7 builds (Since Failed#18600 )
> Took 0.7 sec.
> Error Message
> java.lang.AssertionError: expected: but was:
> Stacktrace
> java.lang.AssertionError: expected: but was:
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:118)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at
> org.apache.kafka.connect.runtime.rest.RestServerTest.checkCORSRequest(RestServerTest.java:221)
> at
> org.apache.kafka.connect.runtime.rest.RestServerTest.testCORSEnabled(RestServerTest.java:84)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.junit.internal.runners.TestMethod.invoke(TestMethod.java:68)
> at
> org.powermock.modules.junit4.internal.impl.PowerMockJUnit44RunnerDelegateImpl$PowerMockJUnit44MethodRunner.runTestMethod(PowerMockJUnit44RunnerDelegateImpl.java:326)
> at org.junit.internal.runners.MethodRoadie$2.run(MethodRoadie.java:89)
> at
> org.junit.internal.runners.MethodRoadie.runBeforesThenTestThenAfters(MethodRoadie.java:97)
> {code}
> If it helps, I see an uncaught exception in the stdout:
> {code}
> [2019-01-08 19:35:23,664] ERROR Uncaught exception in REST call to
> /connector-plugins/FileStreamSource/validate
> (org.apache.kafka.connect.runtime.rest.errors.ConnectExceptionMapper:61)
> javax.ws.rs.NotFoundException: HTTP 404 Not Found
> at
> org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:274)
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:272)
> at org.glassfish.jersey.internal.Errors$1.call(Errors.java:268)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:316)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:298)
> at org.glassfish.jersey.internal.Errors.process(Errors.java:268)
> at
> org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:289)
> at
> org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:256)
> at
> org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:703)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KAFKA-7953) Kafka Producer buffer-full policy should be configurable
Justin Borromeo created KAFKA-7953: -- Summary: Kafka Producer buffer-full policy should be configurable Key: KAFKA-7953 URL: https://issues.apache.org/jira/browse/KAFKA-7953 Project: Kafka Issue Type: Improvement Components: producer Reporter: Justin Borromeo Currently, any attempt to write a message to a full KafkaProducer queue will block until the message is written or `max.block.ms` is reached. For some use cases, it would be preferable to treat the new message as higher-priority and expire the oldest message from the queue. If there isn't a way to do that with the existing implementation, an additional configuration value for expiration policy (either expire the newest message or the oldest message) would be useful. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7932) Streams needs to handle new Producer exceptions
[ https://issues.apache.org/jira/browse/KAFKA-7932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772275#comment-16772275 ] Matthias J. Sax commented on KAFKA-7932: [~vvcephei] Can you split this into two Jiras? On for `initTx()` (based on 6446) so we can back port this fix to older release and one based on 7763. Also, please link all Jiras to each other. Thanks. > Streams needs to handle new Producer exceptions > --- > > Key: KAFKA-7932 > URL: https://issues.apache.org/jira/browse/KAFKA-7932 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.0.0, 2.0.1, 2.1.0, 2.2.0, 2.1.1 >Reporter: John Roesler >Assignee: John Roesler >Priority: Critical > Fix For: 2.3.0 > > > Following on KAFKA-7763, Streams needs to handle the new behavior. > See also https://github.com/apache/kafka/pull/6066 > Streams code (StreamTask.java) needs to be modified to handle the new > exception. > Also, from another upstream change, `initTxn` can also throw TimeoutException > now: default `MAX_BLOCK_MS_CONFIG` in producer is 60 seconds, so I think just > wrapping it as StreamsException should be reasonable, similar to what we do > for `producer#send`'s TimeoutException > ([https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/RecordCollectorImpl.java#L220-L225] > ). > > Note we need to handle in three functions: init/commit/abortTxn. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KAFKA-7921) Instable KafkaStreamsTest
[
https://issues.apache.org/jira/browse/KAFKA-7921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
John Roesler reassigned KAFKA-7921:
---
Assignee: Matthias J. Sax (was: John Roesler)
> Instable KafkaStreamsTest
> -
>
> Key: KAFKA-7921
> URL: https://issues.apache.org/jira/browse/KAFKA-7921
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Major
>
> {{KafkaStreamsTest}} failed multiple times, eg,
> {quote}java.lang.AssertionError: Condition not met within timeout 15000.
> Streams never started.
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:365)
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:325)
> at
> org.apache.kafka.streams.KafkaStreamsTest.shouldThrowOnCleanupWhileRunning(KafkaStreamsTest.java:556){quote}
> or
> {quote}java.lang.AssertionError: Condition not met within timeout 15000.
> Streams never started.
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:365)
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:325)
> at
> org.apache.kafka.streams.KafkaStreamsTest.testStateThreadClose(KafkaStreamsTest.java:255){quote}
>
> The preserved logs are as follows:
> {quote}[2019-02-12 07:02:17,198] INFO Kafka version: 2.3.0-SNAPSHOT
> (org.apache.kafka.common.utils.AppInfoParser:109)
> [2019-02-12 07:02:17,198] INFO Kafka commitId: 08036fa4b1e5b822
> (org.apache.kafka.common.utils.AppInfoParser:110)
> [2019-02-12 07:02:17,199] INFO stream-client [clientId] State transition from
> CREATED to REBALANCING (org.apache.kafka.streams.KafkaStreams:263)
> [2019-02-12 07:02:17,200] INFO stream-thread [clientId-StreamThread-238]
> Starting (org.apache.kafka.streams.processor.internals.StreamThread:767)
> [2019-02-12 07:02:17,200] INFO stream-client [clientId] State transition from
> REBALANCING to PENDING_SHUTDOWN (org.apache.kafka.streams.KafkaStreams:263)
> [2019-02-12 07:02:17,200] INFO stream-thread [clientId-StreamThread-239]
> Starting (org.apache.kafka.streams.processor.internals.StreamThread:767)
> [2019-02-12 07:02:17,200] INFO stream-thread [clientId-StreamThread-238]
> State transition from CREATED to STARTING
> (org.apache.kafka.streams.processor.internals.StreamThread:214)
> [2019-02-12 07:02:17,200] INFO stream-thread [clientId-StreamThread-239]
> State transition from CREATED to STARTING
> (org.apache.kafka.streams.processor.internals.StreamThread:214)
> [2019-02-12 07:02:17,200] INFO stream-thread [clientId-StreamThread-238]
> Informed to shut down
> (org.apache.kafka.streams.processor.internals.StreamThread:1192)
> [2019-02-12 07:02:17,201] INFO stream-thread [clientId-StreamThread-238]
> State transition from STARTING to PENDING_SHUTDOWN
> (org.apache.kafka.streams.processor.internals.StreamThread:214)
> [2019-02-12 07:02:17,201] INFO stream-thread [clientId-StreamThread-239]
> Informed to shut down
> (org.apache.kafka.streams.processor.internals.StreamThread:1192)
> [2019-02-12 07:02:17,201] INFO stream-thread [clientId-StreamThread-239]
> State transition from STARTING to PENDING_SHUTDOWN
> (org.apache.kafka.streams.processor.internals.StreamThread:214)
> [2019-02-12 07:02:17,205] INFO Cluster ID: J8uJhiTKQx-Y_i9LzT0iLg
> (org.apache.kafka.clients.Metadata:365)
> [2019-02-12 07:02:17,205] INFO Cluster ID: J8uJhiTKQx-Y_i9LzT0iLg
> (org.apache.kafka.clients.Metadata:365)
> [2019-02-12 07:02:17,205] INFO [Consumer
> clientId=clientId-StreamThread-238-consumer, groupId=appId] Discovered group
> coordinator localhost:36122 (id: 2147483647 rack: null)
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:675)
> [2019-02-12 07:02:17,205] INFO [Consumer
> clientId=clientId-StreamThread-239-consumer, groupId=appId] Discovered group
> coordinator localhost:36122 (id: 2147483647 rack: null)
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:675)
> [2019-02-12 07:02:17,206] INFO [Consumer
> clientId=clientId-StreamThread-238-consumer, groupId=appId] Revoking
> previously assigned partitions []
> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:458)
> [2019-02-12 07:02:17,206] INFO [Consumer
> clientId=clientId-StreamThread-239-consumer, groupId=appId] Revoking
> previously assigned partitions []
> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:458)
> [2019-02-12 07:02:17,206] INFO [Consumer
> clientId=clientId-StreamThread-238-consumer, groupId=appId] (Re-)joining
> group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:491)
> [2019-02-12 07:02:17,206] INFO [Consumer
> clientId=clientId-StreamThread-239-consumer, groupId=appId] (Re-)joining
> group (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:491)
> [2019-02-12 07:02:17,2
[jira] [Commented] (KAFKA-7921) Instable KafkaStreamsTest
[
https://issues.apache.org/jira/browse/KAFKA-7921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772240#comment-16772240
]
John Roesler commented on KAFKA-7921:
-
I haven't seen a failure since we merged the PR to capture logs.
Checked:
[https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/test_results_analyzer/]
[https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/test_results_analyzer/]
> Instable KafkaStreamsTest
> -
>
> Key: KAFKA-7921
> URL: https://issues.apache.org/jira/browse/KAFKA-7921
> Project: Kafka
> Issue Type: Bug
> Components: streams, unit tests
>Reporter: Matthias J. Sax
>Assignee: John Roesler
>Priority: Major
>
> {{KafkaStreamsTest}} failed multiple times, eg,
> {quote}java.lang.AssertionError: Condition not met within timeout 15000.
> Streams never started.
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:365)
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:325)
> at
> org.apache.kafka.streams.KafkaStreamsTest.shouldThrowOnCleanupWhileRunning(KafkaStreamsTest.java:556){quote}
> or
> {quote}java.lang.AssertionError: Condition not met within timeout 15000.
> Streams never started.
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:365)
> at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:325)
> at
> org.apache.kafka.streams.KafkaStreamsTest.testStateThreadClose(KafkaStreamsTest.java:255){quote}
>
> The preserved logs are as follows:
> {quote}[2019-02-12 07:02:17,198] INFO Kafka version: 2.3.0-SNAPSHOT
> (org.apache.kafka.common.utils.AppInfoParser:109)
> [2019-02-12 07:02:17,198] INFO Kafka commitId: 08036fa4b1e5b822
> (org.apache.kafka.common.utils.AppInfoParser:110)
> [2019-02-12 07:02:17,199] INFO stream-client [clientId] State transition from
> CREATED to REBALANCING (org.apache.kafka.streams.KafkaStreams:263)
> [2019-02-12 07:02:17,200] INFO stream-thread [clientId-StreamThread-238]
> Starting (org.apache.kafka.streams.processor.internals.StreamThread:767)
> [2019-02-12 07:02:17,200] INFO stream-client [clientId] State transition from
> REBALANCING to PENDING_SHUTDOWN (org.apache.kafka.streams.KafkaStreams:263)
> [2019-02-12 07:02:17,200] INFO stream-thread [clientId-StreamThread-239]
> Starting (org.apache.kafka.streams.processor.internals.StreamThread:767)
> [2019-02-12 07:02:17,200] INFO stream-thread [clientId-StreamThread-238]
> State transition from CREATED to STARTING
> (org.apache.kafka.streams.processor.internals.StreamThread:214)
> [2019-02-12 07:02:17,200] INFO stream-thread [clientId-StreamThread-239]
> State transition from CREATED to STARTING
> (org.apache.kafka.streams.processor.internals.StreamThread:214)
> [2019-02-12 07:02:17,200] INFO stream-thread [clientId-StreamThread-238]
> Informed to shut down
> (org.apache.kafka.streams.processor.internals.StreamThread:1192)
> [2019-02-12 07:02:17,201] INFO stream-thread [clientId-StreamThread-238]
> State transition from STARTING to PENDING_SHUTDOWN
> (org.apache.kafka.streams.processor.internals.StreamThread:214)
> [2019-02-12 07:02:17,201] INFO stream-thread [clientId-StreamThread-239]
> Informed to shut down
> (org.apache.kafka.streams.processor.internals.StreamThread:1192)
> [2019-02-12 07:02:17,201] INFO stream-thread [clientId-StreamThread-239]
> State transition from STARTING to PENDING_SHUTDOWN
> (org.apache.kafka.streams.processor.internals.StreamThread:214)
> [2019-02-12 07:02:17,205] INFO Cluster ID: J8uJhiTKQx-Y_i9LzT0iLg
> (org.apache.kafka.clients.Metadata:365)
> [2019-02-12 07:02:17,205] INFO Cluster ID: J8uJhiTKQx-Y_i9LzT0iLg
> (org.apache.kafka.clients.Metadata:365)
> [2019-02-12 07:02:17,205] INFO [Consumer
> clientId=clientId-StreamThread-238-consumer, groupId=appId] Discovered group
> coordinator localhost:36122 (id: 2147483647 rack: null)
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:675)
> [2019-02-12 07:02:17,205] INFO [Consumer
> clientId=clientId-StreamThread-239-consumer, groupId=appId] Discovered group
> coordinator localhost:36122 (id: 2147483647 rack: null)
> (org.apache.kafka.clients.consumer.internals.AbstractCoordinator:675)
> [2019-02-12 07:02:17,206] INFO [Consumer
> clientId=clientId-StreamThread-238-consumer, groupId=appId] Revoking
> previously assigned partitions []
> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:458)
> [2019-02-12 07:02:17,206] INFO [Consumer
> clientId=clientId-StreamThread-239-consumer, groupId=appId] Revoking
> previously assigned partitions []
> (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:458)
> [2019-02-12 07:02:17,206] INFO [Consumer
> clientId=clientId-StreamThread-238-consumer, groupId=appId] (Re-)joining
> group (org.apache.kafka.clients.consumer.internals.Abstract
[jira] [Commented] (KAFKA-7951) Log Cleaner thread stop with "Varint is too long" error
[
https://issues.apache.org/jira/browse/KAFKA-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772175#comment-16772175
]
Ninth Nails commented on KAFKA-7951:
As a temporary fix, I moved the partition on question {{__consumer_offsets-37}}
onto another broker. This allow the cleaner do to it's work on the faulty
broker, but I'm expecting the other broker's log cleaner to fails as well,
eventually.
> Log Cleaner thread stop with "Varint is too long" error
> ---
>
> Key: KAFKA-7951
> URL: https://issues.apache.org/jira/browse/KAFKA-7951
> Project: Kafka
> Issue Type: Bug
> Components: log
>Affects Versions: 0.11.0.3
> Environment: Amazon Linux AMI 2017.09
> confluent-platform-oss-2.11-3.3.2
> OpenJDK Runtime Environment (build 1.8.0_191-b12)
> EC2 instance type i3.xlarge
>Reporter: Ninth Nails
>Priority: Blocker
>
> I have one broker log cleaner thread dying because of a failing type
> conversion. My cluster is running on 0.11.03-cp1 (Confluent Platform). Here's
> the relevant logs in log-cleaner.log
> {code:java}
> [2019-02-19 15:08:23,387] priority=INFO message="[kafka-log-cleaner-thread-0]:
> Log cleaner thread 0 cleaned log __consumer_offsets-36 (dirty section
> = [931148954, 931148954])
> 13,051.5 MB of log processed in 159.0 seconds (82.1 MB/sec).
> Indexed 13,051.5 MB in 81.2 seconds (160.8 Mb/sec, 51.1% of total
> time)
> Buffer utilization: 0.0%
> Cleaned 13,051.5 MB in 77.8 seconds (167.7 Mb/sec, 48.9% of total
> time)
> Start size: 13,051.5 MB (166,676,749 messages)
> End size: 0.0 MB (159 messages)
> 100.0% size reduction (100.0% fewer messages)
> " category=kafka.log.LogCleaner
> [2019-02-19 15:08:23,392] priority=INFO message="Cleaner 0: Beginning
> cleaning of log __consumer_offsets-37." category=kafka.log.LogCleaner
> [2019-02-19 15:08:23,392] priority=INFO message="Cleaner 0: Building offset
> map for __consumer_offsets-37..." category=kafka.log.LogCleaner
> [2019-02-19 15:08:23,410] priority=INFO message="Cleaner 0: Building offset
> map for log __consumer_offsets-37 for 87 segments in offset range [211566448,
> 308245257)." category=kafka.log.LogCleaner
> [2019-02-19 15:08:23,945] priority=ERROR
> message="[kafka-log-cleaner-thread-0]: Error due to"
> category=kafka.log.LogCleaner
> java.lang.IllegalArgumentException: Varint is too long, the most significant
> bit in the 5th byte is set, converted value: 445d418c
> at
> org.apache.kafka.common.utils.ByteUtils.illegalVarintException(ByteUtils.java:326)
> at
> org.apache.kafka.common.utils.ByteUtils.readVarint(ByteUtils.java:148)
> at
> org.apache.kafka.common.record.DefaultRecord.readFrom(DefaultRecord.java:305)
> at
> org.apache.kafka.common.record.DefaultRecordBatch$2.readNext(DefaultRecordBatch.java:299)
> at
> org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:563)
> at
> org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:532)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:752)
> at
> kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:741)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> kafka.log.Cleaner.kafka$log$Cleaner$$buildOffsetMapForSegment(LogCleaner.scala:741)
> at
> kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:707)
> at
> kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:704)
> at
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
> at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:704)
> at kafka.log.Cleaner.do
[jira] [Comment Edited] (KAFKA-7933) KTableKTableLeftJoinTest takes an hour to finish
[
https://issues.apache.org/jira/browse/KAFKA-7933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772119#comment-16772119
]
Matthias J. Sax edited comment on KAFKA-7933 at 2/19/19 5:06 PM:
-
Create https://issues.apache.org/jira/browse/KAFKA-7952 a follow up. Local test
runtime dropped from 15 second to about 1 second switching to in-memory stores.
was (Author: mjsax):
Create https://issues.apache.org/jira/browse/KAFKA-7952 a follow up. Local test
runtime dropped from 15 second to less than 100ms switching to in-memory stores.
> KTableKTableLeftJoinTest takes an hour to finish
>
>
> Key: KAFKA-7933
> URL: https://issues.apache.org/jira/browse/KAFKA-7933
> Project: Kafka
> Issue Type: Improvement
> Components: streams, unit tests
>Reporter: Viktor Somogyi-Vass
>Assignee: Matthias J. Sax
>Priority: Major
> Attachments: jenkins-output-one-hour-test.log
>
>
> PRs might time out as
> {{KTableKTableLeftJoinTest.shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions}}
> took one hour to complete.
> {noformat}
> 11:57:45 org.apache.kafka.streams.kstream.internals.KTableKTableLeftJoinTest
> > shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions STARTED
> 12:53:35
> 12:53:35 org.apache.kafka.streams.kstream.internals.KTableKTableLeftJoinTest
> > shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions PASSED
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7933) KTableKTableLeftJoinTest takes an hour to finish
[
https://issues.apache.org/jira/browse/KAFKA-7933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772127#comment-16772127
]
ASF GitHub Bot commented on KAFKA-7933:
---
mjsax commented on pull request #6292: KAFKA-7933: Switch from persistent to
in-memory store in KTableKTablLeftJoinTest
URL: https://github.com/apache/kafka/pull/6292
Local test runtime drops from 15 sec to about 1 second for
`shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions()`
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> KTableKTableLeftJoinTest takes an hour to finish
>
>
> Key: KAFKA-7933
> URL: https://issues.apache.org/jira/browse/KAFKA-7933
> Project: Kafka
> Issue Type: Improvement
> Components: streams, unit tests
>Reporter: Viktor Somogyi-Vass
>Assignee: Matthias J. Sax
>Priority: Major
> Attachments: jenkins-output-one-hour-test.log
>
>
> PRs might time out as
> {{KTableKTableLeftJoinTest.shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions}}
> took one hour to complete.
> {noformat}
> 11:57:45 org.apache.kafka.streams.kstream.internals.KTableKTableLeftJoinTest
> > shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions STARTED
> 12:53:35
> 12:53:35 org.apache.kafka.streams.kstream.internals.KTableKTableLeftJoinTest
> > shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions PASSED
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Issue Comment Deleted] (KAFKA-1) The log4j appender still uses the SyncProducer API
[ https://issues.apache.org/jira/browse/KAFKA-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-1: Comment: was deleted (was: Partially blocked, because we don't have in-memory window or session stores yet.) > The log4j appender still uses the SyncProducer API > -- > > Key: KAFKA-1 > URL: https://issues.apache.org/jira/browse/KAFKA-1 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.6 >Priority: Major > > The log4j appender still uses the SyncProducer API. Change it to use the > Producer API using the StringEncoder instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7933) KTableKTableLeftJoinTest takes an hour to finish
[
https://issues.apache.org/jira/browse/KAFKA-7933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772119#comment-16772119
]
Matthias J. Sax commented on KAFKA-7933:
Create https://issues.apache.org/jira/browse/KAFKA-7952 a follow up. Local test
runtime dropped from 15 second to less than 100ms switching to in-memory stores.
> KTableKTableLeftJoinTest takes an hour to finish
>
>
> Key: KAFKA-7933
> URL: https://issues.apache.org/jira/browse/KAFKA-7933
> Project: Kafka
> Issue Type: Improvement
> Components: streams, unit tests
>Reporter: Viktor Somogyi-Vass
>Assignee: Matthias J. Sax
>Priority: Major
> Attachments: jenkins-output-one-hour-test.log
>
>
> PRs might time out as
> {{KTableKTableLeftJoinTest.shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions}}
> took one hour to complete.
> {noformat}
> 11:57:45 org.apache.kafka.streams.kstream.internals.KTableKTableLeftJoinTest
> > shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions STARTED
> 12:53:35
> 12:53:35 org.apache.kafka.streams.kstream.internals.KTableKTableLeftJoinTest
> > shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions PASSED
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7952) Consider to switch to in-memory stores in test whenever possible
[ https://issues.apache.org/jira/browse/KAFKA-7952?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772118#comment-16772118 ] Matthias J. Sax commented on KAFKA-7952: Partially blocked, because we don't have in-memory window or session stores atm. > Consider to switch to in-memory stores in test whenever possible > > > Key: KAFKA-7952 > URL: https://issues.apache.org/jira/browse/KAFKA-7952 > Project: Kafka > Issue Type: Improvement > Components: streams, unit tests >Reporter: Matthias J. Sax >Priority: Major > > We observed that tests can be very slow using default RocksDB stores (cf. > KAFKA-7933). > We should consider to switch to in-memory stores whenever possible to reduce > test runtime. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7952) Consider to switch to in-memory stores in test whenever possible
Matthias J. Sax created KAFKA-7952: -- Summary: Consider to switch to in-memory stores in test whenever possible Key: KAFKA-7952 URL: https://issues.apache.org/jira/browse/KAFKA-7952 Project: Kafka Issue Type: Improvement Components: streams, unit tests Reporter: Matthias J. Sax We observed that tests can be very slow using default RocksDB stores (cf. KAFKA-7933). We should consider to switch to in-memory stores whenever possible to reduce test runtime. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-1) The log4j appender still uses the SyncProducer API
[ https://issues.apache.org/jira/browse/KAFKA-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772116#comment-16772116 ] Matthias J. Sax commented on KAFKA-1: - Partially blocked, because we don't have in-memory window or session stores yet. > The log4j appender still uses the SyncProducer API > -- > > Key: KAFKA-1 > URL: https://issues.apache.org/jira/browse/KAFKA-1 > Project: Kafka > Issue Type: Improvement >Affects Versions: 0.6 >Priority: Major > > The log4j appender still uses the SyncProducer API. Change it to use the > Producer API using the StringEncoder instead. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KAFKA-7540) Transient failure: kafka.api.ConsumerBounceTest.testClose
[
https://issues.apache.org/jira/browse/KAFKA-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7540:
---
Priority: Critical (was: Major)
> Transient failure: kafka.api.ConsumerBounceTest.testClose
> -
>
> Key: KAFKA-7540
> URL: https://issues.apache.org/jira/browse/KAFKA-7540
> Project: Kafka
> Issue Type: Bug
>Reporter: John Roesler
>Assignee: Jason Gustafson
>Priority: Critical
> Labels: flaky-test
>
> Observed on Java 8:
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/17314/testReport/junit/kafka.api/ConsumerBounceTest/testClose/]
>
> Stacktrace:
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: -1
> at
> kafka.integration.KafkaServerTestHarness.killBroker(KafkaServerTestHarness.scala:146)
> at
> kafka.api.ConsumerBounceTest.checkCloseWithCoordinatorFailure(ConsumerBounceTest.scala:238)
> at kafka.api.ConsumerBounceTest.testClose(ConsumerBounceTest.scala:211)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:117)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(Messag
[jira] [Commented] (KAFKA-7540) Transient failure: kafka.api.ConsumerBounceTest.testClose
[
https://issues.apache.org/jira/browse/KAFKA-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772093#comment-16772093
]
Matthias J. Sax commented on KAFKA-7540:
Reopening this to get it fixed for `2.2` release.
[https://builds.apache.org/blue/organizations/jenkins/kafka-2.2-jdk8/detail/kafka-2.2-jdk8/22/changes/]
{quote}java.lang.ArrayIndexOutOfBoundsException: -1
at
kafka.integration.KafkaServerTestHarness.killBroker(KafkaServerTestHarness.scala:146)
at
kafka.api.ConsumerBounceTest.checkCloseWithCoordinatorFailure(ConsumerBounceTest.scala:252)
at kafka.api.ConsumerBounceTest.testClose(ConsumerBounceTest.scala:225){quote}
> Transient failure: kafka.api.ConsumerBounceTest.testClose
> -
>
> Key: KAFKA-7540
> URL: https://issues.apache.org/jira/browse/KAFKA-7540
> Project: Kafka
> Issue Type: Bug
>Reporter: John Roesler
>Assignee: Jason Gustafson
>Priority: Major
> Labels: flaky-test
>
> Observed on Java 8:
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/17314/testReport/junit/kafka.api/ConsumerBounceTest/testClose/]
>
> Stacktrace:
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: -1
> at
> kafka.integration.KafkaServerTestHarness.killBroker(KafkaServerTestHarness.scala:146)
> at
> kafka.api.ConsumerBounceTest.checkCloseWithCoordinatorFailure(ConsumerBounceTest.scala:238)
> at kafka.api.ConsumerBounceTest.testClose(ConsumerBounceTest.scala:211)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:117)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Un
[jira] [Commented] (KAFKA-7933) KTableKTableLeftJoinTest takes an hour to finish
[
https://issues.apache.org/jira/browse/KAFKA-7933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772101#comment-16772101
]
Matthias J. Sax commented on KAFKA-7933:
Interesting. I'll do a PR to switch to an in-memory store for this test. Let's
see if it fixes the issue.
> KTableKTableLeftJoinTest takes an hour to finish
>
>
> Key: KAFKA-7933
> URL: https://issues.apache.org/jira/browse/KAFKA-7933
> Project: Kafka
> Issue Type: Improvement
> Components: streams, unit tests
>Reporter: Viktor Somogyi-Vass
>Assignee: Matthias J. Sax
>Priority: Major
> Attachments: jenkins-output-one-hour-test.log
>
>
> PRs might time out as
> {{KTableKTableLeftJoinTest.shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions}}
> took one hour to complete.
> {noformat}
> 11:57:45 org.apache.kafka.streams.kstream.internals.KTableKTableLeftJoinTest
> > shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions STARTED
> 12:53:35
> 12:53:35 org.apache.kafka.streams.kstream.internals.KTableKTableLeftJoinTest
> > shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions PASSED
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KAFKA-7540) Transient failure: kafka.api.ConsumerBounceTest.testClose
[
https://issues.apache.org/jira/browse/KAFKA-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7540:
---
Fix Version/s: 2.2.0
> Transient failure: kafka.api.ConsumerBounceTest.testClose
> -
>
> Key: KAFKA-7540
> URL: https://issues.apache.org/jira/browse/KAFKA-7540
> Project: Kafka
> Issue Type: Bug
>Reporter: John Roesler
>Assignee: Jason Gustafson
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.2.0
>
>
> Observed on Java 8:
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/17314/testReport/junit/kafka.api/ConsumerBounceTest/testClose/]
>
> Stacktrace:
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: -1
> at
> kafka.integration.KafkaServerTestHarness.killBroker(KafkaServerTestHarness.scala:146)
> at
> kafka.api.ConsumerBounceTest.checkCloseWithCoordinatorFailure(ConsumerBounceTest.scala:238)
> at kafka.api.ConsumerBounceTest.testClose(ConsumerBounceTest.scala:211)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:117)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrap
[jira] [Reopened] (KAFKA-7540) Transient failure: kafka.api.ConsumerBounceTest.testClose
[
https://issues.apache.org/jira/browse/KAFKA-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax reopened KAFKA-7540:
> Transient failure: kafka.api.ConsumerBounceTest.testClose
> -
>
> Key: KAFKA-7540
> URL: https://issues.apache.org/jira/browse/KAFKA-7540
> Project: Kafka
> Issue Type: Bug
>Reporter: John Roesler
>Assignee: Jason Gustafson
>Priority: Major
> Labels: flaky-test
>
> Observed on Java 8:
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/17314/testReport/junit/kafka.api/ConsumerBounceTest/testClose/]
>
> Stacktrace:
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: -1
> at
> kafka.integration.KafkaServerTestHarness.killBroker(KafkaServerTestHarness.scala:146)
> at
> kafka.api.ConsumerBounceTest.checkCloseWithCoordinatorFailure(ConsumerBounceTest.scala:238)
> at kafka.api.ConsumerBounceTest.testClose(ConsumerBounceTest.scala:211)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:117)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBackedObjectConnection$DispatchWrapper.dispatch(MessageHubBackedObjectConnection.java:155)
>
[jira] [Updated] (KAFKA-7540) Transient failure: kafka.api.ConsumerBounceTest.testClose
[
https://issues.apache.org/jira/browse/KAFKA-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7540:
---
Component/s: unit tests
clients
> Transient failure: kafka.api.ConsumerBounceTest.testClose
> -
>
> Key: KAFKA-7540
> URL: https://issues.apache.org/jira/browse/KAFKA-7540
> Project: Kafka
> Issue Type: Bug
> Components: clients, unit tests
>Affects Versions: 2.2.0
>Reporter: John Roesler
>Assignee: Jason Gustafson
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.2.0
>
>
> Observed on Java 8:
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/17314/testReport/junit/kafka.api/ConsumerBounceTest/testClose/]
>
> Stacktrace:
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: -1
> at
> kafka.integration.KafkaServerTestHarness.killBroker(KafkaServerTestHarness.scala:146)
> at
> kafka.api.ConsumerBounceTest.checkCloseWithCoordinatorFailure(ConsumerBounceTest.scala:238)
> at kafka.api.ConsumerBounceTest.testClose(ConsumerBounceTest.scala:211)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:117)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
[jira] [Updated] (KAFKA-7540) Transient failure: kafka.api.ConsumerBounceTest.testClose
[
https://issues.apache.org/jira/browse/KAFKA-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias J. Sax updated KAFKA-7540:
---
Affects Version/s: 2.2.0
> Transient failure: kafka.api.ConsumerBounceTest.testClose
> -
>
> Key: KAFKA-7540
> URL: https://issues.apache.org/jira/browse/KAFKA-7540
> Project: Kafka
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: John Roesler
>Assignee: Jason Gustafson
>Priority: Critical
> Labels: flaky-test
> Fix For: 2.2.0
>
>
> Observed on Java 8:
> [https://builds.apache.org/job/kafka-pr-jdk8-scala2.11/17314/testReport/junit/kafka.api/ConsumerBounceTest/testClose/]
>
> Stacktrace:
> {noformat}
> java.lang.ArrayIndexOutOfBoundsException: -1
> at
> kafka.integration.KafkaServerTestHarness.killBroker(KafkaServerTestHarness.scala:146)
> at
> kafka.api.ConsumerBounceTest.checkCloseWithCoordinatorFailure(ConsumerBounceTest.scala:238)
> at kafka.api.ConsumerBounceTest.testClose(ConsumerBounceTest.scala:211)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.runTestClass(JUnitTestClassExecutor.java:106)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:58)
> at
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecutor.execute(JUnitTestClassExecutor.java:38)
> at
> org.gradle.api.internal.tasks.testing.junit.AbstractJUnitTestClassProcessor.processTestClass(AbstractJUnitTestClassProcessor.java:66)
> at
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:51)
> at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
> at
> org.gradle.internal.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
> at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
> at
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:117)
> at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
> at
> org.gradle.internal.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
> at
> org.gradle.internal.remote.internal.hub.MessageHubBa
[jira] [Commented] (KAFKA-7951) Log Cleaner thread stop with "Varint is too long" error
[
https://issues.apache.org/jira/browse/KAFKA-7951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16772045#comment-16772045
]
Ninth Nails commented on KAFKA-7951:
The converted value is 445d418c, 1146962316 in base 10. Obviously in Java the
maximum value for a integer is 2147483647. So I don't understand how the
converted value doesn't fit in a integer.
> Log Cleaner thread stop with "Varint is too long" error
> ---
>
> Key: KAFKA-7951
> URL: https://issues.apache.org/jira/browse/KAFKA-7951
> Project: Kafka
> Issue Type: Bug
> Components: log
>Affects Versions: 0.11.0.3
> Environment: Amazon Linux AMI 2017.09
> confluent-platform-oss-2.11-3.3.2
> OpenJDK Runtime Environment (build 1.8.0_191-b12)
> EC2 instance type i3.xlarge
>Reporter: Ninth Nails
>Priority: Blocker
>
> I have one broker log cleaner thread dying because of a failing type
> conversion. My cluster is running on 0.11.03-cp1 (Confluent Platform). Here's
> the relevant logs in log-cleaner.log
> {code:java}
> [2019-02-19 15:08:23,387] priority=INFO message="[kafka-log-cleaner-thread-0]:
> Log cleaner thread 0 cleaned log __consumer_offsets-36 (dirty section
> = [931148954, 931148954])
> 13,051.5 MB of log processed in 159.0 seconds (82.1 MB/sec).
> Indexed 13,051.5 MB in 81.2 seconds (160.8 Mb/sec, 51.1% of total
> time)
> Buffer utilization: 0.0%
> Cleaned 13,051.5 MB in 77.8 seconds (167.7 Mb/sec, 48.9% of total
> time)
> Start size: 13,051.5 MB (166,676,749 messages)
> End size: 0.0 MB (159 messages)
> 100.0% size reduction (100.0% fewer messages)
> " category=kafka.log.LogCleaner
> [2019-02-19 15:08:23,392] priority=INFO message="Cleaner 0: Beginning
> cleaning of log __consumer_offsets-37." category=kafka.log.LogCleaner
> [2019-02-19 15:08:23,392] priority=INFO message="Cleaner 0: Building offset
> map for __consumer_offsets-37..." category=kafka.log.LogCleaner
> [2019-02-19 15:08:23,410] priority=INFO message="Cleaner 0: Building offset
> map for log __consumer_offsets-37 for 87 segments in offset range [211566448,
> 308245257)." category=kafka.log.LogCleaner
> [2019-02-19 15:08:23,945] priority=ERROR
> message="[kafka-log-cleaner-thread-0]: Error due to"
> category=kafka.log.LogCleaner
> java.lang.IllegalArgumentException: Varint is too long, the most significant
> bit in the 5th byte is set, converted value: 445d418c
> at
> org.apache.kafka.common.utils.ByteUtils.illegalVarintException(ByteUtils.java:326)
> at
> org.apache.kafka.common.utils.ByteUtils.readVarint(ByteUtils.java:148)
> at
> org.apache.kafka.common.record.DefaultRecord.readFrom(DefaultRecord.java:305)
> at
> org.apache.kafka.common.record.DefaultRecordBatch$2.readNext(DefaultRecordBatch.java:299)
> at
> org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:563)
> at
> org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:532)
> at
> scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:752)
> at
> kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:741)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> kafka.log.Cleaner.kafka$log$Cleaner$$buildOffsetMapForSegment(LogCleaner.scala:741)
> at
> kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:707)
> at
> kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:704)
> at
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
> at
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
> at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
> at
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
> at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:704)
> at kafka.log.Cleaner.doClean(LogCleaner.scala:373)
> at kafka.lo
[jira] [Created] (KAFKA-7951) Log Cleaner thread stop with "Varint is too long" error
Ninth Nails created KAFKA-7951:
--
Summary: Log Cleaner thread stop with "Varint is too long" error
Key: KAFKA-7951
URL: https://issues.apache.org/jira/browse/KAFKA-7951
Project: Kafka
Issue Type: Bug
Components: log
Affects Versions: 0.11.0.3
Environment: Amazon Linux AMI 2017.09
confluent-platform-oss-2.11-3.3.2
OpenJDK Runtime Environment (build 1.8.0_191-b12)
EC2 instance type i3.xlarge
Reporter: Ninth Nails
I have one broker log cleaner thread dying because of a failing type
conversion. My cluster is running on 0.11.03-cp1 (Confluent Platform). Here's
the relevant logs in log-cleaner.log
{code:java}
[2019-02-19 15:08:23,387] priority=INFO message="[kafka-log-cleaner-thread-0]:
Log cleaner thread 0 cleaned log __consumer_offsets-36 (dirty section =
[931148954, 931148954])
13,051.5 MB of log processed in 159.0 seconds (82.1 MB/sec).
Indexed 13,051.5 MB in 81.2 seconds (160.8 Mb/sec, 51.1% of total time)
Buffer utilization: 0.0%
Cleaned 13,051.5 MB in 77.8 seconds (167.7 Mb/sec, 48.9% of total time)
Start size: 13,051.5 MB (166,676,749 messages)
End size: 0.0 MB (159 messages)
100.0% size reduction (100.0% fewer messages)
" category=kafka.log.LogCleaner
[2019-02-19 15:08:23,392] priority=INFO message="Cleaner 0: Beginning cleaning
of log __consumer_offsets-37." category=kafka.log.LogCleaner
[2019-02-19 15:08:23,392] priority=INFO message="Cleaner 0: Building offset map
for __consumer_offsets-37..." category=kafka.log.LogCleaner
[2019-02-19 15:08:23,410] priority=INFO message="Cleaner 0: Building offset map
for log __consumer_offsets-37 for 87 segments in offset range [211566448,
308245257)." category=kafka.log.LogCleaner
[2019-02-19 15:08:23,945] priority=ERROR message="[kafka-log-cleaner-thread-0]:
Error due to" category=kafka.log.LogCleaner
java.lang.IllegalArgumentException: Varint is too long, the most significant
bit in the 5th byte is set, converted value: 445d418c
at
org.apache.kafka.common.utils.ByteUtils.illegalVarintException(ByteUtils.java:326)
at
org.apache.kafka.common.utils.ByteUtils.readVarint(ByteUtils.java:148)
at
org.apache.kafka.common.record.DefaultRecord.readFrom(DefaultRecord.java:305)
at
org.apache.kafka.common.record.DefaultRecordBatch$2.readNext(DefaultRecordBatch.java:299)
at
org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:563)
at
org.apache.kafka.common.record.DefaultRecordBatch$RecordIterator.next(DefaultRecordBatch.java:532)
at
scala.collection.convert.Wrappers$JIteratorWrapper.next(Wrappers.scala:43)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:752)
at
kafka.log.Cleaner$$anonfun$kafka$log$Cleaner$$buildOffsetMapForSegment$1.apply(LogCleaner.scala:741)
at scala.collection.Iterator$class.foreach(Iterator.scala:891)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at
kafka.log.Cleaner.kafka$log$Cleaner$$buildOffsetMapForSegment(LogCleaner.scala:741)
at
kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:707)
at
kafka.log.Cleaner$$anonfun$buildOffsetMap$3.apply(LogCleaner.scala:704)
at
scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:733)
at
scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
at
scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:732)
at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:704)
at kafka.log.Cleaner.doClean(LogCleaner.scala:373)
at kafka.log.Cleaner.clean(LogCleaner.scala:361)
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:256)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:236)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
[2019-02-19 15:08:23,964] priority=INFO message="[kafka-log-cleaner-thread-0]:
Stopped" category=kafka.log.LogCleaner{code}
This is the first time we are seeing this error and coincidentally we have
changed our JVM from Oracle to OpenJDK, both 1.8 Java.
Please note that I can't upgrade to newer version of Kafka yet.
Thanks!
--
T
[jira] [Created] (KAFKA-7950) Kafka tools GetOffsetShell -time description
Kartik created KAFKA-7950: - Summary: Kafka tools GetOffsetShell -time description Key: KAFKA-7950 URL: https://issues.apache.org/jira/browse/KAFKA-7950 Project: Kafka Issue Type: Improvement Components: tools Affects Versions: 2.1.0 Reporter: Kartik Assignee: Kartik In Kafka GetOffsetShell tool, The --time description should contain information regarding what happens when the timestamp value > recently committed timestamp is given. Expected: "If timestamp value provided is greater than recently committed timestamp then no offset is returned. " -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KAFKA-7933) KTableKTableLeftJoinTest takes an hour to finish
[
https://issues.apache.org/jira/browse/KAFKA-7933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771836#comment-16771836
]
Ismael Juma commented on KAFKA-7933:
Good point [~chia7712], that could be it.
> KTableKTableLeftJoinTest takes an hour to finish
>
>
> Key: KAFKA-7933
> URL: https://issues.apache.org/jira/browse/KAFKA-7933
> Project: Kafka
> Issue Type: Improvement
> Components: streams, unit tests
>Reporter: Viktor Somogyi-Vass
>Assignee: Matthias J. Sax
>Priority: Major
> Attachments: jenkins-output-one-hour-test.log
>
>
> PRs might time out as
> {{KTableKTableLeftJoinTest.shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions}}
> took one hour to complete.
> {noformat}
> 11:57:45 org.apache.kafka.streams.kstream.internals.KTableKTableLeftJoinTest
> > shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions STARTED
> 12:53:35
> 12:53:35 org.apache.kafka.streams.kstream.internals.KTableKTableLeftJoinTest
> > shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions PASSED
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7933) KTableKTableLeftJoinTest takes an hour to finish
[
https://issues.apache.org/jira/browse/KAFKA-7933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771829#comment-16771829
]
Chia-Ping Tsai commented on KAFKA-7933:
---
Is this jira related to hardware env?
shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions flush rocksdb for
each "input data". I profile the test on my local with normal hdd. It take 0.1
second to complete the flush. Hence, it needs 0.1 s * 1000 (loop) * 23 (inputs)
= 38 minutes to complete
shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions. By contrast,
completing the test only took 5 minutes on my ssd.
> KTableKTableLeftJoinTest takes an hour to finish
>
>
> Key: KAFKA-7933
> URL: https://issues.apache.org/jira/browse/KAFKA-7933
> Project: Kafka
> Issue Type: Improvement
> Components: streams, unit tests
>Reporter: Viktor Somogyi-Vass
>Assignee: Matthias J. Sax
>Priority: Major
> Attachments: jenkins-output-one-hour-test.log
>
>
> PRs might time out as
> {{KTableKTableLeftJoinTest.shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions}}
> took one hour to complete.
> {noformat}
> 11:57:45 org.apache.kafka.streams.kstream.internals.KTableKTableLeftJoinTest
> > shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions STARTED
> 12:53:35
> 12:53:35 org.apache.kafka.streams.kstream.internals.KTableKTableLeftJoinTest
> > shouldNotThrowIllegalStateExceptionWhenMultiCacheEvictions PASSED
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-6755) MaskField SMT should optionally take a literal value to use instead of using null
[
https://issues.apache.org/jira/browse/KAFKA-6755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771800#comment-16771800
]
Valeria Vasylieva commented on KAFKA-6755:
--
hi [~rhauch]! I have submitted the PR, could you please look at it?
> MaskField SMT should optionally take a literal value to use instead of using
> null
> -
>
> Key: KAFKA-6755
> URL: https://issues.apache.org/jira/browse/KAFKA-6755
> Project: Kafka
> Issue Type: Improvement
> Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Randall Hauch
>Assignee: Valeria Vasylieva
>Priority: Major
> Labels: needs-kip, newbie
> Original Estimate: 8h
> Remaining Estimate: 8h
>
> The existing {{org.apache.kafka.connect.transforms.MaskField}} SMT always
> uses the null value for the type of field. It'd be nice to *optionally* be
> able to specify a literal value for the type, where the SMT would convert the
> literal string value in the configuration to the desired type (using the new
> {{Values}} methods).
> Use cases: mask out the IP address, or SSN, or other personally identifiable
> information (PII).
> Since this changes the API, and thus will require a KIP.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KAFKA-7927) Read committed receives aborted events
[
https://issues.apache.org/jira/browse/KAFKA-7927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16771798#comment-16771798
]
Gabor Somogyi commented on KAFKA-7927:
--
I've tested it and works fine. That said a warning/error/whatever would be good
to notify users.
> Read committed receives aborted events
> --
>
> Key: KAFKA-7927
> URL: https://issues.apache.org/jira/browse/KAFKA-7927
> Project: Kafka
> Issue Type: Bug
> Components: consumer, core, producer
>Affects Versions: 1.0.0
>Reporter: Gabor Somogyi
>Priority: Blocker
> Attachments: KafkaProducer.scala, consumer.log, producer.log.gz
>
>
> When a kafka client produces ~30k events and at the end it aborts the
> transaction a consumer can read part of the aborted messages when
> "isolation.level" set to "READ_COMMITTED".
> Kafka client version: 2.0.0
> Kafka broker version: 1.0.0
> Producer:
> {code:java}
> java -jar
> kafka-producer/target/kafka-producer-0.0.1-SNAPSHOT-jar-with-dependencies.jar
> gsomogyi-cdh5144-220cloudera2-1.gce.cloudera.com:9092 src-topic
> {code}
> See attached code.
> Consumer:
> {code:java}
> kafka-console-consumer --zookeeper localhost:2181 --topic src-topic
> --from-beginning --isolation-level read_committed
> {code}
> Same behavior seen when re-implemented the consumer in scala.
> The whole application can be found here:
> https://github.com/gaborgsomogyi/kafka-semantics-tester
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KAFKA-7949) Backport knox trusted proxy support for ambari for 2.6
[ https://issues.apache.org/jira/browse/KAFKA-7949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Attila Magyar resolved KAFKA-7949. -- Resolution: Invalid wrong subproject > Backport knox trusted proxy support for ambari for 2.6 > -- > > Key: KAFKA-7949 > URL: https://issues.apache.org/jira/browse/KAFKA-7949 > Project: Kafka > Issue Type: Task >Reporter: Attila Magyar >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KAFKA-7949) Backport knox trusted proxy support for ambari for 2.6
Attila Magyar created KAFKA-7949: Summary: Backport knox trusted proxy support for ambari for 2.6 Key: KAFKA-7949 URL: https://issues.apache.org/jira/browse/KAFKA-7949 Project: Kafka Issue Type: Task Reporter: Attila Magyar -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (KAFKA-5010) Log cleaner crashed with BufferOverflowException when writing to the writeBuffer
[
https://issues.apache.org/jira/browse/KAFKA-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Li Haijun updated KAFKA-5010:
-
Comment: was deleted
(was: Hi Shuai Lin, we encounterd the relative problem recently, how could I
find the specific topic that triggered the crash of the log cleaner? By the
way, the log.cleaner.log text :
```
1 [2019-02-19 09:59:37,700] INFO Starting the log cleaner (kafka.log.LogCleaner)
2 [2019-02-19 09:59:37,739] INFO [kafka-log-cleaner-thread-0], Starting
(kafka.log.LogCleaner)
3 [2019-02-19 09:59:38,044] INFO Cleaner 0: Beginning cleaning of log
get_coupon_streams_2-get_coupon_agg-changelog-5. (kafka.log.LogCleaner)
4 [2019-02-19 09:59:38,044] INFO Cleaner 0: Building offset map for
get_coupon_streams_2-get_coupon_agg-changelog-5... (kafka.log.LogCleaner)
5 [2019-02-19 09:59:38,072] INFO Cleaner 0: Building offset map for log
get_coupon_streams_2-get_coupon_agg-changelog-5 for 15 segments in offset range
[0, 3481151). (kafka.log.LogCleaner)
6 [2019-02-19 09:59:44,899] INFO Cleaner 0: Offset map for log
get_coupon_streams_2-get_coupon_agg-changelog-5 complete. (kafka.log.LogCleaner)
7 [2019-02-19 09:59:44,925] INFO Cleaner 0: Cleaning log
get_coupon_streams_2-get_coupon_agg-changelog-5 (cleaning prior to Wed Feb 13
16:56:15 CST 2019, discarding tombstones prior to Thu Jan 01 08:00:00 CST
1970)... (kafka.log.LogCleaner)
8 [2019-02-19 09:59:44,946] INFO Cleaner 0: Cleaning segment 0 in log
get_coupon_streams_2-get_coupon_agg-changelog-5 (largest timestamp Wed Nov 07
16:51:37 CST 2018) into 0, retaining deletes. (kafka.log.LogCleaner)
9 [2019-02-19 09:59:45,231] ERROR [kafka-log-cleaner-thread-0], Error due to
(kafka.log.LogCleaner)
10 java.lang.IllegalArgumentException
11 at java.nio.Buffer.position(Buffer.java:236)
12 at
org.apache.kafka.common.record.MemoryRecordsBuilder.(MemoryRecordsBuilder.java:141)
13 at
org.apache.kafka.common.record.MemoryRecords.builder(MemoryRecords.java:300)
14 at
org.apache.kafka.common.record.MemoryRecords.builderWithEntries(MemoryRecords.java:408)
15 at
org.apache.kafka.common.record.MemoryRecords.filterTo(MemoryRecords.java:168)
16 at
org.apache.kafka.common.record.MemoryRecords.filterTo(MemoryRecords.java:111)
17 at kafka.log.Cleaner.cleanInto(LogCleaner.scala:468)
18 at kafka.log.Cleaner$$anonfun$cleanSegments$1.apply(LogCleaner.scala:405)
19 at kafka.log.Cleaner$$anonfun$cleanSegments$1.apply(LogCleaner.scala:401)
20 at scala.collection.immutable.List.foreach(List.scala:381)
21 at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:401)
22 at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:363)
23 at kafka.log.Cleaner$$anonfun$clean$4.apply(LogCleaner.scala:362)
24 at scala.collection.immutable.List.foreach(List.scala:381)
25 at kafka.log.Cleaner.clean(LogCleaner.scala:362)
26 at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:241)
27 at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:220)
28 at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
29 [2019-02-19 09:59:45,274] INFO [kafka-log-cleaner-thread-0], Stopped
(kafka.log.LogCleaner)
```)
> Log cleaner crashed with BufferOverflowException when writing to the
> writeBuffer
>
>
> Key: KAFKA-5010
> URL: https://issues.apache.org/jira/browse/KAFKA-5010
> Project: Kafka
> Issue Type: Bug
> Components: log
>Affects Versions: 0.10.2.0
>Reporter: Shuai Lin
>Priority: Critical
> Labels: reliability
>
> After upgrading from 0.10.0.1 to 0.10.2.0 the log cleaner thread crashed with
> BufferOverflowException when writing the filtered records into the
> writeBuffer:
> {code}
> [2017-03-24 10:41:03,926] INFO [kafka-log-cleaner-thread-0], Starting
> (kafka.log.LogCleaner)
> [2017-03-24 10:41:04,177] INFO Cleaner 0: Beginning cleaning of log
> app-topic-20170317-20. (kafka.log.LogCleaner)
> [2017-03-24 10:41:04,177] INFO Cleaner 0: Building offset map for
> app-topic-20170317-20... (kafka.log.LogCleaner)
> [2017-03-24 10:41:04,387] INFO Cleaner 0: Building offset map for log
> app-topic-20170317-20 for 1 segments in offset range [9737795, 9887707).
> (kafka.log.LogCleaner)
> [2017-03-24 10:41:07,101] INFO Cleaner 0: Offset map for log
> app-topic-20170317-20 complete. (kafka.log.LogCleaner)
> [2017-03-24 10:41:07,106] INFO Cleaner 0: Cleaning log app-topic-20170317-20
> (cleaning prior to Fri Mar 24 10:36:06 GMT 2017, discarding tombstones prior
> to Thu Mar 23 10:18:02 GMT 2017)... (kafka.log.LogCleaner)
> [2017-03-24 10:41:07,110] INFO Cleaner 0: Cleaning segment 0 in log
> app-topic-20170317-20 (largest timestamp Fri Mar 24 09:58:25 GMT 2017) into
> 0, retaining deletes. (kafka.log.LogCleaner)
> [2017-03-24 10:41:07,372] ERROR [kafka-log-cleaner
