[jira] [Commented] (KAFKA-8520) TimeoutException in client side doesn't have stack trace
[ https://issues.apache.org/jira/browse/KAFKA-8520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16969950#comment-16969950 ] Viliam Durina commented on KAFKA-8520: -- Duplicate of https://issues.apache.org/jira/browse/KAFKA-7016 > TimeoutException in client side doesn't have stack trace > > > Key: KAFKA-8520 > URL: https://issues.apache.org/jira/browse/KAFKA-8520 > Project: Kafka > Issue Type: New Feature > Components: clients >Reporter: Shixiong Zhu >Priority: Major > > When a TimeoutException is thrown directly in the client side, it doesn't > have any stack trace because it inherits > "org.apache.kafka.common.errors.ApiException". This makes the user hard to > debug timeout issues, because it's hard to know which line in the user codes > throwing this TimeoutException. > It would be great that adding a new client side TimeoutException which > contains the stack trace. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9160) Use a common stateMachine to make the state transfer more simple and clear
[ https://issues.apache.org/jira/browse/KAFKA-9160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] maobaolong updated KAFKA-9160: -- Description: We can reference the repo https://github.com/maobaolong/statemachine_demo if we can use the statemachine to design the group state, you may just care about the event, the state transfer is the result. was:We can reference the repo https://github.com/maobaolong/statemachine_demo Summary: Use a common stateMachine to make the state transfer more simple and clear (was: Use a common stateMachine to make the state transfer more simple and clear。) > Use a common stateMachine to make the state transfer more simple and clear > -- > > Key: KAFKA-9160 > URL: https://issues.apache.org/jira/browse/KAFKA-9160 > Project: Kafka > Issue Type: Improvement > Components: controller >Affects Versions: 2.5.0 >Reporter: maobaolong >Priority: Major > > We can reference the repo https://github.com/maobaolong/statemachine_demo > if we can use the statemachine to design the group state, you may just care > about the event, the state transfer is the result. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9125) GroupMetadataManager and TransactionStateManager should use metadata cache instead of zkClient
[ https://issues.apache.org/jira/browse/KAFKA-9125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970103#comment-16970103 ] ASF GitHub Bot commented on KAFKA-9125: --- viktorsomogyi commented on pull request #7657: KAFKA-9125: Use metadata cache GroupMetadataManager and TransactionStateManager URL: https://github.com/apache/kafka/pull/7657 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > GroupMetadataManager and TransactionStateManager should use metadata cache > instead of zkClient > -- > > Key: KAFKA-9125 > URL: https://issues.apache.org/jira/browse/KAFKA-9125 > Project: Kafka > Issue Type: Sub-task >Reporter: Viktor Somogyi-Vass >Assignee: Viktor Somogyi-Vass >Priority: Major > > Both classes query their respective topic's partition count via the zkClient. > This however could be queried by the broker's local metadata cache. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8677) Flakey test GroupEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
[
https://issues.apache.org/jira/browse/KAFKA-8677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970409#comment-16970409
]
ASF GitHub Bot commented on KAFKA-8677:
---
guozhangwang commented on pull request #7613: KAFKA-8677: Simplify the
best-effort network client poll to never throw exception
URL: https://github.com/apache/kafka/pull/7613
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Flakey test
> GroupEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
>
>
> Key: KAFKA-8677
> URL: https://issues.apache.org/jira/browse/KAFKA-8677
> Project: Kafka
> Issue Type: Bug
> Components: core, security, unit tests
>Affects Versions: 2.4.0
>Reporter: Boyang Chen
>Priority: Blocker
> Labels: flaky-test
> Fix For: 2.4.0
>
>
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/6325/console]
>
> *18:43:39* kafka.api.GroupEndToEndAuthorizationTest >
> testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl STARTED*18:44:00*
> kafka.api.GroupEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
> failed, log available in
> /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk11-scala2.12/core/build/reports/testOutput/kafka.api.GroupEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl.test.stdout*18:44:00*
> *18:44:00* kafka.api.GroupEndToEndAuthorizationTest >
> testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl FAILED*18:44:00*
> org.scalatest.exceptions.TestFailedException: Consumed 0 records before
> timeout instead of the expected 1 records
> ---
> I found this flaky test is actually exposing a real bug in consumer: within
> {{KafkaConsumer.poll}}, we have an optimization to try to send the next fetch
> request before returning the data in order to pipelining the fetch requests:
> {code}
> if (!records.isEmpty()) {
> // before returning the fetched records, we can send off
> the next round of fetches
> // and avoid block waiting for their responses to enable
> pipelining while the user
> // is handling the fetched records.
> //
> // NOTE: since the consumed position has already been
> updated, we must not allow
> // wakeups or any other errors to be triggered prior to
> returning the fetched records.
> if (fetcher.sendFetches() > 0 ||
> client.hasPendingRequests()) {
> client.pollNoWakeup();
> }
> return this.interceptors.onConsume(new
> ConsumerRecords<>(records));
> }
> {code}
> As the NOTE mentioned, this pollNoWakeup should NOT throw any exceptions,
> since at this point the fetch position has been updated. If an exception is
> thrown here, and the callers decides to capture and continue, those records
> would never be returned again, causing data loss.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-8677) Flakey test GroupEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
[
https://issues.apache.org/jira/browse/KAFKA-8677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Manikumar resolved KAFKA-8677.
--
Assignee: Guozhang Wang
Resolution: Fixed
> Flakey test
> GroupEndToEndAuthorizationTest#testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
>
>
> Key: KAFKA-8677
> URL: https://issues.apache.org/jira/browse/KAFKA-8677
> Project: Kafka
> Issue Type: Bug
> Components: core, security, unit tests
>Affects Versions: 2.4.0
>Reporter: Boyang Chen
>Assignee: Guozhang Wang
>Priority: Blocker
> Labels: flaky-test
> Fix For: 2.4.0
>
>
> [https://builds.apache.org/job/kafka-pr-jdk11-scala2.12/6325/console]
>
> *18:43:39* kafka.api.GroupEndToEndAuthorizationTest >
> testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl STARTED*18:44:00*
> kafka.api.GroupEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl
> failed, log available in
> /home/jenkins/jenkins-slave/workspace/kafka-pr-jdk11-scala2.12/core/build/reports/testOutput/kafka.api.GroupEndToEndAuthorizationTest.testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl.test.stdout*18:44:00*
> *18:44:00* kafka.api.GroupEndToEndAuthorizationTest >
> testNoDescribeProduceOrConsumeWithoutTopicDescribeAcl FAILED*18:44:00*
> org.scalatest.exceptions.TestFailedException: Consumed 0 records before
> timeout instead of the expected 1 records
> ---
> I found this flaky test is actually exposing a real bug in consumer: within
> {{KafkaConsumer.poll}}, we have an optimization to try to send the next fetch
> request before returning the data in order to pipelining the fetch requests:
> {code}
> if (!records.isEmpty()) {
> // before returning the fetched records, we can send off
> the next round of fetches
> // and avoid block waiting for their responses to enable
> pipelining while the user
> // is handling the fetched records.
> //
> // NOTE: since the consumed position has already been
> updated, we must not allow
> // wakeups or any other errors to be triggered prior to
> returning the fetched records.
> if (fetcher.sendFetches() > 0 ||
> client.hasPendingRequests()) {
> client.pollNoWakeup();
> }
> return this.interceptors.onConsume(new
> ConsumerRecords<>(records));
> }
> {code}
> As the NOTE mentioned, this pollNoWakeup should NOT throw any exceptions,
> since at this point the fetch position has been updated. If an exception is
> thrown here, and the callers decides to capture and continue, those records
> would never be returned again, causing data loss.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9164) Don't evict active topics' metadata from the producer's cache
Brian Byrne created KAFKA-9164: -- Summary: Don't evict active topics' metadata from the producer's cache Key: KAFKA-9164 URL: https://issues.apache.org/jira/browse/KAFKA-9164 Project: Kafka Issue Type: Bug Components: producer Reporter: Brian Byrne Assignee: Brian Byrne The producer's metadata currently marks a topic as "not needing to be retained" if it has been 5 minutes since it was first considered, regardless of whether records were being actively produced for the topic. This shouldn't happen and should be fixed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9016) Warn when log dir stopped serving replicas
[
https://issues.apache.org/jira/browse/KAFKA-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970516#comment-16970516
]
ASF GitHub Bot commented on KAFKA-9016:
---
lindong28 commented on pull request #7563: KAFKA-9016:Warn when log dir stopped
serving replicas
URL: https://github.com/apache/kafka/pull/7563
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Warn when log dir stopped serving replicas
> --
>
> Key: KAFKA-9016
> URL: https://issues.apache.org/jira/browse/KAFKA-9016
> Project: Kafka
> Issue Type: Improvement
> Components: log, log cleaner
>Reporter: Viktor Somogyi-Vass
>Priority: Major
> Labels: easy, newbie
>
> Kafka should warn if the log directory stops serving replicas as usually it
> is due to an error and it's much visible if it's on warn level.
> {noformat}
> 2019-09-19 12:39:54,194 ERROR kafka.server.LogDirFailureChannel: Error while
> writing to checkpoint file /kafka/data/diskX/replication-offset-checkpoint
> java.io.SyncFailedException: sync failed
> ..
> 2019-09-19 12:39:54,197 INFO kafka.server.ReplicaManager: [ReplicaManager
> broker=638] Stopping serving replicas in dir /kafka/data/diskX
> ..
> 2019-09-19 12:39:54,205 INFO kafka.server.ReplicaFetcherManager:
> [ReplicaFetcherManager on broker 638] Removed fetcher for partitions
> Set(test1-0, test2-2, test-0, test2-2, test4-14, test4-0, test2-6)
> 2019-09-19 12:39:54,206 INFO kafka.server.ReplicaAlterLogDirsManager:
> [ReplicaAlterLogDirsManager on broker 638] Removed fetcher for partitions
> Set(test1-0, test2-2, test-0, test3-2, test4-14, test4-0, test2-6)
> 2019-09-19 12:39:54,222 INFO kafka.server.ReplicaManager: [ReplicaManager
> broker=638] Broker 638 stopped fetcher for partitions
> test1-0,test2-2,test-0,test3-2,test4-14,test4-0,test2-6 and stopped moving
> logs for partitions because they are in the failed log directory
> /kafka/data/diskX.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KAFKA-9016) Warn when log dir stopped serving replicas
[
https://issues.apache.org/jira/browse/KAFKA-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Lin reassigned KAFKA-9016:
---
Assignee: kumar uttpal
> Warn when log dir stopped serving replicas
> --
>
> Key: KAFKA-9016
> URL: https://issues.apache.org/jira/browse/KAFKA-9016
> Project: Kafka
> Issue Type: Improvement
> Components: log, log cleaner
>Reporter: Viktor Somogyi-Vass
>Assignee: kumar uttpal
>Priority: Major
> Labels: easy, newbie
>
> Kafka should warn if the log directory stops serving replicas as usually it
> is due to an error and it's much visible if it's on warn level.
> {noformat}
> 2019-09-19 12:39:54,194 ERROR kafka.server.LogDirFailureChannel: Error while
> writing to checkpoint file /kafka/data/diskX/replication-offset-checkpoint
> java.io.SyncFailedException: sync failed
> ..
> 2019-09-19 12:39:54,197 INFO kafka.server.ReplicaManager: [ReplicaManager
> broker=638] Stopping serving replicas in dir /kafka/data/diskX
> ..
> 2019-09-19 12:39:54,205 INFO kafka.server.ReplicaFetcherManager:
> [ReplicaFetcherManager on broker 638] Removed fetcher for partitions
> Set(test1-0, test2-2, test-0, test2-2, test4-14, test4-0, test2-6)
> 2019-09-19 12:39:54,206 INFO kafka.server.ReplicaAlterLogDirsManager:
> [ReplicaAlterLogDirsManager on broker 638] Removed fetcher for partitions
> Set(test1-0, test2-2, test-0, test3-2, test4-14, test4-0, test2-6)
> 2019-09-19 12:39:54,222 INFO kafka.server.ReplicaManager: [ReplicaManager
> broker=638] Broker 638 stopped fetcher for partitions
> test1-0,test2-2,test-0,test3-2,test4-14,test4-0,test2-6 and stopped moving
> logs for partitions because they are in the failed log directory
> /kafka/data/diskX.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-9016) Warn when log dir stopped serving replicas
[
https://issues.apache.org/jira/browse/KAFKA-9016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dong Lin resolved KAFKA-9016.
-
Resolution: Fixed
> Warn when log dir stopped serving replicas
> --
>
> Key: KAFKA-9016
> URL: https://issues.apache.org/jira/browse/KAFKA-9016
> Project: Kafka
> Issue Type: Improvement
> Components: log, log cleaner
>Reporter: Viktor Somogyi-Vass
>Assignee: kumar uttpal
>Priority: Major
> Labels: easy, newbie
>
> Kafka should warn if the log directory stops serving replicas as usually it
> is due to an error and it's much visible if it's on warn level.
> {noformat}
> 2019-09-19 12:39:54,194 ERROR kafka.server.LogDirFailureChannel: Error while
> writing to checkpoint file /kafka/data/diskX/replication-offset-checkpoint
> java.io.SyncFailedException: sync failed
> ..
> 2019-09-19 12:39:54,197 INFO kafka.server.ReplicaManager: [ReplicaManager
> broker=638] Stopping serving replicas in dir /kafka/data/diskX
> ..
> 2019-09-19 12:39:54,205 INFO kafka.server.ReplicaFetcherManager:
> [ReplicaFetcherManager on broker 638] Removed fetcher for partitions
> Set(test1-0, test2-2, test-0, test2-2, test4-14, test4-0, test2-6)
> 2019-09-19 12:39:54,206 INFO kafka.server.ReplicaAlterLogDirsManager:
> [ReplicaAlterLogDirsManager on broker 638] Removed fetcher for partitions
> Set(test1-0, test2-2, test-0, test3-2, test4-14, test4-0, test2-6)
> 2019-09-19 12:39:54,222 INFO kafka.server.ReplicaManager: [ReplicaManager
> broker=638] Broker 638 stopped fetcher for partitions
> test1-0,test2-2,test-0,test3-2,test4-14,test4-0,test2-6 and stopped moving
> logs for partitions because they are in the failed log directory
> /kafka/data/diskX.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KAFKA-8933) An unhandled SSL handshake exception in polling event - needed a retry logic
[
https://issues.apache.org/jira/browse/KAFKA-8933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gustafson reassigned KAFKA-8933:
--
Assignee: Jason Gustafson
> An unhandled SSL handshake exception in polling event - needed a retry logic
>
>
> Key: KAFKA-8933
> URL: https://issues.apache.org/jira/browse/KAFKA-8933
> Project: Kafka
> Issue Type: Bug
> Components: clients
>Affects Versions: 2.2.1, 2.4.0
> Environment: software platform
>Reporter: Remigius
>Assignee: Jason Gustafson
>Priority: Critical
>
> Already client is connected and during polling event, SSL handshake failure
> happened. it led to leaving the co-ordinator. Even on SSL handshake failure
> which was actually intermittent issue, polling should have some resilient and
> retry the polling. Leaving group caused all instances of clients to drop and
> left the messages in Kafka for long time until re-subscribe the kafka topic
> manually.
>
>
> {noformat}
> 2019-09-06 04:03:09,016 ERROR [reactive-kafka-]
> org.apache.kafka.clients.NetworkClient [Consumer clientId=aaa, groupId=bbb]
> Connection to node 150 (host:port) failed authentication due to: SSL
> handshake failed
> 2019-09-06 04:03:09,021 ERROR [reactive-kafka-]
> reactor.kafka.receiver.internals.DefaultKafkaReceiver Unexpected exception
> java.lang.NullPointerException: null
> at
> org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.handleCompletedMetadataResponse(NetworkClient.java:1012)
> ~[kafka-clients-2.2.1.jar!/:?]
> at
> org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:822)
> ~[kafka-clients-2.2.1.jar!/:?]
> at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:544)
> ~[kafka-clients-2.2.1.jar!/:?]
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
> ~[kafka-clients-2.2.1.jar!/:?]
> at
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
> ~[kafka-clients-2.2.1.jar!/:?]
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1256)
> ~[kafka-clients-2.2.1.jar!/:?]
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1200)
> ~[kafka-clients-2.2.1.jar!/:?]
> at
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1176)
> ~[kafka-clients-2.2.1.jar!/:?]
> at
> reactor.kafka.receiver.internals.DefaultKafkaReceiver$PollEvent.run(DefaultKafkaReceiver.java:470)
> ~[reactor-kafka-1.1.1.RELEASE.jar!/:1.1.1.RELEASE]
> at
> reactor.kafka.receiver.internals.DefaultKafkaReceiver.doEvent(DefaultKafkaReceiver.java:401)
> ~[reactor-kafka-1.1.1.RELEASE.jar!/:1.1.1.RELEASE]
> at
> reactor.kafka.receiver.internals.DefaultKafkaReceiver.lambda$start$14(DefaultKafkaReceiver.java:335)
> ~[reactor-kafka-1.1.1.RELEASE.jar!/:1.1.1.RELEASE]
> at reactor.core.publisher.LambdaSubscriber.onNext(LambdaSubscriber.java:130)
> ~[reactor-core-3.2.10.RELEASE.jar!/:3.2.10.RELEASE]
> at
> reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.runAsync(FluxPublishOn.java:398)
> ~[reactor-core-3.2.10.RELEASE.jar!/:3.2.10.RELEASE]
> at
> reactor.core.publisher.FluxPublishOn$PublishOnSubscriber.run(FluxPublishOn.java:484)
> ~[reactor-core-3.2.10.RELEASE.jar!/:3.2.10.RELEASE]
> at
> reactor.kafka.receiver.internals.KafkaSchedulers$EventScheduler.lambda$decorate$1(KafkaSchedulers.java:100)
> ~[reactor-kafka-1.1.1.RELEASE.jar!/:1.1.1.RELEASE]
> at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:84)
> ~[reactor-core-3.2.10.RELEASE.jar!/:3.2.10.RELEASE]
> at reactor.core.scheduler.WorkerTask.call(WorkerTask.java:37)
> ~[reactor-core-3.2.10.RELEASE.jar!/:3.2.10.RELEASE]
> at
> org.springframework.cloud.sleuth.instrument.async.TraceCallable.call(TraceCallable.java:70)
> ~[spring-cloud-sleuth-core-2.1.1.RELEASE.jar!/:2.1.1.RELEASE]
> at java.util.concurrent.FutureTask.run(FutureTask.java:264) ~[?:?]
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> ~[?:?]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> ~[?:?]
> at java.lang.Thread.run(Thread.java:834) [?:?]
> 2019-09-06 04:03:09,023 INFO [reactive-kafka-]
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator [Consumer
> clientId=aaa, groupId=bbb] Member x_13-081e61ec-1509-4e0e-819e-58063d1ce8f6
> sending LeaveGroup request to coordinator{noformat}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KAFKA-9165) Fix jersey warnings in Trogdor
Colin McCabe created KAFKA-9165: --- Summary: Fix jersey warnings in Trogdor Key: KAFKA-9165 URL: https://issues.apache.org/jira/browse/KAFKA-9165 Project: Kafka Issue Type: Bug Reporter: Colin McCabe Lately, we have a lot of jersey warnings starting with "SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime." While this doesn't cause any runtime impact, it would be good to avoid having this warning confuse users and clutter up the logs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KAFKA-9165) Fix jersey warnings in Trogdor
[ https://issues.apache.org/jira/browse/KAFKA-9165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin McCabe reassigned KAFKA-9165: --- Assignee: Colin McCabe > Fix jersey warnings in Trogdor > -- > > Key: KAFKA-9165 > URL: https://issues.apache.org/jira/browse/KAFKA-9165 > Project: Kafka > Issue Type: Bug >Reporter: Colin McCabe >Assignee: Colin McCabe >Priority: Minor > > Lately, we have a lot of jersey warnings starting with "SERVER runtime does > not implement any provider interfaces applicable in the SERVER runtime." > While this doesn't cause any runtime impact, it would be good to avoid having > this warning confuse users and clutter up the logs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9165) Fix jersey warnings in Trogdor
[ https://issues.apache.org/jira/browse/KAFKA-9165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970576#comment-16970576 ] ASF GitHub Bot commented on KAFKA-9165: --- cmccabe commented on pull request #7669: KAFKA-9165: Fix jersey warnings in Trogdor URL: https://github.com/apache/kafka/pull/7669 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Fix jersey warnings in Trogdor > -- > > Key: KAFKA-9165 > URL: https://issues.apache.org/jira/browse/KAFKA-9165 > Project: Kafka > Issue Type: Bug >Reporter: Colin McCabe >Assignee: Colin McCabe >Priority: Minor > > Lately, we have a lot of jersey warnings starting with "SERVER runtime does > not implement any provider interfaces applicable in the SERVER runtime." > While this doesn't cause any runtime impact, it would be good to avoid having > this warning confuse users and clutter up the logs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-7016) Reconsider the "avoid the expensive and useless stack trace for api exceptions" practice
[
https://issues.apache.org/jira/browse/KAFKA-7016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970583#comment-16970583
]
Guozhang Wang commented on KAFKA-7016:
--
I think I also agree with this change, primarily driven by all of my debugging
experience on Kafka Consumer: each time I would want to comment out the
``fillInStackTrace`` in my local branch, and then upon submitting the PR I'd
need to revert that change again. I think it is actually worth printing the
stack trace when ApiException is thrown because it is usually considered a
fatal error which causes the client to be shutdown / restart anyways so being a
bit costly would not hurt much.
> Reconsider the "avoid the expensive and useless stack trace for api
> exceptions" practice
>
>
> Key: KAFKA-7016
> URL: https://issues.apache.org/jira/browse/KAFKA-7016
> Project: Kafka
> Issue Type: Bug
>Reporter: Martin Vysny
>Priority: Major
>
> I am trying to write a Kafka Consumer; upon running it only prints out:
> {\{ org.apache.kafka.common.errors.InvalidGroupIdException: The configured
> groupId is invalid}}
> Note that the stack trace is missing, so that I have no information which
> part of my code is bad and need fixing; I also have no information which
> Kafka Client method has been called. Upon closer examination I found this in
> ApiException:
>
> {{/* avoid the expensive and useless stack trace for api exceptions */}}
> {{@Override}}
> {{public Throwable fillInStackTrace() {}}
> \{{ return this;}}
> {{}}}
>
> I think it is a bad practice to hide all useful debugging info and trade it
> for dubious performance gains. Exceptions are for exceptional code flow which
> are allowed to be slow.
>
> This applies to kafka-clients 1.1.0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-8520) TimeoutException in client side doesn't have stack trace
[ https://issues.apache.org/jira/browse/KAFKA-8520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-8520. -- Resolution: Duplicate > TimeoutException in client side doesn't have stack trace > > > Key: KAFKA-8520 > URL: https://issues.apache.org/jira/browse/KAFKA-8520 > Project: Kafka > Issue Type: New Feature > Components: clients >Reporter: Shixiong Zhu >Priority: Major > > When a TimeoutException is thrown directly in the client side, it doesn't > have any stack trace because it inherits > "org.apache.kafka.common.errors.ApiException". This makes the user hard to > debug timeout issues, because it's hard to know which line in the user codes > throwing this TimeoutException. > It would be great that adding a new client side TimeoutException which > contains the stack trace. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8520) TimeoutException in client side doesn't have stack trace
[ https://issues.apache.org/jira/browse/KAFKA-8520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970581#comment-16970581 ] Guozhang Wang commented on KAFKA-8520: -- Thanks [~vilo] good find. > TimeoutException in client side doesn't have stack trace > > > Key: KAFKA-8520 > URL: https://issues.apache.org/jira/browse/KAFKA-8520 > Project: Kafka > Issue Type: New Feature > Components: clients >Reporter: Shixiong Zhu >Priority: Major > > When a TimeoutException is thrown directly in the client side, it doesn't > have any stack trace because it inherits > "org.apache.kafka.common.errors.ApiException". This makes the user hard to > debug timeout issues, because it's hard to know which line in the user codes > throwing this TimeoutException. > It would be great that adding a new client side TimeoutException which > contains the stack trace. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-7016) Reconsider the "avoid the expensive and useless stack trace for api exceptions" practice
[
https://issues.apache.org/jira/browse/KAFKA-7016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970585#comment-16970585
]
Ismael Juma commented on KAFKA-7016:
I think this was done for the broker and not the clients. Even in the broker,
stacktraces are also helpful.
> Reconsider the "avoid the expensive and useless stack trace for api
> exceptions" practice
>
>
> Key: KAFKA-7016
> URL: https://issues.apache.org/jira/browse/KAFKA-7016
> Project: Kafka
> Issue Type: Bug
>Reporter: Martin Vysny
>Priority: Major
>
> I am trying to write a Kafka Consumer; upon running it only prints out:
> {\{ org.apache.kafka.common.errors.InvalidGroupIdException: The configured
> groupId is invalid}}
> Note that the stack trace is missing, so that I have no information which
> part of my code is bad and need fixing; I also have no information which
> Kafka Client method has been called. Upon closer examination I found this in
> ApiException:
>
> {{/* avoid the expensive and useless stack trace for api exceptions */}}
> {{@Override}}
> {{public Throwable fillInStackTrace() {}}
> \{{ return this;}}
> {{}}}
>
> I think it is a bad practice to hide all useful debugging info and trade it
> for dubious performance gains. Exceptions are for exceptional code flow which
> are allowed to be slow.
>
> This applies to kafka-clients 1.1.0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9098) Name Repartition Filter, Source, and Sink Processors
[ https://issues.apache.org/jira/browse/KAFKA-9098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970586#comment-16970586 ] ASF GitHub Bot commented on KAFKA-9098: --- bbejeck commented on pull request #7598: KAFKA-9098: When users name repartition topic, use the name for the repartition filter, source and sink node. URL: https://github.com/apache/kafka/pull/7598 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Name Repartition Filter, Source, and Sink Processors > > > Key: KAFKA-9098 > URL: https://issues.apache.org/jira/browse/KAFKA-9098 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.2.0, 2.3.0 >Reporter: Bill Bejeck >Assignee: Bill Bejeck >Priority: Major > > When users provide a name for repartition topics, we should the same name as > the base for the filter, source and sink operators. While this does not > break a topology, users providing names for all processors in a DSL topology > may find the generated names for the repartition topics filter, source, and > sink operators as inconsistent with the naming approach. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KAFKA-9098) Name Repartition Filter, Source, and Sink Processors
[ https://issues.apache.org/jira/browse/KAFKA-9098?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bill Bejeck updated KAFKA-9098: --- Fix Version/s: 2.5.0 > Name Repartition Filter, Source, and Sink Processors > > > Key: KAFKA-9098 > URL: https://issues.apache.org/jira/browse/KAFKA-9098 > Project: Kafka > Issue Type: Improvement > Components: streams >Affects Versions: 2.2.0, 2.3.0 >Reporter: Bill Bejeck >Assignee: Bill Bejeck >Priority: Major > Fix For: 2.5.0 > > > When users provide a name for repartition topics, we should the same name as > the base for the filter, source and sink operators. While this does not > break a topology, users providing names for all processors in a DSL topology > may find the generated names for the repartition topics filter, source, and > sink operators as inconsistent with the naming approach. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-7106) Remove segment/segmentInterval from Window definition
[ https://issues.apache.org/jira/browse/KAFKA-7106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970598#comment-16970598 ] ASF GitHub Bot commented on KAFKA-7106: --- guozhangwang commented on pull request #7670: KAFKA-7106: Not hide the stack trace for ApiException URL: https://github.com/apache/kafka/pull/7670 Unit tests passed locally, do not think it has any compatibility breakage. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Remove segment/segmentInterval from Window definition > - > > Key: KAFKA-7106 > URL: https://issues.apache.org/jira/browse/KAFKA-7106 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: John Roesler >Priority: Blocker > Labels: needs-kip > Fix For: 3.0.0 > > > Currently, Window configures segment and segmentInterval properties, but > these aren't truly properties of a window in general. > Rather, they are properties of the particular implementation that we > currently have: a segmented store. Therefore, these properties should be > moved to configure only that implementation. > > This may be related to KAFKA-4730, since an in-memory window store wouldn't > necessarily need to be segmented. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9164) Don't evict active topics' metadata from the producer's cache
[ https://issues.apache.org/jira/browse/KAFKA-9164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970627#comment-16970627 ] ASF GitHub Bot commented on KAFKA-9164: --- bdbyrne commented on pull request #7672: KAFKA-9164: Improve producer topic expiry logic to account for last a… URL: https://github.com/apache/kafka/pull/7672 …ccess time. Note one, this changes the ProducerMetadata to longer record a sentinel TOPIC_EXPIRY_NEEDS_UPDATE upon topic map emplacement. What this value was doing was protecting the topic from being evicted on the first retainTopic() call, however I would anticipate that 5 minutes is plenty of guard, especially since a metadata update is forced immediately when a new topic arrives. Note two, in the accumulator, the current time is being calculated under lock, which might suggest that the times need to be ascending. However it doesn't appear that this is necessary, since the tryAppend also calculates the current time outside of the lock, and ProducerBatch is fairly relaxed on how it uses this value (nothing for correctness). Please let me know if there's any error in my understanding. ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build status - [ ] Verify documentation (including upgrade notes) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Don't evict active topics' metadata from the producer's cache > - > > Key: KAFKA-9164 > URL: https://issues.apache.org/jira/browse/KAFKA-9164 > Project: Kafka > Issue Type: Bug > Components: producer >Reporter: Brian Byrne >Assignee: Brian Byrne >Priority: Minor > > The producer's metadata currently marks a topic as "not needing to be > retained" if it has been 5 minutes since it was first considered, regardless > of whether records were being actively produced for the topic. This shouldn't > happen and should be fixed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9134) Refactor the StreamsPartitionAssignor for more code sharing with the FutureStreamsPartitionAssignor
[ https://issues.apache.org/jira/browse/KAFKA-9134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970665#comment-16970665 ] Sophie Blee-Goldman commented on KAFKA-9134: Hm, maybe the wording of this ticket could have been more carefully chosen but I think we're on the same page. There's actually not that much code that needs to differ in the future assignor's case, but the problem is it can currently only call the entire method. We should break up the various steps in StreamsPartitionAssignor#assign and let the future assignor override only what actually changes when the version gets bumped, and let them share the massive amounts of busy work in the middle between parsing the (versioned) subscription and writing the (versioned) assignment. I took a look through your PR and definitely agree it's moving things in the right direction, but I'd still want to do more with the StreamsPartitionAssignor to allow for sensible code sharing (I guess a more accurate description would be, sensible not-sharing-parts-of-code-that-should-be-different ;)) > Refactor the StreamsPartitionAssignor for more code sharing with the > FutureStreamsPartitionAssignor > --- > > Key: KAFKA-9134 > URL: https://issues.apache.org/jira/browse/KAFKA-9134 > Project: Kafka > Issue Type: Improvement > Components: streams >Reporter: Sophie Blee-Goldman >Priority: Major > > Frequently when fixing bugs in the StreamsPartitionAssignor, version probing, > or other assignor-related matters, we make a change in the > StreamsPartitionAssignor and re-run the version_probing_upgrade system test > only to see that the issue is not fixed because we forgot to mirror that > change in the FutureStreamsPartitionAssignor. Worse yet, we are making a new > change or fixing something that doesn't directly affect version probing, so > we update only the StreamsPartitionAssignor and don't even run the version > probing test, and discover later the version probing system test has started > failing. Then we often waste time digging through old changes just to > discover it was just because we forgot to copy any changes to the > StreamsUpgradeTest classes (includes also it's future version of the > SubscriptionInfo and AssignmentInfo classes) > > We should refactor the StreamsPartitionAssignor so that the future version > can rely more heavily on the original class. This will probably involve > either or all of: > * making the class's latestSupportedVersion configurable in the constructor, > to be used only by the system test. this will probably also require making it > configurable in some other classes such as SubscriptionInfo and AssignmentInfo > * breaking up the methods such as onAssignment and assign so that the future > assignor can call smaller pieces at a time – part of the original problem is > that the normal assignor will throw an exception partway through these > methods if the latest version is a "future" one, so we end up having to call > the method, catch the exception, and recode the remainder of the work in that > method -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9106) metrics exposed via JMX shoud be configurable
[ https://issues.apache.org/jira/browse/KAFKA-9106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970693#comment-16970693 ] ASF GitHub Bot commented on KAFKA-9106: --- xvrl commented on pull request #7674: KAFKA-9106 make metrics exposed via jmx configurable URL: https://github.com/apache/kafka/pull/7674 This change implements [KIP-544](https://cwiki.apache.org/confluence/display/KAFKA/KIP-544%3A+Make+metrics+exposed+via+JMX+configurable) (vote pending) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > metrics exposed via JMX shoud be configurable > - > > Key: KAFKA-9106 > URL: https://issues.apache.org/jira/browse/KAFKA-9106 > Project: Kafka > Issue Type: Improvement > Components: metrics >Reporter: Xavier Léauté >Assignee: Xavier Léauté >Priority: Major > > Kafka exposes a very large number of metrics, all of which are always visible > in JMX by default. On large clusters with many partitions, this may result in > tens of thousands of mbeans to be registered, which can lead to timeouts with > some popular monitoring agents that rely on listing JMX metrics via RMI. > Making the set of JMX-visible metrics configurable would allow operators to > decide on the set of critical metrics to collect and workaround limitation of > JMX in those cases. > corresponding KIP > https://cwiki.apache.org/confluence/display/KAFKA/KIP-544%3A+Make+metrics+exposed+via+JMX+configurable -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-9133) LogCleaner thread dies with: currentLog cannot be empty on an unexpected exception
[
https://issues.apache.org/jira/browse/KAFKA-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970702#comment-16970702
]
ASF GitHub Bot commented on KAFKA-9133:
---
hachikuji commented on pull request #7662: KAFKA-9133; Cleaner should handle
log start offset larger than active segment base offset
URL: https://github.com/apache/kafka/pull/7662
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> LogCleaner thread dies with: currentLog cannot be empty on an unexpected
> exception
> --
>
> Key: KAFKA-9133
> URL: https://issues.apache.org/jira/browse/KAFKA-9133
> Project: Kafka
> Issue Type: Bug
> Components: log cleaner
>Affects Versions: 2.3.1
>Reporter: Karolis Pocius
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 2.4.0, 2.3.2
>
>
> Log cleaner thread dies without a clear reference to which log is causing it:
> {code}
> [2019-11-02 11:59:59,078] INFO Starting the log cleaner (kafka.log.LogCleaner)
> [2019-11-02 11:59:59,144] INFO [kafka-log-cleaner-thread-0]: Starting
> (kafka.log.LogCleaner)
> [2019-11-02 11:59:59,199] ERROR [kafka-log-cleaner-thread-0]: Error due to
> (kafka.log.LogCleaner)
> java.lang.IllegalStateException: currentLog cannot be empty on an unexpected
> exception
> at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:346)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:307)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89)
> Caused by: java.lang.IllegalArgumentException: Illegal request for non-active
> segments beginning at offset 5033130, which is larger than the active
> segment's base offset 5019648
> at kafka.log.Log.nonActiveLogSegmentsFrom(Log.scala:1933)
> at
> kafka.log.LogCleanerManager$.maxCompactionDelay(LogCleanerManager.scala:491)
> at
> kafka.log.LogCleanerManager.$anonfun$grabFilthiestCompactedLog$4(LogCleanerManager.scala:184)
> at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at scala.collection.TraversableLike.map(TraversableLike.scala:238)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
> at scala.collection.immutable.List.map(List.scala:298)
> at
> kafka.log.LogCleanerManager.$anonfun$grabFilthiestCompactedLog$1(LogCleanerManager.scala:181)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
> at
> kafka.log.LogCleanerManager.grabFilthiestCompactedLog(LogCleanerManager.scala:171)
> at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:321)
> ... 2 more
> [2019-11-02 11:59:59,200] INFO [kafka-log-cleaner-thread-0]: Stopped
> (kafka.log.LogCleaner)
> {code}
> If I try to ressurect it by dynamically bumping {{log.cleaner.threads}} it
> instantly dies with the exact same error.
> Not sure if this is something KAFKA-8725 is supposed to address.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-9133) LogCleaner thread dies with: currentLog cannot be empty on an unexpected exception
[
https://issues.apache.org/jira/browse/KAFKA-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jason Gustafson resolved KAFKA-9133.
Resolution: Fixed
> LogCleaner thread dies with: currentLog cannot be empty on an unexpected
> exception
> --
>
> Key: KAFKA-9133
> URL: https://issues.apache.org/jira/browse/KAFKA-9133
> Project: Kafka
> Issue Type: Bug
> Components: log cleaner
>Affects Versions: 2.3.1
>Reporter: Karolis Pocius
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 2.4.0, 2.3.2
>
>
> Log cleaner thread dies without a clear reference to which log is causing it:
> {code}
> [2019-11-02 11:59:59,078] INFO Starting the log cleaner (kafka.log.LogCleaner)
> [2019-11-02 11:59:59,144] INFO [kafka-log-cleaner-thread-0]: Starting
> (kafka.log.LogCleaner)
> [2019-11-02 11:59:59,199] ERROR [kafka-log-cleaner-thread-0]: Error due to
> (kafka.log.LogCleaner)
> java.lang.IllegalStateException: currentLog cannot be empty on an unexpected
> exception
> at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:346)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:307)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89)
> Caused by: java.lang.IllegalArgumentException: Illegal request for non-active
> segments beginning at offset 5033130, which is larger than the active
> segment's base offset 5019648
> at kafka.log.Log.nonActiveLogSegmentsFrom(Log.scala:1933)
> at
> kafka.log.LogCleanerManager$.maxCompactionDelay(LogCleanerManager.scala:491)
> at
> kafka.log.LogCleanerManager.$anonfun$grabFilthiestCompactedLog$4(LogCleanerManager.scala:184)
> at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at scala.collection.TraversableLike.map(TraversableLike.scala:238)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
> at scala.collection.immutable.List.map(List.scala:298)
> at
> kafka.log.LogCleanerManager.$anonfun$grabFilthiestCompactedLog$1(LogCleanerManager.scala:181)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
> at
> kafka.log.LogCleanerManager.grabFilthiestCompactedLog(LogCleanerManager.scala:171)
> at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:321)
> ... 2 more
> [2019-11-02 11:59:59,200] INFO [kafka-log-cleaner-thread-0]: Stopped
> (kafka.log.LogCleaner)
> {code}
> If I try to ressurect it by dynamically bumping {{log.cleaner.threads}} it
> instantly dies with the exact same error.
> Not sure if this is something KAFKA-8725 is supposed to address.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KAFKA-9133) LogCleaner thread dies with: currentLog cannot be empty on an unexpected exception
[
https://issues.apache.org/jira/browse/KAFKA-9133?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970705#comment-16970705
]
Jason Gustafson commented on KAFKA-9133:
[~Karolis] We have committed a patch for this issue. Can you try it out when
you have a chance? Note the first 2.4 RC should be available soon.
> LogCleaner thread dies with: currentLog cannot be empty on an unexpected
> exception
> --
>
> Key: KAFKA-9133
> URL: https://issues.apache.org/jira/browse/KAFKA-9133
> Project: Kafka
> Issue Type: Bug
> Components: log cleaner
>Affects Versions: 2.3.1
>Reporter: Karolis Pocius
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 2.4.0, 2.3.2
>
>
> Log cleaner thread dies without a clear reference to which log is causing it:
> {code}
> [2019-11-02 11:59:59,078] INFO Starting the log cleaner (kafka.log.LogCleaner)
> [2019-11-02 11:59:59,144] INFO [kafka-log-cleaner-thread-0]: Starting
> (kafka.log.LogCleaner)
> [2019-11-02 11:59:59,199] ERROR [kafka-log-cleaner-thread-0]: Error due to
> (kafka.log.LogCleaner)
> java.lang.IllegalStateException: currentLog cannot be empty on an unexpected
> exception
> at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:346)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:307)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:89)
> Caused by: java.lang.IllegalArgumentException: Illegal request for non-active
> segments beginning at offset 5033130, which is larger than the active
> segment's base offset 5019648
> at kafka.log.Log.nonActiveLogSegmentsFrom(Log.scala:1933)
> at
> kafka.log.LogCleanerManager$.maxCompactionDelay(LogCleanerManager.scala:491)
> at
> kafka.log.LogCleanerManager.$anonfun$grabFilthiestCompactedLog$4(LogCleanerManager.scala:184)
> at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:238)
> at scala.collection.immutable.List.foreach(List.scala:392)
> at scala.collection.TraversableLike.map(TraversableLike.scala:238)
> at scala.collection.TraversableLike.map$(TraversableLike.scala:231)
> at scala.collection.immutable.List.map(List.scala:298)
> at
> kafka.log.LogCleanerManager.$anonfun$grabFilthiestCompactedLog$1(LogCleanerManager.scala:181)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:253)
> at
> kafka.log.LogCleanerManager.grabFilthiestCompactedLog(LogCleanerManager.scala:171)
> at kafka.log.LogCleaner$CleanerThread.cleanFilthiestLog(LogCleaner.scala:321)
> ... 2 more
> [2019-11-02 11:59:59,200] INFO [kafka-log-cleaner-thread-0]: Stopped
> (kafka.log.LogCleaner)
> {code}
> If I try to ressurect it by dynamically bumping {{log.cleaner.threads}} it
> instantly dies with the exact same error.
> Not sure if this is something KAFKA-8725 is supposed to address.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KAFKA-8984) Improve tagged fields documentation
[ https://issues.apache.org/jira/browse/KAFKA-8984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-8984. -- Fix Version/s: 2.4.0 Resolution: Fixed Issue resolved by pull request 7477 [https://github.com/apache/kafka/pull/7477] > Improve tagged fields documentation > --- > > Key: KAFKA-8984 > URL: https://issues.apache.org/jira/browse/KAFKA-8984 > Project: Kafka > Issue Type: Improvement > Components: protocol >Reporter: Colin McCabe >Assignee: Colin McCabe >Priority: Minor > Fix For: 2.4.0 > > > We should improve the KIP-482 tagged fields documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KAFKA-8984) Improve tagged fields documentation
[ https://issues.apache.org/jira/browse/KAFKA-8984?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16970721#comment-16970721 ] ASF GitHub Bot commented on KAFKA-8984: --- omkreddy commented on pull request #7477: KAFKA-8984: Improve tagged fields documentation URL: https://github.com/apache/kafka/pull/7477 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Improve tagged fields documentation > --- > > Key: KAFKA-8984 > URL: https://issues.apache.org/jira/browse/KAFKA-8984 > Project: Kafka > Issue Type: Improvement > Components: protocol >Reporter: Colin McCabe >Assignee: Colin McCabe >Priority: Minor > > We should improve the KIP-482 tagged fields documentation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
