[jira] [Updated] (KAFKA-1236) Change producer performance tool to optionally use the new producer
[ https://issues.apache.org/jira/browse/KAFKA-1236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-1236: --- Fix Version/s: (was: 0.10.0.0) (was: 0.10.1.0) 0.9.0.0 > Change producer performance tool to optionally use the new producer > --- > > Key: KAFKA-1236 > URL: https://issues.apache.org/jira/browse/KAFKA-1236 > Project: Kafka > Issue Type: Sub-task > Components: producer >Affects Versions: 0.10.1.0 >Reporter: Neha Narkhede >Assignee: Jay Kreps >Priority: Critical > Fix For: 0.9.0.0 > > Attachments: KAFKA-1236-v1.patch, KAFKA-1236.patch > > > The producer performance tool needs the ability to optionally use the new > producer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2375) Implement elasticsearch Copycat sink connector
[ https://issues.apache.org/jira/browse/KAFKA-2375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-2375: --- Fix Version/s: (was: 0.10.0.0) > Implement elasticsearch Copycat sink connector > -- > > Key: KAFKA-2375 > URL: https://issues.apache.org/jira/browse/KAFKA-2375 > Project: Kafka > Issue Type: Sub-task > Components: KafkaConnect >Reporter: Ewen Cheslack-Postava >Assignee: Liquan Pei > > Implement an elasticsearch sink connector for Copycat. This should send > records to elasticsearch with unique document IDs, given appropriate configs > to extract IDs from input records. > The motivation here is to provide a good end-to-end example with built-in > connectors that require minimal dependencies. Because Elasticsearch has a > very simple REST API, an elasticsearch connector shouldn't require any extra > dependencies and logs -> Elasticsearch (in combination with KAFKA-2374) > provides a compelling out-of-the-box Copycat use case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-665) Outgoing responses delayed on a busy Kafka broker
[ https://issues.apache.org/jira/browse/KAFKA-665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-665: -- Fix Version/s: (was: 0.10.0.0) (was: 0.10.1.0) 0.9.0.0 > Outgoing responses delayed on a busy Kafka broker > -- > > Key: KAFKA-665 > URL: https://issues.apache.org/jira/browse/KAFKA-665 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Neha Narkhede >Priority: Critical > Labels: replication-performance > Fix For: 0.9.0.0 > > > In a long running test, I observed that after a few hours of operation, few > requests start timing out, mainly because they spent very long time sitting > in the response queue - > [2012-12-07 22:05:56,670] TRACE Completed request with correlation id 3965966 > and client : TopicMetadataRequest:4009, queueTime:1, localTime:28, > remoteTime:0, sendTime:3980 (kafka.network.RequestChannel$) > [2012-12-07 22:04:12,046] TRACE Completed request with correlation id 3962561 > and client : TopicMetadataRequest:3449, queueTime:0, localTime:29, > remoteTime:0, sendTime:3420 (kafka.network.RequestChannel$) > [2012-12-07 22:05:56,670] TRACE Completed request with correlation id 3965966 > and client : TopicMetadataRequest:4009, queueTime:1, localTime:28, > remoteTime:0, sendTime:3980 (kafka.network.RequestChannel$) > We might have a problem in the way we process outgoing responses. Basically, > if the processor thread blocks on enqueuing requests in the request queue, it > doesn't come around to processing its responses which are ready to go out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2998) Log warnings when client is disconnected from bootstrap brokers
[ https://issues.apache.org/jira/browse/KAFKA-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-2998: --- Summary: Log warnings when client is disconnected from bootstrap brokers (was: New Consumer should not retry indefinitely if no broker is available) > Log warnings when client is disconnected from bootstrap brokers > --- > > Key: KAFKA-2998 > URL: https://issues.apache.org/jira/browse/KAFKA-2998 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 0.9.0.0 >Reporter: Florian Hussonnois >Assignee: Jason Gustafson >Priority: Minor > Fix For: 0.10.0.0 > > > If no broker from bootstrap.servers is available consumer retries > indefinitely with debug log message : > > DEBUG 17:16:13 Give up sending metadata request since no node is available > DEBUG 17:16:13 Initialize connection to node -1 for sending metadata request > DEBUG 17:16:13 Initiating connection to node -1 at localhost:9091. > At least, an ERROR message should be log after a number of retries. > In addition, maybe the consumer should fail in a such case ? This behavior > could be set by a configuration property ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-3088) 0.9.0.0 broker crash on receipt of produce request with empty client ID
[ https://issues.apache.org/jira/browse/KAFKA-3088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3088: --- Fix Version/s: 0.9.0.2 > 0.9.0.0 broker crash on receipt of produce request with empty client ID > --- > > Key: KAFKA-3088 > URL: https://issues.apache.org/jira/browse/KAFKA-3088 > Project: Kafka > Issue Type: Bug > Components: producer >Affects Versions: 0.9.0.0 >Reporter: Dave Peterson >Assignee: Grant Henke > Fix For: 0.10.0.0, 0.9.0.2 > > > Sending a produce request with an empty client ID to a 0.9.0.0 broker causes > the broker to crash as shown below. More details can be found in the > following email thread: > http://mail-archives.apache.org/mod_mbox/kafka-users/201601.mbox/%3c5693ecd9.4050...@dspeterson.com%3e >[2016-01-10 23:03:44,957] ERROR [KafkaApi-3] error when handling request > Name: ProducerRequest; Version: 0; CorrelationId: 1; ClientId: null; > RequiredAcks: 1; AckTimeoutMs: 1 ms; TopicAndPartition: [topic_1,3] -> 37 > (kafka.server.KafkaApis) >java.lang.NullPointerException > at > org.apache.kafka.common.metrics.JmxReporter.getMBeanName(JmxReporter.java:127) > at > org.apache.kafka.common.metrics.JmxReporter.addAttribute(JmxReporter.java:106) > at > org.apache.kafka.common.metrics.JmxReporter.metricChange(JmxReporter.java:76) > at > org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:288) > at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:177) > at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:162) > at > kafka.server.ClientQuotaManager.getOrCreateQuotaSensors(ClientQuotaManager.scala:209) > at > kafka.server.ClientQuotaManager.recordAndMaybeThrottle(ClientQuotaManager.scala:111) > at > kafka.server.KafkaApis.kafka$server$KafkaApis$$sendResponseCallback$2(KafkaApis.scala:353) > at > kafka.server.KafkaApis$$anonfun$handleProducerRequest$1.apply(KafkaApis.scala:371) > at > kafka.server.KafkaApis$$anonfun$handleProducerRequest$1.apply(KafkaApis.scala:371) > at > kafka.server.ReplicaManager.appendMessages(ReplicaManager.scala:348) > at kafka.server.KafkaApis.handleProducerRequest(KafkaApis.scala:366) > at kafka.server.KafkaApis.handle(KafkaApis.scala:68) > at > kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2998) Log warnings when client is disconnected from bootstrap brokers
[ https://issues.apache.org/jira/browse/KAFKA-2998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15283482#comment-15283482 ] Ismael Juma commented on KAFKA-2998: I updated the title to match what was actually implemented. > Log warnings when client is disconnected from bootstrap brokers > --- > > Key: KAFKA-2998 > URL: https://issues.apache.org/jira/browse/KAFKA-2998 > Project: Kafka > Issue Type: Bug > Components: clients >Affects Versions: 0.9.0.0 >Reporter: Florian Hussonnois >Assignee: Jason Gustafson >Priority: Minor > Fix For: 0.10.0.0 > > > If no broker from bootstrap.servers is available consumer retries > indefinitely with debug log message : > > DEBUG 17:16:13 Give up sending metadata request since no node is available > DEBUG 17:16:13 Initialize connection to node -1 for sending metadata request > DEBUG 17:16:13 Initiating connection to node -1 at localhost:9091. > At least, an ERROR message should be log after a number of retries. > In addition, maybe the consumer should fail in a such case ? This behavior > could be set by a configuration property ? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Closed] (KAFKA-665) Outgoing responses delayed on a busy Kafka broker
[ https://issues.apache.org/jira/browse/KAFKA-665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy closed KAFKA-665. - > Outgoing responses delayed on a busy Kafka broker > -- > > Key: KAFKA-665 > URL: https://issues.apache.org/jira/browse/KAFKA-665 > Project: Kafka > Issue Type: Bug >Affects Versions: 0.8.0 >Reporter: Neha Narkhede >Priority: Critical > Labels: replication-performance > Fix For: 0.9.0.0 > > > In a long running test, I observed that after a few hours of operation, few > requests start timing out, mainly because they spent very long time sitting > in the response queue - > [2012-12-07 22:05:56,670] TRACE Completed request with correlation id 3965966 > and client : TopicMetadataRequest:4009, queueTime:1, localTime:28, > remoteTime:0, sendTime:3980 (kafka.network.RequestChannel$) > [2012-12-07 22:04:12,046] TRACE Completed request with correlation id 3962561 > and client : TopicMetadataRequest:3449, queueTime:0, localTime:29, > remoteTime:0, sendTime:3420 (kafka.network.RequestChannel$) > [2012-12-07 22:05:56,670] TRACE Completed request with correlation id 3965966 > and client : TopicMetadataRequest:4009, queueTime:1, localTime:28, > remoteTime:0, sendTime:3980 (kafka.network.RequestChannel$) > We might have a problem in the way we process outgoing responses. Basically, > if the processor thread blocks on enqueuing requests in the request queue, it > doesn't come around to processing its responses which are ready to go out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] kafka pull request: Update doc
GitHub user satendrakumar06 opened a pull request: https://github.com/apache/kafka/pull/1386 Update doc Process grep command has been updated. Previous "ps | grep server-1.properties" command is showing nothing. You can merge this pull request into a Git repository by running: $ git pull https://github.com/satendrakumar06/kafka patch-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1386.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1386 commit fac6975ec347734ae84c53c4ad15c80ae681bd1e Author: Satendra Kumar Date: 2016-05-14T13:34:18Z Update doc Process grep command has been updated. Previous "ps | grep server-1.properties" command is showing nothing. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (KAFKA-3412) Multiple commitAsync() calls causes SendFailedException in commit callback
[ https://issues.apache.org/jira/browse/KAFKA-3412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15283583#comment-15283583 ] Gary Russell commented on KAFKA-3412: - Given that this is considered a Blocker, are there any plans to fix it in 0.9.0.2? > Multiple commitAsync() calls causes SendFailedException in commit callback > -- > > Key: KAFKA-3412 > URL: https://issues.apache.org/jira/browse/KAFKA-3412 > Project: Kafka > Issue Type: Bug > Components: consumer >Affects Versions: 0.9.0.1 >Reporter: Jason Gustafson >Assignee: Jason Gustafson >Priority: Blocker > Fix For: 0.10.1.0, 0.10.0.0 > > > If the user calls commitAsync() multiple times between poll() calls, some of > them will succeed, but many will be rejected with SendFailedException. This > is basically the result of NetworkClient only accepting one request to be > sent at a time and the higher level ConsumerNetworkClient not retrying after > send failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] kafka pull request: Modifier 'public' is redundant for interface m...
GitHub user philipealves opened a pull request: https://github.com/apache/kafka/pull/1387 Modifier 'public' is redundant for interface methods You can merge this pull request into a Git repository by running: $ git pull https://github.com/philipealves/kafka trunk Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/1387.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #1387 commit 58de360fa69be262c0c6c63287fcd8ed52e7b8cd Author: Philipe Alves de Oliveira e Silva Date: 2016-05-14T18:39:00Z Modifier 'public' is redundant for interface methods --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka-site pull request: MINOR: Replace Scala style guide PDF link...
GitHub user RyanCoonan opened a pull request: https://github.com/apache/kafka-site/pull/13 MINOR: Replace Scala style guide PDF link with scala-lang.org link The style guide PDF currently linked to is mostly empty with just "Moved to " links. You can merge this pull request into a Git repository by running: $ git pull https://github.com/RyanCoonan/kafka-site patch-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka-site/pull/13.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #13 commit f1560fc4b6f1fca03fa81f90b1112004bc6d59b7 Author: Ryan Coonan Date: 2016-05-14T19:00:37Z Replace style guide link with scala-lang.org link The style guide PDF linked to currently just points to this page anyway. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[GitHub] kafka-site pull request: MINOR: Fix typos and formatting in coding...
GitHub user RyanCoonan opened a pull request: https://github.com/apache/kafka-site/pull/14 MINOR: Fix typos and formatting in coding-guide.html You can merge this pull request into a Git repository by running: $ git pull https://github.com/RyanCoonan/kafka-site patch-2 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka-site/pull/14.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #14 commit 9d646947cc5a63eaca34f1768f2909207c601340 Author: Ryan Coonan Date: 2016-05-14T20:19:17Z MINOR: Fix typos and formatting in coding-guide.html --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
0.9.0.1 - KafkaProducer keeps requesting for metadata of topic which is no longer present, leads to never ending UNKNOWN_TOPIC_OR_PARTITION logs
Hello Kafka team,
We are using 0.9.0.1 (latest stable) of Kafka server and client
libraries. We use Java client for communicating with the Kafka
installation. Our simple application uses a single instance of
KafkaProducer (since the javadoc of that class states it's thread safe
and recommended to be shared across the threads) for the lifetime of our
application runtime. We seem to be running into a potential issue in the
Kafka producer and the way it expects metadata for topics which it had
communicated before but are no longer around.
The use case we have, where we run into this issue is as follows:
1. Our application is sent the name of the topic to which the
application sends a message using the KafkaProducer
2. The topics is short lived and after a while the topic is deleted via
Kafka tools externally
3. Our application continues to run and the next time it receives a
request to send a message to _some other topic_, it ends up running into
an issue where it endlessly floods the logs with messages:
10:17:53,245 WARN [NetworkClient] - Error while fetching metadata with
correlation id 122 :
{name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION}
10:17:53,347 WARN [NetworkClient] - Error while fetching metadata with
correlation id 123 :
{name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION}
10:17:53,449 WARN [NetworkClient] - Error while fetching metadata with
correlation id 124 :
{name-of-the-topic-that-got-deleted=UNKNOWN_TOPIC_OR_PARTITION}
It appears that the KafkaProducer wants to get hold of the metadata for
this topic which we deleted externally and which we have no intention to
communicate to anymore. These logs never stop, till we bring down the
application or close that producer instance.
This looks like an issue to me since the producer should either be aware
that the topic got deleted and no longer request for metadata (unless
there is an explicit call to send some message to it from the user
application) or it should just ignore the fact that the metadata for
this topic isn't there and move on without logging these logs (unless,
again, there is an explicit call to send some message to the deleted
topic, from the user application).
Looking at the code in the Kafka, it appears that the "topics" set gets
added with the topic name of the topic to which a communication is
established by the KafkaProducer. Once added, that topic name, never
gets removed from that set for the lifetime of that producer, even in
cases like these where the topic is deleted and never again used with
that producer.
Do you think this is a bug in the Kafka code? I have a simple
application which reproduces this easily on a 0.9.0.1 setup here
https://gist.github.com/jaikiran/45e9ce510c259267b28821b84105a25a.
Let me know if you need more details about this.
-Jaikiran
