[jira] [Assigned] (KAFKA-5905) Remove PrincipalBuilder and DefaultPrincipalBuilder

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar reassigned KAFKA-5905:


Assignee: Manikumar

> Remove PrincipalBuilder and DefaultPrincipalBuilder
> ---
>
> Key: KAFKA-5905
> URL: https://issues.apache.org/jira/browse/KAFKA-5905
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jason Gustafson
>Assignee: Manikumar
>Priority: Major
> Fix For: 2.0.0
>
>
> These classes were deprecated after KIP-189: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-189%3A+Improve+principal+builder+interface+and+add+support+for+SASL,
>  which is part of 1.0.0.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6956) Use Java AdminClient in BrokerApiVersionsCommand

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6956?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16492006#comment-16492006
 ] 

Manikumar commented on KAFKA-6956:
--

Existing Jira for this task :  https://issues.apache.org/jira/browse/KAFKA-5723

> Use Java AdminClient in BrokerApiVersionsCommand
> 
>
> Key: KAFKA-6956
> URL: https://issues.apache.org/jira/browse/KAFKA-6956
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Vahid Hashemian
>Priority: Major
>
> The Scala AdminClient was introduced as a stop gap until we had an officially 
> supported API. The Java AdminClient is the supported API so we should migrate 
> all usages to it and remove the Scala AdminClient. This JIRA is for using the 
> Java AdminClient in BrokerApiVersionsCommand. We would need to verify that 
> the necessary APIs are available via the Java AdminClient.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6929) ZkData - Consumers offsets Zookeeper path is not correct

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6929.
--
   Resolution: Fixed
Fix Version/s: 2.0.0

Fixed via https://github.com/apache/kafka/pull/5060

> ZkData - Consumers offsets Zookeeper path is not correct
> 
>
> Key: KAFKA-6929
> URL: https://issues.apache.org/jira/browse/KAFKA-6929
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.1.0
>Reporter: maytal
>Priority: Major
>  Labels: patch-available
> Fix For: 2.0.0
>
>
> ZkData.java contains ConsumerOffset.path which is wrong. should contain the 
> word `offsets` instead of `offset`.
> [https://github.com/maytals/kafka-1/blob/1.1/core/src/main/scala/kafka/zk/ZkData.scala#L411]
>  
> Already created patch:
> [https://github.com/apache/kafka/pull/5060]
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6947) Mirrormaker Closing producer due to send failure

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16491626#comment-16491626
 ] 

Manikumar commented on KAFKA-6947:
--

By default mirror maker config property "abort.on.send.failure" is true. So 
whenever producer gets error, mirror maker force close the producer instance.
"timeoutMillis = 0 ms" in logs indicates force closing of the producer.

We need to find out reason for producer send failure. you can try enabling 
producer debug logs to for more information.

> Mirrormaker Closing producer due to send failure
> 
>
> Key: KAFKA-6947
> URL: https://issues.apache.org/jira/browse/KAFKA-6947
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.11.0.0
>Reporter: Andrew Holford
>Priority: Major
>
> Hi
> On occasion our mirrormakers fail with the below error
> [2018-05-25 05:10:31,695] ERROR Error when sending message to topic 
> com_snapshot--demo with key: 13 bytes, value: 355 bytes with error: 
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> org.apache.kafka.common.errors.TimeoutException: Expiring 38 record(s) for 
> com_snapshot--demo-5: 91886 ms has passed since last append
> [2018-05-25 05:10:31,710] INFO Closing producer due to send failure. 
> (kafka.tools.MirrorMaker$)
> [2018-05-25 05:10:31,710] INFO Closing the Kafka producer with timeoutMillis 
> = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2018-05-25 05:10:31,710] INFO Proceeding to force close the producer since 
> pending requests could not be completed within timeout 0 ms. 
> (org.apache.kafka.clients.producer.KafkaProducer)
> and just after this:
> [2018-05-25 05:10:32,041] ERROR Error when sending message to topic 
> com_snapshot--demo with key: 13 bytes, value: 323 bytes with error: 
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
>  org.apache.kafka.common.errors.TimeoutException: Expiring 15 record(s) for 
> com_snapshot--demo-5: 89429 ms has passed since batch creation plus linger 
> time
>  [2018-05-25 05:10:32,041] INFO Closing producer due to send failure. 
> (kafka.tools.MirrorMaker$)
>  [2018-05-25 05:10:32,041] INFO Closing the Kafka producer with timeoutMillis 
> = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
>  [2018-05-25 05:10:32,041] INFO Proceeding to force close the producer since 
> pending requests could not be completed within timeout 0 ms. 
> (org.apache.kafka.clients.producer.KafkaProducer)
> and sometimes we get:
> [2018-05-25 08:24:29,249] ERROR Error when sending message to topic 
> com_snapshot--demo with key: 13 bytes, value: 354 bytes with error: 
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)
> java.lang.IllegalStateException: Producer is closed forcefully.
>  at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortBatches(RecordAccumulator.java:609)
>  at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortIncompleteBatches(RecordAccumulator.java:596)
>  at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:183)
>  at java.lang.Thread.run(Thread.java:745)
> [2018-05-25 08:24:29,249] INFO Closing producer due to send failure. 
> (kafka.tools.MirrorMaker$)
> [2018-05-25 08:24:29,249] INFO Closing the Kafka producer with timeoutMillis 
> = 0 ms. (org.apache.kafka.clients.producer.KafkaProducer)
> [2018-05-25 08:24:29,249] INFO Proceeding to force close the producer since 
> pending requests could not be completed within timeout 0 ms. 
> (org.apache.kafka.clients.producer.KafkaProducer)
>  then it completely shuts down:
> [2018-05-25 05:10:32,060] INFO Exiting on send failure, skip committing 
> offsets. (kafka.tools.MirrorMaker$)
>  [2018-05-25 05:10:32,083] INFO [mirrormaker-thread-0] Flushing producer. 
> (kafka.tools.MirrorMaker$MirrorMakerThread)
>  [2018-05-25 05:10:32,094] INFO [mirrormaker-thread-0] Committing consumer 
> offsets. (kafka.tools.MirrorMaker$MirrorMakerThread)
>  [2018-05-25 05:10:32,094] INFO Exiting on send failure, skip committing 
> offsets. (kafka.tools.MirrorMaker$)
>  [2018-05-25 05:10:32,105] INFO [mirrormaker-thread-0] Shutting down consumer 
> connectors. (kafka.tools.MirrorMaker$MirrorMakerThread)
>  [2018-05-25 05:10:32,138] INFO Publish thread interrupted! 
> (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor)
>  [2018-05-25 05:10:32,144] INFO Publishing Monitoring Metrics stopped for 
> clientID=peach-prod2prod3-mirrormaker-0 
> (io.confluent.monitoring.clients.interceptor.MonitoringInterceptor)
>  [2018-05-25 05:10:32,148] INFO Closing the Kafka producer with timeoutMillis 
> = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)
>  [2018-05-25 05:10:32,183] INFO Closed monitoring interceptor for client 
> ID=peach-prod2prod3-mirrormaker-0 
> 

[jira] [Resolved] (KAFKA-3649) Add capability to query broker process for configuration properties

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-3649.
--
Resolution: Fixed

Requested features are added in  KIP-133: Describe and Alter Configs Admin APIs 
and KIP-226 - Dynamic Broker Configuration.

> Add capability to query broker process for configuration properties
> ---
>
> Key: KAFKA-3649
> URL: https://issues.apache.org/jira/browse/KAFKA-3649
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin, config, core
>Affects Versions: 0.9.0.1, 0.10.0.0
>Reporter: David Tucker
>Assignee: Liquan Pei
>Priority: Major
>
> Developing an API by which running brokers could be queries for the various 
> configuration settings is an important feature to managing the Kafka cluster.
> Long term, the API could be enhanced to allow updates for those properties 
> that could be changed at run time ... but this involves a more thorough 
> evaluation of configuration properties (which once can be modified in a 
> running broker and which require a restart {of individual nodes or the entire 
> cluster}).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6921) Remove old Scala producer and all related code, tests, and tools

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6921.
--
Resolution: Fixed

> Remove old Scala producer and all related code, tests, and tools
> 
>
> Key: KAFKA-6921
> URL: https://issues.apache.org/jira/browse/KAFKA-6921
> Project: Kafka
>  Issue Type: Task
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>Priority: Major
> Fix For: 2.0.0
>
>




--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6945) Add support to allow users to acquire delegation tokens for other users

[Manikumar (JIRA)]

Manikumar created KAFKA-6945:


 Summary: Add support to allow users to acquire delegation tokens 
for other users
 Key: KAFKA-6945
 URL: https://issues.apache.org/jira/browse/KAFKA-6945
 Project: Kafka
  Issue Type: Sub-task
Reporter: Manikumar
Assignee: Manikumar


Currently, we only allow a user to create delegation token for that user only. 
We should allow users to acquire delegation tokens for other users.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6862) test toolchain

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6862?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6862.
--
Resolution: Invalid

Please reopen the Jira with more details.

> test toolchain
> --
>
> Key: KAFKA-6862
> URL: https://issues.apache.org/jira/browse/KAFKA-6862
> Project: Kafka
>  Issue Type: Test
>  Components: build
>Reporter: ravi
>Priority: Major
>
> test toolchain



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-4505) Cannot get topic lag since kafka upgrade from 0.8.1.0 to 0.10.1.0

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4505.
--
Resolution: Auto Closed

{color:#00}Closing inactive issue. Please reopen if you think the issue 
still exists{color}
 

> Cannot get topic lag since kafka upgrade from 0.8.1.0 to 0.10.1.0
> -
>
> Key: KAFKA-4505
> URL: https://issues.apache.org/jira/browse/KAFKA-4505
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, metrics, offset manager
>Affects Versions: 0.10.1.0
>Reporter: Romaric Parmentier
>Priority: Critical
>
> We were using kafka 0.8.1.1 and we just migrate to version 0.10.1.0.
> Since we migrate we are using the new script kafka-consumer-groups.sh to 
> retreive topic lags but it don't seem to work anymore. 
> Because the application is using the 0.8 driver we have added the following 
> conf to each kafka servers:
> log.message.format.version=0.8.2
> inter.broker.protocol.version=0.10.0.0
> When I'm using the option --list with kafka-consumer-groups.sh I can see 
> every consumer groups I'm using but the --describe is not working:
> /usr/share/kafka$ bin/kafka-consumer-groups.sh --zookeeper ip:2181 --describe 
> --group group_name
> No topic available for consumer group provided
> GROUP  TOPIC  PARTITION  
> CURRENT-OFFSET  LOG-END-OFFSET  LAG OWNER
> When I'm looking into zookeeper I can see the offset increasing for this 
> consumer group.
> Any idea ?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-2036) Consumer and broker have different networks

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2036.
--
Resolution: Duplicate

Resolving this duplicate of KIP-235/KIP-302 

> Consumer and broker have different networks
> ---
>
> Key: KAFKA-2036
> URL: https://issues.apache.org/jira/browse/KAFKA-2036
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 0.8.2.1
> Environment: oracle java {7,8}, ipv6 only consumer, ipv4 + ipv6 broker
>Reporter: Arsenii Krasikov
>Assignee: Jun Rao
>Priority: Major
> Attachments: patch, patch2
>
>
> If broker is connected to several networks ( for example ipv6 and ipv4 ) and 
> not all of them are reachable to consumer then 
> {{kafka.network.BlockingChannel}} gives up to connect after the first 
> "Network is unreachable" error not triyng remaining networks



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-3793) Kafka Python Consumer library messages gets truncated

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3793?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-3793.
--
Resolution: Cannot Reproduce

{color:#00}Closing inactive issue. Please reopen if you think the issue 
still exists{color}
 

> Kafka Python Consumer library messages gets truncated
> -
>
> Key: KAFKA-3793
> URL: https://issues.apache.org/jira/browse/KAFKA-3793
> Project: Kafka
>  Issue Type: Bug
>Reporter: Rahul
>Priority: Major
>
> Snippet code is below:
> from kafka import KafkaConsumer
> from kafka.client import KafkaClient
> from kafka.consumer import SimpleConsumer
> consumer = KafkaConsumer('eventdetails_ingestion' , 
> group_id='1',bootstrap_servers=‘:9092', 
> max_partition_fetch_bytes=1055)
> for msg in consumer:
>try:
>jValue = json.loads(str(msg.value))
>   except ValueError:
>fileErr.write(str(msg.value)+"\n")
> Steps:
> We send/produce large sets of messages to Kafka of around 20 to 30 KB size 
> each messages in JSON format and producing around 200 messages / sec for 1 
> hour duration. We have 3 Kafka Brokers running and I am trying to consume the 
> messages from these 3 Kafka Brokers from the same topic using the above code. 
> The problem is that sometimes some of the messages gets truncated, I am not 
> sure why it happen ?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6356) UnknownTopicOrPartitionException & NotLeaderForPartitionException and log deletion happening with retention bytes kept at -1.

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6356.
--
Resolution: Not A Problem

old data is discarded after log retention period or when the log reaches 
retention time.  In this case, you may need to increase retention period.  
replication errors looks normal. Please reopen if you think the issue still 
exists

Post these kind of queries to us...@kafka.apache.org mailing list 
(http://kafka.apache.org/contact) for quicker response.

> UnknownTopicOrPartitionException & NotLeaderForPartitionException and log 
> deletion happening with retention bytes kept at -1.
> -
>
> Key: KAFKA-6356
> URL: https://issues.apache.org/jira/browse/KAFKA-6356
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
> Environment: Cent OS 7.2,
> HDD : 2Tb,
> CPUs: 56 cores,
> RAM : 256GB
>Reporter: kaushik srinivas
>Priority: Major
> Attachments: configs.txt, stderr_b0, stderr_b1, stderr_b2, stdout_b0, 
> stdout_b1, stdout_b2, topic_description, topic_offsets
>
>
> Facing issues in kafka topic with partitions and replication factor of 3.
> Config used :
> No of partitions : 20
> replication factor : 3
> No of brokers : 3
> Memory for broker : 32GB
> Heap for broker : 12GB
> Producer is run to produce data for 20 partitions of a single topic.
> But observed that partitions for which the leader is one of the 
> broker(broker-1), the offsets are never incremented and also we see log file 
> with 0MB size in the broker disk.
> Seeing below error in the brokers :
> error 1:
> 2017-12-13 07:11:11,191] ERROR [ReplicaFetcherThread-0-2], Error for 
> partition [test2,5] to broker 
> 2:org.apache.kafka.common.errors.UnknownTopicOrPartitionException: This 
> server does not host this topic-partition. (kafka.server.ReplicaFetcherThread)
> error 2:
> [2017-12-11 12:19:41,599] ERROR [ReplicaFetcherThread-0-2], Error for 
> partition [test1,13] to broker 
> 2:org.apache.kafka.common.errors.NotLeaderForPartitionException: This server 
> is not the leader for that topic-partition. 
> (kafka.server.ReplicaFetcherThread)
> Attaching,
> 1. error and std out files of all the brokers.
> 2. kafka config used.
> 3. offsets and topic description.
> Retention bytes was kept to -1 and retention period 96 hours.
> But still observing some of the log files deleting at the broker,
> from logs :
> [2017-12-11 12:20:20,586] INFO Deleting index 
> /var/lib/mesos/slave/slaves/7b319cf4-f06e-4a35-a6fe-fd4fcc0548e6-S7/frameworks/7b319cf4-f06e-4a35-a6fe-fd4fcc0548e6-0006/executors/ckafka__5f085d0c-e296-40f0-a686-8953dd14e4c6/runs/506a1ce7-23d1-45ea-bb7c-84e015405285/kafka-broker-data/broker-1/test1-12/.timeindex
>  (kafka.log.TimeIndex)
> [2017-12-11 12:20:20,587] INFO Deleted log for partition [test1,12] in 
> /var/lib/mesos/slave/slaves/7b319cf4-f06e-4a35-a6fe-fd4fcc0548e6-S7/frameworks/7b319cf4-f06e-4a35-a6fe-fd4fcc0548e6-0006/executors/ckafka__5f085d0c-e296-40f0-a686-8953dd14e4c6/runs/506a1ce7-23d1-45ea-bb7c-84e015405285/kafka-broker-data/broker-1/test1-12.
>  (kafka.log.LogManager)
> We are expecting the logs to be never delete if retention bytes set to -1.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-2943) Transient Failure in kafka.producer.SyncProducerTest.testReachableServer

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2943?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2943.
--
Resolution: Auto Closed

> Transient Failure in kafka.producer.SyncProducerTest.testReachableServer
> 
>
> Key: KAFKA-2943
> URL: https://issues.apache.org/jira/browse/KAFKA-2943
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Priority: Major
>  Labels: transient-unit-test-failure
>
> {code}
> Stacktrace
> java.lang.AssertionError: Unexpected failure sending message to broker. null
>   at org.junit.Assert.fail(Assert.java:88)
>   at 
> kafka.producer.SyncProducerTest.testReachableServer(SyncProducerTest.scala:58)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
>   at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106)
>   at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
>   at 
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
>   at 
> org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at java.lang.Thread.run(Thread.java:744)
> Standard Output
> [2015-12-03 07:10:17,494] ERROR [Replica Manager on Broker 0]: Error 
> processing append operation on partition [minisrtest,0] 

[jira] [Resolved] (KAFKA-5432) producer and consumer SocketTimeoutException

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-5432.
--
Resolution: Auto Closed

Closing inactive issue.  Exceptions observed with deprecated scala clients.  
Please reopen if you think the issue still exists in newer versions.

> producer and consumer  SocketTimeoutException
> -
>
> Key: KAFKA-5432
> URL: https://issues.apache.org/jira/browse/KAFKA-5432
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 0.10.2.0
> Environment: os:Red Hat 4.4.7-17
> java:
> java version "1.8.0_131"
> Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
> Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)
>Reporter: Jian Lin
>Priority: Major
> Attachments: server.properties
>
>
> Hey all, I met a strange problem.
> The program ran normally for a week, and I did not do any changes, but today 
> it reported a mistake suddenly
> Producer error log:
> {code:java}
> 2017-06-12 10:46:01[qtp958382397-80:591423838]-[WARN] Failed to send producer 
> request with correlation id 234645 to broker 176 with data for partitions 
> [sms,3]
> java.net.SocketTimeoutException
>   at 
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:229)
>   at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
>   at 
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
>   at kafka.utils.Utils$.read(Utils.scala:380)
>   at 
> kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54)
>   at kafka.network.Receive$class.readCompletely(Transmission.scala:56)
>   at 
> kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29)
>   at kafka.network.BlockingChannel.receive(BlockingChannel.scala:111)
>   at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:75)
>   at 
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:72)
>   at 
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:103)
>   at 
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:103)
>   at 
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:103)
>   at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
>   at 
> kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.scala:102)
>   at 
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:102)
>   at 
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:102)
>   at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
>   at kafka.producer.SyncProducer.send(SyncProducer.scala:101)
>   at 
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:255)
>   at 
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:106)
>   at 
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$2.apply(DefaultEventHandler.scala:100)
>   at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
>   at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
>   at scala.collection.Iterator$class.foreach(Iterator.scala:772)
>   at 
> scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
>   at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
>   at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45)
>   at scala.collection.mutable.HashMap.foreach(HashMap.scala:95)
>   at 
> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:100)
>   at 
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
>   at kafka.producer.Producer.send(Producer.scala:77)
>   at kafka.javaapi.producer.Producer.send(Producer.scala:33)
> {code}
> Consumer error log:
> {code:java}
> 2017-06-12 
> 10:46:52[ConsumerFetcherThread-sms-consumer-group1_zw_78_64-1496632739724-69516149-0-176:602874523]-[INFO]
>  Reconnect due to socket error: java.net.SocketTimeoutException
> 2017-06-12 
> 10:47:22[ConsumerFetcherThread-sms-consumer-group1_zw_78_64-1496632739724-69516149-0-176:602904544]-[WARN]
>  
> [ConsumerFetcherThread-sms-consumer-group1_zw_78_64-1496632739724-69516149-0-176],
>  Error in fetch Name: FetchRequest; Version: 0; CorrelationId: 6060231; 
> ClientId: sms-consumer-group1; ReplicaId: -1; MaxWait: 100 ms; MinBytes: 1 
> bytes; RequestInfo: [sms,0] -> PartitionFetchInfo(81132,1048576),[sms,3] -> 
> 

[jira] [Resolved] (KAFKA-4419) Unable to GetOffset when the ACL of topic is defined

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4419.
--
Resolution: Duplicate

Resolving as duplicate of  KAFKA-3355.

> Unable to GetOffset when the ACL of topic is defined 
> -
>
> Key: KAFKA-4419
> URL: https://issues.apache.org/jira/browse/KAFKA-4419
> Project: Kafka
>  Issue Type: Bug
>  Components: admin, security
>Affects Versions: 0.9.0.1
> Environment: kafka 0.9.0.1 
> centos 7
> kafka server with kerberos (zokeeper also with kerberos)
> listeners=PLAINTEXT://:9092,SASL_PLAINTEXT://:9093
>Reporter: Mohammed amine GARMES
>Priority: Critical
>  Labels: security
>   Original Estimate: 12h
>  Remaining Estimate: 12h
>
> I have a kafka server with kerberos enable 
> (listeners=PLAINTEXT://:9092,SASL_PLAINTEXT://:9093), I create a test topic 
> and I pushed some data to in. I run the GetOffsetShell  to get  topic offset :
> [root@kafka1 ~]#  /opt/kafka/bin/kafka-run-class.sh 
> kafka.tools.GetOffsetShell --broker-list kafka1:9092 --topic test-topic 
> --time -1 [2016-11-17 16:52:02,471] INFO 
> Verifying properties (kafka.utils.VerifiableProperties)
> [2016-11-17 16:52:02,479] INFO Property client.id is overridden to 
> GetOffsetShell (kafka.utils.VerifiableProperties)
> [2016-11-17 16:52:02,479] INFO Property metadata.broker.list is overridden to 
> kafka1:9092 (kafka.utils.VerifiableProperties)
> [2016-11-17 16:52:02,480] INFO Property request.timeout.ms is overridden to 
> 1000 (kafka.utils.VerifiableProperties)
> [2016-11-17 16:52:02,513] INFO Fetching metadata from broker 
> BrokerEndPoint(0,kafka1,9092) with correlation id 0 for 1 topic(s) 
> Set(test-topic) (kafka.client.ClientUtils$)
> [2016-11-17 16:52:02,561] INFO Connected to kafka1:9092 for producing 
> (kafka.producer.SyncProducer)
> [2016-11-17 16:52:02,573] INFO Disconnecting from kafka1:9092 
> (kafka.producer.SyncProducer)
> test-topic:2:773
> test-topic:1:773
> test-topic:0:772
> I added an user to ACL for my test topic:
> [root@kafka1 ~]#   $KAFKA_HOME/bin/kafka-acls.sh --authorizer-properties 
> zookeeper.connect=kafka1:2181,kafka2:2181,kafka3:2181   --add 
> --allow-principal User:garmes  --operation All  --topic test-topic
> I pushed some data again. I run the GetOffsetShell  to get  topic offset but 
> this time I dont have offset :
> [root@kafka1 ~]#  /opt/kafka/bin/kafka-run-class.sh 
> kafka.tools.GetOffsetShell --broker-list kafka1:9092 --topic test-topic 
> --time -1
> [2016-11-17 16:43:31,289] INFO Verifying properties 
> (kafka.utils.VerifiableProperties)
> [2016-11-17 16:43:31,305] INFO Property client.id is overridden to 
> GetOffsetShell (kafka.utils.VerifiableProperties)
> [2016-11-17 16:43:31,305] INFO Property metadata.broker.list is overridden to 
> kafka1:9092 (kafka.utils.VerifiableProperties)
> [2016-11-17 16:43:31,305] INFO Property request.timeout.ms is overridden to 
> 1000 (kafka.utils.VerifiableProperties)
> [2016-11-17 16:43:31,339] INFO Fetching metadata from broker 
> BrokerEndPoint(0,kafka1,9092) with correlation id 0 for 1 topic(s) 
> Set(test-topic) (kafka.client.ClientUtils$)
> [2016-11-17 16:43:31,382] INFO Connected to kafka1:9092 for producing 
> (kafka.producer.SyncProducer)
> [2016-11-17 16:43:31,394] INFO Disconnecting from kafka1:9092 
> (kafka.producer.SyncProducer)
> [root@kafka1 ~]# 
> I changed the broker port from 9092 to 9093, but I have below  error :
> [root@kafka1 ~]# /opt/kafka/bin/kafka-run-class.sh kafka.tools.GetOffsetShell 
> --broker-list kafka1:9093 --topic test-topic --time -1
>  [2016-11-17 16:59:18,112] INFO Verifying properties 
> (kafka.utils.VerifiableProperties)
> [2016-11-17 16:59:18,129] INFO Property client.id is overridden to 
> GetOffsetShell (kafka.utils.VerifiableProperties)
> [2016-11-17 16:59:18,129] INFO Property metadata.broker.list is overridden to 
> kafka1:9093 (kafka.utils.VerifiableProperties)
> [2016-11-17 16:59:18,129] INFO Property request.timeout.ms is overridden to 
> 1000 (kafka.utils.VerifiableProperties)
> [2016-11-17 16:59:18,162] INFO Fetching metadata from broker 
> BrokerEndPoint(0,kafka1,9093) with correlation id 0 for 1 topic(s) 
> Set(test-topic) (kafka.client.ClientUtils$)
> [2016-11-17 16:59:18,206] INFO Connected to kafka1:9093 for producing 
> (kafka.producer.SyncProducer)
> [2016-11-17 16:59:18,210] INFO Disconnecting from kafka1:9093 
> (kafka.producer.SyncProducer)
> [2016-11-17 16:59:18,212] WARN Fetching topic metadata with correlation id 0 
> for topics [Set(test-topic)] from broker [BrokerEndPoint(0,kafka1,9093)] 
> failed (kafka.client.ClientUtils$)
> java.io.EOFException
> at 
> 

[jira] [Resolved] (KAFKA-1460) NoReplicaOnlineException: No replica for partition

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1460.
--
Resolution: Auto Closed

Closing inactive issue.  Please reopen if you think the issue still exists in 
newer versions.

> NoReplicaOnlineException: No replica for partition
> --
>
> Key: KAFKA-1460
> URL: https://issues.apache.org/jira/browse/KAFKA-1460
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Artur Denysenko
>Priority: Critical
> Attachments: state-change.log
>
>
> We have a standalone kafka server.
> After several days of running we get:
> {noformat}
> kafka.common.NoReplicaOnlineException: No replica for partition 
> [gk.q.module,1] is alive. Live brokers are: [Set()], Assigned replicas are: 
> [List(0)]
>   at 
> kafka.controller.OfflinePartitionLeaderSelector.selectLeader(PartitionLeaderSelector.scala:61)
>   at 
> kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:336)
>   at 
> kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:185)
>   at 
> kafka.controller.PartitionStateMachine$$anonfun$triggerOnlinePartitionStateChange$3.apply(PartitionStateMachine.scala:99)
>   at 
> kafka.controller.PartitionStateMachine$$anonfun$triggerOnlinePartitionStateChange$3.apply(PartitionStateMachine.scala:96)
>   at 
> scala.collection.TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:743)
>   at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
>   at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:95)
>   at scala.collection.Iterator$class.foreach(Iterator.scala:772)
>   at 
> scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:157)
>   at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:190)
>   at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:45)
>   at scala.collection.mutable.HashMap.foreach(HashMap.scala:95)
>   at 
> scala.collection.TraversableLike$WithFilter.foreach(TraversableLike.scala:742)
>   at 
> kafka.controller.PartitionStateMachine.triggerOnlinePartitionStateChange(PartitionStateMachine.scala:96)
>   at 
> kafka.controller.PartitionStateMachine.startup(PartitionStateMachine.scala:68)
>   at 
> kafka.controller.KafkaController.onControllerFailover(KafkaController.scala:312)
>   at 
> kafka.controller.KafkaController$$anonfun$1.apply$mcV$sp(KafkaController.scala:162)
>   at 
> kafka.server.ZookeeperLeaderElector.elect(ZookeeperLeaderElector.scala:63)
>   at 
> kafka.controller.KafkaController$SessionExpirationListener$$anonfun$handleNewSession$1.apply$mcZ$sp(KafkaController.scala:1068)
>   at 
> kafka.controller.KafkaController$SessionExpirationListener$$anonfun$handleNewSession$1.apply(KafkaController.scala:1066)
>   at 
> kafka.controller.KafkaController$SessionExpirationListener$$anonfun$handleNewSession$1.apply(KafkaController.scala:1066)
>   at kafka.utils.Utils$.inLock(Utils.scala:538)
>   at 
> kafka.controller.KafkaController$SessionExpirationListener.handleNewSession(KafkaController.scala:1066)
>   at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472)
>   at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> {noformat}
> Please see attached [state-change.log]
> You can find all server logs (450mb) here: 
> http://46.4.114.35:/deploy/kafka-logs.2014-05-14-16.tgz
> On client we get:
> {noformat}
> 16:28:36,843 [ool-12-thread-2] WARN  ZookeeperConsumerConnector - 
> [dev_dev-1400257716132-e7b8240c], no brokers found when trying to rebalance.
> {noformat}
> If we try to send message using 'kafka-console-producer.sh':
> {noformat}
> [root@dev kafka]# /srv/kafka/bin/kafka-console-producer.sh --broker-list 
> localhost:9092 --topic test
> message
> SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further 
> details.
> [2014-05-16 19:45:30,950] WARN Fetching topic metadata with correlation id 0 
> for topics [Set(test)] from broker [id:0,host:localhost,port:9092] failed 
> (kafka.client.ClientUtils$)
> java.net.SocketTimeoutException
> at 
> sun.nio.ch.SocketAdaptor$SocketInputStream.read(SocketAdaptor.java:229)
> at sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103)
> at 
> java.nio.channels.Channels$ReadableByteChannelImpl.read(Channels.java:385)
> at kafka.utils.Utils$.read(Utils.scala:375)
> at 
> 

[jira] [Updated] (KAFKA-3355) GetOffsetShell command doesn't work with SASL enabled Kafka

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar updated KAFKA-3355:
-
Fix Version/s: 2.0.0

> GetOffsetShell command doesn't work with SASL enabled Kafka
> ---
>
> Key: KAFKA-3355
> URL: https://issues.apache.org/jira/browse/KAFKA-3355
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.9.0.1
> Environment: Kafka 0.9.0.1
>Reporter: TAO XIAO
>Assignee: Ashish Singh
>Priority: Major
> Fix For: 2.0.0
>
>
> I found that GetOffsetShell doesn't work with SASL enabled Kafka. I believe 
> this is due to old producer being used in GetOffsetShell.
> Kafka version 0.9.0.1
> Exception
> % bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 
> localhost:9092 --topic test --time -1
> [2016-03-04 21:43:56,597] INFO Verifying properties 
> (kafka.utils.VerifiableProperties)
> [2016-03-04 21:43:56,613] INFO Property client.id is overridden to 
> GetOffsetShell (kafka.utils.VerifiableProperties)
> [2016-03-04 21:43:56,613] INFO Property metadata.broker.list is overridden to 
> localhost:9092 (kafka.utils.VerifiableProperties)
> [2016-03-04 21:43:56,613] INFO Property request.timeout.ms is overridden to 
> 1000 (kafka.utils.VerifiableProperties)
> [2016-03-04 21:43:56,674] INFO Fetching metadata from broker 
> BrokerEndPoint(0,localhost,9092) with correlation id 0 for 1 topic(s) 
> Set(test) (kafka.client.ClientUtils$)
> [2016-03-04 21:43:56,689] INFO Connected to localhost:9092 for producing 
> (kafka.producer.SyncProducer)
> [2016-03-04 21:43:56,705] WARN Fetching topic metadata with correlation id 0 
> for topics [Set(test)] from broker [BrokerEndPoint(0,localhost,9092)] failed 
> (kafka.client.ClientUtils$)
> java.nio.BufferUnderflowException
>   at java.nio.Buffer.nextGetIndex(Buffer.java:498)
>   at java.nio.HeapByteBuffer.getShort(HeapByteBuffer.java:304)
>   at kafka.api.ApiUtils$.readShortString(ApiUtils.scala:36)
>   at kafka.cluster.BrokerEndPoint$.readFrom(BrokerEndPoint.scala:52)
>   at 
> kafka.api.TopicMetadataResponse$$anonfun$1.apply(TopicMetadataResponse.scala:28)
>   at 
> kafka.api.TopicMetadataResponse$$anonfun$1.apply(TopicMetadataResponse.scala:28)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:245)
>   at scala.collection.immutable.Range.foreach(Range.scala:166)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:245)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:104)
>   at 
> kafka.api.TopicMetadataResponse$.readFrom(TopicMetadataResponse.scala:28)
>   at kafka.producer.SyncProducer.send(SyncProducer.scala:120)
>   at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
>   at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:94)
>   at kafka.tools.GetOffsetShell$.main(GetOffsetShell.scala:78)
>   at kafka.tools.GetOffsetShell.main(GetOffsetShell.scala)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-2174) Wrong TopicMetadata deserialization

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2174.
--
Resolution: Auto Closed

Closing inactive issue.

> Wrong TopicMetadata deserialization
> ---
>
> Key: KAFKA-2174
> URL: https://issues.apache.org/jira/browse/KAFKA-2174
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Alexey Ozeritskiy
>Priority: Major
> Attachments: KAFKA-2174.patch
>
>
> TopicMetadata.readFrom assumes that ByteBuffer always contains the full set 
> of partitions but it is not true. On incomplete metadata we will get 
> java.lang.ArrayIndexOutOfBoundsException:
> {code}
> java.lang.ArrayIndexOutOfBoundsException: 47
> at 
> kafka.api.TopicMetadata$$anonfun$readFrom$1.apply$mcVI$sp(TopicMetadata.scala:38)
> at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:141)
> at kafka.api.TopicMetadata$.readFrom(TopicMetadata.scala:36)
> at 
> kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
> at 
> kafka.api.TopicMetadataResponse$$anonfun$3.apply(TopicMetadataResponse.scala:31)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at scala.collection.immutable.Range.foreach(Range.scala:141)
> at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at 
> kafka.api.TopicMetadataResponse$.readFrom(TopicMetadataResponse.scala:31)
> {code}
> We sometimes get this exceptions on any broker restart (kill -TERM, 
> controlled.shutdown.enable=false).



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-3345) ProducerResponse could gracefully handle no throttle time provided

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-3345.
--
Resolution: Auto Closed

Closing as scala producer is removed from codebase.
[|https://issues.apache.org/jira/secure/AddComment!default.jspa?id=13062336]

> ProducerResponse could gracefully handle no throttle time provided
> --
>
> Key: KAFKA-3345
> URL: https://issues.apache.org/jira/browse/KAFKA-3345
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Bryan Baugher
>Priority: Minor
>
> When doing some compatibility testing between kafka 0.8 and 0.9 I found that 
> the old producer using 0.9 libraries could write to a cluster running 0.8 if 
> 'request.required.acks' was set to 0. If it was set to anything else it would 
> fail with,
> {code}
> java.nio.BufferUnderflowException
>   at java.nio.Buffer.nextGetIndex(Buffer.java:506) 
>   at java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:361) 
>   at kafka.api.ProducerResponse$.readFrom(ProducerResponse.scala:41) 
>   at kafka.producer.SyncProducer.send(SyncProducer.scala:109) 
> {code}
> In 0.9 there was a one line change to the response here[1] to look for a 
> throttle time value in the response. It seems if the 0.9 code gracefully 
> handled throttle time not being provided this would work. Would you be open 
> to this change?
> [1] - 
> https://github.com/apache/kafka/blob/0.9.0.1/core/src/main/scala/kafka/api/ProducerResponse.scala#L41



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-5039) Logging in BlockingChannel and SyncProducer connect

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-5039.
--
Resolution: Auto Closed

Closing as scala producer is removed from codebase.

> Logging in BlockingChannel and SyncProducer connect
> ---
>
> Key: KAFKA-5039
> URL: https://issues.apache.org/jira/browse/KAFKA-5039
> Project: Kafka
>  Issue Type: Bug
>Reporter: Arun Mahadevan
>Assignee: Arun Mahadevan
>Priority: Minor
>
> When an exception is thrown in BlockingChannel::connect, the connection is 
> disconnected but the actual exception is not logged. This later manifests as 
> ClosedChannelException when trying to send. Also the SyncProducer wrongfully 
> logs "Connected to host:port for producing" even in case of exceptions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-3246) Transient Failure in kafka.producer.SyncProducerTest.testProduceCorrectlyReceivesResponse

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-3246.
--
Resolution: Auto Closed

> Transient Failure in 
> kafka.producer.SyncProducerTest.testProduceCorrectlyReceivesResponse
> -
>
> Key: KAFKA-3246
> URL: https://issues.apache.org/jira/browse/KAFKA-3246
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Priority: Major
>  Labels: transient-unit-test-failure
>
> {code}
> java.lang.AssertionError: expected:<0> but was:<3>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at org.junit.Assert.assertEquals(Assert.java:631)
>   at 
> kafka.producer.SyncProducerTest.testProduceCorrectlyReceivesResponse(SyncProducerTest.scala:182)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
>   at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106)
>   at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
>   at 
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
>   at 
> org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> 

[jira] [Resolved] (KAFKA-5728) Stopping consumer thread cause loosing message in the partition

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-5728.
--
Resolution: Auto Closed

looks like this is related to spring kafka config issue. must be related to 
committing offsets. Pls take a look at spring kafka docs.

> Stopping consumer thread cause loosing message in the partition
> ---
>
> Key: KAFKA-5728
> URL: https://issues.apache.org/jira/browse/KAFKA-5728
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.1.0
>Reporter: Vasudevan Karnan
>Priority: Major
>
> Currently using Spring boot Kafka listener thread to consume the message from 
> partition.
> Having 10 partitions and concurrency to 10 in the consumer group.
> In testing, I have 2 messages in the single partition (say for ex: partition 
> 4). Created listener to read the message and post to service. During normal 
> days, read the message and post to service, and working as expected. No 
> issues on that.
> Suppose if the service is down, then I am doing Spring Retry template to 
> retry to post the message to service (repeatedly) for number of retry and 
> backoff time in ms. If I stop the listener, then getting 
> org.springframework.retry.backoff.BackOffInterruptedException: Thread 
> interrupted while sleeping; nested exception is 
> java.lang.InterruptedException: sleep interrupted
>   at 
> org.springframework.retry.backoff.FixedBackOffPolicy.doBackOff(FixedBackOffPolicy.java:86)
>  ~[spring-retry-1.1.4.RELEASE.jar:na]
>   at 
> org.springframework.retry.backoff.StatelessBackOffPolicy.backOff(StatelessBackOffPolicy.java:36)
>  ~[spring-retry-1.1.4.RELEASE.jar:na]
> After that I am loosing the message from particular partition (message that 
> are got retried is lost in the middle) and lag is reduced. (This is happening 
> during the end of stopping the listener).
> Is there any way, we don't loose the message even I am getting the sleep 
> interrupted exception?
> Suppose during stopping the server, if I dont face sleep interrupt exception, 
> in the next time listener startup, face the same issue and loosing the 
> message again.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-4091) Unable to produce or consume on any topic

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4091.
--
Resolution: Cannot Reproduce

{color:#00}Closing inactive issue. Please reopen if you think the issue 
still exists.{color}
 

> Unable to produce or consume on any topic
> -
>
> Key: KAFKA-4091
> URL: https://issues.apache.org/jira/browse/KAFKA-4091
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
> Environment: Amazon Linux, t2.micro
>Reporter: Avi Chopra
>Priority: Critical
>
> While trying to set kafka on 2 slave and 1 master box, got a weird condition 
> where I was not able to consume or produce to a topic.
> Using Mirror Maker to sync data between slave <--> Master. Getting following 
> logs unending :
> [2016-08-26 14:28:33,897] WARN Bootstrap broker localhost:9092 disconnected 
> (org.apache.kafka.clients.NetworkClient) [2016-08-26 14:28:43,515] WARN 
> Bootstrap broker localhost:9092 disconnected 
> (org.apache.kafka.clients.NetworkClient) [2016-08-26 14:28:45,118] WARN 
> Bootstrap broker localhost:9092 disconnected 
> (org.apache.kafka.clients.NetworkClient) [2016-08-26 14:28:46,721] WARN 
> Bootstrap broker localhost:9092 disconnected 
> (org.apache.kafka.clients.NetworkClient) [2016-08-26 14:28:48,324] WARN 
> Bootstrap broker localhost:9092 disconnected 
> (org.apache.kafka.clients.NetworkClient) [2016-08-26 14:28:49,927] WARN 
> Bootstrap broker localhost:9092 disconnected 
> (org.apache.kafka.clients.NetworkClient) [2016-08-26 14:28:53,029] WARN 
> Bootstrap broker localhost:9092 disconnected 
> (org.apache.kafka.clients.NetworkClient)
> Only way I could recover was by restarting Kafka which produced this kind of 
> logs :
> [2016-08-26 14:30:54,856] WARN Found a corrupted index file, 
> /tmp/kafka-logs/__consumer_offsets-43/.index, deleting 
> and rebuilding index... (kafka.log.Log) [2016-08-26 14:30:54,856] INFO 
> Recovering unflushed segment 0 in log __consumer_offsets-43. (kafka.log.Log) 
> [2016-08-26 14:30:54,857] INFO Completed load of log __consumer_offsets-43 
> with log end offset 0 (kafka.log.Log) [2016-08-26 14:30:54,860] WARN Found a 
> corrupted index file, 
> /tmp/kafka-logs/__consumer_offsets-26/.index, deleting 
> and rebuilding index... (kafka.log.Log) [2016-08-26 14:30:54,860] INFO 
> Recovering unflushed segment 0 in log __consumer_offsets-26. (kafka.log.Log) 
> [2016-08-26 14:30:54,861] INFO Completed load of log __consumer_offsets-26 
> with log end offset 0 (kafka.log.Log) [2016-08-26 14:30:54,864] WARN Found a 
> corrupted index file, 
> /tmp/kafka-logs/__consumer_offsets-35/.index, deleting 
> and rebuilding index... (kafka.log.Log)
> ERROR Error when sending message to topic dr_ubr_analytics_limits with key: 
> null, value: 1 bytes with error: 
> (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback) 
> org.apache.kafka.common.errors.TimeoutException: Failed to update metadata 
> after 6 ms.
> The consumer group command was showing a major lag.
> This is my test phase so I was able to restart and recover from the master 
> box but I want know what caused this issue and how can it be avoided. Is 
> there a way to debug this issue?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-3843) Endless consuming messages in Kafka cluster

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-3843.
--
Resolution: Auto Closed

The Scala consumers have been deprecated and no further work is planned, please 
upgrade to the Java consumer whenever possible.

> Endless consuming messages in Kafka cluster
> ---
>
> Key: KAFKA-3843
> URL: https://issues.apache.org/jira/browse/KAFKA-3843
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.1
>Reporter: Tomas Benc
>Assignee: Neha Narkhede
>Priority: Major
>
> We are running Kafka in cluster (3 virtual machines). Kafka is configured  
> min.insync.replicas = 2 and topics are configured replication factor = 3. 
> This configuration means, there must be at least 2 brokers of 3 in cluster up 
> and running to receive any messages. This works as expected.
> Our consumers are high level consumers and offsets are commited manually 
> (auto.commit disabled) and stored in Kafka.
> Reproducing the issue:
> 1. Kafka cluster up and running and receives messages
> 2. Consumers are disabled (messages in Kafka are in lag)
> 3. Disable 2 Kafka brokers in cluster
> 4. Enable consumers
> Consumers are consuming messages in batch, and commiting offsets after 
> processing. But commit offsets fails in Kafka, because of 
> NotEnoughReplicasException. That is correct. What is not correct, high level 
> consumer has no idea, that offset are not commited and consumes same messages 
> again and again.
> It would be helpful, that method commitOffsets() in interface 
> kafka.javaapi.consumer.ConsumerConnector should return some information 
> (return boolean or throw exception) about this operation.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-3620) Clean up Protocol class.

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-3620.
--
Resolution: Fixed

Closing this as some cleanup done in newer versions.  Please reopen if you 
think otherwise.

> Clean up Protocol class.
> 
>
> Key: KAFKA-3620
> URL: https://issues.apache.org/jira/browse/KAFKA-3620
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ashish Singh
>Assignee: Ashish Singh
>Priority: Major
>
> This came up on PR of KAFKA-3307. Below is excerpt.
> {quote}
> With the versioning getting a little more complex in Protocol class, it makes 
> sense to try and encapsulate some of its logic a little better. For example, 
> rather than using raw arrays for each request type, we could have something 
> like this:
> {code}
> class KafkaApi {
>   private ApiKey api;
>   private Schema[] requests;
>   private Schema[] responses;
>   Schema currentSchema();
>   Schema schemaFor(int version);
>   int minVersion();
>   int currentVersion();
> }
> {code}
> {quote}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-3180) issue for extracting JSON from https web page

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3180?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-3180.
--
Resolution: Not A Problem

 I suggest to post these kind of queries to 
[[us...@kafka.apache.org|mailto:us...@kafka.apache.org]|mailto:[us...@kafka.apache.org|mailto:us...@kafka.apache.org]]
 mailing list ([[http://kafka.apache.org/contact]]) for more visibility.

> issue for extracting JSON from https web page
> -
>
> Key: KAFKA-3180
> URL: https://issues.apache.org/jira/browse/KAFKA-3180
> Project: Kafka
>  Issue Type: Test
>  Components: clients
>Affects Versions: 0.9.0.0
> Environment: cloudera 5.4.2.0
>Reporter: swayam
>Priority: Critical
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Hi Team,
> Could you help me how to extract JSON info from https web page by help of 
> kafka into HDFS . 
> here is the json available URL : 
> https://affiliate-api.flipkart.net/affiliate/api/8924b177d4c64fcab4db860b94fbcea2.json
> Please help me to get the info ..



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-1977) Make logEndOffset available in the Zookeeper consumer

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1977.
--
Resolution: Auto Closed

Closing this as the Scala consumers have been deprecated and no further work is 
planned. This requirement will be tracked in KAFKA-2500 for java consumer.

> Make logEndOffset available in the Zookeeper consumer
> -
>
> Key: KAFKA-1977
> URL: https://issues.apache.org/jira/browse/KAFKA-1977
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Reporter: Will Funnell
>Priority: Minor
> Attachments: 
> Make_logEndOffset_available_in_the_Zookeeper_consumer.patch
>
>
> The requirement is to create a snapshot from the Kafka topic but NOT do 
> continual reads after that point. For example you might be creating a backup 
> of the data to a file.
> In order to achieve that, a recommended solution by Joel Koshy and Jay Kreps 
> was to expose the high watermark, as maxEndOffset, from the FetchResponse 
> object through to each MessageAndMetadata object in order to be aware when 
> the consumer has reached the end of each partition.
> The submitted patch achieves this by adding the maxEndOffset to the 
> PartitionTopicInfo, which is updated when a new message arrives in the 
> ConsumerFetcherThread and then exposed in MessageAndMetadata.
> See here for discussion:
> http://search-hadoop.com/m/4TaT4TpJy71



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6426) Kafka SASL/SCRAM authentication does not fail for incorrect username or password.

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6426.
--
Resolution: Cannot Reproduce

This looks like configuration issue. Please reopen if you think the issue still 
exists. 

> Kafka SASL/SCRAM authentication does not fail for incorrect username or 
> password.
> -
>
> Key: KAFKA-6426
> URL: https://issues.apache.org/jira/browse/KAFKA-6426
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.1
> Environment: Ubuntu 16.04, JDK 1.8, Kafka_2.10-0.10.2.1
>Reporter: Menaka Madushanka
>Priority: Major
> Attachments: broker-jaas.conf, client-jaas.conf, consumer.properties, 
> producer.properties, server.properties
>
>
> Hi,
> I configured Kafka 0.10.2.1 for SASL/SCRAM by following the documentation 
> [1]. 
> But it does work when I use incorrect username or password in the client as 
> well. 
> I have attached the server.properties, consumer.properties, 
> producer.properties, jass config files for broker and client. 
> Also, in my producer, I have set
>  {{props.put("sasl.mechanism", "SCRAM-SHA-256");}}
> but when running, it shows,
> {{kafka.utils.VerifiableProperties  - Property sasl.mechanism is not valid}}
> [1] 
> [https://kafka.apache.org/documentation/#security_sasl_scram|https://kafka.apache.org/documentation/#security_sasl_scram]
> Thanks and Regards,
> Menaka



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6682) Kafka reconnection after broker restart

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6682.
--
Resolution: Duplicate

Resolving this as duplicate of KAFKA-6260.  Please reopen if the issue still 
exists.

> Kafka reconnection after broker restart
> ---
>
> Key: KAFKA-6682
> URL: https://issues.apache.org/jira/browse/KAFKA-6682
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 1.0.0
>Reporter: madi
>Priority: Major
>
> I am using kafka producer plugin for logback (danielwegener) with the clients 
> library 1.0.0 and after restart of broker all my JVMs connected to it get 
> tons of the exceptions:
> {code:java}
> 11:22:48.738 [kafka-producer-network-thread | app-logback-relaxed] cid: 
> clid: E [    @] a: o.a.k.c.p.internals.Sender - [Producer 
> clientId=id-id-logback-relaxed] Uncaught error in kafka producer I/O 
> thread:  ex:java.lang.NullPointerException: null
>     at 
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:436)
>     at org.apache.kafka.common.network.Selector.poll(Selector.java:399)
>     at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460)
>     at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
>     at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
>     at java.lang.Thread.run(Thread.java:798){code}
> During restart there are still other brokers available behind LB.    
> Dosen't matter kafka is up again, only restarting JVM helps
> {code:java}
>      class="com.github.danielwegener.logback.kafka.KafkaAppender">
>     
>     
>    
>  %date{"-MM-dd'T'HH:mm:ss.SSS'Z'"} ${HOSTNAME} 
> [%thread] %logger{32} - %message ex:%exf%n
>     
>     mytopichere
>     
>      class="com.github.danielwegener.logback.kafka.keying.HostNameKeyingStrategy" 
> />
>     
>      class="com.github.danielwegener.logback.kafka.delivery.AsynchronousDeliveryStrategy"
>  />
>     
>    
>  
>     
>     bootstrap.servers=10.99.99.1:9092
>     
>     acks=0
>     
>     block.on.buffer.full=false
>     
>     
> client.id=${HOSTNAME}-${CONTEXT_NAME}-logback-relaxed
>     
>     
>     compression.type=none
>    
>  
>     max.block.ms=0
>     {code}
> I provide loadbalancer address in bootstrap servers here. There are three 
> kafka brokers behind.
> {code:java}
> java version "1.7.0"
> Java(TM) SE Runtime Environment (build pap6470sr9fp60ifix-20161110_01(SR9 
> FP60)+IV90630+IV90578))
> IBM J9 VM (build 2.6, JRE 1.7.0 AIX ppc64-64 Compressed References 
> 20161005_321282 (JIT enabled, AOT enabled)
> J9VM - R26_Java726_SR9_20161005_1259_B321282
> JIT  - tr.r11_20161001_125404
> GC   - R26_Java726_SR9_20161005_1259_B321282_CMPRSS
> J9CL - 20161005_321282)
> JCL - 20161021_01 based on Oracle jdk7u121-b15{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6787) enumerate SampledStat.samples got null item

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474473#comment-16474473
 ] 

Manikumar commented on KAFKA-6787:
--

This might have fixed in KAFKA-6870.

> enumerate SampledStat.samples got null item
> ---
>
> Key: KAFKA-6787
> URL: https://issues.apache.org/jira/browse/KAFKA-6787
> Project: Kafka
>  Issue Type: Bug
>  Components: metrics
>Affects Versions: 1.1.0
>Reporter: Qiang Zhao
>Priority: Critical
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> {code}
> java.lang.NullPointerException at 
> org.apache.kafka.common.metrics.stats.SampledStat.purgeObsoleteSamples(SampledStat.java:104)
>  at 
> org.apache.kafka.common.metrics.stats.SampledStat.measure(SampledStat.java:74)
>  at 
> org.apache.kafka.common.metrics.KafkaMetric.metricValue(KafkaMetric.java:68) 
> at kafka.api.QuotaTestClients$.metricValue(BaseQuotaTest.scala:163) at 
> kafka.api.QuotaTestClients.produceUntilThrottled(BaseQuotaTest.scala:193) at 
> kafka.api.BaseQuotaTest.testThrottledProducerConsumer(BaseQuotaTest.scala:84)
>  {code}
> {code:java}
> protected void purgeObsoleteSamples(MetricConfig config, long now) {
> long expireAge = config.samples() * config.timeWindowMs();
> for (Sample sample : samples) {
> if (now - sample.lastWindowMs >= expireAge)  // line 104, sample 
> is null, probably is a concurrent issue
> sample.reset(now);
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KAFKA-6787) enumerate SampledStat.samples got null item

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar updated KAFKA-6787:
-
Description: 
{code}
java.lang.NullPointerException at 
org.apache.kafka.common.metrics.stats.SampledStat.purgeObsoleteSamples(SampledStat.java:104)
 at 
org.apache.kafka.common.metrics.stats.SampledStat.measure(SampledStat.java:74) 
at org.apache.kafka.common.metrics.KafkaMetric.metricValue(KafkaMetric.java:68) 
at kafka.api.QuotaTestClients$.metricValue(BaseQuotaTest.scala:163) at 
kafka.api.QuotaTestClients.produceUntilThrottled(BaseQuotaTest.scala:193) at 
kafka.api.BaseQuotaTest.testThrottledProducerConsumer(BaseQuotaTest.scala:84)
 {code}

{code:java}
protected void purgeObsoleteSamples(MetricConfig config, long now) {
long expireAge = config.samples() * config.timeWindowMs();
for (Sample sample : samples) {
if (now - sample.lastWindowMs >= expireAge)  // line 104, sample is 
null, probably is a concurrent issue
sample.reset(now);
}
}
{code}

  was:
java.lang.NullPointerException at 
org.apache.kafka.common.metrics.stats.SampledStat.purgeObsoleteSamples(SampledStat.java:104)
 at 
org.apache.kafka.common.metrics.stats.SampledStat.measure(SampledStat.java:74) 
at org.apache.kafka.common.metrics.KafkaMetric.metricValue(KafkaMetric.java:68) 
at kafka.api.QuotaTestClients$.metricValue(BaseQuotaTest.scala:163) at 
kafka.api.QuotaTestClients.produceUntilThrottled(BaseQuotaTest.scala:193) at 
kafka.api.BaseQuotaTest.testThrottledProducerConsumer(BaseQuotaTest.scala:84)

 
{code:java}
protected void purgeObsoleteSamples(MetricConfig config, long now) {
long expireAge = config.samples() * config.timeWindowMs();
for (Sample sample : samples) {
if (now - sample.lastWindowMs >= expireAge)  // line 104, sample is 
null, probably is a concurrent issue
sample.reset(now);
}
}
{code}


> enumerate SampledStat.samples got null item
> ---
>
> Key: KAFKA-6787
> URL: https://issues.apache.org/jira/browse/KAFKA-6787
> Project: Kafka
>  Issue Type: Bug
>  Components: metrics
>Affects Versions: 1.1.0
>Reporter: Qiang Zhao
>Priority: Critical
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> {code}
> java.lang.NullPointerException at 
> org.apache.kafka.common.metrics.stats.SampledStat.purgeObsoleteSamples(SampledStat.java:104)
>  at 
> org.apache.kafka.common.metrics.stats.SampledStat.measure(SampledStat.java:74)
>  at 
> org.apache.kafka.common.metrics.KafkaMetric.metricValue(KafkaMetric.java:68) 
> at kafka.api.QuotaTestClients$.metricValue(BaseQuotaTest.scala:163) at 
> kafka.api.QuotaTestClients.produceUntilThrottled(BaseQuotaTest.scala:193) at 
> kafka.api.BaseQuotaTest.testThrottledProducerConsumer(BaseQuotaTest.scala:84)
>  {code}
> {code:java}
> protected void purgeObsoleteSamples(MetricConfig config, long now) {
> long expireAge = config.samples() * config.timeWindowMs();
> for (Sample sample : samples) {
> if (now - sample.lastWindowMs >= expireAge)  // line 104, sample 
> is null, probably is a concurrent issue
> sample.reset(now);
> }
> }
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-5907) Support aggregatedJavadoc in Java 9

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16474101#comment-16474101
 ] 

Manikumar commented on KAFKA-5907:
--

[~ijuma] I am able to generate Java 9 javadocs on trunk.  

> Support aggregatedJavadoc in Java 9
> ---
>
> Key: KAFKA-5907
> URL: https://issues.apache.org/jira/browse/KAFKA-5907
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>Priority: Major
> Fix For: 2.0.0
>
>
> The Java 9 Javadoc tool has some improvements including a search bar. 
> However, it currently fails with a number of errors like:
> {code}
> > Task :aggregatedJavadoc
> /Users/ijuma/src/kafka/streams/src/main/java/org/apache/kafka/streams/Topology.java:29:
>  error: package org.apache.kafka.streams.processor.internals does not exist
> import org.apache.kafka.streams.processor.internals.InternalTopologyBuilder;
>^
> /Users/ijuma/src/kafka/streams/src/main/java/org/apache/kafka/streams/Topology.java:30:
>  error: package org.apache.kafka.streams.processor.internals does not exist
> import org.apache.kafka.streams.processor.internals.ProcessorNode;
>^
> /Users/ijuma/src/kafka/streams/src/main/java/org/apache/kafka/streams/Topology.java:31:
>  error: package org.apache.kafka.streams.processor.internals does not exist
> import org.apache.kafka.streams.processor.internals.ProcessorTopology;
>^
> /Users/ijuma/src/kafka/streams/src/main/java/org/apache/kafka/streams/Topology.java:32:
>  error: package org.apache.kafka.streams.processor.internals does not exist
> import org.apache.kafka.streams.processor.internals.SinkNode;
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6141) Errors logs when running integration/kafka/tools/MirrorMakerIntegrationTest

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6141.
--
Resolution: Fixed

Closing this as log level changed to debug in ZOOKEEPER-2795 / Zookeeper 3.4.12

> Errors logs when running integration/kafka/tools/MirrorMakerIntegrationTest
> ---
>
> Key: KAFKA-6141
> URL: https://issues.apache.org/jira/browse/KAFKA-6141
> Project: Kafka
>  Issue Type: Improvement
>  Components: unit tests
>Reporter: Pavel
>Priority: Trivial
>
> There are some error logs when running Tests extended from 
> ZooKeeperTestHarness, for example 
> integration/kafka/tools/MirrorMakerIntegrationTest:
> [2017-10-27 18:28:02,557] ERROR ZKShutdownHandler is not registered, so 
> ZooKeeper server won't take any action on ERROR or SHUTDOWN server state 
> changes (org.apache.zookeeper.server.ZooKeeperServer:472)
> [2017-10-27 18:28:09,110] ERROR ZKShutdownHandler is not registered, so 
> ZooKeeper server won't take any action on ERROR or SHUTDOWN server state 
> changes (org.apache.zookeeper.server.ZooKeeperServer:472)
> And these logs have no impact on test results. I think it would be great to 
> eliminate these logs from output by providing a ZKShutdownHandler.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6885) DescribeConfigs synonyms are are identical to parent entry for BROKER resources

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16467896#comment-16467896
 ] 

Manikumar commented on KAFKA-6885:
--

In case of topics,  there is a difference between names of topic override 
configs and server default configs (ex:  "max.message.bytes" is topic override 
config  name for  "message.max.bytes" server property). In this case synonym 
name will be different.

But in case of broker there is no name difference between Dynamic broker 
configs and server default configs (Both places we use "message.max.bytes"  
config name. We can differentiate by using synonyms source property 
(DEFAULT_CONFIG, STATIC_BROKER_CONFIG etc..)



> DescribeConfigs synonyms are are identical to parent entry for BROKER 
> resources
> ---
>
> Key: KAFKA-6885
> URL: https://issues.apache.org/jira/browse/KAFKA-6885
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 1.1.0
>Reporter: Magnus Edenhill
>Priority: Major
>
> The DescribeConfigs protocol response for BROKER resources returns synonyms 
> for various configuration entries, such as "log.dir".
> The list of synonyms returned are identical to their parent configuration 
> entry, rather than the actual synonyms.
> For example, for the broker "log.dir" config entry it returns one synonym, 
> also named "log.dir" rather than "log.dirs" or whatever the synonym is 
> supposed to be.
>  
> This does not seem to happen for TOPIC resources.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6881) Kafka 1.1 Broker version crashes when deleting log

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6881?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16466911#comment-16466911
 ] 

Manikumar commented on KAFKA-6881:
--

By default Kafka is configured to store the data logs to /tmp/ directory.  /tmp 
gets cleared during system reboots/OS cleanups etc..
For production deployments, you will need to change the "log.dirs" property in 
your broker's server.properties file with valid dirs.

> Kafka 1.1 Broker version crashes when deleting log
> --
>
> Key: KAFKA-6881
> URL: https://issues.apache.org/jira/browse/KAFKA-6881
> Project: Kafka
>  Issue Type: Bug
> Environment: Linux
>Reporter: K B Parthasarathy
>Priority: Critical
>
> Hello
> We are running Kafka 1.1 version in Linux from past 3 weeks. Today Kafka 
> crashed. When we checked server.log file the following log was found
> [2018-05-07 16:53:06,721] ERROR Failed to clean up log for 
> __consumer_offsets-24 in dir /tmp/kafka-logs due to IOException 
> (kafka.server.LogDirFailureChannel)
>  java.nio.file.NoSuchFileException: 
> /tmp/kafka-logs/__consumer_offsets-24/.log
>  at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>  at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>  at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
>  at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:409)
>  at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
>  at java.nio.file.Files.move(Files.java:1395)
>  at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:697)
>  at org.apache.kafka.common.record.FileRecords.renameTo(FileRecords.java:212)
>  at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:415)
>  at kafka.log.Log.asyncDeleteSegment(Log.scala:1601)
>  at kafka.log.Log.$anonfun$replaceSegments$1(Log.scala:1653)
>  at kafka.log.Log.$anonfun$replaceSegments$1$adapted(Log.scala:1648)
>  at scala.collection.immutable.List.foreach(List.scala:389)
>  at kafka.log.Log.replaceSegments(Log.scala:1648)
>  at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:535)
>  at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:462)
>  at kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:461)
>  at scala.collection.immutable.List.foreach(List.scala:389)
>  at kafka.log.Cleaner.doClean(LogCleaner.scala:461)
>  at kafka.log.Cleaner.clean(LogCleaner.scala:438)
>  at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:305)
>  at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:291)
>  at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:82)
>  Suppressed: java.nio.file.NoSuchFileException: 
> /tmp/kafka-logs/__consumer_offsets-24/.log -> 
> /tmp/kafka-logs/__consumer_offsets-24/.log.deleted
>  at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
>  at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
>  at sun.nio.fs.UnixCopyFile.move(UnixCopyFile.java:396)
>  at sun.nio.fs.UnixFileSystemProvider.move(UnixFileSystemProvider.java:262)
>  at java.nio.file.Files.move(Files.java:1395)
>  at org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:694)
>  ... 16 more
>  [2018-05-07 16:53:06,725] INFO [ReplicaManager broker=0] Stopping serving 
> replicas in dir /tmp/kafka-logs (kafka.server.ReplicaManager)
>  [2018-05-07 16:53:06,762] INFO Stopping serving logs in dir /tmp/kafka-logs 
> (kafka.log.LogManager)
>  [2018-05-07 16:53:07,032] ERROR Shutdown broker because all log dirs in 
> /tmp/kafka-logs have failed (kafka.log.LogManager)
>  
> Please let me know what may be the issue
>  
> Partha



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-3921) Periodic refresh of metadata causes spurious log messages

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-3921.
--
Resolution: Auto Closed

Closing inactive issue. The old producer is no longer supported. Please upgrade 
to the Java producer whenever possible.


> Periodic refresh of metadata causes spurious log messages
> -
>
> Key: KAFKA-3921
> URL: https://issues.apache.org/jira/browse/KAFKA-3921
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.9.0.1
>Reporter: Steven Schlansker
>Priority: Major
>
> Kafka cluster metadata has a configurable expiry period.  (I don't understand 
> why this is -- cluster updates can happen at any time, and we have to pick 
> those up quicker than every 10 minutes?  But this ticket isn't about that.)
> When this interval expires, the ClientUtils class spins up a SyncProducer, 
> which sends a special message to retrieve metadata.  The producer is then 
> closed immediately after processing this message.
> This causes the SyncProducer to log both a connection open and close at INFO 
> level:
> {code}
> 2016-06-30T17:50:19.408Z INFO <> 
> [ProducerSendThread-central-buzzsaw-1-myhost] kafka.client.ClientUtils$ - 
> Fetching metadata from broker BrokerEndPoint(2,broker-3.mycorp.com,9092) with 
> correlation id 17188 for 1 topic(s) Set(logstash)
> 2016-06-30T17:50:19.410Z INFO <> 
> [ProducerSendThread-central-buzzsaw-1-myhost] kafka.producer.SyncProducer - 
> Connected to broker-3.mycorp.com:9092 for producing
> 2016-06-30T17:50:19.411Z INFO <> 
> [ProducerSendThread-central-buzzsaw-1-myhost] kafka.producer.SyncProducer - 
> Disconnecting from broker-3.mycorp.com:9092
> 2016-06-30T17:50:19.411Z INFO <> 
> [ProducerSendThread-central-buzzsaw-1-myhost] kafka.producer.SyncProducer - 
> Disconnecting from broker-14.mycorp.com:9092
> 2016-06-30T17:50:19.411Z INFO <> 
> [ProducerSendThread-central-buzzsaw-1-myhost] kafka.producer.SyncProducer - 
> Disconnecting from broker-logkafka-13.mycorp.com:9092
> 2016-06-30T17:50:19.411Z INFO <> 
> [ProducerSendThread-central-buzzsaw-1-myhost] kafka.producer.SyncProducer - 
> Disconnecting from broker-12.mycorp.com:9092
> 2016-06-30T17:50:19.413Z INFO <> 
> [ProducerSendThread-central-buzzsaw-1-myhost] kafka.producer.SyncProducer - 
> Connected to broker-12.mycorp.com:9092 for producing
> {code}
> When you are reading the logs, this appears periodically.  We've had more 
> than one administrator then think that the cluster is unhealthy, and client 
> connections are getting dropped -- it's disconnecting from the broker so 
> frequently!  What is wrong???  But in reality, it is just this harmless / 
> expected metadata update.
> Can we tweak the log levels so that the periodic background refresh does not 
> log unless something goes wrong?  The log messages are misleading and easy to 
> misinterpret.  I had to read the code pretty thoroughly to convince myself 
> that these messages are actually harmless.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6762) log-cleaner thread terminates due to java.lang.IllegalStateException

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465471#comment-16465471
 ] 

Manikumar commented on KAFKA-6762:
--

looks like this is related to KAFKA-6834

> log-cleaner thread terminates due to java.lang.IllegalStateException
> 
>
> Key: KAFKA-6762
> URL: https://issues.apache.org/jira/browse/KAFKA-6762
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 1.0.0
> Environment: os: GNU/Linux 
> arch: x86_64 
> Kernel: 4.9.77 
> jvm: OpenJDK 1.8.0
>Reporter: Ricardo Bartolome
>Priority: Major
>
> We are experiencing some problems with kafka log-cleaner thread on Kafka 
> 1.0.0. We have planned to update this cluster to 1.1.0 by next week in order 
> to fix KAFKA-6683, but until then we can only confirm that it happens in 
> 1.0.0.
> log-cleaner thread crashes after a while with the following error:
> {code:java}
> [2018-03-28 11:14:40,199] INFO Cleaner 0: Beginning cleaning of log 
> __consumer_offsets-31. (kafka.log.LogCleaner)
> [2018-03-28 11:14:40,199] INFO Cleaner 0: Building offset map for 
> __consumer_offsets-31... (kafka.log.LogCleaner)
> [2018-03-28 11:14:40,218] INFO Cleaner 0: Building offset map for log 
> __consumer_offsets-31 for 16 segments in offset range [1612869, 14282934). 
> (kafka.log.LogCleaner)
> [2018-03-28 11:14:58,566] INFO Cleaner 0: Offset map for log 
> __consumer_offsets-31 complete. (kafka.log.LogCleaner)
> [2018-03-28 11:14:58,566] INFO Cleaner 0: Cleaning log __consumer_offsets-31 
> (cleaning prior to Tue Mar 27 09:25:09 GMT 2018, discarding tombstones prior 
> to Sat Feb 24 11:04:21 GMT 2018
> )... (kafka.log.LogCleaner)
> [2018-03-28 11:14:58,567] INFO Cleaner 0: Cleaning segment 0 in log 
> __consumer_offsets-31 (largest timestamp Fri Feb 23 11:40:54 GMT 2018) into 
> 0, discarding deletes. (kafka.log.LogClea
> ner)
> [2018-03-28 11:14:58,570] INFO Cleaner 0: Growing cleaner I/O buffers from 
> 262144bytes to 524288 bytes. (kafka.log.LogCleaner)
> [2018-03-28 11:14:58,576] INFO Cleaner 0: Growing cleaner I/O buffers from 
> 524288bytes to 112 bytes. (kafka.log.LogCleaner)
> [2018-03-28 11:14:58,593] ERROR [kafka-log-cleaner-thread-0]: Error due to 
> (kafka.log.LogCleaner)
> java.lang.IllegalStateException: This log contains a message larger than 
> maximum allowable size of 112.
> at kafka.log.Cleaner.growBuffers(LogCleaner.scala:622)
> at kafka.log.Cleaner.cleanInto(LogCleaner.scala:574)
> at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:459)
> at kafka.log.Cleaner.$anonfun$doClean$6(LogCleaner.scala:396)
> at kafka.log.Cleaner.$anonfun$doClean$6$adapted(LogCleaner.scala:395)
> at scala.collection.immutable.List.foreach(List.scala:389)
> at kafka.log.Cleaner.doClean(LogCleaner.scala:395)
> at kafka.log.Cleaner.clean(LogCleaner.scala:372)
> at 
> kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:263)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:243)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:64)
> [2018-03-28 11:14:58,601] INFO [kafka-log-cleaner-thread-0]: Stopped 
> (kafka.log.LogCleaner)
> [2018-04-04 14:25:12,773] INFO The cleaning for partition 
> __broker-11-health-check-0 is aborted and paused (kafka.log.LogCleaner)
> [2018-04-04 14:25:12,773] INFO Compaction for partition 
> __broker-11-health-check-0 is resumed (kafka.log.LogCleaner)
> [2018-04-04 14:25:12,774] INFO The cleaning for partition 
> __broker-11-health-check-0 is aborted (kafka.log.LogCleaner)
> [2018-04-04 14:25:22,850] INFO Shutting down the log cleaner. 
> (kafka.log.LogCleaner)
> [2018-04-04 14:25:22,850] INFO [kafka-log-cleaner-thread-0]: Shutting down 
> (kafka.log.LogCleaner)
> [2018-04-04 14:25:22,850] INFO [kafka-log-cleaner-thread-0]: Shutdown 
> completed (kafka.log.LogCleaner)
> {code}
> What we know so far is:
>  * We are unable to reproduce it yet in a consistent manner.
>  * It only happens in the PRO cluster and not in the PRE cluster for the same 
> customer (which message payloads are very similar)
>  * Checking our Kafka logs, it only happened on the internal topics 
> *__consumer_offsets-**
>  * When we restart the broker process the log-cleaner starts working again 
> but it can take between 3 minutes and some hours to die again.
>  * We workaround it by temporary increasing the message.max.bytes and 
> replica.fetch.max.bytes values to 10485760 (10MB) from default 112 (~1MB).
> ** Before message.max.bytes = 10MB, we tried to match message.max.size with 
> the value of replica.fetch.max.size (1048576), but log-cleaned died with the 
> same error but different limit.
>  ** This allowed the log-cleaner not to die and compact 

[jira] [Assigned] (KAFKA-6394) Prevent misconfiguration of advertised listeners

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar reassigned KAFKA-6394:


Assignee: Manikumar

> Prevent misconfiguration of advertised listeners
> 
>
> Key: KAFKA-6394
> URL: https://issues.apache.org/jira/browse/KAFKA-6394
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Manikumar
>Priority: Major
>
> We don't really have any protection from misconfiguration of the advertised 
> listeners. Sometimes users will copy the config from one host to another 
> during an upgrade. They may remember to update the broker id, but forget 
> about the advertised listeners. It can be surprisingly difficult to detect 
> this unless you know to look for it (e.g. you might just see a lot of 
> NotLeaderForPartition errors as the fetchers connect to the wrong broker). It 
> may not be totally foolproof, but it's probably enough for the common 
> misconfiguration case to check existing brokers to see whether there are any 
> which have already registered the advertised listener.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6837) Apache kafka broker got stopped.

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6837.
--
Resolution: Information Provided

> Apache kafka broker got stopped.
> 
>
> Key: KAFKA-6837
> URL: https://issues.apache.org/jira/browse/KAFKA-6837
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.11.0.2
>Reporter: Rajendra Jangir
>Priority: Major
>
> We are using kafka version 2.11-0.11.0.2 and zookeeper version 3.3.6. And 
> they are they are running on Ubuntu 16.0.4.
> We are producing data with rate of 3k messages per second and the size of 
> each message is around 150 Bytes. we have 7 brokers and 1 zookeeper, and 
> sometimes,  some  brokers get down.
> So how we can find out the reason that why a particular broker was got down? 
> And where the broker stop logs are stored from where we can check the logs 
> and can find why it was stopped?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6837) Apache kafka broker got stopped.

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16465460#comment-16465460
 ] 

Manikumar commented on KAFKA-6837:
--

As discussed offline, I suggest you to post these kind of queries to 
[us...@kafka.apache.org|mailto:us...@kafka.apache.org] mailing list 
([http://kafka.apache.org/contact]).
 

> Apache kafka broker got stopped.
> 
>
> Key: KAFKA-6837
> URL: https://issues.apache.org/jira/browse/KAFKA-6837
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.11.0.2
>Reporter: Rajendra Jangir
>Priority: Major
>
> We are using kafka version 2.11-0.11.0.2 and zookeeper version 3.3.6. And 
> they are they are running on Ubuntu 16.0.4.
> We are producing data with rate of 3k messages per second and the size of 
> each message is around 150 Bytes. we have 7 brokers and 1 zookeeper, and 
> sometimes,  some  brokers get down.
> So how we can find out the reason that why a particular broker was got down? 
> And where the broker stop logs are stored from where we can check the logs 
> and can find why it was stopped?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6863) Kafka clients should try to use multiple DNS resolved IP addresses if the first one fails

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16464080#comment-16464080
 ] 

Manikumar commented on KAFKA-6863:
--

Is this not related to  KIP-235 / KAFKA-6195?

https://cwiki.apache.org/confluence/display/KAFKA/KIP-235%3A+Add+DNS+alias+support+for+secured+connection

> Kafka clients should try to use multiple DNS resolved IP addresses if the 
> first one fails
> -
>
> Key: KAFKA-6863
> URL: https://issues.apache.org/jira/browse/KAFKA-6863
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 1.0.0, 1.1.0
>Reporter: Edoardo Comar
>Assignee: Edoardo Comar
>Priority: Major
> Fix For: 2.0.0
>
>
> Currently Kafka clients resolve a symbolic hostname using
>   {{new InetSocketAddress(String hostname, int port)}}
> which only picks one IP address even if the DNS has multiple records for the 
> hostname, as it calls
>  {{InetAddress.getAllByName(host)[0]}}
> For some environments where the hostnames are mapped by the DNS to multiple 
> IPs, e.g. in clouds where the IPs point to the external load balancers, it 
> would be preferable that the client, on failing to connect to one of the IPs, 
> would try the other ones before giving up the connection.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-3417) Invalid characters in config properties not being validated?

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-3417.
--
   Resolution: Fixed
 Assignee: Mickael Maison  (was: Grant Henke)
Fix Version/s: 2.0.0

> Invalid characters in config properties not being validated?
> 
>
> Key: KAFKA-3417
> URL: https://issues.apache.org/jira/browse/KAFKA-3417
> Project: Kafka
>  Issue Type: Bug
>  Components: config
>Affects Versions: 0.9.0.1
>Reporter: Byron Ruth
>Assignee: Mickael Maison
>Priority: Minor
> Fix For: 2.0.0
>
>
> I ran into an error using a {{client.id}} with invalid characters (per the 
> [config 
> validator|https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/common/Config.scala#L25-L35]).
>  I was able to get that exact error using the {{kafka-console-consumer}} 
> script, presumably because I supplied a consumer properties file and it 
> validated prior to hitting the server. However, when I use a client library 
> (sarama for Go in this case), an error in the metrics subsystem is thrown 
> [here|https://github.com/apache/kafka/blob/977ebbe9bafb6c1a6e1be69620f745712118fe80/clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java#L380].
> The stacktrace is:
> {code:title=stack.java}
> [2016-03-17 17:43:47,342] ERROR [KafkaApi-0] error when handling request 
> Name: FetchRequest; Version: 0; CorrelationId: 2; ClientId: foo:bar; 
> ReplicaId: -1; MaxWait: 250 ms; MinBytes: 1 bytes; RequestInfo: [foo,0] -> 
> PartitionFetchInfo(0,32768) (kafka.server.KafkaApis)
> org.apache.kafka.common.KafkaException: Error creating mbean attribute for 
> metricName :MetricName [name=throttle-time, group=Fetch, description=Tracking 
> average throttle-time per client, tags={client-id=foo:bar}]
>   at 
> org.apache.kafka.common.metrics.JmxReporter.addAttribute(JmxReporter.java:113)
>   at 
> org.apache.kafka.common.metrics.JmxReporter.metricChange(JmxReporter.java:76)
>   at 
> org.apache.kafka.common.metrics.Metrics.registerMetric(Metrics.java:288)
>   at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:177)
>   at org.apache.kafka.common.metrics.Sensor.add(Sensor.java:162)
> ...
> {code}
> Assuming the cause os related to the invalid characters, when the request 
> header is decoded, the {{clientId}} should be validated prior to being used?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-3143) inconsistent state in ZK when all replicas are dead

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459817#comment-16459817
 ] 

Manikumar commented on KAFKA-3143:
--

>From 1.1.0 (KAFKA-5083),  last ISR is preserved in ZK, irrespective of unclean 
>leader election is enabled or not  .

> inconsistent state in ZK when all replicas are dead
> ---
>
> Key: KAFKA-3143
> URL: https://issues.apache.org/jira/browse/KAFKA-3143
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jun Rao
>Assignee: Ismael Juma
>Priority: Major
>  Labels: reliability
> Fix For: 2.0.0
>
>
> This issue can be recreated in the following steps.
> 1. Start 3 brokers, 1, 2 and 3.
> 2. Create a topic with a single partition and 2 replicas, say on broker 1 and 
> 2.
> If we stop both replicas 1 and 2, depending on where the controller is, the 
> leader and isr stored in ZK in the end are different.
> If the controller is on broker 3, what's stored in ZK will be -1 for leader 
> and an empty set for ISR.
> On the other hand, if the controller is on broker 2 and we stop broker 1 
> followed by broker 2, what's stored in ZK will be 2 for leader and 2 for ISR.
> The issue is that in the first case, the controller will call 
> ReplicaStateMachine to transition to OfflineReplica, which will change the 
> leader and isr. However, in the second case, the controller fails over, but 
> we don't transition ReplicaStateMachine to OfflineReplica during controller 
> initialization.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-4701) Allow kafka brokers to dynamically reload truststore without restarting.

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459756#comment-16459756
 ] 

Manikumar commented on KAFKA-4701:
--

Can this be resolved as duplicate of KAFKA-6810?

> Allow kafka brokers to dynamically reload truststore without restarting.
> 
>
> Key: KAFKA-4701
> URL: https://issues.apache.org/jira/browse/KAFKA-4701
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Reporter: Allen Xiang
>Priority: Major
>  Labels: security
> Fix For: 2.0.0
>
>
> Right now in order to add SSL clients(update broker truststores), a rolling 
> restart of all brokers is required. This is very time consuming and 
> unnecessary. A dynamic truststore manager is needed to reload truststore from 
> file system without restarting brokers.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (KAFKA-6835) Enable topic unclean leader election to be enabled without controller change

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar reassigned KAFKA-6835:


Assignee: Manikumar

> Enable topic unclean leader election to be enabled without controller change
> 
>
> Key: KAFKA-6835
> URL: https://issues.apache.org/jira/browse/KAFKA-6835
> Project: Kafka
>  Issue Type: Task
>  Components: core
>Reporter: Rajini Sivaram
>Assignee: Manikumar
>Priority: Major
>
> Dynamic update of broker's default unclean.leader.election.enable will be 
> processed without controller change (KAFKA-6526). We should probably do the 
> same for topic overrides as well.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6798) Kafka leader rebalance failures

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442861#comment-16442861
 ] 

Manikumar commented on KAFKA-6798:
--

Yes, rebalances are due to zookeeper disconnects. Since it is happening on all 
brokers at the same, this may due to some network configuration/timeouts in 
your environment.  Can you check zookeeper logs for issues?

> Kafka leader rebalance failures
> ---
>
> Key: KAFKA-6798
> URL: https://issues.apache.org/jira/browse/KAFKA-6798
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.1, 1.0.1
>Reporter: Riley Zimmerman
>Priority: Critical
>
> I am running 3 Kafka (version 0.10.2.1 and more recently moved to 1.0.1) with 
> 3 Zookeeper (v3.4.9) as statefulsets in a kubernetes v1.9.1 deployment.  My 
> partitions are replication factor 3.  My main workload involves a kafka 
> streams consumer/producer (storing offsets in kafka) and a second kafka 
> consumer storing offsets in zookeeper (only commits every 30 seconds).  There 
> are ~200,000 kafka messages going through each per minute.  The log.retention 
> settings are all 4 hours.  I have auto.leader.rebalance.enabled.  
> I am randomly having failures during the rebalances.  The result is that 
> partitions for both topics and consumer_offsets go out of sync and the 
> partition leader becomes -1.  After 4 hours there is another (auto?) 
> rebalance and sometimes it sorts itself out.  Sometimes it runs for weeks 
> without problems, other times it it happens multiple times in a few days.  It 
> appears to happen earlier in test runs if it is going to happen.   
> {noformat}
> Topic:__consumer_offsetsPartitionCount:50   ReplicationFactor:3   
>   
> Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
> Topic: __consumer_offsets   Partition: 0Leader: -1  
> Replicas: 2,0,1 Isr:
> Topic: __consumer_offsets   Partition: 1Leader: 0   
> Replicas: 0,1,2 Isr: 1,2,0
> Topic: __consumer_offsets   Partition: 2Leader: 1   
> Replicas: 1,2,0 Isr: 2,1,0
> Topic: __consumer_offsets   Partition: 3Leader: -1  
> Replicas: 2,1,0 Isr:
> {noformat}
> {noformat}
> [2018-03-20 12:42:32,180] WARN [Controller 2]: Partition [agent.metadata,5] 
> failed to complete preferred replica leader election. Leader is -1 
> (kafka.controller.KafkaController)
> {noformat}
> {noformat}
> [2018-03-20 11:02:32,099] TRACE Controller 2 epoch 27 started leader election 
> for partition [__consumer_offsets,30] (state.change.logger)
> [2018-03-20 11:02:32,101] ERROR Controller 2 epoch 27 encountered error while 
> electing leader for partition [__consumer_offsets,30] due to: Preferred 
> replica 2 for partition [__consumer_offsets,30] is either not alive or not in 
> the isr. Current leader and ISR: [{"leader":-1,"leader_epoch":59,"isr":[]}]. 
> (state.change.logger)
> [2018-03-20 11:02:32,101] ERROR Controller 2 epoch 27 initiated state change 
> for partition [__consumer_offsets,30] from OnlinePartition to OnlinePartition 
> failed (state.change.logger)
> kafka.common.StateChangeFailedException: encountered error while electing 
> leader for partition [__consumer_offsets,30] due to: Preferred replica 2 for 
> partition [__consumer_offsets,30] is either not alive or not in the isr. 
> Current leader and ISR: [{"leader":-1,"leader_epoch":59,"isr":[]}].
>   at 
> kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:362)
>   at 
> kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:202)
>   at 
> kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:141)
>   at 
> kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:140)
>   at scala.collection.immutable.Set$Set1.foreach(Set.scala:94)
>   at 
> kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:140)
>   at 
> kafka.controller.KafkaController.onPreferredReplicaElection(KafkaController.scala:662)
>   at 
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$16$$anonfun$apply$5.apply$mcV$sp(KafkaController.scala:1230)
>   at 
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$16$$anonfun$apply$5.apply(KafkaController.scala:1225)
>   at 
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$16$$anonfun$apply$5.apply(KafkaController.scala:1225)
>   at 

[jira] [Commented] (KAFKA-6798) Kafka leader rebalance failures

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442656#comment-16442656
 ] 

Manikumar commented on KAFKA-6798:
--

It does look like zk session timeout issue. Not sure why happening for every 
four hours. Are you observing same pattern on all broker nodes?  may be  
Network work or GC issue..  We can check GC logs. Can you try increasing zk 
session timeout and check?

> Kafka leader rebalance failures
> ---
>
> Key: KAFKA-6798
> URL: https://issues.apache.org/jira/browse/KAFKA-6798
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.1, 1.0.1
>Reporter: Riley Zimmerman
>Priority: Critical
>
> I am running 3 Kafka (version 0.10.2.1 and more recently moved to 1.0.1) with 
> 3 Zookeeper (v3.4.9) as statefulsets in a kubernetes v1.9.1 deployment.  My 
> partitions are replication factor 3.  My main workload involves a kafka 
> streams consumer/producer (storing offsets in kafka) and a second kafka 
> consumer storing offsets in zookeeper (only commits every 30 seconds).  There 
> are ~200,000 kafka messages going through each per minute.  The log.retention 
> settings are all 4 hours.  I have auto.leader.rebalance.enabled.  
> I am randomly having failures during the rebalances.  The result is that 
> partitions for both topics and consumer_offsets go out of sync and the 
> partition leader becomes -1.  After 4 hours there is another (auto?) 
> rebalance and sometimes it sorts itself out.  Sometimes it runs for weeks 
> without problems, other times it it happens multiple times in a few days.  It 
> appears to happen earlier in test runs if it is going to happen.   
> {noformat}
> Topic:__consumer_offsetsPartitionCount:50   ReplicationFactor:3   
>   
> Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
> Topic: __consumer_offsets   Partition: 0Leader: -1  
> Replicas: 2,0,1 Isr:
> Topic: __consumer_offsets   Partition: 1Leader: 0   
> Replicas: 0,1,2 Isr: 1,2,0
> Topic: __consumer_offsets   Partition: 2Leader: 1   
> Replicas: 1,2,0 Isr: 2,1,0
> Topic: __consumer_offsets   Partition: 3Leader: -1  
> Replicas: 2,1,0 Isr:
> {noformat}
> {noformat}
> [2018-03-20 12:42:32,180] WARN [Controller 2]: Partition [agent.metadata,5] 
> failed to complete preferred replica leader election. Leader is -1 
> (kafka.controller.KafkaController)
> {noformat}
> {noformat}
> [2018-03-20 11:02:32,099] TRACE Controller 2 epoch 27 started leader election 
> for partition [__consumer_offsets,30] (state.change.logger)
> [2018-03-20 11:02:32,101] ERROR Controller 2 epoch 27 encountered error while 
> electing leader for partition [__consumer_offsets,30] due to: Preferred 
> replica 2 for partition [__consumer_offsets,30] is either not alive or not in 
> the isr. Current leader and ISR: [{"leader":-1,"leader_epoch":59,"isr":[]}]. 
> (state.change.logger)
> [2018-03-20 11:02:32,101] ERROR Controller 2 epoch 27 initiated state change 
> for partition [__consumer_offsets,30] from OnlinePartition to OnlinePartition 
> failed (state.change.logger)
> kafka.common.StateChangeFailedException: encountered error while electing 
> leader for partition [__consumer_offsets,30] due to: Preferred replica 2 for 
> partition [__consumer_offsets,30] is either not alive or not in the isr. 
> Current leader and ISR: [{"leader":-1,"leader_epoch":59,"isr":[]}].
>   at 
> kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:362)
>   at 
> kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:202)
>   at 
> kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:141)
>   at 
> kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:140)
>   at scala.collection.immutable.Set$Set1.foreach(Set.scala:94)
>   at 
> kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:140)
>   at 
> kafka.controller.KafkaController.onPreferredReplicaElection(KafkaController.scala:662)
>   at 
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$16$$anonfun$apply$5.apply$mcV$sp(KafkaController.scala:1230)
>   at 
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$16$$anonfun$apply$5.apply(KafkaController.scala:1225)
>   at 
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$16$$anonfun$apply$5.apply(KafkaController.scala:1225)
>   

[jira] [Commented] (KAFKA-6335) SimpleAclAuthorizerTest#testHighConcurrencyModificationOfResourceAcls fails intermittently

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6335?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442490#comment-16442490
 ] 

Manikumar commented on KAFKA-6335:
--

[~Sonia] Can you give more details? Are you seeing test failure or an issue 
with SimpleAclAuthorizer? Can you also confirm the Kafka version? 

> SimpleAclAuthorizerTest#testHighConcurrencyModificationOfResourceAcls fails 
> intermittently
> --
>
> Key: KAFKA-6335
> URL: https://issues.apache.org/jira/browse/KAFKA-6335
> Project: Kafka
>  Issue Type: Test
>Reporter: Ted Yu
>Assignee: Manikumar
>Priority: Major
> Fix For: 1.2.0
>
>
> From 
> https://builds.apache.org/job/kafka-pr-jdk9-scala2.12/3045/testReport/junit/kafka.security.auth/SimpleAclAuthorizerTest/testHighConcurrencyModificationOfResourceAcls/
>  :
> {code}
> java.lang.AssertionError: expected acls Set(User:36 has Allow permission for 
> operations: Read from hosts: *, User:7 has Allow permission for operations: 
> Read from hosts: *, User:21 has Allow permission for operations: Read from 
> hosts: *, User:39 has Allow permission for operations: Read from hosts: *, 
> User:43 has Allow permission for operations: Read from hosts: *, User:3 has 
> Allow permission for operations: Read from hosts: *, User:35 has Allow 
> permission for operations: Read from hosts: *, User:15 has Allow permission 
> for operations: Read from hosts: *, User:16 has Allow permission for 
> operations: Read from hosts: *, User:22 has Allow permission for operations: 
> Read from hosts: *, User:26 has Allow permission for operations: Read from 
> hosts: *, User:11 has Allow permission for operations: Read from hosts: *, 
> User:38 has Allow permission for operations: Read from hosts: *, User:8 has 
> Allow permission for operations: Read from hosts: *, User:28 has Allow 
> permission for operations: Read from hosts: *, User:32 has Allow permission 
> for operations: Read from hosts: *, User:25 has Allow permission for 
> operations: Read from hosts: *, User:41 has Allow permission for operations: 
> Read from hosts: *, User:44 has Allow permission for operations: Read from 
> hosts: *, User:48 has Allow permission for operations: Read from hosts: *, 
> User:2 has Allow permission for operations: Read from hosts: *, User:9 has 
> Allow permission for operations: Read from hosts: *, User:14 has Allow 
> permission for operations: Read from hosts: *, User:46 has Allow permission 
> for operations: Read from hosts: *, User:13 has Allow permission for 
> operations: Read from hosts: *, User:5 has Allow permission for operations: 
> Read from hosts: *, User:29 has Allow permission for operations: Read from 
> hosts: *, User:45 has Allow permission for operations: Read from hosts: *, 
> User:6 has Allow permission for operations: Read from hosts: *, User:37 has 
> Allow permission for operations: Read from hosts: *, User:23 has Allow 
> permission for operations: Read from hosts: *, User:19 has Allow permission 
> for operations: Read from hosts: *, User:24 has Allow permission for 
> operations: Read from hosts: *, User:17 has Allow permission for operations: 
> Read from hosts: *, User:34 has Allow permission for operations: Read from 
> hosts: *, User:12 has Allow permission for operations: Read from hosts: *, 
> User:42 has Allow permission for operations: Read from hosts: *, User:4 has 
> Allow permission for operations: Read from hosts: *, User:47 has Allow 
> permission for operations: Read from hosts: *, User:18 has Allow permission 
> for operations: Read from hosts: *, User:31 has Allow permission for 
> operations: Read from hosts: *, User:49 has Allow permission for operations: 
> Read from hosts: *, User:33 has Allow permission for operations: Read from 
> hosts: *, User:1 has Allow permission for operations: Read from hosts: *, 
> User:27 has Allow permission for operations: Read from hosts: *) but got 
> Set(User:36 has Allow permission for operations: Read from hosts: *, User:7 
> has Allow permission for operations: Read from hosts: *, User:21 has Allow 
> permission for operations: Read from hosts: *, User:39 has Allow permission 
> for operations: Read from hosts: *, User:43 has Allow permission for 
> operations: Read from hosts: *, User:3 has Allow permission for operations: 
> Read from hosts: *, User:35 has Allow permission for operations: Read from 
> hosts: *, User:15 has Allow permission for operations: Read from hosts: *, 
> User:16 has Allow permission for operations: Read from hosts: *, User:22 has 
> Allow permission for operations: Read from hosts: *, User:26 has Allow 
> permission for operations: Read from hosts: *, User:11 has Allow permission 
> for operations: Read from hosts: *, User:38 has Allow permission for 
> operations: Read 

[jira] [Commented] (KAFKA-5813) Unexpected unclean leader election due to leader/controller's unusual event handling order

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16442343#comment-16442343
 ] 

Manikumar commented on KAFKA-5813:
--

This might have fixed in async zk controller changes.

> Unexpected unclean leader election due to leader/controller's unusual event 
> handling order 
> ---
>
> Key: KAFKA-5813
> URL: https://issues.apache.org/jira/browse/KAFKA-5813
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.10.2.1
>Reporter: Allen Wang
>Priority: Minor
>
> We experienced an unexpected unclean leader election after network glitch 
> happened on the leader of partition. We have replication factor 2.
> Here is the sequence of event gathered from various logs:
> 1. ZK session timeout happens for leader of partition 
> 2. New ZK session is established for leader 
> 3. Leader removes the follower from ISR (which might be caused by replication 
> delay due to the network problem) and updates the ISR in ZK 
> 4. Controller processes the BrokerChangeListener event happened at step 1 
> where the leader seems to be offline 
> 5. Because the ISR in ZK is already updated by leader to remove the follower, 
> controller makes an unclean leader election 
> 6. Controller processes the second BrokerChangeListener event happened at 
> step 2 to mark the broker online again
> It seems to me that step 4 happens too late. If it happens right after step 
> 1, it will be a clean leader election and hopefully the producer will 
> immediately switch to the new leader, thus avoiding consumer offset reset. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-1716) hang during shutdown of ZookeeperConsumerConnector

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1716.
--
Resolution: Auto Closed

Closing inactive issue. The Scala consumers have been deprecated and no further 
work is planned, please upgrade to the Java consumer whenever possible.

> hang during shutdown of ZookeeperConsumerConnector
> --
>
> Key: KAFKA-1716
> URL: https://issues.apache.org/jira/browse/KAFKA-1716
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.1.1
>Reporter: Sean Fay
>Assignee: Neha Narkhede
>Priority: Major
> Attachments: after-shutdown.log, before-shutdown.log, 
> kafka-shutdown-stuck.log
>
>
> It appears to be possible for {{ZookeeperConsumerConnector.shutdown()}} to 
> wedge in the case that some consumer fetcher threads receive messages during 
> the shutdown process.
> Shutdown thread:
> {code}-- Parking to wait for: 
> java/util/concurrent/CountDownLatch$Sync@0x2aaaf3ef06d0
> at jrockit/vm/Locks.park0(J)V(Native Method)
> at jrockit/vm/Locks.park(Locks.java:2230)
> at sun/misc/Unsafe.park(ZJ)V(Native Method)
> at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
> at 
> java/util/concurrent/locks/AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
> at 
> java/util/concurrent/locks/AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
> at 
> java/util/concurrent/locks/AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
> at java/util/concurrent/CountDownLatch.await(CountDownLatch.java:207)
> at kafka/utils/ShutdownableThread.shutdown(ShutdownableThread.scala:36)
> at 
> kafka/server/AbstractFetcherThread.shutdown(AbstractFetcherThread.scala:71)
> at 
> kafka/server/AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:121)
> at 
> kafka/server/AbstractFetcherManager$$anonfun$closeAllFetchers$2.apply(AbstractFetcherManager.scala:120)
> at 
> scala/collection/TraversableLike$WithFilter$$anonfun$foreach$1.apply(TraversableLike.scala:772)
> at 
> scala/collection/mutable/HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala/collection/mutable/HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala/collection/mutable/HashTable$class.foreachEntry(HashTable.scala:226)
> at scala/collection/mutable/HashMap.foreachEntry(HashMap.scala:39)
> at scala/collection/mutable/HashMap.foreach(HashMap.scala:98)
> at 
> scala/collection/TraversableLike$WithFilter.foreach(TraversableLike.scala:771)
> at 
> kafka/server/AbstractFetcherManager.closeAllFetchers(AbstractFetcherManager.scala:120)
> ^-- Holding lock: java/lang/Object@0x2aaaebcc7318[thin lock]
> at 
> kafka/consumer/ConsumerFetcherManager.stopConnections(ConsumerFetcherManager.scala:148)
> at 
> kafka/consumer/ZookeeperConsumerConnector.liftedTree1$1(ZookeeperConsumerConnector.scala:171)
> at 
> kafka/consumer/ZookeeperConsumerConnector.shutdown(ZookeeperConsumerConnector.scala:167){code}
> ConsumerFetcherThread:
> {code}-- Parking to wait for: 
> java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject@0x2aaaebcc7568
> at jrockit/vm/Locks.park0(J)V(Native Method)
> at jrockit/vm/Locks.park(Locks.java:2230)
> at sun/misc/Unsafe.park(ZJ)V(Native Method)
> at java/util/concurrent/locks/LockSupport.park(LockSupport.java:156)
> at 
> java/util/concurrent/locks/AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
> at 
> java/util/concurrent/LinkedBlockingQueue.put(LinkedBlockingQueue.java:306)
> at kafka/consumer/PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:60)
> at 
> kafka/consumer/ConsumerFetcherThread.processPartitionData(ConsumerFetcherThread.scala:49)
> at 
> kafka/server/AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:130)
> at 
> kafka/server/AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfun$apply$mcV$sp$2.apply(AbstractFetcherThread.scala:111)
> at scala/collection/immutable/HashMap$HashMap1.foreach(HashMap.scala:224)
> at 
> scala/collection/immutable/HashMap$HashTrieMap.foreach(HashMap.scala:403)
> at 
> kafka/server/AbstractFetcherThread$$anonfun$processFetchRequest$1.apply$mcV$sp(AbstractFetcherThread.scala:111)
> at 
> kafka/server/AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:111)
> at 
> kafka/server/AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(AbstractFetcherThread.scala:111)
> at kafka/utils/Utils$.inLock(Utils.scala:538)
> at 

[jira] [Resolved] (KAFKA-5287) Messages getting repeated in kafka

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-5287.
--
Resolution: Cannot Reproduce

Closing inactive issue. Please reopen with more details., if you think the 
issue still exists 

> Messages getting repeated in kafka
> --
>
> Key: KAFKA-5287
> URL: https://issues.apache.org/jira/browse/KAFKA-5287
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.10.2.0
> Environment: Hardware specification(8 Cores , 16 GB RAM,1 TB Harddisk)
>Reporter: Abhimanyu Nagrath
>Priority: Major
>
> I have a topic with 200 partition in which messages contains the total of 3 
> Million messages. It took 5 days to completely process all the messages and 
> as soon as message got processed i.e. Kafka-consumer-groups.sh showed 0 lag 
> in all the partition of the topic I stopped the consumer .but after 6 hrs 
> again it was showing the lag of 2 million message which I found that were 
> duplicate messages. This thing is happening very frequently. My offsets are 
> stored on Kafka broker itself. 
> My server configuration is:
> broker.id=1
> delete.topic.enable=true
> #listeners=PLAINTEXT://:9092
> #advertised.listeners=PLAINTEXT://your.host.name:9092
> num.network.threads=3
> num.io.threads=8
> socket.send.buffer.bytes=102400
> socket.receive.buffer.bytes=102400
> socket.request.max.bytes=104857600
> log.dirs=/kafka/data/logs
> num.partitions=1
> num.recovery.threads.per.data.dir=5
> log.flush.interval.messages=1
> #log.flush.interval.ms=1000
> log.retention.hours=480
> log.retention.bytes=1073741824
> log.segment.bytes=1073741824
> log.retention.check.interval.ms=30
> zookeeper.connect=:2181
> zookeeper.connection.timeout.ms=6000
> Is there in the configuration that I am missing? 
> Any help is appreciated 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-5262) Can't find some consumer group information

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-5262.
--
Resolution: Cannot Reproduce

Closing inactive issue. Please reopen with more details., if you think the 
issue still exists 


> Can't  find  some  consumer group   information
> ---
>
> Key: KAFKA-5262
> URL: https://issues.apache.org/jira/browse/KAFKA-5262
> Project: Kafka
>  Issue Type: Bug
>  Components: offset manager
>Affects Versions: 0.10.1.0
>Reporter: miaozhiyong
>Priority: Major
>
> The  kafka client use  broker to connect with kafka ,  i had install  two 
> kafka-manager.  the consumer don't display in the kafka-manager .and   can''t 
>  work with   the commmand line:
> kafka-consumer-groups.sh --new-consumer  --bootstrap-serveer
> but the client is ok . where is consumer store the lag?



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-3476) -Xloggc is not recognised by IBM java

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-3476.
--
Resolution: Won't Fix

Closing this as we can export the GC values and performance opts.  Please 
reopen if you think otherwise

>  -Xloggc is not recognised by IBM java
> --
>
> Key: KAFKA-3476
> URL: https://issues.apache.org/jira/browse/KAFKA-3476
> Project: Kafka
>  Issue Type: Bug
>  Components: admin, tools
>Affects Versions: 0.9.0.0
>Reporter: Khirod Patra
>Priority: Major
>
> Getting below error on AIX server.
> NOTE : java version is :
> --
> java version "1.8.0"
> Java(TM) SE Runtime Environment (build pap6480-20150129_02)
> IBM J9 VM (build 2.8, JRE 1.8.0 AIX ppc64-64 Compressed References 
> 20150116_231420 (JIT enabled, AOT enabled)
> J9VM - R28_Java8_GA_20150116_2030_B231420
> JIT  - tr.r14.java_20150109_82886.02
> GC   - R28_Java8_GA_20150116_2030_B231420_CMPRSS
> J9CL - 20150116_231420)
> JCL - 20150123_01 based on Oracle jdk8u31-b12
> Error :
> ---
> kafka-run-class.sh -name zookeeper -loggc  
> org.apache.zookeeper.server.quorum.QuorumPeerMain 
> ../config/zookeeper.properties
> 
> http://www.ibm.com/j9/verbosegc; 
> version="R28_Java8_GA_20150116_2030_B231420_CMPRSS">
> JVMJ9VM007E Command-line option unrecognised: 
> -Xloggc:/home/test_user/containers/kafka_2.11-0.9.0.0/bin/../logs/zookeeper-gc.log
> 
> Error: Could not create the Java Virtual Machine.
> Error: A fatal exception has occurred. Program will exit.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6107) SCRAM user add appears to fail if Kafka has never been started

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6107.
--
   Resolution: Fixed
Fix Version/s: 1.1.0

This was fixed in KafkaZkClient changes work.

> SCRAM user add appears to fail if Kafka has never been started
> --
>
> Key: KAFKA-6107
> URL: https://issues.apache.org/jira/browse/KAFKA-6107
> Project: Kafka
>  Issue Type: Bug
>  Components: tools, zkclient
>Affects Versions: 0.11.0.0
>Reporter: Dustin Cote
>Priority: Minor
> Fix For: 1.1.0
>
>
> When trying to add a SCRAM user in ZooKeeper without having ever starting 
> Kafka, the kafka-configs tool does not handle it well. This is a common use 
> case because starting a new cluster where you want SCRAM for inter broker 
> communication would generally result in seeing this problem. Today, the 
> workaround is to start Kafka, add the user, then restart Kafka. Here's how to 
> reproduce:
> 1) Start ZooKeeper
> 2) Run 
> {code}
> bin/kafka-configs --zookeeper localhost:2181 --alter --add-config 
> 'SCRAM-SHA-256=[iterations=8192,password=broker_pwd],SCRAM-SHA-512=[password=broker_pwd]'
>  --entity-type users --entity-name broker
> {code}
> This will result in:
> {code}
> bin/kafka-configs --zookeeper localhost:2181 --alter --add-config 
> 'SCRAM-SHA-256=[iterations=8192,password=broker_pwd],SCRAM-SHA-512=[password=broker_pwd]'
>  --entity-type users --entity-name broker
> Error while executing config command 
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = 
> NoNode for /config/changes/config_change_
> org.I0Itec.zkclient.exception.ZkNoNodeException: 
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = 
> NoNode for /config/changes/config_change_
>   at org.I0Itec.zkclient.exception.ZkException.create(ZkException.java:47)
>   at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:1001)
>   at org.I0Itec.zkclient.ZkClient.create(ZkClient.java:528)
>   at 
> org.I0Itec.zkclient.ZkClient.createPersistentSequential(ZkClient.java:444)
>   at kafka.utils.ZkPath.createPersistentSequential(ZkUtils.scala:1045)
>   at kafka.utils.ZkUtils.createSequentialPersistentPath(ZkUtils.scala:527)
>   at 
> kafka.admin.AdminUtils$.kafka$admin$AdminUtils$$changeEntityConfig(AdminUtils.scala:600)
>   at 
> kafka.admin.AdminUtils$.changeUserOrUserClientIdConfig(AdminUtils.scala:551)
>   at kafka.admin.AdminUtilities$class.changeConfigs(AdminUtils.scala:63)
>   at kafka.admin.AdminUtils$.changeConfigs(AdminUtils.scala:72)
>   at kafka.admin.ConfigCommand$.alterConfig(ConfigCommand.scala:101)
>   at kafka.admin.ConfigCommand$.main(ConfigCommand.scala:68)
>   at kafka.admin.ConfigCommand.main(ConfigCommand.scala)
> Caused by: org.apache.zookeeper.KeeperException$NoNodeException: 
> KeeperErrorCode = NoNode for /config/changes/config_change_
>   at org.apache.zookeeper.KeeperException.create(KeeperException.java:111)
>   at org.apache.zookeeper.KeeperException.create(KeeperException.java:51)
>   at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:783)
>   at org.I0Itec.zkclient.ZkConnection.create(ZkConnection.java:100)
>   at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:531)
>   at org.I0Itec.zkclient.ZkClient$3.call(ZkClient.java:528)
>   at org.I0Itec.zkclient.ZkClient.retryUntilConnected(ZkClient.java:991)
>   ... 11 more
> {code}
> The command doesn't appear to fail but it does throw an exception. The return 
> code of the script is still 0 and the user is created in ZooKeeper but this 
> should be cleaned up since it's misleading.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KAFKA-6798) Kafka leader rebalance failures

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441972#comment-16441972
 ] 

Manikumar edited comment on KAFKA-6798 at 4/18/18 6:43 AM:
---

ISR is set to empty  if all replicas are out of sync and unclean leader 
election is disabled. We need to find out the reason for replicas going async.  
check for any zk connection/network errors, GC issues. monitoring under 
replicated partitions and correlating with errors, may help to identify the 
issue.


was (Author: omkreddy):
ISR is set to empty if all replicas are out of sync and unclean leader election 
is disabled. We need to find out the reason for replicas going async.  check 
for any zk connection/network errors, GC issues. monitoring under partitions 
may help to identify the issue.

> Kafka leader rebalance failures
> ---
>
> Key: KAFKA-6798
> URL: https://issues.apache.org/jira/browse/KAFKA-6798
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.1, 1.0.1
>Reporter: Riley Zimmerman
>Priority: Critical
>
> I am running 3 Kafka (version 0.10.2.1 and more recently moved to 1.0.1) with 
> 3 Zookeeper (v3.4.9) as statefulsets in a kubernetes v1.9.1 deployment.  My 
> partitions are replication factor 3.  My main workload involves a kafka 
> streams consumer/producer (storing offsets in kafka) and a second kafka 
> consumer storing offsets in zookeeper (only commits every 30 seconds).  There 
> are ~200,000 kafka messages going through each per minute.  The log.retention 
> settings are all 4 hours.  I have auto.leader.rebalance.enabled.  
> I am randomly having failures during the rebalances.  The result is that 
> partitions for both topics and consumer_offsets go out of sync and the 
> partition leader becomes -1.  After 4 hours there is another (auto?) 
> rebalance and sometimes it sorts itself out.  Sometimes it runs for weeks 
> without problems, other times it it happens multiple times in a few days.  It 
> appears to happen earlier in test runs if it is going to happen.   
> {noformat}
> Topic:__consumer_offsetsPartitionCount:50   ReplicationFactor:3   
>   
> Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
> Topic: __consumer_offsets   Partition: 0Leader: -1  
> Replicas: 2,0,1 Isr:
> Topic: __consumer_offsets   Partition: 1Leader: 0   
> Replicas: 0,1,2 Isr: 1,2,0
> Topic: __consumer_offsets   Partition: 2Leader: 1   
> Replicas: 1,2,0 Isr: 2,1,0
> Topic: __consumer_offsets   Partition: 3Leader: -1  
> Replicas: 2,1,0 Isr:
> {noformat}
> {noformat}
> [2018-03-20 12:42:32,180] WARN [Controller 2]: Partition [agent.metadata,5] 
> failed to complete preferred replica leader election. Leader is -1 
> (kafka.controller.KafkaController)
> {noformat}
> {noformat}
> [2018-03-20 11:02:32,099] TRACE Controller 2 epoch 27 started leader election 
> for partition [__consumer_offsets,30] (state.change.logger)
> [2018-03-20 11:02:32,101] ERROR Controller 2 epoch 27 encountered error while 
> electing leader for partition [__consumer_offsets,30] due to: Preferred 
> replica 2 for partition [__consumer_offsets,30] is either not alive or not in 
> the isr. Current leader and ISR: [{"leader":-1,"leader_epoch":59,"isr":[]}]. 
> (state.change.logger)
> [2018-03-20 11:02:32,101] ERROR Controller 2 epoch 27 initiated state change 
> for partition [__consumer_offsets,30] from OnlinePartition to OnlinePartition 
> failed (state.change.logger)
> kafka.common.StateChangeFailedException: encountered error while electing 
> leader for partition [__consumer_offsets,30] due to: Preferred replica 2 for 
> partition [__consumer_offsets,30] is either not alive or not in the isr. 
> Current leader and ISR: [{"leader":-1,"leader_epoch":59,"isr":[]}].
>   at 
> kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:362)
>   at 
> kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:202)
>   at 
> kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:141)
>   at 
> kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:140)
>   at scala.collection.immutable.Set$Set1.foreach(Set.scala:94)
>   at 
> kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:140)
>   at 
> kafka.controller.KafkaController.onPreferredReplicaElection(KafkaController.scala:662)
>   at 
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$16$$anonfun$apply$5.apply$mcV$sp(KafkaController.scala:1230)

[jira] [Commented] (KAFKA-6798) Kafka leader rebalance failures

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6798?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441972#comment-16441972
 ] 

Manikumar commented on KAFKA-6798:
--

ISR is set to empty if all replicas are out of sync and unclean leader election 
is disabled. We need to find out the reason for replicas going async.  check 
for any zk connection/network errors, GC issues. monitoring under partitions 
may help to identify the issue.

> Kafka leader rebalance failures
> ---
>
> Key: KAFKA-6798
> URL: https://issues.apache.org/jira/browse/KAFKA-6798
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.1, 1.0.1
>Reporter: Riley Zimmerman
>Priority: Critical
>
> I am running 3 Kafka (version 0.10.2.1 and more recently moved to 1.0.1) with 
> 3 Zookeeper (v3.4.9) as statefulsets in a kubernetes v1.9.1 deployment.  My 
> partitions are replication factor 3.  My main workload involves a kafka 
> streams consumer/producer (storing offsets in kafka) and a second kafka 
> consumer storing offsets in zookeeper (only commits every 30 seconds).  There 
> are ~200,000 kafka messages going through each per minute.  The log.retention 
> settings are all 4 hours.  I have auto.leader.rebalance.enabled.  
> I am randomly having failures during the rebalances.  The result is that 
> partitions for both topics and consumer_offsets go out of sync and the 
> partition leader becomes -1.  After 4 hours there is another (auto?) 
> rebalance and sometimes it sorts itself out.  Sometimes it runs for weeks 
> without problems, other times it it happens multiple times in a few days.  It 
> appears to happen earlier in test runs if it is going to happen.   
> {noformat}
> Topic:__consumer_offsetsPartitionCount:50   ReplicationFactor:3   
>   
> Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
> Topic: __consumer_offsets   Partition: 0Leader: -1  
> Replicas: 2,0,1 Isr:
> Topic: __consumer_offsets   Partition: 1Leader: 0   
> Replicas: 0,1,2 Isr: 1,2,0
> Topic: __consumer_offsets   Partition: 2Leader: 1   
> Replicas: 1,2,0 Isr: 2,1,0
> Topic: __consumer_offsets   Partition: 3Leader: -1  
> Replicas: 2,1,0 Isr:
> {noformat}
> {noformat}
> [2018-03-20 12:42:32,180] WARN [Controller 2]: Partition [agent.metadata,5] 
> failed to complete preferred replica leader election. Leader is -1 
> (kafka.controller.KafkaController)
> {noformat}
> {noformat}
> [2018-03-20 11:02:32,099] TRACE Controller 2 epoch 27 started leader election 
> for partition [__consumer_offsets,30] (state.change.logger)
> [2018-03-20 11:02:32,101] ERROR Controller 2 epoch 27 encountered error while 
> electing leader for partition [__consumer_offsets,30] due to: Preferred 
> replica 2 for partition [__consumer_offsets,30] is either not alive or not in 
> the isr. Current leader and ISR: [{"leader":-1,"leader_epoch":59,"isr":[]}]. 
> (state.change.logger)
> [2018-03-20 11:02:32,101] ERROR Controller 2 epoch 27 initiated state change 
> for partition [__consumer_offsets,30] from OnlinePartition to OnlinePartition 
> failed (state.change.logger)
> kafka.common.StateChangeFailedException: encountered error while electing 
> leader for partition [__consumer_offsets,30] due to: Preferred replica 2 for 
> partition [__consumer_offsets,30] is either not alive or not in the isr. 
> Current leader and ISR: [{"leader":-1,"leader_epoch":59,"isr":[]}].
>   at 
> kafka.controller.PartitionStateMachine.electLeaderForPartition(PartitionStateMachine.scala:362)
>   at 
> kafka.controller.PartitionStateMachine.kafka$controller$PartitionStateMachine$$handleStateChange(PartitionStateMachine.scala:202)
>   at 
> kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:141)
>   at 
> kafka.controller.PartitionStateMachine$$anonfun$handleStateChanges$2.apply(PartitionStateMachine.scala:140)
>   at scala.collection.immutable.Set$Set1.foreach(Set.scala:94)
>   at 
> kafka.controller.PartitionStateMachine.handleStateChanges(PartitionStateMachine.scala:140)
>   at 
> kafka.controller.KafkaController.onPreferredReplicaElection(KafkaController.scala:662)
>   at 
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$16$$anonfun$apply$5.apply$mcV$sp(KafkaController.scala:1230)
>   at 
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$16$$anonfun$apply$5.apply(KafkaController.scala:1225)
>   at 
> 

[jira] [Commented] (KAFKA-6404) OldConsumer FetchRequest apiVersion not match resulting in broker RequestHandler socket leak

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441820#comment-16441820
 ] 

Manikumar commented on KAFKA-6404:
--

SocketServer error handling is improved in KAFKA-5607. This must have been 
handled in KAFKA-5607. 

> OldConsumer FetchRequest apiVersion not match resulting in broker 
> RequestHandler socket leak
> 
>
> Key: KAFKA-6404
> URL: https://issues.apache.org/jira/browse/KAFKA-6404
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.1
>Reporter: Yu Gan
>Priority: Critical
>
> *kafka broker version*: 0.10.0.1
> *cluster info*: 200+ nodes, no acls, any client in the same LAN could access
> *situation*: someone uses high released version (such as 0.11.x) of 
> bin/kafka-console-consumer.sh with parameter "--zookeeper" to continuously 
> consume a topic with partitions spread all the brokers
> *phenomenon*: 
> 1.broker server log:
> errors like: 
> 1) Connection to 2 was disconnected before the response was read;
> 2) Shrinking ISR for partition [abc, 21] from 33,13,14 to 33;
> 3) ERROR Processor got uncaught exception. (kafka.network.Processor) 
> java.nio.BufferUnderflowException
> 2.common consumers keeping in rebalance status:
> errors like:
> 1) c.p.b.f.l.c.FiberTopoWorkerThread : got uncaught exception
> org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be 
> completed since the group has already rebalanced and assigned the partitions 
> to another member. This means that the time between subsequent calls to 
> poll() was longer than the configured session.timeout.ms, which typically 
> implies that the poll loop is spending too much time message processing. You 
> can address this either by increasing the session timeout or by reducing the 
> maximum size of batches returned in poll() with max.poll.records.
> 2) java.lang.IllegalStateException: Correlation id for response (1246203) 
> does not match request (1246122)
> *bad results*: kafka brokers in sick
> *root cause*: 
> 1) OldConsumer after 0.10.1 in ConsumerFetcherThread.scala setting 
> requestVersion 3:
> {code:java}
> private val fetchRequestBuilder = new FetchRequestBuilder().
> clientId(clientId).
> replicaId(Request.OrdinaryConsumerId).
> maxWait(config.fetchWaitMaxMs).
> minBytes(config.fetchMinBytes).
> requestVersion(3) // for now, the old consumer is pinned to the old 
> message format through the fetch request
> {code}
> but in 0.10.0.1 FetchRequest.CurrentVersion=2, FetchRequst.readFrom wouldn't 
> read the field "max_bytes" from version 3, then throws 
> "BufferUnderflowException" :
> {code:java}
>   def readFrom(buffer: ByteBuffer): FetchRequest = {
> val versionId = buffer.getShort
> val correlationId = buffer.getInt
> val clientId = readShortString(buffer)
> val replicaId = buffer.getInt
> val maxWait = buffer.getInt
> val minBytes = buffer.getInt
> val topicCount = buffer.getInt
> val pairs = (1 to topicCount).flatMap(_ => {
>   val topic = readShortString(buffer)
>   val partitionCount = buffer.getInt
>   (1 to partitionCount).map(_ => {
> val partitionId = buffer.getInt
> val offset = buffer.getLong
> val fetchSize = buffer.getInt
> (TopicAndPartition(topic, partitionId), PartitionFetchInfo(offset, 
> fetchSize))
>   })
> })
> FetchRequest(versionId, correlationId, clientId, replicaId, maxWait, 
> minBytes, Map(pairs:_*))
>   }
> {code}
> 2) when the FetchRequst.readFrom crashed with throwable like 
> "BufferUnderflowException" not in "(InvalidRequestException, 
> SchemaException)", the socket wouldn't be closed;
> SocketServer.processCompletedReceives:
> {code:java}
>   private def processCompletedReceives() {
> selector.completedReceives.asScala.foreach { receive =>
>   try {
> val openChannel = selector.channel(receive.source)
> // Only methods that are safe to call on a disconnected channel 
> should be invoked on 'openOrClosingChannel'.
> val openOrClosingChannel = if (openChannel != null) openChannel else 
> selector.closingChannel(receive.source)
> val session = RequestChannel.Session(new 
> KafkaPrincipal(KafkaPrincipal.USER_TYPE, 
> openOrClosingChannel.principal.getName), openOrClosingChannel.socketAddress)
> val req = RequestChannel.Request(processor = id, connectionId = 
> receive.source, session = session,
>   buffer = receive.payload, startTimeNanos = time.nanoseconds,
>   listenerName = listenerName, securityProtocol = securityProtocol)
> requestChannel.sendRequest(req)
> selector.mute(receive.source)
>   } catch {
> case e @ (_: InvalidRequestException | _: 

[jira] [Commented] (KAFKA-6479) Broker file descriptor leak after consumer request timeout

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6479?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16441366#comment-16441366
 ] 

Manikumar commented on KAFKA-6479:
--

One of the option is to adjust "connections.max.idle.ms"  config value 
(Default: 10mins). Server closes the connections that are idle more than this 
config value.

> Broker file descriptor leak after consumer request timeout
> --
>
> Key: KAFKA-6479
> URL: https://issues.apache.org/jira/browse/KAFKA-6479
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 1.0.0
>Reporter: Ryan Leslie
>Priority: Major
>
> When a consumer request times out, i.e. takes longer than request.timeout.ms, 
> and the client disconnects from the coordinator, the coordinator may leak 
> file descriptors. The following code produces this behavior:
> {code:java}
> Properties config = new Properties();
> config.put("bootstrap.servers", BROKERS);
> config.put("group.id", "leak-test");
> config.put("key.deserializer", 
> "org.apache.kafka.common.serialization.StringDeserializer");
> config.put("value.deserializer", 
> "org.apache.kafka.common.serialization.StringDeserializer");
> config.put("max.poll.interval.ms", Integer.MAX_VALUE);
> config.put("request.timeout.ms", 12000);
> KafkaConsumer consumer1 = new KafkaConsumer<>(config);
> KafkaConsumer consumer2 = new KafkaConsumer<>(config);
> List topics = Collections.singletonList("leak-test");
> consumer1.subscribe(topics);
> consumer2.subscribe(topics);
> consumer1.poll(100); 
> consumer2.poll(100);
> {code}
> When the above executes, consumer 2 will attempt to rebalance indefinitely 
> (blocked by the inactive consumer 1), logging a _Marking the coordinator 
> dead_ message every 12 seconds after giving up on the JOIN_GROUP request and 
> disconnecting. Unless the consumer exits or times out, this will cause a 
> socket in CLOSE_WAIT to leak in the coordinator and the broker will 
> eventually run out of file descriptors and crash.
> Aside from faulty code as in the example above, or an intentional DoS, any 
> client bug causing a consumer to block, e.g. KAFKA-6397, could also result in 
> this leak.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-2661) Add a unit test for disconnecting idle socket connections

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2661.
--
Resolution: Fixed

SelectorTest.testCloseConnectionInClosingState/testCloseOldestConnection tests 
covers unit test case for idle connections.

> Add a unit test for disconnecting idle socket connections 
> --
>
> Key: KAFKA-2661
> URL: https://issues.apache.org/jira/browse/KAFKA-2661
> Project: Kafka
>  Issue Type: Test
>  Components: core
>Reporter: Jun Rao
>Priority: Major
>
> The logic for disconnecting idle connections is now moved to Selector. We 
> just need to add a unit test to verify that it works.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (KAFKA-6778) DescribeConfigs does not return error for non-existent topic

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6778?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar reassigned KAFKA-6778:


Assignee: Manikumar

> DescribeConfigs does not return error for non-existent topic
> 
>
> Key: KAFKA-6778
> URL: https://issues.apache.org/jira/browse/KAFKA-6778
> Project: Kafka
>  Issue Type: Bug
>  Components: admin
>Affects Versions: 1.1.0
>Reporter: Magnus Edenhill
>Assignee: Manikumar
>Priority: Major
>
> Sending a DescribeConfigsRequest with a ConfigResource(TOPIC, 
> "non-existent-topic") returns a fully populated ConfigResource back in the 
> response with 24 configuration entries.
> A resource-level error_code of UnknownTopic.. would be expected instead.
>  
> {code:java}
> [0081_admin / 1.143s] ConfigResource #0: type TOPIC (2), 
> "rdkafkatest_rnd3df408bf5d94d696_DescribeConfigs_notexist": 24 ConfigEntries, 
> error NO_ERROR ()
> [0081_admin / 1.144s] #0/24: Source UNKNOWN (5): 
> "compression.type"="producer" [is read-only=n, default=n, sensitive=n, 
> synonym=n] with 1 synonym(s)
> 
> {code}
> But the topic does not exist:
> {code:java}
> $ $KAFKA_PATH/bin/kafka-topics.sh --zookeeper $ZK_ADDRESS --list | grep 
> rdkafkatest_rnd3df408bf5d94d696_DescribeConfigs_notexist ; echo $?
> 1
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-4373) Kafka Consumer API jumping offsets

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4373.
--
Resolution: Cannot Reproduce

Closing inactive issue. Please reopen if you think the issue still exists


> Kafka Consumer API jumping offsets
> --
>
> Key: KAFKA-4373
> URL: https://issues.apache.org/jira/browse/KAFKA-4373
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Reporter: Srinivasan Venkatraman
>Priority: Major
>
> Hi,
> I am using Kafka Version 0.10.0.1 and java consumer API to consume messages 
> from a topic. We are using a single node kafka and zookeeper. It is sometimes 
> observed that the consumer is losing a bulk of message. We are unable to find 
> the exact reason to replicate the issue.
> The scenario is:
> Consumer polls the topic.
> Fetches the messages and gives it to a thread pool to handle the message.
> Waits for the threads to return and then commits the offsets.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-2284) ConsumerRebalanceListener receives wrong type in partitionOwnership values

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2284.
--
Resolution: Auto Closed

The Scala consumers have been deprecated and no further work is planned, please 
upgrade to the Java consumer whenever possible.

> ConsumerRebalanceListener receives wrong type in partitionOwnership values
> --
>
> Key: KAFKA-2284
> URL: https://issues.apache.org/jira/browse/KAFKA-2284
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.2.0
>Reporter: E. Sammer
>Assignee: Neha Narkhede
>Priority: Major
>
> The ConsumerRebalanceListener's beforeReleasingPartitions() method is 
> supposed to receive an arg of Map (topic -> 
> Set(partitions)). Even though the type of the map value is specified as a 
> java.util.Set, a scala.collection.convert.Wrappers$JSetWrapper is passed 
> instead which does not implement Set causing a class cast exception as 
> soon as one attempts to access any value of the map. It looks as if this 
> method was never tested against the actual types specified by the interface.
> Here's what happens if you call {{Set foo = 
> partitionOwnership.get(topic)}}:
> {code}
> 2015-06-18 07:28:43,776 
> (search-consumer_esammer-mbp.local-1434637723383-12126c1b_watcher_executor) 
> [WARN - 
> com.rocana.search.consumer.ConsumerService$1.beforeReleasingPartitions(ConsumerService.java:246)]
>  Exception while rebalancing!
> java.lang.ClassCastException: scala.collection.convert.Wrappers$JSetWrapper 
> cannot be cast to java.util.Set
>   at 
> com.rocana.search.consumer.IndexConsumerWorker.onRebalance(IndexConsumerWorker.java:80)
>   at 
> com.rocana.search.consumer.ConsumerService$1.beforeReleasingPartitions(ConsumerService.java:244)
>   at 
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.kafka$consumer$ZookeeperConsumerConnector$ZKRebalancerListener$$rebalance(ZookeeperConsumerConnector.scala:675)
>   at 
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1$$anonfun$apply$mcV$sp$1.apply$mcVI$sp(ZookeeperConsumerConnector.scala:625)
>   at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:142)
>   at 
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply$mcV$sp(ZookeeperConsumerConnector.scala:619)
>   at 
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply(ZookeeperConsumerConnector.scala:616)
>   at 
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anonfun$syncedRebalance$1.apply(ZookeeperConsumerConnector.scala:616)
>   at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
>   at 
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:615)
>   at 
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:568)
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-4347) outputStreamId in Kafka spout

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4347?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4347.
--
Resolution: Invalid

Looks like this is related to Apache Storm Kafka connector. Request you raise 
the issue on Storm project.

> outputStreamId in Kafka spout
> -
>
> Key: KAFKA-4347
> URL: https://issues.apache.org/jira/browse/KAFKA-4347
> Project: Kafka
>  Issue Type: Bug
>Reporter: Rahim Dastar
>Priority: Major
>
> If you set outputStreamId in SpoutConfig for kafka spout, topology sends 
> tuples to another stream. The wrong stream name is topicName.
> In ParttionManager class  collector.emit(_spoutConfig.topic, tup, new 
> KafkaMessageId(_partition, toEmit.offset())); should change to 
> collector.emit(_spoutConfig.outputStreamId, tup, new 
> KafkaMessageId(_partition, toEmit.offset()));.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-4124) Handle disk failures gracefully

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4124.
--
Resolution: Duplicate

Some of these requirements are implemented in KIP-112/KIP-113. Pls raise new 
JIRAs for any missing/additional requirements/improvements.

> Handle disk failures gracefully
> ---
>
> Key: KAFKA-4124
> URL: https://issues.apache.org/jira/browse/KAFKA-4124
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Gokul
>Priority: Major
>
> Currently when a disk goes down, the broker also goes down with it. This 
> causes too much reshuffle of data over the network to replace the broker. 
> Make the broker resilient to disk failure. 
> The broker can detect a disk failure, mark it bad and then re-replicate the 
> under replicated data in all other available disks in the node. If the bad 
> disk is replaced with new one, the broker can rebalance the data among all 
> other disks it has. The broker can also tolerate upto n disk failures.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-1786) implement a global configuration feature for brokers

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16435598#comment-16435598
 ] 

Manikumar commented on KAFKA-1786:
--

looks like this requirement is covered in Dynamic Broker config feature 
(KIP-226).

> implement a global configuration feature for brokers
> 
>
> Key: KAFKA-1786
> URL: https://issues.apache.org/jira/browse/KAFKA-1786
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joe Stein
>Assignee: Andrii Biletskyi
>Priority: Major
> Attachments: KAFKA_1786.patch
>
>
> Global level configurations (much like topic level) for brokers are managed 
> by humans and automation systems through server.properties.  
> Some configuration make sense to use default (like it is now) or override 
> from central location (zookeeper for now). We can modify this through the new 
> CLI tool so that every broker can have exact same setting.  Some 
> configurations we should allow to be overriden from server.properties (like 
> port) but others we should use the global store as source of truth (e.g. auto 
> topic enable, fetch replica message size, etc). Since most configuration I 
> believe are going to fall into this category we should have the list of 
> server.properties that can override the global config in the code in a list 
> which we can manage... everything else the global takes precedence. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-611) Migration tool and Mirror Maker ignore mirror.topics.whitelist in config

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-611.
-
Resolution: Auto Closed

mirror.topics.whitelist/blacklist were part of old consumer. also these configs 
were not used in old consumer.  The old consumer is no longer supported.. 
MirrorMaker supports whitelist option. 

> Migration tool and Mirror Maker ignore mirror.topics.whitelist in config
> 
>
> Key: KAFKA-611
> URL: https://issues.apache.org/jira/browse/KAFKA-611
> Project: Kafka
>  Issue Type: Bug
>  Components: config
>Affects Versions: 0.7, 0.8.0
>Reporter: Chris Riccomini
>Priority: Major
>
> Apparently Kafka ignores the "mirror.topics.whitelist" setting in both 0.7 
> MirrorMaker and 0.8 KafkaMigrationTool. It only appears to pay attention to 
> --whitelist.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-2339) broker becomes unavailable if bad data is passed through the protocol

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2339?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2339.
--
Resolution: Cannot Reproduce

Not able to reproduce on latest server. Please reopen if you think the issue 
still exists.

> broker becomes unavailable if bad data is passed through the protocol
> -
>
> Key: KAFKA-2339
> URL: https://issues.apache.org/jira/browse/KAFKA-2339
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>Assignee: Timothy Chen
>Priority: Critical
>
> I ran into a situation that a non integer value got past for the partition 
> and the brokers went bonkers.
> reproducible
> {code}
> ah="1..2"
> echo "don't do this in production"|kafkacat -b localhost:9092 -p $ah
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-2302) Currently iterated chunk is not cleared during consumer shutdown

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-2302.
--
Resolution: Auto Closed

Closing inactive issue. The old consumer is no longer supported, please upgrade 
to the Java consumer whenever possible.

> Currently iterated chunk is not cleared during consumer shutdown
> 
>
> Key: KAFKA-2302
> URL: https://issues.apache.org/jira/browse/KAFKA-2302
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.1.1
>Reporter: Matthieu Sprunck
>Assignee: Joel Koshy
>Priority: Major
>
> During consumer connector shutdown, fetch queues are cleared, but the 
> currently iterated chunk is not cleared.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KAFKA-2021) Consolidate test classes for KafkaConfig

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2021?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar updated KAFKA-2021:
-
Description: 
* We have kafka.server.KafkaConfigTest, KafkaConfigConfigDefTest and 
kafka.unit.KafkaTest (in a file called KafkaConfigTest.scala)

I think consolidating them into one test class (or at list renaming so it will 
be clear how they are different) will make a lot of sense.

  was:
We have kafka.server.KafkaConfigTest, KafkaConfigConfigDefTest and 
kafka.unit.KafkaTest (in a file called KafkaConfigTest.scala)

I think consolidating them into one test class (or at list renaming so it will 
be clear how they are different) will make a lot of sense.


> Consolidate test classes for KafkaConfig
> 
>
> Key: KAFKA-2021
> URL: https://issues.apache.org/jira/browse/KAFKA-2021
> Project: Kafka
>  Issue Type: Task
>Reporter: Gwen Shapira
>Assignee: Andrii Biletskyi
>Priority: Minor
> Attachments: KAFKA-2021.patch
>
>
> * We have kafka.server.KafkaConfigTest, KafkaConfigConfigDefTest and 
> kafka.unit.KafkaTest (in a file called KafkaConfigTest.scala)
> I think consolidating them into one test class (or at list renaming so it 
> will be clear how they are different) will make a lot of sense.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-1643) message.send.max.retries not respected when no brokers are up

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1643.
--
Resolution: Auto Closed

The Scala producers have been deprecated for a while and no further work is 
planned. Please upgrade to the Java producer whenever possible.


> message.send.max.retries not respected when no brokers are up
> -
>
> Key: KAFKA-1643
> URL: https://issues.apache.org/jira/browse/KAFKA-1643
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.1.1
>Reporter: Alex Shafer
>Assignee: Jun Rao
>Priority: Major
>
> {noformat}
> 2014-09-19 20:20:04,320 WARN kafka.producer.async.DefaultEventHandler: Failed 
> to send producer request with correlation id 1944531 to broker 6405 with data 
> for partitions [lva1-spades-hdfs-audit,2]
> java.io.IOException: Broken pipe
> at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
> at sun.nio.ch.IOUtil.write(IOUtil.java:148)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:499)
> at java.nio.channels.SocketChannel.write(SocketChannel.java:502)
> at 
> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:56)
> at kafka.network.Send$class.writeCompletely(Transmission.scala:75)
> at 
> kafka.network.BoundedByteBufferSend.writeCompletely(BoundedByteBufferSend.scala:26)
> at kafka.network.BlockingChannel.send(BlockingChannel.scala:92)
> at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:72)
> at 
> kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:71)
> at 
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(SyncProducer.scala:102)
> at 
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
> at 
> kafka.producer.SyncProducer$$anonfun$send$1$$anonfun$apply$mcV$sp$1.apply(SyncProducer.scala:102)
> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> at 
> kafka.producer.SyncProducer$$anonfun$send$1.apply$mcV$sp(SyncProducer.scala:101)
> at 
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
> at 
> kafka.producer.SyncProducer$$anonfun$send$1.apply(SyncProducer.scala:101)
> at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
> at kafka.producer.SyncProducer.send(SyncProducer.scala:100)
> at 
> kafka.producer.async.DefaultEventHandler.kafka$producer$async$DefaultEventHandler$$send(DefaultEventHandler.scala:255)
> at 
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:106)
> at 
> kafka.producer.async.DefaultEventHandler$$anonfun$dispatchSerializedData$1.apply(DefaultEventHandler.scala:100)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:80)
> at scala.collection.Iterator$class.foreach(Iterator.scala:631)
> at 
> scala.collection.mutable.HashTable$$anon$1.foreach(HashTable.scala:161)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:194)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:80)
> at 
> kafka.producer.async.DefaultEventHandler.dispatchSerializedData(DefaultEventHandler.scala:100)
> at 
> kafka.producer.async.DefaultEventHandler.handle(DefaultEventHandler.scala:72)
> at kafka.producer.Producer.send(Producer.scala:76)
> at 
> kafka.producer.KafkaLog4jAppender.append(KafkaLog4jAppender.scala:93)
> at 
> org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
> at 
> org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
> at org.apache.log4j.Category.callAppenders(Category.java:206)
> at org.apache.log4j.Category.forcedLog(Category.java:391)
> at org.apache.log4j.Category.log(Category.java:856)
> at 
> org.apache.commons.logging.impl.Log4JLogger.info(Log4JLogger.java:176)
> 
> 2014-09-19 20:20:04,331 INFO kafka.producer.async.DefaultEventHandler: Back 
> off for 100 ms before retrying send. Remaining retries = 3
> 2014-09-19 20:20:04,433 INFO kafka.client.ClientUtils$: Fetching metadata 
> from broker id:17,host:eat1-hcl6393.grid.linkedin.com,port:9092 with 
> correlation id 1944532 for 1 topic(s) Set(lva1-spades-hdfs-audit)
> 2014-09-19 20:20:04,451 ERROR 

[jira] [Resolved] (KAFKA-1405) Global JSON.globalNumberParser screws up other libraries

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1405.
--
Resolution: Fixed

Old Scala JSON  usage is removed in new versions of Kafka

> Global JSON.globalNumberParser screws up other libraries
> 
>
> Key: KAFKA-1405
> URL: https://issues.apache.org/jira/browse/KAFKA-1405
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0
>Reporter: Vadim Chekan
>Priority: Major
>  Labels: json
>
> I'm getting exception "kafka.common.ConsumerRebalanceFailedException" but it 
> only happens when I do a call to "scala/pickling" serialization library. What 
> the connection you might ask? The underly exception is 
> "ZookeeperConsumerConnector:76, exception during rebalance 
> kafka.common.KafkaException: Failed to parse the broker info from zookeeper: 
> {"jmx_port":-1,"timestamp":"1397514497053","host":"xxx","version":1,"port":9092}
> Caused by: java.lang.ClassCastException: java.lang.Double cannot be cast to 
> java.lang.Integer"
> A little bit looking at  kafka code lead me to this line:
> In 
> https://github.com/apache/kafka/blob/0.8.0/core/src/main/scala/kafka/utils/Json.scala#L27
> there is JSON.globalNumberParser redefined. It is terrible idea to change 
> global variable. This JSON library is used by other libraries and this global 
> assignment messes it up.
> My 5-minutes research shows that scala's JSON library was considered almost 
> of demo quality and most people prefer ligt-json implementation.
> https://groups.google.com/forum/#!topic/scala-user/P7-8PEUUj6A
> Also it is my understanding, that scala JSON is deprecated in scala-2.11, so 
> this change is needed anyway.
> If no objections to this ticket in general, I can work on a patch to use 3rd 
> party JSON library usage in kafka. Pleas let me know...



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-4883) invalid client sasl.jaas.config triggers NullPointerException

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4883.
--
   Resolution: Fixed
 Assignee: Manikumar
Fix Version/s: 1.2.0

> invalid client sasl.jaas.config triggers NullPointerException
> -
>
> Key: KAFKA-4883
> URL: https://issues.apache.org/jira/browse/KAFKA-4883
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.0
>Reporter: Xavier Léauté
>Assignee: Manikumar
>Priority: Minor
> Fix For: 1.2.0
>
>
> Using an invalid sasl.jaas.config property causes a NullPointerException to 
> be thrown, which makes it difficult to understand the root of the problem.
> Example of an invalid config
> {code}
> sasl.jaas.config=KafkaClient {\n\
>   org.apache.kafka.common.security.plain.PlainLoginModule required\n\
>   username="x"\n\
>   password="";\n\
> }
> {code}
> Resulting stack trace:
> {code}
> org.apache.kafka.common.KafkaException: Failed to construct kafka producer
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:342)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:166)
> [...]
> Caused by: org.apache.kafka.common.KafkaException: 
> java.lang.NullPointerException
>   at 
> org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:94)
>   at 
> org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:93)
>   at 
> org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:51)
>   at 
> org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:84)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:305)
>   ... 41 more
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.kafka.common.security.JaasConfig.loginModuleControlFlag(JaasConfig.java:87)
>   at 
> org.apache.kafka.common.security.JaasConfig.parseAppConfigurationEntry(JaasConfig.java:110)
>   at 
> org.apache.kafka.common.security.JaasConfig.(JaasConfig.java:65)
>   at 
> org.apache.kafka.common.security.JaasUtils.jaasConfig(JaasUtils.java:59)
>   at 
> org.apache.kafka.common.network.SaslChannelBuilder.configure(SaslChannelBuilder.java:85)
>   ... 45 more
> {code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Assigned] (KAFKA-6752) Unclean leader election metric no longer working

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar reassigned KAFKA-6752:


Assignee: Manikumar

> Unclean leader election metric no longer working
> 
>
> Key: KAFKA-6752
> URL: https://issues.apache.org/jira/browse/KAFKA-6752
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 1.1.0
>Reporter: Jason Gustafson
>Assignee: Manikumar
>Priority: Major
>
> Happened to notice that the unclean leader election meter is no longer being 
> updated. This was probably lost during the controller overhaul.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6413) ReassignPartitionsCommand#parsePartitionReassignmentData() should give better error message when JSON is malformed

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6413.
--
Resolution: Duplicate

Resolving as duplicate of KAFKA-6084.

> ReassignPartitionsCommand#parsePartitionReassignmentData() should give better 
> error message when JSON is malformed
> --
>
> Key: KAFKA-6413
> URL: https://issues.apache.org/jira/browse/KAFKA-6413
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ted Yu
>Priority: Minor
>  Labels: json
>
> In this thread: 
> http://search-hadoop.com/m/Kafka/uyzND1J9Hizcxo0X?subj=Partition+reassignment+data+file+is+empty
>  , Allen gave an example JSON string with extra comma where 
> partitionsToBeReassigned returned by 
> ReassignPartitionsCommand#parsePartitionReassignmentData() was empty.
> I tried the following example where a right bracket is removed:
> {code}
> val (partitionsToBeReassigned, replicaAssignment) = 
> ReassignPartitionsCommand.parsePartitionReassignmentData(
> 
> "{\"version\":1,\"partitions\":[{\"topic\":\"metrics\",\"partition\":0,\"replicas\":[1,2]},{\"topic\":\"metrics\",\"partition\":1,\"replicas\":[2,3]},}");
> {code}
> The returned partitionsToBeReassigned is empty (and no exception was thrown).
> The parser should give better error message for malformed JSON string.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Updated] (KAFKA-6741) Transient test failure: SslTransportLayerTest.testNetworkThreadTimeRecorded

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar updated KAFKA-6741:
-
Description: 
debug logs:

{code}
 [2018-04-03 14:51:33,365] DEBUG Added sensor with name connections-closed: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,368] DEBUG Added sensor with name connections-created: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,368] DEBUG Added sensor with name 
successful-authentication: (org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,368] DEBUG Added sensor with name failed-authentication: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,368] DEBUG Added sensor with name bytes-sent-received: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,369] DEBUG Added sensor with name bytes-sent: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,370] DEBUG Added sensor with name bytes-received: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,370] DEBUG Added sensor with name select-time: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,371] DEBUG Added sensor with name io-time: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,379] DEBUG Removed sensor with name connections-closed: 
(org.apache.kafka.common.metrics.Metrics:447)
 [2018-04-03 14:51:33,379] DEBUG Removed sensor with name connections-created: 
(org.apache.kafka.common.metrics.Metrics:447)
 [2018-04-03 14:51:33,379] DEBUG Removed sensor with name 
successful-authentication: (org.apache.kafka.common.metrics.Metrics:447)
 [2018-04-03 14:51:33,380] DEBUG Removed sensor with name 
failed-authentication: (org.apache.kafka.common.metrics.Metrics:447)
 [2018-04-03 14:51:33,380] DEBUG Removed sensor with name bytes-sent-received: 
(org.apache.kafka.common.metrics.Metrics:447)
 [2018-04-03 14:51:33,380] DEBUG Removed sensor with name bytes-sent: 
(org.apache.kafka.common.metrics.Metrics:447)
 [2018-04-03 14:51:33,380] DEBUG Removed sensor with name bytes-received: 
(org.apache.kafka.common.metrics.Metrics:447)
 [2018-04-03 14:51:33,382] DEBUG Removed sensor with name select-time: 
(org.apache.kafka.common.metrics.Metrics:447)
 [2018-04-03 14:51:33,382] DEBUG Removed sensor with name io-time: 
(org.apache.kafka.common.metrics.Metrics:447)
 [2018-04-03 14:51:33,382] DEBUG Added sensor with name connections-closed: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,382] DEBUG Added sensor with name connections-created: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,383] DEBUG Added sensor with name 
successful-authentication: (org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,383] DEBUG Added sensor with name failed-authentication: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,383] DEBUG Added sensor with name bytes-sent-received: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,384] DEBUG Added sensor with name bytes-sent: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,384] DEBUG Added sensor with name bytes-received: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,384] DEBUG Added sensor with name select-time: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,384] DEBUG Added sensor with name io-time: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,444] DEBUG Added sensor with name connections-closed: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,445] DEBUG Added sensor with name connections-created: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,445] DEBUG Added sensor with name 
successful-authentication: (org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,445] DEBUG Added sensor with name failed-authentication: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,446] DEBUG Added sensor with name bytes-sent-received: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,446] DEBUG Added sensor with name bytes-sent: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,446] DEBUG Added sensor with name bytes-received: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,447] DEBUG Added sensor with name select-time: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,447] DEBUG Added sensor with name io-time: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,890] DEBUG Created socket with SO_RCVBUF = 326640, 
SO_SNDBUF = 65328, SO_TIMEOUT = 0 to node 0 
(org.apache.kafka.common.network.Selector:474)
 [2018-04-03 14:51:33,892] DEBUG Added sensor with name 
node-127.0.0.1:53543-127.0.0.1:53544.bytes-sent 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,893] DEBUG Added sensor with name 

[jira] [Created] (KAFKA-6741) Transient test failure: SslTransportLayerTest.testNetworkThreadTimeRecorded

[Manikumar (JIRA)]

Manikumar created KAFKA-6741:


 Summary: Transient test failure: 
SslTransportLayerTest.testNetworkThreadTimeRecorded
 Key: KAFKA-6741
 URL: https://issues.apache.org/jira/browse/KAFKA-6741
 Project: Kafka
  Issue Type: Bug
Reporter: Manikumar


debug logs:

{code}
 [2018-04-03 14:51:33,365] DEBUG Added sensor with name connections-closed: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,368] DEBUG Added sensor with name connections-created: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,368] DEBUG Added sensor with name 
successful-authentication: (org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,368] DEBUG Added sensor with name failed-authentication: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,368] DEBUG Added sensor with name bytes-sent-received: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,369] DEBUG Added sensor with name bytes-sent: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,370] DEBUG Added sensor with name bytes-received: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,370] DEBUG Added sensor with name select-time: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,371] DEBUG Added sensor with name io-time: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,379] DEBUG Removed sensor with name connections-closed: 
(org.apache.kafka.common.metrics.Metrics:447)
 [2018-04-03 14:51:33,379] DEBUG Removed sensor with name connections-created: 
(org.apache.kafka.common.metrics.Metrics:447)
 [2018-04-03 14:51:33,379] DEBUG Removed sensor with name 
successful-authentication: (org.apache.kafka.common.metrics.Metrics:447)
 [2018-04-03 14:51:33,380] DEBUG Removed sensor with name 
failed-authentication: (org.apache.kafka.common.metrics.Metrics:447)
 [2018-04-03 14:51:33,380] DEBUG Removed sensor with name bytes-sent-received: 
(org.apache.kafka.common.metrics.Metrics:447)
 [2018-04-03 14:51:33,380] DEBUG Removed sensor with name bytes-sent: 
(org.apache.kafka.common.metrics.Metrics:447)
 [2018-04-03 14:51:33,380] DEBUG Removed sensor with name bytes-received: 
(org.apache.kafka.common.metrics.Metrics:447)
 [2018-04-03 14:51:33,382] DEBUG Removed sensor with name select-time: 
(org.apache.kafka.common.metrics.Metrics:447)
 [2018-04-03 14:51:33,382] DEBUG Removed sensor with name io-time: 
(org.apache.kafka.common.metrics.Metrics:447)
 [2018-04-03 14:51:33,382] DEBUG Added sensor with name connections-closed: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,382] DEBUG Added sensor with name connections-created: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,383] DEBUG Added sensor with name 
successful-authentication: (org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,383] DEBUG Added sensor with name failed-authentication: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,383] DEBUG Added sensor with name bytes-sent-received: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,384] DEBUG Added sensor with name bytes-sent: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,384] DEBUG Added sensor with name bytes-received: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,384] DEBUG Added sensor with name select-time: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,384] DEBUG Added sensor with name io-time: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,444] DEBUG Added sensor with name connections-closed: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,445] DEBUG Added sensor with name connections-created: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,445] DEBUG Added sensor with name 
successful-authentication: (org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,445] DEBUG Added sensor with name failed-authentication: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,446] DEBUG Added sensor with name bytes-sent-received: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,446] DEBUG Added sensor with name bytes-sent: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,446] DEBUG Added sensor with name bytes-received: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,447] DEBUG Added sensor with name select-time: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,447] DEBUG Added sensor with name io-time: 
(org.apache.kafka.common.metrics.Metrics:414)
 [2018-04-03 14:51:33,890] DEBUG Created socket with SO_RCVBUF = 326640, 
SO_SNDBUF = 65328, SO_TIMEOUT = 0 to node 0 
(org.apache.kafka.common.network.Selector:474)
 [2018-04-03 14:51:33,892] DEBUG Added sensor with name 

[jira] [Resolved] (KAFKA-6716) discardChannel should be released in MockSelector#completeSend

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6716.
--
   Resolution: Fixed
Fix Version/s: 1.2.0

> discardChannel should be released in MockSelector#completeSend
> --
>
> Key: KAFKA-6716
> URL: https://issues.apache.org/jira/browse/KAFKA-6716
> Project: Kafka
>  Issue Type: Test
>Reporter: Ted Yu
>Assignee: huxihx
>Priority: Minor
> Fix For: 1.2.0
>
>
> {code}
> private void completeSend(Send send) throws IOException {
> // Consume the send so that we will be able to send more requests to 
> the destination
> ByteBufferChannel discardChannel = new ByteBufferChannel(send.size());
> while (!send.completed()) {
> send.writeTo(discardChannel);
> }
> completedSends.add(send);
> }
> {code}
> The {{discardChannel}} should be closed before returning from the method



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6630) Speed up the processing of TopicDeletionStopReplicaResponseReceived events on the controller

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6630.
--
   Resolution: Fixed
Fix Version/s: 1.2.0

> Speed up the processing of TopicDeletionStopReplicaResponseReceived events on 
> the controller
> 
>
> Key: KAFKA-6630
> URL: https://issues.apache.org/jira/browse/KAFKA-6630
> Project: Kafka
>  Issue Type: Improvement
>  Components: core
>Reporter: Lucas Wang
>Assignee: Lucas Wang
>Priority: Minor
> Fix For: 1.2.0
>
>
> Problem Statement:
> We find in a large cluster with many partition replicas, it takes a long time 
> to successfully delete a topic. 
> Root cause:
> Further analysis shows that for a topic with N replicas, the controller 
> receives all the N StopReplicaResponses from brokers within a short time, 
> however sequentially handling all the N 
> TopicDeletionStopReplicaResponseReceived events one by one takes a long time.
> Specifically the functions triggered while handling every single 
> TopicDeletionStopReplicaResponseReceived event include:
> TopicDeletionStopReplicaResponseReceived.process calls 
> TopicDeletionManager.completeReplicaDeletion, which calls 
> TopicDeletionManager.resumeDeletions, which calls several inefficient 
> functions.
> The inefficient functions called inside TopicDeletionManager.resumeDeletions 
> include
> ReplicaStateMachine.areAllReplicasForTopicDeleted
> ReplicaStateMachine.isAtLeastOneReplicaInDeletionStartedState
> ReplicaStateMachine.replicasInState
> Each of the 3 inefficient functions above will iterate through all the 
> replicas in the cluster, and filter out the replicas belonging to a topic. In 
> a large cluster with many replicas, these functions can be quite slow. 
> Total deletion time for a topic becomes long in single threaded controller 
> processing model:
> Since the controller needs to sequentially process the queued 
> TopicDeletionStopReplicaResponseReceived events, if the time cost to process 
> one event is t, the total time to process all events for all replicas of a 
> topic is N * t.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6702) Wrong className in LoggerFactory.getLogger method

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6702.
--
Resolution: Fixed

> Wrong className in LoggerFactory.getLogger method
> -
>
> Key: KAFKA-6702
> URL: https://issues.apache.org/jira/browse/KAFKA-6702
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, streams
>Affects Versions: 1.0.1, 1.2.0
>Reporter: JieFang.He
>Assignee: JieFang.He
>Priority: Major
>
> Wrong className in LoggerFactory.getLogger method



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6707) The default value for config of Type.LONG should be *L

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6707.
--
Resolution: Fixed

> The default value for config of Type.LONG should be *L
> --
>
> Key: KAFKA-6707
> URL: https://issues.apache.org/jira/browse/KAFKA-6707
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 1.2.0
>Reporter: JieFang.He
>Assignee: JieFang.He
>Priority: Major
>
> h1. The default value for config of Type.LONG should be *L



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6705) producer.send() should not block due to metadata not available

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16412512#comment-16412512
 ] 

Manikumar commented on KAFKA-6705:
--

Existing JIRA: KAFKA-3539

> producer.send() should not block due to metadata not available
> --
>
> Key: KAFKA-6705
> URL: https://issues.apache.org/jira/browse/KAFKA-6705
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Dong Lin
>Assignee: Dong Lin
>Priority: Major
>
> Currently producer.send() may block on metadata for up to max.block.ms. This 
> behavior is well documented but it is a bit sub-optimal. Since we return a 
> future we should be able to make producer.send() completely non-blocking. One 
> idea is to simply insert the record into a global queue shared across all 
> partitions, and let the sender thread fetch record from this queue and send 
> to broker.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6676) System tests do not handle ZK chroot properly with SCRAM

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6676.
--
   Resolution: Fixed
Fix Version/s: 1.1.0

> System tests do not handle ZK chroot properly with SCRAM
> 
>
> Key: KAFKA-6676
> URL: https://issues.apache.org/jira/browse/KAFKA-6676
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
>Priority: Major
> Fix For: 1.1.0
>
>
> This is related to the issue observed in KAFKA-6672. There, we are now 
> automatically creating parent nodes if they do not exist. However, if using a 
> chroot within ZK and that chroot does not yet exist, you get an error message 
> about "Path length must be > 0" as it tries to create all the parent paths.
> It would probably be better to be able to detect this issue and account for 
> it, but currently system test code will fail if you use SCRAM and a chroot 
> because while Kafka will create the chroot when it starts up, there are some 
> commands related to security that may need to be executed before that and 
> assume the chroot will already be there.
> We're currently missing this because while the chroot option is there, 
> nothing in Kafka's tests are currently exercising it. So given what is 
> apparently a common assumption in tools that the chroot already exists (since 
> I think the core kafka server is the only thing that handles creating it if 
> needed), I think the fix here would be two-fold:
>  # Make KafkaService ensure the chroot exists before running any commands 
> that might need it.
>  # On at least one test that exercises security support, use a zk_chroot so 
> that functionality is at least reasonably well exercised.
> It would be good to have this in both trunk and 1.1 branches.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6680) Fix config initialization in DynamicBrokerConfig

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6680.
--
   Resolution: Fixed
Fix Version/s: (was: 1.2.0)
   1.1.0

> Fix config initialization in DynamicBrokerConfig
> 
>
> Key: KAFKA-6680
> URL: https://issues.apache.org/jira/browse/KAFKA-6680
> Project: Kafka
>  Issue Type: Bug
>Reporter: Manikumar
>Assignee: Manikumar
>Priority: Major
> Fix For: 1.1.0
>
>
> Below issues observed while testing dynamic config update feature
> 1. {{kafkaConfig}} doesn't get updated during {{initialize}} if there are no 
> dynamic configs defined in ZK.
> 2.  update DynamicListenerConfig.validateReconfiguration() to check new 
> Listeners must be subset of listener map



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KAFKA-6682) Kafka reconnection after broker restart

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406717#comment-16406717
 ] 

Manikumar edited comment on KAFKA-6682 at 3/20/18 5:14 PM:
---

Looks like this fixed in : https://issues.apache.org/jira/browse/KAFKA-6260
You can try using 1.0.1 client library.


was (Author: omkreddy):
Looks like this fixed in : https://issues.apache.org/jira/browse/KAFKA-6260

> Kafka reconnection after broker restart
> ---
>
> Key: KAFKA-6682
> URL: https://issues.apache.org/jira/browse/KAFKA-6682
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 1.0.0
>Reporter: madi
>Priority: Major
>
> I am using kafka producer plugin for logback (danielwegener) with the clients 
> library 1.0.0 and after restart of broker all my JVMs connected to it get 
> tons of the exceptions:
> {code:java}
> 11:22:48.738 [kafka-producer-network-thread | app-logback-relaxed] cid: 
> clid: E [    @] a: o.a.k.c.p.internals.Sender - [Producer 
> clientId=id-id-logback-relaxed] Uncaught error in kafka producer I/O 
> thread:  ex:java.lang.NullPointerException: null
>     at 
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:436)
>     at org.apache.kafka.common.network.Selector.poll(Selector.java:399)
>     at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460)
>     at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
>     at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
>     at java.lang.Thread.run(Thread.java:798){code}
> During restart there are still other brokers available behind LB.    
> Dosen't matter kafka is up again, only restarting JVM helps
> {code:java}
>      class="com.github.danielwegener.logback.kafka.KafkaAppender">
>     
>     
>    
>  %date{"-MM-dd'T'HH:mm:ss.SSS'Z'"} ${HOSTNAME} 
> [%thread] %logger{32} - %message ex:%exf%n
>     
>     mytopichere
>     
>      class="com.github.danielwegener.logback.kafka.keying.HostNameKeyingStrategy" 
> />
>     
>      class="com.github.danielwegener.logback.kafka.delivery.AsynchronousDeliveryStrategy"
>  />
>     
>    
>  
>     
>     bootstrap.servers=10.99.99.1:9092
>     
>     acks=0
>     
>     block.on.buffer.full=false
>     
>     
> client.id=${HOSTNAME}-${CONTEXT_NAME}-logback-relaxed
>     
>     
>     compression.type=none
>    
>  
>     max.block.ms=0
>     {code}
> I provide loadbalancer address in bootstrap servers here. There are three 
> kafka brokers behind.
> {code:java}
> java version "1.7.0"
> Java(TM) SE Runtime Environment (build pap6470sr9fp60ifix-20161110_01(SR9 
> FP60)+IV90630+IV90578))
> IBM J9 VM (build 2.6, JRE 1.7.0 AIX ppc64-64 Compressed References 
> 20161005_321282 (JIT enabled, AOT enabled)
> J9VM - R26_Java726_SR9_20161005_1259_B321282
> JIT  - tr.r11_20161001_125404
> GC   - R26_Java726_SR9_20161005_1259_B321282_CMPRSS
> J9CL - 20161005_321282)
> JCL - 20161021_01 based on Oracle jdk7u121-b15{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6682) Kafka reconnection after broker restart

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16406717#comment-16406717
 ] 

Manikumar commented on KAFKA-6682:
--

Looks like this fixed in : https://issues.apache.org/jira/browse/KAFKA-6260

> Kafka reconnection after broker restart
> ---
>
> Key: KAFKA-6682
> URL: https://issues.apache.org/jira/browse/KAFKA-6682
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 1.0.0
>Reporter: madi
>Priority: Major
>
> I am using kafka producer plugin for logback (danielwegener) with the clients 
> library 1.0.0 and after restart of broker all my JVMs connected to it get 
> tons of the exceptions:
> {code:java}
> 11:22:48.738 [kafka-producer-network-thread | app-logback-relaxed] cid: 
> clid: E [    @] a: o.a.k.c.p.internals.Sender - [Producer 
> clientId=id-id-logback-relaxed] Uncaught error in kafka producer I/O 
> thread:  ex:java.lang.NullPointerException: null
>     at 
> org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:436)
>     at org.apache.kafka.common.network.Selector.poll(Selector.java:399)
>     at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:460)
>     at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:239)
>     at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:163)
>     at java.lang.Thread.run(Thread.java:798){code}
> During restart there are still other brokers available behind LB.    
> Dosen't matter kafka is up again, only restarting JVM helps
> {code:java}
>      class="com.github.danielwegener.logback.kafka.KafkaAppender">
>     
>     
>    
>  %date{"-MM-dd'T'HH:mm:ss.SSS'Z'"} ${HOSTNAME} 
> [%thread] %logger{32} - %message ex:%exf%n
>     
>     mytopichere
>     
>      class="com.github.danielwegener.logback.kafka.keying.HostNameKeyingStrategy" 
> />
>     
>      class="com.github.danielwegener.logback.kafka.delivery.AsynchronousDeliveryStrategy"
>  />
>     
>    
>  
>     
>     bootstrap.servers=10.99.99.1:9092
>     
>     acks=0
>     
>     block.on.buffer.full=false
>     
>     
> client.id=${HOSTNAME}-${CONTEXT_NAME}-logback-relaxed
>     
>     
>     compression.type=none
>    
>  
>     max.block.ms=0
>     {code}
> I provide loadbalancer address in bootstrap servers here. There are three 
> kafka brokers behind.
> {code:java}
> java version "1.7.0"
> Java(TM) SE Runtime Environment (build pap6470sr9fp60ifix-20161110_01(SR9 
> FP60)+IV90630+IV90578))
> IBM J9 VM (build 2.6, JRE 1.7.0 AIX ppc64-64 Compressed References 
> 20161005_321282 (JIT enabled, AOT enabled)
> J9VM - R26_Java726_SR9_20161005_1259_B321282
> JIT  - tr.r11_20161001_125404
> GC   - R26_Java726_SR9_20161005_1259_B321282_CMPRSS
> J9CL - 20161005_321282)
> JCL - 20161021_01 based on Oracle jdk7u121-b15{code}



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Comment Edited] (KAFKA-6689) Kafka not release .deleted file.

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405870#comment-16405870
 ] 

Manikumar edited comment on KAFKA-6689 at 3/20/18 7:05 AM:
---

These temp files will be deleted after file.delete.delay.ms (default: 6) 
milliseconds after file creation.
 file count increasing continuously? Are you observing any exceptions in logs?


was (Author: omkreddy):
There temp files will be deleted after file.delete.delay.ms (default: 6) 
milliseconds after file creation.
 file count increasing continuously? Are you observing any exceptions in logs?

> Kafka not release .deleted file.
> 
>
> Key: KAFKA-6689
> URL: https://issues.apache.org/jira/browse/KAFKA-6689
> Project: Kafka
>  Issue Type: Bug
>  Components: config, controller, log
>Affects Versions: 0.10.1.1
> Environment: 3 Kafka Broker running on CentOS6.9(rungi3 VMs) 
>Reporter: A
>Priority: Critical
>  Labels: newbie
> Fix For: 0.10.1.1
>
>
>          After Kafka cleaned log  .timeindex / .index files based on topic 
> retention. I can
> still lsof a lot of .index.deleted and .timeindex.deleted files. 
>        We have 3 Brokers on 3 VMs ,  It's happen only 2 brokers.  
> [brokeer-01 ~]$ lsof -p 24324 | grep deleted | wc -l
> 28
> [broker-02 ~]$ lsof -p 12131 | grep deleted | wc -l
> 14599
> [broker-03 ~]$ lsof -p 3349 | grep deleted | wc -l
> 14523
>  
> Configuration on 3 broker is same.  (Rolllog every hour, Retention time 11 
> Hours)
>  * INFO KafkaConfig values:
>  advertised.host.name = null
>  advertised.listeners = PLAINTEXT://Broker-02:9092
>  advertised.port = null
>  authorizer.class.name =
>  auto.create.topics.enable = true
>  auto.leader.rebalance.enable = true
>  background.threads = 10
>  broker.id = 2
>  broker.id.generation.enable = true
>  broker.rack = null
>  compression.type = producer
>  connections.max.idle.ms = 60
>  controlled.shutdown.enable = true
>  controlled.shutdown.max.retries = 3
>  controlled.shutdown.retry.backoff.ms = 5000
>  controller.socket.timeout.ms = 3
>  default.replication.factor = 3
>  delete.topic.enable = true
>  fetch.purgatory.purge.interval.requests = 1000
>  group.max.session.timeout.ms = 30
>  group.min.session.timeout.ms = 6000
>  host.name =
>  inter.broker.protocol.version = 0.10.1-IV2
>  leader.imbalance.check.interval.seconds = 300
>  leader.imbalance.per.broker.percentage = 10
>  listeners = null
>  log.cleaner.backoff.ms = 15000
>  log.cleaner.dedupe.buffer.size = 134217728
>  log.cleaner.delete.retention.ms = 8640
>  log.cleaner.enable = true
>  log.cleaner.io.buffer.load.factor = 0.9
>  log.cleaner.io.buffer.size = 524288
>  log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
>  log.cleaner.min.cleanable.ratio = 0.5
>  log.cleaner.min.compaction.lag.ms = 0
>  log.cleaner.threads = 1
>  log.cleanup.policy = [delete]
>  log.dir = /tmp/kafka-logs
>  log.dirs = /data/appdata/kafka/data
>  log.flush.interval.messages = 9223372036854775807
>  log.flush.interval.ms = null
>  log.flush.offset.checkpoint.interval.ms = 6
>  log.flush.scheduler.interval.ms = 9223372036854775807
>  log.index.interval.bytes = 4096
>  log.index.size.max.bytes = 10485760
>  log.message.format.version = 0.10.1-IV2
>  log.message.timestamp.difference.max.ms = 9223372036854775807
>  log.message.timestamp.type = CreateTime
>  log.preallocate = false
>  log.retention.bytes = -1
>  log.retention.check.interval.ms = 30
>  log.retention.hours = 11
>  log.retention.minutes = 660
>  log.retention.ms = 3960
>  log.roll.hours = 1
>  log.roll.jitter.hours = 0
>  log.roll.jitter.ms = null
>  log.roll.ms = null
>  log.segment.bytes = 1073741824
>  log.segment.delete.delay.ms = 6
>  max.connections.per.ip = 2147483647
>  max.connections.per.ip.overrides =
>  message.max.bytes = 112
>  metric.reporters = []
>  metrics.num.samples = 2
>  metrics.sample.window.ms = 3
>  min.insync.replicas = 2
>  num.io.threads = 16
>  num.network.threads = 16
>  num.partitions = 10
>  num.recovery.threads.per.data.dir = 3
>  num.replica.fetchers = 1
>  offset.metadata.max.bytes = 4096
>  offsets.commit.required.acks = -1
>  offsets.commit.timeout.ms = 5000
>  offsets.load.buffer.size = 5242880
>  offsets.retention.check.interval.ms = 60
>  offsets.retention.minutes = 1440
>  offsets.topic.compression.codec = 0
>  offsets.topic.num.partitions = 50
>  offsets.topic.replication.factor = 3
>  offsets.topic.segment.bytes = 104857600
>  port = 9092
>  principal.builder.class = class 
> org.apache.kafka.common.security.auth.DefaultPrincipalBuilder
>  producer.purgatory.purge.interval.requests = 1000
>  queued.max.requests = 3
>  quota.consumer.default = 9223372036854775807
>  quota.producer.default = 9223372036854775807
>  

[jira] [Commented] (KAFKA-6689) Kafka not release .deleted file.

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405870#comment-16405870
 ] 

Manikumar commented on KAFKA-6689:
--

There temp files will be deleted after file.delete.delay.ms (default: 6) 
milliseconds after file creation.
 file count increasing continuously? Are you observing any exceptions in logs?

> Kafka not release .deleted file.
> 
>
> Key: KAFKA-6689
> URL: https://issues.apache.org/jira/browse/KAFKA-6689
> Project: Kafka
>  Issue Type: Bug
>  Components: config, controller, log
>Affects Versions: 0.10.1.1
> Environment: 3 Kafka Broker running on CentOS6.9(rungi3 VMs) 
>Reporter: A
>Priority: Critical
>  Labels: newbie
> Fix For: 0.10.1.1
>
>
>          After Kafka cleaned log  .timeindex / .index files based on topic 
> retention. I can
> still lsof a lot of .index.deleted and .timeindex.deleted files. 
>        We have 3 Brokers on 3 VMs ,  It's happen only 2 brokers.  
> [brokeer-01 ~]$ lsof -p 24324 | grep deleted | wc -l
> 28
> [broker-02 ~]$ lsof -p 12131 | grep deleted | wc -l
> 14599
> [broker-03 ~]$ lsof -p 3349 | grep deleted | wc -l
> 14523
>  
> Configuration on 3 broker is same.  (Rolllog every hour, Retention time 11 
> Hours)
>  * INFO KafkaConfig values:
>  advertised.host.name = null
>  advertised.listeners = PLAINTEXT://Broker-02:9092
>  advertised.port = null
>  authorizer.class.name =
>  auto.create.topics.enable = true
>  auto.leader.rebalance.enable = true
>  background.threads = 10
>  broker.id = 2
>  broker.id.generation.enable = true
>  broker.rack = null
>  compression.type = producer
>  connections.max.idle.ms = 60
>  controlled.shutdown.enable = true
>  controlled.shutdown.max.retries = 3
>  controlled.shutdown.retry.backoff.ms = 5000
>  controller.socket.timeout.ms = 3
>  default.replication.factor = 3
>  delete.topic.enable = true
>  fetch.purgatory.purge.interval.requests = 1000
>  group.max.session.timeout.ms = 30
>  group.min.session.timeout.ms = 6000
>  host.name =
>  inter.broker.protocol.version = 0.10.1-IV2
>  leader.imbalance.check.interval.seconds = 300
>  leader.imbalance.per.broker.percentage = 10
>  listeners = null
>  log.cleaner.backoff.ms = 15000
>  log.cleaner.dedupe.buffer.size = 134217728
>  log.cleaner.delete.retention.ms = 8640
>  log.cleaner.enable = true
>  log.cleaner.io.buffer.load.factor = 0.9
>  log.cleaner.io.buffer.size = 524288
>  log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
>  log.cleaner.min.cleanable.ratio = 0.5
>  log.cleaner.min.compaction.lag.ms = 0
>  log.cleaner.threads = 1
>  log.cleanup.policy = [delete]
>  log.dir = /tmp/kafka-logs
>  log.dirs = /data/appdata/kafka/data
>  log.flush.interval.messages = 9223372036854775807
>  log.flush.interval.ms = null
>  log.flush.offset.checkpoint.interval.ms = 6
>  log.flush.scheduler.interval.ms = 9223372036854775807
>  log.index.interval.bytes = 4096
>  log.index.size.max.bytes = 10485760
>  log.message.format.version = 0.10.1-IV2
>  log.message.timestamp.difference.max.ms = 9223372036854775807
>  log.message.timestamp.type = CreateTime
>  log.preallocate = false
>  log.retention.bytes = -1
>  log.retention.check.interval.ms = 30
>  log.retention.hours = 11
>  log.retention.minutes = 660
>  log.retention.ms = 3960
>  log.roll.hours = 1
>  log.roll.jitter.hours = 0
>  log.roll.jitter.ms = null
>  log.roll.ms = null
>  log.segment.bytes = 1073741824
>  log.segment.delete.delay.ms = 6
>  max.connections.per.ip = 2147483647
>  max.connections.per.ip.overrides =
>  message.max.bytes = 112
>  metric.reporters = []
>  metrics.num.samples = 2
>  metrics.sample.window.ms = 3
>  min.insync.replicas = 2
>  num.io.threads = 16
>  num.network.threads = 16
>  num.partitions = 10
>  num.recovery.threads.per.data.dir = 3
>  num.replica.fetchers = 1
>  offset.metadata.max.bytes = 4096
>  offsets.commit.required.acks = -1
>  offsets.commit.timeout.ms = 5000
>  offsets.load.buffer.size = 5242880
>  offsets.retention.check.interval.ms = 60
>  offsets.retention.minutes = 1440
>  offsets.topic.compression.codec = 0
>  offsets.topic.num.partitions = 50
>  offsets.topic.replication.factor = 3
>  offsets.topic.segment.bytes = 104857600
>  port = 9092
>  principal.builder.class = class 
> org.apache.kafka.common.security.auth.DefaultPrincipalBuilder
>  producer.purgatory.purge.interval.requests = 1000
>  queued.max.requests = 3
>  quota.consumer.default = 9223372036854775807
>  quota.producer.default = 9223372036854775807
>  quota.window.num = 11
>  quota.window.size.seconds = 1
>  replica.fetch.backoff.ms = 1000
>  replica.fetch.max.bytes = 1048576
>  replica.fetch.min.bytes = 1
>  replica.fetch.response.max.bytes = 10485760
>  replica.fetch.wait.max.ms = 500
>  

[jira] [Assigned] (KAFKA-6680) Fix config initialization in DynamicBrokerConfig

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar reassigned KAFKA-6680:


 Assignee: Manikumar
Fix Version/s: 1.2.0

PR: https://github.com/apache/kafka/pull/4731

> Fix config initialization in DynamicBrokerConfig
> 
>
> Key: KAFKA-6680
> URL: https://issues.apache.org/jira/browse/KAFKA-6680
> Project: Kafka
>  Issue Type: Bug
>Reporter: Manikumar
>Assignee: Manikumar
>Priority: Major
> Fix For: 1.2.0
>
>
> Below issues observed while testing dynamic config update feature
> 1. {{kafkaConfig}} doesn't get updated during {{initialize}} if there are no 
> dynamic configs defined in ZK.
> 2.  update DynamicListenerConfig.validateReconfiguration() to check new 
> Listeners must be subset of listener map



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KAFKA-6680) Fix config initialization in DynamicBrokerConfig

[Manikumar (JIRA)]

Manikumar created KAFKA-6680:


 Summary: Fix config initialization in DynamicBrokerConfig
 Key: KAFKA-6680
 URL: https://issues.apache.org/jira/browse/KAFKA-6680
 Project: Kafka
  Issue Type: Bug
Reporter: Manikumar


Below issues observed while testing dynamic config update feature

1. {{kafkaConfig}} doesn't get updated during {{initialize}} if there are no 
dynamic configs defined in ZK.

2.  update DynamicListenerConfig.validateReconfiguration() to check new 
Listeners must be subset of listener map



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6338) java.lang.NoClassDefFoundError: org/apache/kafka/common/network/LoginType

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6338.
--
Resolution: Not A Problem

Closing this  in-favor of RANGER-1964

> java.lang.NoClassDefFoundError: org/apache/kafka/common/network/LoginType
> -
>
> Key: KAFKA-6338
> URL: https://issues.apache.org/jira/browse/KAFKA-6338
> Project: Kafka
>  Issue Type: Test
>Affects Versions: 1.0.0
>Reporter: Ronald van de Kuil
>Priority: Minor
>
> I have just setup a kerberized Kafa cluster with Ranger 0.7.1 and Kafka 
> 1.0.0. 
> It all seems to work fine as I see that authorisation policies are enforced 
> and auditlogging is present.
> On startup of a kafka server I see a stack trace but it does not seem to 
> matter.
> My wish is to keep the logs tidy and free of false alerts.
> I wonder whether I have an issue somewhere.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-6596) Why we can not see any consumer offset produce rate in kafka manager

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-6596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-6596.
--
Resolution: Not A Problem

I suggest you to reach out kafka-manager mailing list/github community for any 
help on kafka-manager.  

> Why we can not see any consumer offset produce rate in kafka manager 
> -
>
> Key: KAFKA-6596
> URL: https://issues.apache.org/jira/browse/KAFKA-6596
> Project: Kafka
>  Issue Type: Bug
>Reporter: Rajendra Jangir
>Priority: Major
>
> I am using kafka v0.10.2  and kafka manager for monitoring.
> But I am not able to see any consumer offset producer rate.
> Please help me out. Thanks in advance



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-6613) The controller shouldn't stop partition reassignment after an exception is being thrown

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-6613?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16403454#comment-16403454
 ] 

Manikumar commented on KAFKA-6613:
--

This might have fixed in KAFKA-5758

> The controller shouldn't stop partition reassignment after an exception is 
> being thrown
> ---
>
> Key: KAFKA-6613
> URL: https://issues.apache.org/jira/browse/KAFKA-6613
> Project: Kafka
>  Issue Type: Bug
>  Components: admin, config, controller, core
>Affects Versions: 0.11.0.2
>Reporter: chandra kasiraju
>Priority: Major
> Fix For: 0.11.0.2
>
>
> I issued a partition reassignment command . It created the following entries 
> in the zookeeper .
> But the entry never gets deleted because the partition reassigment hangs gets 
> some exceptions in kafka logs . After that no matter how many hours the 
> movement of partitions to other brokers never happens .
>  
> *Path in Zookeeper*
> get /admin/reassign_partitions
> {"version":1,"partitions":[{"topic":"__consumer_offsets","partition":44,"replicas":([1003,1001,1004,1002]},\{"topic":"683ad5e0-3775-4adc-ab55-82fda0761ba9_newTopic9","partition":0,"replicas":[1003,1004,1001,1002]},\{"topic":"683ad5e0-3775-4adc-ab55-82fda0761ba9_newTopic1","partition":0,"replicas":[1003,1004,1001,1002]},\{"topic":"__CruiseControlMetrics","partition":0,"replicas":[1002,1001,1004,1003]},\{"topic":"b1c39c85-aee5-4ea0-90a1-9fc7eedc635b_topic","partition":0,"replicas":[1003,1004,1001,1002]},\{"topic":"88ec4bd5-e149-4c98-8e8e-952e86ba5fae_topic","partition":4,"replicas":[1002,1004,1003,1001]},\{"topic":"c8c56723-73a5-4a37-93bf-b8ecaf766429_topic","partition":4,"replicas":[1002,1003,1004,1001]},\{"topic":"683ad5e0-3775-4adc-ab55-82fda0761ba9_newTopic9","partition":4,"replicas":[1002,1004,1003,1001]},\{"topic":"b1c39c85-aee5-4ea0-90a1-9fc7eedc635b_topic","partition":4,"replicas":[1003,1001,1004,1002]},\{"topic":"9db0cad2-69f8-4e85-b663-cd3987bd90fe_topic","partition":0,"replicas":[1003,1001,1004]},\{"topic":"683ad5e0-3775-4adc-ab55-82fda0761ba9_topic","partition":1,"replicas":[1003,1004,1001,1002]}]}
> cZxid = 0x552f8
> ctime = Tue Mar 06 01:27:54 UTC 2018
> mZxid = 0x55359
> mtime = Tue Mar 06 01:28:06 UTC 2018
> pZxid = 0x552f8
> cversion = 0
> dataVersion = 13
> aclVersion = 0
> ephemeralOwner = 0x0
> dataLength = 1114
> numChildren = 0
>  
>  
> *Exception* 
>  
> ERROR [KafkaApi-1002] Error when handling request 
> \{replica_id=1005,max_wait_time=500,min_bytes=1,max_bytes=10485760,isolation_level=0,topics=[{topic=__consumer_offsets,partitions=[{partition=41,fetch_offset=0,log_start_offset=0,max_bytes=1048576}]}]}
>  (kafka.server.KafkaApis)
> kafka.common.NotAssignedReplicaException: Leader 1002 failed to record 
> follower 1005's position 0 since the replica is not recognized to be one of 
> the assigned replicas 1001,1002,1004 for partition __consumer_offsets-41.
>  at kafka.cluster.Partition.updateReplicaLogReadResult(Partition.scala:274)
>  at 
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:1092)
>  at 
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:1089)
>  at 
> scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:59)
>  at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:48)
>  at 
> kafka.server.ReplicaManager.updateFollowerLogReadResults(ReplicaManager.scala:1089)
>  at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:623)
>  at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:606)
>  at kafka.server.KafkaApis.handle(KafkaApis.scala:98)
>  at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:66)
>  at java.lang.Thread.run(Thread.java:745)
>  
>  
>  
> I was expecting it would be recover from that exception move the partitions 
> to other nodes and finally remove the entries in /admin/reassign_partitions 
> after the move has happened.
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-5187) producer.close() times out at 30 seconds

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-5187.
--
Resolution: Not A Problem

Please reopen if the issue still exists.

> producer.close() times out at 30 seconds
> 
>
> Key: KAFKA-5187
> URL: https://issues.apache.org/jira/browse/KAFKA-5187
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.1.1
>Reporter: Somnath Choudhuri
>Priority: Major
>
> In the code below, send() is successful and returns immediately. However 
> producer.close() hangs and times out after 30 seconds.
>producer.send(new ProducerRecord(topic, "0", 
> "test string 0")); 
>System.out.println("Send successful");
>
>long start_time = System.currentTimeMillis();
>producer.close();
>long end_time = System.currentTimeMillis();
>
>System.out.println("Time spent in closing Producer: " + 
> (end_time - start_time));



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-4782) change chroot for a kafka instance

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4782.
--
Resolution: Not A Problem

> change chroot for a kafka instance
> --
>
> Key: KAFKA-4782
> URL: https://issues.apache.org/jira/browse/KAFKA-4782
> Project: Kafka
>  Issue Type: Improvement
>Reporter: polarbear
>Priority: Major
>
> Hello
> I am a beginner of the kafka.
> Our kafka is "kafka_2.11-0.10.0.1.jar.asc", should be 0.10.0.1.
> chroot is default "/".
> Our user ask us to change the chroot from "/" to "/kafka".
> I change the parameter zookeeper.connect to add /kafka, but we cannot find  
> the old topic in "/kafka", so I think it should be a new instance.
> Which tool can we use to migrate the topic from the old "/" to new "/kafka"?
> Thank you



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-3553) Issue with getting data of next offset after restarting the consumer and producer is always running

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-3553.
--
Resolution: Auto Closed

{color:#00}Closing inactive issue. Please raise issue with kafka-python 
project,  if the issue still exists with newer versions.{color}
 

> Issue with getting data of next offset after restarting the consumer and 
> producer is always running
> ---
>
> Key: KAFKA-3553
> URL: https://issues.apache.org/jira/browse/KAFKA-3553
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, offset manager
>Affects Versions: 0.9.0.0
> Environment: operating system : ubuntu 14.04
> using kafka-python
>Reporter: pooja sadashiv deokar
>Assignee: Neha Narkhede
>Priority: Major
>  Labels: performance
>
> I am pushing data to kafka topic after every 1 second in python. And written 
> consumer to fetch data from topic with consumer timeout as 500 ms and 
> enable_auto_commit as false.
> Following are the scripts:
> 1) prod.py
> from kafka import KafkaClient,SimpleConsumer
> from kafka.producer import SimpleProducer
> import random, time
> kafka =  KafkaClient("localhost:9092")
> producer = SimpleProducer(kafka)
> i=0
> while(True):
>   l1 = ['Rahul','Narendra','NaMo','ManMohan','Sonia']
>   msg = str(i) + ',' + str(random.choice(l1))
>   print('putting data : '+msg) 
>   producer.send_messages("test", msg)
>   i = i + 1
>   time.sleep(1)
> 2) con.py
> from kafka import KafkaConsumer
> consumer = 
> KafkaConsumer('test',bootstrap_servers=['localhost:9092'],consumer_timeout_ms 
> = 500, enable_auto_commit = False)
> for message in consumer:
> print ("%s:%d:%d: key=%s value=%s" % (message.topic, message.partition,
>   message.offset, message.key,
>   message.value))
> consumer.commit()
> My producer is continuously running. And I am  running consumer again when it 
> stops. For first time consumer works well and gets data for me on time. But 
> after first time its taking minimum 1 minute to maximum  5 minutes (or above 
> 5 min also) to get next data. It should give me data after 1 second or 
> maximum after 5 seconds. But its giving problem :( . 
> Also 1 thing which I observe is If I wait for starting consumer about 1 min 
> then data comes as expected.
> Please correct me if I am going wrong.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-3550) Broker does not honor MetadataRequest api version; always returns v0 MetadataResponse

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-3550.
--
Resolution: Fixed

Closing as per above comment.

> Broker does not honor MetadataRequest api version; always returns v0 
> MetadataResponse
> -
>
> Key: KAFKA-3550
> URL: https://issues.apache.org/jira/browse/KAFKA-3550
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.0, 0.8.1.1, 0.8.2.2, 0.9.0.1
>Reporter: Dana Powers
>Assignee: Grant Henke
>Priority: Major
>
> To reproduce:
> Send a MetadataRequest (api key 3) with incorrect api version (e.g., 1234).
> The expected behavior is for the broker to reject the request as unrecognized.
> Broker (incorrectly) responds with MetadataResponse v0.
> The problem here is that any request for a "new" MetadataRequest (i.e., 
> KIP-4) sent to an old broker will generate an incorrect response.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-3238) Deadlock Mirrormaker consumer not fetching any messages

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-3238.
--
Resolution: Auto Closed

Closing inactive issue.  old consumer client is no longer supported.  In newer 
releases, we can use mirror maker with Java consumer . Please open a JIRA if 
the issue still exists in newer releases.

> Deadlock Mirrormaker consumer not fetching any messages
> ---
>
> Key: KAFKA-3238
> URL: https://issues.apache.org/jira/browse/KAFKA-3238
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0
>Reporter: Rekha Joshi
>Priority: Critical
>
> Hi,
> We have been seeing consistent issue mirroring between our DataCenters 
> happening randomly.Below are the details.
> Thanks
> Rekha
> {code}
> Source: AWS (13 Brokers)
> Destination: OTHER-DC (20 Brokers)
> Topic: mirrortest (source: 260 partitions, destination: 200 partitions)
> Connectivity: AWS Direct Connect (max 6Gbps)
> Data details: Source is receiving 40,000 msg/sec, each message is around
> 5KB
> Mirroring
> 
> Node: is at our OTHER-DC (24 Cores, 48 GB Memory)
> JVM Details: 1.8.0_51, -Xmx8G -Xms8G -XX:PermSize=96m -XX:MaxPermSize=96m
> -XX:+UseG1GC -XX:MaxGCPauseMillis=20
> -XX:InitiatingHeapOccupancyPercent=35
> Launch script: kafka.tools.MirrorMaker --consumer.config
> consumer.properties --producer.config producer.properties --num.producers
> 1 --whitelist mirrortest --num.streams 1 --queue.size 10
> consumer.properties
> ---
> zookeeper.connect=
> group.id=KafkaMirror
> auto.offset.reset=smallest
> fetch.message.max.bytes=900
> zookeeper.connection.timeout.ms=6
> rebalance.max.retries=4
> rebalance.backoff.ms=5000
> producer.properties
> --
> metadata.broker.list=
> partitioner.class=
> producer.type=async
> When we start the mirroring job everything works fine as expected,
> Eventually we hit an issue where the job stops consuming no more.
> At this stage:
> 1. No Error seen in the mirrormaker logs
> 2. consumer threads are not fetching any messages and we see thread dumps
> as follows:
> "ConsumerFetcherThread-KafkaMirror_1454375262613-7b760d87-0-26" - Thread
> t@73
> java.lang.Thread.State: WAITING
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <79b6d3ce> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awa
> i
> t(AbstractQueuedSynchronizer.java:2039)
> at
> java.util.concurrent.LinkedBlockingQueue.put(LinkedBlockingQueue.java:350
> )
> at kafka.consumer.PartitionTopicInfo.enqueue(PartitionTopicInfo.scala:60)
> at
> kafka.consumer.ConsumerFetcherThread.processPartitionData(ConsumerFetcher
> T
> hread.scala:49)
> at
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfu
> n
> $apply$mcV$sp$2.apply(AbstractFetcherThread.scala:128)
> at
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1$$anonfu
> n
> $apply$mcV$sp$2.apply(AbstractFetcherThread.scala:109)
> at scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:224)
> at
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:403)
> at
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply$m
> c
> V$sp(AbstractFetcherThread.scala:109)
> at
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(A
> b
> stractFetcherThread.scala:109)
> at
> kafka.server.AbstractFetcherThread$$anonfun$processFetchRequest$1.apply(A
> b
> stractFetcherThread.scala:109)
> at kafka.utils.Utils$.inLock(Utils.scala:535)
> at
> kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThr
> e
> ad.scala:108)
> at
> kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:86)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
> Locked ownable synchronizers:
> - locked <199dc92d> (a
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> 3. Producer stops producing, in trace mode we notice it's handling 0
> events and Thread dump as follows:
> "ProducerSendThread--0" - Thread t@53
>   java.lang.Thread.State: RUNNABLE
> at sun.nio.ch.FileDispatcherImpl.writev0(Native Method)
> at sun.nio.ch.SocketDispatcher.writev(SocketDispatcher.java:51)
> at sun.nio.ch.IOUtil.write(IOUtil.java:148)
> at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:504)
> - locked <5ae2fc40> (a java.lang.Object)
> at java.nio.channels.SocketChannel.write(SocketChannel.java:502)
> at
> kafka.network.BoundedByteBufferSend.writeTo(BoundedByteBufferSend.scala:5
> 6
> )
> at kafka.network.Send$class.writeCompletely(Transmission.scala:75)
> at
> 

[jira] [Resolved] (KAFKA-1106) HighwaterMarkCheckpoint failure puting broker into a bad state

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1106?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-1106.
--
Resolution: Auto Closed

Relevant parts of the code is changed an similar exceptions are fixed in the 
new versions.  Please open a new issue if the still exists.

> HighwaterMarkCheckpoint failure puting broker into a bad state
> --
>
> Key: KAFKA-1106
> URL: https://issues.apache.org/jira/browse/KAFKA-1106
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.0
>Reporter: David Lao
>Priority: Major
>  Labels: reliability
> Attachments: KAFKA-1106-patch, kafka.log
>
>
> I'm encountering a case where broker get stuck due to HighwaterMarkCheckpoint 
> failing to recover from reading what appear to be corrupted isr entries. Once 
> in this state, leader election can never succeed and hence stalling the 
> entire cluster. 
> Please see the detailed stack trace from the attached log.  Perhaps failing 
> fast when HighwaterMarkCheckpoint fails to read would force the broker to 
> restart and recover.  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KAFKA-4027) Leader for a cetain partition unavailable forever

[Manikumar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16397208#comment-16397208
 ] 

Manikumar commented on KAFKA-4027:
--

More details about similar exception are here: 
https://issues.apache.org/jira/browse/KAFKA-5758

> Leader for a cetain partition unavailable forever
> -
>
> Key: KAFKA-4027
> URL: https://issues.apache.org/jira/browse/KAFKA-4027
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: tu nguyen khac
>Priority: Major
>
> I have a cluster of brokers ( 9 box) , i 'm naming it from 0 --> 8 . 
> Yesterday some servers went down ( hard reset ) i regularly restart these 
> server ( down servers ) but after that some topics cannot assign leader 
> i checked server log and retrieved these logging : 
> kafka.common.NotAssignedReplicaException: Leader 1 failed to record follower 
> 6's position -1 since the replica is not recognized to be one of the assigned 
> replicas 1 for partition [tos_htv3tv.com,31].
>   at 
> kafka.cluster.Partition.updateReplicaLogReadResult(Partition.scala:251)
>   at 
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:864)
>   at 
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:861)
>   at 
> scala.collection.immutable.HashMap$HashMap1.foreach(HashMap.scala:221)
>   at 
> scala.collection.immutable.HashMap$HashTrieMap.foreach(HashMap.scala:428)
>   at 
> kafka.server.ReplicaManager.updateFollowerLogReadResults(ReplicaManager.scala:861)
>   at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:470)
>   at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:496)
>   at kafka.server.KafkaApis.handle(KafkaApis.scala:77)
>   at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
>   at java.lang.Thread.run(Thread.java:745)
> i tried to run Prefered Leader but it didn't work ( some partitions has node 
> leader ) :(



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Resolved] (KAFKA-4984) Unable to produce or consume when enabling authentication SASL/Kerberos

[Manikumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar resolved KAFKA-4984.
--
Resolution: Cannot Reproduce

{color:#00}Closing inactive issue. Please reopen if you think the issue 
still exists{color}
 

> Unable to produce or consume when enabling authentication SASL/Kerberos
> ---
>
> Key: KAFKA-4984
> URL: https://issues.apache.org/jira/browse/KAFKA-4984
> Project: Kafka
>  Issue Type: Bug
> Environment: Ubuntu 16.04LTS running in VirtualBox
>Reporter: Ait haj Slimane
>Priority: Critical
> Attachments: Screenshot from 2017-03-30 15-36-30.png, logKafka.txt, 
> logZookeeper.txt
>
>
> I have a problem while trying to produce or consume on kerberos enabled 
> cluster.
> I launched a single broker and a console producer,
> using the SASL authentication between producer and broker.
> When i run the producer ,I got the result attached below
> Any advice on what can cause the problem.
> Thanks!
> 
> configuration used:
> server.properties:
> listeners=SASL_PLAINTEXT://kafka.example.com:9092
> security.inter.broker.protocol=SASL_PLAINTEXT
> sasl.mechanism.inter.broker.protocol=GSSAPI
> sasl.enabled.mechanism=GSSAPI
> sasl.kerberos.service.name=kafka
> producer.properties
> bootstrap.servers=kafka.example.com:9092
> sasl.kerberos.service.name=kafka
> security.protocol=SASL_PLAINTEXT
> kafka_client_jaas.conf
> KafkaClient {
> com.sun.security.auth.module.Krb5LoginModule required
> useKeyTab=true
> storeKey=true
> keyTab="/etc/kafka/keytabs/kafkaclient.keytab"
> principal="kafkaclient/kafka.example@example.com";
> };



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)