[jira] [Created] (KAFKA-2874) zookeeper-server-stop.sh may fail to shutdown ZK and/or may stop unrelated processes

[Michael Noll (JIRA)]

Michael Noll created KAFKA-2874:
---

 Summary: zookeeper-server-stop.sh may fail to shutdown ZK and/or 
may stop unrelated processes
 Key: KAFKA-2874
 URL: https://issues.apache.org/jira/browse/KAFKA-2874
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.1, 0.9.0.0
Reporter: Michael Noll


We have run into the problem of ZK not shutting down properly when the included 
{{bin/zookeeper-server-stop.sh}} is being used.  In a nutshell, ZK may not 
shutdown when you send only a SIGINT;  instead, there are certain situations 
(which unfortunately are a bit hard to pin down) where for some reason only a 
SIGTERM will shut ZK down.

Similarly, the current 
[zookeeper-server-stop|https://github.com/apache/kafka/blob/trunk/bin/zookeeper-server-stop.sh#L16]
 script uses a very broad grep statement (`grep -i zookeeper`) that might cause 
the script to shutdown other processes on the machine as well, think: 
collateral damage.

For reference this is the current command to stop ZK:

{code}
ps ax | grep -i 'zookeeper' | grep -v grep | awk '{print $1}' | xargs kill 
-SIGINT
{code}

Disclaimer: I don't know whether there are any unwanted side effects of 
switching from SIGINT to SIGTERM.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2874: shutdown ZK process reliably

[miguno]

GitHub user miguno opened a pull request:

https://github.com/apache/kafka/pull/573

KAFKA-2874: shutdown ZK process reliably



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/miguno/kafka KAFKA-2874

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/573.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #573


commit 8775d9bae5c1abf490cbe0c256944544aeda7420
Author: Michael G. Noll 
Date:   2015-11-23T09:10:25Z

KAFKA-2874: shutdown ZK process reliably




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2874) zookeeper-server-stop.sh may fail to shutdown ZK and/or may stop unrelated processes

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15021804#comment-15021804
 ] 

ASF GitHub Bot commented on KAFKA-2874:
---

GitHub user miguno opened a pull request:

https://github.com/apache/kafka/pull/573

KAFKA-2874: shutdown ZK process reliably



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/miguno/kafka KAFKA-2874

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/573.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #573


commit 8775d9bae5c1abf490cbe0c256944544aeda7420
Author: Michael G. Noll 
Date:   2015-11-23T09:10:25Z

KAFKA-2874: shutdown ZK process reliably




> zookeeper-server-stop.sh may fail to shutdown ZK and/or may stop unrelated 
> processes
> 
>
> Key: KAFKA-2874
> URL: https://issues.apache.org/jira/browse/KAFKA-2874
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1, 0.9.0.0
>Reporter: Michael Noll
>
> We have run into the problem of ZK not shutting down properly when the 
> included {{bin/zookeeper-server-stop.sh}} is being used.  In a nutshell, ZK 
> may not shutdown when you send only a SIGINT;  instead, there are certain 
> situations (which unfortunately are a bit hard to pin down) where for some 
> reason only a SIGTERM will shut ZK down.
> Similarly, the current 
> [zookeeper-server-stop|https://github.com/apache/kafka/blob/trunk/bin/zookeeper-server-stop.sh#L16]
>  script uses a very broad grep statement (`grep -i zookeeper`) that might 
> cause the script to shutdown other processes on the machine as well, think: 
> collateral damage.
> For reference this is the current command to stop ZK:
> {code}
> ps ax | grep -i 'zookeeper' | grep -v grep | awk '{print $1}' | xargs kill 
> -SIGINT
> {code}
> Disclaimer: I don't know whether there are any unwanted side effects of 
> switching from SIGINT to SIGTERM.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: Introduce `installAll` and accept major...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/574

MINOR: Introduce `installAll` and accept major instead of full Scala version

We can take advantage of the fact that major Scala versions are binary 
compatible (since 2.10) to make the build a little more user-friendly.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
install-all-and-major-instead-of-full-version

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/574.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #574


commit 3795f1e3bc8e6a367ef831c68e63e41b2cf0f915
Author: Ismael Juma 
Date:   2015-11-23T13:04:35Z

Introduce `installAll` and make it possible for users to pass a major 
version instead of a full version




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2334) Prevent HW from going back during leader failover

[jin xing (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15022117#comment-15022117
 ] 

jin xing commented on KAFKA-2334:
-

after receiving LeaderAndIsrRequest, Broker B2 will finally call " 
Partition::makeLeader", part of code is as below:
...
 zkVersion = leaderAndIsr.zkVersion
  leaderReplicaIdOpt = Some(localBrokerId)
  // construct the high watermark metadata for the new leader replica
  val newLeaderReplica = getReplica().get
  newLeaderReplica.convertHWToLocalOffsetMetadata()
  // reset log end offset for remote replicas
  assignedReplicas.foreach(r => if (r.brokerId != localBrokerId) 
r.logEndOffset = LogOffsetMetadata.UnknownOffsetMetadata)
  // we may need to increment high watermark since ISR could be down to 1
  maybeIncrementLeaderHW(newLeaderReplica)
  if (topic == OffsetManager.OffsetsTopicName)
offsetManager.loadOffsetsFromLog(partitionId)
...
I can tell Broker B2 will first set 'leaderReplicaIdOpt = Some(localBrokerId)', 
and then try to update high watermark;
by setting leaderReplicaIdOpt, Broker B2 will be available for consumer(if the 
consumer send fetchReqeust, there will be no NotLeaderForPartitionException);
In the short interval which after 'leaderReplicaIdOpt = Some(localBrokerId)' 
and before setting up hw, what the consumer get is the "gone back" hw;
If my understanding is wright, just reverse the order of setting up 
leaderReplicaIdOpt and updating high watermark will fix this issue;
am I wrong ?

> Prevent HW from going back during leader failover 
> --
>
> Key: KAFKA-2334
> URL: https://issues.apache.org/jira/browse/KAFKA-2334
> Project: Kafka
>  Issue Type: Bug
>  Components: replication
>Affects Versions: 0.8.2.1
>Reporter: Guozhang Wang
>Assignee: Neha Narkhede
> Fix For: 0.10.0.0
>
>
> Consider the following scenario:
> 0. Kafka use replication factor of 2, with broker B1 as the leader, and B2 as 
> the follower. 
> 1. A producer keep sending to Kafka with ack=-1.
> 2. A consumer repeat issuing ListOffset request to Kafka.
> And the following sequence:
> 0. B1 current log-end-offset (LEO) 0, HW-offset 0; and same with B2.
> 1. B1 receive a ProduceRequest of 100 messages, append to local log (LEO 
> becomes 100) and hold the request in purgatory.
> 2. B1 receive a FetchRequest starting at offset 0 from follower B2, and 
> returns the 100 messages.
> 3. B2 append its received message to local log (LEO becomes 100).
> 4. B1 receive another FetchRequest starting at offset 100 from B2, knowing 
> that B2's LEO has caught up to 100, and hence update its own HW, and 
> satisfying the ProduceRequest in purgatory, and sending the FetchResponse 
> with HW 100 back to B2 ASYNCHRONOUSLY.
> 5. B1 successfully sends the ProduceResponse to the producer, and then fails, 
> hence the FetchResponse did not reach B2, whose HW remains 0.
> From the consumer's point of view, it could first see the latest offset of 
> 100 (from B1), and then see the latest offset of 0 (from B2), and then the 
> latest offset gradually catch up to 100.
> This is because we use HW to guard the ListOffset and 
> Fetch-from-ordinary-consumer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: Bump version to 0.9.1.0-SNAPSHOT

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/575

MINOR: Bump version to 0.9.1.0-SNAPSHOT

Since we created the 0.9.0 branch a while back.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka bump-to-0.9.1.0-SNAPSHOT

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/575.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #575


commit c278d99726dcca4d416cd2867487880c2e0dc23a
Author: Ismael Juma 
Date:   2015-11-23T14:05:58Z

Bump version to 0.9.1.0-SNAPSHOT

Since we created the 0.9.0 branch a while back.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-2875) Class path contains multiple SLF4J bindings warnings when using scripts under bin

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-2875:
--

 Summary: Class path contains multiple SLF4J bindings warnings when 
using scripts under bin
 Key: KAFKA-2875
 URL: https://issues.apache.org/jira/browse/KAFKA-2875
 Project: Kafka
  Issue Type: Bug
  Components: tools
Affects Versions: 0.9.0.0
Reporter: Ismael Juma


This adds a lot of noise when running the scripts, see example when running 
kafka-console-producer.sh:

{code}
~/D/s/kafka-0.9.0.0-src ❯❯❯ ./bin/kafka-console-producer.sh --topic topic 
--broker-list localhost:9092 ⏎
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in 
[jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/tools/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/api/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/runtime/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/file/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in 
[jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/json/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: Bump version to 0.9.1.0-SNAPSHOT

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/575


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: 0.9.0.0 RC4

[Ismael Juma]

+1 (non-binding).

Verified source and binary artifacts, ran ./gradlew testAll with JDK 7u80,
quick start on source artifact and Scala 2.11 binary artifact.

On Sat, Nov 21, 2015 at 1:21 AM, Jun Rao  wrote:

> This is the fourth candidate for release of Apache Kafka 0.9.0.0. This a
> major release that includes (1) authentication (through SSL and SASL) and
> authorization, (2) a new java consumer, (3) a Kafka connect framework for
> data ingestion and egression, and (4) quotas. Since this is a major
> release, we will give people a bit more time for trying this out.
>
> Release Notes for the 0.9.0.0 release
>
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, Nov. 23, 6pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS in addition to the md5, sha1
> and sha2 (SHA256) checksum.
>
> * Release artifacts to be voted upon (source and binary):
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/
>
> * Maven artifacts to be voted upon prior to release:
> https://repository.apache.org/content/groups/staging/
>
> * scala-doc
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/scaladoc/
>
> * java-doc
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/javadoc/
>
> * The tag to be voted upon (off the 0.9.0 branch) is the 0.9.0.0 tag
>
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=132943b0f83831132cd46ac961cf6f1c00132565
>
> * Documentation
> http://kafka.apache.org/090/documentation.html
>
> /***
>
> Thanks,
>
> Jun
>

[jira] [Updated] (KAFKA-2875) Class path contains multiple SLF4J bindings warnings when using scripts under bin

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2875:
---
Priority: Minor  (was: Major)

> Class path contains multiple SLF4J bindings warnings when using scripts under 
> bin
> -
>
> Key: KAFKA-2875
> URL: https://issues.apache.org/jira/browse/KAFKA-2875
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.9.0.0
>Reporter: Ismael Juma
>Priority: Minor
>
> This adds a lot of noise when running the scripts, see example when running 
> kafka-console-producer.sh:
> {code}
> ~/D/s/kafka-0.9.0.0-src ❯❯❯ ./bin/kafka-console-producer.sh --topic topic 
> --broker-list localhost:9092 ⏎
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/tools/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/api/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/runtime/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/file/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/json/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2875) Class path contains multiple SLF4J bindings warnings when using scripts under bin

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15022306#comment-15022306
 ] 

Ismael Juma commented on KAFKA-2875:


This only seems to happen when running it from the source distribution or a git 
checkout, it didn't happen when I tested the binary distribution for Scala 
2.11. Reducing severity as a result.

> Class path contains multiple SLF4J bindings warnings when using scripts under 
> bin
> -
>
> Key: KAFKA-2875
> URL: https://issues.apache.org/jira/browse/KAFKA-2875
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.9.0.0
>Reporter: Ismael Juma
>Priority: Minor
>
> This adds a lot of noise when running the scripts, see example when running 
> kafka-console-producer.sh:
> {code}
> ~/D/s/kafka-0.9.0.0-src ❯❯❯ ./bin/kafka-console-producer.sh --topic topic 
> --broker-list localhost:9092 ⏎
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/tools/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/api/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/runtime/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/file/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/json/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-2875) Class path contains multiple SLF4J bindings warnings when using scripts under bin

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15022306#comment-15022306
 ] 

Ismael Juma edited comment on KAFKA-2875 at 11/23/15 3:46 PM:
--

This only seems to happen when running it from the source distribution or a git 
checkout, it didn't happen when I tested the binary distribution for Scala 
2.11. Reducing priority as a result.


was (Author: ijuma):
This only seems to happen when running it from the source distribution or a git 
checkout, it didn't happen when I tested the binary distribution for Scala 
2.11. Reducing severity as a result.

> Class path contains multiple SLF4J bindings warnings when using scripts under 
> bin
> -
>
> Key: KAFKA-2875
> URL: https://issues.apache.org/jira/browse/KAFKA-2875
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.9.0.0
>Reporter: Ismael Juma
>Priority: Minor
>
> This adds a lot of noise when running the scripts, see example when running 
> kafka-console-producer.sh:
> {code}
> ~/D/s/kafka-0.9.0.0-src ❯❯❯ ./bin/kafka-console-producer.sh --topic topic 
> --broker-list localhost:9092 ⏎
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/tools/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/api/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/runtime/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/file/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/json/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-493) High CPU usage on inactive server

[Cosmin Marginean (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-493?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15022364#comment-15022364
 ] 

Cosmin Marginean commented on KAFKA-493:


Hi guys

This seems to have been inactive for a while but I can confirm that I met 
similar conditions (high number of partitions - 256 per topic) and get a high 
CPU usage, but only while having producers/consumers wired. As soon as I 
shutdown the client apps, CPU usage goes considerably low. Just wanted to know 
if this is related or if this is something I should investigate and provide 
diagnosis for in a separate ticket.

Below I managed to identify the threads that generate this high CPU usage.

{code}
"kafka-network-thread-9092-2" #24 prio=5 os_prio=0 tid=0x7f3f085a 
nid=0xe96 runnable [0x7f3ee8ef3000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0xd4d6e6c8> (a sun.nio.ch.Util$2)
- locked <0xd4d6e6b0> (a java.util.Collections$UnmodifiableSet)
- locked <0xd4d77ac8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at kafka.network.Processor.run(SocketServer.scala:320)
at java.lang.Thread.run(Thread.java:745)

"kafka-network-thread-9092-1" #23 prio=5 os_prio=0 tid=0x7f3f08585000 
nid=0xe95 runnable [0x7f3ee8ff4000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0xd4d6e6f8> (a sun.nio.ch.Util$2)
- locked <0xd4d6e6e0> (a java.util.Collections$UnmodifiableSet)
- locked <0xd4d77b58> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at kafka.network.Processor.run(SocketServer.scala:320)
at java.lang.Thread.run(Thread.java:745)

"kafka-network-thread-9092-0" #22 prio=5 os_prio=0 tid=0x7f3f0856a800 
nid=0xe94 runnable [0x7f3ee90f5000]
   java.lang.Thread.State: RUNNABLE
at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
- locked <0xd4d7c4e8> (a sun.nio.ch.Util$2)
- locked <0xd4d7c4d0> (a java.util.Collections$UnmodifiableSet)
- locked <0xd4d77be8> (a sun.nio.ch.EPollSelectorImpl)
at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
at kafka.network.Processor.run(SocketServer.scala:320)
at java.lang.Thread.run(Thread.java:745)

{code}

> High CPU usage on inactive server
> -
>
> Key: KAFKA-493
> URL: https://issues.apache.org/jira/browse/KAFKA-493
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.0
>Reporter: Jay Kreps
> Fix For: 0.10.0.0
>
> Attachments: Kafka-2014-11-10.snapshot.zip, Kafka-sampling1.zip, 
> Kafka-sampling2.zip, Kafka-sampling3.zip, Kafka-trace1.zip, Kafka-trace2.zip, 
> Kafka-trace3.zip, backtraces.txt, stacktrace.txt
>
>
> > I've been playing with the 0.8 branch of Kafka and noticed that idle CPU 
> > usage is fairly high (13% of a 
> > core). Is that to be expected? I did look at the stack, but didn't see 
> > anything obvious. A background 
> > task?
> > I wanted to mention how I am getting into this state. I've set up two 
> > machines with the latest 0.8 
> > code base and am using a replication factor of 2. On starting the brokers 
> > there is no idle CPU activity. 
> > Then I run a test that essential does 10k publish operations followed by 
> > immediate consume operations 
> > (I was measuring latency). Once this has run the kafka nodes seem to 
> > consistently be consuming CPU 
> > essentially forever.
> hprof results:
> THREAD START (obj=53ae, id = 24, name="RMI TCP Accept-0", 
> group="system")
> THREAD START (obj=53ae, id = 25, name="RMI TCP Accept-", 
> group="system")
> THREAD START (obj=53ae, id = 26, name="RMI TCP Accept-0", 
> group="system")
> THREAD START (obj=53ae, id = 21, name="main", group="main")
> THREAD START (obj=53ae, id = 27, name="Thread-2", group="main")
> THREAD START (obj=53ae, id = 28

[GitHub] kafka-site pull request: Add 0.9.0.0 to downloads page

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka-site/pull/4

Add 0.9.0.0 to downloads page



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka-site 0.9.0.0-in-downloads

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka-site/pull/4.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #4


commit 56d48ce6b8fa3df3fa40d3843d6267ea4b829c57
Author: Ismael Juma 
Date:   2015-11-23T16:12:03Z

Add 0.9.0.0 to downloads page




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2608) Recommend kafka_2.11 for 0.9.0.0 on the website

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2608?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15022371#comment-15022371
 ] 

ASF GitHub Bot commented on KAFKA-2608:
---

Github user ijuma commented on the pull request:

https://github.com/apache/kafka-site/pull/4#issuecomment-158983344
  
This should only be merged after Kafka 0.9.0.0 is released. It recommends 
the Scala 2.11 version (as per 
https://issues.apache.org/jira/browse/KAFKA-2608).


> Recommend kafka_2.11 for 0.9.0.0 on the website
> ---
>
> Key: KAFKA-2608
> URL: https://issues.apache.org/jira/browse/KAFKA-2608
> Project: Kafka
>  Issue Type: Task
>  Components: website
>Affects Versions: 0.9.0.0
>Reporter: Ismael Juma
>
> Scala 2.11 has been out for 17 months and Scala 2.10 is not being updated 
> anymore. We should recommend the Scala 2.11 version of 0.9.0.0 on the website:
> http://kafka.apache.org/downloads.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka-site pull request: Add 0.9.0.0 to downloads page

[ijuma]

Github user ijuma commented on the pull request:

https://github.com/apache/kafka-site/pull/4#issuecomment-158983344
  
This should only be merged after Kafka 0.9.0.0 is released. It recommends 
the Scala 2.11 version (as per 
https://issues.apache.org/jira/browse/KAFKA-2608).


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-2608) Recommend kafka_2.11 for 0.9.0.0 on the website

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2608?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2608:
---
Assignee: Ismael Juma
Reviewer: Jun Rao
  Status: Patch Available  (was: Open)

> Recommend kafka_2.11 for 0.9.0.0 on the website
> ---
>
> Key: KAFKA-2608
> URL: https://issues.apache.org/jira/browse/KAFKA-2608
> Project: Kafka
>  Issue Type: Task
>  Components: website
>Affects Versions: 0.9.0.0
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>
> Scala 2.11 has been out for 17 months and Scala 2.10 is not being updated 
> anymore. We should recommend the Scala 2.11 version of 0.9.0.0 on the website:
> http://kafka.apache.org/downloads.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: 0.9.0.0 RC4

[Ismael Juma]

On Mon, Nov 23, 2015 at 4:15 PM, hsy...@gmail.com  wrote:

> In http://kafka.apache.org/090/documentation.html#newconsumerconfigs
> partition.assignment.strategy should string, not a list of string?
>

List is correct, I believe, see how it's used in the code:

List assignors = config.getConfiguredInstances(
ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
PartitionAssignor.class);

The documentation for that config needs to be improved though.

Ismael

Re: 0.9.0.0 RC4

[hsy...@gmail.com]

In http://kafka.apache.org/090/documentation.html#newconsumerconfigs
partition.assignment.strategy should string, not a list of string?

On Fri, Nov 20, 2015 at 5:21 PM, Jun Rao  wrote:

> This is the fourth candidate for release of Apache Kafka 0.9.0.0. This a
> major release that includes (1) authentication (through SSL and SASL) and
> authorization, (2) a new java consumer, (3) a Kafka connect framework for
> data ingestion and egression, and (4) quotas. Since this is a major
> release, we will give people a bit more time for trying this out.
>
> Release Notes for the 0.9.0.0 release
>
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, Nov. 23, 6pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS in addition to the md5, sha1
> and sha2 (SHA256) checksum.
>
> * Release artifacts to be voted upon (source and binary):
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/
>
> * Maven artifacts to be voted upon prior to release:
> https://repository.apache.org/content/groups/staging/
>
> * scala-doc
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/scaladoc/
>
> * java-doc
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/javadoc/
>
> * The tag to be voted upon (off the 0.9.0 branch) is the 0.9.0.0 tag
>
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=132943b0f83831132cd46ac961cf6f1c00132565
>
> * Documentation
> http://kafka.apache.org/090/documentation.html
>
> /***
>
> Thanks,
>
> Jun
>

Build failed in Jenkins: kafka-trunk-jdk7 #849

[Apache Jenkins Server]

See 

Changes:

[junrao] MINOR: Bump version to 0.9.1.0-SNAPSHOT

--
[...truncated 1370 lines...]

kafka.log.LogTest > testTruncateTo PASSED

kafka.log.LogTest > testCleanShutdownFile PASSED

kafka.log.OffsetIndexTest > lookupExtremeCases PASSED

kafka.log.OffsetIndexTest > appendTooMany PASSED

kafka.log.OffsetIndexTest > randomLookupTest PASSED

kafka.log.OffsetIndexTest > testReopen PASSED

kafka.log.OffsetIndexTest > appendOutOfOrder PASSED

kafka.log.OffsetIndexTest > truncate PASSED

kafka.log.LogSegmentTest > testRecoveryWithCorruptMessage PASSED

kafka.log.LogSegmentTest > testRecoveryFixesCorruptIndex PASSED

kafka.log.LogSegmentTest > testReadFromGap PASSED

kafka.log.LogSegmentTest > testTruncate PASSED

kafka.log.LogSegmentTest > testReadBeforeFirstOffset PASSED

kafka.log.LogSegmentTest > testCreateWithInitFileSizeAppendMessage PASSED

kafka.log.LogSegmentTest > testChangeFileSuffixes PASSED

kafka.log.LogSegmentTest > testMaxOffset PASSED

kafka.log.LogSegmentTest > testNextOffsetCalculation PASSED

kafka.log.LogSegmentTest > testReadOnEmptySegment PASSED

kafka.log.LogSegmentTest > testReadAfterLast PASSED

kafka.log.LogSegmentTest > testCreateWithInitFileSizeClearShutdown PASSED

kafka.log.LogSegmentTest > testTruncateFull PASSED

kafka.log.CleanerTest > testBuildOffsetMap PASSED

kafka.log.CleanerTest > testSegmentGrouping PASSED

kafka.log.CleanerTest > testCleanSegmentsWithAbort PASSED

kafka.log.CleanerTest > testSegmentGroupingWithSparseOffsets PASSED

kafka.log.CleanerTest > testRecoveryAfterCrash PASSED

kafka.log.CleanerTest > testLogToClean PASSED

kafka.log.CleanerTest > testCleaningWithDeletes PASSED

kafka.log.CleanerTest > testCleanSegments PASSED

kafka.log.CleanerTest > testCleaningWithUnkeyedMessages PASSED

kafka.log.OffsetMapTest > testClear PASSED

kafka.log.OffsetMapTest > testBasicValidation PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testDescribeGroupStable PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatIllegalGeneration 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testDescribeGroupWrongCoordinator PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testDescribeGroupRebalancing 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaderFailureInSyncGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testGenerationIdIncrementsOnRebalance PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testSyncGroupFromIllegalGeneration PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testInvalidGroupId PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testListGroupsIncludesStableGroups PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testHeartbeatDuringRebalanceCausesRebalanceInProgress PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupInconsistentGroupProtocol PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupSessionTimeoutTooLarge PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupSessionTimeoutTooSmall PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testSyncGroupEmptyAssignment 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testCommitOffsetWithDefaultGeneration PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupFromUnchangedLeaderShouldRebalance PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testHeartbeatRebalanceInProgress PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaveGroupUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testListGroupsIncludesRebalancingGroups PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testSyncGroupFollowerAfterLeader PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testCommitOffsetInAwaitingSync 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testJoinGroupWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupUnknownConsumerExistingGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testSyncGroupFromUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupInconsistentProtocolType PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testCommitOffsetFromUnknownGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaveGroupWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testLeaveGroupUnknownConsumerExistingGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupUnknownConsumerNewGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupFromUnchangedFollowerDoesNotRebalance PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testValidJoinGroup 

[GitHub] kafka pull request: Minor: Fix KafkaConsumer Constructor Summary j...

[jholoman]

GitHub user jholoman opened a pull request:

https://github.com/apache/kafka/pull/576

Minor: Fix KafkaConsumer Constructor Summary javadoc



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jholoman/kafka 
minor-consumer-constructor-javadoc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/576.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #576


commit f030066a2f795ea9320de3bd7a5eb2384e7e18d3
Author: jholoman 
Date:   2015-11-23T12:29:37Z

Minor: Fix KafkaConsumer Constructor Summary javadoc




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: 0.9.0.0 RC4

[Guozhang Wang]

I think we should update the release notes to remove the Kafka Streams
tickets, I have marked them as 0.9.1.0.

Guozhang

On Fri, Nov 20, 2015 at 5:21 PM, Jun Rao  wrote:

> This is the fourth candidate for release of Apache Kafka 0.9.0.0. This a
> major release that includes (1) authentication (through SSL and SASL) and
> authorization, (2) a new java consumer, (3) a Kafka connect framework for
> data ingestion and egression, and (4) quotas. Since this is a major
> release, we will give people a bit more time for trying this out.
>
> Release Notes for the 0.9.0.0 release
>
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, Nov. 23, 6pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS in addition to the md5, sha1
> and sha2 (SHA256) checksum.
>
> * Release artifacts to be voted upon (source and binary):
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/
>
> * Maven artifacts to be voted upon prior to release:
> https://repository.apache.org/content/groups/staging/
>
> * scala-doc
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/scaladoc/
>
> * java-doc
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/javadoc/
>
> * The tag to be voted upon (off the 0.9.0 branch) is the 0.9.0.0 tag
>
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=132943b0f83831132cd46ac961cf6f1c00132565
>
> * Documentation
> http://kafka.apache.org/090/documentation.html
>
> /***
>
> Thanks,
>
> Jun
>



-- 
-- Guozhang

Build failed in Jenkins: kafka-trunk-jdk8 #178

[Apache Jenkins Server]

See 

Changes:

[junrao] MINOR: Bump version to 0.9.1.0-SNAPSHOT

--
[...truncated 4428 lines...]
org.apache.kafka.connect.json.JsonConverterTest > timeToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > structToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
testConnectSchemaMetadataTranslation PASSED

org.apache.kafka.connect.json.JsonConverterTest > shortToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > dateToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > doubleToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > timeToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > mapToConnectStringKeys PASSED

org.apache.kafka.connect.json.JsonConverterTest > floatToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > decimalToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > arrayToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
testCacheSchemaToConnectConversion PASSED

org.apache.kafka.connect.json.JsonConverterTest > booleanToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > bytesToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > doubleToConnect PASSED
:connect:runtime:checkstyleMain
:connect:runtime:compileTestJavawarning: [options] bootstrap class path not set 
in conjunction with -source 1.7
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
1 warning

:connect:runtime:processTestResources
:connect:runtime:testClasses
:connect:runtime:checkstyleTest
:connect:runtime:test

org.apache.kafka.connect.util.KafkaBasedLogTest > testReloadOnStart PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testSendAndReadToEnd FAILED
org.junit.ComparisonFailure: expected: but was:
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.kafka.connect.util.KafkaBasedLogTest.testSendAndReadToEnd(KafkaBasedLogTest.java:312)

org.apache.kafka.connect.util.KafkaBasedLogTest > testConsumerError PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testProducerError PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testStartStop PASSED

org.apache.kafka.connect.util.ShutdownableThreadTest > testGracefulShutdown 
PASSED

org.apache.kafka.connect.util.ShutdownableThreadTest > testForcibleShutdown 
PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testListConnectors PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testCreateConnectorNotLeader PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testCreateConnectorExists PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testDeleteConnectorNotLeader PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testListConnectorsNotSynced PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testPutConnectorConfig PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testGetConnectorTaskConfigs PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testGetConnectorTaskConfigsConnectorNotFound PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testPutConnectorTaskConfigs PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testPutConnectorTaskConfigsConnectorNotFound PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testDeleteConnector PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testGetConnector PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testDeleteConnectorNotFound PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testCreateConnector PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testGetConnectorConfig PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testGetConnectorConfigConnectorNotFound PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testListConnectorsNotLeader PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testPollsInBackground PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > testCommit PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testCommitTaskFlushFailure PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testCommitTaskSuccessAndFlushFailure PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testCommitConsumerFailure PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > testCommitTimeout 
PASSED

org.apache.kafka.co

kafka java consumer not consuming messsages produced by remote client

[Kudumula, Surender]

Hi all
I have a remote java producer producing messages to the same topic where my 
local kafka java client is subscribed to but it doesn't consume any messages 
whereas if I ran the consumer from command line I can consume the messages 
produced by remote client.
The following command line consumer works but my kafka java consumer doesn't 
consume messages produced by remote client but when I produce messages from 
command line in my local machine where kafka client is subscribed to the same 
topic it works and consumes all the messages.
bin/kafka-console-consumer.sh --zookeeper ap3.apdomain.com:2181 --topic 
RequestPdfa --security-protocol PLAINTEXTSASL --from-beginning

Any ideas please?

Regards

Surender

[jira] [Commented] (KAFKA-2873) o.a.k.connect.util.KafkaBasedLogTest failure in 0.9 RC4

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15022533#comment-15022533
 ] 

Jason Gustafson commented on KAFKA-2873:


Duplicate of KAFKA-2667?

> o.a.k.connect.util.KafkaBasedLogTest failure in  0.9 RC4
> 
>
> Key: KAFKA-2873
> URL: https://issues.apache.org/jira/browse/KAFKA-2873
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>
> Saw the following test failure when run unit test on RC4.
> org.apache.kafka.connect.util.KafkaBasedLogTest > testSendAndReadToEnd FAILED
> org.junit.ComparisonFailure: expected: but was:
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at 
> org.apache.kafka.connect.util.KafkaBasedLogTest.testSendAndReadToEnd(KafkaBasedLogTest.java:312)
> Also, it looks the following command does not work with connect package.
> ./gradlew -Dtest.single=KafkaBasedLogTest connect:test



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: 0.9.0.0 RC4

[Jun Rao]

I updated the release notes. Since this doesn't affect the release
artifacts to be voted upon, we don't have to do another RC.

Please vote by 6pm PT today.

Thanks,

Jun

On Mon, Nov 23, 2015 at 8:43 AM, Guozhang Wang  wrote:

> I think we should update the release notes to remove the Kafka Streams
> tickets, I have marked them as 0.9.1.0.
>
> Guozhang
>
> On Fri, Nov 20, 2015 at 5:21 PM, Jun Rao  wrote:
>
> > This is the fourth candidate for release of Apache Kafka 0.9.0.0. This a
> > major release that includes (1) authentication (through SSL and SASL) and
> > authorization, (2) a new java consumer, (3) a Kafka connect framework for
> > data ingestion and egression, and (4) quotas. Since this is a major
> > release, we will give people a bit more time for trying this out.
> >
> > Release Notes for the 0.9.0.0 release
> >
> >
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Monday, Nov. 23, 6pm PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > http://kafka.apache.org/KEYS in addition to the md5, sha1
> > and sha2 (SHA256) checksum.
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/
> >
> > * Maven artifacts to be voted upon prior to release:
> > https://repository.apache.org/content/groups/staging/
> >
> > * scala-doc
> > https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/scaladoc/
> >
> > * java-doc
> > https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/javadoc/
> >
> > * The tag to be voted upon (off the 0.9.0 branch) is the 0.9.0.0 tag
> >
> >
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=132943b0f83831132cd46ac961cf6f1c00132565
> >
> > * Documentation
> > http://kafka.apache.org/090/documentation.html
> >
> > /***
> >
> > Thanks,
> >
> > Jun
> >
>
>
>
> --
> -- Guozhang
>

Re: 0.9.0.0 RC4

[Neha Narkhede]

+1 (binding).

Verified source and binary artifacts, ran unit tests.

On Mon, Nov 23, 2015 at 9:32 AM, Jun Rao  wrote:

> I updated the release notes. Since this doesn't affect the release
> artifacts to be voted upon, we don't have to do another RC.
>
> Please vote by 6pm PT today.
>
> Thanks,
>
> Jun
>
> On Mon, Nov 23, 2015 at 8:43 AM, Guozhang Wang  wrote:
>
> > I think we should update the release notes to remove the Kafka Streams
> > tickets, I have marked them as 0.9.1.0.
> >
> > Guozhang
> >
> > On Fri, Nov 20, 2015 at 5:21 PM, Jun Rao  wrote:
> >
> > > This is the fourth candidate for release of Apache Kafka 0.9.0.0. This
> a
> > > major release that includes (1) authentication (through SSL and SASL)
> and
> > > authorization, (2) a new java consumer, (3) a Kafka connect framework
> for
> > > data ingestion and egression, and (4) quotas. Since this is a major
> > > release, we will give people a bit more time for trying this out.
> > >
> > > Release Notes for the 0.9.0.0 release
> > >
> > >
> >
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Monday, Nov. 23, 6pm PT
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > http://kafka.apache.org/KEYS in addition to the md5, sha1
> > > and sha2 (SHA256) checksum.
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/
> > >
> > > * Maven artifacts to be voted upon prior to release:
> > > https://repository.apache.org/content/groups/staging/
> > >
> > > * scala-doc
> > > https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/scaladoc/
> > >
> > > * java-doc
> > > https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/javadoc/
> > >
> > > * The tag to be voted upon (off the 0.9.0 branch) is the 0.9.0.0 tag
> > >
> > >
> >
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=132943b0f83831132cd46ac961cf6f1c00132565
> > >
> > > * Documentation
> > > http://kafka.apache.org/090/documentation.html
> > >
> > > /***
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>



-- 
Thanks,
Neha

[jira] [Commented] (KAFKA-2082) Kafka Replication ends up in a bad state

[Drew Zagieboylo (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15022662#comment-15022662
 ] 

Drew Zagieboylo commented on KAFKA-2082:


I have also experience this behavior on 0.8.2.1
What is the current state of this issue?
Is there a fix planned for the next release or is it still being decided?

Thanks/

> Kafka Replication ends up in a bad state
> 
>
> Key: KAFKA-2082
> URL: https://issues.apache.org/jira/browse/KAFKA-2082
> Project: Kafka
>  Issue Type: Bug
>  Components: replication
>Affects Versions: 0.8.2.1
>Reporter: Evan Huus
>Assignee: Sriharsha Chintalapani
>Priority: Critical
>  Labels: zkclient-problems
> Attachments: KAFKA-2082.patch
>
>
> While running integration tests for Sarama (the go client) we came across a 
> pattern of connection losses that reliably puts kafka into a bad state: 
> several of the brokers start spinning, chewing ~30% CPU and spamming the logs 
> with hundreds of thousands of lines like:
> {noformat}
> [2015-04-01 13:08:40,070] WARN [Replica Manager on Broker 9093]: Fetch 
> request with correlation id 111094 from client ReplicaFetcherThread-0-9093 on 
> partition [many_partition,1] failed due to Leader not local for partition 
> [many_partition,1] on broker 9093 (kafka.server.ReplicaManager)
> [2015-04-01 13:08:40,070] WARN [Replica Manager on Broker 9093]: Fetch 
> request with correlation id 111094 from client ReplicaFetcherThread-0-9093 on 
> partition [many_partition,6] failed due to Leader not local for partition 
> [many_partition,6] on broker 9093 (kafka.server.ReplicaManager)
> [2015-04-01 13:08:40,070] WARN [Replica Manager on Broker 9093]: Fetch 
> request with correlation id 111095 from client ReplicaFetcherThread-0-9093 on 
> partition [many_partition,21] failed due to Leader not local for partition 
> [many_partition,21] on broker 9093 (kafka.server.ReplicaManager)
> [2015-04-01 13:08:40,071] WARN [Replica Manager on Broker 9093]: Fetch 
> request with correlation id 111095 from client ReplicaFetcherThread-0-9093 on 
> partition [many_partition,26] failed due to Leader not local for partition 
> [many_partition,26] on broker 9093 (kafka.server.ReplicaManager)
> [2015-04-01 13:08:40,071] WARN [Replica Manager on Broker 9093]: Fetch 
> request with correlation id 111095 from client ReplicaFetcherThread-0-9093 on 
> partition [many_partition,1] failed due to Leader not local for partition 
> [many_partition,1] on broker 9093 (kafka.server.ReplicaManager)
> [2015-04-01 13:08:40,071] WARN [Replica Manager on Broker 9093]: Fetch 
> request with correlation id 111095 from client ReplicaFetcherThread-0-9093 on 
> partition [many_partition,6] failed due to Leader not local for partition 
> [many_partition,6] on broker 9093 (kafka.server.ReplicaManager)
> [2015-04-01 13:08:40,072] WARN [Replica Manager on Broker 9093]: Fetch 
> request with correlation id 111096 from client ReplicaFetcherThread-0-9093 on 
> partition [many_partition,21] failed due to Leader not local for partition 
> [many_partition,21] on broker 9093 (kafka.server.ReplicaManager)
> [2015-04-01 13:08:40,072] WARN [Replica Manager on Broker 9093]: Fetch 
> request with correlation id 111096 from client ReplicaFetcherThread-0-9093 on 
> partition [many_partition,26] failed due to Leader not local for partition 
> [many_partition,26] on broker 9093 (kafka.server.ReplicaManager)
> {noformat}
> This can be easily and reliably reproduced using the {{toxiproxy-final}} 
> branch of https://github.com/Shopify/sarama which includes a vagrant script 
> for provisioning the appropriate cluster: 
> - {{git clone https://github.com/Shopify/sarama.git}}
> - {{git checkout test-jira-kafka-2082}}
> - {{vagrant up}}
> - {{TEST_SEED=1427917826425719059 DEBUG=true go test -v}}
> After the test finishes (it fails because the cluster ends up in a bad 
> state), you can log into the cluster machine with {{vagrant ssh}} and inspect 
> the bad nodes. The vagrant script provisions five zookeepers and five brokers 
> in {{/opt/kafka-9091/}} through {{/opt/kafka-9095/}}.
> Additional context: the test produces continually to the cluster while 
> randomly cutting and restoring zookeeper connections (all connections to 
> zookeeper are run through a simple proxy on the same vm to make this easy). 
> The majority of the time this works very well and does a good job exercising 
> our producer's retry and failover code. However, under certain patterns of 
> connection loss (the {{TEST_SEED}} in the instructions is important), kafka 
> gets confused. The test never cuts more than two connections at a time, so 
> zookeeper should always have quorum, and the topic (with three replicas) 
> should always be writable.
> Completely restarting the cluster via {

Re: 0.9.0.0 RC4

[Harsha]

+1 (binding).

1. Ran unit tests
2. Ran SSL & SASL tests using vagrant cluster.

Thanks,
Harsha

On Mon, Nov 23, 2015, at 09:34 AM, Neha Narkhede wrote:
> +1 (binding).
> 
> Verified source and binary artifacts, ran unit tests.
> 
> On Mon, Nov 23, 2015 at 9:32 AM, Jun Rao  wrote:
> 
> > I updated the release notes. Since this doesn't affect the release
> > artifacts to be voted upon, we don't have to do another RC.
> >
> > Please vote by 6pm PT today.
> >
> > Thanks,
> >
> > Jun
> >
> > On Mon, Nov 23, 2015 at 8:43 AM, Guozhang Wang  wrote:
> >
> > > I think we should update the release notes to remove the Kafka Streams
> > > tickets, I have marked them as 0.9.1.0.
> > >
> > > Guozhang
> > >
> > > On Fri, Nov 20, 2015 at 5:21 PM, Jun Rao  wrote:
> > >
> > > > This is the fourth candidate for release of Apache Kafka 0.9.0.0. This
> > a
> > > > major release that includes (1) authentication (through SSL and SASL)
> > and
> > > > authorization, (2) a new java consumer, (3) a Kafka connect framework
> > for
> > > > data ingestion and egression, and (4) quotas. Since this is a major
> > > > release, we will give people a bit more time for trying this out.
> > > >
> > > > Release Notes for the 0.9.0.0 release
> > > >
> > > >
> > >
> > https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/RELEASE_NOTES.html
> > > >
> > > > *** Please download, test and vote by Monday, Nov. 23, 6pm PT
> > > >
> > > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > > http://kafka.apache.org/KEYS in addition to the md5, sha1
> > > > and sha2 (SHA256) checksum.
> > > >
> > > > * Release artifacts to be voted upon (source and binary):
> > > > https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/
> > > >
> > > > * Maven artifacts to be voted upon prior to release:
> > > > https://repository.apache.org/content/groups/staging/
> > > >
> > > > * scala-doc
> > > > https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/scaladoc/
> > > >
> > > > * java-doc
> > > > https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/javadoc/
> > > >
> > > > * The tag to be voted upon (off the 0.9.0 branch) is the 0.9.0.0 tag
> > > >
> > > >
> > >
> > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=132943b0f83831132cd46ac961cf6f1c00132565
> > > >
> > > > * Documentation
> > > > http://kafka.apache.org/090/documentation.html
> > > >
> > > > /***
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
> 
> 
> 
> -- 
> Thanks,
> Neha

[jira] [Commented] (KAFKA-2873) o.a.k.connect.util.KafkaBasedLogTest failure in 0.9 RC4

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15023066#comment-15023066
 ] 

Jiangjie Qin commented on KAFKA-2873:
-

[~hachikuji] Ah, looks it is duplicate. BTW, does the single test command works 
for you?

> o.a.k.connect.util.KafkaBasedLogTest failure in  0.9 RC4
> 
>
> Key: KAFKA-2873
> URL: https://issues.apache.org/jira/browse/KAFKA-2873
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>
> Saw the following test failure when run unit test on RC4.
> org.apache.kafka.connect.util.KafkaBasedLogTest > testSendAndReadToEnd FAILED
> org.junit.ComparisonFailure: expected: but was:
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at 
> org.apache.kafka.connect.util.KafkaBasedLogTest.testSendAndReadToEnd(KafkaBasedLogTest.java:312)
> Also, it looks the following command does not work with connect package.
> ./gradlew -Dtest.single=KafkaBasedLogTest connect:test



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2876) Broker allows startup with incompatible listen port/inter-broker protocol settings

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-2876:
--

 Summary: Broker allows startup with incompatible listen 
port/inter-broker protocol settings
 Key: KAFKA-2876
 URL: https://issues.apache.org/jira/browse/KAFKA-2876
 Project: Kafka
  Issue Type: Bug
Reporter: Jason Gustafson


Currently the broker allows startup with an incompatible inter-broker security 
setting. For example, if the only listening port is enabled for SSL and no 
"security.inter.broker.protocol" is set, then the broker will still attempt to 
use the default PLAINTEXT protocol. When the broker then attempts to send 
LeaderAndIsr and other requests over plain text to itself (which can happen if 
it becomes the controller), it will silently catch the error since it cannot 
find the corresponding endpoint. It would be better to raise a configuration 
error in this case since there's no way that the broker can work correctly when 
it becomes controller.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-2876) Broker allows startup with incompatible listen port/inter-broker protocol settings

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reassigned KAFKA-2876:
--

Assignee: Ismael Juma

> Broker allows startup with incompatible listen port/inter-broker protocol 
> settings
> --
>
> Key: KAFKA-2876
> URL: https://issues.apache.org/jira/browse/KAFKA-2876
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Ismael Juma
>
> Currently the broker allows startup with an incompatible inter-broker 
> security setting. For example, if the only listening port is enabled for SSL 
> and no "security.inter.broker.protocol" is set, then the broker will still 
> attempt to use the default PLAINTEXT protocol. When the broker then attempts 
> to send LeaderAndIsr and other requests over plain text to itself (which can 
> happen if it becomes the controller), it will silently catch the error since 
> it cannot find the corresponding endpoint. It would be better to raise a 
> configuration error in this case since there's no way that the broker can 
> work correctly when it becomes controller.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2877) Messages sometimes not delivered by new consumer after Kafka restart

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-2877:
-

 Summary: Messages sometimes not delivered by new consumer after 
Kafka restart 
 Key: KAFKA-2877
 URL: https://issues.apache.org/jira/browse/KAFKA-2877
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 0.9.0.0
Reporter: Rajini Sivaram
Assignee: Neha Narkhede
Priority: Critical


After a Kafka restart, our health check consumer which subscribes to five 
topics with one partition each, was receiving messages from four out of the 
five topics. This has happened twice, the second time today was on 0.9.0.0 RC3. 

Some of the system test failures in 
http://jenkins.confluent.io/job/kafka_system_tests_branch_builder/220/ when the 
replication test was modified to use SSL/SASL clients and the throughput of the 
producer was reduced, also show a similar problem. Many of the replication 
tests  fail intermittently when new consumer is used in order to run clients 
with SSL/SASL. 




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-2877) Messages sometimes not delivered by new consumer after Kafka restart

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson reassigned KAFKA-2877:
--

Assignee: Jason Gustafson  (was: Neha Narkhede)

> Messages sometimes not delivered by new consumer after Kafka restart 
> -
>
> Key: KAFKA-2877
> URL: https://issues.apache.org/jira/browse/KAFKA-2877
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Jason Gustafson
>Priority: Critical
>
> After a Kafka restart, our health check consumer which subscribes to five 
> topics with one partition each, was receiving messages from four out of the 
> five topics. This has happened twice, the second time today was on 0.9.0.0 
> RC3. 
> Some of the system test failures in 
> http://jenkins.confluent.io/job/kafka_system_tests_branch_builder/220/ when 
> the replication test was modified to use SSL/SASL clients and the throughput 
> of the producer was reduced, also show a similar problem. Many of the 
> replication tests  fail intermittently when new consumer is used in order to 
> run clients with SSL/SASL. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2873) o.a.k.connect.util.KafkaBasedLogTest failure in 0.9 RC4

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15023157#comment-15023157
 ] 

Jason Gustafson commented on KAFKA-2873:


[~becket_qin] Looks like you need to include the connect package:
{code}
./gradlew -Dtest.single=KafkaBasedLogTest connect:runtime:test
{code}

> o.a.k.connect.util.KafkaBasedLogTest failure in  0.9 RC4
> 
>
> Key: KAFKA-2873
> URL: https://issues.apache.org/jira/browse/KAFKA-2873
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>
> Saw the following test failure when run unit test on RC4.
> org.apache.kafka.connect.util.KafkaBasedLogTest > testSendAndReadToEnd FAILED
> org.junit.ComparisonFailure: expected: but was:
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at 
> org.apache.kafka.connect.util.KafkaBasedLogTest.testSendAndReadToEnd(KafkaBasedLogTest.java:312)
> Also, it looks the following command does not work with connect package.
> ./gradlew -Dtest.single=KafkaBasedLogTest connect:test



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2878) Kafka broker throws OutOfMemory exception with invalid join group request

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-2878:
-

 Summary: Kafka broker throws OutOfMemory exception with invalid 
join group request
 Key: KAFKA-2878
 URL: https://issues.apache.org/jira/browse/KAFKA-2878
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 0.9.0.0
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
Priority: Critical


Array allocation for join group request doesn't have any checks and hence can 
result in OutOfMemory exception in the broker. Array size from the request 
should be validated to avoid DoS attacks on a secure installation of Kafka.

{quote}
at org/apache/kafka/common/protocol/types/ArrayOf.read(ArrayOf.java:44)
at org/apache/kafka/common/protocol/types/Schema.read(Schema.java:69)
at org/apache/kafka/common/protocol/ProtoUtils.parseRequest(ProtoUtils.java:60)
at 
org/apache/kafka/common/requests/JoinGroupRequest.parse(JoinGroupRequest.java:144)
at 
org/apache/kafka/common/requests/AbstractRequest.getRequest(AbstractRequest.java:55)
 
at kafka/network/RequestChannel$Request.(RequestChannel.scala:78)
{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2873) o.a.k.connect.util.KafkaBasedLogTest failure in 0.9 RC4

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15023188#comment-15023188
 ] 

Jiangjie Qin commented on KAFKA-2873:
-

Cool, that works. Thanks. I'll close this ticket as duplicate.

> o.a.k.connect.util.KafkaBasedLogTest failure in  0.9 RC4
> 
>
> Key: KAFKA-2873
> URL: https://issues.apache.org/jira/browse/KAFKA-2873
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>
> Saw the following test failure when run unit test on RC4.
> org.apache.kafka.connect.util.KafkaBasedLogTest > testSendAndReadToEnd FAILED
> org.junit.ComparisonFailure: expected: but was:
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at 
> org.apache.kafka.connect.util.KafkaBasedLogTest.testSendAndReadToEnd(KafkaBasedLogTest.java:312)
> Also, it looks the following command does not work with connect package.
> ./gradlew -Dtest.single=KafkaBasedLogTest connect:test



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2873) o.a.k.connect.util.KafkaBasedLogTest failure in 0.9 RC4

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2873?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin resolved KAFKA-2873.
-
Resolution: Duplicate

Duplicate for KAFKA-2667

> o.a.k.connect.util.KafkaBasedLogTest failure in  0.9 RC4
> 
>
> Key: KAFKA-2873
> URL: https://issues.apache.org/jira/browse/KAFKA-2873
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>
> Saw the following test failure when run unit test on RC4.
> org.apache.kafka.connect.util.KafkaBasedLogTest > testSendAndReadToEnd FAILED
> org.junit.ComparisonFailure: expected: but was:
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at 
> org.apache.kafka.connect.util.KafkaBasedLogTest.testSendAndReadToEnd(KafkaBasedLogTest.java:312)
> Also, it looks the following command does not work with connect package.
> ./gradlew -Dtest.single=KafkaBasedLogTest connect:test



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

All brokers are running but some partitions' leader is -1

[Qi Xu]

Hi folks,
We have a 10 node cluster and have several topics. Each topic has about 256
partitions with 3 replica factor. Now we run into an issue that in some
topic, a few partition (< 10)'s leader is -1 and all of them has only one
synced partition.

>From the Kafka manager, here's the snapshot:
[image: Inline image 2]

[image: Inline image 1]

here's the state log:
[2015-11-23 21:57:58,598] ERROR Controller 1 epoch 435499 initiated state
change for partition [userlogs,84] from OnlinePartition to OnlinePartition
failed (state.change.logger)
kafka.common.StateChangeFailedException: encountered error while electing
leader for partition [userlogs,84] due to: Preferred replica 0 for
partition [userlogs,84] is either not alive or not in the isr. Current
leader and ISR: [{"leader":-1,"leader_epoch":203,"isr":[1]}].
Caused by: kafka.common.StateChangeFailedException: Preferred replica 0 for
partition [userlogs,84] is either not alive or not in the isr. Current
leader and ISR: [{"leader":-1,"leader_epoch":203,"isr":[1]}]

My question is:
1) how could this happen and how can I fix it or work around it?
2) Is 256 partitions too big? We have about 200+ cores for spark streaming
job.

Thanks,
Qi

Re: All brokers are running but some partitions' leader is -1

[Qi Xu]

Loop another guy from our team.

On Mon, Nov 23, 2015 at 2:26 PM, Qi Xu  wrote:

> Hi folks,
> We have a 10 node cluster and have several topics. Each topic has about
> 256 partitions with 3 replica factor. Now we run into an issue that in some
> topic, a few partition (< 10)'s leader is -1 and all of them has only one
> synced partition.
>
> From the Kafka manager, here's the snapshot:
> [image: Inline image 2]
>
> [image: Inline image 1]
>
> here's the state log:
> [2015-11-23 21:57:58,598] ERROR Controller 1 epoch 435499 initiated state
> change for partition [userlogs,84] from OnlinePartition to OnlinePartition
> failed (state.change.logger)
> kafka.common.StateChangeFailedException: encountered error while electing
> leader for partition [userlogs,84] due to: Preferred replica 0 for
> partition [userlogs,84] is either not alive or not in the isr. Current
> leader and ISR: [{"leader":-1,"leader_epoch":203,"isr":[1]}].
> Caused by: kafka.common.StateChangeFailedException: Preferred replica 0
> for partition [userlogs,84] is either not alive or not in the isr. Current
> leader and ISR: [{"leader":-1,"leader_epoch":203,"isr":[1]}]
>
> My question is:
> 1) how could this happen and how can I fix it or work around it?
> 2) Is 256 partitions too big? We have about 200+ cores for spark streaming
> job.
>
> Thanks,
> Qi
>
>

[GitHub] kafka pull request: KAFKA-2878: Guard against OutOfMemory in Kafka...

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/577

KAFKA-2878: Guard against OutOfMemory in Kafka broker 

Sanity check array size in requests before allocation

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-2878

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/577.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #577


commit 8d1a8a9d4e0444929db8218ded5949a0b827c42d
Author: Rajini Sivaram 
Date:   2015-11-23T22:42:51Z

KAFKA-2878: Guard against OutOfMemory in Kafka broker with invalid requests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2878) Kafka broker throws OutOfMemory exception with invalid join group request

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15023264#comment-15023264
 ] 

ASF GitHub Bot commented on KAFKA-2878:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/577

KAFKA-2878: Guard against OutOfMemory in Kafka broker 

Sanity check array size in requests before allocation

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-2878

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/577.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #577


commit 8d1a8a9d4e0444929db8218ded5949a0b827c42d
Author: Rajini Sivaram 
Date:   2015-11-23T22:42:51Z

KAFKA-2878: Guard against OutOfMemory in Kafka broker with invalid requests




> Kafka broker throws OutOfMemory exception with invalid join group request
> -
>
> Key: KAFKA-2878
> URL: https://issues.apache.org/jira/browse/KAFKA-2878
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>Priority: Critical
>
> Array allocation for join group request doesn't have any checks and hence can 
> result in OutOfMemory exception in the broker. Array size from the request 
> should be validated to avoid DoS attacks on a secure installation of Kafka.
> {quote}
> at org/apache/kafka/common/protocol/types/ArrayOf.read(ArrayOf.java:44)
> at org/apache/kafka/common/protocol/types/Schema.read(Schema.java:69)
> at 
> org/apache/kafka/common/protocol/ProtoUtils.parseRequest(ProtoUtils.java:60)
> at 
> org/apache/kafka/common/requests/JoinGroupRequest.parse(JoinGroupRequest.java:144)
> at 
> org/apache/kafka/common/requests/AbstractRequest.getRequest(AbstractRequest.java:55)
>  
> at kafka/network/RequestChannel$Request.(RequestChannel.scala:78)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: All brokers are running but some partitions' leader is -1

[Qi Xu]

Forgot to mention is that the Kafka version we're using is from Aug's Trunk
branch---which has the SSL support.

Thanks again,
Qi


On Mon, Nov 23, 2015 at 2:29 PM, Qi Xu  wrote:

> Loop another guy from our team.
>
> On Mon, Nov 23, 2015 at 2:26 PM, Qi Xu  wrote:
>
>> Hi folks,
>> We have a 10 node cluster and have several topics. Each topic has about
>> 256 partitions with 3 replica factor. Now we run into an issue that in some
>> topic, a few partition (< 10)'s leader is -1 and all of them has only one
>> synced partition.
>>
>> From the Kafka manager, here's the snapshot:
>> [image: Inline image 2]
>>
>> [image: Inline image 1]
>>
>> here's the state log:
>> [2015-11-23 21:57:58,598] ERROR Controller 1 epoch 435499 initiated state
>> change for partition [userlogs,84] from OnlinePartition to OnlinePartition
>> failed (state.change.logger)
>> kafka.common.StateChangeFailedException: encountered error while electing
>> leader for partition [userlogs,84] due to: Preferred replica 0 for
>> partition [userlogs,84] is either not alive or not in the isr. Current
>> leader and ISR: [{"leader":-1,"leader_epoch":203,"isr":[1]}].
>> Caused by: kafka.common.StateChangeFailedException: Preferred replica 0
>> for partition [userlogs,84] is either not alive or not in the isr. Current
>> leader and ISR: [{"leader":-1,"leader_epoch":203,"isr":[1]}]
>>
>> My question is:
>> 1) how could this happen and how can I fix it or work around it?
>> 2) Is 256 partitions too big? We have about 200+ cores for spark
>> streaming job.
>>
>> Thanks,
>> Qi
>>
>>
>

Re: 0.9.0.0 RC4

[Flavio Junqueira]

+1 (non-binding). Built it, ran tests, ran the rat tool separately, checked 
license and notice, checked digests, ran a few local smoke tests including zk 
auth. Looks good!

-Flavio

> On 23 Nov 2015, at 19:11, Harsha  wrote:
> 
> +1 (binding).
> 
> 1. Ran unit tests
> 2. Ran SSL & SASL tests using vagrant cluster.
> 
> Thanks,
> Harsha
> 
> On Mon, Nov 23, 2015, at 09:34 AM, Neha Narkhede wrote:
>> +1 (binding).
>> 
>> Verified source and binary artifacts, ran unit tests.
>> 
>> On Mon, Nov 23, 2015 at 9:32 AM, Jun Rao  wrote:
>> 
>>> I updated the release notes. Since this doesn't affect the release
>>> artifacts to be voted upon, we don't have to do another RC.
>>> 
>>> Please vote by 6pm PT today.
>>> 
>>> Thanks,
>>> 
>>> Jun
>>> 
>>> On Mon, Nov 23, 2015 at 8:43 AM, Guozhang Wang  wrote:
>>> 
 I think we should update the release notes to remove the Kafka Streams
 tickets, I have marked them as 0.9.1.0.
 
 Guozhang
 
 On Fri, Nov 20, 2015 at 5:21 PM, Jun Rao  wrote:
 
> This is the fourth candidate for release of Apache Kafka 0.9.0.0. This
>>> a
> major release that includes (1) authentication (through SSL and SASL)
>>> and
> authorization, (2) a new java consumer, (3) a Kafka connect framework
>>> for
> data ingestion and egression, and (4) quotas. Since this is a major
> release, we will give people a bit more time for trying this out.
> 
> Release Notes for the 0.9.0.0 release
> 
> 
 
>>> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/RELEASE_NOTES.html
> 
> *** Please download, test and vote by Monday, Nov. 23, 6pm PT
> 
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS in addition to the md5, sha1
> and sha2 (SHA256) checksum.
> 
> * Release artifacts to be voted upon (source and binary):
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/
> 
> * Maven artifacts to be voted upon prior to release:
> https://repository.apache.org/content/groups/staging/
> 
> * scala-doc
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/scaladoc/
> 
> * java-doc
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/javadoc/
> 
> * The tag to be voted upon (off the 0.9.0 branch) is the 0.9.0.0 tag
> 
> 
 
>>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=132943b0f83831132cd46ac961cf6f1c00132565
> 
> * Documentation
> http://kafka.apache.org/090/documentation.html
> 
> /***
> 
> Thanks,
> 
> Jun
> 
 
 
 
 --
 -- Guozhang
 
>>> 
>> 
>> 
>> 
>> -- 
>> Thanks,
>> Neha

[jira] [Created] (KAFKA-2879) Make MiniKDC test service slightly more generic

[Gwen Shapira (JIRA)]

Gwen Shapira created KAFKA-2879:
---

 Summary: Make MiniKDC test service slightly more generic
 Key: KAFKA-2879
 URL: https://issues.apache.org/jira/browse/KAFKA-2879
 Project: Kafka
  Issue Type: Improvement
Reporter: Gwen Shapira


Allow users of MiniKDC in tests to pass in principals to be added to keytab, in 
addition to the Kafka principals that are generated by MiniKDC.

This will allow using MiniKDC to be used when testing additional services with 
their own principals.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2879: Make MiniKDC test service slightly...

[gwenshap]

GitHub user gwenshap opened a pull request:

https://github.com/apache/kafka/pull/578

KAFKA-2879: Make MiniKDC test service slightly more generic



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gwenshap/kafka KAFKA-2879

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/578.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #578


commit ff4a248b76e900a142f16b7bf3a95958813ee596
Author: Gwen Shapira 
Date:   2015-11-23T23:24:43Z

KAFKA-2879: Make MiniKDC test service slightly more generic




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2879) Make MiniKDC test service slightly more generic

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15023327#comment-15023327
 ] 

ASF GitHub Bot commented on KAFKA-2879:
---

GitHub user gwenshap opened a pull request:

https://github.com/apache/kafka/pull/578

KAFKA-2879: Make MiniKDC test service slightly more generic



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/gwenshap/kafka KAFKA-2879

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/578.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #578


commit ff4a248b76e900a142f16b7bf3a95958813ee596
Author: Gwen Shapira 
Date:   2015-11-23T23:24:43Z

KAFKA-2879: Make MiniKDC test service slightly more generic




> Make MiniKDC test service slightly more generic
> ---
>
> Key: KAFKA-2879
> URL: https://issues.apache.org/jira/browse/KAFKA-2879
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Gwen Shapira
>
> Allow users of MiniKDC in tests to pass in principals to be added to keytab, 
> in addition to the Kafka principals that are generated by MiniKDC.
> This will allow using MiniKDC to be used when testing additional services 
> with their own principals.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-2692) Add ducktape tests for SASL/Kerberos

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reassigned KAFKA-2692:
--

Assignee: Ismael Juma

> Add ducktape tests for SASL/Kerberos
> 
>
> Key: KAFKA-2692
> URL: https://issues.apache.org/jira/browse/KAFKA-2692
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Rajini Sivaram
>Assignee: Ismael Juma
> Fix For: 0.9.1.0
>
>
> KAFKA-2644 runs replication tests and benchmarks with SASL/Kerberos using 
> MiniKdc. Additional Kerberos-specific tests are required, particularly to 
> test error scenarios. This may require replacing MiniKdc with full KDC.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2804: manage changelog topics through ZK...

[guozhangwang]

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/579

KAFKA-2804: manage changelog topics through ZK in PartitionAssignor



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka K2804

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/579.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #579






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2804) Create / Update changelog topics upon state store initialization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15023433#comment-15023433
 ] 

ASF GitHub Bot commented on KAFKA-2804:
---

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/579

KAFKA-2804: manage changelog topics through ZK in PartitionAssignor



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka K2804

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/579.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #579






> Create / Update changelog topics upon state store initialization
> 
>
> Key: KAFKA-2804
> URL: https://issues.apache.org/jira/browse/KAFKA-2804
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>
> When state store instances that are logging-backed are initialized, we need 
> to check if the corresponding change log topics have been created with the 
> right number of partitions:
> 1) If not exist, create topic
> 2) If expected #.partitions < actual #.partitions, delete and re-create topic.
> 3) If expected #.partitions > actual #.partitions, add partitions.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2880) Fetcher.getTopicMetadata NullPointerException when broker cannot be reached

[Ewen Cheslack-Postava (JIRA)]

Ewen Cheslack-Postava created KAFKA-2880:


 Summary: Fetcher.getTopicMetadata NullPointerException when broker 
cannot be reached
 Key: KAFKA-2880
 URL: https://issues.apache.org/jira/browse/KAFKA-2880
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 0.9.0.0
Reporter: Ewen Cheslack-Postava
Assignee: Jason Gustafson


The Fetcher class will throw a NullPointerException if a broker cannot be 
reached:

{quote}
Exception in thread "main" java.lang.NullPointerException
at 
org.apache.kafka.common.requests.MetadataResponse.(MetadataResponse.java:130)
at 
org.apache.kafka.clients.consumer.internals.Fetcher.getTopicMetadata(Fetcher.java:203)
at 
org.apache.kafka.clients.consumer.KafkaConsumer.partitionsFor(KafkaConsumer.java:1143)
at 
org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:126)
at 
org.apache.kafka.connect.storage.KafkaOffsetBackingStore.start(KafkaOffsetBackingStore.java:85)
at org.apache.kafka.connect.runtime.Worker.start(Worker.java:108)
at org.apache.kafka.connect.runtime.Connect.start(Connect.java:56)
at 
org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:62)
{quote}

This is trivially reproduced by trying to start Kafka Connect in distributed 
mode (i.e. connect-distributed.sh config/connect-distributed.properties) with 
no broker running. However, it's not specific to Kafka Connect, it just happens 
to use the consumer in a way that triggers it reliably.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2879: Make MiniKDC test service slightly...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/578


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: 0.9.0.0 RC4

[Ewen Cheslack-Postava]

+1 non-binding. Built in a fresh VM, ran some smoke tests for both the
brokers and Kafka Connect.

-Ewen

On Mon, Nov 23, 2015 at 3:11 PM, Flavio Junqueira  wrote:

> +1 (non-binding). Built it, ran tests, ran the rat tool separately,
> checked license and notice, checked digests, ran a few local smoke tests
> including zk auth. Looks good!
>
> -Flavio
>
> > On 23 Nov 2015, at 19:11, Harsha  wrote:
> >
> > +1 (binding).
> >
> > 1. Ran unit tests
> > 2. Ran SSL & SASL tests using vagrant cluster.
> >
> > Thanks,
> > Harsha
> >
> > On Mon, Nov 23, 2015, at 09:34 AM, Neha Narkhede wrote:
> >> +1 (binding).
> >>
> >> Verified source and binary artifacts, ran unit tests.
> >>
> >> On Mon, Nov 23, 2015 at 9:32 AM, Jun Rao  wrote:
> >>
> >>> I updated the release notes. Since this doesn't affect the release
> >>> artifacts to be voted upon, we don't have to do another RC.
> >>>
> >>> Please vote by 6pm PT today.
> >>>
> >>> Thanks,
> >>>
> >>> Jun
> >>>
> >>> On Mon, Nov 23, 2015 at 8:43 AM, Guozhang Wang 
> wrote:
> >>>
>  I think we should update the release notes to remove the Kafka Streams
>  tickets, I have marked them as 0.9.1.0.
> 
>  Guozhang
> 
>  On Fri, Nov 20, 2015 at 5:21 PM, Jun Rao  wrote:
> 
> > This is the fourth candidate for release of Apache Kafka 0.9.0.0.
> This
> >>> a
> > major release that includes (1) authentication (through SSL and SASL)
> >>> and
> > authorization, (2) a new java consumer, (3) a Kafka connect framework
> >>> for
> > data ingestion and egression, and (4) quotas. Since this is a major
> > release, we will give people a bit more time for trying this out.
> >
> > Release Notes for the 0.9.0.0 release
> >
> >
> 
> >>>
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Monday, Nov. 23, 6pm PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > http://kafka.apache.org/KEYS in addition to the md5, sha1
> > and sha2 (SHA256) checksum.
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/
> >
> > * Maven artifacts to be voted upon prior to release:
> > https://repository.apache.org/content/groups/staging/
> >
> > * scala-doc
> > https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/scaladoc/
> >
> > * java-doc
> > https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/javadoc/
> >
> > * The tag to be voted upon (off the 0.9.0 branch) is the 0.9.0.0 tag
> >
> >
> 
> >>>
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=132943b0f83831132cd46ac961cf6f1c00132565
> >
> > * Documentation
> > http://kafka.apache.org/090/documentation.html
> >
> > /***
> >
> > Thanks,
> >
> > Jun
> >
> 
> 
> 
>  --
>  -- Guozhang
> 
> >>>
> >>
> >>
> >>
> >> --
> >> Thanks,
> >> Neha
>
>


-- 
Thanks,
Ewen

[jira] [Resolved] (KAFKA-2879) Make MiniKDC test service slightly more generic

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-2879.
--
   Resolution: Fixed
Fix Version/s: 0.9.1.0

Issue resolved by pull request 578
[https://github.com/apache/kafka/pull/578]

> Make MiniKDC test service slightly more generic
> ---
>
> Key: KAFKA-2879
> URL: https://issues.apache.org/jira/browse/KAFKA-2879
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Gwen Shapira
> Fix For: 0.9.1.0
>
>
> Allow users of MiniKDC in tests to pass in principals to be added to keytab, 
> in addition to the Kafka principals that are generated by MiniKDC.
> This will allow using MiniKDC to be used when testing additional services 
> with their own principals.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2879) Make MiniKDC test service slightly more generic

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15023444#comment-15023444
 ] 

ASF GitHub Bot commented on KAFKA-2879:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/578


> Make MiniKDC test service slightly more generic
> ---
>
> Key: KAFKA-2879
> URL: https://issues.apache.org/jira/browse/KAFKA-2879
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Gwen Shapira
> Fix For: 0.9.1.0
>
>
> Allow users of MiniKDC in tests to pass in principals to be added to keytab, 
> in addition to the Kafka principals that are generated by MiniKDC.
> This will allow using MiniKDC to be used when testing additional services 
> with their own principals.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2879) Make MiniKDC test service slightly more generic

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2879:
-
Assignee: Gwen Shapira

> Make MiniKDC test service slightly more generic
> ---
>
> Key: KAFKA-2879
> URL: https://issues.apache.org/jira/browse/KAFKA-2879
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Gwen Shapira
>Assignee: Gwen Shapira
> Fix For: 0.9.1.0
>
>
> Allow users of MiniKDC in tests to pass in principals to be added to keytab, 
> in addition to the Kafka principals that are generated by MiniKDC.
> This will allow using MiniKDC to be used when testing additional services 
> with their own principals.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #179

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-2879: Make MiniKDC test service slightly more generic

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu3 (Ubuntu ubuntu legacy-ubuntu) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision e32df4722ca476ab373cca45a824f712cf4943ef 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e32df4722ca476ab373cca45a824f712cf4943ef
 > git rev-list f81dd4a4fbb5c1c78dc5fc38a13d2adaf7eff2e7 # timeout=10
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson5750116615495886701.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.4-rc-2/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:downloadWrapper

BUILD SUCCESSFUL

Total time: 14.022 secs
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson956717940552020964.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew -Dorg.gradle.project.maxParallelForks=1 clean jarAll testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.9/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:clean UP-TO-DATE
:clients:clean
:connect:clean UP-TO-DATE
:core:clean
:examples:clean
:log4j-appender:clean
:streams:clean
:tools:clean
:connect:api:clean
:connect:file:clean
:connect:json:clean
:connect:runtime:clean
:jar_core_2_10_6
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk8:clients:compileJava
:jar_core_2_10_6 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Could not read entry ':clients:compileJava' from cache taskArtifacts.bin 
(/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk8/.gradle/2.9/taskArtifacts/taskArtifacts.bin).
> java.io.UTFDataFormatException (no error message)

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 11.433 secs
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
ERROR: Publisher 'Publish JUnit test result report' failed: No test report 
files were found. Configuration error?
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2

Re: All brokers are running but some partitions' leader is -1

[Gwen Shapira]

We fixed many many bugs since August. Since we are about to release 0.9.0
(with SSL!), maybe wait a day and go with a released and tested version.

On Mon, Nov 23, 2015 at 3:01 PM, Qi Xu  wrote:

> Forgot to mention is that the Kafka version we're using is from Aug's
> Trunk branch---which has the SSL support.
>
> Thanks again,
> Qi
>
>
> On Mon, Nov 23, 2015 at 2:29 PM, Qi Xu  wrote:
>
>> Loop another guy from our team.
>>
>> On Mon, Nov 23, 2015 at 2:26 PM, Qi Xu  wrote:
>>
>>> Hi folks,
>>> We have a 10 node cluster and have several topics. Each topic has about
>>> 256 partitions with 3 replica factor. Now we run into an issue that in some
>>> topic, a few partition (< 10)'s leader is -1 and all of them has only one
>>> synced partition.
>>>
>>> From the Kafka manager, here's the snapshot:
>>> [image: Inline image 2]
>>>
>>> [image: Inline image 1]
>>>
>>> here's the state log:
>>> [2015-11-23 21:57:58,598] ERROR Controller 1 epoch 435499 initiated
>>> state change for partition [userlogs,84] from OnlinePartition to
>>> OnlinePartition failed (state.change.logger)
>>> kafka.common.StateChangeFailedException: encountered error while
>>> electing leader for partition [userlogs,84] due to: Preferred replica 0 for
>>> partition [userlogs,84] is either not alive or not in the isr. Current
>>> leader and ISR: [{"leader":-1,"leader_epoch":203,"isr":[1]}].
>>> Caused by: kafka.common.StateChangeFailedException: Preferred replica 0
>>> for partition [userlogs,84] is either not alive or not in the isr. Current
>>> leader and ISR: [{"leader":-1,"leader_epoch":203,"isr":[1]}]
>>>
>>> My question is:
>>> 1) how could this happen and how can I fix it or work around it?
>>> 2) Is 256 partitions too big? We have about 200+ cores for spark
>>> streaming job.
>>>
>>> Thanks,
>>> Qi
>>>
>>>
>>
>

Re: 0.9.0.0 RC4

[Guozhang Wang]

+1 (binding), ran quickstart and the topics / consumer-groups tools with
2.11 and 2.10 versions.

Guozhang

On Mon, Nov 23, 2015 at 4:14 PM, Ewen Cheslack-Postava 
wrote:

> +1 non-binding. Built in a fresh VM, ran some smoke tests for both the
> brokers and Kafka Connect.
>
> -Ewen
>
> On Mon, Nov 23, 2015 at 3:11 PM, Flavio Junqueira  wrote:
>
> > +1 (non-binding). Built it, ran tests, ran the rat tool separately,
> > checked license and notice, checked digests, ran a few local smoke tests
> > including zk auth. Looks good!
> >
> > -Flavio
> >
> > > On 23 Nov 2015, at 19:11, Harsha  wrote:
> > >
> > > +1 (binding).
> > >
> > > 1. Ran unit tests
> > > 2. Ran SSL & SASL tests using vagrant cluster.
> > >
> > > Thanks,
> > > Harsha
> > >
> > > On Mon, Nov 23, 2015, at 09:34 AM, Neha Narkhede wrote:
> > >> +1 (binding).
> > >>
> > >> Verified source and binary artifacts, ran unit tests.
> > >>
> > >> On Mon, Nov 23, 2015 at 9:32 AM, Jun Rao  wrote:
> > >>
> > >>> I updated the release notes. Since this doesn't affect the release
> > >>> artifacts to be voted upon, we don't have to do another RC.
> > >>>
> > >>> Please vote by 6pm PT today.
> > >>>
> > >>> Thanks,
> > >>>
> > >>> Jun
> > >>>
> > >>> On Mon, Nov 23, 2015 at 8:43 AM, Guozhang Wang 
> > wrote:
> > >>>
> >  I think we should update the release notes to remove the Kafka
> Streams
> >  tickets, I have marked them as 0.9.1.0.
> > 
> >  Guozhang
> > 
> >  On Fri, Nov 20, 2015 at 5:21 PM, Jun Rao  wrote:
> > 
> > > This is the fourth candidate for release of Apache Kafka 0.9.0.0.
> > This
> > >>> a
> > > major release that includes (1) authentication (through SSL and
> SASL)
> > >>> and
> > > authorization, (2) a new java consumer, (3) a Kafka connect
> framework
> > >>> for
> > > data ingestion and egression, and (4) quotas. Since this is a major
> > > release, we will give people a bit more time for trying this out.
> > >
> > > Release Notes for the 0.9.0.0 release
> > >
> > >
> > 
> > >>>
> >
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/RELEASE_NOTES.html
> > >
> > > *** Please download, test and vote by Monday, Nov. 23, 6pm PT
> > >
> > > Kafka's KEYS file containing PGP keys we use to sign the release:
> > > http://kafka.apache.org/KEYS in addition to the md5, sha1
> > > and sha2 (SHA256) checksum.
> > >
> > > * Release artifacts to be voted upon (source and binary):
> > > https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/
> > >
> > > * Maven artifacts to be voted upon prior to release:
> > > https://repository.apache.org/content/groups/staging/
> > >
> > > * scala-doc
> > >
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/scaladoc/
> > >
> > > * java-doc
> > >
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/javadoc/
> > >
> > > * The tag to be voted upon (off the 0.9.0 branch) is the 0.9.0.0
> tag
> > >
> > >
> > 
> > >>>
> >
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=132943b0f83831132cd46ac961cf6f1c00132565
> > >
> > > * Documentation
> > > http://kafka.apache.org/090/documentation.html
> > >
> > > /***
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > 
> > 
> > 
> >  --
> >  -- Guozhang
> > 
> > >>>
> > >>
> > >>
> > >>
> > >> --
> > >> Thanks,
> > >> Neha
> >
> >
>
>
> --
> Thanks,
> Ewen
>



-- 
-- Guozhang

Jenkins build is back to normal : kafka-trunk-jdk7 #850

[Apache Jenkins Server]

See 

[jira] [Reopened] (KAFKA-2718) Reuse of temporary directories leading to transient unit test failures

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang reopened KAFKA-2718:
--

> Reuse of temporary directories leading to transient unit test failures
> --
>
> Key: KAFKA-2718
> URL: https://issues.apache.org/jira/browse/KAFKA-2718
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Affects Versions: 0.9.1.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.9.0.0
>
>
> Stack traces in some of the transient unit test failures indicate that 
> temporary directories used for Zookeeper are being reused.
> {quote}
> kafka.common.TopicExistsException: Topic "topic" already exists.
>   at 
> kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:253)
>   at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:237)
>   at kafka.utils.TestUtils$.createTopic(TestUtils.scala:231)
>   at kafka.api.BaseConsumerTest.setUp(BaseConsumerTest.scala:63)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2718) Reuse of temporary directories leading to transient unit test failures

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15023598#comment-15023598
 ] 

Guozhang Wang commented on KAFKA-2718:
--

Saw another case with this issue:

https://builds.apache.org/job/kafka-trunk-git-pr-jdk7/1485/testReport/junit/kafka.api/PlaintextConsumerTest/testAutoCommitOnRebalance/

Re-open it for now until we have some further investigation.

> Reuse of temporary directories leading to transient unit test failures
> --
>
> Key: KAFKA-2718
> URL: https://issues.apache.org/jira/browse/KAFKA-2718
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Affects Versions: 0.9.1.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.9.0.0
>
>
> Stack traces in some of the transient unit test failures indicate that 
> temporary directories used for Zookeeper are being reused.
> {quote}
> kafka.common.TopicExistsException: Topic "topic" already exists.
>   at 
> kafka.admin.AdminUtils$.createOrUpdateTopicPartitionAssignmentPathInZK(AdminUtils.scala:253)
>   at kafka.admin.AdminUtils$.createTopic(AdminUtils.scala:237)
>   at kafka.utils.TestUtils$.createTopic(TestUtils.scala:231)
>   at kafka.api.BaseConsumerTest.setUp(BaseConsumerTest.scala:63)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: Minor: Fix KafkaConsumer Constructor Summary j...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/576


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-2881) Documentation improvement

[Gwen Shapira (JIRA)]

Gwen Shapira created KAFKA-2881:
---

 Summary: Documentation improvement
 Key: KAFKA-2881
 URL: https://issues.apache.org/jira/browse/KAFKA-2881
 Project: Kafka
  Issue Type: Improvement
Reporter: Gwen Shapira



1. We seem to be missing section 4.9 (Quotas) in the table of contents 

2. The consumer documentation could use a little meta-guidance. There are three 
sections "High Level Consumer", "Simple Consumer", "New Consumer". If I came 
into this fresh, I would be super confused as to what these things were and 
which I should use. I think we need to add a section that introduces the state 
of things--i.e. when should you use each?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #851

[Apache Jenkins Server]

See 

Changes:

[cshapi] Minor: Fix KafkaConsumer Constructor Summary javadoc

--
[...truncated 1371 lines...]

kafka.log.LogTest > testTruncateTo PASSED

kafka.log.LogTest > testCleanShutdownFile PASSED

kafka.log.OffsetIndexTest > lookupExtremeCases PASSED

kafka.log.OffsetIndexTest > appendTooMany PASSED

kafka.log.OffsetIndexTest > randomLookupTest PASSED

kafka.log.OffsetIndexTest > testReopen PASSED

kafka.log.OffsetIndexTest > appendOutOfOrder PASSED

kafka.log.OffsetIndexTest > truncate PASSED

kafka.log.LogSegmentTest > testRecoveryWithCorruptMessage PASSED

kafka.log.LogSegmentTest > testRecoveryFixesCorruptIndex PASSED

kafka.log.LogSegmentTest > testReadFromGap PASSED

kafka.log.LogSegmentTest > testTruncate PASSED

kafka.log.LogSegmentTest > testReadBeforeFirstOffset PASSED

kafka.log.LogSegmentTest > testCreateWithInitFileSizeAppendMessage PASSED

kafka.log.LogSegmentTest > testChangeFileSuffixes PASSED

kafka.log.LogSegmentTest > testMaxOffset PASSED

kafka.log.LogSegmentTest > testNextOffsetCalculation PASSED

kafka.log.LogSegmentTest > testReadOnEmptySegment PASSED

kafka.log.LogSegmentTest > testReadAfterLast PASSED

kafka.log.LogSegmentTest > testCreateWithInitFileSizeClearShutdown PASSED

kafka.log.LogSegmentTest > testTruncateFull PASSED

kafka.log.CleanerTest > testBuildOffsetMap PASSED

kafka.log.CleanerTest > testSegmentGrouping PASSED

kafka.log.CleanerTest > testCleanSegmentsWithAbort PASSED

kafka.log.CleanerTest > testSegmentGroupingWithSparseOffsets PASSED

kafka.log.CleanerTest > testRecoveryAfterCrash PASSED

kafka.log.CleanerTest > testLogToClean PASSED

kafka.log.CleanerTest > testCleaningWithDeletes PASSED

kafka.log.CleanerTest > testCleanSegments PASSED

kafka.log.CleanerTest > testCleaningWithUnkeyedMessages PASSED

kafka.log.OffsetMapTest > testClear PASSED

kafka.log.OffsetMapTest > testBasicValidation PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testDescribeGroupStable PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatIllegalGeneration 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testDescribeGroupWrongCoordinator PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testDescribeGroupRebalancing 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaderFailureInSyncGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testGenerationIdIncrementsOnRebalance PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testSyncGroupFromIllegalGeneration PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testInvalidGroupId PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testHeartbeatUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testListGroupsIncludesStableGroups PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testHeartbeatDuringRebalanceCausesRebalanceInProgress PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupInconsistentGroupProtocol PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupSessionTimeoutTooLarge PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupSessionTimeoutTooSmall PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testSyncGroupEmptyAssignment 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testCommitOffsetWithDefaultGeneration PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupFromUnchangedLeaderShouldRebalance PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testHeartbeatRebalanceInProgress PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaveGroupUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testListGroupsIncludesRebalancingGroups PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testSyncGroupFollowerAfterLeader PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testCommitOffsetInAwaitingSync 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testJoinGroupWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupUnknownConsumerExistingGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testSyncGroupFromUnknownGroup 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupInconsistentProtocolType PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testCommitOffsetFromUnknownGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testLeaveGroupWrongCoordinator 
PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testLeaveGroupUnknownConsumerExistingGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupUnknownConsumerNewGroup PASSED

kafka.coordinator.GroupCoordinatorResponseTest > 
testJoinGroupFromUnchangedFollowerDoesNotRebalance PASSED

kafka.coordinator.GroupCoordinatorResponseTest > testVa

[jira] [Commented] (KAFKA-2880) Fetcher.getTopicMetadata NullPointerException when broker cannot be reached

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15023729#comment-15023729
 ] 

Jason Gustafson commented on KAFKA-2880:


[~ewencp] Looks like we're missing a disconnect check on the client response. 
Looking over the code, error handling for the listTopics() and partitionsFor() 
API clearly hasn't been given enough attention. How about we expand the scope 
of this ticket to include the following cases?

1. Unauthorized topics: raise TopicAuthorizationException
2. Miscellaneous topic errors (e.g. leader/replica not available): retry until 
request timeout expires
3. Disconnect errors: retry until request timeout expires
4. Request timeout: raise TimeoutException


> Fetcher.getTopicMetadata NullPointerException when broker cannot be reached
> ---
>
> Key: KAFKA-2880
> URL: https://issues.apache.org/jira/browse/KAFKA-2880
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
>
> The Fetcher class will throw a NullPointerException if a broker cannot be 
> reached:
> {quote}
> Exception in thread "main" java.lang.NullPointerException
> at 
> org.apache.kafka.common.requests.MetadataResponse.(MetadataResponse.java:130)
> at 
> org.apache.kafka.clients.consumer.internals.Fetcher.getTopicMetadata(Fetcher.java:203)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.partitionsFor(KafkaConsumer.java:1143)
> at 
> org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:126)
> at 
> org.apache.kafka.connect.storage.KafkaOffsetBackingStore.start(KafkaOffsetBackingStore.java:85)
> at org.apache.kafka.connect.runtime.Worker.start(Worker.java:108)
> at org.apache.kafka.connect.runtime.Connect.start(Connect.java:56)
> at 
> org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:62)
> {quote}
> This is trivially reproduced by trying to start Kafka Connect in distributed 
> mode (i.e. connect-distributed.sh config/connect-distributed.properties) with 
> no broker running. However, it's not specific to Kafka Connect, it just 
> happens to use the consumer in a way that triggers it reliably.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #180

[Apache Jenkins Server]

See 

Changes:

[cshapi] Minor: Fix KafkaConsumer Constructor Summary javadoc

--
[...truncated 4428 lines...]
org.apache.kafka.connect.json.JsonConverterTest > timeToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > structToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
testConnectSchemaMetadataTranslation PASSED

org.apache.kafka.connect.json.JsonConverterTest > shortToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > dateToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > doubleToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > timeToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > mapToConnectStringKeys PASSED

org.apache.kafka.connect.json.JsonConverterTest > floatToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > decimalToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > arrayToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > 
testCacheSchemaToConnectConversion PASSED

org.apache.kafka.connect.json.JsonConverterTest > booleanToJson PASSED

org.apache.kafka.connect.json.JsonConverterTest > bytesToConnect PASSED

org.apache.kafka.connect.json.JsonConverterTest > doubleToConnect PASSED
:connect:runtime:checkstyleMain
:connect:runtime:compileTestJavawarning: [options] bootstrap class path not set 
in conjunction with -source 1.7
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
1 warning

:connect:runtime:processTestResources
:connect:runtime:testClasses
:connect:runtime:checkstyleTest
:connect:runtime:test

org.apache.kafka.connect.util.KafkaBasedLogTest > testConsumerError PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testProducerError PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testStartStop PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testReloadOnStart PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testSendAndReadToEnd FAILED
org.junit.ComparisonFailure: expected: but was:
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at 
org.apache.kafka.connect.util.KafkaBasedLogTest.testSendAndReadToEnd(KafkaBasedLogTest.java:312)

org.apache.kafka.connect.util.ShutdownableThreadTest > testGracefulShutdown 
PASSED

org.apache.kafka.connect.util.ShutdownableThreadTest > testForcibleShutdown 
PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testListConnectorsNotLeader PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testCreateConnector PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testListConnectorsNotSynced PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testGetConnectorConfig PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testDeleteConnectorNotFound PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testGetConnector PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testGetConnectorConfigConnectorNotFound PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testPutConnectorConfig PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testGetConnectorTaskConfigs PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testPutConnectorTaskConfigs PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testGetConnectorTaskConfigsConnectorNotFound PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testPutConnectorTaskConfigsConnectorNotFound PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testListConnectors PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testCreateConnectorNotLeader PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testCreateConnectorExists PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testDeleteConnector PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testDeleteConnectorNotLeader PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testPollsInBackground PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > testCommit PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testCommitTaskFlushFailure PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testCommitTaskSuccessAndFlushFailure PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > 
testCommitConsumerFailure PASSED

org.apache.kafka.connect.runtime.WorkerSinkTaskThreadedTest > testCommitTimeout 
PASSED

org.ap

Re: 0.9.0.0 RC4

[Jun Rao]

+1

Thanks,

Jun

On Fri, Nov 20, 2015 at 5:21 PM, Jun Rao  wrote:

> This is the fourth candidate for release of Apache Kafka 0.9.0.0. This a
> major release that includes (1) authentication (through SSL and SASL) and
> authorization, (2) a new java consumer, (3) a Kafka connect framework for
> data ingestion and egression, and (4) quotas. Since this is a major
> release, we will give people a bit more time for trying this out.
>
> Release Notes for the 0.9.0.0 release
>
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, Nov. 23, 6pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS in addition to the md5, sha1
> and sha2 (SHA256) checksum.
>
> * Release artifacts to be voted upon (source and binary):
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/
>
> * Maven artifacts to be voted upon prior to release:
> https://repository.apache.org/content/groups/staging/
>
> * scala-doc
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/scaladoc/
>
> * java-doc
> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/javadoc/
>
> * The tag to be voted upon (off the 0.9.0 branch) is the 0.9.0.0 tag
>
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=132943b0f83831132cd46ac961cf6f1c00132565
>
> * Documentation
> http://kafka.apache.org/090/documentation.html
>
> /***
>
> Thanks,
>
> Jun
>
>

[jira] [Commented] (KAFKA-2880) Fetcher.getTopicMetadata NullPointerException when broker cannot be reached

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2880?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15023766#comment-15023766
 ] 

Ewen Cheslack-Postava commented on KAFKA-2880:
--

[~hachikuji] Agreed, makes sense to round out error handling in general for 
those APIs.

> Fetcher.getTopicMetadata NullPointerException when broker cannot be reached
> ---
>
> Key: KAFKA-2880
> URL: https://issues.apache.org/jira/browse/KAFKA-2880
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Ewen Cheslack-Postava
>Assignee: Jason Gustafson
>
> The Fetcher class will throw a NullPointerException if a broker cannot be 
> reached:
> {quote}
> Exception in thread "main" java.lang.NullPointerException
> at 
> org.apache.kafka.common.requests.MetadataResponse.(MetadataResponse.java:130)
> at 
> org.apache.kafka.clients.consumer.internals.Fetcher.getTopicMetadata(Fetcher.java:203)
> at 
> org.apache.kafka.clients.consumer.KafkaConsumer.partitionsFor(KafkaConsumer.java:1143)
> at 
> org.apache.kafka.connect.util.KafkaBasedLog.start(KafkaBasedLog.java:126)
> at 
> org.apache.kafka.connect.storage.KafkaOffsetBackingStore.start(KafkaOffsetBackingStore.java:85)
> at org.apache.kafka.connect.runtime.Worker.start(Worker.java:108)
> at org.apache.kafka.connect.runtime.Connect.start(Connect.java:56)
> at 
> org.apache.kafka.connect.cli.ConnectDistributed.main(ConnectDistributed.java:62)
> {quote}
> This is trivially reproduced by trying to start Kafka Connect in distributed 
> mode (i.e. connect-distributed.sh config/connect-distributed.properties) with 
> no broker running. However, it's not specific to Kafka Connect, it just 
> happens to use the consumer in a way that triggers it reliably.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: 0.9.0.0 RC4

[Jun Rao]

Thanks everyone for voting.

The following are the results of the votes.

+1 binding = 4 votes (Neha Narkhede, Sriharsha Chintalapani, Guozhang Wang,
Jun Rao)
+1 non-binding = 3 votes
-1 = 0 votes
0 = 0 votes

The vote passes.

I will release artifacts to maven central, update the dist svn and download
site. Will send out an announce after that.

Jun

On Mon, Nov 23, 2015 at 8:46 PM, Jun Rao  wrote:

> +1
>
> Thanks,
>
> Jun
>
> On Fri, Nov 20, 2015 at 5:21 PM, Jun Rao  wrote:
>
>> This is the fourth candidate for release of Apache Kafka 0.9.0.0. This a
>> major release that includes (1) authentication (through SSL and SASL) and
>> authorization, (2) a new java consumer, (3) a Kafka connect framework for
>> data ingestion and egression, and (4) quotas. Since this is a major
>> release, we will give people a bit more time for trying this out.
>>
>> Release Notes for the 0.9.0.0 release
>>
>> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Monday, Nov. 23, 6pm PT
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS in addition to the md5, sha1
>> and sha2 (SHA256) checksum.
>>
>> * Release artifacts to be voted upon (source and binary):
>> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/
>>
>> * Maven artifacts to be voted upon prior to release:
>> https://repository.apache.org/content/groups/staging/
>>
>> * scala-doc
>> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/scaladoc/
>>
>> * java-doc
>> https://people.apache.org/~junrao/kafka-0.9.0.0-candidate4/javadoc/
>>
>> * The tag to be voted upon (off the 0.9.0 branch) is the 0.9.0.0 tag
>>
>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=132943b0f83831132cd46ac961cf6f1c00132565
>>
>> * Documentation
>> http://kafka.apache.org/090/documentation.html
>>
>> /***
>>
>> Thanks,
>>
>> Jun
>>
>>
>

[jira] [Reopened] (KAFKA-2842) BrokerEndPoint regex does't support hostname with _

[Sachin Pasalkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sachin Pasalkar reopened KAFKA-2842:


I just saw the console-consumer & simpleConsumer works with '_' in hostname? 
Shouldn't this be consistent?

> BrokerEndPoint regex does't support hostname with _
> ---
>
> Key: KAFKA-2842
> URL: https://issues.apache.org/jira/browse/KAFKA-2842
> Project: Kafka
>  Issue Type: Bug
>Reporter: Sachin Pasalkar
>Priority: Minor
>
> If you look at the code of BrokerEndPoint.scala, it has regex uriParseExp. 
> This regex is used for validation of brokers. However, it fails to validate 
> hostname with _ in it. e.g. adfs_212:9092



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (KAFKA-2842) BrokerEndPoint regex does't support hostname with _

[Sachin Pasalkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Sachin Pasalkar updated KAFKA-2842:
---
Comment: was deleted

(was: I just saw the console-consumer & simpleConsumer works with '_' in 
hostname? Shouldn't this be consistent?)

> BrokerEndPoint regex does't support hostname with _
> ---
>
> Key: KAFKA-2842
> URL: https://issues.apache.org/jira/browse/KAFKA-2842
> Project: Kafka
>  Issue Type: Bug
>Reporter: Sachin Pasalkar
>Priority: Minor
>
> If you look at the code of BrokerEndPoint.scala, it has regex uriParseExp. 
> This regex is used for validation of brokers. However, it fails to validate 
> hostname with _ in it. e.g. adfs_212:9092



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: All brokers are running but some partitions' leader is -1

[Prabhjot Bharaj]

Hi,

With the information provided, these are the steps I can think of (based on
the experience I had with kafka):-

1. do a describe on the topic. See if the partitions and replicas are
evenly distributed amongst all. If not, you might want to try the 'Reassign
Partitions Tool' -
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-6.ReassignPartitionsTool
2. is/are some partition(s) getting more data than others leading to an
imbalance of disk space amongst the nodes in the cluster, to an extent that
the kafka server process goes down on one or more machines in the cluster ?
3. From what I understand, your kafka and spark machines are the same ?? !!
how much memory usage the replica-0 has when your spark cluster is running
full throttle ?

Workaround -
Try running the Preferred Replica Leader Election Tool -
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools#Replicationtools-1.PreferredReplicaLeaderElectionTool
to make some replica (the one that you noticed earlier when the cluster was
all good) as the leader for this partition

Regards,
Prabhjot

On Tue, Nov 24, 2015 at 7:11 AM, Gwen Shapira  wrote:

> We fixed many many bugs since August. Since we are about to release 0.9.0
> (with SSL!), maybe wait a day and go with a released and tested version.
>
> On Mon, Nov 23, 2015 at 3:01 PM, Qi Xu  wrote:
>
> > Forgot to mention is that the Kafka version we're using is from Aug's
> > Trunk branch---which has the SSL support.
> >
> > Thanks again,
> > Qi
> >
> >
> > On Mon, Nov 23, 2015 at 2:29 PM, Qi Xu  wrote:
> >
> >> Loop another guy from our team.
> >>
> >> On Mon, Nov 23, 2015 at 2:26 PM, Qi Xu  wrote:
> >>
> >>> Hi folks,
> >>> We have a 10 node cluster and have several topics. Each topic has about
> >>> 256 partitions with 3 replica factor. Now we run into an issue that in
> some
> >>> topic, a few partition (< 10)'s leader is -1 and all of them has only
> one
> >>> synced partition.
> >>>
> >>> From the Kafka manager, here's the snapshot:
> >>> [image: Inline image 2]
> >>>
> >>> [image: Inline image 1]
> >>>
> >>> here's the state log:
> >>> [2015-11-23 21:57:58,598] ERROR Controller 1 epoch 435499 initiated
> >>> state change for partition [userlogs,84] from OnlinePartition to
> >>> OnlinePartition failed (state.change.logger)
> >>> kafka.common.StateChangeFailedException: encountered error while
> >>> electing leader for partition [userlogs,84] due to: Preferred replica
> 0 for
> >>> partition [userlogs,84] is either not alive or not in the isr. Current
> >>> leader and ISR: [{"leader":-1,"leader_epoch":203,"isr":[1]}].
> >>> Caused by: kafka.common.StateChangeFailedException: Preferred replica 0
> >>> for partition [userlogs,84] is either not alive or not in the isr.
> Current
> >>> leader and ISR: [{"leader":-1,"leader_epoch":203,"isr":[1]}]
> >>>
> >>> My question is:
> >>> 1) how could this happen and how can I fix it or work around it?
> >>> 2) Is 256 partitions too big? We have about 200+ cores for spark
> >>> streaming job.
> >>>
> >>> Thanks,
> >>> Qi
> >>>
> >>>
> >>
> >
>



-- 
-
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"