[jira] [Commented] (KAFKA-2231) Deleting a topic fails

[Bogdan Popa (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2231?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14577095#comment-14577095
 ] 

Bogdan Popa commented on KAFKA-2231:


I'm having the same situation.

It's a staging machine on AWS with one kafka broker and one zookeeper node. I'm 
using kafka 8.2.1.

On --list all topics remain as "marked for deletion" even after multiple 
restarts of kafka / zookeeper. It's worth noting that the actual files got 
deleted but not for all topics.

Another thing that might be relevant is that I've obtained this server by 
creating an image of an existing node form a production cluster and then 
changing the config to make it it's own cluster.

delete.topic.enable is true

> Deleting a topic fails
> --
>
> Key: KAFKA-2231
> URL: https://issues.apache.org/jira/browse/KAFKA-2231
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
> Environment: Windows 8.1
>Reporter: James G. Haberly
>Priority: Minor
>
> delete.topic.enable=true is in config\server.properties.
> Using --list shows the topic "marked for deletion".
> Stopping and restarting kafka and zookeeper does not delete the topic; it 
> remains "marked for deletion".
> Trying to recreate the topic fails with "Topic XXX already exists".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

[Aditya Auradkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14577509#comment-14577509
 ] 

Aditya Auradkar commented on KAFKA-1367:


[~jjkoshy] [~junrao] KAFKA-2225, even if we leave the ISR in the 
TopicMetadataRequest, how do the consumers detect which of the replicas in ISR 
to fetch from right? The consumers need to know which "zone" each of the 
brokers live in and their own in order to fetch from the closest replica (which 
mitigates with the bandwidth issues described in 2225).

> Broker topic metadata not kept in sync with ZooKeeper
> -
>
> Key: KAFKA-1367
> URL: https://issues.apache.org/jira/browse/KAFKA-1367
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0, 0.8.1
>Reporter: Ryan Berdeen
>Assignee: Ashish K Singh
>  Labels: newbie++
> Fix For: 0.8.3
>
> Attachments: KAFKA-1367.txt
>
>
> When a broker is restarted, the topic metadata responses from the brokers 
> will be incorrect (different from ZooKeeper) until a preferred replica leader 
> election.
> In the metadata, it looks like leaders are correctly removed from the ISR 
> when a broker disappears, but followers are not. Then, when a broker 
> reappears, the ISR is never updated.
> I used a variation of the Vagrant setup created by Joe Stein to reproduce 
> this with latest from the 0.8.1 branch: 
> https://github.com/also/kafka/commit/dba36a503a5e22ea039df0f9852560b4fb1e067c



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

[Aditya Auradkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14577509#comment-14577509
 ] 

Aditya Auradkar edited comment on KAFKA-1367 at 6/8/15 5:40 PM:


[~jjkoshy] [~junrao] KAFKA-2225, even if we leave the ISR in the 
TopicMetadataRequest, how do the consumers detect which of the replicas in ISR 
to fetch from right? The consumers need to know which "zone" each of the 
brokers live in and their own in order to fetch from the closest replica (which 
mitigates with the bandwidth issues described in 2225).

Couple of options:
1. Return it in BrokerMetadataRequest (KIP-24)
2. Piggyback it along with the ISR field in TMR. i.e. isr : {0: "zone1", 1: 
"zone2"}

If we choose to do (2), then the TMR will evolve anyway.


was (Author: aauradkar):
[~jjkoshy] [~junrao] KAFKA-2225, even if we leave the ISR in the 
TopicMetadataRequest, how do the consumers detect which of the replicas in ISR 
to fetch from right? The consumers need to know which "zone" each of the 
brokers live in and their own in order to fetch from the closest replica (which 
mitigates with the bandwidth issues described in 2225).

> Broker topic metadata not kept in sync with ZooKeeper
> -
>
> Key: KAFKA-1367
> URL: https://issues.apache.org/jira/browse/KAFKA-1367
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0, 0.8.1
>Reporter: Ryan Berdeen
>Assignee: Ashish K Singh
>  Labels: newbie++
> Fix For: 0.8.3
>
> Attachments: KAFKA-1367.txt
>
>
> When a broker is restarted, the topic metadata responses from the brokers 
> will be incorrect (different from ZooKeeper) until a preferred replica leader 
> election.
> In the metadata, it looks like leaders are correctly removed from the ISR 
> when a broker disappears, but followers are not. Then, when a broker 
> reappears, the ISR is never updated.
> I used a variation of the Vagrant setup created by Joe Stein to reproduce 
> this with latest from the 0.8.1 branch: 
> https://github.com/also/kafka/commit/dba36a503a5e22ea039df0f9852560b4fb1e067c



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14577563#comment-14577563
 ] 

Jun Rao commented on KAFKA-1367:


Yes, we need some kind of notion of zones for both the brokers and the clients. 
Each broker and each client (producer/consumer) need a configuration for which 
zone it belongs to. It's probably simpler to just return the zone info in TMR. 
We will need to evolve TMR, but that can probably be done separately from 
fixing the ISR in TMR. We probably should move these design discussions to 
KAFKA-2225 itself.

> Broker topic metadata not kept in sync with ZooKeeper
> -
>
> Key: KAFKA-1367
> URL: https://issues.apache.org/jira/browse/KAFKA-1367
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0, 0.8.1
>Reporter: Ryan Berdeen
>Assignee: Ashish K Singh
>  Labels: newbie++
> Fix For: 0.8.3
>
> Attachments: KAFKA-1367.txt
>
>
> When a broker is restarted, the topic metadata responses from the brokers 
> will be incorrect (different from ZooKeeper) until a preferred replica leader 
> election.
> In the metadata, it looks like leaders are correctly removed from the ISR 
> when a broker disappears, but followers are not. Then, when a broker 
> reappears, the ISR is never updated.
> I used a variation of the Vagrant setup created by Joe Stein to reproduce 
> this with latest from the 0.8.1 branch: 
> https://github.com/also/kafka/commit/dba36a503a5e22ea039df0f9852560b4fb1e067c



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2225) Allow fetching from ISR

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2225?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14577571#comment-14577571
 ] 

Jun Rao commented on KAFKA-2225:


A few notes on this.

1. We probably should only do this on the new java consumer.
2. We will need some kind of notion of zones in both the brokers and the 
clients.
3. We probably need to fix ISR in TMR first. There is some discussion about 
this in KAFKA-1367. 

> Allow fetching from ISR
> ---
>
> Key: KAFKA-2225
> URL: https://issues.apache.org/jira/browse/KAFKA-2225
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Theo Hultberg
>Assignee: Parth Brahmbhatt
>
> Currently clients are not allowed to fetch from replicas, even if they are in 
> sync with the master. If I'm not missing anything significant it shouldn't be 
> any difference fetching from the leader or an ISR, besides maybe some extra 
> latency.
> For our use case it would be very beneficial to be able to fetch from 
> replicas instead of just the leader. We run Kafka clusters that replicate 
> across EC2 availability zones, and each byte sent between zones costs money. 
> This bandwith usage costs us about the same as it costs to run the instances. 
> If we could fetch from a replica in the same zone as the client we could 
> avoid some of this cost.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 35201: Fix KAFKA-2253

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35201/
---

(Updated June 8, 2015, 6:47 p.m.)


Review request for kafka.


Bugs: KAFKA-2253
https://issues.apache.org/jira/browse/KAFKA-2253


Repository: kafka


Description
---

Incorporated Jiangjie and Onur's comments


Diffs (updated)
-

  core/src/main/scala/kafka/server/DelayedOperation.scala 
123078d97a7bfe2121655c00f3b2c6af21c53015 

Diff: https://reviews.apache.org/r/35201/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Updated] (KAFKA-2253) Deadlock in delayed operation purgatory

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2253:
-
Attachment: KAFKA-2253_2015-06-08_11:47:40.patch

> Deadlock in delayed operation purgatory
> ---
>
> Key: KAFKA-2253
> URL: https://issues.apache.org/jira/browse/KAFKA-2253
> Project: Kafka
>  Issue Type: Bug
>Reporter: Mayuresh Gharat
>Assignee: Guozhang Wang
> Fix For: 0.8.3
>
> Attachments: KAFKA-2253.patch, KAFKA-2253.patch, 
> KAFKA-2253_2015-06-08_11:47:40.patch
>
>
> We hit a deadlock while running brokers with git hash: 
> 9e894aa0173b14d64a900bcf780d6b7809368384
> There's a circular wait between the removeWatchersLock and an operations 
> intrinsic lock.
> {code}
> Found one Java-level deadlock:
> =
> "kafka-request-handler-a":
>   waiting for ownable synchronizer 0x0006da08f9e0, (a 
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
>   which is held by "ExpirationReaper-xyz"
> "ExpirationReaper-xyz":
>   waiting to lock monitor 0x7f454e18 (object 0x0006b0563fe8, a 
> java.util.LinkedList),
>   which is held by "kafka-request-handler-b"
> "kafka-request-handler-b":
>   waiting for ownable synchronizer 0x0006da08f9e0, (a 
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
>   which is held by "ExpirationReaper-xyz"
> "kafka-request-handler-a":
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0006da08f9e0> (a 
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
> at 
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:296)
> at kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:304)
> at 
> kafka.server.DelayedOperationPurgatory.checkAndComplete(DelayedOperation.scala:224)
> at 
> kafka.server.ReplicaManager.tryCompleteDelayedFetch(ReplicaManager.scala:166)
> at 
> kafka.cluster.Partition.kafka$cluster$Partition$$maybeIncrementLeaderHW(Partition.scala:358)
> at 
> kafka.cluster.Partition$$anonfun$maybeExpandIsr$1.apply$mcV$sp(Partition.scala:288)
> at 
> kafka.cluster.Partition$$anonfun$maybeExpandIsr$1.apply(Partition.scala:270)
> at 
> kafka.cluster.Partition$$anonfun$maybeExpandIsr$1.apply(Partition.scala:270)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:298)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:306)
> at kafka.cluster.Partition.maybeExpandIsr(Partition.scala:268)
> at 
> kafka.cluster.Partition.updateReplicaLogReadResult(Partition.scala:244)
> at 
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:790)
> at 
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:787)
> at scala.collection.immutable.Map$Map4.foreach(Map.scala:181)
> at 
> kafka.server.ReplicaManager.updateFollowerLogReadResults(ReplicaManager.scala:787)
> at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:432)
> at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:312)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:60)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> "ExpirationReaper-xyz":
> at 
> kafka.server.DelayedOperationPurgatory$Watchers.watched(DelayedOperation.scala:278)
> - waiting to lock <0x0006b0563fe8> (a java.util.LinkedList)
> at 
> kafka.server.DelayedOperationPurgatory$$anonfun$kafka$server$DelayedOperationPurgatory$$removeKeyIfEmpty$1.apply(DelayedOperation.scala:258)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:298)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:306)
> at 
> kafka.server.DelayedOperationPurgatory.kafka$server$DelayedOperationPurgatory$$removeKeyIfEmpty(DelayedOperation.scala:256)
> at 
> kafka.server.DelayedOperationPurgatory$Watchers.purgeCompleted(DelayedOperation.scala:322)
> - locked <0x00071a86a478> (a java.util.LinkedList)
> at 
> kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper$$anonfun$3.apply(DelayedOperat

[jira] [Commented] (KAFKA-2253) Deadlock in delayed operation purgatory

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2253?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14577639#comment-14577639
 ] 

Guozhang Wang commented on KAFKA-2253:
--

Updated reviewboard https://reviews.apache.org/r/35201/diff/
 against branch origin/trunk

> Deadlock in delayed operation purgatory
> ---
>
> Key: KAFKA-2253
> URL: https://issues.apache.org/jira/browse/KAFKA-2253
> Project: Kafka
>  Issue Type: Bug
>Reporter: Mayuresh Gharat
>Assignee: Guozhang Wang
> Fix For: 0.8.3
>
> Attachments: KAFKA-2253.patch, KAFKA-2253.patch, 
> KAFKA-2253_2015-06-08_11:47:40.patch
>
>
> We hit a deadlock while running brokers with git hash: 
> 9e894aa0173b14d64a900bcf780d6b7809368384
> There's a circular wait between the removeWatchersLock and an operations 
> intrinsic lock.
> {code}
> Found one Java-level deadlock:
> =
> "kafka-request-handler-a":
>   waiting for ownable synchronizer 0x0006da08f9e0, (a 
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
>   which is held by "ExpirationReaper-xyz"
> "ExpirationReaper-xyz":
>   waiting to lock monitor 0x7f454e18 (object 0x0006b0563fe8, a 
> java.util.LinkedList),
>   which is held by "kafka-request-handler-b"
> "kafka-request-handler-b":
>   waiting for ownable synchronizer 0x0006da08f9e0, (a 
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
>   which is held by "ExpirationReaper-xyz"
> "kafka-request-handler-a":
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for  <0x0006da08f9e0> (a 
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:967)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1283)
> at 
> java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:727)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:296)
> at kafka.utils.CoreUtils$.inReadLock(CoreUtils.scala:304)
> at 
> kafka.server.DelayedOperationPurgatory.checkAndComplete(DelayedOperation.scala:224)
> at 
> kafka.server.ReplicaManager.tryCompleteDelayedFetch(ReplicaManager.scala:166)
> at 
> kafka.cluster.Partition.kafka$cluster$Partition$$maybeIncrementLeaderHW(Partition.scala:358)
> at 
> kafka.cluster.Partition$$anonfun$maybeExpandIsr$1.apply$mcV$sp(Partition.scala:288)
> at 
> kafka.cluster.Partition$$anonfun$maybeExpandIsr$1.apply(Partition.scala:270)
> at 
> kafka.cluster.Partition$$anonfun$maybeExpandIsr$1.apply(Partition.scala:270)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:298)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:306)
> at kafka.cluster.Partition.maybeExpandIsr(Partition.scala:268)
> at 
> kafka.cluster.Partition.updateReplicaLogReadResult(Partition.scala:244)
> at 
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:790)
> at 
> kafka.server.ReplicaManager$$anonfun$updateFollowerLogReadResults$2.apply(ReplicaManager.scala:787)
> at scala.collection.immutable.Map$Map4.foreach(Map.scala:181)
> at 
> kafka.server.ReplicaManager.updateFollowerLogReadResults(ReplicaManager.scala:787)
> at kafka.server.ReplicaManager.fetchMessages(ReplicaManager.scala:432)
> at kafka.server.KafkaApis.handleFetchRequest(KafkaApis.scala:312)
> at kafka.server.KafkaApis.handle(KafkaApis.scala:60)
> at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
> at java.lang.Thread.run(Thread.java:745)
> "ExpirationReaper-xyz":
> at 
> kafka.server.DelayedOperationPurgatory$Watchers.watched(DelayedOperation.scala:278)
> - waiting to lock <0x0006b0563fe8> (a java.util.LinkedList)
> at 
> kafka.server.DelayedOperationPurgatory$$anonfun$kafka$server$DelayedOperationPurgatory$$removeKeyIfEmpty$1.apply(DelayedOperation.scala:258)
> at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:298)
> at kafka.utils.CoreUtils$.inWriteLock(CoreUtils.scala:306)
> at 
> kafka.server.DelayedOperationPurgatory.kafka$server$DelayedOperationPurgatory$$removeKeyIfEmpty(DelayedOperation.scala:256)
> at 
> kafka.server.DelayedOperationPurgatory$Watchers.purgeCompleted(DelayedOperation.scala:322)
> - locked <0x00071a86a478> (a java.util.LinkedList)
> at 

Re: Review Request 35201: Fix KAFKA-2253

[Jiangjie Qin]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35201/#review87065
---

Ship it!


Ship It!

- Jiangjie Qin


On June 8, 2015, 6:47 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35201/
> ---
> 
> (Updated June 8, 2015, 6:47 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2253
> https://issues.apache.org/jira/browse/KAFKA-2253
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Incorporated Jiangjie and Onur's comments
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/server/DelayedOperation.scala 
> 123078d97a7bfe2121655c00f3b2c6af21c53015 
> 
> Diff: https://reviews.apache.org/r/35201/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14577728#comment-14577728
 ] 

Gwen Shapira commented on KAFKA-1367:
-

By "zones" do we mean rack-awareness? Or more general locality notion?
Sounds like something that may need its own JIRA and design.

> Broker topic metadata not kept in sync with ZooKeeper
> -
>
> Key: KAFKA-1367
> URL: https://issues.apache.org/jira/browse/KAFKA-1367
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0, 0.8.1
>Reporter: Ryan Berdeen
>Assignee: Ashish K Singh
>  Labels: newbie++
> Fix For: 0.8.3
>
> Attachments: KAFKA-1367.txt
>
>
> When a broker is restarted, the topic metadata responses from the brokers 
> will be incorrect (different from ZooKeeper) until a preferred replica leader 
> election.
> In the metadata, it looks like leaders are correctly removed from the ISR 
> when a broker disappears, but followers are not. Then, when a broker 
> reappears, the ISR is never updated.
> I used a variation of the Vagrant setup created by Joe Stein to reproduce 
> this with latest from the 0.8.1 branch: 
> https://github.com/also/kafka/commit/dba36a503a5e22ea039df0f9852560b4fb1e067c



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14577758#comment-14577758
 ] 

Jiangjie Qin commented on KAFKA-1367:
-

I agree with [~gwenshap], it sounds this deserves a KIP.

> Broker topic metadata not kept in sync with ZooKeeper
> -
>
> Key: KAFKA-1367
> URL: https://issues.apache.org/jira/browse/KAFKA-1367
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0, 0.8.1
>Reporter: Ryan Berdeen
>Assignee: Ashish K Singh
>  Labels: newbie++
> Fix For: 0.8.3
>
> Attachments: KAFKA-1367.txt
>
>
> When a broker is restarted, the topic metadata responses from the brokers 
> will be incorrect (different from ZooKeeper) until a preferred replica leader 
> election.
> In the metadata, it looks like leaders are correctly removed from the ISR 
> when a broker disappears, but followers are not. Then, when a broker 
> reappears, the ISR is never updated.
> I used a variation of the Vagrant setup created by Joe Stein to reproduce 
> this with latest from the 0.8.1 branch: 
> https://github.com/also/kafka/commit/dba36a503a5e22ea039df0f9852560b4fb1e067c



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume performance under windows and some old Linux file system

[Jay Kreps]

Honghai,

You are going to do a vote on this, right? I think the patch is ready to go
so we are just waiting on the KIP adoption I think.

-Jay

On Thu, May 21, 2015 at 8:50 AM, Jun Rao  wrote:

> Honghai,
>
> Could you update the wiki on the preallocated size? Instead of
> config.segmentSize
> - 2 * config.maxMessageSize, we just want to use config.segmentSize.
>
> Thanks,
>
> Jun
>
> On Mon, May 11, 2015 at 8:16 PM, Honghai Chen 
> wrote:
>
> > All issues fixed, test cases added, performance result on windows
> > attached.  The patch can help improve the consume performance around
> > 25%~50%.
> >
> > Thanks, Honghai Chen
> >
> > -Original Message-
> > From: Jun Rao [mailto:j...@confluent.io]
> > Sent: Wednesday, May 6, 2015 5:39 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve consume
> > performance under windows and some old Linux file system
> >
> > Thanks. Could you updated the wiki? Also, commented on the jira.
> >
> > Jun
> >
> > On Tue, May 5, 2015 at 12:48 AM, Honghai Chen <
> honghai.c...@microsoft.com>
> > wrote:
> >
> > > Use config.segmentSize should be ok.   Previously add that one for make
> > > sure the file not exceed config.segmentSize, actually the function
> > > maybeRoll already make sure that.
> > > When try add test case for recover, blocked by the rename related
> > > issue, just open one jira at
> > > https://issues.apache.org/jira/browse/KAFKA-2170 , any recommendation
> > for fix that issue?
> > >
> > > Thanks, Honghai Chen
> > >
> > > -Original Message-
> > > From: Jun Rao [mailto:j...@confluent.io]
> > > Sent: Tuesday, May 5, 2015 12:51 PM
> > > To: dev@kafka.apache.org
> > > Subject: Re: [DISCUSS] KIP 20 Enable log preallocate to improve
> > > consume performance under windows and some old Linux file system
> > >
> > > This seems similar to what's in
> > > https://issues.apache.org/jira/browse/KAFKA-1065.
> > >
> > > Also, could you explain why the preallocated size is set to
> > > config.segmentSize
> > > - 2 * config.maxMessageSize, instead of just config.segmentSize?
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Mon, May 4, 2015 at 8:12 PM, Honghai Chen
> > > 
> > > wrote:
> > >
> > > >   Hi guys,
> > > > I'm trying add test cases, but below case crashed at line "
> > > > segReopen.recover(64*1024)--> index.trimToValidSize()  ", any idea
> > > > for
> > > it?
> > > > Appreciate your help.
> > > > The case assume kafka suddenly crash, and need recover the
> > > > last segment.
> > > >
> > > > kafka.log.LogSegmentTest > testCreateWithInitFileSizeCrash FAILED
> > > > java.io.IOException: The requested operation cannot be performed
> > > > on a file w ith a user-mapped section open
> > > > at java.io.RandomAccessFile.setLength(Native Method)
> > > > at
> > > > kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:292)
> > > > at
> > > > kafka.log.OffsetIndex$$anonfun$resize$1.apply(OffsetIndex.scala:283)
> > > > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:288)
> > > > at kafka.log.OffsetIndex.resize(OffsetIndex.scala:283)
> > > > at
> > > > kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply$mcV$sp(Offset
> > > > I
> > > > ndex.scala:272)
> > > > at
> > > > kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply(OffsetIndex.s
> > > > c
> > > > ala:272)
> > > > at
> > > > kafka.log.OffsetIndex$$anonfun$trimToValidSize$1.apply(OffsetIndex.s
> > > > c
> > > > ala:272)
> > > > at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:288)
> > > > at
> kafka.log.OffsetIndex.trimToValidSize(OffsetIndex.scala:271)
> > > > at kafka.log.LogSegment.recover(LogSegment.scala:199)
> > > > at
> > > > kafka.log.LogSegmentTest.testCreateWithInitFileSizeCrash(LogSegmentT
> > > > e
> > > > st.scala:306)
> > > >
> > > >   def recover(maxMessageSize: Int): Int = {
> > > > index.truncate()
> > > > index.resize(index.maxIndexSize)
> > > > var validBytes = 0
> > > > var lastIndexEntry = 0
> > > > val iter = log.iterator(maxMessageSize)
> > > > try {
> > > >   while(iter.hasNext) {
> > > > val entry = iter.next
> > > > entry.message.ensureValid()
> > > > if(validBytes - lastIndexEntry > indexIntervalBytes) {
> > > >   // we need to decompress the message, if required, to get
> > > > the offset of the first uncompressed message
> > > >   val startOffset =
> > > > entry.message.compressionCodec match {
> > > >   case NoCompressionCodec =>
> > > > entry.offset
> > > >   case _ =>
> > > >
> > > > ByteBufferMessageSet.deepIterator(entry.message).next().offset
> > > >   }
> > > >   index.append(startOffset, validBytes)
> > > >   lastIndexEntry = validBytes
> > > > }
> > > > validBytes += MessageSet.entrySize(entry.message)
> > > >   }
> > > > } cat

Re: Review Request 35201: Fix KAFKA-2253

[Onur Karaman]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35201/#review87077
---

Ship it!


Ship It!

- Onur Karaman


On June 8, 2015, 6:47 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35201/
> ---
> 
> (Updated June 8, 2015, 6:47 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2253
> https://issues.apache.org/jira/browse/KAFKA-2253
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Incorporated Jiangjie and Onur's comments
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/server/DelayedOperation.scala 
> 123078d97a7bfe2121655c00f3b2c6af21c53015 
> 
> Diff: https://reviews.apache.org/r/35201/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

[GitHub] kafka pull request: Make MockProducer generic

[apakulov]

GitHub user apakulov opened a pull request:

https://github.com/apache/kafka/pull/68

Make MockProducer generic

MockConsumer and MockProducer have been moved to test source set.
KeySerializer and ValueSerializer have been added to mimic actual 
KafkaProducer behavior.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apakulov/kafka KAFKA-2232

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/68.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #68


commit 2e06015752b1b6ccae263c03778668959271e152
Author: Alexander Pakulov 
Date:   2015-06-08T20:40:26Z

Make MockProducer generic




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2232) make MockProducer generic

[Alexander Pakulov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14577806#comment-14577806
 ] 

Alexander Pakulov commented on KAFKA-2232:
--

Pull request has been submitted: https://github.com/apache/kafka/pull/68
I've moved MockProducer and MockConsumer to test source set, since they only 
have been used for tests.

> make MockProducer generic
> -
>
> Key: KAFKA-2232
> URL: https://issues.apache.org/jira/browse/KAFKA-2232
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.8.2.1
>Reporter: Jun Rao
>  Labels: newbie
> Fix For: 0.8.3
>
>
> Currently, MockProducer implements Producer. Instead, we 
> should implement MockProducer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14577881#comment-14577881
 ] 

Jun Rao commented on KAFKA-1690:


[~sriharsha], thanks for the latest patch. Haven't looked at it in details. A 
couple of comments.

1. Could you rebase now that KAFKA-1928 is done?
2. We need to think a bit more about supporting the zero-copy transfer on the 
consumer. Currently, the broker sends data to the consumer client using the 
socket.writeTo() api (in PartitionDataSend, TopicDataSend, and 
FetchResponseSend). We want to preserve this optimization for plain socket 
channels. Only for SSL channel, we will copy the bytes from the file channel 
into the application and then send it to the remote socket. One way is probably 
to have Channel implement GatheringByteChannel and ScatteringByteChannel.

> new java producer needs ssl support as a client
> ---
>
> Key: KAFKA-1690
> URL: https://issues.apache.org/jira/browse/KAFKA-1690
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Joe Stein
>Assignee: Sriharsha Chintalapani
> Fix For: 0.8.3
>
> Attachments: KAFKA-1690.patch, KAFKA-1690.patch, 
> KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, 
> KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, 
> KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, 
> KAFKA-1690_2015-05-21_10:37:08.patch, KAFKA-1690_2015-06-03_18:52:29.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client

[Sriharsha Chintalapani (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14577912#comment-14577912
 ] 

Sriharsha Chintalapani commented on KAFKA-1690:
---

[~junrao] currently working on rebasing on KAFKA-1928. 
2. My earlier  version have channel implementing GatheringByteChannel and 
ScatteringByteChannel . I thought from our previous conversation you wanted 
Channel to own NetworkReceive and Send. I am currently keeping Send and Receive 
to work with socketChannel . I'll try to upload the new patch with KAFKA-1928 
soon.

> new java producer needs ssl support as a client
> ---
>
> Key: KAFKA-1690
> URL: https://issues.apache.org/jira/browse/KAFKA-1690
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Joe Stein
>Assignee: Sriharsha Chintalapani
> Fix For: 0.8.3
>
> Attachments: KAFKA-1690.patch, KAFKA-1690.patch, 
> KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, 
> KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, 
> KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, 
> KAFKA-1690_2015-05-21_10:37:08.patch, KAFKA-1690_2015-06-03_18:52:29.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 34492: Patch for KAFKA-2210

[Jun Rao]

> On June 3, 2015, 11:36 p.m., Parth Brahmbhatt wrote:
> > core/src/main/scala/kafka/security/auth/Acl.scala, lines 57-62
> > 
> >
> > I tried exactly that but it tunrs out our current Json parser does not 
> > work when a json string has other special characters, somehow gets into 
> > some double parsing and fails. Has been long since I wrote this code so 
> > dont exactly remember why it was failing but I did try it and with current 
> > JsonUtil it does not work.

Could you explain a bit which part doesn't work? The following simple test 
works for me.

scala> val a = "[{\"a\": \"aa\"}]"
a: String = [{"a": "aa"}]

scala> JSON.parseFull(a)
res4: Option[Any] = Some(List(Map(a -> aa)))


> On June 3, 2015, 11:36 p.m., Parth Brahmbhatt wrote:
> > core/src/main/scala/kafka/security/auth/KafkaPrincipal.scala, line 22
> > 
> >
> > I haven't added Group support yet but they will be of the form 
> > Group:. Why did you get the impression that groups will not 
> > have ":"

Oh, I was just saying that if the group name itself can contain ":", parsing 
will be more difficult if ":" is the separator.


> On June 3, 2015, 11:36 p.m., Parth Brahmbhatt wrote:
> > core/src/main/scala/kafka/security/auth/KafkaPrincipal.scala, line 41
> > 
> >
> > Yes we can and as mentioned in the design doc when no authentication is 
> > configured it will be set as User:DrWho?.

So, I guess authentication will always authenticate at the user level and it's 
up to the Authorization model to implement the user to group mapping?


> On June 3, 2015, 11:36 p.m., Parth Brahmbhatt wrote:
> > core/src/main/scala/kafka/security/auth/Operation.java, line 22
> > 
> >
> > I grepped through kafka code base to understand how enums were used in 
> > other parts and all places used java enums. I assumed that was the 
> > convention . If that is not the case I can change all enum classes in core 
> > to use http://www.scala-lang.org/api/current/index.html#scala.Enumeration

Under core/, we don't have java files except when defining the java api. We 
implement enum using case object in scala (see BrokerStates as an example).


> On June 3, 2015, 11:36 p.m., Parth Brahmbhatt wrote:
> > core/src/test/scala/unit/kafka/security/auth/AclTest.scala, line 36
> > 
> >
> > can you elloborate why do you think that is a better approach?

I was thinking of just embedding the acl json string in the code.


> On June 3, 2015, 11:36 p.m., Parth Brahmbhatt wrote:
> > core/src/main/scala/kafka/security/auth/Authorizer.scala, line 36
> > 
> >
> > In the KIP dicussion it was proposed to add a config 
> > authoizer.config.path which will contain path to a property files on all 
> > broker hosts. This is how the plugin specific property file gets passed on. 
> > Do we want to instead use configurable?

Sorry, but I missed this in the KIP review. I think it's probably better NOT to 
have another config.path inside a configuration file. We already have other 
pluggable logic such as the MetrisReporter and will be adding other pluggable 
logic such as PrincipalExtractor in KAFKA-1690. Introducing a separate config 
path for each pluggable logic may not be ideal. Also, currently, we allow 
people to instantiate KafkaServerStartble directly so that people can obtain 
the properties from any configuration system and pass them to Kafka, instead of 
assuming that the properties are always specified in a file. So, it's probably 
better to specify the properties needed by any pluggable logic in the same 
property file, then pass them to the pluggable logic through the configure() 
api. We have KAFKA-2249 filed to allow KafkaConfig to do this. Perhaps, we can 
fix KAFKA-2249 first.


- Jun


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34492/#review86267
---


On June 4, 2015, 11:07 p.m., Parth Brahmbhatt wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34492/
> ---
> 
> (Updated June 4, 2015, 11:07 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2210
> https://issues.apache.org/jira/browse/KAFKA-2210
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Addressing review comments from Jun.
> 
> 
> Adding CREATE check for offset topic only if the topic does not exist already.
> 

[jira] [Created] (KAFKA-2256) Port system tests

[Geoffrey Anderson (JIRA)]

Geoffrey Anderson created KAFKA-2256:


 Summary: Port system tests
 Key: KAFKA-2256
 URL: https://issues.apache.org/jira/browse/KAFKA-2256
 Project: Kafka
  Issue Type: Improvement
  Components: system tests
Reporter: Geoffrey Anderson
Assignee: Geoffrey Anderson
 Fix For: 0.8.3


This is a tracking issue for the system test suites to be ported per KIP-25



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2257) port replication_testsuite

[Geoffrey Anderson (JIRA)]

Geoffrey Anderson created KAFKA-2257:


 Summary: port replication_testsuite
 Key: KAFKA-2257
 URL: https://issues.apache.org/jira/browse/KAFKA-2257
 Project: Kafka
  Issue Type: Sub-task
Reporter: Geoffrey Anderson
Assignee: Geoffrey Anderson
 Fix For: 0.8.3


Port subset of replication_testsuite to run on ducktape. Details to follow



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2258) Port mirrormaker_testsuite

[Geoffrey Anderson (JIRA)]

Geoffrey Anderson created KAFKA-2258:


 Summary: Port mirrormaker_testsuite
 Key: KAFKA-2258
 URL: https://issues.apache.org/jira/browse/KAFKA-2258
 Project: Kafka
  Issue Type: Sub-task
Reporter: Geoffrey Anderson
Assignee: Geoffrey Anderson


Port mirrormaker_testsuite to run on ducktape



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14578020#comment-14578020
 ] 

Jun Rao commented on KAFKA-1690:


2. I am not sure what the best approach is. Currently, Send.writeTo works with 
GatheringByteChannel, which is simpler than SocketChannel. So, it's probably 
easier to keep it this way.

> new java producer needs ssl support as a client
> ---
>
> Key: KAFKA-1690
> URL: https://issues.apache.org/jira/browse/KAFKA-1690
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Joe Stein
>Assignee: Sriharsha Chintalapani
> Fix For: 0.8.3
>
> Attachments: KAFKA-1690.patch, KAFKA-1690.patch, 
> KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, 
> KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, 
> KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, 
> KAFKA-1690_2015-05-21_10:37:08.patch, KAFKA-1690_2015-06-03_18:52:29.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2259) port offset_management_testsuite

[Geoffrey Anderson (JIRA)]

Geoffrey Anderson created KAFKA-2259:


 Summary: port offset_management_testsuite
 Key: KAFKA-2259
 URL: https://issues.apache.org/jira/browse/KAFKA-2259
 Project: Kafka
  Issue Type: Sub-task
Reporter: Geoffrey Anderson
Assignee: Geoffrey Anderson
 Fix For: 0.8.3


Port to run on ducktape



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 35231: Fix KAFKA-1740

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35231/
---

Review request for kafka.


Bugs: KAFKA-2208
https://issues.apache.org/jira/browse/KAFKA-2208


Repository: kafka


Description
---

Move offset manager to coordinator, add validation logic for offset commit and 
fetch


Diffs
-

  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Coordinator.java
 c1496a0851526f3c7d3905ce4bdff2129c83a6c1 
  clients/src/main/java/org/apache/kafka/common/protocol/Errors.java 
5b898c8f8ad5d0469f469b600c4b2eb13d1fc662 
  
clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitResponse.java
 70844d65369f6ff300cbeb513dbb6650050c7eec 
  
clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchRequest.java 
deec1fa480d5a5c5884a1c007b076aa64e902472 
  
clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchResponse.java 
512a0ef7e619d54e74122c38119209f5cf9590e3 
  
clients/src/test/java/org/apache/kafka/clients/consumer/internals/CoordinatorTest.java
 b06c4a73e2b4e9472cd772c8bc32bf4a29f431bb 
  core/src/main/scala/kafka/admin/TopicCommand.scala 
dacbdd089f96c67b7bbc1c0cd36490d4fe094c1b 
  core/src/main/scala/kafka/cluster/Partition.scala 
730a232482fdf77be5704cdf5941cfab3828db88 
  core/src/main/scala/kafka/common/OffsetMetadataAndError.scala 
6b4242c7cd1df9b3465db0fec35a25102c76cd60 
  core/src/main/scala/kafka/common/Topic.scala 
ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 
  core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala 
51e89c87ee2c20fc7f976536f01fa1055fb8e670 
  core/src/main/scala/kafka/coordinator/CoordinatorMetadata.scala 
c39e6de34ee531c6dfa9107b830752bd7f8fbe59 
  core/src/main/scala/kafka/server/KafkaApis.scala 
d63bc18d795a6f0e6994538ca55a9a46f7fb8ffd 
  core/src/main/scala/kafka/server/KafkaServer.scala 
b320ce9f6a12c0ee392e91beb82e8804d167f9f4 
  core/src/main/scala/kafka/server/OffsetManager.scala 
5cca85cf727975f6d3acb2223fd186753ad761dc 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/test/scala/integration/kafka/api/ConsumerTest.scala 
17b17b9b1520c7cc2e2ba96cdb1f9ff06e47bcad 
  core/src/test/scala/integration/kafka/api/IntegrationTestHarness.scala 
07b1ff47bfc3cd3f948c9533c8dc977fa36d996f 
  core/src/test/scala/unit/kafka/admin/TopicCommandTest.scala 
c7136f20972614ac47aa57ab13e3c94ef775a4b7 
  core/src/test/scala/unit/kafka/consumer/TopicFilterTest.scala 
4f124af5c3e946045a78ad1519c37372a72c8985 
  core/src/test/scala/unit/kafka/coordinator/CoordinatorMetadataTest.scala 
08854c5e6ec249368206298b2ac2623df18f266a 
  core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala 
528525b719ec916e16f8b3ae3715bec4b5dcc47d 

Diff: https://reviews.apache.org/r/35231/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Updated] (KAFKA-2208) Consumer should detect coordinator failures

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2208:
-
Attachment: KAFKA-2208.patch

> Consumer should detect coordinator failures
> ---
>
> Key: KAFKA-2208
> URL: https://issues.apache.org/jira/browse/KAFKA-2208
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Onur Karaman
>Assignee: Guozhang Wang
> Attachments: KAFKA-2208.patch, KAFKA-2208.patch, 
> KAFKA-2208_2015-05-20_19:14:58.patch, KAFKA-2208_2015-05-31_17:05:06.patch, 
> KAFKA-2208_2015-06-02_10:45:34.patch, KAFKA-2208_2015-06-02_14:06:58.patch
>
>
> Add coordinator discovery and failure detection to the consumer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2208) Consumer should detect coordinator failures

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14578030#comment-14578030
 ] 

Guozhang Wang commented on KAFKA-2208:
--

Created reviewboard https://reviews.apache.org/r/35231/diff/
 against branch origin/trunk

> Consumer should detect coordinator failures
> ---
>
> Key: KAFKA-2208
> URL: https://issues.apache.org/jira/browse/KAFKA-2208
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Onur Karaman
>Assignee: Guozhang Wang
> Attachments: KAFKA-2208.patch, KAFKA-2208.patch, 
> KAFKA-2208_2015-05-20_19:14:58.patch, KAFKA-2208_2015-05-31_17:05:06.patch, 
> KAFKA-2208_2015-06-02_10:45:34.patch, KAFKA-2208_2015-06-02_14:06:58.patch
>
>
> Add coordinator discovery and failure detection to the consumer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 35231: Fix KAFKA-1740

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35231/
---

(Updated June 8, 2015, 11:12 p.m.)


Review request for kafka.


Bugs: KAFKA-1740
https://issues.apache.org/jira/browse/KAFKA-1740


Repository: kafka


Description
---

Move offset manager to coordinator, add validation logic for offset commit and 
fetch


Diffs
-

  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Coordinator.java
 c1496a0851526f3c7d3905ce4bdff2129c83a6c1 
  clients/src/main/java/org/apache/kafka/common/protocol/Errors.java 
5b898c8f8ad5d0469f469b600c4b2eb13d1fc662 
  
clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitResponse.java
 70844d65369f6ff300cbeb513dbb6650050c7eec 
  
clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchRequest.java 
deec1fa480d5a5c5884a1c007b076aa64e902472 
  
clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchResponse.java 
512a0ef7e619d54e74122c38119209f5cf9590e3 
  
clients/src/test/java/org/apache/kafka/clients/consumer/internals/CoordinatorTest.java
 b06c4a73e2b4e9472cd772c8bc32bf4a29f431bb 
  core/src/main/scala/kafka/admin/TopicCommand.scala 
dacbdd089f96c67b7bbc1c0cd36490d4fe094c1b 
  core/src/main/scala/kafka/cluster/Partition.scala 
730a232482fdf77be5704cdf5941cfab3828db88 
  core/src/main/scala/kafka/common/OffsetMetadataAndError.scala 
6b4242c7cd1df9b3465db0fec35a25102c76cd60 
  core/src/main/scala/kafka/common/Topic.scala 
ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 
  core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala 
51e89c87ee2c20fc7f976536f01fa1055fb8e670 
  core/src/main/scala/kafka/coordinator/CoordinatorMetadata.scala 
c39e6de34ee531c6dfa9107b830752bd7f8fbe59 
  core/src/main/scala/kafka/server/KafkaApis.scala 
d63bc18d795a6f0e6994538ca55a9a46f7fb8ffd 
  core/src/main/scala/kafka/server/KafkaServer.scala 
b320ce9f6a12c0ee392e91beb82e8804d167f9f4 
  core/src/main/scala/kafka/server/OffsetManager.scala 
5cca85cf727975f6d3acb2223fd186753ad761dc 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/test/scala/integration/kafka/api/ConsumerTest.scala 
17b17b9b1520c7cc2e2ba96cdb1f9ff06e47bcad 
  core/src/test/scala/integration/kafka/api/IntegrationTestHarness.scala 
07b1ff47bfc3cd3f948c9533c8dc977fa36d996f 
  core/src/test/scala/unit/kafka/admin/TopicCommandTest.scala 
c7136f20972614ac47aa57ab13e3c94ef775a4b7 
  core/src/test/scala/unit/kafka/consumer/TopicFilterTest.scala 
4f124af5c3e946045a78ad1519c37372a72c8985 
  core/src/test/scala/unit/kafka/coordinator/CoordinatorMetadataTest.scala 
08854c5e6ec249368206298b2ac2623df18f266a 
  core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala 
528525b719ec916e16f8b3ae3715bec4b5dcc47d 

Diff: https://reviews.apache.org/r/35231/diff/


Testing
---


Thanks,

Guozhang Wang

[jira] [Work started] (KAFKA-1740) Merge Offset manager into Coordinator

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-1740 started by Guozhang Wang.

> Merge Offset manager into Coordinator
> -
>
> Key: KAFKA-1740
> URL: https://issues.apache.org/jira/browse/KAFKA-1740
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.0
>
>
> This JIRA involves refactoring offset manager and merge it into coordinator, 
> including adding the logic for consumer-id / generation-id checking.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client

[Sriharsha Chintalapani (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14578031#comment-14578031
 ] 

Sriharsha Chintalapani commented on KAFKA-1690:
---

[~junrao] meant to say GatheringByteChannel. Keeping those interfaces same as 
they are trunk now.

> new java producer needs ssl support as a client
> ---
>
> Key: KAFKA-1690
> URL: https://issues.apache.org/jira/browse/KAFKA-1690
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Joe Stein
>Assignee: Sriharsha Chintalapani
> Fix For: 0.8.3
>
> Attachments: KAFKA-1690.patch, KAFKA-1690.patch, 
> KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, 
> KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, 
> KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, 
> KAFKA-1690_2015-05-21_10:37:08.patch, KAFKA-1690_2015-06-03_18:52:29.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1740) Merge Offset manager into Coordinator

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-1740:
-
Status: Patch Available  (was: In Progress)

> Merge Offset manager into Coordinator
> -
>
> Key: KAFKA-1740
> URL: https://issues.apache.org/jira/browse/KAFKA-1740
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.0
>
>
> This JIRA involves refactoring offset manager and merge it into coordinator, 
> including adding the logic for consumer-id / generation-id checking.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1740) Merge Offset manager into Coordinator

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14578032#comment-14578032
 ] 

Guozhang Wang commented on KAFKA-1740:
--

Created reviewboard https://reviews.apache.org/r/35231/diff/
against branch origin/trunk

> Merge Offset manager into Coordinator
> -
>
> Key: KAFKA-1740
> URL: https://issues.apache.org/jira/browse/KAFKA-1740
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.0
>
>
> This JIRA involves refactoring offset manager and merge it into coordinator, 
> including adding the logic for consumer-id / generation-id checking.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (KAFKA-2208) Consumer should detect coordinator failures

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2208:
-
Comment: was deleted

(was: Created reviewboard https://reviews.apache.org/r/35231/diff/
 against branch origin/trunk)

> Consumer should detect coordinator failures
> ---
>
> Key: KAFKA-2208
> URL: https://issues.apache.org/jira/browse/KAFKA-2208
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Onur Karaman
>Assignee: Guozhang Wang
> Attachments: KAFKA-2208.patch, KAFKA-2208.patch, 
> KAFKA-2208_2015-05-20_19:14:58.patch, KAFKA-2208_2015-05-31_17:05:06.patch, 
> KAFKA-2208_2015-06-02_10:45:34.patch, KAFKA-2208_2015-06-02_14:06:58.patch
>
>
> Add coordinator discovery and failure detection to the consumer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14578053#comment-14578053
 ] 

Jun Rao commented on KAFKA-1367:


Yes, perhaps some kind of more general locality could be useful. That can be 
done in a separate jira.

Here, we just want to figure out whether it's useful to maintain ISR in TMR.

[~jjkoshy], another issue without ISR is that initially a client will have no 
idea which replica is in sync and can only guess.

> Broker topic metadata not kept in sync with ZooKeeper
> -
>
> Key: KAFKA-1367
> URL: https://issues.apache.org/jira/browse/KAFKA-1367
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.0, 0.8.1
>Reporter: Ryan Berdeen
>Assignee: Ashish K Singh
>  Labels: newbie++
> Fix For: 0.8.3
>
> Attachments: KAFKA-1367.txt
>
>
> When a broker is restarted, the topic metadata responses from the brokers 
> will be incorrect (different from ZooKeeper) until a preferred replica leader 
> election.
> In the metadata, it looks like leaders are correctly removed from the ISR 
> when a broker disappears, but followers are not. Then, when a broker 
> reappears, the ISR is never updated.
> I used a variation of the Vagrant setup created by Joe Stein to reproduce 
> this with latest from the 0.8.1 branch: 
> https://github.com/also/kafka/commit/dba36a503a5e22ea039df0f9852560b4fb1e067c



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2254) The shell script should be optimized , even kafka-run-class.sh has a syntax error.

[Bo Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Wang updated KAFKA-2254:
---
Status: Open  (was: Patch Available)

> The shell script should be optimized , even kafka-run-class.sh has a syntax 
> error.
> --
>
> Key: KAFKA-2254
> URL: https://issues.apache.org/jira/browse/KAFKA-2254
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.8.2.1
> Environment: linux
>Reporter: Bo Wang
>Priority: Critical
>  Labels: client-script, kafka-run-class.sh, shell-script
> Fix For: 0.8.2.1
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
>  kafka-run-class.sh 128 line has a syntax error(missing a space):
> 127-loggc)
> 128 if [ -z "$KAFKA_GC_LOG_OPTS"] ; then
> 129GC_LOG_ENABLED="true"
> 130 fi
> And use the ShellCheck to check the shell scripts, the results shows some 
> errors 、 warnings and notes：
> https://github.com/koalaman/shellcheck/wiki/SC2068
> https://github.com/koalaman/shellcheck/wiki/Sc2046
> https://github.com/koalaman/shellcheck/wiki/Sc2086



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2254) The shell script should be optimized , even kafka-run-class.sh has a syntax error.

[Bo Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Wang updated KAFKA-2254:
---
Status: Patch Available  (was: Open)

> The shell script should be optimized , even kafka-run-class.sh has a syntax 
> error.
> --
>
> Key: KAFKA-2254
> URL: https://issues.apache.org/jira/browse/KAFKA-2254
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.8.2.1
> Environment: linux
>Reporter: Bo Wang
>Priority: Critical
>  Labels: client-script, kafka-run-class.sh, shell-script
> Fix For: 0.8.2.1
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
>  kafka-run-class.sh 128 line has a syntax error(missing a space):
> 127-loggc)
> 128 if [ -z "$KAFKA_GC_LOG_OPTS"] ; then
> 129GC_LOG_ENABLED="true"
> 130 fi
> And use the ShellCheck to check the shell scripts, the results shows some 
> errors 、 warnings and notes：
> https://github.com/koalaman/shellcheck/wiki/SC2068
> https://github.com/koalaman/shellcheck/wiki/Sc2046
> https://github.com/koalaman/shellcheck/wiki/Sc2086



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2254) The shell script should be optimized , even kafka-run-class.sh has a syntax error.

[Bo Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Wang updated KAFKA-2254:
---
Attachment: kafka-shell-script.patch

> The shell script should be optimized , even kafka-run-class.sh has a syntax 
> error.
> --
>
> Key: KAFKA-2254
> URL: https://issues.apache.org/jira/browse/KAFKA-2254
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.8.2.1
> Environment: linux
>Reporter: Bo Wang
>Priority: Critical
>  Labels: client-script, kafka-run-class.sh, shell-script
> Fix For: 0.8.2.1
>
> Attachments: kafka-shell-script.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
>  kafka-run-class.sh 128 line has a syntax error(missing a space):
> 127-loggc)
> 128 if [ -z "$KAFKA_GC_LOG_OPTS"] ; then
> 129GC_LOG_ENABLED="true"
> 130 fi
> And use the ShellCheck to check the shell scripts, the results shows some 
> errors 、 warnings and notes：
> https://github.com/koalaman/shellcheck/wiki/SC2068
> https://github.com/koalaman/shellcheck/wiki/Sc2046
> https://github.com/koalaman/shellcheck/wiki/Sc2086



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2254) The shell script should be optimized , even kafka-run-class.sh has a syntax error.

[Bo Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Wang updated KAFKA-2254:
---
Status: Patch Available  (was: Open)

I create a patch kafka-shell-script.patch of the trunk .

> The shell script should be optimized , even kafka-run-class.sh has a syntax 
> error.
> --
>
> Key: KAFKA-2254
> URL: https://issues.apache.org/jira/browse/KAFKA-2254
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.8.2.1
> Environment: linux
>Reporter: Bo Wang
>Priority: Critical
>  Labels: client-script, kafka-run-class.sh, shell-script
> Fix For: 0.8.2.1
>
> Attachments: kafka-shell-script.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
>  kafka-run-class.sh 128 line has a syntax error(missing a space):
> 127-loggc)
> 128 if [ -z "$KAFKA_GC_LOG_OPTS"] ; then
> 129GC_LOG_ENABLED="true"
> 130 fi
> And use the ShellCheck to check the shell scripts, the results shows some 
> errors 、 warnings and notes：
> https://github.com/koalaman/shellcheck/wiki/SC2068
> https://github.com/koalaman/shellcheck/wiki/Sc2046
> https://github.com/koalaman/shellcheck/wiki/Sc2086



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-25 System test improvements

[Geoffrey Anderson]

Hi Gwen,

I don't see any problem with this as long as we're convinced there's a good
use case, which seems to be true.

Cheers,
Geoff

On Thu, Jun 4, 2015 at 5:20 PM, Gwen Shapira  wrote:

> Not completely random places :)
> People may use Cloudera / HWX distributions which include Kafka, but want
> to verify that these bits match a specific upstream release.
>
> I think having the tests separately will be useful for this. In this case,
> finding the tests are not a big issue - we'll add a download link :)
>
> On Thu, Jun 4, 2015 at 5:00 PM, Jiangjie Qin 
> wrote:
>
> > Hey Gwen,
> >
> > Currently the test and code are downloaded at the same time. Supposedly
> > the tests in the same repository should cover match the code.
> > Are you saying people downloaded a release from some random place and
> want
> > to verify it? If that is the case, does that mean people still need to
> > find the correct place to download the right test artifact?
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> >
> > On 6/4/15, 4:29 PM, "Gwen Shapira"  wrote:
> >
> > >Hi,
> > >
> > >Reviving the discussion a bit :)
> > >
> > >I think it will be nice if each Kafka version that we release will also
> > >have a separate "tests" artifact that users can download, untar and
> easily
> > >run against a Kafka cluster of the same version.
> > >
> > >The idea is that if someone downloads packages that claim to contain
> > >something of a specific Kafka version (i.e. Kafka 0.8.2.0 + patches),
> > >users
> > >can easily download the tests and verify that it indeed passes the tests
> > >for this version and therefore behaves the way this version is expected
> to
> > >behave.
> > >
> > >Does it make sense?
> > >
> > >Gwen
> > >
> > >On Thu, May 21, 2015 at 3:26 PM, Geoffrey Anderson 
> > >wrote:
> > >
> > >> Hi Ashish,
> > >>
> > >> Looks like Ewen already hit the main points, but a few additions:
> > >>
> > >> 1. ducktape repo is here: https://github.com/confluentinc/ducktape
> > >> ducktape itself will be pip installable in the near future, and Kafka
> > >> system tests will be able to depend on a particular version of
> ducktape.
> > >>
> > >> 2.  The reporting is nothing fancy. We're definitely open to feedback,
> > >>but
> > >> it consists of:
> > >> - top level summary of the test run (simple PASS/FAIL for each test)
> > >> - top level info and debug logs
> > >> - per-test info and debug logs
> > >> - per-test "service" logs gathered from each service used in the test.
> > >>For
> > >> example, if your test pulls up a Kafka cluster with 5 brokers, the end
> > >> result will have the Kafka logs from each of those 5 machines.
> > >>
> > >> Cheers,
> > >> Geoff
> > >>
> > >> On Thu, May 21, 2015 at 3:15 PM, Ewen Cheslack-Postava
> > >>
> > >> wrote:
> > >>
> > >> > Ashish,
> > >> >
> > >> > 1. That was the plan. We put some effort into cleanly separating the
> > >> > framework so it would be reusable across many projects.
> > >> > 2. I think you're seeing a test in progress where the final report
> > >>hasn't
> > >> > been created yet. If you visit one of the older ones you'll see it
> > >>has a
> > >> > landing page with links:
> > >> > http://testing.confluent.io/confluent_platform/2015-05-20--001/
> > >> Apparently
> > >> > we need to adjust when we update the 'latest' symlink. The logs that
> > >>are
> > >> > collected for tests are configurable, and service implementations
> > >>include
> > >> > sane defaults (so, e.g., you will always get the normal log file for
> > >> Kafka,
> > >> > but only get the data files if the test asks for them).
> > >> > 3. No code coverage support. Haven't looked into it, so I couldn't
> > >> comment
> > >> > on how hard it would be to add.
> > >> >
> > >> > -Ewen
> > >> >
> > >> > On Thu, May 21, 2015 at 2:38 PM, Ashish Singh 
> > >> wrote:
> > >> >
> > >> > > Geoffrey,
> > >> > >
> > >> > > This looks great!
> > >> > >
> > >> > > A few questions.
> > >> > > 1. Will ducktape be maintained separately as a github repo?
> > >> > > 2. How easy is viewing the test results and logs. The link in KIP,
> > >> > > http://testing.confluent.io/confluent_platform/latest/, lists a
> > >>bunch
> > >> of
> > >> > > files and dirs. Could you add to KIP how the result and logs for
> the
> > >> > tests
> > >> > > will be organized.
> > >> > > 3. Does it support code coverage? If not, how easy/ difficult
> would
> > >>it
> > >> > be?
> > >> > >
> > >> > > On Thu, May 21, 2015 at 2:03 PM, Geoffrey Anderson
> > >> > >> >
> > >> > > wrote:
> > >> > >
> > >> > > > Great, I'll work on putting together a more detailed map of this
> > >> > > > replacement process.
> > >> > > >
> > >> > > > On Thu, May 21, 2015 at 11:13 AM, Gwen Shapira <
> > >> gshap...@cloudera.com>
> > >> > > > wrote:
> > >> > > >
> > >> > > > > Love this idea :)
> > >> > > > >
> > >> > > > > I took a look at Ducktape API and it looks like a good fit -
> > >>clean
> > >> > API,
> > >> > > > > extensible, easy to use and powerful enough for our use-case.
> 

[jira] [Updated] (KAFKA-2254) The shell script should be optimized , even kafka-run-class.sh has a syntax error.

[Bo Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Wang updated KAFKA-2254:
---
Priority: Major  (was: Critical)

> The shell script should be optimized , even kafka-run-class.sh has a syntax 
> error.
> --
>
> Key: KAFKA-2254
> URL: https://issues.apache.org/jira/browse/KAFKA-2254
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.8.2.1
> Environment: linux
>Reporter: Bo Wang
>  Labels: client-script, kafka-run-class.sh, shell-script
> Fix For: 0.8.2.1
>
> Attachments: kafka-shell-script.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
>  kafka-run-class.sh 128 line has a syntax error(missing a space):
> 127-loggc)
> 128 if [ -z "$KAFKA_GC_LOG_OPTS"] ; then
> 129GC_LOG_ENABLED="true"
> 130 fi
> And use the ShellCheck to check the shell scripts, the results shows some 
> errors 、 warnings and notes：
> https://github.com/koalaman/shellcheck/wiki/SC2068
> https://github.com/koalaman/shellcheck/wiki/Sc2046
> https://github.com/koalaman/shellcheck/wiki/Sc2086



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 33378: Patch for KAFKA-2136

[Aditya Auradkar]

> On June 5, 2015, 2:43 a.m., Joel Koshy wrote:
> > core/src/main/scala/kafka/api/FetchResponse.scala, line 143
> > 
> >
> > follow-up

Can you elaborate?


> On June 5, 2015, 2:43 a.m., Joel Koshy wrote:
> > core/src/main/scala/kafka/server/DelayedFetch.scala, line 58
> > 
> >
> > This is slightly unwieldy. Perhaps we can hold this patch especially 
> > since this will be impacted by the main patch (KAFKA-2084)

Sure, the plan is to commit this after the main patch. I can make this simpler 
if I added a class to represent the arguments for the callback but I don't 
think it will add a great deal of value


> On June 5, 2015, 2:43 a.m., Joel Koshy wrote:
> > core/src/main/scala/kafka/api/ProducerResponse.scala, line 40
> > 
> >
> > We should do this based on the response version as well right?

I gather the readFrom is only used on the client side. A client running this 
code will only send V1 style requests and will always get the throttleTime in 
return.
This isn't the case for consumers because they will send an old version of the 
request from the ReplicaFetcherThread (if the 
intra.cluster.replication.protocol is old).


- Aditya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33378/#review86428
---


On May 12, 2015, 9:42 p.m., Aditya Auradkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33378/
> ---
> 
> (Updated May 12, 2015, 9:42 p.m.)
> 
> 
> Review request for kafka, Joel Koshy and Jun Rao.
> 
> 
> Bugs: KAFKA-2136
> https://issues.apache.org/jira/browse/KAFKA-2136
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Changes are
> - protocol changes to the fetch request and response to return the 
> throttle_time_ms to clients
> - New producer/consumer metrics to expose the avg and max delay time for a 
> client
> - Test cases.
> 
> For now the patch will publish a zero delay and return a response
> 
> 
> Diffs
> -
> 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java
>  ef9dd5238fbc771496029866ece1d85db6d7b7a5 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> b2db91ca14bbd17fef5ce85839679144fff3f689 
>   clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 
> 3dc8b015afd2347a41c9a9dbc02b8e367da5f75f 
>   clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java 
> 8686d83aa52e435c6adafbe9ff4bd1602281072a 
>   clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java 
> eb8951fba48c335095cc43fc3672de1c733e07ff 
>   clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java 
> fabeae3083a8ea55cdacbb9568f3847ccd85bab4 
>   clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java 
> 37ec0b79beafcf5735c386b066eb319fb697eff5 
>   
> clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java
>  419541011d652becf0cda7a5e62ce813cddb1732 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
>   
> clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java
>  e3cc1967e407b64cc734548c19e30de700b64ba8 
>   core/src/main/scala/kafka/api/FetchRequest.scala 
> b038c15186c0cbcc65b59479324052498361b717 
>   core/src/main/scala/kafka/api/FetchResponse.scala 
> 75aaf57fb76ec01660d93701a57ae953d877d81c 
>   core/src/main/scala/kafka/api/ProducerRequest.scala 
> 570b2da1d865086f9830aa919a49063abbbe574d 
>   core/src/main/scala/kafka/api/ProducerResponse.scala 
> 5d1fac4cb8943f5bfaa487f8e9d9d2856efbd330 
>   core/src/main/scala/kafka/consumer/SimpleConsumer.scala 
> 31a2639477bf66f9a05d2b9b07794572d7ec393b 
>   core/src/main/scala/kafka/server/AbstractFetcherThread.scala 
> 83fc47417dd7fe4edf030217fa7fd69d99b170b0 
>   core/src/main/scala/kafka/server/DelayedFetch.scala 
> de6cf5bdaa0e70394162febc63b50b55ca0a92db 
>   core/src/main/scala/kafka/server/DelayedProduce.scala 
> 05078b24ef28f2f4e099afa943e43f1d00359fda 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> 417960dd1ab407ebebad8fdb0e97415db3e91a2f 
>   core/src/main/scala/kafka/server/OffsetManager.scala 
> 18680ce100f10035175cc0263ba7787ab0f6a17a 
>   core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
> b31b432a226ba79546dd22ef1d2acbb439c2e9a3 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
>   core/src/test/scala/unit/kafka/api/RequestRe

Re: Review Request 35231: Fix KAFKA-1740

[Onur Karaman]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/35231/#review87107
---


I'll try to do a more detailed review soon. I just want to get this one comment 
out of the way.


core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala


I don't think the ConsumerCoordinator constructor should change.

The coordinator just needs a KafkaConfig, ZkClient, and OffsetManager in 
order to work. It shouldn't care about ReplicaManagers, KakfaSchedulers, or 
OffsetManagerConfigs.

Passing in these lower-level dependencies makes testing the 
ConsumerCoordinator harder. As an example, think about how 
https://reviews.apache.org/r/35086/diff/ would work with this change. You'd 
have to mock out dependencies(ReplicaManager) that are a level deeper than what 
you actually care about(OffsetManager) with the hope that OffsetManager would 
react nicely to your mock instead of directly mocking the OffsetManager itself.


- Onur Karaman


On June 8, 2015, 11:12 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/35231/
> ---
> 
> (Updated June 8, 2015, 11:12 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1740
> https://issues.apache.org/jira/browse/KAFKA-1740
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Move offset manager to coordinator, add validation logic for offset commit 
> and fetch
> 
> 
> Diffs
> -
> 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/Coordinator.java
>  c1496a0851526f3c7d3905ce4bdff2129c83a6c1 
>   clients/src/main/java/org/apache/kafka/common/protocol/Errors.java 
> 5b898c8f8ad5d0469f469b600c4b2eb13d1fc662 
>   
> clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitResponse.java
>  70844d65369f6ff300cbeb513dbb6650050c7eec 
>   
> clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchRequest.java
>  deec1fa480d5a5c5884a1c007b076aa64e902472 
>   
> clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchResponse.java
>  512a0ef7e619d54e74122c38119209f5cf9590e3 
>   
> clients/src/test/java/org/apache/kafka/clients/consumer/internals/CoordinatorTest.java
>  b06c4a73e2b4e9472cd772c8bc32bf4a29f431bb 
>   core/src/main/scala/kafka/admin/TopicCommand.scala 
> dacbdd089f96c67b7bbc1c0cd36490d4fe094c1b 
>   core/src/main/scala/kafka/cluster/Partition.scala 
> 730a232482fdf77be5704cdf5941cfab3828db88 
>   core/src/main/scala/kafka/common/OffsetMetadataAndError.scala 
> 6b4242c7cd1df9b3465db0fec35a25102c76cd60 
>   core/src/main/scala/kafka/common/Topic.scala 
> ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 
>   core/src/main/scala/kafka/coordinator/ConsumerCoordinator.scala 
> 51e89c87ee2c20fc7f976536f01fa1055fb8e670 
>   core/src/main/scala/kafka/coordinator/CoordinatorMetadata.scala 
> c39e6de34ee531c6dfa9107b830752bd7f8fbe59 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> d63bc18d795a6f0e6994538ca55a9a46f7fb8ffd 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> b320ce9f6a12c0ee392e91beb82e8804d167f9f4 
>   core/src/main/scala/kafka/server/OffsetManager.scala 
> 5cca85cf727975f6d3acb2223fd186753ad761dc 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
>   core/src/test/scala/integration/kafka/api/ConsumerTest.scala 
> 17b17b9b1520c7cc2e2ba96cdb1f9ff06e47bcad 
>   core/src/test/scala/integration/kafka/api/IntegrationTestHarness.scala 
> 07b1ff47bfc3cd3f948c9533c8dc977fa36d996f 
>   core/src/test/scala/unit/kafka/admin/TopicCommandTest.scala 
> c7136f20972614ac47aa57ab13e3c94ef775a4b7 
>   core/src/test/scala/unit/kafka/consumer/TopicFilterTest.scala 
> 4f124af5c3e946045a78ad1519c37372a72c8985 
>   core/src/test/scala/unit/kafka/coordinator/CoordinatorMetadataTest.scala 
> 08854c5e6ec249368206298b2ac2623df18f266a 
>   core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala 
> 528525b719ec916e16f8b3ae3715bec4b5dcc47d 
> 
> Diff: https://reviews.apache.org/r/35231/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

[jira] [Commented] (KAFKA-2254) The shell script should be optimized , even kafka-run-class.sh has a syntax error.

[Bo Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14578153#comment-14578153
 ] 

Bo Wang commented on KAFKA-2254:


The trunk have fixed kafka-run-class.sh  syntax error.

> The shell script should be optimized , even kafka-run-class.sh has a syntax 
> error.
> --
>
> Key: KAFKA-2254
> URL: https://issues.apache.org/jira/browse/KAFKA-2254
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.8.2.1
> Environment: linux
>Reporter: Bo Wang
>  Labels: client-script, kafka-run-class.sh, shell-script
> Fix For: 0.8.2.1
>
> Attachments: kafka-shell-script.patch
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
>  kafka-run-class.sh 128 line has a syntax error(missing a space):
> 127-loggc)
> 128 if [ -z "$KAFKA_GC_LOG_OPTS"] ; then
> 129GC_LOG_ENABLED="true"
> 130 fi
> And use the ShellCheck to check the shell scripts, the results shows some 
> errors 、 warnings and notes：
> https://github.com/koalaman/shellcheck/wiki/SC2068
> https://github.com/koalaman/shellcheck/wiki/Sc2046
> https://github.com/koalaman/shellcheck/wiki/Sc2086



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 33378: Patch for KAFKA-2136

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33378/
---

(Updated June 9, 2015, 1:37 a.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Bugs: KAFKA-2136
https://issues.apache.org/jira/browse/KAFKA-2136


Repository: kafka


Description
---

Changes are
- protocol changes to the fetch request and response to return the 
throttle_time_ms to clients
- New producer/consumer metrics to expose the avg and max delay time for a 
client
- Test cases.

For now the patch will publish a zero delay and return a response


Diffs (updated)
-

  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java 
56281ee15cc33dfc96ff64d5b1e596047c7132a4 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
07e65d4a54ba4eef5b787eba3e71cbe9f6a920bd 
  clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 
3dc8b015afd2347a41c9a9dbc02b8e367da5f75f 
  clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java 
8686d83aa52e435c6adafbe9ff4bd1602281072a 
  clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java 
eb8951fba48c335095cc43fc3672de1c733e07ff 
  clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java 
fabeae3083a8ea55cdacbb9568f3847ccd85bab4 
  clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java 
37ec0b79beafcf5735c386b066eb319fb697eff5 
  
clients/src/test/java/org/apache/kafka/clients/consumer/internals/FetcherTest.java
 419541011d652becf0cda7a5e62ce813cddb1732 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
  
clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java 
e3cc1967e407b64cc734548c19e30de700b64ba8 
  core/src/main/scala/kafka/api/FetchRequest.scala 
5b38f8554898e54800abd65a7415dd0ac41fd958 
  core/src/main/scala/kafka/api/FetchResponse.scala 
0b6b33ab6f7a732ff1322b6f48abd4c43e0d7215 
  core/src/main/scala/kafka/api/ProducerRequest.scala 
c866180d3680da03e48d374415f10220f6ca68c4 
  core/src/main/scala/kafka/api/ProducerResponse.scala 
5d1fac4cb8943f5bfaa487f8e9d9d2856efbd330 
  core/src/main/scala/kafka/consumer/SimpleConsumer.scala 
c16f7edd322709060e54c77eb505c44cbd77a4ec 
  core/src/main/scala/kafka/server/AbstractFetcherThread.scala 
83fc47417dd7fe4edf030217fa7fd69d99b170b0 
  core/src/main/scala/kafka/server/DelayedFetch.scala 
de6cf5bdaa0e70394162febc63b50b55ca0a92db 
  core/src/main/scala/kafka/server/DelayedProduce.scala 
05078b24ef28f2f4e099afa943e43f1d00359fda 
  core/src/main/scala/kafka/server/KafkaApis.scala 
d63bc18d795a6f0e6994538ca55a9a46f7fb8ffd 
  core/src/main/scala/kafka/server/OffsetManager.scala 
5cca85cf727975f6d3acb2223fd186753ad761dc 
  core/src/main/scala/kafka/server/ReplicaFetcherThread.scala 
b31b432a226ba79546dd22ef1d2acbb439c2e9a3 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
59c9bc3ac3a8afc07a6f8c88c5871304db588d17 
  core/src/test/scala/unit/kafka/api/RequestResponseSerializationTest.scala 
5717165f2344823fabe8f7cfafae4bb8af2d949a 
  core/src/test/scala/unit/kafka/server/ReplicaManagerTest.scala 
00d59337a99ac135e8689bd1ecd928f7b1423d79 

Diff: https://reviews.apache.org/r/33378/diff/


Testing
---

New tests added


Thanks,

Aditya Auradkar

Re: Review Request 33204: Patch for KAFKA-1646 merge to latest trunk

[Honghai Chen]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33204/
---

(Updated June 9, 2015, 3:02 a.m.)


Review request for kafka.


Summary (updated)
-

Patch for KAFKA-1646 merge to latest trunk


Bugs: KAFKA-1646
https://issues.apache.org/jira/browse/KAFKA-1646


Repository: kafka


Description (updated)
---

Patch for KAFKA-1646 merge to latest trunk


Diffs (updated)
-

  core/src/main/scala/kafka/log/FileMessageSet.scala 
2522604bd985c513527fa0c863a7df677ff7a503 
  core/src/main/scala/kafka/log/Log.scala 
84e7b8fe9dd014884b60c4fbe13c835cf02a40e4 
  core/src/main/scala/kafka/log/LogConfig.scala 
a907da09e1ccede3b446459225e407cd1ae6d8b3 
  core/src/main/scala/kafka/log/LogSegment.scala 
ed039539ac18ea4d65144073915cf112f7374631 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
2d75186a110075e0c322db4b9f7a8c964a7a3e88 
  core/src/main/scala/kafka/server/KafkaServer.scala 
b320ce9f6a12c0ee392e91beb82e8804d167f9f4 
  core/src/main/scala/kafka/utils/CoreUtils.scala 
d0a8fa701564b4c13b3cd6501e1b6218d77e8e06 
  core/src/test/scala/unit/kafka/log/FileMessageSetTest.scala 
cec1caecc51507ae339ebf8f3b8a028b12a1a056 
  core/src/test/scala/unit/kafka/log/LogSegmentTest.scala 
03fb3512c4a4450eac83d4cd4b0919baeaa22942 

Diff: https://reviews.apache.org/r/33204/diff/


Testing
---


Thanks,

Honghai Chen

[jira] [Updated] (KAFKA-1646) Improve consumer read performance for Windows

[Honghai Chen (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Honghai Chen updated KAFKA-1646:

Attachment: KAFKA-1646_20150609_MergeToLatestTrunk.patch

> Improve consumer read performance for Windows
> -
>
> Key: KAFKA-1646
> URL: https://issues.apache.org/jira/browse/KAFKA-1646
> Project: Kafka
>  Issue Type: Improvement
>  Components: log
>Affects Versions: 0.8.1.1
> Environment: Windows
>Reporter: xueqiang wang
>Assignee: xueqiang wang
>  Labels: newbie, patch
> Attachments: Improve consumer read performance for Windows.patch, 
> KAFKA-1646-truncate-off-trailing-zeros-on-broker-restart-if-bro.patch, 
> KAFKA-1646_20141216_163008.patch, KAFKA-1646_20150306_005526.patch, 
> KAFKA-1646_20150511_AddTestcases.patch, 
> KAFKA-1646_20150609_MergeToLatestTrunk.patch
>
>
> This patch is for Window platform only. In Windows platform, if there are 
> more than one replicas writing to disk, the segment log files will not be 
> consistent in disk and then consumer reading performance will be dropped down 
> greatly. This fix allocates more disk spaces when rolling a new segment, and 
> then it will improve the consumer reading performance in NTFS file system.
> This patch doesn't affect file allocation of other filesystems, for it only 
> adds statements like 'if(Os.iswindow)' or adds methods used on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1646) Improve consumer read performance for Windows

[Honghai Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14578235#comment-14578235
 ] 

Honghai Chen commented on KAFKA-1646:
-

Created reviewboard  against branch origin/trunk

> Improve consumer read performance for Windows
> -
>
> Key: KAFKA-1646
> URL: https://issues.apache.org/jira/browse/KAFKA-1646
> Project: Kafka
>  Issue Type: Improvement
>  Components: log
>Affects Versions: 0.8.1.1
> Environment: Windows
>Reporter: xueqiang wang
>Assignee: xueqiang wang
>  Labels: newbie, patch
> Attachments: Improve consumer read performance for Windows.patch, 
> KAFKA-1646-truncate-off-trailing-zeros-on-broker-restart-if-bro.patch, 
> KAFKA-1646_20141216_163008.patch, KAFKA-1646_20150306_005526.patch, 
> KAFKA-1646_20150511_AddTestcases.patch, 
> KAFKA-1646_20150609_MergeToLatestTrunk.patch
>
>
> This patch is for Window platform only. In Windows platform, if there are 
> more than one replicas writing to disk, the segment log files will not be 
> consistent in disk and then consumer reading performance will be dropped down 
> greatly. This fix allocates more disk spaces when rolling a new segment, and 
> then it will improve the consumer reading performance in NTFS file system.
> This patch doesn't affect file allocation of other filesystems, for it only 
> adds statements like 'if(Os.iswindow)' or adds methods used on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-1646) Improve consumer read performance for Windows

[Honghai Chen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14578235#comment-14578235
 ] 

Honghai Chen edited comment on KAFKA-1646 at 6/9/15 3:07 AM:
-

Merge to latest trunk, patch attached.

code review no change   https://reviews.apache.org/r/33204/diff/4/



was (Author: waldenchen):
Created reviewboard  against branch origin/trunk

> Improve consumer read performance for Windows
> -
>
> Key: KAFKA-1646
> URL: https://issues.apache.org/jira/browse/KAFKA-1646
> Project: Kafka
>  Issue Type: Improvement
>  Components: log
>Affects Versions: 0.8.1.1
> Environment: Windows
>Reporter: xueqiang wang
>Assignee: xueqiang wang
>  Labels: newbie, patch
> Attachments: Improve consumer read performance for Windows.patch, 
> KAFKA-1646-truncate-off-trailing-zeros-on-broker-restart-if-bro.patch, 
> KAFKA-1646_20141216_163008.patch, KAFKA-1646_20150306_005526.patch, 
> KAFKA-1646_20150511_AddTestcases.patch, 
> KAFKA-1646_20150609_MergeToLatestTrunk.patch
>
>
> This patch is for Window platform only. In Windows platform, if there are 
> more than one replicas writing to disk, the segment log files will not be 
> consistent in disk and then consumer reading performance will be dropped down 
> greatly. This fix allocates more disk spaces when rolling a new segment, and 
> then it will improve the consumer reading performance in NTFS file system.
> This patch doesn't affect file allocation of other filesystems, for it only 
> adds statements like 'if(Os.iswindow)' or adds methods used on Windows.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2260) Allow specifying expected offset on produce

[Ben Kirwin (JIRA)]

Ben Kirwin created KAFKA-2260:
-

 Summary: Allow specifying expected offset on produce
 Key: KAFKA-2260
 URL: https://issues.apache.org/jira/browse/KAFKA-2260
 Project: Kafka
  Issue Type: Improvement
Reporter: Ben Kirwin
Priority: Minor


I'd like to propose a change that adds a simple CAS-like mechanism to the Kafka 
producer. This update has a small footprint, but enables a bunch of interesting 
uses in stream processing or as a commit log for process state.

h4. Proposed Change

In short:

- Allow the user to attach a specific offset to each message produced.

- The server assigns offsets to messages in the usual way. However, if the 
expected offset doesn't match the actual offset, the server should fail the 
produce request instead of completing the write.

This is a form of optimistic concurrency control, like the ubiquitous 
check-and-set -- but instead of checking the current value of some state, it 
checks the current offset of the log.

h4. Motivation

Much like check-and-set, this feature is only useful when there's very low 
contention. Happily, when Kafka is used as a commit log or as a 
stream-processing transport, it's common to have just one producer (or a small 
number) for a given partition -- and in many of these cases, predicting offsets 
turns out to be quite useful.

- We get the same benefits as the 'idempotent producer' proposal: a producer 
can retry a write indefinitely and be sure that at most one of those attempts 
will succeed; and if two producers accidentally write to the end of the 
partition at once, we can be certain that at least one of them will fail.

- It's possible to 'bulk load' Kafka this way -- you can write a list of n 
messages consecutively to a partition, even if the list is much larger than the 
buffer size or the producer has to be restarted.

- If a process is using Kafka as a commit log -- reading from a partition to 
bootstrap, then writing any updates to that same partition -- it can be sure 
that it's seen all of the messages in that partition at the moment it does its 
first (successful) write.

There's a bunch of other similar use-cases here, but they all have roughly the 
same flavour.

h4. Implementation

The major advantage of this proposal over other suggested transaction / 
idempotency mechanisms is its minimality: it gives the 'obvious' meaning to a 
currently-unused field, adds no new APIs, and requires very little new code or 
additional work from the server.

- Produced messages already carry an offset field, which is currently ignored 
by the server. This field could be used for the 'expected offset', with a sigil 
value for the current behaviour. (-1 is a natural choice, since it's already 
used to mean 'next available offset'.)

- We'd need a new error and error code for a 'CAS failure'.

- The server assigns offsets to produced messages in 
{{ByteBufferMessageSet.validateMessagesAndAssignOffsets}}. After this changed, 
this method would assign offsets in the same way -- but if they don't match the 
offset in the message, we'd return an error instead of completing the write.

- To avoid breaking existing clients, this behaviour would need to live behind 
some config flag. (Possibly global, but probably more useful per-topic?)

I understand all this is unsolicited and possibly strange: happy to answer 
questions, and if this seems interesting, I'd be glad to flesh this out into a 
full KIP or patch. (And apologies if this is the wrong venue for this sort of 
thing!)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-25 System test improvements

[Geoffrey Anderson]

Hi KIP-25 thread,

I consolidated some of the questions from this thread and elsewhere.

Q: Can we see a map of what system-test currently tests, which ones we want
to replace and JIRAs for replacing?
A: Initial draft here:
https://cwiki.apache.org/confluence/display/KAFKA/Roadmap+-+port+existing+system+tests

Q: Will ducktape be maintained separately as a github repo?
A: Yes https://github.com/confluentinc/ducktape

Q: How easy is viewing the test results and logs, how will test output be
structured?
A: Hierarchical structure as outlined here:
https://github.com/confluentinc/ducktape/wiki/Design-overview#output

Q: Does it support code coverage? If not, how easy/ difficult would it be
to support?
A: It does not, and we have no immediate plans to support this. Difficulty
unclear.

Q: It would be nice if each Kafka version that we release will also
have a separate "tests" artifact that users can download, untar and easily
run against a Kafka cluster of the same version.
A: This seems reasonable and not too much extra work. Definitely open to
discussion on this.

Q: Why not share running services across multiple tests?
A: Prefer to optimize for simplicity and correctness over what might be a
questionable improvement in run-time.

Q: Are regressions - in the road map?
A: yes

Q: Are Jepsen style tests involving network failures in the road map?
A: yes

Thanks much,
Geoff



On Mon, Jun 8, 2015 at 4:55 PM, Geoffrey Anderson 
wrote:

> Hi Gwen,
>
> I don't see any problem with this as long as we're convinced there's a
> good use case, which seems to be true.
>
> Cheers,
> Geoff
>
> On Thu, Jun 4, 2015 at 5:20 PM, Gwen Shapira 
> wrote:
>
>> Not completely random places :)
>> People may use Cloudera / HWX distributions which include Kafka, but want
>> to verify that these bits match a specific upstream release.
>>
>> I think having the tests separately will be useful for this. In this case,
>> finding the tests are not a big issue - we'll add a download link :)
>>
>> On Thu, Jun 4, 2015 at 5:00 PM, Jiangjie Qin 
>> wrote:
>>
>> > Hey Gwen,
>> >
>> > Currently the test and code are downloaded at the same time. Supposedly
>> > the tests in the same repository should cover match the code.
>> > Are you saying people downloaded a release from some random place and
>> want
>> > to verify it? If that is the case, does that mean people still need to
>> > find the correct place to download the right test artifact?
>> >
>> > Thanks,
>> >
>> > Jiangjie (Becket) Qin
>> >
>> >
>> >
>> > On 6/4/15, 4:29 PM, "Gwen Shapira"  wrote:
>> >
>> > >Hi,
>> > >
>> > >Reviving the discussion a bit :)
>> > >
>> > >I think it will be nice if each Kafka version that we release will also
>> > >have a separate "tests" artifact that users can download, untar and
>> easily
>> > >run against a Kafka cluster of the same version.
>> > >
>> > >The idea is that if someone downloads packages that claim to contain
>> > >something of a specific Kafka version (i.e. Kafka 0.8.2.0 + patches),
>> > >users
>> > >can easily download the tests and verify that it indeed passes the
>> tests
>> > >for this version and therefore behaves the way this version is
>> expected to
>> > >behave.
>> > >
>> > >Does it make sense?
>> > >
>> > >Gwen
>> > >
>> > >On Thu, May 21, 2015 at 3:26 PM, Geoffrey Anderson > >
>> > >wrote:
>> > >
>> > >> Hi Ashish,
>> > >>
>> > >> Looks like Ewen already hit the main points, but a few additions:
>> > >>
>> > >> 1. ducktape repo is here: https://github.com/confluentinc/ducktape
>> > >> ducktape itself will be pip installable in the near future, and Kafka
>> > >> system tests will be able to depend on a particular version of
>> ducktape.
>> > >>
>> > >> 2.  The reporting is nothing fancy. We're definitely open to
>> feedback,
>> > >>but
>> > >> it consists of:
>> > >> - top level summary of the test run (simple PASS/FAIL for each test)
>> > >> - top level info and debug logs
>> > >> - per-test info and debug logs
>> > >> - per-test "service" logs gathered from each service used in the
>> test.
>> > >>For
>> > >> example, if your test pulls up a Kafka cluster with 5 brokers, the
>> end
>> > >> result will have the Kafka logs from each of those 5 machines.
>> > >>
>> > >> Cheers,
>> > >> Geoff
>> > >>
>> > >> On Thu, May 21, 2015 at 3:15 PM, Ewen Cheslack-Postava
>> > >>
>> > >> wrote:
>> > >>
>> > >> > Ashish,
>> > >> >
>> > >> > 1. That was the plan. We put some effort into cleanly separating
>> the
>> > >> > framework so it would be reusable across many projects.
>> > >> > 2. I think you're seeing a test in progress where the final report
>> > >>hasn't
>> > >> > been created yet. If you visit one of the older ones you'll see it
>> > >>has a
>> > >> > landing page with links:
>> > >> > http://testing.confluent.io/confluent_platform/2015-05-20--001/
>> > >> Apparently
>> > >> > we need to adjust when we update the 'latest' symlink. The logs
>> that
>> > >>are
>> > >> > collected for tests are configurable, and service implementations
>>