[jira] [Commented] (KAFKA-2478) KafkaConsumer javadoc example seems wrong

2015-08-26 Thread Dmitry Stratiychuk (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715362#comment-14715362
 ] 

Dmitry Stratiychuk commented on KAFKA-2478:
---

Or flush the buffer and commit at the end of processing all records. 
Is there a way to control how many records are returned by the poll() method?


 KafkaConsumer javadoc example seems wrong
 -

 Key: KAFKA-2478
 URL: https://issues.apache.org/jira/browse/KAFKA-2478
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 0.8.3
Reporter: Dmitry Stratiychuk
Assignee: Neha Narkhede

 I was looking at this KafkaConsumer example in the javadoc:
 https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L199
 As I understand, commit() method commits the maximum offsets returned by the 
 most recent invocation of poll() method.
 In this example, there's a danger of losing the data.
 Imagine the case where 300 records are returned by consumer.poll()
 The commit will happen after inserting 200 records into the database.
 But it will also commit the offsets for 100 records that are still 
 unprocessed.
 So if consumer fails before buffer is dumped into the database again,
 then those 100 records will never be processed.
 If I'm wrong, could you please clarify the behaviour of commit() method?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (KAFKA-1977) Make logEndOffset available in the Zookeeper consumer

2015-08-26 Thread Will Funnell (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715430#comment-14715430
 ] 

Will Funnell edited comment on KAFKA-1977 at 8/26/15 8:12 PM:
--

I would still definitely like to see this in the new consumer, I think its a 
small thing to include, but very useful, especially when determining when you 
have reached the end of our log compacted topics.

Our implementation is as follows:

{code}
IteratorConsumerMessage iterator = new IteratorConsumerMessage() {

public boolean finished;
private Integer partition;
private ConsumerIteratorbyte[], byte[] it = stream.iterator();

@Override
public boolean hasNext() {

if (finished) {
return false;
} else {
try {
return it.hasNext();
} catch (Exception e) {

if (hasBeenForciblyShutdownByClient(e)) {

consumer.shutdown();

return false;
}
LOG.error(partition={} description=\Error while 
fetching from Kafka\, partition, e);

throw e;
}
}
}

@Override
public ConsumerMessage next() {

MessageAndMetadatabyte[], byte[] messageAndMetadata = 
it.next();
count++;

if (partition == null) {
partition = messageAndMetadata.partition();
}

if (messageAndMetadata.offset() == 
messageAndMetadata.logEndOffset() - 1) {

finished = true;
LOG.info(partition=\{}\, description=\Finished with 
partition\, messageAndMetadata.partition());
}

return toConsumedMessage(messageAndMetadata);
}

@Override
public void remove() {
it.remove();
}

private boolean hasBeenForciblyShutdownByClient(Exception e) {
return e instanceof InterruptedException;
}

};

{code}

Not quite sure how this translates to the new Consumer yet.


was (Author: willf):
I would still definitely like to see this in the new consumer, I think its a 
small thing to include, but very useful, especially when determining when you 
have reached the end of our log compacted topics.

Our implementation is as follows:

{code}
IteratorConsumerMessage iterator = new IteratorConsumerMessage() {

public boolean finished;
private Integer partition;
private ConsumerIteratorbyte[], byte[] it = stream.iterator();

private long count;

@Override
public boolean hasNext() {

if (finished) {
return false;
} else {
try {
return it.hasNext();
} catch (Exception e) {

if (hasBeenForciblyShutdownByClient(e)) {

consumer.shutdown();

return false;
}
LOG.error(partition={} description=\Error while 
fetching from Kafka\, partition, e);

throw e;
}
}
}

@Override
public ConsumerMessage next() {

MessageAndMetadatabyte[], byte[] messageAndMetadata = 
it.next();
count++;

if (partition == null) {
partition = messageAndMetadata.partition();
}

if (LOG.isDebugEnabled()) {
LOG.debug(count={} partition={} description=\Messages 
read from Kafka\, count, messageAndMetadata.partition());
}

if (messageAndMetadata.offset() == 
messageAndMetadata.logEndOffset() - 1) {

finished = true;
LOG.info(partition=\{}\, description=\Finished with 
partition\, messageAndMetadata.partition());
}

return toConsumedMessage(messageAndMetadata);
}

@Override
public void remove() {
it.remove();
}

private boolean hasBeenForciblyShutdownByClient(Exception e) {
return e instanceof InterruptedException;
}

};

{code}

Not quite sure how this translates to the new Consumer yet.

 Make logEndOffset available in the Zookeeper consumer
 -

 Key: KAFKA-1977
 URL: https://issues.apache.org/jira/browse/KAFKA-1977
 Project: 

[jira] [Commented] (KAFKA-1977) Make logEndOffset available in the Zookeeper consumer

2015-08-26 Thread Gwen Shapira (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715409#comment-14715409
 ] 

Gwen Shapira commented on KAFKA-1977:
-

I lost track of the new consumer APIs, so I'm ok with whatever [~hachikuji] 
thinks :)

 Make logEndOffset available in the Zookeeper consumer
 -

 Key: KAFKA-1977
 URL: https://issues.apache.org/jira/browse/KAFKA-1977
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Will Funnell
Priority: Minor
 Attachments: 
 Make_logEndOffset_available_in_the_Zookeeper_consumer.patch


 The requirement is to create a snapshot from the Kafka topic but NOT do 
 continual reads after that point. For example you might be creating a backup 
 of the data to a file.
 In order to achieve that, a recommended solution by Joel Koshy and Jay Kreps 
 was to expose the high watermark, as maxEndOffset, from the FetchResponse 
 object through to each MessageAndMetadata object in order to be aware when 
 the consumer has reached the end of each partition.
 The submitted patch achieves this by adding the maxEndOffset to the 
 PartitionTopicInfo, which is updated when a new message arrives in the 
 ConsumerFetcherThread and then exposed in MessageAndMetadata.
 See here for discussion:
 http://search-hadoop.com/m/4TaT4TpJy71



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-1977) Make logEndOffset available in the Zookeeper consumer

2015-08-26 Thread Will Funnell (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715430#comment-14715430
 ] 

Will Funnell commented on KAFKA-1977:
-

I would still definitely like to see this in the new consumer, I think its a 
small thing to include, but very useful, especially when determining when you 
have reached the end of our log compacted topics.

Our implementation is as follows:

{code}
IteratorConsumerMessage iterator = new IteratorConsumerMessage() {

public boolean finished;
private Integer partition;
private ConsumerIteratorbyte[], byte[] it = stream.iterator();

private long count;

@Override
public boolean hasNext() {

if (finished) {
return false;
} else {
try {
return it.hasNext();
} catch (Exception e) {

if (hasBeenForciblyShutdownByClient(e)) {

consumer.shutdown();

return false;
}
LOG.error(partition={} description=\Error while 
fetching from Kafka\, partition, e);

throw e;
}
}
}

@Override
public ConsumerMessage next() {

MessageAndMetadatabyte[], byte[] messageAndMetadata = 
it.next();
count++;

if (partition == null) {
partition = messageAndMetadata.partition();
}

if (LOG.isDebugEnabled()) {
LOG.debug(count={} partition={} description=\Messages 
read from Kafka\, count, messageAndMetadata.partition());
}

if (messageAndMetadata.offset() == 
messageAndMetadata.logEndOffset() - 1) {

finished = true;
LOG.info(partition=\{}\, description=\Finished with 
partition\, messageAndMetadata.partition());
}

return toConsumedMessage(messageAndMetadata);
}

@Override
public void remove() {
it.remove();
}

private boolean hasBeenForciblyShutdownByClient(Exception e) {
return e instanceof InterruptedException;
}

};

{code}

Not quite sure how this translates to the new Consumer yet.

 Make logEndOffset available in the Zookeeper consumer
 -

 Key: KAFKA-1977
 URL: https://issues.apache.org/jira/browse/KAFKA-1977
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Will Funnell
Priority: Minor
 Attachments: 
 Make_logEndOffset_available_in_the_Zookeeper_consumer.patch


 The requirement is to create a snapshot from the Kafka topic but NOT do 
 continual reads after that point. For example you might be creating a backup 
 of the data to a file.
 In order to achieve that, a recommended solution by Joel Koshy and Jay Kreps 
 was to expose the high watermark, as maxEndOffset, from the FetchResponse 
 object through to each MessageAndMetadata object in order to be aware when 
 the consumer has reached the end of each partition.
 The submitted patch achieves this by adding the maxEndOffset to the 
 PartitionTopicInfo, which is updated when a new message arrives in the 
 ConsumerFetcherThread and then exposed in MessageAndMetadata.
 See here for discussion:
 http://search-hadoop.com/m/4TaT4TpJy71



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: Review Request 36858: Patch for KAFKA-2120

2015-08-26 Thread Joel Koshy

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review96095
---


Can you rebase?


clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java (line 130)
https://reviews.apache.org/r/36858/#comment151310

Returns a list of nodes with pending inflight requests that need to be 
timed out



clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java (line 138)
https://reviews.apache.org/r/36858/#comment151311

Can do without this guard.



clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java (line 141)
https://reviews.apache.org/r/36858/#comment151323

Is this right? i.e., `lastSent`?



clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
(line 228)
https://reviews.apache.org/r/36858/#comment152086

This logic is a bit confusing. Is this block necessary here? i.e., vs being 
written once below?



clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
(line 245)
https://reviews.apache.org/r/36858/#comment152089

I'm not sure we should remove the _replication timeout_ though. i.e., sure 
the replication timeout should not be used for request timeout going forward, 
but we still need a replication timeout in the producer request.



clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
(line 428)
https://reviews.apache.org/r/36858/#comment152144

Minor point: given that there may be a custom serializer and custom 
partitioner, the elapsed check should probably be made after each step.



clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
(line 100)
https://reviews.apache.org/r/36858/#comment152146

See comment above on the (continued) need for replication timeout. Also, 
can you fix whitespace on all your comments? i.e., `* @deprecated` instead of 
`*@deprecated`.



clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
(line 196)
https://reviews.apache.org/r/36858/#comment152148

Let us make this doc string as exhaustive and clear as possible - i.e., 
drop the etc. and enumerate everything. You can also add something along the 
lines of - also see the request timeout config, with a doc link.



clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 (line 94)
https://reviews.apache.org/r/36858/#comment152158

Can you add the new param to the javadoc?



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 (line 155)
https://reviews.apache.org/r/36858/#comment152162

same here



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 (line 205)
https://reviews.apache.org/r/36858/#comment152168

Maybe drop the due to This would be just one possibility right? E.g., 
you could have a low request timeout and high linger time... possibly other 
scenarios as well.



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 (line 214)
https://reviews.apache.org/r/36858/#comment152170

typo in comment



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 (line 221)
https://reviews.apache.org/r/36858/#comment152172

If you do a reverse iteration and a batch has _not_ expired, then we can 
break early right?


- Joel Koshy


On Aug. 12, 2015, 5:59 p.m., Mayuresh Gharat wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/36858/
 ---
 
 (Updated Aug. 12, 2015, 5:59 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-2120
 https://issues.apache.org/jira/browse/KAFKA-2120
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Solved compile error
 
 
 Addressed Jason's comments for Kip-19
 
 
 Addressed Jun's comments
 
 
 Addressed Jason's comments about the default values for requestTimeout
 
 
 Diffs
 -
 
   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
 dc8f0f115bcda893c95d17c0a57be8d14518d034 
   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
 2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
 7ab2503794ff3aab39df881bd9fbae6547827d3b 
   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
 0e51d7bd461d253f4396a5b6ca7cd391658807fa 
   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
 d35b421a515074d964c7fccb73d260b847ea5f00 
   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
 ed99e9bdf7c4ea7a6d4555d4488cf8ed0b80641b 
   
 

[jira] [Commented] (KAFKA-2477) Replicas spuriously deleting all segments in partition

2015-08-26 Thread Jiangjie Qin (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715280#comment-14715280
 ] 

Jiangjie Qin commented on KAFKA-2477:
-

Maybe related to KAFKA-2143.

 Replicas spuriously deleting all segments in partition
 --

 Key: KAFKA-2477
 URL: https://issues.apache.org/jira/browse/KAFKA-2477
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.1
Reporter: Håkon Hitland
 Attachments: kafka_log.txt


 We're seeing some strange behaviour in brokers: a replica will sometimes 
 schedule all segments in a partition for deletion, and then immediately start 
 replicating them back, triggering our check for under-replicating topics.
 This happens on average a couple of times a week, for different brokers and 
 topics.
 We have per-topic retention.ms and retention.bytes configuration, the topics 
 where we've seen this happen are hitting the size limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-1977) Make logEndOffset available in the Zookeeper consumer

2015-08-26 Thread James Cheng (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715394#comment-14715394
 ] 

James Cheng commented on KAFKA-1977:


Ping [~gwenshap]. Do you think logEndOffset can/should/will be exposed in the 
new Java consumer?

 Make logEndOffset available in the Zookeeper consumer
 -

 Key: KAFKA-1977
 URL: https://issues.apache.org/jira/browse/KAFKA-1977
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Will Funnell
Priority: Minor
 Attachments: 
 Make_logEndOffset_available_in_the_Zookeeper_consumer.patch


 The requirement is to create a snapshot from the Kafka topic but NOT do 
 continual reads after that point. For example you might be creating a backup 
 of the data to a file.
 In order to achieve that, a recommended solution by Joel Koshy and Jay Kreps 
 was to expose the high watermark, as maxEndOffset, from the FetchResponse 
 object through to each MessageAndMetadata object in order to be aware when 
 the consumer has reached the end of each partition.
 The submitted patch achieves this by adding the maxEndOffset to the 
 PartitionTopicInfo, which is updated when a new message arrives in the 
 ConsumerFetcherThread and then exposed in MessageAndMetadata.
 See here for discussion:
 http://search-hadoop.com/m/4TaT4TpJy71



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: Can I get permission to create proposal

2015-08-26 Thread Guozhang Wang
Done.

Guozhang

On Tue, Aug 25, 2015 at 11:33 AM, Abhishek Nigam 
ani...@linkedin.com.invalid wrote:

 For pinning the controller to a broker KAFKA-1778 I want to create a
 proposal detailing the design.

 My user id is:anigam

 -Abhishek




-- 
-- Guozhang


[jira] [Commented] (KAFKA-2478) KafkaConsumer javadoc example seems wrong

2015-08-26 Thread Jason Gustafson (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14714505#comment-14714505
 ] 

Jason Gustafson commented on KAFKA-2478:


[~devstr] I agree that the example seems incorrect. Probably we should only be 
committing the max offset stored in the buffer.


 KafkaConsumer javadoc example seems wrong
 -

 Key: KAFKA-2478
 URL: https://issues.apache.org/jira/browse/KAFKA-2478
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 0.8.3
Reporter: Dmitry Stratiychuk
Assignee: Neha Narkhede

 I was looking at this KafkaConsumer example in the javadoc:
 https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L199
 As I understand, commit() method commits the maximum offsets returned by the 
 most recent invocation of poll() method.
 In this example, there's a danger of losing the data.
 Imagine the case where 300 records are returned by consumer.poll()
 The commit will happen after inserting 200 records into the database.
 But it will also commit the offsets for 100 records that are still 
 unprocessed.
 So if consumer fails before buffer is dumped into the database again,
 then those 100 records will never be processed.
 If I'm wrong, could you please clarify the behaviour of commit() method?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] kafka pull request: KAFKA-2072: Add StopReplica request/response t...

2015-08-26 Thread dajac
GitHub user dajac opened a pull request:

https://github.com/apache/kafka/pull/170

KAFKA-2072: Add StopReplica request/response to o.a.k.common.requests

This PR adds StopReplica request and response as it is required by @ijuma 
for KAFKA-2411. Migration of core module is addressed a separate PR (#141).

@ijuma Could you review it? @gwenshap Could you take a look as well?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dajac/kafka KAFKA-2072-part-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/170.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #170


commit f8f4d691d43cc260377e630fcaf67d31923b37e3
Author: David Jacot david.ja...@gmail.com
Date:   2015-08-13T15:36:49Z

Add o.a.k.c.r.StopReplicaRequest and o.a.k.c.r.StopReplicaResponse.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (KAFKA-2072) Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module

2015-08-26 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14714470#comment-14714470
 ] 

ASF GitHub Bot commented on KAFKA-2072:
---

GitHub user dajac opened a pull request:

https://github.com/apache/kafka/pull/170

KAFKA-2072: Add StopReplica request/response to o.a.k.common.requests

This PR adds StopReplica request and response as it is required by @ijuma 
for KAFKA-2411. Migration of core module is addressed a separate PR (#141).

@ijuma Could you review it? @gwenshap Could you take a look as well?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dajac/kafka KAFKA-2072-part-1

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/170.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #170


commit f8f4d691d43cc260377e630fcaf67d31923b37e3
Author: David Jacot david.ja...@gmail.com
Date:   2015-08-13T15:36:49Z

Add o.a.k.c.r.StopReplicaRequest and o.a.k.c.r.StopReplicaResponse.




 Add StopReplica request/response to o.a.k.common.requests and replace the 
 usage in core module
 --

 Key: KAFKA-2072
 URL: https://issues.apache.org/jira/browse/KAFKA-2072
 Project: Kafka
  Issue Type: Sub-task
Reporter: Gwen Shapira
Assignee: David Jacot
 Fix For: 0.8.3


 Add StopReplica request/response to o.a.k.common.requests and replace the 
 usage in core module



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (KAFKA-2477) Replicas spuriously deleting all segments in partition

2015-08-26 Thread Jiangjie Qin (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715604#comment-14715604
 ] 

Jiangjie Qin edited comment on KAFKA-2477 at 8/26/15 10:13 PM:
---

What is the partition replication factor?
Also, can you search for start offset in the server log of the broker who 
truncates its log?


was (Author: becket_qin):
What is the partition replication factor?

 Replicas spuriously deleting all segments in partition
 --

 Key: KAFKA-2477
 URL: https://issues.apache.org/jira/browse/KAFKA-2477
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.1
Reporter: Håkon Hitland
 Attachments: kafka_log.txt


 We're seeing some strange behaviour in brokers: a replica will sometimes 
 schedule all segments in a partition for deletion, and then immediately start 
 replicating them back, triggering our check for under-replicating topics.
 This happens on average a couple of times a week, for different brokers and 
 topics.
 We have per-topic retention.ms and retention.bytes configuration, the topics 
 where we've seen this happen are hitting the size limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-2390) OffsetOutOfRangeException should contain the Offset and Partition info.

2015-08-26 Thread Jiangjie Qin (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-2390:

Summary: OffsetOutOfRangeException should contain the Offset and Partition 
info.  (was: Seek() should take a callback.)

 OffsetOutOfRangeException should contain the Offset and Partition info.
 ---

 Key: KAFKA-2390
 URL: https://issues.apache.org/jira/browse/KAFKA-2390
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jiangjie Qin
Assignee: Dong Lin

 Currently seek is an async call. To have the same interface as other calls 
 like commit(), seek() should take a callback. This callback will be invoked 
 if the position to seek triggers OFFSET_OUT_OF_RANGE exception from broker.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-2390) Seek() should take a callback.

2015-08-26 Thread Jiangjie Qin (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715682#comment-14715682
 ] 

Jiangjie Qin commented on KAFKA-2390:
-

Makes sense, so if user needs the offsets, they can get it by themselves. If 
they don't need it, no extra requests will be wasted. I'll change the ticket 
title.

 Seek() should take a callback.
 --

 Key: KAFKA-2390
 URL: https://issues.apache.org/jira/browse/KAFKA-2390
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jiangjie Qin
Assignee: Dong Lin

 Currently seek is an async call. To have the same interface as other calls 
 like commit(), seek() should take a callback. This callback will be invoked 
 if the position to seek triggers OFFSET_OUT_OF_RANGE exception from broker.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-2120) Add a request timeout to NetworkClient

2015-08-26 Thread Joel Koshy (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy updated KAFKA-2120:
--
Status: In Progress  (was: Patch Available)

 Add a request timeout to NetworkClient
 --

 Key: KAFKA-2120
 URL: https://issues.apache.org/jira/browse/KAFKA-2120
 Project: Kafka
  Issue Type: New Feature
Reporter: Jiangjie Qin
Assignee: Mayuresh Gharat
 Fix For: 0.8.3

 Attachments: KAFKA-2120.patch, KAFKA-2120_2015-07-27_15:31:19.patch, 
 KAFKA-2120_2015-07-29_15:57:02.patch, KAFKA-2120_2015-08-10_19:55:18.patch, 
 KAFKA-2120_2015-08-12_10:59:09.patch


 Currently NetworkClient does not have a timeout setting for requests. So if 
 no response is received for a request due to reasons such as broker is down, 
 the request will never be completed.
 Request timeout will also be used as implicit timeout for some methods such 
 as KafkaProducer.flush() and kafkaProducer.close().
 KIP-19 is created for this public interface change.
 https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: Review Request 36858: Patch for KAFKA-2120

2015-08-26 Thread Joel Koshy

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review96586
---



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 (line 240)
https://reviews.apache.org/r/36858/#comment152173

Also, per the KIP agreement, the request timeout will reset for retries - 
are you handling that?


- Joel Koshy


On Aug. 12, 2015, 5:59 p.m., Mayuresh Gharat wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/36858/
 ---
 
 (Updated Aug. 12, 2015, 5:59 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-2120
 https://issues.apache.org/jira/browse/KAFKA-2120
 
 
 Repository: kafka
 
 
 Description
 ---
 
 Solved compile error
 
 
 Addressed Jason's comments for Kip-19
 
 
 Addressed Jun's comments
 
 
 Addressed Jason's comments about the default values for requestTimeout
 
 
 Diffs
 -
 
   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
 dc8f0f115bcda893c95d17c0a57be8d14518d034 
   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
 2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b 
   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
 7ab2503794ff3aab39df881bd9fbae6547827d3b 
   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
 0e51d7bd461d253f4396a5b6ca7cd391658807fa 
   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
 d35b421a515074d964c7fccb73d260b847ea5f00 
   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
 ed99e9bdf7c4ea7a6d4555d4488cf8ed0b80641b 
   
 clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
 03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 
   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
 aa264202f2724907924985a5ecbe74afc4c6c04b 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
 0baf16e55046a2f49f6431e01d52c323c95eddf0 
   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
 ce20111ac434eb8c74585e9c63757bb9d60a832f 
   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
 9133d85342b11ba2c9888d4d2804d181831e7a8e 
   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
 43238ceaad0322e39802b615bb805b895336a009 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
  5b2e4ffaeab7127648db608c179703b27b577414 
   
 clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
  8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 
   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
 158f9829ff64a969008f699e40c51e918287859e 
   core/src/main/scala/kafka/tools/ProducerPerformance.scala 
 0335cc64013ffe2cdf1c4879e86e11ec8c526712 
   core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
 ee94011894b46864614b97bbd2a98375a7d3f20b 
   core/src/test/scala/unit/kafka/utils/TestUtils.scala 
 eb169d8b33c27d598cc24e5a2e5f78b789fa38d3 
 
 Diff: https://reviews.apache.org/r/36858/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Mayuresh Gharat
 




[GitHub] kafka pull request: KAFKA-2072: Add StopReplica request/response t...

2015-08-26 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/170


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server

2015-08-26 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715489#comment-14715489
 ] 

ASF GitHub Bot commented on KAFKA-1683:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/155


 Implement a session concept in the socket server
 --

 Key: KAFKA-1683
 URL: https://issues.apache.org/jira/browse/KAFKA-1683
 Project: Kafka
  Issue Type: Sub-task
  Components: security
Affects Versions: 0.8.2.1
Reporter: Jay Kreps
Assignee: Gwen Shapira
 Fix For: 0.8.3

 Attachments: KAFKA-1683.patch, KAFKA-1683.patch


 To implement authentication we need a way to keep track of some things 
 between requests. The initial use for this would be remembering the 
 authenticated user/principle info, but likely more uses would come up (for 
 example we will also need to remember whether and which encryption or 
 integrity measures are in place on the socket so we can wrap and unwrap 
 writes and reads).
 I was thinking we could just add a Session object that might have a user 
 field. The session object would need to get added to RequestChannel.Request 
 so it is passed down to the API layer with each request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Resolved] (KAFKA-1683) Implement a session concept in the socket server

2015-08-26 Thread Gwen Shapira (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira resolved KAFKA-1683.
-
Resolution: Fixed

Issue resolved by pull request 155
[https://github.com/apache/kafka/pull/155]

 Implement a session concept in the socket server
 --

 Key: KAFKA-1683
 URL: https://issues.apache.org/jira/browse/KAFKA-1683
 Project: Kafka
  Issue Type: Sub-task
  Components: security
Affects Versions: 0.8.2.1
Reporter: Jay Kreps
Assignee: Gwen Shapira
 Fix For: 0.8.3

 Attachments: KAFKA-1683.patch, KAFKA-1683.patch


 To implement authentication we need a way to keep track of some things 
 between requests. The initial use for this would be remembering the 
 authenticated user/principle info, but likely more uses would come up (for 
 example we will also need to remember whether and which encryption or 
 integrity measures are in place on the socket so we can wrap and unwrap 
 writes and reads).
 I was thinking we could just add a Session object that might have a user 
 field. The session object would need to get added to RequestChannel.Request 
 so it is passed down to the API layer with each request.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-2210) KafkaAuthorizer: Add all public entities, config changes and changes to KafkaAPI and kafkaServer to allow pluggable authorizer implementation.

2015-08-26 Thread Parth Brahmbhatt (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Parth Brahmbhatt updated KAFKA-2210:

Attachment: KAFKA-2210_2015-08-26_14:29:02.patch

 KafkaAuthorizer: Add all public entities, config changes and changes to 
 KafkaAPI and kafkaServer to allow pluggable authorizer implementation.
 --

 Key: KAFKA-2210
 URL: https://issues.apache.org/jira/browse/KAFKA-2210
 Project: Kafka
  Issue Type: Sub-task
  Components: security
Reporter: Parth Brahmbhatt
Assignee: Parth Brahmbhatt
 Fix For: 0.8.3

 Attachments: KAFKA-2210.patch, KAFKA-2210_2015-06-03_16:36:11.patch, 
 KAFKA-2210_2015-06-04_16:07:39.patch, KAFKA-2210_2015-07-09_18:00:34.patch, 
 KAFKA-2210_2015-07-14_10:02:19.patch, KAFKA-2210_2015-07-14_14:13:19.patch, 
 KAFKA-2210_2015-07-20_16:42:18.patch, KAFKA-2210_2015-07-21_17:08:21.patch, 
 KAFKA-2210_2015-08-10_18:31:54.patch, KAFKA-2210_2015-08-20_11:27:18.patch, 
 KAFKA-2210_2015-08-25_17:59:22.patch, KAFKA-2210_2015-08-26_14:29:02.patch


 This is the first subtask for Kafka-1688. As Part of this jira we intend to 
 agree on all the public entities, configs and changes to existing kafka 
 classes to allow pluggable authorizer implementation.
 Please see KIP-11 
 https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface
  for detailed design. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: Review Request 34492: Patch for KAFKA-2210

2015-08-26 Thread Parth Brahmbhatt

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34492/
---

(Updated Aug. 26, 2015, 9:29 p.m.)


Review request for kafka.


Bugs: KAFKA-2210
https://issues.apache.org/jira/browse/KAFKA-2210


Repository: kafka


Description (updated)
---

Addressing review comments from Jun.


Adding CREATE check for offset topic only if the topic does not exist already.


Addressing some more comments.


Removing acl.json file


Moving PermissionType to trait instead of enum. Following the convention for 
defining constants.


Adding authorizer.config.path back.


Addressing more comments from Jun.


Addressing more comments.


Now addressing Ismael's comments. Case sensitive checks.


Addressing Jun's comments.


Merge remote-tracking branch 'origin/trunk' into az

Conflicts:
core/src/main/scala/kafka/server/KafkaApis.scala
core/src/main/scala/kafka/server/KafkaServer.scala

Deleting KafkaConfigDefTest


Addressing comments from Ismael.


Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into az


Consolidating KafkaPrincipal.


Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into az

Conflicts:

clients/src/main/java/org/apache/kafka/common/network/PlaintextTransportLayer.java

clients/src/main/java/org/apache/kafka/common/security/auth/KafkaPrincipal.java
core/src/main/scala/kafka/server/KafkaApis.scala


Diffs (updated)
-

  
clients/src/main/java/org/apache/kafka/common/network/PlaintextTransportLayer.java
 35d41685dd178bbdf77b2476e03ad51fc4adcbb6 
  
clients/src/main/java/org/apache/kafka/common/security/auth/KafkaPrincipal.java 
b640ea0f4bdb694fc5524ef594aa125cc1ba4cf3 
  
clients/src/test/java/org/apache/kafka/common/security/auth/KafkaPrincipalTest.java
 PRE-CREATION 
  core/src/main/scala/kafka/api/OffsetRequest.scala 
f418868046f7c99aefdccd9956541a0cb72b1500 
  core/src/main/scala/kafka/common/AuthorizationException.scala PRE-CREATION 
  core/src/main/scala/kafka/common/ErrorMapping.scala 
c75c68589681b2c9d6eba2b440ce5e58cddf6370 
  core/src/main/scala/kafka/security/auth/Acl.scala PRE-CREATION 
  core/src/main/scala/kafka/security/auth/Authorizer.scala PRE-CREATION 
  core/src/main/scala/kafka/security/auth/Operation.scala PRE-CREATION 
  core/src/main/scala/kafka/security/auth/PermissionType.scala PRE-CREATION 
  core/src/main/scala/kafka/security/auth/Resource.scala PRE-CREATION 
  core/src/main/scala/kafka/security/auth/ResourceType.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
a3a8df0545c3f9390e0e04b8d2fab0134f5fd019 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
d547a01cf7098f216a3775e1e1901c5794e1b24c 
  core/src/main/scala/kafka/server/KafkaServer.scala 
17db4fa3c3a146f03a35dd7671ad1b06d122bb59 
  core/src/test/scala/unit/kafka/security/auth/AclTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/security/auth/OperationTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/security/auth/PermissionTypeTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/security/auth/ResourceTypeTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
3da666f73227fc7ef7093e3790546344065f6825 

Diff: https://reviews.apache.org/r/34492/diff/


Testing
---


Thanks,

Parth Brahmbhatt



[jira] [Comment Edited] (KAFKA-2389) CommitType seems not necessary in commit().

2015-08-26 Thread Guozhang Wang (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715565#comment-14715565
 ] 

Guozhang Wang edited comment on KAFKA-2389 at 8/26/15 9:39 PM:
---

I'm late on this discussion, but here are my two cents: [~jkreps]

1. We need to rename the async commits to commitAsync or something similar, 
you can't have two methods with the same name that behave totally differently 
and have different post-conditions.

I feel it is OK to have sync / async differentiated by the callback rather than 
by their names. For example ZooKeeper has a similar approach regarding sync / 
async APIs:

http://zookeeper.apache.org/doc/r3.4.5/api/org/apache/zookeeper/ZooKeeper.html#exists(java.lang.String,
 org.apache.zookeeper.Watcher, org.apache.zookeeper.AsyncCallback.StatCallback, 
java.lang.Object)

2. We need to include variants of asyncCommit that don't take the callback. 
Having the user implement or discover a NoOpCallback to be able to use the api 
is not good.

With commit(OffsetCommitCallback callback), users can just call commit(null) 
and do not need to implement a NoOpCallback, right?

I am personally not favor of making commitSync / commitAsync function names.


was (Author: guozhang):
I'm late on this discussion, but here are my two cents:

1. We need to rename the async commits to commitAsync or something similar, 
you can't have two methods with the same name that behave totally differently 
and have different post-conditions.

I feel it is OK to have sync / async differentiated by the callback rather than 
by their names. For example ZooKeeper has a similar approach regarding sync / 
async APIs:

http://zookeeper.apache.org/doc/r3.4.5/api/org/apache/zookeeper/ZooKeeper.html#exists(java.lang.String,
 org.apache.zookeeper.Watcher, org.apache.zookeeper.AsyncCallback.StatCallback, 
java.lang.Object)

2. We need to include variants of asyncCommit that don't take the callback. 
Having the user implement or discover a NoOpCallback to be able to use the api 
is not good.

With commit(OffsetCommitCallback callback), users can just call commit(null) 
and do not need to implement a NoOpCallback, right?

I am personally not favor of making commitSync / commitAsync function names.

 CommitType seems not necessary in commit().
 ---

 Key: KAFKA-2389
 URL: https://issues.apache.org/jira/browse/KAFKA-2389
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jiangjie Qin
Assignee: Jiangjie Qin

 The CommitType does not seem to be necessary in for commit(), it can be 
 inferred from whether user passed in a callback or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Resolved] (KAFKA-2072) Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module

2015-08-26 Thread Jun Rao (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-2072.

Resolution: Fixed

Issue resolved by pull request 170
[https://github.com/apache/kafka/pull/170]

 Add StopReplica request/response to o.a.k.common.requests and replace the 
 usage in core module
 --

 Key: KAFKA-2072
 URL: https://issues.apache.org/jira/browse/KAFKA-2072
 Project: Kafka
  Issue Type: Sub-task
Reporter: Gwen Shapira
Assignee: David Jacot
 Fix For: 0.8.3


 Add StopReplica request/response to o.a.k.common.requests and replace the 
 usage in core module



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-2072) Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module

2015-08-26 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715627#comment-14715627
 ] 

ASF GitHub Bot commented on KAFKA-2072:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/170


 Add StopReplica request/response to o.a.k.common.requests and replace the 
 usage in core module
 --

 Key: KAFKA-2072
 URL: https://issues.apache.org/jira/browse/KAFKA-2072
 Project: Kafka
  Issue Type: Sub-task
Reporter: Gwen Shapira
Assignee: David Jacot
 Fix For: 0.8.3


 Add StopReplica request/response to o.a.k.common.requests and replace the 
 usage in core module



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (KAFKA-2477) Replicas spuriously deleting all segments in partition

2015-08-26 Thread JIRA
Håkon Hitland created KAFKA-2477:


 Summary: Replicas spuriously deleting all segments in partition
 Key: KAFKA-2477
 URL: https://issues.apache.org/jira/browse/KAFKA-2477
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.1
Reporter: Håkon Hitland


We're seeing some strange behaviour in brokers: a replica will sometimes 
schedule all segments in a partition for deletion, and then immediately start 
replicating them back, triggering our check for under-replicating topics.

This happens on average a couple of times a week, for different brokers and 
topics.

We have per-topic retention.ms and retention.bytes configuration, the topics 
where we've seen this happen are hitting the size limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Created] (KAFKA-2478) KafkaConsumer javadoc example seems wrong

2015-08-26 Thread Dmitry Stratiychuk (JIRA)
Dmitry Stratiychuk created KAFKA-2478:
-

 Summary: KafkaConsumer javadoc example seems wrong
 Key: KAFKA-2478
 URL: https://issues.apache.org/jira/browse/KAFKA-2478
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 0.8.3
Reporter: Dmitry Stratiychuk
Assignee: Neha Narkhede


I was looking at this KafkaConsumer example in the javadoc:
https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L199

As I understand, commit() method commits the maximum offsets returned by the 
most recent invocation of poll() method.

In this example, there's a danger of losing the data.
Imagine the case where 300 records are returned by consumer.poll()
The commit will happen after inserting 200 records into the database.
But it will also commit the offsets for 100 records that are still unprocessed.

So if consumer fails before buffer is dumped into the database again,
then those 100 records will never be processed.

If I'm wrong, could you please clarify the behaviour of commit() method?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: KAFKA-2364 migrate docs from SVN to git

2015-08-26 Thread Ismael Juma
Hi Kumar,

One note: we need to update the documentation on how to submit changes to
the website here:

http://kafka.apache.org/contributing.html

Ismael

On Wed, Aug 26, 2015 at 3:13 PM, Manikumar Reddy ku...@nmsworks.co.in
wrote:

 Hi Guozhang,

   Our plan is to follow Gwen's suggested approach and migrate the existing
 svn site repo to new git repo.

   (1) Gwen's suggestion will help to us maintain latest docs in Kafka repo
 itself.  We periodically need to copy these latest docs to site repo. I
 will submit patch for this.

   (2)  svn repo - git repo  migration will help us to integrate site repo
 to git tooling/github. It will be easy to maintain the site repo and
 changes.  So we have created new git repo for docs and need committer help
 to create a branch asf-site.

new git repo: https://git-wip-us.apache.org/repos/asf/kafka-site.git

   Hope this clears the confusion.

 Kumar
 I thought Gwen's suggestion was to us a separate folder in the same repo
 for docs instead of a separate branch, Gwen can correct me if I was wrong?

 Guozhang

 On Mon, Aug 24, 2015 at 10:31 AM, Manikumar Reddy ku...@nmsworks.co.in
 wrote:

  Hi,
 
 Infra team created git repo for kafka site docs.
 
 Gwen/Guozhang,
 Need your help to create a branch asf-site and copy the exiting
  svn contents to that branch.
 
  git repo: https://git-wip-us.apache.org/repos/asf/kafka-site.git
 
 
 

 https://issues.apache.org/jira/browse/INFRA-10143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14709630#comment-14709630
 
  Kumar
 
  On Fri, Aug 21, 2015 at 6:16 PM, Ismael Juma ism...@juma.me.uk wrote:
 
   My preference would be to do `2` because it reduces the number of tools
  we
   need to know. If we want to clone the repo for the generated site, we
 can
   use the same tools as we do for the code repo and we can watch for
  changes
   on GitHub, etc.
  
   Ismael
  
   On Fri, Aug 21, 2015 at 1:34 PM, Manikumar Reddy ku...@nmsworks.co.in
 
   wrote:
  
Hi All,
   
Can we finalize the  approach? So that we can proceed further.
   
1. Gwen's suggestion + existing svn repo
2. Gwen's suggestion + new git repo for docs
   
kumar
   
On Thu, Aug 20, 2015 at 11:48 PM, Manikumar Reddy 
  ku...@nmsworks.co.in
wrote:
   
   Also can we migrate svn repo to git repo? This will help us to
 fix
 occasional  doc changes/bug fixes through github PR.

 On Thu, Aug 20, 2015 at 4:04 AM, Guozhang Wang wangg...@gmail.com
 
wrote:

 Gwen: I remembered it wrong. We would not need another round of
   voting.

 On Wed, Aug 19, 2015 at 3:06 PM, Gwen Shapira g...@confluent.io
wrote:

  Looking back at this thread, the +1 mention same repo, so I'm
  not
 sure a
  new vote is required.
 
  On Wed, Aug 19, 2015 at 3:00 PM, Guozhang Wang 
  wangg...@gmail.com
 wrote:
 
   So I think we have two different approaches here. The original
 proposal
   from Aseem is to move website from SVN to a separate Git repo,
  and
 hence
   have separate commits on code / doc changes. For that we have
 accumulated
   enough binging +1s to move on; Gwen's proposal is to move
  website
into
  the
   same repo under a different folder. If people feel they prefer
   this
 over
   the previous approach I would like to call for another round
 of
 voting.
  
   Guozhang
  
   On Wed, Aug 19, 2015 at 10:24 AM, Ashish 
  paliwalash...@gmail.com
   
  wrote:
  
+1 to what Gwen has suggested. This is what we follow in
  Flume.
   
All the latest doc changes are in git, once ready you move
   changes
 to
svn to update website.
The only catch is, when you need to update specific changes
 to
 website
outside release cycle, need to be a bit careful :)
   
On Wed, Aug 19, 2015 at 9:06 AM, Gwen Shapira 
   g...@confluent.io
  wrote:
 Yeah, so the way this works in few other projects I worked
  on
is:

 * The code repo has a /docs directory with the latest
  revision
of
 the
docs
 (not multiple versions, just one that matches the latest
  state
of
  code)
 * When you submit a patch that requires doc modification,
  you
 modify
   all
 relevant files in same patch and they get reviewed and
   committed
   together
 (ideally)
 * When we release, we copy the docs matching the release
 and
 commit
  to
SVN
 website. We also do this occasionally to fix bugs in
 earlier
docs.
 * Release artifacts include a copy of the docs

 Nice to have:
 * Docs are in Asciidoc and build generates the HTML.
  Asciidoc
   is
  easier
to
 edit and review.

 I suggest a similar process for Kafka.

 On 

Re: Issue when enabling SSL on broker

2015-08-26 Thread Xiang Zhou (Samuel)
Hi, Harsha,

I appreciate you very much for your response and the bash script you
provided to generate the keystores works for me and solve the problem. I
was wondering it was caused by the cipher suite differences between openjdk
and oracle-jdk, anyway it is not that case. Finally I got it worked under
both openjdk and oracle-jdk.

Thanks,
Samuel

On Tue, Aug 25, 2015 at 9:55 PM, Sriharsha Chintalapani 
harsh...@fastmail.fm wrote:

 Hi,
   Turns out to be a bug in the instructions in the wiki . I fixed it
 can you please retry generating the truststore and keystore

 https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka
  .

 checkout this section All of the above steps in a bash script” to
 generate the keystores.
 Thanks,
 Harsha


 On August 25, 2015 at 8:56:24 PM, Sriharsha Chintalapani (ka...@harsha.io)
 wrote:

 Hi Xiang,
  Did you try following the instructions here
 https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka
  .
 Whats the output of openssl s_client and which version of java and OS are
 you using.

 Thanks,
 Harsha


 On August 25, 2015 at 8:42:18 PM, Xiang Zhou (Samuel) (zhou...@gmail.com)
 wrote:

 no cipher suites in common




[jira] [Commented] (KAFKA-1690) Add SSL support to Kafka Broker, Producer and Consumer

2015-08-26 Thread Navjot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14713473#comment-14713473
 ] 

Navjot commented on KAFKA-1690:
---

Great [~harsha_ch] seems that it's time to recompile everything. I'll confirm 
post testing.

 Add SSL support to Kafka Broker, Producer and Consumer
 --

 Key: KAFKA-1690
 URL: https://issues.apache.org/jira/browse/KAFKA-1690
 Project: Kafka
  Issue Type: Sub-task
Reporter: Joe Stein
Assignee: Sriharsha Chintalapani
 Fix For: 0.8.3

 Attachments: KAFKA-1690.patch, KAFKA-1690.patch, 
 KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, 
 KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, 
 KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, 
 KAFKA-1690_2015-05-21_10:37:08.patch, KAFKA-1690_2015-06-03_18:52:29.patch, 
 KAFKA-1690_2015-06-23_13:18:20.patch, KAFKA-1690_2015-07-20_06:10:42.patch, 
 KAFKA-1690_2015-07-20_11:59:57.patch, KAFKA-1690_2015-07-25_12:10:55.patch, 
 KAFKA-1690_2015-08-16_20:41:02.patch, KAFKA-1690_2015-08-17_08:12:50.patch, 
 KAFKA-1690_2015-08-17_09:28:52.patch, KAFKA-1690_2015-08-17_12:20:53.patch, 
 KAFKA-1690_2015-08-18_11:24:46.patch, KAFKA-1690_2015-08-18_17:24:48.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-2470) kafka-producer-perf-test.sh can't configure all to request-num-acks

2015-08-26 Thread Bo Wang (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bo Wang updated KAFKA-2470:
---
Fix Version/s: 0.8.3

 kafka-producer-perf-test.sh can't configure all to request-num-acks
 ---

 Key: KAFKA-2470
 URL: https://issues.apache.org/jira/browse/KAFKA-2470
 Project: Kafka
  Issue Type: Bug
  Components: clients, tools
Affects Versions: 0.8.2.1
 Environment: Linux
Reporter: Bo Wang
 Fix For: 0.8.3

   Original Estimate: 24h
  Remaining Estimate: 24h

 For New Producer API, kafka-producer-perf-test.sh can't configure all to 
 request-num-acks :
 bin]# ./kafka-producer-perf-test.sh --topic test --broker-list host:port 
 --messages 100 --message-size 200 --new-producer --sync --batch-size 1
  --request-num-acks all
 Exception in thread main joptsimple.OptionArgumentConversionException: 
 Cannot convert argument 'all' of option ['request-num-acks'] to class 
 java.lang.Integer
   at 
 joptsimple.ArgumentAcceptingOptionSpec.convert(ArgumentAcceptingOptionSpec.java:237)
   at joptsimple.OptionSet.valuesOf(OptionSet.java:226)
   at joptsimple.OptionSet.valueOf(OptionSet.java:170)
   at 
 kafka.tools.ProducerPerformance$ProducerPerfConfig.init(ProducerPerformance.scala:146)
   at kafka.tools.ProducerPerformance$.main(ProducerPerformance.scala:42)
   at kafka.tools.ProducerPerformance.main(ProducerPerformance.scala)
 Caused by: joptsimple.internal.ReflectionException: 
 java.lang.NumberFormatException: For input string: all
   at 
 joptsimple.internal.Reflection.reflectionException(Reflection.java:136)
   at joptsimple.internal.Reflection.invoke(Reflection.java:123)
   at 
 joptsimple.internal.MethodInvokingValueConverter.convert(MethodInvokingValueConverter.java:48)
   at 
 joptsimple.ArgumentAcceptingOptionSpec.convert(ArgumentAcceptingOptionSpec.java:234)
   ... 5 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-2477) Replicas spuriously deleting all segments in partition

2015-08-26 Thread JIRA

 [ 
https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Håkon Hitland updated KAFKA-2477:
-
Attachment: kafka_log.txt

Attached example log from a broker

 Replicas spuriously deleting all segments in partition
 --

 Key: KAFKA-2477
 URL: https://issues.apache.org/jira/browse/KAFKA-2477
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.1
Reporter: Håkon Hitland
 Attachments: kafka_log.txt


 We're seeing some strange behaviour in brokers: a replica will sometimes 
 schedule all segments in a partition for deletion, and then immediately start 
 replicating them back, triggering our check for under-replicating topics.
 This happens on average a couple of times a week, for different brokers and 
 topics.
 We have per-topic retention.ms and retention.bytes configuration, the topics 
 where we've seen this happen are hitting the size limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: KAFKA-2364 migrate docs from SVN to git

2015-08-26 Thread Manikumar Reddy
Hi Guozhang,

  Our plan is to follow Gwen's suggested approach and migrate the existing
svn site repo to new git repo.

  (1) Gwen's suggestion will help to us maintain latest docs in Kafka repo
itself.  We periodically need to copy these latest docs to site repo. I
will submit patch for this.

  (2)  svn repo - git repo  migration will help us to integrate site repo
to git tooling/github. It will be easy to maintain the site repo and
changes.  So we have created new git repo for docs and need committer help
to create a branch asf-site.

   new git repo: https://git-wip-us.apache.org/repos/asf/kafka-site.git

  Hope this clears the confusion.

Kumar
I thought Gwen's suggestion was to us a separate folder in the same repo
for docs instead of a separate branch, Gwen can correct me if I was wrong?

Guozhang

On Mon, Aug 24, 2015 at 10:31 AM, Manikumar Reddy ku...@nmsworks.co.in
wrote:

 Hi,

Infra team created git repo for kafka site docs.

Gwen/Guozhang,
Need your help to create a branch asf-site and copy the exiting
 svn contents to that branch.

 git repo: https://git-wip-us.apache.org/repos/asf/kafka-site.git



https://issues.apache.org/jira/browse/INFRA-10143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14709630#comment-14709630

 Kumar

 On Fri, Aug 21, 2015 at 6:16 PM, Ismael Juma ism...@juma.me.uk wrote:

  My preference would be to do `2` because it reduces the number of tools
 we
  need to know. If we want to clone the repo for the generated site, we
can
  use the same tools as we do for the code repo and we can watch for
 changes
  on GitHub, etc.
 
  Ismael
 
  On Fri, Aug 21, 2015 at 1:34 PM, Manikumar Reddy ku...@nmsworks.co.in
  wrote:
 
   Hi All,
  
   Can we finalize the  approach? So that we can proceed further.
  
   1. Gwen's suggestion + existing svn repo
   2. Gwen's suggestion + new git repo for docs
  
   kumar
  
   On Thu, Aug 20, 2015 at 11:48 PM, Manikumar Reddy 
 ku...@nmsworks.co.in
   wrote:
  
  Also can we migrate svn repo to git repo? This will help us to fix
occasional  doc changes/bug fixes through github PR.
   
On Thu, Aug 20, 2015 at 4:04 AM, Guozhang Wang wangg...@gmail.com
   wrote:
   
Gwen: I remembered it wrong. We would not need another round of
  voting.
   
On Wed, Aug 19, 2015 at 3:06 PM, Gwen Shapira g...@confluent.io
   wrote:
   
 Looking back at this thread, the +1 mention same repo, so I'm
 not
sure a
 new vote is required.

 On Wed, Aug 19, 2015 at 3:00 PM, Guozhang Wang 
 wangg...@gmail.com
wrote:

  So I think we have two different approaches here. The original
proposal
  from Aseem is to move website from SVN to a separate Git repo,
 and
hence
  have separate commits on code / doc changes. For that we have
accumulated
  enough binging +1s to move on; Gwen's proposal is to move
 website
   into
 the
  same repo under a different folder. If people feel they prefer
  this
over
  the previous approach I would like to call for another round of
voting.
 
  Guozhang
 
  On Wed, Aug 19, 2015 at 10:24 AM, Ashish 
 paliwalash...@gmail.com
  
 wrote:
 
   +1 to what Gwen has suggested. This is what we follow in
 Flume.
  
   All the latest doc changes are in git, once ready you move
  changes
to
   svn to update website.
   The only catch is, when you need to update specific changes
to
website
   outside release cycle, need to be a bit careful :)
  
   On Wed, Aug 19, 2015 at 9:06 AM, Gwen Shapira 
  g...@confluent.io
 wrote:
Yeah, so the way this works in few other projects I worked
 on
   is:
   
* The code repo has a /docs directory with the latest
 revision
   of
the
   docs
(not multiple versions, just one that matches the latest
 state
   of
 code)
* When you submit a patch that requires doc modification,
 you
modify
  all
relevant files in same patch and they get reviewed and
  committed
  together
(ideally)
* When we release, we copy the docs matching the release
and
commit
 to
   SVN
website. We also do this occasionally to fix bugs in
earlier
   docs.
* Release artifacts include a copy of the docs
   
Nice to have:
* Docs are in Asciidoc and build generates the HTML.
 Asciidoc
  is
 easier
   to
edit and review.
   
I suggest a similar process for Kafka.
   
On Wed, Aug 19, 2015 at 8:53 AM, Ismael Juma 
  ism...@juma.me.uk
   
  wrote:
   
I should clarify: it's not possible unless we add an
  additional
step
   that
moves the docs from the code repo to the website repo.
   
Ismael
   
On Wed, Aug 19, 2015 at 4:42 PM, Ismael Juma 
   ism...@juma.me.uk
  wrote:
   
 Hi all,

 

[jira] [Commented] (KAFKA-1451) Broker stuck due to leader election race

2015-08-26 Thread Jason Kania (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14713405#comment-14713405
 ] 

Jason Kania commented on KAFKA-1451:


I too am seeing this issue in 0.8.2.1.

 Broker stuck due to leader election race 
 -

 Key: KAFKA-1451
 URL: https://issues.apache.org/jira/browse/KAFKA-1451
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.1.1
Reporter: Maciek Makowski
Assignee: Manikumar Reddy
Priority: Minor
  Labels: newbie
 Fix For: 0.8.2.0

 Attachments: KAFKA-1451.patch, KAFKA-1451_2014-07-28_20:27:32.patch, 
 KAFKA-1451_2014-07-29_10:13:23.patch


 h3. Symptoms
 The broker does not become available due to being stuck in an infinite loop 
 while electing leader. This can be recognised by the following line being 
 repeatedly written to server.log:
 {code}
 [2014-05-14 04:35:09,187] INFO I wrote this conflicted ephemeral node 
 [{version:1,brokerid:1,timestamp:1400060079108}] at /controller a 
 while back in a different session, hence I will backoff for this node to be 
 deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
 {code}
 h3. Steps to Reproduce
 In a single kafka 0.8.1.1 node, single zookeeper 3.4.6 (but will likely 
 behave the same with the ZK version included in Kafka distribution) node 
 setup:
 # start both zookeeper and kafka (in any order)
 # stop zookeeper
 # stop kafka
 # start kafka
 # start zookeeper
 h3. Likely Cause
 {{ZookeeperLeaderElector}} subscribes to data changes on startup, and then 
 triggers an election. if the deletion of ephemeral {{/controller}} node 
 associated with previous zookeeper session of the broker happens after 
 subscription to changes in new session, election will be invoked twice, once 
 from {{startup}} and once from {{handleDataDeleted}}:
 * {{startup}}: acquire {{controllerLock}}
 * {{startup}}: subscribe to data changes
 * zookeeper: delete {{/controller}} since the session that created it timed 
 out
 * {{handleDataDeleted}}: {{/controller}} was deleted
 * {{handleDataDeleted}}: wait on {{controllerLock}}
 * {{startup}}: elect -- writes {{/controller}}
 * {{startup}}: release {{controllerLock}}
 * {{handleDataDeleted}}: acquire {{controllerLock}}
 * {{handleDataDeleted}}: elect -- attempts to write {{/controller}} and then 
 gets into infinite loop as a result of conflict
 {{createEphemeralPathExpectConflictHandleZKBug}} assumes that the existing 
 znode was written from different session, which is not true in this case; it 
 was written from the same session. That adds to the confusion.
 h3. Suggested Fix
 In {{ZookeeperLeaderElector.startup}} first run {{elect}} and then subscribe 
 to data changes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Issue when enabling SSL on broker

2015-08-26 Thread Xiang Zhou (Samuel)
Hi, folks,

I am just following up to see if anybody has clue on the following issue
when I was following the guidelines provided in the step of running openssl
to verify the SSL setup:

[2015-08-25 22:29:15,385] WARN Error in I/O with connection to X
(org.apache.kafka.common.network.Selector)
javax.net.ssl.SSLHandshakeException: no cipher suites in common
at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1348)
at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:519)
at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1200)
at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1172)
at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469)
at
org.apache.kafka.common.network.SSLTransportLayer.handshakeWrap(SSLTransportLayer.java:345)
at
org.apache.kafka.common.network.SSLTransportLayer.handshake(SSLTransportLayer.java:222)
at
org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:69)
at org.apache.kafka.common.network.Selector.poll(Selector.java:290)
at kafka.network.Processor.run(SocketServer.scala:393)
at java.lang.Thread.run(Thread.java:745)
Caused by: javax.net.ssl.SSLHandshakeException: no cipher suites in common
at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1650)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:281)
at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:269)
at
sun.security.ssl.ServerHandshaker.chooseCipherSuite(ServerHandshaker.java:970)
at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:684)
at
sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:222)
at sun.security.ssl.Handshaker.processLoop(Handshaker.java:913)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:853)
at sun.security.ssl.Handshaker$1.run(Handshaker.java:851)
at java.security.AccessController.doPrivileged(Native Method)
at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1285)
at
org.apache.kafka.common.network.SSLTransportLayer.runDelegatedTasks(SSLTransportLayer.java:303)
at
org.apache.kafka.common.network.SSLTransportLayer.handshakeUnwrap(SSLTransportLayer.java:381)
at
org.apache.kafka.common.network.SSLTransportLayer.handshake(SSLTransportLayer.java:246)
... 4 more

Above exception was shown on broker side and there is no certificate shown
up when verifying with openssl command. And when I was trying to connect
broker by producer and consumer command, it got the same error message. I
tried to let ssl.cipher.suites unset and set some values in it, neither way
has luck. Appreciate any help.

Thanks,
Samuel


On Fri, Aug 21, 2015 at 6:28 AM, Ben Stopford b...@confluent.io wrote:

 Hi Qi

 Trunk seems fairly stable.

 There are guidelines here which includes how to generate keys
 https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka
 https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka
 

 Your server config needs these properties (also on the webpage):

 listeners=PLAINTEXT://:9092,SSL://:9093

 ssl.protocol = TLS
 ssl.keystore.type = JKS
 ssl.keystore.location = path/keystore.jks
 ssl.keystore.password = pass
 ssl.key.password = pass
 ssl.truststore.type = JKS
 ssl.truststore.location = path/truststore.jks
 ssl.truststore.password = pass

 To get yourself going it’s easiest to just generate a set of certs locally
 and spark up the console producer/consumer pair. You’ll need the latest cut
 from trunk (from today) to get a console consumer that works.

 Hope that helps

 Ben


  On 21 Aug 2015, at 07:10, Qi Xu shkir...@gmail.com wrote:
 
  Hi folks,
  I tried to clone the latest version of kafka truck and try to enable the
  SSL. The server.properties seems not having any security related
 settings,
  and it seems there's no other config file relevant to SSL either.
  So may I know is this feature ready to use now in truck branch?
 
  BTW, we're using the SSL feature from the branch :
  https://github.com/relango/kafka/tree/0.8.2. Is there any significant
  difference between Kafka-truck and relango's branch?
 
  Thanks,
  Qi




[jira] [Commented] (KAFKA-1690) Add SSL support to Kafka Broker, Producer and Consumer

2015-08-26 Thread Navjot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14713445#comment-14713445
 ] 

Navjot commented on KAFKA-1690:
---

[~harsha_ch] Finally I got some luck with Testing SSL Enabled Kafka Server but 
got stuck with the Consumer. I found one new confluence page written by you at 
[https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka] and 
followed the same. Now my Server and Producers are SSL enabled (Can see that by 
enabling trace) but when I run Consumer by using :
{{kafka-console-consumer.bat --bootstrap-server localhost:9093 --topic test 
--new-consumer --consumer.config security.protocol=SSL  --consumer.config 
ssl.truststore.location=client.truststore.jks --consumer.config 
ssl.truststore.password=test1234}}

{{It fails with error: 'bootstrap-server' is not a recognized option}}

Also Normal Consumer is running fine and is reading data from Producer (SSL 
enabled producer). Please suggest what might be wrong here.

 Add SSL support to Kafka Broker, Producer and Consumer
 --

 Key: KAFKA-1690
 URL: https://issues.apache.org/jira/browse/KAFKA-1690
 Project: Kafka
  Issue Type: Sub-task
Reporter: Joe Stein
Assignee: Sriharsha Chintalapani
 Fix For: 0.8.3

 Attachments: KAFKA-1690.patch, KAFKA-1690.patch, 
 KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, 
 KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, 
 KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, 
 KAFKA-1690_2015-05-21_10:37:08.patch, KAFKA-1690_2015-06-03_18:52:29.patch, 
 KAFKA-1690_2015-06-23_13:18:20.patch, KAFKA-1690_2015-07-20_06:10:42.patch, 
 KAFKA-1690_2015-07-20_11:59:57.patch, KAFKA-1690_2015-07-25_12:10:55.patch, 
 KAFKA-1690_2015-08-16_20:41:02.patch, KAFKA-1690_2015-08-17_08:12:50.patch, 
 KAFKA-1690_2015-08-17_09:28:52.patch, KAFKA-1690_2015-08-17_12:20:53.patch, 
 KAFKA-1690_2015-08-18_11:24:46.patch, KAFKA-1690_2015-08-18_17:24:48.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-1690) Add SSL support to Kafka Broker, Producer and Consumer

2015-08-26 Thread Sriharsha Chintalapani (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14713464#comment-14713464
 ] 

Sriharsha Chintalapani commented on KAFKA-1690:
---

[~NavjotBhardwaj] are you using the latest trunk since that option added 
recently. 

 Add SSL support to Kafka Broker, Producer and Consumer
 --

 Key: KAFKA-1690
 URL: https://issues.apache.org/jira/browse/KAFKA-1690
 Project: Kafka
  Issue Type: Sub-task
Reporter: Joe Stein
Assignee: Sriharsha Chintalapani
 Fix For: 0.8.3

 Attachments: KAFKA-1690.patch, KAFKA-1690.patch, 
 KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, 
 KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, 
 KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, 
 KAFKA-1690_2015-05-21_10:37:08.patch, KAFKA-1690_2015-06-03_18:52:29.patch, 
 KAFKA-1690_2015-06-23_13:18:20.patch, KAFKA-1690_2015-07-20_06:10:42.patch, 
 KAFKA-1690_2015-07-20_11:59:57.patch, KAFKA-1690_2015-07-25_12:10:55.patch, 
 KAFKA-1690_2015-08-16_20:41:02.patch, KAFKA-1690_2015-08-17_08:12:50.patch, 
 KAFKA-1690_2015-08-17_09:28:52.patch, KAFKA-1690_2015-08-17_12:20:53.patch, 
 KAFKA-1690_2015-08-18_11:24:46.patch, KAFKA-1690_2015-08-18_17:24:48.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-2478) KafkaConsumer javadoc example seems wrong

2015-08-26 Thread Jason Gustafson (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715459#comment-14715459
 ] 

Jason Gustafson commented on KAFKA-2478:


[~devstr] Not that I'm aware of. You can control the maximum fetch size in 
configuration, but that only affects individual fetches and poll() can send out 
many of these. Do you want to submit a patch for this?

 KafkaConsumer javadoc example seems wrong
 -

 Key: KAFKA-2478
 URL: https://issues.apache.org/jira/browse/KAFKA-2478
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 0.8.3
Reporter: Dmitry Stratiychuk
Assignee: Neha Narkhede

 I was looking at this KafkaConsumer example in the javadoc:
 https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L199
 As I understand, commit() method commits the maximum offsets returned by the 
 most recent invocation of poll() method.
 In this example, there's a danger of losing the data.
 Imagine the case where 300 records are returned by consumer.poll()
 The commit will happen after inserting 200 records into the database.
 But it will also commit the offsets for 100 records that are still 
 unprocessed.
 So if consumer fails before buffer is dumped into the database again,
 then those 100 records will never be processed.
 If I'm wrong, could you please clarify the behaviour of commit() method?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] kafka pull request: KAFKA-1683: persisting session information in ...

2015-08-26 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/155


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (KAFKA-1977) Make logEndOffset available in the Zookeeper consumer

2015-08-26 Thread Jason Gustafson (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715470#comment-14715470
 ] 

Jason Gustafson commented on KAFKA-1977:


Seems like there should be some options for exposing it in the ConsumerRecords 
object that is returned in the new consumer's poll. [~willf] Do you want to 
create a jira for that?

 Make logEndOffset available in the Zookeeper consumer
 -

 Key: KAFKA-1977
 URL: https://issues.apache.org/jira/browse/KAFKA-1977
 Project: Kafka
  Issue Type: Improvement
  Components: core
Reporter: Will Funnell
Priority: Minor
 Attachments: 
 Make_logEndOffset_available_in_the_Zookeeper_consumer.patch


 The requirement is to create a snapshot from the Kafka topic but NOT do 
 continual reads after that point. For example you might be creating a backup 
 of the data to a file.
 In order to achieve that, a recommended solution by Joel Koshy and Jay Kreps 
 was to expose the high watermark, as maxEndOffset, from the FetchResponse 
 object through to each MessageAndMetadata object in order to be aware when 
 the consumer has reached the end of each partition.
 The submitted patch achieves this by adding the maxEndOffset to the 
 PartitionTopicInfo, which is updated when a new message arrives in the 
 ConsumerFetcherThread and then exposed in MessageAndMetadata.
 See here for discussion:
 http://search-hadoop.com/m/4TaT4TpJy71



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-2390) OffsetOutOfRangeException should contain the Offset and Partition info.

2015-08-26 Thread Jiangjie Qin (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-2390:

Description: Currently when seek() finishes, the offset seek to is not 
verified and OffsetOutOfRangeException.   (was: Currently seek is an async 
call. To have the same interface as other calls like commit(), seek() should 
take a callback. This callback will be invoked if the position to seek triggers 
OFFSET_OUT_OF_RANGE exception from broker.)

 OffsetOutOfRangeException should contain the Offset and Partition info.
 ---

 Key: KAFKA-2390
 URL: https://issues.apache.org/jira/browse/KAFKA-2390
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jiangjie Qin
Assignee: Dong Lin

 Currently when seek() finishes, the offset seek to is not verified and 
 OffsetOutOfRangeException. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-2390) OffsetOutOfRangeException should contain the Offset and Partition info.

2015-08-26 Thread Jiangjie Qin (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-2390:

Description: Currently when seek() finishes, the offset seek to is not 
verified and OffsetOutOfRangeException might be thrown later. To let the users 
take actions when the OffsetOutOfRangeException is thrown, we need to provide 
more information in the Exception.  (was: Currently when seek() finishes, the 
offset seek to is not verified and OffsetOutOfRangeException. )

 OffsetOutOfRangeException should contain the Offset and Partition info.
 ---

 Key: KAFKA-2390
 URL: https://issues.apache.org/jira/browse/KAFKA-2390
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jiangjie Qin
Assignee: Dong Lin

 Currently when seek() finishes, the offset seek to is not verified and 
 OffsetOutOfRangeException might be thrown later. To let the users take 
 actions when the OffsetOutOfRangeException is thrown, we need to provide more 
 information in the Exception.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Comment Edited] (KAFKA-2477) Replicas spuriously deleting all segments in partition

2015-08-26 Thread JIRA

[ 
https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715742#comment-14715742
 ] 

Håkon Hitland edited comment on KAFKA-2477 at 8/26/15 11:38 PM:


We use a replication factor of 3.
The only line with start offset that day is the one in the attached log:
[2015-08-24 18:32:32,299] WARN [ReplicaFetcherThread-3-0], Replica 3 for 
partition [log.event,3] reset its fetch offset from 10200597616 to current 
leader 0's start offset 10200597616 (kafka.server.ReplicaFetcherThread)

e: the leader error reads:
[2015-08-24 18:32:32,145] ERROR [Replica Manager on Broker 0]: Error when 
processing fetch request for partition [log.event,3] offset 10349592111 from 
follower with correlation id 141609587. Possible cause: Request for offset 
10349592111 but we only have log segments in the range 10200597616 to 
10349592109. (kafka.server.ReplicaManager)


was (Author: hakon):
We use a replication factor of 3.
The only line with start offset that day is the one in the attached log:
[2015-08-24 18:32:32,299] WARN [ReplicaFetcherThread-3-0], Replica 3 for 
partition [log.event,3] reset its fetch offset from 10200597616 to current 
leader 0's start offset 10200597616 (kafka.server.ReplicaFetcherThread)

 Replicas spuriously deleting all segments in partition
 --

 Key: KAFKA-2477
 URL: https://issues.apache.org/jira/browse/KAFKA-2477
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.1
Reporter: Håkon Hitland
 Attachments: kafka_log.txt


 We're seeing some strange behaviour in brokers: a replica will sometimes 
 schedule all segments in a partition for deletion, and then immediately start 
 replicating them back, triggering our check for under-replicating topics.
 This happens on average a couple of times a week, for different brokers and 
 topics.
 We have per-topic retention.ms and retention.bytes configuration, the topics 
 where we've seen this happen are hitting the size limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-2388) subscribe(topic)/unsubscribe(topic) should either take a callback to allow user to handle exceptions or it should be synchronous.

2015-08-26 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715798#comment-14715798
 ] 

ASF GitHub Bot commented on KAFKA-2388:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/139


 subscribe(topic)/unsubscribe(topic) should either take a callback to allow 
 user to handle exceptions or it should be synchronous.
 -

 Key: KAFKA-2388
 URL: https://issues.apache.org/jira/browse/KAFKA-2388
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jiangjie Qin
Assignee: Jason Gustafson
 Fix For: 0.8.3


 According to the mailing list discussion on the consumer interface, we'll 
 replace:
 {code}
 public void subscribe(String... topics);
 public void subscribe(TopicPartition... partitions);
 public SetTopicPartition subscriptions();
 {code}
 with:
 {code}
 void subscribe(ListString topics, RebalanceCallback callback);
 void assign(ListTopicPartition partitions);
 ListString subscriptions();
 ListTopicPartition assignments();
 {code}
 We don't need the unsubscribe APIs anymore.
 The RebalanceCallback would look like:
 {code}
 interface RebalanceCallback {
   void onAssignment(ListTopicPartition partitions);
   void onRevocation(ListTopicPartition partitions);
   // handle non-existing topics, etc.
   void onError(Exception e);
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Resolved] (KAFKA-2388) subscribe(topic)/unsubscribe(topic) should either take a callback to allow user to handle exceptions or it should be synchronous.

2015-08-26 Thread Guozhang Wang (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-2388.
--
   Resolution: Fixed
Fix Version/s: 0.8.3

Issue resolved by pull request 139
[https://github.com/apache/kafka/pull/139]

 subscribe(topic)/unsubscribe(topic) should either take a callback to allow 
 user to handle exceptions or it should be synchronous.
 -

 Key: KAFKA-2388
 URL: https://issues.apache.org/jira/browse/KAFKA-2388
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jiangjie Qin
Assignee: Jason Gustafson
 Fix For: 0.8.3


 According to the mailing list discussion on the consumer interface, we'll 
 replace:
 {code}
 public void subscribe(String... topics);
 public void subscribe(TopicPartition... partitions);
 public SetTopicPartition subscriptions();
 {code}
 with:
 {code}
 void subscribe(ListString topics, RebalanceCallback callback);
 void assign(ListTopicPartition partitions);
 ListString subscriptions();
 ListTopicPartition assignments();
 {code}
 We don't need the unsubscribe APIs anymore.
 The RebalanceCallback would look like:
 {code}
 interface RebalanceCallback {
   void onAssignment(ListTopicPartition partitions);
   void onRevocation(ListTopicPartition partitions);
   // handle non-existing topics, etc.
   void onError(Exception e);
 }
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-2467) ConsoleConsumer regressions

2015-08-26 Thread Ewen Cheslack-Postava (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-2467:
-
Status: Patch Available  (was: Open)

[~gwenshap] or [~guozhang] Can one of you review and commit this? It's 
currently breaking system tests.

 ConsoleConsumer regressions
 ---

 Key: KAFKA-2467
 URL: https://issues.apache.org/jira/browse/KAFKA-2467
 Project: Kafka
  Issue Type: Bug
  Components: tools
Reporter: Ewen Cheslack-Postava
Assignee: Ewen Cheslack-Postava

 It seems that the patch for KAFKA-2015 caused a few changes in the behavior 
 of the console consumer. I picked this up because it caused the new mirror 
 maker sanity system test to hang. We need a separate fix for ducktape to 
 address the lack of a timeout where it got stuck, but I'd also like to get 
 this fixed ASAP since it affects pretty much all system test efforts since 
 they commonly use console consumer to validate data produced to Kafka.
 I've tracked down a couple of changes so far:
 1. The --consumer.config option handling was changed entirely. I think the 
 new approach was trying to parse it as key=value parameters, but it's 
 supposed to be a properties file *containing* key=value pairs.
 2. A few different exceptions during message processing are not handled the 
 same way. The skipMessageOnErrorOpt is not longer being used at all (it's 
 parsed, but that option is never checked anymore). Also, exceptions during 
 iteration are not caught. After fixing the consumer.config issue, which was 
 keeping the consumer.timeout.ms setting from making it into the consumer 
 config, this also caused the process to hang. It killed the main thread, but 
 there must be another non-daemon thread still running (presumably the 
 consumer threads?)
 3. The consumed X messages message changed from stderr to stdout.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-2467) ConsoleConsumer regressions

2015-08-26 Thread Guozhang Wang (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715879#comment-14715879
 ] 

Guozhang Wang commented on KAFKA-2467:
--

I should be the guy apologizing here.. My original patch on KAFKA-2015 was not 
complete but only sufficient for some of my local validations long time ago 
which is essentially a very small subset of the old system test. When 
[~benstopford] picked it up I did not clearly mention it to him. Sorry about 
that.

 ConsoleConsumer regressions
 ---

 Key: KAFKA-2467
 URL: https://issues.apache.org/jira/browse/KAFKA-2467
 Project: Kafka
  Issue Type: Bug
  Components: tools
Reporter: Ewen Cheslack-Postava
Assignee: Ewen Cheslack-Postava

 It seems that the patch for KAFKA-2015 caused a few changes in the behavior 
 of the console consumer. I picked this up because it caused the new mirror 
 maker sanity system test to hang. We need a separate fix for ducktape to 
 address the lack of a timeout where it got stuck, but I'd also like to get 
 this fixed ASAP since it affects pretty much all system test efforts since 
 they commonly use console consumer to validate data produced to Kafka.
 I've tracked down a couple of changes so far:
 1. The --consumer.config option handling was changed entirely. I think the 
 new approach was trying to parse it as key=value parameters, but it's 
 supposed to be a properties file *containing* key=value pairs.
 2. A few different exceptions during message processing are not handled the 
 same way. The skipMessageOnErrorOpt is not longer being used at all (it's 
 parsed, but that option is never checked anymore). Also, exceptions during 
 iteration are not caught. After fixing the consumer.config issue, which was 
 keeping the consumer.timeout.ms setting from making it into the consumer 
 config, this also caused the process to hang. It killed the main thread, but 
 there must be another non-daemon thread still running (presumably the 
 consumer threads?)
 3. The consumed X messages message changed from stderr to stdout.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-2067) Add LeaderAndISR request/response to org.apache.kafka.common.requests and replace usage in core module

2015-08-26 Thread Ismael Juma (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715785#comment-14715785
 ] 

Ismael Juma commented on KAFKA-2067:


Needed for KAFKA-2411

 Add LeaderAndISR request/response to org.apache.kafka.common.requests and 
 replace usage in core module
 --

 Key: KAFKA-2067
 URL: https://issues.apache.org/jira/browse/KAFKA-2067
 Project: Kafka
  Issue Type: Sub-task
Reporter: Gwen Shapira
Assignee: Ismael Juma
 Fix For: 0.8.3


 Add LeaderAndISR request/response to org.apache.kafka.common.requests and 
 replace usage in core module.
 Note that this will require adding a bunch of new objects to o.a.k.common - 
 LeaderAndISR, LeaderISRAndEpoch and possibly others.
 It may be nice to have a scala implicit to translate those objects from their 
 old (core) implementation to the o.a.k.common implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Re: [DISCUSS] Client-side Assignment for New Consumer

2015-08-26 Thread Jiangjie Qin
Hi folks,

After further discussion in LinkedIn, we found that while having a more
general group management protocol is very useful, the vast majority of the
clients will not use customized partition assignment strategy. In that
case, the broker side partition assignment would be ideal because it avoids
issues like metadata inconsistency / split brain / exploding subscription
set propagation.

So we have the following proposal that satisfies the majority of the
clients' needs without changing the currently proposed binary protocol.
i.e., Continue to support broker-side assignment if the assignment strategy
is recognized by the coordinator.

1. Keep the binary protocol as currently proposed.

2. Change the way we interpret ProtocolMetadata:
2.1 On consumer side, change partition.assignment.strategy to
partition.assignor.class. Implement the something like the following
PartitionAssignor Interface:

public interface PartitionAssignor {
  ListString protocolTypes();
  byte[] protocolMetadata();
  // return the Topic-ListPartition map that are assigned to this
consumer.
  ListTopicPartition assignPartitions(String protocolType, byte[]
responseProtocolMetadata);
}

public abstract class AbstractPartitionAssignor implements
PartitionAssignor {
  protected final KafkaConsumer consumer;
  AbstractPartitionAssignor(KafkaConsumer consumer) {
this.consumer = consumer;
  }
}

2.2 The ProtocolMetadata in JoinGroupRequest will be
partitionAssignor.protocolMetadata(). When partition.assignor.class is
range or roundrobin, the ProtocolMetadata in JoinGroupRequest will be a
JSON subscription set. (range, roundrobin will be reserved words, we
can also consider reserving some Prefix such as broker- to be more clear)
2.3 On broker side when ProtocolType is range or roundroubin,
coordinator will parse the ProtocolMetadata in the JoinGroupRequest and
assign the partitions for consumers. In the JoinGroupResponse, the
ProtocolMetadata will be the global assignment of partitions.
2.4 On client side, after receiving the JoinGroupResponse,
partitionAssignor.assignPartitions() will be invoked to return the actual
assignment. If the assignor is RangeAssignor or RoundRobinAssignor, they
will parse the assignment from the ProtocolMetadata returned by coordinator.

This approach has a few merits:
1. Does not change the proposed binary protocol, which is still general.
2. The majority of the consumers will not suffer from inconsistent metadata
/ split brain / exploding subscription set propagation. This is
specifically to deal with the issue that the current proposal caters to a
20% use-case while adversely impacting the more common 80% use-cases.
3. Easy to implement. The only thing needed is implement a partitioner
class. For most users, the default range and roundrobin partitioner are
good enough.

Thoughts?

Thanks,

Jiangjie (Becket) Qin

On Tue, Aug 18, 2015 at 2:51 PM, Jason Gustafson ja...@confluent.io wrote:

 Follow-up from the kip call:

 1. Onur brought up the question of whether this protocol provides enough
 coordination capabilities to be generally useful in practice (is that
 accurate, Onur?). If it doesn't, then each use case would probably need a
 dependence on zookeeper anyway, and we haven't really gained anything. The
 group membership provided by this protocol is a useful primitive for
 coordination, but it's limited in the sense that everything shared among
 the group has to be communicated at the time the group is created. If any
 shared data changes, then the only way the group can ensure agreement is to
 force a rebalance. This is expensive since all members must stall while the
 rebalancing takes place. As we have also seen, there is a practical limit
 on the amount of metadata that can be sent through this protocol when
 groups get a little larger. This protocol is therefore not suitable to
 cases which require frequent communication or which require a large amount
 of data to be communicated. For the use cases listed on the wiki, neither
 of these appear to be an issue, but there may be other limitations which
 would limit reuse of the protocol. Perhaps it would be sufficient to sketch
 how these cases might work?

 2. We talked a little bit about the issue of metadata churn. Becket brought
 up the interesting point that not only do we depend on topic metadata
 changing relatively infrequently, but we also expect timely agreement among
 the brokers on what that metadata is. To resolve this, we can have the
 consumers fetch metadata from the coordinator. We still depend on topic
 metadata not changing frequently, but this should resolve any disagreement
 among the brokers themselves. In fact, since we expect that disagreement is
 relatively rare, we can have the consumers fetch from the coordinator only
 when when a disagreement occurs. The nice thing about this proposal is that
 it doesn't affect the join group semantics, so the coordinator would remain
 oblivious to the metadata used by the group for 

[jira] [Comment Edited] (KAFKA-2065) Add ControlledShutdown to org.apache.kafka.common.requests and replace current use in core module

2015-08-26 Thread Ismael Juma (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715780#comment-14715780
 ] 

Ismael Juma edited comment on KAFKA-2065 at 8/27/15 12:04 AM:
--

I've done this as part of KAFKA-2411, I hope you don't mind.


was (Author: ijuma):
I've done this as part of KAFKA-2417, I hope you don't mind.

 Add ControlledShutdown to  org.apache.kafka.common.requests and replace 
 current use in core module
 --

 Key: KAFKA-2065
 URL: https://issues.apache.org/jira/browse/KAFKA-2065
 Project: Kafka
  Issue Type: Sub-task
Reporter: Gwen Shapira
Assignee: Ismael Juma
 Fix For: 0.8.3


 Add ControlledShutdown to  org.apache.kafka.common.requests and replace 
 current use in core module



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-2065) Add ControlledShutdown to org.apache.kafka.common.requests and replace current use in core module

2015-08-26 Thread Ismael Juma (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715780#comment-14715780
 ] 

Ismael Juma commented on KAFKA-2065:


I've done this as part of KAFKA-2417, I hope you don't mind.

 Add ControlledShutdown to  org.apache.kafka.common.requests and replace 
 current use in core module
 --

 Key: KAFKA-2065
 URL: https://issues.apache.org/jira/browse/KAFKA-2065
 Project: Kafka
  Issue Type: Sub-task
Reporter: Gwen Shapira
Assignee: Ismael Juma
 Fix For: 0.8.3


 Add ControlledShutdown to  org.apache.kafka.common.requests and replace 
 current use in core module



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Assigned] (KAFKA-2065) Add ControlledShutdown to org.apache.kafka.common.requests and replace current use in core module

2015-08-26 Thread Ismael Juma (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reassigned KAFKA-2065:
--

Assignee: Ismael Juma  (was: Ashish K Singh)

 Add ControlledShutdown to  org.apache.kafka.common.requests and replace 
 current use in core module
 --

 Key: KAFKA-2065
 URL: https://issues.apache.org/jira/browse/KAFKA-2065
 Project: Kafka
  Issue Type: Sub-task
Reporter: Gwen Shapira
Assignee: Ismael Juma
 Fix For: 0.8.3


 Add ControlledShutdown to  org.apache.kafka.common.requests and replace 
 current use in core module



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-2389) CommitType seems not necessary in commit().

2015-08-26 Thread Ewen Cheslack-Postava (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715805#comment-14715805
 ] 

Ewen Cheslack-Postava commented on KAFKA-2389:
--

I'd also say that just because the Zookeeper API has it doesn't make it good 
API design... I don't know about other people, but I raised the concern about 
naming again in the code review because I actually did get confused about what 
the semantics of each method signature was.

 CommitType seems not necessary in commit().
 ---

 Key: KAFKA-2389
 URL: https://issues.apache.org/jira/browse/KAFKA-2389
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jiangjie Qin
Assignee: Jiangjie Qin

 The CommitType does not seem to be necessary in for commit(), it can be 
 inferred from whether user passed in a callback or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-2389) CommitType seems not necessary in commit().

2015-08-26 Thread Guozhang Wang (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715846#comment-14715846
 ] 

Guozhang Wang commented on KAFKA-2389:
--

Yeah, I did not think that Zookeeper API is the golden standard and we should 
just align with it, I guess I am just subjectively leaning towards the same 
function names for sync / async since for sync methods users can always put 
their callback logic after the commit() call hence they do not need a callback 
parameter, and wanted to use Zookeeper API as a backing point. Regarding 
[~becket_qin]'s point I agree that would be the issue. Anyways since this is 
pure personal taste I am willing to go with the majority vote.

 CommitType seems not necessary in commit().
 ---

 Key: KAFKA-2389
 URL: https://issues.apache.org/jira/browse/KAFKA-2389
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jiangjie Qin
Assignee: Jiangjie Qin

 The CommitType does not seem to be necessary in for commit(), it can be 
 inferred from whether user passed in a callback or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Build failed in Jenkins: Kafka-trunk #602

2015-08-26 Thread Apache Jenkins Server
See https://builds.apache.org/job/Kafka-trunk/602/changes

Changes:

[wangguoz] KAFKA-2388: refactor KafkaConsumer subscribe API

--
[...truncated 1633 lines...]
kafka.coordinator.ConsumerCoordinatorResponseTest  
testHeartbeatIllegalGeneration PASSED

kafka.admin.DeleteConsumerGroupTest  testTopicWideDeleteInZK PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest  
testHeartbeatDuringRebalanceCausesIllegalGeneration PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest  testValidJoinGroup PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest  
testHeartbeatUnknownConsumerExistingGroup PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest  
testJoinGroupWrongCoordinator PASSED

kafka.integration.TopicMetadataTest  
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.admin.AdminTest  testPartitionReassignmentWithLeaderNotInNewReplicas 
PASSED

kafka.api.QuotasTest  testProducerConsumerOverrideUnthrottled PASSED

kafka.utils.timer.TimerTaskListTest  testAll PASSED

kafka.integration.TopicMetadataTest  testBasicTopicMetadata PASSED

kafka.api.ConsumerTest  testPartitionPauseAndResume PASSED

kafka.admin.DeleteTopicTest  testRecreateTopicAfterDeletion PASSED

kafka.integration.TopicMetadataTest  testGetAllTopicMetadata PASSED

kafka.api.ProducerBounceTest  testBrokerFailure PASSED

kafka.admin.AddPartitionsTest  testTopicDoesNotExist PASSED

kafka.integration.AutoOffsetResetTest  testResetToLatestWhenOffsetTooLow PASSED

kafka.admin.DeleteConsumerGroupTest  
testConsumptionOnRecreatedTopicAfterTopicWideDeleteInZK PASSED

kafka.api.ProducerSendTest  testCloseWithZeroTimeoutFromSenderThread PASSED

kafka.utils.SchedulerTest  testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest  testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest  testNonPeriodicTask PASSED

kafka.utils.SchedulerTest  testPeriodicTask PASSED

kafka.utils.SchedulerTest  testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest  testRestart PASSED

kafka.utils.timer.TimerTest  testTaskExpiration PASSED

kafka.utils.timer.TimerTest  testAlreadyExpiredTask PASSED

kafka.integration.TopicMetadataTest  testTopicMetadataRequest PASSED

kafka.log.LogTest  testCorruptLog PASSED

kafka.integration.AutoOffsetResetTest  testResetToEarliestWhenOffsetTooLow 
PASSED

kafka.integration.FetcherTest  testFetcher PASSED

kafka.common.ConfigTest  testInvalidClientIds PASSED

kafka.common.ConfigTest  testInvalidGroupIds PASSED

kafka.log.LogTest  testCleanShutdownFile PASSED

kafka.log.LogTest  testParseTopicPartitionName PASSED

kafka.log.LogTest  testParseTopicPartitionNameForEmptyName PASSED

kafka.log.LogTest  testParseTopicPartitionNameForNull PASSED

kafka.log.LogTest  testParseTopicPartitionNameForMissingSeparator PASSED

kafka.log.LogTest  testParseTopicPartitionNameForMissingTopic PASSED

kafka.log.LogTest  testParseTopicPartitionNameForMissingPartition PASSED

kafka.admin.AdminTest  testPartitionReassignmentNonOverlappingReplicas PASSED

kafka.admin.DeleteTopicTest  testDeleteNonExistingTopic PASSED

kafka.log.LogCleanerIntegrationTest  cleanerTest[0] PASSED

kafka.api.ConsumerTest  testPauseStateNotPreservedByRebalance PASSED

kafka.integration.AutoOffsetResetTest  testResetToLatestWhenOffsetTooHigh 
PASSED

kafka.api.ProducerFailureHandlingTest  testTooLargeRecordWithAckZero PASSED

kafka.admin.AdminTest  testReassigningNonExistingPartition PASSED

kafka.integration.AutoOffsetResetTest  testResetToEarliestWhenOffsetTooHigh 
PASSED

kafka.admin.DeleteTopicTest  testResumeDeleteTopicWithRecoveredFollower PASSED

kafka.admin.AddPartitionsTest  testReplicaPlacement PASSED

kafka.integration.UncleanLeaderElectionTest  
testUncleanLeaderElectionInvalidTopicOverride PASSED

kafka.api.ProducerFailureHandlingTest  testTooLargeRecordWithAckOne PASSED

kafka.api.ProducerFailureHandlingTest  testNonExistentTopic PASSED

kafka.api.ConsumerBounceTest  testConsumptionWithBrokerFailures PASSED

kafka.utils.IteratorTemplateTest  testIterator PASSED

kafka.integration.RollingBounceTest  testRollingBounce PASSED

kafka.common.TopicTest  testTopicHasCollisionChars PASSED

kafka.common.TopicTest  testInvalidTopicNames PASSED

kafka.common.TopicTest  testTopicHasCollision PASSED

kafka.api.ConsumerTest  testSimpleConsumption PASSED

kafka.integration.UncleanLeaderElectionTest  
testUncleanLeaderElectionEnabledByTopicOverride PASSED

kafka.api.ProducerFailureHandlingTest  testWrongBrokerList PASSED

kafka.api.ConsumerTest  testCommitSpecifiedOffsets PASSED

kafka.log.LogCleanerIntegrationTest  cleanerTest[1] PASSED

kafka.api.ProducerFailureHandlingTest  testNoResponse PASSED

kafka.admin.DeleteTopicTest  testDeleteTopicWithCleaner PASSED

kafka.integration.UncleanLeaderElectionTest  testUncleanLeaderElectionEnabled 
PASSED

kafka.api.ConsumerTest  testAutoOffsetReset PASSED

kafka.api.ProducerFailureHandlingTest  testInvalidPartition PASSED


Build failed in Jenkins: Kafka-trunk #601

2015-08-26 Thread Apache Jenkins Server
See https://builds.apache.org/job/Kafka-trunk/601/changes

Changes:

[junrao] KAFKA-2072: Add StopReplica request/response to o.a.k.common.requests

--
[...truncated 2936 lines...]
kafka.api.ProducerSendTest  testCloseWithZeroTimeoutFromCallerThread PASSED

kafka.integration.TopicMetadataTest  
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest  
testJoinGroupWrongCoordinator PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest  
testGenerationIdIncrementsOnRebalance PASSED

kafka.admin.DeleteTopicTest  testResumeDeleteTopicOnControllerFailover PASSED

kafka.api.ConsumerTest  testPartitionPauseAndResume PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest  testValidHeartbeat PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest  
testJoinGroupSessionTimeoutTooLarge PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest  
testJoinGroupSessionTimeoutTooSmall PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest  
testHeartbeatDuringRebalanceCausesIllegalGeneration PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest  
testJoinGroupUnknownPartitionAssignmentStrategy PASSED

kafka.admin.DeleteConsumerGroupTest  testTopicWideDeleteInZK PASSED

kafka.integration.TopicMetadataTest  testTopicMetadataRequest PASSED

kafka.api.ProducerBounceTest  testBrokerFailure PASSED

kafka.utils.timer.TimerTest  testTaskExpiration PASSED

kafka.utils.timer.TimerTest  testAlreadyExpiredTask PASSED

kafka.integration.TopicMetadataTest  testAutoCreateTopicWithCollision PASSED

kafka.integration.AutoOffsetResetTest  testResetToEarliestWhenOffsetTooHigh 
PASSED

kafka.admin.AddPartitionsTest  testTopicDoesNotExist PASSED

kafka.admin.AdminTest  testBasicPreferredReplicaElection PASSED

kafka.api.ConsumerTest  testPauseStateNotPreservedByRebalance PASSED

kafka.admin.DeleteTopicTest  testPartitionReassignmentDuringDeleteTopic PASSED

kafka.integration.AutoOffsetResetTest  testResetToEarliestWhenOffsetTooLow 
PASSED

kafka.admin.DeleteConsumerGroupTest  
testConsumptionOnRecreatedTopicAfterTopicWideDeleteInZK PASSED

kafka.log.LogTest  testCorruptLog PASSED

kafka.admin.AdminTest  testShutdownBroker PASSED

kafka.log.LogTest  testCleanShutdownFile PASSED

kafka.log.LogTest  testParseTopicPartitionName PASSED

kafka.log.LogTest  testParseTopicPartitionNameForEmptyName PASSED

kafka.log.LogTest  testParseTopicPartitionNameForNull PASSED

kafka.log.LogTest  testParseTopicPartitionNameForMissingSeparator PASSED

kafka.log.LogTest  testParseTopicPartitionNameForMissingTopic PASSED

kafka.log.LogTest  testParseTopicPartitionNameForMissingPartition PASSED

kafka.integration.FetcherTest  testFetcher PASSED

kafka.common.ConfigTest  testInvalidGroupIds PASSED

kafka.common.ConfigTest  testInvalidClientIds PASSED

kafka.api.ProducerSendTest  testCloseWithZeroTimeoutFromSenderThread PASSED

kafka.utils.SchedulerTest  testPeriodicTask PASSED

kafka.utils.SchedulerTest  testRestart PASSED

kafka.utils.SchedulerTest  testNonPeriodicTask PASSED

kafka.utils.SchedulerTest  testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest  testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest  testReentrantTaskInMockScheduler PASSED

kafka.admin.DeleteTopicTest  testDeleteNonExistingTopic PASSED

kafka.log.LogCleanerIntegrationTest  cleanerTest[0] PASSED

kafka.admin.AddPartitionsTest  testIncrementPartitions PASSED

kafka.integration.AutoOffsetResetTest  testResetToLatestWhenOffsetTooHigh 
PASSED

kafka.api.ConsumerTest  testPartitionsFor PASSED

kafka.admin.AdminTest  testTopicConfigChange PASSED

kafka.api.ProducerFailureHandlingTest  testTooLargeRecordWithAckZero PASSED

kafka.integration.AutoOffsetResetTest  testResetToLatestWhenOffsetTooLow PASSED

kafka.api.ProducerFailureHandlingTest  testTooLargeRecordWithAckOne PASSED

kafka.api.ConsumerTest  testListTopics PASSED

kafka.api.ConsumerBounceTest  testConsumptionWithBrokerFailures PASSED

kafka.utils.IteratorTemplateTest  testIterator PASSED

kafka.api.ProducerFailureHandlingTest  testNonExistentTopic PASSED

kafka.integration.RollingBounceTest  testRollingBounce PASSED

kafka.common.TopicTest  testInvalidTopicNames PASSED

kafka.common.TopicTest  testTopicHasCollisionChars PASSED

kafka.common.TopicTest  testTopicHasCollision PASSED

kafka.api.ConsumerTest  testPartitionReassignmentCallback PASSED

kafka.api.ProducerFailureHandlingTest  testWrongBrokerList PASSED

kafka.admin.DeleteTopicTest  testDeleteTopicWithCleaner PASSED

kafka.api.ConsumerTest  testUnsubscribeTopic PASSED

kafka.integration.UncleanLeaderElectionTest  
testCleanLeaderElectionDisabledByTopicOverride PASSED

kafka.log.LogCleanerIntegrationTest  cleanerTest[1] PASSED

kafka.admin.DeleteTopicTest  testDeleteTopicAlreadyMarkedAsDeleted PASSED

kafka.api.ProducerFailureHandlingTest  testNoResponse PASSED

kafka.integration.PrimitiveApiTest  testProduceAndMultiFetch PASSED


[jira] [Commented] (KAFKA-2477) Replicas spuriously deleting all segments in partition

2015-08-26 Thread JIRA

[ 
https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715742#comment-14715742
 ] 

Håkon Hitland commented on KAFKA-2477:
--

We use a replication factor of 3.
The only line with start offset that day is the one in the attached log:
[2015-08-24 18:32:32,299] WARN [ReplicaFetcherThread-3-0], Replica 3 for 
partition [log.event,3] reset its fetch offset from 10200597616 to current 
leader 0's start offset 10200597616 (kafka.server.ReplicaFetcherThread)

 Replicas spuriously deleting all segments in partition
 --

 Key: KAFKA-2477
 URL: https://issues.apache.org/jira/browse/KAFKA-2477
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.1
Reporter: Håkon Hitland
 Attachments: kafka_log.txt


 We're seeing some strange behaviour in brokers: a replica will sometimes 
 schedule all segments in a partition for deletion, and then immediately start 
 replicating them back, triggering our check for under-replicating topics.
 This happens on average a couple of times a week, for different brokers and 
 topics.
 We have per-topic retention.ms and retention.bytes configuration, the topics 
 where we've seen this happen are hitting the size limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Assigned] (KAFKA-2067) Add LeaderAndISR request/response to org.apache.kafka.common.requests and replace usage in core module

2015-08-26 Thread Ismael Juma (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reassigned KAFKA-2067:
--

Assignee: Ismael Juma

 Add LeaderAndISR request/response to org.apache.kafka.common.requests and 
 replace usage in core module
 --

 Key: KAFKA-2067
 URL: https://issues.apache.org/jira/browse/KAFKA-2067
 Project: Kafka
  Issue Type: Sub-task
Reporter: Gwen Shapira
Assignee: Ismael Juma
 Fix For: 0.8.3


 Add LeaderAndISR request/response to org.apache.kafka.common.requests and 
 replace usage in core module.
 Note that this will require adding a bunch of new objects to o.a.k.common - 
 LeaderAndISR, LeaderISRAndEpoch and possibly others.
 It may be nice to have a scala implicit to translate those objects from their 
 old (core) implementation to the o.a.k.common implementation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[GitHub] kafka pull request: KAFKA-2388 [WIP]; refactor KafkaConsumer subsc...

2015-08-26 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/139


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


[jira] [Commented] (KAFKA-2477) Replicas spuriously deleting all segments in partition

2015-08-26 Thread JIRA

[ 
https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715518#comment-14715518
 ] 

Håkon Hitland commented on KAFKA-2477:
--

Thanks for the reply. Checking the logs, we did get the Error when processing 
fetch request error in the leader mentioned in KAFKA-2143, so it could be the 
same issue.

I don't see anything in our logs about a leader change, so I don't think it is 
caused by an unclean election, like some of the comments suggest.

 Replicas spuriously deleting all segments in partition
 --

 Key: KAFKA-2477
 URL: https://issues.apache.org/jira/browse/KAFKA-2477
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.1
Reporter: Håkon Hitland
 Attachments: kafka_log.txt


 We're seeing some strange behaviour in brokers: a replica will sometimes 
 schedule all segments in a partition for deletion, and then immediately start 
 replicating them back, triggering our check for under-replicating topics.
 This happens on average a couple of times a week, for different brokers and 
 topics.
 We have per-topic retention.ms and retention.bytes configuration, the topics 
 where we've seen this happen are hitting the size limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-2389) CommitType seems not necessary in commit().

2015-08-26 Thread Guozhang Wang (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715565#comment-14715565
 ] 

Guozhang Wang commented on KAFKA-2389:
--

I'm late on this discussion, but here are my two cents:

1. We need to rename the async commits to commitAsync or something similar, 
you can't have two methods with the same name that behave totally differently 
and have different post-conditions.

I feel it is OK to have sync / async differentiated by the callback rather than 
by their names. For example ZooKeeper has a similar approach regarding sync / 
async APIs:

http://zookeeper.apache.org/doc/r3.4.5/api/org/apache/zookeeper/ZooKeeper.html#exists(java.lang.String,
 org.apache.zookeeper.Watcher, org.apache.zookeeper.AsyncCallback.StatCallback, 
java.lang.Object)

2. We need to include variants of asyncCommit that don't take the callback. 
Having the user implement or discover a NoOpCallback to be able to use the api 
is not good.

With commit(OffsetCommitCallback callback), users can just call commit(null) 
and do not need to implement a NoOpCallback, right?

I am personally not favor of making commitSync / commitAsync function names.

 CommitType seems not necessary in commit().
 ---

 Key: KAFKA-2389
 URL: https://issues.apache.org/jira/browse/KAFKA-2389
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jiangjie Qin
Assignee: Jiangjie Qin

 The CommitType does not seem to be necessary in for commit(), it can be 
 inferred from whether user passed in a callback or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-2477) Replicas spuriously deleting all segments in partition

2015-08-26 Thread Jiangjie Qin (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715604#comment-14715604
 ] 

Jiangjie Qin commented on KAFKA-2477:
-

What is the partition replication factor?

 Replicas spuriously deleting all segments in partition
 --

 Key: KAFKA-2477
 URL: https://issues.apache.org/jira/browse/KAFKA-2477
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.1
Reporter: Håkon Hitland
 Attachments: kafka_log.txt


 We're seeing some strange behaviour in brokers: a replica will sometimes 
 schedule all segments in a partition for deletion, and then immediately start 
 replicating them back, triggering our check for under-replicating topics.
 This happens on average a couple of times a week, for different brokers and 
 topics.
 We have per-topic retention.ms and retention.bytes configuration, the topics 
 where we've seen this happen are hitting the size limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Work started] (KAFKA-2072) Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module

2015-08-26 Thread David Jacot (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-2072 started by David Jacot.
--
 Add StopReplica request/response to o.a.k.common.requests and replace the 
 usage in core module
 --

 Key: KAFKA-2072
 URL: https://issues.apache.org/jira/browse/KAFKA-2072
 Project: Kafka
  Issue Type: Sub-task
Reporter: Gwen Shapira
Assignee: David Jacot
 Fix For: 0.8.3


 Add StopReplica request/response to o.a.k.common.requests and replace the 
 usage in core module



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Reopened] (KAFKA-2072) Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module

2015-08-26 Thread David Jacot (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

David Jacot reopened KAFKA-2072:


Reopen this JIRA as https://github.com/apache/kafka/pull/170 adds StopReplica 
request/response to o.a.k.common.requests (required for KAFKA-2411) but doesn't 
replace the usage in core module. The replacement will be addressed in a second 
PR once KAFKA-2411 gets in.

Is it ok like this? Or should it be addressed in follow up JIRA?





 Add StopReplica request/response to o.a.k.common.requests and replace the 
 usage in core module
 --

 Key: KAFKA-2072
 URL: https://issues.apache.org/jira/browse/KAFKA-2072
 Project: Kafka
  Issue Type: Sub-task
Reporter: Gwen Shapira
Assignee: David Jacot
 Fix For: 0.8.3


 Add StopReplica request/response to o.a.k.common.requests and replace the 
 usage in core module



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-1690) Add SSL support to Kafka Broker, Producer and Consumer

2015-08-26 Thread Navjot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14713667#comment-14713667
 ] 

Navjot commented on KAFKA-1690:
---

And it started working. I'll now try to do it via API. Thanks for all your help 
[~harsha_ch]

 Add SSL support to Kafka Broker, Producer and Consumer
 --

 Key: KAFKA-1690
 URL: https://issues.apache.org/jira/browse/KAFKA-1690
 Project: Kafka
  Issue Type: Sub-task
Reporter: Joe Stein
Assignee: Sriharsha Chintalapani
 Fix For: 0.8.3

 Attachments: KAFKA-1690.patch, KAFKA-1690.patch, 
 KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, 
 KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, 
 KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, 
 KAFKA-1690_2015-05-21_10:37:08.patch, KAFKA-1690_2015-06-03_18:52:29.patch, 
 KAFKA-1690_2015-06-23_13:18:20.patch, KAFKA-1690_2015-07-20_06:10:42.patch, 
 KAFKA-1690_2015-07-20_11:59:57.patch, KAFKA-1690_2015-07-25_12:10:55.patch, 
 KAFKA-1690_2015-08-16_20:41:02.patch, KAFKA-1690_2015-08-17_08:12:50.patch, 
 KAFKA-1690_2015-08-17_09:28:52.patch, KAFKA-1690_2015-08-17_12:20:53.patch, 
 KAFKA-1690_2015-08-18_11:24:46.patch, KAFKA-1690_2015-08-18_17:24:48.patch






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Updated] (KAFKA-2467) ConsoleConsumer regressions

2015-08-26 Thread Guozhang Wang (JIRA)

 [ 
https://issues.apache.org/jira/browse/KAFKA-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2467:
-
   Resolution: Fixed
Fix Version/s: 0.8.3
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 166
[https://github.com/apache/kafka/pull/166]

 ConsoleConsumer regressions
 ---

 Key: KAFKA-2467
 URL: https://issues.apache.org/jira/browse/KAFKA-2467
 Project: Kafka
  Issue Type: Bug
  Components: tools
Reporter: Ewen Cheslack-Postava
Assignee: Ewen Cheslack-Postava
 Fix For: 0.8.3


 It seems that the patch for KAFKA-2015 caused a few changes in the behavior 
 of the console consumer. I picked this up because it caused the new mirror 
 maker sanity system test to hang. We need a separate fix for ducktape to 
 address the lack of a timeout where it got stuck, but I'd also like to get 
 this fixed ASAP since it affects pretty much all system test efforts since 
 they commonly use console consumer to validate data produced to Kafka.
 I've tracked down a couple of changes so far:
 1. The --consumer.config option handling was changed entirely. I think the 
 new approach was trying to parse it as key=value parameters, but it's 
 supposed to be a properties file *containing* key=value pairs.
 2. A few different exceptions during message processing are not handled the 
 same way. The skipMessageOnErrorOpt is not longer being used at all (it's 
 parsed, but that option is never checked anymore). Also, exceptions during 
 iteration are not caught. After fixing the consumer.config issue, which was 
 keeping the consumer.timeout.ms setting from making it into the consumer 
 config, this also caused the process to hang. It killed the main thread, but 
 there must be another non-daemon thread still running (presumably the 
 consumer threads?)
 3. The consumed X messages message changed from stderr to stdout.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-2467) ConsoleConsumer regressions

2015-08-26 Thread ASF GitHub Bot (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715886#comment-14715886
 ] 

ASF GitHub Bot commented on KAFKA-2467:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/166


 ConsoleConsumer regressions
 ---

 Key: KAFKA-2467
 URL: https://issues.apache.org/jira/browse/KAFKA-2467
 Project: Kafka
  Issue Type: Bug
  Components: tools
Reporter: Ewen Cheslack-Postava
Assignee: Ewen Cheslack-Postava
 Fix For: 0.8.3


 It seems that the patch for KAFKA-2015 caused a few changes in the behavior 
 of the console consumer. I picked this up because it caused the new mirror 
 maker sanity system test to hang. We need a separate fix for ducktape to 
 address the lack of a timeout where it got stuck, but I'd also like to get 
 this fixed ASAP since it affects pretty much all system test efforts since 
 they commonly use console consumer to validate data produced to Kafka.
 I've tracked down a couple of changes so far:
 1. The --consumer.config option handling was changed entirely. I think the 
 new approach was trying to parse it as key=value parameters, but it's 
 supposed to be a properties file *containing* key=value pairs.
 2. A few different exceptions during message processing are not handled the 
 same way. The skipMessageOnErrorOpt is not longer being used at all (it's 
 parsed, but that option is never checked anymore). Also, exceptions during 
 iteration are not caught. After fixing the consumer.config issue, which was 
 keeping the consumer.timeout.ms setting from making it into the consumer 
 config, this also caused the process to hang. It killed the main thread, but 
 there must be another non-daemon thread still running (presumably the 
 consumer threads?)
 3. The consumed X messages message changed from stderr to stdout.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


[jira] [Commented] (KAFKA-2466) ConsoleConsumer throws ConcurrentModificationException on termination

2015-08-26 Thread Guozhang Wang (JIRA)

[ 
https://issues.apache.org/jira/browse/KAFKA-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715889#comment-14715889
 ] 

Guozhang Wang commented on KAFKA-2466:
--

This is fixed as part of KAFKA-2466.

 ConsoleConsumer throws ConcurrentModificationException on termination
 -

 Key: KAFKA-2466
 URL: https://issues.apache.org/jira/browse/KAFKA-2466
 Project: Kafka
  Issue Type: Bug
  Components: tools
Reporter: Ashish K Singh
Assignee: Ashish K Singh
 Fix For: 0.8.3


 ConsoleConsumer throws ConcurrentModificationException on termination.
 ST:
 {code}
 Exception in thread Thread-1 java.util.ConcurrentModificationException: 
 KafkaConsumer is not safe for multi-threaded access
   at 
 org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:1169)
   at 
 org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1087)
   at kafka.consumer.NewShinyConsumer.close(BaseConsumer.scala:50)
   at kafka.tools.ConsoleConsumer$$anon$1.run(ConsoleConsumer.scala:74)
 {code}
 Other thread which constantly tries to consume is
 {code}
 main prio=10 tid=0x7f3aa800c000 nid=0x1314 runnable [0x7f3aae37d000]
java.lang.Thread.State: RUNNABLE
   at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
   at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
   at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79)
   at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87)
   - locked 0xdd1df130 (a sun.nio.ch.Util$2)
   - locked 0xdd1df120 (a java.util.Collections$UnmodifiableSet)
   - locked 0xdd0af720 (a sun.nio.ch.EPollSelectorImpl)
   at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98)
   at org.apache.kafka.common.network.Selector.select(Selector.java:440)
   at org.apache.kafka.common.network.Selector.poll(Selector.java:263)
   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:221)
   at 
 org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:274)
   at 
 org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:182)
   at 
 org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:172)
   at 
 org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:779)
   at 
 org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:730)
   at kafka.consumer.NewShinyConsumer.receive(BaseConsumer.scala:43)
   at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:87)
   at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:54)
   at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:39)
   at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
 {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)


Build failed in Jenkins: Kafka-trunk #603

2015-08-26 Thread Apache Jenkins Server
See https://builds.apache.org/job/Kafka-trunk/603/changes

Changes:

[wangguoz] KAFKA-2467: Fix changes to behavior in ConsoleConsumer: properly 
parse consumer.config option, handle exceptions during message processing, and 
print number of processed messages to stderr.

--
[...truncated 897 lines...]
kafka.utils.timer.TimerTest  testTaskExpiration PASSED

kafka.server.OffsetCommitTest  testUpdateOffsets PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest  testHeartbeatUnknownGroup 
PASSED

kafka.api.ProducerSendTest  testCloseWithZeroTimeoutFromCallerThread PASSED

kafka.admin.DeleteConsumerGroupTest  testTopicWideDeleteInZK PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest  
testJoinGroupUnknownConsumerExistingGroup PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest  
testJoinGroupSessionTimeoutTooSmall PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest  
testHeartbeatIllegalGeneration PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest  
testGenerationIdIncrementsOnRebalance PASSED

kafka.coordinator.ConsumerCoordinatorResponseTest  
testHeartbeatUnknownConsumerExistingGroup PASSED

kafka.integration.TopicMetadataTest  testIsrAfterBrokerShutDownAndJoinsBack 
PASSED

kafka.admin.AdminTest  testBasicPreferredReplicaElection PASSED

kafka.admin.DeleteTopicTest  testAddPartitionDuringDeleteTopic PASSED

kafka.admin.AddPartitionsTest  testWrongReplicaCount PASSED

kafka.integration.AutoOffsetResetTest  testResetToLatestWhenOffsetTooHigh 
PASSED

kafka.admin.DeleteConsumerGroupTest  
testConsumptionOnRecreatedTopicAfterTopicWideDeleteInZK PASSED

kafka.admin.DeleteTopicTest  testRecreateTopicAfterDeletion PASSED

kafka.admin.AdminTest  testShutdownBroker PASSED

kafka.admin.AdminTest  testReplicaAssignment PASSED

kafka.integration.FetcherTest  testFetcher PASSED

kafka.common.ConfigTest  testInvalidGroupIds PASSED

kafka.common.ConfigTest  testInvalidClientIds PASSED

kafka.admin.AdminTest  testTopicCreationWithCollision PASSED

kafka.api.ProducerSendTest  testCloseWithZeroTimeoutFromSenderThread PASSED

kafka.utils.SchedulerTest  testPeriodicTask PASSED

kafka.utils.SchedulerTest  testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest  testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest  testNonPeriodicTask PASSED

kafka.utils.SchedulerTest  testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest  testRestart PASSED

kafka.api.ConsumerTest  testSimpleConsumption PASSED

kafka.integration.AutoOffsetResetTest  testResetToLatestWhenOffsetTooLow PASSED

kafka.admin.AdminTest  testTopicCreationInZK PASSED

kafka.admin.AddPartitionsTest  testIncrementPartitions PASSED

kafka.admin.DeleteTopicTest  testDeleteNonExistingTopic PASSED

kafka.admin.AdminTest  testTopicConfigChange PASSED

kafka.integration.AutoOffsetResetTest  testResetToEarliestWhenOffsetTooHigh 
PASSED

kafka.log.LogTest  testCorruptLog PASSED

kafka.log.LogTest  testCleanShutdownFile PASSED

kafka.log.LogTest  testParseTopicPartitionName PASSED

kafka.log.LogTest  testParseTopicPartitionNameForEmptyName PASSED

kafka.log.LogTest  testParseTopicPartitionNameForNull PASSED

kafka.log.LogTest  testParseTopicPartitionNameForMissingSeparator PASSED

kafka.log.LogTest  testParseTopicPartitionNameForMissingTopic PASSED

kafka.log.LogTest  testParseTopicPartitionNameForMissingPartition PASSED

kafka.log.LogCleanerIntegrationTest  cleanerTest[0] PASSED

kafka.integration.AutoOffsetResetTest  testResetToEarliestWhenOffsetTooLow 
PASSED

kafka.api.ConsumerTest  testCommitSpecifiedOffsets PASSED

kafka.api.ConsumerBounceTest  testConsumptionWithBrokerFailures PASSED

kafka.utils.IteratorTemplateTest  testIterator PASSED

kafka.api.ProducerFailureHandlingTest  testNoResponse PASSED

kafka.api.ConsumerTest  testPositionAndCommit PASSED

kafka.integration.RollingBounceTest  testRollingBounce PASSED

kafka.common.TopicTest  testTopicHasCollisionChars PASSED

kafka.common.TopicTest  testTopicHasCollision PASSED

kafka.common.TopicTest  testInvalidTopicNames PASSED

kafka.api.ProducerFailureHandlingTest  testInvalidPartition PASSED

kafka.api.ConsumerTest  testPartitionsFor PASSED

kafka.api.ProducerFailureHandlingTest  testSendAfterClosed PASSED

kafka.integration.UncleanLeaderElectionTest  
testCleanLeaderElectionDisabledByTopicOverride PASSED

kafka.admin.DeleteTopicTest  testDeleteTopicWithCleaner PASSED

kafka.api.ProducerFailureHandlingTest  testCannotSendToInternalTopic PASSED

kafka.api.ProducerFailureHandlingTest  testNotEnoughReplicas PASSED

kafka.api.ConsumerTest  testListTopics PASSED

kafka.admin.DeleteTopicTest  testDeleteTopicAlreadyMarkedAsDeleted PASSED

kafka.log.LogCleanerIntegrationTest  cleanerTest[1] PASSED

kafka.integration.PrimitiveApiTest  testEmptyFetchRequest PASSED

kafka.integration.PrimitiveApiTest  testConsumerEmptyTopic PASSED

kafka.integration.UncleanLeaderElectionTest  
testUncleanLeaderElectionEnabledByTopicOverride 

[GitHub] kafka pull request: KAFKA-2467: Fix changes to behavior in Console...

2015-08-26 Thread asfgit
Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/166


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---


Re: [DISCUSS] Client-side Assignment for New Consumer

2015-08-26 Thread Neha Narkhede
Hey Becket,

In that case, the broker side partition assignment would be ideal because
 it avoids
 issues like metadata inconsistency / split brain / exploding subscription
 set propagation.


As per our previous discussions regarding each of those concerns (referring
to this email thread, KIP calls and JIRA comments), we are going to run a
set of tests using the LinkedIn deployment numbers that we will wait for
you to share. The purpose is to see if those concerns are really valid or
not. I'd prefer to see that before making any more changes that will
complicate the protocol.

On Wed, Aug 26, 2015 at 4:57 PM, Jiangjie Qin j...@linkedin.com.invalid
wrote:

 Hi folks,

 After further discussion in LinkedIn, we found that while having a more
 general group management protocol is very useful, the vast majority of the
 clients will not use customized partition assignment strategy. In that
 case, the broker side partition assignment would be ideal because it avoids
 issues like metadata inconsistency / split brain / exploding subscription
 set propagation.

 So we have the following proposal that satisfies the majority of the
 clients' needs without changing the currently proposed binary protocol.
 i.e., Continue to support broker-side assignment if the assignment strategy
 is recognized by the coordinator.

 1. Keep the binary protocol as currently proposed.

 2. Change the way we interpret ProtocolMetadata:
 2.1 On consumer side, change partition.assignment.strategy to
 partition.assignor.class. Implement the something like the following
 PartitionAssignor Interface:

 public interface PartitionAssignor {
   ListString protocolTypes();
   byte[] protocolMetadata();
   // return the Topic-ListPartition map that are assigned to this
 consumer.
   ListTopicPartition assignPartitions(String protocolType, byte[]
 responseProtocolMetadata);
 }

 public abstract class AbstractPartitionAssignor implements
 PartitionAssignor {
   protected final KafkaConsumer consumer;
   AbstractPartitionAssignor(KafkaConsumer consumer) {
 this.consumer = consumer;
   }
 }

 2.2 The ProtocolMetadata in JoinGroupRequest will be
 partitionAssignor.protocolMetadata(). When partition.assignor.class is
 range or roundrobin, the ProtocolMetadata in JoinGroupRequest will be a
 JSON subscription set. (range, roundrobin will be reserved words, we
 can also consider reserving some Prefix such as broker- to be more clear)
 2.3 On broker side when ProtocolType is range or roundroubin,
 coordinator will parse the ProtocolMetadata in the JoinGroupRequest and
 assign the partitions for consumers. In the JoinGroupResponse, the
 ProtocolMetadata will be the global assignment of partitions.
 2.4 On client side, after receiving the JoinGroupResponse,
 partitionAssignor.assignPartitions() will be invoked to return the actual
 assignment. If the assignor is RangeAssignor or RoundRobinAssignor, they
 will parse the assignment from the ProtocolMetadata returned by
 coordinator.

 This approach has a few merits:
 1. Does not change the proposed binary protocol, which is still general.
 2. The majority of the consumers will not suffer from inconsistent metadata
 / split brain / exploding subscription set propagation. This is
 specifically to deal with the issue that the current proposal caters to a
 20% use-case while adversely impacting the more common 80% use-cases.
 3. Easy to implement. The only thing needed is implement a partitioner
 class. For most users, the default range and roundrobin partitioner are
 good enough.

 Thoughts?

 Thanks,

 Jiangjie (Becket) Qin

 On Tue, Aug 18, 2015 at 2:51 PM, Jason Gustafson ja...@confluent.io
 wrote:

  Follow-up from the kip call:
 
  1. Onur brought up the question of whether this protocol provides enough
  coordination capabilities to be generally useful in practice (is that
  accurate, Onur?). If it doesn't, then each use case would probably need a
  dependence on zookeeper anyway, and we haven't really gained anything.
 The
  group membership provided by this protocol is a useful primitive for
  coordination, but it's limited in the sense that everything shared among
  the group has to be communicated at the time the group is created. If any
  shared data changes, then the only way the group can ensure agreement is
 to
  force a rebalance. This is expensive since all members must stall while
 the
  rebalancing takes place. As we have also seen, there is a practical limit
  on the amount of metadata that can be sent through this protocol when
  groups get a little larger. This protocol is therefore not suitable to
  cases which require frequent communication or which require a large
 amount
  of data to be communicated. For the use cases listed on the wiki, neither
  of these appear to be an issue, but there may be other limitations which
  would limit reuse of the protocol. Perhaps it would be sufficient to
 sketch
  how these cases might work?
 
  2. We talked a little bit about the issue of metadata