[jira] [Commented] (KAFKA-2478) KafkaConsumer javadoc example seems wrong
[ https://issues.apache.org/jira/browse/KAFKA-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715362#comment-14715362 ] Dmitry Stratiychuk commented on KAFKA-2478: --- Or flush the buffer and commit at the end of processing all records. Is there a way to control how many records are returned by the poll() method? KafkaConsumer javadoc example seems wrong - Key: KAFKA-2478 URL: https://issues.apache.org/jira/browse/KAFKA-2478 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8.3 Reporter: Dmitry Stratiychuk Assignee: Neha Narkhede I was looking at this KafkaConsumer example in the javadoc: https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L199 As I understand, commit() method commits the maximum offsets returned by the most recent invocation of poll() method. In this example, there's a danger of losing the data. Imagine the case where 300 records are returned by consumer.poll() The commit will happen after inserting 200 records into the database. But it will also commit the offsets for 100 records that are still unprocessed. So if consumer fails before buffer is dumped into the database again, then those 100 records will never be processed. If I'm wrong, could you please clarify the behaviour of commit() method? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1977) Make logEndOffset available in the Zookeeper consumer
[ https://issues.apache.org/jira/browse/KAFKA-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715430#comment-14715430 ] Will Funnell edited comment on KAFKA-1977 at 8/26/15 8:12 PM: -- I would still definitely like to see this in the new consumer, I think its a small thing to include, but very useful, especially when determining when you have reached the end of our log compacted topics. Our implementation is as follows: {code} IteratorConsumerMessage iterator = new IteratorConsumerMessage() { public boolean finished; private Integer partition; private ConsumerIteratorbyte[], byte[] it = stream.iterator(); @Override public boolean hasNext() { if (finished) { return false; } else { try { return it.hasNext(); } catch (Exception e) { if (hasBeenForciblyShutdownByClient(e)) { consumer.shutdown(); return false; } LOG.error(partition={} description=\Error while fetching from Kafka\, partition, e); throw e; } } } @Override public ConsumerMessage next() { MessageAndMetadatabyte[], byte[] messageAndMetadata = it.next(); count++; if (partition == null) { partition = messageAndMetadata.partition(); } if (messageAndMetadata.offset() == messageAndMetadata.logEndOffset() - 1) { finished = true; LOG.info(partition=\{}\, description=\Finished with partition\, messageAndMetadata.partition()); } return toConsumedMessage(messageAndMetadata); } @Override public void remove() { it.remove(); } private boolean hasBeenForciblyShutdownByClient(Exception e) { return e instanceof InterruptedException; } }; {code} Not quite sure how this translates to the new Consumer yet. was (Author: willf): I would still definitely like to see this in the new consumer, I think its a small thing to include, but very useful, especially when determining when you have reached the end of our log compacted topics. Our implementation is as follows: {code} IteratorConsumerMessage iterator = new IteratorConsumerMessage() { public boolean finished; private Integer partition; private ConsumerIteratorbyte[], byte[] it = stream.iterator(); private long count; @Override public boolean hasNext() { if (finished) { return false; } else { try { return it.hasNext(); } catch (Exception e) { if (hasBeenForciblyShutdownByClient(e)) { consumer.shutdown(); return false; } LOG.error(partition={} description=\Error while fetching from Kafka\, partition, e); throw e; } } } @Override public ConsumerMessage next() { MessageAndMetadatabyte[], byte[] messageAndMetadata = it.next(); count++; if (partition == null) { partition = messageAndMetadata.partition(); } if (LOG.isDebugEnabled()) { LOG.debug(count={} partition={} description=\Messages read from Kafka\, count, messageAndMetadata.partition()); } if (messageAndMetadata.offset() == messageAndMetadata.logEndOffset() - 1) { finished = true; LOG.info(partition=\{}\, description=\Finished with partition\, messageAndMetadata.partition()); } return toConsumedMessage(messageAndMetadata); } @Override public void remove() { it.remove(); } private boolean hasBeenForciblyShutdownByClient(Exception e) { return e instanceof InterruptedException; } }; {code} Not quite sure how this translates to the new Consumer yet. Make logEndOffset available in the Zookeeper consumer - Key: KAFKA-1977 URL: https://issues.apache.org/jira/browse/KAFKA-1977 Project:
[jira] [Commented] (KAFKA-1977) Make logEndOffset available in the Zookeeper consumer
[ https://issues.apache.org/jira/browse/KAFKA-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715409#comment-14715409 ] Gwen Shapira commented on KAFKA-1977: - I lost track of the new consumer APIs, so I'm ok with whatever [~hachikuji] thinks :) Make logEndOffset available in the Zookeeper consumer - Key: KAFKA-1977 URL: https://issues.apache.org/jira/browse/KAFKA-1977 Project: Kafka Issue Type: Improvement Components: core Reporter: Will Funnell Priority: Minor Attachments: Make_logEndOffset_available_in_the_Zookeeper_consumer.patch The requirement is to create a snapshot from the Kafka topic but NOT do continual reads after that point. For example you might be creating a backup of the data to a file. In order to achieve that, a recommended solution by Joel Koshy and Jay Kreps was to expose the high watermark, as maxEndOffset, from the FetchResponse object through to each MessageAndMetadata object in order to be aware when the consumer has reached the end of each partition. The submitted patch achieves this by adding the maxEndOffset to the PartitionTopicInfo, which is updated when a new message arrives in the ConsumerFetcherThread and then exposed in MessageAndMetadata. See here for discussion: http://search-hadoop.com/m/4TaT4TpJy71 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1977) Make logEndOffset available in the Zookeeper consumer
[ https://issues.apache.org/jira/browse/KAFKA-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715430#comment-14715430 ] Will Funnell commented on KAFKA-1977: - I would still definitely like to see this in the new consumer, I think its a small thing to include, but very useful, especially when determining when you have reached the end of our log compacted topics. Our implementation is as follows: {code} IteratorConsumerMessage iterator = new IteratorConsumerMessage() { public boolean finished; private Integer partition; private ConsumerIteratorbyte[], byte[] it = stream.iterator(); private long count; @Override public boolean hasNext() { if (finished) { return false; } else { try { return it.hasNext(); } catch (Exception e) { if (hasBeenForciblyShutdownByClient(e)) { consumer.shutdown(); return false; } LOG.error(partition={} description=\Error while fetching from Kafka\, partition, e); throw e; } } } @Override public ConsumerMessage next() { MessageAndMetadatabyte[], byte[] messageAndMetadata = it.next(); count++; if (partition == null) { partition = messageAndMetadata.partition(); } if (LOG.isDebugEnabled()) { LOG.debug(count={} partition={} description=\Messages read from Kafka\, count, messageAndMetadata.partition()); } if (messageAndMetadata.offset() == messageAndMetadata.logEndOffset() - 1) { finished = true; LOG.info(partition=\{}\, description=\Finished with partition\, messageAndMetadata.partition()); } return toConsumedMessage(messageAndMetadata); } @Override public void remove() { it.remove(); } private boolean hasBeenForciblyShutdownByClient(Exception e) { return e instanceof InterruptedException; } }; {code} Not quite sure how this translates to the new Consumer yet. Make logEndOffset available in the Zookeeper consumer - Key: KAFKA-1977 URL: https://issues.apache.org/jira/browse/KAFKA-1977 Project: Kafka Issue Type: Improvement Components: core Reporter: Will Funnell Priority: Minor Attachments: Make_logEndOffset_available_in_the_Zookeeper_consumer.patch The requirement is to create a snapshot from the Kafka topic but NOT do continual reads after that point. For example you might be creating a backup of the data to a file. In order to achieve that, a recommended solution by Joel Koshy and Jay Kreps was to expose the high watermark, as maxEndOffset, from the FetchResponse object through to each MessageAndMetadata object in order to be aware when the consumer has reached the end of each partition. The submitted patch achieves this by adding the maxEndOffset to the PartitionTopicInfo, which is updated when a new message arrives in the ConsumerFetcherThread and then exposed in MessageAndMetadata. See here for discussion: http://search-hadoop.com/m/4TaT4TpJy71 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 36858: Patch for KAFKA-2120
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36858/#review96095 --- Can you rebase? clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java (line 130) https://reviews.apache.org/r/36858/#comment151310 Returns a list of nodes with pending inflight requests that need to be timed out clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java (line 138) https://reviews.apache.org/r/36858/#comment151311 Can do without this guard. clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java (line 141) https://reviews.apache.org/r/36858/#comment151323 Is this right? i.e., `lastSent`? clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java (line 228) https://reviews.apache.org/r/36858/#comment152086 This logic is a bit confusing. Is this block necessary here? i.e., vs being written once below? clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java (line 245) https://reviews.apache.org/r/36858/#comment152089 I'm not sure we should remove the _replication timeout_ though. i.e., sure the replication timeout should not be used for request timeout going forward, but we still need a replication timeout in the producer request. clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java (line 428) https://reviews.apache.org/r/36858/#comment152144 Minor point: given that there may be a custom serializer and custom partitioner, the elapsed check should probably be made after each step. clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java (line 100) https://reviews.apache.org/r/36858/#comment152146 See comment above on the (continued) need for replication timeout. Also, can you fix whitespace on all your comments? i.e., `* @deprecated` instead of `*@deprecated`. clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java (line 196) https://reviews.apache.org/r/36858/#comment152148 Let us make this doc string as exhaustive and clear as possible - i.e., drop the etc. and enumerate everything. You can also add something along the lines of - also see the request timeout config, with a doc link. clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java (line 94) https://reviews.apache.org/r/36858/#comment152158 Can you add the new param to the javadoc? clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java (line 155) https://reviews.apache.org/r/36858/#comment152162 same here clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java (line 205) https://reviews.apache.org/r/36858/#comment152168 Maybe drop the due to This would be just one possibility right? E.g., you could have a low request timeout and high linger time... possibly other scenarios as well. clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java (line 214) https://reviews.apache.org/r/36858/#comment152170 typo in comment clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java (line 221) https://reviews.apache.org/r/36858/#comment152172 If you do a reverse iteration and a batch has _not_ expired, then we can break early right? - Joel Koshy On Aug. 12, 2015, 5:59 p.m., Mayuresh Gharat wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36858/ --- (Updated Aug. 12, 2015, 5:59 p.m.) Review request for kafka. Bugs: KAFKA-2120 https://issues.apache.org/jira/browse/KAFKA-2120 Repository: kafka Description --- Solved compile error Addressed Jason's comments for Kip-19 Addressed Jun's comments Addressed Jason's comments about the default values for requestTimeout Diffs - clients/src/main/java/org/apache/kafka/clients/ClientRequest.java dc8f0f115bcda893c95d17c0a57be8d14518d034 clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 7ab2503794ff3aab39df881bd9fbae6547827d3b clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 0e51d7bd461d253f4396a5b6ca7cd391658807fa clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java d35b421a515074d964c7fccb73d260b847ea5f00 clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java ed99e9bdf7c4ea7a6d4555d4488cf8ed0b80641b
[jira] [Commented] (KAFKA-2477) Replicas spuriously deleting all segments in partition
[ https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715280#comment-14715280 ] Jiangjie Qin commented on KAFKA-2477: - Maybe related to KAFKA-2143. Replicas spuriously deleting all segments in partition -- Key: KAFKA-2477 URL: https://issues.apache.org/jira/browse/KAFKA-2477 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.1 Reporter: Håkon Hitland Attachments: kafka_log.txt We're seeing some strange behaviour in brokers: a replica will sometimes schedule all segments in a partition for deletion, and then immediately start replicating them back, triggering our check for under-replicating topics. This happens on average a couple of times a week, for different brokers and topics. We have per-topic retention.ms and retention.bytes configuration, the topics where we've seen this happen are hitting the size limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1977) Make logEndOffset available in the Zookeeper consumer
[ https://issues.apache.org/jira/browse/KAFKA-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715394#comment-14715394 ] James Cheng commented on KAFKA-1977: Ping [~gwenshap]. Do you think logEndOffset can/should/will be exposed in the new Java consumer? Make logEndOffset available in the Zookeeper consumer - Key: KAFKA-1977 URL: https://issues.apache.org/jira/browse/KAFKA-1977 Project: Kafka Issue Type: Improvement Components: core Reporter: Will Funnell Priority: Minor Attachments: Make_logEndOffset_available_in_the_Zookeeper_consumer.patch The requirement is to create a snapshot from the Kafka topic but NOT do continual reads after that point. For example you might be creating a backup of the data to a file. In order to achieve that, a recommended solution by Joel Koshy and Jay Kreps was to expose the high watermark, as maxEndOffset, from the FetchResponse object through to each MessageAndMetadata object in order to be aware when the consumer has reached the end of each partition. The submitted patch achieves this by adding the maxEndOffset to the PartitionTopicInfo, which is updated when a new message arrives in the ConsumerFetcherThread and then exposed in MessageAndMetadata. See here for discussion: http://search-hadoop.com/m/4TaT4TpJy71 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Can I get permission to create proposal
Done. Guozhang On Tue, Aug 25, 2015 at 11:33 AM, Abhishek Nigam ani...@linkedin.com.invalid wrote: For pinning the controller to a broker KAFKA-1778 I want to create a proposal detailing the design. My user id is:anigam -Abhishek -- -- Guozhang
[jira] [Commented] (KAFKA-2478) KafkaConsumer javadoc example seems wrong
[ https://issues.apache.org/jira/browse/KAFKA-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14714505#comment-14714505 ] Jason Gustafson commented on KAFKA-2478: [~devstr] I agree that the example seems incorrect. Probably we should only be committing the max offset stored in the buffer. KafkaConsumer javadoc example seems wrong - Key: KAFKA-2478 URL: https://issues.apache.org/jira/browse/KAFKA-2478 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8.3 Reporter: Dmitry Stratiychuk Assignee: Neha Narkhede I was looking at this KafkaConsumer example in the javadoc: https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L199 As I understand, commit() method commits the maximum offsets returned by the most recent invocation of poll() method. In this example, there's a danger of losing the data. Imagine the case where 300 records are returned by consumer.poll() The commit will happen after inserting 200 records into the database. But it will also commit the offsets for 100 records that are still unprocessed. So if consumer fails before buffer is dumped into the database again, then those 100 records will never be processed. If I'm wrong, could you please clarify the behaviour of commit() method? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] kafka pull request: KAFKA-2072: Add StopReplica request/response t...
GitHub user dajac opened a pull request: https://github.com/apache/kafka/pull/170 KAFKA-2072: Add StopReplica request/response to o.a.k.common.requests This PR adds StopReplica request and response as it is required by @ijuma for KAFKA-2411. Migration of core module is addressed a separate PR (#141). @ijuma Could you review it? @gwenshap Could you take a look as well? You can merge this pull request into a Git repository by running: $ git pull https://github.com/dajac/kafka KAFKA-2072-part-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/170.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #170 commit f8f4d691d43cc260377e630fcaf67d31923b37e3 Author: David Jacot david.ja...@gmail.com Date: 2015-08-13T15:36:49Z Add o.a.k.c.r.StopReplicaRequest and o.a.k.c.r.StopReplicaResponse. --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (KAFKA-2072) Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module
[ https://issues.apache.org/jira/browse/KAFKA-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14714470#comment-14714470 ] ASF GitHub Bot commented on KAFKA-2072: --- GitHub user dajac opened a pull request: https://github.com/apache/kafka/pull/170 KAFKA-2072: Add StopReplica request/response to o.a.k.common.requests This PR adds StopReplica request and response as it is required by @ijuma for KAFKA-2411. Migration of core module is addressed a separate PR (#141). @ijuma Could you review it? @gwenshap Could you take a look as well? You can merge this pull request into a Git repository by running: $ git pull https://github.com/dajac/kafka KAFKA-2072-part-1 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/170.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #170 commit f8f4d691d43cc260377e630fcaf67d31923b37e3 Author: David Jacot david.ja...@gmail.com Date: 2015-08-13T15:36:49Z Add o.a.k.c.r.StopReplicaRequest and o.a.k.c.r.StopReplicaResponse. Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module -- Key: KAFKA-2072 URL: https://issues.apache.org/jira/browse/KAFKA-2072 Project: Kafka Issue Type: Sub-task Reporter: Gwen Shapira Assignee: David Jacot Fix For: 0.8.3 Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-2477) Replicas spuriously deleting all segments in partition
[ https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715604#comment-14715604 ] Jiangjie Qin edited comment on KAFKA-2477 at 8/26/15 10:13 PM: --- What is the partition replication factor? Also, can you search for start offset in the server log of the broker who truncates its log? was (Author: becket_qin): What is the partition replication factor? Replicas spuriously deleting all segments in partition -- Key: KAFKA-2477 URL: https://issues.apache.org/jira/browse/KAFKA-2477 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.1 Reporter: Håkon Hitland Attachments: kafka_log.txt We're seeing some strange behaviour in brokers: a replica will sometimes schedule all segments in a partition for deletion, and then immediately start replicating them back, triggering our check for under-replicating topics. This happens on average a couple of times a week, for different brokers and topics. We have per-topic retention.ms and retention.bytes configuration, the topics where we've seen this happen are hitting the size limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2390) OffsetOutOfRangeException should contain the Offset and Partition info.
[ https://issues.apache.org/jira/browse/KAFKA-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-2390: Summary: OffsetOutOfRangeException should contain the Offset and Partition info. (was: Seek() should take a callback.) OffsetOutOfRangeException should contain the Offset and Partition info. --- Key: KAFKA-2390 URL: https://issues.apache.org/jira/browse/KAFKA-2390 Project: Kafka Issue Type: Sub-task Reporter: Jiangjie Qin Assignee: Dong Lin Currently seek is an async call. To have the same interface as other calls like commit(), seek() should take a callback. This callback will be invoked if the position to seek triggers OFFSET_OUT_OF_RANGE exception from broker. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2390) Seek() should take a callback.
[ https://issues.apache.org/jira/browse/KAFKA-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715682#comment-14715682 ] Jiangjie Qin commented on KAFKA-2390: - Makes sense, so if user needs the offsets, they can get it by themselves. If they don't need it, no extra requests will be wasted. I'll change the ticket title. Seek() should take a callback. -- Key: KAFKA-2390 URL: https://issues.apache.org/jira/browse/KAFKA-2390 Project: Kafka Issue Type: Sub-task Reporter: Jiangjie Qin Assignee: Dong Lin Currently seek is an async call. To have the same interface as other calls like commit(), seek() should take a callback. This callback will be invoked if the position to seek triggers OFFSET_OUT_OF_RANGE exception from broker. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2120) Add a request timeout to NetworkClient
[ https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-2120: -- Status: In Progress (was: Patch Available) Add a request timeout to NetworkClient -- Key: KAFKA-2120 URL: https://issues.apache.org/jira/browse/KAFKA-2120 Project: Kafka Issue Type: New Feature Reporter: Jiangjie Qin Assignee: Mayuresh Gharat Fix For: 0.8.3 Attachments: KAFKA-2120.patch, KAFKA-2120_2015-07-27_15:31:19.patch, KAFKA-2120_2015-07-29_15:57:02.patch, KAFKA-2120_2015-08-10_19:55:18.patch, KAFKA-2120_2015-08-12_10:59:09.patch Currently NetworkClient does not have a timeout setting for requests. So if no response is received for a request due to reasons such as broker is down, the request will never be completed. Request timeout will also be used as implicit timeout for some methods such as KafkaProducer.flush() and kafkaProducer.close(). KIP-19 is created for this public interface change. https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 36858: Patch for KAFKA-2120
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36858/#review96586 --- clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java (line 240) https://reviews.apache.org/r/36858/#comment152173 Also, per the KIP agreement, the request timeout will reset for retries - are you handling that? - Joel Koshy On Aug. 12, 2015, 5:59 p.m., Mayuresh Gharat wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/36858/ --- (Updated Aug. 12, 2015, 5:59 p.m.) Review request for kafka. Bugs: KAFKA-2120 https://issues.apache.org/jira/browse/KAFKA-2120 Repository: kafka Description --- Solved compile error Addressed Jason's comments for Kip-19 Addressed Jun's comments Addressed Jason's comments about the default values for requestTimeout Diffs - clients/src/main/java/org/apache/kafka/clients/ClientRequest.java dc8f0f115bcda893c95d17c0a57be8d14518d034 clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 2c421f42ed3fc5d61cf9c87a7eaa7bb23e26f63b clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 7ab2503794ff3aab39df881bd9fbae6547827d3b clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 0e51d7bd461d253f4396a5b6ca7cd391658807fa clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java d35b421a515074d964c7fccb73d260b847ea5f00 clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java ed99e9bdf7c4ea7a6d4555d4488cf8ed0b80641b clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java 9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 03b8dd23df63a8d8a117f02eabcce4a2d48c44f7 clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java aa264202f2724907924985a5ecbe74afc4c6c04b clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java 4cb1e50d6c4ed55241aeaef1d3af09def5274103 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java a152bd7697dca55609a9ec4cfe0a82c10595fbc3 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java 06182db1c3a5da85648199b4c0c98b80ea7c6c0c clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 0baf16e55046a2f49f6431e01d52c323c95eddf0 clients/src/main/java/org/apache/kafka/common/network/Selector.java ce20111ac434eb8c74585e9c63757bb9d60a832f clients/src/test/java/org/apache/kafka/clients/MockClient.java 9133d85342b11ba2c9888d4d2804d181831e7a8e clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 43238ceaad0322e39802b615bb805b895336a009 clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java 5b2e4ffaeab7127648db608c179703b27b577414 clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java 8b1805d3d2bcb9fe2bacb37d870c3236aa9532c4 clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 158f9829ff64a969008f699e40c51e918287859e core/src/main/scala/kafka/tools/ProducerPerformance.scala 0335cc64013ffe2cdf1c4879e86e11ec8c526712 core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala ee94011894b46864614b97bbd2a98375a7d3f20b core/src/test/scala/unit/kafka/utils/TestUtils.scala eb169d8b33c27d598cc24e5a2e5f78b789fa38d3 Diff: https://reviews.apache.org/r/36858/diff/ Testing --- Thanks, Mayuresh Gharat
[GitHub] kafka pull request: KAFKA-2072: Add StopReplica request/response t...
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/170 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715489#comment-14715489 ] ASF GitHub Bot commented on KAFKA-1683: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/155 Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.8.2.1 Reporter: Jay Kreps Assignee: Gwen Shapira Fix For: 0.8.3 Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KAFKA-1683) Implement a session concept in the socket server
[ https://issues.apache.org/jira/browse/KAFKA-1683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gwen Shapira resolved KAFKA-1683. - Resolution: Fixed Issue resolved by pull request 155 [https://github.com/apache/kafka/pull/155] Implement a session concept in the socket server -- Key: KAFKA-1683 URL: https://issues.apache.org/jira/browse/KAFKA-1683 Project: Kafka Issue Type: Sub-task Components: security Affects Versions: 0.8.2.1 Reporter: Jay Kreps Assignee: Gwen Shapira Fix For: 0.8.3 Attachments: KAFKA-1683.patch, KAFKA-1683.patch To implement authentication we need a way to keep track of some things between requests. The initial use for this would be remembering the authenticated user/principle info, but likely more uses would come up (for example we will also need to remember whether and which encryption or integrity measures are in place on the socket so we can wrap and unwrap writes and reads). I was thinking we could just add a Session object that might have a user field. The session object would need to get added to RequestChannel.Request so it is passed down to the API layer with each request. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2210) KafkaAuthorizer: Add all public entities, config changes and changes to KafkaAPI and kafkaServer to allow pluggable authorizer implementation.
[ https://issues.apache.org/jira/browse/KAFKA-2210?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Brahmbhatt updated KAFKA-2210: Attachment: KAFKA-2210_2015-08-26_14:29:02.patch KafkaAuthorizer: Add all public entities, config changes and changes to KafkaAPI and kafkaServer to allow pluggable authorizer implementation. -- Key: KAFKA-2210 URL: https://issues.apache.org/jira/browse/KAFKA-2210 Project: Kafka Issue Type: Sub-task Components: security Reporter: Parth Brahmbhatt Assignee: Parth Brahmbhatt Fix For: 0.8.3 Attachments: KAFKA-2210.patch, KAFKA-2210_2015-06-03_16:36:11.patch, KAFKA-2210_2015-06-04_16:07:39.patch, KAFKA-2210_2015-07-09_18:00:34.patch, KAFKA-2210_2015-07-14_10:02:19.patch, KAFKA-2210_2015-07-14_14:13:19.patch, KAFKA-2210_2015-07-20_16:42:18.patch, KAFKA-2210_2015-07-21_17:08:21.patch, KAFKA-2210_2015-08-10_18:31:54.patch, KAFKA-2210_2015-08-20_11:27:18.patch, KAFKA-2210_2015-08-25_17:59:22.patch, KAFKA-2210_2015-08-26_14:29:02.patch This is the first subtask for Kafka-1688. As Part of this jira we intend to agree on all the public entities, configs and changes to existing kafka classes to allow pluggable authorizer implementation. Please see KIP-11 https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization+Interface for detailed design. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 34492: Patch for KAFKA-2210
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/34492/ --- (Updated Aug. 26, 2015, 9:29 p.m.) Review request for kafka. Bugs: KAFKA-2210 https://issues.apache.org/jira/browse/KAFKA-2210 Repository: kafka Description (updated) --- Addressing review comments from Jun. Adding CREATE check for offset topic only if the topic does not exist already. Addressing some more comments. Removing acl.json file Moving PermissionType to trait instead of enum. Following the convention for defining constants. Adding authorizer.config.path back. Addressing more comments from Jun. Addressing more comments. Now addressing Ismael's comments. Case sensitive checks. Addressing Jun's comments. Merge remote-tracking branch 'origin/trunk' into az Conflicts: core/src/main/scala/kafka/server/KafkaApis.scala core/src/main/scala/kafka/server/KafkaServer.scala Deleting KafkaConfigDefTest Addressing comments from Ismael. Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into az Consolidating KafkaPrincipal. Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into az Conflicts: clients/src/main/java/org/apache/kafka/common/network/PlaintextTransportLayer.java clients/src/main/java/org/apache/kafka/common/security/auth/KafkaPrincipal.java core/src/main/scala/kafka/server/KafkaApis.scala Diffs (updated) - clients/src/main/java/org/apache/kafka/common/network/PlaintextTransportLayer.java 35d41685dd178bbdf77b2476e03ad51fc4adcbb6 clients/src/main/java/org/apache/kafka/common/security/auth/KafkaPrincipal.java b640ea0f4bdb694fc5524ef594aa125cc1ba4cf3 clients/src/test/java/org/apache/kafka/common/security/auth/KafkaPrincipalTest.java PRE-CREATION core/src/main/scala/kafka/api/OffsetRequest.scala f418868046f7c99aefdccd9956541a0cb72b1500 core/src/main/scala/kafka/common/AuthorizationException.scala PRE-CREATION core/src/main/scala/kafka/common/ErrorMapping.scala c75c68589681b2c9d6eba2b440ce5e58cddf6370 core/src/main/scala/kafka/security/auth/Acl.scala PRE-CREATION core/src/main/scala/kafka/security/auth/Authorizer.scala PRE-CREATION core/src/main/scala/kafka/security/auth/Operation.scala PRE-CREATION core/src/main/scala/kafka/security/auth/PermissionType.scala PRE-CREATION core/src/main/scala/kafka/security/auth/Resource.scala PRE-CREATION core/src/main/scala/kafka/security/auth/ResourceType.scala PRE-CREATION core/src/main/scala/kafka/server/KafkaApis.scala a3a8df0545c3f9390e0e04b8d2fab0134f5fd019 core/src/main/scala/kafka/server/KafkaConfig.scala d547a01cf7098f216a3775e1e1901c5794e1b24c core/src/main/scala/kafka/server/KafkaServer.scala 17db4fa3c3a146f03a35dd7671ad1b06d122bb59 core/src/test/scala/unit/kafka/security/auth/AclTest.scala PRE-CREATION core/src/test/scala/unit/kafka/security/auth/OperationTest.scala PRE-CREATION core/src/test/scala/unit/kafka/security/auth/PermissionTypeTest.scala PRE-CREATION core/src/test/scala/unit/kafka/security/auth/ResourceTypeTest.scala PRE-CREATION core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 3da666f73227fc7ef7093e3790546344065f6825 Diff: https://reviews.apache.org/r/34492/diff/ Testing --- Thanks, Parth Brahmbhatt
[jira] [Comment Edited] (KAFKA-2389) CommitType seems not necessary in commit().
[ https://issues.apache.org/jira/browse/KAFKA-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715565#comment-14715565 ] Guozhang Wang edited comment on KAFKA-2389 at 8/26/15 9:39 PM: --- I'm late on this discussion, but here are my two cents: [~jkreps] 1. We need to rename the async commits to commitAsync or something similar, you can't have two methods with the same name that behave totally differently and have different post-conditions. I feel it is OK to have sync / async differentiated by the callback rather than by their names. For example ZooKeeper has a similar approach regarding sync / async APIs: http://zookeeper.apache.org/doc/r3.4.5/api/org/apache/zookeeper/ZooKeeper.html#exists(java.lang.String, org.apache.zookeeper.Watcher, org.apache.zookeeper.AsyncCallback.StatCallback, java.lang.Object) 2. We need to include variants of asyncCommit that don't take the callback. Having the user implement or discover a NoOpCallback to be able to use the api is not good. With commit(OffsetCommitCallback callback), users can just call commit(null) and do not need to implement a NoOpCallback, right? I am personally not favor of making commitSync / commitAsync function names. was (Author: guozhang): I'm late on this discussion, but here are my two cents: 1. We need to rename the async commits to commitAsync or something similar, you can't have two methods with the same name that behave totally differently and have different post-conditions. I feel it is OK to have sync / async differentiated by the callback rather than by their names. For example ZooKeeper has a similar approach regarding sync / async APIs: http://zookeeper.apache.org/doc/r3.4.5/api/org/apache/zookeeper/ZooKeeper.html#exists(java.lang.String, org.apache.zookeeper.Watcher, org.apache.zookeeper.AsyncCallback.StatCallback, java.lang.Object) 2. We need to include variants of asyncCommit that don't take the callback. Having the user implement or discover a NoOpCallback to be able to use the api is not good. With commit(OffsetCommitCallback callback), users can just call commit(null) and do not need to implement a NoOpCallback, right? I am personally not favor of making commitSync / commitAsync function names. CommitType seems not necessary in commit(). --- Key: KAFKA-2389 URL: https://issues.apache.org/jira/browse/KAFKA-2389 Project: Kafka Issue Type: Sub-task Reporter: Jiangjie Qin Assignee: Jiangjie Qin The CommitType does not seem to be necessary in for commit(), it can be inferred from whether user passed in a callback or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KAFKA-2072) Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module
[ https://issues.apache.org/jira/browse/KAFKA-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-2072. Resolution: Fixed Issue resolved by pull request 170 [https://github.com/apache/kafka/pull/170] Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module -- Key: KAFKA-2072 URL: https://issues.apache.org/jira/browse/KAFKA-2072 Project: Kafka Issue Type: Sub-task Reporter: Gwen Shapira Assignee: David Jacot Fix For: 0.8.3 Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2072) Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module
[ https://issues.apache.org/jira/browse/KAFKA-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715627#comment-14715627 ] ASF GitHub Bot commented on KAFKA-2072: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/170 Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module -- Key: KAFKA-2072 URL: https://issues.apache.org/jira/browse/KAFKA-2072 Project: Kafka Issue Type: Sub-task Reporter: Gwen Shapira Assignee: David Jacot Fix For: 0.8.3 Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2477) Replicas spuriously deleting all segments in partition
Håkon Hitland created KAFKA-2477: Summary: Replicas spuriously deleting all segments in partition Key: KAFKA-2477 URL: https://issues.apache.org/jira/browse/KAFKA-2477 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.1 Reporter: Håkon Hitland We're seeing some strange behaviour in brokers: a replica will sometimes schedule all segments in a partition for deletion, and then immediately start replicating them back, triggering our check for under-replicating topics. This happens on average a couple of times a week, for different brokers and topics. We have per-topic retention.ms and retention.bytes configuration, the topics where we've seen this happen are hitting the size limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (KAFKA-2478) KafkaConsumer javadoc example seems wrong
Dmitry Stratiychuk created KAFKA-2478: - Summary: KafkaConsumer javadoc example seems wrong Key: KAFKA-2478 URL: https://issues.apache.org/jira/browse/KAFKA-2478 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8.3 Reporter: Dmitry Stratiychuk Assignee: Neha Narkhede I was looking at this KafkaConsumer example in the javadoc: https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L199 As I understand, commit() method commits the maximum offsets returned by the most recent invocation of poll() method. In this example, there's a danger of losing the data. Imagine the case where 300 records are returned by consumer.poll() The commit will happen after inserting 200 records into the database. But it will also commit the offsets for 100 records that are still unprocessed. So if consumer fails before buffer is dumped into the database again, then those 100 records will never be processed. If I'm wrong, could you please clarify the behaviour of commit() method? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: KAFKA-2364 migrate docs from SVN to git
Hi Kumar, One note: we need to update the documentation on how to submit changes to the website here: http://kafka.apache.org/contributing.html Ismael On Wed, Aug 26, 2015 at 3:13 PM, Manikumar Reddy ku...@nmsworks.co.in wrote: Hi Guozhang, Our plan is to follow Gwen's suggested approach and migrate the existing svn site repo to new git repo. (1) Gwen's suggestion will help to us maintain latest docs in Kafka repo itself. We periodically need to copy these latest docs to site repo. I will submit patch for this. (2) svn repo - git repo migration will help us to integrate site repo to git tooling/github. It will be easy to maintain the site repo and changes. So we have created new git repo for docs and need committer help to create a branch asf-site. new git repo: https://git-wip-us.apache.org/repos/asf/kafka-site.git Hope this clears the confusion. Kumar I thought Gwen's suggestion was to us a separate folder in the same repo for docs instead of a separate branch, Gwen can correct me if I was wrong? Guozhang On Mon, Aug 24, 2015 at 10:31 AM, Manikumar Reddy ku...@nmsworks.co.in wrote: Hi, Infra team created git repo for kafka site docs. Gwen/Guozhang, Need your help to create a branch asf-site and copy the exiting svn contents to that branch. git repo: https://git-wip-us.apache.org/repos/asf/kafka-site.git https://issues.apache.org/jira/browse/INFRA-10143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14709630#comment-14709630 Kumar On Fri, Aug 21, 2015 at 6:16 PM, Ismael Juma ism...@juma.me.uk wrote: My preference would be to do `2` because it reduces the number of tools we need to know. If we want to clone the repo for the generated site, we can use the same tools as we do for the code repo and we can watch for changes on GitHub, etc. Ismael On Fri, Aug 21, 2015 at 1:34 PM, Manikumar Reddy ku...@nmsworks.co.in wrote: Hi All, Can we finalize the approach? So that we can proceed further. 1. Gwen's suggestion + existing svn repo 2. Gwen's suggestion + new git repo for docs kumar On Thu, Aug 20, 2015 at 11:48 PM, Manikumar Reddy ku...@nmsworks.co.in wrote: Also can we migrate svn repo to git repo? This will help us to fix occasional doc changes/bug fixes through github PR. On Thu, Aug 20, 2015 at 4:04 AM, Guozhang Wang wangg...@gmail.com wrote: Gwen: I remembered it wrong. We would not need another round of voting. On Wed, Aug 19, 2015 at 3:06 PM, Gwen Shapira g...@confluent.io wrote: Looking back at this thread, the +1 mention same repo, so I'm not sure a new vote is required. On Wed, Aug 19, 2015 at 3:00 PM, Guozhang Wang wangg...@gmail.com wrote: So I think we have two different approaches here. The original proposal from Aseem is to move website from SVN to a separate Git repo, and hence have separate commits on code / doc changes. For that we have accumulated enough binging +1s to move on; Gwen's proposal is to move website into the same repo under a different folder. If people feel they prefer this over the previous approach I would like to call for another round of voting. Guozhang On Wed, Aug 19, 2015 at 10:24 AM, Ashish paliwalash...@gmail.com wrote: +1 to what Gwen has suggested. This is what we follow in Flume. All the latest doc changes are in git, once ready you move changes to svn to update website. The only catch is, when you need to update specific changes to website outside release cycle, need to be a bit careful :) On Wed, Aug 19, 2015 at 9:06 AM, Gwen Shapira g...@confluent.io wrote: Yeah, so the way this works in few other projects I worked on is: * The code repo has a /docs directory with the latest revision of the docs (not multiple versions, just one that matches the latest state of code) * When you submit a patch that requires doc modification, you modify all relevant files in same patch and they get reviewed and committed together (ideally) * When we release, we copy the docs matching the release and commit to SVN website. We also do this occasionally to fix bugs in earlier docs. * Release artifacts include a copy of the docs Nice to have: * Docs are in Asciidoc and build generates the HTML. Asciidoc is easier to edit and review. I suggest a similar process for Kafka. On
Re: Issue when enabling SSL on broker
Hi, Harsha, I appreciate you very much for your response and the bash script you provided to generate the keystores works for me and solve the problem. I was wondering it was caused by the cipher suite differences between openjdk and oracle-jdk, anyway it is not that case. Finally I got it worked under both openjdk and oracle-jdk. Thanks, Samuel On Tue, Aug 25, 2015 at 9:55 PM, Sriharsha Chintalapani harsh...@fastmail.fm wrote: Hi, Turns out to be a bug in the instructions in the wiki . I fixed it can you please retry generating the truststore and keystore https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka . checkout this section All of the above steps in a bash script” to generate the keystores. Thanks, Harsha On August 25, 2015 at 8:56:24 PM, Sriharsha Chintalapani (ka...@harsha.io) wrote: Hi Xiang, Did you try following the instructions here https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka . Whats the output of openssl s_client and which version of java and OS are you using. Thanks, Harsha On August 25, 2015 at 8:42:18 PM, Xiang Zhou (Samuel) (zhou...@gmail.com) wrote: no cipher suites in common
[jira] [Commented] (KAFKA-1690) Add SSL support to Kafka Broker, Producer and Consumer
[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14713473#comment-14713473 ] Navjot commented on KAFKA-1690: --- Great [~harsha_ch] seems that it's time to recompile everything. I'll confirm post testing. Add SSL support to Kafka Broker, Producer and Consumer -- Key: KAFKA-1690 URL: https://issues.apache.org/jira/browse/KAFKA-1690 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein Assignee: Sriharsha Chintalapani Fix For: 0.8.3 Attachments: KAFKA-1690.patch, KAFKA-1690.patch, KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, KAFKA-1690_2015-05-21_10:37:08.patch, KAFKA-1690_2015-06-03_18:52:29.patch, KAFKA-1690_2015-06-23_13:18:20.patch, KAFKA-1690_2015-07-20_06:10:42.patch, KAFKA-1690_2015-07-20_11:59:57.patch, KAFKA-1690_2015-07-25_12:10:55.patch, KAFKA-1690_2015-08-16_20:41:02.patch, KAFKA-1690_2015-08-17_08:12:50.patch, KAFKA-1690_2015-08-17_09:28:52.patch, KAFKA-1690_2015-08-17_12:20:53.patch, KAFKA-1690_2015-08-18_11:24:46.patch, KAFKA-1690_2015-08-18_17:24:48.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2470) kafka-producer-perf-test.sh can't configure all to request-num-acks
[ https://issues.apache.org/jira/browse/KAFKA-2470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bo Wang updated KAFKA-2470: --- Fix Version/s: 0.8.3 kafka-producer-perf-test.sh can't configure all to request-num-acks --- Key: KAFKA-2470 URL: https://issues.apache.org/jira/browse/KAFKA-2470 Project: Kafka Issue Type: Bug Components: clients, tools Affects Versions: 0.8.2.1 Environment: Linux Reporter: Bo Wang Fix For: 0.8.3 Original Estimate: 24h Remaining Estimate: 24h For New Producer API, kafka-producer-perf-test.sh can't configure all to request-num-acks : bin]# ./kafka-producer-perf-test.sh --topic test --broker-list host:port --messages 100 --message-size 200 --new-producer --sync --batch-size 1 --request-num-acks all Exception in thread main joptsimple.OptionArgumentConversionException: Cannot convert argument 'all' of option ['request-num-acks'] to class java.lang.Integer at joptsimple.ArgumentAcceptingOptionSpec.convert(ArgumentAcceptingOptionSpec.java:237) at joptsimple.OptionSet.valuesOf(OptionSet.java:226) at joptsimple.OptionSet.valueOf(OptionSet.java:170) at kafka.tools.ProducerPerformance$ProducerPerfConfig.init(ProducerPerformance.scala:146) at kafka.tools.ProducerPerformance$.main(ProducerPerformance.scala:42) at kafka.tools.ProducerPerformance.main(ProducerPerformance.scala) Caused by: joptsimple.internal.ReflectionException: java.lang.NumberFormatException: For input string: all at joptsimple.internal.Reflection.reflectionException(Reflection.java:136) at joptsimple.internal.Reflection.invoke(Reflection.java:123) at joptsimple.internal.MethodInvokingValueConverter.convert(MethodInvokingValueConverter.java:48) at joptsimple.ArgumentAcceptingOptionSpec.convert(ArgumentAcceptingOptionSpec.java:234) ... 5 more -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2477) Replicas spuriously deleting all segments in partition
[ https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Håkon Hitland updated KAFKA-2477: - Attachment: kafka_log.txt Attached example log from a broker Replicas spuriously deleting all segments in partition -- Key: KAFKA-2477 URL: https://issues.apache.org/jira/browse/KAFKA-2477 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.1 Reporter: Håkon Hitland Attachments: kafka_log.txt We're seeing some strange behaviour in brokers: a replica will sometimes schedule all segments in a partition for deletion, and then immediately start replicating them back, triggering our check for under-replicating topics. This happens on average a couple of times a week, for different brokers and topics. We have per-topic retention.ms and retention.bytes configuration, the topics where we've seen this happen are hitting the size limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: KAFKA-2364 migrate docs from SVN to git
Hi Guozhang, Our plan is to follow Gwen's suggested approach and migrate the existing svn site repo to new git repo. (1) Gwen's suggestion will help to us maintain latest docs in Kafka repo itself. We periodically need to copy these latest docs to site repo. I will submit patch for this. (2) svn repo - git repo migration will help us to integrate site repo to git tooling/github. It will be easy to maintain the site repo and changes. So we have created new git repo for docs and need committer help to create a branch asf-site. new git repo: https://git-wip-us.apache.org/repos/asf/kafka-site.git Hope this clears the confusion. Kumar I thought Gwen's suggestion was to us a separate folder in the same repo for docs instead of a separate branch, Gwen can correct me if I was wrong? Guozhang On Mon, Aug 24, 2015 at 10:31 AM, Manikumar Reddy ku...@nmsworks.co.in wrote: Hi, Infra team created git repo for kafka site docs. Gwen/Guozhang, Need your help to create a branch asf-site and copy the exiting svn contents to that branch. git repo: https://git-wip-us.apache.org/repos/asf/kafka-site.git https://issues.apache.org/jira/browse/INFRA-10143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14709630#comment-14709630 Kumar On Fri, Aug 21, 2015 at 6:16 PM, Ismael Juma ism...@juma.me.uk wrote: My preference would be to do `2` because it reduces the number of tools we need to know. If we want to clone the repo for the generated site, we can use the same tools as we do for the code repo and we can watch for changes on GitHub, etc. Ismael On Fri, Aug 21, 2015 at 1:34 PM, Manikumar Reddy ku...@nmsworks.co.in wrote: Hi All, Can we finalize the approach? So that we can proceed further. 1. Gwen's suggestion + existing svn repo 2. Gwen's suggestion + new git repo for docs kumar On Thu, Aug 20, 2015 at 11:48 PM, Manikumar Reddy ku...@nmsworks.co.in wrote: Also can we migrate svn repo to git repo? This will help us to fix occasional doc changes/bug fixes through github PR. On Thu, Aug 20, 2015 at 4:04 AM, Guozhang Wang wangg...@gmail.com wrote: Gwen: I remembered it wrong. We would not need another round of voting. On Wed, Aug 19, 2015 at 3:06 PM, Gwen Shapira g...@confluent.io wrote: Looking back at this thread, the +1 mention same repo, so I'm not sure a new vote is required. On Wed, Aug 19, 2015 at 3:00 PM, Guozhang Wang wangg...@gmail.com wrote: So I think we have two different approaches here. The original proposal from Aseem is to move website from SVN to a separate Git repo, and hence have separate commits on code / doc changes. For that we have accumulated enough binging +1s to move on; Gwen's proposal is to move website into the same repo under a different folder. If people feel they prefer this over the previous approach I would like to call for another round of voting. Guozhang On Wed, Aug 19, 2015 at 10:24 AM, Ashish paliwalash...@gmail.com wrote: +1 to what Gwen has suggested. This is what we follow in Flume. All the latest doc changes are in git, once ready you move changes to svn to update website. The only catch is, when you need to update specific changes to website outside release cycle, need to be a bit careful :) On Wed, Aug 19, 2015 at 9:06 AM, Gwen Shapira g...@confluent.io wrote: Yeah, so the way this works in few other projects I worked on is: * The code repo has a /docs directory with the latest revision of the docs (not multiple versions, just one that matches the latest state of code) * When you submit a patch that requires doc modification, you modify all relevant files in same patch and they get reviewed and committed together (ideally) * When we release, we copy the docs matching the release and commit to SVN website. We also do this occasionally to fix bugs in earlier docs. * Release artifacts include a copy of the docs Nice to have: * Docs are in Asciidoc and build generates the HTML. Asciidoc is easier to edit and review. I suggest a similar process for Kafka. On Wed, Aug 19, 2015 at 8:53 AM, Ismael Juma ism...@juma.me.uk wrote: I should clarify: it's not possible unless we add an additional step that moves the docs from the code repo to the website repo. Ismael On Wed, Aug 19, 2015 at 4:42 PM, Ismael Juma ism...@juma.me.uk wrote: Hi all,
[jira] [Commented] (KAFKA-1451) Broker stuck due to leader election race
[ https://issues.apache.org/jira/browse/KAFKA-1451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14713405#comment-14713405 ] Jason Kania commented on KAFKA-1451: I too am seeing this issue in 0.8.2.1. Broker stuck due to leader election race - Key: KAFKA-1451 URL: https://issues.apache.org/jira/browse/KAFKA-1451 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Maciek Makowski Assignee: Manikumar Reddy Priority: Minor Labels: newbie Fix For: 0.8.2.0 Attachments: KAFKA-1451.patch, KAFKA-1451_2014-07-28_20:27:32.patch, KAFKA-1451_2014-07-29_10:13:23.patch h3. Symptoms The broker does not become available due to being stuck in an infinite loop while electing leader. This can be recognised by the following line being repeatedly written to server.log: {code} [2014-05-14 04:35:09,187] INFO I wrote this conflicted ephemeral node [{version:1,brokerid:1,timestamp:1400060079108}] at /controller a while back in a different session, hence I will backoff for this node to be deleted by Zookeeper and retry (kafka.utils.ZkUtils$) {code} h3. Steps to Reproduce In a single kafka 0.8.1.1 node, single zookeeper 3.4.6 (but will likely behave the same with the ZK version included in Kafka distribution) node setup: # start both zookeeper and kafka (in any order) # stop zookeeper # stop kafka # start kafka # start zookeeper h3. Likely Cause {{ZookeeperLeaderElector}} subscribes to data changes on startup, and then triggers an election. if the deletion of ephemeral {{/controller}} node associated with previous zookeeper session of the broker happens after subscription to changes in new session, election will be invoked twice, once from {{startup}} and once from {{handleDataDeleted}}: * {{startup}}: acquire {{controllerLock}} * {{startup}}: subscribe to data changes * zookeeper: delete {{/controller}} since the session that created it timed out * {{handleDataDeleted}}: {{/controller}} was deleted * {{handleDataDeleted}}: wait on {{controllerLock}} * {{startup}}: elect -- writes {{/controller}} * {{startup}}: release {{controllerLock}} * {{handleDataDeleted}}: acquire {{controllerLock}} * {{handleDataDeleted}}: elect -- attempts to write {{/controller}} and then gets into infinite loop as a result of conflict {{createEphemeralPathExpectConflictHandleZKBug}} assumes that the existing znode was written from different session, which is not true in this case; it was written from the same session. That adds to the confusion. h3. Suggested Fix In {{ZookeeperLeaderElector.startup}} first run {{elect}} and then subscribe to data changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Issue when enabling SSL on broker
Hi, folks, I am just following up to see if anybody has clue on the following issue when I was following the guidelines provided in the step of running openssl to verify the SSL setup: [2015-08-25 22:29:15,385] WARN Error in I/O with connection to X (org.apache.kafka.common.network.Selector) javax.net.ssl.SSLHandshakeException: no cipher suites in common at sun.security.ssl.Handshaker.checkThrown(Handshaker.java:1348) at sun.security.ssl.SSLEngineImpl.checkTaskThrown(SSLEngineImpl.java:519) at sun.security.ssl.SSLEngineImpl.writeAppRecord(SSLEngineImpl.java:1200) at sun.security.ssl.SSLEngineImpl.wrap(SSLEngineImpl.java:1172) at javax.net.ssl.SSLEngine.wrap(SSLEngine.java:469) at org.apache.kafka.common.network.SSLTransportLayer.handshakeWrap(SSLTransportLayer.java:345) at org.apache.kafka.common.network.SSLTransportLayer.handshake(SSLTransportLayer.java:222) at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:69) at org.apache.kafka.common.network.Selector.poll(Selector.java:290) at kafka.network.Processor.run(SocketServer.scala:393) at java.lang.Thread.run(Thread.java:745) Caused by: javax.net.ssl.SSLHandshakeException: no cipher suites in common at sun.security.ssl.Alerts.getSSLException(Alerts.java:192) at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1650) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:281) at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:269) at sun.security.ssl.ServerHandshaker.chooseCipherSuite(ServerHandshaker.java:970) at sun.security.ssl.ServerHandshaker.clientHello(ServerHandshaker.java:684) at sun.security.ssl.ServerHandshaker.processMessage(ServerHandshaker.java:222) at sun.security.ssl.Handshaker.processLoop(Handshaker.java:913) at sun.security.ssl.Handshaker$1.run(Handshaker.java:853) at sun.security.ssl.Handshaker$1.run(Handshaker.java:851) at java.security.AccessController.doPrivileged(Native Method) at sun.security.ssl.Handshaker$DelegatedTask.run(Handshaker.java:1285) at org.apache.kafka.common.network.SSLTransportLayer.runDelegatedTasks(SSLTransportLayer.java:303) at org.apache.kafka.common.network.SSLTransportLayer.handshakeUnwrap(SSLTransportLayer.java:381) at org.apache.kafka.common.network.SSLTransportLayer.handshake(SSLTransportLayer.java:246) ... 4 more Above exception was shown on broker side and there is no certificate shown up when verifying with openssl command. And when I was trying to connect broker by producer and consumer command, it got the same error message. I tried to let ssl.cipher.suites unset and set some values in it, neither way has luck. Appreciate any help. Thanks, Samuel On Fri, Aug 21, 2015 at 6:28 AM, Ben Stopford b...@confluent.io wrote: Hi Qi Trunk seems fairly stable. There are guidelines here which includes how to generate keys https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka Your server config needs these properties (also on the webpage): listeners=PLAINTEXT://:9092,SSL://:9093 ssl.protocol = TLS ssl.keystore.type = JKS ssl.keystore.location = path/keystore.jks ssl.keystore.password = pass ssl.key.password = pass ssl.truststore.type = JKS ssl.truststore.location = path/truststore.jks ssl.truststore.password = pass To get yourself going it’s easiest to just generate a set of certs locally and spark up the console producer/consumer pair. You’ll need the latest cut from trunk (from today) to get a console consumer that works. Hope that helps Ben On 21 Aug 2015, at 07:10, Qi Xu shkir...@gmail.com wrote: Hi folks, I tried to clone the latest version of kafka truck and try to enable the SSL. The server.properties seems not having any security related settings, and it seems there's no other config file relevant to SSL either. So may I know is this feature ready to use now in truck branch? BTW, we're using the SSL feature from the branch : https://github.com/relango/kafka/tree/0.8.2. Is there any significant difference between Kafka-truck and relango's branch? Thanks, Qi
[jira] [Commented] (KAFKA-1690) Add SSL support to Kafka Broker, Producer and Consumer
[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14713445#comment-14713445 ] Navjot commented on KAFKA-1690: --- [~harsha_ch] Finally I got some luck with Testing SSL Enabled Kafka Server but got stuck with the Consumer. I found one new confluence page written by you at [https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka] and followed the same. Now my Server and Producers are SSL enabled (Can see that by enabling trace) but when I run Consumer by using : {{kafka-console-consumer.bat --bootstrap-server localhost:9093 --topic test --new-consumer --consumer.config security.protocol=SSL --consumer.config ssl.truststore.location=client.truststore.jks --consumer.config ssl.truststore.password=test1234}} {{It fails with error: 'bootstrap-server' is not a recognized option}} Also Normal Consumer is running fine and is reading data from Producer (SSL enabled producer). Please suggest what might be wrong here. Add SSL support to Kafka Broker, Producer and Consumer -- Key: KAFKA-1690 URL: https://issues.apache.org/jira/browse/KAFKA-1690 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein Assignee: Sriharsha Chintalapani Fix For: 0.8.3 Attachments: KAFKA-1690.patch, KAFKA-1690.patch, KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, KAFKA-1690_2015-05-21_10:37:08.patch, KAFKA-1690_2015-06-03_18:52:29.patch, KAFKA-1690_2015-06-23_13:18:20.patch, KAFKA-1690_2015-07-20_06:10:42.patch, KAFKA-1690_2015-07-20_11:59:57.patch, KAFKA-1690_2015-07-25_12:10:55.patch, KAFKA-1690_2015-08-16_20:41:02.patch, KAFKA-1690_2015-08-17_08:12:50.patch, KAFKA-1690_2015-08-17_09:28:52.patch, KAFKA-1690_2015-08-17_12:20:53.patch, KAFKA-1690_2015-08-18_11:24:46.patch, KAFKA-1690_2015-08-18_17:24:48.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1690) Add SSL support to Kafka Broker, Producer and Consumer
[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14713464#comment-14713464 ] Sriharsha Chintalapani commented on KAFKA-1690: --- [~NavjotBhardwaj] are you using the latest trunk since that option added recently. Add SSL support to Kafka Broker, Producer and Consumer -- Key: KAFKA-1690 URL: https://issues.apache.org/jira/browse/KAFKA-1690 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein Assignee: Sriharsha Chintalapani Fix For: 0.8.3 Attachments: KAFKA-1690.patch, KAFKA-1690.patch, KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, KAFKA-1690_2015-05-21_10:37:08.patch, KAFKA-1690_2015-06-03_18:52:29.patch, KAFKA-1690_2015-06-23_13:18:20.patch, KAFKA-1690_2015-07-20_06:10:42.patch, KAFKA-1690_2015-07-20_11:59:57.patch, KAFKA-1690_2015-07-25_12:10:55.patch, KAFKA-1690_2015-08-16_20:41:02.patch, KAFKA-1690_2015-08-17_08:12:50.patch, KAFKA-1690_2015-08-17_09:28:52.patch, KAFKA-1690_2015-08-17_12:20:53.patch, KAFKA-1690_2015-08-18_11:24:46.patch, KAFKA-1690_2015-08-18_17:24:48.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2478) KafkaConsumer javadoc example seems wrong
[ https://issues.apache.org/jira/browse/KAFKA-2478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715459#comment-14715459 ] Jason Gustafson commented on KAFKA-2478: [~devstr] Not that I'm aware of. You can control the maximum fetch size in configuration, but that only affects individual fetches and poll() can send out many of these. Do you want to submit a patch for this? KafkaConsumer javadoc example seems wrong - Key: KAFKA-2478 URL: https://issues.apache.org/jira/browse/KAFKA-2478 Project: Kafka Issue Type: Bug Components: consumer Affects Versions: 0.8.3 Reporter: Dmitry Stratiychuk Assignee: Neha Narkhede I was looking at this KafkaConsumer example in the javadoc: https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java#L199 As I understand, commit() method commits the maximum offsets returned by the most recent invocation of poll() method. In this example, there's a danger of losing the data. Imagine the case where 300 records are returned by consumer.poll() The commit will happen after inserting 200 records into the database. But it will also commit the offsets for 100 records that are still unprocessed. So if consumer fails before buffer is dumped into the database again, then those 100 records will never be processed. If I'm wrong, could you please clarify the behaviour of commit() method? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] kafka pull request: KAFKA-1683: persisting session information in ...
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/155 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (KAFKA-1977) Make logEndOffset available in the Zookeeper consumer
[ https://issues.apache.org/jira/browse/KAFKA-1977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715470#comment-14715470 ] Jason Gustafson commented on KAFKA-1977: Seems like there should be some options for exposing it in the ConsumerRecords object that is returned in the new consumer's poll. [~willf] Do you want to create a jira for that? Make logEndOffset available in the Zookeeper consumer - Key: KAFKA-1977 URL: https://issues.apache.org/jira/browse/KAFKA-1977 Project: Kafka Issue Type: Improvement Components: core Reporter: Will Funnell Priority: Minor Attachments: Make_logEndOffset_available_in_the_Zookeeper_consumer.patch The requirement is to create a snapshot from the Kafka topic but NOT do continual reads after that point. For example you might be creating a backup of the data to a file. In order to achieve that, a recommended solution by Joel Koshy and Jay Kreps was to expose the high watermark, as maxEndOffset, from the FetchResponse object through to each MessageAndMetadata object in order to be aware when the consumer has reached the end of each partition. The submitted patch achieves this by adding the maxEndOffset to the PartitionTopicInfo, which is updated when a new message arrives in the ConsumerFetcherThread and then exposed in MessageAndMetadata. See here for discussion: http://search-hadoop.com/m/4TaT4TpJy71 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2390) OffsetOutOfRangeException should contain the Offset and Partition info.
[ https://issues.apache.org/jira/browse/KAFKA-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-2390: Description: Currently when seek() finishes, the offset seek to is not verified and OffsetOutOfRangeException. (was: Currently seek is an async call. To have the same interface as other calls like commit(), seek() should take a callback. This callback will be invoked if the position to seek triggers OFFSET_OUT_OF_RANGE exception from broker.) OffsetOutOfRangeException should contain the Offset and Partition info. --- Key: KAFKA-2390 URL: https://issues.apache.org/jira/browse/KAFKA-2390 Project: Kafka Issue Type: Sub-task Reporter: Jiangjie Qin Assignee: Dong Lin Currently when seek() finishes, the offset seek to is not verified and OffsetOutOfRangeException. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2390) OffsetOutOfRangeException should contain the Offset and Partition info.
[ https://issues.apache.org/jira/browse/KAFKA-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-2390: Description: Currently when seek() finishes, the offset seek to is not verified and OffsetOutOfRangeException might be thrown later. To let the users take actions when the OffsetOutOfRangeException is thrown, we need to provide more information in the Exception. (was: Currently when seek() finishes, the offset seek to is not verified and OffsetOutOfRangeException. ) OffsetOutOfRangeException should contain the Offset and Partition info. --- Key: KAFKA-2390 URL: https://issues.apache.org/jira/browse/KAFKA-2390 Project: Kafka Issue Type: Sub-task Reporter: Jiangjie Qin Assignee: Dong Lin Currently when seek() finishes, the offset seek to is not verified and OffsetOutOfRangeException might be thrown later. To let the users take actions when the OffsetOutOfRangeException is thrown, we need to provide more information in the Exception. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-2477) Replicas spuriously deleting all segments in partition
[ https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715742#comment-14715742 ] Håkon Hitland edited comment on KAFKA-2477 at 8/26/15 11:38 PM: We use a replication factor of 3. The only line with start offset that day is the one in the attached log: [2015-08-24 18:32:32,299] WARN [ReplicaFetcherThread-3-0], Replica 3 for partition [log.event,3] reset its fetch offset from 10200597616 to current leader 0's start offset 10200597616 (kafka.server.ReplicaFetcherThread) e: the leader error reads: [2015-08-24 18:32:32,145] ERROR [Replica Manager on Broker 0]: Error when processing fetch request for partition [log.event,3] offset 10349592111 from follower with correlation id 141609587. Possible cause: Request for offset 10349592111 but we only have log segments in the range 10200597616 to 10349592109. (kafka.server.ReplicaManager) was (Author: hakon): We use a replication factor of 3. The only line with start offset that day is the one in the attached log: [2015-08-24 18:32:32,299] WARN [ReplicaFetcherThread-3-0], Replica 3 for partition [log.event,3] reset its fetch offset from 10200597616 to current leader 0's start offset 10200597616 (kafka.server.ReplicaFetcherThread) Replicas spuriously deleting all segments in partition -- Key: KAFKA-2477 URL: https://issues.apache.org/jira/browse/KAFKA-2477 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.1 Reporter: Håkon Hitland Attachments: kafka_log.txt We're seeing some strange behaviour in brokers: a replica will sometimes schedule all segments in a partition for deletion, and then immediately start replicating them back, triggering our check for under-replicating topics. This happens on average a couple of times a week, for different brokers and topics. We have per-topic retention.ms and retention.bytes configuration, the topics where we've seen this happen are hitting the size limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2388) subscribe(topic)/unsubscribe(topic) should either take a callback to allow user to handle exceptions or it should be synchronous.
[ https://issues.apache.org/jira/browse/KAFKA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715798#comment-14715798 ] ASF GitHub Bot commented on KAFKA-2388: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/139 subscribe(topic)/unsubscribe(topic) should either take a callback to allow user to handle exceptions or it should be synchronous. - Key: KAFKA-2388 URL: https://issues.apache.org/jira/browse/KAFKA-2388 Project: Kafka Issue Type: Sub-task Reporter: Jiangjie Qin Assignee: Jason Gustafson Fix For: 0.8.3 According to the mailing list discussion on the consumer interface, we'll replace: {code} public void subscribe(String... topics); public void subscribe(TopicPartition... partitions); public SetTopicPartition subscriptions(); {code} with: {code} void subscribe(ListString topics, RebalanceCallback callback); void assign(ListTopicPartition partitions); ListString subscriptions(); ListTopicPartition assignments(); {code} We don't need the unsubscribe APIs anymore. The RebalanceCallback would look like: {code} interface RebalanceCallback { void onAssignment(ListTopicPartition partitions); void onRevocation(ListTopicPartition partitions); // handle non-existing topics, etc. void onError(Exception e); } {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (KAFKA-2388) subscribe(topic)/unsubscribe(topic) should either take a callback to allow user to handle exceptions or it should be synchronous.
[ https://issues.apache.org/jira/browse/KAFKA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-2388. -- Resolution: Fixed Fix Version/s: 0.8.3 Issue resolved by pull request 139 [https://github.com/apache/kafka/pull/139] subscribe(topic)/unsubscribe(topic) should either take a callback to allow user to handle exceptions or it should be synchronous. - Key: KAFKA-2388 URL: https://issues.apache.org/jira/browse/KAFKA-2388 Project: Kafka Issue Type: Sub-task Reporter: Jiangjie Qin Assignee: Jason Gustafson Fix For: 0.8.3 According to the mailing list discussion on the consumer interface, we'll replace: {code} public void subscribe(String... topics); public void subscribe(TopicPartition... partitions); public SetTopicPartition subscriptions(); {code} with: {code} void subscribe(ListString topics, RebalanceCallback callback); void assign(ListTopicPartition partitions); ListString subscriptions(); ListTopicPartition assignments(); {code} We don't need the unsubscribe APIs anymore. The RebalanceCallback would look like: {code} interface RebalanceCallback { void onAssignment(ListTopicPartition partitions); void onRevocation(ListTopicPartition partitions); // handle non-existing topics, etc. void onError(Exception e); } {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2467) ConsoleConsumer regressions
[ https://issues.apache.org/jira/browse/KAFKA-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2467: - Status: Patch Available (was: Open) [~gwenshap] or [~guozhang] Can one of you review and commit this? It's currently breaking system tests. ConsoleConsumer regressions --- Key: KAFKA-2467 URL: https://issues.apache.org/jira/browse/KAFKA-2467 Project: Kafka Issue Type: Bug Components: tools Reporter: Ewen Cheslack-Postava Assignee: Ewen Cheslack-Postava It seems that the patch for KAFKA-2015 caused a few changes in the behavior of the console consumer. I picked this up because it caused the new mirror maker sanity system test to hang. We need a separate fix for ducktape to address the lack of a timeout where it got stuck, but I'd also like to get this fixed ASAP since it affects pretty much all system test efforts since they commonly use console consumer to validate data produced to Kafka. I've tracked down a couple of changes so far: 1. The --consumer.config option handling was changed entirely. I think the new approach was trying to parse it as key=value parameters, but it's supposed to be a properties file *containing* key=value pairs. 2. A few different exceptions during message processing are not handled the same way. The skipMessageOnErrorOpt is not longer being used at all (it's parsed, but that option is never checked anymore). Also, exceptions during iteration are not caught. After fixing the consumer.config issue, which was keeping the consumer.timeout.ms setting from making it into the consumer config, this also caused the process to hang. It killed the main thread, but there must be another non-daemon thread still running (presumably the consumer threads?) 3. The consumed X messages message changed from stderr to stdout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2467) ConsoleConsumer regressions
[ https://issues.apache.org/jira/browse/KAFKA-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715879#comment-14715879 ] Guozhang Wang commented on KAFKA-2467: -- I should be the guy apologizing here.. My original patch on KAFKA-2015 was not complete but only sufficient for some of my local validations long time ago which is essentially a very small subset of the old system test. When [~benstopford] picked it up I did not clearly mention it to him. Sorry about that. ConsoleConsumer regressions --- Key: KAFKA-2467 URL: https://issues.apache.org/jira/browse/KAFKA-2467 Project: Kafka Issue Type: Bug Components: tools Reporter: Ewen Cheslack-Postava Assignee: Ewen Cheslack-Postava It seems that the patch for KAFKA-2015 caused a few changes in the behavior of the console consumer. I picked this up because it caused the new mirror maker sanity system test to hang. We need a separate fix for ducktape to address the lack of a timeout where it got stuck, but I'd also like to get this fixed ASAP since it affects pretty much all system test efforts since they commonly use console consumer to validate data produced to Kafka. I've tracked down a couple of changes so far: 1. The --consumer.config option handling was changed entirely. I think the new approach was trying to parse it as key=value parameters, but it's supposed to be a properties file *containing* key=value pairs. 2. A few different exceptions during message processing are not handled the same way. The skipMessageOnErrorOpt is not longer being used at all (it's parsed, but that option is never checked anymore). Also, exceptions during iteration are not caught. After fixing the consumer.config issue, which was keeping the consumer.timeout.ms setting from making it into the consumer config, this also caused the process to hang. It killed the main thread, but there must be another non-daemon thread still running (presumably the consumer threads?) 3. The consumed X messages message changed from stderr to stdout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2067) Add LeaderAndISR request/response to org.apache.kafka.common.requests and replace usage in core module
[ https://issues.apache.org/jira/browse/KAFKA-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715785#comment-14715785 ] Ismael Juma commented on KAFKA-2067: Needed for KAFKA-2411 Add LeaderAndISR request/response to org.apache.kafka.common.requests and replace usage in core module -- Key: KAFKA-2067 URL: https://issues.apache.org/jira/browse/KAFKA-2067 Project: Kafka Issue Type: Sub-task Reporter: Gwen Shapira Assignee: Ismael Juma Fix For: 0.8.3 Add LeaderAndISR request/response to org.apache.kafka.common.requests and replace usage in core module. Note that this will require adding a bunch of new objects to o.a.k.common - LeaderAndISR, LeaderISRAndEpoch and possibly others. It may be nice to have a scala implicit to translate those objects from their old (core) implementation to the o.a.k.common implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] Client-side Assignment for New Consumer
Hi folks, After further discussion in LinkedIn, we found that while having a more general group management protocol is very useful, the vast majority of the clients will not use customized partition assignment strategy. In that case, the broker side partition assignment would be ideal because it avoids issues like metadata inconsistency / split brain / exploding subscription set propagation. So we have the following proposal that satisfies the majority of the clients' needs without changing the currently proposed binary protocol. i.e., Continue to support broker-side assignment if the assignment strategy is recognized by the coordinator. 1. Keep the binary protocol as currently proposed. 2. Change the way we interpret ProtocolMetadata: 2.1 On consumer side, change partition.assignment.strategy to partition.assignor.class. Implement the something like the following PartitionAssignor Interface: public interface PartitionAssignor { ListString protocolTypes(); byte[] protocolMetadata(); // return the Topic-ListPartition map that are assigned to this consumer. ListTopicPartition assignPartitions(String protocolType, byte[] responseProtocolMetadata); } public abstract class AbstractPartitionAssignor implements PartitionAssignor { protected final KafkaConsumer consumer; AbstractPartitionAssignor(KafkaConsumer consumer) { this.consumer = consumer; } } 2.2 The ProtocolMetadata in JoinGroupRequest will be partitionAssignor.protocolMetadata(). When partition.assignor.class is range or roundrobin, the ProtocolMetadata in JoinGroupRequest will be a JSON subscription set. (range, roundrobin will be reserved words, we can also consider reserving some Prefix such as broker- to be more clear) 2.3 On broker side when ProtocolType is range or roundroubin, coordinator will parse the ProtocolMetadata in the JoinGroupRequest and assign the partitions for consumers. In the JoinGroupResponse, the ProtocolMetadata will be the global assignment of partitions. 2.4 On client side, after receiving the JoinGroupResponse, partitionAssignor.assignPartitions() will be invoked to return the actual assignment. If the assignor is RangeAssignor or RoundRobinAssignor, they will parse the assignment from the ProtocolMetadata returned by coordinator. This approach has a few merits: 1. Does not change the proposed binary protocol, which is still general. 2. The majority of the consumers will not suffer from inconsistent metadata / split brain / exploding subscription set propagation. This is specifically to deal with the issue that the current proposal caters to a 20% use-case while adversely impacting the more common 80% use-cases. 3. Easy to implement. The only thing needed is implement a partitioner class. For most users, the default range and roundrobin partitioner are good enough. Thoughts? Thanks, Jiangjie (Becket) Qin On Tue, Aug 18, 2015 at 2:51 PM, Jason Gustafson ja...@confluent.io wrote: Follow-up from the kip call: 1. Onur brought up the question of whether this protocol provides enough coordination capabilities to be generally useful in practice (is that accurate, Onur?). If it doesn't, then each use case would probably need a dependence on zookeeper anyway, and we haven't really gained anything. The group membership provided by this protocol is a useful primitive for coordination, but it's limited in the sense that everything shared among the group has to be communicated at the time the group is created. If any shared data changes, then the only way the group can ensure agreement is to force a rebalance. This is expensive since all members must stall while the rebalancing takes place. As we have also seen, there is a practical limit on the amount of metadata that can be sent through this protocol when groups get a little larger. This protocol is therefore not suitable to cases which require frequent communication or which require a large amount of data to be communicated. For the use cases listed on the wiki, neither of these appear to be an issue, but there may be other limitations which would limit reuse of the protocol. Perhaps it would be sufficient to sketch how these cases might work? 2. We talked a little bit about the issue of metadata churn. Becket brought up the interesting point that not only do we depend on topic metadata changing relatively infrequently, but we also expect timely agreement among the brokers on what that metadata is. To resolve this, we can have the consumers fetch metadata from the coordinator. We still depend on topic metadata not changing frequently, but this should resolve any disagreement among the brokers themselves. In fact, since we expect that disagreement is relatively rare, we can have the consumers fetch from the coordinator only when when a disagreement occurs. The nice thing about this proposal is that it doesn't affect the join group semantics, so the coordinator would remain oblivious to the metadata used by the group for
[jira] [Comment Edited] (KAFKA-2065) Add ControlledShutdown to org.apache.kafka.common.requests and replace current use in core module
[ https://issues.apache.org/jira/browse/KAFKA-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715780#comment-14715780 ] Ismael Juma edited comment on KAFKA-2065 at 8/27/15 12:04 AM: -- I've done this as part of KAFKA-2411, I hope you don't mind. was (Author: ijuma): I've done this as part of KAFKA-2417, I hope you don't mind. Add ControlledShutdown to org.apache.kafka.common.requests and replace current use in core module -- Key: KAFKA-2065 URL: https://issues.apache.org/jira/browse/KAFKA-2065 Project: Kafka Issue Type: Sub-task Reporter: Gwen Shapira Assignee: Ismael Juma Fix For: 0.8.3 Add ControlledShutdown to org.apache.kafka.common.requests and replace current use in core module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2065) Add ControlledShutdown to org.apache.kafka.common.requests and replace current use in core module
[ https://issues.apache.org/jira/browse/KAFKA-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715780#comment-14715780 ] Ismael Juma commented on KAFKA-2065: I've done this as part of KAFKA-2417, I hope you don't mind. Add ControlledShutdown to org.apache.kafka.common.requests and replace current use in core module -- Key: KAFKA-2065 URL: https://issues.apache.org/jira/browse/KAFKA-2065 Project: Kafka Issue Type: Sub-task Reporter: Gwen Shapira Assignee: Ismael Juma Fix For: 0.8.3 Add ControlledShutdown to org.apache.kafka.common.requests and replace current use in core module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (KAFKA-2065) Add ControlledShutdown to org.apache.kafka.common.requests and replace current use in core module
[ https://issues.apache.org/jira/browse/KAFKA-2065?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reassigned KAFKA-2065: -- Assignee: Ismael Juma (was: Ashish K Singh) Add ControlledShutdown to org.apache.kafka.common.requests and replace current use in core module -- Key: KAFKA-2065 URL: https://issues.apache.org/jira/browse/KAFKA-2065 Project: Kafka Issue Type: Sub-task Reporter: Gwen Shapira Assignee: Ismael Juma Fix For: 0.8.3 Add ControlledShutdown to org.apache.kafka.common.requests and replace current use in core module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2389) CommitType seems not necessary in commit().
[ https://issues.apache.org/jira/browse/KAFKA-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715805#comment-14715805 ] Ewen Cheslack-Postava commented on KAFKA-2389: -- I'd also say that just because the Zookeeper API has it doesn't make it good API design... I don't know about other people, but I raised the concern about naming again in the code review because I actually did get confused about what the semantics of each method signature was. CommitType seems not necessary in commit(). --- Key: KAFKA-2389 URL: https://issues.apache.org/jira/browse/KAFKA-2389 Project: Kafka Issue Type: Sub-task Reporter: Jiangjie Qin Assignee: Jiangjie Qin The CommitType does not seem to be necessary in for commit(), it can be inferred from whether user passed in a callback or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2389) CommitType seems not necessary in commit().
[ https://issues.apache.org/jira/browse/KAFKA-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715846#comment-14715846 ] Guozhang Wang commented on KAFKA-2389: -- Yeah, I did not think that Zookeeper API is the golden standard and we should just align with it, I guess I am just subjectively leaning towards the same function names for sync / async since for sync methods users can always put their callback logic after the commit() call hence they do not need a callback parameter, and wanted to use Zookeeper API as a backing point. Regarding [~becket_qin]'s point I agree that would be the issue. Anyways since this is pure personal taste I am willing to go with the majority vote. CommitType seems not necessary in commit(). --- Key: KAFKA-2389 URL: https://issues.apache.org/jira/browse/KAFKA-2389 Project: Kafka Issue Type: Sub-task Reporter: Jiangjie Qin Assignee: Jiangjie Qin The CommitType does not seem to be necessary in for commit(), it can be inferred from whether user passed in a callback or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Build failed in Jenkins: Kafka-trunk #602
See https://builds.apache.org/job/Kafka-trunk/602/changes Changes: [wangguoz] KAFKA-2388: refactor KafkaConsumer subscribe API -- [...truncated 1633 lines...] kafka.coordinator.ConsumerCoordinatorResponseTest testHeartbeatIllegalGeneration PASSED kafka.admin.DeleteConsumerGroupTest testTopicWideDeleteInZK PASSED kafka.coordinator.ConsumerCoordinatorResponseTest testHeartbeatDuringRebalanceCausesIllegalGeneration PASSED kafka.coordinator.ConsumerCoordinatorResponseTest testValidJoinGroup PASSED kafka.coordinator.ConsumerCoordinatorResponseTest testHeartbeatUnknownConsumerExistingGroup PASSED kafka.coordinator.ConsumerCoordinatorResponseTest testJoinGroupWrongCoordinator PASSED kafka.integration.TopicMetadataTest testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED kafka.admin.AdminTest testPartitionReassignmentWithLeaderNotInNewReplicas PASSED kafka.api.QuotasTest testProducerConsumerOverrideUnthrottled PASSED kafka.utils.timer.TimerTaskListTest testAll PASSED kafka.integration.TopicMetadataTest testBasicTopicMetadata PASSED kafka.api.ConsumerTest testPartitionPauseAndResume PASSED kafka.admin.DeleteTopicTest testRecreateTopicAfterDeletion PASSED kafka.integration.TopicMetadataTest testGetAllTopicMetadata PASSED kafka.api.ProducerBounceTest testBrokerFailure PASSED kafka.admin.AddPartitionsTest testTopicDoesNotExist PASSED kafka.integration.AutoOffsetResetTest testResetToLatestWhenOffsetTooLow PASSED kafka.admin.DeleteConsumerGroupTest testConsumptionOnRecreatedTopicAfterTopicWideDeleteInZK PASSED kafka.api.ProducerSendTest testCloseWithZeroTimeoutFromSenderThread PASSED kafka.utils.SchedulerTest testMockSchedulerPeriodicTask PASSED kafka.utils.SchedulerTest testMockSchedulerNonPeriodicTask PASSED kafka.utils.SchedulerTest testNonPeriodicTask PASSED kafka.utils.SchedulerTest testPeriodicTask PASSED kafka.utils.SchedulerTest testReentrantTaskInMockScheduler PASSED kafka.utils.SchedulerTest testRestart PASSED kafka.utils.timer.TimerTest testTaskExpiration PASSED kafka.utils.timer.TimerTest testAlreadyExpiredTask PASSED kafka.integration.TopicMetadataTest testTopicMetadataRequest PASSED kafka.log.LogTest testCorruptLog PASSED kafka.integration.AutoOffsetResetTest testResetToEarliestWhenOffsetTooLow PASSED kafka.integration.FetcherTest testFetcher PASSED kafka.common.ConfigTest testInvalidClientIds PASSED kafka.common.ConfigTest testInvalidGroupIds PASSED kafka.log.LogTest testCleanShutdownFile PASSED kafka.log.LogTest testParseTopicPartitionName PASSED kafka.log.LogTest testParseTopicPartitionNameForEmptyName PASSED kafka.log.LogTest testParseTopicPartitionNameForNull PASSED kafka.log.LogTest testParseTopicPartitionNameForMissingSeparator PASSED kafka.log.LogTest testParseTopicPartitionNameForMissingTopic PASSED kafka.log.LogTest testParseTopicPartitionNameForMissingPartition PASSED kafka.admin.AdminTest testPartitionReassignmentNonOverlappingReplicas PASSED kafka.admin.DeleteTopicTest testDeleteNonExistingTopic PASSED kafka.log.LogCleanerIntegrationTest cleanerTest[0] PASSED kafka.api.ConsumerTest testPauseStateNotPreservedByRebalance PASSED kafka.integration.AutoOffsetResetTest testResetToLatestWhenOffsetTooHigh PASSED kafka.api.ProducerFailureHandlingTest testTooLargeRecordWithAckZero PASSED kafka.admin.AdminTest testReassigningNonExistingPartition PASSED kafka.integration.AutoOffsetResetTest testResetToEarliestWhenOffsetTooHigh PASSED kafka.admin.DeleteTopicTest testResumeDeleteTopicWithRecoveredFollower PASSED kafka.admin.AddPartitionsTest testReplicaPlacement PASSED kafka.integration.UncleanLeaderElectionTest testUncleanLeaderElectionInvalidTopicOverride PASSED kafka.api.ProducerFailureHandlingTest testTooLargeRecordWithAckOne PASSED kafka.api.ProducerFailureHandlingTest testNonExistentTopic PASSED kafka.api.ConsumerBounceTest testConsumptionWithBrokerFailures PASSED kafka.utils.IteratorTemplateTest testIterator PASSED kafka.integration.RollingBounceTest testRollingBounce PASSED kafka.common.TopicTest testTopicHasCollisionChars PASSED kafka.common.TopicTest testInvalidTopicNames PASSED kafka.common.TopicTest testTopicHasCollision PASSED kafka.api.ConsumerTest testSimpleConsumption PASSED kafka.integration.UncleanLeaderElectionTest testUncleanLeaderElectionEnabledByTopicOverride PASSED kafka.api.ProducerFailureHandlingTest testWrongBrokerList PASSED kafka.api.ConsumerTest testCommitSpecifiedOffsets PASSED kafka.log.LogCleanerIntegrationTest cleanerTest[1] PASSED kafka.api.ProducerFailureHandlingTest testNoResponse PASSED kafka.admin.DeleteTopicTest testDeleteTopicWithCleaner PASSED kafka.integration.UncleanLeaderElectionTest testUncleanLeaderElectionEnabled PASSED kafka.api.ConsumerTest testAutoOffsetReset PASSED kafka.api.ProducerFailureHandlingTest testInvalidPartition PASSED
Build failed in Jenkins: Kafka-trunk #601
See https://builds.apache.org/job/Kafka-trunk/601/changes Changes: [junrao] KAFKA-2072: Add StopReplica request/response to o.a.k.common.requests -- [...truncated 2936 lines...] kafka.api.ProducerSendTest testCloseWithZeroTimeoutFromCallerThread PASSED kafka.integration.TopicMetadataTest testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED kafka.coordinator.ConsumerCoordinatorResponseTest testJoinGroupWrongCoordinator PASSED kafka.coordinator.ConsumerCoordinatorResponseTest testGenerationIdIncrementsOnRebalance PASSED kafka.admin.DeleteTopicTest testResumeDeleteTopicOnControllerFailover PASSED kafka.api.ConsumerTest testPartitionPauseAndResume PASSED kafka.coordinator.ConsumerCoordinatorResponseTest testValidHeartbeat PASSED kafka.coordinator.ConsumerCoordinatorResponseTest testJoinGroupSessionTimeoutTooLarge PASSED kafka.coordinator.ConsumerCoordinatorResponseTest testJoinGroupSessionTimeoutTooSmall PASSED kafka.coordinator.ConsumerCoordinatorResponseTest testHeartbeatDuringRebalanceCausesIllegalGeneration PASSED kafka.coordinator.ConsumerCoordinatorResponseTest testJoinGroupUnknownPartitionAssignmentStrategy PASSED kafka.admin.DeleteConsumerGroupTest testTopicWideDeleteInZK PASSED kafka.integration.TopicMetadataTest testTopicMetadataRequest PASSED kafka.api.ProducerBounceTest testBrokerFailure PASSED kafka.utils.timer.TimerTest testTaskExpiration PASSED kafka.utils.timer.TimerTest testAlreadyExpiredTask PASSED kafka.integration.TopicMetadataTest testAutoCreateTopicWithCollision PASSED kafka.integration.AutoOffsetResetTest testResetToEarliestWhenOffsetTooHigh PASSED kafka.admin.AddPartitionsTest testTopicDoesNotExist PASSED kafka.admin.AdminTest testBasicPreferredReplicaElection PASSED kafka.api.ConsumerTest testPauseStateNotPreservedByRebalance PASSED kafka.admin.DeleteTopicTest testPartitionReassignmentDuringDeleteTopic PASSED kafka.integration.AutoOffsetResetTest testResetToEarliestWhenOffsetTooLow PASSED kafka.admin.DeleteConsumerGroupTest testConsumptionOnRecreatedTopicAfterTopicWideDeleteInZK PASSED kafka.log.LogTest testCorruptLog PASSED kafka.admin.AdminTest testShutdownBroker PASSED kafka.log.LogTest testCleanShutdownFile PASSED kafka.log.LogTest testParseTopicPartitionName PASSED kafka.log.LogTest testParseTopicPartitionNameForEmptyName PASSED kafka.log.LogTest testParseTopicPartitionNameForNull PASSED kafka.log.LogTest testParseTopicPartitionNameForMissingSeparator PASSED kafka.log.LogTest testParseTopicPartitionNameForMissingTopic PASSED kafka.log.LogTest testParseTopicPartitionNameForMissingPartition PASSED kafka.integration.FetcherTest testFetcher PASSED kafka.common.ConfigTest testInvalidGroupIds PASSED kafka.common.ConfigTest testInvalidClientIds PASSED kafka.api.ProducerSendTest testCloseWithZeroTimeoutFromSenderThread PASSED kafka.utils.SchedulerTest testPeriodicTask PASSED kafka.utils.SchedulerTest testRestart PASSED kafka.utils.SchedulerTest testNonPeriodicTask PASSED kafka.utils.SchedulerTest testMockSchedulerNonPeriodicTask PASSED kafka.utils.SchedulerTest testMockSchedulerPeriodicTask PASSED kafka.utils.SchedulerTest testReentrantTaskInMockScheduler PASSED kafka.admin.DeleteTopicTest testDeleteNonExistingTopic PASSED kafka.log.LogCleanerIntegrationTest cleanerTest[0] PASSED kafka.admin.AddPartitionsTest testIncrementPartitions PASSED kafka.integration.AutoOffsetResetTest testResetToLatestWhenOffsetTooHigh PASSED kafka.api.ConsumerTest testPartitionsFor PASSED kafka.admin.AdminTest testTopicConfigChange PASSED kafka.api.ProducerFailureHandlingTest testTooLargeRecordWithAckZero PASSED kafka.integration.AutoOffsetResetTest testResetToLatestWhenOffsetTooLow PASSED kafka.api.ProducerFailureHandlingTest testTooLargeRecordWithAckOne PASSED kafka.api.ConsumerTest testListTopics PASSED kafka.api.ConsumerBounceTest testConsumptionWithBrokerFailures PASSED kafka.utils.IteratorTemplateTest testIterator PASSED kafka.api.ProducerFailureHandlingTest testNonExistentTopic PASSED kafka.integration.RollingBounceTest testRollingBounce PASSED kafka.common.TopicTest testInvalidTopicNames PASSED kafka.common.TopicTest testTopicHasCollisionChars PASSED kafka.common.TopicTest testTopicHasCollision PASSED kafka.api.ConsumerTest testPartitionReassignmentCallback PASSED kafka.api.ProducerFailureHandlingTest testWrongBrokerList PASSED kafka.admin.DeleteTopicTest testDeleteTopicWithCleaner PASSED kafka.api.ConsumerTest testUnsubscribeTopic PASSED kafka.integration.UncleanLeaderElectionTest testCleanLeaderElectionDisabledByTopicOverride PASSED kafka.log.LogCleanerIntegrationTest cleanerTest[1] PASSED kafka.admin.DeleteTopicTest testDeleteTopicAlreadyMarkedAsDeleted PASSED kafka.api.ProducerFailureHandlingTest testNoResponse PASSED kafka.integration.PrimitiveApiTest testProduceAndMultiFetch PASSED
[jira] [Commented] (KAFKA-2477) Replicas spuriously deleting all segments in partition
[ https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715742#comment-14715742 ] Håkon Hitland commented on KAFKA-2477: -- We use a replication factor of 3. The only line with start offset that day is the one in the attached log: [2015-08-24 18:32:32,299] WARN [ReplicaFetcherThread-3-0], Replica 3 for partition [log.event,3] reset its fetch offset from 10200597616 to current leader 0's start offset 10200597616 (kafka.server.ReplicaFetcherThread) Replicas spuriously deleting all segments in partition -- Key: KAFKA-2477 URL: https://issues.apache.org/jira/browse/KAFKA-2477 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.1 Reporter: Håkon Hitland Attachments: kafka_log.txt We're seeing some strange behaviour in brokers: a replica will sometimes schedule all segments in a partition for deletion, and then immediately start replicating them back, triggering our check for under-replicating topics. This happens on average a couple of times a week, for different brokers and topics. We have per-topic retention.ms and retention.bytes configuration, the topics where we've seen this happen are hitting the size limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (KAFKA-2067) Add LeaderAndISR request/response to org.apache.kafka.common.requests and replace usage in core module
[ https://issues.apache.org/jira/browse/KAFKA-2067?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reassigned KAFKA-2067: -- Assignee: Ismael Juma Add LeaderAndISR request/response to org.apache.kafka.common.requests and replace usage in core module -- Key: KAFKA-2067 URL: https://issues.apache.org/jira/browse/KAFKA-2067 Project: Kafka Issue Type: Sub-task Reporter: Gwen Shapira Assignee: Ismael Juma Fix For: 0.8.3 Add LeaderAndISR request/response to org.apache.kafka.common.requests and replace usage in core module. Note that this will require adding a bunch of new objects to o.a.k.common - LeaderAndISR, LeaderISRAndEpoch and possibly others. It may be nice to have a scala implicit to translate those objects from their old (core) implementation to the o.a.k.common implementation. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[GitHub] kafka pull request: KAFKA-2388 [WIP]; refactor KafkaConsumer subsc...
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/139 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
[jira] [Commented] (KAFKA-2477) Replicas spuriously deleting all segments in partition
[ https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715518#comment-14715518 ] Håkon Hitland commented on KAFKA-2477: -- Thanks for the reply. Checking the logs, we did get the Error when processing fetch request error in the leader mentioned in KAFKA-2143, so it could be the same issue. I don't see anything in our logs about a leader change, so I don't think it is caused by an unclean election, like some of the comments suggest. Replicas spuriously deleting all segments in partition -- Key: KAFKA-2477 URL: https://issues.apache.org/jira/browse/KAFKA-2477 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.1 Reporter: Håkon Hitland Attachments: kafka_log.txt We're seeing some strange behaviour in brokers: a replica will sometimes schedule all segments in a partition for deletion, and then immediately start replicating them back, triggering our check for under-replicating topics. This happens on average a couple of times a week, for different brokers and topics. We have per-topic retention.ms and retention.bytes configuration, the topics where we've seen this happen are hitting the size limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2389) CommitType seems not necessary in commit().
[ https://issues.apache.org/jira/browse/KAFKA-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715565#comment-14715565 ] Guozhang Wang commented on KAFKA-2389: -- I'm late on this discussion, but here are my two cents: 1. We need to rename the async commits to commitAsync or something similar, you can't have two methods with the same name that behave totally differently and have different post-conditions. I feel it is OK to have sync / async differentiated by the callback rather than by their names. For example ZooKeeper has a similar approach regarding sync / async APIs: http://zookeeper.apache.org/doc/r3.4.5/api/org/apache/zookeeper/ZooKeeper.html#exists(java.lang.String, org.apache.zookeeper.Watcher, org.apache.zookeeper.AsyncCallback.StatCallback, java.lang.Object) 2. We need to include variants of asyncCommit that don't take the callback. Having the user implement or discover a NoOpCallback to be able to use the api is not good. With commit(OffsetCommitCallback callback), users can just call commit(null) and do not need to implement a NoOpCallback, right? I am personally not favor of making commitSync / commitAsync function names. CommitType seems not necessary in commit(). --- Key: KAFKA-2389 URL: https://issues.apache.org/jira/browse/KAFKA-2389 Project: Kafka Issue Type: Sub-task Reporter: Jiangjie Qin Assignee: Jiangjie Qin The CommitType does not seem to be necessary in for commit(), it can be inferred from whether user passed in a callback or not. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2477) Replicas spuriously deleting all segments in partition
[ https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715604#comment-14715604 ] Jiangjie Qin commented on KAFKA-2477: - What is the partition replication factor? Replicas spuriously deleting all segments in partition -- Key: KAFKA-2477 URL: https://issues.apache.org/jira/browse/KAFKA-2477 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.1 Reporter: Håkon Hitland Attachments: kafka_log.txt We're seeing some strange behaviour in brokers: a replica will sometimes schedule all segments in a partition for deletion, and then immediately start replicating them back, triggering our check for under-replicating topics. This happens on average a couple of times a week, for different brokers and topics. We have per-topic retention.ms and retention.bytes configuration, the topics where we've seen this happen are hitting the size limit. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (KAFKA-2072) Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module
[ https://issues.apache.org/jira/browse/KAFKA-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-2072 started by David Jacot. -- Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module -- Key: KAFKA-2072 URL: https://issues.apache.org/jira/browse/KAFKA-2072 Project: Kafka Issue Type: Sub-task Reporter: Gwen Shapira Assignee: David Jacot Fix For: 0.8.3 Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (KAFKA-2072) Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module
[ https://issues.apache.org/jira/browse/KAFKA-2072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David Jacot reopened KAFKA-2072: Reopen this JIRA as https://github.com/apache/kafka/pull/170 adds StopReplica request/response to o.a.k.common.requests (required for KAFKA-2411) but doesn't replace the usage in core module. The replacement will be addressed in a second PR once KAFKA-2411 gets in. Is it ok like this? Or should it be addressed in follow up JIRA? Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module -- Key: KAFKA-2072 URL: https://issues.apache.org/jira/browse/KAFKA-2072 Project: Kafka Issue Type: Sub-task Reporter: Gwen Shapira Assignee: David Jacot Fix For: 0.8.3 Add StopReplica request/response to o.a.k.common.requests and replace the usage in core module -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1690) Add SSL support to Kafka Broker, Producer and Consumer
[ https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14713667#comment-14713667 ] Navjot commented on KAFKA-1690: --- And it started working. I'll now try to do it via API. Thanks for all your help [~harsha_ch] Add SSL support to Kafka Broker, Producer and Consumer -- Key: KAFKA-1690 URL: https://issues.apache.org/jira/browse/KAFKA-1690 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein Assignee: Sriharsha Chintalapani Fix For: 0.8.3 Attachments: KAFKA-1690.patch, KAFKA-1690.patch, KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, KAFKA-1690_2015-05-21_10:37:08.patch, KAFKA-1690_2015-06-03_18:52:29.patch, KAFKA-1690_2015-06-23_13:18:20.patch, KAFKA-1690_2015-07-20_06:10:42.patch, KAFKA-1690_2015-07-20_11:59:57.patch, KAFKA-1690_2015-07-25_12:10:55.patch, KAFKA-1690_2015-08-16_20:41:02.patch, KAFKA-1690_2015-08-17_08:12:50.patch, KAFKA-1690_2015-08-17_09:28:52.patch, KAFKA-1690_2015-08-17_12:20:53.patch, KAFKA-1690_2015-08-18_11:24:46.patch, KAFKA-1690_2015-08-18_17:24:48.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2467) ConsoleConsumer regressions
[ https://issues.apache.org/jira/browse/KAFKA-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-2467: - Resolution: Fixed Fix Version/s: 0.8.3 Status: Resolved (was: Patch Available) Issue resolved by pull request 166 [https://github.com/apache/kafka/pull/166] ConsoleConsumer regressions --- Key: KAFKA-2467 URL: https://issues.apache.org/jira/browse/KAFKA-2467 Project: Kafka Issue Type: Bug Components: tools Reporter: Ewen Cheslack-Postava Assignee: Ewen Cheslack-Postava Fix For: 0.8.3 It seems that the patch for KAFKA-2015 caused a few changes in the behavior of the console consumer. I picked this up because it caused the new mirror maker sanity system test to hang. We need a separate fix for ducktape to address the lack of a timeout where it got stuck, but I'd also like to get this fixed ASAP since it affects pretty much all system test efforts since they commonly use console consumer to validate data produced to Kafka. I've tracked down a couple of changes so far: 1. The --consumer.config option handling was changed entirely. I think the new approach was trying to parse it as key=value parameters, but it's supposed to be a properties file *containing* key=value pairs. 2. A few different exceptions during message processing are not handled the same way. The skipMessageOnErrorOpt is not longer being used at all (it's parsed, but that option is never checked anymore). Also, exceptions during iteration are not caught. After fixing the consumer.config issue, which was keeping the consumer.timeout.ms setting from making it into the consumer config, this also caused the process to hang. It killed the main thread, but there must be another non-daemon thread still running (presumably the consumer threads?) 3. The consumed X messages message changed from stderr to stdout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2467) ConsoleConsumer regressions
[ https://issues.apache.org/jira/browse/KAFKA-2467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715886#comment-14715886 ] ASF GitHub Bot commented on KAFKA-2467: --- Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/166 ConsoleConsumer regressions --- Key: KAFKA-2467 URL: https://issues.apache.org/jira/browse/KAFKA-2467 Project: Kafka Issue Type: Bug Components: tools Reporter: Ewen Cheslack-Postava Assignee: Ewen Cheslack-Postava Fix For: 0.8.3 It seems that the patch for KAFKA-2015 caused a few changes in the behavior of the console consumer. I picked this up because it caused the new mirror maker sanity system test to hang. We need a separate fix for ducktape to address the lack of a timeout where it got stuck, but I'd also like to get this fixed ASAP since it affects pretty much all system test efforts since they commonly use console consumer to validate data produced to Kafka. I've tracked down a couple of changes so far: 1. The --consumer.config option handling was changed entirely. I think the new approach was trying to parse it as key=value parameters, but it's supposed to be a properties file *containing* key=value pairs. 2. A few different exceptions during message processing are not handled the same way. The skipMessageOnErrorOpt is not longer being used at all (it's parsed, but that option is never checked anymore). Also, exceptions during iteration are not caught. After fixing the consumer.config issue, which was keeping the consumer.timeout.ms setting from making it into the consumer config, this also caused the process to hang. It killed the main thread, but there must be another non-daemon thread still running (presumably the consumer threads?) 3. The consumed X messages message changed from stderr to stdout. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2466) ConsoleConsumer throws ConcurrentModificationException on termination
[ https://issues.apache.org/jira/browse/KAFKA-2466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14715889#comment-14715889 ] Guozhang Wang commented on KAFKA-2466: -- This is fixed as part of KAFKA-2466. ConsoleConsumer throws ConcurrentModificationException on termination - Key: KAFKA-2466 URL: https://issues.apache.org/jira/browse/KAFKA-2466 Project: Kafka Issue Type: Bug Components: tools Reporter: Ashish K Singh Assignee: Ashish K Singh Fix For: 0.8.3 ConsoleConsumer throws ConcurrentModificationException on termination. ST: {code} Exception in thread Thread-1 java.util.ConcurrentModificationException: KafkaConsumer is not safe for multi-threaded access at org.apache.kafka.clients.consumer.KafkaConsumer.acquire(KafkaConsumer.java:1169) at org.apache.kafka.clients.consumer.KafkaConsumer.close(KafkaConsumer.java:1087) at kafka.consumer.NewShinyConsumer.close(BaseConsumer.scala:50) at kafka.tools.ConsoleConsumer$$anon$1.run(ConsoleConsumer.scala:74) {code} Other thread which constantly tries to consume is {code} main prio=10 tid=0x7f3aa800c000 nid=0x1314 runnable [0x7f3aae37d000] java.lang.Thread.State: RUNNABLE at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method) at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269) at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:79) at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:87) - locked 0xdd1df130 (a sun.nio.ch.Util$2) - locked 0xdd1df120 (a java.util.Collections$UnmodifiableSet) - locked 0xdd0af720 (a sun.nio.ch.EPollSelectorImpl) at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:98) at org.apache.kafka.common.network.Selector.select(Selector.java:440) at org.apache.kafka.common.network.Selector.poll(Selector.java:263) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:221) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:274) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:182) at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:172) at org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:779) at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:730) at kafka.consumer.NewShinyConsumer.receive(BaseConsumer.scala:43) at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:87) at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:54) at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:39) at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala) {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Build failed in Jenkins: Kafka-trunk #603
See https://builds.apache.org/job/Kafka-trunk/603/changes Changes: [wangguoz] KAFKA-2467: Fix changes to behavior in ConsoleConsumer: properly parse consumer.config option, handle exceptions during message processing, and print number of processed messages to stderr. -- [...truncated 897 lines...] kafka.utils.timer.TimerTest testTaskExpiration PASSED kafka.server.OffsetCommitTest testUpdateOffsets PASSED kafka.coordinator.ConsumerCoordinatorResponseTest testHeartbeatUnknownGroup PASSED kafka.api.ProducerSendTest testCloseWithZeroTimeoutFromCallerThread PASSED kafka.admin.DeleteConsumerGroupTest testTopicWideDeleteInZK PASSED kafka.coordinator.ConsumerCoordinatorResponseTest testJoinGroupUnknownConsumerExistingGroup PASSED kafka.coordinator.ConsumerCoordinatorResponseTest testJoinGroupSessionTimeoutTooSmall PASSED kafka.coordinator.ConsumerCoordinatorResponseTest testHeartbeatIllegalGeneration PASSED kafka.coordinator.ConsumerCoordinatorResponseTest testGenerationIdIncrementsOnRebalance PASSED kafka.coordinator.ConsumerCoordinatorResponseTest testHeartbeatUnknownConsumerExistingGroup PASSED kafka.integration.TopicMetadataTest testIsrAfterBrokerShutDownAndJoinsBack PASSED kafka.admin.AdminTest testBasicPreferredReplicaElection PASSED kafka.admin.DeleteTopicTest testAddPartitionDuringDeleteTopic PASSED kafka.admin.AddPartitionsTest testWrongReplicaCount PASSED kafka.integration.AutoOffsetResetTest testResetToLatestWhenOffsetTooHigh PASSED kafka.admin.DeleteConsumerGroupTest testConsumptionOnRecreatedTopicAfterTopicWideDeleteInZK PASSED kafka.admin.DeleteTopicTest testRecreateTopicAfterDeletion PASSED kafka.admin.AdminTest testShutdownBroker PASSED kafka.admin.AdminTest testReplicaAssignment PASSED kafka.integration.FetcherTest testFetcher PASSED kafka.common.ConfigTest testInvalidGroupIds PASSED kafka.common.ConfigTest testInvalidClientIds PASSED kafka.admin.AdminTest testTopicCreationWithCollision PASSED kafka.api.ProducerSendTest testCloseWithZeroTimeoutFromSenderThread PASSED kafka.utils.SchedulerTest testPeriodicTask PASSED kafka.utils.SchedulerTest testReentrantTaskInMockScheduler PASSED kafka.utils.SchedulerTest testMockSchedulerPeriodicTask PASSED kafka.utils.SchedulerTest testNonPeriodicTask PASSED kafka.utils.SchedulerTest testMockSchedulerNonPeriodicTask PASSED kafka.utils.SchedulerTest testRestart PASSED kafka.api.ConsumerTest testSimpleConsumption PASSED kafka.integration.AutoOffsetResetTest testResetToLatestWhenOffsetTooLow PASSED kafka.admin.AdminTest testTopicCreationInZK PASSED kafka.admin.AddPartitionsTest testIncrementPartitions PASSED kafka.admin.DeleteTopicTest testDeleteNonExistingTopic PASSED kafka.admin.AdminTest testTopicConfigChange PASSED kafka.integration.AutoOffsetResetTest testResetToEarliestWhenOffsetTooHigh PASSED kafka.log.LogTest testCorruptLog PASSED kafka.log.LogTest testCleanShutdownFile PASSED kafka.log.LogTest testParseTopicPartitionName PASSED kafka.log.LogTest testParseTopicPartitionNameForEmptyName PASSED kafka.log.LogTest testParseTopicPartitionNameForNull PASSED kafka.log.LogTest testParseTopicPartitionNameForMissingSeparator PASSED kafka.log.LogTest testParseTopicPartitionNameForMissingTopic PASSED kafka.log.LogTest testParseTopicPartitionNameForMissingPartition PASSED kafka.log.LogCleanerIntegrationTest cleanerTest[0] PASSED kafka.integration.AutoOffsetResetTest testResetToEarliestWhenOffsetTooLow PASSED kafka.api.ConsumerTest testCommitSpecifiedOffsets PASSED kafka.api.ConsumerBounceTest testConsumptionWithBrokerFailures PASSED kafka.utils.IteratorTemplateTest testIterator PASSED kafka.api.ProducerFailureHandlingTest testNoResponse PASSED kafka.api.ConsumerTest testPositionAndCommit PASSED kafka.integration.RollingBounceTest testRollingBounce PASSED kafka.common.TopicTest testTopicHasCollisionChars PASSED kafka.common.TopicTest testTopicHasCollision PASSED kafka.common.TopicTest testInvalidTopicNames PASSED kafka.api.ProducerFailureHandlingTest testInvalidPartition PASSED kafka.api.ConsumerTest testPartitionsFor PASSED kafka.api.ProducerFailureHandlingTest testSendAfterClosed PASSED kafka.integration.UncleanLeaderElectionTest testCleanLeaderElectionDisabledByTopicOverride PASSED kafka.admin.DeleteTopicTest testDeleteTopicWithCleaner PASSED kafka.api.ProducerFailureHandlingTest testCannotSendToInternalTopic PASSED kafka.api.ProducerFailureHandlingTest testNotEnoughReplicas PASSED kafka.api.ConsumerTest testListTopics PASSED kafka.admin.DeleteTopicTest testDeleteTopicAlreadyMarkedAsDeleted PASSED kafka.log.LogCleanerIntegrationTest cleanerTest[1] PASSED kafka.integration.PrimitiveApiTest testEmptyFetchRequest PASSED kafka.integration.PrimitiveApiTest testConsumerEmptyTopic PASSED kafka.integration.UncleanLeaderElectionTest testUncleanLeaderElectionEnabledByTopicOverride
[GitHub] kafka pull request: KAFKA-2467: Fix changes to behavior in Console...
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/166 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enabled but not working, please contact infrastructure at infrastruct...@apache.org or file a JIRA ticket with INFRA. ---
Re: [DISCUSS] Client-side Assignment for New Consumer
Hey Becket, In that case, the broker side partition assignment would be ideal because it avoids issues like metadata inconsistency / split brain / exploding subscription set propagation. As per our previous discussions regarding each of those concerns (referring to this email thread, KIP calls and JIRA comments), we are going to run a set of tests using the LinkedIn deployment numbers that we will wait for you to share. The purpose is to see if those concerns are really valid or not. I'd prefer to see that before making any more changes that will complicate the protocol. On Wed, Aug 26, 2015 at 4:57 PM, Jiangjie Qin j...@linkedin.com.invalid wrote: Hi folks, After further discussion in LinkedIn, we found that while having a more general group management protocol is very useful, the vast majority of the clients will not use customized partition assignment strategy. In that case, the broker side partition assignment would be ideal because it avoids issues like metadata inconsistency / split brain / exploding subscription set propagation. So we have the following proposal that satisfies the majority of the clients' needs without changing the currently proposed binary protocol. i.e., Continue to support broker-side assignment if the assignment strategy is recognized by the coordinator. 1. Keep the binary protocol as currently proposed. 2. Change the way we interpret ProtocolMetadata: 2.1 On consumer side, change partition.assignment.strategy to partition.assignor.class. Implement the something like the following PartitionAssignor Interface: public interface PartitionAssignor { ListString protocolTypes(); byte[] protocolMetadata(); // return the Topic-ListPartition map that are assigned to this consumer. ListTopicPartition assignPartitions(String protocolType, byte[] responseProtocolMetadata); } public abstract class AbstractPartitionAssignor implements PartitionAssignor { protected final KafkaConsumer consumer; AbstractPartitionAssignor(KafkaConsumer consumer) { this.consumer = consumer; } } 2.2 The ProtocolMetadata in JoinGroupRequest will be partitionAssignor.protocolMetadata(). When partition.assignor.class is range or roundrobin, the ProtocolMetadata in JoinGroupRequest will be a JSON subscription set. (range, roundrobin will be reserved words, we can also consider reserving some Prefix such as broker- to be more clear) 2.3 On broker side when ProtocolType is range or roundroubin, coordinator will parse the ProtocolMetadata in the JoinGroupRequest and assign the partitions for consumers. In the JoinGroupResponse, the ProtocolMetadata will be the global assignment of partitions. 2.4 On client side, after receiving the JoinGroupResponse, partitionAssignor.assignPartitions() will be invoked to return the actual assignment. If the assignor is RangeAssignor or RoundRobinAssignor, they will parse the assignment from the ProtocolMetadata returned by coordinator. This approach has a few merits: 1. Does not change the proposed binary protocol, which is still general. 2. The majority of the consumers will not suffer from inconsistent metadata / split brain / exploding subscription set propagation. This is specifically to deal with the issue that the current proposal caters to a 20% use-case while adversely impacting the more common 80% use-cases. 3. Easy to implement. The only thing needed is implement a partitioner class. For most users, the default range and roundrobin partitioner are good enough. Thoughts? Thanks, Jiangjie (Becket) Qin On Tue, Aug 18, 2015 at 2:51 PM, Jason Gustafson ja...@confluent.io wrote: Follow-up from the kip call: 1. Onur brought up the question of whether this protocol provides enough coordination capabilities to be generally useful in practice (is that accurate, Onur?). If it doesn't, then each use case would probably need a dependence on zookeeper anyway, and we haven't really gained anything. The group membership provided by this protocol is a useful primitive for coordination, but it's limited in the sense that everything shared among the group has to be communicated at the time the group is created. If any shared data changes, then the only way the group can ensure agreement is to force a rebalance. This is expensive since all members must stall while the rebalancing takes place. As we have also seen, there is a practical limit on the amount of metadata that can be sent through this protocol when groups get a little larger. This protocol is therefore not suitable to cases which require frequent communication or which require a large amount of data to be communicated. For the use cases listed on the wiki, neither of these appear to be an issue, but there may be other limitations which would limit reuse of the protocol. Perhaps it would be sufficient to sketch how these cases might work? 2. We talked a little bit about the issue of metadata