[jira] [Commented] (KAFKA-1530) howto update continuously
[ https://issues.apache.org/jira/browse/KAFKA-1530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064815#comment-14064815 ] Oleg Golovin commented on KAFKA-1530: - Thank you for mentioning the option unclean.leader.election.enable. It seems to be a new option we didn't know of. We will need some time to test it. We will report how it went as soon as we perform this testing. howto update continuously - Key: KAFKA-1530 URL: https://issues.apache.org/jira/browse/KAFKA-1530 Project: Kafka Issue Type: Wish Reporter: Stanislav Gilmulin Assignee: Guozhang Wang Priority: Minor Labels: operating_manual, performance Hi, Could I ask you a question about the Kafka update procedure? Is there a way to update software, which doesn't require service interruption or lead to data losses? We can't stop message brokering during the update as we have a strict SLA. Best regards Stanislav Gilmulin -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (KAFKA-1543) Changing replication factor
Alexey Ozeritskiy created KAFKA-1543: Summary: Changing replication factor Key: KAFKA-1543 URL: https://issues.apache.org/jira/browse/KAFKA-1543 Project: Kafka Issue Type: Improvement Reporter: Alexey Ozeritskiy Attachments: can-change-replication.patch It is difficult to change replication factor by manual editing json config. I propose to add a key to kafka-reassign-partitions.sh command to automatically create json config. Example of usage %% kafka-reassign-partitions.sh --zookeeper zk --replicas new-replication-factor --topics-to-move-json-file topics-file --broker-list 1,2,3,4 --generate output %% -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1543) Changing replication factor
[
https://issues.apache.org/jira/browse/KAFKA-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Ozeritskiy updated KAFKA-1543:
-
Attachment: can-change-replication.patch
Changing replication factor
---
Key: KAFKA-1543
URL: https://issues.apache.org/jira/browse/KAFKA-1543
Project: Kafka
Issue Type: Improvement
Reporter: Alexey Ozeritskiy
Attachments: can-change-replication.patch
It is difficult to change replication factor by manual editing json config.
I propose to add a key to kafka-reassign-partitions.sh command to
automatically create json config.
Example of usage
{code}
kafka-reassign-partitions.sh --zookeeper zk --replicas new-replication-factor
--topics-to-move-json-file topics-file --broker-list 1,2,3,4 --generate
output
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (KAFKA-1543) Changing replication factor
[
https://issues.apache.org/jira/browse/KAFKA-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Ozeritskiy updated KAFKA-1543:
-
Description:
It is difficult to change replication factor by manual editing json config.
I propose to add a key to kafka-reassign-partitions.sh command to automatically
create json config.
Example of usage
{code}
kafka-reassign-partitions.sh --zookeeper zk --replicas new-replication-factor
--topics-to-move-json-file topics-file --broker-list 1,2,3,4 --generate output
{code}
was:
It is difficult to change replication factor by manual editing json config.
I propose to add a key to kafka-reassign-partitions.sh command to automatically
create json config.
Example of usage
%%
kafka-reassign-partitions.sh --zookeeper zk --replicas new-replication-factor
--topics-to-move-json-file topics-file --broker-list 1,2,3,4 --generate output
%%
Changing replication factor
---
Key: KAFKA-1543
URL: https://issues.apache.org/jira/browse/KAFKA-1543
Project: Kafka
Issue Type: Improvement
Reporter: Alexey Ozeritskiy
Attachments: can-change-replication.patch
It is difficult to change replication factor by manual editing json config.
I propose to add a key to kafka-reassign-partitions.sh command to
automatically create json config.
Example of usage
{code}
kafka-reassign-partitions.sh --zookeeper zk --replicas new-replication-factor
--topics-to-move-json-file topics-file --broker-list 1,2,3,4 --generate
output
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (KAFKA-1543) Changing replication factor
[
https://issues.apache.org/jira/browse/KAFKA-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Ozeritskiy updated KAFKA-1543:
-
Status: Patch Available (was: Open)
Changing replication factor
---
Key: KAFKA-1543
URL: https://issues.apache.org/jira/browse/KAFKA-1543
Project: Kafka
Issue Type: Improvement
Reporter: Alexey Ozeritskiy
Attachments: can-change-replication.patch
It is difficult to change replication factor by manual editing json config.
I propose to add a key to kafka-reassign-partitions.sh command to
automatically create json config.
Example of usage
{code}
kafka-reassign-partitions.sh --zookeeper zk --replicas new-replication-factor
--topics-to-move-json-file topics-file --broker-list 1,2,3,4 --generate
output
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (KAFKA-1544) LogCleaner may take a long time to shutdown
[
https://issues.apache.org/jira/browse/KAFKA-1544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14064989#comment-14064989
]
Jun Rao commented on KAFKA-1544:
One solution is to simply put the sleep in a loop. In each iteration, we sleep
for a small amount of time, say 500ms. The loop finishes if the thread is not
runnable or if the backoff time is reached.
LogCleaner may take a long time to shutdown
---
Key: KAFKA-1544
URL: https://issues.apache.org/jira/browse/KAFKA-1544
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.2
Reporter: Jun Rao
Labels: newbie
We have the following code in LogCleaner. Since the cleaner thread is
shutdown w/o interrupt. If may take up to backoff time for the cleaner thread
to detect the shutdown flag.
private def cleanOrSleep() {
cleanerManager.grabFilthiestLog() match {
case None =
// there are no cleanable logs, sleep a while
time.sleep(config.backOffMs)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
Re: Review Request 23568: Patch for KAFKA-1523
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23568/#review48011 --- core/src/main/scala/kafka/server/KafkaApis.scala https://reviews.apache.org/r/23568/#comment84260 Need to specify txId = txRequest.requestInfo.txId in Message construction. - Dong Lin On July 16, 2014, 9:29 p.m., Dong Lin wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23568/ --- (Updated July 16, 2014, 9:29 p.m.) Review request for kafka. Bugs: KAFKA-1523 https://issues.apache.org/jira/browse/KAFKA-1523 Repository: kafka Description --- KAFKA-1523 transaction manager module Diffs - core/src/main/scala/kafka/admin/TopicCommand.scala 8d5c2e7088fc6e8bf69e775ea7f5893b94580fdf core/src/main/scala/kafka/common/Topic.scala ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 core/src/main/scala/kafka/controller/ControllerChannelManager.scala 8763968fbff697e4c5c98ab1274627c192a4d26a core/src/main/scala/kafka/message/Message.scala d2a7293c7be4022af30884330924791340acc5c1 core/src/main/scala/kafka/server/KafkaApis.scala 0b668f230c8556fdf08654ce522a11847d0bf39b core/src/main/scala/kafka/server/KafkaConfig.scala ef75b67b67676ae5b8931902cbc8c0c2cc72c0d3 core/src/main/scala/kafka/server/KafkaServer.scala c22e51e0412843ec993721ad3230824c0aadd2ba core/src/main/scala/kafka/server/TransactionManager.scala PRE-CREATION core/src/main/scala/kafka/utils/ZkUtils.scala dcdc1ce2b02c996294e19cf480736106aaf29511 Diff: https://reviews.apache.org/r/23568/diff/ Testing --- Thanks, Dong Lin
[jira] [Commented] (KAFKA-1451) Broker stuck due to leader election race
[
https://issues.apache.org/jira/browse/KAFKA-1451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065055#comment-14065055
]
Jun Rao commented on KAFKA-1451:
Thanks for reporting this. Very interesting. That does sound like a potential
problem. The problem is that ZookeeperLeaderElector.elect assumes that no
controller exists. However, this may not be true. One possible solution is to
first check the existence of the controller from ZK before creating the
ephemeral node.
Broker stuck due to leader election race
-
Key: KAFKA-1451
URL: https://issues.apache.org/jira/browse/KAFKA-1451
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: Maciek Makowski
Priority: Minor
h3. Symptoms
The broker does not become available due to being stuck in an infinite loop
while electing leader. This can be recognised by the following line being
repeatedly written to server.log:
{code}
[2014-05-14 04:35:09,187] INFO I wrote this conflicted ephemeral node
[{version:1,brokerid:1,timestamp:1400060079108}] at /controller a
while back in a different session, hence I will backoff for this node to be
deleted by Zookeeper and retry (kafka.utils.ZkUtils$)
{code}
h3. Steps to Reproduce
In a single kafka 0.8.1.1 node, single zookeeper 3.4.6 (but will likely
behave the same with the ZK version included in Kafka distribution) node
setup:
# start both zookeeper and kafka (in any order)
# stop zookeeper
# stop kafka
# start kafka
# start zookeeper
h3. Likely Cause
{{ZookeeperLeaderElector}} subscribes to data changes on startup, and then
triggers an election. if the deletion of ephemeral {{/controller}} node
associated with previous zookeeper session of the broker happens after
subscription to changes in new session, election will be invoked twice, once
from {{startup}} and once from {{handleDataDeleted}}:
* {{startup}}: acquire {{controllerLock}}
* {{startup}}: subscribe to data changes
* zookeeper: delete {{/controller}} since the session that created it timed
out
* {{handleDataDeleted}}: {{/controller}} was deleted
* {{handleDataDeleted}}: wait on {{controllerLock}}
* {{startup}}: elect -- writes {{/controller}}
* {{startup}}: release {{controllerLock}}
* {{handleDataDeleted}}: acquire {{controllerLock}}
* {{handleDataDeleted}}: elect -- attempts to write {{/controller}} and then
gets into infinite loop as a result of conflict
{{createEphemeralPathExpectConflictHandleZKBug}} assumes that the existing
znode was written from different session, which is not true in this case; it
was written from the same session. That adds to the confusion.
h3. Suggested Fix
In {{ZookeeperLeaderElector.startup}} first run {{elect}} and then subscribe
to data changes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (KAFKA-1541) Add transactional request definitions to clients package
[ https://issues.apache.org/jira/browse/KAFKA-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raul Castro Fernandez updated KAFKA-1541: - Attachment: (was: KAFKA-1541.patch) Add transactional request definitions to clients package - Key: KAFKA-1541 URL: https://issues.apache.org/jira/browse/KAFKA-1541 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Raul Castro Fernandez Labels: transactions Separate jira for this since KAFKA-1522 only adds definitions to the core package. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1541) Add transactional request definitions to clients package
[ https://issues.apache.org/jira/browse/KAFKA-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raul Castro Fernandez updated KAFKA-1541: - Attachment: KAFKA-1541.patch Add transactional request definitions to clients package - Key: KAFKA-1541 URL: https://issues.apache.org/jira/browse/KAFKA-1541 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Raul Castro Fernandez Labels: transactions Attachments: KAFKA-1541.patch Separate jira for this since KAFKA-1522 only adds definitions to the core package. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (KAFKA-1536) Change the status of the JIRA to Patch Available in the kafka-review-tool
[ https://issues.apache.org/jira/browse/KAFKA-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede reassigned KAFKA-1536: Assignee: Neha Narkhede (was: Manikumar Reddy) Assigning to myself for review Change the status of the JIRA to Patch Available in the kafka-review-tool --- Key: KAFKA-1536 URL: https://issues.apache.org/jira/browse/KAFKA-1536 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Neha Narkhede Fix For: 0.9.0 Attachments: KAFKA-1536.patch, KAFKA-1536.patch When using the kafka-review-tool to upload a patch to certain jira, the status remains OPEN. It makes searching for JIRAs that needs review a bit hard. Would be better to make the tool also change the status of the jira. -- This message was sent by Atlassian JIRA (v6.2#6252)
Review Request 23641: Patch for KAFKA-1536
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23641/ --- Review request for kafka. Bugs: KAFKA-1536 https://issues.apache.org/jira/browse/KAFKA-1536 Repository: kafka Description --- committing v1 for testing Diffs - kafka-patch-review.py dc45549f886440f1721c60aab9aa0a4af9b4cbef Diff: https://reviews.apache.org/r/23641/diff/ Testing --- Thanks, Neha Narkhede
[jira] [Commented] (KAFKA-1536) Change the status of the JIRA to Patch Available in the kafka-review-tool
[ https://issues.apache.org/jira/browse/KAFKA-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065123#comment-14065123 ] Neha Narkhede commented on KAFKA-1536: -- Created reviewboard https://reviews.apache.org/r/23641/ against branch trunk Change the status of the JIRA to Patch Available in the kafka-review-tool --- Key: KAFKA-1536 URL: https://issues.apache.org/jira/browse/KAFKA-1536 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Neha Narkhede Fix For: 0.9.0 Attachments: KAFKA-1536.patch, KAFKA-1536.patch, KAFKA-1536.patch When using the kafka-review-tool to upload a patch to certain jira, the status remains OPEN. It makes searching for JIRAs that needs review a bit hard. Would be better to make the tool also change the status of the jira. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1536) Change the status of the JIRA to Patch Available in the kafka-review-tool
[ https://issues.apache.org/jira/browse/KAFKA-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1536: - Attachment: KAFKA-1536.patch Change the status of the JIRA to Patch Available in the kafka-review-tool --- Key: KAFKA-1536 URL: https://issues.apache.org/jira/browse/KAFKA-1536 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Neha Narkhede Fix For: 0.9.0 Attachments: KAFKA-1536.patch, KAFKA-1536.patch, KAFKA-1536.patch When using the kafka-review-tool to upload a patch to certain jira, the status remains OPEN. It makes searching for JIRAs that needs review a bit hard. Would be better to make the tool also change the status of the jira. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1536) Change the status of the JIRA to Patch Available in the kafka-review-tool
[ https://issues.apache.org/jira/browse/KAFKA-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1536: - Status: Patch Available (was: In Progress) Change the status of the JIRA to Patch Available in the kafka-review-tool --- Key: KAFKA-1536 URL: https://issues.apache.org/jira/browse/KAFKA-1536 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Neha Narkhede Fix For: 0.9.0 Attachments: KAFKA-1536.patch, KAFKA-1536.patch, KAFKA-1536.patch When using the kafka-review-tool to upload a patch to certain jira, the status remains OPEN. It makes searching for JIRAs that needs review a bit hard. Would be better to make the tool also change the status of the jira. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (KAFKA-1536) Change the status of the JIRA to Patch Available in the kafka-review-tool
[ https://issues.apache.org/jira/browse/KAFKA-1536?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065123#comment-14065123 ] Neha Narkhede edited comment on KAFKA-1536 at 7/17/14 4:56 PM: --- For testing the patch. Created reviewboard https://reviews.apache.org/r/23641/ against branch trunk was (Author: nehanarkhede): Created reviewboard https://reviews.apache.org/r/23641/ against branch trunk Change the status of the JIRA to Patch Available in the kafka-review-tool --- Key: KAFKA-1536 URL: https://issues.apache.org/jira/browse/KAFKA-1536 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Neha Narkhede Fix For: 0.9.0 Attachments: KAFKA-1536.patch, KAFKA-1536.patch, KAFKA-1536.patch When using the kafka-review-tool to upload a patch to certain jira, the status remains OPEN. It makes searching for JIRAs that needs review a bit hard. Would be better to make the tool also change the status of the jira. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 23440: Patch for KAFKA-1536
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23440/#review48017 --- Other than the comment below, this patch looks great! kafka-patch-review.py https://reviews.apache.org/r/23440/#comment84268 our naming convention is not really camel case in this python code. Could you change transitionsMap to jira_transitions? - Neha Narkhede On July 12, 2014, 1:51 p.m., Manikumar Reddy O wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23440/ --- (Updated July 12, 2014, 1:51 p.m.) Review request for kafka. Bugs: KAFKA-1536 https://issues.apache.org/jira/browse/KAFKA-1536 Repository: kafka Description --- JIRA status set to Patch Available in kafka-patch-review script Diffs - kafka-patch-review.py dc45549f886440f1721c60aab9aa0a4af9b4cbef Diff: https://reviews.apache.org/r/23440/diff/ Testing --- Thanks, Manikumar Reddy O
[jira] [Updated] (KAFKA-1538) TEST JIRA for KAFKA-1536
[ https://issues.apache.org/jira/browse/KAFKA-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1538: - Resolution: Invalid Status: Resolved (was: Patch Available) Closing this as the test succeeded. TEST JIRA for KAFKA-1536 Key: KAFKA-1538 URL: https://issues.apache.org/jira/browse/KAFKA-1538 Project: Kafka Issue Type: Test Reporter: Guozhang Wang Assignee: Guozhang Wang Attachments: KAFKA-1538.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Closed] (KAFKA-1538) TEST JIRA for KAFKA-1536
[ https://issues.apache.org/jira/browse/KAFKA-1538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede closed KAFKA-1538. TEST JIRA for KAFKA-1536 Key: KAFKA-1538 URL: https://issues.apache.org/jira/browse/KAFKA-1538 Project: Kafka Issue Type: Test Reporter: Guozhang Wang Assignee: Guozhang Wang Attachments: KAFKA-1538.patch -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1059) Improve the patch review tool to use OAuth for JIRA access
[ https://issues.apache.org/jira/browse/KAFKA-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065137#comment-14065137 ] Neha Narkhede commented on KAFKA-1059: -- hey [~omkreddy], thanks for proactively looking into this. I punted on this since I couldn't figure it out in the 1 hour that I dedicated to writing this tool :-) I'm wondering if you need any help for next steps on this? For step 2, are you saying that you contacted the JIRA admin already? Did you hear back? Improve the patch review tool to use OAuth for JIRA access -- Key: KAFKA-1059 URL: https://issues.apache.org/jira/browse/KAFKA-1059 Project: Kafka Issue Type: Improvement Components: tools Reporter: Neha Narkhede jira-python seems to support oauth for accessing jira. It will be nice to do that instead of storing the password in clear text http://jira-python.readthedocs.org/en/latest/#oauth -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1059) Improve the patch review tool to use OAuth for JIRA access
[ https://issues.apache.org/jira/browse/KAFKA-1059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1059: - Labels: newbie (was: ) Improve the patch review tool to use OAuth for JIRA access -- Key: KAFKA-1059 URL: https://issues.apache.org/jira/browse/KAFKA-1059 Project: Kafka Issue Type: Improvement Components: tools Reporter: Neha Narkhede Assignee: Manikumar Reddy Labels: newbie jira-python seems to support oauth for accessing jira. It will be nice to do that instead of storing the password in clear text http://jira-python.readthedocs.org/en/latest/#oauth -- This message was sent by Atlassian JIRA (v6.2#6252)
Review Request 23647: Patch for KAFKA-1526
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23647/ --- Review request for kafka. Bugs: KAFKA-1526 https://issues.apache.org/jira/browse/KAFKA-1526 Repository: kafka Description --- KAFKA-1526; Producer perf tool has an option to enable transactions Diffs - core/src/main/scala/kafka/producer/BaseProducer.scala b0207930dd0543f2c51f0b35002e13bf104340ff core/src/main/scala/kafka/tools/ProducerPerformance.scala fc3e7241dda0672deb8c5ce5d7c983c320ec45ee Diff: https://reviews.apache.org/r/23647/diff/ Testing --- Thanks, Raul Castro Fernandez
[jira] [Commented] (KAFKA-1526) Producer performance tool should have an option to enable transactions
[ https://issues.apache.org/jira/browse/KAFKA-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065192#comment-14065192 ] Raul Castro Fernandez commented on KAFKA-1526: -- Created reviewboard https://reviews.apache.org/r/23647/diff/ against branch origin/transactional_messaging Producer performance tool should have an option to enable transactions -- Key: KAFKA-1526 URL: https://issues.apache.org/jira/browse/KAFKA-1526 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Raul Castro Fernandez Labels: transactions Attachments: KAFKA-1526.patch, KAFKA-1526.patch If this flag is enabled the producer could start/commit/abort transactions randomly - we could add more configs/parameters for more control on transaction boundaries. -- This message was sent by Atlassian JIRA (v6.2#6252)
Review Request 23648: Patch for KAFKA-1524
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23648/ --- Review request for kafka. Bugs: KAFKA-1524 https://issues.apache.org/jira/browse/KAFKA-1524 Repository: kafka Description --- KAFKA-1524; Implement transactional producer Diffs - clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 522881c972ca42ff4dfb6237a2db15b625334d7e clients/src/main/java/org/apache/kafka/clients/producer/InvalidTransactionStatusException.java PRE-CREATION clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 00775abbcac850b0f2bb9a70b6fbc7cdf319bcf6 clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java c0f1d57e0feb894d9f246058cd0396461afe3225 clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 36e8398416036cab84faad1f07159e5adefd8086 clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java f9de4af426449cceca12a8de9a9f54a6241d28d8 clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java 1ed3c28b436d28381d9402896e32d16f2586c65e clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java dd0af8aee98abed5d4a0dc50989e37888bb353fe clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionContext.java PRE-CREATION clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionControl.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/errors/TransactionCoordinatorNotAvailableException.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/errors/TransactionFailedException.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/record/Compressor.java 0323f5f7032dceb49d820c17a41b78c56591ffc4 clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 759f577eaf0e7d28a84926d4aa30f4ef0cb27bc2 clients/src/main/java/org/apache/kafka/common/record/Record.java 10df9fd8d3f4ec8c277650fa7eab269f3ea30d85 clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java 93b58d02eac0f8ca28440e3e0ebea28ed3a7673c clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java 5489acac6806b3ae5e6d568d401d5a20c86cac05 clients/src/test/java/org/apache/kafka/clients/producer/TransactionContextTest.java PRE-CREATION clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java 94a11121e207d5cf94dbc94443a8aa7edf387782 Diff: https://reviews.apache.org/r/23648/diff/ Testing --- Thanks, Raul Castro Fernandez
[jira] [Updated] (KAFKA-1524) Implement transactional producer
[ https://issues.apache.org/jira/browse/KAFKA-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raul Castro Fernandez updated KAFKA-1524: - Attachment: KAFKA-1524.patch Implement transactional producer Key: KAFKA-1524 URL: https://issues.apache.org/jira/browse/KAFKA-1524 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Raul Castro Fernandez Labels: transactions Attachments: KAFKA-1524.patch Implement the basic transactional producer functionality as outlined in https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka The scope of this jira is basic functionality (i.e., to be able to begin and commit or abort a transaction) without the failure scenarios. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1524) Implement transactional producer
[ https://issues.apache.org/jira/browse/KAFKA-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raul Castro Fernandez updated KAFKA-1524: - Attachment: (was: KAFKA-1524.patch) Implement transactional producer Key: KAFKA-1524 URL: https://issues.apache.org/jira/browse/KAFKA-1524 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Raul Castro Fernandez Labels: transactions Attachments: KAFKA-1524.patch Implement the basic transactional producer functionality as outlined in https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka The scope of this jira is basic functionality (i.e., to be able to begin and commit or abort a transaction) without the failure scenarios. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1526) Producer performance tool should have an option to enable transactions
[ https://issues.apache.org/jira/browse/KAFKA-1526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Raul Castro Fernandez updated KAFKA-1526: - Attachment: (was: KAFKA-1526.patch) Producer performance tool should have an option to enable transactions -- Key: KAFKA-1526 URL: https://issues.apache.org/jira/browse/KAFKA-1526 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Raul Castro Fernandez Labels: transactions Attachments: KAFKA-1526.patch If this flag is enabled the producer could start/commit/abort transactions randomly - we could add more configs/parameters for more control on transaction boundaries. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1524) Implement transactional producer
[ https://issues.apache.org/jira/browse/KAFKA-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065194#comment-14065194 ] Raul Castro Fernandez commented on KAFKA-1524: -- Created reviewboard https://reviews.apache.org/r/23648/diff/ against branch origin/transactional_messaging Implement transactional producer Key: KAFKA-1524 URL: https://issues.apache.org/jira/browse/KAFKA-1524 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Raul Castro Fernandez Labels: transactions Attachments: KAFKA-1524.patch Implement the basic transactional producer functionality as outlined in https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka The scope of this jira is basic functionality (i.e., to be able to begin and commit or abort a transaction) without the failure scenarios. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 23474: Patch for KAFKA-1483
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23474/#review48028 --- Ship it! - Guozhang Wang On July 16, 2014, 6:07 p.m., Sriharsha Chintalapani wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23474/ --- (Updated July 16, 2014, 6:07 p.m.) Review request for kafka. Bugs: KAFKA-1483 https://issues.apache.org/jira/browse/KAFKA-1483 Repository: kafka Description --- KAFKA-1483. Split Brain about Leader Partitions. Diffs - core/src/main/scala/kafka/server/ReplicaManager.scala 6a56a772c134dbf1e70c1bfe067223009bfdbac8 Diff: https://reviews.apache.org/r/23474/diff/ Testing --- Thanks, Sriharsha Chintalapani
[jira] [Created] (KAFKA-1545) java.net.InetAddress.getLocalHost in KafkaHealthcheck.register may fail on some irregular hostnames
Guozhang Wang created KAFKA-1545: Summary: java.net.InetAddress.getLocalHost in KafkaHealthcheck.register may fail on some irregular hostnames Key: KAFKA-1545 URL: https://issues.apache.org/jira/browse/KAFKA-1545 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Assignee: Guozhang Wang Fix For: 0.8.2 For example: kafka.server.LogOffsetTest testGetOffsetsForUnknownTopic FAILED java.net.UnknownHostException: guwang-mn2: guwang-mn2: nodename nor servname provided, or not known at java.net.InetAddress.getLocalHost(InetAddress.java:1473) at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:59) at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:45) at kafka.server.KafkaServer.startup(KafkaServer.scala:121) at kafka.utils.TestUtils$.createServer(TestUtils.scala:130) at kafka.server.LogOffsetTest.setUp(LogOffsetTest.scala:53) Caused by: java.net.UnknownHostException: guwang-mn2: nodename nor servname provided, or not known at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method) at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901) at java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293) at java.net.InetAddress.getLocalHost(InetAddress.java:1469) ... 5 more -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (KAFKA-1546) Automate replica lag tuning
Neha Narkhede created KAFKA-1546: Summary: Automate replica lag tuning Key: KAFKA-1546 URL: https://issues.apache.org/jira/browse/KAFKA-1546 Project: Kafka Issue Type: Improvement Components: replication Affects Versions: 0.8.1.1, 0.8.0, 0.8.1 Reporter: Neha Narkhede Currently, there is no good way to tune the replica lag configs to automatically account for high and low volume topics on the same cluster. For the low-volume topic it will take a very long time to detect a lagging replica, and for the high-volume topic it will have false-positives. One approach to making this easier would be to have the configuration be something like replica.lag.max.ms and translate this into a number of messages dynamically based on the throughput of the partition. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (KAFKA-1547) maven sources jar still empty
lee mighdoll created KAFKA-1547:
---
Summary: maven sources jar still empty
Key: KAFKA-1547
URL: https://issues.apache.org/jira/browse/KAFKA-1547
Project: Kafka
Issue Type: Bug
Components: packaging
Reporter: lee mighdoll
A lot like KAFKA-1174, the published sources jar is empty.
see: http://search.maven.org/#browse%7C329602347
{noformat}
kafka_2.10-0.8.1.1-sources.jar 22-Apr-2014 4.4 K
{noformat}
I've heard kafka is very concise and efficient, but still it probably doesn't
fit in 4.4K of source code...
As a temporary workaround, building the source jar locally seems to work:
{noformat}
./gradlew -PscalaVersion=2.10.4 srcJar
{noformat}
Tagging as packaging.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (KAFKA-1547) maven sources jar still empty
[
https://issues.apache.org/jira/browse/KAFKA-1547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065320#comment-14065320
]
lee mighdoll commented on KAFKA-1547:
-
ah, looks like a dup of KAFKA-1502
maven sources jar still empty
--
Key: KAFKA-1547
URL: https://issues.apache.org/jira/browse/KAFKA-1547
Project: Kafka
Issue Type: Bug
Components: packaging
Reporter: lee mighdoll
A lot like KAFKA-1174, the published sources jar is empty.
see: http://search.maven.org/#browse%7C329602347
{noformat}
kafka_2.10-0.8.1.1-sources.jar22-Apr-2014 4.4 K
{noformat}
I've heard kafka is very concise and efficient, but still it probably doesn't
fit in 4.4K of source code...
As a temporary workaround, building the source jar locally seems to work:
{noformat}
./gradlew -PscalaVersion=2.10.4 srcJar
{noformat}
Tagging as packaging.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Reopened] (KAFKA-1377) transient unit test failure in LogOffsetTest
[ https://issues.apache.org/jira/browse/KAFKA-1377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede reopened KAFKA-1377: -- [~junrao], reopening this as per [~omkreddy]'s observation transient unit test failure in LogOffsetTest Key: KAFKA-1377 URL: https://issues.apache.org/jira/browse/KAFKA-1377 Project: Kafka Issue Type: Bug Components: core Reporter: Jun Rao Assignee: Jun Rao Fix For: 0.8.2 Attachments: KAFKA-1377.patch, KAFKA-1377_2014-04-11_17:42:13.patch, KAFKA-1377_2014-04-11_18:14:45.patch Saw the following transient unit test failure. kafka.server.LogOffsetTest testGetOffsetsBeforeEarliestTime FAILED junit.framework.AssertionFailedError: expected:List(0) but was:Vector() at junit.framework.Assert.fail(Assert.java:47) at junit.framework.Assert.failNotEquals(Assert.java:277) at junit.framework.Assert.assertEquals(Assert.java:64) at junit.framework.Assert.assertEquals(Assert.java:71) at kafka.server.LogOffsetTest.testGetOffsetsBeforeEarliestTime(LogOffsetTest.scala:198) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (KAFKA-1479) Logs filling up while Kafka ReplicaFetcherThread tries to retrieve partition info for deleted topics
[
https://issues.apache.org/jira/browse/KAFKA-1479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neha Narkhede resolved KAFKA-1479.
--
Resolution: Not a Problem
Assignee: (was: Jay Kreps)
This is not a problem once delete topic works.
Logs filling up while Kafka ReplicaFetcherThread tries to retrieve partition
info for deleted topics
Key: KAFKA-1479
URL: https://issues.apache.org/jira/browse/KAFKA-1479
Project: Kafka
Issue Type: Bug
Components: log
Affects Versions: 0.8.1
Environment: CentOS
Reporter: Manasi Manasi
Started noticing that logs are filling up fast with lines like this:
{quote}
[2014-06-01 15:18:08,218] WARN [KafkaApi-2] Fetch request with correlation id
10049 from client ReplicaFetcherThread-0-2 on partition [sams_2014-05-27,26]
failed due to Topic sams_2014-05-27 either doesn't exist or is in the process
of being deleted (kafka.server.KafkaApis)
[2014-06-01 15:18:08,218] WARN [KafkaApi-2] Fetch request with correlation id
10049 from client ReplicaFetcherThread-0-2 on partition [sams_2014-05-28,38]
failed due to Topic sams_2014-05-28 either doesn't exist or is in the process
of being deleted (kafka.server.KafkaApis)
[2014-06-01 15:18:08,219] WARN [KafkaApi-2] Fetch request with correlation id
10049 from client ReplicaFetcherThread-0-2 on partition [sams_2014-05-30,20]
failed due to Topic sams_2014-05-30 either doesn't exist or is in the process
of being deleted (kafka.server.KafkaApis)
[2014-06-01 15:18:08,219] WARN [KafkaApi-2] Fetch request with correlation id
10049 from client ReplicaFetcherThread-0-2 on partition [sams_2014-05-22,46]
failed due to Topic sams_2014-05-22 either doesn't exist or is in the process
of being deleted (kafka.server.KafkaApis)
[2014-06-01 15:18:08,219] WARN [KafkaApi-2] Fetch request with correlation id
10049 from client ReplicaFetcherThread-0-2 on partition [sams_2014-05-27,8]
failed due to Topic sams_2014-05-27 either doesn't exist or is in the process
of being deleted (kafka.server.KafkaApis)
{quote}
The above is from kafkaServer.out. Also seeing errors in server.log:
{quote}
[2014-06-01 15:23:52,788] ERROR [ReplicaFetcherThread-0-0], Error for
partition [sams_2014-05-26,19] to broker 0:class
kafka.common.UnknownTopicOrPartitionException
(kafka.server.ReplicaFetcherThread)
[2014-06-01 15:23:52,788] WARN [KafkaApi-2] Fetch request with correlation id
10887 from client ReplicaFetcherThread-0-2 on partition [sams_2014-05-30,4]
failed due to Topic sams_2014-05-30 either doesn't exist or is in the process
of being deleted (kafka.server.KafkaApis)
[2014-06-01 15:23:52,788] ERROR [ReplicaFetcherThread-0-0], Error for
partition [sams_2014-05-24,34] to broker 0:class
kafka.common.UnknownTopicOrPartitionException
(kafka.server.ReplicaFetcherThread)
[2014-06-01 15:23:52,788] ERROR [ReplicaFetcherThread-0-0], Error for
partition [sams_2014-05-26,41] to broker 0:class
kafka.common.UnknownTopicOrPartitionException
(kafka.server.ReplicaFetcherThread)
[2014-06-01 15:23:52,788] WARN [KafkaApi-2] Fetch request with correlation id
10887 from client ReplicaFetcherThread-0-2 on partition [2014-05-21,0] failed
due to Topic 2014-05-21 either doesn't exist or is in the process of being
deleted (kafka.server.KafkaApis)
[2014-06-01 15:23:52,788] ERROR [ReplicaFetcherThread-0-0], Error for
partition [sams_2014-05-28,42] to broker 0:class
kafka.common.UnknownTopicOrPartitionException
(kafka.server.ReplicaFetcherThread)
[2014-06-01 15:23:52,788] ERROR [ReplicaFetcherThread-0-0], Error for
partition [sams_2014-05-22,21] to broker 0:class
kafka.common.UnknownTopicOrPartitionException
(kafka.server.ReplicaFetcherThread)
[2014-06-01 15:23:52,788] WARN [KafkaApi-2] Fetch request with correlation id
10887 from client ReplicaFetcherThread-0-2 on partition [sams_2014-05-20,26]
failed due to Topic sams_2014-05-20 either doesn't exist or is in the process
of being deleted (kafka.server.KafkaApis)
{quote}
All these partitions belong to deleted topics. Nothing changed on our end
when we started noticing these logs filling up. Any ideas what is going on?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (KAFKA-1535) return all live brokers in TopicMetadataResponse
[ https://issues.apache.org/jira/browse/KAFKA-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nicu marasoiu updated KAFKA-1535: - Attachment: (was: KAFKA-1535__return_all_live_brokers_in_TopicMetadataResponse1.patch) return all live brokers in TopicMetadataResponse Key: KAFKA-1535 URL: https://issues.apache.org/jira/browse/KAFKA-1535 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Labels: newbie Currently, we only return the brokers that have assigned replicas for a topic in TopicMetadataResponse. The new producer will use those brokers for refreshing metadata. Now suppose that we stop all those brokers, copy all local data to some new hosts and then restart those hosts (with the original broker id). There is no way for the new producer to automatically get the information about the new brokers since all old brokers are gone. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1535) return all live brokers in TopicMetadataResponse
[ https://issues.apache.org/jira/browse/KAFKA-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nicu marasoiu updated KAFKA-1535: - Attachment: KAFKA-1535__return_all_live_brokers_in_TopicMetadataResponse.patch return all live brokers in TopicMetadataResponse Key: KAFKA-1535 URL: https://issues.apache.org/jira/browse/KAFKA-1535 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Labels: newbie Attachments: KAFKA-1535__return_all_live_brokers_in_TopicMetadataResponse.patch Currently, we only return the brokers that have assigned replicas for a topic in TopicMetadataResponse. The new producer will use those brokers for refreshing metadata. Now suppose that we stop all those brokers, copy all local data to some new hosts and then restart those hosts (with the original broker id). There is no way for the new producer to automatically get the information about the new brokers since all old brokers are gone. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] Subscription: outstanding kafka patches
Issue Subscription Filter: outstanding kafka patches (109 issues) The list of outstanding kafka patches Subscriber: kafka-mailing-list Key Summary KAFKA-1543 Changing replication factor https://issues.apache.org/jira/browse/KAFKA-1543 KAFKA-1541 Add transactional request definitions to clients package https://issues.apache.org/jira/browse/KAFKA-1541 KAFKA-1536 Change the status of the JIRA to Patch Available in the kafka-review-tool https://issues.apache.org/jira/browse/KAFKA-1536 KAFKA-1535 return all live brokers in TopicMetadataResponse https://issues.apache.org/jira/browse/KAFKA-1535 KAFKA-1533 transient unit test failure in ProducerFailureHandlingTest https://issues.apache.org/jira/browse/KAFKA-1533 KAFKA-1528 Normalize all the line endings https://issues.apache.org/jira/browse/KAFKA-1528 KAFKA-1526 Producer performance tool should have an option to enable transactions https://issues.apache.org/jira/browse/KAFKA-1526 KAFKA-1525 DumpLogSegments should print transaction IDs https://issues.apache.org/jira/browse/KAFKA-1525 KAFKA-1524 Implement transactional producer https://issues.apache.org/jira/browse/KAFKA-1524 KAFKA-1523 Implement transaction manager module https://issues.apache.org/jira/browse/KAFKA-1523 KAFKA-1522 Transactional messaging request/response definitions https://issues.apache.org/jira/browse/KAFKA-1522 KAFKA-1509 Restart of destination broker after unreplicated partition move leaves partitions without leader https://issues.apache.org/jira/browse/KAFKA-1509 KAFKA-1507 Using GetOffsetShell against non-existent topic creates the topic unintentionally https://issues.apache.org/jira/browse/KAFKA-1507 KAFKA-1500 adding new consumer requests using the new protocol https://issues.apache.org/jira/browse/KAFKA-1500 KAFKA-1498 new producer performance and bug improvements https://issues.apache.org/jira/browse/KAFKA-1498 KAFKA-1496 Using batch message in sync producer only sends the first message if we use a Scala Stream as the argument https://issues.apache.org/jira/browse/KAFKA-1496 KAFKA-1483 Split Brain about Leader Partitions https://issues.apache.org/jira/browse/KAFKA-1483 KAFKA-1481 Stop using dashes AND underscores as separators in MBean names https://issues.apache.org/jira/browse/KAFKA-1481 KAFKA-1477 add authentication layer and initial JKS x509 implementation for brokers, producers and consumer for network communication https://issues.apache.org/jira/browse/KAFKA-1477 KAFKA-1475 Kafka consumer stops LeaderFinder/FetcherThreads, but application does not know https://issues.apache.org/jira/browse/KAFKA-1475 KAFKA-1471 Add Producer Unit Tests for LZ4 and LZ4HC compression https://issues.apache.org/jira/browse/KAFKA-1471 KAFKA-1468 Improve perf tests https://issues.apache.org/jira/browse/KAFKA-1468 KAFKA-1462 Add new request and response formats for the new consumer and coordinator communication https://issues.apache.org/jira/browse/KAFKA-1462 KAFKA-1460 NoReplicaOnlineException: No replica for partition https://issues.apache.org/jira/browse/KAFKA-1460 KAFKA-1450 check invalid leader in a more robust way https://issues.apache.org/jira/browse/KAFKA-1450 KAFKA-1430 Purgatory redesign https://issues.apache.org/jira/browse/KAFKA-1430 KAFKA-1414 Speedup broker startup after hard reset https://issues.apache.org/jira/browse/KAFKA-1414 KAFKA-1394 Ensure last segment isn't deleted on expiration when there are unflushed messages https://issues.apache.org/jira/browse/KAFKA-1394 KAFKA-1372 Upgrade to Gradle 1.10 https://issues.apache.org/jira/browse/KAFKA-1372 KAFKA-1367 Broker topic metadata not kept in sync with ZooKeeper https://issues.apache.org/jira/browse/KAFKA-1367 KAFKA-1351 String.format is very expensive in Scala https://issues.apache.org/jira/browse/KAFKA-1351 KAFKA-1343 Kafka consumer iterator thread stalls https://issues.apache.org/jira/browse/KAFKA-1343 KAFKA-1330 Implement subscribe(TopicPartition...partitions) in the new consumer https://issues.apache.org/jira/browse/KAFKA-1330 KAFKA-1329 Add metadata fetch and refresh functionality to the consumer https://issues.apache.org/jira/browse/KAFKA-1329 KAFKA-1324 Debian packaging https://issues.apache.org/jira/browse/KAFKA-1324 KAFKA-1303 metadata request in the new producer can be delayed https://issues.apache.org/jira/browse/KAFKA-1303 KAFKA-1300 Added WaitForReplaction admin tool. https://issues.apache.org/jira/browse/KAFKA-1300 KAFKA-1235 Enable server to indefinitely retry on controlled shutdown
[jira] [Updated] (KAFKA-1535) return all live brokers in TopicMetadataResponse
[ https://issues.apache.org/jira/browse/KAFKA-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nicu marasoiu updated KAFKA-1535: - Attachment: (was: KAFKA-1535__return_all_live_brokers_in_TopicMetadataResponse.patch) return all live brokers in TopicMetadataResponse Key: KAFKA-1535 URL: https://issues.apache.org/jira/browse/KAFKA-1535 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Labels: newbie Currently, we only return the brokers that have assigned replicas for a topic in TopicMetadataResponse. The new producer will use those brokers for refreshing metadata. Now suppose that we stop all those brokers, copy all local data to some new hosts and then restart those hosts (with the original broker id). There is no way for the new producer to automatically get the information about the new brokers since all old brokers are gone. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1535) return all live brokers in TopicMetadataResponse
[ https://issues.apache.org/jira/browse/KAFKA-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nicu marasoiu updated KAFKA-1535: - Attachment: KAFKA-1535__return_all_live_brokers_in_TopicMetadataResponse_.patch return all live brokers in TopicMetadataResponse Key: KAFKA-1535 URL: https://issues.apache.org/jira/browse/KAFKA-1535 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Labels: newbie Attachments: KAFKA-1535__return_all_live_brokers_in_TopicMetadataResponse_.patch Currently, we only return the brokers that have assigned replicas for a topic in TopicMetadataResponse. The new producer will use those brokers for refreshing metadata. Now suppose that we stop all those brokers, copy all local data to some new hosts and then restart those hosts (with the original broker id). There is no way for the new producer to automatically get the information about the new brokers since all old brokers are gone. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1535) return all live brokers in TopicMetadataResponse
[ https://issues.apache.org/jira/browse/KAFKA-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065424#comment-14065424 ] nicu marasoiu commented on KAFKA-1535: -- updated patch, tests fine, one question though, I have not find where is the topicMetadataResponse.brokers read in producers (or anywhere in the non-test code)! return all live brokers in TopicMetadataResponse Key: KAFKA-1535 URL: https://issues.apache.org/jira/browse/KAFKA-1535 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Labels: newbie Attachments: KAFKA-1535__return_all_live_brokers_in_TopicMetadataResponse_.patch Currently, we only return the brokers that have assigned replicas for a topic in TopicMetadataResponse. The new producer will use those brokers for refreshing metadata. Now suppose that we stop all those brokers, copy all local data to some new hosts and then restart those hosts (with the original broker id). There is no way for the new producer to automatically get the information about the new brokers since all old brokers are gone. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Comment Edited] (KAFKA-1535) return all live brokers in TopicMetadataResponse
[ https://issues.apache.org/jira/browse/KAFKA-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065424#comment-14065424 ] nicu marasoiu edited comment on KAFKA-1535 at 7/17/14 7:32 PM: --- updated trunk patch, tests fine, one question though, I have not find where is the topicMetadataResponse.brokers read in producers (or anywhere in the non-test code)! was (Author: nmarasoi): updated patch, tests fine, one question though, I have not find where is the topicMetadataResponse.brokers read in producers (or anywhere in the non-test code)! return all live brokers in TopicMetadataResponse Key: KAFKA-1535 URL: https://issues.apache.org/jira/browse/KAFKA-1535 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Labels: newbie Attachments: KAFKA-1535__return_all_live_brokers_in_TopicMetadataResponse_.patch Currently, we only return the brokers that have assigned replicas for a topic in TopicMetadataResponse. The new producer will use those brokers for refreshing metadata. Now suppose that we stop all those brokers, copy all local data to some new hosts and then restart those hosts (with the original broker id). There is no way for the new producer to automatically get the information about the new brokers since all old brokers are gone. -- This message was sent by Atlassian JIRA (v6.2#6252)
Review Request 23655: Provide alternate (round-robin-style) rebalance algorithms.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23655/ --- Review request for kafka. Bugs: KAFKA-687 https://issues.apache.org/jira/browse/KAFKA-687 Repository: kafka Description --- tweaks Ready to submit Diffs - core/src/main/scala/kafka/consumer/ConsumerConfig.scala 1cf2f62ba02e4aa66bfa7575865e5d57baf82212 core/src/main/scala/kafka/consumer/PartitionAllocator.scala PRE-CREATION core/src/main/scala/kafka/consumer/TopicCount.scala c79311097c5bd6718cb6a7fc403f804a1a939353 core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 65f518d47c7555c42c4bff39c211814831f4b8b6 core/src/main/scala/kafka/utils/ZkUtils.scala dcdc1ce2b02c996294e19cf480736106aaf29511 core/src/test/scala/unit/kafka/consumer/PartitionAllocatorTest.scala PRE-CREATION Diff: https://reviews.apache.org/r/23655/diff/ Testing --- Thanks, Joel Koshy
[jira] [Updated] (KAFKA-687) Rebalance algorithm should consider partitions from all topics
[ https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy updated KAFKA-687: - Attachment: KAFKA-687.patch Rebalance algorithm should consider partitions from all topics -- Key: KAFKA-687 URL: https://issues.apache.org/jira/browse/KAFKA-687 Project: Kafka Issue Type: Improvement Affects Versions: 0.9.0 Reporter: Pablo Barrera Assignee: Sriharsha Chintalapani Attachments: KAFKA-687.patch The current rebalance step, as stated in the original Kafka paper [1], splits the partitions per topic between all the consumers. So if you have 100 topics with 2 partitions each and 10 consumers only two consumers will be used. That is, for each topic all partitions will be listed and shared between the consumers in the consumer group in order (not randomly). If the consumer group is reading from several topics at the same time it makes sense to split all the partitions from all topics between all the consumer. Following the example, we will have 200 partitions in total, 20 per consumer, using the 10 consumers. The load per topic could be different and the division should consider this. However even a random division should be better than the current algorithm while reading from several topics and should harm reading from a few topics with several partitions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-687) Rebalance algorithm should consider partitions from all topics
[ https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065446#comment-14065446 ] Joel Koshy commented on KAFKA-687: -- Created reviewboard https://reviews.apache.org/r/23655/diff/ against branch origin/trunk Rebalance algorithm should consider partitions from all topics -- Key: KAFKA-687 URL: https://issues.apache.org/jira/browse/KAFKA-687 Project: Kafka Issue Type: Improvement Affects Versions: 0.9.0 Reporter: Pablo Barrera Assignee: Sriharsha Chintalapani Attachments: KAFKA-687.patch The current rebalance step, as stated in the original Kafka paper [1], splits the partitions per topic between all the consumers. So if you have 100 topics with 2 partitions each and 10 consumers only two consumers will be used. That is, for each topic all partitions will be listed and shared between the consumers in the consumer group in order (not randomly). If the consumer group is reading from several topics at the same time it makes sense to split all the partitions from all topics between all the consumer. Following the example, we will have 200 partitions in total, 20 per consumer, using the 10 consumers. The load per topic could be different and the division should consider this. However even a random division should be better than the current algorithm while reading from several topics and should harm reading from a few topics with several partitions. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 23655: Provide alternate (round-robin-style) rebalance algorithms.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23655/ --- (Updated July 17, 2014, 7:50 p.m.) Review request for kafka. Bugs: KAFKA-687 https://issues.apache.org/jira/browse/KAFKA-687 Repository: kafka Description (updated) --- The comments in the code and the summary are pretty self-explanatory. Things to think about: * Naming - do symmetric/range/roundrobin make sense? * The comments briefly summarize why we needed a separate symmetric mode but let me know if that is unclear. * Rebalance time will be slightly higher - I have not measured (will do that) * I would like to add some mbeans to show ownership counts. Diffs - core/src/main/scala/kafka/consumer/ConsumerConfig.scala 1cf2f62ba02e4aa66bfa7575865e5d57baf82212 core/src/main/scala/kafka/consumer/PartitionAllocator.scala PRE-CREATION core/src/main/scala/kafka/consumer/TopicCount.scala c79311097c5bd6718cb6a7fc403f804a1a939353 core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 65f518d47c7555c42c4bff39c211814831f4b8b6 core/src/main/scala/kafka/utils/ZkUtils.scala dcdc1ce2b02c996294e19cf480736106aaf29511 core/src/test/scala/unit/kafka/consumer/PartitionAllocatorTest.scala PRE-CREATION Diff: https://reviews.apache.org/r/23655/diff/ Testing (updated) --- * I did the unit tests (including the new one) as well as mirror maker system test suite with roundrobin. While this is being reviewed I will run the system tests with symmetric Thanks, Joel Koshy
Re: Review Request 23655: Provide alternate (round-robin-style) rebalance algorithms.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23655/#review48043 --- core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala https://reviews.apache.org/r/23655/#comment84294 (FYI: while this is being reviewed I will look into adding some mbeans to report topic-level and total owner counts for the consumer.) - Joel Koshy On July 17, 2014, 7:50 p.m., Joel Koshy wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23655/ --- (Updated July 17, 2014, 7:50 p.m.) Review request for kafka. Bugs: KAFKA-687 https://issues.apache.org/jira/browse/KAFKA-687 Repository: kafka Description --- The comments in the code and the summary are pretty self-explanatory. Things to think about: * Naming - do symmetric/range/roundrobin make sense? * The comments briefly summarize why we needed a separate symmetric mode but let me know if that is unclear. * Rebalance time will be slightly higher - I have not measured (will do that) * I would like to add some mbeans to show ownership counts. Diffs - core/src/main/scala/kafka/consumer/ConsumerConfig.scala 1cf2f62ba02e4aa66bfa7575865e5d57baf82212 core/src/main/scala/kafka/consumer/PartitionAllocator.scala PRE-CREATION core/src/main/scala/kafka/consumer/TopicCount.scala c79311097c5bd6718cb6a7fc403f804a1a939353 core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 65f518d47c7555c42c4bff39c211814831f4b8b6 core/src/main/scala/kafka/utils/ZkUtils.scala dcdc1ce2b02c996294e19cf480736106aaf29511 core/src/test/scala/unit/kafka/consumer/PartitionAllocatorTest.scala PRE-CREATION Diff: https://reviews.apache.org/r/23655/diff/ Testing --- * I did the unit tests (including the new one) as well as mirror maker system test suite with roundrobin. While this is being reviewed I will run the system tests with symmetric Thanks, Joel Koshy
Re: [DISCUSS] Kafka Security Specific Features
Thanks Raja, it was helpful Now I am able to start zookeeper and broker in secure mode ready for SSL handshake. I get *java.lang.OutOfMemoryError: Java heap space* on producer. I using the default configuration and keystore. Is there anything missing *Start broker:* *bin/kafka-server-start.sh config/server.properties* *broker.log:* [2014-07-17 15:34:46,281] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient) [2014-07-17 15:34:46,523] INFO Loading log 'secure.test-0' (kafka.log.LogManager) [2014-07-17 15:34:46,558] INFO Recovering unflushed segment 0 in log secure.test-0. (kafka.log.Log) [2014-07-17 15:34:46,571] INFO Completed load of log secure.test-0 with log end offset 0 (kafka.log.Log) [2014-07-17 15:34:46,582] INFO Starting log cleanup with a period of 6 ms. (kafka.log.LogManager) [2014-07-17 15:34:46,587] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager) [2014-07-17 15:34:46,614] INFO Initializing secure authentication (kafka.network.security.SecureAuth$) [2014-07-17 15:34:46,678] INFO Secure authentication initialization has been successfully completed (kafka.network.security.SecureAuth$) [2014-07-17 15:34:46,691] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor) [2014-07-17 15:34:46,692] INFO [Socket Server on Broker 0], Started (kafka.network.SocketServer) [2014-07-17 15:34:46,794] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$) [2014-07-17 15:34:46,837] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector) [2014-07-17 15:34:47,057] INFO Registered broker 0 at path /brokers/ids/0 with address 10.1.100.130:9092. (kafka.utils.ZkUtils$) [2014-07-17 15:34:47,059] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener) *[2014-07-17 15:34:47,068] INFO [Kafka Server 0], started (kafka.server.KafkaServer)* *[2014-07-17 15:34:47,383] INFO begin ssl handshake for /10.1.100.130:9092//10.1.100.130:51685 http://10.1.100.130:9092//10.1.100.130:51685 (kafka.network.security.SSLSocketChannel)* *[2014-07-17 15:34:47,392] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51685//10.1.100.130:9092 http://10.1.100.130/10.1.100.130:51685//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel)* *[2014-07-17 15:34:47,465] INFO finished ssl handshake for 10.1.100.130/10.1.100.130:51685//10.1.100.130:9092 http://10.1.100.130/10.1.100.130:51685//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel)* *[2014-07-17 15:34:47,465] INFO finished ssl handshake for /10.1.100.130:9092//10.1.100.130:51685 http://10.1.100.130:9092//10.1.100.130:51685 (kafka.network.security.SSLSocketChannel)* *[2014-07-17 15:34:47,617] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions (kafka.server.ReplicaFetcherManager)* *[2014-07-17 15:34:47,627] INFO [ReplicaFetcherManager on broker 0] Added fetcher for partitions List() (kafka.server.ReplicaFetcherManager)* *[2014-07-17 15:34:47,656] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [secure.test,0] (kafka.server.ReplicaFetcherManager)* [2014-07-17 15:37:15,970] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51689//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:16,075] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51690//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:16,434] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51691//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:16,530] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51692//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:16,743] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51693//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:16,834] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51694//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:17,043] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51695//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:17,137] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51696//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:17,342] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51697//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) *Start producer* *bin/kafka-console-producer.sh --broker-list 10.1.100.130:9092:true --topic secure.test* *producer.log:* bin/kafka-console-producer.sh --broker-list 10.1.100.130:9092:true --topic secure.test [2014-07-17 15:37:46,889] WARN Property topic is not valid (kafka.utils.VerifiableProperties) Hello Secure Kafka *[2014-07-17 15:38:14,186] ERROR OOME with size 352518400 (kafka.network.BoundedByteBufferReceive)* *java.lang.OutOfMemoryError: Java heap space* at
[jira] [Commented] (KAFKA-687) Rebalance algorithm should consider partitions from all topics
[ https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065457#comment-14065457 ] Joel Koshy commented on KAFKA-687: -- I ended up abandoning the earlier approach I was thinking of in the above gist and went with I think a simpler approach. The layout algorithms are the result of discussions with [~clarkhaskins] Rebalance algorithm should consider partitions from all topics -- Key: KAFKA-687 URL: https://issues.apache.org/jira/browse/KAFKA-687 Project: Kafka Issue Type: Improvement Affects Versions: 0.9.0 Reporter: Pablo Barrera Assignee: Sriharsha Chintalapani Attachments: KAFKA-687.patch The current rebalance step, as stated in the original Kafka paper [1], splits the partitions per topic between all the consumers. So if you have 100 topics with 2 partitions each and 10 consumers only two consumers will be used. That is, for each topic all partitions will be listed and shared between the consumers in the consumer group in order (not randomly). If the consumer group is reading from several topics at the same time it makes sense to split all the partitions from all topics between all the consumer. Following the example, we will have 200 partitions in total, 20 per consumer, using the 10 consumers. The load per topic could be different and the division should consider this. However even a random division should be better than the current algorithm while reading from several topics and should harm reading from a few topics with several partitions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (KAFKA-687) Rebalance algorithm should consider partitions from all topics
[ https://issues.apache.org/jira/browse/KAFKA-687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Joel Koshy reassigned KAFKA-687: Assignee: Joel Koshy (was: Sriharsha Chintalapani) Rebalance algorithm should consider partitions from all topics -- Key: KAFKA-687 URL: https://issues.apache.org/jira/browse/KAFKA-687 Project: Kafka Issue Type: Improvement Affects Versions: 0.9.0 Reporter: Pablo Barrera Assignee: Joel Koshy Attachments: KAFKA-687.patch The current rebalance step, as stated in the original Kafka paper [1], splits the partitions per topic between all the consumers. So if you have 100 topics with 2 partitions each and 10 consumers only two consumers will be used. That is, for each topic all partitions will be listed and shared between the consumers in the consumer group in order (not randomly). If the consumer group is reading from several topics at the same time it makes sense to split all the partitions from all topics between all the consumer. Following the example, we will have 200 partitions in total, 20 per consumer, using the 10 consumers. The load per topic could be different and the division should consider this. However even a random division should be better than the current algorithm while reading from several topics and should harm reading from a few topics with several partitions. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (KAFKA-1483) Split Brain about Leader Partitions
[ https://issues.apache.org/jira/browse/KAFKA-1483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede reassigned KAFKA-1483: Assignee: Neha Narkhede (was: Sriharsha Chintalapani) Assigning to myself for review Split Brain about Leader Partitions --- Key: KAFKA-1483 URL: https://issues.apache.org/jira/browse/KAFKA-1483 Project: Kafka Issue Type: Improvement Reporter: Guozhang Wang Assignee: Neha Narkhede Labels: newbie++ Fix For: 0.9.0 Attachments: KAFKA-1483.patch, KAFKA-1483_2014-07-16_11:07:44.patch Today in the server there are two places storing the leader partition info: 1) leaderPartitions list in the ReplicaManager. 2) leaderBrokerIdOpt in the Partition. 1) is used as the ground truth to decide if the server is the current leader for serving requests; 2) is used as the ground truth for reporting leader counts metrics, etc and for the background Shrinking-ISR thread to decide which partition to check. There is a risk that these two ground truth caches are not consistent, and we'd better only make one of them as the ground truth. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 23516: Patch for KAFKA-1462
On July 16, 2014, 11:09 p.m., Guozhang Wang wrote: clients/src/main/java/org/apache/kafka/common/Cluster.java, line 18 https://reviews.apache.org/r/23516/diff/1/?file=632640#file632640line18 Do we ever want to use * in imports? Jun Rao wrote: IDE did the optimization since there are too many imports from the same package. Yeah. What I am wondering though is if we want to use wildcards in our imports. We used to do so in scala files, which I think messed up some imports, especially with immutable/mutable imports. So I am wondering if we should set it as a coding convention to not use wildcards for non-kafka classes? - Guozhang --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23516/#review47940 --- On July 17, 2014, 4:39 a.m., Jun Rao wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23516/ --- (Updated July 17, 2014, 4:39 a.m.) Review request for kafka. Bugs: KAFKA-1462 https://issues.apache.org/jira/browse/KAFKA-1462 Repository: kafka Description --- address Jay's comments remove partition from all PartitionData since it's redundant minor fixes initial patch Diffs - clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java a016269512b6d6d6e0fd3fab997e9c8265024eb4 clients/src/main/java/org/apache/kafka/common/Cluster.java c62707ab3aba26771fc4b993df28bf8c44f32309 clients/src/main/java/org/apache/kafka/common/protocol/ApiKeys.java 6fe7573973832615976defa37fe0dfbb8f911939 clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 044b03061802ee5e8ea4f1995fb0988e1a70e9a7 clients/src/main/java/org/apache/kafka/common/protocol/types/Struct.java 8cecba50bf067713184208552af36469962cd628 clients/src/main/java/org/apache/kafka/common/requests/AbstractRequestResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/ConsumerMetadataRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/ConsumerMetadataResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/HeartbeatRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/HeartbeatResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/JoinGroupRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/JoinGroupResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/ListOffsetRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/ListOffsetResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/MetadataRequest.java f35bd87cf0c52a30ed779b25d60bbe64a60b9502 clients/src/main/java/org/apache/kafka/common/requests/MetadataResponse.java 2652c32f123b3bc4b0456d4bc9fbba52c051724c clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java 6036f6af1c55c1b0a15471e79b229b17f50ce31c clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java 6cf4fb714916f1a318d788cde8fc0aad9dfb83ca clients/src/main/java/org/apache/kafka/common/requests/RequestHeader.java 66cc2fea6443968e525419a203dbc4227e0b1cdf clients/src/main/java/org/apache/kafka/common/requests/ResponseHeader.java 257b8287757e40349ea041ed7a651993007a55a8 clients/src/main/java/org/apache/kafka/common/utils/CollectionUtils.java PRE-CREATION clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 2f98192b064d1ce7c0779e901293edb8c3801915 clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java PRE-CREATION core/src/main/scala/kafka/api/ConsumerMetadataRequest.scala dfad6e6534dd9b00099d110804899080e8d832ab core/src/main/scala/kafka/api/ConsumerMetadataResponse.scala c72ca14708a3625cb89d5fb92630138d2afa2bf0 core/src/main/scala/kafka/api/ControlledShutdownRequest.scala 7dacb2023788064b736df8b775aaf12281d545b5
[jira] [Created] (KAFKA-1548) Refactor the replica_id in requests
Guozhang Wang created KAFKA-1548: Summary: Refactor the replica_id in requests Key: KAFKA-1548 URL: https://issues.apache.org/jira/browse/KAFKA-1548 Project: Kafka Issue Type: Bug Reporter: Guozhang Wang Fix For: 0.9.0 Today in many requests like fetch and offset we have a integer replica_id field, if the request is from a follower consumer it is the broker id from that follower replica, if it is from a regular consumer it could be one of the two values: -1 for ordinary consumer, or -2 for debugging consumer. Hence this replica_id field is also used in two folds: 1) Logging for trouble shooting in request logs, which can be helpful only when this is from a follower replica, 2) Deciding if it is from the consumer or a replica to logically handle the request in different ways. For this purpose we do not really care about the actually id value. We probably would like to do the following improvements: 1) Rename replica_id to sth. less confusing? 2) Change the request.toString() function based on the replica_id, whether it is a positive integer (meaning from a broker replica fetcher) or -1/-2 (meaning from a regular consumer). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1476) Get a list of consumer groups
[ https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065534#comment-14065534 ] BalajiSeshadri commented on KAFKA-1476: --- I'm working on this JIRA ticket to create tool to list consumer groups based on topic provided. Get a list of consumer groups - Key: KAFKA-1476 URL: https://issues.apache.org/jira/browse/KAFKA-1476 Project: Kafka Issue Type: Wish Components: tools Affects Versions: 0.8.1.1 Reporter: Ryan Williams Labels: newbie Fix For: 0.9.0 It would be useful to have a way to get a list of consumer groups currently active via some tool/script that ships with kafka. This would be helpful so that the system tools can be explored more easily. For example, when running the ConsumerOffsetChecker, it requires a group option bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --topic test --group ? But, when just getting started with kafka, using the console producer and consumer, it is not clear what value to use for the group option. If a list of consumer groups could be listed, then it would be clear what value to use. Background: http://mail-archives.apache.org/mod_mbox/kafka-users/201405.mbox/%3cCAOq_b1w=slze5jrnakxvak0gu9ctdkpazak1g4dygvqzbsg...@mail.gmail.com%3e -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Interested in contributing to Kafka?
Hey Neha, I like all these ideas. I think people have been doing a good job with the newbie label, which is great, we should keep doing that. I think pairing people up is a great idea. The trick is actually figuring out who has an interest in becoming a committer and would like a pair, and who is just dropping a one-off-patch. I like the idea of having someone assign JIRAs with patches to committers for review. So the proposed workflow would be: 1. Have a designated triage person go through all JIRAs. If a JIRA comes in with a patch or the person says they are working on something, the triage person will assign the JIRA to a random committer to handle review. If the committer is too busy they can hand it off to another random contributor until it lands on someone who can do it. It would be nice if the committer could get to it in 2 days (even if they are just punting it to someone else) so that people don't have to wait too long. 2. The assigned committer then reviews the patch. If it needs more work they assign it back to the contributor. If it is good as is they commit it and close the JIRA. If no objections, let's use this approach for managing patches. I think it is kind of what we have been doing informally, but this will make it clear. -Jay On Thu, Jul 17, 2014 at 7:00 AM, Neha Narkhede neha.narkh...@gmail.com wrote: Thanks Jay for bringing this up. A couple things might help - 1. Be diligent in marking newbie/newbie++ labels. I've seen us do pretty well here. 2. Pair up contributors to committers for a few initial patches to ensure a smoother ramp up. I've recently done this and have seen it work pretty well. Happy to help more. 3. Jay and I talked about ways of improving patch review turnaround time. Mostly, the problem is that committers are either swamped or not sure which patches need review. What might work is to assign the JIRA to a committer for review and have the committer shepherd the patch to checkin and reassign the JIRA back to the contributor. I can help with triaging and assigning committers to patch reviews and over time most of the committers will be able to do this. Thanks, Neha On Wed, Jul 16, 2014 at 10:26 PM, pushkar priyadarshi priyadarshi.push...@gmail.com wrote: I have been using kafka for quite some time now and would really be interested to contribute to this awesome code base. Regards, Pushkar On Thu, Jul 17, 2014 at 7:17 AM, Joe Stein joe.st...@stealth.ly wrote: ./gradlew scaladoc Builds the scala doc, perhaps we can start to publish this again with the next release and link it on the website. For more related check out the README /*** Joe Stein Founder, Principal Consultant Big Data Open Source Security LLC http://www.stealth.ly Twitter: @allthingshadoop http://www.twitter.com/allthingshadoop / On Wed, Jul 16, 2014 at 8:39 PM, hsy...@gmail.com hsy...@gmail.com wrote: Is there a scala API doc for the entire kafka library? On Wed, Jul 16, 2014 at 5:34 PM, hsy...@gmail.com hsy...@gmail.com wrote: Hi Jay, I would like to take a look at the code base and maybe start working on some jiras. Best, Siyuan On Wed, Jul 16, 2014 at 3:09 PM, Jay Kreps jay.kr...@gmail.com wrote: Hey All, A number of people have been submitting really nice patches recently. If you are interested in contributing and are looking for something to work on, or if you are contributing and are interested in ramping up to be a committer on the project, please let us know--we are happy to help you help us :-). It is often hard to know what JIRAs or projects would be good to work on, how hard those will be, and where to get started. Feel free to reach out to me, Neha, Jun, or any of the other committers for help with this. Cheers, -Jay
[jira] [Commented] (KAFKA-1543) Changing replication factor
[
https://issues.apache.org/jira/browse/KAFKA-1543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065616#comment-14065616
]
Jay Kreps commented on KAFKA-1543:
--
I wonder if it would make more sense to have the alter topic command do this.
Something like:
{code}
bin/kafka-topics.sh --zookeeper host:port --alter --topic name
--replication-factor 3
{code}
Changing replication factor
---
Key: KAFKA-1543
URL: https://issues.apache.org/jira/browse/KAFKA-1543
Project: Kafka
Issue Type: Improvement
Reporter: Alexey Ozeritskiy
Attachments: can-change-replication.patch
It is difficult to change replication factor by manual editing json config.
I propose to add a key to kafka-reassign-partitions.sh command to
automatically create json config.
Example of usage
{code}
kafka-reassign-partitions.sh --zookeeper zk --replicas new-replication-factor
--topics-to-move-json-file topics-file --broker-list 1,2,3,4 --generate
output
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (KAFKA-169) Layering violations in Kafka code
[ https://issues.apache.org/jira/browse/KAFKA-169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-169. - Resolution: Won't Fix Layering violations in Kafka code - Key: KAFKA-169 URL: https://issues.apache.org/jira/browse/KAFKA-169 Project: Kafka Issue Type: Bug Components: core Reporter: Jay Kreps Priority: Minor Attachments: draw_deps.py, kafka_deps.svg I am noticing lot of layering violations creeping into the code. For example the log implementation depends on zookeeper code now, the network server depends on the kafka api, etc. This stuff is messy and makes it hard to test or reason about the pieces in isolation. I have run a quick analysis on the imports to look at problems and there are a few. Let's try to keep this graph in good shape and think about the layering in the code. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: [DISCUSS] Kafka Security Specific Features
Can you try with turning off security to check if this error happens only on secure mode? Thanks, Raja. On Thu, Jul 17, 2014 at 3:51 PM, Pramod Deshmukh dpram...@gmail.com wrote: Thanks Raja, it was helpful Now I am able to start zookeeper and broker in secure mode ready for SSL handshake. I get *java.lang.OutOfMemoryError: Java heap space* on producer. I using the default configuration and keystore. Is there anything missing *Start broker:* *bin/kafka-server-start.sh config/server.properties* *broker.log:* [2014-07-17 15:34:46,281] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient) [2014-07-17 15:34:46,523] INFO Loading log 'secure.test-0' (kafka.log.LogManager) [2014-07-17 15:34:46,558] INFO Recovering unflushed segment 0 in log secure.test-0. (kafka.log.Log) [2014-07-17 15:34:46,571] INFO Completed load of log secure.test-0 with log end offset 0 (kafka.log.Log) [2014-07-17 15:34:46,582] INFO Starting log cleanup with a period of 6 ms. (kafka.log.LogManager) [2014-07-17 15:34:46,587] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager) [2014-07-17 15:34:46,614] INFO Initializing secure authentication (kafka.network.security.SecureAuth$) [2014-07-17 15:34:46,678] INFO Secure authentication initialization has been successfully completed (kafka.network.security.SecureAuth$) [2014-07-17 15:34:46,691] INFO Awaiting socket connections on 0.0.0.0:9092 . (kafka.network.Acceptor) [2014-07-17 15:34:46,692] INFO [Socket Server on Broker 0], Started (kafka.network.SocketServer) [2014-07-17 15:34:46,794] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$) [2014-07-17 15:34:46,837] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector) [2014-07-17 15:34:47,057] INFO Registered broker 0 at path /brokers/ids/0 with address 10.1.100.130:9092. (kafka.utils.ZkUtils$) [2014-07-17 15:34:47,059] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener) *[2014-07-17 15:34:47,068] INFO [Kafka Server 0], started (kafka.server.KafkaServer)* *[2014-07-17 15:34:47,383] INFO begin ssl handshake for /10.1.100.130:9092//10.1.100.130:51685 http://10.1.100.130:9092//10.1.100.130:51685 (kafka.network.security.SSLSocketChannel)* *[2014-07-17 15:34:47,392] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51685//10.1.100.130:9092 http://10.1.100.130/10.1.100.130:51685//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel)* *[2014-07-17 15:34:47,465] INFO finished ssl handshake for 10.1.100.130/10.1.100.130:51685//10.1.100.130:9092 http://10.1.100.130/10.1.100.130:51685//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel)* *[2014-07-17 15:34:47,465] INFO finished ssl handshake for /10.1.100.130:9092//10.1.100.130:51685 http://10.1.100.130:9092//10.1.100.130:51685 (kafka.network.security.SSLSocketChannel)* *[2014-07-17 15:34:47,617] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions (kafka.server.ReplicaFetcherManager)* *[2014-07-17 15:34:47,627] INFO [ReplicaFetcherManager on broker 0] Added fetcher for partitions List() (kafka.server.ReplicaFetcherManager)* *[2014-07-17 15:34:47,656] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [secure.test,0] (kafka.server.ReplicaFetcherManager)* [2014-07-17 15:37:15,970] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51689//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:16,075] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51690//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:16,434] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51691//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:16,530] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51692//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:16,743] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51693//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:16,834] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51694//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:17,043] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51695//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:17,137] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51696//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:17,342] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51697//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) *Start producer* *bin/kafka-console-producer.sh --broker-list 10.1.100.130:9092:true --topic secure.test* *producer.log:* bin/kafka-console-producer.sh --broker-list 10.1.100.130:9092:true
Re: Review Request 23516: Patch for KAFKA-1462
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23516/#review48071 --- I am not sure why we need to remove correlationId from the ReuqestOrResponse in order to do ser/deser. Shouldn't we just add correlation_id to the AbstractRequestResponse java class? core/src/main/scala/kafka/controller/ControllerChannelManager.scala https://reviews.apache.org/r/23516/#comment84348 Should we use toString or describe(false) here? core/src/main/scala/kafka/controller/ControllerChannelManager.scala https://reviews.apache.org/r/23516/#comment84349 Ditto above. - Guozhang Wang On July 17, 2014, 4:39 a.m., Jun Rao wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23516/ --- (Updated July 17, 2014, 4:39 a.m.) Review request for kafka. Bugs: KAFKA-1462 https://issues.apache.org/jira/browse/KAFKA-1462 Repository: kafka Description --- address Jay's comments remove partition from all PartitionData since it's redundant minor fixes initial patch Diffs - clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java a016269512b6d6d6e0fd3fab997e9c8265024eb4 clients/src/main/java/org/apache/kafka/common/Cluster.java c62707ab3aba26771fc4b993df28bf8c44f32309 clients/src/main/java/org/apache/kafka/common/protocol/ApiKeys.java 6fe7573973832615976defa37fe0dfbb8f911939 clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 044b03061802ee5e8ea4f1995fb0988e1a70e9a7 clients/src/main/java/org/apache/kafka/common/protocol/types/Struct.java 8cecba50bf067713184208552af36469962cd628 clients/src/main/java/org/apache/kafka/common/requests/AbstractRequestResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/ConsumerMetadataRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/ConsumerMetadataResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/HeartbeatRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/HeartbeatResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/JoinGroupRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/JoinGroupResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/ListOffsetRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/ListOffsetResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/MetadataRequest.java f35bd87cf0c52a30ed779b25d60bbe64a60b9502 clients/src/main/java/org/apache/kafka/common/requests/MetadataResponse.java 2652c32f123b3bc4b0456d4bc9fbba52c051724c clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java 6036f6af1c55c1b0a15471e79b229b17f50ce31c clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java 6cf4fb714916f1a318d788cde8fc0aad9dfb83ca clients/src/main/java/org/apache/kafka/common/requests/RequestHeader.java 66cc2fea6443968e525419a203dbc4227e0b1cdf clients/src/main/java/org/apache/kafka/common/requests/ResponseHeader.java 257b8287757e40349ea041ed7a651993007a55a8 clients/src/main/java/org/apache/kafka/common/utils/CollectionUtils.java PRE-CREATION clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 2f98192b064d1ce7c0779e901293edb8c3801915 clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java PRE-CREATION core/src/main/scala/kafka/api/ConsumerMetadataRequest.scala dfad6e6534dd9b00099d110804899080e8d832ab core/src/main/scala/kafka/api/ConsumerMetadataResponse.scala c72ca14708a3625cb89d5fb92630138d2afa2bf0 core/src/main/scala/kafka/api/ControlledShutdownRequest.scala 7dacb2023788064b736df8b775aaf12281d545b5 core/src/main/scala/kafka/api/ControlledShutdownResponse.scala 46ec3db28f88bbf9e0b0de2133807dc552bcae13 core/src/main/scala/kafka/api/FetchRequest.scala
[jira] [Commented] (KAFKA-1535) return all live brokers in TopicMetadataResponse
[ https://issues.apache.org/jira/browse/KAFKA-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065646#comment-14065646 ] Jay Kreps commented on KAFKA-1535: -- Our current model of nodes is that they are permanent. That is if there is a node 1, if it dies, it will come back or be replaced. It need not literally be the same machine, just that if a node dies you will eventually add a new node with id 1 which will take over the work 1 used to do. The metadata response is read by the producer and consumer clients. For example in the new java code it is in clients/src/main/java/org/apache/kafka/common/requests/MetadataResponse.java. return all live brokers in TopicMetadataResponse Key: KAFKA-1535 URL: https://issues.apache.org/jira/browse/KAFKA-1535 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Labels: newbie Attachments: KAFKA-1535__return_all_live_brokers_in_TopicMetadataResponse_.patch Currently, we only return the brokers that have assigned replicas for a topic in TopicMetadataResponse. The new producer will use those brokers for refreshing metadata. Now suppose that we stop all those brokers, copy all local data to some new hosts and then restart those hosts (with the original broker id). There is no way for the new producer to automatically get the information about the new brokers since all old brokers are gone. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: [DISCUSS] Kafka Security Specific Features
Correct, I don't see any exceptions when i turn off security. Consumer is able to consume the message. I still see warning for topic property. [2014-07-17 18:04:38,360] WARN Property topic is not valid (kafka.utils.VerifiableProperties) On Thu, Jul 17, 2014 at 5:49 PM, Rajasekar Elango rela...@salesforce.com wrote: Can you try with turning off security to check if this error happens only on secure mode? Thanks, Raja. On Thu, Jul 17, 2014 at 3:51 PM, Pramod Deshmukh dpram...@gmail.com wrote: Thanks Raja, it was helpful Now I am able to start zookeeper and broker in secure mode ready for SSL handshake. I get *java.lang.OutOfMemoryError: Java heap space* on producer. I using the default configuration and keystore. Is there anything missing *Start broker:* *bin/kafka-server-start.sh config/server.properties* *broker.log:* [2014-07-17 15:34:46,281] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient) [2014-07-17 15:34:46,523] INFO Loading log 'secure.test-0' (kafka.log.LogManager) [2014-07-17 15:34:46,558] INFO Recovering unflushed segment 0 in log secure.test-0. (kafka.log.Log) [2014-07-17 15:34:46,571] INFO Completed load of log secure.test-0 with log end offset 0 (kafka.log.Log) [2014-07-17 15:34:46,582] INFO Starting log cleanup with a period of 6 ms. (kafka.log.LogManager) [2014-07-17 15:34:46,587] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager) [2014-07-17 15:34:46,614] INFO Initializing secure authentication (kafka.network.security.SecureAuth$) [2014-07-17 15:34:46,678] INFO Secure authentication initialization has been successfully completed (kafka.network.security.SecureAuth$) [2014-07-17 15:34:46,691] INFO Awaiting socket connections on 0.0.0.0:9092 . (kafka.network.Acceptor) [2014-07-17 15:34:46,692] INFO [Socket Server on Broker 0], Started (kafka.network.SocketServer) [2014-07-17 15:34:46,794] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$) [2014-07-17 15:34:46,837] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector) [2014-07-17 15:34:47,057] INFO Registered broker 0 at path /brokers/ids/0 with address 10.1.100.130:9092. (kafka.utils.ZkUtils$) [2014-07-17 15:34:47,059] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener) *[2014-07-17 15:34:47,068] INFO [Kafka Server 0], started (kafka.server.KafkaServer)* *[2014-07-17 15:34:47,383] INFO begin ssl handshake for /10.1.100.130:9092//10.1.100.130:51685 http://10.1.100.130:9092//10.1.100.130:51685 (kafka.network.security.SSLSocketChannel)* *[2014-07-17 15:34:47,392] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51685//10.1.100.130:9092 http://10.1.100.130/10.1.100.130:51685//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel)* *[2014-07-17 15:34:47,465] INFO finished ssl handshake for 10.1.100.130/10.1.100.130:51685//10.1.100.130:9092 http://10.1.100.130/10.1.100.130:51685//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel)* *[2014-07-17 15:34:47,465] INFO finished ssl handshake for /10.1.100.130:9092//10.1.100.130:51685 http://10.1.100.130:9092//10.1.100.130:51685 (kafka.network.security.SSLSocketChannel)* *[2014-07-17 15:34:47,617] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions (kafka.server.ReplicaFetcherManager)* *[2014-07-17 15:34:47,627] INFO [ReplicaFetcherManager on broker 0] Added fetcher for partitions List() (kafka.server.ReplicaFetcherManager)* *[2014-07-17 15:34:47,656] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [secure.test,0] (kafka.server.ReplicaFetcherManager)* [2014-07-17 15:37:15,970] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51689//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:16,075] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51690//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:16,434] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51691//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:16,530] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51692//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:16,743] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51693//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:16,834] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51694//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:17,043] INFO begin ssl handshake for 10.1.100.130/10.1.100.130:51695//10.1.100.130:9092 (kafka.network.security.SSLSocketChannel) [2014-07-17 15:37:17,137] INFO begin ssl handshake
[jira] [Commented] (KAFKA-1070) Auto-assign node id
[ https://issues.apache.org/jira/browse/KAFKA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065670#comment-14065670 ] Sriharsha Chintalapani commented on KAFKA-1070: --- [~jkreps] [~nehanarkhede] I am working on this JIRA. Following the proposed design in the comments. I see there might be a issue where one of the broker configs have broker.id defined as 0 and another one doesn't have broker.id. If the second broker started first we fetch a global sequence id from zookeeper that starts with 0 and the next one has already have a defined config we will use that in this case we have two brokers with same id. Should we consider this case in which a kafka cluster's broker might have a inconsistent config interms of broker.id. Instead of using zookeeper for generating a global sequence number why shouldn't we be using UUID and make brokerId Long type. Thanks. Auto-assign node id --- Key: KAFKA-1070 URL: https://issues.apache.org/jira/browse/KAFKA-1070 Project: Kafka Issue Type: Bug Reporter: Jay Kreps Assignee: Sriharsha Chintalapani Labels: usability It would be nice to have Kafka brokers auto-assign node ids rather than having that be a configuration. Having a configuration is irritating because (1) you have to generate a custom config for each broker and (2) even though it is in configuration, changing the node id can cause all kinds of bad things to happen. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1070) Auto-assign node id
[ https://issues.apache.org/jira/browse/KAFKA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065702#comment-14065702 ] Jay Kreps commented on KAFKA-1070: -- Hey [~harsha_ch], that is a great point. I'd like to avoid changing the type of the id to a long or UUID so as to not have to bump up the protocol format for the metadata request which hands these out to clients (we would need a way to handle compatibility with older clients that don't expect the longer types). I think we can get around the problem you point out by just defaulting the node id sequence to 1000. This could theoretically conflict but most people number from 0 or 1 and we can discuss this in the release notes. Our plan will be to release with support for both configured node ids and assigned node ids for compatibility. After a couple of releases we will remove the config. So the behavior would be this: If there is a node id in the config we will validate it against the node id in the data directory If it matches that good, we'll use that. If it doesn't match that is bad, we'll crash with an error. If there is a node id in the data directory but none in the config, we'll use whatever is in the data directory. If there is no node id in the data directory yet but there is one in the config we'll write that to the data directory and use it. If there is neither a node id in the data directory nor in the config we'll allocate a node id and write it to the data directory. Auto-assign node id --- Key: KAFKA-1070 URL: https://issues.apache.org/jira/browse/KAFKA-1070 Project: Kafka Issue Type: Bug Reporter: Jay Kreps Assignee: Sriharsha Chintalapani Labels: usability It would be nice to have Kafka brokers auto-assign node ids rather than having that be a configuration. Having a configuration is irritating because (1) you have to generate a custom config for each broker and (2) even though it is in configuration, changing the node id can cause all kinds of bad things to happen. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1414) Speedup broker startup after hard reset
[
https://issues.apache.org/jira/browse/KAFKA-1414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065721#comment-14065721
]
Jay Kreps commented on KAFKA-1414:
--
Hey [~aozeritsky], any before/after perf numbers for your setup?
[~ataraxer] How many threads were being used when that out of memory error
occurred? My understanding is that that happens when java requests memory from
the OS and the OS is physically out of memory and not willing to give virtual
memory. Can you confirm that this is a reproducible thing? If so we may need to
kind of warn people about that...however it is somewhat counterintuitive that
on a machine with sufficient memory calling flush, say, from 4 threads would
crash the process.
Speedup broker startup after hard reset
---
Key: KAFKA-1414
URL: https://issues.apache.org/jira/browse/KAFKA-1414
Project: Kafka
Issue Type: Improvement
Components: log
Affects Versions: 0.8.2, 0.9.0, 0.8.1.1
Reporter: Dmitry Bugaychenko
Assignee: Jay Kreps
Attachments:
0001-KAFKA-1414-Speedup-broker-startup-after-hard-reset-a.patch,
KAFKA-1414-rev1.patch, parallel-dir-loading-0.8.patch,
parallel-dir-loading-trunk-fixed-threadpool.patch,
parallel-dir-loading-trunk-threadpool.patch, parallel-dir-loading-trunk.patch
After hard reset due to power failure broker takes way too much time
recovering unflushed segments in a single thread. This could be easiliy
improved launching multiple threads (one per data dirrectory, assuming that
typically each data directory is on a dedicated drive). Localy we trie this
simple patch to LogManager.loadLogs and it seems to work, however I'm too new
to scala, so do not take it literally:
{code}
/**
* Recover and load all logs in the given data directories
*/
private def loadLogs(dirs: Seq[File]) {
val threads : Array[Thread] = new Array[Thread](dirs.size)
var i: Int = 0
val me = this
for(dir - dirs) {
val thread = new Thread( new Runnable {
def run()
{
val recoveryPoints = me.recoveryPointCheckpoints(dir).read
/* load the logs */
val subDirs = dir.listFiles()
if(subDirs != null) {
val cleanShutDownFile = new File(dir, Log.CleanShutdownFile)
if(cleanShutDownFile.exists())
info(Found clean shutdown file. Skipping recovery for all logs
in data directory '%s'.format(dir.getAbsolutePath))
for(dir - subDirs) {
if(dir.isDirectory) {
info(Loading log ' + dir.getName + ')
val topicPartition = Log.parseTopicPartitionName(dir.getName)
val config = topicConfigs.getOrElse(topicPartition.topic,
defaultConfig)
val log = new Log(dir,
config,
recoveryPoints.getOrElse(topicPartition, 0L),
scheduler,
time)
val previous = addLogWithLock(topicPartition, log)
if(previous != null)
throw new IllegalArgumentException(Duplicate log
directories found: %s, %s!.format(log.dir.getAbsolutePath,
previous.dir.getAbsolutePath))
}
}
cleanShutDownFile.delete()
}
}
})
thread.start()
threads(i) = thread
i = i + 1
}
for(thread - threads) {
thread.join()
}
}
def addLogWithLock(topicPartition: TopicAndPartition, log: Log): Log = {
logCreationOrDeletionLock synchronized {
this.logs.put(topicPartition, log)
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (KAFKA-1535) return all live brokers in TopicMetadataResponse
[ https://issues.apache.org/jira/browse/KAFKA-1535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps updated KAFKA-1535: - Resolution: Fixed Assignee: Jay Kreps Status: Resolved (was: Patch Available) Patch looks good to me! return all live brokers in TopicMetadataResponse Key: KAFKA-1535 URL: https://issues.apache.org/jira/browse/KAFKA-1535 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Assignee: Jay Kreps Labels: newbie Attachments: KAFKA-1535__return_all_live_brokers_in_TopicMetadataResponse_.patch Currently, we only return the brokers that have assigned replicas for a topic in TopicMetadataResponse. The new producer will use those brokers for refreshing metadata. Now suppose that we stop all those brokers, copy all local data to some new hosts and then restart those hosts (with the original broker id). There is no way for the new producer to automatically get the information about the new brokers since all old brokers are gone. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1476) Get a list of consumer groups
[ https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BalajiSeshadri updated KAFKA-1476: -- Status: Patch Available (was: Open) Created ConsumerGroupLister class that lists all ConsumerGroups related to topic if no topic is provided it shows all consumer groups. Get a list of consumer groups - Key: KAFKA-1476 URL: https://issues.apache.org/jira/browse/KAFKA-1476 Project: Kafka Issue Type: Wish Components: tools Affects Versions: 0.8.1.1 Reporter: Ryan Williams Labels: newbie Fix For: 0.9.0 Attachments: KAFKA-1476.patch It would be useful to have a way to get a list of consumer groups currently active via some tool/script that ships with kafka. This would be helpful so that the system tools can be explored more easily. For example, when running the ConsumerOffsetChecker, it requires a group option bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --topic test --group ? But, when just getting started with kafka, using the console producer and consumer, it is not clear what value to use for the group option. If a list of consumer groups could be listed, then it would be clear what value to use. Background: http://mail-archives.apache.org/mod_mbox/kafka-users/201405.mbox/%3cCAOq_b1w=slze5jrnakxvak0gu9ctdkpazak1g4dygvqzbsg...@mail.gmail.com%3e -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1476) Get a list of consumer groups
[ https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BalajiSeshadri updated KAFKA-1476: -- Attachment: KAFKA-1476.patch ConsumerGroupLister.scala Get a list of consumer groups - Key: KAFKA-1476 URL: https://issues.apache.org/jira/browse/KAFKA-1476 Project: Kafka Issue Type: Wish Components: tools Affects Versions: 0.8.1.1 Reporter: Ryan Williams Labels: newbie Fix For: 0.9.0 Attachments: KAFKA-1476.patch It would be useful to have a way to get a list of consumer groups currently active via some tool/script that ships with kafka. This would be helpful so that the system tools can be explored more easily. For example, when running the ConsumerOffsetChecker, it requires a group option bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --topic test --group ? But, when just getting started with kafka, using the console producer and consumer, it is not clear what value to use for the group option. If a list of consumer groups could be listed, then it would be clear what value to use. Background: http://mail-archives.apache.org/mod_mbox/kafka-users/201405.mbox/%3cCAOq_b1w=slze5jrnakxvak0gu9ctdkpazak1g4dygvqzbsg...@mail.gmail.com%3e -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1476) Get a list of consumer groups
[ https://issues.apache.org/jira/browse/KAFKA-1476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] BalajiSeshadri updated KAFKA-1476: -- Attachment: (was: ConsumerGroupLister.scala) Get a list of consumer groups - Key: KAFKA-1476 URL: https://issues.apache.org/jira/browse/KAFKA-1476 Project: Kafka Issue Type: Wish Components: tools Affects Versions: 0.8.1.1 Reporter: Ryan Williams Labels: newbie Fix For: 0.9.0 Attachments: KAFKA-1476.patch It would be useful to have a way to get a list of consumer groups currently active via some tool/script that ships with kafka. This would be helpful so that the system tools can be explored more easily. For example, when running the ConsumerOffsetChecker, it requires a group option bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --topic test --group ? But, when just getting started with kafka, using the console producer and consumer, it is not clear what value to use for the group option. If a list of consumer groups could be listed, then it would be clear what value to use. Background: http://mail-archives.apache.org/mod_mbox/kafka-users/201405.mbox/%3cCAOq_b1w=slze5jrnakxvak0gu9ctdkpazak1g4dygvqzbsg...@mail.gmail.com%3e -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (KAFKA-183) Expose offset vector to the consumer
[ https://issues.apache.org/jira/browse/KAFKA-183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jay Kreps resolved KAFKA-183. - Resolution: Fixed This is being done in the new consumer. Expose offset vector to the consumer Key: KAFKA-183 URL: https://issues.apache.org/jira/browse/KAFKA-183 Project: Kafka Issue Type: Sub-task Reporter: Jay Kreps Assignee: Jay Kreps We should enable consumers to save their position themselves. This would be useful for consumers that need to store consumed data so they can store the data and the position together, this gives a poor man's transactionality since any data loss on the consumer will also rewind the position to the previous position so the two are always in sync. Two ways to do this: 1. Add an OffsetStorage interface and have the zk storage implement this. The user can override this by providing an OffsetStorage implementation of their own to change how values are stored. 2. Make commit() return the position offset vector and add a setPosition(ListLong) method to initialize the position. Let's figure out any potential problems with this, and work out the best approach. -- This message was sent by Atlassian JIRA (v6.2#6252)
Build failed in Jenkins: Kafka-trunk #225
See https://builds.apache.org/job/Kafka-trunk/225/changes Changes: [jay.kreps] KAFKA-1535 Have the metadata response contain all alive brokers rather than just the ones needed for the given topics. -- [...truncated 503 lines...] org.apache.kafka.common.record.RecordTest testEquality[55] PASSED org.apache.kafka.common.record.RecordTest testFields[56] PASSED org.apache.kafka.common.record.RecordTest testChecksum[56] PASSED org.apache.kafka.common.record.RecordTest testEquality[56] PASSED org.apache.kafka.common.record.RecordTest testFields[57] PASSED org.apache.kafka.common.record.RecordTest testChecksum[57] PASSED org.apache.kafka.common.record.RecordTest testEquality[57] PASSED org.apache.kafka.common.record.RecordTest testFields[58] PASSED org.apache.kafka.common.record.RecordTest testChecksum[58] PASSED org.apache.kafka.common.record.RecordTest testEquality[58] PASSED org.apache.kafka.common.record.RecordTest testFields[59] PASSED org.apache.kafka.common.record.RecordTest testChecksum[59] PASSED org.apache.kafka.common.record.RecordTest testEquality[59] PASSED org.apache.kafka.common.record.RecordTest testFields[60] PASSED org.apache.kafka.common.record.RecordTest testChecksum[60] PASSED org.apache.kafka.common.record.RecordTest testEquality[60] PASSED org.apache.kafka.common.record.RecordTest testFields[61] PASSED org.apache.kafka.common.record.RecordTest testChecksum[61] PASSED org.apache.kafka.common.record.RecordTest testEquality[61] PASSED org.apache.kafka.common.record.RecordTest testFields[62] PASSED org.apache.kafka.common.record.RecordTest testChecksum[62] PASSED org.apache.kafka.common.record.RecordTest testEquality[62] PASSED org.apache.kafka.common.record.RecordTest testFields[63] PASSED org.apache.kafka.common.record.RecordTest testChecksum[63] PASSED org.apache.kafka.common.record.RecordTest testEquality[63] PASSED org.apache.kafka.common.record.RecordTest testFields[64] PASSED org.apache.kafka.common.record.RecordTest testChecksum[64] PASSED org.apache.kafka.common.record.RecordTest testEquality[64] PASSED org.apache.kafka.common.record.RecordTest testFields[65] PASSED org.apache.kafka.common.record.RecordTest testChecksum[65] PASSED org.apache.kafka.common.record.RecordTest testEquality[65] PASSED org.apache.kafka.common.record.RecordTest testFields[66] PASSED org.apache.kafka.common.record.RecordTest testChecksum[66] PASSED org.apache.kafka.common.record.RecordTest testEquality[66] PASSED org.apache.kafka.common.record.RecordTest testFields[67] PASSED org.apache.kafka.common.record.RecordTest testChecksum[67] PASSED org.apache.kafka.common.record.RecordTest testEquality[67] PASSED org.apache.kafka.common.record.RecordTest testFields[68] PASSED org.apache.kafka.common.record.RecordTest testChecksum[68] PASSED org.apache.kafka.common.record.RecordTest testEquality[68] PASSED org.apache.kafka.common.record.RecordTest testFields[69] PASSED org.apache.kafka.common.record.RecordTest testChecksum[69] PASSED org.apache.kafka.common.record.RecordTest testEquality[69] PASSED org.apache.kafka.common.record.RecordTest testFields[70] PASSED org.apache.kafka.common.record.RecordTest testChecksum[70] PASSED org.apache.kafka.common.record.RecordTest testEquality[70] PASSED org.apache.kafka.common.record.RecordTest testFields[71] PASSED org.apache.kafka.common.record.RecordTest testChecksum[71] PASSED org.apache.kafka.common.record.RecordTest testEquality[71] PASSED org.apache.kafka.common.record.RecordTest testFields[72] PASSED org.apache.kafka.common.record.RecordTest testChecksum[72] PASSED org.apache.kafka.common.record.RecordTest testEquality[72] PASSED org.apache.kafka.common.record.RecordTest testFields[73] PASSED org.apache.kafka.common.record.RecordTest testChecksum[73] PASSED org.apache.kafka.common.record.RecordTest testEquality[73] PASSED org.apache.kafka.common.record.RecordTest testFields[74] PASSED org.apache.kafka.common.record.RecordTest testChecksum[74] PASSED org.apache.kafka.common.record.RecordTest testEquality[74] PASSED org.apache.kafka.common.record.RecordTest testFields[75] PASSED org.apache.kafka.common.record.RecordTest testChecksum[75] PASSED org.apache.kafka.common.record.RecordTest testEquality[75] PASSED org.apache.kafka.common.record.RecordTest testFields[76] PASSED org.apache.kafka.common.record.RecordTest testChecksum[76] PASSED org.apache.kafka.common.record.RecordTest testEquality[76] PASSED org.apache.kafka.common.record.RecordTest testFields[77] PASSED org.apache.kafka.common.record.RecordTest testChecksum[77] PASSED org.apache.kafka.common.record.RecordTest testEquality[77] PASSED org.apache.kafka.common.record.RecordTest testFields[78] PASSED org.apache.kafka.common.record.RecordTest testChecksum[78] PASSED org.apache.kafka.common.record.RecordTest testEquality[78] PASSED
[jira] [Commented] (KAFKA-1546) Automate replica lag tuning
[ https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065806#comment-14065806 ] Jay Kreps commented on KAFKA-1546: -- I think this is actually a really important thing to get right to make replication reliable. There are some subtleties. It would be good to work out the basics of how this could work on this JIRA. For example the throughput on a partition might be 1 msg/sec. But that is because only 1 msg/sec is being written by the producer. However if someone writes a batch of 1000 messages, that doesn't mean we are necessarily 1000 seconds behind. We already track the time since the last fetch request. So if the fetcher stops entirely for too long it will be caught. I think the other condition we want to be able to catch is one where the fetcher is still fetching but it is behind and likely won't catch up. One way to make caught-up concrete is to say that the last fetch went to the end of the log. We potentially reduce this to one config and just have replica.lag.time.ms which would both be the maximum time since a fetch or the maximum amount of time without catching up to the leader. The implementation would be that every time a fetch didn't go to the logEndOffset we would set the lag clock and it would only reset when a fetch request finally went all the way to the logEndOffset. Automate replica lag tuning --- Key: KAFKA-1546 URL: https://issues.apache.org/jira/browse/KAFKA-1546 Project: Kafka Issue Type: Improvement Components: replication Affects Versions: 0.8.0, 0.8.1, 0.8.1.1 Reporter: Neha Narkhede Labels: newbie++ Currently, there is no good way to tune the replica lag configs to automatically account for high and low volume topics on the same cluster. For the low-volume topic it will take a very long time to detect a lagging replica, and for the high-volume topic it will have false-positives. One approach to making this easier would be to have the configuration be something like replica.lag.max.ms and translate this into a number of messages dynamically based on the throughput of the partition. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 23516: Patch for KAFKA-1462
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23516/#review48092 --- Ship it! Ship It! - Jay Kreps On July 17, 2014, 4:39 a.m., Jun Rao wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23516/ --- (Updated July 17, 2014, 4:39 a.m.) Review request for kafka. Bugs: KAFKA-1462 https://issues.apache.org/jira/browse/KAFKA-1462 Repository: kafka Description --- address Jay's comments remove partition from all PartitionData since it's redundant minor fixes initial patch Diffs - clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java a016269512b6d6d6e0fd3fab997e9c8265024eb4 clients/src/main/java/org/apache/kafka/common/Cluster.java c62707ab3aba26771fc4b993df28bf8c44f32309 clients/src/main/java/org/apache/kafka/common/protocol/ApiKeys.java 6fe7573973832615976defa37fe0dfbb8f911939 clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 044b03061802ee5e8ea4f1995fb0988e1a70e9a7 clients/src/main/java/org/apache/kafka/common/protocol/types/Struct.java 8cecba50bf067713184208552af36469962cd628 clients/src/main/java/org/apache/kafka/common/requests/AbstractRequestResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/ConsumerMetadataRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/ConsumerMetadataResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/FetchRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/FetchResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/HeartbeatRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/HeartbeatResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/JoinGroupRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/JoinGroupResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/ListOffsetRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/ListOffsetResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/MetadataRequest.java f35bd87cf0c52a30ed779b25d60bbe64a60b9502 clients/src/main/java/org/apache/kafka/common/requests/MetadataResponse.java 2652c32f123b3bc4b0456d4bc9fbba52c051724c clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchRequest.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/OffsetFetchResponse.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/requests/ProduceRequest.java 6036f6af1c55c1b0a15471e79b229b17f50ce31c clients/src/main/java/org/apache/kafka/common/requests/ProduceResponse.java 6cf4fb714916f1a318d788cde8fc0aad9dfb83ca clients/src/main/java/org/apache/kafka/common/requests/RequestHeader.java 66cc2fea6443968e525419a203dbc4227e0b1cdf clients/src/main/java/org/apache/kafka/common/requests/ResponseHeader.java 257b8287757e40349ea041ed7a651993007a55a8 clients/src/main/java/org/apache/kafka/common/utils/CollectionUtils.java PRE-CREATION clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 2f98192b064d1ce7c0779e901293edb8c3801915 clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java PRE-CREATION core/src/main/scala/kafka/api/ConsumerMetadataRequest.scala dfad6e6534dd9b00099d110804899080e8d832ab core/src/main/scala/kafka/api/ConsumerMetadataResponse.scala c72ca14708a3625cb89d5fb92630138d2afa2bf0 core/src/main/scala/kafka/api/ControlledShutdownRequest.scala 7dacb2023788064b736df8b775aaf12281d545b5 core/src/main/scala/kafka/api/ControlledShutdownResponse.scala 46ec3db28f88bbf9e0b0de2133807dc552bcae13 core/src/main/scala/kafka/api/FetchRequest.scala a8b73acd1a813284744359e8434cb52d22063c99 core/src/main/scala/kafka/api/GenericRequestOrResponseAndHeader.scala PRE-CREATION core/src/main/scala/kafka/api/HeartbeatRequestAndHeader.scala PRE-CREATION core/src/main/scala/kafka/api/HeartbeatResponseAndHeader.scala PRE-CREATION core/src/main/scala/kafka/api/JoinGroupRequestAndHeader.scala PRE-CREATION core/src/main/scala/kafka/api/JoinGroupResponseAndHeader.scala PRE-CREATION core/src/main/scala/kafka/api/LeaderAndIsrRequest.scala
[jira] [Resolved] (KAFKA-1462) Add new request and response formats for the new consumer and coordinator communication
[ https://issues.apache.org/jira/browse/KAFKA-1462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-1462. Resolution: Fixed Fix Version/s: (was: 0.9.0) 0.8.2 Thanks for the reviews. Committed to trunk. Add new request and response formats for the new consumer and coordinator communication --- Key: KAFKA-1462 URL: https://issues.apache.org/jira/browse/KAFKA-1462 Project: Kafka Issue Type: Sub-task Components: consumer Reporter: Guozhang Wang Assignee: Jun Rao Fix For: 0.8.2 Attachments: KAFKA-1462.patch, KAFKA-1462_2014-07-16_21:39:07.patch We need to add the request / response formats according to the new format protocol once their design is final: https://cwiki.apache.org/confluence/display/KAFKA /Kafka+0.9+Consumer+Rewrite+Design -- This message was sent by Atlassian JIRA (v6.2#6252)
Build failed in Jenkins: Kafka-trunk #226
See https://builds.apache.org/job/Kafka-trunk/226/changes Changes: [junrao] kafka-1462; Add new request and response formats for the new consumer and coordinator communication; patched by Jun Rao; reviewed by Guozhang Wang and Jay Kreps -- [...truncated 1612 lines...] kafka.log.LogManagerTest testCleanupExpiredSegments PASSED kafka.log.LogManagerTest testCleanupSegmentsToMaintainSize PASSED kafka.log.LogManagerTest testTimeBasedFlush PASSED kafka.log.LogManagerTest testLeastLoadedAssignment PASSED kafka.log.LogManagerTest testTwoLogManagersUsingSameDirFails PASSED kafka.log.LogManagerTest testCheckpointRecoveryPoints PASSED kafka.log.LogManagerTest testRecoveryDirectoryMappingWithTrailingSlash PASSED kafka.log.LogManagerTest testRecoveryDirectoryMappingWithRelativeDirectory PASSED kafka.log.FileMessageSetTest testTruncate PASSED kafka.log.FileMessageSetTest testWrittenEqualsRead PASSED kafka.log.FileMessageSetTest testIteratorIsConsistent PASSED kafka.log.FileMessageSetTest testSizeInBytes PASSED kafka.log.FileMessageSetTest testWriteTo PASSED kafka.log.FileMessageSetTest testFileSize PASSED kafka.log.FileMessageSetTest testIterationOverPartialAndTruncation PASSED kafka.log.FileMessageSetTest testIterationDoesntChangePosition PASSED kafka.log.FileMessageSetTest testRead PASSED kafka.log.FileMessageSetTest testSearch PASSED kafka.log.FileMessageSetTest testIteratorWithLimits PASSED kafka.log4j.KafkaLog4jAppenderTest testKafkaLog4jConfigs PASSED kafka.log4j.KafkaLog4jAppenderTest testLog4jAppends PASSED kafka.zk.ZKEphemeralTest testEphemeralNodeCleanup PASSED kafka.producer.AsyncProducerTest testProducerQueueSize PASSED kafka.producer.AsyncProducerTest testProduceAfterClosed PASSED kafka.producer.AsyncProducerTest testBatchSize PASSED kafka.producer.AsyncProducerTest testQueueTimeExpired PASSED kafka.producer.AsyncProducerTest testPartitionAndCollateEvents PASSED kafka.producer.AsyncProducerTest testSerializeEvents PASSED kafka.producer.AsyncProducerTest testInvalidPartition PASSED kafka.producer.AsyncProducerTest testNoBroker PASSED kafka.producer.AsyncProducerTest testIncompatibleEncoder PASSED kafka.producer.AsyncProducerTest testRandomPartitioner PASSED kafka.producer.AsyncProducerTest testFailedSendRetryLogic PASSED kafka.producer.AsyncProducerTest testJavaProducer PASSED kafka.producer.AsyncProducerTest testInvalidConfiguration PASSED kafka.producer.ProducerTest testUpdateBrokerPartitionInfo PASSED kafka.producer.ProducerTest testSendToNewTopic PASSED kafka.producer.ProducerTest testSendWithDeadBroker PASSED kafka.producer.ProducerTest testAsyncSendCanCorrectlyFailWithTimeout PASSED kafka.producer.ProducerTest testSendNullMessage PASSED kafka.producer.SyncProducerTest testReachableServer PASSED kafka.producer.SyncProducerTest testEmptyProduceRequest PASSED kafka.producer.SyncProducerTest testMessageSizeTooLarge PASSED kafka.producer.SyncProducerTest testMessageSizeTooLargeWithAckZero PASSED kafka.producer.SyncProducerTest testProduceCorrectlyReceivesResponse PASSED kafka.producer.SyncProducerTest testProducerCanTimeout PASSED kafka.producer.SyncProducerTest testProduceRequestWithNoResponse PASSED kafka.network.SocketServerTest simpleRequest PASSED kafka.network.SocketServerTest tooBigRequestIsRejected PASSED kafka.network.SocketServerTest testNullResponse PASSED kafka.network.SocketServerTest testSocketsCloseOnShutdown PASSED kafka.network.SocketServerTest testMaxConnectionsPerIp PASSED kafka.javaapi.consumer.ZookeeperConsumerConnectorTest testBasic PASSED kafka.javaapi.message.ByteBufferMessageSetTest testWrittenEqualsRead PASSED kafka.javaapi.message.ByteBufferMessageSetTest testIteratorIsConsistent PASSED kafka.javaapi.message.ByteBufferMessageSetTest testSizeInBytes PASSED kafka.javaapi.message.ByteBufferMessageSetTest testIteratorIsConsistentWithCompression PASSED kafka.javaapi.message.ByteBufferMessageSetTest testSizeInBytesWithCompression PASSED kafka.javaapi.message.ByteBufferMessageSetTest testEquals PASSED kafka.javaapi.message.ByteBufferMessageSetTest testEqualsWithCompression PASSED kafka.api.RequestResponseSerializationTest testSerializationAndDeserialization PASSED kafka.api.ApiUtilsTest testShortStringNonASCII PASSED kafka.api.ApiUtilsTest testShortStringASCII PASSED kafka.api.ProducerFailureHandlingTest testInvalidPartition PASSED kafka.api.ProducerFailureHandlingTest testTooLargeRecordWithAckZero PASSED kafka.api.ProducerFailureHandlingTest testTooLargeRecordWithAckOne PASSED kafka.api.ProducerFailureHandlingTest testNonExistentTopic PASSED kafka.api.ProducerFailureHandlingTest testWrongBrokerList PASSED kafka.api.ProducerFailureHandlingTest testNoResponse PASSED kafka.api.ProducerFailureHandlingTest testSendAfterClosed PASSED kafka.api.ProducerFailureHandlingTest testBrokerFailure PASSED
Re: Review Request 23568: Patch for KAFKA-1523
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23568/ --- (Updated July 18, 2014, 2:26 a.m.) Review request for kafka. Bugs: KAFKA-1523 https://issues.apache.org/jira/browse/KAFKA-1523 Repository: kafka Description (updated) --- KAFKA-1523 transaction manager module (version 2) Diffs (updated) - core/src/main/scala/kafka/admin/TopicCommand.scala 8d5c2e7088fc6e8bf69e775ea7f5893b94580fdf core/src/main/scala/kafka/common/Topic.scala ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 core/src/main/scala/kafka/controller/ControllerChannelManager.scala 8763968fbff697e4c5c98ab1274627c192a4d26a core/src/main/scala/kafka/message/Message.scala d2a7293c7be4022af30884330924791340acc5c1 core/src/main/scala/kafka/server/KafkaApis.scala 0b668f230c8556fdf08654ce522a11847d0bf39b core/src/main/scala/kafka/server/KafkaConfig.scala ef75b67b67676ae5b8931902cbc8c0c2cc72c0d3 core/src/main/scala/kafka/server/KafkaServer.scala c22e51e0412843ec993721ad3230824c0aadd2ba core/src/main/scala/kafka/server/TransactionManager.scala PRE-CREATION core/src/main/scala/kafka/utils/ZkUtils.scala dcdc1ce2b02c996294e19cf480736106aaf29511 Diff: https://reviews.apache.org/r/23568/diff/ Testing --- Thanks, Dong Lin
[jira] [Updated] (KAFKA-1523) Implement transaction manager module
[ https://issues.apache.org/jira/browse/KAFKA-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-1523: Attachment: KAFKA-1523_2014-07-17_19:26:34.patch Implement transaction manager module Key: KAFKA-1523 URL: https://issues.apache.org/jira/browse/KAFKA-1523 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Dong Lin Labels: transactions Attachments: KAFKA-1523.patch, KAFKA-1523_2014-07-17_19:26:34.patch * Entry point for transaction requests * Appends transaction control records to the transaction journal * Sends transaction control records to data brokers * Responsible for expiring transactions * Supports fail-over: for which it needs to maintain a transaction HW which is the offset of the BEGIN control record of the earliest pending transaction. It should checkpoint the HW periodically either to ZK/separate topic/offset commit. The scope of this ticket will be the basic transaction coordinator functionality. E.g., we can just have a basic transaction manager that can begin/commit/abort transactions. No expiration, no fail-over. We will add failure handling in subsequent jiras. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1523) Implement transaction manager module
[ https://issues.apache.org/jira/browse/KAFKA-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065912#comment-14065912 ] Dong Lin commented on KAFKA-1523: - Updated reviewboard https://reviews.apache.org/r/23568/diff/ against branch origin/transactional_messaging Implement transaction manager module Key: KAFKA-1523 URL: https://issues.apache.org/jira/browse/KAFKA-1523 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Dong Lin Labels: transactions Attachments: KAFKA-1523.patch, KAFKA-1523_2014-07-17_19:26:34.patch * Entry point for transaction requests * Appends transaction control records to the transaction journal * Sends transaction control records to data brokers * Responsible for expiring transactions * Supports fail-over: for which it needs to maintain a transaction HW which is the offset of the BEGIN control record of the earliest pending transaction. It should checkpoint the HW periodically either to ZK/separate topic/offset commit. The scope of this ticket will be the basic transaction coordinator functionality. E.g., we can just have a basic transaction manager that can begin/commit/abort transactions. No expiration, no fail-over. We will add failure handling in subsequent jiras. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 23567: Patch for KAFKA-1522
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23567/ --- (Updated July 18, 2014, 2:30 a.m.) Review request for kafka. Bugs: KAFKA-1522 https://issues.apache.org/jira/browse/KAFKA-1522 Repository: kafka Description (updated) --- KAFKA-1522 Tansactional messaging request/response definitions (version 2) Diffs (updated) - core/src/main/scala/kafka/admin/TopicCommand.scala 8d5c2e7088fc6e8bf69e775ea7f5893b94580fdf core/src/main/scala/kafka/api/RequestKeys.scala fbfc9d3aeaffed4ca85902125fcc1050086835db core/src/main/scala/kafka/api/TransactionRequest.scala PRE-CREATION core/src/main/scala/kafka/api/TransactionResponse.scala PRE-CREATION core/src/main/scala/kafka/api/TxCoordinatorMetadataRequest.scala PRE-CREATION core/src/main/scala/kafka/api/TxCoordinatorMetadataResponse.scala PRE-CREATION core/src/main/scala/kafka/common/ErrorMapping.scala 5559d26ba2b96059f719754a351fa4598ca8a70b core/src/main/scala/kafka/common/Topic.scala ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 core/src/main/scala/kafka/controller/ControllerChannelManager.scala 8763968fbff697e4c5c98ab1274627c192a4d26a core/src/main/scala/kafka/message/Message.scala d2a7293c7be4022af30884330924791340acc5c1 core/src/main/scala/kafka/server/KafkaApis.scala 0b668f230c8556fdf08654ce522a11847d0bf39b core/src/main/scala/kafka/server/KafkaConfig.scala ef75b67b67676ae5b8931902cbc8c0c2cc72c0d3 core/src/main/scala/kafka/server/KafkaServer.scala c22e51e0412843ec993721ad3230824c0aadd2ba core/src/main/scala/kafka/server/TransactionManager.scala PRE-CREATION core/src/main/scala/kafka/utils/ZkUtils.scala dcdc1ce2b02c996294e19cf480736106aaf29511 Diff: https://reviews.apache.org/r/23567/diff/ Testing --- Thanks, Dong Lin
[jira] [Commented] (KAFKA-1522) Transactional messaging request/response definitions
[ https://issues.apache.org/jira/browse/KAFKA-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065918#comment-14065918 ] Dong Lin commented on KAFKA-1522: - Updated reviewboard against branch origin/transactional_messaging Transactional messaging request/response definitions Key: KAFKA-1522 URL: https://issues.apache.org/jira/browse/KAFKA-1522 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Dong Lin Labels: transactions Attachments: KAFKA-1522_2014-07-17_19:30:29.patch * Add the TransactionRequest * Add TransactionResponse * Add transaction-id field to OffsetCommitRequest * Add error code for errors such as non-initiated transaction (this will be used for the ProducerResponse and OffsetCommitResponse - say if a producer sends a request and its messages have a transaction ID that has not been initiated. * TxCoordinatorMetadataRequest (to look up the transaction coordinator for a transaction group) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1522) Transactional messaging request/response definitions
[ https://issues.apache.org/jira/browse/KAFKA-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-1522: Attachment: (was: KAFKA-1522_2014-07-17_19:30:39.patch) Transactional messaging request/response definitions Key: KAFKA-1522 URL: https://issues.apache.org/jira/browse/KAFKA-1522 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Dong Lin Labels: transactions Attachments: KAFKA-1522_2014-07-17_19:30:29.patch * Add the TransactionRequest * Add TransactionResponse * Add transaction-id field to OffsetCommitRequest * Add error code for errors such as non-initiated transaction (this will be used for the ProducerResponse and OffsetCommitResponse - say if a producer sends a request and its messages have a transaction ID that has not been initiated. * TxCoordinatorMetadataRequest (to look up the transaction coordinator for a transaction group) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1522) Transactional messaging request/response definitions
[ https://issues.apache.org/jira/browse/KAFKA-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-1522: Attachment: KAFKA-1522_2014-07-17_19:30:39.patch Transactional messaging request/response definitions Key: KAFKA-1522 URL: https://issues.apache.org/jira/browse/KAFKA-1522 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Dong Lin Labels: transactions Attachments: KAFKA-1522_2014-07-17_19:30:29.patch * Add the TransactionRequest * Add TransactionResponse * Add transaction-id field to OffsetCommitRequest * Add error code for errors such as non-initiated transaction (this will be used for the ProducerResponse and OffsetCommitResponse - say if a producer sends a request and its messages have a transaction ID that has not been initiated. * TxCoordinatorMetadataRequest (to look up the transaction coordinator for a transaction group) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1522) Transactional messaging request/response definitions
[ https://issues.apache.org/jira/browse/KAFKA-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-1522: Attachment: (was: KAFKA-1522.patch) Transactional messaging request/response definitions Key: KAFKA-1522 URL: https://issues.apache.org/jira/browse/KAFKA-1522 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Dong Lin Labels: transactions Attachments: KAFKA-1522_2014-07-17_19:30:29.patch * Add the TransactionRequest * Add TransactionResponse * Add transaction-id field to OffsetCommitRequest * Add error code for errors such as non-initiated transaction (this will be used for the ProducerResponse and OffsetCommitResponse - say if a producer sends a request and its messages have a transaction ID that has not been initiated. * TxCoordinatorMetadataRequest (to look up the transaction coordinator for a transaction group) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1522) Transactional messaging request/response definitions
[ https://issues.apache.org/jira/browse/KAFKA-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-1522: Attachment: KAFKA-1522_2014-07-17_19:30:29.patch Transactional messaging request/response definitions Key: KAFKA-1522 URL: https://issues.apache.org/jira/browse/KAFKA-1522 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Dong Lin Labels: transactions Attachments: KAFKA-1522_2014-07-17_19:30:29.patch * Add the TransactionRequest * Add TransactionResponse * Add transaction-id field to OffsetCommitRequest * Add error code for errors such as non-initiated transaction (this will be used for the ProducerResponse and OffsetCommitResponse - say if a producer sends a request and its messages have a transaction ID that has not been initiated. * TxCoordinatorMetadataRequest (to look up the transaction coordinator for a transaction group) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1522) Transactional messaging request/response definitions
[ https://issues.apache.org/jira/browse/KAFKA-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065919#comment-14065919 ] Dong Lin commented on KAFKA-1522: - Updated reviewboard https://reviews.apache.org/r/23567/diff/ against branch origin/transactional_messaging Transactional messaging request/response definitions Key: KAFKA-1522 URL: https://issues.apache.org/jira/browse/KAFKA-1522 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Dong Lin Labels: transactions Attachments: KAFKA-1522_2014-07-17_19:30:29.patch * Add the TransactionRequest * Add TransactionResponse * Add transaction-id field to OffsetCommitRequest * Add error code for errors such as non-initiated transaction (this will be used for the ProducerResponse and OffsetCommitResponse - say if a producer sends a request and its messages have a transaction ID that has not been initiated. * TxCoordinatorMetadataRequest (to look up the transaction coordinator for a transaction group) -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 23567: Patch for KAFKA-1522
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23567/ --- (Updated July 18, 2014, 2:36 a.m.) Review request for kafka. Bugs: KAFKA-1522 https://issues.apache.org/jira/browse/KAFKA-1522 Repository: kafka Description --- KAFKA-1522 Tansactional messaging request/response definitions (version 2) This patch involves changes for transaction manager by mistake. Will re-upload. Diffs - core/src/main/scala/kafka/admin/TopicCommand.scala 8d5c2e7088fc6e8bf69e775ea7f5893b94580fdf core/src/main/scala/kafka/api/RequestKeys.scala fbfc9d3aeaffed4ca85902125fcc1050086835db core/src/main/scala/kafka/api/TransactionRequest.scala PRE-CREATION core/src/main/scala/kafka/api/TransactionResponse.scala PRE-CREATION core/src/main/scala/kafka/api/TxCoordinatorMetadataRequest.scala PRE-CREATION core/src/main/scala/kafka/api/TxCoordinatorMetadataResponse.scala PRE-CREATION core/src/main/scala/kafka/common/ErrorMapping.scala 5559d26ba2b96059f719754a351fa4598ca8a70b core/src/main/scala/kafka/common/Topic.scala ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 core/src/main/scala/kafka/controller/ControllerChannelManager.scala 8763968fbff697e4c5c98ab1274627c192a4d26a core/src/main/scala/kafka/message/Message.scala d2a7293c7be4022af30884330924791340acc5c1 core/src/main/scala/kafka/server/KafkaApis.scala 0b668f230c8556fdf08654ce522a11847d0bf39b core/src/main/scala/kafka/server/KafkaConfig.scala ef75b67b67676ae5b8931902cbc8c0c2cc72c0d3 core/src/main/scala/kafka/server/KafkaServer.scala c22e51e0412843ec993721ad3230824c0aadd2ba core/src/main/scala/kafka/server/TransactionManager.scala PRE-CREATION core/src/main/scala/kafka/utils/ZkUtils.scala dcdc1ce2b02c996294e19cf480736106aaf29511 Diff: https://reviews.apache.org/r/23567/diff/ Testing --- Thanks, Dong Lin
Re: Review Request 23567: Patch for KAFKA-1522
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23567/ --- (Updated July 18, 2014, 2:36 a.m.) Review request for kafka. Bugs: KAFKA-1522 https://issues.apache.org/jira/browse/KAFKA-1522 Repository: kafka Description (updated) --- KAFKA-1522 Tansactional messaging request/response definitions (version 2) This patch involves changes for transaction manager by mistake. Will re-upload. Diffs - core/src/main/scala/kafka/admin/TopicCommand.scala 8d5c2e7088fc6e8bf69e775ea7f5893b94580fdf core/src/main/scala/kafka/api/RequestKeys.scala fbfc9d3aeaffed4ca85902125fcc1050086835db core/src/main/scala/kafka/api/TransactionRequest.scala PRE-CREATION core/src/main/scala/kafka/api/TransactionResponse.scala PRE-CREATION core/src/main/scala/kafka/api/TxCoordinatorMetadataRequest.scala PRE-CREATION core/src/main/scala/kafka/api/TxCoordinatorMetadataResponse.scala PRE-CREATION core/src/main/scala/kafka/common/ErrorMapping.scala 5559d26ba2b96059f719754a351fa4598ca8a70b core/src/main/scala/kafka/common/Topic.scala ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 core/src/main/scala/kafka/controller/ControllerChannelManager.scala 8763968fbff697e4c5c98ab1274627c192a4d26a core/src/main/scala/kafka/message/Message.scala d2a7293c7be4022af30884330924791340acc5c1 core/src/main/scala/kafka/server/KafkaApis.scala 0b668f230c8556fdf08654ce522a11847d0bf39b core/src/main/scala/kafka/server/KafkaConfig.scala ef75b67b67676ae5b8931902cbc8c0c2cc72c0d3 core/src/main/scala/kafka/server/KafkaServer.scala c22e51e0412843ec993721ad3230824c0aadd2ba core/src/main/scala/kafka/server/TransactionManager.scala PRE-CREATION core/src/main/scala/kafka/utils/ZkUtils.scala dcdc1ce2b02c996294e19cf480736106aaf29511 Diff: https://reviews.apache.org/r/23567/diff/ Testing --- Thanks, Dong Lin
Re: Review Request 23567: Patch for KAFKA-1522
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23567/ --- (Updated July 18, 2014, 2:36 a.m.) Review request for kafka. Bugs: KAFKA-1522 https://issues.apache.org/jira/browse/KAFKA-1522 Repository: kafka Description --- KAFKA-1522 Tansactional messaging request/response definitions (version 2) This patch involves changes for transaction manager by mistake. Will re-upload. Diffs - core/src/main/scala/kafka/admin/TopicCommand.scala 8d5c2e7088fc6e8bf69e775ea7f5893b94580fdf core/src/main/scala/kafka/api/RequestKeys.scala fbfc9d3aeaffed4ca85902125fcc1050086835db core/src/main/scala/kafka/api/TransactionRequest.scala PRE-CREATION core/src/main/scala/kafka/api/TransactionResponse.scala PRE-CREATION core/src/main/scala/kafka/api/TxCoordinatorMetadataRequest.scala PRE-CREATION core/src/main/scala/kafka/api/TxCoordinatorMetadataResponse.scala PRE-CREATION core/src/main/scala/kafka/common/ErrorMapping.scala 5559d26ba2b96059f719754a351fa4598ca8a70b core/src/main/scala/kafka/common/Topic.scala ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 core/src/main/scala/kafka/controller/ControllerChannelManager.scala 8763968fbff697e4c5c98ab1274627c192a4d26a core/src/main/scala/kafka/message/Message.scala d2a7293c7be4022af30884330924791340acc5c1 core/src/main/scala/kafka/server/KafkaApis.scala 0b668f230c8556fdf08654ce522a11847d0bf39b core/src/main/scala/kafka/server/KafkaConfig.scala ef75b67b67676ae5b8931902cbc8c0c2cc72c0d3 core/src/main/scala/kafka/server/KafkaServer.scala c22e51e0412843ec993721ad3230824c0aadd2ba core/src/main/scala/kafka/server/TransactionManager.scala PRE-CREATION core/src/main/scala/kafka/utils/ZkUtils.scala dcdc1ce2b02c996294e19cf480736106aaf29511 Diff: https://reviews.apache.org/r/23567/diff/ Testing --- Thanks, Dong Lin
Re: Review Request 23567: Patch for KAFKA-1522
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23567/ --- (Updated July 18, 2014, 2:37 a.m.) Review request for kafka. Bugs: KAFKA-1522 https://issues.apache.org/jira/browse/KAFKA-1522 Repository: kafka Description (updated) --- KAFKA-1522 Tansactional messaging request/response definitions (version 2) This patch involves changes for another ticket. Will re-upload. Diffs - core/src/main/scala/kafka/admin/TopicCommand.scala 8d5c2e7088fc6e8bf69e775ea7f5893b94580fdf core/src/main/scala/kafka/api/RequestKeys.scala fbfc9d3aeaffed4ca85902125fcc1050086835db core/src/main/scala/kafka/api/TransactionRequest.scala PRE-CREATION core/src/main/scala/kafka/api/TransactionResponse.scala PRE-CREATION core/src/main/scala/kafka/api/TxCoordinatorMetadataRequest.scala PRE-CREATION core/src/main/scala/kafka/api/TxCoordinatorMetadataResponse.scala PRE-CREATION core/src/main/scala/kafka/common/ErrorMapping.scala 5559d26ba2b96059f719754a351fa4598ca8a70b core/src/main/scala/kafka/common/Topic.scala ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 core/src/main/scala/kafka/controller/ControllerChannelManager.scala 8763968fbff697e4c5c98ab1274627c192a4d26a core/src/main/scala/kafka/message/Message.scala d2a7293c7be4022af30884330924791340acc5c1 core/src/main/scala/kafka/server/KafkaApis.scala 0b668f230c8556fdf08654ce522a11847d0bf39b core/src/main/scala/kafka/server/KafkaConfig.scala ef75b67b67676ae5b8931902cbc8c0c2cc72c0d3 core/src/main/scala/kafka/server/KafkaServer.scala c22e51e0412843ec993721ad3230824c0aadd2ba core/src/main/scala/kafka/server/TransactionManager.scala PRE-CREATION core/src/main/scala/kafka/utils/ZkUtils.scala dcdc1ce2b02c996294e19cf480736106aaf29511 Diff: https://reviews.apache.org/r/23567/diff/ Testing --- Thanks, Dong Lin
Re: Review Request 23567: Patch for KAFKA-1522
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23567/ --- (Updated July 18, 2014, 2:38 a.m.) Review request for kafka. Bugs: KAFKA-1522 https://issues.apache.org/jira/browse/KAFKA-1522 Repository: kafka Description (updated) --- KAFKA-1522 Tansactional messaging request/response definitions (version 2) Diffs (updated) - core/src/main/scala/kafka/api/RequestKeys.scala fbfc9d3aeaffed4ca85902125fcc1050086835db core/src/main/scala/kafka/api/TransactionRequest.scala PRE-CREATION core/src/main/scala/kafka/api/TransactionResponse.scala PRE-CREATION core/src/main/scala/kafka/api/TxCoordinatorMetadataRequest.scala PRE-CREATION core/src/main/scala/kafka/api/TxCoordinatorMetadataResponse.scala PRE-CREATION core/src/main/scala/kafka/common/ErrorMapping.scala 5559d26ba2b96059f719754a351fa4598ca8a70b Diff: https://reviews.apache.org/r/23567/diff/ Testing --- Thanks, Dong Lin
[jira] [Commented] (KAFKA-1522) Transactional messaging request/response definitions
[ https://issues.apache.org/jira/browse/KAFKA-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065926#comment-14065926 ] Dong Lin commented on KAFKA-1522: - Updated reviewboard https://reviews.apache.org/r/23567/diff/ against branch origin/transactional_messaging Transactional messaging request/response definitions Key: KAFKA-1522 URL: https://issues.apache.org/jira/browse/KAFKA-1522 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Dong Lin Labels: transactions Attachments: KAFKA-1522_2014-07-17_19:30:29.patch, KAFKA-1522_2014-07-17_19:38:36.patch * Add the TransactionRequest * Add TransactionResponse * Add transaction-id field to OffsetCommitRequest * Add error code for errors such as non-initiated transaction (this will be used for the ProducerResponse and OffsetCommitResponse - say if a producer sends a request and its messages have a transaction ID that has not been initiated. * TxCoordinatorMetadataRequest (to look up the transaction coordinator for a transaction group) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1522) Transactional messaging request/response definitions
[ https://issues.apache.org/jira/browse/KAFKA-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-1522: Attachment: (was: KAFKA-1522_2014-07-17_19:30:29.patch) Transactional messaging request/response definitions Key: KAFKA-1522 URL: https://issues.apache.org/jira/browse/KAFKA-1522 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Dong Lin Labels: transactions Attachments: KAFKA-1522_2014-07-17_19:38:36.patch * Add the TransactionRequest * Add TransactionResponse * Add transaction-id field to OffsetCommitRequest * Add error code for errors such as non-initiated transaction (this will be used for the ProducerResponse and OffsetCommitResponse - say if a producer sends a request and its messages have a transaction ID that has not been initiated. * TxCoordinatorMetadataRequest (to look up the transaction coordinator for a transaction group) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1523) Implement transaction manager module
[ https://issues.apache.org/jira/browse/KAFKA-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-1523: Attachment: (was: KAFKA-1523_2014-07-17_19:26:34.patch) Implement transaction manager module Key: KAFKA-1523 URL: https://issues.apache.org/jira/browse/KAFKA-1523 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Dong Lin Labels: transactions Attachments: KAFKA-1523.patch * Entry point for transaction requests * Appends transaction control records to the transaction journal * Sends transaction control records to data brokers * Responsible for expiring transactions * Supports fail-over: for which it needs to maintain a transaction HW which is the offset of the BEGIN control record of the earliest pending transaction. It should checkpoint the HW periodically either to ZK/separate topic/offset commit. The scope of this ticket will be the basic transaction coordinator functionality. E.g., we can just have a basic transaction manager that can begin/commit/abort transactions. No expiration, no fail-over. We will add failure handling in subsequent jiras. -- This message was sent by Atlassian JIRA (v6.2#6252)
Re: Review Request 23521: Fix KAFKA-1533
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23521/#review48102 --- clients/src/main/java/org/apache/kafka/clients/NetworkClient.java https://reviews.apache.org/r/23521/#comment84403 how about renaming this to metadataLastUpdateAttemptMs vs MS to conform with convention? - Neha Narkhede On July 15, 2014, 8:40 p.m., Guozhang Wang wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23521/ --- (Updated July 15, 2014, 8:40 p.m.) Review request for kafka. Bugs: KAFKA-1533 https://issues.apache.org/jira/browse/KAFKA-1533 Repository: kafka Description --- 1. Add the metadataRefreshAttemptMS in NetworkClient for backing off; 2. Refactor Producer API tests using KafkaTestHarness; 3. Change default backoff time to 100ms for test utils Diffs - clients/src/main/java/org/apache/kafka/clients/NetworkClient.java d8f9ce663ee24d2b0852c974136741280c39f8f8 clients/src/main/java/org/apache/kafka/clients/producer/internals/Metadata.java 4aa5b01d611631db72df47d50bbe30edb8c478db core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 15fd5bcbaf175a0f7d7cf0b142e63f705ca9b6ae core/src/test/scala/integration/kafka/api/ProducerSendTest.scala 34a7db4b4ea2b720476c2b1f22a623a997faffbc core/src/test/scala/unit/kafka/integration/KafkaServerTestHarness.scala 194dd70919a5f301d3131c56594e40a0ebb27311 core/src/test/scala/unit/kafka/utils/TestUtils.scala 3faa884f8eb83c7c00baab416d0acfb488dc39c1 Diff: https://reviews.apache.org/r/23521/diff/ Testing --- Thanks, Guozhang Wang
Re: Review Request 23568: Patch for KAFKA-1523
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23568/ --- (Updated July 18, 2014, 3:01 a.m.) Review request for kafka. Bugs: KAFKA-1523 https://issues.apache.org/jira/browse/KAFKA-1523 Repository: kafka Description --- KAFKA-1523 transaction manager module (version 2) Diffs (updated) - core/src/main/scala/kafka/admin/TopicCommand.scala 8d5c2e7088fc6e8bf69e775ea7f5893b94580fdf core/src/main/scala/kafka/common/Topic.scala ad759786d1c22f67c47808c0b8f227eb2b1a9aa8 core/src/main/scala/kafka/controller/ControllerChannelManager.scala 8763968fbff697e4c5c98ab1274627c192a4d26a core/src/main/scala/kafka/message/Message.scala d2a7293c7be4022af30884330924791340acc5c1 core/src/main/scala/kafka/server/KafkaApis.scala 0b668f230c8556fdf08654ce522a11847d0bf39b core/src/main/scala/kafka/server/KafkaConfig.scala ef75b67b67676ae5b8931902cbc8c0c2cc72c0d3 core/src/main/scala/kafka/server/KafkaServer.scala c22e51e0412843ec993721ad3230824c0aadd2ba core/src/main/scala/kafka/server/TransactionManager.scala PRE-CREATION core/src/main/scala/kafka/utils/ZkUtils.scala dcdc1ce2b02c996294e19cf480736106aaf29511 Diff: https://reviews.apache.org/r/23568/diff/ Testing --- Thanks, Dong Lin
[jira] [Commented] (KAFKA-1523) Implement transaction manager module
[ https://issues.apache.org/jira/browse/KAFKA-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065952#comment-14065952 ] Dong Lin commented on KAFKA-1523: - python kafka-patch-review.py incorrectly involves needed changes... Need to do it again. Implement transaction manager module Key: KAFKA-1523 URL: https://issues.apache.org/jira/browse/KAFKA-1523 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Dong Lin Labels: transactions Attachments: KAFKA-1523.patch, KAFKA-1523_2014-07-17_20:01:21.patch * Entry point for transaction requests * Appends transaction control records to the transaction journal * Sends transaction control records to data brokers * Responsible for expiring transactions * Supports fail-over: for which it needs to maintain a transaction HW which is the offset of the BEGIN control record of the earliest pending transaction. It should checkpoint the HW periodically either to ZK/separate topic/offset commit. The scope of this ticket will be the basic transaction coordinator functionality. E.g., we can just have a basic transaction manager that can begin/commit/abort transactions. No expiration, no fail-over. We will add failure handling in subsequent jiras. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (KAFKA-1523) Implement transaction manager module
[ https://issues.apache.org/jira/browse/KAFKA-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14065948#comment-14065948 ] Dong Lin commented on KAFKA-1523: - Updated reviewboard https://reviews.apache.org/r/23568/diff/ against branch origin/transactional_messaging Implement transaction manager module Key: KAFKA-1523 URL: https://issues.apache.org/jira/browse/KAFKA-1523 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Dong Lin Labels: transactions Attachments: KAFKA-1523.patch, KAFKA-1523_2014-07-17_20:01:21.patch * Entry point for transaction requests * Appends transaction control records to the transaction journal * Sends transaction control records to data brokers * Responsible for expiring transactions * Supports fail-over: for which it needs to maintain a transaction HW which is the offset of the BEGIN control record of the earliest pending transaction. It should checkpoint the HW periodically either to ZK/separate topic/offset commit. The scope of this ticket will be the basic transaction coordinator functionality. E.g., we can just have a basic transaction manager that can begin/commit/abort transactions. No expiration, no fail-over. We will add failure handling in subsequent jiras. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1523) Implement transaction manager module
[ https://issues.apache.org/jira/browse/KAFKA-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-1523: Attachment: KAFKA-1523_2014-07-17_20:01:21.patch Implement transaction manager module Key: KAFKA-1523 URL: https://issues.apache.org/jira/browse/KAFKA-1523 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Dong Lin Labels: transactions Attachments: KAFKA-1523.patch, KAFKA-1523_2014-07-17_20:01:21.patch * Entry point for transaction requests * Appends transaction control records to the transaction journal * Sends transaction control records to data brokers * Responsible for expiring transactions * Supports fail-over: for which it needs to maintain a transaction HW which is the offset of the BEGIN control record of the earliest pending transaction. It should checkpoint the HW periodically either to ZK/separate topic/offset commit. The scope of this ticket will be the basic transaction coordinator functionality. E.g., we can just have a basic transaction manager that can begin/commit/abort transactions. No expiration, no fail-over. We will add failure handling in subsequent jiras. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1523) Implement transaction manager module
[ https://issues.apache.org/jira/browse/KAFKA-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-1523: Attachment: (was: KAFKA-1523_2014-07-17_20:01:21.patch) Implement transaction manager module Key: KAFKA-1523 URL: https://issues.apache.org/jira/browse/KAFKA-1523 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Dong Lin Labels: transactions * Entry point for transaction requests * Appends transaction control records to the transaction journal * Sends transaction control records to data brokers * Responsible for expiring transactions * Supports fail-over: for which it needs to maintain a transaction HW which is the offset of the BEGIN control record of the earliest pending transaction. It should checkpoint the HW periodically either to ZK/separate topic/offset commit. The scope of this ticket will be the basic transaction coordinator functionality. E.g., we can just have a basic transaction manager that can begin/commit/abort transactions. No expiration, no fail-over. We will add failure handling in subsequent jiras. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1523) Implement transaction manager module
[ https://issues.apache.org/jira/browse/KAFKA-1523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-1523: Attachment: (was: KAFKA-1523.patch) Implement transaction manager module Key: KAFKA-1523 URL: https://issues.apache.org/jira/browse/KAFKA-1523 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Dong Lin Labels: transactions * Entry point for transaction requests * Appends transaction control records to the transaction journal * Sends transaction control records to data brokers * Responsible for expiring transactions * Supports fail-over: for which it needs to maintain a transaction HW which is the offset of the BEGIN control record of the earliest pending transaction. It should checkpoint the HW periodically either to ZK/separate topic/offset commit. The scope of this ticket will be the basic transaction coordinator functionality. E.g., we can just have a basic transaction manager that can begin/commit/abort transactions. No expiration, no fail-over. We will add failure handling in subsequent jiras. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (KAFKA-1522) Transactional messaging request/response definitions
[ https://issues.apache.org/jira/browse/KAFKA-1522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated KAFKA-1522: Attachment: (was: KAFKA-1522_2014-07-17_19:38:36.patch) Transactional messaging request/response definitions Key: KAFKA-1522 URL: https://issues.apache.org/jira/browse/KAFKA-1522 Project: Kafka Issue Type: New Feature Reporter: Joel Koshy Assignee: Dong Lin Labels: transactions * Add the TransactionRequest * Add TransactionResponse * Add transaction-id field to OffsetCommitRequest * Add error code for errors such as non-initiated transaction (this will be used for the ProducerResponse and OffsetCommitResponse - say if a producer sends a request and its messages have a transaction ID that has not been initiated. * TxCoordinatorMetadataRequest (to look up the transaction coordinator for a transaction group) -- This message was sent by Atlassian JIRA (v6.2#6252)
