Hi,
We have recently studied the problem of load balancing in distributed
stream processing systems such as Samza [1].
In particular, we focused on what happens when the key distribution of the
stream is skewed when using key grouping.
We developed a new stream partitioning scheme (which we call
Thanks for the update.
10. About whether to return a new field in the response to indicate
throttling. Earlier, the plan was to not change the response format and
just have a metric on the broker to indicate whether a clientId is
throttled or not. The issue is that we don't know whether a
Update, I added a proposal on doing dynamic client based configuration that can
be used for quotas.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-13+-+Quotas
Please take a look and let me know if there are any concerns.
Thanks,
Aditya
From:
Gianmarco,
I am coming from storm community. I think PKG is a very
interesting and we can provide an implementation of Partitioner for PKG. Can
you open a JIRA for this.
--
Harsha
Sent with Airmail
On April 3, 2015 at 4:49:15 AM, Gianmarco De Francisci Morales
[
https://issues.apache.org/jira/browse/KAFKA-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Malaska updated KAFKA-1961:
---
Attachment: KAFKA-1961-6.patch
Updated based on comments
Looks like its possible to delete
Jun and Jay,
Have addressed all the comments and put up another patch set. Can you
please take a look again? Really appreciate it.
Thanks.
Tong Li
OpenStack Kafka Community Development
Building 501/B205
liton...@us.ibm.com
This is fantastic.
A couple of minor things:
1. I think we use commas rather than semicolons for list item separators in
config
2. I would advocate for adding the return flag when we next bump the
request format version just to avoid proliferation. I agree this is a good
thing to know about, but
Hi, Everyone,
We plan to have a KIP discussion on Google hangout on Apr. 7 at 11am PST.
If you are interested in participating and have not already received a
calendar invitation, please let me know. The following is the agenda.
Agenda:
KIP-4 (admin commands):
* wrap up any remaining issues
[
https://issues.apache.org/jira/browse/KAFKA-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14394856#comment-14394856
]
Neha Narkhede commented on KAFKA-1961:
--
+1. Checking it in...
Looks like its
[
https://issues.apache.org/jira/browse/KAFKA-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neha Narkhede updated KAFKA-1961:
-
Resolution: Fixed
Assignee: Ted Malaska (was: Gwen Shapira)
Status: Resolved (was:
Thanks Jun.
Some thoughts:
10) I think it is better we throttle regardless of the produce/fetch version.
This is a nice feature where clients can tell if they are being throttled or
not. If we only throttle newer clients, then we have inconsistent behavior
across clients in a multi-tenant
Tim Brooks created KAFKA-2090:
-
Summary: Remove duplicate check to metadataFetchInProgress
Key: KAFKA-2090
URL: https://issues.apache.org/jira/browse/KAFKA-2090
Project: Kafka
Issue Type:
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32823/
---
Review request for kafka.
Bugs: KAFKA-2090
[
https://issues.apache.org/jira/browse/KAFKA-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tim Brooks updated KAFKA-2090:
--
Attachment: KAFKA-2090.patch
Remove duplicate check to metadataFetchInProgress
[
https://issues.apache.org/jira/browse/KAFKA-2090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14394750#comment-14394750
]
Tim Brooks commented on KAFKA-2090:
---
Created reviewboard
[
https://issues.apache.org/jira/browse/KAFKA-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14394934#comment-14394934
]
Aravind edited comment on KAFKA-2078 at 4/3/15 8:32 PM:
we have 3
On April 3, 2015, 12:41 a.m., Joel Koshy wrote:
clients/src/main/java/org/apache/kafka/common/protocol/SecurityProtocol.java,
line 26
https://reviews.apache.org/r/28769/diff/22/?file=914062#file914062line26
Can we go with TRACE(Short.MAX_VALUE, TRACE) and start plaintext at
0?
Aditya, thanks for the updated KIP and Jay/Jun thanks for the
comments. Couple of comments in-line:
2. I would advocate for adding the return flag when we next bump the
request format version just to avoid proliferation. I agree this is a good
thing to know about, but at the moment I don't
11. We have to be careful not to double count when recording the metrics.
Currently, metric.record(fetchSize) is called on completion of a fetch
request. If we call metric.record(fetchSize) before the fetch request is
added to purgatory, then we shouldn't call it again on completion. However,
the
[
https://issues.apache.org/jira/browse/KAFKA-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14394934#comment-14394934
]
Aravind commented on KAFKA-2078:
You can see from one server it is EOF and other it is
See https://builds.apache.org/job/Kafka-trunk/440/changes
Changes:
[neha.narkhede] KAFKA-1961 Prevent deletion of _consumer_offsets topic;
reviewed by Neha Narkhede, Gwen Shapira and Jun Rao
--
[...truncated 1325 lines...]
kafka.message.MessageWriterTest
Some responses to Jay's points.
1. Using commas - Cool.
2. Adding return flag - I'm inclined to agree with Joel that this is good to
have in the initial implementation.
3. Config - +1. I'll remove it from the KIP. We can discuss this in parallel.
4. Purgatory vs Delay queue - I feel that it
[
https://issues.apache.org/jira/browse/KAFKA-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395266#comment-14395266
]
Joel Koshy commented on KAFKA-1809:
---
Re: ApiVersion - ok, but I just thought it would be
[
https://issues.apache.org/jira/browse/KAFKA-2082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395533#comment-14395533
]
Sriharsha Chintalapani commented on KAFKA-2082:
---
[~eapache] can you please
See https://builds.apache.org/job/KafkaPreCommit/53/changes
Changes:
[jjkoshy] KAFKA-1546; Automate replica lag tuning; reviewed by Joel Koshy, Neha
--
[...truncated 2606 lines...]
at
[
https://issues.apache.org/jira/browse/KAFKA-2082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395504#comment-14395504
]
Sriharsha Chintalapani commented on KAFKA-2082:
---
[~eapache] trying to
[
https://issues.apache.org/jira/browse/KAFKA-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated KAFKA-2077:
Comment: was deleted
(was: Thanks , I posted to reviewBoard - https://reviews.apache.org/r/32842/)
[
https://issues.apache.org/jira/browse/KAFKA-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395257#comment-14395257
]
Benoy Antony commented on KAFKA-2077:
-
Thanks , I posted to reviewBoard -
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28769/#review78867
---
Could you also make a pass of unused imports? I saw unused imports
Thanks so much. New patch set come up after rebase.
Sent from my iPhone
On Apr 3, 2015, at 7:50 PM, Jun Rao j...@confluent.io wrote:
Reviewed.
Thanks,
Jun
On Fri, Apr 3, 2015 at 10:38 AM, Tong Li liton...@us.ibm.com wrote:
Jun and Jay,
Have addressed all the comments and put
[
https://issues.apache.org/jira/browse/KAFKA-2088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhiqiang He updated KAFKA-2088:
---
Description:
1. set server.properties
server.properties:
zookeeper.connect =
[
https://issues.apache.org/jira/browse/KAFKA-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395511#comment-14395511
]
Sriharsha Chintalapani commented on KAFKA-1461:
---
Updated reviewboard
[
https://issues.apache.org/jira/browse/KAFKA-1461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sriharsha Chintalapani updated KAFKA-1461:
--
Attachment: KAFKA-1461_2015-04-03_20:48:34.patch
Replica fetcher thread does
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31366/
---
(Updated April 4, 2015, 3:48 a.m.)
Review request for kafka.
Bugs:
[
https://issues.apache.org/jira/browse/KAFKA-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated KAFKA-2077:
Attachment: KAFKA-2077.patch
Add ability to specify a TopicPicker class for KafkaLog4jApender
[
https://issues.apache.org/jira/browse/KAFKA-2077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395256#comment-14395256
]
Benoy Antony commented on KAFKA-2077:
-
Created reviewboard
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32842/
---
Review request for kafka.
Bugs: KAFKA-2077
See https://builds.apache.org/job/Kafka-trunk/441/changes
[
https://issues.apache.org/jira/browse/KAFKA-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated KAFKA-2041:
Attachment: kafka-2041-003.patch
Thanks for catching that . I am attaching the fixed patch
Add
[
https://issues.apache.org/jira/browse/KAFKA-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Benoy Antony updated KAFKA-2041:
Attachment: KAFKA-2041.patch
Add ability to specify a KeyClass for KafkaLog4jAppender
[
https://issues.apache.org/jira/browse/KAFKA-2041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395250#comment-14395250
]
Benoy Antony commented on KAFKA-2041:
-
Created reviewboard
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32841/
---
Review request for kafka.
Bugs: KAFKA-2041
[
https://issues.apache.org/jira/browse/KAFKA-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395251#comment-14395251
]
Jun Rao commented on KAFKA-1809:
The latest patch looks good to me. Left a few comments on
[
https://issues.apache.org/jira/browse/KAFKA-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395269#comment-14395269
]
Gwen Shapira commented on KAFKA-1809:
-
Yay on KAFKA-1546 :)
Refactor brokers to
[
https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395268#comment-14395268
]
Joel Koshy commented on KAFKA-1546:
---
+1 on the latest doc updates. I can check in the
Reviewed.
Thanks,
Jun
On Fri, Apr 3, 2015 at 10:38 AM, Tong Li liton...@us.ibm.com wrote:
Jun and Jay,
Have addressed all the comments and put up another patch set. Can you
please take a look again? Really appreciate it.
Thanks.
Tong Li
OpenStack Kafka Community Development
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32740/#review78858
---
Thanks for the new patch. Just a couple of more minor comments.
[
https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Joel Koshy updated KAFKA-1546:
--
Resolution: Fixed
Status: Resolved (was: Patch Available)
Thanks for the patches.
Committed
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28769/#review78839
---
Thanks for the patch. Looks good. Just a few more minor comments
[
https://issues.apache.org/jira/browse/KAFKA-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395260#comment-14395260
]
Ted Malaska commented on KAFKA-1961:
Thank you Neha for the final review.
Thank you
[
https://issues.apache.org/jira/browse/KAFKA-1809?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395262#comment-14395262
]
Gwen Shapira commented on KAFKA-1809:
-
Yep, will submit this weekend.
I'll address
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32823/#review78853
---
Ship it!
Ship It!
- Ewen Cheslack-Postava
On April 3, 2015,
[
https://issues.apache.org/jira/browse/KAFKA-2082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sriharsha Chintalapani updated KAFKA-2082:
--
Attachment: KAFKA-2082.patch
Kafka Replication ends up in a bad state
[
https://issues.apache.org/jira/browse/KAFKA-2082?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sriharsha Chintalapani updated KAFKA-2082:
--
Status: Patch Available (was: Open)
Kafka Replication ends up in a bad state
[
https://issues.apache.org/jira/browse/KAFKA-2082?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14395530#comment-14395530
]
Sriharsha Chintalapani commented on KAFKA-2082:
---
Created reviewboard
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/32852/
---
Review request for kafka.
Bugs: KAFKA-2082
I have a kafka cluster which has 40 partitions, and 2M requests every
second, for each, it is kinda 1.5k bytes, using 4 consumer machine, 10
partitions for each machine
there are other consumers consuming. in other several machines
the tricky thing is, some of the parts will get lag surely, but
57 matches
Mail list logo
