[jira] [Comment Edited] (KAFKA-1932) kafka topic (creation) templates
[ https://issues.apache.org/jira/browse/KAFKA-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388208#comment-14388208 ] Ahmet AKYOL edited comment on KAFKA-1932 at 3/31/15 8:41 AM: - [~becket_qin] , thanks, I wasn't aware of KIP's. By the way, templates are a little more than that and very handy but it fits ElasticSearch's schema design (for example daily logs) better than Kafka topics. I offered templates as a fast but dirty solution; [KIP-4|https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations] is way better. So, it's OK to close this issue. was (Author: liqusha): [~becket_qin] , thanks, I wasn't aware of KIP's. By the way, templates are a little more than that and very handy but it fits ElasticSearch's schema design (for example daily logs) better than Kafka topics. I offered templates as a fast but dirty solution; KIPs are way better. So, it's OK to close this issue. kafka topic (creation) templates Key: KAFKA-1932 URL: https://issues.apache.org/jira/browse/KAFKA-1932 Project: Kafka Issue Type: Wish Reporter: Ahmet AKYOL AFAIK, the only way to create a Kafka topic (without using the default settings) is using the provided bash script. Even though, a client support could be nice, I would prefer to see a template mechanism similar to [Elasticsearch Index Templates|http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-templates.html] . What I have in my mind is very simple and adding something like this into server properties : template.name=pattern,numOfReplica,NumberOfPartition and pattern can only contain * meaning starts with, ends with or contains. example: template.logtopics=*_log,2,20 template.loaders=*_loader,1,5 so,when some producer sends a message to a topic for the first time which ends with _logs , then, kafka can use above settings. thanks in advance update: On second thought, maybe a command like kafka-create-template.sh could be more practical for cluster deployments, rather than adding to server.properties. Kafka internally registers this to ZK. About use cases, I can understand an opposing argument like creating many topics is not a good design decision. Besides, my point is not to create so many topics, just to automate an important process by giving the responsibility to Kafka. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1932) kafka topic (creation) templates
[ https://issues.apache.org/jira/browse/KAFKA-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388208#comment-14388208 ] Ahmet AKYOL commented on KAFKA-1932: [~becket_qin] , thanks, I wasn't aware of KIP's. By the way, templates are a little more than that and very handy but it fits ElasticSearch's schema design (for example daily logs) better than Kafka topics. So, it's OK to close this issue. kafka topic (creation) templates Key: KAFKA-1932 URL: https://issues.apache.org/jira/browse/KAFKA-1932 Project: Kafka Issue Type: Wish Reporter: Ahmet AKYOL AFAIK, the only way to create a Kafka topic (without using the default settings) is using the provided bash script. Even though, a client support could be nice, I would prefer to see a template mechanism similar to [Elasticsearch Index Templates|http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-templates.html] . What I have in my mind is very simple and adding something like this into server properties : template.name=pattern,numOfReplica,NumberOfPartition and pattern can only contain * meaning starts with, ends with or contains. example: template.logtopics=*_log,2,20 template.loaders=*_loader,1,5 so,when some producer sends a message to a topic for the first time which ends with _logs , then, kafka can use above settings. thanks in advance update: On second thought, maybe a command like kafka-create-template.sh could be more practical for cluster deployments, rather than adding to server.properties. Kafka internally registers this to ZK. About use cases, I can understand an opposing argument like creating many topics is not a good design decision. Besides, my point is not to create so many topics, just to automate an important process by giving the responsibility to Kafka. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1932) kafka topic (creation) templates
[ https://issues.apache.org/jira/browse/KAFKA-1932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388208#comment-14388208 ] Ahmet AKYOL edited comment on KAFKA-1932 at 3/31/15 8:32 AM: - [~becket_qin] , thanks, I wasn't aware of KIP's. By the way, templates are a little more than that and very handy but it fits ElasticSearch's schema design (for example daily logs) better than Kafka topics. I offered templates as a fast but dirty solution; KIPs are way better. So, it's OK to close this issue. was (Author: liqusha): [~becket_qin] , thanks, I wasn't aware of KIP's. By the way, templates are a little more than that and very handy but it fits ElasticSearch's schema design (for example daily logs) better than Kafka topics. So, it's OK to close this issue. kafka topic (creation) templates Key: KAFKA-1932 URL: https://issues.apache.org/jira/browse/KAFKA-1932 Project: Kafka Issue Type: Wish Reporter: Ahmet AKYOL AFAIK, the only way to create a Kafka topic (without using the default settings) is using the provided bash script. Even though, a client support could be nice, I would prefer to see a template mechanism similar to [Elasticsearch Index Templates|http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-templates.html] . What I have in my mind is very simple and adding something like this into server properties : template.name=pattern,numOfReplica,NumberOfPartition and pattern can only contain * meaning starts with, ends with or contains. example: template.logtopics=*_log,2,20 template.loaders=*_loader,1,5 so,when some producer sends a message to a topic for the first time which ends with _logs , then, kafka can use above settings. thanks in advance update: On second thought, maybe a command like kafka-create-template.sh could be more practical for cluster deployments, rather than adding to server.properties. Kafka internally registers this to ZK. About use cases, I can understand an opposing argument like creating many topics is not a good design decision. Besides, my point is not to create so many topics, just to automate an important process by giving the responsibility to Kafka. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 28769: Patch for KAFKA-1809
On March 29, 2015, 7:33 p.m., Jun Rao wrote: clients/src/main/java/org/apache/kafka/common/protocol/ApiVersion.java, line 28 https://reviews.apache.org/r/28769/diff/21/?file=908349#file908349line28 Since this is for intra-broker communication, should we move this class to core? Sure. At first I thought we'll need it when we move all requests/responses to o.a.k.common, but on second look, it doesn't seem like the requests/responses themselves need it. On March 29, 2015, 7:33 p.m., Jun Rao wrote: core/src/main/scala/kafka/server/KafkaConfig.scala, lines 345-347 https://reviews.apache.org/r/28769/diff/21/?file=908381#file908381line345 It seems that a 3-digit value like 0.8.2 is also valid? yes, and also 0.8.2.X or 0.8.2.Y. We treat them all as equivalent. Do you think its an issue? or just want to document this fact? On March 29, 2015, 7:33 p.m., Jun Rao wrote: core/src/main/scala/kafka/utils/Utils.scala, line 614 https://reviews.apache.org/r/28769/diff/21/?file=908392#file908392line614 Should we just return scala.collection.Map? Isn't it a bit confusing? A Map could be mutable, immutable, or generic (also immutable but a different type...) depending on what is imported. We pass the results of this around a bit between classes, sometimes relying on type-inference, often with different collections imported. I'd rather be stricter here, unless you see an issue with this? On March 29, 2015, 7:33 p.m., Jun Rao wrote: core/src/test/scala/unit/kafka/cluster/BrokerTest.scala, lines 62-64 https://reviews.apache.org/r/28769/diff/21/?file=908400#file908400line62 Is this necessary given the above tests on equal and hashCode? Probably not. Since we typically care most about the ability to find the object in a hashmap, I usually test for it explicitly. If you think it doesn't belong in Kafka, I can remove it. On March 29, 2015, 7:33 p.m., Jun Rao wrote: system_test/run_all.sh, lines 5-6 https://reviews.apache.org/r/28769/diff/21/?file=908434#file908434line5 Would it be better to just have testcase_to_run.json be testcase_to_run_all.json? Also, could we change README accordingly? Will update the readme. We can't just have testcase_to_run.json be testcase_to_run_all.json. system_test_runner executes testcase_to_run.json by default. We have several testsuites we may want to run testcase_to_run_all.json, testcase_to_run_all_replica.json (which I missed in this patch), and testcase_to_run_sanity.json. The shellscript backsup the existing test configuration, replaces with the tests we want to run and executes system_test_runner.sh (This is already done in run_sanity.sh) I agree that modifying system_test_runner to take the script as parameter will be cleaner, but this is beyond the scope of this patch. That is, I don't want to dive into the python spaggetti that will be replaced soon :) - Gwen --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28769/#review78160 --- On March 27, 2015, 10:04 p.m., Gwen Shapira wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28769/ --- (Updated March 27, 2015, 10:04 p.m.) Review request for kafka. Bugs: KAFKA-1809 https://issues.apache.org/jira/browse/KAFKA-1809 Repository: kafka Description --- forgot rest of patch merge with trunk Diffs - clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java fa9daaef66ff7961e1c46cd0cd8fed18a53bccd8 clients/src/main/java/org/apache/kafka/common/protocol/ApiVersion.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/protocol/SecurityProtocol.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/utils/Utils.java 920b51a6c3c99639fbc9dc0656373c19fabd clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java c899813d55b9c4786adde3d840f040d6645d27c8 config/server.properties 1614260b71a658b405bb24157c8f12b1f1031aa5 core/src/main/scala/kafka/admin/AdminUtils.scala b700110f2d7f1ede235af55d8e37e1b5592c6c7d core/src/main/scala/kafka/admin/TopicCommand.scala f400b71f8444fffd3fc1d8398a283682390eba4e core/src/main/scala/kafka/api/ConsumerMetadataResponse.scala 24aaf954dc42e2084454fa5fc9e8f388ea95c756 core/src/main/scala/kafka/api/LeaderAndIsrRequest.scala 4ff7e8f8cc695551dd5d2fe65c74f6b6c571e340 core/src/main/scala/kafka/api/TopicMetadata.scala 0190076df0adf906ecd332284f222ff974b315fc core/src/main/scala/kafka/api/TopicMetadataResponse.scala 92ac4e687be22e4800199c0666bfac5e0059e5bb core/src/main/scala/kafka/api/UpdateMetadataRequest.scala
Does system_test work on MacOs?
Hi, The first line of the system_test README says: This test framework currently doesn't support MacOS due to different ps argument options from Linux. The correct ps execution is required to terminate the background running processes properly. I use system_test on MacOS and never had any issues. Did anyone else run into issues? If it works for all of us, I'll take the warning out. Gwen
Re: Does system_test work on MacOs?
It's been awhile and I mostly ran them in VMs, but I don't recall having any issues running them on OS X. -Ewen On Tue, Mar 31, 2015 at 6:00 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi, The first line of the system_test README says: This test framework currently doesn't support MacOS due to different ps argument options from Linux. The correct ps execution is required to terminate the background running processes properly. I use system_test on MacOS and never had any issues. Did anyone else run into issues? If it works for all of us, I'll take the warning out. Gwen -- Thanks, Ewen
[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.
[
https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389839#comment-14389839
]
Jiangjie Qin commented on KAFKA-2076:
-
Return a map of time-offset would be useful.
I cannot think of use cases where people want to know the value of earliest
offset. But it is covered by the map anyway.
I like the interface. It looks cleaner.
One concern I have is that most of the time user only cares about log end
offsets which is the only continuously updated offset. Using this interface, in
order to get the LEO, they have to get the entire map. This is probably OK if
we have coarse granularity for timestamps, but could have some overhead if we
have finer granularity, because the LEO might potentially be queried frequently
and the returned map might be large.
Maybe we can add an interface:
{code}
PartitionOffsets offsetTimes(TopicPartition tp, int numOffsets)
{code}
So it returns the numOffsets entries of time-offset back traced from LEO. So
people set numOffsets=1 will only get LEO.
And I think batched request might be better. Otherwise for a consumer consuming
from many partitions, it has to send many separate requests. It is not
efficient and might also complicate the user code.
Add an API to new consumer to allow user get high watermark of partitions.
--
Key: KAFKA-2076
URL: https://issues.apache.org/jira/browse/KAFKA-2076
Project: Kafka
Issue Type: Improvement
Reporter: Jiangjie Qin
We have a use case that user wants to know how far it is behind a particular
partition on startup. Currently in each fetch response, we have high
watermark for each partition, we only keep a global max-lag metric. It would
be better that we keep a record of high watermark per partition and update it
on each fetch response. We can add a new API to let user query the high
watermark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Metrics package discussion
(2) Not sure why we can't do this in 0.8.3. We changed the metrics names in 0.8.2 already. Given that we need to share code btw the client and the core, and we need to keep the metrics consistent on the broker, it seems that we have no choice but to migrate to KM. If so, it seems that the sooner that we do this, the better. It is important to give people an easy path for migration. However, it may not be easy to keep the mbean names exactly the same. For example, YM has hardcoded attributes (e.g. 1-min-rate, 5-min-rate, 15-min-rate, etc for rates) that are not available in KM. One benefit out of this migration is that one can get the metrics in the client and the broker in the same way. Thanks, Jun On Mon, Mar 30, 2015 at 9:26 PM, Gwen Shapira gshap...@cloudera.com wrote: (1) It will be interesting to see what others use for monitoring integration, to see what is already covered with existing JMX integrations and what needs special support. (2) I think the migration story is more important - this is a non-compatible change, right? So we can't do it in 0.8.3 timeframe, it has to be in 0.9? And we need to figure out how will users migrate - do we just tell everyone please reconfigure all your monitors from scratch - don't worry, it is worth it? I know you keep saying we did it before and our users are used to it, but I think there are a lot more users now, and some of them have different compatibility expectations. We probably need to find: * A least painful way to migrate - can we keep the names of at least most of the metrics intact? * Good explanation of what users gain from this painful migration (i.e. more accurate statistics due to gazillion histograms) On Mon, Mar 30, 2015 at 6:29 PM, Jun Rao j...@confluent.io wrote: If we are committed to migrating the broker side metrics to KM for the next release, we will need to (1) have a story on supporting common reporters (as listed in KAFKA-1930), and (2) see if the current histogram support is good enough for measuring things like request time. Thanks, Jun On Mon, Mar 30, 2015 at 3:03 PM, Aditya Auradkar aaurad...@linkedin.com.invalid wrote: If we do plan to use the network code in client, I think that is a good reason in favor of migration. It will be unnecessary to have metrics from multiple libraries coexist since our users will have to start monitoring these new metrics anyway. I also agree with Jay that in multi-tenant clusters people care about detailed statistics for their own application over global numbers. Based on the arguments so far, I'm +1 for migrating to KM. Thanks, Aditya From: Jun Rao [j...@confluent.io] Sent: Sunday, March 29, 2015 9:44 AM To: dev@kafka.apache.org Subject: Re: Metrics package discussion There is another thing to consider. We plan to reuse the client components on the server side over time. For example, as part of the security work, we are looking into replacing the server side network code with the client network code (KAFKA-1928). However, the client network already has metrics based on KM. Thanks, Jun On Sat, Mar 28, 2015 at 1:34 PM, Jay Kreps jay.kr...@gmail.com wrote: I think Joel's summary is good. I'll add a few more points: As discussed memory matter a lot if we want to be able to give percentiles at the client or topic level, in which case we will have thousands of them. If we just do histograms at the global level then it is not a concern. The argument for doing histograms at the client and topic level is that averages are often very misleading, especially for latency information or other asymmetric distributions. Most people who care about this kind of thing would say the same. If you are a user of a multi-tenant cluster then you probably care a lot more about stats for your application or your topic rather than the global, so it could be nice to have histograms for these. I don't feel super strongly about this. The ExponentiallyDecayingSample is internally a ConcurrentSkipListMapDouble, Long. This seems to have an overhead of about 64 bytes per entry. So a 1000 element sample is 64KB. For global metrics this is fine, but for granular metrics not workable. Two other issues I'm not sure about: 1. Is there a way to get metric descriptions into the coda hale JMX output? One of the really nicest practical things about the new client metrics is that if you look at them in jconsole each metric has an associated description that explains what it means. I think this is a nice usability thing--it is really hard to know what to make of the current metrics without this kind of documentation and keeping separate docs up-to-date is really hard and even if you do it most people won't find it. 2. I'm not clear if the sample decay in the histogram is
Re: [DISCUSSION] Keep docs updated per jira
For the wire protocol in the wiki, we need to tag each new request version with the first release when it's supported. So, currently, any new request version introduced in trunk will be tagged with 0.8.3. Thanks, Jun On Mon, Mar 30, 2015 at 5:17 PM, Joel Koshy jjkosh...@gmail.com wrote: Also for the wikis - those should probably correspond to the latest released version right? So for e.g., if we add or modify the protocol on trunk we can add it to the wiki but mark it with some highlight or similar just to make it clear that it is a change on trunk only. Thanks, Joel On Thu, Mar 26, 2015 at 06:27:25PM -0700, Jun Rao wrote: Hi, Everyone, Quite a few jiras these days require documentation changes (e.g., wire protocol, ZK layout, configs, jmx, etc). Historically, we have been updating the documentation just before we do a release. The issue is that some of the changes will be missed since they were done a while back. Another way to do that is to keep the docs updated as we complete each jira. Currently, our documentations are in the following places. wire protocol: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol ZK layout: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper configs/jmx: https://svn.apache.org/repos/asf/kafka/site/083 We probably don't need to update configs already ported to ConfigDef since they can be generated automatically. However, for the rest of the doc related changes, keeping they updated per jira seems a better approach. What do people think? Thanks, Jun
Re: [DISCUSS] KIP-11- Authorization design for kafka security
Thanks for the writeup. A few more comments. 20. I agree that it would be better to do this after KIP-4 (admin commands) is done. With KIP-4, all admin operations will be sent as requests to the brokers instead of accessing ZK directly. This will make authorization easier. 21. Operation: What about other types of requests not covered in the list, such as committing and fetching offsets, list topics, fetching consumer metadata, heartbeat, join group, etc? 22. TopicConfigCache: We will need such a cache in KIP-4 as well. It would be useful to make sure that the implementation can be reused. 23. Authorizer: 23.1 Do cluster level operations go through authorize() too? If so, what will be the resource? 23.2 I assume that the authorize() check will be called on every request. So, we will have to make sure that the check is cheap. 24. The acl json string in the config: Should we version this so that we can evolve it in the future (e.g., adding group support)? Jun On Sun, Mar 29, 2015 at 3:56 PM, Parth Brahmbhatt pbrahmbh...@hortonworks.com wrote: Hi Gwen, Thanks a lot for taking the time to review this. I have tried to address all your questions below. Thanks Parth On 3/28/15, 8:08 PM, Gwen Shapira gshap...@cloudera.commailto: gshap...@cloudera.com wrote: Preparing for Tuesday meeting, I went over the KIP :) First, Parth did an amazing job, the KIP is fantastic - detailed and readable. Thank you! Second, I have a lng list of questions :) No objections, just some things I'm unclear on and random minor comments. In general, I like the design, I just feel I'm missing parts of the picture. 1. Yes, Create topic will have an optional acls, the output of describe will display owner and acls and alter topic will allow to modify the acls.” - will be nice to see what the CLI will look like. * I will modify the KIP but I was going to add “—acl acl-file.json” to create-topic and alter-topic. 2. I like the addition of Topic owner. We made the mistake of forgetting about it when adding authorization to Sqoop2. We probably want to add “chown” command to the topic commands. * Again we can add “—owner user-name” to alter topic. 3. Kafka server will read authorizer.class” config value at startup time, create an instance of the specified class and call initialize method. We’ll need to validate that users specify only one of those. * The config type will be string so type validation should take care of it. 4. One added assumption is that on non-secure connections the session will have principal set to an object whose name() method will return Anonymous”. Can we keep DrWho? :) * Sure, its up to you actually as you are the owner of the jira that introduces session concept. 5. For cluster actions that do not apply to a specific topic like CREATE we have 2 options. We can either add a broker config called broker.acls which will point to a json file. This file will be available on all broker hosts and authorizer will read the acls on initialization and keep refreshing it every X minutes. Any changes will require re-distribution of the acl json file. Alternatively we can add a zookeeper path /brokers/acls and store the acl json as data. Authorizer can refresh the acl from json every X minutes. In absence of broker acls the authorizer will fail open, in other words it will allow all users from all hosts to perform all cluster actions” I prefer a file to ZK - since thats where we store all use-defined configurations for now. Everyone knows how to secure a file system :) * I will let everyone vote, file system is fine by me. 6. When an Acl is missing , this implementation will always fail open for backward compatibility. “ - agree, but we need to document that this makes the default authorizer non-secure * Sure. 7. If the value of authorizer.class.name is null, in secure mode the cluster will fail with ConfigException. In non secure mode in absence of config value forauthorizer.class.name the server will allow all requests to all topics that , even if the topic has configured acls” - I don’t think Kafka has “secure mode” - it can support SSL and plaintext (un-authenticated) on two different ports simultaneously. I don’t object to adding such configuration, but we need to decide exactly what it means. * This is one area of confusion so I will add an open question. 8. Currently all topic creation/modification/deletion actions are performed using KafkaAdminUtil which mostly interacts directly with zookeeper instead of forwarding requests to a broker host. Given all the code is executed on client side there is no easy way to perform authorization. “ - since we are adding the admin protocol requests in KIP-4, perhaps addressing those makes sense. * Yes, We will have to wait for KIP-4 to be delivered. 9. I didn’t see a specification of what is “resource”, I assume its an enum with things like Topic and… ? *
Re: Metrics package discussion
(2) I believe we agreed that our metrics are a public API. I believe we also agree we don't break API in minor releases. So, it seems obvious to me that we can't make breaking changes to metrics in minor releases. I'm not convinced we did it in the past is a good reason to do it again. Is there a strong reason to do it in a 0.8.3 time-frame? On Tue, Mar 31, 2015 at 7:59 AM, Jun Rao j...@confluent.io wrote: (2) Not sure why we can't do this in 0.8.3. We changed the metrics names in 0.8.2 already. Given that we need to share code btw the client and the core, and we need to keep the metrics consistent on the broker, it seems that we have no choice but to migrate to KM. If so, it seems that the sooner that we do this, the better. It is important to give people an easy path for migration. However, it may not be easy to keep the mbean names exactly the same. For example, YM has hardcoded attributes (e.g. 1-min-rate, 5-min-rate, 15-min-rate, etc for rates) that are not available in KM. One benefit out of this migration is that one can get the metrics in the client and the broker in the same way. Thanks, Jun On Mon, Mar 30, 2015 at 9:26 PM, Gwen Shapira gshap...@cloudera.com wrote: (1) It will be interesting to see what others use for monitoring integration, to see what is already covered with existing JMX integrations and what needs special support. (2) I think the migration story is more important - this is a non-compatible change, right? So we can't do it in 0.8.3 timeframe, it has to be in 0.9? And we need to figure out how will users migrate - do we just tell everyone please reconfigure all your monitors from scratch - don't worry, it is worth it? I know you keep saying we did it before and our users are used to it, but I think there are a lot more users now, and some of them have different compatibility expectations. We probably need to find: * A least painful way to migrate - can we keep the names of at least most of the metrics intact? * Good explanation of what users gain from this painful migration (i.e. more accurate statistics due to gazillion histograms) On Mon, Mar 30, 2015 at 6:29 PM, Jun Rao j...@confluent.io wrote: If we are committed to migrating the broker side metrics to KM for the next release, we will need to (1) have a story on supporting common reporters (as listed in KAFKA-1930), and (2) see if the current histogram support is good enough for measuring things like request time. Thanks, Jun On Mon, Mar 30, 2015 at 3:03 PM, Aditya Auradkar aaurad...@linkedin.com.invalid wrote: If we do plan to use the network code in client, I think that is a good reason in favor of migration. It will be unnecessary to have metrics from multiple libraries coexist since our users will have to start monitoring these new metrics anyway. I also agree with Jay that in multi-tenant clusters people care about detailed statistics for their own application over global numbers. Based on the arguments so far, I'm +1 for migrating to KM. Thanks, Aditya From: Jun Rao [j...@confluent.io] Sent: Sunday, March 29, 2015 9:44 AM To: dev@kafka.apache.org Subject: Re: Metrics package discussion There is another thing to consider. We plan to reuse the client components on the server side over time. For example, as part of the security work, we are looking into replacing the server side network code with the client network code (KAFKA-1928). However, the client network already has metrics based on KM. Thanks, Jun On Sat, Mar 28, 2015 at 1:34 PM, Jay Kreps jay.kr...@gmail.com wrote: I think Joel's summary is good. I'll add a few more points: As discussed memory matter a lot if we want to be able to give percentiles at the client or topic level, in which case we will have thousands of them. If we just do histograms at the global level then it is not a concern. The argument for doing histograms at the client and topic level is that averages are often very misleading, especially for latency information or other asymmetric distributions. Most people who care about this kind of thing would say the same. If you are a user of a multi-tenant cluster then you probably care a lot more about stats for your application or your topic rather than the global, so it could be nice to have histograms for these. I don't feel super strongly about this. The ExponentiallyDecayingSample is internally a ConcurrentSkipListMapDouble, Long. This seems to have an overhead of about 64 bytes per entry. So a 1000 element sample is 64KB. For global metrics this is fine, but for granular metrics not workable. Two other issues I'm not sure about: 1. Is there a way to get metric descriptions into the coda hale JMX output? One of the really nicest practical things about the new client metrics is that
Re: Metrics package discussion
My main concern is that we don't do the migration in 0.8.3, we will be left with some metrics in YM format and some others in KM format (as we start sharing client code on the broker). This is probably a worse situation to be in. +1. I am not sure how our servo adaptor will work if there are two formats for metrics? unless there is an easy way to check the format (YM/KM). On Tue, Mar 31, 2015 at 9:40 AM, Jun Rao j...@confluent.io wrote: (2) The metrics are clearly part of the client API and we are not changing that (at least for the new client). Arguably, the metrics are also part of the broker side API. However, since they affect fewer parties (mostly just the Kafka admins), it may be easier to make those changes. My main concern is that we don't do the migration in 0.8.3, we will be left with some metrics in YM format and some others in KM format (as we start sharing client code on the broker). This is probably a worse situation to be in. Thanks, Jun On Tue, Mar 31, 2015 at 9:26 AM, Gwen Shapira gshap...@cloudera.com wrote: (2) I believe we agreed that our metrics are a public API. I believe we also agree we don't break API in minor releases. So, it seems obvious to me that we can't make breaking changes to metrics in minor releases. I'm not convinced we did it in the past is a good reason to do it again. Is there a strong reason to do it in a 0.8.3 time-frame? On Tue, Mar 31, 2015 at 7:59 AM, Jun Rao j...@confluent.io wrote: (2) Not sure why we can't do this in 0.8.3. We changed the metrics names in 0.8.2 already. Given that we need to share code btw the client and the core, and we need to keep the metrics consistent on the broker, it seems that we have no choice but to migrate to KM. If so, it seems that the sooner that we do this, the better. It is important to give people an easy path for migration. However, it may not be easy to keep the mbean names exactly the same. For example, YM has hardcoded attributes (e.g. 1-min-rate, 5-min-rate, 15-min-rate, etc for rates) that are not available in KM. One benefit out of this migration is that one can get the metrics in the client and the broker in the same way. Thanks, Jun On Mon, Mar 30, 2015 at 9:26 PM, Gwen Shapira gshap...@cloudera.com wrote: (1) It will be interesting to see what others use for monitoring integration, to see what is already covered with existing JMX integrations and what needs special support. (2) I think the migration story is more important - this is a non-compatible change, right? So we can't do it in 0.8.3 timeframe, it has to be in 0.9? And we need to figure out how will users migrate - do we just tell everyone please reconfigure all your monitors from scratch - don't worry, it is worth it? I know you keep saying we did it before and our users are used to it, but I think there are a lot more users now, and some of them have different compatibility expectations. We probably need to find: * A least painful way to migrate - can we keep the names of at least most of the metrics intact? * Good explanation of what users gain from this painful migration (i.e. more accurate statistics due to gazillion histograms) On Mon, Mar 30, 2015 at 6:29 PM, Jun Rao j...@confluent.io wrote: If we are committed to migrating the broker side metrics to KM for the next release, we will need to (1) have a story on supporting common reporters (as listed in KAFKA-1930), and (2) see if the current histogram support is good enough for measuring things like request time. Thanks, Jun On Mon, Mar 30, 2015 at 3:03 PM, Aditya Auradkar aaurad...@linkedin.com.invalid wrote: If we do plan to use the network code in client, I think that is a good reason in favor of migration. It will be unnecessary to have metrics from multiple libraries coexist since our users will have to start monitoring these new metrics anyway. I also agree with Jay that in multi-tenant clusters people care about detailed statistics for their own application over global numbers. Based on the arguments so far, I'm +1 for migrating to KM. Thanks, Aditya From: Jun Rao [j...@confluent.io] Sent: Sunday, March 29, 2015 9:44 AM To: dev@kafka.apache.org Subject: Re: Metrics package discussion There is another thing to consider. We plan to reuse the client components on the server side over time. For example, as part of the security work, we are looking into replacing the server side network code with the client network code (KAFKA-1928). However, the client network already has metrics based on KM. Thanks, Jun On Sat, Mar 28, 2015 at 1:34 PM, Jay Kreps jay.kr...@gmail.com wrote: I
Re: KIP discussion Mar 24 at 11am PST
Jun, I would like to participate but have not received the invite. Can you please send to me? Thanks. Tong Li OpenStack Kafka Community Development Building 501/B205 liton...@us.ibm.com From: Jun Rao j...@confluent.io To: dev@kafka.apache.org dev@kafka.apache.org Date: 03/27/2015 10:35 PM Subject:Re: KIP discussion Mar 24 at 11am PST Just sent out an invite for Mar 31 at 11am PST. The following is the tentative agenda. Let me know if you want to attend, but haven't received an invite. Agenda: KIP-4 (admin commands): * wrap up any remaining issues KIP-13 (quota): * dependency on using the new metrics package * dependency KIP-5 (broker configuration) Perhaps security related KIPS. Thanks, Jun On Fri, Mar 27, 2015 at 6:30 PM, Joel Koshy jjkosh...@gmail.com wrote: I think we decided that we should reconvene next Tuesday to collect everyone's thoughts on metrics and discuss other KIPs. Can everyone still do Tuesday or do people feel we need to push that further out by a few days? Thanks, Joel On Tue, Mar 24, 2015 at 12:28:04PM -0700, Jun Rao wrote: Just to keep everyone posted. The following is a summary of what's being discussed in the KIP hangout today. KIP-4 (admin commands): * Gwen is uploading a patch in KAFKA-1927 (refactoring requests) so that we can get unblocked of adding new requests. * We will combine DescribeTopic and TopicMetadata in the future. * We will leave the admin requests async for now. * We will not add a VerifyReassignPartitionRequest for now. We can do that later when we improve the verification process. * We need to discuss a bit more on how to expose the controller info to the client. * Andrii will send out more details on the KIP thread. KIP-15 (close): * If close() or close with a non-zero timeout is called from the send thread, we will log it as an error. * Jiangjie will follow up on the KIP thread. KIP-13 (quota): * Need a separate discussion on whether to use the new metrics package on the broker on the mailing list. * There are a few other details being discuss and Aditya will follow up on the KIP thread. Thanks, Jun On Fri, Mar 20, 2015 at 2:44 PM, Jun Rao j...@confluent.io wrote: Hi, Everyone, We plan to have a KIP discussion on Google hangout on Mar 24 at 11am PST. If you are interested in participating and have not already received a calendar invitation, please let me know. The following is the agenda. KIP-4 (admin commands): 10 mins * status of KAFKA-1927 (refactoring requests). which blocks this KIP * status of KAFKA-1634 (improve OffsetCommitRequest), which blocks KAFKA-1927 * any remaining issues for discussion KIP-15 (close): 10 mins * semantics of close() and close(timeout) KIP-13 (quota): * protocol change to reflect the state of throttling * dependency on using the new metrics package * dependency KIP-5 (broker configuration) Thanks, Jun
Re: [DISCUSSION] Keep docs updated per jira
+1 On Tue, Mar 31, 2015 at 9:23 AM, Jay Kreps jay.kr...@gmail.com wrote: Yeah the protocol should probably move off the wiki and into the release-versioned docs. -Jay On Mon, Mar 30, 2015 at 5:17 PM, Joel Koshy jjkosh...@gmail.com wrote: Also for the wikis - those should probably correspond to the latest released version right? So for e.g., if we add or modify the protocol on trunk we can add it to the wiki but mark it with some highlight or similar just to make it clear that it is a change on trunk only. Thanks, Joel On Thu, Mar 26, 2015 at 06:27:25PM -0700, Jun Rao wrote: Hi, Everyone, Quite a few jiras these days require documentation changes (e.g., wire protocol, ZK layout, configs, jmx, etc). Historically, we have been updating the documentation just before we do a release. The issue is that some of the changes will be missed since they were done a while back. Another way to do that is to keep the docs updated as we complete each jira. Currently, our documentations are in the following places. wire protocol: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol ZK layout: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper configs/jmx: https://svn.apache.org/repos/asf/kafka/site/083 We probably don't need to update configs already ported to ConfigDef since they can be generated automatically. However, for the rest of the doc related changes, keeping they updated per jira seems a better approach. What do people think? Thanks, Jun
Re: [DISCUSSION] Keep docs updated per jira
Yeah the protocol should probably move off the wiki and into the release-versioned docs. -Jay On Mon, Mar 30, 2015 at 5:17 PM, Joel Koshy jjkosh...@gmail.com wrote: Also for the wikis - those should probably correspond to the latest released version right? So for e.g., if we add or modify the protocol on trunk we can add it to the wiki but mark it with some highlight or similar just to make it clear that it is a change on trunk only. Thanks, Joel On Thu, Mar 26, 2015 at 06:27:25PM -0700, Jun Rao wrote: Hi, Everyone, Quite a few jiras these days require documentation changes (e.g., wire protocol, ZK layout, configs, jmx, etc). Historically, we have been updating the documentation just before we do a release. The issue is that some of the changes will be missed since they were done a while back. Another way to do that is to keep the docs updated as we complete each jira. Currently, our documentations are in the following places. wire protocol: https://cwiki.apache.org/confluence/display/KAFKA/A+Guide+To+The+Kafka+Protocol ZK layout: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+data+structures+in+Zookeeper configs/jmx: https://svn.apache.org/repos/asf/kafka/site/083 We probably don't need to update configs already ported to ConfigDef since they can be generated automatically. However, for the rest of the doc related changes, keeping they updated per jira seems a better approach. What do people think? Thanks, Jun
Re: Metrics package discussion
(2) The metrics are clearly part of the client API and we are not changing that (at least for the new client). Arguably, the metrics are also part of the broker side API. However, since they affect fewer parties (mostly just the Kafka admins), it may be easier to make those changes. My main concern is that we don't do the migration in 0.8.3, we will be left with some metrics in YM format and some others in KM format (as we start sharing client code on the broker). This is probably a worse situation to be in. Thanks, Jun On Tue, Mar 31, 2015 at 9:26 AM, Gwen Shapira gshap...@cloudera.com wrote: (2) I believe we agreed that our metrics are a public API. I believe we also agree we don't break API in minor releases. So, it seems obvious to me that we can't make breaking changes to metrics in minor releases. I'm not convinced we did it in the past is a good reason to do it again. Is there a strong reason to do it in a 0.8.3 time-frame? On Tue, Mar 31, 2015 at 7:59 AM, Jun Rao j...@confluent.io wrote: (2) Not sure why we can't do this in 0.8.3. We changed the metrics names in 0.8.2 already. Given that we need to share code btw the client and the core, and we need to keep the metrics consistent on the broker, it seems that we have no choice but to migrate to KM. If so, it seems that the sooner that we do this, the better. It is important to give people an easy path for migration. However, it may not be easy to keep the mbean names exactly the same. For example, YM has hardcoded attributes (e.g. 1-min-rate, 5-min-rate, 15-min-rate, etc for rates) that are not available in KM. One benefit out of this migration is that one can get the metrics in the client and the broker in the same way. Thanks, Jun On Mon, Mar 30, 2015 at 9:26 PM, Gwen Shapira gshap...@cloudera.com wrote: (1) It will be interesting to see what others use for monitoring integration, to see what is already covered with existing JMX integrations and what needs special support. (2) I think the migration story is more important - this is a non-compatible change, right? So we can't do it in 0.8.3 timeframe, it has to be in 0.9? And we need to figure out how will users migrate - do we just tell everyone please reconfigure all your monitors from scratch - don't worry, it is worth it? I know you keep saying we did it before and our users are used to it, but I think there are a lot more users now, and some of them have different compatibility expectations. We probably need to find: * A least painful way to migrate - can we keep the names of at least most of the metrics intact? * Good explanation of what users gain from this painful migration (i.e. more accurate statistics due to gazillion histograms) On Mon, Mar 30, 2015 at 6:29 PM, Jun Rao j...@confluent.io wrote: If we are committed to migrating the broker side metrics to KM for the next release, we will need to (1) have a story on supporting common reporters (as listed in KAFKA-1930), and (2) see if the current histogram support is good enough for measuring things like request time. Thanks, Jun On Mon, Mar 30, 2015 at 3:03 PM, Aditya Auradkar aaurad...@linkedin.com.invalid wrote: If we do plan to use the network code in client, I think that is a good reason in favor of migration. It will be unnecessary to have metrics from multiple libraries coexist since our users will have to start monitoring these new metrics anyway. I also agree with Jay that in multi-tenant clusters people care about detailed statistics for their own application over global numbers. Based on the arguments so far, I'm +1 for migrating to KM. Thanks, Aditya From: Jun Rao [j...@confluent.io] Sent: Sunday, March 29, 2015 9:44 AM To: dev@kafka.apache.org Subject: Re: Metrics package discussion There is another thing to consider. We plan to reuse the client components on the server side over time. For example, as part of the security work, we are looking into replacing the server side network code with the client network code (KAFKA-1928). However, the client network already has metrics based on KM. Thanks, Jun On Sat, Mar 28, 2015 at 1:34 PM, Jay Kreps jay.kr...@gmail.com wrote: I think Joel's summary is good. I'll add a few more points: As discussed memory matter a lot if we want to be able to give percentiles at the client or topic level, in which case we will have thousands of them. If we just do histograms at the global level then it is not a concern. The argument for doing histograms at the client and topic level is that averages are often very misleading, especially for latency information or other asymmetric distributions. Most people who care about this kind of thing would say the
KIP Hangout - March 31, 2015 - TL;DR;
Hi, Short notes from today's discussion for those who missed it. Attendees, feel free to correct or add: KIP-4: * Agreed to bump TopicMetadataRequest version, leave V0 with automatic topic-creation and add warnings that we are deprecating the feature in future releases. * Agreed to document all API additions and how client developers will use new API. * Decided not to have the server parse regular expressions when listing topics Quotas and Metrics: * Quotas should use new KM metrics - those will be missing from existing reporters * Security requires re-using client classes in core and this will bring more KM metrics * We don’t want to block security with metrics requirement * So we can shim KM into YM for security * After security we can start replacing everything KIP-5: Shaping up to be a very large feature. Decided that Quotas won’t need to wait on this. Waiting for a more in-depth design doc KIP-13: - Depends on KIP-4 for admin side KIP-11: - Want to get the network client reuse done first - Gwen needs to check if its possible to share - blocked on generic channel implementation Others: * Agreed to use KIP call to discuss JIRAs blocked on reviews * New replica lag is almost ready! Gwen
[jira] [Commented] (KAFKA-2078) Getting Selector [WARN] Error in I/O with host java.io.EOFException
[ https://issues.apache.org/jira/browse/KAFKA-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389231#comment-14389231 ] Aravind commented on KAFKA-2078: Sorry, above error is for some other test. For this test there were many java.io.IOException: Connection reset by peer java.io.IOException: Connection reset by peer at sun.nio.ch.FileChannelImpl.transferTo0(Native Method) at sun.nio.ch.FileChannelImpl.transferToDirectly(FileChannelImpl.java:433) at sun.nio.ch.FileChannelImpl.transferTo(FileChannelImpl.java:565) at kafka.log.FileMessageSet.writeTo(FileMessageSet.scala:147) at kafka.api.PartitionDataSend.writeTo(FetchResponse.scala:70) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.TopicDataSend.writeTo(FetchResponse.scala:125) at kafka.network.MultiSend.writeTo(Transmission.scala:101) at kafka.api.FetchResponseSend.writeTo(FetchResponse.scala:231) at kafka.network.Processor.write(SocketServer.scala:472) at kafka.network.Processor.run(SocketServer.scala:342) at java.lang.Thread.run(Thread.java:744) Thanks, Aravind Getting Selector [WARN] Error in I/O with host java.io.EOFException --- Key: KAFKA-2078 URL: https://issues.apache.org/jira/browse/KAFKA-2078 Project: Kafka Issue Type: Bug Components: producer Affects Versions: 0.8.2.0 Environment: OS Version: 2.6.39-400.209.1.el5uek and Hardware: 8 x Intel(R) Xeon(R) CPU X5660 @ 2.80GHz/44GB Reporter: Aravind Assignee: Jun Rao When trying to Produce 1000 (10 MB) messages, getting this below error some where between 997 to 1000th message. There is no pattern but able to reproduce. [PDT] 2015-03-31 13:53:50 Selector [WARN] Error in I/O with our host java.io.EOFException at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62) at org.apache.kafka.common.network.Selector.poll(Selector.java:248) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) at java.lang.Thread.run(Thread.java:724) This error I am getting some times @ 997th message or 999th message. There is no pattern but able to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] KIP-11- Authorization design for kafka security
21. Operation: What about other types of requests not covered in the list, such as committing and fetching offsets, list topics, fetching consumer metadata, heartbeat, join group, etc? Would “CONFIGURE”, “DESCRIBE”, etc take care of this? Or should we add high level grouping like “ADMIN”, “OPERATIONS/MANAGEMENT” to cover related permissions? Bosco On 3/31/15, 9:21 AM, Jun Rao j...@confluent.io wrote: Thanks for the writeup. A few more comments. 20. I agree that it would be better to do this after KIP-4 (admin commands) is done. With KIP-4, all admin operations will be sent as requests to the brokers instead of accessing ZK directly. This will make authorization easier. 21. Operation: What about other types of requests not covered in the list, such as committing and fetching offsets, list topics, fetching consumer metadata, heartbeat, join group, etc? 22. TopicConfigCache: We will need such a cache in KIP-4 as well. It would be useful to make sure that the implementation can be reused. 23. Authorizer: 23.1 Do cluster level operations go through authorize() too? If so, what will be the resource? 23.2 I assume that the authorize() check will be called on every request. So, we will have to make sure that the check is cheap. 24. The acl json string in the config: Should we version this so that we can evolve it in the future (e.g., adding group support)? Jun On Sun, Mar 29, 2015 at 3:56 PM, Parth Brahmbhatt pbrahmbh...@hortonworks.com wrote: Hi Gwen, Thanks a lot for taking the time to review this. I have tried to address all your questions below. Thanks Parth On 3/28/15, 8:08 PM, Gwen Shapira gshap...@cloudera.commailto: gshap...@cloudera.com wrote: Preparing for Tuesday meeting, I went over the KIP :) First, Parth did an amazing job, the KIP is fantastic - detailed and readable. Thank you! Second, I have a lng list of questions :) No objections, just some things I'm unclear on and random minor comments. In general, I like the design, I just feel I'm missing parts of the picture. 1. Yes, Create topic will have an optional acls, the output of describe will display owner and acls and alter topic will allow to modify the acls.” - will be nice to see what the CLI will look like. * I will modify the KIP but I was going to add “—acl acl-file.json” to create-topic and alter-topic. 2. I like the addition of Topic owner. We made the mistake of forgetting about it when adding authorization to Sqoop2. We probably want to add “chown” command to the topic commands. * Again we can add “—owner user-name” to alter topic. 3. Kafka server will read authorizer.class” config value at startup time, create an instance of the specified class and call initialize method. We’ll need to validate that users specify only one of those. * The config type will be string so type validation should take care of it. 4. One added assumption is that on non-secure connections the session will have principal set to an object whose name() method will return Anonymous”. Can we keep DrWho? :) * Sure, its up to you actually as you are the owner of the jira that introduces session concept. 5. For cluster actions that do not apply to a specific topic like CREATE we have 2 options. We can either add a broker config called broker.acls which will point to a json file. This file will be available on all broker hosts and authorizer will read the acls on initialization and keep refreshing it every X minutes. Any changes will require re-distribution of the acl json file. Alternatively we can add a zookeeper path /brokers/acls and store the acl json as data. Authorizer can refresh the acl from json every X minutes. In absence of broker acls the authorizer will fail open, in other words it will allow all users from all hosts to perform all cluster actions” I prefer a file to ZK - since thats where we store all use-defined configurations for now. Everyone knows how to secure a file system :) * I will let everyone vote, file system is fine by me. 6. When an Acl is missing , this implementation will always fail open for backward compatibility. “ - agree, but we need to document that this makes the default authorizer non-secure * Sure. 7. If the value of authorizer.class.name is null, in secure mode the cluster will fail with ConfigException. In non secure mode in absence of config value forauthorizer.class.name the server will allow all requests to all topics that , even if the topic has configured acls” - I don’t think Kafka has “secure mode” - it can support SSL and plaintext (un-authenticated) on two different ports simultaneously. I don’t object to adding such configuration, but we need to decide exactly what it means. * This is one area of confusion so I will add an open question. 8. Currently all topic creation/modification/deletion actions are performed using KafkaAdminUtil which mostly interacts directly with zookeeper instead of
[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.
[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388879#comment-14388879 ] Jay Kreps commented on KAFKA-2076: -- This makes sense, and would be useful. The question is whether we want to do this now or later. We have talked about generalizing the existing offset request and exposing that (e.g. to allow also querying for the first offset, and getting offsets by time). If we are going to do that we almost need to think through that full api, as this will be closely related. However I am not very happy with either the protocol that that uses (the request is super hacky) and the way we assign times is quite buggy (since it is based on server time not event time), so I am not wild about exposing it until we have a better story. Two corner cases to consider the behavior for: 1. I ask for the log end offset for a partition I am not subscribed to 2. I ask for the log end offset for a partition I am subscribed to but for which a fetch request has not yet been issued Add an API to new consumer to allow user get high watermark of partitions. -- Key: KAFKA-2076 URL: https://issues.apache.org/jira/browse/KAFKA-2076 Project: Kafka Issue Type: Improvement Reporter: Jiangjie Qin We have a use case that user wants to know how far it is behind a particular partition on startup. Currently in each fetch response, we have high watermark for each partition, we only keep a global max-lag metric. It would be better that we keep a record of high watermark per partition and update it on each fetch response. We can add a new API to let user query the high watermark. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2035) Add a topic config cache.
[ https://issues.apache.org/jira/browse/KAFKA-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388945#comment-14388945 ] Parth Brahmbhatt commented on KAFKA-2035: - Updated reviewboard https://reviews.apache.org/r/32460/diff/ against branch origin/trunk Add a topic config cache. - Key: KAFKA-2035 URL: https://issues.apache.org/jira/browse/KAFKA-2035 Project: Kafka Issue Type: Task Reporter: Parth Brahmbhatt Assignee: Parth Brahmbhatt Attachments: KAFKA-2035_2015-03-31_10:52:12.patch Currently the topic config is all about Log configuration so we have a TopicConfigManager which takes in a Log instance and keeps updating that instance's config instance as and when the topic config is updated. The topic config update notifications are sent using zk watchers by Controller. I propose to introduce a TopicConfigCache which will be updated by TopicConfigManager on any config changes. The log instance and any other component (like the authorizer mentioned in KAFKA-1688) will have a reference to TopicConfigCache using which they will access the topic configs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.
[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14388947#comment-14388947 ] Grant Henke commented on KAFKA-2076: This functionality and the edge cases that [~jkreps] mentioned was part of the motivation behind KIP-17: https://cwiki.apache.org/confluence/display/KAFKA/KIP-17+-+Add+HighwaterMarkOffset+to+OffsetFetchResponse Perhaps the title of that KIP should be made more generic, and the content adjusted, and the discussion around this can happen there? Add an API to new consumer to allow user get high watermark of partitions. -- Key: KAFKA-2076 URL: https://issues.apache.org/jira/browse/KAFKA-2076 Project: Kafka Issue Type: Improvement Reporter: Jiangjie Qin We have a use case that user wants to know how far it is behind a particular partition on startup. Currently in each fetch response, we have high watermark for each partition, we only keep a global max-lag metric. It would be better that we keep a record of high watermark per partition and update it on each fetch response. We can add a new API to let user query the high watermark. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] KIP-11- Authorization design for kafka security
Thanks for reviewing, comments inline: On 3/31/15, 9:21 AM, Jun Rao j...@confluent.iomailto:j...@confluent.io wrote: Thanks for the writeup. A few more comments. 20. I agree that it would be better to do this after KIP-4 (admin commands) is done. With KIP-4, all admin operations will be sent as requests to the brokers instead of accessing ZK directly. This will make authorization easier. 21. Operation: What about other types of requests not covered in the list, such as committing and fetching offsets, list topics, fetching consumer metadata, heartbeat, join group, etc? * I was actually considering any kind of write (like commit offset) as WRITE operation, and kind of read (fetching offset, get consumer metadata) as READ and any kind of list(list topics) as DESCRIBE. We can either create a one to one mapping between API and operation or classify each API as one of the operation. I was going with the classification but if you think one to one mapping will be easier to understand I am open to that. 22. TopicConfigCache: We will need such a cache in KIP-4 as well. It would be useful to make sure that the implementation can be reused. * I already opened a separate jirahttps://issues.apache.org/jira/browse/KAFKA-2035 for this and posted a reviewhttps://reviews.apache.org/r/32460/diff/#. I plan to add Acl and owner as instance variables of TopicConfig class as part of authZ patch. 23. Authorizer: 23.1 Do cluster level operations go through authorize() too? If so, what will be the resource? * Yes and I was considering to use a constant string like “Kafka-Cluster” for cluster operations. 23.2 I assume that the authorize() check will be called on every request. So, we will have to make sure that the check is cheap. * Yes , that is why by design we will trade off for speed and cache all acls, which means if you update acls it may take a few minutes before the changes take effect. 24. The acl json string in the config: Should we version this so that we can evolve it in the future (e.g., adding group support)? * I am looking into this right now but this seemed like implementation details so I did not capture it in design. I will update the json format once I have settled on a solution. What are your thoughts on using some existing libraries that support json parsing with versioning? The current json encoding/decoding used by kafka is already failing for me when I try to parse a map that has an already json encoded string as value for some key. Jun On Sun, Mar 29, 2015 at 3:56 PM, Parth Brahmbhatt pbrahmbh...@hortonworks.commailto:pbrahmbh...@hortonworks.com wrote: Hi Gwen, Thanks a lot for taking the time to review this. I have tried to address all your questions below. Thanks Parth On 3/28/15, 8:08 PM, Gwen Shapira gshap...@cloudera.commailto:gshap...@cloudera.commailto: gshap...@cloudera.commailto:gshap...@cloudera.com wrote: Preparing for Tuesday meeting, I went over the KIP :) First, Parth did an amazing job, the KIP is fantastic - detailed and readable. Thank you! Second, I have a lng list of questions :) No objections, just some things I'm unclear on and random minor comments. In general, I like the design, I just feel I'm missing parts of the picture. 1. Yes, Create topic will have an optional acls, the output of describe will display owner and acls and alter topic will allow to modify the acls.” - will be nice to see what the CLI will look like. * I will modify the KIP but I was going to add “—acl acl-file.json” to create-topic and alter-topic. 2. I like the addition of Topic owner. We made the mistake of forgetting about it when adding authorization to Sqoop2. We probably want to add “chown” command to the topic commands. * Again we can add “—owner user-name” to alter topic. 3. Kafka server will read authorizer.class” config value at startup time, create an instance of the specified class and call initialize method. We’ll need to validate that users specify only one of those. * The config type will be string so type validation should take care of it. 4. One added assumption is that on non-secure connections the session will have principal set to an object whose name() method will return Anonymous”. Can we keep DrWho? :) * Sure, its up to you actually as you are the owner of the jira that introduces session concept. 5. For cluster actions that do not apply to a specific topic like CREATE we have 2 options. We can either add a broker config called broker.acls which will point to a json file. This file will be available on all broker hosts and authorizer will read the acls on initialization and keep refreshing it every X minutes. Any changes will require re-distribution of the acl json file. Alternatively we can add a zookeeper path /brokers/acls and store the acl json as data. Authorizer can refresh the acl from json every X minutes. In absence of broker acls the authorizer will fail open, in other words it will allow
[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.
[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389020#comment-14389020 ] Jay Kreps commented on KAFKA-2076: -- Hey [~granthenke] I don't really get the relationship between that and the existing offset request, it would be good to flesh that out. That existing request serves the same purpose and also does other things but KIP-17 doesn't actually replace all those use cases. Add an API to new consumer to allow user get high watermark of partitions. -- Key: KAFKA-2076 URL: https://issues.apache.org/jira/browse/KAFKA-2076 Project: Kafka Issue Type: Improvement Reporter: Jiangjie Qin We have a use case that user wants to know how far it is behind a particular partition on startup. Currently in each fetch response, we have high watermark for each partition, we only keep a global max-lag metric. It would be better that we keep a record of high watermark per partition and update it on each fetch response. We can add a new API to let user query the high watermark. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-2078) Getting Selector [WARN] Error in I/O with host java.io.EOFException
[ https://issues.apache.org/jira/browse/KAFKA-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14387612#comment-14387612 ] Aravind edited comment on KAFKA-2078 at 3/31/15 5:18 PM: - Hi Harsha, thanks for reply. On a other note: This is occurring only when the size of the message is 10 MB and more. 1 MB, 2 MB, 5 MB are good. Memory on producer client is 6 GB. below are the details: Producer config: kafka.compression.codec=gzip producer.metadata.broker.list=host:9092 producer.logger=KafkaProducer producer.max.request.size=104857600 producer.batch.num.messages=50 producer.queue.buffering.max.ms=5000 Server.properties: broker.id=0 port=9092 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/data/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 log.retention.hours=1 log.segment.bytes=1073741824 log.retention.check.interval.ms=30 log.cleaner.enable=false zookeeper.connect=host1:2181 zookeeper.connection.timeout.ms=6000 message.max.bytes=104857600 replica.fetch.max.bytes=104857600 Thanks, Aravind was (Author: aravind2015): Hi Harsha, thanks for reply. below are the details: Producer config: kafka.compression.codec=gzip producer.metadata.broker.list=host:9092 producer.logger=KafkaProducer producer.max.request.size=1073741824 producer.batch.num.messages=50 producer.queue.buffering.max.ms=5000 Server.properties: broker.id=0 port=9092 num.network.threads=3 num.io.threads=8 socket.send.buffer.bytes=102400 socket.receive.buffer.bytes=102400 socket.request.max.bytes=104857600 log.dirs=/data/kafka-logs num.partitions=1 num.recovery.threads.per.data.dir=1 log.retention.hours=1 log.segment.bytes=1073741824 log.retention.check.interval.ms=30 log.cleaner.enable=false zookeeper.connect=host1:2181 zookeeper.connection.timeout.ms=6000 message.max.bytes=104857600 replica.fetch.max.bytes=104857600 Thanks, Aravind Getting Selector [WARN] Error in I/O with host java.io.EOFException --- Key: KAFKA-2078 URL: https://issues.apache.org/jira/browse/KAFKA-2078 Project: Kafka Issue Type: Bug Components: producer Affects Versions: 0.8.2.0 Environment: OS Version: 2.6.39-400.209.1.el5uek and Hardware: 8 x Intel(R) Xeon(R) CPU X5660 @ 2.80GHz/44GB Reporter: Aravind Assignee: Jun Rao When trying to Produce 1000 (10 MB) messages, getting this below error some where between 997 to 1000th message. There is no pattern but able to reproduce. [PDT] 2015-03-31 13:53:50 Selector [WARN] Error in I/O with our host java.io.EOFException at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62) at org.apache.kafka.common.network.Selector.poll(Selector.java:248) at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) at java.lang.Thread.run(Thread.java:724) This error I am getting some times @ 997th message or 999th message. There is no pattern but able to reproduce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2035) Add a topic config cache.
[ https://issues.apache.org/jira/browse/KAFKA-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Brahmbhatt updated KAFKA-2035: Attachment: KAFKA-2035_2015-03-31_10:52:12.patch Add a topic config cache. - Key: KAFKA-2035 URL: https://issues.apache.org/jira/browse/KAFKA-2035 Project: Kafka Issue Type: Task Reporter: Parth Brahmbhatt Assignee: Parth Brahmbhatt Attachments: KAFKA-2035_2015-03-31_10:52:12.patch Currently the topic config is all about Log configuration so we have a TopicConfigManager which takes in a Log instance and keeps updating that instance's config instance as and when the topic config is updated. The topic config update notifications are sent using zk watchers by Controller. I propose to introduce a TopicConfigCache which will be updated by TopicConfigManager on any config changes. The log instance and any other component (like the authorizer mentioned in KAFKA-1688) will have a reference to TopicConfigCache using which they will access the topic configs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2035) Add a topic config cache.
[ https://issues.apache.org/jira/browse/KAFKA-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Parth Brahmbhatt updated KAFKA-2035: Status: Patch Available (was: Open) Add a topic config cache. - Key: KAFKA-2035 URL: https://issues.apache.org/jira/browse/KAFKA-2035 Project: Kafka Issue Type: Task Reporter: Parth Brahmbhatt Assignee: Parth Brahmbhatt Attachments: KAFKA-2035_2015-03-31_10:52:12.patch Currently the topic config is all about Log configuration so we have a TopicConfigManager which takes in a Log instance and keeps updating that instance's config instance as and when the topic config is updated. The topic config update notifications are sent using zk watchers by Controller. I propose to introduce a TopicConfigCache which will be updated by TopicConfigManager on any config changes. The log instance and any other component (like the authorizer mentioned in KAFKA-1688) will have a reference to TopicConfigCache using which they will access the topic configs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] KIP-11- Authorization design for kafka security
From the design doc , one of the added config: * kafka.superusers: list of users that will be given superuser access. These users will have access to everything. Users should set this to the user kafka broker processes are running as to avoid duplicate configuration for every single topic like ALLOW REPLICATION to BROKER_USER for TOPIC from ALL hosts. Thanks Parth On 3/31/15, 10:20 AM, Gwen Shapira gshap...@cloudera.com wrote: Related interesting question: Since a broker is a consumer (of lead replicas), how do we handle the broker level of permissions? Do we hardcode a broker-principal name and automatically authorize brokers to do anything? Or is there a cleaner way? On Tue, Mar 31, 2015 at 10:17 AM, Don Bosco Durai bo...@apache.org wrote: 21. Operation: What about other types of requests not covered in the list, such as committing and fetching offsets, list topics, fetching consumer metadata, heartbeat, join group, etc? Would “CONFIGURE”, “DESCRIBE”, etc take care of this? Or should we add high level grouping like “ADMIN”, “OPERATIONS/MANAGEMENT” to cover related permissions? Bosco On 3/31/15, 9:21 AM, Jun Rao j...@confluent.io wrote: Thanks for the writeup. A few more comments. 20. I agree that it would be better to do this after KIP-4 (admin commands) is done. With KIP-4, all admin operations will be sent as requests to the brokers instead of accessing ZK directly. This will make authorization easier. 21. Operation: What about other types of requests not covered in the list, such as committing and fetching offsets, list topics, fetching consumer metadata, heartbeat, join group, etc? 22. TopicConfigCache: We will need such a cache in KIP-4 as well. It would be useful to make sure that the implementation can be reused. 23. Authorizer: 23.1 Do cluster level operations go through authorize() too? If so, what will be the resource? 23.2 I assume that the authorize() check will be called on every request. So, we will have to make sure that the check is cheap. 24. The acl json string in the config: Should we version this so that we can evolve it in the future (e.g., adding group support)? Jun On Sun, Mar 29, 2015 at 3:56 PM, Parth Brahmbhatt pbrahmbh...@hortonworks.com wrote: Hi Gwen, Thanks a lot for taking the time to review this. I have tried to address all your questions below. Thanks Parth On 3/28/15, 8:08 PM, Gwen Shapira gshap...@cloudera.commailto: gshap...@cloudera.com wrote: Preparing for Tuesday meeting, I went over the KIP :) First, Parth did an amazing job, the KIP is fantastic - detailed and readable. Thank you! Second, I have a lng list of questions :) No objections, just some things I'm unclear on and random minor comments. In general, I like the design, I just feel I'm missing parts of the picture. 1. Yes, Create topic will have an optional acls, the output of describe will display owner and acls and alter topic will allow to modify the acls.” - will be nice to see what the CLI will look like. * I will modify the KIP but I was going to add “—acl acl-file.json” to create-topic and alter-topic. 2. I like the addition of Topic owner. We made the mistake of forgetting about it when adding authorization to Sqoop2. We probably want to add “chown” command to the topic commands. * Again we can add “—owner user-name” to alter topic. 3. Kafka server will read authorizer.class” config value at startup time, create an instance of the specified class and call initialize method. We’ll need to validate that users specify only one of those. * The config type will be string so type validation should take care of it. 4. One added assumption is that on non-secure connections the session will have principal set to an object whose name() method will return Anonymous”. Can we keep DrWho? :) * Sure, its up to you actually as you are the owner of the jira that introduces session concept. 5. For cluster actions that do not apply to a specific topic like CREATE we have 2 options. We can either add a broker config called broker.acls which will point to a json file. This file will be available on all broker hosts and authorizer will read the acls on initialization and keep refreshing it every X minutes. Any changes will require re-distribution of the acl json file. Alternatively we can add a zookeeper path /brokers/acls and store the acl json as data. Authorizer can refresh the acl from json every X minutes. In absence of broker acls the authorizer will fail open, in other words it will allow all users from all hosts to perform all cluster actions” I prefer a file to ZK - since thats where we store all use-defined configurations for now. Everyone knows how to secure a file system :) * I will let everyone vote, file system is fine by me. 6. When an Acl is missing , this implementation will always fail open for backward compatibility. “ - agree, but we need to document that this makes the default
Re: [DISCUSS] KIP-11- Authorization design for kafka security
Related interesting question: Since a broker is a consumer (of lead replicas), how do we handle the broker level of permissions? Do we hardcode a broker-principal name and automatically authorize brokers to do anything? Or is there a cleaner way? I feel, in Kerberos environment, “kafka” keytab would be the ideal solution. And “kafka” principal will need to be white listed. SSL certificate is another option, but it would be painful to set it up. IP whitelisting is another low impact, but less secure option. Bosco On 3/31/15, 10:20 AM, Gwen Shapira gshap...@cloudera.com wrote: Related interesting question: Since a broker is a consumer (of lead replicas), how do we handle the broker level of permissions? Do we hardcode a broker-principal name and automatically authorize brokers to do anything? Or is there a cleaner way? On Tue, Mar 31, 2015 at 10:17 AM, Don Bosco Durai bo...@apache.org wrote: 21. Operation: What about other types of requests not covered in the list, such as committing and fetching offsets, list topics, fetching consumer metadata, heartbeat, join group, etc? Would “CONFIGURE”, “DESCRIBE”, etc take care of this? Or should we add high level grouping like “ADMIN”, “OPERATIONS/MANAGEMENT” to cover related permissions? Bosco On 3/31/15, 9:21 AM, Jun Rao j...@confluent.io wrote: Thanks for the writeup. A few more comments. 20. I agree that it would be better to do this after KIP-4 (admin commands) is done. With KIP-4, all admin operations will be sent as requests to the brokers instead of accessing ZK directly. This will make authorization easier. 21. Operation: What about other types of requests not covered in the list, such as committing and fetching offsets, list topics, fetching consumer metadata, heartbeat, join group, etc? 22. TopicConfigCache: We will need such a cache in KIP-4 as well. It would be useful to make sure that the implementation can be reused. 23. Authorizer: 23.1 Do cluster level operations go through authorize() too? If so, what will be the resource? 23.2 I assume that the authorize() check will be called on every request. So, we will have to make sure that the check is cheap. 24. The acl json string in the config: Should we version this so that we can evolve it in the future (e.g., adding group support)? Jun On Sun, Mar 29, 2015 at 3:56 PM, Parth Brahmbhatt pbrahmbh...@hortonworks.com wrote: Hi Gwen, Thanks a lot for taking the time to review this. I have tried to address all your questions below. Thanks Parth On 3/28/15, 8:08 PM, Gwen Shapira gshap...@cloudera.commailto: gshap...@cloudera.com wrote: Preparing for Tuesday meeting, I went over the KIP :) First, Parth did an amazing job, the KIP is fantastic - detailed and readable. Thank you! Second, I have a lng list of questions :) No objections, just some things I'm unclear on and random minor comments. In general, I like the design, I just feel I'm missing parts of the picture. 1. Yes, Create topic will have an optional acls, the output of describe will display owner and acls and alter topic will allow to modify the acls.” - will be nice to see what the CLI will look like. * I will modify the KIP but I was going to add “—acl acl-file.json” to create-topic and alter-topic. 2. I like the addition of Topic owner. We made the mistake of forgetting about it when adding authorization to Sqoop2. We probably want to add “chown” command to the topic commands. * Again we can add “—owner user-name” to alter topic. 3. Kafka server will read authorizer.class” config value at startup time, create an instance of the specified class and call initialize method. We’ll need to validate that users specify only one of those. * The config type will be string so type validation should take care of it. 4. One added assumption is that on non-secure connections the session will have principal set to an object whose name() method will return Anonymous”. Can we keep DrWho? :) * Sure, its up to you actually as you are the owner of the jira that introduces session concept. 5. For cluster actions that do not apply to a specific topic like CREATE we have 2 options. We can either add a broker config called broker.acls which will point to a json file. This file will be available on all broker hosts and authorizer will read the acls on initialization and keep refreshing it every X minutes. Any changes will require re-distribution of the acl json file. Alternatively we can add a zookeeper path /brokers/acls and store the acl json as data. Authorizer can refresh the acl from json every X minutes. In absence of broker acls the authorizer will fail open, in other words it will allow all users from all hosts to perform all cluster actions” I prefer a file to ZK - since thats where we store all use-defined configurations for now. Everyone knows how to secure a file system :) * I will let everyone vote, file system is fine by me. 6. When an Acl is
Re: [DISCUSS] KIP-11- Authorization design for kafka security
Related interesting question: Since a broker is a consumer (of lead replicas), how do we handle the broker level of permissions? Do we hardcode a broker-principal name and automatically authorize brokers to do anything? Or is there a cleaner way? On Tue, Mar 31, 2015 at 10:17 AM, Don Bosco Durai bo...@apache.org wrote: 21. Operation: What about other types of requests not covered in the list, such as committing and fetching offsets, list topics, fetching consumer metadata, heartbeat, join group, etc? Would “CONFIGURE”, “DESCRIBE”, etc take care of this? Or should we add high level grouping like “ADMIN”, “OPERATIONS/MANAGEMENT” to cover related permissions? Bosco On 3/31/15, 9:21 AM, Jun Rao j...@confluent.io wrote: Thanks for the writeup. A few more comments. 20. I agree that it would be better to do this after KIP-4 (admin commands) is done. With KIP-4, all admin operations will be sent as requests to the brokers instead of accessing ZK directly. This will make authorization easier. 21. Operation: What about other types of requests not covered in the list, such as committing and fetching offsets, list topics, fetching consumer metadata, heartbeat, join group, etc? 22. TopicConfigCache: We will need such a cache in KIP-4 as well. It would be useful to make sure that the implementation can be reused. 23. Authorizer: 23.1 Do cluster level operations go through authorize() too? If so, what will be the resource? 23.2 I assume that the authorize() check will be called on every request. So, we will have to make sure that the check is cheap. 24. The acl json string in the config: Should we version this so that we can evolve it in the future (e.g., adding group support)? Jun On Sun, Mar 29, 2015 at 3:56 PM, Parth Brahmbhatt pbrahmbh...@hortonworks.com wrote: Hi Gwen, Thanks a lot for taking the time to review this. I have tried to address all your questions below. Thanks Parth On 3/28/15, 8:08 PM, Gwen Shapira gshap...@cloudera.commailto: gshap...@cloudera.com wrote: Preparing for Tuesday meeting, I went over the KIP :) First, Parth did an amazing job, the KIP is fantastic - detailed and readable. Thank you! Second, I have a lng list of questions :) No objections, just some things I'm unclear on and random minor comments. In general, I like the design, I just feel I'm missing parts of the picture. 1. Yes, Create topic will have an optional acls, the output of describe will display owner and acls and alter topic will allow to modify the acls.” - will be nice to see what the CLI will look like. * I will modify the KIP but I was going to add “—acl acl-file.json” to create-topic and alter-topic. 2. I like the addition of Topic owner. We made the mistake of forgetting about it when adding authorization to Sqoop2. We probably want to add “chown” command to the topic commands. * Again we can add “—owner user-name” to alter topic. 3. Kafka server will read authorizer.class” config value at startup time, create an instance of the specified class and call initialize method. We’ll need to validate that users specify only one of those. * The config type will be string so type validation should take care of it. 4. One added assumption is that on non-secure connections the session will have principal set to an object whose name() method will return Anonymous”. Can we keep DrWho? :) * Sure, its up to you actually as you are the owner of the jira that introduces session concept. 5. For cluster actions that do not apply to a specific topic like CREATE we have 2 options. We can either add a broker config called broker.acls which will point to a json file. This file will be available on all broker hosts and authorizer will read the acls on initialization and keep refreshing it every X minutes. Any changes will require re-distribution of the acl json file. Alternatively we can add a zookeeper path /brokers/acls and store the acl json as data. Authorizer can refresh the acl from json every X minutes. In absence of broker acls the authorizer will fail open, in other words it will allow all users from all hosts to perform all cluster actions” I prefer a file to ZK - since thats where we store all use-defined configurations for now. Everyone knows how to secure a file system :) * I will let everyone vote, file system is fine by me. 6. When an Acl is missing , this implementation will always fail open for backward compatibility. “ - agree, but we need to document that this makes the default authorizer non-secure * Sure. 7. If the value of authorizer.class.name is null, in secure mode the cluster will fail with ConfigException. In non secure mode in absence of config value forauthorizer.class.name the server will allow all requests to all topics that , even if the topic has configured acls” - I don’t think Kafka has “secure mode” - it can support SSL and plaintext (un-authenticated) on two different ports
[jira] [Commented] (KAFKA-1910) Refactor KafkaConsumer
[ https://issues.apache.org/jira/browse/KAFKA-1910?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=1433#comment-1433 ] Joel Koshy commented on KAFKA-1910: --- BTW, just to be clear, what I would like to discuss is: * Is the error code change necessary? * If it is, then should we bump up the OffsetFetchRequest version (without actually changing the request content)? i.e., since the response is different across different broker versions. Refactor KafkaConsumer -- Key: KAFKA-1910 URL: https://issues.apache.org/jira/browse/KAFKA-1910 Project: Kafka Issue Type: Sub-task Components: consumer Reporter: Guozhang Wang Assignee: Guozhang Wang Fix For: 0.8.3 Attachments: KAFKA-1910.patch, KAFKA-1910.patch, KAFKA-1910_2015-03-05_14:55:33.patch KafkaConsumer now contains all the logic on the consumer side, making it a very huge class file, better re-factoring it to have multiple layers on top of KafkaClient. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] KIP-11- Authorization design for kafka security
Yes in case of kerberos we will use superACL and this will be equivalent to kafka broker’s principal name. But in SSL as two-way auth is not mandatory the only option if we want enforce authorizer in case of ssl is to force two-way auth. Again this can be an issue on client side , lets say if a producer doesn’t want to provide client auth and just needs wire encryption there won’t be any identity , in this case and we won’t be able to enforce an authorizer as the client will be anonymous. -- Harsha On March 31, 2015 at 10:29:33 AM, Don Bosco Durai (bo...@apache.org) wrote: Related interesting question: Since a broker is a consumer (of lead replicas), how do we handle the broker level of permissions? Do we hardcode a broker-principal name and automatically authorize brokers to do anything? Or is there a cleaner way? I feel, in Kerberos environment, “kafka” keytab would be the ideal solution. And “kafka” principal will need to be white listed. SSL certificate is another option, but it would be painful to set it up. IP whitelisting is another low impact, but less secure option. Bosco On 3/31/15, 10:20 AM, Gwen Shapira gshap...@cloudera.com wrote: Related interesting question: Since a broker is a consumer (of lead replicas), how do we handle the broker level of permissions? Do we hardcode a broker-principal name and automatically authorize brokers to do anything? Or is there a cleaner way? On Tue, Mar 31, 2015 at 10:17 AM, Don Bosco Durai bo...@apache.org wrote: 21. Operation: What about other types of requests not covered in the list, such as committing and fetching offsets, list topics, fetching consumer metadata, heartbeat, join group, etc? Would “CONFIGURE”, “DESCRIBE”, etc take care of this? Or should we add high level grouping like “ADMIN”, “OPERATIONS/MANAGEMENT” to cover related permissions? Bosco On 3/31/15, 9:21 AM, Jun Rao j...@confluent.io wrote: Thanks for the writeup. A few more comments. 20. I agree that it would be better to do this after KIP-4 (admin commands) is done. With KIP-4, all admin operations will be sent as requests to the brokers instead of accessing ZK directly. This will make authorization easier. 21. Operation: What about other types of requests not covered in the list, such as committing and fetching offsets, list topics, fetching consumer metadata, heartbeat, join group, etc? 22. TopicConfigCache: We will need such a cache in KIP-4 as well. It would be useful to make sure that the implementation can be reused. 23. Authorizer: 23.1 Do cluster level operations go through authorize() too? If so, what will be the resource? 23.2 I assume that the authorize() check will be called on every request. So, we will have to make sure that the check is cheap. 24. The acl json string in the config: Should we version this so that we can evolve it in the future (e.g., adding group support)? Jun On Sun, Mar 29, 2015 at 3:56 PM, Parth Brahmbhatt pbrahmbh...@hortonworks.com wrote: Hi Gwen, Thanks a lot for taking the time to review this. I have tried to address all your questions below. Thanks Parth On 3/28/15, 8:08 PM, Gwen Shapira gshap...@cloudera.commailto: gshap...@cloudera.com wrote: Preparing for Tuesday meeting, I went over the KIP :) First, Parth did an amazing job, the KIP is fantastic - detailed and readable. Thank you! Second, I have a lng list of questions :) No objections, just some things I'm unclear on and random minor comments. In general, I like the design, I just feel I'm missing parts of the picture. 1. Yes, Create topic will have an optional acls, the output of describe will display owner and acls and alter topic will allow to modify the acls.” - will be nice to see what the CLI will look like. * I will modify the KIP but I was going to add “—acl acl-file.json” to create-topic and alter-topic. 2. I like the addition of Topic owner. We made the mistake of forgetting about it when adding authorization to Sqoop2. We probably want to add “chown” command to the topic commands. * Again we can add “—owner user-name” to alter topic. 3. Kafka server will read authorizer.class” config value at startup time, create an instance of the specified class and call initialize method. We’ll need to validate that users specify only one of those. * The config type will be string so type validation should take care of it. 4. One added assumption is that on non-secure connections the session will have principal set to an object whose name() method will return Anonymous”. Can we keep DrWho? :) * Sure, its up to you actually as you are the owner of the jira that introduces session concept. 5. For cluster actions that do not apply to a specific topic like CREATE we have
Re: Review Request 32460: Patch for KAFKA-2035
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/32460/ --- (Updated March 31, 2015, 5:52 p.m.) Review request for kafka. Summary (updated) - Patch for KAFKA-2035 Bugs: KAFKA-2032 and KAFKA-2035 https://issues.apache.org/jira/browse/KAFKA-2032 https://issues.apache.org/jira/browse/KAFKA-2035 Repository: kafka Description (updated) --- Merge remote-tracking branch 'origin/trunk' into 2032 KAFKA-2035: Added TopicConfigCache. Merge remote-tracking branch 'origin/trunk' into 2032 Diffs (updated) - core/src/main/scala/kafka/server/KafkaServer.scala 4db3384545be8c237d6fc9646716ab67d5193ec5 core/src/main/scala/kafka/server/TopicConfigCache.scala PRE-CREATION core/src/main/scala/kafka/server/TopicConfigManager.scala 47295d40131492aaac786273819b7bc6e22e5486 core/src/test/scala/unit/kafka/server/DynamicConfigChangeTest.scala 93182aeb342729d420d2e7d59a1035994164b7db core/src/test/scala/unit/kafka/server/TopicConfigCacheTest.scala PRE-CREATION Diff: https://reviews.apache.org/r/32460/diff/ Testing (updated) --- unit tests added. Thanks, Parth Brahmbhatt
Re: KIP Hangout - March 31, 2015 - TL;DR;
We will be using KM for quota'ing on the new client-id-specific metrics. On Tue, Mar 31, 2015 at 08:44:44PM +, Jiangjie Qin wrote: Thanks a lot for the summary, Gwen! About the Quota, does that mean the first quota implementation will be based on YM? I¹m thinking can we pursue a quota solution that has a loose coupling with metrics interfaces? Like something operating system does for FUSE, so we don¹t need to care about which underlying metric we use. In that case, we can implement the Quota base on KM and wrap YM to resemble KM. Quota management itself can also be extracted as a separate package in this case. Jiangjie (Becket) Qin On 3/31/15, 12:04 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi, Short notes from today's discussion for those who missed it. Attendees, feel free to correct or add: KIP-4: * Agreed to bump TopicMetadataRequest version, leave V0 with automatic topic-creation and add warnings that we are deprecating the feature in future releases. * Agreed to document all API additions and how client developers will use new API. * Decided not to have the server parse regular expressions when listing topics Quotas and Metrics: * Quotas should use new KM metrics - those will be missing from existing reporters * Security requires re-using client classes in core and this will bring more KM metrics * We don¹t want to block security with metrics requirement * So we can shim KM into YM for security * After security we can start replacing everything KIP-5: Shaping up to be a very large feature. Decided that Quotas won¹t need to wait on this. Waiting for a more in-depth design doc KIP-13: - Depends on KIP-4 for admin side KIP-11: - Want to get the network client reuse done first - Gwen needs to check if its possible to share - blocked on generic channel implementation Others: * Agreed to use KIP call to discuss JIRAs blocked on reviews * New replica lag is almost ready! Gwen
[jira] [Commented] (KAFKA-1546) Automate replica lag tuning
[ https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389322#comment-14389322 ] Jun Rao commented on KAFKA-1546: For the doc change, do we need to make the following changes? We didn't remove the MaxLag jmx, right? - tdMax lag in messages btw follower and leader replicas/td - tdkafka.server:type=ReplicaFetcherManager,name=MaxLag,clientId=Replica/td - tdlt replica.lag.max.messages/td -/tr -tr - tdLag in messages per follower replica/td - tdkafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=([-.\w]+),topic=([-.\w]+),partition=([0-9]+)/td - tdlt replica.lag.max.messages/td -/tr -tr tdRequests waiting in the producer purgatory/td Automate replica lag tuning --- Key: KAFKA-1546 URL: https://issues.apache.org/jira/browse/KAFKA-1546 Project: Kafka Issue Type: Improvement Components: replication Affects Versions: 0.8.0, 0.8.1, 0.8.1.1 Reporter: Neha Narkhede Assignee: Aditya Auradkar Labels: newbie++ Fix For: 0.8.3 Attachments: KAFKA-1546.patch, KAFKA-1546_2015-03-11_18:48:09.patch, KAFKA-1546_2015-03-12_13:42:01.patch, KAFKA-1546_2015-03-16_11:31:39.patch, KAFKA-1546_2015-03-17_14:46:10.patch, KAFKA-1546_2015-03-25_13:27:40.patch, KAFKA-1546_2015-03-26_17:44:08.patch, KAFKA-1546_2015-03-27_11:57:56.patch, documentation.diff Currently, there is no good way to tune the replica lag configs to automatically account for high and low volume topics on the same cluster. For the low-volume topic it will take a very long time to detect a lagging replica, and for the high-volume topic it will have false-positives. One approach to making this easier would be to have the configuration be something like replica.lag.max.ms and translate this into a number of messages dynamically based on the throughput of the partition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Can I be added as a contributor?
Done for both. Thanks, Jun On Mon, Mar 30, 2015 at 9:23 AM, Brock Noland br...@apache.org wrote: Hi, Could I be added as a contributor and to confluence? I am brocknoland on JIRA and brockn at gmail on confluence. Cheers! Brock
[jira] [Updated] (KAFKA-1546) Automate replica lag tuning
[ https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya Auradkar updated KAFKA-1546: --- Attachment: documentation.diff Automate replica lag tuning --- Key: KAFKA-1546 URL: https://issues.apache.org/jira/browse/KAFKA-1546 Project: Kafka Issue Type: Improvement Components: replication Affects Versions: 0.8.0, 0.8.1, 0.8.1.1 Reporter: Neha Narkhede Assignee: Aditya Auradkar Labels: newbie++ Fix For: 0.8.3 Attachments: KAFKA-1546.patch, KAFKA-1546_2015-03-11_18:48:09.patch, KAFKA-1546_2015-03-12_13:42:01.patch, KAFKA-1546_2015-03-16_11:31:39.patch, KAFKA-1546_2015-03-17_14:46:10.patch, KAFKA-1546_2015-03-25_13:27:40.patch, KAFKA-1546_2015-03-26_17:44:08.patch, KAFKA-1546_2015-03-27_11:57:56.patch, documentation.diff, documentation.diff Currently, there is no good way to tune the replica lag configs to automatically account for high and low volume topics on the same cluster. For the low-volume topic it will take a very long time to detect a lagging replica, and for the high-volume topic it will have false-positives. One approach to making this easier would be to have the configuration be something like replica.lag.max.ms and translate this into a number of messages dynamically based on the throughput of the partition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2016) RollingBounceTest takes long
[ https://issues.apache.org/jira/browse/KAFKA-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389421#comment-14389421 ] Gwen Shapira commented on KAFKA-2016: - I believe Ted addressed your comments, [~harsha_ch], mind taking a look? When we tested, this saves 40s. Thats pretty significant. [~junrao], any chance you can check this out? Its fairly simple but saves lots of time on our tests. RollingBounceTest takes long Key: KAFKA-2016 URL: https://issues.apache.org/jira/browse/KAFKA-2016 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.1 Reporter: Jun Rao Labels: newbie Fix For: 0.8.3 Attachments: KAFKA-2016-1.patch, KAFKA-2016-2.patch RollingBounceTest.testRollingBounce() currently takes about 48 secs. This is a bit too long. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 30809: Patch for KAFKA-1888
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30809/#review78270 --- bin/kafka-run-class.sh https://reviews.apache.org/r/30809/#comment126788 Can you remove this (and the echo that Ashish pointed out)? core/src/main/scala/kafka/tools/ContinuousValidationTest.java https://reviews.apache.org/r/30809/#comment126790 This should definitely not be in tools - this should probably live somewhere under clients/test. I don't think those are currently exported though, so we will need to modify build.gradle. However, per other comments below I'm not sure this should be part of system tests since it is (by definition long running). core/src/main/scala/kafka/tools/ContinuousValidationTest.java https://reviews.apache.org/r/30809/#comment126791 Can remove this core/src/main/scala/kafka/tools/ContinuousValidationTest.java https://reviews.apache.org/r/30809/#comment126822 It would help a lot if you could add comments describing what validation is done. For e.g., I'm unclear on why we need the complicated file-based signaling mechanism. So a high-level description would help a lot. More importantly, I really think we should separate continuous validation from broker upgrade which is the focus of KAFKA-1888 In order to do a broker upgrade test, we don't need any additional code. We just instantiate the producer performance and consumer via system test utils. Keep those on the old jar. The cluster will start with the old jar as well and during the test we bounce in the latest jar (the system test utils will need to be updated to support this). We then do the standard system test validation - that all messages sent were received. core/src/main/scala/kafka/tools/ContinuousValidationTest.java https://reviews.apache.org/r/30809/#comment126794 This appears to be for rate-limiting the producer but can be more general than that. It would help to add a comment describing its purpose. Also, should probably be private core/src/main/scala/kafka/tools/ContinuousValidationTest.java https://reviews.apache.org/r/30809/#comment126793 1 - one core/src/main/scala/kafka/tools/ContinuousValidationTest.java https://reviews.apache.org/r/30809/#comment126792 Stray println core/src/main/scala/kafka/tools/ContinuousValidationTest.java https://reviews.apache.org/r/30809/#comment126795 Stray println core/src/main/scala/kafka/tools/ContinuousValidationTest.java https://reviews.apache.org/r/30809/#comment126829 Can you use the logger formatter we use elsewhere? i.e., curly braces instead of an explicit String.format system_test/broker_upgrade/bin/test-broker-upgrade.sh https://reviews.apache.org/r/30809/#comment127013 This appears to be a one-off script to set up the test. This needs to be done within the system test framework which already has a number of utilities that do similar things. One other comment is that the patch is for an upgrade test, but I think it is a bit confusing to mix this with CVT. system_test/broker_upgrade/bin/test-broker-upgrade.sh https://reviews.apache.org/r/30809/#comment127010 may be better to just use mktemp for temporary files system_test/broker_upgrade/bin/test-broker-upgrade.sh https://reviews.apache.org/r/30809/#comment127011 should these (and below) be in basedir? That said I don't see this created anywhere. - Joel Koshy On March 23, 2015, 6:54 p.m., Abhishek Nigam wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/30809/ --- (Updated March 23, 2015, 6:54 p.m.) Review request for kafka. Bugs: KAFKA-1888 https://issues.apache.org/jira/browse/KAFKA-1888 Repository: kafka Description --- Updated the RB with Gwen's comments, Beckett's comments and a subset of Guozhang's comments Diffs - bin/kafka-run-class.sh 881f578a8f5c796fe23415b978c1ad35869af76e core/src/main/scala/kafka/tools/ContinuousValidationTest.java PRE-CREATION core/src/main/scala/kafka/utils/ShutdownableThread.scala fc226c863095b7761290292cd8755cd7ad0f155c system_test/broker_upgrade/bin/test-broker-upgrade.sh PRE-CREATION Diff: https://reviews.apache.org/r/30809/diff/ Testing --- Scripted it to run 20 times without any failures. Command-line: broker-upgrade/bin/test.sh dir1 dir2 Thanks, Abhishek Nigam
[jira] [Commented] (KAFKA-1546) Automate replica lag tuning
[ https://issues.apache.org/jira/browse/KAFKA-1546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389441#comment-14389441 ] Aditya Auradkar commented on KAFKA-1546: Good point Jun. I've added those metrics back and have changed the Normal Value section to remove references to replica.lag.max.messages. Automate replica lag tuning --- Key: KAFKA-1546 URL: https://issues.apache.org/jira/browse/KAFKA-1546 Project: Kafka Issue Type: Improvement Components: replication Affects Versions: 0.8.0, 0.8.1, 0.8.1.1 Reporter: Neha Narkhede Assignee: Aditya Auradkar Labels: newbie++ Fix For: 0.8.3 Attachments: KAFKA-1546.patch, KAFKA-1546_2015-03-11_18:48:09.patch, KAFKA-1546_2015-03-12_13:42:01.patch, KAFKA-1546_2015-03-16_11:31:39.patch, KAFKA-1546_2015-03-17_14:46:10.patch, KAFKA-1546_2015-03-25_13:27:40.patch, KAFKA-1546_2015-03-26_17:44:08.patch, KAFKA-1546_2015-03-27_11:57:56.patch, documentation.diff, documentation.diff Currently, there is no good way to tune the replica lag configs to automatically account for high and low volume topics on the same cluster. For the low-volume topic it will take a very long time to detect a lagging replica, and for the high-volume topic it will have false-positives. One approach to making this easier would be to have the configuration be something like replica.lag.max.ms and translate this into a number of messages dynamically based on the throughput of the partition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.
[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389652#comment-14389652 ] Jiangjie Qin commented on KAFKA-2076: - Got it. Thanks for the explanation, [~jkreps]. The strategy looks very good. So as the first step, I guess we can add the following interface, given we agree it is useful from user point of view. We can Implement them with current existing screwy protocols we have now. MapTopicPartition, Long latestOffsetsFor(ListTopicPartition partitions) Later on, when we have a fully baked protocol we can replace the underlying implementation. Considering we are already using OffsetRequest in new consumer (ListOffsetRequest), this probably won't add an extra unwanted protocol dependency. As for the corner cases you mentioned: 1. I ask for the log end offset for a partition I am not subscribed to. - We probably want to provide this, otherwise people will just subscribe first then get the log end offset. It adds unnecessary steps to user. 2. I ask for the log end offset for a partition I am subscribed to but for which a fetch request has not yet been issued. - My implementation proposal covers this, but we might want to add a back-off time for fetch LEO from broker, say, if the LEO in local map has not been updated for more than Log.end.offset.fetch.backoff.ms, we issue another request to broker. In a normally consuming consumer, hopefully no additional request will be sent for LEO. Any thoughts? Add an API to new consumer to allow user get high watermark of partitions. -- Key: KAFKA-2076 URL: https://issues.apache.org/jira/browse/KAFKA-2076 Project: Kafka Issue Type: Improvement Reporter: Jiangjie Qin We have a use case that user wants to know how far it is behind a particular partition on startup. Currently in each fetch response, we have high watermark for each partition, we only keep a global max-lag metric. It would be better that we keep a record of high watermark per partition and update it on each fetch response. We can add a new API to let user query the high watermark. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31893: Patch for KAFKA-2013
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31893/#review76975 --- Thanks for the new patch. I really like the desgin of this test! A few minor comments below. core/src/test/scala/other/kafka/TestPurgatoryPerformance.scala https://reviews.apache.org/r/31893/#comment127091 The Try syntax doesn't seem to be supported in scala 2.9. ./gradlew -PscalaVersion=2.9.1 testJar kafka/core/src/test/scala/other/kafka/TestPurgatoryPerformance.scala:29: Try is not a member of scala.util import scala.util.Try core/src/test/scala/other/kafka/TestPurgatoryPerformance.scala https://reviews.apache.org/r/31893/#comment127122 Could we note that all times are in millisecs? core/src/test/scala/other/kafka/TestPurgatoryPerformance.scala https://reviews.apache.org/r/31893/#comment127085 why = way core/src/test/scala/other/kafka/TestPurgatoryPerformance.scala https://reviews.apache.org/r/31893/#comment127086 Could you add a comment on what 0.674490d stands for? core/src/test/scala/other/kafka/TestPurgatoryPerformance.scala https://reviews.apache.org/r/31893/#comment127110 Now that we are completing the request in CompleteQueue explicitly, do we still need to override isCompleted()? core/src/test/scala/other/kafka/TestPurgatoryPerformance.scala https://reviews.apache.org/r/31893/#comment127104 We probably can make this a ShutdownableThread. core/src/test/scala/other/kafka/TestPurgatoryPerformance.scala https://reviews.apache.org/r/31893/#comment127102 Do we need the while loop? Could we just do a if test and go back to the outer loop? core/src/test/scala/other/kafka/TestPurgatoryPerformance.scala https://reviews.apache.org/r/31893/#comment127105 Would it be more reliable to call forceComplete() here? Not sure if delayQueue always returns an item at the precise time. core/src/test/scala/other/kafka/TestPurgatoryPerformance.scala https://reviews.apache.org/r/31893/#comment127107 Could we just put FakeOperation to the delayQueue directly instead of wrapping it under Scheduled? - Jun Rao On March 19, 2015, 11:31 p.m., Yasuhiro Matsuda wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31893/ --- (Updated March 19, 2015, 11:31 p.m.) Review request for kafka. Bugs: KAFKA-2013 https://issues.apache.org/jira/browse/KAFKA-2013 Repository: kafka Description --- purgatory micro benchmark Diffs - core/src/test/scala/other/kafka/TestPurgatoryPerformance.scala PRE-CREATION Diff: https://reviews.apache.org/r/31893/diff/ Testing --- Thanks, Yasuhiro Matsuda
[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.
[
https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389700#comment-14389700
]
Jay Kreps commented on KAFKA-2076:
--
I guess the question we need to answer is whether we will want to add other
offset-related APIs, in particular:
- get the map of known time-offset
- get the least offset
I think if we have a picture for where we want to go with this it will be
easier to know what to do now for getting the last offset.
If we want to do both of these things then maybe the api should be
{code}
PartitionOffsets offsetTimes(TopicPartition tp)
{code}
where PartitionOffsets would basically be SortedMapLong, Long mapping
timestamp to offset (with some additional helper methods to make usage clearer).
A separate question is whether the api needs to be batched or not--this should
ideally be as consistent as possible across the methods. Many of the other
methods don't need to batch as they refresh all subscribed partitions whenever
you ask for any one, so the actual lookup is just in-memory. But this approach
may not work here as I think we are saying we want you to be able to check
non-subscribed partitions.
Thoughts?
Add an API to new consumer to allow user get high watermark of partitions.
--
Key: KAFKA-2076
URL: https://issues.apache.org/jira/browse/KAFKA-2076
Project: Kafka
Issue Type: Improvement
Reporter: Jiangjie Qin
We have a use case that user wants to know how far it is behind a particular
partition on startup. Currently in each fetch response, we have high
watermark for each partition, we only keep a global max-lag metric. It would
be better that we keep a record of high watermark per partition and update it
on each fetch response. We can add a new API to let user query the high
watermark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Any idea why acceptor is volatile in SocketServer?
Hi, In SocketServer.scala, acceptor is defined as a volatile var: https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/network/SocketServer.scala#L54 My understanding is that @volatile is used to protect variables that are accessed and modified by multiple threads. However, it looks like acceptor variable is only accessed by the main KafkaServer thread. Am I missing anything? What are we protecting the acceptor from? The reason I'm asking is that when modifying the code for multi-port access, I turned a single acceptor var into a ConcurrentHashMap. However, if there are no threads accessing this variable, it probably doesn't need to be a ConcurrentHashMap, and it probably doesn't need to be a volatile either. Debugging concurrency issues after the fact is tricky, so it will be nice if someone who worked on this can comment :) Gwen
[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.
[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389466#comment-14389466 ] Jiangjie Qin commented on KAFKA-2076: - [~jkreps] Totally agree that we need to think about improvement on new consumer API to fix the gap between new consumer and old consumer. I think the new consumer API should provide all reasonable functions simple consumer supports. Obviously it won't be as strong because currently simple consumer supports sending every type of request and responses. I think we can create the following new APIs: long earliestOffsetsFor(ListTopicPartition partitions) long latestOffsetsFor(ListTopicPartition partitions) long offsetsBefore(MapTopicPartition, Long partitions) Not sure if we need a KIP on this, if you think so please let me know. I went through all the possible requests simple consumer can send. 1. ConsumerMetadataRequest - Should be supported after coordinator implementation 2. FetchRequest - supported with seek() and poll() 3. OffsetCommitRequest - supported with commit() and commit(offsetMap) 4. OffsetRequest - not supported 5. OffsetFetchRequest - supported by committed() 6. TopicMetadataRequest - supported with partitionsFor() From what I can see and also as you mentioned, there are only following functions that are useful but not yet supported by new consumer yet: 1. getOffsetBefore 2. earliestOrLatest 3. high watermark Implementation wise, I kind of think high watermark is a little bit different from other offsets in the following ways: 1. Consumer offset is determined by consumer, while HW is determined by producer. This means consumer offsets needs only minimum communication with broker, but HW needs frequent communication. 2. Typically user will only fetch offsets when starting consumption but user may care about HW both before starting consumption and during the consuming as it reflects lags. This means the HW updates should be cheap otherwise the overhead would be big. Based on the above two points. I think The implementation I have in mind now is to store HW in a separate map. The value will be updated when: 1. latestOffsetFor(partition) is called, which always sends a OffsetRequest. 2. a piggybacked HW from fetch request is received. I don't quite understand why the server side offset time stamp would be a blocker for adding this API. It looks transparent to the user, right? Add an API to new consumer to allow user get high watermark of partitions. -- Key: KAFKA-2076 URL: https://issues.apache.org/jira/browse/KAFKA-2076 Project: Kafka Issue Type: Improvement Reporter: Jiangjie Qin We have a use case that user wants to know how far it is behind a particular partition on startup. Currently in each fetch response, we have high watermark for each partition, we only keep a global max-lag metric. It would be better that we keep a record of high watermark per partition and update it on each fetch response. We can add a new API to let user query the high watermark. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2035) Add a topic config cache.
[ https://issues.apache.org/jira/browse/KAFKA-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389468#comment-14389468 ] Jay Kreps commented on KAFKA-2035: -- Currently this is the log itself--Log.config always gives the latest config. I think you are proposing Log.cache.config, but I'm not sure if I see the need for two layers of indirection here. Add a topic config cache. - Key: KAFKA-2035 URL: https://issues.apache.org/jira/browse/KAFKA-2035 Project: Kafka Issue Type: Task Reporter: Parth Brahmbhatt Assignee: Parth Brahmbhatt Attachments: KAFKA-2035_2015-03-31_10:52:12.patch Currently the topic config is all about Log configuration so we have a TopicConfigManager which takes in a Log instance and keeps updating that instance's config instance as and when the topic config is updated. The topic config update notifications are sent using zk watchers by Controller. I propose to introduce a TopicConfigCache which will be updated by TopicConfigManager on any config changes. The log instance and any other component (like the authorizer mentioned in KAFKA-1688) will have a reference to TopicConfigCache using which they will access the topic configs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.
[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389483#comment-14389483 ] Jay Kreps commented on KAFKA-2076: -- Actually rather than trying to ensure we have everything the simple consumer has our criteria was to make sure we added no api we would regret later. It is okay to do a first version of the consumer that still has a few gaps (we can add them in the next version), but if we add something that is half thought out now we can't remove it later so it will be a permanent blight. Here is a summary of the prior discussion on this: 1. Currently we have OffsetRequest, our most screwy request. It is per-partition (you have to route it to the exact correct node to get an answer). We want to retire this request as it is so hacky. 2. There are many other questions you can ask on a per-partition basis. You might want to know the log end offset, the size on disk, segment timestamps, compaction point, etc. One possible generalization of the OffsetRequest would be PartitionMetadataRequest which would give all this stuff back. 3. Seeking to the beginning and end of the stream are things we can't ship even a first version of the consumer without, but those are so useful we can just add special purpose shortcuts (seekToEnd, seekToBeginning) and we won't regret these as they are the most common usage anyway. So basically I agree it would be good to add these APIs but we just need to figure out what the protocol end state and how we will expose this functionality. Add an API to new consumer to allow user get high watermark of partitions. -- Key: KAFKA-2076 URL: https://issues.apache.org/jira/browse/KAFKA-2076 Project: Kafka Issue Type: Improvement Reporter: Jiangjie Qin We have a use case that user wants to know how far it is behind a particular partition on startup. Currently in each fetch response, we have high watermark for each partition, we only keep a global max-lag metric. It would be better that we keep a record of high watermark per partition and update it on each fetch response. We can add a new API to let user query the high watermark. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2016) RollingBounceTest takes long
[ https://issues.apache.org/jira/browse/KAFKA-2016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389452#comment-14389452 ] Sriharsha Chintalapani commented on KAFKA-2016: --- [~malaskat] Sorry for the delay thought I already gave +1. RollingBounceTest takes long Key: KAFKA-2016 URL: https://issues.apache.org/jira/browse/KAFKA-2016 Project: Kafka Issue Type: Improvement Affects Versions: 0.8.2.1 Reporter: Jun Rao Labels: newbie Fix For: 0.8.3 Attachments: KAFKA-2016-1.patch, KAFKA-2016-2.patch RollingBounceTest.testRollingBounce() currently takes about 48 secs. This is a bit too long. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2035) Add a topic config cache.
[ https://issues.apache.org/jira/browse/KAFKA-2035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389471#comment-14389471 ] Parth Brahmbhatt commented on KAFKA-2035: - As stated in description, the current TopicConfig is only about LogConfig which is why Log.config makes sense. We plan to add Topic acls and Topic owner as part of TopicConfig. With added entities I think it makes more sense to decouple to the LogEntity from the TopicConfig entiry and add a class like the following: class TopicConfig(val owner: String,val logConfig: LogConfig,val acls: Set[Acl], val overrideProperties: Properties) This is essentially the first step for moving in that direction. Add a topic config cache. - Key: KAFKA-2035 URL: https://issues.apache.org/jira/browse/KAFKA-2035 Project: Kafka Issue Type: Task Reporter: Parth Brahmbhatt Assignee: Parth Brahmbhatt Attachments: KAFKA-2035_2015-03-31_10:52:12.patch Currently the topic config is all about Log configuration so we have a TopicConfigManager which takes in a Log instance and keeps updating that instance's config instance as and when the topic config is updated. The topic config update notifications are sent using zk watchers by Controller. I propose to introduce a TopicConfigCache which will be updated by TopicConfigManager on any config changes. The log instance and any other component (like the authorizer mentioned in KAFKA-1688) will have a reference to TopicConfigCache using which they will access the topic configs. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: KIP Hangout - March 31, 2015 - TL;DR;
Thanks for clarification Joel. Just wondering if we are going to depend on any KM specific features? Asking this because KM metric config has quota in side it. On 3/31/15, 1:53 PM, Joel Koshy jjkosh...@gmail.com wrote: We will be using KM for quota'ing on the new client-id-specific metrics. On Tue, Mar 31, 2015 at 08:44:44PM +, Jiangjie Qin wrote: Thanks a lot for the summary, Gwen! About the Quota, does that mean the first quota implementation will be based on YM? I¹m thinking can we pursue a quota solution that has a loose coupling with metrics interfaces? Like something operating system does for FUSE, so we don¹t need to care about which underlying metric we use. In that case, we can implement the Quota base on KM and wrap YM to resemble KM. Quota management itself can also be extracted as a separate package in this case. Jiangjie (Becket) Qin On 3/31/15, 12:04 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi, Short notes from today's discussion for those who missed it. Attendees, feel free to correct or add: KIP-4: * Agreed to bump TopicMetadataRequest version, leave V0 with automatic topic-creation and add warnings that we are deprecating the feature in future releases. * Agreed to document all API additions and how client developers will use new API. * Decided not to have the server parse regular expressions when listing topics Quotas and Metrics: * Quotas should use new KM metrics - those will be missing from existing reporters * Security requires re-using client classes in core and this will bring more KM metrics * We don¹t want to block security with metrics requirement * So we can shim KM into YM for security * After security we can start replacing everything KIP-5: Shaping up to be a very large feature. Decided that Quotas won¹t need to wait on this. Waiting for a more in-depth design doc KIP-13: - Depends on KIP-4 for admin side KIP-11: - Want to get the network client reuse done first - Gwen needs to check if its possible to share - blocked on generic channel implementation Others: * Agreed to use KIP call to discuss JIRAs blocked on reviews * New replica lag is almost ready! Gwen
Re: [DISCUSS] KIP-11- Authorization design for kafka security
21. What you suggested makes sense. Could you include the categorization of each request in the wiki? 24. We have a jira (KAFKA-1595) to look for a better json parser. However, it depends on dropping the scala 2.9.x support, which is being discussed. Thanks, Jun On Tue, Mar 31, 2015 at 10:56 AM, Parth Brahmbhatt pbrahmbh...@hortonworks.com wrote: Thanks for reviewing, comments inline: On 3/31/15, 9:21 AM, Jun Rao j...@confluent.iomailto:j...@confluent.io wrote: Thanks for the writeup. A few more comments. 20. I agree that it would be better to do this after KIP-4 (admin commands) is done. With KIP-4, all admin operations will be sent as requests to the brokers instead of accessing ZK directly. This will make authorization easier. 21. Operation: What about other types of requests not covered in the list, such as committing and fetching offsets, list topics, fetching consumer metadata, heartbeat, join group, etc? * I was actually considering any kind of write (like commit offset) as WRITE operation, and kind of read (fetching offset, get consumer metadata) as READ and any kind of list(list topics) as DESCRIBE. We can either create a one to one mapping between API and operation or classify each API as one of the operation. I was going with the classification but if you think one to one mapping will be easier to understand I am open to that. 22. TopicConfigCache: We will need such a cache in KIP-4 as well. It would be useful to make sure that the implementation can be reused. * I already opened a separate jira https://issues.apache.org/jira/browse/KAFKA-2035 for this and posted a reviewhttps://reviews.apache.org/r/32460/diff/#. I plan to add Acl and owner as instance variables of TopicConfig class as part of authZ patch. 23. Authorizer: 23.1 Do cluster level operations go through authorize() too? If so, what will be the resource? * Yes and I was considering to use a constant string like “Kafka-Cluster” for cluster operations. 23.2 I assume that the authorize() check will be called on every request. So, we will have to make sure that the check is cheap. * Yes , that is why by design we will trade off for speed and cache all acls, which means if you update acls it may take a few minutes before the changes take effect. 24. The acl json string in the config: Should we version this so that we can evolve it in the future (e.g., adding group support)? * I am looking into this right now but this seemed like implementation details so I did not capture it in design. I will update the json format once I have settled on a solution. What are your thoughts on using some existing libraries that support json parsing with versioning? The current json encoding/decoding used by kafka is already failing for me when I try to parse a map that has an already json encoded string as value for some key. Jun On Sun, Mar 29, 2015 at 3:56 PM, Parth Brahmbhatt pbrahmbh...@hortonworks.commailto:pbrahmbh...@hortonworks.com wrote: Hi Gwen, Thanks a lot for taking the time to review this. I have tried to address all your questions below. Thanks Parth On 3/28/15, 8:08 PM, Gwen Shapira gshap...@cloudera.commailto: gshap...@cloudera.commailto: gshap...@cloudera.commailto:gshap...@cloudera.com wrote: Preparing for Tuesday meeting, I went over the KIP :) First, Parth did an amazing job, the KIP is fantastic - detailed and readable. Thank you! Second, I have a lng list of questions :) No objections, just some things I'm unclear on and random minor comments. In general, I like the design, I just feel I'm missing parts of the picture. 1. Yes, Create topic will have an optional acls, the output of describe will display owner and acls and alter topic will allow to modify the acls.” - will be nice to see what the CLI will look like. * I will modify the KIP but I was going to add “—acl acl-file.json” to create-topic and alter-topic. 2. I like the addition of Topic owner. We made the mistake of forgetting about it when adding authorization to Sqoop2. We probably want to add “chown” command to the topic commands. * Again we can add “—owner user-name” to alter topic. 3. Kafka server will read authorizer.class” config value at startup time, create an instance of the specified class and call initialize method. We’ll need to validate that users specify only one of those. * The config type will be string so type validation should take care of it. 4. One added assumption is that on non-secure connections the session will have principal set to an object whose name() method will return Anonymous”. Can we keep DrWho? :) * Sure, its up to you actually as you are the owner of the jira that introduces session concept. 5. For cluster actions that do not apply to a specific topic like CREATE we have 2 options. We can either add a broker config called broker.acls which will point to a json file. This file
[jira] [Created] (KAFKA-2080) quick cleanup of producer performance scripts
Gwen Shapira created KAFKA-2080: --- Summary: quick cleanup of producer performance scripts Key: KAFKA-2080 URL: https://issues.apache.org/jira/browse/KAFKA-2080 Project: Kafka Issue Type: Bug Reporter: Gwen Shapira Assignee: Gwen Shapira We have two producer performance tools at the moment: one at o.a.k.client.tools and one at kafka.tools bin/kafka-producer-perf-test.sh is calling the kafka.tools one. org.apache.kafka.clients.tools.ProducerPerformance has --messages listed as optional (with default) while leaving the parameter out results in an error. Cleanup will include: * Removing the kafka.tools performance tool * Changing the shellscript to use new tool * Fix the misleading documentation for --messages * Adding both performance tools to the kafka docs -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: KIP Hangout - March 31, 2015 - TL;DR;
Thanks a lot for the summary, Gwen! About the Quota, does that mean the first quota implementation will be based on YM? I¹m thinking can we pursue a quota solution that has a loose coupling with metrics interfaces? Like something operating system does for FUSE, so we don¹t need to care about which underlying metric we use. In that case, we can implement the Quota base on KM and wrap YM to resemble KM. Quota management itself can also be extracted as a separate package in this case. Jiangjie (Becket) Qin On 3/31/15, 12:04 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi, Short notes from today's discussion for those who missed it. Attendees, feel free to correct or add: KIP-4: * Agreed to bump TopicMetadataRequest version, leave V0 with automatic topic-creation and add warnings that we are deprecating the feature in future releases. * Agreed to document all API additions and how client developers will use new API. * Decided not to have the server parse regular expressions when listing topics Quotas and Metrics: * Quotas should use new KM metrics - those will be missing from existing reporters * Security requires re-using client classes in core and this will bring more KM metrics * We don¹t want to block security with metrics requirement * So we can shim KM into YM for security * After security we can start replacing everything KIP-5: Shaping up to be a very large feature. Decided that Quotas won¹t need to wait on this. Waiting for a more in-depth design doc KIP-13: - Depends on KIP-4 for admin side KIP-11: - Want to get the network client reuse done first - Gwen needs to check if its possible to share - blocked on generic channel implementation Others: * Agreed to use KIP call to discuss JIRAs blocked on reviews * New replica lag is almost ready! Gwen
[jira] [Commented] (KAFKA-2078) Getting Selector [WARN] Error in I/O with host java.io.EOFException
[
https://issues.apache.org/jira/browse/KAFKA-2078?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389407#comment-14389407
]
Aravind commented on KAFKA-2078:
Also continuously getting this INFO logged:
[PDT] 2015-03-31 00:02:36 ZkUtils$ [INFO] conflict in
/consumers/demoCG4_3p3r/ids/demoCG4_3p3r_host-1427756678066-837b99e0 data:
{version:1,subscription:{3p_3rf:3},pattern:static,timestamp:1427759418520}
stored data:
{version:1,subscription:{3p_3rf:3},pattern:static,timestamp:1427759390994}
[PDT] 2015-03-31 00:02:36 ZkUtils$ [INFO] I wrote this conflicted ephemeral
node
[{version:1,subscription:{3p_3rf:3},pattern:static,timestamp:1427759418520}]
at /consumers/demoCG4_3p3r/ids/demoCG4_3p3r_host-1427756678066-837b99e0 a
while back in a different session, hence I will backoff for this node to be
deleted by Zookeeper and retry
Getting Selector [WARN] Error in I/O with host java.io.EOFException
---
Key: KAFKA-2078
URL: https://issues.apache.org/jira/browse/KAFKA-2078
Project: Kafka
Issue Type: Bug
Components: producer
Affects Versions: 0.8.2.0
Environment: OS Version: 2.6.39-400.209.1.el5uek and Hardware: 8 x
Intel(R) Xeon(R) CPU X5660 @ 2.80GHz/44GB
Reporter: Aravind
Assignee: Jun Rao
When trying to Produce 1000 (10 MB) messages, getting this below error some
where between 997 to 1000th message. There is no pattern but able to
reproduce.
[PDT] 2015-03-31 13:53:50 Selector [WARN] Error in I/O with our host
java.io.EOFException at
org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:62)
at org.apache.kafka.common.network.Selector.poll(Selector.java:248) at
org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:192) at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:191) at
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:122) at
java.lang.Thread.run(Thread.java:724)
This error I am getting some times @ 997th message or 999th message. There is
no pattern but able to reproduce.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1961) Looks like its possible to delete _consumer_offsets topic
[ https://issues.apache.org/jira/browse/KAFKA-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389409#comment-14389409 ] Gwen Shapira commented on KAFKA-1961: - pinging for review again [~nehanarkhede]... Looks like its possible to delete _consumer_offsets topic - Key: KAFKA-1961 URL: https://issues.apache.org/jira/browse/KAFKA-1961 Project: Kafka Issue Type: Bug Affects Versions: 0.8.2.0 Reporter: Gwen Shapira Assignee: Gwen Shapira Labels: newbie Attachments: KAFKA-1961.3.patch, KAFKA-1961.4.patch Noticed that kafka-topics.sh --delete can successfully delete internal topics (__consumer_offsets). I'm pretty sure we want to prevent that, to avoid users shooting themselves in the foot. Topic admin command should check for internal topics, just like ReplicaManager does and not let users delete them. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-2076) Add an API to new consumer to allow user get high watermark of partitions.
[ https://issues.apache.org/jira/browse/KAFKA-2076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389553#comment-14389553 ] Jay Kreps commented on KAFKA-2076: -- In terms of mechanics we did a discussion on the consumer api a while back but there are several changes since then and I think we should do a KIP on it from scratch. If there is a proposal for adding this functionality that we can work out fairly quickly on this ticket I think it is fine to fold that in to the overall consumer KIP. Add an API to new consumer to allow user get high watermark of partitions. -- Key: KAFKA-2076 URL: https://issues.apache.org/jira/browse/KAFKA-2076 Project: Kafka Issue Type: Improvement Reporter: Jiangjie Qin We have a use case that user wants to know how far it is behind a particular partition on startup. Currently in each fetch response, we have high watermark for each partition, we only keep a global max-lag metric. It would be better that we keep a record of high watermark per partition and update it on each fetch response. We can add a new API to let user query the high watermark. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31893: Patch for KAFKA-2013
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31893/ --- (Updated April 1, 2015, 12:31 a.m.) Review request for kafka. Bugs: KAFKA-2013 https://issues.apache.org/jira/browse/KAFKA-2013 Repository: kafka Description --- purgatory micro benchmark Diffs (updated) - core/src/test/scala/other/kafka/TestPurgatoryPerformance.scala PRE-CREATION Diff: https://reviews.apache.org/r/31893/diff/ Testing --- Thanks, Yasuhiro Matsuda
[jira] [Commented] (KAFKA-2013) benchmark test for the purgatory
[ https://issues.apache.org/jira/browse/KAFKA-2013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14389764#comment-14389764 ] Yasuhiro Matsuda commented on KAFKA-2013: - Updated reviewboard https://reviews.apache.org/r/31893/diff/ against branch origin/trunk benchmark test for the purgatory Key: KAFKA-2013 URL: https://issues.apache.org/jira/browse/KAFKA-2013 Project: Kafka Issue Type: Test Components: purgatory Reporter: Yasuhiro Matsuda Assignee: Yasuhiro Matsuda Priority: Trivial Attachments: KAFKA-2013.patch, KAFKA-2013_2015-03-16_13:23:38.patch, KAFKA-2013_2015-03-16_14:13:20.patch, KAFKA-2013_2015-03-16_14:39:07.patch, KAFKA-2013_2015-03-19_16:30:52.patch, KAFKA-2013_2015-03-31_17:30:56.patch We need a micro benchmark test for measuring the purgatory performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-2013) benchmark test for the purgatory
[ https://issues.apache.org/jira/browse/KAFKA-2013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yasuhiro Matsuda updated KAFKA-2013: Attachment: KAFKA-2013_2015-03-31_17:30:56.patch benchmark test for the purgatory Key: KAFKA-2013 URL: https://issues.apache.org/jira/browse/KAFKA-2013 Project: Kafka Issue Type: Test Components: purgatory Reporter: Yasuhiro Matsuda Assignee: Yasuhiro Matsuda Priority: Trivial Attachments: KAFKA-2013.patch, KAFKA-2013_2015-03-16_13:23:38.patch, KAFKA-2013_2015-03-16_14:13:20.patch, KAFKA-2013_2015-03-16_14:39:07.patch, KAFKA-2013_2015-03-19_16:30:52.patch, KAFKA-2013_2015-03-31_17:30:56.patch We need a micro benchmark test for measuring the purgatory performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 31893: Patch for KAFKA-2013
On March 31, 2015, 11:47 p.m., Jun Rao wrote: core/src/test/scala/other/kafka/TestPurgatoryPerformance.scala, line 261 https://reviews.apache.org/r/31893/diff/5/?file=900766#file900766line261 Could we just put FakeOperation to the delayQueue directly instead of wrapping it under Scheduled? We have to use completion time instead of expiration time for CompletionQueue. That is why the operaion is wrapped. - Yasuhiro --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31893/#review76975 --- On April 1, 2015, 12:31 a.m., Yasuhiro Matsuda wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/31893/ --- (Updated April 1, 2015, 12:31 a.m.) Review request for kafka. Bugs: KAFKA-2013 https://issues.apache.org/jira/browse/KAFKA-2013 Repository: kafka Description --- purgatory micro benchmark Diffs - core/src/test/scala/other/kafka/TestPurgatoryPerformance.scala PRE-CREATION Diff: https://reviews.apache.org/r/31893/diff/ Testing --- Thanks, Yasuhiro Matsuda
Re: KIP Hangout - March 31, 2015 - TL;DR;
Yes - we will be using the quota manager integration with KM. The main reason for using KM is that the end-state we agreed on is for all metrics to move over to KM and remove our dependency on YM completely. Thanks, Joel On Tue, Mar 31, 2015 at 09:51:56PM +, Jiangjie Qin wrote: Thanks for clarification Joel. Just wondering if we are going to depend on any KM specific features? Asking this because KM metric config has quota in side it. On 3/31/15, 1:53 PM, Joel Koshy jjkosh...@gmail.com wrote: We will be using KM for quota'ing on the new client-id-specific metrics. On Tue, Mar 31, 2015 at 08:44:44PM +, Jiangjie Qin wrote: Thanks a lot for the summary, Gwen! About the Quota, does that mean the first quota implementation will be based on YM? I¹m thinking can we pursue a quota solution that has a loose coupling with metrics interfaces? Like something operating system does for FUSE, so we don¹t need to care about which underlying metric we use. In that case, we can implement the Quota base on KM and wrap YM to resemble KM. Quota management itself can also be extracted as a separate package in this case. Jiangjie (Becket) Qin On 3/31/15, 12:04 PM, Gwen Shapira gshap...@cloudera.com wrote: Hi, Short notes from today's discussion for those who missed it. Attendees, feel free to correct or add: KIP-4: * Agreed to bump TopicMetadataRequest version, leave V0 with automatic topic-creation and add warnings that we are deprecating the feature in future releases. * Agreed to document all API additions and how client developers will use new API. * Decided not to have the server parse regular expressions when listing topics Quotas and Metrics: * Quotas should use new KM metrics - those will be missing from existing reporters * Security requires re-using client classes in core and this will bring more KM metrics * We don¹t want to block security with metrics requirement * So we can shim KM into YM for security * After security we can start replacing everything KIP-5: Shaping up to be a very large feature. Decided that Quotas won¹t need to wait on this. Waiting for a more in-depth design doc KIP-13: - Depends on KIP-4 for admin side KIP-11: - Want to get the network client reuse done first - Gwen needs to check if its possible to share - blocked on generic channel implementation Others: * Agreed to use KIP call to discuss JIRAs blocked on reviews * New replica lag is almost ready! Gwen -- Joel
