Re: Copycat (KIP-26. PR-99) - plan on moving forward

[Neha Narkhede]

+1 on resolving some of these open issues separately. It is a pain to
maintain, review, discuss and commit these type of large patches. Typically
the set of improvements to really stabilize new features like this go well
beyond one large initial patch anyway.

On Thu, Aug 13, 2015 at 10:24 PM, Gwen Shapira  wrote:

> Reminder:
> If no one objects, I'm planning on merging PR-99 tomorrow.
>
> On Wed, Aug 12, 2015 at 11:17 AM, Gwen Shapira  wrote:
>
> > Hi Kafka fans,
> >
> > I'd like to move ahead with getting the initial Copycat patches into
> Kafka.
> >
> > If you'll look at the PR, we still have a number of open questions on the
> > API (batch connectors, connector upgrades, data formats). I believe it
> will
> > be easier to iterate on those independently of getting the basic patch
> in,
> > but this means that the API we are committing may change in the next few
> > month (before the 0.8.3 release).
> >
> > Here is my plan:
> > 1. Add Stability attributes (The type that Hadoop and Guavas have) to
> > Kafka.
> > 2. Ewen agreed to mark the data API and connector APIs as "unstable", so
> > no one will get caught by surprise if they implement a connector and the
> > APIs move
> > 3. I will start few concurrent discussions on the mailing list on what we
> > see as open issues (the data API discussion already started by Ewen).
> > Please chime in.
> > 4. While the discussions are still on-going we will commit the initial
> > CopyCat patch to allow people to try it out and give feedback.
> >
> > If someone objects to this plan, please respond ASAP. We'd like to get
> the
> > plan done by Friday.
> >
> > Thanks,
> > Gwen
> >
> >
>



-- 
Thanks,
Neha

[jira] [Commented] (KAFKA-2397) leave group request

[Onur Karaman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696551#comment-14696551
 ] 

Onur Karaman commented on KAFKA-2397:
-

Sooo... ship it?!

> leave group request
> ---
>
> Key: KAFKA-2397
> URL: https://issues.apache.org/jira/browse/KAFKA-2397
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Onur Karaman
>Assignee: Onur Karaman
>Priority: Minor
> Fix For: 0.8.3
>
>
> Let's say every consumer in a group has session timeout s. Currently, if a 
> consumer leaves the group, the worst case time to stabilize the group is 2s 
> (s to detect the consumer failure + s for the rebalance window). If a 
> consumer instead can declare they are leaving the group, the worst case time 
> to stabilize the group would just be the s associated with the rebalance 
> window.
> This is a low priority optimization!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [Copycat] How will copycat serialize its metadata

[Neha Narkhede]

Copycat enables streaming data in and out of Kafka. Connector writers need
to define the serde of the data as it is different per system. Metadata
should be entirely hidden by the copycat framework and isn't something
users or connector implementors need to serialize differently as long as we
provide tools/REST APIs to access the metadata where required. Moreover, as
you suggest, evolution, maintenance and configs are much simpler if it
remains hidden.

+1 on keeping just the serializers for data configurable.

On Thu, Aug 13, 2015 at 9:59 PM, Gwen Shapira  wrote:

> Hi Team Kafka,
>
> As you know from KIP-26 and PR-99, when you will use Copycat to move data
> from an external system to Kafka, in addition to storing the data itself,
> Copycat will also need to store some metadata.
>
> This metadata is currently offsets on the source system (say, SCN# from
> Oracle redo log), but I can imagine to store a bit more.
>
> When storing data, we obviously want pluggable serializers, so users will
> get the data in a format they like.
>
> But the metadata seems internal. i.e users shouldn't care about it and if
> we want them to read or change anything, we want to provide them with tools
> to do it.
>
> Moreover, by controlling the format we can do three important things:
> * Read the metadata for monitoring / audit purposes
> * Evolve / modify it. If users serialize it in their own format, and
> actually write clients to use this metadata, we don't know if its safe to
> evolve.
> * Keep configuration a bit simpler. This adds at least 4 new configuration
> items...
>
> What do you guys think?
>
> Gwen
>



-- 
Thanks,
Neha

[jira] [Commented] (KAFKA-2367) Add Copycat runtime data API

[James Cheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696540#comment-14696540
 ] 

James Cheng commented on KAFKA-2367:


I just read through PR-99. I saw that even just the schema-related stuff added 
up to 3000+ lines of code. So if we used Avro, we could get rid of all that.

> Add Copycat runtime data API
> 
>
> Key: KAFKA-2367
> URL: https://issues.apache.org/jira/browse/KAFKA-2367
> Project: Kafka
>  Issue Type: Sub-task
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.8.3
>
>
> Design the API used for runtime data in Copycat. This API is used to 
> construct schemas and records that Copycat processes. This needs to be a 
> fairly general data model (think Avro, JSON, Protobufs, Thrift) in order to 
> support complex, varied data types that may be input from/output to many data 
> systems.
> This should issue should also address the serialization interfaces used 
> within Copycat, which translate the runtime data into serialized byte[] form. 
> It is important that these be considered together because the data format can 
> be used in multiple ways (records, partition IDs, partition offsets), so it 
> and the corresponding serializers must be sufficient for all these use cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Client-side Assignment for New Consumer

[Neha Narkhede]

Becket,

As you say, the metadata hash addresses the concern you originally raised
about large topic subscriptions. Can you please list other problems you are
raising more clearly? It is more helpful to know problems that the proposal
does not address or addresses poorly.

Regarding other things you said -

it is required that each
> consumer refresh their metadata before sending a JoinGroupRequest
>

This is required for wildcard topic subscriptions anyway. So this proposal
does not introduce a regression. We had agreed earlier that it does not
make sense for the server to deserialize regular expressions sent by the
consumer.


> the metadata might still be inconsistent if there is a topic or partition
> change because the
> UpdateMetadataRequest from controller might be handled at different time.
>

Topic metadata does not change frequently and even if it did, a couple
rebalance attempts will be needed whether the coordinator drives the
assignments or the consumer. Because guess how the coordinator knows about
the topic metadata changes -- indirectly through either a zk callback or
UpdateMetadataRequest, so it is completely possible the coordinator sees
the topic metadata changes in batches, not all at once.

On Thu, Aug 13, 2015 at 10:50 PM, Neha Narkhede  wrote:

> Ewen/Jason,
>
> The metadata hash is a clever approach and certainly addresses the problem
> of large metadata for consumers like mirror maker. Few comments -
>
>
>1. In the interest of simplifying the format of the consumer's
>metadata - Why not just always include only the topic names in the metadata
>followed by the metadata hash? If the metadata hash check succeeds, each
>consumer uses the # of partitions it had fetched. If it fails, a rebalance
>happens and the metadata is not used anyway.
>2. Do you need a topic list and topic regex to be separate? A single
>topic or list of topics can be expressed as a regex.
>3. Let's include a version explicitly at the beginning of the
>ProtocolMetadata. The version dictates how to deserialize the
>ProtocolMetadata blob and is consistent with the rest of Kafka.
>
> That can simplify the metadata format to the following:
>
> GroupType => "consumer"
>>
>> Protocol => AssignmentStrategy
>>   AssignmentStrategy   => String
>>
>> ProtocolMetadata => Version Subscription AssignmentStrategyMetadata
>
> Version=> String
>
>   Subscription => TopicRegex MetadataHash
>> TopicRegex => String
>> MetadataHash   => String
>>   AssignmentStrategyMetadata   => bytes
>
>
> On Thu, Aug 13, 2015 at 6:28 PM, Jiangjie Qin 
> wrote:
>
>> Ewen and Jason,
>>
>> Thanks for the reply. Sorry I missed the metadata hash. Yes, that is a
>> clever approach and would solve my concern about the data passing around.
>> I
>> can see both pros and cons from doing this, though. The advantage is we
>> don't need the topic metadata in JoinGroupResponse anymore. The downside
>> is
>> that now rebalance have extra dependency on the consensus of metadata of
>> all consumers, which is obtained separately. So it is required that each
>> consumer refresh their metadata before sending a JoinGroupRequest,
>> otherwise in some cases (e.g. wildcard consumers) will almost certainly
>> fail for the first rebalance attempt. Even if we do that, since the
>> consumers are getting metadata from different brokers, the metadata might
>> still be inconsistent if there is a topic or partition change because the
>> UpdateMetadataRequest from controller might be handled at different time.
>> Just want to make sure we think through the cases so the protocol does not
>> cause us unexpected issues.
>>
>> About the number of consumers, I think with the current liveliness
>> definition, we can tolerate churns by bumping up the session timeout. Also
>> I guess we will see an increasing number of consumers for new consumer,
>> because every the old consumer thread will probably become a consumer.
>>
>> It is a valid concern for consumers that have large subscription set. This
>> might not be avoided though for client side assignment approach. One
>> solution is having topic names associate with a topic ID. And only use
>> topic ID in JoinGroupRequest and JoinGroupResponse, There is a discussion
>> thread about this to solve the topic renaming case but this is a
>> completely
>> different discussion.
>>
>> Thanks,
>>
>> Jiangjie (Becket) Qin
>>
>>
>>
>> On Thu, Aug 13, 2015 at 2:14 PM, Jason Gustafson 
>> wrote:
>>
>> > Thanks Jiangjie, that information helps. I agree the protocol must
>> consider
>> > scalability. My point was that the synchronization barrier in the
>> current
>> > protocol already effectively limits the number of consumers since it
>> > provides no way to gracefully handle churn. It wouldn't be worth
>> worrying
>> > about scaling up to 100,000 members, for example, because there's no way
>> > the group would be stable. So we just nee

Re: [DISCUSS] Client-side Assignment for New Consumer

[Neha Narkhede]

Ewen/Jason,

The metadata hash is a clever approach and certainly addresses the problem
of large metadata for consumers like mirror maker. Few comments -


   1. In the interest of simplifying the format of the consumer's metadata
   - Why not just always include only the topic names in the metadata followed
   by the metadata hash? If the metadata hash check succeeds, each consumer
   uses the # of partitions it had fetched. If it fails, a rebalance happens
   and the metadata is not used anyway.
   2. Do you need a topic list and topic regex to be separate? A single
   topic or list of topics can be expressed as a regex.
   3. Let's include a version explicitly at the beginning of the
   ProtocolMetadata. The version dictates how to deserialize the
   ProtocolMetadata blob and is consistent with the rest of Kafka.

That can simplify the metadata format to the following:

GroupType => "consumer"
>
> Protocol => AssignmentStrategy
>   AssignmentStrategy   => String
>
> ProtocolMetadata => Version Subscription AssignmentStrategyMetadata

Version=> String

  Subscription => TopicRegex MetadataHash
> TopicRegex => String
> MetadataHash   => String
>   AssignmentStrategyMetadata   => bytes


On Thu, Aug 13, 2015 at 6:28 PM, Jiangjie Qin 
wrote:

> Ewen and Jason,
>
> Thanks for the reply. Sorry I missed the metadata hash. Yes, that is a
> clever approach and would solve my concern about the data passing around. I
> can see both pros and cons from doing this, though. The advantage is we
> don't need the topic metadata in JoinGroupResponse anymore. The downside is
> that now rebalance have extra dependency on the consensus of metadata of
> all consumers, which is obtained separately. So it is required that each
> consumer refresh their metadata before sending a JoinGroupRequest,
> otherwise in some cases (e.g. wildcard consumers) will almost certainly
> fail for the first rebalance attempt. Even if we do that, since the
> consumers are getting metadata from different brokers, the metadata might
> still be inconsistent if there is a topic or partition change because the
> UpdateMetadataRequest from controller might be handled at different time.
> Just want to make sure we think through the cases so the protocol does not
> cause us unexpected issues.
>
> About the number of consumers, I think with the current liveliness
> definition, we can tolerate churns by bumping up the session timeout. Also
> I guess we will see an increasing number of consumers for new consumer,
> because every the old consumer thread will probably become a consumer.
>
> It is a valid concern for consumers that have large subscription set. This
> might not be avoided though for client side assignment approach. One
> solution is having topic names associate with a topic ID. And only use
> topic ID in JoinGroupRequest and JoinGroupResponse, There is a discussion
> thread about this to solve the topic renaming case but this is a completely
> different discussion.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
>
> On Thu, Aug 13, 2015 at 2:14 PM, Jason Gustafson 
> wrote:
>
> > Thanks Jiangjie, that information helps. I agree the protocol must
> consider
> > scalability. My point was that the synchronization barrier in the current
> > protocol already effectively limits the number of consumers since it
> > provides no way to gracefully handle churn. It wouldn't be worth worrying
> > about scaling up to 100,000 members, for example, because there's no way
> > the group would be stable. So we just need to set some clear expectations
> > on the size we can scale to, and that can help inform the discussion on
> the
> > size of messages in this protocol.
> >
> > Ewen and I were discussing this morning along similar lines to what
> you're
> > suggesting. However, even if the coordinator decides on the metadata for
> > the group, each member still needs to communicate its subscriptions to
> the
> > rest of the group. This is nice for the regex case since the regex is
> > probably small, but if the members have a large topic list, then we have
> > the same problem. One thing I was thinking about was whether we really
> need
> > to handle different subscriptions for every member. If the coordinator
> > could guarantee that all members had the same subscription, then there
> > would be no need for the coordinator to return the subscriptions for each
> > member. However, this would prevent graceful upgrades. We might be able
> to
> > fix that problem by allowing the consumer to provide two subscriptions to
> > allowing rolling updates, but that starts to sound pretty nasty.
> >
> > -Jason
> >
> > On Thu, Aug 13, 2015 at 1:41 PM, Jiangjie Qin  >
> > wrote:
> >
> > > Jason,
> > >
> > > The protocol has to consider the scalability. The protocol in the wiki
> > > means the JoinGroupResoponse size would be:
> > > NumberOfTopics * (AvgTopicNameLength + 4) * (NumberOfConsumers)^2
> > >
> > > T

[jira] [Updated] (KAFKA-2433) Remove documentation on dead configuration item: replica.lag.max.messages

[Gwen Shapira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-2433:

Reviewer: Gwen Shapira

> Remove documentation on dead configuration item: replica.lag.max.messages
> -
>
> Key: KAFKA-2433
> URL: https://issues.apache.org/jira/browse/KAFKA-2433
> Project: Kafka
>  Issue Type: Improvement
>  Components: website
>Affects Versions: 0.8.1.2, 0.8.2.1
>Reporter: Stephen Powis
>Priority: Trivial
> Attachments: documentation-update.diff
>
>
> Digging thru the source code it looks like the configuration property 
> replica.lag.max.messages is no longer used.  The website should be updated to 
> no longer advertise this as a configurable option.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2433) Remove documentation on dead configuration item: replica.lag.max.messages

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696505#comment-14696505
 ] 

Gwen Shapira commented on KAFKA-2433:
-

Thank you!

Your patch modifies the docs both for version 0.8.2 and 0.8.3, but I believe 
the configuration was still alive in 0.8.2.
Can you double check?



> Remove documentation on dead configuration item: replica.lag.max.messages
> -
>
> Key: KAFKA-2433
> URL: https://issues.apache.org/jira/browse/KAFKA-2433
> Project: Kafka
>  Issue Type: Improvement
>  Components: website
>Affects Versions: 0.8.1.2, 0.8.2.1
>Reporter: Stephen Powis
>Priority: Trivial
> Attachments: documentation-update.diff
>
>
> Digging thru the source code it looks like the configuration property 
> replica.lag.max.messages is no longer used.  The website should be updated to 
> no longer advertise this as a configurable option.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Copycat (KIP-26. PR-99) - plan on moving forward

[Gwen Shapira]

Reminder:
If no one objects, I'm planning on merging PR-99 tomorrow.

On Wed, Aug 12, 2015 at 11:17 AM, Gwen Shapira  wrote:

> Hi Kafka fans,
>
> I'd like to move ahead with getting the initial Copycat patches into Kafka.
>
> If you'll look at the PR, we still have a number of open questions on the
> API (batch connectors, connector upgrades, data formats). I believe it will
> be easier to iterate on those independently of getting the basic patch in,
> but this means that the API we are committing may change in the next few
> month (before the 0.8.3 release).
>
> Here is my plan:
> 1. Add Stability attributes (The type that Hadoop and Guavas have) to
> Kafka.
> 2. Ewen agreed to mark the data API and connector APIs as "unstable", so
> no one will get caught by surprise if they implement a connector and the
> APIs move
> 3. I will start few concurrent discussions on the mailing list on what we
> see as open issues (the data API discussion already started by Ewen).
> Please chime in.
> 4. While the discussions are still on-going we will commit the initial
> CopyCat patch to allow people to try it out and give feedback.
>
> If someone objects to this plan, please respond ASAP. We'd like to get the
> plan done by Friday.
>
> Thanks,
> Gwen
>
>

[copycat] support for "batch" jobs

[Gwen Shapira]

Hi Team Kafka,

(sorry for the flood, this is last one! promise!)

If you tried out PR-99, you know that CopyCat now does on-going
export/import. So it will continuously read data from a source and write it
to Kafka (or vice versa). This is great for tailing logs and replicating
from MySQL binlog.

But, I'm wondering if there's a need for a batch-mode too.
This can be useful for:
* Camus-like thing. You can stream data to HDFS, but the benefits are
limited and there are some known issues there.
* Dump large parts of an RDBMS at once.

Do you agree that this need exist? or is stream export/import good enough?

Also, anyone has ideas how he would like the batch mode to work?

Gwen

[copycat] How to upgrade connectors

[Gwen Shapira]

Hi Team Kafka,

This may be a slightly premature discussion, but forgetting about upgrades
is a newbie mistake I'd like us to avoid :)

So, we have connector-plugins, and users use them to create
connector-instances. This requires some configuration - database connection
string, HDFS namenode, etc.

Now suppose we have a SystemX connector-plugin, and it takes just
"connection string" as argument. And people happily use it. Now imagine
that the guy who wrong the plugin wants to release SystemX-plugin-2.0.

Ideally, we'd want users to be able to drop 2.0 jar instead of 2.0 and
restart their connector-instances and keep on running with the existing
configuration.

But what if SystemX-plugin-2.0 made changes to the configuration? What if
it now has a new mandatory parameter? Or if the connection string format
changed?

I'd like to give connector developers a way to upgrade existing
configuration when they release a new version.

My proposal:
1. Connector API now includes 2 new methods - int getVersion() and
configuration upgrade(configuration)
2. When the framework persists configuration for the connector (I'm talking
mostly about cluster mode where we want to keep the configuration in
Kafka), it also persists the version #.
3. When starting a connector-instance, if the version from getVersion()
doesn't match the version in the persisted configs, the framework will call
upgrade() with existing configs so the connector can upgrade them and
return the new configs which will then be persisted with the new version.
4. If the upgrade fails, the connector-instance will not run.

Does that make sense?

Gwen

[Copycat] How will copycat serialize its metadata

[Gwen Shapira]

Hi Team Kafka,

As you know from KIP-26 and PR-99, when you will use Copycat to move data
from an external system to Kafka, in addition to storing the data itself,
Copycat will also need to store some metadata.

This metadata is currently offsets on the source system (say, SCN# from
Oracle redo log), but I can imagine to store a bit more.

When storing data, we obviously want pluggable serializers, so users will
get the data in a format they like.

But the metadata seems internal. i.e users shouldn't care about it and if
we want them to read or change anything, we want to provide them with tools
to do it.

Moreover, by controlling the format we can do three important things:
* Read the metadata for monitoring / audit purposes
* Evolve / modify it. If users serialize it in their own format, and
actually write clients to use this metadata, we don't know if its safe to
evolve.
* Keep configuration a bit simpler. This adds at least 4 new configuration
items...

What do you guys think?

Gwen

Re: NEW: API Stability annotations!

[Gwen Shapira]

Got it!
I was under the impression that deprecated means "there is something better
now and we'll remove this in few releases". But yeah, lets avoid marking it
for now.

On Thu, Aug 13, 2015 at 4:58 PM, Neha Narkhede  wrote:

> Thanks for starting the discussion around API annotations, Gwen! I think it
> is better to add the deprecated annotation after the new consumer API is
> successfully deployed at a couple places.
>
> On Thu, Aug 13, 2015 at 3:42 PM, Ismael Juma  wrote:
>
> > On Thu, Aug 13, 2015 at 10:41 PM, Gwen Shapira 
> wrote:
> >
> > > IMO, both old producer and old consumer APIs should be marked as
> > deprecated
> > > for 0.8.3 (since the new code will be in and we want to encourage the
> > > switch).
> > >
> >
> > I can see the appeal of this, but it's also worth considering the
> downsides
> > for users too:
> >
> > * It will introduce a number of deprecation warnings to everyone that
> > upgrades to 0.8.3 even though the old Consumer APIs still work fine (also
> > worth keeping in mind that in many projects, warnings cause a build
> > failure)
> > * The new Consumer is still marked as `Unstable` so it seems a bit odd to
> > deprecate the old one
> >
> > I think a more conservative option would be to update the documentation
> to
> > encourage users to move to the new consumer without adding deprecation
> > annotations to the old consumer APIs. Some features that are only
> available
> > in the new consumer (e.g. SSL support) provide further incentive to move.
> > As Ewen suggested, the old consumer would then be deprecated in the
> > following release. And removed in the one after that. The main downside
> > would be having to maintain the old consumer for a little while longer.
> >
> > Something to think about. :)
> >
> > Best,
> > Ismael
> >
>
>
>
> --
> Thanks,
> Neha
>

Re: Review Request 34766: Patch for KAFKA-2229

[Grant Henke]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34766/#review95376
---



core/src/main/scala/kafka/server/TopicCommandHelper.scala (line 57)


This could make a lot of calls. Since it is used in createTopic do we want 
to filter topics that have not been deleted yet?



core/src/main/scala/kafka/server/TopicCommandHelper.scala (line 84)


Do we need to check? why not?



core/src/main/scala/kafka/server/TopicCommandHelper.scala (line 89)


or Partitions and ReplicationFactor must be defined



core/src/main/scala/kafka/server/TopicCommandHelper.scala (lines 146 - 153)


This logic is fairly complex and difficult to track. I am not sure the best 
way to resolve it. Breaking it into smaller functions may help.


- Grant Henke


On June 30, 2015, 1:59 p.m., Andrii Biletskyi wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34766/
> ---
> 
> (Updated June 30, 2015, 1:59 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2229
> https://issues.apache.org/jira/browse/KAFKA-2229
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KIP-4 Phase 1 Rebase
> 
> 
> Diffs
> -
> 
>   checkstyle/import-control.xml f2e6cec267e67ce8e261341e373718e14a8e8e03 
>   clients/src/main/java/org/apache/kafka/common/ConfigEntry.java PRE-CREATION 
>   
> clients/src/main/java/org/apache/kafka/common/PartitionReplicaAssignment.java 
> PRE-CREATION 
>   clients/src/main/java/org/apache/kafka/common/TopicConfigInfo.java 
> PRE-CREATION 
>   clients/src/main/java/org/apache/kafka/common/protocol/ApiKeys.java 
> b39e9bb53dd51f1ce931691e23aabc9ebaebf1e7 
>   clients/src/main/java/org/apache/kafka/common/protocol/Errors.java 
> 5b898c8f8ad5d0469f469b600c4b2eb13d1fc662 
>   clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 
> 3dc8b015afd2347a41c9a9dbc02b8e367da5f75f 
>   clients/src/main/java/org/apache/kafka/common/requests/AbstractRequest.java 
> 5d3d52859587ce0981d702f04042b0f6e1bc3704 
>   
> clients/src/main/java/org/apache/kafka/common/requests/admin/AlterTopicRequest.java
>  PRE-CREATION 
>   
> clients/src/main/java/org/apache/kafka/common/requests/admin/AlterTopicResponse.java
>  PRE-CREATION 
>   
> clients/src/main/java/org/apache/kafka/common/requests/admin/CreateTopicRequest.java
>  PRE-CREATION 
>   
> clients/src/main/java/org/apache/kafka/common/requests/admin/CreateTopicResponse.java
>  PRE-CREATION 
>   
> clients/src/main/java/org/apache/kafka/common/requests/admin/DeleteTopicRequest.java
>  PRE-CREATION 
>   
> clients/src/main/java/org/apache/kafka/common/requests/admin/DeleteTopicResponse.java
>  PRE-CREATION 
>   
> clients/src/test/java/org/apache/kafka/common/requests/RequestResponseTest.java
>  8b2aca85fa738180e5420985fddc39a4bf9681ea 
>   core/src/main/scala/kafka/admin/AdminUtils.scala 
> f06edf41c732a7b794e496d0048b0ce6f897e72b 
>   core/src/main/scala/kafka/api/RequestKeys.scala 
> 155cb650e9cffe2c950326cfc25b1480cda819db 
>   core/src/main/scala/kafka/common/InvalidPartitionsException.scala 
> PRE-CREATION 
>   core/src/main/scala/kafka/common/InvalidReplicaAssignmentException.scala 
> PRE-CREATION 
>   core/src/main/scala/kafka/common/InvalidReplicationFactorException.scala 
> PRE-CREATION 
>   core/src/main/scala/kafka/common/InvalidTopicConfigurationException.scala 
> PRE-CREATION 
>   
> core/src/main/scala/kafka/common/ReassignPartitionsInProgressException.scala 
> PRE-CREATION 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> ad6f05807c61c971e5e60d24bc0c87e989115961 
>   core/src/main/scala/kafka/server/TopicCommandHelper.scala PRE-CREATION 
>   core/src/test/scala/unit/kafka/admin/AdminTest.scala 
> 252ac813c8df1780c2dc5fa9e698fb43bb6d5cf8 
>   core/src/test/scala/unit/kafka/server/TopicCommandHelperTest.scala 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/34766/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Andrii Biletskyi
> 
>

Nightly build snapshots?

[Jaikiran Pai]

There are certain features in the latest trunk for 0.8.3 release which 
we would like to test in our dev environments. Is there some place where 
we can get the official nightly snapshot packaged builds or is the 
recommended way to always fetch latest source code and build it locally 
to try out the latest?


-Jaikiran

[jira] [Commented] (KAFKA-2397) leave group request

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696351#comment-14696351
 ] 

Jason Gustafson commented on KAFKA-2397:


[~ewencp] The only case I can come up with is an application timeout on the 
client (e.g. if the heartbeat was delayed by a transient network issue), and 
that can be fixed by always ensuring that the timeout for coordinator requests 
is longer than the session timeout. My unease mostly has to do with 
proxy/tunnel situations where I don't know that TCP always behaves properly. 
Perhaps you know enough to know whether this is an issue? In any case, it seems 
like we all agree that we need a way to leave the group properly. My preference 
is probably for Onur's patch as it stands now.

> leave group request
> ---
>
> Key: KAFKA-2397
> URL: https://issues.apache.org/jira/browse/KAFKA-2397
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Onur Karaman
>Assignee: Onur Karaman
>Priority: Minor
> Fix For: 0.8.3
>
>
> Let's say every consumer in a group has session timeout s. Currently, if a 
> consumer leaves the group, the worst case time to stabilize the group is 2s 
> (s to detect the consumer failure + s for the rebalance window). If a 
> consumer instead can declare they are leaving the group, the worst case time 
> to stabilize the group would just be the s associated with the rebalance 
> window.
> This is a low priority optimization!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2433) Remove documentation on dead configuration item: replica.lag.max.messages

[Stephen Powis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Powis updated KAFKA-2433:
-
Status: Patch Available  (was: Open)

Attached a patch

> Remove documentation on dead configuration item: replica.lag.max.messages
> -
>
> Key: KAFKA-2433
> URL: https://issues.apache.org/jira/browse/KAFKA-2433
> Project: Kafka
>  Issue Type: Improvement
>  Components: website
>Affects Versions: 0.8.2.1, 0.8.1.2
>Reporter: Stephen Powis
>Priority: Trivial
> Attachments: documentation-update.diff
>
>
> Digging thru the source code it looks like the configuration property 
> replica.lag.max.messages is no longer used.  The website should be updated to 
> no longer advertise this as a configurable option.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2433) Remove documentation on dead configuration item: replica.lag.max.messages

[Stephen Powis (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Stephen Powis updated KAFKA-2433:
-
Attachment: documentation-update.diff

> Remove documentation on dead configuration item: replica.lag.max.messages
> -
>
> Key: KAFKA-2433
> URL: https://issues.apache.org/jira/browse/KAFKA-2433
> Project: Kafka
>  Issue Type: Improvement
>  Components: website
>Affects Versions: 0.8.1.2, 0.8.2.1
>Reporter: Stephen Powis
>Priority: Trivial
> Attachments: documentation-update.diff
>
>
> Digging thru the source code it looks like the configuration property 
> replica.lag.max.messages is no longer used.  The website should be updated to 
> no longer advertise this as a configurable option.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2433) Remove documentation on dead configuration item: replica.lag.max.messages

[Stephen Powis (JIRA)]

Stephen Powis created KAFKA-2433:


 Summary: Remove documentation on dead configuration item: 
replica.lag.max.messages
 Key: KAFKA-2433
 URL: https://issues.apache.org/jira/browse/KAFKA-2433
 Project: Kafka
  Issue Type: Improvement
  Components: website
Affects Versions: 0.8.2.1, 0.8.1.2
Reporter: Stephen Powis
Priority: Trivial


Digging thru the source code it looks like the configuration property 
replica.lag.max.messages is no longer used.  The website should be updated to 
no longer advertise this as a configurable option.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated Aug. 14, 2015, 2:20 a.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

Updated patch for quotas. This patch does the following: 
1. Add per-client metrics for both producer and consumers 
2. Add configuration for quotas 
3. Compute delay times in the metrics package and return the delay times in 
QuotaViolationException 
4. Add a DelayQueue in KafkaApi's that can be used to throttle any type of 
request. Implemented request throttling for produce and fetch requests. 
5. Added unit and integration test cases for both producer and consumer
6. This doesn't include a system test. There is a separate ticket for that
7. Fixed KAFKA-2191 - (Included fix from : https://reviews.apache.org/r/34418/ )

Addressed comments from Joel and Jun


Diffs
-

  build.gradle 1b67e628c2fca897177c12b6afad9a8700fffd1f 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java 
98429da34418f7f1deba1b5e44e2e6025212edb3 
  clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java 
544e120594de78c43581a980b1e4087b4fb98ccb 
  core/src/main/scala/kafka/server/ClientQuotaManager.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
7ea509c2c41acc00430c74e025e069a833aac4e7 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
a06f0bd40e2f90972b44b80a106f98f3d50e5e2b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
84d4730ac634f9a5bf12a656e422fea03ad72da8 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
2e0bbcd6e4f0e38997ea18202b249ee3553640ec 
  core/src/main/scala/kafka/server/ThrottledResponse.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
fc226c863095b7761290292cd8755cd7ad0f155c 
  core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaManagerTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
e26a7306a6ea3104b3fa3df60006c0a473bfb2cc 
  core/src/test/scala/unit/kafka/server/ThrottledResponseExpirationTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

[jira] [Commented] (KAFKA-2084) byte rate metrics per client ID (producer and consumer)

[Aditya A Auradkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696339#comment-14696339
 ] 

Aditya A Auradkar commented on KAFKA-2084:
--

Updated reviewboard https://reviews.apache.org/r/33049/diff/
 against branch origin/trunk

> byte rate metrics per client ID (producer and consumer)
> ---
>
> Key: KAFKA-2084
> URL: https://issues.apache.org/jira/browse/KAFKA-2084
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Aditya Auradkar
>Assignee: Aditya Auradkar
>  Labels: quotas
> Attachments: KAFKA-2084.patch, KAFKA-2084_2015-04-09_18:10:56.patch, 
> KAFKA-2084_2015-04-10_17:24:34.patch, KAFKA-2084_2015-04-21_12:21:18.patch, 
> KAFKA-2084_2015-04-21_12:28:05.patch, KAFKA-2084_2015-05-05_15:27:35.patch, 
> KAFKA-2084_2015-05-05_17:52:02.patch, KAFKA-2084_2015-05-11_16:16:01.patch, 
> KAFKA-2084_2015-05-26_11:50:50.patch, KAFKA-2084_2015-06-02_17:02:00.patch, 
> KAFKA-2084_2015-06-02_17:09:28.patch, KAFKA-2084_2015-06-02_17:10:52.patch, 
> KAFKA-2084_2015-06-04_16:31:22.patch, KAFKA-2084_2015-06-12_10:39:35.patch, 
> KAFKA-2084_2015-06-29_17:53:44.patch, KAFKA-2084_2015-08-04_18:50:51.patch, 
> KAFKA-2084_2015-08-04_19:07:46.patch, KAFKA-2084_2015-08-07_11:27:51.patch, 
> KAFKA-2084_2015-08-10_13:48:50.patch, KAFKA-2084_2015-08-10_21:57:48.patch, 
> KAFKA-2084_2015-08-12_12:02:33.patch, KAFKA-2084_2015-08-12_12:04:51.patch, 
> KAFKA-2084_2015-08-12_12:08:17.patch, KAFKA-2084_2015-08-12_21:24:07.patch, 
> KAFKA-2084_2015-08-13_19:08:27.patch, KAFKA-2084_2015-08-13_19:19:16.patch
>
>
> We need to be able to track the bytes-in/bytes-out rate on a per-client ID 
> basis. This is necessary for quotas.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2084) byte rate metrics per client ID (producer and consumer)

[Aditya A Auradkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aditya A Auradkar updated KAFKA-2084:
-
Attachment: KAFKA-2084_2015-08-13_19:19:16.patch

> byte rate metrics per client ID (producer and consumer)
> ---
>
> Key: KAFKA-2084
> URL: https://issues.apache.org/jira/browse/KAFKA-2084
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Aditya Auradkar
>Assignee: Aditya Auradkar
>  Labels: quotas
> Attachments: KAFKA-2084.patch, KAFKA-2084_2015-04-09_18:10:56.patch, 
> KAFKA-2084_2015-04-10_17:24:34.patch, KAFKA-2084_2015-04-21_12:21:18.patch, 
> KAFKA-2084_2015-04-21_12:28:05.patch, KAFKA-2084_2015-05-05_15:27:35.patch, 
> KAFKA-2084_2015-05-05_17:52:02.patch, KAFKA-2084_2015-05-11_16:16:01.patch, 
> KAFKA-2084_2015-05-26_11:50:50.patch, KAFKA-2084_2015-06-02_17:02:00.patch, 
> KAFKA-2084_2015-06-02_17:09:28.patch, KAFKA-2084_2015-06-02_17:10:52.patch, 
> KAFKA-2084_2015-06-04_16:31:22.patch, KAFKA-2084_2015-06-12_10:39:35.patch, 
> KAFKA-2084_2015-06-29_17:53:44.patch, KAFKA-2084_2015-08-04_18:50:51.patch, 
> KAFKA-2084_2015-08-04_19:07:46.patch, KAFKA-2084_2015-08-07_11:27:51.patch, 
> KAFKA-2084_2015-08-10_13:48:50.patch, KAFKA-2084_2015-08-10_21:57:48.patch, 
> KAFKA-2084_2015-08-12_12:02:33.patch, KAFKA-2084_2015-08-12_12:04:51.patch, 
> KAFKA-2084_2015-08-12_12:08:17.patch, KAFKA-2084_2015-08-12_21:24:07.patch, 
> KAFKA-2084_2015-08-13_19:08:27.patch, KAFKA-2084_2015-08-13_19:19:16.patch
>
>
> We need to be able to track the bytes-in/bytes-out rate on a per-client ID 
> basis. This is necessary for quotas.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated Aug. 14, 2015, 2:19 a.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

Signed-off-by: Aditya Auradkar 

Addressing Joel's comments


Minor imports changes


Added testcase to verify that replication traffic is not throttled


Tmp commit


Fixing test failure


Minor


Addressing Joel's comments


Addressing comments


Addressing comments


Addressing Juns comments


Minor checkstyle changes


fixed test case


Addressing Juns comments


Addressing Juns comments


Addressing Juns comments


Minor change


Minor change


Moving MockTime back to test


Addressing Joels comments


Minor import change


Diffs (updated)
-

  build.gradle 1b67e628c2fca897177c12b6afad9a8700fffd1f 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java 
98429da34418f7f1deba1b5e44e2e6025212edb3 
  clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java 
544e120594de78c43581a980b1e4087b4fb98ccb 
  core/src/main/scala/kafka/server/ClientQuotaManager.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
7ea509c2c41acc00430c74e025e069a833aac4e7 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
a06f0bd40e2f90972b44b80a106f98f3d50e5e2b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
84d4730ac634f9a5bf12a656e422fea03ad72da8 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
2e0bbcd6e4f0e38997ea18202b249ee3553640ec 
  core/src/main/scala/kafka/server/ThrottledResponse.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
fc226c863095b7761290292cd8755cd7ad0f155c 
  core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaManagerTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
e26a7306a6ea3104b3fa3df60006c0a473bfb2cc 
  core/src/test/scala/unit/kafka/server/ThrottledResponseExpirationTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

> On Aug. 14, 2015, 1:57 a.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java, 
> > lines 69-77
> > 
> >
> > This is probably not the right place to throw QuotaViolationException. 
> > Rate.measure() can be called when the JMX value is polled and we don't want 
> > to throw an exception there.
> > 
> > I was thinking that another way to do this is to perhaps add the logic 
> > in Quota.acceptable(). We can probably change it to be
> > 
> > // if the check passes, return -1; otherwise return a delay to make the 
> > check pass.
> > int Quota.check(KafkaMetrics, long now).
> > 
> > In the implementation, we can then check if the KafkaMetrics passed in 
> > has a measurable that's an instance of Rate. If so, we will compute the 
> > delay for rate (the window should be elapsedCurrent and elapsedPrior as 
> > computed in Rate.measure()).

Rate/Measurable stat does not expose the windows currently. So we cannot access 
elapsedCurrent and elapsedPrior from the KafkaMetric object in Sensor. Couple 
of options:
1. We can simply record it in Rate.record() which is called from Sensor.record
2. We can check quotas in Sensor.java but not track the precise window 
boundaries.


- Aditya


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/#review95369
---


On Aug. 14, 2015, 2:09 a.m., Aditya Auradkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33049/
> ---
> 
> (Updated Aug. 14, 2015, 2:09 a.m.)
> 
> 
> Review request for kafka, Joel Koshy and Jun Rao.
> 
> 
> Bugs: KAFKA-2084
> https://issues.apache.org/jira/browse/KAFKA-2084
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Updated patch for quotas. This patch does the following: 
> 1. Add per-client metrics for both producer and consumers 
> 2. Add configuration for quotas 
> 3. Compute delay times in the metrics package and return the delay times in 
> QuotaViolationException 
> 4. Add a DelayQueue in KafkaApi's that can be used to throttle any type of 
> request. Implemented request throttling for produce and fetch requests. 
> 5. Added unit and integration test cases for both producer and consumer
> 6. This doesn't include a system test. There is a separate ticket for that
> 7. Fixed KAFKA-2191 - (Included fix from : 
> https://reviews.apache.org/r/34418/ )
> 
> Addressed comments from Joel and Jun
> 
> 
> Diffs
> -
> 
>   build.gradle 1b67e628c2fca897177c12b6afad9a8700fffd1f 
>   clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
> d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
>   
> clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
>  a451e5385c9eca76b38b425e8ac856b2715fcffe 
>   clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
> ca823fd4639523018311b814fde69b6177e73b97 
>   clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java 
> 98429da34418f7f1deba1b5e44e2e6025212edb3 
>   clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java 
> 544e120594de78c43581a980b1e4087b4fb98ccb 
>   core/src/main/scala/kafka/server/ClientQuotaManager.scala PRE-CREATION 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> 7ea509c2c41acc00430c74e025e069a833aac4e7 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> a06f0bd40e2f90972b44b80a106f98f3d50e5e2b 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 84d4730ac634f9a5bf12a656e422fea03ad72da8 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 2e0bbcd6e4f0e38997ea18202b249ee3553640ec 
>   core/src/main/scala/kafka/server/ThrottledResponse.scala PRE-CREATION 
>   core/src/main/scala/kafka/utils/ShutdownableThread.scala 
> fc226c863095b7761290292cd8755cd7ad0f155c 
>   core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
>   core/src/test/scala/unit/kafka/server/ClientQuotaManagerTest.scala 
> PRE-CREATION 
>   core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
> e26a7306a6ea3104b3fa3df60006c0a473bfb2cc 
>   core/src/test/scala/unit/kafka/server/ThrottledResponseExpirationTest.scala 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/33049/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aditya Auradkar
> 
>

[jira] [Commented] (KAFKA-2397) leave group request

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696334#comment-14696334
 ] 

Jiangjie Qin commented on KAFKA-2397:
-

[~ewencp] Are you saying that the coordinator should kick the consumer out of 
group once its TCP connection is closed? I think the problem here is this 
breaks the layers we have on broker side. So the TCP connections are only 
maintained by SocketServer and not exposed to KafkaApiThreads. So SocketServer 
does not know about which consumer a particular connection is associated with. 
If you want to let Coordinator knows about TCP connection closure, the 
coordinator needs to keep a consumer-socket map and SocketServer needs to 
produce some event back to the request queue to notify a disconnection and 
coordinator needs to check if that socket is associated with some consumer or 
not.

> leave group request
> ---
>
> Key: KAFKA-2397
> URL: https://issues.apache.org/jira/browse/KAFKA-2397
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Onur Karaman
>Assignee: Onur Karaman
>Priority: Minor
> Fix For: 0.8.3
>
>
> Let's say every consumer in a group has session timeout s. Currently, if a 
> consumer leaves the group, the worst case time to stabilize the group is 2s 
> (s to detect the consumer failure + s for the rebalance window). If a 
> consumer instead can declare they are leaving the group, the worst case time 
> to stabilize the group would just be the s associated with the rebalance 
> window.
> This is a low priority optimization!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated Aug. 14, 2015, 2:09 a.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

Updated patch for quotas. This patch does the following: 
1. Add per-client metrics for both producer and consumers 
2. Add configuration for quotas 
3. Compute delay times in the metrics package and return the delay times in 
QuotaViolationException 
4. Add a DelayQueue in KafkaApi's that can be used to throttle any type of 
request. Implemented request throttling for produce and fetch requests. 
5. Added unit and integration test cases for both producer and consumer
6. This doesn't include a system test. There is a separate ticket for that
7. Fixed KAFKA-2191 - (Included fix from : https://reviews.apache.org/r/34418/ )

Addressed comments from Joel and Jun


Diffs
-

  build.gradle 1b67e628c2fca897177c12b6afad9a8700fffd1f 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java 
98429da34418f7f1deba1b5e44e2e6025212edb3 
  clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java 
544e120594de78c43581a980b1e4087b4fb98ccb 
  core/src/main/scala/kafka/server/ClientQuotaManager.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
7ea509c2c41acc00430c74e025e069a833aac4e7 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
a06f0bd40e2f90972b44b80a106f98f3d50e5e2b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
84d4730ac634f9a5bf12a656e422fea03ad72da8 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
2e0bbcd6e4f0e38997ea18202b249ee3553640ec 
  core/src/main/scala/kafka/server/ThrottledResponse.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
fc226c863095b7761290292cd8755cd7ad0f155c 
  core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaManagerTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
e26a7306a6ea3104b3fa3df60006c0a473bfb2cc 
  core/src/test/scala/unit/kafka/server/ThrottledResponseExpirationTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

[jira] [Updated] (KAFKA-2084) byte rate metrics per client ID (producer and consumer)

[Aditya A Auradkar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Aditya A Auradkar updated KAFKA-2084:
-
Attachment: KAFKA-2084_2015-08-13_19:08:27.patch

> byte rate metrics per client ID (producer and consumer)
> ---
>
> Key: KAFKA-2084
> URL: https://issues.apache.org/jira/browse/KAFKA-2084
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Aditya Auradkar
>Assignee: Aditya Auradkar
>  Labels: quotas
> Attachments: KAFKA-2084.patch, KAFKA-2084_2015-04-09_18:10:56.patch, 
> KAFKA-2084_2015-04-10_17:24:34.patch, KAFKA-2084_2015-04-21_12:21:18.patch, 
> KAFKA-2084_2015-04-21_12:28:05.patch, KAFKA-2084_2015-05-05_15:27:35.patch, 
> KAFKA-2084_2015-05-05_17:52:02.patch, KAFKA-2084_2015-05-11_16:16:01.patch, 
> KAFKA-2084_2015-05-26_11:50:50.patch, KAFKA-2084_2015-06-02_17:02:00.patch, 
> KAFKA-2084_2015-06-02_17:09:28.patch, KAFKA-2084_2015-06-02_17:10:52.patch, 
> KAFKA-2084_2015-06-04_16:31:22.patch, KAFKA-2084_2015-06-12_10:39:35.patch, 
> KAFKA-2084_2015-06-29_17:53:44.patch, KAFKA-2084_2015-08-04_18:50:51.patch, 
> KAFKA-2084_2015-08-04_19:07:46.patch, KAFKA-2084_2015-08-07_11:27:51.patch, 
> KAFKA-2084_2015-08-10_13:48:50.patch, KAFKA-2084_2015-08-10_21:57:48.patch, 
> KAFKA-2084_2015-08-12_12:02:33.patch, KAFKA-2084_2015-08-12_12:04:51.patch, 
> KAFKA-2084_2015-08-12_12:08:17.patch, KAFKA-2084_2015-08-12_21:24:07.patch, 
> KAFKA-2084_2015-08-13_19:08:27.patch
>
>
> We need to be able to track the bytes-in/bytes-out rate on a per-client ID 
> basis. This is necessary for quotas.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 33049: Patch for KAFKA-2084

[Aditya Auradkar]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/
---

(Updated Aug. 14, 2015, 2:08 a.m.)


Review request for kafka, Joel Koshy and Jun Rao.


Bugs: KAFKA-2084
https://issues.apache.org/jira/browse/KAFKA-2084


Repository: kafka


Description (updated)
---

Signed-off-by: Aditya Auradkar 

Addressing Joel's comments


Minor imports changes


Added testcase to verify that replication traffic is not throttled


Tmp commit


Fixing test failure


Minor


Addressing Joel's comments


Addressing comments


Addressing comments


Addressing Juns comments


Minor checkstyle changes


fixed test case


Addressing Juns comments


Addressing Juns comments


Addressing Juns comments


Minor change


Minor change


Moving MockTime back to test


Addressing Joels comments


Diffs (updated)
-

  build.gradle 1b67e628c2fca897177c12b6afad9a8700fffd1f 
  clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
  
clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
 a451e5385c9eca76b38b425e8ac856b2715fcffe 
  clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
ca823fd4639523018311b814fde69b6177e73b97 
  clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java 
98429da34418f7f1deba1b5e44e2e6025212edb3 
  clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java 
544e120594de78c43581a980b1e4087b4fb98ccb 
  core/src/main/scala/kafka/server/ClientQuotaManager.scala PRE-CREATION 
  core/src/main/scala/kafka/server/KafkaApis.scala 
7ea509c2c41acc00430c74e025e069a833aac4e7 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
a06f0bd40e2f90972b44b80a106f98f3d50e5e2b 
  core/src/main/scala/kafka/server/KafkaServer.scala 
84d4730ac634f9a5bf12a656e422fea03ad72da8 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
2e0bbcd6e4f0e38997ea18202b249ee3553640ec 
  core/src/main/scala/kafka/server/ThrottledResponse.scala PRE-CREATION 
  core/src/main/scala/kafka/utils/ShutdownableThread.scala 
fc226c863095b7761290292cd8755cd7ad0f155c 
  core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/server/ClientQuotaManagerTest.scala 
PRE-CREATION 
  core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
e26a7306a6ea3104b3fa3df60006c0a473bfb2cc 
  core/src/test/scala/unit/kafka/server/ThrottledResponseExpirationTest.scala 
PRE-CREATION 

Diff: https://reviews.apache.org/r/33049/diff/


Testing
---


Thanks,

Aditya Auradkar

[jira] [Commented] (KAFKA-2084) byte rate metrics per client ID (producer and consumer)

[Aditya A Auradkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696333#comment-14696333
 ] 

Aditya A Auradkar commented on KAFKA-2084:
--

Updated reviewboard https://reviews.apache.org/r/33049/diff/
 against branch origin/trunk

> byte rate metrics per client ID (producer and consumer)
> ---
>
> Key: KAFKA-2084
> URL: https://issues.apache.org/jira/browse/KAFKA-2084
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Aditya Auradkar
>Assignee: Aditya Auradkar
>  Labels: quotas
> Attachments: KAFKA-2084.patch, KAFKA-2084_2015-04-09_18:10:56.patch, 
> KAFKA-2084_2015-04-10_17:24:34.patch, KAFKA-2084_2015-04-21_12:21:18.patch, 
> KAFKA-2084_2015-04-21_12:28:05.patch, KAFKA-2084_2015-05-05_15:27:35.patch, 
> KAFKA-2084_2015-05-05_17:52:02.patch, KAFKA-2084_2015-05-11_16:16:01.patch, 
> KAFKA-2084_2015-05-26_11:50:50.patch, KAFKA-2084_2015-06-02_17:02:00.patch, 
> KAFKA-2084_2015-06-02_17:09:28.patch, KAFKA-2084_2015-06-02_17:10:52.patch, 
> KAFKA-2084_2015-06-04_16:31:22.patch, KAFKA-2084_2015-06-12_10:39:35.patch, 
> KAFKA-2084_2015-06-29_17:53:44.patch, KAFKA-2084_2015-08-04_18:50:51.patch, 
> KAFKA-2084_2015-08-04_19:07:46.patch, KAFKA-2084_2015-08-07_11:27:51.patch, 
> KAFKA-2084_2015-08-10_13:48:50.patch, KAFKA-2084_2015-08-10_21:57:48.patch, 
> KAFKA-2084_2015-08-12_12:02:33.patch, KAFKA-2084_2015-08-12_12:04:51.patch, 
> KAFKA-2084_2015-08-12_12:08:17.patch, KAFKA-2084_2015-08-12_21:24:07.patch, 
> KAFKA-2084_2015-08-13_19:08:27.patch
>
>
> We need to be able to track the bytes-in/bytes-out rate on a per-client ID 
> basis. This is necessary for quotas.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2397) leave group request

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696327#comment-14696327
 ] 

Ewen Cheslack-Postava commented on KAFKA-2397:
--

Can we be more concrete about what we think the odd side effects would be if it 
were tied to the TCP session? What would happen that would cause the TCP 
connection to actually close rather than waiting around for a long time for the 
normal TCP timeout? I'm struggling to come up with a scenario that would 
actually kill the connection and I wouldn't want to kick the member out of the 
group.

I'm running into annoying issues since we don't have any mechanism to leave the 
group currently. Initially it was just manifesting in my manual testing with 
Copycat where if I restarted a sink task, it would take awhile for it to start 
processing data because it had to wait for the process that I had just killed 
to be kicked out of the group. This is a bit annoying with the default session 
timeout of 30s, but workable. However, I'm also working on system tests and now 
it's making me set quite large timeouts for some steps (which otherwise should 
be more like < 1s to complete) and therefore makes the tests run a lot slower.

> leave group request
> ---
>
> Key: KAFKA-2397
> URL: https://issues.apache.org/jira/browse/KAFKA-2397
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Onur Karaman
>Assignee: Onur Karaman
>Priority: Minor
> Fix For: 0.8.3
>
>
> Let's say every consumer in a group has session timeout s. Currently, if a 
> consumer leaves the group, the worst case time to stabilize the group is 2s 
> (s to detect the consumer failure + s for the rebalance window). If a 
> consumer instead can declare they are leaving the group, the worst case time 
> to stabilize the group would just be the s associated with the rebalance 
> window.
> This is a low priority optimization!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 33049: Patch for KAFKA-2084

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/#review95369
---



clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java (lines 69 
- 77)


This is probably not the right place to throw QuotaViolationException. 
Rate.measure() can be called when the JMX value is polled and we don't want to 
throw an exception there.

I was thinking that another way to do this is to perhaps add the logic in 
Quota.acceptable(). We can probably change it to be

// if the check passes, return -1; otherwise return a delay to make the 
check pass.
int Quota.check(KafkaMetrics, long now).

In the implementation, we can then check if the KafkaMetrics passed in has 
a measurable that's an instance of Rate. If so, we will compute the delay for 
rate (the window should be elapsedCurrent and elapsedPrior as computed in 
Rate.measure()).


- Jun Rao


On Aug. 13, 2015, 4:25 a.m., Aditya Auradkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33049/
> ---
> 
> (Updated Aug. 13, 2015, 4:25 a.m.)
> 
> 
> Review request for kafka, Joel Koshy and Jun Rao.
> 
> 
> Bugs: KAFKA-2084
> https://issues.apache.org/jira/browse/KAFKA-2084
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Updated patch for quotas. This patch does the following: 
> 1. Add per-client metrics for both producer and consumers 
> 2. Add configuration for quotas 
> 3. Compute delay times in the metrics package and return the delay times in 
> QuotaViolationException 
> 4. Add a DelayQueue in KafkaApi's that can be used to throttle any type of 
> request. Implemented request throttling for produce and fetch requests. 
> 5. Added unit and integration test cases for both producer and consumer
> 6. This doesn't include a system test. There is a separate ticket for that
> 7. Fixed KAFKA-2191 - (Included fix from : 
> https://reviews.apache.org/r/34418/ )
> 
> Addressed comments from Joel and Jun
> 
> 
> Diffs
> -
> 
>   build.gradle 1b67e628c2fca897177c12b6afad9a8700fffd1f 
>   clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
> d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
>   
> clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
>  a451e5385c9eca76b38b425e8ac856b2715fcffe 
>   clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
> ca823fd4639523018311b814fde69b6177e73b97 
>   clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java 
> 98429da34418f7f1deba1b5e44e2e6025212edb3 
>   clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java 
> 544e120594de78c43581a980b1e4087b4fb98ccb 
>   core/src/main/scala/kafka/server/ClientQuotaManager.scala PRE-CREATION 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> 7ea509c2c41acc00430c74e025e069a833aac4e7 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> dbe170f87331f43e2dc30165080d2cb7dfe5fdbf 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 84d4730ac634f9a5bf12a656e422fea03ad72da8 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 795220e7f63d163be90738b4c1a39687b44c1395 
>   core/src/main/scala/kafka/server/ThrottledResponse.scala PRE-CREATION 
>   core/src/main/scala/kafka/utils/ShutdownableThread.scala 
> fc226c863095b7761290292cd8755cd7ad0f155c 
>   core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
>   core/src/test/scala/unit/kafka/server/ClientQuotaManagerTest.scala 
> PRE-CREATION 
>   core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
> f32d206d3f52f3f9f4d649c213edd7058f4b6150 
>   core/src/test/scala/unit/kafka/server/ThrottledResponseExpirationTest.scala 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/33049/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aditya Auradkar
> 
>

Re: Review Request 33049: Patch for KAFKA-2084

[Joel Koshy]

> On Aug. 13, 2015, 11:13 p.m., Joel Koshy wrote:
> > build.gradle, line 383
> > 
> >
> > I don't think these changes 383-385 are needed.
> > 
> > Also, (unrelated to this change) I'm seeing NPE errors in running unit 
> > tests (e.g., QuotasTest) - can you verify?

Not sure why it didn't seem to affect your environment, but you need the 
`@Before` and `@After` annotations on your `setUp` and `tearDown` methods.


- Joel


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/#review95344
---


On Aug. 13, 2015, 4:25 a.m., Aditya Auradkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33049/
> ---
> 
> (Updated Aug. 13, 2015, 4:25 a.m.)
> 
> 
> Review request for kafka, Joel Koshy and Jun Rao.
> 
> 
> Bugs: KAFKA-2084
> https://issues.apache.org/jira/browse/KAFKA-2084
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Updated patch for quotas. This patch does the following: 
> 1. Add per-client metrics for both producer and consumers 
> 2. Add configuration for quotas 
> 3. Compute delay times in the metrics package and return the delay times in 
> QuotaViolationException 
> 4. Add a DelayQueue in KafkaApi's that can be used to throttle any type of 
> request. Implemented request throttling for produce and fetch requests. 
> 5. Added unit and integration test cases for both producer and consumer
> 6. This doesn't include a system test. There is a separate ticket for that
> 7. Fixed KAFKA-2191 - (Included fix from : 
> https://reviews.apache.org/r/34418/ )
> 
> Addressed comments from Joel and Jun
> 
> 
> Diffs
> -
> 
>   build.gradle 1b67e628c2fca897177c12b6afad9a8700fffd1f 
>   clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
> d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
>   
> clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
>  a451e5385c9eca76b38b425e8ac856b2715fcffe 
>   clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
> ca823fd4639523018311b814fde69b6177e73b97 
>   clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java 
> 98429da34418f7f1deba1b5e44e2e6025212edb3 
>   clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java 
> 544e120594de78c43581a980b1e4087b4fb98ccb 
>   core/src/main/scala/kafka/server/ClientQuotaManager.scala PRE-CREATION 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> 7ea509c2c41acc00430c74e025e069a833aac4e7 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> dbe170f87331f43e2dc30165080d2cb7dfe5fdbf 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 84d4730ac634f9a5bf12a656e422fea03ad72da8 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 795220e7f63d163be90738b4c1a39687b44c1395 
>   core/src/main/scala/kafka/server/ThrottledResponse.scala PRE-CREATION 
>   core/src/main/scala/kafka/utils/ShutdownableThread.scala 
> fc226c863095b7761290292cd8755cd7ad0f155c 
>   core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
>   core/src/test/scala/unit/kafka/server/ClientQuotaManagerTest.scala 
> PRE-CREATION 
>   core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
> f32d206d3f52f3f9f4d649c213edd7058f4b6150 
>   core/src/test/scala/unit/kafka/server/ThrottledResponseExpirationTest.scala 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/33049/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aditya Auradkar
> 
>

Re: [DISCUSS] Client-side Assignment for New Consumer

[Jiangjie Qin]

Ewen and Jason,

Thanks for the reply. Sorry I missed the metadata hash. Yes, that is a
clever approach and would solve my concern about the data passing around. I
can see both pros and cons from doing this, though. The advantage is we
don't need the topic metadata in JoinGroupResponse anymore. The downside is
that now rebalance have extra dependency on the consensus of metadata of
all consumers, which is obtained separately. So it is required that each
consumer refresh their metadata before sending a JoinGroupRequest,
otherwise in some cases (e.g. wildcard consumers) will almost certainly
fail for the first rebalance attempt. Even if we do that, since the
consumers are getting metadata from different brokers, the metadata might
still be inconsistent if there is a topic or partition change because the
UpdateMetadataRequest from controller might be handled at different time.
Just want to make sure we think through the cases so the protocol does not
cause us unexpected issues.

About the number of consumers, I think with the current liveliness
definition, we can tolerate churns by bumping up the session timeout. Also
I guess we will see an increasing number of consumers for new consumer,
because every the old consumer thread will probably become a consumer.

It is a valid concern for consumers that have large subscription set. This
might not be avoided though for client side assignment approach. One
solution is having topic names associate with a topic ID. And only use
topic ID in JoinGroupRequest and JoinGroupResponse, There is a discussion
thread about this to solve the topic renaming case but this is a completely
different discussion.

Thanks,

Jiangjie (Becket) Qin



On Thu, Aug 13, 2015 at 2:14 PM, Jason Gustafson  wrote:

> Thanks Jiangjie, that information helps. I agree the protocol must consider
> scalability. My point was that the synchronization barrier in the current
> protocol already effectively limits the number of consumers since it
> provides no way to gracefully handle churn. It wouldn't be worth worrying
> about scaling up to 100,000 members, for example, because there's no way
> the group would be stable. So we just need to set some clear expectations
> on the size we can scale to, and that can help inform the discussion on the
> size of messages in this protocol.
>
> Ewen and I were discussing this morning along similar lines to what you're
> suggesting. However, even if the coordinator decides on the metadata for
> the group, each member still needs to communicate its subscriptions to the
> rest of the group. This is nice for the regex case since the regex is
> probably small, but if the members have a large topic list, then we have
> the same problem. One thing I was thinking about was whether we really need
> to handle different subscriptions for every member. If the coordinator
> could guarantee that all members had the same subscription, then there
> would be no need for the coordinator to return the subscriptions for each
> member. However, this would prevent graceful upgrades. We might be able to
> fix that problem by allowing the consumer to provide two subscriptions to
> allowing rolling updates, but that starts to sound pretty nasty.
>
> -Jason
>
> On Thu, Aug 13, 2015 at 1:41 PM, Jiangjie Qin 
> wrote:
>
> > Jason,
> >
> > The protocol has to consider the scalability. The protocol in the wiki
> > means the JoinGroupResoponse size would be:
> > NumberOfTopics * (AvgTopicNameLength + 4) * (NumberOfConsumers)^2
> >
> > To give some real number, we have 26-node Mirror Maker cluster, each
> with 4
> > consumers. That is 104 consumers using regex ".*". And most of our
> clusters
> > have around 3000 topics, whose topic name are typically around 20
> > characters.
> >
> > I think the key issue for client side partition assignment logic is to
> make
> > sure 1) all the clients run the same algorithm. 2) all the clients make
> > decision on the same topic metadata. The second purpose can be done by
> > simply letting coordinator provide the topic metadata and all then member
> > information as source of truth. Is it necessary to pass topic metadata of
> > each consumer around? Can we keep the protocol metadata field completely
> > independent of topic metadata? I think In the JoinGroupResponse, we
> should
> > have only one copy of topic metadata provided by coordinator and is
> outside
> > of protocol metadata. If user decides to put some metadata in the
> > JoinGroupRequest and let coordinator pass around, they are responsible
> for
> > understanding the risk.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> >
> > On Thu, Aug 13, 2015 at 12:41 PM, Jason Gustafson 
> > wrote:
> >
> > > Hey Onur and Jiangjie,
> > >
> > > I've updated that wiki with a proposal to add regex subscriptions to
> the
> > > consumer metadata. Can you have a look to see if it addresses your
> > concern?
> > > In general, I think we should be a little careful when we are talking
> > about
> > > the scalabil

[jira] [Commented] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696196#comment-14696196
 ] 

Jiangjie Qin commented on KAFKA-2406:
-

Thanks, [~junrao].

> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Fix For: 0.8.3
>
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: NEW: API Stability annotations!

[Neha Narkhede]

Thanks for starting the discussion around API annotations, Gwen! I think it
is better to add the deprecated annotation after the new consumer API is
successfully deployed at a couple places.

On Thu, Aug 13, 2015 at 3:42 PM, Ismael Juma  wrote:

> On Thu, Aug 13, 2015 at 10:41 PM, Gwen Shapira  wrote:
>
> > IMO, both old producer and old consumer APIs should be marked as
> deprecated
> > for 0.8.3 (since the new code will be in and we want to encourage the
> > switch).
> >
>
> I can see the appeal of this, but it's also worth considering the downsides
> for users too:
>
> * It will introduce a number of deprecation warnings to everyone that
> upgrades to 0.8.3 even though the old Consumer APIs still work fine (also
> worth keeping in mind that in many projects, warnings cause a build
> failure)
> * The new Consumer is still marked as `Unstable` so it seems a bit odd to
> deprecate the old one
>
> I think a more conservative option would be to update the documentation to
> encourage users to move to the new consumer without adding deprecation
> annotations to the old consumer APIs. Some features that are only available
> in the new consumer (e.g. SSL support) provide further incentive to move.
> As Ewen suggested, the old consumer would then be deprecated in the
> following release. And removed in the one after that. The main downside
> would be having to maintain the old consumer for a little while longer.
>
> Something to think about. :)
>
> Best,
> Ismael
>



-- 
Thanks,
Neha

[jira] [Commented] (KAFKA-1782) Junit3 Misusage

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696161#comment-14696161
 ] 

Guozhang Wang commented on KAFKA-1782:
--

[~apakulov] we are currently moving to the PR for contribution / reviews, you 
can take a look at this wiki (section "Contributor and Reviewer Workflow"):

https://cwiki.apache.org/confluence/display/KAFKA/Patch+submission+and+review

> Junit3 Misusage
> ---
>
> Key: KAFKA-1782
> URL: https://issues.apache.org/jira/browse/KAFKA-1782
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Alexander Pakulov
>  Labels: newbie
> Fix For: 0.8.3
>
> Attachments: KAFKA-1782.patch, KAFKA-1782.patch, 
> KAFKA-1782_2015-06-18_11:52:49.patch, KAFKA-1782_2015-07-15_16:57:44.patch, 
> KAFKA-1782_2015-07-16_11:50:05.patch, KAFKA-1782_2015-07-16_11:56:11.patch
>
>
> This is found while I was working on KAFKA-1580: in many of our cases where 
> we explicitly extend from junit3suite (e.g. ProducerFailureHandlingTest), we 
> are actually misusing a bunch of features that only exist in Junit4, such as 
> (expected=classOf). For example, the following code
> {code}
> import org.scalatest.junit.JUnit3Suite
> import org.junit.Test
> import java.io.IOException
> class MiscTest extends JUnit3Suite {
>   @Test (expected = classOf[IOException])
>   def testSendOffset() {
>   }
> }
> {code}
> will actually pass even though IOException was not thrown since this 
> annotation is not supported in Junit3. Whereas
> {code}
> import org.junit._
> import java.io.IOException
> class MiscTest extends JUnit3Suite {
>   @Test (expected = classOf[IOException])
>   def testSendOffset() {
>   }
> }
> {code}
> or
> {code}
> import org.scalatest.junit.JUnitSuite
> import org.junit._
> import java.io.IOException
> class MiscTest extends JUnit3Suite {
>   @Test (expected = classOf[IOException])
>   def testSendOffset() {
>   }
> }
> {code}
> or
> {code}
> import org.junit._
> import java.io.IOException
> class MiscTest {
>   @Test (expected = classOf[IOException])
>   def testSendOffset() {
>   }
> }
> {code}
> will fail.
> I would propose to not rely on Junit annotations other than @Test itself but 
> use scala unit test annotations instead, for example:
> {code}
> import org.junit._
> import java.io.IOException
> class MiscTest {
>   @Test
>   def testSendOffset() {
> intercept[IOException] {
>   //nothing
> }
>   }
> }
> {code}
> will fail with a clearer stacktrace.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1782) Junit3 Misusage

[Alexander Pakulov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696141#comment-14696141
 ] 

Alexander Pakulov commented on KAFKA-1782:
--

[~guozhang] am I able to use github PR to send changset or kafka-review tool is 
still a preferred way?

> Junit3 Misusage
> ---
>
> Key: KAFKA-1782
> URL: https://issues.apache.org/jira/browse/KAFKA-1782
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Alexander Pakulov
>  Labels: newbie
> Fix For: 0.8.3
>
> Attachments: KAFKA-1782.patch, KAFKA-1782.patch, 
> KAFKA-1782_2015-06-18_11:52:49.patch, KAFKA-1782_2015-07-15_16:57:44.patch, 
> KAFKA-1782_2015-07-16_11:50:05.patch, KAFKA-1782_2015-07-16_11:56:11.patch
>
>
> This is found while I was working on KAFKA-1580: in many of our cases where 
> we explicitly extend from junit3suite (e.g. ProducerFailureHandlingTest), we 
> are actually misusing a bunch of features that only exist in Junit4, such as 
> (expected=classOf). For example, the following code
> {code}
> import org.scalatest.junit.JUnit3Suite
> import org.junit.Test
> import java.io.IOException
> class MiscTest extends JUnit3Suite {
>   @Test (expected = classOf[IOException])
>   def testSendOffset() {
>   }
> }
> {code}
> will actually pass even though IOException was not thrown since this 
> annotation is not supported in Junit3. Whereas
> {code}
> import org.junit._
> import java.io.IOException
> class MiscTest extends JUnit3Suite {
>   @Test (expected = classOf[IOException])
>   def testSendOffset() {
>   }
> }
> {code}
> or
> {code}
> import org.scalatest.junit.JUnitSuite
> import org.junit._
> import java.io.IOException
> class MiscTest extends JUnit3Suite {
>   @Test (expected = classOf[IOException])
>   def testSendOffset() {
>   }
> }
> {code}
> or
> {code}
> import org.junit._
> import java.io.IOException
> class MiscTest {
>   @Test (expected = classOf[IOException])
>   def testSendOffset() {
>   }
> }
> {code}
> will fail.
> I would propose to not rely on Junit annotations other than @Test itself but 
> use scala unit test annotations instead, for example:
> {code}
> import org.junit._
> import java.io.IOException
> class MiscTest {
>   @Test
>   def testSendOffset() {
> intercept[IOException] {
>   //nothing
> }
>   }
> }
> {code}
> will fail with a clearer stacktrace.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 33049: Patch for KAFKA-2084

[Joel Koshy]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/33049/#review95344
---



build.gradle (line 383)


I don't think these changes 383-385 are needed.

Also, (unrelated to this change) I'm seeing NPE errors in running unit 
tests (e.g., QuotasTest) - can you verify?


- Joel Koshy


On Aug. 13, 2015, 4:25 a.m., Aditya Auradkar wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/33049/
> ---
> 
> (Updated Aug. 13, 2015, 4:25 a.m.)
> 
> 
> Review request for kafka, Joel Koshy and Jun Rao.
> 
> 
> Bugs: KAFKA-2084
> https://issues.apache.org/jira/browse/KAFKA-2084
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Updated patch for quotas. This patch does the following: 
> 1. Add per-client metrics for both producer and consumers 
> 2. Add configuration for quotas 
> 3. Compute delay times in the metrics package and return the delay times in 
> QuotaViolationException 
> 4. Add a DelayQueue in KafkaApi's that can be used to throttle any type of 
> request. Implemented request throttling for produce and fetch requests. 
> 5. Added unit and integration test cases for both producer and consumer
> 6. This doesn't include a system test. There is a separate ticket for that
> 7. Fixed KAFKA-2191 - (Included fix from : 
> https://reviews.apache.org/r/34418/ )
> 
> Addressed comments from Joel and Jun
> 
> 
> Diffs
> -
> 
>   build.gradle 1b67e628c2fca897177c12b6afad9a8700fffd1f 
>   clients/src/main/java/org/apache/kafka/common/metrics/Quota.java 
> d82bb0c055e631425bc1ebbc7d387baac76aeeaa 
>   
> clients/src/main/java/org/apache/kafka/common/metrics/QuotaViolationException.java
>  a451e5385c9eca76b38b425e8ac856b2715fcffe 
>   clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 
> ca823fd4639523018311b814fde69b6177e73b97 
>   clients/src/main/java/org/apache/kafka/common/metrics/stats/Rate.java 
> 98429da34418f7f1deba1b5e44e2e6025212edb3 
>   clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java 
> 544e120594de78c43581a980b1e4087b4fb98ccb 
>   core/src/main/scala/kafka/server/ClientQuotaManager.scala PRE-CREATION 
>   core/src/main/scala/kafka/server/KafkaApis.scala 
> 7ea509c2c41acc00430c74e025e069a833aac4e7 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> dbe170f87331f43e2dc30165080d2cb7dfe5fdbf 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 84d4730ac634f9a5bf12a656e422fea03ad72da8 
>   core/src/main/scala/kafka/server/ReplicaManager.scala 
> 795220e7f63d163be90738b4c1a39687b44c1395 
>   core/src/main/scala/kafka/server/ThrottledResponse.scala PRE-CREATION 
>   core/src/main/scala/kafka/utils/ShutdownableThread.scala 
> fc226c863095b7761290292cd8755cd7ad0f155c 
>   core/src/test/scala/integration/kafka/api/QuotasTest.scala PRE-CREATION 
>   core/src/test/scala/unit/kafka/server/ClientQuotaManagerTest.scala 
> PRE-CREATION 
>   core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
> f32d206d3f52f3f9f4d649c213edd7058f4b6150 
>   core/src/test/scala/unit/kafka/server/ThrottledResponseExpirationTest.scala 
> PRE-CREATION 
> 
> Diff: https://reviews.apache.org/r/33049/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Aditya Auradkar
> 
>

Re: NEW: API Stability annotations!

[Ismael Juma]

On Thu, Aug 13, 2015 at 10:41 PM, Gwen Shapira  wrote:

> IMO, both old producer and old consumer APIs should be marked as deprecated
> for 0.8.3 (since the new code will be in and we want to encourage the
> switch).
>

I can see the appeal of this, but it's also worth considering the downsides
for users too:

* It will introduce a number of deprecation warnings to everyone that
upgrades to 0.8.3 even though the old Consumer APIs still work fine (also
worth keeping in mind that in many projects, warnings cause a build failure)
* The new Consumer is still marked as `Unstable` so it seems a bit odd to
deprecate the old one

I think a more conservative option would be to update the documentation to
encourage users to move to the new consumer without adding deprecation
annotations to the old consumer APIs. Some features that are only available
in the new consumer (e.g. SSL support) provide further incentive to move.
As Ewen suggested, the old consumer would then be deprecated in the
following release. And removed in the one after that. The main downside
would be having to maintain the old consumer for a little while longer.

Something to think about. :)

Best,
Ismael

[jira] [Commented] (KAFKA-2367) Add Copycat runtime data API

[James Cheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696072#comment-14696072
 ] 

James Cheng commented on KAFKA-2367:


Using Avro makes sense to me, instead of writing something ourselves. As 
[~gwenshap] pointed out, it seems to do most of what we need: allow building a 
runtime schema, and allow building data objects that match that schema. And 
using Avro means we wouldn't have to maintain our own thing.


> Add Copycat runtime data API
> 
>
> Key: KAFKA-2367
> URL: https://issues.apache.org/jira/browse/KAFKA-2367
> Project: Kafka
>  Issue Type: Sub-task
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.8.3
>
>
> Design the API used for runtime data in Copycat. This API is used to 
> construct schemas and records that Copycat processes. This needs to be a 
> fairly general data model (think Avro, JSON, Protobufs, Thrift) in order to 
> support complex, varied data types that may be input from/output to many data 
> systems.
> This should issue should also address the serialization interfaces used 
> within Copycat, which translate the runtime data into serialized byte[] form. 
> It is important that these be considered together because the data format can 
> be used in multiple ways (records, partition IDs, partition offsets), so it 
> and the corresponding serializers must be sufficient for all these use cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: expose vagrant base box as variable

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/137


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-1215) Rack-Aware replica assignment option

[Allen Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696026#comment-14696026
 ] 

Allen Wang commented on KAFKA-1215:
---

[~junrao] Can you review the GitHub pull request or have someone take a look?


> Rack-Aware replica assignment option
> 
>
> Key: KAFKA-1215
> URL: https://issues.apache.org/jira/browse/KAFKA-1215
> Project: Kafka
>  Issue Type: Improvement
>  Components: replication
>Affects Versions: 0.8.0
>Reporter: Joris Van Remoortere
>Assignee: Jun Rao
> Fix For: 0.9.0
>
> Attachments: rack_aware_replica_assignment_v1.patch, 
> rack_aware_replica_assignment_v2.patch
>
>
> Adding a rack-id to kafka config. This rack-id can be used during replica 
> assignment by using the max-rack-replication argument in the admin scripts 
> (create topic, etc.). By default the original replication assignment 
> algorithm is used because max-rack-replication defaults to -1. 
> max-rack-replication > -1 is not honored if you are doing manual replica 
> assignment (preffered).
> If this looks good I can add some test cases specific to the rack-aware 
> assignment.
> I can also port this to trunk. We are currently running 0.8.0 in production 
> and need this, so i wrote the patch against that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-2406.

   Resolution: Fixed
Fix Version/s: 0.8.3

Issue resolved by pull request 114
[https://github.com/apache/kafka/pull/114]

> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Fix For: 0.8.3
>
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14696012#comment-14696012
 ] 

ASF GitHub Bot commented on KAFKA-2406:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/114


> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
> Fix For: 0.8.3
>
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2406: Throttle ISR propagation

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/114


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: NEW: API Stability annotations!

[Gwen Shapira]

IMO, both old producer and old consumer APIs should be marked as deprecated
for 0.8.3 (since the new code will be in and we want to encourage the
switch).

I'd like to see 0.9 released with just the new java clients.


On Thu, Aug 13, 2015 at 6:47 AM, Ismael Juma  wrote:

> On Thu, Aug 13, 2015 at 8:36 AM, Ewen Cheslack-Postava 
> wrote:
>
> > On deprecation, I think we should definitely use the standard annotation
> to
> > handle this.
>
>
> Thanks Ewen. I agree that we should use @Deprecated for language-level
> deprecations. My question was regarding the following:
>
> - unstable - can change at any time
> > - evolving - can break compatibility at minor releases (i.e. 0.9 may be
> > incompatible with 0.8)
> > - stable - will only break compatibility at major releases (1.0, 2.0,
> etc)
> >
>
>
> Let's pick `evolving`, as an example. Does it mean that we can break
> compatibility at minor releases without a deprecation cycle? Or does it
> mean that we would need to deprecate it for one minor release and then
> remove in the next? The latter is nicer for users as they have one release
> where both the deprecated and recommended still work, but it adds more
> overhead to Kafka development. What is the current thinking around this?
>
> The annotation doesn't necessarily work for everything (e.g.,
> > deprecating older versions of the protocol doesn't work via Java
> > deprecation annotations), but we should definitely use it for deprecated
> > Java/Scala APIs. Perhaps we should start by marking all the old producer
> > APIs deprecated
>
>
> Seems sensible assuming that users have found it easy to migrate to the new
> producer in the latest release.
>
> and, after the next release, the old consumer APIs too? :)
>
>
> Similar to above, it would be good to do that once we have some reports
> from users who moved from one API to the next.
>
> Best,
> Ismael
>

[GitHub] kafka pull request: MINOR: expose vagrant base box as variable

[granders]

GitHub user granders opened a pull request:

https://github.com/apache/kafka/pull/137

MINOR: expose vagrant base box as variable

Added base_box variable to Vagrantfile. This makes it possible to override 
the base box in Vagrantfile.local.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/confluentinc/kafka minor-expose-vagrant-box

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/137.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #137


commit 44936f7918a80aed42729d466d348b99703395ff
Author: Geoff Anderson 
Date:   2015-08-13T21:18:41Z

Added base_box variable to Vagrantfile. This makes it possible to override 
the base box in Vagrantfile.local.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] Client-side Assignment for New Consumer

[Jason Gustafson]

Thanks Jiangjie, that information helps. I agree the protocol must consider
scalability. My point was that the synchronization barrier in the current
protocol already effectively limits the number of consumers since it
provides no way to gracefully handle churn. It wouldn't be worth worrying
about scaling up to 100,000 members, for example, because there's no way
the group would be stable. So we just need to set some clear expectations
on the size we can scale to, and that can help inform the discussion on the
size of messages in this protocol.

Ewen and I were discussing this morning along similar lines to what you're
suggesting. However, even if the coordinator decides on the metadata for
the group, each member still needs to communicate its subscriptions to the
rest of the group. This is nice for the regex case since the regex is
probably small, but if the members have a large topic list, then we have
the same problem. One thing I was thinking about was whether we really need
to handle different subscriptions for every member. If the coordinator
could guarantee that all members had the same subscription, then there
would be no need for the coordinator to return the subscriptions for each
member. However, this would prevent graceful upgrades. We might be able to
fix that problem by allowing the consumer to provide two subscriptions to
allowing rolling updates, but that starts to sound pretty nasty.

-Jason

On Thu, Aug 13, 2015 at 1:41 PM, Jiangjie Qin 
wrote:

> Jason,
>
> The protocol has to consider the scalability. The protocol in the wiki
> means the JoinGroupResoponse size would be:
> NumberOfTopics * (AvgTopicNameLength + 4) * (NumberOfConsumers)^2
>
> To give some real number, we have 26-node Mirror Maker cluster, each with 4
> consumers. That is 104 consumers using regex ".*". And most of our clusters
> have around 3000 topics, whose topic name are typically around 20
> characters.
>
> I think the key issue for client side partition assignment logic is to make
> sure 1) all the clients run the same algorithm. 2) all the clients make
> decision on the same topic metadata. The second purpose can be done by
> simply letting coordinator provide the topic metadata and all then member
> information as source of truth. Is it necessary to pass topic metadata of
> each consumer around? Can we keep the protocol metadata field completely
> independent of topic metadata? I think In the JoinGroupResponse, we should
> have only one copy of topic metadata provided by coordinator and is outside
> of protocol metadata. If user decides to put some metadata in the
> JoinGroupRequest and let coordinator pass around, they are responsible for
> understanding the risk.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
> On Thu, Aug 13, 2015 at 12:41 PM, Jason Gustafson 
> wrote:
>
> > Hey Onur and Jiangjie,
> >
> > I've updated that wiki with a proposal to add regex subscriptions to the
> > consumer metadata. Can you have a look to see if it addresses your
> concern?
> > In general, I think we should be a little careful when we are talking
> about
> > the scalability of the protocol. Regardless of whether assignment is done
> > on the server or the client, the protocol assumes a relatively stable
> > configuration. When the number of consumers increases beyond a certain
> > limit, then membership churn becomes a major concern. Similarly there is
> a
> > notion of metadata churn when topics are added, deleted, or resized. If
> > either membership or metadata changes, then the protocol forces all
> > consumers to stop consumption and rejoin the group. If this happens often
> > enough, then it can severely impact the ability of the consumer to make
> > progress. The point is that the protocol may already be unsuited to cases
> > where there are either a large number of consumers or topics. I wonder if
> > you guys can share your thoughts about your scaling expectations?
> >
> > -Jason
> >
> >
> >
> >
> > On Wed, Aug 12, 2015 at 12:28 PM, Jason Gustafson 
> > wrote:
> >
> > > Hey Jiangjie,
> > >
> > > That's a great point. In the worst case (the mirror maker case I
> guess),
> > > the join group response can be massive. This would be especially deadly
> > > when there is a lot of churn in the group (e.g. in a rolling upgrade).
> > The
> > > current protocol is not great for this case either, but it's
> > significantly
> > > better. Here are a couple ways to deal with the size:
> > >
> > > 1. First, we could have the coordinator compress the responses. This
> > would
> > > probably be pretty effective if applied across the metadata from all
> > > members.
> > >
> > > 2. I think the regex case is the main problem. Is that right? We could
> > > extend the metadata to allow the consumer to embed its regex
> subscription
> > > in the metadata directly (note this might be a good idea regardless of
> > the
> > > rest of this proposal). To support regex on the consumer, we must fetch
> > > metadata for all topics. Rather than having all 

Re: [DISCUSS] Client-side Assignment for New Consumer

[Ewen Cheslack-Postava]

Becket,

Did you see the updates to the "ConsumerEmbeddedProtocol" section,
specifically on how regex subscriptions like you're describing are handled?
The updated version removes almost all the metadata in the regex case by
taking advantage of exactly the point you made in (2). The clients don't
actually need to transmit all the metadata, they just need to make sure
they are all making decisions using the same metadata. They can verify this
with a hash of the metadata they are using instead of sending the full
metadata.

-Ewen

On Thu, Aug 13, 2015 at 1:41 PM, Jiangjie Qin 
wrote:

> Jason,
>
> The protocol has to consider the scalability. The protocol in the wiki
> means the JoinGroupResoponse size would be:
> NumberOfTopics * (AvgTopicNameLength + 4) * (NumberOfConsumers)^2
>
> To give some real number, we have 26-node Mirror Maker cluster, each with 4
> consumers. That is 104 consumers using regex ".*". And most of our clusters
> have around 3000 topics, whose topic name are typically around 20
> characters.
>
> I think the key issue for client side partition assignment logic is to make
> sure 1) all the clients run the same algorithm. 2) all the clients make
> decision on the same topic metadata. The second purpose can be done by
> simply letting coordinator provide the topic metadata and all then member
> information as source of truth. Is it necessary to pass topic metadata of
> each consumer around? Can we keep the protocol metadata field completely
> independent of topic metadata? I think In the JoinGroupResponse, we should
> have only one copy of topic metadata provided by coordinator and is outside
> of protocol metadata. If user decides to put some metadata in the
> JoinGroupRequest and let coordinator pass around, they are responsible for
> understanding the risk.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
>
> On Thu, Aug 13, 2015 at 12:41 PM, Jason Gustafson 
> wrote:
>
> > Hey Onur and Jiangjie,
> >
> > I've updated that wiki with a proposal to add regex subscriptions to the
> > consumer metadata. Can you have a look to see if it addresses your
> concern?
> > In general, I think we should be a little careful when we are talking
> about
> > the scalability of the protocol. Regardless of whether assignment is done
> > on the server or the client, the protocol assumes a relatively stable
> > configuration. When the number of consumers increases beyond a certain
> > limit, then membership churn becomes a major concern. Similarly there is
> a
> > notion of metadata churn when topics are added, deleted, or resized. If
> > either membership or metadata changes, then the protocol forces all
> > consumers to stop consumption and rejoin the group. If this happens often
> > enough, then it can severely impact the ability of the consumer to make
> > progress. The point is that the protocol may already be unsuited to cases
> > where there are either a large number of consumers or topics. I wonder if
> > you guys can share your thoughts about your scaling expectations?
> >
> > -Jason
> >
> >
> >
> >
> > On Wed, Aug 12, 2015 at 12:28 PM, Jason Gustafson 
> > wrote:
> >
> > > Hey Jiangjie,
> > >
> > > That's a great point. In the worst case (the mirror maker case I
> guess),
> > > the join group response can be massive. This would be especially deadly
> > > when there is a lot of churn in the group (e.g. in a rolling upgrade).
> > The
> > > current protocol is not great for this case either, but it's
> > significantly
> > > better. Here are a couple ways to deal with the size:
> > >
> > > 1. First, we could have the coordinator compress the responses. This
> > would
> > > probably be pretty effective if applied across the metadata from all
> > > members.
> > >
> > > 2. I think the regex case is the main problem. Is that right? We could
> > > extend the metadata to allow the consumer to embed its regex
> subscription
> > > in the metadata directly (note this might be a good idea regardless of
> > the
> > > rest of this proposal). To support regex on the consumer, we must fetch
> > > metadata for all topics. Rather than having all regex subscribers embed
> > all
> > > of this metadata in their join group requests, they could instead
> embed a
> > > hash of it. Then after the join group responses are received, they just
> > > need to check that the hashes are the same. If there is a mismatch
> (which
> > > should only occur when topics are created, deleted, or resized), then
> the
> > > group members must refetch the metadata and rejoin the group. This is
> > also
> > > how the current protocol behaves when there is a change in the topic
> > > metadata affecting the group--someone (either the coordinator or the
> > > consumer) detects the change and forces the group to rebalance.
> > >
> > > What do you think?
> > >
> > > (Also I think adding groupId/generationId to fetch and produce requests
> > > seems like an interesting line of thought.)
> > >
> > > -Jason
> > >
> > >
> > >
> > > On Wed, Aug 12, 2015 at 10:57

[jira] [Commented] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695932#comment-14695932
 ] 

Jiangjie Qin commented on KAFKA-2406:
-

[~junrao] ping for review.

> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Client-side Assignment for New Consumer

[Jiangjie Qin]

Jason,

The protocol has to consider the scalability. The protocol in the wiki
means the JoinGroupResoponse size would be:
NumberOfTopics * (AvgTopicNameLength + 4) * (NumberOfConsumers)^2

To give some real number, we have 26-node Mirror Maker cluster, each with 4
consumers. That is 104 consumers using regex ".*". And most of our clusters
have around 3000 topics, whose topic name are typically around 20
characters.

I think the key issue for client side partition assignment logic is to make
sure 1) all the clients run the same algorithm. 2) all the clients make
decision on the same topic metadata. The second purpose can be done by
simply letting coordinator provide the topic metadata and all then member
information as source of truth. Is it necessary to pass topic metadata of
each consumer around? Can we keep the protocol metadata field completely
independent of topic metadata? I think In the JoinGroupResponse, we should
have only one copy of topic metadata provided by coordinator and is outside
of protocol metadata. If user decides to put some metadata in the
JoinGroupRequest and let coordinator pass around, they are responsible for
understanding the risk.

Thanks,

Jiangjie (Becket) Qin


On Thu, Aug 13, 2015 at 12:41 PM, Jason Gustafson 
wrote:

> Hey Onur and Jiangjie,
>
> I've updated that wiki with a proposal to add regex subscriptions to the
> consumer metadata. Can you have a look to see if it addresses your concern?
> In general, I think we should be a little careful when we are talking about
> the scalability of the protocol. Regardless of whether assignment is done
> on the server or the client, the protocol assumes a relatively stable
> configuration. When the number of consumers increases beyond a certain
> limit, then membership churn becomes a major concern. Similarly there is a
> notion of metadata churn when topics are added, deleted, or resized. If
> either membership or metadata changes, then the protocol forces all
> consumers to stop consumption and rejoin the group. If this happens often
> enough, then it can severely impact the ability of the consumer to make
> progress. The point is that the protocol may already be unsuited to cases
> where there are either a large number of consumers or topics. I wonder if
> you guys can share your thoughts about your scaling expectations?
>
> -Jason
>
>
>
>
> On Wed, Aug 12, 2015 at 12:28 PM, Jason Gustafson 
> wrote:
>
> > Hey Jiangjie,
> >
> > That's a great point. In the worst case (the mirror maker case I guess),
> > the join group response can be massive. This would be especially deadly
> > when there is a lot of churn in the group (e.g. in a rolling upgrade).
> The
> > current protocol is not great for this case either, but it's
> significantly
> > better. Here are a couple ways to deal with the size:
> >
> > 1. First, we could have the coordinator compress the responses. This
> would
> > probably be pretty effective if applied across the metadata from all
> > members.
> >
> > 2. I think the regex case is the main problem. Is that right? We could
> > extend the metadata to allow the consumer to embed its regex subscription
> > in the metadata directly (note this might be a good idea regardless of
> the
> > rest of this proposal). To support regex on the consumer, we must fetch
> > metadata for all topics. Rather than having all regex subscribers embed
> all
> > of this metadata in their join group requests, they could instead embed a
> > hash of it. Then after the join group responses are received, they just
> > need to check that the hashes are the same. If there is a mismatch (which
> > should only occur when topics are created, deleted, or resized), then the
> > group members must refetch the metadata and rejoin the group. This is
> also
> > how the current protocol behaves when there is a change in the topic
> > metadata affecting the group--someone (either the coordinator or the
> > consumer) detects the change and forces the group to rebalance.
> >
> > What do you think?
> >
> > (Also I think adding groupId/generationId to fetch and produce requests
> > seems like an interesting line of thought.)
> >
> > -Jason
> >
> >
> >
> > On Wed, Aug 12, 2015 at 10:57 AM, Jiangjie Qin  >
> > wrote:
> >
> >> Hey Ewen,
> >>
> >> Onur and I discussed this a little bit more. And we are still worrying
> >> about passing all the metadata of all consumers around.
> >>
> >> Let's say I have a cluster has 10,000 topics, the average topic name
> >> length
> >> is 10 bytes. In this case, the opaque metadata will have 10 * 10,000 =
> >> 100KB for topic name, for each topic, there is a 4-byte integer of
> number
> >> of partitions, that's another 40KB. So one global topic metadata will
> have
> >> 140KB data. If I have 100 consumers who are using wildcard to consume
> from
> >> all the topics. That means the protocol metadata end up in the
> >> JoinGroupResponse will be 140KB * 100 = 14MB. And the JoinGroupResponse
> >> will need to be sent to 100 different c

[jira] [Commented] (KAFKA-2388) subscribe(topic)/unsubscribe(topic) should either take a callback to allow user to handle exceptions or it should be synchronous.

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695877#comment-14695877
 ] 

Jason Gustafson commented on KAFKA-2388:


[~onurkaraman] I'll take a shot at it. If I can't get a patch together pretty 
quickly, I'll send it back to you. Perhaps instead of the generic onError, we 
handle just the specific case of the topic not existing (for example: 
onUnknownTopic())? Are there any other relevant error cases?

> subscribe(topic)/unsubscribe(topic) should either take a callback to allow 
> user to handle exceptions or it should be synchronous.
> -
>
> Key: KAFKA-2388
> URL: https://issues.apache.org/jira/browse/KAFKA-2388
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>Assignee: Onur Karaman
>
> According to the mailing list discussion on the consumer interface, we'll 
> replace:
> {code}
> public void subscribe(String... topics);
> public void subscribe(TopicPartition... partitions);
> public Set subscriptions();
> {code}
> with:
> {code}
> void subscribe(List topics, RebalanceCallback callback);
> void assign(List partitions);
> List subscriptions();
> List assignments();
> {code}
> We don't need the unsubscribe APIs anymore.
> The RebalanceCallback would look like:
> {code}
> interface RebalanceCallback {
>   void onAssignment(List partitions);
>   void onRevocation(List partitions);
>   // handle non-existing topics, etc.
>   void onError(Exception e);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2351) Brokers are having a problem shutting down correctly

[Mayuresh Gharat (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695859#comment-14695859
 ] 

Mayuresh Gharat commented on KAFKA-2351:


Updated reviewboard https://reviews.apache.org/r/36652/diff/
 against branch origin/trunk

> Brokers are having a problem shutting down correctly
> 
>
> Key: KAFKA-2351
> URL: https://issues.apache.org/jira/browse/KAFKA-2351
> Project: Kafka
>  Issue Type: Bug
>Reporter: Mayuresh Gharat
>Assignee: Mayuresh Gharat
> Attachments: KAFKA-2351.patch, KAFKA-2351_2015-07-21_14:58:13.patch, 
> KAFKA-2351_2015-07-23_21:36:52.patch, KAFKA-2351_2015-08-13_13:10:05.patch
>
>
> The run() in Acceptor during shutdown might throw an exception that is not 
> caught and it never reaches shutdownComplete due to which the latch is not 
> counted down and the broker will not be able to shutdown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2351) Brokers are having a problem shutting down correctly

[Mayuresh Gharat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayuresh Gharat updated KAFKA-2351:
---
Status: Patch Available  (was: In Progress)

> Brokers are having a problem shutting down correctly
> 
>
> Key: KAFKA-2351
> URL: https://issues.apache.org/jira/browse/KAFKA-2351
> Project: Kafka
>  Issue Type: Bug
>Reporter: Mayuresh Gharat
>Assignee: Mayuresh Gharat
> Attachments: KAFKA-2351.patch, KAFKA-2351_2015-07-21_14:58:13.patch, 
> KAFKA-2351_2015-07-23_21:36:52.patch, KAFKA-2351_2015-08-13_13:10:05.patch
>
>
> The run() in Acceptor during shutdown might throw an exception that is not 
> caught and it never reaches shutdownComplete due to which the latch is not 
> counted down and the broker will not be able to shutdown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2351) Brokers are having a problem shutting down correctly

[Mayuresh Gharat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayuresh Gharat updated KAFKA-2351:
---
Attachment: KAFKA-2351_2015-08-13_13:10:05.patch

> Brokers are having a problem shutting down correctly
> 
>
> Key: KAFKA-2351
> URL: https://issues.apache.org/jira/browse/KAFKA-2351
> Project: Kafka
>  Issue Type: Bug
>Reporter: Mayuresh Gharat
>Assignee: Mayuresh Gharat
> Attachments: KAFKA-2351.patch, KAFKA-2351_2015-07-21_14:58:13.patch, 
> KAFKA-2351_2015-07-23_21:36:52.patch, KAFKA-2351_2015-08-13_13:10:05.patch
>
>
> The run() in Acceptor during shutdown might throw an exception that is not 
> caught and it never reaches shutdownComplete due to which the latch is not 
> counted down and the broker will not be able to shutdown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 36652: Patch for KAFKA-2351

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36652/
---

(Updated Aug. 13, 2015, 8:10 p.m.)


Review request for kafka.


Bugs: KAFKA-2351
https://issues.apache.org/jira/browse/KAFKA-2351


Repository: kafka


Description (updated)
---

Added a try-catch to catch any exceptions thrown by the nioSelector


Addressed comments on the Jira ticket


Addressed Jun's comments


Diffs (updated)
-

  core/src/main/scala/kafka/network/SocketServer.scala 
dbe784b63817fd94e1593136926db17fac6fa3d7 

Diff: https://reviews.apache.org/r/36652/diff/


Testing
---


Thanks,

Mayuresh Gharat

Re: [DISCUSS] Client-side Assignment for New Consumer

[Jason Gustafson]

Hey Onur and Jiangjie,

I've updated that wiki with a proposal to add regex subscriptions to the
consumer metadata. Can you have a look to see if it addresses your concern?
In general, I think we should be a little careful when we are talking about
the scalability of the protocol. Regardless of whether assignment is done
on the server or the client, the protocol assumes a relatively stable
configuration. When the number of consumers increases beyond a certain
limit, then membership churn becomes a major concern. Similarly there is a
notion of metadata churn when topics are added, deleted, or resized. If
either membership or metadata changes, then the protocol forces all
consumers to stop consumption and rejoin the group. If this happens often
enough, then it can severely impact the ability of the consumer to make
progress. The point is that the protocol may already be unsuited to cases
where there are either a large number of consumers or topics. I wonder if
you guys can share your thoughts about your scaling expectations?

-Jason




On Wed, Aug 12, 2015 at 12:28 PM, Jason Gustafson 
wrote:

> Hey Jiangjie,
>
> That's a great point. In the worst case (the mirror maker case I guess),
> the join group response can be massive. This would be especially deadly
> when there is a lot of churn in the group (e.g. in a rolling upgrade). The
> current protocol is not great for this case either, but it's significantly
> better. Here are a couple ways to deal with the size:
>
> 1. First, we could have the coordinator compress the responses. This would
> probably be pretty effective if applied across the metadata from all
> members.
>
> 2. I think the regex case is the main problem. Is that right? We could
> extend the metadata to allow the consumer to embed its regex subscription
> in the metadata directly (note this might be a good idea regardless of the
> rest of this proposal). To support regex on the consumer, we must fetch
> metadata for all topics. Rather than having all regex subscribers embed all
> of this metadata in their join group requests, they could instead embed a
> hash of it. Then after the join group responses are received, they just
> need to check that the hashes are the same. If there is a mismatch (which
> should only occur when topics are created, deleted, or resized), then the
> group members must refetch the metadata and rejoin the group. This is also
> how the current protocol behaves when there is a change in the topic
> metadata affecting the group--someone (either the coordinator or the
> consumer) detects the change and forces the group to rebalance.
>
> What do you think?
>
> (Also I think adding groupId/generationId to fetch and produce requests
> seems like an interesting line of thought.)
>
> -Jason
>
>
>
> On Wed, Aug 12, 2015 at 10:57 AM, Jiangjie Qin 
> wrote:
>
>> Hey Ewen,
>>
>> Onur and I discussed this a little bit more. And we are still worrying
>> about passing all the metadata of all consumers around.
>>
>> Let's say I have a cluster has 10,000 topics, the average topic name
>> length
>> is 10 bytes. In this case, the opaque metadata will have 10 * 10,000 =
>> 100KB for topic name, for each topic, there is a 4-byte integer of number
>> of partitions, that's another 40KB. So one global topic metadata will have
>> 140KB data. If I have 100 consumers who are using wildcard to consume from
>> all the topics. That means the protocol metadata end up in the
>> JoinGroupResponse will be 140KB * 100 = 14MB. And the JoinGroupResponse
>> will need to be sent to 100 different consumers, that means 14MB * 100 =
>> 1.4GB need to be sent by the consumer coordinator for one rebalance. How
>> would that work?
>>
>> Also, having two consumers (old owner and new owner) consuming from the
>> same partition might also be a problem. e.g. people are updating database.
>> One thing might worth doing is to add GroupId and Generation ID to
>> ProducerRequest and FetchRequest as well. This will also help with the
>> single producer use case. However, this is probably orthogonal to this
>> thread given the current new consumer also has this problem and I believe
>> we need to fix it.
>>
>> Thanks,
>>
>> Jiangjie (Becket) Qin
>>
>> On Tue, Aug 11, 2015 at 11:43 PM, Ewen Cheslack-Postava <
>> e...@confluent.io>
>> wrote:
>>
>> > On Tue, Aug 11, 2015 at 11:29 PM, Jiangjie Qin
>> 
>> > wrote:
>> >
>> > > Ewen,
>> > >
>> > > Thanks for the explanation.
>> > >
>> > > For (1), I am more concerned about the failure case instead of normal
>> > case.
>> > > What if a consumer somehow was kick out of a group but is still
>> consuming
>> > > and committing offsets? Does that mean the new owner and old owner
>> might
>> > > potentially consuming from and committing offsets for the same
>> partition?
>> > > In the old consumer, this won't happen because the new consumer will
>> not
>> > be
>> > > able to start consumption unless the previous owner has released its
>> > > ownership. Basically, without the ownership g

[jira] [Reopened] (KAFKA-2398) Transient test failure for SocketServerTest - Socket closed.

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava reopened KAFKA-2398:
--

> Transient test failure for SocketServerTest - Socket closed.
> 
>
> Key: KAFKA-2398
> URL: https://issues.apache.org/jira/browse/KAFKA-2398
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>
> See the following transient test failure for SocketServerTest.
> kafka.network.SocketServerTest > simpleRequest FAILED
> java.net.SocketException: Socket closed
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at 
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at 
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at 
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at java.net.Socket.connect(Socket.java:528)
> at java.net.Socket.(Socket.java:425)
> at java.net.Socket.(Socket.java:208)
> at kafka.network.SocketServerTest.connect(SocketServerTest.scala:84)
> at 
> kafka.network.SocketServerTest.simpleRequest(SocketServerTest.scala:94)
> kafka.network.SocketServerTest > tooBigRequestIsRejected FAILED
> java.net.SocketException: Socket closed
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at 
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at 
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at 
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at java.net.Socket.connect(Socket.java:528)
> at java.net.Socket.(Socket.java:425)
> at java.net.Socket.(Socket.java:208)
> at kafka.network.SocketServerTest.connect(SocketServerTest.scala:84)
> at 
> kafka.network.SocketServerTest.tooBigRequestIsRejected(SocketServerTest.scala:124)
> kafka.network.SocketServerTest > testSocketsCloseOnShutdown FAILED
> java.net.SocketException: Socket closed
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at 
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at 
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at 
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at java.net.Socket.connect(Socket.java:528)
> at java.net.Socket.(Socket.java:425)
> at java.net.Socket.(Socket.java:208)
> at kafka.network.SocketServerTest.connect(SocketServerTest.scala:84)
> at 
> kafka.network.SocketServerTest.testSocketsCloseOnShutdown(SocketServerTest.scala:136)
> kafka.network.SocketServerTest > testMaxConnectionsPerIp FAILED
> java.net.SocketException: Socket closed
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at 
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at 
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at 
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at java.net.Socket.connect(Socket.java:528)
> at java.net.Socket.(Socket.java:425)
> at java.net.Socket.(Socket.java:208)
> at kafka.network.SocketServerTest.connect(SocketServerTest.scala:84)
> at 
> kafka.network.SocketServerTest$$anonfun$1.apply(SocketServerTest.scala:170)
> at 
> kafka.network.SocketServerTest$$anonfun$1.apply(SocketServerTest.scala:170)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at scala.collection.immutable.Range.foreach(Range.scala:141)
> at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at 
> kafka.network.SocketServerTest.testMaxConnectionsPerIp(SocketServerTest.scala:170)
> kafka.network.SocketServerTest > testMaxConnectionsPerIPOverrides FAILED
> java.net.SocketException: Socket closed
>   

[jira] [Commented] (KAFKA-2398) Transient test failure for SocketServerTest - Socket closed.

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695775#comment-14695775
 ] 

Ewen Cheslack-Postava commented on KAFKA-2398:
--

Sadly this does not seem to have fully resolved the issue. These tests passed a 
couple of times after I applied the patch, but now they are back to failing. 
Reopening this.

> Transient test failure for SocketServerTest - Socket closed.
> 
>
> Key: KAFKA-2398
> URL: https://issues.apache.org/jira/browse/KAFKA-2398
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>
> See the following transient test failure for SocketServerTest.
> kafka.network.SocketServerTest > simpleRequest FAILED
> java.net.SocketException: Socket closed
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at 
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at 
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at 
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at java.net.Socket.connect(Socket.java:528)
> at java.net.Socket.(Socket.java:425)
> at java.net.Socket.(Socket.java:208)
> at kafka.network.SocketServerTest.connect(SocketServerTest.scala:84)
> at 
> kafka.network.SocketServerTest.simpleRequest(SocketServerTest.scala:94)
> kafka.network.SocketServerTest > tooBigRequestIsRejected FAILED
> java.net.SocketException: Socket closed
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at 
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at 
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at 
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at java.net.Socket.connect(Socket.java:528)
> at java.net.Socket.(Socket.java:425)
> at java.net.Socket.(Socket.java:208)
> at kafka.network.SocketServerTest.connect(SocketServerTest.scala:84)
> at 
> kafka.network.SocketServerTest.tooBigRequestIsRejected(SocketServerTest.scala:124)
> kafka.network.SocketServerTest > testSocketsCloseOnShutdown FAILED
> java.net.SocketException: Socket closed
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at 
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at 
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at 
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at java.net.Socket.connect(Socket.java:528)
> at java.net.Socket.(Socket.java:425)
> at java.net.Socket.(Socket.java:208)
> at kafka.network.SocketServerTest.connect(SocketServerTest.scala:84)
> at 
> kafka.network.SocketServerTest.testSocketsCloseOnShutdown(SocketServerTest.scala:136)
> kafka.network.SocketServerTest > testMaxConnectionsPerIp FAILED
> java.net.SocketException: Socket closed
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at 
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at 
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at 
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at java.net.Socket.connect(Socket.java:528)
> at java.net.Socket.(Socket.java:425)
> at java.net.Socket.(Socket.java:208)
> at kafka.network.SocketServerTest.connect(SocketServerTest.scala:84)
> at 
> kafka.network.SocketServerTest$$anonfun$1.apply(SocketServerTest.scala:170)
> at 
> kafka.network.SocketServerTest$$anonfun$1.apply(SocketServerTest.scala:170)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at scala.collection.immutable.Range.foreach(Range.scala:141)
> at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:10

[jira] [Commented] (KAFKA-2388) subscribe(topic)/unsubscribe(topic) should either take a callback to allow user to handle exceptions or it should be synchronous.

[Onur Karaman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695753#comment-14695753
 ] 

Onur Karaman commented on KAFKA-2388:
-

You could take it over if you want.

Sorry, I started working on it but something else took priority. My scrap work 
is here: 
https://github.com/onurkaraman/kafka/commit/0fd461ba3e24d46b3e8f29275d3dbe6c5038c3d8
Feel free to completely throw it away.

I think this ticket is really trying to address two things:
1. distinguishing subscriptions from assignments. This is something that should 
definitely happen.
2. Introduce an onError callback to subscribe(). This one is less clear to me. 
Regardless of what we decide here, we definitely need documentation of 
exceptions in KafkaConsumer.

> subscribe(topic)/unsubscribe(topic) should either take a callback to allow 
> user to handle exceptions or it should be synchronous.
> -
>
> Key: KAFKA-2388
> URL: https://issues.apache.org/jira/browse/KAFKA-2388
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>Assignee: Onur Karaman
>
> According to the mailing list discussion on the consumer interface, we'll 
> replace:
> {code}
> public void subscribe(String... topics);
> public void subscribe(TopicPartition... partitions);
> public Set subscriptions();
> {code}
> with:
> {code}
> void subscribe(List topics, RebalanceCallback callback);
> void assign(List partitions);
> List subscriptions();
> List assignments();
> {code}
> We don't need the unsubscribe APIs anymore.
> The RebalanceCallback would look like:
> {code}
> interface RebalanceCallback {
>   void onAssignment(List partitions);
>   void onRevocation(List partitions);
>   // handle non-existing topics, etc.
>   void onError(Exception e);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1782) Junit3 Misusage

[Alexander Pakulov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695654#comment-14695654
 ] 

Alexander Pakulov commented on KAFKA-1782:
--

[~ewencp] [~guozhang] thanks for taking care of it.

> Junit3 Misusage
> ---
>
> Key: KAFKA-1782
> URL: https://issues.apache.org/jira/browse/KAFKA-1782
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Alexander Pakulov
>  Labels: newbie
> Fix For: 0.8.3
>
> Attachments: KAFKA-1782.patch, KAFKA-1782.patch, 
> KAFKA-1782_2015-06-18_11:52:49.patch, KAFKA-1782_2015-07-15_16:57:44.patch, 
> KAFKA-1782_2015-07-16_11:50:05.patch, KAFKA-1782_2015-07-16_11:56:11.patch
>
>
> This is found while I was working on KAFKA-1580: in many of our cases where 
> we explicitly extend from junit3suite (e.g. ProducerFailureHandlingTest), we 
> are actually misusing a bunch of features that only exist in Junit4, such as 
> (expected=classOf). For example, the following code
> {code}
> import org.scalatest.junit.JUnit3Suite
> import org.junit.Test
> import java.io.IOException
> class MiscTest extends JUnit3Suite {
>   @Test (expected = classOf[IOException])
>   def testSendOffset() {
>   }
> }
> {code}
> will actually pass even though IOException was not thrown since this 
> annotation is not supported in Junit3. Whereas
> {code}
> import org.junit._
> import java.io.IOException
> class MiscTest extends JUnit3Suite {
>   @Test (expected = classOf[IOException])
>   def testSendOffset() {
>   }
> }
> {code}
> or
> {code}
> import org.scalatest.junit.JUnitSuite
> import org.junit._
> import java.io.IOException
> class MiscTest extends JUnit3Suite {
>   @Test (expected = classOf[IOException])
>   def testSendOffset() {
>   }
> }
> {code}
> or
> {code}
> import org.junit._
> import java.io.IOException
> class MiscTest {
>   @Test (expected = classOf[IOException])
>   def testSendOffset() {
>   }
> }
> {code}
> will fail.
> I would propose to not rely on Junit annotations other than @Test itself but 
> use scala unit test annotations instead, for example:
> {code}
> import org.junit._
> import java.io.IOException
> class MiscTest {
>   @Test
>   def testSendOffset() {
> intercept[IOException] {
>   //nothing
> }
>   }
> }
> {code}
> will fail with a clearer stacktrace.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2430) Listing of PR commits in commit message should be optional

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2430:
-
   Resolution: Fixed
Fix Version/s: 0.8.3
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 136
[https://github.com/apache/kafka/pull/136]

> Listing of PR commits in commit message should be optional
> --
>
> Key: KAFKA-2430
> URL: https://issues.apache.org/jira/browse/KAFKA-2430
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.8.3
>
>
> Listing of PR commits is useful for curated branches, but the PRs for the 
> Kafka project are often for organic branches and some of them has a large 
> number of commits that are basically noise. Listing is also not useful if 
> there is a single commit in the PR.
> This change in the PR script will not list the commit if there is a single 
> one and let the merger decide whether listing the commits is useful or not 
> for other cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2430) Listing of PR commits in commit message should be optional

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695623#comment-14695623
 ] 

ASF GitHub Bot commented on KAFKA-2430:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/136


> Listing of PR commits in commit message should be optional
> --
>
> Key: KAFKA-2430
> URL: https://issues.apache.org/jira/browse/KAFKA-2430
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.8.3
>
>
> Listing of PR commits is useful for curated branches, but the PRs for the 
> Kafka project are often for organic branches and some of them has a large 
> number of commits that are basically noise. Listing is also not useful if 
> there is a single commit in the PR.
> This change in the PR script will not list the commit if there is a single 
> one and let the merger decide whether listing the commits is useful or not 
> for other cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2430; Listing of PR commits in commit me...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/136


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2388) subscribe(topic)/unsubscribe(topic) should either take a callback to allow user to handle exceptions or it should be synchronous.

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695608#comment-14695608
 ] 

Jason Gustafson commented on KAFKA-2388:


[~onurkaraman] Have you had a chance to get started on this? If you don't have 
time, I can help out.

> subscribe(topic)/unsubscribe(topic) should either take a callback to allow 
> user to handle exceptions or it should be synchronous.
> -
>
> Key: KAFKA-2388
> URL: https://issues.apache.org/jira/browse/KAFKA-2388
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>Assignee: Onur Karaman
>
> According to the mailing list discussion on the consumer interface, we'll 
> replace:
> {code}
> public void subscribe(String... topics);
> public void subscribe(TopicPartition... partitions);
> public Set subscriptions();
> {code}
> with:
> {code}
> void subscribe(List topics, RebalanceCallback callback);
> void assign(List partitions);
> List subscriptions();
> List assignments();
> {code}
> We don't need the unsubscribe APIs anymore.
> The RebalanceCallback would look like:
> {code}
> interface RebalanceCallback {
>   void onAssignment(List partitions);
>   void onRevocation(List partitions);
>   // handle non-existing topics, etc.
>   void onError(Exception e);
> }
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2432) Controlled shutdown does not proceed successfully while shutting down the broker. Follow up from KAFKA-2351

[Mayuresh Gharat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayuresh Gharat updated KAFKA-2432:
---
Summary: Controlled shutdown does not proceed successfully while shutting 
down the broker. Follow up from KAFKA-2351  (was: Controlled shutdown does not 
proceed successfully while shutting down the broker.)

> Controlled shutdown does not proceed successfully while shutting down the 
> broker. Follow up from KAFKA-2351
> ---
>
> Key: KAFKA-2432
> URL: https://issues.apache.org/jira/browse/KAFKA-2432
> Project: Kafka
>  Issue Type: Bug
>Reporter: Mayuresh Gharat
>Assignee: Mayuresh Gharat
>
> {code}
> [2015-08-05 00:19:09,998] INFO [Offset Manager on Broker 0]: Removed 0 
> expired offsets in 0 milliseconds. (kafka.server.OffsetManager)
> ^C[2015-08-05 00:23:09,144] INFO [Kafka Server 0], shutting down 
> (kafka.server.KafkaServer)
> [2015-08-05 00:23:09,146] INFO [Kafka Server 0], Starting controlled shutdown 
> (kafka.server.KafkaServer)
> [2015-08-05 00:23:09,155] ERROR [KafkaApi-0] error when handling request 
> Name: ControlledShutdownRequest; Version: 0; CorrelationId: 0; BrokerId: 0 
> (kafka.server.KafkaApis)
> kafka.common.ControllerMovedException: Controller moved to another broker. 
> Aborting controlled shutdown
>   at 
> kafka.controller.KafkaController.shutdownBroker(KafkaController.scala:231)
>   at 
> kafka.server.KafkaApis.handleControlledShutdownRequest(KafkaApis.scala:146)
>   at kafka.server.KafkaApis.handle(KafkaApis.scala:63)
>   at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
>   at java.lang.Thread.run(Thread.java:745)
> [2015-08-05 00:23:09,156] INFO [Kafka Server 0], Remaining partitions to 
> move:  (kafka.server.KafkaServer)
> [2015-08-05 00:23:09,156] INFO [Kafka Server 0], Error code from controller: 
> -1 (kafka.server.KafkaServer)
> [2015-08-05 00:23:14,160] WARN [Kafka Server 0], Retrying controlled shutdown 
> after the previous attempt failed... (kafka.server.KafkaServer)
> [2015-08-05 00:23:14,166] ERROR [KafkaApi-0] error when handling request 
> Name: ControlledShutdownRequest; Version: 0; CorrelationId: 1; BrokerId: 0 
> (kafka.server.KafkaApis)
> kafka.common.ControllerMovedException: Controller moved to another broker. 
> Aborting controlled shutdown
>   at 
> kafka.controller.KafkaController.shutdownBroker(KafkaController.scala:231)
>   at 
> kafka.server.KafkaApis.handleControlledShutdownRequest(KafkaApis.scala:146)
>   at kafka.server.KafkaApis.handle(KafkaApis.scala:63)
>   at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
>   at java.lang.Thread.run(Thread.java:745)
> [2015-08-05 00:23:14,167] INFO [Kafka Server 0], Remaining partitions to 
> move:  (kafka.server.KafkaServer)
> [2015-08-05 00:23:14,167] INFO [Kafka Server 0], Error code from controller: 
> -1 (kafka.server.KafkaServer)
> [2015-08-05 00:23:19,169] WARN [Kafka Server 0], Retrying controlled shutdown 
> after the previous attempt failed... (kafka.server.KafkaServer)
> [2015-08-05 00:23:19,172] ERROR [KafkaApi-0] error when handling request 
> Name: ControlledShutdownRequest; Version: 0; CorrelationId: 2; BrokerId: 0 
> (kafka.server.KafkaApis)
> kafka.common.ControllerMovedException: Controller moved to another broker. 
> Aborting controlled shutdown
>   at 
> kafka.controller.KafkaController.shutdownBroker(KafkaController.scala:231)
>   at 
> kafka.server.KafkaApis.handleControlledShutdownRequest(KafkaApis.scala:146)
>   at kafka.server.KafkaApis.handle(KafkaApis.scala:63)
>   at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
>   at java.lang.Thread.run(Thread.java:745)
> [2015-08-05 00:23:19,173] INFO [Kafka Server 0], Remaining partitions to 
> move:  (kafka.server.KafkaServer)
> [2015-08-05 00:23:19,173] INFO [Kafka Server 0], Error code from controller: 
> -1 (kafka.server.KafkaServer)
> [2015-08-05 00:23:24,176] WARN [Kafka Server 0], Retrying controlled shutdown 
> after the previous attempt failed... (kafka.server.KafkaServer)
> [2015-08-05 00:23:24,177] WARN [Kafka Server 0], Proceeding to do an unclean 
> shutdown as all the controlled shutdown attempts failed 
> (kafka.server.KafkaServer)
> [2015-08-05 00:23:24,180] INFO [Socket Server on Broker 0], Shutting down 
> (kafka.network.SocketServer)
> [2015-08-05 00:23:24,189] INFO [Socket Server on Broker 0], Shutdown 
> completed (kafka.network.SocketServer)
> [2015-08-05 00:23:24,190] INFO [Kafka Request Handler on Broker 0], shutting 
> down (kafka.server.KafkaRequestHandlerPool)
> [2015-08-05 00:23:24,193] INFO [Kafka Request Handler on Broker 0], shut down 
> completely (kafka.server.KafkaRequestHandlerPool)
> [2015-08-05 00:23:24,196] IN

[jira] [Commented] (KAFKA-2189) Snappy compression of message batches less efficient in 0.8.2.1

[Andrew Otto (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695589#comment-14695589
 ] 

Andrew Otto commented on KAFKA-2189:


Hi all,

The Wikimedia Foundation had a serious production issue when we upgraded to 
0.8.2.1 because of this bug.  Snappy compression doesn't work at scale in 
0.8.2.1.  I know 0.8.3 is slated for release soon, maybe you should consider 
doing a 0.8.2.2 release just to get this out there in a stable tag, so that 
others don't run into this issue.


> Snappy compression of message batches less efficient in 0.8.2.1
> ---
>
> Key: KAFKA-2189
> URL: https://issues.apache.org/jira/browse/KAFKA-2189
> Project: Kafka
>  Issue Type: Bug
>  Components: build, compression, log
>Affects Versions: 0.8.2.1
>Reporter: Olson,Andrew
>Assignee: Ismael Juma
>Priority: Blocker
>  Labels: trivial
> Fix For: 0.8.3
>
> Attachments: KAFKA-2189.patch
>
>
> We are using snappy compression and noticed a fairly substantial increase 
> (about 2.25x) in log filesystem space consumption after upgrading a Kafka 
> cluster from 0.8.1.1 to 0.8.2.1. We found that this is caused by messages 
> being seemingly recompressed individually (or possibly with a much smaller 
> buffer or dictionary?) instead of as a batch as sent by producers. We 
> eventually tracked down the change in compression ratio/scope to this [1] 
> commit that updated the snappy version from 1.0.5 to 1.1.1.3. The Kafka 
> client version does not appear to be relevant as we can reproduce this with 
> both the 0.8.1.1 and 0.8.2.1 Producer.
> Here are the log files from our troubleshooting that contain the same set of 
> 1000 messages, for batch sizes of 1, 10, 100, and 1000. f9d9b was the last 
> commit with 0.8.1.1-like behavior prior to f5ab8 introducing the issue.
> {noformat}
> -rw-rw-r-- 1 kafka kafka 404967 May 12 11:45 
> /var/kafka2/f9d9b-batch-1-0/.log
> -rw-rw-r-- 1 kafka kafka 119951 May 12 11:45 
> /var/kafka2/f9d9b-batch-10-0/.log
> -rw-rw-r-- 1 kafka kafka  89645 May 12 11:45 
> /var/kafka2/f9d9b-batch-100-0/.log
> -rw-rw-r-- 1 kafka kafka  88279 May 12 11:45 
> /var/kafka2/f9d9b-batch-1000-0/.log
> -rw-rw-r-- 1 kafka kafka 402837 May 12 11:41 
> /var/kafka2/f5ab8-batch-1-0/.log
> -rw-rw-r-- 1 kafka kafka 382437 May 12 11:41 
> /var/kafka2/f5ab8-batch-10-0/.log
> -rw-rw-r-- 1 kafka kafka 364791 May 12 11:41 
> /var/kafka2/f5ab8-batch-100-0/.log
> -rw-rw-r-- 1 kafka kafka 380693 May 12 11:41 
> /var/kafka2/f5ab8-batch-1000-0/.log
> {noformat}
> [1] 
> https://github.com/apache/kafka/commit/f5ab8e1780cf80f267906e3259ad4f9278c32d28
>  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2432) Controlled shutdown does not proceed successfully while shutting down the broker.

[Mayuresh Gharat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayuresh Gharat updated KAFKA-2432:
---
Description: 
{code}
[2015-08-05 00:19:09,998] INFO [Offset Manager on Broker 0]: Removed 0 expired 
offsets in 0 milliseconds. (kafka.server.OffsetManager)
^C[2015-08-05 00:23:09,144] INFO [Kafka Server 0], shutting down 
(kafka.server.KafkaServer)
[2015-08-05 00:23:09,146] INFO [Kafka Server 0], Starting controlled shutdown 
(kafka.server.KafkaServer)
[2015-08-05 00:23:09,155] ERROR [KafkaApi-0] error when handling request Name: 
ControlledShutdownRequest; Version: 0; CorrelationId: 0; BrokerId: 0 
(kafka.server.KafkaApis)
kafka.common.ControllerMovedException: Controller moved to another broker. 
Aborting controlled shutdown
at 
kafka.controller.KafkaController.shutdownBroker(KafkaController.scala:231)
at 
kafka.server.KafkaApis.handleControlledShutdownRequest(KafkaApis.scala:146)
at kafka.server.KafkaApis.handle(KafkaApis.scala:63)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
[2015-08-05 00:23:09,156] INFO [Kafka Server 0], Remaining partitions to move:  
(kafka.server.KafkaServer)
[2015-08-05 00:23:09,156] INFO [Kafka Server 0], Error code from controller: -1 
(kafka.server.KafkaServer)
[2015-08-05 00:23:14,160] WARN [Kafka Server 0], Retrying controlled shutdown 
after the previous attempt failed... (kafka.server.KafkaServer)
[2015-08-05 00:23:14,166] ERROR [KafkaApi-0] error when handling request Name: 
ControlledShutdownRequest; Version: 0; CorrelationId: 1; BrokerId: 0 
(kafka.server.KafkaApis)
kafka.common.ControllerMovedException: Controller moved to another broker. 
Aborting controlled shutdown
at 
kafka.controller.KafkaController.shutdownBroker(KafkaController.scala:231)
at 
kafka.server.KafkaApis.handleControlledShutdownRequest(KafkaApis.scala:146)
at kafka.server.KafkaApis.handle(KafkaApis.scala:63)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
[2015-08-05 00:23:14,167] INFO [Kafka Server 0], Remaining partitions to move:  
(kafka.server.KafkaServer)
[2015-08-05 00:23:14,167] INFO [Kafka Server 0], Error code from controller: -1 
(kafka.server.KafkaServer)
[2015-08-05 00:23:19,169] WARN [Kafka Server 0], Retrying controlled shutdown 
after the previous attempt failed... (kafka.server.KafkaServer)
[2015-08-05 00:23:19,172] ERROR [KafkaApi-0] error when handling request Name: 
ControlledShutdownRequest; Version: 0; CorrelationId: 2; BrokerId: 0 
(kafka.server.KafkaApis)
kafka.common.ControllerMovedException: Controller moved to another broker. 
Aborting controlled shutdown
at 
kafka.controller.KafkaController.shutdownBroker(KafkaController.scala:231)
at 
kafka.server.KafkaApis.handleControlledShutdownRequest(KafkaApis.scala:146)
at kafka.server.KafkaApis.handle(KafkaApis.scala:63)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
[2015-08-05 00:23:19,173] INFO [Kafka Server 0], Remaining partitions to move:  
(kafka.server.KafkaServer)
[2015-08-05 00:23:19,173] INFO [Kafka Server 0], Error code from controller: -1 
(kafka.server.KafkaServer)
[2015-08-05 00:23:24,176] WARN [Kafka Server 0], Retrying controlled shutdown 
after the previous attempt failed... (kafka.server.KafkaServer)
[2015-08-05 00:23:24,177] WARN [Kafka Server 0], Proceeding to do an unclean 
shutdown as all the controlled shutdown attempts failed 
(kafka.server.KafkaServer)
[2015-08-05 00:23:24,180] INFO [Socket Server on Broker 0], Shutting down 
(kafka.network.SocketServer)
[2015-08-05 00:23:24,189] INFO [Socket Server on Broker 0], Shutdown completed 
(kafka.network.SocketServer)
[2015-08-05 00:23:24,190] INFO [Kafka Request Handler on Broker 0], shutting 
down (kafka.server.KafkaRequestHandlerPool)
[2015-08-05 00:23:24,193] INFO [Kafka Request Handler on Broker 0], shut down 
completely (kafka.server.KafkaRequestHandlerPool)
[2015-08-05 00:23:24,196] INFO [Replica Manager on Broker 0]: Shutting down 
(kafka.server.ReplicaManager)
[2015-08-05 00:23:24,196] INFO [ReplicaFetcherManager on broker 0] shutting 
down (kafka.server.ReplicaFetcherManager)
[2015-08-05 00:23:24,197] INFO [ReplicaFetcherManager on broker 0] shutdown 
completed (kafka.server.ReplicaFetcherManager)
[2015-08-05 00:23:24,197] INFO [ExpirationReaper-0], Shutting down 
(kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2015-08-05 00:23:24,310] INFO [ExpirationReaper-0], Stopped  
(kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2015-08-05 00:23:24,310] INFO [ExpirationReaper-0], Shutdown completed 
(kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2015-08-05 00:23:24,310] INFO [ExpirationReaper-0], Shutting

[jira] [Created] (KAFKA-2432) Controlled shutdown does not proceed successfully while shutting down the broker.

[Mayuresh Gharat (JIRA)]

Mayuresh Gharat created KAFKA-2432:
--

 Summary: Controlled shutdown does not proceed successfully while 
shutting down the broker.
 Key: KAFKA-2432
 URL: https://issues.apache.org/jira/browse/KAFKA-2432
 Project: Kafka
  Issue Type: Bug
Reporter: Mayuresh Gharat
Assignee: Mayuresh Gharat






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2432) Controlled shutdown does not proceed successfully while shutting down the broker.

[Mayuresh Gharat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayuresh Gharat updated KAFKA-2432:
---
Description: 
[2015-08-05 00:19:09,998] INFO [Offset Manager on Broker 0]: Removed 0 expired 
offsets in 0 milliseconds. (kafka.server.OffsetManager)
^C[2015-08-05 00:23:09,144] INFO [Kafka Server 0], shutting down 
(kafka.server.KafkaServer)
[2015-08-05 00:23:09,146] INFO [Kafka Server 0], Starting controlled shutdown 
(kafka.server.KafkaServer)
[2015-08-05 00:23:09,155] ERROR [KafkaApi-0] error when handling request Name: 
ControlledShutdownRequest; Version: 0; CorrelationId: 0; BrokerId: 0 
(kafka.server.KafkaApis)
kafka.common.ControllerMovedException: Controller moved to another broker. 
Aborting controlled shutdown
at 
kafka.controller.KafkaController.shutdownBroker(KafkaController.scala:231)
at 
kafka.server.KafkaApis.handleControlledShutdownRequest(KafkaApis.scala:146)
at kafka.server.KafkaApis.handle(KafkaApis.scala:63)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
[2015-08-05 00:23:09,156] INFO [Kafka Server 0], Remaining partitions to move:  
(kafka.server.KafkaServer)
[2015-08-05 00:23:09,156] INFO [Kafka Server 0], Error code from controller: -1 
(kafka.server.KafkaServer)
[2015-08-05 00:23:14,160] WARN [Kafka Server 0], Retrying controlled shutdown 
after the previous attempt failed... (kafka.server.KafkaServer)
[2015-08-05 00:23:14,166] ERROR [KafkaApi-0] error when handling request Name: 
ControlledShutdownRequest; Version: 0; CorrelationId: 1; BrokerId: 0 
(kafka.server.KafkaApis)
kafka.common.ControllerMovedException: Controller moved to another broker. 
Aborting controlled shutdown
at 
kafka.controller.KafkaController.shutdownBroker(KafkaController.scala:231)
at 
kafka.server.KafkaApis.handleControlledShutdownRequest(KafkaApis.scala:146)
at kafka.server.KafkaApis.handle(KafkaApis.scala:63)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
[2015-08-05 00:23:14,167] INFO [Kafka Server 0], Remaining partitions to move:  
(kafka.server.KafkaServer)
[2015-08-05 00:23:14,167] INFO [Kafka Server 0], Error code from controller: -1 
(kafka.server.KafkaServer)
[2015-08-05 00:23:19,169] WARN [Kafka Server 0], Retrying controlled shutdown 
after the previous attempt failed... (kafka.server.KafkaServer)
[2015-08-05 00:23:19,172] ERROR [KafkaApi-0] error when handling request Name: 
ControlledShutdownRequest; Version: 0; CorrelationId: 2; BrokerId: 0 
(kafka.server.KafkaApis)
kafka.common.ControllerMovedException: Controller moved to another broker. 
Aborting controlled shutdown
at 
kafka.controller.KafkaController.shutdownBroker(KafkaController.scala:231)
at 
kafka.server.KafkaApis.handleControlledShutdownRequest(KafkaApis.scala:146)
at kafka.server.KafkaApis.handle(KafkaApis.scala:63)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
[2015-08-05 00:23:19,173] INFO [Kafka Server 0], Remaining partitions to move:  
(kafka.server.KafkaServer)
[2015-08-05 00:23:19,173] INFO [Kafka Server 0], Error code from controller: -1 
(kafka.server.KafkaServer)
[2015-08-05 00:23:24,176] WARN [Kafka Server 0], Retrying controlled shutdown 
after the previous attempt failed... (kafka.server.KafkaServer)
[2015-08-05 00:23:24,177] WARN [Kafka Server 0], Proceeding to do an unclean 
shutdown as all the controlled shutdown attempts failed 
(kafka.server.KafkaServer)
[2015-08-05 00:23:24,180] INFO [Socket Server on Broker 0], Shutting down 
(kafka.network.SocketServer)
[2015-08-05 00:23:24,189] INFO [Socket Server on Broker 0], Shutdown completed 
(kafka.network.SocketServer)
[2015-08-05 00:23:24,190] INFO [Kafka Request Handler on Broker 0], shutting 
down (kafka.server.KafkaRequestHandlerPool)
[2015-08-05 00:23:24,193] INFO [Kafka Request Handler on Broker 0], shut down 
completely (kafka.server.KafkaRequestHandlerPool)
[2015-08-05 00:23:24,196] INFO [Replica Manager on Broker 0]: Shutting down 
(kafka.server.ReplicaManager)
[2015-08-05 00:23:24,196] INFO [ReplicaFetcherManager on broker 0] shutting 
down (kafka.server.ReplicaFetcherManager)
[2015-08-05 00:23:24,197] INFO [ReplicaFetcherManager on broker 0] shutdown 
completed (kafka.server.ReplicaFetcherManager)
[2015-08-05 00:23:24,197] INFO [ExpirationReaper-0], Shutting down 
(kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2015-08-05 00:23:24,310] INFO [ExpirationReaper-0], Stopped  
(kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2015-08-05 00:23:24,310] INFO [ExpirationReaper-0], Shutdown completed 
(kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2015-08-05 00:23:24,310] INFO [ExpirationReaper-0], Shutting down 

[jira] [Updated] (KAFKA-2432) Controlled shutdown does not proceed successfully while shutting down the broker.

[Mayuresh Gharat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayuresh Gharat updated KAFKA-2432:
---
Description: 
{format}
[2015-08-05 00:19:09,998] INFO [Offset Manager on Broker 0]: Removed 0 expired 
offsets in 0 milliseconds. (kafka.server.OffsetManager)
^C[2015-08-05 00:23:09,144] INFO [Kafka Server 0], shutting down 
(kafka.server.KafkaServer)
[2015-08-05 00:23:09,146] INFO [Kafka Server 0], Starting controlled shutdown 
(kafka.server.KafkaServer)
[2015-08-05 00:23:09,155] ERROR [KafkaApi-0] error when handling request Name: 
ControlledShutdownRequest; Version: 0; CorrelationId: 0; BrokerId: 0 
(kafka.server.KafkaApis)
kafka.common.ControllerMovedException: Controller moved to another broker. 
Aborting controlled shutdown
at 
kafka.controller.KafkaController.shutdownBroker(KafkaController.scala:231)
at 
kafka.server.KafkaApis.handleControlledShutdownRequest(KafkaApis.scala:146)
at kafka.server.KafkaApis.handle(KafkaApis.scala:63)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
[2015-08-05 00:23:09,156] INFO [Kafka Server 0], Remaining partitions to move:  
(kafka.server.KafkaServer)
[2015-08-05 00:23:09,156] INFO [Kafka Server 0], Error code from controller: -1 
(kafka.server.KafkaServer)
[2015-08-05 00:23:14,160] WARN [Kafka Server 0], Retrying controlled shutdown 
after the previous attempt failed... (kafka.server.KafkaServer)
[2015-08-05 00:23:14,166] ERROR [KafkaApi-0] error when handling request Name: 
ControlledShutdownRequest; Version: 0; CorrelationId: 1; BrokerId: 0 
(kafka.server.KafkaApis)
kafka.common.ControllerMovedException: Controller moved to another broker. 
Aborting controlled shutdown
at 
kafka.controller.KafkaController.shutdownBroker(KafkaController.scala:231)
at 
kafka.server.KafkaApis.handleControlledShutdownRequest(KafkaApis.scala:146)
at kafka.server.KafkaApis.handle(KafkaApis.scala:63)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
[2015-08-05 00:23:14,167] INFO [Kafka Server 0], Remaining partitions to move:  
(kafka.server.KafkaServer)
[2015-08-05 00:23:14,167] INFO [Kafka Server 0], Error code from controller: -1 
(kafka.server.KafkaServer)
[2015-08-05 00:23:19,169] WARN [Kafka Server 0], Retrying controlled shutdown 
after the previous attempt failed... (kafka.server.KafkaServer)
[2015-08-05 00:23:19,172] ERROR [KafkaApi-0] error when handling request Name: 
ControlledShutdownRequest; Version: 0; CorrelationId: 2; BrokerId: 0 
(kafka.server.KafkaApis)
kafka.common.ControllerMovedException: Controller moved to another broker. 
Aborting controlled shutdown
at 
kafka.controller.KafkaController.shutdownBroker(KafkaController.scala:231)
at 
kafka.server.KafkaApis.handleControlledShutdownRequest(KafkaApis.scala:146)
at kafka.server.KafkaApis.handle(KafkaApis.scala:63)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
[2015-08-05 00:23:19,173] INFO [Kafka Server 0], Remaining partitions to move:  
(kafka.server.KafkaServer)
[2015-08-05 00:23:19,173] INFO [Kafka Server 0], Error code from controller: -1 
(kafka.server.KafkaServer)
[2015-08-05 00:23:24,176] WARN [Kafka Server 0], Retrying controlled shutdown 
after the previous attempt failed... (kafka.server.KafkaServer)
[2015-08-05 00:23:24,177] WARN [Kafka Server 0], Proceeding to do an unclean 
shutdown as all the controlled shutdown attempts failed 
(kafka.server.KafkaServer)
[2015-08-05 00:23:24,180] INFO [Socket Server on Broker 0], Shutting down 
(kafka.network.SocketServer)
[2015-08-05 00:23:24,189] INFO [Socket Server on Broker 0], Shutdown completed 
(kafka.network.SocketServer)
[2015-08-05 00:23:24,190] INFO [Kafka Request Handler on Broker 0], shutting 
down (kafka.server.KafkaRequestHandlerPool)
[2015-08-05 00:23:24,193] INFO [Kafka Request Handler on Broker 0], shut down 
completely (kafka.server.KafkaRequestHandlerPool)
[2015-08-05 00:23:24,196] INFO [Replica Manager on Broker 0]: Shutting down 
(kafka.server.ReplicaManager)
[2015-08-05 00:23:24,196] INFO [ReplicaFetcherManager on broker 0] shutting 
down (kafka.server.ReplicaFetcherManager)
[2015-08-05 00:23:24,197] INFO [ReplicaFetcherManager on broker 0] shutdown 
completed (kafka.server.ReplicaFetcherManager)
[2015-08-05 00:23:24,197] INFO [ExpirationReaper-0], Shutting down 
(kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2015-08-05 00:23:24,310] INFO [ExpirationReaper-0], Stopped  
(kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2015-08-05 00:23:24,310] INFO [ExpirationReaper-0], Shutdown completed 
(kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2015-08-05 00:23:24,310] INFO [ExpirationReaper-0], Shutti

[jira] [Comment Edited] (KAFKA-2411) remove usage of BlockingChannel in the broker

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695575#comment-14695575
 ] 

Ismael Juma edited comment on KAFKA-2411 at 8/13/15 5:09 PM:
-

I pushed an additional commit that updates `ControllerChannelManager` to use 
`Selector` and I've replaced the PR to master with one that targets the SSL/TLS 
branch in my fork (so that it only shows the relevant commits):

https://github.com/ijuma/kafka/pull/5

I made the simplest possible change for now and the tests pass.

I've started experimenting with the changes to `AbstractFetcherThread`, but 
still interested in feedback on the two commits in the PR.


was (Author: ijuma):
I pushed an additional commit that updates `ControllerChannelManager` to use 
`Selector` and I've replaced the PR to master with one that targets the SSL/TLS 
branch in my fork (so that it only shows the relevant commits):

https://github.com/ijuma/kafka/pull/5

I made the simplest possible change for now.

I've started experimenting with the changes to `AbstractFetcherThread`, but 
still interested in feedback on the two commits in the PR.

> remove usage of BlockingChannel in the broker
> -
>
> Key: KAFKA-2411
> URL: https://issues.apache.org/jira/browse/KAFKA-2411
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Jun Rao
>Assignee: Ismael Juma
> Fix For: 0.8.3
>
>
> In KAFKA-1690, we are adding the SSL support at Selector. However, there are 
> still a few places where we use BlockingChannel for inter-broker 
> communication. We need to replace those usage with Selector/NetworkClient to 
> enable inter-broker communication over SSL. Specially, BlockingChannel is 
> currently used in the following places.
> 1. ControllerChannelManager: for the controller to propagate metadata to the 
> brokers.
> 2. KafkaServer: for the broker to send controlled shutdown request to the 
> controller.
> 3. AbstractFetcherThread: for the follower to fetch data from the leader 
> (through SimpleConsumer).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2389) CommitType seems not necessary in commit().

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-2389:

Status: Patch Available  (was: Open)

> CommitType seems not necessary in commit().
> ---
>
> Key: KAFKA-2389
> URL: https://issues.apache.org/jira/browse/KAFKA-2389
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>
> The CommitType does not seem to be necessary in for commit(), it can be 
> inferred from whether user passed in a callback or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2411) remove usage of BlockingChannel in the broker

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695575#comment-14695575
 ] 

Ismael Juma commented on KAFKA-2411:


I pushed an additional commit that updates `ControllerChannelManager` to use 
`Selector` and I've replaced the PR to master with one that targets the SSL/TLS 
branch in my fork (so that it only shows the relevant commits):

https://github.com/ijuma/kafka/pull/5

I made the simplest possible change for now.

I've started experimenting with the changes to `AbstractFetcherThread`, but 
still interested in feedback on the two commits in the PR.

> remove usage of BlockingChannel in the broker
> -
>
> Key: KAFKA-2411
> URL: https://issues.apache.org/jira/browse/KAFKA-2411
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Jun Rao
>Assignee: Ismael Juma
> Fix For: 0.8.3
>
>
> In KAFKA-1690, we are adding the SSL support at Selector. However, there are 
> still a few places where we use BlockingChannel for inter-broker 
> communication. We need to replace those usage with Selector/NetworkClient to 
> enable inter-broker communication over SSL. Specially, BlockingChannel is 
> currently used in the following places.
> 1. ControllerChannelManager: for the controller to propagate metadata to the 
> brokers.
> 2. KafkaServer: for the broker to send controlled shutdown request to the 
> controller.
> 3. AbstractFetcherThread: for the follower to fetch data from the leader 
> (through SimpleConsumer).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-2274) Add integration test for consumer coordinator

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson reassigned KAFKA-2274:
--

Assignee: Jason Gustafson

> Add integration test for consumer coordinator
> -
>
> Key: KAFKA-2274
> URL: https://issues.apache.org/jira/browse/KAFKA-2274
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Guozhang Wang
>Assignee: Jason Gustafson
>Priority: Critical
> Fix For: 0.8.3
>
>
> As discussed with Onur offline, here are some things we could test / simulate:
> - consumer kill -9 (tested in ConsumerTest)
> - broker kill -9 (tested in ConsumerTest)
> - consumer very long GC
> - broker very long GC
> - consumer network cable unplugged
> - broker network cable unplugged
> - consumer power cord unplugged
> - broker power cord unplugged
> Quoting Onur: " Another motivating factor is to verify if 
> response.wasDisconnected is good enough or if we actually need consumers to 
> detect coordinator failures with timeouts.
> GC’s can be simulated with SIGSTOP and SIGCONT. I think we might be able to 
> simulate network cable being unplugged with "ifconfig eth0 down”, but I’m not 
> sure."



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2411) remove usage of BlockingChannel in the broker

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695565#comment-14695565
 ] 

ASF GitHub Bot commented on KAFKA-2411:
---

Github user ijuma closed the pull request at:

https://github.com/apache/kafka/pull/127


> remove usage of BlockingChannel in the broker
> -
>
> Key: KAFKA-2411
> URL: https://issues.apache.org/jira/browse/KAFKA-2411
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Jun Rao
>Assignee: Ismael Juma
> Fix For: 0.8.3
>
>
> In KAFKA-1690, we are adding the SSL support at Selector. However, there are 
> still a few places where we use BlockingChannel for inter-broker 
> communication. We need to replace those usage with Selector/NetworkClient to 
> enable inter-broker communication over SSL. Specially, BlockingChannel is 
> currently used in the following places.
> 1. ControllerChannelManager: for the controller to propagate metadata to the 
> brokers.
> 2. KafkaServer: for the broker to send controlled shutdown request to the 
> controller.
> 3. AbstractFetcherThread: for the follower to fetch data from the leader 
> (through SimpleConsumer).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2411; [WIP] remove usage of blocking cha...

[ijuma]

Github user ijuma closed the pull request at:

https://github.com/apache/kafka/pull/127


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: NEW: API Stability annotations!

[Ismael Juma]

On Thu, Aug 13, 2015 at 8:36 AM, Ewen Cheslack-Postava 
wrote:

> On deprecation, I think we should definitely use the standard annotation to
> handle this.


Thanks Ewen. I agree that we should use @Deprecated for language-level
deprecations. My question was regarding the following:

- unstable - can change at any time
> - evolving - can break compatibility at minor releases (i.e. 0.9 may be
> incompatible with 0.8)
> - stable - will only break compatibility at major releases (1.0, 2.0, etc)
>


Let's pick `evolving`, as an example. Does it mean that we can break
compatibility at minor releases without a deprecation cycle? Or does it
mean that we would need to deprecate it for one minor release and then
remove in the next? The latter is nicer for users as they have one release
where both the deprecated and recommended still work, but it adds more
overhead to Kafka development. What is the current thinking around this?

The annotation doesn't necessarily work for everything (e.g.,
> deprecating older versions of the protocol doesn't work via Java
> deprecation annotations), but we should definitely use it for deprecated
> Java/Scala APIs. Perhaps we should start by marking all the old producer
> APIs deprecated


Seems sensible assuming that users have found it easy to migrate to the new
producer in the latest release.

and, after the next release, the old consumer APIs too? :)


Similar to above, it would be good to do that once we have some reports
from users who moved from one API to the next.

Best,
Ismael

[jira] [Comment Edited] (KAFKA-1782) Junit3 Misusage

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694602#comment-14694602
 ] 

Guozhang Wang edited comment on KAFKA-1782 at 8/13/15 4:35 PM:
---

[~apakulov] Thanks for the patch, [~ewencp] helped fixing another issue and it 
has not been committed to trunk.

EDIT: not => now


was (Author: guozhang):
[~apakulov] Thanks for the patch, [~ewencp] helped fixing another issue and it 
has not been committed to trunk.

> Junit3 Misusage
> ---
>
> Key: KAFKA-1782
> URL: https://issues.apache.org/jira/browse/KAFKA-1782
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Alexander Pakulov
>  Labels: newbie
> Fix For: 0.8.3
>
> Attachments: KAFKA-1782.patch, KAFKA-1782.patch, 
> KAFKA-1782_2015-06-18_11:52:49.patch, KAFKA-1782_2015-07-15_16:57:44.patch, 
> KAFKA-1782_2015-07-16_11:50:05.patch, KAFKA-1782_2015-07-16_11:56:11.patch
>
>
> This is found while I was working on KAFKA-1580: in many of our cases where 
> we explicitly extend from junit3suite (e.g. ProducerFailureHandlingTest), we 
> are actually misusing a bunch of features that only exist in Junit4, such as 
> (expected=classOf). For example, the following code
> {code}
> import org.scalatest.junit.JUnit3Suite
> import org.junit.Test
> import java.io.IOException
> class MiscTest extends JUnit3Suite {
>   @Test (expected = classOf[IOException])
>   def testSendOffset() {
>   }
> }
> {code}
> will actually pass even though IOException was not thrown since this 
> annotation is not supported in Junit3. Whereas
> {code}
> import org.junit._
> import java.io.IOException
> class MiscTest extends JUnit3Suite {
>   @Test (expected = classOf[IOException])
>   def testSendOffset() {
>   }
> }
> {code}
> or
> {code}
> import org.scalatest.junit.JUnitSuite
> import org.junit._
> import java.io.IOException
> class MiscTest extends JUnit3Suite {
>   @Test (expected = classOf[IOException])
>   def testSendOffset() {
>   }
> }
> {code}
> or
> {code}
> import org.junit._
> import java.io.IOException
> class MiscTest {
>   @Test (expected = classOf[IOException])
>   def testSendOffset() {
>   }
> }
> {code}
> will fail.
> I would propose to not rely on Junit annotations other than @Test itself but 
> use scala unit test annotations instead, for example:
> {code}
> import org.junit._
> import java.io.IOException
> class MiscTest {
>   @Test
>   def testSendOffset() {
> intercept[IOException] {
>   //nothing
> }
>   }
> }
> {code}
> will fail with a clearer stacktrace.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2214) kafka-reassign-partitions.sh --verify should return non-zero exit codes when reassignment is not completed yet

[Manikumar Reddy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695421#comment-14695421
 ] 

Manikumar Reddy commented on KAFKA-2214:


[~gwenshap] pinging for review

> kafka-reassign-partitions.sh --verify should return non-zero exit codes when 
> reassignment is not completed yet
> --
>
> Key: KAFKA-2214
> URL: https://issues.apache.org/jira/browse/KAFKA-2214
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>Affects Versions: 0.8.1.1, 0.8.2.0
>Reporter: Michael Noll
>Assignee: Manikumar Reddy
>Priority: Minor
> Attachments: KAFKA-2214.patch, KAFKA-2214_2015-07-10_21:56:04.patch, 
> KAFKA-2214_2015-07-13_21:10:58.patch, KAFKA-2214_2015-07-14_15:31:12.patch, 
> KAFKA-2214_2015-07-14_15:40:49.patch, KAFKA-2214_2015-08-05_20:47:17.patch
>
>
> h4. Background
> The admin script {{kafka-reassign-partitions.sh}} should integrate better 
> with automation tools such as Ansible, which rely on scripts adhering to Unix 
> best practices such as appropriate exit codes on success/failure.
> h4. Current behavior (incorrect)
> When reassignments are still in progress {{kafka-reassign-partitions.sh}} 
> prints {{ERROR}} messages but returns an exit code of zero, which indicates 
> success.  This behavior makes it a bit cumbersome to integrate the script 
> into automation tools such as Ansible.
> {code}
> $ kafka-reassign-partitions.sh --zookeeper zookeeper1:2181 
> --reassignment-json-file partitions-to-move.json --verify
> Status of partition reassignment:
> ERROR: Assigned replicas (316,324,311) don't match the list of replicas for 
> reassignment (316,324) for partition [mytopic,2]
> Reassignment of partition [mytopic,0] completed successfully
> Reassignment of partition [myothertopic,1] completed successfully
> Reassignment of partition [myothertopic,3] completed successfully
> ...
> $ echo $?
> 0
> # But preferably the exit code in the presence of ERRORs should be, say, 1.
> {code}
> h3. How to improve
> I'd suggest that, using the above as the running example, if there are any 
> {{ERROR}} entries in the output (i.e. if there are any assignments remaining 
> that don't match the desired assignments), then the 
> {{kafka-reassign-partitions.sh}}  should return a non-zero exit code.
> h3. Notes
> In Kafka 0.8.2 the output is a bit different: The ERROR messages are now 
> phrased differently.
> Before:
> {code}
> ERROR: Assigned replicas (316,324,311) don't match the list of replicas for 
> reassignment (316,324) for partition [mytopic,2]
> {code}
> Now:
> {code}
> Reassignment of partition [mytopic,2] is still in progress
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: NEW: API Stability annotations!

[Ewen Cheslack-Postava]

We could potentially add more annotations to cover things like
public/private APIs, deprecation, etc. I lifted this initial implementation
from Hadoop (thanks for the pointer Gwen!). Hadoop also has a
public/private annotation that we could lift, and they use this to filter
javadocs. These are useful signals, but we also don't want to make adding
new code too heavyweight, so I'd prefer keeping these annotations minimal.

The immediate benefit of adding stable/unstable annotations was to make it
clear a) that people should stop trying to use our in-progress APIs and b)
so it's clear when we're committing in-progress code that may not be as
good quality as we would normally require but is acceptable for new,
in-development code that we plan to quickly iterate and improve upon before
release. In the latter case, it's a useful signal internally for Kafka
development that an API isn't yet fully baked even for other Kafka devs.

On deprecation, I think we should definitely use the standard annotation to
handle this. The annotation doesn't necessarily work for everything (e.g.,
deprecating older versions of the protocol doesn't work via Java
deprecation annotations), but we should definitely use it for deprecated
Java/Scala APIs. Perhaps we should start by marking all the old producer
APIs deprecated and, after the next release, the old consumer APIs too? :)

-Ewen

On Wed, Aug 12, 2015 at 3:42 PM, Ismael Juma  wrote:

> Hi Gwen,
>
> Nice to see this. There is no deprecation cycle for breakages?
>
> Ismael
>
> On Wed, Aug 12, 2015 at 11:05 PM, Gwen Shapira  wrote:
>
> > Hi Team Kafka,
> >
> > Ewen just added stability annotations to Apache Kafka (KAFKA-2429).
> >
> > In the same PR, we marked the new Consumer API as "unstable" since we are
> > still actively iterating on them. The goal is to mark them as "evolving"
> > before the next release and after one release to validate them, we will
> > mark them as "stable".
> >
> > When adding new public APIs, we encourage you to think of their stage of
> > development and annotate correctly.
> >
> > The usage is:
> >
> > - unstable - can change at any time
> > - evolving - can break compatibility at minor releases (i.e. 0.9 may be
> > incompatible with 0.8)
> > - stable - will only break compatibility at major releases (1.0, 2.0,
> etc)
> >
> > Enjoy!
> >
> > Gwen
> >
>



-- 
Thanks,
Ewen

Re: NEW: API Stability annotations!

[Edward Ribeiro]

Very cool! Congrats! :)

My only comment is that, as Gwen pointed out in the first email, it is a
set of  API compatibility annotations, but the name is InterfaceStability.

Wouldn't ApiStability be a better name? InterfaceStability looks a bit
restrictive, imho.

https://github.com/apache/kafka/pull/133/files

Cheers,
Edward
Good annotations. I can see a few future usages :)

Jiangjie (Becket) Qin

On Wed, Aug 12, 2015 at 3:05 PM, Gwen Shapira  wrote:

> Hi Team Kafka,
>
> Ewen just added stability annotations to Apache Kafka (KAFKA-2429).
>
> In the same PR, we marked the new Consumer API as "unstable" since we are
> still actively iterating on them. The goal is to mark them as "evolving"
> before the next release and after one release to validate them, we will
> mark them as "stable".
>
> When adding new public APIs, we encourage you to think of their stage of
> development and annotate correctly.
>
> The usage is:
>
> - unstable - can change at any time
> - evolving - can break compatibility at minor releases (i.e. 0.9 may be
> incompatible with 0.8)
> - stable - will only break compatibility at major releases (1.0, 2.0, etc)
>
> Enjoy!
>
> Gwen
>

[GitHub] kafka pull request: KAFKA-2430; Listing of PR commits in commit me...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/136

KAFKA-2430; Listing of PR commits in commit message should be optional

If there is a single commit in the PR, then it's never listed.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-2430-optional-listing-commits

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/136.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #136


commit 207ba2e9a3140423fb127a21552c7f7e43abd0c2
Author: Ismael Juma 
Date:   2015-08-13T10:22:03Z

Listing of PR commits in commit message should be optional

If there is a single commit in the PR, then it's never listed.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-2417) Ducktape tests for SSL/TLS

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2417:
---
Description: 
The tests should be complementary to the unit/integration tests written as part 
of KAFKA-1685.

Things to consider:
* Upgrade/downgrade to turning on/off SSL
* Failure testing
* Expired/revoked certificates
* Renegotiation

Some changes to ducktape may be required for upgrade scenarios.

  was:
The tests should be complementary to the unit/integration tests written as part 
of KAFKA-1685.

Things to consider:
* Upgrade/downgrade to turning on/off SSL
* Impact on basic performance
* Failure testing
* Expired/revoked certificates
* Renegotiation

Some changes to ducktape may be required for upgrade scenarios.


> Ducktape tests for SSL/TLS
> --
>
> Key: KAFKA-2417
> URL: https://issues.apache.org/jira/browse/KAFKA-2417
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
> Fix For: 0.8.3
>
>
> The tests should be complementary to the unit/integration tests written as 
> part of KAFKA-1685.
> Things to consider:
> * Upgrade/downgrade to turning on/off SSL
> * Failure testing
> * Expired/revoked certificates
> * Renegotiation
> Some changes to ducktape may be required for upgrade scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2417) Ducktape tests for SSL/TLS

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695087#comment-14695087
 ] 

Ismael Juma commented on KAFKA-2417:


Performance is now being tracked in KAFKA-2431 so removed it from this issue.

> Ducktape tests for SSL/TLS
> --
>
> Key: KAFKA-2417
> URL: https://issues.apache.org/jira/browse/KAFKA-2417
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
> Fix For: 0.8.3
>
>
> The tests should be complementary to the unit/integration tests written as 
> part of KAFKA-1685.
> Things to consider:
> * Upgrade/downgrade to turning on/off SSL
> * Failure testing
> * Expired/revoked certificates
> * Renegotiation
> Some changes to ducktape may be required for upgrade scenarios.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2431) Test SSL/TLS impact on performance

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14695086#comment-14695086
 ] 

Ismael Juma commented on KAFKA-2431:


A sensible way to do this is to allow SSL/TLS to be optionally enabled with our 
existing performance testing tools.

> Test SSL/TLS impact on performance
> --
>
> Key: KAFKA-2431
> URL: https://issues.apache.org/jira/browse/KAFKA-2431
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
> Fix For: 0.8.3
>
>
> Test new Producer and new Consumer performance with and without SSL/TLS once 
> the SSL/TLS branch is integrated.
> The ideal scenario is that SSL/TLS would not have an impact if disabled. When 
> enabled, there will be some overhead (encryption and the inability to use 
> `SendFile`) and it will be good to quantify it. The encryption overhead is 
> reduced if recent JDKs are used with CPUs that support AES-specific 
> instructions (https://en.wikipedia.org/wiki/AES_instruction_set).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2431) Test SSL/TLS impact on performance

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2431:
---
Fix Version/s: 0.8.3

> Test SSL/TLS impact on performance
> --
>
> Key: KAFKA-2431
> URL: https://issues.apache.org/jira/browse/KAFKA-2431
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
> Fix For: 0.8.3
>
>
> Test new Producer and new Consumer performance with and without SSL/TLS once 
> the SSL/TLS branch is integrated.
> The ideal scenario is that SSL/TLS would not have an impact if disabled. When 
> enabled, there will be some overhead (encryption and the inability to use 
> `SendFile`) and it will be good to quantify it. The encryption overhead is 
> reduced if recent JDKs are used with CPUs that support AES-specific 
> instructions (https://en.wikipedia.org/wiki/AES_instruction_set).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2431) Test SSL/TLS impact on performance

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-2431:
--

 Summary: Test SSL/TLS impact on performance
 Key: KAFKA-2431
 URL: https://issues.apache.org/jira/browse/KAFKA-2431
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma


Test new Producer and new Consumer performance with and without SSL/TLS once 
the SSL/TLS branch is integrated.

The ideal scenario is that SSL/TLS would not have an impact if disabled. When 
enabled, there will be some overhead (encryption and the inability to use 
`SendFile`) and it will be good to quantify it. The encryption overhead is 
reduced if recent JDKs are used with CPUs that support AES-specific 
instructions (https://en.wikipedia.org/wiki/AES_instruction_set).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2430) Listing of PR commits in commit message should be optional

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2430?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2430:
---
Description: 
Listing of PR commits is useful for curated branches, but the PRs for the Kafka 
project are often for organic branches and some of them has a large number of 
commits that are basically noise. Listing is also not useful if there is a 
single commit in the PR.

This change in the PR script will not list the commit if there is a single one 
and let the merger decide whether listing the commits is useful or not for 
other cases.

  was:
Listing of PR commits is useful for curated branches, but the PRs for the Kafka 
project are often for organic branches and some of them has a large number of 
commits that are basically noise.

This change in the PR script will let the merger decide whether listing the 
commits is useful or not for a particular PR.


> Listing of PR commits in commit message should be optional
> --
>
> Key: KAFKA-2430
> URL: https://issues.apache.org/jira/browse/KAFKA-2430
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>
> Listing of PR commits is useful for curated branches, but the PRs for the 
> Kafka project are often for organic branches and some of them has a large 
> number of commits that are basically noise. Listing is also not useful if 
> there is a single commit in the PR.
> This change in the PR script will not list the commit if there is a single 
> one and let the merger decide whether listing the commits is useful or not 
> for other cases.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2430) Listing of PR commits in commit message should be optional

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-2430:
--

 Summary: Listing of PR commits in commit message should be optional
 Key: KAFKA-2430
 URL: https://issues.apache.org/jira/browse/KAFKA-2430
 Project: Kafka
  Issue Type: Improvement
Reporter: Ismael Juma
Assignee: Ismael Juma


Listing of PR commits is useful for curated branches, but the PRs for the Kafka 
project are often for organic branches and some of them has a large number of 
commits that are basically noise.

This change in the PR script will let the merger decide whether listing the 
commits is useful or not for a particular PR.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2425) Migrate website from SVN to Git

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694929#comment-14694929
 ] 

Ismael Juma commented on KAFKA-2425:


[~omkreddy], I think it's OK because what appears under `documentation.html` is 
based on an include to a particular version (see the 07, 08, 081, etc. 
directories). So, we could still show the latest released version by default 
and we would have the opportunity to fix things in the documentation for 
released or unreleased versions. But let's see what [~gwenshap] says.

> Migrate website from SVN to Git 
> 
>
> Key: KAFKA-2425
> URL: https://issues.apache.org/jira/browse/KAFKA-2425
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>
> The preference is to share the same Git repo for the code and website as per 
> discussion in the mailing list:
> http://search-hadoop.com/m/uyzND1Dux842dm7vg2
> Useful reference:
> https://blogs.apache.org/infra/entry/git_based_websites_available



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2398) Transient test failure for SocketServerTest - Socket closed.

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-2398.

Resolution: Duplicate

Ewen confirmed that KAFKA-1782 fixes this by avoiding leaks due to teardown not 
being called.

> Transient test failure for SocketServerTest - Socket closed.
> 
>
> Key: KAFKA-2398
> URL: https://issues.apache.org/jira/browse/KAFKA-2398
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>
> See the following transient test failure for SocketServerTest.
> kafka.network.SocketServerTest > simpleRequest FAILED
> java.net.SocketException: Socket closed
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at 
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at 
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at 
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at java.net.Socket.connect(Socket.java:528)
> at java.net.Socket.(Socket.java:425)
> at java.net.Socket.(Socket.java:208)
> at kafka.network.SocketServerTest.connect(SocketServerTest.scala:84)
> at 
> kafka.network.SocketServerTest.simpleRequest(SocketServerTest.scala:94)
> kafka.network.SocketServerTest > tooBigRequestIsRejected FAILED
> java.net.SocketException: Socket closed
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at 
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at 
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at 
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at java.net.Socket.connect(Socket.java:528)
> at java.net.Socket.(Socket.java:425)
> at java.net.Socket.(Socket.java:208)
> at kafka.network.SocketServerTest.connect(SocketServerTest.scala:84)
> at 
> kafka.network.SocketServerTest.tooBigRequestIsRejected(SocketServerTest.scala:124)
> kafka.network.SocketServerTest > testSocketsCloseOnShutdown FAILED
> java.net.SocketException: Socket closed
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at 
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at 
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at 
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at java.net.Socket.connect(Socket.java:528)
> at java.net.Socket.(Socket.java:425)
> at java.net.Socket.(Socket.java:208)
> at kafka.network.SocketServerTest.connect(SocketServerTest.scala:84)
> at 
> kafka.network.SocketServerTest.testSocketsCloseOnShutdown(SocketServerTest.scala:136)
> kafka.network.SocketServerTest > testMaxConnectionsPerIp FAILED
> java.net.SocketException: Socket closed
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at 
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
> at 
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200)
> at 
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:579)
> at java.net.Socket.connect(Socket.java:528)
> at java.net.Socket.(Socket.java:425)
> at java.net.Socket.(Socket.java:208)
> at kafka.network.SocketServerTest.connect(SocketServerTest.scala:84)
> at 
> kafka.network.SocketServerTest$$anonfun$1.apply(SocketServerTest.scala:170)
> at 
> kafka.network.SocketServerTest$$anonfun$1.apply(SocketServerTest.scala:170)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
> at scala.collection.immutable.Range.foreach(Range.scala:141)
> at 
> scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
> at scala.collection.AbstractTraversable.map(Traversable.scala:105)
> at 
> kafka.network.SocketServerTest.testMaxConnectionsPerIp(SocketServerTest.scala:170)
> kafka.network.SocketServe

[jira] [Commented] (KAFKA-1782) Junit3 Misusage

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14694837#comment-14694837
 ] 

Ismael Juma commented on KAFKA-1782:


Guozhang meant that it has now been committed to trunk.

> Junit3 Misusage
> ---
>
> Key: KAFKA-1782
> URL: https://issues.apache.org/jira/browse/KAFKA-1782
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Alexander Pakulov
>  Labels: newbie
> Fix For: 0.8.3
>
> Attachments: KAFKA-1782.patch, KAFKA-1782.patch, 
> KAFKA-1782_2015-06-18_11:52:49.patch, KAFKA-1782_2015-07-15_16:57:44.patch, 
> KAFKA-1782_2015-07-16_11:50:05.patch, KAFKA-1782_2015-07-16_11:56:11.patch
>
>
> This is found while I was working on KAFKA-1580: in many of our cases where 
> we explicitly extend from junit3suite (e.g. ProducerFailureHandlingTest), we 
> are actually misusing a bunch of features that only exist in Junit4, such as 
> (expected=classOf). For example, the following code
> {code}
> import org.scalatest.junit.JUnit3Suite
> import org.junit.Test
> import java.io.IOException
> class MiscTest extends JUnit3Suite {
>   @Test (expected = classOf[IOException])
>   def testSendOffset() {
>   }
> }
> {code}
> will actually pass even though IOException was not thrown since this 
> annotation is not supported in Junit3. Whereas
> {code}
> import org.junit._
> import java.io.IOException
> class MiscTest extends JUnit3Suite {
>   @Test (expected = classOf[IOException])
>   def testSendOffset() {
>   }
> }
> {code}
> or
> {code}
> import org.scalatest.junit.JUnitSuite
> import org.junit._
> import java.io.IOException
> class MiscTest extends JUnit3Suite {
>   @Test (expected = classOf[IOException])
>   def testSendOffset() {
>   }
> }
> {code}
> or
> {code}
> import org.junit._
> import java.io.IOException
> class MiscTest {
>   @Test (expected = classOf[IOException])
>   def testSendOffset() {
>   }
> }
> {code}
> will fail.
> I would propose to not rely on Junit annotations other than @Test itself but 
> use scala unit test annotations instead, for example:
> {code}
> import org.junit._
> import java.io.IOException
> class MiscTest {
>   @Test
>   def testSendOffset() {
> intercept[IOException] {
>   //nothing
> }
>   }
> }
> {code}
> will fail with a clearer stacktrace.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)