Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Tong Li]

same question here as well on hosts. If I would like to secure a topic,
regardless where the topic is, I would like it to be secured. It is hard to
imagine that one would like a topic to be secured on one host, but not on
the other unless a topic spreads over multiple hosts really meant to be
totally different things. Then it will be really needed.

Thanks.

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Jeff Holoman 
To: dev@kafka.apache.org
Cc: Tom Graves 
Date:   04/22/2015 02:40 PM
Subject:Re: [DISCUSS] KIP-11- Authorization design for kafka security



Parth,

This is a long thread, so trying to keep up here, sorry if this has been
covered before. First, great job on the KIP proposal and work so far.

Are we sure that we want to tie host level access to a given user? My
understanding is that the ACL will be (omitting some fields)

user_a, host1, host2, host3
user_b, host1, host2, host3

So there would potentially be a lot of redundancy in the configs. Does it
make sense to have hosts be at the same level as principal in the
hierarchy? This way you could just blanket the allowed / denied hosts and
only have to worry about the users. So if you follow this, then

we can wildcard the user so we can have a separate list of just host-based
access. What's the order that the perms would be evaluated if a there was
more than one match on a principal ?

Is the thought that there wouldn't usually be much overlap on hosts? I
guess I can imagine a scenario where I want to offline/online access to a
particular hosts or set of hosts and if there was overlap, I'm doing a
bunch of alter commands for just a single host. Maybe this is too contrived
an example?

I agree that having this level of granularity gives flexibility but I
wonder if people will actually use it and not just * the hosts for a given
user and create separate "global" list as i mentioned above?

The only other system I know of that ties users with hosts for access is
MySql and I don't love that model. Companies usually standardize on group
authorization anyway, are we complicating that issue with the inclusion of
hosts attached to users? Additionally I worry about the debt of big JSON
configs in the first place, most non-developers find them non-intuitive
already, so anything to ease this I think would be beneficial.


Thanks

Jeff

On Wed, Apr 22, 2015 at 2:22 PM, Parth Brahmbhatt <
pbrahmbh...@hortonworks.com> wrote:

> Sorry I missed your last questions. I am +0 on adding ―host option for
> ―list, we could add it for symmetry. Again if this is only a CLI change
it
> can be added later if you mean adding this in authorizer interface then
we
> should make a decision now.
>
> Given a choice I would like to actually keep only one option which is
> resource based get (remove even the get based on principal). I see those
> (getAcl for principal or host) as special filtering case which can easily
> be achieved by a third party tool by doing "list all topics" and calling
> getAcls for each topic and applying filtering logic on that.  I really
> don’t see the need to make those first class citizens of the authorizer
> interface given these kind of queries will be issued outside of broker
JVM
> so they will not benefit from the caching and because the storage will be
> indexed on resource both these options even as a first class API will
just
> scan all topic acls and apply filtering logic.
>
> Thanks
> Parth
>
> On 4/22/15, 11:08 AM, "Parth Brahmbhatt" 
> wrote:
>
> >Please see all the available options here
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization
+I
> >nterface#KIP-11-AuthorizationInterface-AclManagement(CLI) . I think it
> >covers both hosts and operations and allows to specify a list for both.
> >
> >Thanks
> >Parth
> >
> >From: Tom Graves mailto:tgraves...@yahoo.com>>
> >Reply-To: Tom Graves mailto:tgraves...@yahoo.com>>
> >Date: Wednesday, April 22, 2015 at 11:02 AM
> >To: Parth Brahmbhatt
> >mailto:pbrahmbh...@hortonworks.com>>,
> >"dev@kafka.apache.org<mailto:dev@kafka.apache.org>"
> >mailto:dev@kafka.apache.org>>
> >Subject: Re: [DISCUSS] KIP-11- Authorization design for kafka security
> >
> >Thanks for the explanations Parth.
> >
> >On the configs questions, the way I see it is its more likely to
> >accidentally give everyone access, especially since you have to run a
> >separate command to change the acls. If there was some config for
> >defaults, a cluster admin could change that to be nobody or certain set
> >of users, then grant others permissions.  This would also remove the
race
> >between commands.  This is something you can

Re: KIP hangout on Apr 21

[Tong Li]

Jun,
 Not sure why in these invitations, I am not seeing the google hangout
link. I am using IBM notes which is quite different from gmail and
calendar. Is there anyway that you send the hangout link in the invitation
body? Thanks.

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Jun Rao 
To: "dev@kafka.apache.org" 
Date:   04/18/2015 12:50 AM
Subject:KIP hangout on Apr 21



Hi,

We will have a KIP hangout at 3pm PST on Apr 21. The following is the
tentative agenda. If you'd like to attend but haven't received an invite,
please let me know.

Agenda:
KIP-4 (admin commands): wrap up any remaining issues
KIP-11 (Authorization):
KIP-12 (SSL/Kerberos): See if there is any blocker.

jira backlog check

Thanks,

Jun

Re: [DISCUSSION] KIP-11: ACL Management

[Tong Li]

Gwen,
 There is one product called ElasticSearch which has been quite
successful. They recently added security, what they actually did is quite
nice. They really separated Authentication and Authorization which many
people get really confused about and often mix them up. I looked through
what they did and quite impressed by it, I think there are many things we
can borrow from. Here is a link to it.
http://www.elastic.co/guide/en/shield/current/architecture.html. The
product name is called "shield" which is implemented as an ElasticSearch
plugin. The promise here is that you can have a running ElasticSearch, then
you install this plugin, configure it, then your ElasticSearch service is
secured. The goal should be really the same for Kafka, you have a Kafka
service running, you install a new plugin (in this case security plugin),
configure it, then your Kafka service is secured. I think that the key here
is that we should introduce a true pluggable framework in Kafka which
allows security, quota, encryption,  compression,
serialization/deserialization all being developed as plugins which can be
all easily added and configured onto a running Kafka service, then the
functions/features provided by the plugins will start working. Once we have
this framework in, how a security plugin works internally becomes the
really the concern of that plugin, for example, how a new user gets
registered, permission granted, revoked, all these will be the concern of
that plugin, rest of the Kafka components should not really be concerned
about them. This way we are really following the design principal
(Separation of concerns).  With all that, what I am proposing is a true
pluggable framework introduction into Kafka which I have also talked about
in a previous email. For security we can implement a simple file based
security plugin, other plugins such as LDAP, AD for authentication can come
later, plugin for authorization such as RBAC can also come later if people
care so much about using them.

Thanks.

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Gwen Shapira 
To: "dev@kafka.apache.org" 
Date:   04/16/2015 12:44 PM
Subject:[DISCUSSION] KIP-11: ACL Management



Hi Kafka Authorization Fans,

I'm starting a new thread on a specific sub-topic of KIP-11, since
this is a bit long :)

Currently KIP-11, as I understand it, proposes:
* Authorizers are pluggable, with Kafka providing DefaultAuthorizer.
* Kafka tools allow adding / managing ACLs.
* Those ACLs are stored in ZK and cached in a new TopicCache
* Authorizers can either use the ACLs defined and stored in Kafka, or
define and use their own.

I am concerned of two possible issues with this design:
1. Separation of concerns - only authorizers should worry about ACLs,
and therefore the less code for ACLs that exist in Kafka core, the
better.
2. User confusion - It sounded like we can define ACLs in Kafka itself
but authorizers can also define their own, so "kafka-topics
--describe" may show an ACL different than the one in use. This can be
super confusing for admins.

My alternative suggestion:
* Authorizer API will include:
 grantPrivilege(List, List)
 revokePrivilege(List, List),
 getPrivilegesByPrincipal(Principal, Resource)
 
 (The exact API can be discussed in detail, but you get the idea)
* Kafka tools will simply invoke these APIs when topics are added /
modified / described.
* Each authorizer (including the default one) will be responsible for
storing, caching and using those ACLs.

This way, we keep almost all ACL code with the Authorizer, where it
belongs and users get a nice unified interface that reflects what is
actually getting used in the system.
This is pretty much how Sqoop and Hive implement their authorization APIs.

What do you think?

Gwen

Re: Will it be possible to apply quotas based on a security principal?

[Tong Li]

If it is all possible I think we can introduce request pipeline
architecture for producer, broker and consumer (does not have to be done
all at once), that way, security, quota, encryption, serialization,
compression all can be done as plugable components. One can freely stack
them up or tear them apart at the configuration level. Each component can
have its own design and implementation. Any deployed can choose what
interest him or her. Components have no dependency among each other. New
features can be easily added/tried/removed/abandoned. When I looked at our
code, instrument this so called pipeline can be very easy. Wonder if any of
you guys interested.

Thanks.

Sent from my iPhone

> On Apr 15, 2015, at 4:14 PM, Jay Kreps  wrote:
>
> I think this should be a fairly minor follow-up item to have the quotas
key
> off of user rather than client id. The advantage of starting with
client.id
> is that it decouples the security work from the quota work in the short
> term and provides a mechanism for those using Kafka without
authentication
> to still enforce quotas.
>
> On Wed, Apr 15, 2015 at 6:15 AM, Adrian Preston 
wrote:
>
> > Hi,
> >
> > I've been investigating using Kafka for a multi-user system that
applies
> > quotas at a per-user level.  Reading through KIP-13 and KAFKA-1682, I
> > wondered: are there any plans to link together the security principal
and
> > client identifier in some way?  Currently it appears these are separate
> > concepts - so I can't see any way to apply a quota based on the
> > authenticated identity of a user.
> >
> >
> > Regards
> > - Adrian
> >
> > Unless stated otherwise above:
> > IBM United Kingdom Limited - Registered in England and Wales with
number
> > 741598.
> > Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
> >
> >

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Tong Li]

Parth,
 If one wants to use his or her own access control including
authentication system, with this design what will be needed to be done? Can
one completely turn this off so that the system behaves exactly same as it
is today?

Thanks.

Tong

Sent from my iPhone

> On Apr 15, 2015, at 1:51 PM, Parth Brahmbhatt
 wrote:
>
> Hi Michael,
>
> There is code in kafka codebase that reads and interprets the topic
config JSON which has acls, owner and logconfig properties. There are 3 use
cases that we are supporting with current proposal:
>
>   *   You use out of box simpleAcl authorizer which is tied to the acl
stored in topic config and the format is locked down.
>   *   You have a custom authorizer and a custom ACL store.  Ranger/Argus
falls under this as they have their own acl store and ui that users use to
configure acls on the cluster and cluster resources  like topic. It is upto
the custom authorizer to leverage the kafka acl configs or completely
ignore them as they have set a user expectation that only acls configured
via their ui/system will be effective.
>   *   You have a custom authorizer but no custom Acl store. You are
completely tied to Acl structure that we have provided in out of box
implementation.
>
> Thanks
> Parth
>
> On 4/15/15, 10:31 AM, "Michael Herstine"
mailto:mherst...@linkedin.com.INVALID>>
wrote:
>
> Hi Parth,
>
> One question that occurred to me at the end of today’s hangout: how tied
> are we to a particular ACL representation under your proposal? I know
that
> TopicConfigCache will just contain JSON— if a particular site decides
they
> want to represent their ACLs differently, and swap out the authorizer
> implementation, will that work?  I guess what I’m asking is whether
> there’s any code in the Kafka codebase that will interpret that JSON, or
> does that logic live exclusively in the authorizer?
>
> On 4/14/15, 10:56 PM, "Don Bosco Durai"
mailto:bo...@apache.org>> wrote:
>
> I also feel, having just IP would be more appropriate. Host lookup will
> unnecessary slow things down and would be insecure as you pointed out.
>
> With IP, it will be also able to setup policies (in future if needed)
with
> ranges or netmasks and it would be more scalable.
>
> Bosco
>
>
> On 4/14/15, 1:40 PM, "Michael Herstine"
mailto:mherst...@linkedin.com.INVALID>>
> wrote:
>
> Hi Parth,
>
> Sorry to chime in so late, but I’ve got a minor question on the KIP.
>
> Several methods take a parameter named “host” of type String. Is that
> intended to be a hostname, or an IP address? If the former, I’m curious
> as
> to how that’s found (in my experience, when accepting an incoming socket
> connection, you only know the IP address, and there isn’t a way to map
> that to a hostname without a round trip to a DNS server, which is
> insecure
> anyway).
>
>
> On 3/25/15, 1:07 PM, "Parth Brahmbhatt"
mailto:pbrahmbh...@hortonworks.com>>
> wrote:
>
> Hi all,
>
> I have modified the KIP to reflect the recent change request from the
> reviewers. I have been working on the code and I have the server side
> code
> for authorization ready. I am now modifying the command line utilities.
> I
> would really appreciate if some of the committers can spend sometime to
> review the KIP so we can make progress on this.
>
> Thanks
> Parth
>
> On 3/18/15, 2:20 PM, "Michael Herstine"
mailto:mherst...@linkedin.com.INVALID>>
> wrote:
>
> Hi Parth,
>
> Thanks! A few questions:
>
> 1. Do you want to permit rules in your ACLs that DENY access as well as
> ALLOW? This can be handy setting up rules that have exceptions. E.g.
> “Allow principal P to READ resource R from all hosts” with “Deny
> principal
> P READ access to resource R from host H1” in combination would allow P
> to
> READ R from all hosts *except* H1.
>
> 2. When a topic is newly created, will there be an ACL created for it?
> If
> not, would that not deny subsequent access to it?
>
> (nit) Maybe use Principal instead of String to represent principals?
>
>
> On 3/9/15, 11:48 AM, "Don Bosco Durai"
mailto:bo...@apache.org>> wrote:
>
> Parth
>
> Overall it is looking good. Couple of questionsŠ
>
> - Can you give an example how the policies will look like in the
> default
> implementation?
> - In the operations, can we support ³CONNECT² also? This can be used
> during Session connection
> - Regarding access control for ³Topic Creation², since we can¹t do it
> on
> the server side, can we de-scope it for? And plan it as a future
> feature
> request?
>
> Thanks
>
> Bosco
>
>
> On 3/6/15, 8:10 AM, "Harsha" mailto:ka...@harsha.io>>
wrote:
>
> Hi Parth,
> Thanks for putting this together. Overall it looks good
> to
> me. Although AdminUtils is a concern KIP-4  can probably
> fix
> that part.
> Thanks,
> Harsha
>
> On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
> Forgot to add links to wiki and jira.
> Link to wiki:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
> t
> i
> o
> n
> +
> Interface
> Link to Jira:

Re: KIP discussion Apr 7 at 11am PST

[Tong Li]

Jun Rao,
  Can you include the hangout link in the invitation? I could not find
it anywhere.

Thanks.

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Jun Rao 
To: "dev@kafka.apache.org" 
Date:   04/03/2015 01:27 PM
Subject:KIP discussion Apr 7 at 11am PST



Hi, Everyone,

We plan to have a KIP discussion on Google hangout on Apr. 7 at 11am PST.
If you are interested in participating and have not already received a
calendar invitation, please let me know. The following is the agenda.

Agenda:
KIP-4 (admin commands):
* wrap up any remaining issues

KIP-13 (quota):
* wrap up any remaining issues

Security related KIPS.
KIP-7 (IP filtering)
KIP-11 (Authorization)
KIP-12 (SSL/Kerberos)

Thanks,

Jun

Re: [KIP-DISCUSSION] KIP-13 Quotas

[Tong Li]

see some response inline below.
Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com

Jay Kreps  wrote on 04/07/2015 10:41:19 AM:

> From: Jay Kreps 
> To: "dev@kafka.apache.org" 
> Date: 04/07/2015 10:44 AM
> Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas
>
> Totally. But is that the only use? What I wanted to flesh out was whether
> the goal was:
> 1. Expose throttling in the client metrics
> 2. Enable programmatic response (i.e. stop sending stuff or something
like
> that)
>
> I think I kind of understand (1) but let's get specific on the metric we
> would be adding and what exactly you would expose  in a dashboard. For
> example if the goal is just monitoring do I really want a boolean flag
for
> is_throttled or do I want to know how much I am being throttled (i.e.
> throttle_pct might indicate the percent of your request time that was due
> to throttling or something like that)? If I am 1% throttled that may be
> irrelevant but 99% throttled would be quite relevant? Not sure I agree,
> just throwing that out there...
>
Jay, great point, I think Kafka should really just sent metrics, how to
judge if
a system is throttled should be someone other people's job. I would think
this comes down to design principles, if we follow the principal of
"separation
of the concerns", then this should not be really part of Kafka.
I have been doing monitoring systems for awhile, the system being monitored
normally just
send the fact of itself, such as CPU usage, network usage, disk usage etc
to the
monitoring system, the monitoring system will run various algorithms to
eventually
decide if a system is throttled by setting up threshold and other measures.
The monitoring
system will also send out notifications/alarms if things turns bad. Just
to make this discussion even easier, a set of general purpose of agents
collecting
these data have been developed and available as part of a monitoring system
named
Monasca. If you are interested, I can provide more information. For Kafka
to have
the features such as judging if the system is throttling seems to be a
moving-away
from its core values. Just my 2 cents of course.


> For (2) the prior discussion seemed to kind of allude to this but I can't
> really come up with a use case. Is there one?
>
> If it is just (1) I think the question is whether it really helps much to
> have the metric on the client vs the server. I suppose this is a bit
> environment specific. If you have a central metrics system it shouldn't
> make any difference, but if you don't I suppose it does.
>
> -Jay
>
> On Mon, Apr 6, 2015 at 7:57 PM, Gwen Shapira 
wrote:
>
> > Here's a wild guess:
> >
> > An app developer included a Kafka Producer in his app, and is not happy
> > with the throughput. He doesn't have visibility into the brokers since
they
> > are owned by a different team. Obviously the first instinct of a
developer
> > who knows that throttling exists is to blame throttling for any
slowdown in
> > the app.
> > If he doesn't have a way to know from the responses whether or not his
app
> > is throttled, he may end up calling Aditya at 4am asked "Hey, is my app
> > throttled?".
> >
> > I assume Aditya is trying to avoid this scenario.
> >
> > On Mon, Apr 6, 2015 at 7:47 PM, Jay Kreps  wrote:
> >
> > > Hey Aditya,
> > >
> > > 2. I kind of buy it, but I really like to understand the details of
the
> > use
> > > case before we make protocol changes. What changes are you proposing
in
> > the
> > > clients for monitoring and how would that be used?
> > >
> > > -Jay
> > >
> > > On Mon, Apr 6, 2015 at 10:36 AM, Aditya Auradkar <
> > > aaurad...@linkedin.com.invalid> wrote:
> > >
> > > > Hi Jay,
> > > >
> > > > 2. At this time, the proposed response format changes are only for
> > > > monitoring/informing clients. As Jun mentioned, we get instance
level
> > > > monitoring in this case since each instance that got throttled will
> > have
> > > a
> > > > metric confirming the same. Without client level monitoring for
this,
> > > it's
> > > > hard for application developers to find if they are being throttled
> > since
> > > > they will also have to be aware of all the brokers in the cluster.
This
> > > is
> > > > quite problematic for large clusters.
> > > >
> > > > It seems nice for app developers to not have to think about kafka
> > > internal
> > > > metrics and only focus on the metrics exposed on their inst

Re: [jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li]

Run Rao,
Thanks for reviewing it again. What commands do you run to show unused
imports? When I run "grandle --daemon test", it did not show these (there
were some warnings, but not unused imports). I thought that would have
shown these things.

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   "Jun Rao (JIRA)" 
To: dev@kafka.apache.org
Date:   04/06/2015 12:48 AM
Subject:[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with
o.a.k.common.utils.Utils




 [
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-1926:
---
   Resolution: Fixed
Fix Version/s: 0.8.3
   Status: Resolved  (was: Patch Available)

Thanks for rebasing the latest patch. +1. Committed to trunk after removing
unused imports in a few files.

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>    Reporter: Jay Kreps
>Assignee: Tong Li
>  Labels: newbie, patch
> Fix For: 0.8.3
>
> Attachments: KAFKA-1926.patch, KAFKA-1926.patch,
KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926_2015-04-01_22:16:46.patch,
KAFKA-1926_2015-04-05_23:45:13.patch
>
>
> There is currently a lot of duplication between the Utils class in common
and the one in core.
> Our plan has been to deprecate duplicate code in the server and replace
it with the new common code.
> As such we should evaluate each method in the scala Utils and do one of
the following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general
purpose utility in active use that is not Kafka-specific. If we migrate it
we should really think about the API and make sure there is some test
coverage. A few things in there are kind of funky and we shouldn't just
blindly copy them over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will
hold any utilities that really need to make use of Scala features to be
convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1926:
---
Attachment: KAFKA-1926_2015-04-05_23:45:13.patch

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>Assignee: Tong Li
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, 
> KAFKA-1926.patch, KAFKA-1926_2015-04-01_22:16:46.patch, 
> KAFKA-1926_2015-04-05_23:45:13.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14480907#comment-14480907
 ] 

Tong Li commented on KAFKA-1926:


[~junrao] Jun, thanks again. Noticed you have done a lot over the weekend. what 
an inspiration! So here I did a bit. Since the patch 1809 is in, I rebased the 
patch set and addressed the couple new comments. Here it is in again for your 
review. Really appreciate all the work you did for the project. 

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>Assignee: Tong Li
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, 
> KAFKA-1926.patch, KAFKA-1926_2015-04-01_22:16:46.patch, 
> KAFKA-1926_2015-04-05_23:45:13.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14480906#comment-14480906
 ] 

Tong Li commented on KAFKA-1926:


Updated reviewboard https://reviews.apache.org/r/32740/
 against branch origin/trunk

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>Assignee: Tong Li
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, 
> KAFKA-1926.patch, KAFKA-1926_2015-04-01_22:16:46.patch, 
> KAFKA-1926_2015-04-05_23:45:13.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 32740: Patch for KAFKA-1926

[Tong Li]

/server/LogRecoveryTest.scala 
92e49df6964230753762aa359191a6ede337d3ac 
  core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala 
a6bb69059b548bea3ebefa2a3146600163bcfa4b 
  core/src/test/scala/unit/kafka/server/ServerGenerateBrokerIdTest.scala 
2bfaeb307094b0fba67a1976b7f87591ca36a45c 
  core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
a20321f89defdb3426e159f6c376d4d2194092c2 
  core/src/test/scala/unit/kafka/server/ServerStartupTest.scala 
661ddd504fc4cdccbd5eb129212911296f4102b0 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
a8ed1422ef11737f61fe0b62629e7244c0c11677 
  core/src/test/scala/unit/kafka/utils/UtilsTest.scala 
8c3797a964a2760a00ed60530d1a48e3da473769 
  core/src/test/scala/unit/kafka/zk/EmbeddedZookeeper.scala 
1d87506649f0d1f73a2ea5465339dc586900b626 
  core/src/test/scala/unit/kafka/zk/ZooKeeperTestHarness.scala 
fedefb5287eaf8e0d1ee585a8dd31687bdcf9d88 

Diff: https://reviews.apache.org/r/32740/diff/


Testing
---


Thanks,

Tong Li

Re: Review request at https://reviews.apache.org/r/32740/

[Tong Li]

Thanks so much. New patch set come up after rebase.

Sent from my iPhone

> On Apr 3, 2015, at 7:50 PM, Jun Rao  wrote:
>
> Reviewed.
>
> Thanks,
>
> Jun
>
> On Fri, Apr 3, 2015 at 10:38 AM, Tong Li  wrote:
>
> > Jun and Jay,
> >  Have addressed all the comments and put up another patch set. Can
you
> > please take a look again? Really appreciate it.
> >
> > Thanks.
> >
> > Tong Li
> > OpenStack & Kafka Community Development
> > Building 501/B205
> > liton...@us.ibm.com

Review request at https://reviews.apache.org/r/32740/

[Tong Li]

Jun and Jay,
 Have addressed all the comments and put up another patch set. Can you
please take a look again? Really appreciate it.

Thanks.

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com

Re: Kafka meetup at ApacheCon in Apr

[Tong Li]

Jun,
 Thanks for the information, I am trying to create a travel request to
be there to meet you guys. But I need some information on who and what
company the participators will represent. I wonder if you can give me few
names and their company name so that I can put in my travel request?

Thanks.

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Jun Rao 
To: "dev@kafka.apache.org" ,
"us...@kafka.apache.org" ,
"kafka-clie...@googlegroups.com"

Date:   04/02/2015 02:20 PM
Subject:Kafka meetup at ApacheCon in Apr



Hi,

A few of us will be at ApacheCon in April. We are organizing a Kafka meetup
Tuesday evening (
http://events.linuxfoundation.org/events/apachecon-north-america/extend-the-experience/meetups
).
Every attendee of ApacheCon is welcome to join the meetup. We also plan to
have a a few short presentations in the meetup. If you have something that
you'd like to share, please let me know.

Thanks,

Jun

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14391999#comment-14391999
 ] 

Tong Li commented on KAFKA-1926:


[~junrao]Thanks so much for the quick review. All the new comments were 
addressed in the updated patch set.. Please review when you have time. 

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>Assignee: Tong Li
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, 
> KAFKA-1926.patch, KAFKA-1926_2015-04-01_22:16:46.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14391990#comment-14391990
 ] 

Tong Li commented on KAFKA-1926:


Updated reviewboard https://reviews.apache.org/r/32740/
 against branch origin/trunk

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>Assignee: Tong Li
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, 
> KAFKA-1926.patch, KAFKA-1926_2015-04-01_22:16:46.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1926:
---
Attachment: KAFKA-1926_2015-04-01_22:16:46.patch

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>Assignee: Tong Li
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, 
> KAFKA-1926.patch, KAFKA-1926_2015-04-01_22:16:46.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 32740: Patch for KAFKA-1926

[Tong Li]

/server/OffsetCommitTest.scala 
7654275c5437e1db319f01a8d587d66b42ea8dc7 
  core/src/test/scala/unit/kafka/server/ServerGenerateBrokerIdTest.scala 
96a8a5a8cb1f40f743490f85bf18d441ce86a765 
  core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
71317eb46b8d76060b115e6c1d31a4653459724f 
  core/src/test/scala/unit/kafka/server/ServerStartupTest.scala 
60021ef8d8d7fa172d6b160db48ee5eb9574e1a9 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
1682a77362123449de6bb1d54a55887409990a24 
  core/src/test/scala/unit/kafka/utils/UtilsTest.scala 
8c3797a964a2760a00ed60530d1a48e3da473769 
  core/src/test/scala/unit/kafka/zk/EmbeddedZookeeper.scala 
31515615089385c722325bfe019a47ae3a9a4052 
  core/src/test/scala/unit/kafka/zk/ZooKeeperTestHarness.scala 
67d9c4bab270c852dc05d47aaa4dd96b0f6039d4 

Diff: https://reviews.apache.org/r/32740/diff/


Testing
---


Thanks,

Tong Li

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14391295#comment-14391295
 ] 

Tong Li commented on KAFKA-1926:


[~junrao], all comments have been addressed in the latest patch set other than 
the readByte movement. The move of readByte from core to client will cause many 
other changes since the method uses scala.Byte, any thing uses that method will 
have to have the signature changes. Since this patch set is already huge, I 
would very much like to do that refactor in a later patch set. Thanks.

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>Assignee: Tong Li
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, 
> KAFKA-1926.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 32740: Patch for KAFKA-1926

[Tong Li]

 
7654275c5437e1db319f01a8d587d66b42ea8dc7 
  core/src/test/scala/unit/kafka/server/ServerGenerateBrokerIdTest.scala 
96a8a5a8cb1f40f743490f85bf18d441ce86a765 
  core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
71317eb46b8d76060b115e6c1d31a4653459724f 
  core/src/test/scala/unit/kafka/server/ServerStartupTest.scala 
60021ef8d8d7fa172d6b160db48ee5eb9574e1a9 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
1682a77362123449de6bb1d54a55887409990a24 
  core/src/test/scala/unit/kafka/utils/UtilsTest.scala 
8c3797a964a2760a00ed60530d1a48e3da473769 
  core/src/test/scala/unit/kafka/zk/EmbeddedZookeeper.scala 
31515615089385c722325bfe019a47ae3a9a4052 
  core/src/test/scala/unit/kafka/zk/ZooKeeperTestHarness.scala 
67d9c4bab270c852dc05d47aaa4dd96b0f6039d4 

Diff: https://reviews.apache.org/r/32740/diff/


Testing
---


Thanks,

Tong Li

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14391281#comment-14391281
 ] 

Tong Li commented on KAFKA-1926:


Created reviewboard https://reviews.apache.org/r/32740/
 against branch origin/trunk

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>Assignee: Tong Li
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, 
> KAFKA-1926.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1926:
---
Attachment: KAFKA-1926.patch

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>Assignee: Tong Li
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, 
> KAFKA-1926.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: KIP discussion Mar 24 at 11am PST

[Tong Li]

Jun,
   I would like to participate but have not received the invite. Can you
please send to me?

Thanks.

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Jun Rao 
To: "dev@kafka.apache.org" 
Date:   03/27/2015 10:35 PM
Subject:Re: KIP discussion Mar 24 at 11am PST



Just sent out an invite for Mar 31 at 11am PST. The following is the
tentative agenda. Let me know if you want to attend, but haven't received
an invite.

Agenda:
KIP-4 (admin commands):
* wrap up any remaining issues

KIP-13 (quota):
* dependency on using the new metrics package
* dependency KIP-5 (broker configuration)

Perhaps security related KIPS.

Thanks,

Jun

On Fri, Mar 27, 2015 at 6:30 PM, Joel Koshy  wrote:

> I think we decided that we should reconvene next Tuesday to collect
> everyone's thoughts on metrics and discuss other KIPs. Can everyone
> still do Tuesday or do people feel we need to push that further out by
> a few days?
>
> Thanks,
>
> Joel
>
> On Tue, Mar 24, 2015 at 12:28:04PM -0700, Jun Rao wrote:
> > Just to keep everyone posted. The following is a summary of what's
being
> > discussed in the KIP hangout today.
> >
> > KIP-4 (admin commands):
> > * Gwen is uploading a patch in KAFKA-1927 (refactoring requests) so
that
> we
> > can get unblocked of adding new requests.
> > * We will combine DescribeTopic and TopicMetadata in the future.
> > * We will leave the admin requests async for now.
> > * We will not add a VerifyReassignPartitionRequest for now. We can do
> that
> > later when we improve the verification process.
> > * We need to discuss a bit more on how to expose the controller info to
> the
> > client.
> > * Andrii will send out more details on the KIP thread.
> >
> > KIP-15 (close):
> > * If close() or close with a non-zero timeout is called from the send
> > thread, we will log it as an error.
> > * Jiangjie will follow up on the KIP thread.
> >
> > KIP-13 (quota):
> > * Need a separate discussion on whether to use the new metrics package
on
> > the broker on the mailing list.
> > * There are a few other details being discuss and Aditya will follow up
> on
> > the KIP thread.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Fri, Mar 20, 2015 at 2:44 PM, Jun Rao  wrote:
> >
> > > Hi, Everyone,
> > >
> > > We plan to have a KIP discussion on Google hangout on Mar 24 at 11am
> PST.
> > > If you are interested in participating and have not already received
a
> > > calendar invitation, please let me know. The following is the agenda.
> > >
> > > KIP-4 (admin commands): 10 mins
> > > * status of KAFKA-1927 (refactoring requests). which blocks this KIP
> > > * status of KAFKA-1634 (improve OffsetCommitRequest), which blocks
> > > KAFKA-1927
> > > * any remaining issues for discussion
> > >
> > > KIP-15 (close): 10 mins
> > > * semantics of close() and close(timeout)
> > >
> > > KIP-13 (quota):
> > > * protocol change to reflect the state of throttling
> > > * dependency on using the new metrics package
> > > * dependency KIP-5 (broker configuration)
> > >
> > > Thanks,
> > >
> > > Jun
> > >
>
>

Re: Dropping support for Scala 2.9.x

[Tong Li]

+1. But can we also look at this from the deployment base point view and
find out how many production deployments are still using 2.9? If there is
not any, dropping it is really an easy decision.

Thanks

Sent from my iPhone

> On Mar 27, 2015, at 8:21 AM, Ismael Juma  wrote:
>
> Hi all,
>
> The Kafka build currently includes support for Scala 2.9, which means
that
> it cannot take advantage of features introduced in Scala 2.10 or depend
on
> libraries that require it.
>
> This restricts the solutions available while trying to solve existing
> issues. I was browsing JIRA looking for areas to contribute and I quickly
> ran into two issues where this is the case:
>
> * KAFKA-1351: "String.format is very expensive in Scala" could be solved
> nicely by using the String interpolation feature introduced in Scala
2.10.
>
> * KAFKA-1595: "Remove deprecated and slower scala JSON parser from
> kafka.consumer.TopicCount" could be solved by using an existing JSON
> library, but both jackson-scala and play-json require 2.10 (argonaut
> supports Scala 2.9, but it brings other dependencies like scalaz). We can
> workaround this by writing our own code instead of using libraries, of
> course, but it's not ideal.
>
> Other features like Scala Futures and value classes would also be useful
in
> some situations, I would think (for a more extensive list of new
features,
> see
>
http://scala-language.1934581.n4.nabble.com/Scala-2-10-0-now-available-td4634126.html

> ).
>
> Another pain point of supporting 2.9.x is that it doubles the number of
> build and test configurations required from 2 to 4 (because the 2.9.x
> series was not necessarily binary compatible).
>
> A strong argument for maintaining support for 2.9.x was the client
library,
> but that has been rewritten in Java.
>
> It's also worth mentioning that Scala 2.9.1 was released in August 2011
> (more than 3.5 years ago) and the 2.9.x series hasn't received updates of
> any sort since early 2013. Scala 2.10.0, in turn, was released in January
> 2013 (over 2 years ago) and 2.10.5, the last planned release in the
2.10.x
> series, has been recently released (so even 2.10.x won't be receiving
> updates any longer).
>
> All in all, I think it would not be unreasonable to drop support for
Scala
> 2.9.x in a future release, but I may be missing something. What do others
> think?
>
> Ismael

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14363164#comment-14363164
 ] 

Tong Li commented on KAFKA-1926:


@Jun Rao, awesome comments. I will be following the directions and provide new 
patch set. This also confirms the direction that I am going. Thanks.

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

java.net.BindException: Address already in use

[Tong Li]

Hi guys, when I ran test, I got a lot of these exceptions. I wonder if you
had similar problems running the tests and how resolved the issue.

Thanks.

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14359676#comment-14359676
 ] 

Tong Li commented on KAFKA-1988:




I thought this one was merged. Please let me know what I need to do with
it?

Thanks

Sent from my iPhone

wrote:
[ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14359229#comment-14359229
 ]

negative numbers.


KAFKA-1988.patch, KAFKA-1988_2015-03-03_19:00:21.patch,
KAFKA-1988_2015-03-03_19:03:31.patch
negative numbers. The method only returns intended value for positive
numbers. All negative numbers except the Integer.Min_Value will be returned
an unsigned integer.


> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers.
> 
>
> Key: KAFKA-1988
> URL: https://issues.apache.org/jira/browse/KAFKA-1988
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0
>Reporter: Tong Li
>Assignee: Tong Li
>Priority: Blocker
> Fix For: 0.8.3
>
> Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch, 
> KAFKA-1988_2015-03-03_19:00:21.patch, KAFKA-1988_2015-03-03_19:03:31.patch
>
>
> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers. The method only returns intended value for positive 
> numbers. All negative numbers except the Integer.Min_Value will be returned 
> an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

[Tong Li]

Guozhang,
 augmenting topic is fine, but as soon as we start doing that, other
issues follow, for example, access control, who can access the topic, who
can grant permissions. how the information (metadata) itself gets secured.
Should the information be saved in ZK or a datastore? Will using a metadata
file causing long term problems such as file updates/synchronization, once
we have this metadata file, more people will want to put more stuff in it.
how can we control the format? K-V pair not good for large data set.
Clearly there is a need for it, I wonder if we can make this thing
plugable and provide a default implementation which allows us try different
solutions and also allow people to completely ignore it if they do not want
to deal with any of these.

Thanks.

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Guozhang Wang 
To: "dev@kafka.apache.org" 
Date:   03/12/2015 09:39 AM
Subject:Re: [DISCUSS] KIP-4 - Command line and centralized
administrative operations



Folks,

Just want to elaborate a bit more on the create-topic metadata and batching
describe-topic based on config / metadata in my previous email as we work
on KAFKA-1694. The main motivation is to have some sort of topic management
mechanisms, which I think is quite important in a multi-tenant / cloud
architecture: today anyone can create topics in a shared Kafka cluster, but
there is no concept or "ownership" of topics that are created by different
users. For example, at LinkedIn we basically distinguish topic owners via
some casual topic name prefix, which is a bit awkward and does not fly as
we scale our customers. It would be great to use describe-topics such as:

Describe all topics that is created by me.

Describe all topics whose retention time is overriden to X.

Describe all topics whose writable group include user Y (this is related to
authorization), etc..

One possible way to achieve this is to add a metadata file in the
create-topic request, whose value will also be written ZK as we create the
topic; then describe-topics can choose to batch topics based on 1) name
regex, 2) config K-V matching, 3) metadata regex, etc.

Thoughts?

Guozhang

On Thu, Mar 5, 2015 at 4:37 PM, Guozhang Wang  wrote:

> Thanks for the updated wiki. A few comments below:
>
> 1. Error description in response: I think if some errorCode could
indicate
> several different error cases then we should really change it to multiple
> codes. In general the errorCode itself would be precise and sufficient
for
> describing the server side errors.
>
> 2. Describe topic request: it would be great to go beyond just batching
on
> topic name regex for this request. For example, a very common use case of
> the topic command is to list all topics whose config A's value is B. With
> topic name regex then we have to first retrieve __all__ topics's
> description info and then filter at the client end, which will be a huge
> burden on ZK.
>
> 3. Config K-Vs in create topic: this is related to the previous point;
> maybe we can add another metadata K-V or just a metadata string along
side
> with config K-V in create topic like we did for offset commit request.
This
> field can be quite useful in storing information like "owner" of the
topic
> who issue the create command, etc, which is quite important for a
> multi-tenant setting. Then in the describe topic request we can also
batch
> on regex of the metadata field.
>
> 4. Today all the admin operations are async in the sense that command
will
> return once it is written in ZK, and that is why we need extra
verification
> like testUtil.waitForTopicCreated() / verify partition reassignment
> request, etc. With admin requests we could add a flag to enable / disable
> synchronous requests; when it is turned on, the response will not return
> until the request has been completed. And for async requests we can add a
> "token" field in the response, and then only need a general "admin
> verification request" with the given token to check if the async request
> has been completed.
>
> 5. +1 for extending Metadata request to include controller / coordinator
> information, and then we can remove the ConsumerMetadata /
ClusterMetadata
> requests.
>
> Guozhang
>
> On Tue, Mar 3, 2015 at 10:23 AM, Joel Koshy  wrote:
>
>> Thanks for sending that out Joe - I don't think I will be able to make
>> it today, so if notes can be sent out afterward that would be great.
>>
>> On Mon, Mar 02, 2015 at 09:16:13AM -0800, Gwen Shapira wrote:
>> > Thanks for sending this out Joe. Looking forward to chatting with
>> everyone :)
>> >
>> > On Mon, Mar 2, 2015 at 6:46 AM, Joe Stein 
wrote:
>> > > Hey, I just sent out a

problems submitting patch set for review and a bit of solution

[Tong Li]

While trying to submit a reviewboard request by running kafka-patch-review,
I was getting some errors which do not make any sense. The error claims
that a file did not exist in the repository but it did. Here is the error:


1. How the problems look like:
tong@u1404:/opt/stack/kafka$ python kafka-patch-review.py -b
origin/trunk -j KAFKA-1926
Configuring reviewboard url to https://reviews.apache.org
Updating your remote branches to pull the latest changes
Verifying JIRA connection configurations
ERROR: Error validating diff

core/src/main/scala/kafka/utils/Utils.scala: The file was not found
in the repository. (HTTP 400, API Error 207)
ERROR: reviewboard update failed. Exiting.


2. Struggled a lot trying to figure out what the problem was. after tons of
googling, I decided to upgrading python-rbtools

tong@u1404:/opt/stack/kafka$ sudo apt-get install python-rbtools
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  python-rbtools
0 upgraded, 1 newly installed, 0 to remove and 163 not upgraded.
Need to get 48.6 kB of archives.
After this operation, 269 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ trusty/universe
python-rbtools all 0.3.4-1 [48.6 kB]
Fetched 48.6 kB in 3s (13.3 kB/s)
Selecting previously unselected package python-rbtools.
(Reading database ... 57284 files and directories currently
installed.)
Preparing to unpack .../python-rbtools_0.3.4-1_all.deb ...
Unpacking python-rbtools (0.3.4-1) ...
Processing triggers for man-db (2.6.7.1-1) ...
Setting up python-rbtools (0.3.4-1) ...

3. Tried to run patch-review again. But the command stuck and seemed to me
it waits for some credentials. and it completely ignored the jira.ini file
4. Tried the command again, and it successfully posted patchset and updated
the jira issue after I blindly type in my user name, then provided
password:

tong@u1404:/opt/stack/kafka$ python kafka-patch-review.py -b
origin/trunk -j KAFKA-1926
tongli
Password:
Configuring reviewboard url to https://reviews.apache.org
Updating your remote branches to pull the latest changes
Verifying JIRA connection configurations
Review request #31844 posted.

https://reviews.apache.org/r/31844/

Creating diff against origin/trunk and uploading patch to JIRA
KAFKA-1926
Created a new reviewboard https://reviews.apache.org/r/31844/
5. it may be good to revisit the kafka-patch-review.py and see why it
ignores the jira.ini file.

When I posted the problem few days ago, I got no response from anyone, I
thought this maybe a problem that other may not have encountered. sending
to the mailing list in case anyone else stumbles on the same issue.

cheers.

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com

[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1926:
---
Attachment: KAFKA-1926.patch

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14352488#comment-14352488
 ] 

Tong Li commented on KAFKA-1926:


Created reviewboard https://reviews.apache.org/r/31844/
 against branch origin/trunk

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 31844: Patch for KAFKA-1926

[Tong Li]

/test/scala/unit/kafka/zk/EmbeddedZookeeper.scala 
31515615089385c722325bfe019a47ae3a9a4052 
  core/src/test/scala/unit/kafka/zk/ZooKeeperTestHarness.scala 
67d9c4bab270c852dc05d47aaa4dd96b0f6039d4 

Diff: https://reviews.apache.org/r/31844/diff/


Testing
---


Thanks,

Tong Li

Re: [DISCUSS] KIP-6 - New reassignment partition logic for re-balancing

[Tong Li]

Todd,
I think plugable design is good with solid default. The only issue I
feel is when you use one and switch to another, will we end up with some
unread messages hanging around and no one thinks or knows it is their
responsibility to take care of them?

Thanks.

Tong

Sent from my iPhone

> On Mar 5, 2015, at 10:46 AM, Todd Palino  wrote:
>
> Apologize for the late comment on this...
>
> So fair assignment by count (taking into account the current partition
> count of each broker) is very good. However, it's worth noting that all
> partitions are not created equal. We have actually been performing more
> rebalance work based on the partition size on disk, as given equal
> retention of all topics, the size on disk is a better indicator of the
> amount of traffic a partition gets, both in terms of storage and network
> traffic. Overall, this seems to be a better balance.
>
> In addition to this, I think there is very much a need to have Kafka be
> rack-aware. That is, to be able to assure that for a given cluster, you
> never assign all replicas for a given partition in the same rack. This
> would allow us to guard against maintenances or power failures that
affect
> a full rack of systems (or a given switch).
>
> I think it would make sense to implement the reassignment logic as a
> pluggable component. That way it would be easy to select a scheme when
> performing a reassignment (count, size, rack aware). Configuring a
default
> scheme for a cluster would allow for the brokers to create new topics and
> partitions in compliance with the requested policy.
>
> -Todd
>
>
> On Thu, Jan 22, 2015 at 10:13 PM, Joe Stein  wrote:
>
> > I will go back through the ticket and code and write more up. Should be
> > able to-do that sometime next week. The intention was to not replace
> > existing functionality by issue a WARN on use. The following version it
is
> > released we could then deprecate it... I will fix the KIP for that too.
> >
> > On Fri, Jan 23, 2015 at 12:34 AM, Neha Narkhede 
wrote:
> >
> > > Hey Joe,
> > >
> > > 1. Could you add details to the Public Interface section of the KIP?
This
> > > should include the proposed changes to the partition reassignment
tool.
> > > Also, maybe the new option can be named --rebalance instead of
> > > --re-balance?
> > > 2. It makes sense to list --decommission-broker as part of this KIP.
> > > Similarly, shouldn't we also have an --add-broker option? The way I
see
> > > this is that there are several events when a partition reassignment
is
> > > required. Before this functionality is automated on the broker, the
tool
> > > will generate an ideal replica placement for each such event. The
users
> > > should merely have to specify the nature of the event e.g. adding a
> > broker
> > > or decommissioning an existing broker or merely rebalancing.
> > > 3. If I understand the KIP correctly, the upgrade plan for this
feature
> > > includes removing the existing --generate option on the partition
> > > reassignment tool in 0.8.3 while adding all the new options in the
same
> > > release. Is that correct?
> > >
> > > Thanks,
> > > Neha
> > >
> > > On Thu, Jan 22, 2015 at 9:23 PM, Jay Kreps 
wrote:
> > >
> > > > Ditto on this one. Can you give the algorithm we want to implement?
> > > >
> > > > Also I think in terms of scope this is just proposing to change the
> > logic
> > > > in ReassignPartitionsCommand? I think we've had the discussion
various
> > > > times on the mailing list that what people really want is just for
> > Kafka
> > > to
> > > > do it's best to balance data in an online fashion (for some
definition
> > of
> > > > balance). i.e. if you add a new node partitions would slowly
migrate to
> > > it,
> > > > and if a node dies, partitions slowly migrate off it. This could
> > > > potentially be more work, but I'm not sure how much more. Has
anyone
> > > > thought about how to do it?
> > > >
> > > > -Jay
> > > >
> > > > On Wed, Jan 21, 2015 at 10:11 PM, Joe Stein 
> > > wrote:
> > > >
> > > > > Posted a KIP for --re-balance for partition assignment in
> > reassignment
> > > > > tool.
> > > > >
> > > > >
> > > > >
> > > >
> > >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New
+reassignment+partition+logic+for+re-balancing
> > > > >
> > > > > JIRA https://issues.apache.org/jira/browse/KAFKA-1792
> > > > >
> > > > > While going through the KIP I thought of one thing from the JIRA
that
> > > we
> > > > > should change. We should preserve --generate to be existing
> > > functionality
> > > > > for the next release it is in. If folks want to use --re-balance
then
> > > > > great, it just won't break any upgrade paths, yet.
> > > > >
> > > > > /***
> > > > >  Joe Stein
> > > > >  Founder, Principal Consultant
> > > > >  Big Data Open Source Security LLC
> > > > >  http://www.stealth.ly
> > > > >  Twitter: @allthingshadoop

> > > > > **

Error when submit a patch set which contains a file rename.

[Tong Li]

Hi, I was trying to submit a patch set for review. In this patch set, I
removed one file and created another to replace the removed. However, when
I run kafka-patch-review.py I got an error.

python kafka-patch-review.py -b origin/trunk -j KAFKA-1926
Configuring reviewboard url to https://reviews.apache.org
Updating your remote branches to pull the latest changes
Verifying JIRA connection configurations
ERROR: Error validating diff

core/src/main/scala/kafka/utils/Utils.scala: The file was not found in the
repository. (HTTP 400, API Error 207)
ERROR: reviewboard update failed. Exiting.

I wonder if it is because I do not have enough of permission? the file to
be renamed is clearly there. Any help will be appreciated.

Thanks.

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14348038#comment-14348038
 ] 

Tong Li commented on KAFKA-1926:


Jay, I finally put together a new patch set just for this issue. (the 
SystemTime issue we can address in a separate issue since this patch set is 
already very big). In the new patch set, I will have the following changes made:

This patch set is an attempt to refactor core utils class and remove
the namespace overlap. Utils class was defined by both clients and core
projects. Many methods are duplicate. This patch did the following:

1. Renames the core kafka.utils.Utils class to be kafka.utils.CoreUtils
2. Removed the abs method in kafka.utils.CoreUtils, anywhere using abs
   method will be replaced with the method in o.a.k.c.u.Utils.abs
3. Removed the Crc32 class in kafka.utils package. Also use the one
   defined in o.a.k.c.u.Utils class.
4. Removed asString method in CoreUtils since this was not used.
5. Removed ReadUnisnedInt (two methods), WriteUnsignedInt (two methods) 
since
   they are completely duplicate of the client methods, both signature and
   implementation are absolutely identical.

I will submit the patch set for you to review. Since this patch set touches so 
many
files, very time consuming. if we can get this in faster, then I won't need to 
do so
many rounds of rebase. Thanks so much.




> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch, KAFKA-1926.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31566: Patch for KAFKA-1988

[Tong Li]

Guozhang,
  Yes, I have rebased and updated the patch set to resolve the couple
comments for code comments. Also removed the trailing spaces (which always
annoying. :-) ), Please see the new patch set here. Sorry for the delay.

https://reviews.apache.org/r/31566/

Thanks.

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com

"Guozhang Wang"  wrote on 03/03/2015 11:48:36
AM:

> From: "Guozhang Wang" 
> To: "kafka" , "Guozhang Wang"
> , Tong Li/Raleigh/IBM@IBMUS
> Date: 03/03/2015 11:49 AM
> Subject: Re: Review Request 31566: Patch for KAFKA-1988
> Sent by: "Guozhang Wang" 
>
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31566/#review74973
> ---
>
>
> Tong, could you address Jun's last comments before committing?
>
> - Guozhang Wang
>
>
> On Feb. 27, 2015, 11:16 p.m., Tong Li wrote:
> >
> > ---
> > This is an automatically generated e-mail. To reply, visit:
> > https://reviews.apache.org/r/31566/
> > ---
> >
> > (Updated Feb. 27, 2015, 11:16 p.m.)
> >
> >
> > Review request for kafka.
> >
> >
> > Bugs: KAFKA-1988
> > https://issues.apache.org/jira/browse/KAFKA-1988
> >
> >
> > Repository: kafka
> >
> >
> > Description
> > ---
> >
> > KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns
> wrong value for negative numbers
> >
> >
> > Diffs
> > -
> >
> >   clients/src/main/java/org/apache/kafka/clients/producer/
> internals/Partitioner.java dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8
> >   clients/src/main/java/org/apache/kafka/common/utils/Utils.java
> 69530c187cd1c41b8173b61de6f982aafe65c9fe
> >   clients/src/test/java/org/apache/kafka/common/utils/
> UtilsTest.java 4c2ea34815b63174732d58b699e1a0a9e6ec3b6f
> >
> > Diff: https://reviews.apache.org/r/31566/diff/
> >
> >
> > Testing
> > ---
> >
> >
> > Thanks,
> >
> > Tong Li
> >
> >
>

[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1988:
---
Attachment: KAFKA-1988_2015-03-03_19:03:31.patch

> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers.
> 
>
> Key: KAFKA-1988
> URL: https://issues.apache.org/jira/browse/KAFKA-1988
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0
>Reporter: Tong Li
>Assignee: Tong Li
>Priority: Blocker
> Fix For: 0.8.2.1
>
> Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch, 
> KAFKA-1988_2015-03-03_19:00:21.patch, KAFKA-1988_2015-03-03_19:03:31.patch
>
>
> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers. The method only returns intended value for positive 
> numbers. All negative numbers except the Integer.Min_Value will be returned 
> an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346045#comment-14346045
 ] 

Tong Li commented on KAFKA-1988:


Updated reviewboard https://reviews.apache.org/r/31566/diff/
 against branch origin/trunk

> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers.
> 
>
> Key: KAFKA-1988
> URL: https://issues.apache.org/jira/browse/KAFKA-1988
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0
>Reporter: Tong Li
>Assignee: Tong Li
>Priority: Blocker
> Fix For: 0.8.2.1
>
> Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch, 
> KAFKA-1988_2015-03-03_19:00:21.patch, KAFKA-1988_2015-03-03_19:03:31.patch
>
>
> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers. The method only returns intended value for positive 
> numbers. All negative numbers except the Integer.Min_Value will be returned 
> an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31566: Patch for KAFKA-1988

[Tong Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31566/
---

(Updated March 4, 2015, 12:03 a.m.)


Review request for kafka.


Bugs: KAFKA-1988
https://issues.apache.org/jira/browse/KAFKA-1988


Repository: kafka


Description
---

KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns wrong value 
for negative numbers


Diffs (updated)
-

  
clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java
 dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8 
  clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
69530c187cd1c41b8173b61de6f982aafe65c9fe 
  clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 
4c2ea34815b63174732d58b699e1a0a9e6ec3b6f 

Diff: https://reviews.apache.org/r/31566/diff/


Testing
---


Thanks,

Tong Li

[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1988:
---
Attachment: KAFKA-1988_2015-03-03_19:00:21.patch

> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers.
> 
>
> Key: KAFKA-1988
> URL: https://issues.apache.org/jira/browse/KAFKA-1988
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0
>Reporter: Tong Li
>Assignee: Tong Li
>Priority: Blocker
> Fix For: 0.8.2.1
>
> Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch, 
> KAFKA-1988_2015-03-03_19:00:21.patch
>
>
> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers. The method only returns intended value for positive 
> numbers. All negative numbers except the Integer.Min_Value will be returned 
> an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14346038#comment-14346038
 ] 

Tong Li commented on KAFKA-1988:


Updated reviewboard https://reviews.apache.org/r/31566/diff/
 against branch origin/trunk

> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers.
> 
>
> Key: KAFKA-1988
> URL: https://issues.apache.org/jira/browse/KAFKA-1988
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0
>Reporter: Tong Li
>Assignee: Tong Li
>Priority: Blocker
> Fix For: 0.8.2.1
>
> Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch, 
> KAFKA-1988_2015-03-03_19:00:21.patch
>
>
> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers. The method only returns intended value for positive 
> numbers. All negative numbers except the Integer.Min_Value will be returned 
> an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31566: Patch for KAFKA-1988

[Tong Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31566/
---

(Updated March 4, 2015, midnight)


Review request for kafka.


Bugs: KAFKA-1988
https://issues.apache.org/jira/browse/KAFKA-1988


Repository: kafka


Description
---

KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns wrong value 
for negative numbers


Diffs (updated)
-

  
clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java
 dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8 
  clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
69530c187cd1c41b8173b61de6f982aafe65c9fe 
  clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 
4c2ea34815b63174732d58b699e1a0a9e6ec3b6f 

Diff: https://reviews.apache.org/r/31566/diff/


Testing
---


Thanks,

Tong Li

Re: Review Request 31566: Patch for KAFKA-1988

[Tong Li]

> On March 3, 2015, 4:48 p.m., Guozhang Wang wrote:
> > Tong, could you address Jun's last comments before committing?

Yes. absolutely, doing it now. Thanks.


- Tong


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31566/#review74973
---


On Feb. 27, 2015, 11:16 p.m., Tong Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31566/
> ---
> 
> (Updated Feb. 27, 2015, 11:16 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1988
> https://issues.apache.org/jira/browse/KAFKA-1988
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns wrong value 
> for negative numbers
> 
> 
> Diffs
> -
> 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java
>  dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8 
>   clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
> 69530c187cd1c41b8173b61de6f982aafe65c9fe 
>   clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 
> 4c2ea34815b63174732d58b699e1a0a9e6ec3b6f 
> 
> Diff: https://reviews.apache.org/r/31566/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Tong Li
> 
>

Re: Review Request 31566: Patch for KAFKA-1988

[Tong Li]

Yes. It was addressed in the latest patch set. A private method called 
toPositive was added so that the partition selection is preserved cross this 
and previous versions.

Thanks

Sent from my iPhone

> On Mar 4, 2015, at 12:49 AM, Guozhang Wang  wrote:
> 
> 
> This is an automatically generated e-mail. To reply, visit: 
> https://reviews.apache.org/r/31566/
> 
> Tong, could you address Jun's last comments before committing?
> 
> - Guozhang Wang
> 
> 
> On February 27th, 2015, 11:16 p.m. UTC, Tong Li wrote:
> 
> Review request for kafka.
> By Tong Li.
> Updated Feb. 27, 2015, 11:16 p.m.
> 
> Bugs: KAFKA-1988
> Repository: kafka
> Description
> 
> KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns wrong value 
> for negative numbers
> Diffs
> 
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java
>  (dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8)
> clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
> (69530c187cd1c41b8173b61de6f982aafe65c9fe)
> clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 
> (4c2ea34815b63174732d58b699e1a0a9e6ec3b6f)
> View Diff

Fwd: patch set 1988

[Tong Li]

Folks, 
> Do not want to nag you, but wonder if any of you has couple of minutes to 
> review patch set for 1988 again so that I do not have to rebase this so many 
> times. Guozhang already +1ed(thanks Guozhang!) Here are the links for your 
> convenience. 
> 
> The issue
> https://issues.apache.org/jira/browse/KAFKA-1988
> 
> The patch set
> https://reviews.apache.org/r/31566/diff/
> 
> 
> Tong Li
> OpenStack & Kafka Community Development
> Building 501/B205
> liton...@us.ibm.com

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14343109#comment-14343109
 ] 

Tong Li commented on KAFKA-1988:


[~guozhang] Yes, the o.a.k.c.c.u.Utils.abs used in few places.  in patch set 
for issue 1926, I will consolidate both Utils modules from clients and core 
into one. So that we do not have name conflict all over the place. The patch 
set for issue 1926 will be quite big. I would like to get this thing fixed for 
coming up release first, then we can address issue 1926. Thanks.

> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers.
> 
>
> Key: KAFKA-1988
> URL: https://issues.apache.org/jira/browse/KAFKA-1988
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0
>Reporter: Tong Li
>Assignee: Tong Li
>Priority: Blocker
> Fix For: 0.8.2.1
>
> Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch
>
>
> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers. The method only returns intended value for positive 
> numbers. All negative numbers except the Integer.Min_Value will be returned 
> an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1988:
---
Attachment: KAFKA-1988.patch

> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers.
> 
>
> Key: KAFKA-1988
> URL: https://issues.apache.org/jira/browse/KAFKA-1988
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0
>Reporter: Tong Li
>Assignee: Tong Li
>Priority: Blocker
> Fix For: 0.8.2.1
>
> Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch
>
>
> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers. The method only returns intended value for positive 
> numbers. All negative numbers except the Integer.Min_Value will be returned 
> an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14341004#comment-14341004
 ] 

Tong Li commented on KAFKA-1988:


Created reviewboard https://reviews.apache.org/r/31566/diff/
 against branch origin/trunk

> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers.
> 
>
> Key: KAFKA-1988
> URL: https://issues.apache.org/jira/browse/KAFKA-1988
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0
>Reporter: Tong Li
>Assignee: Tong Li
>Priority: Blocker
> Fix For: 0.8.2.1
>
> Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch
>
>
> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers. The method only returns intended value for positive 
> numbers. All negative numbers except the Integer.Min_Value will be returned 
> an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 31566: Patch for KAFKA-1988

[Tong Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31566/
---

Review request for kafka.


Bugs: KAFKA-1988
https://issues.apache.org/jira/browse/KAFKA-1988


Repository: kafka


Description
---

KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns wrong value 
for negative numbers


Diffs
-

  
clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java
 dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8 
  clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
69530c187cd1c41b8173b61de6f982aafe65c9fe 
  clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 
4c2ea34815b63174732d58b699e1a0a9e6ec3b6f 

Diff: https://reviews.apache.org/r/31566/diff/


Testing
---


Thanks,

Tong Li

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14340959#comment-14340959
 ] 

Tong Li commented on KAFKA-1988:


Run Rao, I got the point now. Thanks so much for your very detailed 
explanation. A patch is coming in few minutes with only that.

> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers.
> 
>
> Key: KAFKA-1988
> URL: https://issues.apache.org/jira/browse/KAFKA-1988
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0
>Reporter: Tong Li
>Assignee: Tong Li
>Priority: Blocker
> Fix For: 0.8.2.1
>
> Attachments: KAFKA-1988.patch, KAFKA-1988.patch
>
>
> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers. The method only returns intended value for positive 
> numbers. All negative numbers except the Integer.Min_Value will be returned 
> an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14339924#comment-14339924
 ] 

Tong Li commented on KAFKA-1988:


Jun Rao,
Thanks for taking time reviewing the patch. Would like to confirm here 
is what you suggested to do:

1. Utils.abs will be renamed to Utils.toPositive
2. Make changes in these methods of class DefaultPartitioner and 
ByteArrayPartitioner so that toPositive gets called in partition method.

def partition(key: Any, numPartitions: Int): Int = {
Utils.abs(key.hashCode) % numPartitions
}


I can certainly make the above changes.  however method abs gets used 
many other places and making the case worse is that there are two abs methods 
both in core and clients. I found this problem when I was in an effort of 
addressing issue 1926. For now, what if I keep that method in clients, remove 
the one in core Utils module and making changes to all other modules where abs 
method gets called so that all use of abs method will point to the clients 
Utils.abs instead of two locations?

The patch set reflects the above thoughts is here 
https://reviews.apache.org/r/31533/diff/

> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers.
> 
>
> Key: KAFKA-1988
> URL: https://issues.apache.org/jira/browse/KAFKA-1988
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0
>Reporter: Tong Li
>Assignee: Tong Li
>Priority: Blocker
> Fix For: 0.8.2.1
>
> Attachments: KAFKA-1988.patch, KAFKA-1988.patch
>
>
> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers. The method only returns intended value for positive 
> numbers. All negative numbers except the Integer.Min_Value will be returned 
> an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14339921#comment-14339921
 ] 

Tong Li commented on KAFKA-1988:


Created reviewboard https://reviews.apache.org/r/31533/diff/
 against branch origin/trunk

> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers.
> 
>
> Key: KAFKA-1988
> URL: https://issues.apache.org/jira/browse/KAFKA-1988
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0
>Reporter: Tong Li
>Assignee: Tong Li
>Priority: Blocker
> Fix For: 0.8.2.1
>
> Attachments: KAFKA-1988.patch, KAFKA-1988.patch
>
>
> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers. The method only returns intended value for positive 
> numbers. All negative numbers except the Integer.Min_Value will be returned 
> an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1988:
---
Attachment: KAFKA-1988.patch

> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers.
> 
>
> Key: KAFKA-1988
> URL: https://issues.apache.org/jira/browse/KAFKA-1988
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0
>Reporter: Tong Li
>Assignee: Tong Li
>Priority: Blocker
> Fix For: 0.8.2.1
>
> Attachments: KAFKA-1988.patch, KAFKA-1988.patch
>
>
> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers. The method only returns intended value for positive 
> numbers. All negative numbers except the Integer.Min_Value will be returned 
> an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 31533: Patch for KAFKA-1988

[Tong Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31533/
---

Review request for kafka.


Bugs: KAFKA-1988
https://issues.apache.org/jira/browse/KAFKA-1988


Repository: kafka


Description
---

KAFKA-1988 abs method is not returning right values for negative numbers


Diffs
-

  clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
69530c187cd1c41b8173b61de6f982aafe65c9fe 
  clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 
4c2ea34815b63174732d58b699e1a0a9e6ec3b6f 
  core/src/main/scala/kafka/log/OffsetMap.scala 
42cdfbb6100b5c89d86144f92f661ebd844b2132 
  core/src/main/scala/kafka/producer/ByteArrayPartitioner.scala 
6a3b02e414eb7d62cfaece3344245feac54cecda 
  core/src/main/scala/kafka/producer/DefaultPartitioner.scala 
3afb22eeb4e3b8ecf49e92bd167f2f67b8f6a961 
  core/src/main/scala/kafka/producer/async/DefaultEventHandler.scala 
821901e4f434dfd9eec6eceabfc2e1e65507a57c 
  core/src/main/scala/kafka/server/AbstractFetcherManager.scala 
20c00cb8cc2351950edbc8cb1752905a0c26e79f 
  core/src/main/scala/kafka/tools/MirrorMaker.scala 
5374280dc97dc8e01e9b3ba61fd036dc13ae48cb 
  core/src/main/scala/kafka/utils/Utils.scala 
738c1af9ef5de16fdf5130daab69757a14c48b5c 
  core/src/test/scala/unit/kafka/utils/UtilsTest.scala 
8c3797a964a2760a00ed60530d1a48e3da473769 

Diff: https://reviews.apache.org/r/31533/diff/


Testing
---


Thanks,

Tong Li

Re: [jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li]

Jun Rao,
Thanks for taking time reviewing the patch. Would like to confirm
here is what you suggested to do:

1. Utils.abs will be renamed to Utils.toPositive
2. Make changes in these methods of class DefaultPartitioner and
ByteArrayPartitioner so that toPositive gets called in partition method.

def partition(key: Any, numPartitions: Int): Int = {
Utils.abs(key.hashCode) % numPartitions
}


I can certainly make the above changes.  however method abs gets used
many other places and making the case worse is that there are two abs
methods both in core and clients. I found this problem when I was in an
effort of addressing issue 1926. For now, what if I keep that method in
clients, remove the one in core Utils module and making changes to all
other modules where abs method gets called so that all use of abs method
will point to the clients Utils.abs instead of two locations?

Thanks.

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com

"Jun Rao (JIRA)"  wrote on 02/27/2015 12:43:04 AM:

> From: "Jun Rao (JIRA)" 
> To: dev@kafka.apache.org
> Date: 02/27/2015 12:43 AM
> Subject: [jira] [Commented] (KAFKA-1988)
> org.apache.kafka.common.utils.Utils.abs method returns wrong value
> for negative numbers.
>
>
> [ https://issues.apache.org/jira/browse/KAFKA-1988?
> page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
> tabpanel&focusedCommentId=14339751#comment-14339751 ]
>
> Jun Rao commented on KAFKA-1988:
> 
>
> Actually, we can probably keep the same logic in determining the
> partition from key in Partitioner. We  can move the current code in
> abs() into Partitioner, and give it a new name like toPositive().
> Then we can replace Utils.abs() with toPositive() it in the
> following code in Partitioner.
>
> // hash the key to choose a partition
> return Utils.abs(Utils.murmur2(key)) % numPartitions;
>
> This way, we will preserve the key distribution of existing users of
> the new producer.
>
> Tong,
>
> Do you want to submit a new patch based on that?
>
>
>
> > org.apache.kafka.common.utils.Utils.abs method returns wrong value
> for negative numbers.
> >
>


> >
> > Key: KAFKA-1988
> > URL: https://issues.apache.org/jira/browse/KAFKA-1988
> > Project: Kafka
> >  Issue Type: Bug
> >Affects Versions: 0.8.2.0
> >Reporter: Tong Li
> >Assignee: Tong Li
> >Priority: Blocker
> > Fix For: 0.8.2.1
> >
> > Attachments: KAFKA-1988.patch
> >
> >
> > org.apache.kafka.common.utils.Utils.abs method returns wrong value
> for negative numbers. The method only returns intended value for
> positive numbers. All negative numbers except the Integer.Min_Value
> will be returned an unsigned integer.
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>

[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1988:
---
Status: Patch Available  (was: Open)

> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers.
> 
>
> Key: KAFKA-1988
> URL: https://issues.apache.org/jira/browse/KAFKA-1988
> Project: Kafka
>  Issue Type: Bug
>    Reporter: Tong Li
>Assignee: Tong Li
> Attachments: KAFKA-1988.patch
>
>
> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers. The method only returns intended value for positive 
> numbers. All negative numbers except the Integer.Min_Value will be returned 
> an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14339478#comment-14339478
 ] 

Tong Li commented on KAFKA-1988:


Created reviewboard https://reviews.apache.org/r/31517/diff/
 against branch origin/trunk

> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers.
> 
>
> Key: KAFKA-1988
> URL: https://issues.apache.org/jira/browse/KAFKA-1988
> Project: Kafka
>  Issue Type: Bug
>Reporter: Tong Li
>Assignee: Tong Li
> Attachments: KAFKA-1988.patch
>
>
> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers. The method only returns intended value for positive 
> numbers. All negative numbers except the Integer.Min_Value will be returned 
> an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1988:
---
Attachment: KAFKA-1988.patch

> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers.
> 
>
> Key: KAFKA-1988
> URL: https://issues.apache.org/jira/browse/KAFKA-1988
> Project: Kafka
>  Issue Type: Bug
>    Reporter: Tong Li
>Assignee: Tong Li
> Attachments: KAFKA-1988.patch
>
>
> org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
> negative numbers. The method only returns intended value for positive 
> numbers. All negative numbers except the Integer.Min_Value will be returned 
> an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 31517: Patch for KAFKA-1988

[Tong Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31517/
---

Review request for kafka.


Bugs: KAFKA-1988
https://issues.apache.org/jira/browse/KAFKA-1988


Repository: kafka


Description
---

KAFKA-1988 abs method is not returning right values for negative numbers


Diffs
-

  clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
69530c187cd1c41b8173b61de6f982aafe65c9fe 
  clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 
4c2ea34815b63174732d58b699e1a0a9e6ec3b6f 

Diff: https://reviews.apache.org/r/31517/diff/


Testing
---


Thanks,

Tong Li

[jira] [Created] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

Tong Li created KAFKA-1988:
--

 Summary: org.apache.kafka.common.utils.Utils.abs method returns 
wrong value for negative numbers.
 Key: KAFKA-1988
 URL: https://issues.apache.org/jira/browse/KAFKA-1988
 Project: Kafka
  Issue Type: Bug
Reporter: Tong Li
Assignee: Tong Li


org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative 
numbers. The method only returns intended value for positive numbers. All 
negative numbers except the Integer.Min_Value will be returned an unsigned 
integer.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Tips for working with Kafka and data streams

[Tong Li]

+2, these kind of articles coming from the ones who created Kafka always
provide great value to Kafka users and developers. For my 2 cents, I would
love to see one or two articles for developers who involved in Kafka
development on the topics of how to develop test cases and how to run them,
what to expect when error occurs, typical system settings, I suspect that
most of us do run it on linux based systems, little pointer probably can
help a lot. and most importantly how to set up your dev environment so that
you are not struggling with the things the pioneers have already figured
out. For example, recommended dev. ide, debug methods, of course, these
will be the preference of the writer, no one is obligated to use but can
certainly get people started quicker. As Kafka draw more interest, I
suspect more developers will join, having something like that can be
extremely helpful.

Jay, articles similar to the one linked in your original email can actually
be submitted to developerworks, and you can get some money out of it if you
like. If you do not know how to do that, I can certainly provide some
pointers if you are interested.

Thanks.

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Jay Kreps 
To: "dev@kafka.apache.org" ,
"us...@kafka.apache.org" 
Date:   02/25/2015 02:52 PM
Subject:Tips for working with Kafka and data streams



Hey guys,

One thing we tried to do along with the product release was start to put
together a practical guide for using Kafka. I wrote this up here:
http://blog.confluent.io/2015/02/25/stream-data-platform-1/

I'd like to keep expanding on this as good practices emerge and we learn
more stuff. So two questions:
1. Anything you think other people should know about working with data
streams? What did you wish you knew when you got started?
2. Anything you don't know about but would like to hear more about?

-Jay

Re: Unit tests in java7 vs java8

[Tong Li]

Gwen,
I have not tried Java 8. Still on Java 7, but I always run into the
test hung problems (no errors on the screen and the system is completely
idle), it may be a different problem. I can recreate that problem every
time when I run "gradle --daemon testAll", I recall that couple of weeks
ago there was one patch saying fixed the problem, but I am still seeing the
problem with latest code. What I noticed is that seems tests always stop at
one of the ConsumerTest test cases. What puzzled me the most is that it was
not always a particular test case. Being very new in this community, I
think that error must be something related to my env. Here is my
environment:

Oracle JDK 7, gradle 2.2.1, scala 2.10.4. Lot of open file handles
and big enough max lock memory,

not complaining, just some observations in case you wonder what other
developers may face.

Thanks.

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Gwen Shapira 
To: "dev@kafka.apache.org" 
Date:   02/25/2015 03:47 PM
Subject:Unit tests in java7 vs java8



Hi,

Anyone running tests on Java 8? I just noticed that they take almost twice
as long to run compared to Java 7 (at least on my box, and with Scala
2.10.4).

Anyone else noticed this? Maybe even did some digging on the causes?

Gwen

Re: [jira] [Resolved] (KAFKA-1959) Class CommitThread overwrite group of Thread class causing compile errors

[Tong Li]

Joel, thanks so much for merging the two sets.

Tong

Sent from my iPhone

> On Feb 18, 2015, at 2:55 AM, Joel Koshy (JIRA)  wrote:
>
>
[ 
https://issues.apache.org/jira/browse/KAFKA-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

>
> Joel Koshy resolved KAFKA-1959.
> ---
> Resolution: Fixed
>   Assignee: Tong Li
>
> Thanks for the patch - committed to trunk.
>
> > Class CommitThread overwrite group of Thread class causing compile
errors
> >
-
> >
> > Key: KAFKA-1959
> > URL: https://issues.apache.org/jira/browse/KAFKA-1959
> > Project: Kafka
> >  Issue Type: Bug
> >  Components: core
> >     Environment: scala 2.10.4
> >Reporter: Tong Li
> >Assignee: Tong Li
> >  Labels: newbie
> > Attachments: KAFKA-1959.patch, compileError.png
> >
> >
> > class CommitThread(id: Int, partitionCount: Int, commitIntervalMs:
Long, zkClient: ZkClient)
> > extends ShutdownableThread("commit-thread")
> > with KafkaMetricsGroup {
> > private val group = "group-" + id
> > group overwrite class Thread group member, causing the following
compile error:
> > overriding variable group in class Thread of type ThreadGroup;  value
group has weaker access privileges; it should not be private
>
>
>
> --
> This message was sent by Atlassian JIRA
> (v6.3.4#6332)
>

Re: two very simple patch sets to be reviewed.

[Tong Li]

Gwen,
Really appreciate it. Thanks so much. Anyone else please review them?
Here are the links again.

> https://reviews.apache.org/r/31088/
>
> https://reviews.apache.org/r/31097/


Tong Li
OpenStack Community Development
Building 501/B205
liton...@us.ibm.com



From:   Gwen Shapira 
To: "dev@kafka.apache.org" 
Date:   02/17/2015 03:29 PM
Subject:Re: two very simple patch sets to be reviewed.



I've reviewed both (but can't commit obviously)

They are both safe (a rename and an addition to .gitignore).
The addition to .gitignore will be very useful for anyone who uses system
tests (which should be all of us).
The rename is useful only to those using IBM JDK (i.e. not all of us), but
since its just a rename, I figured there's no reason not to solve it.

Gwen

On Tue, Feb 17, 2015 at 10:05 AM, Tong Li  wrote:

> Dear kafka developers,
>  New to this community and put up two really small patch sets with open
> issues, can any one please review and comment and get them merged if all
> possible? Thanks
>
>
> https://reviews.apache.org/r/31088/
>
> https://reviews.apache.org/r/31097/
>
>
> Tong Li
> OpenStack Community Development
> Building 501/B205
> liton...@us.ibm.com
>
> [image: Inactive hide details for Tong Li---02/17/2015 01:03:36 PM---Dear
> kafka developers, New to this community and put up two real]Tong
> Li---02/17/2015 01:03:36 PM---Dear kafka developers,   New to this
> community and put up two really small patch sets with open issu
>
> From: Tong Li/Raleigh/IBM
> To: kafka 
> Date: 02/17/2015 01:03 PM
> Subject: two very simple patch sets to be reviewed.
> --
>
>
> Dear kafka developers,
>  New to this community and put up two really small patch sets with open
> issues, can any one please review and comment and get them merged if all
> possible? Thanks
>
> https://reviews.apache.org/r/31097/
>
> https://reviews.apache.org/r/31097/
>
> Tong Li
> OpenStack & Kafka Community Development
> Building 501/B205
> liton...@us.ibm.com
>
>

two very simple patch sets to be reviewed.

[Tong Li]

Dear kafka developers,
New to this community and put up two really small patch sets with
open issues, can any one please review and comment and get them merged if
all possible? Thanks


https://reviews.apache.org/r/31088/

https://reviews.apache.org/r/31097/


Tong Li
OpenStack Community Development
Building 501/B205
liton...@us.ibm.com



From:   Tong Li/Raleigh/IBM
To: kafka 
Date:   02/17/2015 01:03 PM
Subject:two very simple patch sets to be reviewed.


Dear kafka developers,
New to this community and put up two really small patch sets with
open issues, can any one please review and comment and get them merged if
all possible? Thanks

https://reviews.apache.org/r/31097/

https://reviews.apache.org/r/31097/

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com

two very simple patch sets to be reviewed.

[Tong Li]

Dear kafka developers,
New to this community and put up two really small patch sets with
open issues, can any one please review and comment and get them merged if
all possible? Thanks

https://reviews.apache.org/r/31097/

https://reviews.apache.org/r/31097/

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com

[jira] [Commented] (KAFKA-1960) .gitignore does not exclude test generated files and folders.

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14323347#comment-14323347
 ] 

Tong Li commented on KAFKA-1960:


Created reviewboard https://reviews.apache.org/r/31097/diff/
 against branch origin/trunk

> .gitignore does not exclude test generated files and folders.
> -
>
> Key: KAFKA-1960
> URL: https://issues.apache.org/jira/browse/KAFKA-1960
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Reporter: Tong Li
>Priority: Minor
>  Labels: newbie
> Attachments: KAFKA-1960.patch
>
>
> gradle test can create quite few folders, .gitignore should exclude these 
> files for an easier git submit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1960) .gitignore does not exclude test generated files and folders.

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1960:
---
Attachment: KAFKA-1960.patch

> .gitignore does not exclude test generated files and folders.
> -
>
> Key: KAFKA-1960
> URL: https://issues.apache.org/jira/browse/KAFKA-1960
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>    Reporter: Tong Li
>Priority: Minor
>  Labels: newbie
> Attachments: KAFKA-1960.patch
>
>
> gradle test can create quite few folders, .gitignore should exclude these 
> files for an easier git submit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 31097: Patch for KAFKA-1960

[Tong Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31097/
---

Review request for kafka.


Bugs: KAFKA-1960
https://issues.apache.org/jira/browse/KAFKA-1960


Repository: kafka


Description
---

test created files in following folders and the files and folders should
be exclude from being committed to the source repository:

core/data/*
gradle/wrapper/*


Diffs
-

  .gitignore 06a64184eaa531fcbf5586692b78bfd48e4176ba 

Diff: https://reviews.apache.org/r/31097/diff/


Testing
---


Thanks,

Tong Li

[jira] [Created] (KAFKA-1960) .gitignore does not exclude test generated files and folders.

[Tong Li (JIRA)]

Tong Li created KAFKA-1960:
--

 Summary: .gitignore does not exclude test generated files and 
folders.
 Key: KAFKA-1960
 URL: https://issues.apache.org/jira/browse/KAFKA-1960
 Project: Kafka
  Issue Type: Bug
  Components: build
Reporter: Tong Li
Priority: Minor


gradle test can create quite few folders, .gitignore should exclude these files 
for an easier git submit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31088: Patch for KAFKA-1959

[Tong Li]

> On Feb. 16, 2015, 6:22 p.m., Gwen Shapira wrote:
> > The rename does make things a bit clearer, so I don't object to committing 
> > it. 
> > But the test does not override "group" in superclas Thread. Thread.group is 
> > a private variable, so it cannot be overriden (and the classes that inherit 
> > from it can't use it) so there's no conflict here.
> 
> Tong Li wrote:
> Thanks Gwen, completely agreed. What puzzled me is that the compile 
> error. I do not know how to explain that. I have uploaded the screen shot of 
> the error against the jira issue, I wonder if you can give a bit of opinion 
> on that? Here is the link 
> https://issues.apache.org/jira/secure/attachment/12699149/compileError.png
> 
> Gwen Shapira wrote:
> interesting. I didn't see this error on my setup.
> 
> Which Scala / Java versions are you using?
> 
> Tong Li wrote:
> Gwen, thanks again. This error really bugs me. I am using IBM JDK 7 (can 
> not use oracle jdk because I am an IBMer). Scala version 2.10.4.

I replaced the IBM JDK7 with Oracle JDK 7. Not seeing that compile error. I am 
pretty sure it is related to the jdk. Would you still like the patch to be 
merged? I think it still helps if some other people running IBM JDK.


- Tong


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31088/#review72637
---


On Feb. 16, 2015, 4:37 p.m., Tong Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31088/
> ---
> 
> (Updated Feb. 16, 2015, 4:37 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1959
> https://issues.apache.org/jira/browse/KAFKA-1959
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> The original code clearly wants to define a string but override the 
> ThreadGroup of
> the super class member. The patchset will rename the private variable to be 
> group_id
> as it intended.
> 
> 
> Diffs
> -
> 
>   core/src/test/scala/other/kafka/TestOffsetManager.scala 
> 41f334d48897b3027ed54c58bbf4811487d3b191 
> 
> Diff: https://reviews.apache.org/r/31088/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Tong Li
> 
>

Re: Review Request 31088: Patch for KAFKA-1959

[Tong Li]

> On Feb. 16, 2015, 6:22 p.m., Gwen Shapira wrote:
> > The rename does make things a bit clearer, so I don't object to committing 
> > it. 
> > But the test does not override "group" in superclas Thread. Thread.group is 
> > a private variable, so it cannot be overriden (and the classes that inherit 
> > from it can't use it) so there's no conflict here.
> 
> Tong Li wrote:
> Thanks Gwen, completely agreed. What puzzled me is that the compile 
> error. I do not know how to explain that. I have uploaded the screen shot of 
> the error against the jira issue, I wonder if you can give a bit of opinion 
> on that? Here is the link 
> https://issues.apache.org/jira/secure/attachment/12699149/compileError.png
> 
> Gwen Shapira wrote:
> interesting. I didn't see this error on my setup.
> 
> Which Scala / Java versions are you using?

Gwen, thanks again. This error really bugs me. I am using IBM JDK 7 (can not 
use oracle jdk because I am an IBMer). Scala version 2.10.4.


- Tong


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31088/#review72637
---


On Feb. 16, 2015, 4:37 p.m., Tong Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31088/
> ---
> 
> (Updated Feb. 16, 2015, 4:37 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1959
> https://issues.apache.org/jira/browse/KAFKA-1959
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> The original code clearly wants to define a string but override the 
> ThreadGroup of
> the super class member. The patchset will rename the private variable to be 
> group_id
> as it intended.
> 
> 
> Diffs
> -
> 
>   core/src/test/scala/other/kafka/TestOffsetManager.scala 
> 41f334d48897b3027ed54c58bbf4811487d3b191 
> 
> Diff: https://reviews.apache.org/r/31088/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Tong Li
> 
>

Re: Review Request 31088: Patch for KAFKA-1959

[Tong Li]

> On Feb. 16, 2015, 6:22 p.m., Gwen Shapira wrote:
> > The rename does make things a bit clearer, so I don't object to committing 
> > it. 
> > But the test does not override "group" in superclas Thread. Thread.group is 
> > a private variable, so it cannot be overriden (and the classes that inherit 
> > from it can't use it) so there's no conflict here.

Thanks Gwen, completely agreed. What puzzled me is that the compile error. I do 
not know how to explain that. I have uploaded the screen shot of the error 
against the jira issue, I wonder if you can give a bit of opinion on that? Here 
is the link 
https://issues.apache.org/jira/secure/attachment/12699149/compileError.png


- Tong


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31088/#review72637
-------


On Feb. 16, 2015, 4:37 p.m., Tong Li wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/31088/
> ---
> 
> (Updated Feb. 16, 2015, 4:37 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1959
> https://issues.apache.org/jira/browse/KAFKA-1959
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> The original code clearly wants to define a string but override the 
> ThreadGroup of
> the super class member. The patchset will rename the private variable to be 
> group_id
> as it intended.
> 
> 
> Diffs
> -
> 
>   core/src/test/scala/other/kafka/TestOffsetManager.scala 
> 41f334d48897b3027ed54c58bbf4811487d3b191 
> 
> Diff: https://reviews.apache.org/r/31088/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Tong Li
> 
>

[jira] [Updated] (KAFKA-1959) Class CommitThread overwrite group of Thread class causing compile errors

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1959:
---
Attachment: compileError.png

> Class CommitThread overwrite group of Thread class causing compile errors
> -
>
> Key: KAFKA-1959
> URL: https://issues.apache.org/jira/browse/KAFKA-1959
> Project: Kafka
>  Issue Type: Bug
>  Components: core
> Environment: scala 2.10.4
>Reporter: Tong Li
>  Labels: newbie
> Attachments: KAFKA-1959.patch, compileError.png
>
>
> class CommitThread(id: Int, partitionCount: Int, commitIntervalMs: Long, 
> zkClient: ZkClient)
> extends ShutdownableThread("commit-thread")
> with KafkaMetricsGroup {
> private val group = "group-" + id
> group overwrite class Thread group member, causing the following compile 
> error:
> overriding variable group in class Thread of type ThreadGroup;  value group 
> has weaker access privileges; it should not be private



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1959) Class CommitThread overwrite group of Thread class causing compile errors

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322966#comment-14322966
 ] 

Tong Li commented on KAFKA-1959:


Created reviewboard https://reviews.apache.org/r/31088/diff/
 against branch origin/trunk

> Class CommitThread overwrite group of Thread class causing compile errors
> -
>
> Key: KAFKA-1959
> URL: https://issues.apache.org/jira/browse/KAFKA-1959
> Project: Kafka
>  Issue Type: Bug
>  Components: core
> Environment: scala 2.10.4
>Reporter: Tong Li
>  Labels: newbie
> Attachments: KAFKA-1959.patch
>
>
> class CommitThread(id: Int, partitionCount: Int, commitIntervalMs: Long, 
> zkClient: ZkClient)
> extends ShutdownableThread("commit-thread")
> with KafkaMetricsGroup {
> private val group = "group-" + id
> group overwrite class Thread group member, causing the following compile 
> error:
> overriding variable group in class Thread of type ThreadGroup;  value group 
> has weaker access privileges; it should not be private



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1959) Class CommitThread overwrite group of Thread class causing compile errors

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1959:
---
Attachment: KAFKA-1959.patch

> Class CommitThread overwrite group of Thread class causing compile errors
> -
>
> Key: KAFKA-1959
> URL: https://issues.apache.org/jira/browse/KAFKA-1959
> Project: Kafka
>  Issue Type: Bug
>  Components: core
> Environment: scala 2.10.4
>Reporter: Tong Li
>  Labels: newbie
> Attachments: KAFKA-1959.patch
>
>
> class CommitThread(id: Int, partitionCount: Int, commitIntervalMs: Long, 
> zkClient: ZkClient)
> extends ShutdownableThread("commit-thread")
> with KafkaMetricsGroup {
> private val group = "group-" + id
> group overwrite class Thread group member, causing the following compile 
> error:
> overriding variable group in class Thread of type ThreadGroup;  value group 
> has weaker access privileges; it should not be private



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 31088: Patch for KAFKA-1959

[Tong Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31088/
---

Review request for kafka.


Bugs: KAFKA-1959
https://issues.apache.org/jira/browse/KAFKA-1959


Repository: kafka


Description
---

The original code clearly wants to define a string but override the ThreadGroup 
of
the super class member. The patchset will rename the private variable to be 
group_id
as it intended.


Diffs
-

  core/src/test/scala/other/kafka/TestOffsetManager.scala 
41f334d48897b3027ed54c58bbf4811487d3b191 

Diff: https://reviews.apache.org/r/31088/diff/


Testing
---


Thanks,

Tong Li

[jira] [Commented] (KAFKA-1959) Class CommitThread overwrite group of Thread class causing compile errors

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14322933#comment-14322933
 ] 

Tong Li commented on KAFKA-1959:


Though  I saw the error by using scala 2.10.4,  I think it will be the same 
error in other scala versions. 

> Class CommitThread overwrite group of Thread class causing compile errors
> -
>
> Key: KAFKA-1959
> URL: https://issues.apache.org/jira/browse/KAFKA-1959
> Project: Kafka
>  Issue Type: Bug
>  Components: core
> Environment: scala 2.10.4
>Reporter: Tong Li
>  Labels: newbie
>
> class CommitThread(id: Int, partitionCount: Int, commitIntervalMs: Long, 
> zkClient: ZkClient)
> extends ShutdownableThread("commit-thread")
> with KafkaMetricsGroup {
> private val group = "group-" + id
> group overwrite class Thread group member, causing the following compile 
> error:
> overriding variable group in class Thread of type ThreadGroup;  value group 
> has weaker access privileges; it should not be private



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-1959) Class CommitThread overwrite group of Thread class causing compile errors

[Tong Li (JIRA)]

Tong Li created KAFKA-1959:
--

 Summary: Class CommitThread overwrite group of Thread class 
causing compile errors
 Key: KAFKA-1959
 URL: https://issues.apache.org/jira/browse/KAFKA-1959
 Project: Kafka
  Issue Type: Bug
  Components: core
 Environment: scala 2.10.4
Reporter: Tong Li


class CommitThread(id: Int, partitionCount: Int, commitIntervalMs: Long, 
zkClient: ZkClient)
extends ShutdownableThread("commit-thread")
with KafkaMetricsGroup {

private val group = "group-" + id

group overwrite class Thread group member, causing the following compile error:

overriding variable group in class Thread of type ThreadGroup;  value group has 
weaker access privileges; it should not be private



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14320416#comment-14320416
 ] 

Tong Li commented on KAFKA-1926:


[~jkreps] Removal of Time.scala cause a lot of code changes. Since Time.scala 
defined the SystemTime object and get used all over the place. Though another 
class SystemTime defined in client util package as a java class. The use of the 
SystemTime object in the core is like using the property, not method calls, I 
can rework that part, but it will be a lot of code like this
 SystemTime.milliseconds

To be changed to:
new SystemTime().milliseconds()

Since the code in client common util gets defined as a regular class. I feel 
doing that also will cost quite a bit rather than stick with the scala 
SystemTime and simply calls its property, not many instance gets created. Let 
me know what you think and we can always improve.

Thanks.

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch, KAFKA-1926.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14320404#comment-14320404
 ] 

Tong Li commented on KAFKA-1926:


Created reviewboard https://reviews.apache.org/r/31007/diff/
 against branch origin/trunk

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch, KAFKA-1926.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1926:
---
Attachment: KAFKA-1926.patch

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch, KAFKA-1926.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 31007: Patch for KAFKA-1926

[Tong Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31007/
---

Review request for kafka.


Bugs: KAFKA-1926
https://issues.apache.org/jira/browse/KAFKA-1926


Repository: kafka


Description
---

More patch set will be submitted to address utils.java issues
Following things happen in this patch set:

1. Crc32.java was removed so that the core uses the same class
   defined in the client package.
2. Removed Time trait defined in Time.scala to use the interface
   defined in the client Time module.
3. Rewrite the SystemTime object in Time.scala so that it uses
   the client SystemTime class.
4. References to Time has been all refacted to use the interface
   defined in client module.
5. .gitignore does not exclude the following build process created
   folder and files:
  core/data/*
  gradle/wrapper/*


Diffs
-

  .gitignore 06a64184eaa531fcbf5586692b78bfd48e4176ba 
  clients/src/main/java/org/apache/kafka/common/utils/Time.java 
66c44de74521ed13cff75d556ebefa8846485b21 
  core/src/main/scala/kafka/cluster/Partition.scala 
e6ad8be5e33b6fb31c078ad78f8de709869ddc04 
  core/src/main/scala/kafka/cluster/Replica.scala 
bd13c20338ce3d73113224440e858a12814e5adb 
  core/src/main/scala/kafka/log/Log.scala 
846023bb98d0fa0603016466360c97071ac935ea 
  core/src/main/scala/kafka/log/LogCleaner.scala 
f8e7cd5fabce78c248a9027c4bb374a792508675 
  core/src/main/scala/kafka/log/LogManager.scala 
47d250af62c1aa53d11204a332d0684fb4217c8d 
  core/src/main/scala/kafka/log/LogSegment.scala 
ac9643423a28d189133705ba69b16cfce23f0049 
  core/src/main/scala/kafka/network/SocketServer.scala 
76ce41aed6e04ac5ba88395c4d5008aca17f9a73 
  core/src/main/scala/kafka/server/KafkaServer.scala 
7e5ddcb9be8fcef3df6ebc82a13ef44ef95f73ae 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
fb948b9ab28c516e81dab14dcbe211dcd99842b6 
  core/src/main/scala/kafka/server/TopicConfigManager.scala 
47295d40131492aaac786273819b7bc6e22e5486 
  core/src/main/scala/kafka/utils/Crc32.java 
0e0e7bcb33886f47b9770a122d00d6eafdbb89a9 
  core/src/main/scala/kafka/utils/Throttler.scala 
d1a144d7882919426824799ff8e8a47f89c83158 
  core/src/main/scala/kafka/utils/Time.scala 
194cc1fa73b6a68914caf46b7dd7d415b2ff6a9f 
  core/src/main/scala/kafka/utils/Utils.scala 
738c1af9ef5de16fdf5130daab69757a14c48b5c 
  core/src/test/scala/unit/kafka/server/ISRExpirationTest.scala 
a703d2715048c5602635127451593903f8d20576 
  core/src/test/scala/unit/kafka/server/LogOffsetTest.scala 
c06ee756bf0fe07e5d3c92823a476c960b37afd6 
  core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala 
5b93239cdc26b5be7696f4e7863adb9fbe5f0ed5 
  core/src/test/scala/unit/kafka/utils/MockScheduler.scala 
c6740782813cbf7c979bf1b94dd829945ef38458 
  core/src/test/scala/unit/kafka/utils/MockTime.scala 
ee65748afefd5e2d699e68fbd402b0ed9a659589 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
21d0ed2cb7c9459261d3cdc7c21dece5e2079698 

Diff: https://reviews.apache.org/r/31007/diff/


Testing
---


Thanks,

Tong Li

Re: gradle testAll stuck

[Tong Li]

Manikumar, yes, exactly the same issue. Commented on
https://issues.apache.org/jira/browse/KAFKA-1948. Before this problem
happens, the testAll normally still takes about half hour?

Thanks

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Manikumar Reddy 
To: dev@kafka.apache.org
Date:   02/12/2015 09:10 AM
Subject:Re: gradle testAll stuck
Sent by:manikumar.re...@gmail.com



This may be due to the recent issue reported by Gwen.

https://issues.apache.org/jira/browse/KAFKA-1948

On 2/12/15, Joel Koshy  wrote:
> - Can you enable test logging (see the README) and see if you can
>   figure out which test is getting stuck or taking forever?
> - A thread-dump may help.
>
> On Thu, Feb 12, 2015 at 08:57:11AM -0500, Tong Li wrote:
>>
>>
>> Hi, folks,
>> How are you all doing?
>> New bee here. Run gradle --daemon testAll on 0.8.2 worked and
>> finished
>> in about 30 minutes. pulled down trunk and run samething, always stuck.
>> left it run for overnight, checked in the morning still stuck. even on
>> 0.8.2, it still takes over 30 minutes on my modest dev system. Wonder
how
>> you all run gradle and is there any specific settings needed? I am
>> running
>> it on Ubuntu 14.04. Any help or pointer is really appreciated.
>>
>> Thanks
>>
>> Tong Li
>> OpenStack & Kafka Community Development
>> Building 501/B205
>> liton...@us.ibm.com
>
>

[jira] [Commented] (KAFKA-1948) kafka.api.consumerTests are hanging

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14318311#comment-14318311
 ] 

Tong Li commented on KAFKA-1948:


Pulled down the latest trunk and run into this problem. It will be nice if 
someone can take a look. The test does not stuck always at the same place. I 
did notice there are tons of threads getting created and died. What I did is 
when the test hung, I ran this command.

top -H -p 

You will notice that many threads come and go. and the state of these threads 
are always S (meaning sleep). including the parent thread. Not really sure what 
the deal is.

> kafka.api.consumerTests are hanging
> ---
>
> Key: KAFKA-1948
> URL: https://issues.apache.org/jira/browse/KAFKA-1948
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>
> Noticed today that very often when I run the full test suite, it hangs on 
> kafka.api.consumerTest (not always same test though). It doesn't reproduce 
> 100% of the time, but enough to be very annoying.
> I also saw it happening on trunk after KAFKA-1333:
> https://builds.apache.org/view/All/job/Kafka-trunk/389/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

gradle testAll stuck

[Tong Li]

Hi, folks,
How are you all doing?
New bee here. Run gradle --daemon testAll on 0.8.2 worked and finished
in about 30 minutes. pulled down trunk and run samething, always stuck.
left it run for overnight, checked in the morning still stuck. even on
0.8.2, it still takes over 30 minutes on my modest dev system. Wonder how
you all run gradle and is there any specific settings needed? I am running
it on Ubuntu 14.04. Any help or pointer is really appreciated.

Thanks

Tong Li
OpenStack & Kafka Community Development
Building 501/B205
liton...@us.ibm.com

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14316830#comment-14316830
 ] 

Tong Li commented on KAFKA-1926:


[~jkreps] Jay, really appreciate your quick review comments. I will see what I 
can do and will submit another patch based on trunk branch. New patch set will 
come up real soon. Thanks so much.

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14316380#comment-14316380
 ] 

Tong Li commented on KAFKA-1926:


[~harsha_ch]Yeah, will do that. Thanks.

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14316286#comment-14316286
 ] 

Tong Li commented on KAFKA-1926:


Looks like 0.8.2 will be released in few days. Should this patch set to be 
targeting 0.8.2.1? Not sure how all the versions/branches work. Thanks.

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14314962#comment-14314962
 ] 

Tong Li commented on KAFKA-1926:


[~harsha_ch], yes, new bee mistake. used the tool submitted the patch set for 
review. Seems it is working fine now. Thanks.

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1926:
---
Attachment: (was: KAFKA-1926-p1.patch)

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>  Labels: newbie, patch
> Attachments: KAFKA-1926.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1926:
---
Attachment: KAFKA-1926.patch

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>  Labels: newbie, patch
> Attachments: KAFKA-1926-p1.patch, KAFKA-1926.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14314948#comment-14314948
 ] 

Tong Li commented on KAFKA-1926:


Created reviewboard https://reviews.apache.org/r/30845/diff/
 against branch origin/0.8.2

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>  Labels: newbie, patch
> Attachments: KAFKA-1926-p1.patch, KAFKA-1926.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 30845: Patch for KAFKA-1926

[Tong Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30845/
---

Review request for kafka.


Bugs: KAFKA-1926
https://issues.apache.org/jira/browse/KAFKA-1926


Repository: kafka


Description
---

This is the first patch set to address issue KAFKA-1926
Following things happen in this patch set:

1. Crc32.java was removed so that the core uses the same class
   defined in the client package.
2. Removed Time trait defined in Time.scala to use the interface
   defined in the client Time module.
3. Rewrite the SystemTime object in Time.scala so that it uses
   the client SystemTime class.
4. References to Time has been all refacted to use the interface
   defined in client module.


Diffs
-

  core/src/main/scala/kafka/cluster/Partition.scala 
419d336824641e5a4100157dceeba5a59dcb33af 
  core/src/main/scala/kafka/cluster/Replica.scala 
bd13c20338ce3d73113224440e858a12814e5adb 
  core/src/main/scala/kafka/log/Log.scala 
ec192155bec7b643025f8044b0b6565c7b9977d1 
  core/src/main/scala/kafka/log/LogCleaner.scala 
f8e7cd5fabce78c248a9027c4bb374a792508675 
  core/src/main/scala/kafka/log/LogManager.scala 
95ca8dbb9ea8b199c204df89b3a63f9d67e0f21d 
  core/src/main/scala/kafka/log/LogSegment.scala 
ac9643423a28d189133705ba69b16cfce23f0049 
  core/src/main/scala/kafka/network/SocketServer.scala 
39b1651b680b2995cedfde95d74c086d9c6219ef 
  core/src/main/scala/kafka/server/KafkaServer.scala 
1691ad7fc80ca0b112f68e3ea0cbab265c75b26b 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
1afb0cbc403ad6f0bbaba5a35940d031e71fd3cf 
  core/src/main/scala/kafka/server/TopicConfigManager.scala 
47295d40131492aaac786273819b7bc6e22e5486 
  core/src/main/scala/kafka/utils/Crc32.java 
af9fe0d7d4ab215190c2fe4d1d165a8ffea4cdc2 
  core/src/main/scala/kafka/utils/Throttler.scala 
d1a144d7882919426824799ff8e8a47f89c83158 
  core/src/main/scala/kafka/utils/Time.scala 
194cc1fa73b6a68914caf46b7dd7d415b2ff6a9f 
  core/src/main/scala/kafka/utils/Utils.scala 
23aefb4715b177feae1d2f83e8b910653ea10c5f 
  core/src/test/scala/unit/kafka/server/ISRExpirationTest.scala 
cd302aa51eb8377d88b752d48274e403926439f2 
  core/src/test/scala/unit/kafka/server/LogOffsetTest.scala 
c06ee756bf0fe07e5d3c92823a476c960b37afd6 
  core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala 
5b93239cdc26b5be7696f4e7863adb9fbe5f0ed5 
  core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
09ed8f5a7a414ae139803bf82d336c2d80bf4ac5 
  core/src/test/scala/unit/kafka/utils/MockScheduler.scala 
d5896ed4d3b73aecb652436b5dfc80c2835af595 
  core/src/test/scala/unit/kafka/utils/MockTime.scala 
ee65748afefd5e2d699e68fbd402b0ed9a659589 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
c9e8ba257b77f46c5c9b62b451470348b6e58889 

Diff: https://reviews.apache.org/r/30845/diff/


Testing
---


Thanks,

Tong Li

Review Request 30844: Patch for KAFKA-1926

[Tong Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/30844/
---

Review request for kafka.


Bugs: KAFKA-1926
https://issues.apache.org/jira/browse/KAFKA-1926


Repository: kafka


Description
---

This is the first patch set to address issue KAFKA-1926
Following things happen in this patch set:

1. Crc32.java was removed so that the core uses the same class
   defined in the client package.
2. Removed Time trait defined in Time.scala to use the interface
   defined in the client Time module.
3. Rewrite the SystemTime object in Time.scala so that it uses
   the client SystemTime class.
4. References to Time has been all refacted to use the interface
   defined in client module.


Diffs
-

  core/src/main/scala/kafka/cluster/Partition.scala 
419d336824641e5a4100157dceeba5a59dcb33af 
  core/src/main/scala/kafka/cluster/Replica.scala 
bd13c20338ce3d73113224440e858a12814e5adb 
  core/src/main/scala/kafka/log/Log.scala 
ec192155bec7b643025f8044b0b6565c7b9977d1 
  core/src/main/scala/kafka/log/LogCleaner.scala 
f8e7cd5fabce78c248a9027c4bb374a792508675 
  core/src/main/scala/kafka/log/LogManager.scala 
95ca8dbb9ea8b199c204df89b3a63f9d67e0f21d 
  core/src/main/scala/kafka/log/LogSegment.scala 
ac9643423a28d189133705ba69b16cfce23f0049 
  core/src/main/scala/kafka/network/SocketServer.scala 
39b1651b680b2995cedfde95d74c086d9c6219ef 
  core/src/main/scala/kafka/server/KafkaServer.scala 
1691ad7fc80ca0b112f68e3ea0cbab265c75b26b 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
1afb0cbc403ad6f0bbaba5a35940d031e71fd3cf 
  core/src/main/scala/kafka/server/TopicConfigManager.scala 
47295d40131492aaac786273819b7bc6e22e5486 
  core/src/main/scala/kafka/utils/Crc32.java 
af9fe0d7d4ab215190c2fe4d1d165a8ffea4cdc2 
  core/src/main/scala/kafka/utils/Throttler.scala 
d1a144d7882919426824799ff8e8a47f89c83158 
  core/src/main/scala/kafka/utils/Time.scala 
194cc1fa73b6a68914caf46b7dd7d415b2ff6a9f 
  core/src/main/scala/kafka/utils/Utils.scala 
23aefb4715b177feae1d2f83e8b910653ea10c5f 
  core/src/test/scala/unit/kafka/server/ISRExpirationTest.scala 
cd302aa51eb8377d88b752d48274e403926439f2 
  core/src/test/scala/unit/kafka/server/LogOffsetTest.scala 
c06ee756bf0fe07e5d3c92823a476c960b37afd6 
  core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala 
5b93239cdc26b5be7696f4e7863adb9fbe5f0ed5 
  core/src/test/scala/unit/kafka/server/SimpleFetchTest.scala 
09ed8f5a7a414ae139803bf82d336c2d80bf4ac5 
  core/src/test/scala/unit/kafka/utils/MockScheduler.scala 
d5896ed4d3b73aecb652436b5dfc80c2835af595 
  core/src/test/scala/unit/kafka/utils/MockTime.scala 
ee65748afefd5e2d699e68fbd402b0ed9a659589 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
c9e8ba257b77f46c5c9b62b451470348b6e58889 

Diff: https://reviews.apache.org/r/30844/diff/


Testing
---


Thanks,

Tong Li

[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1926:
---
Attachment: KAFKA-1926-p1.patch

This is the first attempt to fix the issue. Would like to get some reviews and 
confirm if the direction is right. Thanks.

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>  Labels: newbie, patch
> Attachments: KAFKA-1926-p1.patch
>
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1926:
---
 Reviewer: Neha Narkhede
   Labels: newbie patch  (was: newbie)
Affects Version/s: 0.8.2.0
   Status: Patch Available  (was: Open)

The first part of the attempt to fix kafka-1926

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.0
>Reporter: Jay Kreps
>  Labels: newbie, patch
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1927) Replace requests in kafka.api with requests in org.apache.kafka.common.requests

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313017#comment-14313017
 ] 

Tong Li commented on KAFKA-1927:


@jay, @Gwen, thanks so much for both of your comments. I will start with 1926, 
once that one is done, I will move on to this one or other ones. Thanks.

> Replace requests in kafka.api with requests in 
> org.apache.kafka.common.requests
> ---
>
> Key: KAFKA-1927
> URL: https://issues.apache.org/jira/browse/KAFKA-1927
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jay Kreps
>
> The common package introduced a better way of defining requests using a new 
> protocol definition DSL and also includes wrapper objects for these.
> We should switch KafkaApis over to use these request definitions and consider 
> the scala classes deprecated (we probably need to retain some of them for a 
> while for the scala clients).
> This will be a big improvement because
> 1. We will have each request now defined in only one place (Protocol.java)
> 2. We will have built-in support for multi-version requests
> 3. We will have much better error messages (no more cryptic underflow errors)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14313014#comment-14313014
 ] 

Tong Li commented on KAFKA-1926:


Jay. awesome comment, really appreciated. I will start with this one.

> Replace kafka.utils.Utils with o.a.k.common.utils.Utils
> ---
>
> Key: KAFKA-1926
> URL: https://issues.apache.org/jira/browse/KAFKA-1926
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jay Kreps
>  Labels: newbie
>
> There is currently a lot of duplication between the Utils class in common and 
> the one in core.
> Our plan has been to deprecate duplicate code in the server and replace it 
> with the new common code.
> As such we should evaluate each method in the scala Utils and do one of the 
> following:
> 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
> utility in active use that is not Kafka-specific. If we migrate it we should 
> really think about the API and make sure there is some test coverage. A few 
> things in there are kind of funky and we shouldn't just blindly copy them 
> over.
> 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
> any utilities that really need to make use of Scala features to be convenient.
> 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)