[gmx-users] Can I calculate thermal conductivity of a material in Gromacs?

[Tong Li]

Dear all,


Is it possible for me to calculate the thermal conductivity of a polymer in 
Gromacs? If yes, how can I do it?


Thanks a lot,
Tong


Dalian University of Technology
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

[gmx-users] is there a suitable FF for perovskite?

[Tong Li]

Dear all,


Do you think there is a suitable FF that can do simulation for 
perovskite(CH3NH3PbI3) in gromacs?


Thanks,
Tong
-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

[gmx-users] stress calculation for version 5.0 and plus

[Tong Li]

Dear all,

 

How can I obtain the stress field of the system using Gromacs 5.0 and plus?
I know some codes using core 4.5, like gromacs_Ls and gromacs_fda. Are there
a good way for version 5.0?

 

Thanks,

Tong

 

-- 
Gromacs Users mailing list

* Please search the archive at 
http://www.gromacs.org/Support/Mailing_Lists/GMX-Users_List before posting!

* Can't post? Read http://www.gromacs.org/Support/Mailing_Lists

* For (un)subscribe requests visit
https://maillist.sys.kth.se/mailman/listinfo/gromacs.org_gmx-users or send a 
mail to gmx-users-requ...@gromacs.org.

[Openstack-operators] OpenStack Operators Milan

[Tong Li]

Can someone please put together a list of urls to all the sessions, there
are so many urls flying around.

Thanks.

Tong Li
IBM Open Technology
Building 501/B205
liton...@us.ibm.com



From:   Christoph Andreas Torlinsky <christ...@nuagenetworks.net>
To: Edgar Magana <edgar.mag...@workday.com>
Cc: "OpenStack-operators@lists.openstack.org"
<OpenStack-operators@lists.openstack.org>
Date:   03/13/2017 01:53 PM
Subject:Re: [Openstack-operators] OpenStack Operators Milan - Neutron
Session



Okay, cool, just wanted to be clear, anything I can contribute from running
our platform http://nuagex.io for the past year on Mitaka/Newton/Liberty
and Kilo, i'm happy to share as an operator - we have gone through many
releases of openstack, and tested
them out, finding some better than others etc., which is where i see our
value come in,
look forward to seeing you guys there.

I hope the sessions are kept technical and openstack operators focussed, as
i would like to see that myself, even i happen to work
for the company below ;)

all the best,

c


Christoph Andreas Torlinski
christ...@nuagenetworks.net



On 13 March 2017 at 17:33, Edgar Magana <edgar.mag...@workday.com> wrote:
  Hello Chris,





  I totally understand your position. Believe me, without vendors support
  we wouldn’t even have an Ops Meetup as successful as we have currently.


  My email was an open question to all Operators about the format of the
  sessions. It was clear that the company that you work for was requested
  an updated and not the other way around.





  Thanks,





  Edgar





  From: Christoph Andreas Torlinsky <christ...@nuagenetworks.net>
  Date: Monday, March 13, 2017 at 10:30 AM
  To: Edgar Magana <edgar.mag...@workday.com>
  Cc: "randy.perry...@dell.com" <randy.perry...@dell.com>, "
  OpenStack-operators@lists.openstack.org" <
  OpenStack-operators@lists.openstack.org>



  Subject: Re: [Openstack-operators] OpenStack Operators Milan - Neutron
  Session





  Hey Edgar, i'm going because we run our NuageX Dev/Test on OpenStack, so
  we are also 'operators', eating our own product


  and contributing back to OpenStack with anything we hit, and I'm also an
  avid user of OpenStack myself,


  hope that clears it up a bit, I'm not going to Vendor pitch or do
  marketing, for that we have other venues.





  We are also one of the sponsors as it happens along with Bloomberg, Intel
  and others,


  which is also a nice thing, to contribute into the community.





  all the best,





  c




  Christoph Andreas Torlinski

  christ...@nuagenetworks.net





  On 13 March 2017 at 17:21, Edgar Magana <edgar.mag...@workday.com> wrote:


  WOW! Something has changed from previous Ops Meet-ups. I remember clearly
  that we (Operators), wanted to have a free-from-vendors environment and
  it seems that now we want them to come and provide updates on their
  offerings.


  I think vendors have plenty opportunities to show their features and
  demos, for instance the Summits and OpenStack Days around the world.





  Thanks,





  Edgar





  From: Christoph Andreas Torlinsky <christ...@nuagenetworks.net>
  Date: Monday, March 13, 2017 at 10:11 AM
  To: "randy.perry...@dell.com" <randy.perry...@dell.com>
  Cc: "OpenStack-operators@lists.openstack.org" <
  OpenStack-operators@lists.openstack.org>
  Subject: Re: [Openstack-operators] OpenStack Operators Milan - Neutron
  Session





  I'll be there from Nuage Networks, as far as I know.










  Christoph Andreas Torlinski

  christ...@nuagenetworks.net





  On 9 March 2017 at 13:36, <randy.perry...@dell.com> wrote:
   Here is the etherpad for the Neutron Session in Milan


   https://etherpad.openstack.org/p/MIL-ops-neutron





   Please add to it anything you would like to cover.





   Also will anyone from SDN Vendors be there?  Please come and provide a 5
   minute update to your offering.





   Thank You





   Randy Perryman


   Technical Staff, Cloud Architect
   Certified OpenStack Administrator



   Dell EMC | OpenSource Solutions Group


   office + 01 603 249 7710


   mobile +01 603 321 5611


   email randy.perry...@dell.com


   http://www.dell.com/cloud





   Please consider the environment before printing this email.





   Confidentiality Notice | This e-mail message, including any attachments,
   is for the sole use of the intended recipient(s) and may contain
   confidential or proprietary information. Any unauthorized review, use,
   disclosure or distribution is prohibited. If you are not the intended
   recipient, immediately contact the sender by reply e-mail and destroy
   all copies of the original message.









   ___
   OpenStack-operators mailing list
   OpenStack-operators@lists.openstack.org
   http://lists.openstack.org/

Re: [Openstack] Newton, OpenvSwitch with DPDK, it is all or nothing! How to make it more modular?

[Tong Li]

Martin, glad that you got it working for you. My impression is that it
won't work for metadata services. Any idea if that is still the case?

Thanks.

Tong Li
IBM Open Technology
Building 501/B205
liton...@us.ibm.com



From:   Martinx - ジェームズ <thiagocmarti...@gmail.com>
To: "openstack@lists.openstack.org" <openstack@lists.openstack.org>
Date:   11/17/2016 04:31 PM
Subject:[Openstack] Newton, OpenvSwitch with DPDK, it is all or
nothing! How to make it more modular?



Hey guys,

 I'm playing with OpenvSwitch with DPDK, in my KVM Ubuntu host, it is
awesome!

 My KVM guest, have 4 vNIC, it is a L2 Bridge (another DPDK App), the first
two (eth0 and eth1), are regular bridges (Libvirt Linux Bridges (or OVS
regular bridges)), and the latest two (eth2 and eth3), are type=vhostuser
with "/var/run/openvswitch/vhost-user-1" and
"/var/run/openvswitch/vhost-user-2".

 So, all good! I can ssh into my VM via its eth0 (public), my VM can talk
with other VMs via its eth1 (private), and my virtualized DPDK-App runs
fine on top of two VirtIO vhostuser NIC (eth2 and eth3).

 Now, time to do this via OpenStack! But, it doesn't work like this, let me
try to explain...

 On OpenStack, without OVS+DPDK, my KVM Guest also have 4 vNIC, first two
(eth0 and eth1), are VXLAN networks (using the OpenvSwitch Agent), and the
latest two (eth2 and eth3), are Provider Networks (also using the
OpenvSwitch Agent).

 So, here comes the problem with OVS and DPDK...

 I've configured Neutron OVS Agent like this:
http://docs.openstack.org/developer/neutron/devref/ovs_vhostuser.html

 My intention, is to use the OVS+DPDK _only_ for the Provider Networks,
but, when I started my Instance, all the 4 vNIC are being tied to the OVS
+DPDK bridges, which is clearly, not what I've configured on my Neutron
Agents.

 So, I'm thinking here, how can I do this?

 Regular "OVS Regular" bridges for VXLAN networks, and "OVS DPDPK" bridges
for Provider Networks?

 OR, how to use OVS+DPDK for VXLAN networks? If this is possible, does it
still supports the ovs-firewall-driver?

 Another workaround, that I'm thinking here, would be to launch 2 OVS
Agents on my Compute Nodes, one for OVS and VXLAN, and another only for OVS
+DPDK... Or, maybe, for VXLAN, I can go back to Linux Bridges...

 I've read that RedHat don't recommend DPDK for VXLAN networks but, I'm
using Newton on Ubuntu, so, things might be different, specially because
Ubuntu have native support for a very recent DPDK version, for free,
thoughts?

 I appreciate any comments!

Thanks!
Thiago___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

[Openstack-operators] [interop-challenge]Patch set review for interop-challenge

[Tong Li]

Can someone please review the patch set for the interop challenge effort?
Thanks.

https://review.openstack.org/#/c/379116/

Tong Li
IBM Open Technology
Building 501/B205
liton...@us.ibm.com
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [Interop-Challenge]Please review the work load test scripts

[Tong Li]

Dear openstack/osop-tools-contrib core reviewers, please review the
following patch which provide ansible and terraform work load tests for
Interop-challenge effort.

https://review.openstack.org/#/c/366784/

any comments will be appreciated and addressed accordingly.

Thanks.

Tong Li
IBM Open Technology
Building 501/B205
liton...@us.ibm.com
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

[Openstack-operators] [OpenStack-DefCore] [openstack-operators][interop-challenge] Ansible work load test for interop patch set

[Tong Li]

The patch set has been submitted to github for awhile, can some one please
review the patch set at the link below?

https://review.openstack.org/#/c/354194/

Thanks very much!

Tong Li
IBM Open Technology
Building 501/B205
liton...@us.ibm.com
___
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators

Re: Call for testing on vagrant-bhyve

[Tong Li]

Hello,

Thanks for your reply. One of my mentor, Steve Wills, has started working
on adding support to vagrant-mutate.

https://github.com/swills/vagrant-mutate/tree/bhyve

But now it's limited to FreeBSD boxes, Linux boxes are more complicated, we
are trying to find better ways. Now we have to convert Linux boxes manually.
Tong.

2016年7月29日(金) 23:17 Christian Schwarz :

> Hi,
>
> thanks for your work!
>
> Regarding the conversion process described in the README,
> you might also want to look into / patch vagrant-mutate:
>
> https://github.com/sciurus/vagrant-mutate
>
> People use it to convert from VirtualBox to libvirt on Linux.
>
> --
> Christian Schwarz
> ___
> freebsd-current@freebsd.org mailing list
> https://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"
>
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Call for testing on vagrant-bhyve

[Tong Li]

Hello Everyone,

As part of this year's GSoC, we created a project named vagrant-bhyve
which enables bhyve as a backend
of Vagrant so that we can use Vagrant to manage bhyve VMs. Now most of
fundamental functions are working, including booting up a box, providing
network access to a box, getting ssh into a box (you can check status in
details on GitHub). My mentors and I have done some testes, but we still
need more hands to help us find bugs. We also need more users' feedbacks to
help us confirm most needed functions other than basical ones and
prioritize their implementation. So, please help us. I have documented a
step by step guide on how to test vagrant-bhyve, you can follow that. When
you find some problems or need some new features, feel free to open an
issue. All your testing and feedback will be much appreciated. :)

Here is another thing. I am using PF to porvide NAT and port forwarding
(thanks to vm-bhyve's hints), but got two problems. One is about PF anthor
and another is about port forwarding. I opened two issues for them, please
also help me figure out these two problems if you have met them before.
Thanks a lot.

Best Regards,
Tong
___
freebsd-ports@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-ports
To unsubscribe, send any mail to "freebsd-ports-unsubscr...@freebsd.org"

Call for testing on vagrant-bhyve

[Tong Li]

Hello Everyone,

As part of this year's GSoC, we created a project named vagrant-bhyve
which enables bhyve as a backend
of Vagrant so that we can use Vagrant to manage bhyve VMs. Now most of
fundamental functions are working, including booting up a box, providing
network access to a box, getting ssh into a box (you can check status in
details on GitHub). My mentors and I have done some testes, but we still
need more hands to help us find bugs. We also need more users' feedbacks to
help us confirm most needed functions other than basical ones and
prioritize their implementation. So, please help us. I have documented a
step by step guide on how to test vagrant-bhyve, you can follow that. When
you find some problems or need some new features, feel free to open an
issue. All your testing and feedback will be much appreciated. :)

Here is another thing. I am using PF to porvide NAT and port forwarding
(thanks to vm-bhyve's hints), but got two problems. One is about PF anthor
and another is about port forwarding. I opened two issues for them, please
also help me figure out these two problems if you have met them before.
Thanks a lot.

Best Regards,
Tong
___
freebsd-current@freebsd.org mailing list
https://lists.freebsd.org/mailman/listinfo/freebsd-current
To unsubscribe, send any mail to "freebsd-current-unsubscr...@freebsd.org"

Re: [Openstack] Can't get data from the neutron metadata proxy

[Tong Li]

if it is an ubuntu env, it should be something like this:

/usr/bin/python /usr/bin/neutron-metadata-agent
--config-file=/etc/neutron/metadata_agent.ini
--log-file=/var/log/neutron/neutron-metadata-agent.log
--config-file=/etc/neutron/neutron.conf

Tong Li
IBM Open Technology



From:   Turbo Fredriksson <tu...@bayour.com>
To: OpenStack Mailing List <openstack@lists.openstack.org>
Date:   07/06/2016 09:30 AM
Subject:Re: [Openstack] Can't get data from the neutron metadata proxy



On Jul 6, 2016, at 1:51 PM, Tong Li wrote:


  service neutron-metadata-agent start


Don't have that service..

What's the command line look like?
--
Att tänka innan man talar, är som att torka sig i röven
innan man skiter.
- Arne Anka
___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Can't get data from the neutron metadata proxy

[Tong Li]

if this is not an install from source, say you installed it by using
apt-get install, then you just need to start the service like this (on
ubuntu, other platform, the miles will vary)

service neutron-dhcp-agent start
service neutron-metadata-agent start

you will need to make sure that neutron.conf, dhcp_agent.ini and
metadata_agent.ini files are in /etc/neutron directory with right
configuration values.

Tong Li
IBM Open Technology




From:   Turbo Fredriksson <tu...@bayour.com>
To: OpenStack Mailing List <openstack@lists.openstack.org>
Date:   07/06/2016 08:14 AM
Subject:Re: [Openstack] Can't get data from the neutron metadata proxy



On Jul 1, 2016, at 2:56 PM, Brian Haley wrote:

> You also need to have the neutron-metadata-agent running anywhere you run
the proxy (well, anywhere the l3-agent is running since it starts them
typically). Each namespace proxy communicates with it over a Unix Domain
Socket, and it communicates with the Nova metadata server.


Ahh, that's probably it!

bladeA03:~# ll /usr/bin/neutron*metadata*
-rwxr-xr-x 1 root root 178 Jun 21 09:54 /usr/bin/neutron-metadata-agent*
-rwxr-xr-x 1 root root 184 Jun 21 09:54 /usr/bin/neutron-ns-metadata-proxy*

Had no idea there where TWO processes I needed!


So how do I get the agent started? Do I possibly miss a config
file?
--
If something's hard to do, then it's not worth doing.
- Homer Simpson


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack



___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Error: No valid host was found. There are not enough hosts available

[Tong Li]

try with a tiny image and see what you get. If you suspect that the
available RAM is the issue, use the tiny flavor to begin with and see what
happens. if tiny one is ok, you at least can remove some of the factors and
focus on your baremetal.

Thanks.

Tong Li
IBM Open Technology
Building 501/B205
liton...@us.ibm.com



From:   Irfan Sayed <irfu.sa...@gmail.com>
To: Vikram Choudhary <viks...@gmail.com>
Cc: "openstack@lists.openstack.org" <openstack@lists.openstack.org>
Date:   05/11/2016 02:53 PM
Subject:Re: [Openstack] Error: No valid host was found. There are not
enough hosts available



thanks.
i observed that, instances_path option was not there in the nova.conf
file , i added the same and restarted the service , still the issue
persist
as per the log, ExactRamFilter filter has some issue
i have created one flavor called baremetal. so does that mean that , the
RAM which is assigned to the flavor show exactly match the RAM which is
there on the physical server ?
my baremetal physical server has near about 15 GB RAM, so the exactly same
RAM should be present in the baremetal flavor ?

@vikram, i did not understand what you said, are these commands?

Regards



On Wed, May 11, 2016 at 6:58 PM, Vikram Choudhary <viks...@gmail.com>
wrote:
  Can you please check q-svc & q-agt logs and ensure no issues are
  observed..

  On Wed, May 11, 2016 at 6:17 PM, Irfan Sayed <irfu.sa...@gmail.com>
  wrote:
   in the nova.conf file i have commented out this and tried creating
   instances but still the issue.

   #scheduler_use_baremetal_filters=True
   #scheduler_use_baremetal_filters=True

   anything else i have to comment as well.?
   Regards,




   On Wed, May 11, 2016 at 5:49 PM, Chris Buccella <
   chris.bucce...@verilume.com> wrote:
 > now i dont understand what is the meaning of "ExactRamFilter
 returned 0 hosts"

 
http://docs.openstack.org/developer/nova/api/nova.scheduler.filters.exact_ram_filter.html


 You could try dropping the filter from scheduler_default_filters in
 nova.conf


 -Chris

 On Wed, May 11, 2016 at 12:11 AM, Irfan Sayed <irfu.sa...@gmail.com>
 wrote:
  Hello,
  I am trying to integrate the Ironic service with other openstack
  services. I am using liberty release of openstack.

  installed and configured the Ironic service.
  here is the setup:

  1: all openstack services are deployed on VM
  2: controller is on one VM (along with other services) and compute is
  on another VM
  3: one bare metal machine / desktop is there which i want to
  provision using the ironic service.
  4: nova compute service which runs on compute node is serving
  hypervisor for VMs and another nova compute service which is running
  on controller is acting hypervisor for bare metal node ...
  5: so actually i have hybrid model of VMs and physical server for
  openstack private cloud.
  6: i am using pxe_wol driver to boot the node. using ironic service
  also i can boot the node.

  so, this is the brief about the environment setup which i have as of
  now ...
  the issue is , when i am trying to launch the instance from horizon
  dashboard .. i am getting above error

  in the scheduler log, here is the output.

  2016-05-11 09:17:30.529 15559 INFO nova.filters
  [req-6c68100d-420f-43ad-878e-7819a43298e6
  5c50e3803049470283f7a9d18e871096 28d5644376ba4d3bab3a30e09a0b28d1 - -
  -] Filter ExactRamFilter returned 0 hosts
  2016-05-11 09:17:30.530 15559 INFO nova.filters
  [req-6c68100d-420f-43ad-878e-7819a43298e6
  5c50e3803049470283f7a9d18e871096 28d5644376ba4d3bab3a30e09a0b28d1 - -
  -] Filtering removed all hosts for the request with reservation ID
  'r-mld1goh8' and instance ID '98c49d72-9d8e-4377-bbe0-6dbef187e75a'.
  Filter results: ['RetryFilter: (start: 3, end: 3)',
  'AvailabilityZoneFilter: (start: 3, end: 3)', 'ComputeFilter: (start:
  3, end: 3)', 'ComputeCapabilitiesFilter: (start: 3, end: 3)',
  'ImagePropertiesFilter: (start: 3, end: 3)', 'ExactRamFilter: (start:
  3, end: 0)']

  now i dont understand what is the meaning of "ExactRamFilter
  returned 0 hosts"

  please help

  Regards,



  ___
  Mailing list:
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
  Post to     : openstack@lists.openstack.org
  Unsubscribe :
  http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack




   ___
   Mailing list:
   http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
   Post to     : openstack@lists.openstack.org
   Unsubscribe :
   http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


___
Mailing list: http:/

Re: [Openstack] Error: No valid host was found. There are not enough hosts available

[Tong Li]

I had similar issues. after a lot of searching, I finally figured that the
compute node uses a location which has very little space left. By default,
the instance_path point to /var/lib/nova/instances, on some of the
machines, you may have very little space there if you use SSD to just to
hold your OS. So I changed compute node configuration /etc/nova/nova.conf
instances_path to point to a location which has plenty of space. Then you
restart your compute node, that solved my problem. Try the same thing and
see if the filters are happy. Also it is entirely possible that your
neutron l2 agent may not work correctly.

good luck.

Tong Li
IBM Open Technology
Building 501/B205
liton...@us.ibm.com



From:   Irfan Sayed <irfu.sa...@gmail.com>
To: openstack@lists.openstack.org
Date:   05/11/2016 12:39 AM
Subject:[Openstack] Error: No valid host was found. There are not
enough  hosts available



Hello,
I am trying to integrate the Ironic service with other openstack services.
I am using liberty release of openstack.

installed and configured the Ironic service.
here is the setup:

1: all openstack services are deployed on VM
2: controller is on one VM (along with other services) and compute is on
another VM
3: one bare metal machine / desktop is there which i want to provision
using the ironic service.
4: nova compute service which runs on compute node is serving hypervisor
for VMs and another nova compute service which is running on controller is
acting hypervisor for bare metal node ...
5: so actually i have hybrid model of VMs and physical server for openstack
private cloud.
6: i am using pxe_wol driver to boot the node. using ironic service also i
can boot the node.

so, this is the brief about the environment setup which i have as of
now ...
the issue is , when i am trying to launch the instance from horizon
dashboard .. i am getting above error

in the scheduler log, here is the output.

2016-05-11 09:17:30.529 15559 INFO nova.filters
[req-6c68100d-420f-43ad-878e-7819a43298e6 5c50e3803049470283f7a9d18e871096
28d5644376ba4d3bab3a30e09a0b28d1 - - -] Filter ExactRamFilter returned 0
hosts
2016-05-11 09:17:30.530 15559 INFO nova.filters
[req-6c68100d-420f-43ad-878e-7819a43298e6 5c50e3803049470283f7a9d18e871096
28d5644376ba4d3bab3a30e09a0b28d1 - - -] Filtering removed all hosts for the
request with reservation ID 'r-mld1goh8' and instance ID
'98c49d72-9d8e-4377-bbe0-6dbef187e75a'. Filter results: ['RetryFilter:
(start: 3, end: 3)', 'AvailabilityZoneFilter: (start: 3, end: 3)',
'ComputeFilter: (start: 3, end: 3)', 'ComputeCapabilitiesFilter: (start: 3,
end: 3)', 'ImagePropertiesFilter: (start: 3, end: 3)', 'ExactRamFilter:
(start: 3, end: 0)']

now i dont understand what is the meaning of "ExactRamFilter  returned 0
hosts"

please help

Regards,

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Telemetry and CADF Auditing

[Tong Li]

Jorge,
Short answer Yes. But pieces for producing and receiving CADF messages
are not all in Ceilometer. To generate CADF messages and be able to direct
the messages somewhere you will need the following pieces.
1. Ceilometer notification agent, collector, http dispatcher
2. keystomemiddleware
3. pycadf library

Then you will need to configure all the pieces correctly to be able to send
cadf messages to a http endpoint. but if you just want to save cadf
messages in ceilometer data store, then http dispatcher is not needed.

Thanks.

Tong Li
IBM Open Technology
Building 501/B205
liton...@us.ibm.com



From:   "Jorge Cardoso (Cloud Operations and Analytics, IT R
Division)" <jorge.card...@huawei.com>
To: "openstack@lists.openstack.org" <openstack@lists.openstack.org>
Date:   03/22/2016 08:18 AM
Subject:[Openstack] Telemetry and CADF Auditing




Hi all,

Does anybody knows if Ceilometer (in OS Liberty) includes the results of
the CADF Audit project?

Ceilometer =
http://docs.openstack.org/liberty/install-guide-ubuntu/ceilometer.html
CADF Audit =
https://wiki.openstack.org/wiki/Ceilometer/blueprints/support-standard-audit-formats

Best,
Jorge


Dr. Jorge Cardoso
Lead Architect for Cloud Management
IT R Division

cid:image001.jpg@01D11267.D40AE3F0
Huawei Technologies Duesseldorf GmbH
European Research Center
Riesstraße 25, 80992 München

Email:jorge.card...@huawei.com
Mobile: +49 172 174 0862
Phone: +49 (0)89 158 834 4029 / Internal: 00494029
Seat: D-3.0G 3.11
HUAWEI TECHNOLOGIES Duesseldorf GmbH
Hansaallee 205, 40549 Düsseldorf, Germany, www.huawei.com
Registered Office: Düsseldorf, Register Court Düsseldorf, HRB 56063,
Managing Director: Bo PENG, Wanzhou MENG, Lifang CHEN
Sitz der Gesellschaft: Düsseldorf, Amtsgericht Düsseldorf, HRB 56063,
Geschäftsführer: Bo PENG, Wanzhou MENG, Lifang CHEN

 ___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [Openstack] Switching to KVM from QEMU

[Tong Li]

If you are using devstack, I think you just need to set libvert_type=kvm,
then run stack.sh, it will set it up for you if you had KVM supported on
that machine. Also an article to help you with it can be found here.

http://docs.openstack.org/developer/devstack/guides/devstack-with-nested-kvm.html

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Morales, Victor victor.mora...@intel.com
To: shine nelson radioactive.l...@gmail.com,
openstack@lists.openstack.org openstack@lists.openstack.org
Date:   07/06/2015 11:01 AM
Subject:Re: [Openstack] Switching to KVM from QEMU



Yes, you can.  The only thing that you have to do is modify
the /etc/nova/nova.conf file

[libvirt]
...
virt_type = kvm


Enable the kvm_intel kernel module and restart the service of nova compute

Regards,

Victor Morales

From: shine nelson radioactive.l...@gmail.com
mailto:radioactive.l...@gmail.com
Date: Friday, July 3, 2015 at 10:21 PM
To: openstack@lists.openstack.orgmailto:openstack@lists.openstack.org
openstack@lists.openstack.orgmailto:openstack@lists.openstack.org
Subject: [Openstack] Switching to KVM from QEMU

Hello,

I just recently installed DevStack on my Dedicated Server running Debian 8
(Jessie). After install, I read that KVM would run better (and faster) on
server infrastructure than QEMU. But I think I was too late to realize it.
I had already initiated stack.sh.

My question is : Can I switch to KVM without having to re-install DevStack?
If yes, can somebody help me out by pointing to the right docs / procedure
to achieve this?


Thanks in advance,
~ shine

___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack


___
Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
Post to : openstack@lists.openstack.org
Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Tong Li]

same question here as well on hosts. If I would like to secure a topic,
regardless where the topic is, I would like it to be secured. It is hard to
imagine that one would like a topic to be secured on one host, but not on
the other unless a topic spreads over multiple hosts really meant to be
totally different things. Then it will be really needed.

Thanks.

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Jeff Holoman jholo...@cloudera.com
To: dev@kafka.apache.org
Cc: Tom Graves tgraves...@yahoo.com
Date:   04/22/2015 02:40 PM
Subject:Re: [DISCUSS] KIP-11- Authorization design for kafka security



Parth,

This is a long thread, so trying to keep up here, sorry if this has been
covered before. First, great job on the KIP proposal and work so far.

Are we sure that we want to tie host level access to a given user? My
understanding is that the ACL will be (omitting some fields)

user_a, host1, host2, host3
user_b, host1, host2, host3

So there would potentially be a lot of redundancy in the configs. Does it
make sense to have hosts be at the same level as principal in the
hierarchy? This way you could just blanket the allowed / denied hosts and
only have to worry about the users. So if you follow this, then

we can wildcard the user so we can have a separate list of just host-based
access. What's the order that the perms would be evaluated if a there was
more than one match on a principal ?

Is the thought that there wouldn't usually be much overlap on hosts? I
guess I can imagine a scenario where I want to offline/online access to a
particular hosts or set of hosts and if there was overlap, I'm doing a
bunch of alter commands for just a single host. Maybe this is too contrived
an example?

I agree that having this level of granularity gives flexibility but I
wonder if people will actually use it and not just * the hosts for a given
user and create separate global list as i mentioned above?

The only other system I know of that ties users with hosts for access is
MySql and I don't love that model. Companies usually standardize on group
authorization anyway, are we complicating that issue with the inclusion of
hosts attached to users? Additionally I worry about the debt of big JSON
configs in the first place, most non-developers find them non-intuitive
already, so anything to ease this I think would be beneficial.


Thanks

Jeff

On Wed, Apr 22, 2015 at 2:22 PM, Parth Brahmbhatt 
pbrahmbh...@hortonworks.com wrote:

 Sorry I missed your last questions. I am +0 on adding ―host option for
 ―list, we could add it for symmetry. Again if this is only a CLI change
it
 can be added later if you mean adding this in authorizer interface then
we
 should make a decision now.

 Given a choice I would like to actually keep only one option which is
 resource based get (remove even the get based on principal). I see those
 (getAcl for principal or host) as special filtering case which can easily
 be achieved by a third party tool by doing list all topics and calling
 getAcls for each topic and applying filtering logic on that.  I really
 don’t see the need to make those first class citizens of the authorizer
 interface given these kind of queries will be issued outside of broker
JVM
 so they will not benefit from the caching and because the storage will be
 indexed on resource both these options even as a first class API will
just
 scan all topic acls and apply filtering logic.

 Thanks
 Parth

 On 4/22/15, 11:08 AM, Parth Brahmbhatt pbrahmbh...@hortonworks.com
 wrote:

 Please see all the available options here
 
 https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authorization
+I
 nterface#KIP-11-AuthorizationInterface-AclManagement(CLI) . I think it
 covers both hosts and operations and allows to specify a list for both.
 
 Thanks
 Parth
 
 From: Tom Graves tgraves...@yahoo.commailto:tgraves...@yahoo.com
 Reply-To: Tom Graves tgraves...@yahoo.commailto:tgraves...@yahoo.com
 Date: Wednesday, April 22, 2015 at 11:02 AM
 To: Parth Brahmbhatt
 pbrahmbh...@hortonworks.commailto:pbrahmbh...@hortonworks.com,
 dev@kafka.apache.orgmailto:dev@kafka.apache.org
 dev@kafka.apache.orgmailto:dev@kafka.apache.org
 Subject: Re: [DISCUSS] KIP-11- Authorization design for kafka security
 
 Thanks for the explanations Parth.
 
 On the configs questions, the way I see it is its more likely to
 accidentally give everyone access, especially since you have to run a
 separate command to change the acls. If there was some config for
 defaults, a cluster admin could change that to be nobody or certain set
 of users, then grant others permissions.  This would also remove the
race
 between commands.  This is something you can always add later though if
 people request it.
 
 So in kafka-acl.sh how do I actually tell it what the operation is?
 kafka-acl.sh --topic testtopic --add --grandprincipal user:joe,user:kate
 
 where does READ, WRITE, etc go?  Can specify as a list so I

Re: KIP hangout on Apr 21

[Tong Li]

Jun,
 Not sure why in these invitations, I am not seeing the google hangout
link. I am using IBM notes which is quite different from gmail and
calendar. Is there anyway that you send the hangout link in the invitation
body? Thanks.

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Jun Rao j...@confluent.io
To: dev@kafka.apache.org dev@kafka.apache.org
Date:   04/18/2015 12:50 AM
Subject:KIP hangout on Apr 21



Hi,

We will have a KIP hangout at 3pm PST on Apr 21. The following is the
tentative agenda. If you'd like to attend but haven't received an invite,
please let me know.

Agenda:
KIP-4 (admin commands): wrap up any remaining issues
KIP-11 (Authorization):
KIP-12 (SSL/Kerberos): See if there is any blocker.

jira backlog check

Thanks,

Jun

Re: [DISCUSS] KIP-11- Authorization design for kafka security

[Tong Li]

Parth,
 If one wants to use his or her own access control including
authentication system, with this design what will be needed to be done? Can
one completely turn this off so that the system behaves exactly same as it
is today?

Thanks.

Tong

Sent from my iPhone

 On Apr 15, 2015, at 1:51 PM, Parth Brahmbhatt
pbrahmbh...@hortonworks.com wrote:

 Hi Michael,

 There is code in kafka codebase that reads and interprets the topic
config JSON which has acls, owner and logconfig properties. There are 3 use
cases that we are supporting with current proposal:

   *   You use out of box simpleAcl authorizer which is tied to the acl
stored in topic config and the format is locked down.
   *   You have a custom authorizer and a custom ACL store.  Ranger/Argus
falls under this as they have their own acl store and ui that users use to
configure acls on the cluster and cluster resources  like topic. It is upto
the custom authorizer to leverage the kafka acl configs or completely
ignore them as they have set a user expectation that only acls configured
via their ui/system will be effective.
   *   You have a custom authorizer but no custom Acl store. You are
completely tied to Acl structure that we have provided in out of box
implementation.

 Thanks
 Parth

 On 4/15/15, 10:31 AM, Michael Herstine
mherst...@linkedin.com.INVALIDmailto:mherst...@linkedin.com.INVALID
wrote:

 Hi Parth,

 One question that occurred to me at the end of today’s hangout: how tied
 are we to a particular ACL representation under your proposal? I know
that
 TopicConfigCache will just contain JSON— if a particular site decides
they
 want to represent their ACLs differently, and swap out the authorizer
 implementation, will that work?  I guess what I’m asking is whether
 there’s any code in the Kafka codebase that will interpret that JSON, or
 does that logic live exclusively in the authorizer?

 On 4/14/15, 10:56 PM, Don Bosco Durai
bo...@apache.orgmailto:bo...@apache.org wrote:

 I also feel, having just IP would be more appropriate. Host lookup will
 unnecessary slow things down and would be insecure as you pointed out.

 With IP, it will be also able to setup policies (in future if needed)
with
 ranges or netmasks and it would be more scalable.

 Bosco


 On 4/14/15, 1:40 PM, Michael Herstine
mherst...@linkedin.com.INVALIDmailto:mherst...@linkedin.com.INVALID
 wrote:

 Hi Parth,

 Sorry to chime in so late, but I’ve got a minor question on the KIP.

 Several methods take a parameter named “host” of type String. Is that
 intended to be a hostname, or an IP address? If the former, I’m curious
 as
 to how that’s found (in my experience, when accepting an incoming socket
 connection, you only know the IP address, and there isn’t a way to map
 that to a hostname without a round trip to a DNS server, which is
 insecure
 anyway).


 On 3/25/15, 1:07 PM, Parth Brahmbhatt
pbrahmbh...@hortonworks.commailto:pbrahmbh...@hortonworks.com
 wrote:

 Hi all,

 I have modified the KIP to reflect the recent change request from the
 reviewers. I have been working on the code and I have the server side
 code
 for authorization ready. I am now modifying the command line utilities.
 I
 would really appreciate if some of the committers can spend sometime to
 review the KIP so we can make progress on this.

 Thanks
 Parth

 On 3/18/15, 2:20 PM, Michael Herstine
mherst...@linkedin.com.INVALIDmailto:mherst...@linkedin.com.INVALID
 wrote:

 Hi Parth,

 Thanks! A few questions:

 1. Do you want to permit rules in your ACLs that DENY access as well as
 ALLOW? This can be handy setting up rules that have exceptions. E.g.
 “Allow principal P to READ resource R from all hosts” with “Deny
 principal
 P READ access to resource R from host H1” in combination would allow P
 to
 READ R from all hosts *except* H1.

 2. When a topic is newly created, will there be an ACL created for it?
 If
 not, would that not deny subsequent access to it?

 (nit) Maybe use Principal instead of String to represent principals?


 On 3/9/15, 11:48 AM, Don Bosco Durai
bo...@apache.orgmailto:bo...@apache.org wrote:

 Parth

 Overall it is looking good. Couple of questionsŠ

 - Can you give an example how the policies will look like in the
 default
 implementation?
 - In the operations, can we support ³CONNECT² also? This can be used
 during Session connection
 - Regarding access control for ³Topic Creation², since we can¹t do it
 on
 the server side, can we de-scope it for? And plan it as a future
 feature
 request?

 Thanks

 Bosco


 On 3/6/15, 8:10 AM, Harsha ka...@harsha.iomailto:ka...@harsha.io
wrote:

 Hi Parth,
 Thanks for putting this together. Overall it looks good
 to
 me. Although AdminUtils is a concern KIP-4  can probably
 fix
 that part.
 Thanks,
 Harsha

 On Thu, Mar 5, 2015, at 10:39 AM, Parth Brahmbhatt wrote:
 Forgot to add links to wiki and jira.
 Link to wiki:
 https://cwiki.apache.org/confluence/display/KAFKA/KIP-11+-+Authoriza
 t
 i
 o
 n

Re: Will it be possible to apply quotas based on a security principal?

[Tong Li]

If it is all possible I think we can introduce request pipeline
architecture for producer, broker and consumer (does not have to be done
all at once), that way, security, quota, encryption, serialization,
compression all can be done as plugable components. One can freely stack
them up or tear them apart at the configuration level. Each component can
have its own design and implementation. Any deployed can choose what
interest him or her. Components have no dependency among each other. New
features can be easily added/tried/removed/abandoned. When I looked at our
code, instrument this so called pipeline can be very easy. Wonder if any of
you guys interested.

Thanks.

Sent from my iPhone

 On Apr 15, 2015, at 4:14 PM, Jay Kreps jay.kr...@gmail.com wrote:

 I think this should be a fairly minor follow-up item to have the quotas
key
 off of user rather than client id. The advantage of starting with
client.id
 is that it decouples the security work from the quota work in the short
 term and provides a mechanism for those using Kafka without
authentication
 to still enforce quotas.

 On Wed, Apr 15, 2015 at 6:15 AM, Adrian Preston prest...@uk.ibm.com
wrote:

  Hi,
 
  I've been investigating using Kafka for a multi-user system that
applies
  quotas at a per-user level.  Reading through KIP-13 and KAFKA-1682, I
  wondered: are there any plans to link together the security principal
and
  client identifier in some way?  Currently it appears these are separate
  concepts - so I can't see any way to apply a quota based on the
  authenticated identity of a user.
 
 
  Regards
  - Adrian
 
  Unless stated otherwise above:
  IBM United Kingdom Limited - Registered in England and Wales with
number
  741598.
  Registered office: PO Box 41, North Harbour, Portsmouth, Hampshire PO6
3AU
 
 

Re: [KIP-DISCUSSION] KIP-13 Quotas

[Tong Li]

see some response inline below.
Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com

Jay Kreps jay.kr...@gmail.com wrote on 04/07/2015 10:41:19 AM:

 From: Jay Kreps jay.kr...@gmail.com
 To: dev@kafka.apache.org dev@kafka.apache.org
 Date: 04/07/2015 10:44 AM
 Subject: Re: [KIP-DISCUSSION] KIP-13 Quotas

 Totally. But is that the only use? What I wanted to flesh out was whether
 the goal was:
 1. Expose throttling in the client metrics
 2. Enable programmatic response (i.e. stop sending stuff or something
like
 that)

 I think I kind of understand (1) but let's get specific on the metric we
 would be adding and what exactly you would expose  in a dashboard. For
 example if the goal is just monitoring do I really want a boolean flag
for
 is_throttled or do I want to know how much I am being throttled (i.e.
 throttle_pct might indicate the percent of your request time that was due
 to throttling or something like that)? If I am 1% throttled that may be
 irrelevant but 99% throttled would be quite relevant? Not sure I agree,
 just throwing that out there...

Jay, great point, I think Kafka should really just sent metrics, how to
judge if
a system is throttled should be someone other people's job. I would think
this comes down to design principles, if we follow the principal of
separation
of the concerns, then this should not be really part of Kafka.
I have been doing monitoring systems for awhile, the system being monitored
normally just
send the fact of itself, such as CPU usage, network usage, disk usage etc
to the
monitoring system, the monitoring system will run various algorithms to
eventually
decide if a system is throttled by setting up threshold and other measures.
The monitoring
system will also send out notifications/alarms if things turns bad. Just
to make this discussion even easier, a set of general purpose of agents
collecting
these data have been developed and available as part of a monitoring system
named
Monasca. If you are interested, I can provide more information. For Kafka
to have
the features such as judging if the system is throttling seems to be a
moving-away
from its core values. Just my 2 cents of course.


 For (2) the prior discussion seemed to kind of allude to this but I can't
 really come up with a use case. Is there one?

 If it is just (1) I think the question is whether it really helps much to
 have the metric on the client vs the server. I suppose this is a bit
 environment specific. If you have a central metrics system it shouldn't
 make any difference, but if you don't I suppose it does.

 -Jay

 On Mon, Apr 6, 2015 at 7:57 PM, Gwen Shapira gshap...@cloudera.com
wrote:

  Here's a wild guess:
 
  An app developer included a Kafka Producer in his app, and is not happy
  with the throughput. He doesn't have visibility into the brokers since
they
  are owned by a different team. Obviously the first instinct of a
developer
  who knows that throttling exists is to blame throttling for any
slowdown in
  the app.
  If he doesn't have a way to know from the responses whether or not his
app
  is throttled, he may end up calling Aditya at 4am asked Hey, is my app
  throttled?.
 
  I assume Aditya is trying to avoid this scenario.
 
  On Mon, Apr 6, 2015 at 7:47 PM, Jay Kreps jay.kr...@gmail.com wrote:
 
   Hey Aditya,
  
   2. I kind of buy it, but I really like to understand the details of
the
  use
   case before we make protocol changes. What changes are you proposing
in
  the
   clients for monitoring and how would that be used?
  
   -Jay
  
   On Mon, Apr 6, 2015 at 10:36 AM, Aditya Auradkar 
   aaurad...@linkedin.com.invalid wrote:
  
Hi Jay,
   
2. At this time, the proposed response format changes are only for
monitoring/informing clients. As Jun mentioned, we get instance
level
monitoring in this case since each instance that got throttled will
  have
   a
metric confirming the same. Without client level monitoring for
this,
   it's
hard for application developers to find if they are being throttled
  since
they will also have to be aware of all the brokers in the cluster.
This
   is
quite problematic for large clusters.
   
It seems nice for app developers to not have to think about kafka
   internal
metrics and only focus on the metrics exposed on their instances.
   Analogous
to having client-sde request latency metrics. Basically, we want an
  easy
way for clients to be aware if they are being throttled.
   
4. For purgatory v delay queue, I think we are on the same page. I
feel
   it
is nicer to use the purgatory but I'm happy to use a DelayQueue if
  there
are performance implications. I don't know enough about the current
and
Yasuhiro's new implementation to be sure one way or the other.
   
Stepping back, I think these two things are the only remaining
point of
discussion within the current proposal. Any concerns if I started a
   voting
thread

Re: KIP discussion Apr 7 at 11am PST

[Tong Li]

Jun Rao,
  Can you include the hangout link in the invitation? I could not find
it anywhere.

Thanks.

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Jun Rao j...@confluent.io
To: dev@kafka.apache.org dev@kafka.apache.org
Date:   04/03/2015 01:27 PM
Subject:KIP discussion Apr 7 at 11am PST



Hi, Everyone,

We plan to have a KIP discussion on Google hangout on Apr. 7 at 11am PST.
If you are interested in participating and have not already received a
calendar invitation, please let me know. The following is the agenda.

Agenda:
KIP-4 (admin commands):
* wrap up any remaining issues

KIP-13 (quota):
* wrap up any remaining issues

Security related KIPS.
KIP-7 (IP filtering)
KIP-11 (Authorization)
KIP-12 (SSL/Kerberos)

Thanks,

Jun

Re: [jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li]

Run Rao,
Thanks for reviewing it again. What commands do you run to show unused
imports? When I run grandle --daemon test, it did not show these (there
were some warnings, but not unused imports). I thought that would have
shown these things.

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Jun Rao (JIRA) j...@apache.org
To: dev@kafka.apache.org
Date:   04/06/2015 12:48 AM
Subject:[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with
o.a.k.common.utils.Utils




 [
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-1926:
---
   Resolution: Fixed
Fix Version/s: 0.8.3
   Status: Resolved  (was: Patch Available)

Thanks for rebasing the latest patch. +1. Committed to trunk after removing
unused imports in a few files.

 Replace kafka.utils.Utils with o.a.k.common.utils.Utils
 ---

 Key: KAFKA-1926
 URL: https://issues.apache.org/jira/browse/KAFKA-1926
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.8.2.0
Reporter: Jay Kreps
Assignee: Tong Li
  Labels: newbie, patch
 Fix For: 0.8.3

 Attachments: KAFKA-1926.patch, KAFKA-1926.patch,
KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926_2015-04-01_22:16:46.patch,
KAFKA-1926_2015-04-05_23:45:13.patch


 There is currently a lot of duplication between the Utils class in common
and the one in core.
 Our plan has been to deprecate duplicate code in the server and replace
it with the new common code.
 As such we should evaluate each method in the scala Utils and do one of
the following:
 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general
purpose utility in active use that is not Kafka-specific. If we migrate it
we should really think about the API and make sure there is some test
coverage. A few things in there are kind of funky and we shouldn't just
blindly copy them over.
 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will
hold any utilities that really need to make use of Scala features to be
convenient.
 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 32740: Patch for KAFKA-1926

[Tong Li]

/server/LogRecoveryTest.scala 
92e49df6964230753762aa359191a6ede337d3ac 
  core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala 
a6bb69059b548bea3ebefa2a3146600163bcfa4b 
  core/src/test/scala/unit/kafka/server/ServerGenerateBrokerIdTest.scala 
2bfaeb307094b0fba67a1976b7f87591ca36a45c 
  core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
a20321f89defdb3426e159f6c376d4d2194092c2 
  core/src/test/scala/unit/kafka/server/ServerStartupTest.scala 
661ddd504fc4cdccbd5eb129212911296f4102b0 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
a8ed1422ef11737f61fe0b62629e7244c0c11677 
  core/src/test/scala/unit/kafka/utils/UtilsTest.scala 
8c3797a964a2760a00ed60530d1a48e3da473769 
  core/src/test/scala/unit/kafka/zk/EmbeddedZookeeper.scala 
1d87506649f0d1f73a2ea5465339dc586900b626 
  core/src/test/scala/unit/kafka/zk/ZooKeeperTestHarness.scala 
fedefb5287eaf8e0d1ee585a8dd31687bdcf9d88 

Diff: https://reviews.apache.org/r/32740/diff/


Testing
---


Thanks,

Tong Li

[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1926:
---
Attachment: KAFKA-1926_2015-04-05_23:45:13.patch

 Replace kafka.utils.Utils with o.a.k.common.utils.Utils
 ---

 Key: KAFKA-1926
 URL: https://issues.apache.org/jira/browse/KAFKA-1926
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.8.2.0
Reporter: Jay Kreps
Assignee: Tong Li
  Labels: newbie, patch
 Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, 
 KAFKA-1926.patch, KAFKA-1926_2015-04-01_22:16:46.patch, 
 KAFKA-1926_2015-04-05_23:45:13.patch


 There is currently a lot of duplication between the Utils class in common and 
 the one in core.
 Our plan has been to deprecate duplicate code in the server and replace it 
 with the new common code.
 As such we should evaluate each method in the scala Utils and do one of the 
 following:
 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
 utility in active use that is not Kafka-specific. If we migrate it we should 
 really think about the API and make sure there is some test coverage. A few 
 things in there are kind of funky and we shouldn't just blindly copy them 
 over.
 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
 any utilities that really need to make use of Scala features to be convenient.
 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14480907#comment-14480907
 ] 

Tong Li commented on KAFKA-1926:


[~junrao] Jun, thanks again. Noticed you have done a lot over the weekend. what 
an inspiration! So here I did a bit. Since the patch 1809 is in, I rebased the 
patch set and addressed the couple new comments. Here it is in again for your 
review. Really appreciate all the work you did for the project. 

 Replace kafka.utils.Utils with o.a.k.common.utils.Utils
 ---

 Key: KAFKA-1926
 URL: https://issues.apache.org/jira/browse/KAFKA-1926
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.8.2.0
Reporter: Jay Kreps
Assignee: Tong Li
  Labels: newbie, patch
 Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, 
 KAFKA-1926.patch, KAFKA-1926_2015-04-01_22:16:46.patch, 
 KAFKA-1926_2015-04-05_23:45:13.patch


 There is currently a lot of duplication between the Utils class in common and 
 the one in core.
 Our plan has been to deprecate duplicate code in the server and replace it 
 with the new common code.
 As such we should evaluate each method in the scala Utils and do one of the 
 following:
 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
 utility in active use that is not Kafka-specific. If we migrate it we should 
 really think about the API and make sure there is some test coverage. A few 
 things in there are kind of funky and we shouldn't just blindly copy them 
 over.
 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
 any utilities that really need to make use of Scala features to be convenient.
 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14480906#comment-14480906
 ] 

Tong Li commented on KAFKA-1926:


Updated reviewboard https://reviews.apache.org/r/32740/
 against branch origin/trunk

 Replace kafka.utils.Utils with o.a.k.common.utils.Utils
 ---

 Key: KAFKA-1926
 URL: https://issues.apache.org/jira/browse/KAFKA-1926
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.8.2.0
Reporter: Jay Kreps
Assignee: Tong Li
  Labels: newbie, patch
 Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, 
 KAFKA-1926.patch, KAFKA-1926_2015-04-01_22:16:46.patch, 
 KAFKA-1926_2015-04-05_23:45:13.patch


 There is currently a lot of duplication between the Utils class in common and 
 the one in core.
 Our plan has been to deprecate duplicate code in the server and replace it 
 with the new common code.
 As such we should evaluate each method in the scala Utils and do one of the 
 following:
 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
 utility in active use that is not Kafka-specific. If we migrate it we should 
 really think about the API and make sure there is some test coverage. A few 
 things in there are kind of funky and we shouldn't just blindly copy them 
 over.
 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
 any utilities that really need to make use of Scala features to be convenient.
 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review request at https://reviews.apache.org/r/32740/

[Tong Li]

Jun and Jay,
 Have addressed all the comments and put up another patch set. Can you
please take a look again? Really appreciate it.

Thanks.

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com

Re: Review request at https://reviews.apache.org/r/32740/

[Tong Li]

Thanks so much. New patch set come up after rebase.

Sent from my iPhone

 On Apr 3, 2015, at 7:50 PM, Jun Rao j...@confluent.io wrote:

 Reviewed.

 Thanks,

 Jun

 On Fri, Apr 3, 2015 at 10:38 AM, Tong Li liton...@us.ibm.com wrote:

  Jun and Jay,
   Have addressed all the comments and put up another patch set. Can
you
  please take a look again? Really appreciate it.
 
  Thanks.
 
  Tong Li
  OpenStack  Kafka Community Development
  Building 501/B205
  liton...@us.ibm.com

Re: Kafka meetup at ApacheCon in Apr

[Tong Li]

Jun,
 Thanks for the information, I am trying to create a travel request to
be there to meet you guys. But I need some information on who and what
company the participators will represent. I wonder if you can give me few
names and their company name so that I can put in my travel request?

Thanks.

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Jun Rao j...@confluent.io
To: dev@kafka.apache.org dev@kafka.apache.org,
us...@kafka.apache.org us...@kafka.apache.org,
kafka-clie...@googlegroups.com
kafka-clie...@googlegroups.com
Date:   04/02/2015 02:20 PM
Subject:Kafka meetup at ApacheCon in Apr



Hi,

A few of us will be at ApacheCon in April. We are organizing a Kafka meetup
Tuesday evening (
http://events.linuxfoundation.org/events/apachecon-north-america/extend-the-experience/meetups
).
Every attendee of ApacheCon is welcome to join the meetup. We also plan to
have a a few short presentations in the meetup. If you have something that
you'd like to share, please let me know.

Thanks,

Jun

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391999#comment-14391999
 ] 

Tong Li commented on KAFKA-1926:


[~junrao]Thanks so much for the quick review. All the new comments were 
addressed in the updated patch set.. Please review when you have time. 

 Replace kafka.utils.Utils with o.a.k.common.utils.Utils
 ---

 Key: KAFKA-1926
 URL: https://issues.apache.org/jira/browse/KAFKA-1926
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.8.2.0
Reporter: Jay Kreps
Assignee: Tong Li
  Labels: newbie, patch
 Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, 
 KAFKA-1926.patch, KAFKA-1926_2015-04-01_22:16:46.patch


 There is currently a lot of duplication between the Utils class in common and 
 the one in core.
 Our plan has been to deprecate duplicate code in the server and replace it 
 with the new common code.
 As such we should evaluate each method in the scala Utils and do one of the 
 following:
 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
 utility in active use that is not Kafka-specific. If we migrate it we should 
 really think about the API and make sure there is some test coverage. A few 
 things in there are kind of funky and we shouldn't just blindly copy them 
 over.
 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
 any utilities that really need to make use of Scala features to be convenient.
 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391990#comment-14391990
 ] 

Tong Li commented on KAFKA-1926:


Updated reviewboard https://reviews.apache.org/r/32740/
 against branch origin/trunk

 Replace kafka.utils.Utils with o.a.k.common.utils.Utils
 ---

 Key: KAFKA-1926
 URL: https://issues.apache.org/jira/browse/KAFKA-1926
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.8.2.0
Reporter: Jay Kreps
Assignee: Tong Li
  Labels: newbie, patch
 Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, 
 KAFKA-1926.patch, KAFKA-1926_2015-04-01_22:16:46.patch


 There is currently a lot of duplication between the Utils class in common and 
 the one in core.
 Our plan has been to deprecate duplicate code in the server and replace it 
 with the new common code.
 As such we should evaluate each method in the scala Utils and do one of the 
 following:
 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
 utility in active use that is not Kafka-specific. If we migrate it we should 
 really think about the API and make sure there is some test coverage. A few 
 things in there are kind of funky and we shouldn't just blindly copy them 
 over.
 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
 any utilities that really need to make use of Scala features to be convenient.
 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1926:
---
Attachment: KAFKA-1926_2015-04-01_22:16:46.patch

 Replace kafka.utils.Utils with o.a.k.common.utils.Utils
 ---

 Key: KAFKA-1926
 URL: https://issues.apache.org/jira/browse/KAFKA-1926
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.8.2.0
Reporter: Jay Kreps
Assignee: Tong Li
  Labels: newbie, patch
 Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, 
 KAFKA-1926.patch, KAFKA-1926_2015-04-01_22:16:46.patch


 There is currently a lot of duplication between the Utils class in common and 
 the one in core.
 Our plan has been to deprecate duplicate code in the server and replace it 
 with the new common code.
 As such we should evaluate each method in the scala Utils and do one of the 
 following:
 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
 utility in active use that is not Kafka-specific. If we migrate it we should 
 really think about the API and make sure there is some test coverage. A few 
 things in there are kind of funky and we shouldn't just blindly copy them 
 over.
 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
 any utilities that really need to make use of Scala features to be convenient.
 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 32740: Patch for KAFKA-1926

[Tong Li]

/server/OffsetCommitTest.scala 
7654275c5437e1db319f01a8d587d66b42ea8dc7 
  core/src/test/scala/unit/kafka/server/ServerGenerateBrokerIdTest.scala 
96a8a5a8cb1f40f743490f85bf18d441ce86a765 
  core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
71317eb46b8d76060b115e6c1d31a4653459724f 
  core/src/test/scala/unit/kafka/server/ServerStartupTest.scala 
60021ef8d8d7fa172d6b160db48ee5eb9574e1a9 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
1682a77362123449de6bb1d54a55887409990a24 
  core/src/test/scala/unit/kafka/utils/UtilsTest.scala 
8c3797a964a2760a00ed60530d1a48e3da473769 
  core/src/test/scala/unit/kafka/zk/EmbeddedZookeeper.scala 
31515615089385c722325bfe019a47ae3a9a4052 
  core/src/test/scala/unit/kafka/zk/ZooKeeperTestHarness.scala 
67d9c4bab270c852dc05d47aaa4dd96b0f6039d4 

Diff: https://reviews.apache.org/r/32740/diff/


Testing
---


Thanks,

Tong Li

Review Request 32740: Patch for KAFKA-1926

[Tong Li]

 
7654275c5437e1db319f01a8d587d66b42ea8dc7 
  core/src/test/scala/unit/kafka/server/ServerGenerateBrokerIdTest.scala 
96a8a5a8cb1f40f743490f85bf18d441ce86a765 
  core/src/test/scala/unit/kafka/server/ServerShutdownTest.scala 
71317eb46b8d76060b115e6c1d31a4653459724f 
  core/src/test/scala/unit/kafka/server/ServerStartupTest.scala 
60021ef8d8d7fa172d6b160db48ee5eb9574e1a9 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
1682a77362123449de6bb1d54a55887409990a24 
  core/src/test/scala/unit/kafka/utils/UtilsTest.scala 
8c3797a964a2760a00ed60530d1a48e3da473769 
  core/src/test/scala/unit/kafka/zk/EmbeddedZookeeper.scala 
31515615089385c722325bfe019a47ae3a9a4052 
  core/src/test/scala/unit/kafka/zk/ZooKeeperTestHarness.scala 
67d9c4bab270c852dc05d47aaa4dd96b0f6039d4 

Diff: https://reviews.apache.org/r/32740/diff/


Testing
---


Thanks,

Tong Li

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14391295#comment-14391295
 ] 

Tong Li commented on KAFKA-1926:


[~junrao], all comments have been addressed in the latest patch set other than 
the readByte movement. The move of readByte from core to client will cause many 
other changes since the method uses scala.Byte, any thing uses that method will 
have to have the signature changes. Since this patch set is already huge, I 
would very much like to do that refactor in a later patch set. Thanks.

 Replace kafka.utils.Utils with o.a.k.common.utils.Utils
 ---

 Key: KAFKA-1926
 URL: https://issues.apache.org/jira/browse/KAFKA-1926
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.8.2.0
Reporter: Jay Kreps
Assignee: Tong Li
  Labels: newbie, patch
 Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, 
 KAFKA-1926.patch


 There is currently a lot of duplication between the Utils class in common and 
 the one in core.
 Our plan has been to deprecate duplicate code in the server and replace it 
 with the new common code.
 As such we should evaluate each method in the scala Utils and do one of the 
 following:
 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
 utility in active use that is not Kafka-specific. If we migrate it we should 
 really think about the API and make sure there is some test coverage. A few 
 things in there are kind of funky and we shouldn't just blindly copy them 
 over.
 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
 any utilities that really need to make use of Scala features to be convenient.
 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1926:
---
Attachment: KAFKA-1926.patch

 Replace kafka.utils.Utils with o.a.k.common.utils.Utils
 ---

 Key: KAFKA-1926
 URL: https://issues.apache.org/jira/browse/KAFKA-1926
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.8.2.0
Reporter: Jay Kreps
Assignee: Tong Li
  Labels: newbie, patch
 Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch, 
 KAFKA-1926.patch


 There is currently a lot of duplication between the Utils class in common and 
 the one in core.
 Our plan has been to deprecate duplicate code in the server and replace it 
 with the new common code.
 As such we should evaluate each method in the scala Utils and do one of the 
 following:
 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
 utility in active use that is not Kafka-specific. If we migrate it we should 
 really think about the API and make sure there is some test coverage. A few 
 things in there are kind of funky and we shouldn't just blindly copy them 
 over.
 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
 any utilities that really need to make use of Scala features to be convenient.
 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: KIP discussion Mar 24 at 11am PST

[Tong Li]

Jun,
   I would like to participate but have not received the invite. Can you
please send to me?

Thanks.

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Jun Rao j...@confluent.io
To: dev@kafka.apache.org dev@kafka.apache.org
Date:   03/27/2015 10:35 PM
Subject:Re: KIP discussion Mar 24 at 11am PST



Just sent out an invite for Mar 31 at 11am PST. The following is the
tentative agenda. Let me know if you want to attend, but haven't received
an invite.

Agenda:
KIP-4 (admin commands):
* wrap up any remaining issues

KIP-13 (quota):
* dependency on using the new metrics package
* dependency KIP-5 (broker configuration)

Perhaps security related KIPS.

Thanks,

Jun

On Fri, Mar 27, 2015 at 6:30 PM, Joel Koshy jjkosh...@gmail.com wrote:

 I think we decided that we should reconvene next Tuesday to collect
 everyone's thoughts on metrics and discuss other KIPs. Can everyone
 still do Tuesday or do people feel we need to push that further out by
 a few days?

 Thanks,

 Joel

 On Tue, Mar 24, 2015 at 12:28:04PM -0700, Jun Rao wrote:
  Just to keep everyone posted. The following is a summary of what's
being
  discussed in the KIP hangout today.
 
  KIP-4 (admin commands):
  * Gwen is uploading a patch in KAFKA-1927 (refactoring requests) so
that
 we
  can get unblocked of adding new requests.
  * We will combine DescribeTopic and TopicMetadata in the future.
  * We will leave the admin requests async for now.
  * We will not add a VerifyReassignPartitionRequest for now. We can do
 that
  later when we improve the verification process.
  * We need to discuss a bit more on how to expose the controller info to
 the
  client.
  * Andrii will send out more details on the KIP thread.
 
  KIP-15 (close):
  * If close() or close with a non-zero timeout is called from the send
  thread, we will log it as an error.
  * Jiangjie will follow up on the KIP thread.
 
  KIP-13 (quota):
  * Need a separate discussion on whether to use the new metrics package
on
  the broker on the mailing list.
  * There are a few other details being discuss and Aditya will follow up
 on
  the KIP thread.
 
  Thanks,
 
  Jun
 
 
  On Fri, Mar 20, 2015 at 2:44 PM, Jun Rao j...@confluent.io wrote:
 
   Hi, Everyone,
  
   We plan to have a KIP discussion on Google hangout on Mar 24 at 11am
 PST.
   If you are interested in participating and have not already received
a
   calendar invitation, please let me know. The following is the agenda.
  
   KIP-4 (admin commands): 10 mins
   * status of KAFKA-1927 (refactoring requests). which blocks this KIP
   * status of KAFKA-1634 (improve OffsetCommitRequest), which blocks
   KAFKA-1927
   * any remaining issues for discussion
  
   KIP-15 (close): 10 mins
   * semantics of close() and close(timeout)
  
   KIP-13 (quota):
   * protocol change to reflect the state of throttling
   * dependency on using the new metrics package
   * dependency KIP-5 (broker configuration)
  
   Thanks,
  
   Jun
  

Re: Dropping support for Scala 2.9.x

[Tong Li]

+1. But can we also look at this from the deployment base point view and
find out how many production deployments are still using 2.9? If there is
not any, dropping it is really an easy decision.

Thanks

Sent from my iPhone

 On Mar 27, 2015, at 8:21 AM, Ismael Juma mli...@juma.me.uk wrote:

 Hi all,

 The Kafka build currently includes support for Scala 2.9, which means
that
 it cannot take advantage of features introduced in Scala 2.10 or depend
on
 libraries that require it.

 This restricts the solutions available while trying to solve existing
 issues. I was browsing JIRA looking for areas to contribute and I quickly
 ran into two issues where this is the case:

 * KAFKA-1351: String.format is very expensive in Scala could be solved
 nicely by using the String interpolation feature introduced in Scala
2.10.

 * KAFKA-1595: Remove deprecated and slower scala JSON parser from
 kafka.consumer.TopicCount could be solved by using an existing JSON
 library, but both jackson-scala and play-json require 2.10 (argonaut
 supports Scala 2.9, but it brings other dependencies like scalaz). We can
 workaround this by writing our own code instead of using libraries, of
 course, but it's not ideal.

 Other features like Scala Futures and value classes would also be useful
in
 some situations, I would think (for a more extensive list of new
features,
 see

http://scala-language.1934581.n4.nabble.com/Scala-2-10-0-now-available-td4634126.html

 ).

 Another pain point of supporting 2.9.x is that it doubles the number of
 build and test configurations required from 2 to 4 (because the 2.9.x
 series was not necessarily binary compatible).

 A strong argument for maintaining support for 2.9.x was the client
library,
 but that has been rewritten in Java.

 It's also worth mentioning that Scala 2.9.1 was released in August 2011
 (more than 3.5 years ago) and the 2.9.x series hasn't received updates of
 any sort since early 2013. Scala 2.10.0, in turn, was released in January
 2013 (over 2 years ago) and 2.10.5, the last planned release in the
2.10.x
 series, has been recently released (so even 2.10.x won't be receiving
 updates any longer).

 All in all, I think it would not be unreasonable to drop support for
Scala
 2.9.x in a future release, but I may be missing something. What do others
 think?

 Ismael

Re: Dropping support for Scala 2.9.x

[Tong Li]

+1. But can we also look at this from the deployment base point view and
find out how many production deployments are still using 2.9? If there is
not any, dropping it is really an easy decision.

Thanks

Sent from my iPhone

 On Mar 27, 2015, at 8:21 AM, Ismael Juma mli...@juma.me.uk wrote:

 Hi all,

 The Kafka build currently includes support for Scala 2.9, which means
that
 it cannot take advantage of features introduced in Scala 2.10 or depend
on
 libraries that require it.

 This restricts the solutions available while trying to solve existing
 issues. I was browsing JIRA looking for areas to contribute and I quickly
 ran into two issues where this is the case:

 * KAFKA-1351: String.format is very expensive in Scala could be solved
 nicely by using the String interpolation feature introduced in Scala
2.10.

 * KAFKA-1595: Remove deprecated and slower scala JSON parser from
 kafka.consumer.TopicCount could be solved by using an existing JSON
 library, but both jackson-scala and play-json require 2.10 (argonaut
 supports Scala 2.9, but it brings other dependencies like scalaz). We can
 workaround this by writing our own code instead of using libraries, of
 course, but it's not ideal.

 Other features like Scala Futures and value classes would also be useful
in
 some situations, I would think (for a more extensive list of new
features,
 see

http://scala-language.1934581.n4.nabble.com/Scala-2-10-0-now-available-td4634126.html

 ).

 Another pain point of supporting 2.9.x is that it doubles the number of
 build and test configurations required from 2 to 4 (because the 2.9.x
 series was not necessarily binary compatible).

 A strong argument for maintaining support for 2.9.x was the client
library,
 but that has been rewritten in Java.

 It's also worth mentioning that Scala 2.9.1 was released in August 2011
 (more than 3.5 years ago) and the 2.9.x series hasn't received updates of
 any sort since early 2013. Scala 2.10.0, in turn, was released in January
 2013 (over 2 years ago) and 2.10.5, the last planned release in the
2.10.x
 series, has been recently released (so even 2.10.x won't be receiving
 updates any longer).

 All in all, I think it would not be unreasonable to drop support for
Scala
 2.9.x in a future release, but I may be missing something. What do others
 think?

 Ismael

java.net.BindException: Address already in use

[Tong Li]

Hi guys, when I ran test, I got a lot of these exceptions. I wonder if you
had similar problems running the tests and how resolved the issue.

Thanks.

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14363164#comment-14363164
 ] 

Tong Li commented on KAFKA-1926:


@Jun Rao, awesome comments. I will be following the directions and provide new 
patch set. This also confirms the direction that I am going. Thanks.

 Replace kafka.utils.Utils with o.a.k.common.utils.Utils
 ---

 Key: KAFKA-1926
 URL: https://issues.apache.org/jira/browse/KAFKA-1926
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.8.2.0
Reporter: Jay Kreps
  Labels: newbie, patch
 Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch


 There is currently a lot of duplication between the Utils class in common and 
 the one in core.
 Our plan has been to deprecate duplicate code in the server and replace it 
 with the new common code.
 As such we should evaluate each method in the scala Utils and do one of the 
 following:
 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
 utility in active use that is not Kafka-specific. If we migrate it we should 
 really think about the API and make sure there is some test coverage. A few 
 things in there are kind of funky and we shouldn't just blindly copy them 
 over.
 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
 any utilities that really need to make use of Scala features to be convenient.
 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-4 - Command line and centralized administrative operations

[Tong Li]

Guozhang,
 augmenting topic is fine, but as soon as we start doing that, other
issues follow, for example, access control, who can access the topic, who
can grant permissions. how the information (metadata) itself gets secured.
Should the information be saved in ZK or a datastore? Will using a metadata
file causing long term problems such as file updates/synchronization, once
we have this metadata file, more people will want to put more stuff in it.
how can we control the format? K-V pair not good for large data set.
Clearly there is a need for it, I wonder if we can make this thing
plugable and provide a default implementation which allows us try different
solutions and also allow people to completely ignore it if they do not want
to deal with any of these.

Thanks.

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Guozhang Wang wangg...@gmail.com
To: dev@kafka.apache.org dev@kafka.apache.org
Date:   03/12/2015 09:39 AM
Subject:Re: [DISCUSS] KIP-4 - Command line and centralized
administrative operations



Folks,

Just want to elaborate a bit more on the create-topic metadata and batching
describe-topic based on config / metadata in my previous email as we work
on KAFKA-1694. The main motivation is to have some sort of topic management
mechanisms, which I think is quite important in a multi-tenant / cloud
architecture: today anyone can create topics in a shared Kafka cluster, but
there is no concept or ownership of topics that are created by different
users. For example, at LinkedIn we basically distinguish topic owners via
some casual topic name prefix, which is a bit awkward and does not fly as
we scale our customers. It would be great to use describe-topics such as:

Describe all topics that is created by me.

Describe all topics whose retention time is overriden to X.

Describe all topics whose writable group include user Y (this is related to
authorization), etc..

One possible way to achieve this is to add a metadata file in the
create-topic request, whose value will also be written ZK as we create the
topic; then describe-topics can choose to batch topics based on 1) name
regex, 2) config K-V matching, 3) metadata regex, etc.

Thoughts?

Guozhang

On Thu, Mar 5, 2015 at 4:37 PM, Guozhang Wang wangg...@gmail.com wrote:

 Thanks for the updated wiki. A few comments below:

 1. Error description in response: I think if some errorCode could
indicate
 several different error cases then we should really change it to multiple
 codes. In general the errorCode itself would be precise and sufficient
for
 describing the server side errors.

 2. Describe topic request: it would be great to go beyond just batching
on
 topic name regex for this request. For example, a very common use case of
 the topic command is to list all topics whose config A's value is B. With
 topic name regex then we have to first retrieve __all__ topics's
 description info and then filter at the client end, which will be a huge
 burden on ZK.

 3. Config K-Vs in create topic: this is related to the previous point;
 maybe we can add another metadata K-V or just a metadata string along
side
 with config K-V in create topic like we did for offset commit request.
This
 field can be quite useful in storing information like owner of the
topic
 who issue the create command, etc, which is quite important for a
 multi-tenant setting. Then in the describe topic request we can also
batch
 on regex of the metadata field.

 4. Today all the admin operations are async in the sense that command
will
 return once it is written in ZK, and that is why we need extra
verification
 like testUtil.waitForTopicCreated() / verify partition reassignment
 request, etc. With admin requests we could add a flag to enable / disable
 synchronous requests; when it is turned on, the response will not return
 until the request has been completed. And for async requests we can add a
 token field in the response, and then only need a general admin
 verification request with the given token to check if the async request
 has been completed.

 5. +1 for extending Metadata request to include controller / coordinator
 information, and then we can remove the ConsumerMetadata /
ClusterMetadata
 requests.

 Guozhang

 On Tue, Mar 3, 2015 at 10:23 AM, Joel Koshy jjkosh...@gmail.com wrote:

 Thanks for sending that out Joe - I don't think I will be able to make
 it today, so if notes can be sent out afterward that would be great.

 On Mon, Mar 02, 2015 at 09:16:13AM -0800, Gwen Shapira wrote:
  Thanks for sending this out Joe. Looking forward to chatting with
 everyone :)
 
  On Mon, Mar 2, 2015 at 6:46 AM, Joe Stein joe.st...@stealth.ly
wrote:
   Hey, I just sent out a google hangout invite to all pmc, committers
 and
   everyone I found working on a KIP. If I missed anyone in the invite
 please
   let me know and can update it, np.
  
   We should do this every Tuesday @ 2pm Eastern Time. Maybe we can

Review Request 31844: Patch for KAFKA-1926

[Tong Li]

/test/scala/unit/kafka/zk/EmbeddedZookeeper.scala 
31515615089385c722325bfe019a47ae3a9a4052 
  core/src/test/scala/unit/kafka/zk/ZooKeeperTestHarness.scala 
67d9c4bab270c852dc05d47aaa4dd96b0f6039d4 

Diff: https://reviews.apache.org/r/31844/diff/


Testing
---


Thanks,

Tong Li

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14352488#comment-14352488
 ] 

Tong Li commented on KAFKA-1926:


Created reviewboard https://reviews.apache.org/r/31844/
 against branch origin/trunk

 Replace kafka.utils.Utils with o.a.k.common.utils.Utils
 ---

 Key: KAFKA-1926
 URL: https://issues.apache.org/jira/browse/KAFKA-1926
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.8.2.0
Reporter: Jay Kreps
  Labels: newbie, patch
 Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch


 There is currently a lot of duplication between the Utils class in common and 
 the one in core.
 Our plan has been to deprecate duplicate code in the server and replace it 
 with the new common code.
 As such we should evaluate each method in the scala Utils and do one of the 
 following:
 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
 utility in active use that is not Kafka-specific. If we migrate it we should 
 really think about the API and make sure there is some test coverage. A few 
 things in there are kind of funky and we shouldn't just blindly copy them 
 over.
 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
 any utilities that really need to make use of Scala features to be convenient.
 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1926:
---
Attachment: KAFKA-1926.patch

 Replace kafka.utils.Utils with o.a.k.common.utils.Utils
 ---

 Key: KAFKA-1926
 URL: https://issues.apache.org/jira/browse/KAFKA-1926
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.8.2.0
Reporter: Jay Kreps
  Labels: newbie, patch
 Attachments: KAFKA-1926.patch, KAFKA-1926.patch, KAFKA-1926.patch


 There is currently a lot of duplication between the Utils class in common and 
 the one in core.
 Our plan has been to deprecate duplicate code in the server and replace it 
 with the new common code.
 As such we should evaluate each method in the scala Utils and do one of the 
 following:
 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
 utility in active use that is not Kafka-specific. If we migrate it we should 
 really think about the API and make sure there is some test coverage. A few 
 things in there are kind of funky and we shouldn't just blindly copy them 
 over.
 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
 any utilities that really need to make use of Scala features to be convenient.
 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

problems submitting patch set for review and a bit of solution

[Tong Li]

While trying to submit a reviewboard request by running kafka-patch-review,
I was getting some errors which do not make any sense. The error claims
that a file did not exist in the repository but it did. Here is the error:


1. How the problems look like:
tong@u1404:/opt/stack/kafka$ python kafka-patch-review.py -b
origin/trunk -j KAFKA-1926
Configuring reviewboard url to https://reviews.apache.org
Updating your remote branches to pull the latest changes
Verifying JIRA connection configurations
ERROR: Error validating diff

core/src/main/scala/kafka/utils/Utils.scala: The file was not found
in the repository. (HTTP 400, API Error 207)
ERROR: reviewboard update failed. Exiting.


2. Struggled a lot trying to figure out what the problem was. after tons of
googling, I decided to upgrading python-rbtools

tong@u1404:/opt/stack/kafka$ sudo apt-get install python-rbtools
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following NEW packages will be installed:
  python-rbtools
0 upgraded, 1 newly installed, 0 to remove and 163 not upgraded.
Need to get 48.6 kB of archives.
After this operation, 269 kB of additional disk space will be used.
Get:1 http://us.archive.ubuntu.com/ubuntu/ trusty/universe
python-rbtools all 0.3.4-1 [48.6 kB]
Fetched 48.6 kB in 3s (13.3 kB/s)
Selecting previously unselected package python-rbtools.
(Reading database ... 57284 files and directories currently
installed.)
Preparing to unpack .../python-rbtools_0.3.4-1_all.deb ...
Unpacking python-rbtools (0.3.4-1) ...
Processing triggers for man-db (2.6.7.1-1) ...
Setting up python-rbtools (0.3.4-1) ...

3. Tried to run patch-review again. But the command stuck and seemed to me
it waits for some credentials. and it completely ignored the jira.ini file
4. Tried the command again, and it successfully posted patchset and updated
the jira issue after I blindly type in my user name, then provided
password:

tong@u1404:/opt/stack/kafka$ python kafka-patch-review.py -b
origin/trunk -j KAFKA-1926
tongli
Password:
Configuring reviewboard url to https://reviews.apache.org
Updating your remote branches to pull the latest changes
Verifying JIRA connection configurations
Review request #31844 posted.

https://reviews.apache.org/r/31844/

Creating diff against origin/trunk and uploading patch to JIRA
KAFKA-1926
Created a new reviewboard https://reviews.apache.org/r/31844/
5. it may be good to revisit the kafka-patch-review.py and see why it
ignores the jira.ini file.

When I posted the problem few days ago, I got no response from anyone, I
thought this maybe a problem that other may not have encountered. sending
to the mailing list in case anyone else stumbles on the same issue.

cheers.

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com

Error when submit a patch set which contains a file rename.

[Tong Li]

Hi, I was trying to submit a patch set for review. In this patch set, I
removed one file and created another to replace the removed. However, when
I run kafka-patch-review.py I got an error.

python kafka-patch-review.py -b origin/trunk -j KAFKA-1926
Configuring reviewboard url to https://reviews.apache.org
Updating your remote branches to pull the latest changes
Verifying JIRA connection configurations
ERROR: Error validating diff

core/src/main/scala/kafka/utils/Utils.scala: The file was not found in the
repository. (HTTP 400, API Error 207)
ERROR: reviewboard update failed. Exiting.

I wonder if it is because I do not have enough of permission? the file to
be renamed is clearly there. Any help will be appreciated.

Thanks.

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14348038#comment-14348038
 ] 

Tong Li commented on KAFKA-1926:


Jay, I finally put together a new patch set just for this issue. (the 
SystemTime issue we can address in a separate issue since this patch set is 
already very big). In the new patch set, I will have the following changes made:

This patch set is an attempt to refactor core utils class and remove
the namespace overlap. Utils class was defined by both clients and core
projects. Many methods are duplicate. This patch did the following:

1. Renames the core kafka.utils.Utils class to be kafka.utils.CoreUtils
2. Removed the abs method in kafka.utils.CoreUtils, anywhere using abs
   method will be replaced with the method in o.a.k.c.u.Utils.abs
3. Removed the Crc32 class in kafka.utils package. Also use the one
   defined in o.a.k.c.u.Utils class.
4. Removed asString method in CoreUtils since this was not used.
5. Removed ReadUnisnedInt (two methods), WriteUnsignedInt (two methods) 
since
   they are completely duplicate of the client methods, both signature and
   implementation are absolutely identical.

I will submit the patch set for you to review. Since this patch set touches so 
many
files, very time consuming. if we can get this in faster, then I won't need to 
do so
many rounds of rebase. Thanks so much.




 Replace kafka.utils.Utils with o.a.k.common.utils.Utils
 ---

 Key: KAFKA-1926
 URL: https://issues.apache.org/jira/browse/KAFKA-1926
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.8.2.0
Reporter: Jay Kreps
  Labels: newbie, patch
 Attachments: KAFKA-1926.patch, KAFKA-1926.patch


 There is currently a lot of duplication between the Utils class in common and 
 the one in core.
 Our plan has been to deprecate duplicate code in the server and replace it 
 with the new common code.
 As such we should evaluate each method in the scala Utils and do one of the 
 following:
 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
 utility in active use that is not Kafka-specific. If we migrate it we should 
 really think about the API and make sure there is some test coverage. A few 
 things in there are kind of funky and we shouldn't just blindly copy them 
 over.
 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
 any utilities that really need to make use of Scala features to be convenient.
 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-6 - New reassignment partition logic for re-balancing

[Tong Li]

Todd,
I think plugable design is good with solid default. The only issue I
feel is when you use one and switch to another, will we end up with some
unread messages hanging around and no one thinks or knows it is their
responsibility to take care of them?

Thanks.

Tong

Sent from my iPhone

 On Mar 5, 2015, at 10:46 AM, Todd Palino tpal...@gmail.com wrote:

 Apologize for the late comment on this...

 So fair assignment by count (taking into account the current partition
 count of each broker) is very good. However, it's worth noting that all
 partitions are not created equal. We have actually been performing more
 rebalance work based on the partition size on disk, as given equal
 retention of all topics, the size on disk is a better indicator of the
 amount of traffic a partition gets, both in terms of storage and network
 traffic. Overall, this seems to be a better balance.

 In addition to this, I think there is very much a need to have Kafka be
 rack-aware. That is, to be able to assure that for a given cluster, you
 never assign all replicas for a given partition in the same rack. This
 would allow us to guard against maintenances or power failures that
affect
 a full rack of systems (or a given switch).

 I think it would make sense to implement the reassignment logic as a
 pluggable component. That way it would be easy to select a scheme when
 performing a reassignment (count, size, rack aware). Configuring a
default
 scheme for a cluster would allow for the brokers to create new topics and
 partitions in compliance with the requested policy.

 -Todd


 On Thu, Jan 22, 2015 at 10:13 PM, Joe Stein joe.st...@stealth.ly wrote:

  I will go back through the ticket and code and write more up. Should be
  able to-do that sometime next week. The intention was to not replace
  existing functionality by issue a WARN on use. The following version it
is
  released we could then deprecate it... I will fix the KIP for that too.
 
  On Fri, Jan 23, 2015 at 12:34 AM, Neha Narkhede n...@confluent.io
wrote:
 
   Hey Joe,
  
   1. Could you add details to the Public Interface section of the KIP?
This
   should include the proposed changes to the partition reassignment
tool.
   Also, maybe the new option can be named --rebalance instead of
   --re-balance?
   2. It makes sense to list --decommission-broker as part of this KIP.
   Similarly, shouldn't we also have an --add-broker option? The way I
see
   this is that there are several events when a partition reassignment
is
   required. Before this functionality is automated on the broker, the
tool
   will generate an ideal replica placement for each such event. The
users
   should merely have to specify the nature of the event e.g. adding a
  broker
   or decommissioning an existing broker or merely rebalancing.
   3. If I understand the KIP correctly, the upgrade plan for this
feature
   includes removing the existing --generate option on the partition
   reassignment tool in 0.8.3 while adding all the new options in the
same
   release. Is that correct?
  
   Thanks,
   Neha
  
   On Thu, Jan 22, 2015 at 9:23 PM, Jay Kreps jay.kr...@gmail.com
wrote:
  
Ditto on this one. Can you give the algorithm we want to implement?
   
Also I think in terms of scope this is just proposing to change the
  logic
in ReassignPartitionsCommand? I think we've had the discussion
various
times on the mailing list that what people really want is just for
  Kafka
   to
do it's best to balance data in an online fashion (for some
definition
  of
balance). i.e. if you add a new node partitions would slowly
migrate to
   it,
and if a node dies, partitions slowly migrate off it. This could
potentially be more work, but I'm not sure how much more. Has
anyone
thought about how to do it?
   
-Jay
   
On Wed, Jan 21, 2015 at 10:11 PM, Joe Stein joe.st...@stealth.ly
   wrote:
   
 Posted a KIP for --re-balance for partition assignment in
  reassignment
 tool.



   
  
  https://cwiki.apache.org/confluence/display/KAFKA/KIP-6+-+New
+reassignment+partition+logic+for+re-balancing

 JIRA https://issues.apache.org/jira/browse/KAFKA-1792

 While going through the KIP I thought of one thing from the JIRA
that
   we
 should change. We should preserve --generate to be existing
   functionality
 for the next release it is in. If folks want to use --re-balance
then
 great, it just won't break any upgrade paths, yet.

 /***
  Joe Stein
  Founder, Principal Consultant
  Big Data Open Source Security LLC
  http://www.stealth.ly
  Twitter: @allthingshadoop
http://www.twitter.com/allthingshadoop
 /

   
  
  
  
   --
   Thanks,
   Neha
  
 

Re: Review Request 31566: Patch for KAFKA-1988

[Tong Li]

Yes. It was addressed in the latest patch set. A private method called 
toPositive was added so that the partition selection is preserved cross this 
and previous versions.

Thanks

Sent from my iPhone

 On Mar 4, 2015, at 12:49 AM, Guozhang Wang wangg...@gmail.com wrote:
 
 
 This is an automatically generated e-mail. To reply, visit: 
 https://reviews.apache.org/r/31566/
 
 Tong, could you address Jun's last comments before committing?
 
 - Guozhang Wang
 
 
 On February 27th, 2015, 11:16 p.m. UTC, Tong Li wrote:
 
 Review request for kafka.
 By Tong Li.
 Updated Feb. 27, 2015, 11:16 p.m.
 
 Bugs: KAFKA-1988
 Repository: kafka
 Description
 
 KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns wrong value 
 for negative numbers
 Diffs
 
 clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java
  (dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8)
 clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
 (69530c187cd1c41b8173b61de6f982aafe65c9fe)
 clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 
 (4c2ea34815b63174732d58b699e1a0a9e6ec3b6f)
 View Diff

Re: Review Request 31566: Patch for KAFKA-1988

[Tong Li]

 On March 3, 2015, 4:48 p.m., Guozhang Wang wrote:
  Tong, could you address Jun's last comments before committing?

Yes. absolutely, doing it now. Thanks.


- Tong


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31566/#review74973
---


On Feb. 27, 2015, 11:16 p.m., Tong Li wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/31566/
 ---
 
 (Updated Feb. 27, 2015, 11:16 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1988
 https://issues.apache.org/jira/browse/KAFKA-1988
 
 
 Repository: kafka
 
 
 Description
 ---
 
 KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns wrong value 
 for negative numbers
 
 
 Diffs
 -
 
   
 clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java
  dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8 
   clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
 69530c187cd1c41b8173b61de6f982aafe65c9fe 
   clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 
 4c2ea34815b63174732d58b699e1a0a9e6ec3b6f 
 
 Diff: https://reviews.apache.org/r/31566/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Tong Li
 

[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1988:
---
Attachment: KAFKA-1988_2015-03-03_19:00:21.patch

 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers.
 

 Key: KAFKA-1988
 URL: https://issues.apache.org/jira/browse/KAFKA-1988
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.0
Reporter: Tong Li
Assignee: Tong Li
Priority: Blocker
 Fix For: 0.8.2.1

 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch, 
 KAFKA-1988_2015-03-03_19:00:21.patch


 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers. The method only returns intended value for positive 
 numbers. All negative numbers except the Integer.Min_Value will be returned 
 an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346038#comment-14346038
 ] 

Tong Li commented on KAFKA-1988:


Updated reviewboard https://reviews.apache.org/r/31566/diff/
 against branch origin/trunk

 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers.
 

 Key: KAFKA-1988
 URL: https://issues.apache.org/jira/browse/KAFKA-1988
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.0
Reporter: Tong Li
Assignee: Tong Li
Priority: Blocker
 Fix For: 0.8.2.1

 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch, 
 KAFKA-1988_2015-03-03_19:00:21.patch


 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers. The method only returns intended value for positive 
 numbers. All negative numbers except the Integer.Min_Value will be returned 
 an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31566: Patch for KAFKA-1988

[Tong Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31566/
---

(Updated March 4, 2015, midnight)


Review request for kafka.


Bugs: KAFKA-1988
https://issues.apache.org/jira/browse/KAFKA-1988


Repository: kafka


Description
---

KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns wrong value 
for negative numbers


Diffs (updated)
-

  
clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java
 dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8 
  clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
69530c187cd1c41b8173b61de6f982aafe65c9fe 
  clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 
4c2ea34815b63174732d58b699e1a0a9e6ec3b6f 

Diff: https://reviews.apache.org/r/31566/diff/


Testing
---


Thanks,

Tong Li

Re: Review Request 31566: Patch for KAFKA-1988

[Tong Li]

Guozhang,
  Yes, I have rebased and updated the patch set to resolve the couple
comments for code comments. Also removed the trailing spaces (which always
annoying. :-) ), Please see the new patch set here. Sorry for the delay.

https://reviews.apache.org/r/31566/

Thanks.

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com

Guozhang Wang nore...@reviews.apache.org wrote on 03/03/2015 11:48:36
AM:

 From: Guozhang Wang wangg...@gmail.com
 To: kafka dev@kafka.apache.org, Guozhang Wang
 wangg...@gmail.com, Tong Li/Raleigh/IBM@IBMUS
 Date: 03/03/2015 11:49 AM
 Subject: Re: Review Request 31566: Patch for KAFKA-1988
 Sent by: Guozhang Wang nore...@reviews.apache.org


 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/31566/#review74973
 ---


 Tong, could you address Jun's last comments before committing?

 - Guozhang Wang


 On Feb. 27, 2015, 11:16 p.m., Tong Li wrote:
 
  ---
  This is an automatically generated e-mail. To reply, visit:
  https://reviews.apache.org/r/31566/
  ---
 
  (Updated Feb. 27, 2015, 11:16 p.m.)
 
 
  Review request for kafka.
 
 
  Bugs: KAFKA-1988
  https://issues.apache.org/jira/browse/KAFKA-1988
 
 
  Repository: kafka
 
 
  Description
  ---
 
  KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns
 wrong value for negative numbers
 
 
  Diffs
  -
 
clients/src/main/java/org/apache/kafka/clients/producer/
 internals/Partitioner.java dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8
clients/src/main/java/org/apache/kafka/common/utils/Utils.java
 69530c187cd1c41b8173b61de6f982aafe65c9fe
clients/src/test/java/org/apache/kafka/common/utils/
 UtilsTest.java 4c2ea34815b63174732d58b699e1a0a9e6ec3b6f
 
  Diff: https://reviews.apache.org/r/31566/diff/
 
 
  Testing
  ---
 
 
  Thanks,
 
  Tong Li
 
 

[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1988:
---
Attachment: KAFKA-1988_2015-03-03_19:03:31.patch

 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers.
 

 Key: KAFKA-1988
 URL: https://issues.apache.org/jira/browse/KAFKA-1988
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.0
Reporter: Tong Li
Assignee: Tong Li
Priority: Blocker
 Fix For: 0.8.2.1

 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch, 
 KAFKA-1988_2015-03-03_19:00:21.patch, KAFKA-1988_2015-03-03_19:03:31.patch


 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers. The method only returns intended value for positive 
 numbers. All negative numbers except the Integer.Min_Value will be returned 
 an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14346045#comment-14346045
 ] 

Tong Li commented on KAFKA-1988:


Updated reviewboard https://reviews.apache.org/r/31566/diff/
 against branch origin/trunk

 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers.
 

 Key: KAFKA-1988
 URL: https://issues.apache.org/jira/browse/KAFKA-1988
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.0
Reporter: Tong Li
Assignee: Tong Li
Priority: Blocker
 Fix For: 0.8.2.1

 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch, 
 KAFKA-1988_2015-03-03_19:00:21.patch, KAFKA-1988_2015-03-03_19:03:31.patch


 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers. The method only returns intended value for positive 
 numbers. All negative numbers except the Integer.Min_Value will be returned 
 an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31566: Patch for KAFKA-1988

[Tong Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31566/
---

(Updated March 4, 2015, 12:03 a.m.)


Review request for kafka.


Bugs: KAFKA-1988
https://issues.apache.org/jira/browse/KAFKA-1988


Repository: kafka


Description
---

KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns wrong value 
for negative numbers


Diffs (updated)
-

  
clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java
 dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8 
  clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
69530c187cd1c41b8173b61de6f982aafe65c9fe 
  clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 
4c2ea34815b63174732d58b699e1a0a9e6ec3b6f 

Diff: https://reviews.apache.org/r/31566/diff/


Testing
---


Thanks,

Tong Li

Fwd: patch set 1988

[Tong Li]

Folks, 
 Do not want to nag you, but wonder if any of you has couple of minutes to 
 review patch set for 1988 again so that I do not have to rebase this so many 
 times. Guozhang already +1ed(thanks Guozhang!) Here are the links for your 
 convenience. 
 
 The issue
 https://issues.apache.org/jira/browse/KAFKA-1988
 
 The patch set
 https://reviews.apache.org/r/31566/diff/
 
 
 Tong Li
 OpenStack  Kafka Community Development
 Building 501/B205
 liton...@us.ibm.com

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14343109#comment-14343109
 ] 

Tong Li commented on KAFKA-1988:


[~guozhang] Yes, the o.a.k.c.c.u.Utils.abs used in few places.  in patch set 
for issue 1926, I will consolidate both Utils modules from clients and core 
into one. So that we do not have name conflict all over the place. The patch 
set for issue 1926 will be quite big. I would like to get this thing fixed for 
coming up release first, then we can address issue 1926. Thanks.

 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers.
 

 Key: KAFKA-1988
 URL: https://issues.apache.org/jira/browse/KAFKA-1988
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.0
Reporter: Tong Li
Assignee: Tong Li
Priority: Blocker
 Fix For: 0.8.2.1

 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch


 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers. The method only returns intended value for positive 
 numbers. All negative numbers except the Integer.Min_Value will be returned 
 an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1988:
---
Attachment: KAFKA-1988.patch

 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers.
 

 Key: KAFKA-1988
 URL: https://issues.apache.org/jira/browse/KAFKA-1988
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.0
Reporter: Tong Li
Assignee: Tong Li
Priority: Blocker
 Fix For: 0.8.2.1

 Attachments: KAFKA-1988.patch, KAFKA-1988.patch


 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers. The method only returns intended value for positive 
 numbers. All negative numbers except the Integer.Min_Value will be returned 
 an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339921#comment-14339921
 ] 

Tong Li commented on KAFKA-1988:


Created reviewboard https://reviews.apache.org/r/31533/diff/
 against branch origin/trunk

 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers.
 

 Key: KAFKA-1988
 URL: https://issues.apache.org/jira/browse/KAFKA-1988
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.0
Reporter: Tong Li
Assignee: Tong Li
Priority: Blocker
 Fix For: 0.8.2.1

 Attachments: KAFKA-1988.patch, KAFKA-1988.patch


 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers. The method only returns intended value for positive 
 numbers. All negative numbers except the Integer.Min_Value will be returned 
 an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339924#comment-14339924
 ] 

Tong Li commented on KAFKA-1988:


Jun Rao,
Thanks for taking time reviewing the patch. Would like to confirm here 
is what you suggested to do:

1. Utils.abs will be renamed to Utils.toPositive
2. Make changes in these methods of class DefaultPartitioner and 
ByteArrayPartitioner so that toPositive gets called in partition method.

def partition(key: Any, numPartitions: Int): Int = {
Utils.abs(key.hashCode) % numPartitions
}


I can certainly make the above changes.  however method abs gets used 
many other places and making the case worse is that there are two abs methods 
both in core and clients. I found this problem when I was in an effort of 
addressing issue 1926. For now, what if I keep that method in clients, remove 
the one in core Utils module and making changes to all other modules where abs 
method gets called so that all use of abs method will point to the clients 
Utils.abs instead of two locations?

The patch set reflects the above thoughts is here 
https://reviews.apache.org/r/31533/diff/

 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers.
 

 Key: KAFKA-1988
 URL: https://issues.apache.org/jira/browse/KAFKA-1988
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.0
Reporter: Tong Li
Assignee: Tong Li
Priority: Blocker
 Fix For: 0.8.2.1

 Attachments: KAFKA-1988.patch, KAFKA-1988.patch


 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers. The method only returns intended value for positive 
 numbers. All negative numbers except the Integer.Min_Value will be returned 
 an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li]

Jun Rao,
Thanks for taking time reviewing the patch. Would like to confirm
here is what you suggested to do:

1. Utils.abs will be renamed to Utils.toPositive
2. Make changes in these methods of class DefaultPartitioner and
ByteArrayPartitioner so that toPositive gets called in partition method.

def partition(key: Any, numPartitions: Int): Int = {
Utils.abs(key.hashCode) % numPartitions
}


I can certainly make the above changes.  however method abs gets used
many other places and making the case worse is that there are two abs
methods both in core and clients. I found this problem when I was in an
effort of addressing issue 1926. For now, what if I keep that method in
clients, remove the one in core Utils module and making changes to all
other modules where abs method gets called so that all use of abs method
will point to the clients Utils.abs instead of two locations?

Thanks.

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com

Jun Rao (JIRA) j...@apache.org wrote on 02/27/2015 12:43:04 AM:

 From: Jun Rao (JIRA) j...@apache.org
 To: dev@kafka.apache.org
 Date: 02/27/2015 12:43 AM
 Subject: [jira] [Commented] (KAFKA-1988)
 org.apache.kafka.common.utils.Utils.abs method returns wrong value
 for negative numbers.


 [ https://issues.apache.org/jira/browse/KAFKA-1988?
 page=com.atlassian.jira.plugin.system.issuetabpanels:comment-
 tabpanelfocusedCommentId=14339751#comment-14339751 ]

 Jun Rao commented on KAFKA-1988:
 

 Actually, we can probably keep the same logic in determining the
 partition from key in Partitioner. We  can move the current code in
 abs() into Partitioner, and give it a new name like toPositive().
 Then we can replace Utils.abs() with toPositive() it in the
 following code in Partitioner.

 // hash the key to choose a partition
 return Utils.abs(Utils.murmur2(key)) % numPartitions;

 This way, we will preserve the key distribution of existing users of
 the new producer.

 Tong,

 Do you want to submit a new patch based on that?



  org.apache.kafka.common.utils.Utils.abs method returns wrong value
 for negative numbers.
 



 
  Key: KAFKA-1988
  URL: https://issues.apache.org/jira/browse/KAFKA-1988
  Project: Kafka
   Issue Type: Bug
 Affects Versions: 0.8.2.0
 Reporter: Tong Li
 Assignee: Tong Li
 Priority: Blocker
  Fix For: 0.8.2.1
 
  Attachments: KAFKA-1988.patch
 
 
  org.apache.kafka.common.utils.Utils.abs method returns wrong value
 for negative numbers. The method only returns intended value for
 positive numbers. All negative numbers except the Integer.Min_Value
 will be returned an unsigned integer.



 --
 This message was sent by Atlassian JIRA
 (v6.3.4#6332)

Review Request 31533: Patch for KAFKA-1988

[Tong Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31533/
---

Review request for kafka.


Bugs: KAFKA-1988
https://issues.apache.org/jira/browse/KAFKA-1988


Repository: kafka


Description
---

KAFKA-1988 abs method is not returning right values for negative numbers


Diffs
-

  clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
69530c187cd1c41b8173b61de6f982aafe65c9fe 
  clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 
4c2ea34815b63174732d58b699e1a0a9e6ec3b6f 
  core/src/main/scala/kafka/log/OffsetMap.scala 
42cdfbb6100b5c89d86144f92f661ebd844b2132 
  core/src/main/scala/kafka/producer/ByteArrayPartitioner.scala 
6a3b02e414eb7d62cfaece3344245feac54cecda 
  core/src/main/scala/kafka/producer/DefaultPartitioner.scala 
3afb22eeb4e3b8ecf49e92bd167f2f67b8f6a961 
  core/src/main/scala/kafka/producer/async/DefaultEventHandler.scala 
821901e4f434dfd9eec6eceabfc2e1e65507a57c 
  core/src/main/scala/kafka/server/AbstractFetcherManager.scala 
20c00cb8cc2351950edbc8cb1752905a0c26e79f 
  core/src/main/scala/kafka/tools/MirrorMaker.scala 
5374280dc97dc8e01e9b3ba61fd036dc13ae48cb 
  core/src/main/scala/kafka/utils/Utils.scala 
738c1af9ef5de16fdf5130daab69757a14c48b5c 
  core/src/test/scala/unit/kafka/utils/UtilsTest.scala 
8c3797a964a2760a00ed60530d1a48e3da473769 

Diff: https://reviews.apache.org/r/31533/diff/


Testing
---


Thanks,

Tong Li

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14340959#comment-14340959
 ] 

Tong Li commented on KAFKA-1988:


Run Rao, I got the point now. Thanks so much for your very detailed 
explanation. A patch is coming in few minutes with only that.

 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers.
 

 Key: KAFKA-1988
 URL: https://issues.apache.org/jira/browse/KAFKA-1988
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.0
Reporter: Tong Li
Assignee: Tong Li
Priority: Blocker
 Fix For: 0.8.2.1

 Attachments: KAFKA-1988.patch, KAFKA-1988.patch


 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers. The method only returns intended value for positive 
 numbers. All negative numbers except the Integer.Min_Value will be returned 
 an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 31566: Patch for KAFKA-1988

[Tong Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31566/
---

Review request for kafka.


Bugs: KAFKA-1988
https://issues.apache.org/jira/browse/KAFKA-1988


Repository: kafka


Description
---

KAFKA-1988 org.apache.kafka.common.utils.Utils.abs method returns wrong value 
for negative numbers


Diffs
-

  
clients/src/main/java/org/apache/kafka/clients/producer/internals/Partitioner.java
 dfb936d8f0d5842ee5c7a7f1584c5ed7463c4cf8 
  clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
69530c187cd1c41b8173b61de6f982aafe65c9fe 
  clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 
4c2ea34815b63174732d58b699e1a0a9e6ec3b6f 

Diff: https://reviews.apache.org/r/31566/diff/


Testing
---


Thanks,

Tong Li

[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1988:
---
Attachment: KAFKA-1988.patch

 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers.
 

 Key: KAFKA-1988
 URL: https://issues.apache.org/jira/browse/KAFKA-1988
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.0
Reporter: Tong Li
Assignee: Tong Li
Priority: Blocker
 Fix For: 0.8.2.1

 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch


 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers. The method only returns intended value for positive 
 numbers. All negative numbers except the Integer.Min_Value will be returned 
 an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14341004#comment-14341004
 ] 

Tong Li commented on KAFKA-1988:


Created reviewboard https://reviews.apache.org/r/31566/diff/
 against branch origin/trunk

 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers.
 

 Key: KAFKA-1988
 URL: https://issues.apache.org/jira/browse/KAFKA-1988
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.0
Reporter: Tong Li
Assignee: Tong Li
Priority: Blocker
 Fix For: 0.8.2.1

 Attachments: KAFKA-1988.patch, KAFKA-1988.patch, KAFKA-1988.patch


 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers. The method only returns intended value for positive 
 numbers. All negative numbers except the Integer.Min_Value will be returned 
 an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1988:
---
Status: Patch Available  (was: Open)

 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers.
 

 Key: KAFKA-1988
 URL: https://issues.apache.org/jira/browse/KAFKA-1988
 Project: Kafka
  Issue Type: Bug
Reporter: Tong Li
Assignee: Tong Li
 Attachments: KAFKA-1988.patch


 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers. The method only returns intended value for positive 
 numbers. All negative numbers except the Integer.Min_Value will be returned 
 an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1988:
---
Attachment: KAFKA-1988.patch

 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers.
 

 Key: KAFKA-1988
 URL: https://issues.apache.org/jira/browse/KAFKA-1988
 Project: Kafka
  Issue Type: Bug
Reporter: Tong Li
Assignee: Tong Li
 Attachments: KAFKA-1988.patch


 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers. The method only returns intended value for positive 
 numbers. All negative numbers except the Integer.Min_Value will be returned 
 an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14339478#comment-14339478
 ] 

Tong Li commented on KAFKA-1988:


Created reviewboard https://reviews.apache.org/r/31517/diff/
 against branch origin/trunk

 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers.
 

 Key: KAFKA-1988
 URL: https://issues.apache.org/jira/browse/KAFKA-1988
 Project: Kafka
  Issue Type: Bug
Reporter: Tong Li
Assignee: Tong Li
 Attachments: KAFKA-1988.patch


 org.apache.kafka.common.utils.Utils.abs method returns wrong value for 
 negative numbers. The method only returns intended value for positive 
 numbers. All negative numbers except the Integer.Min_Value will be returned 
 an unsigned integer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 31517: Patch for KAFKA-1988

[Tong Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31517/
---

Review request for kafka.


Bugs: KAFKA-1988
https://issues.apache.org/jira/browse/KAFKA-1988


Repository: kafka


Description
---

KAFKA-1988 abs method is not returning right values for negative numbers


Diffs
-

  clients/src/main/java/org/apache/kafka/common/utils/Utils.java 
69530c187cd1c41b8173b61de6f982aafe65c9fe 
  clients/src/test/java/org/apache/kafka/common/utils/UtilsTest.java 
4c2ea34815b63174732d58b699e1a0a9e6ec3b6f 

Diff: https://reviews.apache.org/r/31517/diff/


Testing
---


Thanks,

Tong Li

[jira] [Created] (KAFKA-1988) org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative numbers.

[Tong Li (JIRA)]

Tong Li created KAFKA-1988:
--

 Summary: org.apache.kafka.common.utils.Utils.abs method returns 
wrong value for negative numbers.
 Key: KAFKA-1988
 URL: https://issues.apache.org/jira/browse/KAFKA-1988
 Project: Kafka
  Issue Type: Bug
Reporter: Tong Li
Assignee: Tong Li


org.apache.kafka.common.utils.Utils.abs method returns wrong value for negative 
numbers. The method only returns intended value for positive numbers. All 
negative numbers except the Integer.Min_Value will be returned an unsigned 
integer.




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Tips for working with Kafka and data streams

[Tong Li]

+2, these kind of articles coming from the ones who created Kafka always
provide great value to Kafka users and developers. For my 2 cents, I would
love to see one or two articles for developers who involved in Kafka
development on the topics of how to develop test cases and how to run them,
what to expect when error occurs, typical system settings, I suspect that
most of us do run it on linux based systems, little pointer probably can
help a lot. and most importantly how to set up your dev environment so that
you are not struggling with the things the pioneers have already figured
out. For example, recommended dev. ide, debug methods, of course, these
will be the preference of the writer, no one is obligated to use but can
certainly get people started quicker. As Kafka draw more interest, I
suspect more developers will join, having something like that can be
extremely helpful.

Jay, articles similar to the one linked in your original email can actually
be submitted to developerworks, and you can get some money out of it if you
like. If you do not know how to do that, I can certainly provide some
pointers if you are interested.

Thanks.

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Jay Kreps j...@confluent.io
To: dev@kafka.apache.org dev@kafka.apache.org,
us...@kafka.apache.org us...@kafka.apache.org
Date:   02/25/2015 02:52 PM
Subject:Tips for working with Kafka and data streams



Hey guys,

One thing we tried to do along with the product release was start to put
together a practical guide for using Kafka. I wrote this up here:
http://blog.confluent.io/2015/02/25/stream-data-platform-1/

I'd like to keep expanding on this as good practices emerge and we learn
more stuff. So two questions:
1. Anything you think other people should know about working with data
streams? What did you wish you knew when you got started?
2. Anything you don't know about but would like to hear more about?

-Jay

Re: Tips for working with Kafka and data streams

[Tong Li]

+2, these kind of articles coming from the ones who created Kafka always
provide great value to Kafka users and developers. For my 2 cents, I would
love to see one or two articles for developers who involved in Kafka
development on the topics of how to develop test cases and how to run them,
what to expect when error occurs, typical system settings, I suspect that
most of us do run it on linux based systems, little pointer probably can
help a lot. and most importantly how to set up your dev environment so that
you are not struggling with the things the pioneers have already figured
out. For example, recommended dev. ide, debug methods, of course, these
will be the preference of the writer, no one is obligated to use but can
certainly get people started quicker. As Kafka draw more interest, I
suspect more developers will join, having something like that can be
extremely helpful.

Jay, articles similar to the one linked in your original email can actually
be submitted to developerworks, and you can get some money out of it if you
like. If you do not know how to do that, I can certainly provide some
pointers if you are interested.

Thanks.

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Jay Kreps j...@confluent.io
To: d...@kafka.apache.org d...@kafka.apache.org,
users@kafka.apache.org users@kafka.apache.org
Date:   02/25/2015 02:52 PM
Subject:Tips for working with Kafka and data streams



Hey guys,

One thing we tried to do along with the product release was start to put
together a practical guide for using Kafka. I wrote this up here:
http://blog.confluent.io/2015/02/25/stream-data-platform-1/

I'd like to keep expanding on this as good practices emerge and we learn
more stuff. So two questions:
1. Anything you think other people should know about working with data
streams? What did you wish you knew when you got started?
2. Anything you don't know about but would like to hear more about?

-Jay

Re: Unit tests in java7 vs java8

[Tong Li]

Gwen,
I have not tried Java 8. Still on Java 7, but I always run into the
test hung problems (no errors on the screen and the system is completely
idle), it may be a different problem. I can recreate that problem every
time when I run gradle --daemon testAll, I recall that couple of weeks
ago there was one patch saying fixed the problem, but I am still seeing the
problem with latest code. What I noticed is that seems tests always stop at
one of the ConsumerTest test cases. What puzzled me the most is that it was
not always a particular test case. Being very new in this community, I
think that error must be something related to my env. Here is my
environment:

Oracle JDK 7, gradle 2.2.1, scala 2.10.4. Lot of open file handles
and big enough max lock memory,

not complaining, just some observations in case you wonder what other
developers may face.

Thanks.

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Gwen Shapira gshap...@cloudera.com
To: dev@kafka.apache.org dev@kafka.apache.org
Date:   02/25/2015 03:47 PM
Subject:Unit tests in java7 vs java8



Hi,

Anyone running tests on Java 8? I just noticed that they take almost twice
as long to run compared to Java 7 (at least on my box, and with Scala
2.10.4).

Anyone else noticed this? Maybe even did some digging on the causes?

Gwen

Re: [jira] [Resolved] (KAFKA-1959) Class CommitThread overwrite group of Thread class causing compile errors

[Tong Li]

Joel, thanks so much for merging the two sets.

Tong

Sent from my iPhone

 On Feb 18, 2015, at 2:55 AM, Joel Koshy (JIRA) j...@apache.org wrote:


[ 
https://issues.apache.org/jira/browse/KAFKA-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]


 Joel Koshy resolved KAFKA-1959.
 ---
 Resolution: Fixed
   Assignee: Tong Li

 Thanks for the patch - committed to trunk.

  Class CommitThread overwrite group of Thread class causing compile
errors
 
-
 
  Key: KAFKA-1959
  URL: https://issues.apache.org/jira/browse/KAFKA-1959
  Project: Kafka
   Issue Type: Bug
   Components: core
  Environment: scala 2.10.4
 Reporter: Tong Li
 Assignee: Tong Li
   Labels: newbie
  Attachments: KAFKA-1959.patch, compileError.png
 
 
  class CommitThread(id: Int, partitionCount: Int, commitIntervalMs:
Long, zkClient: ZkClient)
  extends ShutdownableThread(commit-thread)
  with KafkaMetricsGroup {
  private val group = group- + id
  group overwrite class Thread group member, causing the following
compile error:
  overriding variable group in class Thread of type ThreadGroup;  value
group has weaker access privileges; it should not be private



 --
 This message was sent by Atlassian JIRA
 (v6.3.4#6332)

two very simple patch sets to be reviewed.

[Tong Li]

Dear kafka developers,
New to this community and put up two really small patch sets with
open issues, can any one please review and comment and get them merged if
all possible? Thanks

https://reviews.apache.org/r/31097/

https://reviews.apache.org/r/31097/

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com

two very simple patch sets to be reviewed.

[Tong Li]

Dear kafka developers,
New to this community and put up two really small patch sets with
open issues, can any one please review and comment and get them merged if
all possible? Thanks


https://reviews.apache.org/r/31088/

https://reviews.apache.org/r/31097/


Tong Li
OpenStack Community Development
Building 501/B205
liton...@us.ibm.com



From:   Tong Li/Raleigh/IBM
To: kafka dev@kafka.apache.org
Date:   02/17/2015 01:03 PM
Subject:two very simple patch sets to be reviewed.


Dear kafka developers,
New to this community and put up two really small patch sets with
open issues, can any one please review and comment and get them merged if
all possible? Thanks

https://reviews.apache.org/r/31097/

https://reviews.apache.org/r/31097/

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com

Re: two very simple patch sets to be reviewed.

[Tong Li]

Gwen,
Really appreciate it. Thanks so much. Anyone else please review them?
Here are the links again.

 https://reviews.apache.org/r/31088/

 https://reviews.apache.org/r/31097/


Tong Li
OpenStack Community Development
Building 501/B205
liton...@us.ibm.com



From:   Gwen Shapira gshap...@cloudera.com
To: dev@kafka.apache.org dev@kafka.apache.org
Date:   02/17/2015 03:29 PM
Subject:Re: two very simple patch sets to be reviewed.



I've reviewed both (but can't commit obviously)

They are both safe (a rename and an addition to .gitignore).
The addition to .gitignore will be very useful for anyone who uses system
tests (which should be all of us).
The rename is useful only to those using IBM JDK (i.e. not all of us), but
since its just a rename, I figured there's no reason not to solve it.

Gwen

On Tue, Feb 17, 2015 at 10:05 AM, Tong Li liton...@us.ibm.com wrote:

 Dear kafka developers,
  New to this community and put up two really small patch sets with open
 issues, can any one please review and comment and get them merged if all
 possible? Thanks


 https://reviews.apache.org/r/31088/

 https://reviews.apache.org/r/31097/


 Tong Li
 OpenStack Community Development
 Building 501/B205
 liton...@us.ibm.com

 [image: Inactive hide details for Tong Li---02/17/2015 01:03:36 PM---Dear
 kafka developers, New to this community and put up two real]Tong
 Li---02/17/2015 01:03:36 PM---Dear kafka developers,   New to this
 community and put up two really small patch sets with open issu

 From: Tong Li/Raleigh/IBM
 To: kafka dev@kafka.apache.org
 Date: 02/17/2015 01:03 PM
 Subject: two very simple patch sets to be reviewed.
 --


 Dear kafka developers,
  New to this community and put up two really small patch sets with open
 issues, can any one please review and comment and get them merged if all
 possible? Thanks

 https://reviews.apache.org/r/31097/

 https://reviews.apache.org/r/31097/

 Tong Li
 OpenStack  Kafka Community Development
 Building 501/B205
 liton...@us.ibm.com

Review Request 31088: Patch for KAFKA-1959

[Tong Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31088/
---

Review request for kafka.


Bugs: KAFKA-1959
https://issues.apache.org/jira/browse/KAFKA-1959


Repository: kafka


Description
---

The original code clearly wants to define a string but override the ThreadGroup 
of
the super class member. The patchset will rename the private variable to be 
group_id
as it intended.


Diffs
-

  core/src/test/scala/other/kafka/TestOffsetManager.scala 
41f334d48897b3027ed54c58bbf4811487d3b191 

Diff: https://reviews.apache.org/r/31088/diff/


Testing
---


Thanks,

Tong Li

[jira] [Commented] (KAFKA-1959) Class CommitThread overwrite group of Thread class causing compile errors

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14322966#comment-14322966
 ] 

Tong Li commented on KAFKA-1959:


Created reviewboard https://reviews.apache.org/r/31088/diff/
 against branch origin/trunk

 Class CommitThread overwrite group of Thread class causing compile errors
 -

 Key: KAFKA-1959
 URL: https://issues.apache.org/jira/browse/KAFKA-1959
 Project: Kafka
  Issue Type: Bug
  Components: core
 Environment: scala 2.10.4
Reporter: Tong Li
  Labels: newbie
 Attachments: KAFKA-1959.patch


 class CommitThread(id: Int, partitionCount: Int, commitIntervalMs: Long, 
 zkClient: ZkClient)
 extends ShutdownableThread(commit-thread)
 with KafkaMetricsGroup {
 private val group = group- + id
 group overwrite class Thread group member, causing the following compile 
 error:
 overriding variable group in class Thread of type ThreadGroup;  value group 
 has weaker access privileges; it should not be private



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-1959) Class CommitThread overwrite group of Thread class causing compile errors

[Tong Li (JIRA)]

Tong Li created KAFKA-1959:
--

 Summary: Class CommitThread overwrite group of Thread class 
causing compile errors
 Key: KAFKA-1959
 URL: https://issues.apache.org/jira/browse/KAFKA-1959
 Project: Kafka
  Issue Type: Bug
  Components: core
 Environment: scala 2.10.4
Reporter: Tong Li


class CommitThread(id: Int, partitionCount: Int, commitIntervalMs: Long, 
zkClient: ZkClient)
extends ShutdownableThread(commit-thread)
with KafkaMetricsGroup {

private val group = group- + id

group overwrite class Thread group member, causing the following compile error:

overriding variable group in class Thread of type ThreadGroup;  value group has 
weaker access privileges; it should not be private



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1959) Class CommitThread overwrite group of Thread class causing compile errors

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14322933#comment-14322933
 ] 

Tong Li commented on KAFKA-1959:


Though  I saw the error by using scala 2.10.4,  I think it will be the same 
error in other scala versions. 

 Class CommitThread overwrite group of Thread class causing compile errors
 -

 Key: KAFKA-1959
 URL: https://issues.apache.org/jira/browse/KAFKA-1959
 Project: Kafka
  Issue Type: Bug
  Components: core
 Environment: scala 2.10.4
Reporter: Tong Li
  Labels: newbie

 class CommitThread(id: Int, partitionCount: Int, commitIntervalMs: Long, 
 zkClient: ZkClient)
 extends ShutdownableThread(commit-thread)
 with KafkaMetricsGroup {
 private val group = group- + id
 group overwrite class Thread group member, causing the following compile 
 error:
 overriding variable group in class Thread of type ThreadGroup;  value group 
 has weaker access privileges; it should not be private



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1959) Class CommitThread overwrite group of Thread class causing compile errors

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1959:
---
Attachment: KAFKA-1959.patch

 Class CommitThread overwrite group of Thread class causing compile errors
 -

 Key: KAFKA-1959
 URL: https://issues.apache.org/jira/browse/KAFKA-1959
 Project: Kafka
  Issue Type: Bug
  Components: core
 Environment: scala 2.10.4
Reporter: Tong Li
  Labels: newbie
 Attachments: KAFKA-1959.patch


 class CommitThread(id: Int, partitionCount: Int, commitIntervalMs: Long, 
 zkClient: ZkClient)
 extends ShutdownableThread(commit-thread)
 with KafkaMetricsGroup {
 private val group = group- + id
 group overwrite class Thread group member, causing the following compile 
 error:
 overriding variable group in class Thread of type ThreadGroup;  value group 
 has weaker access privileges; it should not be private



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 31097: Patch for KAFKA-1960

[Tong Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31097/
---

Review request for kafka.


Bugs: KAFKA-1960
https://issues.apache.org/jira/browse/KAFKA-1960


Repository: kafka


Description
---

test created files in following folders and the files and folders should
be exclude from being committed to the source repository:

core/data/*
gradle/wrapper/*


Diffs
-

  .gitignore 06a64184eaa531fcbf5586692b78bfd48e4176ba 

Diff: https://reviews.apache.org/r/31097/diff/


Testing
---


Thanks,

Tong Li

[jira] [Commented] (KAFKA-1960) .gitignore does not exclude test generated files and folders.

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14323347#comment-14323347
 ] 

Tong Li commented on KAFKA-1960:


Created reviewboard https://reviews.apache.org/r/31097/diff/
 against branch origin/trunk

 .gitignore does not exclude test generated files and folders.
 -

 Key: KAFKA-1960
 URL: https://issues.apache.org/jira/browse/KAFKA-1960
 Project: Kafka
  Issue Type: Bug
  Components: build
Reporter: Tong Li
Priority: Minor
  Labels: newbie
 Attachments: KAFKA-1960.patch


 gradle test can create quite few folders, .gitignore should exclude these 
 files for an easier git submit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 31088: Patch for KAFKA-1959

[Tong Li]

 On Feb. 16, 2015, 6:22 p.m., Gwen Shapira wrote:
  The rename does make things a bit clearer, so I don't object to committing 
  it. 
  But the test does not override group in superclas Thread. Thread.group is 
  a private variable, so it cannot be overriden (and the classes that inherit 
  from it can't use it) so there's no conflict here.
 
 Tong Li wrote:
 Thanks Gwen, completely agreed. What puzzled me is that the compile 
 error. I do not know how to explain that. I have uploaded the screen shot of 
 the error against the jira issue, I wonder if you can give a bit of opinion 
 on that? Here is the link 
 https://issues.apache.org/jira/secure/attachment/12699149/compileError.png
 
 Gwen Shapira wrote:
 interesting. I didn't see this error on my setup.
 
 Which Scala / Java versions are you using?
 
 Tong Li wrote:
 Gwen, thanks again. This error really bugs me. I am using IBM JDK 7 (can 
 not use oracle jdk because I am an IBMer). Scala version 2.10.4.

I replaced the IBM JDK7 with Oracle JDK 7. Not seeing that compile error. I am 
pretty sure it is related to the jdk. Would you still like the patch to be 
merged? I think it still helps if some other people running IBM JDK.


- Tong


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31088/#review72637
---


On Feb. 16, 2015, 4:37 p.m., Tong Li wrote:
 
 ---
 This is an automatically generated e-mail. To reply, visit:
 https://reviews.apache.org/r/31088/
 ---
 
 (Updated Feb. 16, 2015, 4:37 p.m.)
 
 
 Review request for kafka.
 
 
 Bugs: KAFKA-1959
 https://issues.apache.org/jira/browse/KAFKA-1959
 
 
 Repository: kafka
 
 
 Description
 ---
 
 The original code clearly wants to define a string but override the 
 ThreadGroup of
 the super class member. The patchset will rename the private variable to be 
 group_id
 as it intended.
 
 
 Diffs
 -
 
   core/src/test/scala/other/kafka/TestOffsetManager.scala 
 41f334d48897b3027ed54c58bbf4811487d3b191 
 
 Diff: https://reviews.apache.org/r/31088/diff/
 
 
 Testing
 ---
 
 
 Thanks,
 
 Tong Li
 

[jira] [Updated] (KAFKA-1960) .gitignore does not exclude test generated files and folders.

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1960?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1960:
---
Attachment: KAFKA-1960.patch

 .gitignore does not exclude test generated files and folders.
 -

 Key: KAFKA-1960
 URL: https://issues.apache.org/jira/browse/KAFKA-1960
 Project: Kafka
  Issue Type: Bug
  Components: build
Reporter: Tong Li
Priority: Minor
  Labels: newbie
 Attachments: KAFKA-1960.patch


 gradle test can create quite few folders, .gitignore should exclude these 
 files for an easier git submit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Review Request 31007: Patch for KAFKA-1926

[Tong Li]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/31007/
---

Review request for kafka.


Bugs: KAFKA-1926
https://issues.apache.org/jira/browse/KAFKA-1926


Repository: kafka


Description
---

More patch set will be submitted to address utils.java issues
Following things happen in this patch set:

1. Crc32.java was removed so that the core uses the same class
   defined in the client package.
2. Removed Time trait defined in Time.scala to use the interface
   defined in the client Time module.
3. Rewrite the SystemTime object in Time.scala so that it uses
   the client SystemTime class.
4. References to Time has been all refacted to use the interface
   defined in client module.
5. .gitignore does not exclude the following build process created
   folder and files:
  core/data/*
  gradle/wrapper/*


Diffs
-

  .gitignore 06a64184eaa531fcbf5586692b78bfd48e4176ba 
  clients/src/main/java/org/apache/kafka/common/utils/Time.java 
66c44de74521ed13cff75d556ebefa8846485b21 
  core/src/main/scala/kafka/cluster/Partition.scala 
e6ad8be5e33b6fb31c078ad78f8de709869ddc04 
  core/src/main/scala/kafka/cluster/Replica.scala 
bd13c20338ce3d73113224440e858a12814e5adb 
  core/src/main/scala/kafka/log/Log.scala 
846023bb98d0fa0603016466360c97071ac935ea 
  core/src/main/scala/kafka/log/LogCleaner.scala 
f8e7cd5fabce78c248a9027c4bb374a792508675 
  core/src/main/scala/kafka/log/LogManager.scala 
47d250af62c1aa53d11204a332d0684fb4217c8d 
  core/src/main/scala/kafka/log/LogSegment.scala 
ac9643423a28d189133705ba69b16cfce23f0049 
  core/src/main/scala/kafka/network/SocketServer.scala 
76ce41aed6e04ac5ba88395c4d5008aca17f9a73 
  core/src/main/scala/kafka/server/KafkaServer.scala 
7e5ddcb9be8fcef3df6ebc82a13ef44ef95f73ae 
  core/src/main/scala/kafka/server/ReplicaManager.scala 
fb948b9ab28c516e81dab14dcbe211dcd99842b6 
  core/src/main/scala/kafka/server/TopicConfigManager.scala 
47295d40131492aaac786273819b7bc6e22e5486 
  core/src/main/scala/kafka/utils/Crc32.java 
0e0e7bcb33886f47b9770a122d00d6eafdbb89a9 
  core/src/main/scala/kafka/utils/Throttler.scala 
d1a144d7882919426824799ff8e8a47f89c83158 
  core/src/main/scala/kafka/utils/Time.scala 
194cc1fa73b6a68914caf46b7dd7d415b2ff6a9f 
  core/src/main/scala/kafka/utils/Utils.scala 
738c1af9ef5de16fdf5130daab69757a14c48b5c 
  core/src/test/scala/unit/kafka/server/ISRExpirationTest.scala 
a703d2715048c5602635127451593903f8d20576 
  core/src/test/scala/unit/kafka/server/LogOffsetTest.scala 
c06ee756bf0fe07e5d3c92823a476c960b37afd6 
  core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala 
5b93239cdc26b5be7696f4e7863adb9fbe5f0ed5 
  core/src/test/scala/unit/kafka/utils/MockScheduler.scala 
c6740782813cbf7c979bf1b94dd829945ef38458 
  core/src/test/scala/unit/kafka/utils/MockTime.scala 
ee65748afefd5e2d699e68fbd402b0ed9a659589 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
21d0ed2cb7c9459261d3cdc7c21dece5e2079698 

Diff: https://reviews.apache.org/r/31007/diff/


Testing
---


Thanks,

Tong Li

[jira] [Updated] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tong Li updated KAFKA-1926:
---
Attachment: KAFKA-1926.patch

 Replace kafka.utils.Utils with o.a.k.common.utils.Utils
 ---

 Key: KAFKA-1926
 URL: https://issues.apache.org/jira/browse/KAFKA-1926
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.8.2.0
Reporter: Jay Kreps
  Labels: newbie, patch
 Attachments: KAFKA-1926.patch, KAFKA-1926.patch


 There is currently a lot of duplication between the Utils class in common and 
 the one in core.
 Our plan has been to deprecate duplicate code in the server and replace it 
 with the new common code.
 As such we should evaluate each method in the scala Utils and do one of the 
 following:
 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
 utility in active use that is not Kafka-specific. If we migrate it we should 
 really think about the API and make sure there is some test coverage. A few 
 things in there are kind of funky and we shouldn't just blindly copy them 
 over.
 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
 any utilities that really need to make use of Scala features to be convenient.
 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320416#comment-14320416
 ] 

Tong Li commented on KAFKA-1926:


[~jkreps] Removal of Time.scala cause a lot of code changes. Since Time.scala 
defined the SystemTime object and get used all over the place. Though another 
class SystemTime defined in client util package as a java class. The use of the 
SystemTime object in the core is like using the property, not method calls, I 
can rework that part, but it will be a lot of code like this
 SystemTime.milliseconds

To be changed to:
new SystemTime().milliseconds()

Since the code in client common util gets defined as a regular class. I feel 
doing that also will cost quite a bit rather than stick with the scala 
SystemTime and simply calls its property, not many instance gets created. Let 
me know what you think and we can always improve.

Thanks.

 Replace kafka.utils.Utils with o.a.k.common.utils.Utils
 ---

 Key: KAFKA-1926
 URL: https://issues.apache.org/jira/browse/KAFKA-1926
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.8.2.0
Reporter: Jay Kreps
  Labels: newbie, patch
 Attachments: KAFKA-1926.patch, KAFKA-1926.patch


 There is currently a lot of duplication between the Utils class in common and 
 the one in core.
 Our plan has been to deprecate duplicate code in the server and replace it 
 with the new common code.
 As such we should evaluate each method in the scala Utils and do one of the 
 following:
 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
 utility in active use that is not Kafka-specific. If we migrate it we should 
 really think about the API and make sure there is some test coverage. A few 
 things in there are kind of funky and we shouldn't just blindly copy them 
 over.
 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
 any utilities that really need to make use of Scala features to be convenient.
 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14320404#comment-14320404
 ] 

Tong Li commented on KAFKA-1926:


Created reviewboard https://reviews.apache.org/r/31007/diff/
 against branch origin/trunk

 Replace kafka.utils.Utils with o.a.k.common.utils.Utils
 ---

 Key: KAFKA-1926
 URL: https://issues.apache.org/jira/browse/KAFKA-1926
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.8.2.0
Reporter: Jay Kreps
  Labels: newbie, patch
 Attachments: KAFKA-1926.patch, KAFKA-1926.patch


 There is currently a lot of duplication between the Utils class in common and 
 the one in core.
 Our plan has been to deprecate duplicate code in the server and replace it 
 with the new common code.
 As such we should evaluate each method in the scala Utils and do one of the 
 following:
 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
 utility in active use that is not Kafka-specific. If we migrate it we should 
 really think about the API and make sure there is some test coverage. A few 
 things in there are kind of funky and we shouldn't just blindly copy them 
 over.
 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
 any utilities that really need to make use of Scala features to be convenient.
 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

gradle testAll stuck

[Tong Li]

Hi, folks,
How are you all doing?
New bee here. Run gradle --daemon testAll on 0.8.2 worked and finished
in about 30 minutes. pulled down trunk and run samething, always stuck.
left it run for overnight, checked in the morning still stuck. even on
0.8.2, it still takes over 30 minutes on my modest dev system. Wonder how
you all run gradle and is there any specific settings needed? I am running
it on Ubuntu 14.04. Any help or pointer is really appreciated.

Thanks

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com

Re: gradle testAll stuck

[Tong Li]

Manikumar, yes, exactly the same issue. Commented on
https://issues.apache.org/jira/browse/KAFKA-1948. Before this problem
happens, the testAll normally still takes about half hour?

Thanks

Tong Li
OpenStack  Kafka Community Development
Building 501/B205
liton...@us.ibm.com



From:   Manikumar Reddy ku...@nmsworks.co.in
To: dev@kafka.apache.org
Date:   02/12/2015 09:10 AM
Subject:Re: gradle testAll stuck
Sent by:manikumar.re...@gmail.com



This may be due to the recent issue reported by Gwen.

https://issues.apache.org/jira/browse/KAFKA-1948

On 2/12/15, Joel Koshy jjkosh...@gmail.com wrote:
 - Can you enable test logging (see the README) and see if you can
   figure out which test is getting stuck or taking forever?
 - A thread-dump may help.

 On Thu, Feb 12, 2015 at 08:57:11AM -0500, Tong Li wrote:


 Hi, folks,
 How are you all doing?
 New bee here. Run gradle --daemon testAll on 0.8.2 worked and
 finished
 in about 30 minutes. pulled down trunk and run samething, always stuck.
 left it run for overnight, checked in the morning still stuck. even on
 0.8.2, it still takes over 30 minutes on my modest dev system. Wonder
how
 you all run gradle and is there any specific settings needed? I am
 running
 it on Ubuntu 14.04. Any help or pointer is really appreciated.

 Thanks

 Tong Li
 OpenStack  Kafka Community Development
 Building 501/B205
 liton...@us.ibm.com

[jira] [Commented] (KAFKA-1948) kafka.api.consumerTests are hanging

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1948?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14318311#comment-14318311
 ] 

Tong Li commented on KAFKA-1948:


Pulled down the latest trunk and run into this problem. It will be nice if 
someone can take a look. The test does not stuck always at the same place. I 
did notice there are tons of threads getting created and died. What I did is 
when the test hung, I ran this command.

top -H -p theprocessid of the test java

You will notice that many threads come and go. and the state of these threads 
are always S (meaning sleep). including the parent thread. Not really sure what 
the deal is.

 kafka.api.consumerTests are hanging
 ---

 Key: KAFKA-1948
 URL: https://issues.apache.org/jira/browse/KAFKA-1948
 Project: Kafka
  Issue Type: Bug
Reporter: Gwen Shapira

 Noticed today that very often when I run the full test suite, it hangs on 
 kafka.api.consumerTest (not always same test though). It doesn't reproduce 
 100% of the time, but enough to be very annoying.
 I also saw it happening on trunk after KAFKA-1333:
 https://builds.apache.org/view/All/job/Kafka-trunk/389/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1926) Replace kafka.utils.Utils with o.a.k.common.utils.Utils

[Tong Li (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14316830#comment-14316830
 ] 

Tong Li commented on KAFKA-1926:


[~jkreps] Jay, really appreciate your quick review comments. I will see what I 
can do and will submit another patch based on trunk branch. New patch set will 
come up real soon. Thanks so much.

 Replace kafka.utils.Utils with o.a.k.common.utils.Utils
 ---

 Key: KAFKA-1926
 URL: https://issues.apache.org/jira/browse/KAFKA-1926
 Project: Kafka
  Issue Type: Improvement
Affects Versions: 0.8.2.0
Reporter: Jay Kreps
  Labels: newbie, patch
 Attachments: KAFKA-1926.patch


 There is currently a lot of duplication between the Utils class in common and 
 the one in core.
 Our plan has been to deprecate duplicate code in the server and replace it 
 with the new common code.
 As such we should evaluate each method in the scala Utils and do one of the 
 following:
 1. Migrate it to o.a.k.common.utils.Utils if it is a sensible general purpose 
 utility in active use that is not Kafka-specific. If we migrate it we should 
 really think about the API and make sure there is some test coverage. A few 
 things in there are kind of funky and we shouldn't just blindly copy them 
 over.
 2. Create a new class ServerUtils or ScalaUtils in kafka.utils that will hold 
 any utilities that really need to make use of Scala features to be convenient.
 3. Delete it if it is not used, or has a bad api.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)