[jira] [Comment Edited] (KAFKA-1070) Auto-assign node id

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085808#comment-14085808
 ] 

Joe Stein edited comment on KAFKA-1070 at 8/5/14 5:08 AM:
--

This sounds like a very cool and VERY useful feature. Excited to use it myself 
often.

I know of a few (>10) different clusters that not only use varying sized 
numbers for their broker.id but do so in what is a seemingly random (but not 
really when you think about it) way.

so in a cluster there may be broker.id that is 1721632 and another 172164875 
and another 172162240 . Making your brokers by replacing "." in 
chef/puppet/salt/ansemble/etc type scripts and sometimes folks get more fancy 
just doing 2288, 2388, 17, 177 (where just the last two octets get used and "." 
is replaced). 

I am not saying I agree with this approach and I actively advocate away from 
doing this but in some/many scenarios it is the best/only way to automate their 
deploys for how things are setup.  It is also what seems to make sense when 
folks are building their automation scripts since they have no other option 
without doing more than they should be expected to-do (and the IP replace "." 
is so obvious to them, and it is).

So, for folks in these cases they would just pick the upper bound to be, lets 
say 17216255256 and then it would auto assign from there?

Is there some better way to go about this where you might have a start 
increment and and some exclusion list? or a way to see broker.id already had 
that and not use it?  I think a lot of folks would like to get having broker id 
be more continious and be easier to communicate but the desire to automate 
everything will outweigh that.  We could give them some sanity back with 
brokers 1,2,3,4,5,6,7,8,9,10,11,12 for a 12 node cluster.

not crucial and you may have already accounted for the scenario I brought up, 
but wanted to bring it up as a real use case for how people automate things.

it might be better for folks to manually migrate their scripts but not sure 
they would do it and if they did would have to decommission brokers which in a 
prod environment could take a few weeks/months.  If we let them start at 1 and 
exclude what they have then they can do it one at a time.  After taking down 
the first broker and bring it back up (empty) it is broker.id=1, and so on (and 
if they have a 5 they don't have to take it down), etc.

For new clusters this is a slam dunk and wouldn't want to hold up the feature 
for existing users that have already decided a work around as I don't know what 
the intent of this was or not. Some folks might not change knowing 
broker.id=17216520 sometimes is nice you just login to that box but talking 
about broker 17216520 over and over again is a pita.




was (Author: joestein):
This sounds like a very cool and VERY useful feature. Excited to use it myself 
often.

I know of a few (>10) different clusters that not only use varying sized 
numbers for their broker.id but do so in what is a seemingly random (but not 
really when you think about it) way.

so in a cluster there may be broker.id that is 1721632 and another 172164875 
and another 172162240 . Making your brokers by replacing "." in 
chef/puppet/salt/ansemble/etc type scripts and sometimes folks get more fancy 
just doing 2288, 2388, 17, 177 (where just the last two octets get used and "." 
is replaced). 

I am not saying I agree with this approach and I actively advocate away from 
doing this but in some/many scenarios it is the best/only way to automate their 
deploys for how things are setup.  It is also what seems to make sense when 
folks are building their automation scripts since they have no other option 
without doing more than they should be expected to-do (and the IP replace "." 
is so obvious to them, and it is).

So, for folks in these cases they would just pick the upper bound to be, lets 
say 17216255256 and then it would auto assign from there?

Is there some better way to go about this where you might have a start 
increment and and some exclusion list? or a way to see broker.id already had 
that and not use it?  I think a lot of folks would like to get having broker id 
be more continious and be easier to communicate but the desire to automate 
everything will outweigh that.  We could give them some sanity back with 
brokers 1,2,3,4,5,6,7,8,9,10,11,12 for a 12 node cluster.

not crucial and you may have already accounted for the scenario I brought up, 
but wanted to bring it up as a real use case for how people automate things.

it might be better for folks to manually migrate their scripts but not sure 
they would do it and if they did would have to decommission brokers which in a 
prod environment could take a few weeks/months.  If we let them start at 1 and 
exclude what they have then they can do it one at a time.  After taking down 
the first broker and bring it ba

[jira] [Comment Edited] (KAFKA-1070) Auto-assign node id

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085808#comment-14085808
 ] 

Joe Stein edited comment on KAFKA-1070 at 8/5/14 5:06 AM:
--

This sounds like a very cool and VERY useful feature. Excited to use it myself 
often.

I know of a few (>10) different clusters that not only use varying sized 
numbers for their broker.id but do so in what is a seemingly random (but not 
really when you think about it) way.

so in a cluster there may be broker.id that is 1721632 and another 172164875 
and another 172162240 . Making your brokers by replacing "." in 
chef/puppet/salt/ansemble/etc type scripts and sometimes folks get more fancy 
just doing 2288, 2388, 17, 177 (where just the last two octets get used and "." 
is replaced). 

I am not saying I agree with this approach and I actively advocate away from 
doing this but in some/many scenarios it is the best/only way to automate their 
deploys for how things are setup.  It is also what seems to make sense when 
folks are building their automation scripts since they have no other option 
without doing more than they should be expected to-do (and the IP replace "." 
is so obvious to them, and it is).

So, for folks in these cases they would just pick the upper bound to be, lets 
say 17216255256 and then it would auto assign from there?

Is there some better way to go about this where you might have a start 
increment and and some exclusion list? or a way to see broker.id already had 
that and not use it?  I think a lot of folks would like to get having broker id 
be more continious and be easier to communicate but the desire to automate 
everything will outweigh that.  We could give them some sanity back with 
brokers 1,2,3,4,5,6,7,8,9,10,11,12 for a 12 node cluster.

not crucial and you may have already accounted for the scenario I brought up, 
but wanted to bring it up as a real use case for how people automate things.

it might be better for folks to manually migrate their scripts but not sure 
they would do it and if they did would have to decommission brokers which in a 
prod environment could take a few weeks/months.  If we let them start at 1 and 
exclude what they have then they can do it one at a time.  After taking down 
the first broker and bring it back up (empty) it is broker.id=1, and so on (and 
if they have a 5 they don't have to take it down), etc.






was (Author: joestein):
This sounds like a very cool and VERY useful feature. Excited to use it myself 
often.

I know of a few (>10) different clusters that not only use varying sized 
numbers for their broker.id but do so in what is a seemingly random (but not 
really when you think about it) way.

so in a cluster there may be broker.id that is 1721632 and another 172164875 
and another 172162240 . Making your brokers by replacing "." in 
chef/puppet/salt/ansemble/etc type scripts and sometimes folks get more fancy 
just doing 2288, 2388, 17, 177 (where just the last two octets get used and "." 
is replaced). 

I am not saying I agree with this approach and I actively advocate away from 
doing this but in some/many scenarios it is the best/only way to automate their 
deploys for how things are setup.  It is also what seems to make sense when 
folks are building their automation scripts since they have no other option 
without doing more than they should be expected to-do (and the IP replace "." 
is so obvious to them, and it is).

So, for folks in these cases they would just pick the upper bound to be, lets 
say 17216255256 and then it would auto assign from there?

Is there some better way to go about this where you might have a start 
increment and and some exclusion list? or a way to see broker.id already had 
that and not use it?  I think a lot of folks would like to get having broker id 
be more continious and be easier to communicate but the desire to automate 
everything will outweigh that.  We could give them some sanity back with 
brokers 1,2,3,4,5,6,7,8,9,10,11,12 for a 12 node cluster.

not crucial and you may have already accounted for the scenario I brought up, 
but wanted to bring it up as a real use case for how people automate things.

it might be better for folks to manually migrate off what they have and then 
moving forward in their automation deal with the lower number or something.  It 
is hard to say how folks find creative solutions to common problems without 
every speaking to each other.  I don't know how this will work for them though.



> Auto-assign node id
> ---
>
> Key: KAFKA-1070
> URL: https://issues.apache.org/jira/browse/KAFKA-1070
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jay Kreps
>Assignee: Sriharsha Chintalapani
>  Labels: usability
> Attachments: KAFKA-1070.patch, KAFKA-1070_2014-07-19_16

[jira] [Commented] (KAFKA-1070) Auto-assign node id

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085808#comment-14085808
 ] 

Joe Stein commented on KAFKA-1070:
--

This sounds like a very cool and VERY useful feature. Excited to use it myself 
often.

I know of a few (>10) different clusters that not only use varying sized 
numbers for their broker.id but do so in what is a seemingly random (but not 
really when you think about it) way.

so in a cluster there may be broker.id that is 1721632 and another 172164875 
and another 172162240 . Making your brokers by replacing "." in 
chef/puppet/salt/ansemble/etc type scripts and sometimes folks get more fancy 
just doing 2288, 2388, 17, 177 (where just the last two octets get used and "." 
is replaced). 

I am not saying I agree with this approach and I actively advocate away from 
doing this but in some/many scenarios it is the best/only way to automate their 
deploys for how things are setup.  It is also what seems to make sense when 
folks are building their automation scripts since they have no other option 
without doing more than they should be expected to-do (and the IP replace "." 
is so obvious to them, and it is).

So, for folks in these cases they would just pick the upper bound to be, lets 
say 17216255256 and then it would auto assign from there?

Is there some better way to go about this where you might have a start 
increment and and some exclusion list? or a way to see broker.id already had 
that and not use it?  I think a lot of folks would like to get having broker id 
be more continious and be easier to communicate but the desire to automate 
everything will outweigh that.  We could give them some sanity back with 
brokers 1,2,3,4,5,6,7,8,9,10,11,12 for a 12 node cluster.

not crucial and you may have already accounted for the scenario I brought up, 
but wanted to bring it up as a real use case for how people automate things.

it might be better for folks to manually migrate off what they have and then 
moving forward in their automation deal with the lower number or something.  It 
is hard to say how folks find creative solutions to common problems without 
every speaking to each other.  I don't know how this will work for them though.



> Auto-assign node id
> ---
>
> Key: KAFKA-1070
> URL: https://issues.apache.org/jira/browse/KAFKA-1070
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jay Kreps
>Assignee: Sriharsha Chintalapani
>  Labels: usability
> Attachments: KAFKA-1070.patch, KAFKA-1070_2014-07-19_16:06:13.patch, 
> KAFKA-1070_2014-07-22_11:34:18.patch, KAFKA-1070_2014-07-24_20:58:17.patch, 
> KAFKA-1070_2014-07-24_21:05:33.patch
>
>
> It would be nice to have Kafka brokers auto-assign node ids rather than 
> having that be a configuration. Having a configuration is irritating because 
> (1) you have to generate a custom config for each broker and (2) even though 
> it is in configuration, changing the node id can cause all kinds of bad 
> things to happen.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Comment Edited] (KAFKA-1387) Kafka getting stuck creating ephemeral node it has already created when two zookeeper sessions are established in a very short period of time

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085792#comment-14085792
 ] 

Joe Stein edited comment on KAFKA-1387 at 8/5/14 4:41 AM:
--

Here is another way to reproduce this issue.  I have seen it a few times now 
with folks getting going with their clusters.

steps to reproduce.  install a 3 node zk ensemble with 3 brokers cluster

e.g. 

git clone https://github.com/stealthly/scala-kafka
git checkout -b zkbk3 origin/zkbk3
vagrant up provider=virtualbox

now setup each node in the cluster as you would broker 1,2,3 and the ensemble

e.g.

vagrant ssh zkbkOne
sudo su
cd /vagrant/vagrant/ && ./up.sh
vagrant ssh zkbkTwo
sudo su
cd /vagrant/vagrant/ && ./up.sh
vagrant ssh zkbkThree
sudo su
cd /vagrant/vagrant/ && ./up.sh

start up zookeeper on all 3 nodes
cd /opt/apache/kafka && bin/zookeeper-server-start.sh 
config/zookeeper.properties 1>>/tmp/zk.log 2>>/tmp/zk.log &

now, start up broker on node 2 only
cd /opt/apache/kafka && bin/kafka-server-start.sh config/server.properties 
1>>/tmp/bk.log 2>>/tmp/bk.log &

ok, now here is where it gets wonky

on server 3 change from broker.id=3 to broker.id=2 
now you need to start up server 1 and 3 (even though it is broker.id=2) at the 
same time

cd /opt/apache/kafka && bin/kafka-server-start.sh config/server.properties 
1>>/tmp/bk.log 2>>/tmp/bk.log &
cd /opt/apache/kafka && bin/kafka-server-start.sh config/server.properties 
1>>/tmp/bk.log 2>>/tmp/bk.log &
( you can have two tabs, hit enter in one switch to other tab and hit enter is 
close enough to same time)

and you get this looping forever

2014-08-05 04:34:38,591] INFO I wrote this conflicted ephemeral node 
[{"version":1,"brokerid":2,"timestamp":"1407212148186"}] at /controller a while 
back in a different session, hence I will backoff for this node to be deleted 
by Zookeeper and retry (kafka.utils.ZkUtils$)
[2014-08-05 04:34:44,598] INFO conflict in /controller data: 
{"version":1,"brokerid":2,"timestamp":"1407212148186"} stored data: 
{"version":1,"brokerid":2,"timestamp":"1407211911014"} (kafka.utils.ZkUtils$)
[2014-08-05 04:34:44,601] INFO I wrote this conflicted ephemeral node 
[{"version":1,"brokerid":2,"timestamp":"1407212148186"}] at /controller a while 
back in a different session, hence I will backoff for this node to be deleted 
by Zookeeper and retry (kafka.utils.ZkUtils$)
[2014-08-05 04:34:50,610] INFO conflict in /controller data: 
{"version":1,"brokerid":2,"timestamp":"1407212148186"} stored data: 
{"version":1,"brokerid":2,"timestamp":"1407211911014"} (kafka.utils.ZkUtils$)
[2014-08-05 04:34:50,614] INFO I wrote this conflicted ephemeral node 
[{"version":1,"brokerid":2,"timestamp":"1407212148186"}] at /controller a while 
back in a different session, hence I will backoff for this node to be deleted 
by Zookeeper and retry (kafka.utils.ZkUtils$)
[2014-08-05 04:34:56,621] INFO conflict in /controller data: 
{"version":1,"brokerid":2,"timestamp":"1407212148186"} stored data: 
{"version":1,"brokerid":2,"timestamp":"1407211911014"} (kafka.utils.ZkUtils$)

the expected result that you get should be

[2014-08-05 04:07:20,917] INFO conflict in /brokers/ids/2 data: 
{"jmx_port":-1,"timestamp":"1407211640900","host":"192.168.30.3","version":1,"port":9092}
 stored data: {"jmx_port":-1,"timestamp":"140721119
9464","host":"192.168.30.2","version":1,"port":9092} (kafka.utils.ZkUtils$)
[2014-08-05 04:07:20,949] FATAL Fatal error during KafkaServerStable startup. 
Prepare to shutdown (kafka.server.KafkaServerStartable)
java.lang.RuntimeException: A broker is already registered on the path 
/brokers/ids/2. This probably indicates that you either have configured a 
brokerid that is already in use, or else you have shutdown 
this broker and restarted it faster than the zookeeper timeout so it appears to 
be re-registering.
at kafka.utils.ZkUtils$.registerBrokerInZk(ZkUtils.scala:205)
at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:57)
at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:44)
at kafka.server.KafkaServer.startup(KafkaServer.scala:103)
at 
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
[2014-08-05 04:07:20,952] INFO [Kafka Server 2], shutting down 
(kafka.server.KafkaServer)
[2014-08-05 04:07:20,954] INFO [Socket Server on Broker 2], Shutting down 
(kafka.network.SocketServer)
[2014-08-05 04:07:20,959] INFO [Socket Server on Broker 2], Shutdown completed 
(kafka.network.SocketServer)
[2014-08-05 04:07:20,960] INFO [Kafka Request Handler on Broker 2], shutting 
down (kafka.server.KafkaRequestHandlerPool)
[2014-08-05 04:07:20,992] INFO [Kafka Request Handler on Broker 2], shut down 
completely (kafka.server.KafkaRequestHandlerPool)
[2014-08-05 04:07:21,263] INFO [Replic

[jira] [Commented] (KAFKA-1387) Kafka getting stuck creating ephemeral node it has already created when two zookeeper sessions are established in a very short period of time

[Joe Stein (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085792#comment-14085792
 ] 

Joe Stein commented on KAFKA-1387:
--

Here is another way to reproduce this issue.  I have seen it a few times now 
with folks getting going with their clusters.

steps to reproduce.  install a 3 node zk ensemble with 3 brokers cluster

e.g. 

git clone https://github.com/stealthly/scala-kafka
git checkout -b zkbk3 origin/zkbk3
vagrant up provider=virtualbox

now setup each node in the cluster as you would broker 1,2,3 and the ensemble

e.g.

vagrant ssh zkbkOne
sudo su
cd /vagrant/vagrant/ && ./up.sh
vagrant ssh zkbkTwo
sudo su
cd /vagrant/vagrant/ && ./up.sh
vagrant ssh zkbkThree
sudo su
cd /vagrant/vagrant/ && ./up.sh

start up zookeeper on all 3 nodes
cd /opt/apache/kafka && bin/zookeeper-server-start.sh 
config/zookeeper.properties 1>>/tmp/zk.log 2>>/tmp/zk.log &

now, start up broker on node 2 only
cd /opt/apache/kafka && bin/kafka-server-start.sh config/server.properties 
1>>/tmp/bk.log 2>>/tmp/bk.log &

ok, now here is where it gets wonky

- change the broker.id int server 3 to = 2 
now you need to start up server 1 and 3 (even though it is 2) at the same time

cd /opt/apache/kafka && bin/kafka-server-start.sh config/server.properties 
1>>/tmp/bk.log 2>>/tmp/bk.log &
cd /opt/apache/kafka && bin/kafka-server-start.sh config/server.properties 
1>>/tmp/bk.log 2>>/tmp/bk.log &
( you can have two tabs, hit enter in one switch to other tab and hit enter is 
close enough to same time)

and you get this looping forever

2014-08-05 04:34:38,591] INFO I wrote this conflicted ephemeral node 
[{"version":1,"brokerid":2,"timestamp":"1407212148186"}] at /controller a while 
back in a different session, hence I will backoff for this node to be deleted 
by Zookeeper and retry (kafka.utils.ZkUtils$)
[2014-08-05 04:34:44,598] INFO conflict in /controller data: 
{"version":1,"brokerid":2,"timestamp":"1407212148186"} stored data: 
{"version":1,"brokerid":2,"timestamp":"1407211911014"} (kafka.utils.ZkUtils$)
[2014-08-05 04:34:44,601] INFO I wrote this conflicted ephemeral node 
[{"version":1,"brokerid":2,"timestamp":"1407212148186"}] at /controller a while 
back in a different session, hence I will backoff for this node to be deleted 
by Zookeeper and retry (kafka.utils.ZkUtils$)
[2014-08-05 04:34:50,610] INFO conflict in /controller data: 
{"version":1,"brokerid":2,"timestamp":"1407212148186"} stored data: 
{"version":1,"brokerid":2,"timestamp":"1407211911014"} (kafka.utils.ZkUtils$)
[2014-08-05 04:34:50,614] INFO I wrote this conflicted ephemeral node 
[{"version":1,"brokerid":2,"timestamp":"1407212148186"}] at /controller a while 
back in a different session, hence I will backoff for this node to be deleted 
by Zookeeper and retry (kafka.utils.ZkUtils$)
[2014-08-05 04:34:56,621] INFO conflict in /controller data: 
{"version":1,"brokerid":2,"timestamp":"1407212148186"} stored data: 
{"version":1,"brokerid":2,"timestamp":"1407211911014"} (kafka.utils.ZkUtils$)

the expected result that you get should be

[2014-08-05 04:07:20,917] INFO conflict in /brokers/ids/2 data: 
{"jmx_port":-1,"timestamp":"1407211640900","host":"192.168.30.3","version":1,"port":9092}
 stored data: {"jmx_port":-1,"timestamp":"140721119
9464","host":"192.168.30.2","version":1,"port":9092} (kafka.utils.ZkUtils$)
[2014-08-05 04:07:20,949] FATAL Fatal error during KafkaServerStable startup. 
Prepare to shutdown (kafka.server.KafkaServerStartable)
java.lang.RuntimeException: A broker is already registered on the path 
/brokers/ids/2. This probably indicates that you either have configured a 
brokerid that is already in use, or else you have shutdown 
this broker and restarted it faster than the zookeeper timeout so it appears to 
be re-registering.
at kafka.utils.ZkUtils$.registerBrokerInZk(ZkUtils.scala:205)
at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:57)
at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:44)
at kafka.server.KafkaServer.startup(KafkaServer.scala:103)
at 
kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:34)
at kafka.Kafka$.main(Kafka.scala:46)
at kafka.Kafka.main(Kafka.scala)
[2014-08-05 04:07:20,952] INFO [Kafka Server 2], shutting down 
(kafka.server.KafkaServer)
[2014-08-05 04:07:20,954] INFO [Socket Server on Broker 2], Shutting down 
(kafka.network.SocketServer)
[2014-08-05 04:07:20,959] INFO [Socket Server on Broker 2], Shutdown completed 
(kafka.network.SocketServer)
[2014-08-05 04:07:20,960] INFO [Kafka Request Handler on Broker 2], shutting 
down (kafka.server.KafkaRequestHandlerPool)
[2014-08-05 04:07:20,992] INFO [Kafka Request Handler on Broker 2], shut down 
completely (kafka.server.KafkaRequestHandlerPool)
[2014-08-05 04:07:21,263] INFO [Replica Manager on Broker 2]: Shut down 
(kafka.server.ReplicaManager)
[

[jira] [Resolved] (KAFKA-1571) MetadataeTest hangs

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao resolved KAFKA-1571.


   Resolution: Fixed
Fix Version/s: 0.8.2

Thanks for the review. Committed to trunk.

> MetadataeTest hangs
> ---
>
> Key: KAFKA-1571
> URL: https://issues.apache.org/jira/browse/KAFKA-1571
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Jun Rao
> Fix For: 0.8.2
>
> Attachments: KAFKA-1571.patch
>
>
> Saw the following stacktrace. 
> "Thread-47" prio=10 tid=0x7fb5b00a5000 nid=0x25de in Object.wait() 
> [0x7fb5af9f8000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0006b0925e40> (a 
> org.apache.kafka.clients.producer.internals.Metadata)
> at 
> org.apache.kafka.clients.producer.internals.Metadata.awaitUpdate(Metadata.java:107)
> - locked <0x0006b0925e40> (a 
> org.apache.kafka.clients.producer.internals.Metadata)
> at 
> org.apache.kafka.clients.producer.MetadataTest$1.run(MetadataTest.java:57)
> "Thread-46" prio=10 tid=0x7fb5b00a3800 nid=0x25dd in Object.wait() 
> [0x7fb5afbfa000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0006b0925e40> (a 
> org.apache.kafka.clients.producer.internals.Metadata)
> at 
> org.apache.kafka.clients.producer.internals.Metadata.awaitUpdate(Metadata.java:107)
> - locked <0x0006b0925e40> (a 
> org.apache.kafka.clients.producer.internals.Metadata)
> at 
> org.apache.kafka.clients.producer.MetadataTest$1.run(MetadataTest.java:57)
> "Test worker" prio=10 tid=0x7fb610891000 nid=0x25b1 in Object.wait() 
> [0x7fb5d4a5f000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0006b0926700> (a 
> org.apache.kafka.clients.producer.MetadataTest$1)
> at java.lang.Thread.join(Thread.java:1186)
> - locked <0x0006b0926700> (a 
> org.apache.kafka.clients.producer.MetadataTest$1)
> at java.lang.Thread.join(Thread.java:1239)
> at 
> org.apache.kafka.clients.producer.MetadataTest.testMetadata(MetadataTest.java:46)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 23895: Patch for KAFKA-1419

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23895/#review49559
---


Look good overall. 

We need to change the default scala version in bin/kafka-run-class.sh. 
Otherwise, the quickstart script will fail.


build.gradle


Since you are touching this part, could you also add clients:uploadArchives 
to uploadArchivesAll?


- Jun Rao


On Aug. 4, 2014, 2:43 p.m., Ivan Lyutov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23895/
> ---
> 
> (Updated Aug. 4, 2014, 2:43 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1419
> https://issues.apache.org/jira/browse/KAFKA-1419
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1419 - cross build for scala 2.11 - dropped scala 2.8 support - minor 
> bug fixes
> 
> 
> KAFKA-1419 - cross build for scala 2.11 - changed 2.11 specific dependency 
> version - updated scala version to 2.11.2 - added getBuffer to 
> ByteBufferMessageSet classes
> 
> 
> KAFKA-1419 - cross build for scala 2.11 - changed 2.11 specific dependency 
> version - updated scala version to 2.11.2 - added getBuffer to 
> ByteBufferMessageSet classes - removed annotations 2.8 file
> 
> 
> Diffs
> -
> 
>   build.gradle a72905df824ba68bed5d5170d18873c23e1782c9 
>   core/src/main/scala/kafka/javaapi/message/ByteBufferMessageSet.scala 
> fecee8d5f7b32f483bb1bfc6a5080d589906f9c4 
>   core/src/main/scala/kafka/message/ByteBufferMessageSet.scala 
> 73401c5ff34d08abce22267aa9c4d86632c6fb74 
>   core/src/main/scala/kafka/utils/Annotations_2.8.scala 
> 28269eb037109f7680b9da732e4baa51c9a594b6 
>   core/src/main/scala/kafka/utils/Annotations_2.9+.scala  
>   gradle.properties 4827769a3f8e34f0fe7e783eb58e44d4db04859b 
>   gradle/buildscript.gradle 225e0a82708bc5f390e5e2c1d4d9a0d06f491b95 
>   gradle/wrapper/gradle-wrapper.properties 
> 610282a699afc89a82203ef0e4e71ecc53761039 
>   scala.gradle ebd21b870c0746aade63248344ab65d9b5baf820 
> 
> Diff: https://reviews.apache.org/r/23895/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ivan Lyutov
> 
>

Re: Newer Zookeeper?

[Joe Stein]

I found an already open ticket in regards to this
https://issues.apache.org/jira/browse/KAFKA-1485

It also references a conflict with storm upgrading and testing and some
other conflicts too.

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/


On Mon, Aug 4, 2014 at 9:57 PM, Joe Stein  wrote:

> Thanks Todd, very good to know and learn!
>
> - Joestein
>
>
> On Mon, Aug 4, 2014 at 9:44 PM, Todd Palino 
> wrote:
>
>> We¹ve started running our test cluster against a Zookeeper 3.4.6 ensemble.
>> So far, we¹ve had no problems with it that were specific to ZK (since
>> we¹re using it for testing trunk version of Kafka, as well as mirror
>> maker, we have plenty of problems with it. Just none that are ZK). We¹re
>> probably going to start rolling that out to our Kafka clusters in the
>> staging environments in the next month or so. That¹s a bigger step, since
>> we treat those clusters more like production (they¹re staging for everyone
>> else, but we¹re infrastructure).
>>
>> -Todd
>>
>>
>> On 8/3/14, 9:56 PM, "Gwen Shapira"  wrote:
>>
>> >Hi,
>> >
>> >Kafka currently builds against Zookeeper 3.3.4, which is quite old.
>> >
>> >Perhaps we should move to the more recent 3.4.x branch?
>> >
>> >I tested the change on my system and the only impact is to
>> >EmbeddedZookeeper used in tests (it uses NIOServerCnxn.factory, which
>> >was refactored into its own class in 3.4).
>> >
>> >Here's what the change looks like:
>> >https://gist.github.com/gwenshap/d95b36e0bced53cab5bb
>> >
>> >Gwen
>>
>>
>

[jira] [Updated] (KAFKA-1485) Upgrade to Zookeeper 3.4.x

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1485:
-

Labels: newbie  (was: )

> Upgrade to Zookeeper 3.4.x
> --
>
> Key: KAFKA-1485
> URL: https://issues.apache.org/jira/browse/KAFKA-1485
> Project: Kafka
>  Issue Type: Wish
>Affects Versions: 0.8.1.1
>Reporter: Machiel Groeneveld
>  Labels: newbie
> Fix For: 0.8.2
>
>
> I can't run projects alongside Kafka that use zookeeper 3.4 jars. 3.4 has 
> been out for 2.5 years and seems to be ready for adoption.
> In particular Apache Storm will upgrade to Zookeeper 3.4.x in their next 
> 0.9.2 release. I can't run both versions in my tests at the same time. 
> The only compile problem I saw was in EmbeddedZookeeper.scala 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1485) Upgrade to Zookeeper 3.4.x

[Joe Stein (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joe Stein updated KAFKA-1485:
-

Fix Version/s: 0.8.2

> Upgrade to Zookeeper 3.4.x
> --
>
> Key: KAFKA-1485
> URL: https://issues.apache.org/jira/browse/KAFKA-1485
> Project: Kafka
>  Issue Type: Wish
>Affects Versions: 0.8.1.1
>Reporter: Machiel Groeneveld
> Fix For: 0.8.2
>
>
> I can't run projects alongside Kafka that use zookeeper 3.4 jars. 3.4 has 
> been out for 2.5 years and seems to be ready for adoption.
> In particular Apache Storm will upgrade to Zookeeper 3.4.x in their next 
> 0.9.2 release. I can't run both versions in my tests at the same time. 
> The only compile problem I saw was in EmbeddedZookeeper.scala 



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: [New Feature Request] Ability to Inject Queue Implementation Async Mode

[Joe Stein]

Is it possible there is another solution to the problem? I think if you
could better describe the problem(s) you are facing and how you are
architected some then you may get responses from others that perhaps have
faced the same problem with similar architectures ... or maybe folks can
chime in with solution(s) to the problem(s).  When only being presented
with solutions it is hard to say much about if it is problem that folks
will have and if this solution will work for them.

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/


On Mon, Aug 4, 2014 at 8:52 PM, Bhavesh Mistry 
wrote:

> Kafka Version:  0.8.x
>
> 1) Ability to define which messages get drooped (least recently instead of
> most recent in queue)
> 2) Try Unbounded Queue to find out the Upper Limit without drooping any
> messages for application (use case Stress test)
> 3) Priority Blocking Queue ( meaning a single Producer can send messages to
> multiple topic and I would like to give Delivery Priority to message for
> particular message for topic)
>
> We have use case to support #3 and #1 since we would like to deliver the
> Application Heartbeat first then any other event within the queue for any
> topics. To lower TCP connections, we only use one producer for 4 topics but
> one of the topics has priority for delivery.
>
> Please let me know if this is useful feature to have or not.
>
> Thanks in advance for great support !!
>
> Thanks,
>
> Bhavesh
>
> P.S.  Sorry for asking this question again, but last time there was no
> conclusion.
>

Re: Newer Zookeeper?

[Joe Stein]

Thanks Todd, very good to know and learn!

- Joestein

On Mon, Aug 4, 2014 at 9:44 PM, Todd Palino 
wrote:

> We¹ve started running our test cluster against a Zookeeper 3.4.6 ensemble.
> So far, we¹ve had no problems with it that were specific to ZK (since
> we¹re using it for testing trunk version of Kafka, as well as mirror
> maker, we have plenty of problems with it. Just none that are ZK). We¹re
> probably going to start rolling that out to our Kafka clusters in the
> staging environments in the next month or so. That¹s a bigger step, since
> we treat those clusters more like production (they¹re staging for everyone
> else, but we¹re infrastructure).
>
> -Todd
>
>
> On 8/3/14, 9:56 PM, "Gwen Shapira"  wrote:
>
> >Hi,
> >
> >Kafka currently builds against Zookeeper 3.3.4, which is quite old.
> >
> >Perhaps we should move to the more recent 3.4.x branch?
> >
> >I tested the change on my system and the only impact is to
> >EmbeddedZookeeper used in tests (it uses NIOServerCnxn.factory, which
> >was refactored into its own class in 3.4).
> >
> >Here's what the change looks like:
> >https://gist.github.com/gwenshap/d95b36e0bced53cab5bb
> >
> >Gwen
>
>

Re: Uniform Distribution of Messages for Topic Across Partitions Without Effecting Performance

[Joe Stein]

Bhavesh, take a look at
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Whyisdatanotevenlydistributedamongpartitionswhenapartitioningkeyisnotspecified
?

Maybe the root cause issue is something else? Even if producers produce
more or less than what they are producing you should be able to make it
random enough with a partitioner and a key.  I don't think you should need
more than what is in the FAQ but incase so maybe look into
http://en.wikipedia.org/wiki/MurmurHash as another hash option.

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/


On Mon, Aug 4, 2014 at 9:12 PM, Bhavesh Mistry 
wrote:

> How to achieve uniform distribution of non-keyed messages per topic across
> all partitions?
>
> We have tried to do this uniform distribution across partition using custom
> partitioning from each producer instance using round robing (
> count(messages) % number of partition for topic). This strategy results in
> very poor performance.  So we have switched back to random stickiness that
> Kafka provide out of box per some interval ( 10 minutes not sure exactly )
> per topic.
>
> The above strategy results in consumer side lags sometime for some
> partitions because we have some applications/producers  producing more
> messages for same topic than other servers.
>
> Can Kafka provide out of box uniform distribution by using coordination
> among all producers and rely on measure rate such as  # messages per minute
> or # of bytes produce per minute to achieve uniform distribution and
> coordinate stickiness of partition among hundreds of producers for same
> topic ?
>
> Thanks,
>
> Bhavesh
>

Re: Newer Zookeeper?

[Todd Palino]

We¹ve started running our test cluster against a Zookeeper 3.4.6 ensemble.
So far, we¹ve had no problems with it that were specific to ZK (since
we¹re using it for testing trunk version of Kafka, as well as mirror
maker, we have plenty of problems with it. Just none that are ZK). We¹re
probably going to start rolling that out to our Kafka clusters in the
staging environments in the next month or so. That¹s a bigger step, since
we treat those clusters more like production (they¹re staging for everyone
else, but we¹re infrastructure).

-Todd


On 8/3/14, 9:56 PM, "Gwen Shapira"  wrote:

>Hi,
>
>Kafka currently builds against Zookeeper 3.3.4, which is quite old.
>
>Perhaps we should move to the more recent 3.4.x branch?
>
>I tested the change on my system and the only impact is to
>EmbeddedZookeeper used in tests (it uses NIOServerCnxn.factory, which
>was refactored into its own class in 3.4).
>
>Here's what the change looks like:
>https://gist.github.com/gwenshap/d95b36e0bced53cab5bb
>
>Gwen

Uniform Distribution of Messages for Topic Across Partitions Without Effecting Performance

[Bhavesh Mistry]

How to achieve uniform distribution of non-keyed messages per topic across
all partitions?

We have tried to do this uniform distribution across partition using custom
partitioning from each producer instance using round robing (
count(messages) % number of partition for topic). This strategy results in
very poor performance.  So we have switched back to random stickiness that
Kafka provide out of box per some interval ( 10 minutes not sure exactly )
per topic.

The above strategy results in consumer side lags sometime for some
partitions because we have some applications/producers  producing more
messages for same topic than other servers.

Can Kafka provide out of box uniform distribution by using coordination
among all producers and rely on measure rate such as  # messages per minute
or # of bytes produce per minute to achieve uniform distribution and
coordinate stickiness of partition among hundreds of producers for same
topic ?

Thanks,

Bhavesh

[New Feature Request] Ability to Inject Queue Implementation Async Mode

[Bhavesh Mistry]

Kafka Version:  0.8.x

1) Ability to define which messages get drooped (least recently instead of
most recent in queue)
2) Try Unbounded Queue to find out the Upper Limit without drooping any
messages for application (use case Stress test)
3) Priority Blocking Queue ( meaning a single Producer can send messages to
multiple topic and I would like to give Delivery Priority to message for
particular message for topic)

We have use case to support #3 and #1 since we would like to deliver the
Application Heartbeat first then any other event within the queue for any
topics. To lower TCP connections, we only use one producer for 4 topics but
one of the topics has priority for delivery.

Please let me know if this is useful feature to have or not.

Thanks in advance for great support !!

Thanks,

Bhavesh

P.S.  Sorry for asking this question again, but last time there was no
conclusion.

Re: Review Request 24287: Patch for KAFKA-1571

[Guozhang Wang]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24287/#review49547
---

Ship it!


Ship It!

- Guozhang Wang


On Aug. 5, 2014, 12:39 a.m., Jun Rao wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/24287/
> ---
> 
> (Updated Aug. 5, 2014, 12:39 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1571
> https://issues.apache.org/jira/browse/KAFKA-1571
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Fix the race condition btw the main thread and the asyncFetch threads.
> 
> 
> Diffs
> -
> 
>   clients/src/test/java/org/apache/kafka/clients/producer/MetadataTest.java 
> 543304c8bb71d90b4af71b519d830a52595c4885 
> 
> Diff: https://reviews.apache.org/r/24287/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jun Rao
> 
>

[jira] [Updated] (KAFKA-1571) MetadataeTest hangs

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1571?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-1571:
---

Attachment: KAFKA-1571.patch

> MetadataeTest hangs
> ---
>
> Key: KAFKA-1571
> URL: https://issues.apache.org/jira/browse/KAFKA-1571
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Jun Rao
> Attachments: KAFKA-1571.patch
>
>
> Saw the following stacktrace. 
> "Thread-47" prio=10 tid=0x7fb5b00a5000 nid=0x25de in Object.wait() 
> [0x7fb5af9f8000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0006b0925e40> (a 
> org.apache.kafka.clients.producer.internals.Metadata)
> at 
> org.apache.kafka.clients.producer.internals.Metadata.awaitUpdate(Metadata.java:107)
> - locked <0x0006b0925e40> (a 
> org.apache.kafka.clients.producer.internals.Metadata)
> at 
> org.apache.kafka.clients.producer.MetadataTest$1.run(MetadataTest.java:57)
> "Thread-46" prio=10 tid=0x7fb5b00a3800 nid=0x25dd in Object.wait() 
> [0x7fb5afbfa000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0006b0925e40> (a 
> org.apache.kafka.clients.producer.internals.Metadata)
> at 
> org.apache.kafka.clients.producer.internals.Metadata.awaitUpdate(Metadata.java:107)
> - locked <0x0006b0925e40> (a 
> org.apache.kafka.clients.producer.internals.Metadata)
> at 
> org.apache.kafka.clients.producer.MetadataTest$1.run(MetadataTest.java:57)
> "Test worker" prio=10 tid=0x7fb610891000 nid=0x25b1 in Object.wait() 
> [0x7fb5d4a5f000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0006b0926700> (a 
> org.apache.kafka.clients.producer.MetadataTest$1)
> at java.lang.Thread.join(Thread.java:1186)
> - locked <0x0006b0926700> (a 
> org.apache.kafka.clients.producer.MetadataTest$1)
> at java.lang.Thread.join(Thread.java:1239)
> at 
> org.apache.kafka.clients.producer.MetadataTest.testMetadata(MetadataTest.java:46)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1571) MetadataeTest hangs

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1571?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085576#comment-14085576
 ] 

Jun Rao commented on KAFKA-1571:


Created reviewboard https://reviews.apache.org/r/24287/
 against branch origin/trunk

> MetadataeTest hangs
> ---
>
> Key: KAFKA-1571
> URL: https://issues.apache.org/jira/browse/KAFKA-1571
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: Jun Rao
> Attachments: KAFKA-1571.patch
>
>
> Saw the following stacktrace. 
> "Thread-47" prio=10 tid=0x7fb5b00a5000 nid=0x25de in Object.wait() 
> [0x7fb5af9f8000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0006b0925e40> (a 
> org.apache.kafka.clients.producer.internals.Metadata)
> at 
> org.apache.kafka.clients.producer.internals.Metadata.awaitUpdate(Metadata.java:107)
> - locked <0x0006b0925e40> (a 
> org.apache.kafka.clients.producer.internals.Metadata)
> at 
> org.apache.kafka.clients.producer.MetadataTest$1.run(MetadataTest.java:57)
> "Thread-46" prio=10 tid=0x7fb5b00a3800 nid=0x25dd in Object.wait() 
> [0x7fb5afbfa000]
>java.lang.Thread.State: TIMED_WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0006b0925e40> (a 
> org.apache.kafka.clients.producer.internals.Metadata)
> at 
> org.apache.kafka.clients.producer.internals.Metadata.awaitUpdate(Metadata.java:107)
> - locked <0x0006b0925e40> (a 
> org.apache.kafka.clients.producer.internals.Metadata)
> at 
> org.apache.kafka.clients.producer.MetadataTest$1.run(MetadataTest.java:57)
> "Test worker" prio=10 tid=0x7fb610891000 nid=0x25b1 in Object.wait() 
> [0x7fb5d4a5f000]
>java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x0006b0926700> (a 
> org.apache.kafka.clients.producer.MetadataTest$1)
> at java.lang.Thread.join(Thread.java:1186)
> - locked <0x0006b0926700> (a 
> org.apache.kafka.clients.producer.MetadataTest$1)
> at java.lang.Thread.join(Thread.java:1239)
> at 
> org.apache.kafka.clients.producer.MetadataTest.testMetadata(MetadataTest.java:46)



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 24287: Patch for KAFKA-1571

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24287/
---

Review request for kafka.


Bugs: KAFKA-1571
https://issues.apache.org/jira/browse/KAFKA-1571


Repository: kafka


Description
---

Fix the race condition btw the main thread and the asyncFetch threads.


Diffs
-

  clients/src/test/java/org/apache/kafka/clients/producer/MetadataTest.java 
543304c8bb71d90b4af71b519d830a52595c4885 

Diff: https://reviews.apache.org/r/24287/diff/


Testing
---


Thanks,

Jun Rao

[jira] [Created] (KAFKA-1571) MetadataeTest hangs

[Jun Rao (JIRA)]

Jun Rao created KAFKA-1571:
--

 Summary: MetadataeTest hangs
 Key: KAFKA-1571
 URL: https://issues.apache.org/jira/browse/KAFKA-1571
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.8.2
Reporter: Jun Rao
Assignee: Jun Rao


Saw the following stacktrace. 

"Thread-47" prio=10 tid=0x7fb5b00a5000 nid=0x25de in Object.wait() 
[0x7fb5af9f8000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0006b0925e40> (a 
org.apache.kafka.clients.producer.internals.Metadata)
at 
org.apache.kafka.clients.producer.internals.Metadata.awaitUpdate(Metadata.java:107)
- locked <0x0006b0925e40> (a 
org.apache.kafka.clients.producer.internals.Metadata)
at 
org.apache.kafka.clients.producer.MetadataTest$1.run(MetadataTest.java:57)

"Thread-46" prio=10 tid=0x7fb5b00a3800 nid=0x25dd in Object.wait() 
[0x7fb5afbfa000]
   java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0006b0925e40> (a 
org.apache.kafka.clients.producer.internals.Metadata)
at 
org.apache.kafka.clients.producer.internals.Metadata.awaitUpdate(Metadata.java:107)
- locked <0x0006b0925e40> (a 
org.apache.kafka.clients.producer.internals.Metadata)
at 
org.apache.kafka.clients.producer.MetadataTest$1.run(MetadataTest.java:57)

"Test worker" prio=10 tid=0x7fb610891000 nid=0x25b1 in Object.wait() 
[0x7fb5d4a5f000]
   java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x0006b0926700> (a 
org.apache.kafka.clients.producer.MetadataTest$1)
at java.lang.Thread.join(Thread.java:1186)
- locked <0x0006b0926700> (a 
org.apache.kafka.clients.producer.MetadataTest$1)
at java.lang.Thread.join(Thread.java:1239)
at 
org.apache.kafka.clients.producer.MetadataTest.testMetadata(MetadataTest.java:46)




--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1569) Create tool to test correctness of transactions end-to-end

[Raul Castro Fernandez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raul Castro Fernandez updated KAFKA-1569:
-

Attachment: KAFKA-1569.patch

> Create tool to test correctness of transactions end-to-end
> --
>
> Key: KAFKA-1569
> URL: https://issues.apache.org/jira/browse/KAFKA-1569
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Raul Castro Fernandez
>Assignee: Raul Castro Fernandez
>  Labels: transactions
> Attachments: KAFKA-1569.patch, KAFKA-1569.patch
>
>
> A producer tool that creates an input file, reads it and sends it to the 
> brokers according to some transaction configuration. And a consumer tool that 
> read data from brokers with transaction boundaries and writes it to a file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1569) Create tool to test correctness of transactions end-to-end

[Raul Castro Fernandez (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085385#comment-14085385
 ] 

Raul Castro Fernandez commented on KAFKA-1569:
--

Created reviewboard https://reviews.apache.org/r/24268/diff/
 against branch origin/transactional_messaging

> Create tool to test correctness of transactions end-to-end
> --
>
> Key: KAFKA-1569
> URL: https://issues.apache.org/jira/browse/KAFKA-1569
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Raul Castro Fernandez
>Assignee: Raul Castro Fernandez
>  Labels: transactions
> Attachments: KAFKA-1569.patch, KAFKA-1569.patch
>
>
> A producer tool that creates an input file, reads it and sends it to the 
> brokers according to some transaction configuration. And a consumer tool that 
> read data from brokers with transaction boundaries and writes it to a file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Issues getting IntelliJ set up

[Jonathan Natkins]

Yep, it looks like you win the game. Managed to get a test running by doing
this. Thanks a bunch, Gwen!


On Mon, Aug 4, 2014 at 1:57 PM, Gwen Shapira  wrote:

> I think I found it :)
>
> Go to "project settings"
> Pick your module (I picked "core")
> "Paths" tab
> And change the paths to:
> /Users/Natty/apache/kafka-new/core/build/classes/main
> and
> /Users/Natty/apache/kafka-new/core/build/classes/test
>
> Don't use the project inherited paths, because we have a build dir per
> module.
>
> Gwen
>
> On Mon, Aug 4, 2014 at 12:54 PM, Jonathan Natkins 
> wrote:
> > I did. I actually tried this from a completely clean repo (cloned a new
> > repo from github, changed gradle.properties, ran `gradlew idea`, then
> > imported into IntelliJ)
> >
> >
> > On Mon, Aug 4, 2014 at 12:18 PM, Timothy Chen  wrote:
> >
> >> Hi Johnathan,
> >>
> >> Did you update your scala version before you run gradle idea?
> >>
> >> Also try cleaning up all the artifacts and try it again, as perhaps
> >> your intellij is not picking up the right version and from the right
> >> build folder.
> >>
> >> Tim
> >>
> >> On Mon, Aug 4, 2014 at 12:09 PM, Jonathan Natkins  >
> >> wrote:
> >> > Hi,
> >> >
> >> > I've been having some issues getting IntelliJ set up...I followed all
> the
> >> > instructions on the wiki, and I've successfully imported the project,
> and
> >> > run the jar Gradle target successfully. However, when I try to run a
> test
> >> > in the IDE, I get a number of errors:
> >> >
> >> >
> >>
> /Users/Natty/apache/kafka-new/core/src/main/scala/kafka/tools/KafkaMigrationTool.java
> >> > Error:(21, 30) java: package kafka.javaapi.producer does not exist
> >> > Error:(22, 22) java: package kafka.producer does not exist
> >> > Error:(23, 22) java: package kafka.producer does not exist
> >> > Error:(24, 19) java: cannot find symbol
> >> >   symbol:   class Utils
> >> >   location: package kafka.utils
> >> > Error:(303, 39) java: cannot find symbol
> >> >   symbol:   class KeyedMessage
> >> >   location: class kafka.tools.KafkaMigrationTool.MigrationThread
> >> >
> >> > And so on.
> >> >
> >> > The two classes that seem to be causing trouble are KafkaMigrationTool
> >> and
> >> > ConsumerConnector. Has anybody run into this? Anyone know how to get
> >> around
> >> > this issue?
> >> >
> >> > Thanks a lot,
> >> > Natty
> >>
>

Review Request 24268: Patch for KAFKA-1569

[Raul Castro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24268/
---

Review request for kafka.


Bugs: KAFKA-1569
https://issues.apache.org/jira/browse/KAFKA-1569


Repository: kafka


Description
---

KAFKA-1569; end to end correctness test for transactions


Diffs
-

  bin/kafka-tx-consumer-test.sh PRE-CREATION 
  bin/kafka-tx-producer-test.sh PRE-CREATION 
  core/src/main/scala/kafka/tools/TransactionalConsumerTest.scala PRE-CREATION 
  core/src/main/scala/kafka/tools/TransactionalProducerTest.scala PRE-CREATION 

Diff: https://reviews.apache.org/r/24268/diff/


Testing
---


Thanks,

Raul Castro Fernandez

[jira] [Commented] (KAFKA-1541) Add transactional request definitions to clients package

[Raul Castro Fernandez (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085369#comment-14085369
 ] 

Raul Castro Fernandez commented on KAFKA-1541:
--

Created reviewboard https://reviews.apache.org/r/24267/diff/
 against branch origin/transactional_messaging

>  Add transactional request definitions to clients package
> -
>
> Key: KAFKA-1541
> URL: https://issues.apache.org/jira/browse/KAFKA-1541
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joel Koshy
>Assignee: Raul Castro Fernandez
>  Labels: transactions
> Attachments: KAFKA-1541.patch, KAFKA-1541.patch, KAFKA-1541.patch
>
>
> Separate jira for this since KAFKA-1522 only adds definitions to the core 
> package.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1541) Add transactional request definitions to clients package

[Raul Castro Fernandez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raul Castro Fernandez updated KAFKA-1541:
-

Attachment: KAFKA-1541.patch

>  Add transactional request definitions to clients package
> -
>
> Key: KAFKA-1541
> URL: https://issues.apache.org/jira/browse/KAFKA-1541
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joel Koshy
>Assignee: Raul Castro Fernandez
>  Labels: transactions
> Attachments: KAFKA-1541.patch, KAFKA-1541.patch, KAFKA-1541.patch
>
>
> Separate jira for this since KAFKA-1522 only adds definitions to the core 
> package.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 24267: Patch for KAFKA-1541

[Raul Castro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24267/
---

Review request for kafka.


Bugs: KAFKA-1541
https://issues.apache.org/jira/browse/KAFKA-1541


Repository: kafka


Description
---

KAFKA-1541; tx request definitions in client package


Diffs
-

  clients/src/main/java/org/apache/kafka/common/protocol/ApiKeys.java 
6fe7573973832615976defa37fe0dfbb8f911939 
  clients/src/main/java/org/apache/kafka/common/protocol/Errors.java 
3374bd98be8e565608c4e764ed10afdae383fb6f 
  clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 
044b03061802ee5e8ea4f1995fb0988e1a70e9a7 
  
clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitRequest.java 
PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/requests/TransactionCoordinatorMetadataRequest.java
 PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/requests/TransactionCoordinatorMetadataResponse.java
 PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/requests/TransactionRequest.java 
PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/requests/TransactionResponse.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/24267/diff/


Testing
---


Thanks,

Raul Castro Fernandez

[jira] [Commented] (KAFKA-1524) Implement transactional producer

[Raul Castro Fernandez (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085352#comment-14085352
 ] 

Raul Castro Fernandez commented on KAFKA-1524:
--

Created reviewboard https://reviews.apache.org/r/24265/diff/
 against branch origin/transactional_messaging

> Implement transactional producer
> 
>
> Key: KAFKA-1524
> URL: https://issues.apache.org/jira/browse/KAFKA-1524
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joel Koshy
>Assignee: Raul Castro Fernandez
>  Labels: transactions
> Attachments: KAFKA-1524.patch, KAFKA-1524.patch, KAFKA-1524.patch
>
>
> Implement the basic transactional producer functionality as outlined in 
> https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka
> The scope of this jira is basic functionality (i.e., to be able to begin and 
> commit or abort a transaction) without the failure scenarios.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1524) Implement transactional producer

[Raul Castro Fernandez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raul Castro Fernandez updated KAFKA-1524:
-

Attachment: KAFKA-1524.patch

> Implement transactional producer
> 
>
> Key: KAFKA-1524
> URL: https://issues.apache.org/jira/browse/KAFKA-1524
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joel Koshy
>Assignee: Raul Castro Fernandez
>  Labels: transactions
> Attachments: KAFKA-1524.patch, KAFKA-1524.patch, KAFKA-1524.patch
>
>
> Implement the basic transactional producer functionality as outlined in 
> https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka
> The scope of this jira is basic functionality (i.e., to be able to begin and 
> commit or abort a transaction) without the failure scenarios.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 24265: Patch for KAFKA-1524

[Raul Castro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24265/
---

Review request for kafka.


Bugs: KAFKA-1524
https://issues.apache.org/jira/browse/KAFKA-1524


Repository: kafka


Description
---

KAFKA-1524; transactional producer


Diffs
-

  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
522881c972ca42ff4dfb6237a2db15b625334d7e 
  
clients/src/main/java/org/apache/kafka/clients/producer/AbortTransactionException.java
 PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/clients/producer/InvalidTransactionStatusException.java
 PRE-CREATION 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
00775abbcac850b0f2bb9a70b6fbc7cdf319bcf6 
  clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 
c0f1d57e0feb894d9f246058cd0396461afe3225 
  clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 
36e8398416036cab84faad1f07159e5adefd8086 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
f9de4af426449cceca12a8de9a9f54a6241d28d8 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 1ed3c28b436d28381d9402896e32d16f2586c65e 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 dd0af8aee98abed5d4a0dc50989e37888bb353fe 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
6fb5b82dedb48d946d1ac1ec7a535bddfdc693fa 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionContext.java
 PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionControl.java
 PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/errors/TransactionCoordinatorNotAvailableException.java
 PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/errors/TransactionFailedException.java
 PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/record/Compressor.java 
0323f5f7032dceb49d820c17a41b78c56591ffc4 
  clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 
759f577eaf0e7d28a84926d4aa30f4ef0cb27bc2 
  clients/src/main/java/org/apache/kafka/common/record/Record.java 
10df9fd8d3f4ec8c277650fa7eab269f3ea30d85 
  
clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java
 93b58d02eac0f8ca28440e3e0ebea28ed3a7673c 
  clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java 
5489acac6806b3ae5e6d568d401d5a20c86cac05 
  
clients/src/test/java/org/apache/kafka/clients/producer/TransactionContextTest.java
 PRE-CREATION 
  clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java 
94a11121e207d5cf94dbc94443a8aa7edf387782 

Diff: https://reviews.apache.org/r/24265/diff/


Testing
---


Thanks,

Raul Castro Fernandez

Re: Issues getting IntelliJ set up

[Gwen Shapira]

I think I found it :)

Go to "project settings"
Pick your module (I picked "core")
"Paths" tab
And change the paths to:
/Users/Natty/apache/kafka-new/core/build/classes/main
and
/Users/Natty/apache/kafka-new/core/build/classes/test

Don't use the project inherited paths, because we have a build dir per module.

Gwen

On Mon, Aug 4, 2014 at 12:54 PM, Jonathan Natkins  wrote:
> I did. I actually tried this from a completely clean repo (cloned a new
> repo from github, changed gradle.properties, ran `gradlew idea`, then
> imported into IntelliJ)
>
>
> On Mon, Aug 4, 2014 at 12:18 PM, Timothy Chen  wrote:
>
>> Hi Johnathan,
>>
>> Did you update your scala version before you run gradle idea?
>>
>> Also try cleaning up all the artifacts and try it again, as perhaps
>> your intellij is not picking up the right version and from the right
>> build folder.
>>
>> Tim
>>
>> On Mon, Aug 4, 2014 at 12:09 PM, Jonathan Natkins 
>> wrote:
>> > Hi,
>> >
>> > I've been having some issues getting IntelliJ set up...I followed all the
>> > instructions on the wiki, and I've successfully imported the project, and
>> > run the jar Gradle target successfully. However, when I try to run a test
>> > in the IDE, I get a number of errors:
>> >
>> >
>> /Users/Natty/apache/kafka-new/core/src/main/scala/kafka/tools/KafkaMigrationTool.java
>> > Error:(21, 30) java: package kafka.javaapi.producer does not exist
>> > Error:(22, 22) java: package kafka.producer does not exist
>> > Error:(23, 22) java: package kafka.producer does not exist
>> > Error:(24, 19) java: cannot find symbol
>> >   symbol:   class Utils
>> >   location: package kafka.utils
>> > Error:(303, 39) java: cannot find symbol
>> >   symbol:   class KeyedMessage
>> >   location: class kafka.tools.KafkaMigrationTool.MigrationThread
>> >
>> > And so on.
>> >
>> > The two classes that seem to be causing trouble are KafkaMigrationTool
>> and
>> > ConsumerConnector. Has anybody run into this? Anyone know how to get
>> around
>> > this issue?
>> >
>> > Thanks a lot,
>> > Natty
>>

[jira] [Updated] (KAFKA-1570) sbt assembly-package-dependency fails with errors

[Xuri Nagarin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xuri Nagarin updated KAFKA-1570:


  Component/s: (was: website)
   build
  Description: 
xnag@xnag-linux:~/Downloads/kafka-0.8.1.1-src$ sbt assembly-package-dependency
[info] Set current project to kafka-0-8-1-1-src (in build 
file:/home/xnag/Downloads/kafka-0.8.1.1-src/)
[error] Not a valid command: assembly-package-dependency
[error] Not a valid project ID: assembly-package-dependency
[error] Expected ':' (if selecting a configuration)
[error] Not a valid key: assembly-package-dependency (similar: sbt-dependency)
[error] assembly-package-dependency
[error]^


xnag@xnag-linux:~/Downloads/kafka-0.8.1.1-src$ sbt about
[info] Set current project to kafka-0-8-1-1-src (in build 
file:/home/xnag/Downloads/kafka-0.8.1.1-src/)
[info] This is sbt 0.13.5
[info] The current project is 
{file:/home/xnag/Downloads/kafka-0.8.1.1-src/}kafka-0-8-1-1-src 0.1-SNAPSHOT
[info] The current project is built against Scala 2.10.4
[info] Available Plugins: sbt.plugins.IvyPlugin, sbt.plugins.JvmPlugin, 
sbt.plugins.CorePlugin, sbt.plugins.JUnitXmlReportPlugin
[info] sbt, sbt plugins, and build definitions are using Scala 2.10.4


xnag@xnag-linux:~/Downloads/kafka-0.8.1.1-src$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:Ubuntu 12.04.4 LTS
Release:12.04
Codename:   precise

xnag@xnag-linux:~/Downloads/kafka-0.8.1.1-src$ sbt tasks
[info] Set current project to kafka-0-8-1-1-src (in build 
file:/home/xnag/Downloads/kafka-0.8.1.1-src/)

This is a list of tasks defined for the current project.
It does not list the scopes the tasks are defined in; use the 'inspect' command 
for that.
Tasks produce values.  Use the 'show' command to run the task and print the 
resulting value.

  cleanDeletes files produced by the build, such as generated 
sources, compiled classes, and task caches.
  compile  Compiles sources.
  console  Starts the Scala interpreter with the project classes on the 
classpath.
  consoleProject   Starts the Scala interpreter with the sbt and the build 
definition on the classpath and useful imports.
  consoleQuick Starts the Scala interpreter with the project dependencies 
on the classpath.
  copyResourcesCopies resources to the output directory.
  doc  Generates API documentation.
  package  Produces the main artifact, such as a binary jar.  This is 
typically an alias for the task that actually does the packaging.
  packageBin   Produces a main artifact, such as a binary jar.
  packageDoc   Produces a documentation artifact, such as a jar containing 
API documentation.
  packageSrc   Produces a source artifact, such as a jar containing sources 
and resources.
  publish  Publishes artifacts to a repository.
  publishLocal Publishes artifacts to the local Ivy repository.
  publishM2Publishes artifacts to the local Maven repository.
  run  Runs a main class, passing along arguments provided on the 
command line.
  runMain  Runs the main class selected by the first argument, passing 
the remaining arguments to the main method.
  test Executes all tests.
  testOnly Executes the tests provided as arguments or all tests if no 
arguments are provided.
  testQuickExecutes the tests that either failed before, were not run 
or whose transitive dependencies changed, among those provided as arguments.
  update   Resolves and optionally retrieves dependencies, producing a 
report.

More tasks may be viewed by increasing verbosity.  See 'help tasks'.



  was:
https://kafka.apache.org/08/quickstart.html says:
> ./sbt update
> ./sbt package
> ./sbt assembly-package-dependency

but  assembly-package-dependency fails and is actually not needed to run the 
rest of the code.

  Environment: (was: 0.8 Git Revision 731ba90)
Affects Version/s: (was: 0.8.0)
   0.8.1.1

> sbt assembly-package-dependency fails with errors
> -
>
> Key: KAFKA-1570
> URL: https://issues.apache.org/jira/browse/KAFKA-1570
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.8.1.1
>Reporter: Xuri Nagarin
>
> xnag@xnag-linux:~/Downloads/kafka-0.8.1.1-src$ sbt assembly-package-dependency
> [info] Set current project to kafka-0-8-1-1-src (in build 
> file:/home/xnag/Downloads/kafka-0.8.1.1-src/)
> [error] Not a valid command: assembly-package-dependency
> [error] Not a valid project ID: assembly-package-dependency
> [error] Expected ':' (if selecting a configuration)
> [error] Not a valid key: assembly-package-dependency (similar: sbt-dependency)
> [error] assembly-pack

Re: Newer Zookeeper?

[Joe Stein]

If Kafka installations are missing something(s) by not having or using the
latest Zookeeper from a feature or stability perspective that would be
something to understand maybe you could help with that Gwen?

I know one of the implementations used this Hadoop version
http://docs.hortonworks.com/HDPDocuments/HDP2/HDP-2.1.3/bk_releasenotes_hdp_2.1/content/ch_relnotes-hdp-2.1.3-product.html
which appears to be using Zk 3.4.5.  I will have to check on the other two
(someone reminded me we saw this more than twice after I sent the email).
 I think maybe one of them was CDH but don't recall off the top of my head
it was a while ago.

A reason why another zookeeper cluster for Kafka vs other software systems
(Hadoop, Mesos, etc) is to separate risk of dependent services. One
zookeeper cluster can now take down more systems when it goes down (for
whatever reason, rogue server/code, upgrade, whatever) and becomes one big
single point of failure for everything.  If you aren't using zookeeper for
anything else that is mission critical it might not matter, it is relative
(and have seen this too of course).

We have also found deploying zookeeper to Mesos very (very (very)))
fruitful for dealing with and managing multiple zookeeper ensembles without
any headaches of course you can't do that with the Zookeeper ensemble
for Mesos but that goes back to my separation.

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/


On Mon, Aug 4, 2014 at 12:36 PM, Gwen Shapira  wrote:

> Also, specific Zookeeper 3.4.X version where loss of quorum occurred will
> help.
> 3.4.5 fixed some pretty serious issues around hanging.
>
> Gwen
>
> On Mon, Aug 4, 2014 at 9:29 AM, Gwen Shapira 
> wrote:
> > Thanks for the heads-up, Joe.
> >
> > We've been shipping Zookeeper 3.4.X for over  two years now (since
> > CDH4.0) and have many production customers. I'll check if there are
> > any known issues with breaking quorum. In any case I will take your
> > comments into account and see if I can arrange for extra testing.
> >
> > Can you share more information about the 3.4.X issues you were seeing?
> > Was there especially large clusters involved? large number of
> > consumers?
> >
> > Also, I'm curious to hear more about the reasons for separate ZK
> > cluster. I can see why you'll want it if you have thousands of
> > consumers, but are there other reasons? Multiple zookeeper installs
> > can be a pain to manage.
> >
> > Gwen
> >
> >
> >
> > On Mon, Aug 4, 2014 at 7:52 AM, Joe Stein  wrote:
> >> I have heard issues from installations running 3.4.X that I have not
> heard
> >> from installations running 3.3.X (i.e. zk breaking quorum and cluster
> going
> >> down).
> >>
> >> In none of these cases did I have an opportunity to isolate and
> reproduce
> >> and confirm the issue happening and caused by 3.4.X. Moving to 3.3.x was
> >> agreed to being a lower risk/cost solution to the problem. Once on 3.3.X
> >> the issues didn't happen again.
> >>
> >> So I can't say for sure if there are issues with running 3.4.X but I
> would
> >> suggest some due diligence in testing and production operation to
> validate
> >> that every case that Kafka requires operates correctly (and over some
> >> time).  There is a cost to this so some company(s) will have to take
> that
> >> investment and do some cost vs the benefit of moving to 3.4.x.
> >>
> >> I currently recommend running a separate ZK cluster for Kafka production
> >> and not chroot into an existing one except for test/qa/dev.
> >>
> >> I don't know what others experience is with 3.4.X as I said the issues I
> >> have seen could have been coincidence.
> >>
> >> /***
> >>  Joe Stein
> >>  Founder, Principal Consultant
> >>  Big Data Open Source Security LLC
> >>  http://www.stealth.ly
> >>  Twitter: @allthingshadoop 
> >> /
> >>
> >>
> >> On Mon, Aug 4, 2014 at 12:56 AM, Gwen Shapira 
> wrote:
> >>
> >>> Hi,
> >>>
> >>> Kafka currently builds against Zookeeper 3.3.4, which is quite old.
> >>>
> >>> Perhaps we should move to the more recent 3.4.x branch?
> >>>
> >>> I tested the change on my system and the only impact is to
> >>> EmbeddedZookeeper used in tests (it uses NIOServerCnxn.factory, which
> >>> was refactored into its own class in 3.4).
> >>>
> >>> Here's what the change looks like:
> >>> https://gist.github.com/gwenshap/d95b36e0bced53cab5bb
> >>>
> >>> Gwen
> >>>
>

[jira] [Commented] (KAFKA-1569) Create tool to test correctness of transactions end-to-end

[Raul Castro Fernandez (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085193#comment-14085193
 ] 

Raul Castro Fernandez commented on KAFKA-1569:
--

Created reviewboard https://reviews.apache.org/r/24255/diff/
 against branch origin/transactional_messaging

> Create tool to test correctness of transactions end-to-end
> --
>
> Key: KAFKA-1569
> URL: https://issues.apache.org/jira/browse/KAFKA-1569
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Raul Castro Fernandez
>Assignee: Raul Castro Fernandez
>  Labels: transactions
> Attachments: KAFKA-1569.patch
>
>
> A producer tool that creates an input file, reads it and sends it to the 
> brokers according to some transaction configuration. And a consumer tool that 
> read data from brokers with transaction boundaries and writes it to a file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1569) Create tool to test correctness of transactions end-to-end

[Raul Castro Fernandez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raul Castro Fernandez updated KAFKA-1569:
-

Attachment: KAFKA-1569.patch

> Create tool to test correctness of transactions end-to-end
> --
>
> Key: KAFKA-1569
> URL: https://issues.apache.org/jira/browse/KAFKA-1569
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Raul Castro Fernandez
>Assignee: Raul Castro Fernandez
>  Labels: transactions
> Attachments: KAFKA-1569.patch
>
>
> A producer tool that creates an input file, reads it and sends it to the 
> brokers according to some transaction configuration. And a consumer tool that 
> read data from brokers with transaction boundaries and writes it to a file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Created] (KAFKA-1570) sbt assembly-package-dependency fails with errors

[Xuri Nagarin (JIRA)]

Xuri Nagarin created KAFKA-1570:
---

 Summary: sbt assembly-package-dependency fails with errors
 Key: KAFKA-1570
 URL: https://issues.apache.org/jira/browse/KAFKA-1570
 Project: Kafka
  Issue Type: Bug
  Components: website
Affects Versions: 0.8.0
 Environment: 0.8 Git Revision 731ba90
Reporter: Xuri Nagarin


https://kafka.apache.org/08/quickstart.html says:
> ./sbt update
> ./sbt package
> ./sbt assembly-package-dependency

but  assembly-package-dependency fails and is actually not needed to run the 
rest of the code.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 24255: Patch for KAFKA-1569

[Raul Castro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24255/
---

Review request for kafka.


Bugs: KAFKA-1569
https://issues.apache.org/jira/browse/KAFKA-1569


Repository: kafka


Description
---

KAFKA-1569; Create tool to test correctness of transactions end to end


Diffs
-

  bin/kafka-tx-consumer-test.sh PRE-CREATION 
  bin/kafka-tx-producer-test.sh PRE-CREATION 
  core/src/main/scala/kafka/tools/TransactionalConsumerTest.scala PRE-CREATION 
  core/src/main/scala/kafka/tools/TransactionalProducerTest.scala PRE-CREATION 

Diff: https://reviews.apache.org/r/24255/diff/


Testing
---


Thanks,

Raul Castro Fernandez

Re: Issues getting IntelliJ set up

[Jonathan Natkins]

I did. I actually tried this from a completely clean repo (cloned a new
repo from github, changed gradle.properties, ran `gradlew idea`, then
imported into IntelliJ)


On Mon, Aug 4, 2014 at 12:18 PM, Timothy Chen  wrote:

> Hi Johnathan,
>
> Did you update your scala version before you run gradle idea?
>
> Also try cleaning up all the artifacts and try it again, as perhaps
> your intellij is not picking up the right version and from the right
> build folder.
>
> Tim
>
> On Mon, Aug 4, 2014 at 12:09 PM, Jonathan Natkins 
> wrote:
> > Hi,
> >
> > I've been having some issues getting IntelliJ set up...I followed all the
> > instructions on the wiki, and I've successfully imported the project, and
> > run the jar Gradle target successfully. However, when I try to run a test
> > in the IDE, I get a number of errors:
> >
> >
> /Users/Natty/apache/kafka-new/core/src/main/scala/kafka/tools/KafkaMigrationTool.java
> > Error:(21, 30) java: package kafka.javaapi.producer does not exist
> > Error:(22, 22) java: package kafka.producer does not exist
> > Error:(23, 22) java: package kafka.producer does not exist
> > Error:(24, 19) java: cannot find symbol
> >   symbol:   class Utils
> >   location: package kafka.utils
> > Error:(303, 39) java: cannot find symbol
> >   symbol:   class KeyedMessage
> >   location: class kafka.tools.KafkaMigrationTool.MigrationThread
> >
> > And so on.
> >
> > The two classes that seem to be causing trouble are KafkaMigrationTool
> and
> > ConsumerConnector. Has anybody run into this? Anyone know how to get
> around
> > this issue?
> >
> > Thanks a lot,
> > Natty
>

[jira] [Updated] (KAFKA-1569) Create tool to test correctness of transactions end-to-end

[Raul Castro Fernandez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raul Castro Fernandez updated KAFKA-1569:
-

Summary: Create tool to test correctness of transactions end-to-end  (was: 
Create tool to test end-to-end correctness when using transactions)

> Create tool to test correctness of transactions end-to-end
> --
>
> Key: KAFKA-1569
> URL: https://issues.apache.org/jira/browse/KAFKA-1569
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Raul Castro Fernandez
>Assignee: Raul Castro Fernandez
>  Labels: transactions
>
> A producer tool that creates an input file, reads it and sends it to the 
> brokers according to some transaction configuration. And a consumer tool that 
> read data from brokers with transaction boundaries and writes it to a file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1569) Create tool to test end-to-end correctness when using transactions

[Raul Castro Fernandez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raul Castro Fernandez updated KAFKA-1569:
-

Summary: Create tool to test end-to-end correctness when using transactions 
 (was: Creat tool to test end-to-end correctness in transactional mode)

> Create tool to test end-to-end correctness when using transactions
> --
>
> Key: KAFKA-1569
> URL: https://issues.apache.org/jira/browse/KAFKA-1569
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Raul Castro Fernandez
>Assignee: Raul Castro Fernandez
>  Labels: transactions
>
> A producer tool that creates an input file, reads it and sends it to the 
> brokers according to some transaction configuration. And a consumer tool that 
> read data from brokers with transaction boundaries and writes it to a file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1541) Add transactional request definitions to clients package

[Raul Castro Fernandez (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085171#comment-14085171
 ] 

Raul Castro Fernandez commented on KAFKA-1541:
--

Created reviewboard https://reviews.apache.org/r/24253/diff/
 against branch origin/transactional_messaging

>  Add transactional request definitions to clients package
> -
>
> Key: KAFKA-1541
> URL: https://issues.apache.org/jira/browse/KAFKA-1541
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joel Koshy
>Assignee: Raul Castro Fernandez
>  Labels: transactions
> Attachments: KAFKA-1541.patch, KAFKA-1541.patch
>
>
> Separate jira for this since KAFKA-1522 only adds definitions to the core 
> package.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1541) Add transactional request definitions to clients package

[Raul Castro Fernandez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raul Castro Fernandez updated KAFKA-1541:
-

Attachment: KAFKA-1541.patch

>  Add transactional request definitions to clients package
> -
>
> Key: KAFKA-1541
> URL: https://issues.apache.org/jira/browse/KAFKA-1541
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joel Koshy
>Assignee: Raul Castro Fernandez
>  Labels: transactions
> Attachments: KAFKA-1541.patch, KAFKA-1541.patch
>
>
> Separate jira for this since KAFKA-1522 only adds definitions to the core 
> package.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 24253: Patch for KAFKA-1541

[Raul Castro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24253/
---

Review request for kafka.


Bugs: KAFKA-1541
https://issues.apache.org/jira/browse/KAFKA-1541


Repository: kafka


Description
---

KAFKA-1541; add transactional req definitions to clients package


Diffs
-

  clients/src/main/java/org/apache/kafka/common/protocol/ApiKeys.java 
6fe7573973832615976defa37fe0dfbb8f911939 
  clients/src/main/java/org/apache/kafka/common/protocol/Errors.java 
3374bd98be8e565608c4e764ed10afdae383fb6f 
  clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 
044b03061802ee5e8ea4f1995fb0988e1a70e9a7 
  
clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitRequest.java 
PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/requests/TransactionCoordinatorMetadataRequest.java
 PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/requests/TransactionCoordinatorMetadataResponse.java
 PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/requests/TransactionRequest.java 
PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/requests/TransactionResponse.java 
PRE-CREATION 

Diff: https://reviews.apache.org/r/24253/diff/


Testing
---


Thanks,

Raul Castro Fernandez

Re: Issues getting IntelliJ set up

[Timothy Chen]

Hi Johnathan,

Did you update your scala version before you run gradle idea?

Also try cleaning up all the artifacts and try it again, as perhaps
your intellij is not picking up the right version and from the right
build folder.

Tim

On Mon, Aug 4, 2014 at 12:09 PM, Jonathan Natkins  wrote:
> Hi,
>
> I've been having some issues getting IntelliJ set up...I followed all the
> instructions on the wiki, and I've successfully imported the project, and
> run the jar Gradle target successfully. However, when I try to run a test
> in the IDE, I get a number of errors:
>
> /Users/Natty/apache/kafka-new/core/src/main/scala/kafka/tools/KafkaMigrationTool.java
> Error:(21, 30) java: package kafka.javaapi.producer does not exist
> Error:(22, 22) java: package kafka.producer does not exist
> Error:(23, 22) java: package kafka.producer does not exist
> Error:(24, 19) java: cannot find symbol
>   symbol:   class Utils
>   location: package kafka.utils
> Error:(303, 39) java: cannot find symbol
>   symbol:   class KeyedMessage
>   location: class kafka.tools.KafkaMigrationTool.MigrationThread
>
> And so on.
>
> The two classes that seem to be causing trouble are KafkaMigrationTool and
> ConsumerConnector. Has anybody run into this? Anyone know how to get around
> this issue?
>
> Thanks a lot,
> Natty

Issues getting IntelliJ set up

[Jonathan Natkins]

Hi,

I've been having some issues getting IntelliJ set up...I followed all the
instructions on the wiki, and I've successfully imported the project, and
run the jar Gradle target successfully. However, when I try to run a test
in the IDE, I get a number of errors:

/Users/Natty/apache/kafka-new/core/src/main/scala/kafka/tools/KafkaMigrationTool.java
Error:(21, 30) java: package kafka.javaapi.producer does not exist
Error:(22, 22) java: package kafka.producer does not exist
Error:(23, 22) java: package kafka.producer does not exist
Error:(24, 19) java: cannot find symbol
  symbol:   class Utils
  location: package kafka.utils
Error:(303, 39) java: cannot find symbol
  symbol:   class KeyedMessage
  location: class kafka.tools.KafkaMigrationTool.MigrationThread

And so on.

The two classes that seem to be causing trouble are KafkaMigrationTool and
ConsumerConnector. Has anybody run into this? Anyone know how to get around
this issue?

Thanks a lot,
Natty

[jira] Subscription: outstanding kafka patches

[jira]

Issue Subscription
Filter: outstanding kafka patches (117 issues)
The list of outstanding kafka patches
Subscriber: kafka-mailing-list

Key Summary
KAFKA-1567  Metric memory leaking after closing the clients
https://issues.apache.org/jira/browse/KAFKA-1567
KAFKA-1561  Data Loss for Incremented Replica Factor and Leader Election
https://issues.apache.org/jira/browse/KAFKA-1561
KAFKA-1560  Make arguments to jira-python API more explicit in 
kafka-patch-review's get_jira() 
https://issues.apache.org/jira/browse/KAFKA-1560
KAFKA-1559  Upgrade Gradle wrapper to Gradle 2.0
https://issues.apache.org/jira/browse/KAFKA-1559
KAFKA-1550  Patch review tool should use git format-patch to generate patch
https://issues.apache.org/jira/browse/KAFKA-1550
KAFKA-1543  Changing replication factor
https://issues.apache.org/jira/browse/KAFKA-1543
KAFKA-1541   Add transactional request definitions to clients package
https://issues.apache.org/jira/browse/KAFKA-1541
KAFKA-1536  Change the status of the JIRA to "Patch Available" in the 
kafka-review-tool
https://issues.apache.org/jira/browse/KAFKA-1536
KAFKA-1528  Normalize all the line endings
https://issues.apache.org/jira/browse/KAFKA-1528
KAFKA-1527  SimpleConsumer should be transaction-aware
https://issues.apache.org/jira/browse/KAFKA-1527
KAFKA-1526  Producer performance tool should have an option to enable 
transactions
https://issues.apache.org/jira/browse/KAFKA-1526
KAFKA-1525  DumpLogSegments should print transaction IDs
https://issues.apache.org/jira/browse/KAFKA-1525
KAFKA-1524  Implement transactional producer
https://issues.apache.org/jira/browse/KAFKA-1524
KAFKA-1523  Implement transaction manager module
https://issues.apache.org/jira/browse/KAFKA-1523
KAFKA-1522  Transactional messaging request/response definitions
https://issues.apache.org/jira/browse/KAFKA-1522
KAFKA-1517  Messages is a required argument to Producer Performance Test
https://issues.apache.org/jira/browse/KAFKA-1517
KAFKA-1510  Force offset commits when migrating consumer offsets from zookeeper 
to kafka
https://issues.apache.org/jira/browse/KAFKA-1510
KAFKA-1509  Restart of destination broker after unreplicated partition move 
leaves partitions without leader
https://issues.apache.org/jira/browse/KAFKA-1509
KAFKA-1507  Using GetOffsetShell against non-existent topic creates the topic 
unintentionally
https://issues.apache.org/jira/browse/KAFKA-1507
KAFKA-1500  adding new consumer requests using the new protocol
https://issues.apache.org/jira/browse/KAFKA-1500
KAFKA-1498  new producer performance and bug improvements
https://issues.apache.org/jira/browse/KAFKA-1498
KAFKA-1496  Using batch message in sync producer only sends the first message 
if we use a Scala Stream as the argument 
https://issues.apache.org/jira/browse/KAFKA-1496
KAFKA-1481  Stop using dashes AND underscores as separators in MBean names
https://issues.apache.org/jira/browse/KAFKA-1481
KAFKA-1477  add authentication layer and initial JKS x509 implementation for 
brokers, producers and consumer for network communication
https://issues.apache.org/jira/browse/KAFKA-1477
KAFKA-1476  Get a list of consumer groups
https://issues.apache.org/jira/browse/KAFKA-1476
KAFKA-1475  Kafka consumer stops LeaderFinder/FetcherThreads, but application 
does not know
https://issues.apache.org/jira/browse/KAFKA-1475
KAFKA-1471  Add Producer Unit Tests for LZ4 and LZ4HC compression
https://issues.apache.org/jira/browse/KAFKA-1471
KAFKA-1468  Improve perf tests
https://issues.apache.org/jira/browse/KAFKA-1468
KAFKA-1460  NoReplicaOnlineException: No replica for partition
https://issues.apache.org/jira/browse/KAFKA-1460
KAFKA-1450  check invalid leader in a more robust way
https://issues.apache.org/jira/browse/KAFKA-1450
KAFKA-1430  Purgatory redesign
https://issues.apache.org/jira/browse/KAFKA-1430
KAFKA-1420  Replace AdminUtils.createOrUpdateTopicPartitionAssignmentPathInZK 
with TestUtils.createTopic in unit tests
https://issues.apache.org/jira/browse/KAFKA-1420
KAFKA-1419  cross build for scala 2.11
https://issues.apache.org/jira/browse/KAFKA-1419
KAFKA-1394  Ensure last segment isn't deleted on expiration when there are 
unflushed messages
https://issues.apache.org/jira/browse/KAFKA-1394
KAFKA-1374  LogCleaner (compaction) does not support compressed topics
https://issues.apache.org/jira/browse/KAFKA-1374
KAFKA-1372  Upgrade to Gradle 1.10
https://issues.apache.org/jira/browse/KAFKA-1372
KAFKA-1367  Broker topic metadata not kept in sync with ZooKeeper
https://issues.apache.org/jira/browse/KAFKA-1367
KAFKA-

[jira] [Updated] (KAFKA-1524) Implement transactional producer

[Raul Castro Fernandez (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Raul Castro Fernandez updated KAFKA-1524:
-

Attachment: KAFKA-1524.patch

> Implement transactional producer
> 
>
> Key: KAFKA-1524
> URL: https://issues.apache.org/jira/browse/KAFKA-1524
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joel Koshy
>Assignee: Raul Castro Fernandez
>  Labels: transactions
> Attachments: KAFKA-1524.patch, KAFKA-1524.patch
>
>
> Implement the basic transactional producer functionality as outlined in 
> https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka
> The scope of this jira is basic functionality (i.e., to be able to begin and 
> commit or abort a transaction) without the failure scenarios.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1524) Implement transactional producer

[Raul Castro Fernandez (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14085063#comment-14085063
 ] 

Raul Castro Fernandez commented on KAFKA-1524:
--

Created reviewboard https://reviews.apache.org/r/24245/diff/
 against branch origin/transactional_messaging

> Implement transactional producer
> 
>
> Key: KAFKA-1524
> URL: https://issues.apache.org/jira/browse/KAFKA-1524
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joel Koshy
>Assignee: Raul Castro Fernandez
>  Labels: transactions
> Attachments: KAFKA-1524.patch, KAFKA-1524.patch
>
>
> Implement the basic transactional producer functionality as outlined in 
> https://cwiki.apache.org/confluence/display/KAFKA/Transactional+Messaging+in+Kafka
> The scope of this jira is basic functionality (i.e., to be able to begin and 
> commit or abort a transaction) without the failure scenarios.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Review Request 24245: Patch for KAFKA-1524

[Raul Castro Fernandez]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/24245/
---

Review request for kafka.


Bugs: KAFKA-1524
https://issues.apache.org/jira/browse/KAFKA-1524


Repository: kafka


Description
---

KAFKA-1524; implement transactional producer


Diffs
-

  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
522881c972ca42ff4dfb6237a2db15b625334d7e 
  
clients/src/main/java/org/apache/kafka/clients/producer/AbortTransactionException.java
 PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/clients/producer/InvalidTransactionStatusException.java
 PRE-CREATION 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
00775abbcac850b0f2bb9a70b6fbc7cdf319bcf6 
  clients/src/main/java/org/apache/kafka/clients/producer/MockProducer.java 
c0f1d57e0feb894d9f246058cd0396461afe3225 
  clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 
36e8398416036cab84faad1f07159e5adefd8086 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
f9de4af426449cceca12a8de9a9f54a6241d28d8 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 1ed3c28b436d28381d9402896e32d16f2586c65e 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 dd0af8aee98abed5d4a0dc50989e37888bb353fe 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
6fb5b82dedb48d946d1ac1ec7a535bddfdc693fa 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionContext.java
 PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/TransactionControl.java
 PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/errors/TransactionCoordinatorNotAvailableException.java
 PRE-CREATION 
  
clients/src/main/java/org/apache/kafka/common/errors/TransactionFailedException.java
 PRE-CREATION 
  clients/src/main/java/org/apache/kafka/common/record/Compressor.java 
0323f5f7032dceb49d820c17a41b78c56591ffc4 
  clients/src/main/java/org/apache/kafka/common/record/MemoryRecords.java 
759f577eaf0e7d28a84926d4aa30f4ef0cb27bc2 
  clients/src/main/java/org/apache/kafka/common/record/Record.java 
10df9fd8d3f4ec8c277650fa7eab269f3ea30d85 
  
clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java
 93b58d02eac0f8ca28440e3e0ebea28ed3a7673c 
  clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java 
5489acac6806b3ae5e6d568d401d5a20c86cac05 
  
clients/src/test/java/org/apache/kafka/clients/producer/TransactionContextTest.java
 PRE-CREATION 
  clients/src/test/java/org/apache/kafka/common/record/MemoryRecordsTest.java 
94a11121e207d5cf94dbc94443a8aa7edf387782 

Diff: https://reviews.apache.org/r/24245/diff/


Testing
---


Thanks,

Raul Castro Fernandez

[jira] [Created] (KAFKA-1569) Creat tool to test end-to-end correctness in transactional mode

[Raul Castro Fernandez (JIRA)]

Raul Castro Fernandez created KAFKA-1569:


 Summary: Creat tool to test end-to-end correctness in 
transactional mode
 Key: KAFKA-1569
 URL: https://issues.apache.org/jira/browse/KAFKA-1569
 Project: Kafka
  Issue Type: New Feature
Reporter: Raul Castro Fernandez
Assignee: Raul Castro Fernandez


A producer tool that creates an input file, reads it and sends it to the 
brokers according to some transaction configuration. And a consumer tool that 
read data from brokers with transaction boundaries and writes it to a file.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Newer Zookeeper?

[Gwen Shapira]

Also, specific Zookeeper 3.4.X version where loss of quorum occurred will help.
3.4.5 fixed some pretty serious issues around hanging.

Gwen

On Mon, Aug 4, 2014 at 9:29 AM, Gwen Shapira  wrote:
> Thanks for the heads-up, Joe.
>
> We've been shipping Zookeeper 3.4.X for over  two years now (since
> CDH4.0) and have many production customers. I'll check if there are
> any known issues with breaking quorum. In any case I will take your
> comments into account and see if I can arrange for extra testing.
>
> Can you share more information about the 3.4.X issues you were seeing?
> Was there especially large clusters involved? large number of
> consumers?
>
> Also, I'm curious to hear more about the reasons for separate ZK
> cluster. I can see why you'll want it if you have thousands of
> consumers, but are there other reasons? Multiple zookeeper installs
> can be a pain to manage.
>
> Gwen
>
>
>
> On Mon, Aug 4, 2014 at 7:52 AM, Joe Stein  wrote:
>> I have heard issues from installations running 3.4.X that I have not heard
>> from installations running 3.3.X (i.e. zk breaking quorum and cluster going
>> down).
>>
>> In none of these cases did I have an opportunity to isolate and reproduce
>> and confirm the issue happening and caused by 3.4.X. Moving to 3.3.x was
>> agreed to being a lower risk/cost solution to the problem. Once on 3.3.X
>> the issues didn't happen again.
>>
>> So I can't say for sure if there are issues with running 3.4.X but I would
>> suggest some due diligence in testing and production operation to validate
>> that every case that Kafka requires operates correctly (and over some
>> time).  There is a cost to this so some company(s) will have to take that
>> investment and do some cost vs the benefit of moving to 3.4.x.
>>
>> I currently recommend running a separate ZK cluster for Kafka production
>> and not chroot into an existing one except for test/qa/dev.
>>
>> I don't know what others experience is with 3.4.X as I said the issues I
>> have seen could have been coincidence.
>>
>> /***
>>  Joe Stein
>>  Founder, Principal Consultant
>>  Big Data Open Source Security LLC
>>  http://www.stealth.ly
>>  Twitter: @allthingshadoop 
>> /
>>
>>
>> On Mon, Aug 4, 2014 at 12:56 AM, Gwen Shapira  wrote:
>>
>>> Hi,
>>>
>>> Kafka currently builds against Zookeeper 3.3.4, which is quite old.
>>>
>>> Perhaps we should move to the more recent 3.4.x branch?
>>>
>>> I tested the change on my system and the only impact is to
>>> EmbeddedZookeeper used in tests (it uses NIOServerCnxn.factory, which
>>> was refactored into its own class in 3.4).
>>>
>>> Here's what the change looks like:
>>> https://gist.github.com/gwenshap/d95b36e0bced53cab5bb
>>>
>>> Gwen
>>>

Re: Newer Zookeeper?

[Gwen Shapira]

Thanks for the heads-up, Joe.

We've been shipping Zookeeper 3.4.X for over  two years now (since
CDH4.0) and have many production customers. I'll check if there are
any known issues with breaking quorum. In any case I will take your
comments into account and see if I can arrange for extra testing.

Can you share more information about the 3.4.X issues you were seeing?
Was there especially large clusters involved? large number of
consumers?

Also, I'm curious to hear more about the reasons for separate ZK
cluster. I can see why you'll want it if you have thousands of
consumers, but are there other reasons? Multiple zookeeper installs
can be a pain to manage.

Gwen



On Mon, Aug 4, 2014 at 7:52 AM, Joe Stein  wrote:
> I have heard issues from installations running 3.4.X that I have not heard
> from installations running 3.3.X (i.e. zk breaking quorum and cluster going
> down).
>
> In none of these cases did I have an opportunity to isolate and reproduce
> and confirm the issue happening and caused by 3.4.X. Moving to 3.3.x was
> agreed to being a lower risk/cost solution to the problem. Once on 3.3.X
> the issues didn't happen again.
>
> So I can't say for sure if there are issues with running 3.4.X but I would
> suggest some due diligence in testing and production operation to validate
> that every case that Kafka requires operates correctly (and over some
> time).  There is a cost to this so some company(s) will have to take that
> investment and do some cost vs the benefit of moving to 3.4.x.
>
> I currently recommend running a separate ZK cluster for Kafka production
> and not chroot into an existing one except for test/qa/dev.
>
> I don't know what others experience is with 3.4.X as I said the issues I
> have seen could have been coincidence.
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
>
>
> On Mon, Aug 4, 2014 at 12:56 AM, Gwen Shapira  wrote:
>
>> Hi,
>>
>> Kafka currently builds against Zookeeper 3.3.4, which is quite old.
>>
>> Perhaps we should move to the more recent 3.4.x branch?
>>
>> I tested the change on my system and the only impact is to
>> EmbeddedZookeeper used in tests (it uses NIOServerCnxn.factory, which
>> was refactored into its own class in 3.4).
>>
>> Here's what the change looks like:
>> https://gist.github.com/gwenshap/d95b36e0bced53cab5bb
>>
>> Gwen
>>

Build failed in Jenkins: Kafka-trunk #238

[Apache Jenkins Server]

See 

Changes:

[junrao] kafka-1562; kafka-topics.sh alter add partitions resets 
cleanup.policy; patched by Jonathan Natkins; reviewed by Jun Rao

--
[...truncated 671 lines...]
there were 12 feature warning(s); re-run with -feature for details
7 warnings found
:core:processResources UP-TO-DATE
:core:classes
:core:compileTestJava UP-TO-DATE
:core:compileTestScala
:core:processTestResources UP-TO-DATE
:core:testClasses
:core:test

unit.kafka.utils.CommandLineUtilsTest > testParseEmptyArg PASSED

unit.kafka.utils.CommandLineUtilsTest > testParseSingleArg PASSED

unit.kafka.utils.CommandLineUtilsTest > testParseArgs PASSED

unit.kafka.common.TopicTest > testInvalidTopicNames PASSED

unit.kafka.common.ConfigTest > testInvalidClientIds PASSED

unit.kafka.common.ConfigTest > testInvalidGroupIds PASSED

kafka.server.LeaderElectionTest > testLeaderElectionAndEpoch PASSED

kafka.server.LeaderElectionTest > testLeaderElectionWithStaleControllerEpoch 
PASSED

kafka.server.LogRecoveryTest > testHWCheckpointNoFailuresSingleLogSegment PASSED

kafka.server.LogRecoveryTest > testHWCheckpointWithFailuresSingleLogSegment 
PASSED

kafka.server.LogRecoveryTest > testHWCheckpointNoFailuresMultipleLogSegments 
PASSED

kafka.server.LogRecoveryTest > testHWCheckpointWithFailuresMultipleLogSegments 
PASSED

kafka.server.IsrExpirationTest > testIsrExpirationForStuckFollowers PASSED

kafka.server.IsrExpirationTest > testIsrExpirationForSlowFollowers PASSED

kafka.server.HighwatermarkPersistenceTest > 
testHighWatermarkPersistenceSinglePartition PASSED

kafka.server.HighwatermarkPersistenceTest > 
testHighWatermarkPersistenceMultiplePartitions PASSED

kafka.server.DynamicConfigChangeTest > testConfigChange PASSED

kafka.server.DynamicConfigChangeTest > testConfigChangeOnNonExistingTopic PASSED

kafka.server.ServerShutdownTest > testCleanShutdown PASSED

kafka.server.ServerShutdownTest > testCleanShutdownWithDeleteTopicEnabled PASSED

kafka.server.AdvertiseBrokerTest > testBrokerAdvertiseToZK PASSED

kafka.server.SimpleFetchTest > testNonReplicaSeesHwWhenFetching PASSED

kafka.server.SimpleFetchTest > testReplicaSeesLeoWhenFetching PASSED

kafka.server.OffsetCommitTest > testUpdateOffsets PASSED

kafka.server.OffsetCommitTest > testCommitAndFetchOffsets PASSED

kafka.server.OffsetCommitTest > testLargeMetadataPayload PASSED

kafka.server.ReplicaManagerTest > testHighWaterMarkDirectoryMapping PASSED

kafka.server.ReplicaManagerTest > testHighwaterMarkRelativeDirectoryMapping 
PASSED

kafka.server.LogOffsetTest > testGetOffsetsForUnknownTopic PASSED

kafka.server.LogOffsetTest > testGetOffsetsBeforeLatestTime PASSED

kafka.server.LogOffsetTest > testEmptyLogsGetOffsets PASSED

kafka.server.LogOffsetTest > testGetOffsetsBeforeNow PASSED

kafka.server.LogOffsetTest > testGetOffsetsBeforeEarliestTime PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeHoursProvided PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeMinutesProvided PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeMsProvided PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeNoConfigProvided PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndHoursProvided 
PASSED

kafka.server.KafkaConfigTest > testLogRetentionTimeBothMinutesAndMsProvided 
PASSED

kafka.server.KafkaConfigTest > testAdvertiseDefaults PASSED

kafka.server.KafkaConfigTest > testAdvertiseConfigured PASSED

kafka.server.KafkaConfigTest > testUncleanLeaderElectionDefault PASSED

kafka.server.KafkaConfigTest > testUncleanElectionDisabled PASSED

kafka.server.KafkaConfigTest > testUncleanElectionEnabled PASSED

kafka.server.KafkaConfigTest > testUncleanElectionInvalid PASSED

kafka.server.KafkaConfigTest > testLogRollTimeMsProvided PASSED

kafka.server.KafkaConfigTest > testLogRollTimeBothMsAndHoursProvided PASSED

kafka.server.KafkaConfigTest > testLogRollTimeNoConfigProvided PASSED

kafka.server.ReplicaFetchTest > testReplicaFetcherThread PASSED

kafka.server.RequestPurgatoryTest > testRequestSatisfaction PASSED

kafka.server.RequestPurgatoryTest > testRequestExpiry PASSED

kafka.utils.ReplicationUtilsTest > testUpdateLeaderAndIsr PASSED

kafka.utils.IteratorTemplateTest > testIterator PASSED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest > testNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.utils.UtilsTest > testSwallow PASSED

kafka.utils.UtilsTest > testCircularIterator PASSED

kafka.utils.UtilsTest > testReadBytes PASSED

kafka.utils.UtilsTest > testAbs PASSED

kafka.utils.UtilsTest > testReplaceSuffix PASSED

kafka.utils.UtilsTest > testReadInt PASSED

kafka.utils.UtilsTest > testCsvList PASSED

kafka.utils.UtilsTest > testInLock PASSED

kafka.ut

[jira] [Commented] (KAFKA-1566) Kafka environment configuration (kafka-env.sh)

[Cosmin Lehene (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084788#comment-14084788
 ] 

Cosmin Lehene commented on KAFKA-1566:
--

Somehow related: KAFKA_LOG_DIR would be better than setting KAFKA_LOG4J_OPTS. 
kafka-run-class.sh would check for this
{code}
 LOG_DIR=${KAFKA_LOG_DIR:-"$base_dir/logs"}
{code}
Perhaps a new issue may be useful (otherwise I could just add them to this 
patch).

> Kafka environment configuration (kafka-env.sh)
> --
>
> Key: KAFKA-1566
> URL: https://issues.apache.org/jira/browse/KAFKA-1566
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Reporter: Cosmin Lehene
>Assignee: Cosmin Lehene
>  Labels: newbie
> Fix For: 0.8.2, 0.9.0
>
>
> It would be useful (especially for automated deployments) to have an 
> environment configuration file that could be sourced from the launcher files 
> (e.g. kafka-run-server.sh). 
> This is how this could look like kafka-env.sh 
> {code}
> export KAFKA_JVM_PERFORMANCE_OPTS="-XX:+UseCompressedOops 
> -XX:+DisableExplicitGC -Djava.awt.headless=true \ -XX:+UseG1GC 
> -XX:PermSize=48m -XX:MaxPermSize=48m -XX:MaxGCPauseMillis=20 
> -XX:InitiatingHeapOccupancyPercent=35' %>" 
> export KAFKA_HEAP_OPTS="'-Xmx1G -Xms1G' %>" 
> export KAFKA_LOG4J_OPTS="-Dkafka.logs.dir=/var/log/kafka" 
> {code} 
> kafka-server-start.sh 
> {code} 
> ... 
> source $base_dir/config/kafka-env.sh 
> ... 
> {code} 
> This approach is consistent with Hadoop and HBase. However the idea here is 
> to be able to set these values in a single place without having to edit 
> startup scripts.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Comment Edited] (KAFKA-1499) Broker-side compression configuration

[Manikumar Reddy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084777#comment-14084777
 ] 

Manikumar Reddy edited comment on KAFKA-1499 at 8/4/14 3:43 PM:


I would like to propose new server property "*log.compression.type*" and topic 
override property "*compression.type*"
If set, this property is used for compression type at server-side. All 
non-compressed messages on the broker will be compressed to this compression 
type.

log.compression.type=none|gzip|snappy (default = none)
compression.type=none|gzip|snappy


was (Author: omkreddy):
I would like to propose new server property "*log.compression.type*" and topic 
override property "*compression.type*"
If set, this property is used for compression type at server-side. All 
non-compressed messages on the broker will be compressed to this compression 
type.

log.compression.type=none|gzip|snappy (default = none)
compression.type=compression.type

> Broker-side compression configuration
> -
>
> Key: KAFKA-1499
> URL: https://issues.apache.org/jira/browse/KAFKA-1499
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joel Koshy
>Assignee: Manikumar Reddy
>  Labels: newbie++
> Fix For: 0.8.2
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> A given topic can have messages in mixed compression codecs. i.e., it can
> also have a mix of uncompressed/compressed messages.
> It will be useful to support a broker-side configuration to recompress
> messages to a specific compression codec. i.e., all messages (for all
> topics) on the broker will be compressed to this codec. We could have
> per-topic overrides as well.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Assigned] (KAFKA-1499) Broker-side compression configuration

[Manikumar Reddy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar Reddy reassigned KAFKA-1499:
--

Assignee: Manikumar Reddy

> Broker-side compression configuration
> -
>
> Key: KAFKA-1499
> URL: https://issues.apache.org/jira/browse/KAFKA-1499
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joel Koshy
>Assignee: Manikumar Reddy
>  Labels: newbie++
> Fix For: 0.8.2
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> A given topic can have messages in mixed compression codecs. i.e., it can
> also have a mix of uncompressed/compressed messages.
> It will be useful to support a broker-side configuration to recompress
> messages to a specific compression codec. i.e., all messages (for all
> topics) on the broker will be compressed to this codec. We could have
> per-topic overrides as well.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1566) Kafka environment configuration (kafka-env.sh)

[Cosmin Lehene (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084778#comment-14084778
 ] 

Cosmin Lehene commented on KAFKA-1566:
--

The convention is to have it in conf/
We can have an env variable for it's location. 

We generate these files at deploy time (they are Puppet .erb templates in our 
case) based on some versioned configuration. 

> Kafka environment configuration (kafka-env.sh)
> --
>
> Key: KAFKA-1566
> URL: https://issues.apache.org/jira/browse/KAFKA-1566
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Reporter: Cosmin Lehene
>Assignee: Cosmin Lehene
>  Labels: newbie
> Fix For: 0.8.2, 0.9.0
>
>
> It would be useful (especially for automated deployments) to have an 
> environment configuration file that could be sourced from the launcher files 
> (e.g. kafka-run-server.sh). 
> This is how this could look like kafka-env.sh 
> {code}
> export KAFKA_JVM_PERFORMANCE_OPTS="-XX:+UseCompressedOops 
> -XX:+DisableExplicitGC -Djava.awt.headless=true \ -XX:+UseG1GC 
> -XX:PermSize=48m -XX:MaxPermSize=48m -XX:MaxGCPauseMillis=20 
> -XX:InitiatingHeapOccupancyPercent=35' %>" 
> export KAFKA_HEAP_OPTS="'-Xmx1G -Xms1G' %>" 
> export KAFKA_LOG4J_OPTS="-Dkafka.logs.dir=/var/log/kafka" 
> {code} 
> kafka-server-start.sh 
> {code} 
> ... 
> source $base_dir/config/kafka-env.sh 
> ... 
> {code} 
> This approach is consistent with Hadoop and HBase. However the idea here is 
> to be able to set these values in a single place without having to edit 
> startup scripts.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1499) Broker-side compression configuration

[Manikumar Reddy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084777#comment-14084777
 ] 

Manikumar Reddy commented on KAFKA-1499:


I would like to propose new server property "*log.compression.type*" and topic 
override property "*compression.type*"
If set, this property is used for compression type at server-side. All 
non-compressed messages on the broker will be compressed to this compression 
type.

log.compression.type=none|gzip|snappy (default = none)
compression.type=compression.type

> Broker-side compression configuration
> -
>
> Key: KAFKA-1499
> URL: https://issues.apache.org/jira/browse/KAFKA-1499
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Joel Koshy
>Assignee: Manikumar Reddy
>  Labels: newbie++
> Fix For: 0.8.2
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> A given topic can have messages in mixed compression codecs. i.e., it can
> also have a mix of uncompressed/compressed messages.
> It will be useful to support a broker-side configuration to recompress
> messages to a specific compression codec. i.e., all messages (for all
> topics) on the broker will be compressed to this codec. We could have
> per-topic overrides as well.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Newer Zookeeper?

[Joe Stein]

I have heard issues from installations running 3.4.X that I have not heard
from installations running 3.3.X (i.e. zk breaking quorum and cluster going
down).

In none of these cases did I have an opportunity to isolate and reproduce
and confirm the issue happening and caused by 3.4.X. Moving to 3.3.x was
agreed to being a lower risk/cost solution to the problem. Once on 3.3.X
the issues didn't happen again.

So I can't say for sure if there are issues with running 3.4.X but I would
suggest some due diligence in testing and production operation to validate
that every case that Kafka requires operates correctly (and over some
time).  There is a cost to this so some company(s) will have to take that
investment and do some cost vs the benefit of moving to 3.4.x.

I currently recommend running a separate ZK cluster for Kafka production
and not chroot into an existing one except for test/qa/dev.

I don't know what others experience is with 3.4.X as I said the issues I
have seen could have been coincidence.

/***
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop 
/


On Mon, Aug 4, 2014 at 12:56 AM, Gwen Shapira  wrote:

> Hi,
>
> Kafka currently builds against Zookeeper 3.3.4, which is quite old.
>
> Perhaps we should move to the more recent 3.4.x branch?
>
> I tested the change on my system and the only impact is to
> EmbeddedZookeeper used in tests (it uses NIOServerCnxn.factory, which
> was refactored into its own class in 3.4).
>
> Here's what the change looks like:
> https://gist.github.com/gwenshap/d95b36e0bced53cab5bb
>
> Gwen
>

[jira] [Commented] (KAFKA-1419) cross build for scala 2.11

[Ivan Lyutov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084715#comment-14084715
 ] 

Ivan Lyutov commented on KAFKA-1419:


Updated reviewboard https://reviews.apache.org/r/23895/diff/
 against branch apache/trunk

> cross build for scala 2.11
> --
>
> Key: KAFKA-1419
> URL: https://issues.apache.org/jira/browse/KAFKA-1419
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.8.1
>Reporter: Scott Clasen
>Assignee: Ivan Lyutov
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: KAFKA-1419.patch, KAFKA-1419.patch, 
> KAFKA-1419_2014-07-28_15:05:16.patch, KAFKA-1419_2014-07-29_15:13:43.patch, 
> KAFKA-1419_2014-08-04_14:43:26.patch
>
>
> Please publish builds for scala 2.11, hopefully just needs a small tweak to 
> the gradle conf?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Updated] (KAFKA-1419) cross build for scala 2.11

[Ivan Lyutov (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ivan Lyutov updated KAFKA-1419:
---

Attachment: KAFKA-1419_2014-08-04_14:43:26.patch

> cross build for scala 2.11
> --
>
> Key: KAFKA-1419
> URL: https://issues.apache.org/jira/browse/KAFKA-1419
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.8.1
>Reporter: Scott Clasen
>Assignee: Ivan Lyutov
>Priority: Blocker
> Fix For: 0.8.2
>
> Attachments: KAFKA-1419.patch, KAFKA-1419.patch, 
> KAFKA-1419_2014-07-28_15:05:16.patch, KAFKA-1419_2014-07-29_15:13:43.patch, 
> KAFKA-1419_2014-08-04_14:43:26.patch
>
>
> Please publish builds for scala 2.11, hopefully just needs a small tweak to 
> the gradle conf?



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Re: Review Request 23895: Patch for KAFKA-1419

[Ivan Lyutov]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23895/
---

(Updated Aug. 4, 2014, 2:43 p.m.)


Review request for kafka.


Bugs: KAFKA-1419
https://issues.apache.org/jira/browse/KAFKA-1419


Repository: kafka


Description (updated)
---

KAFKA-1419 - cross build for scala 2.11 - dropped scala 2.8 support - minor bug 
fixes


KAFKA-1419 - cross build for scala 2.11 - changed 2.11 specific dependency 
version - updated scala version to 2.11.2 - added getBuffer to 
ByteBufferMessageSet classes


KAFKA-1419 - cross build for scala 2.11 - changed 2.11 specific dependency 
version - updated scala version to 2.11.2 - added getBuffer to 
ByteBufferMessageSet classes - removed annotations 2.8 file


Diffs (updated)
-

  build.gradle a72905df824ba68bed5d5170d18873c23e1782c9 
  core/src/main/scala/kafka/javaapi/message/ByteBufferMessageSet.scala 
fecee8d5f7b32f483bb1bfc6a5080d589906f9c4 
  core/src/main/scala/kafka/message/ByteBufferMessageSet.scala 
73401c5ff34d08abce22267aa9c4d86632c6fb74 
  core/src/main/scala/kafka/utils/Annotations_2.8.scala 
28269eb037109f7680b9da732e4baa51c9a594b6 
  core/src/main/scala/kafka/utils/Annotations_2.9+.scala  
  gradle.properties 4827769a3f8e34f0fe7e783eb58e44d4db04859b 
  gradle/buildscript.gradle 225e0a82708bc5f390e5e2c1d4d9a0d06f491b95 
  gradle/wrapper/gradle-wrapper.properties 
610282a699afc89a82203ef0e4e71ecc53761039 
  scala.gradle ebd21b870c0746aade63248344ab65d9b5baf820 

Diff: https://reviews.apache.org/r/23895/diff/


Testing
---


Thanks,

Ivan Lyutov

Re: Review Request 23895: Patch for KAFKA-1419

[Jun Rao]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/23895/#review49454
---



build.gradle


Can we remove Annotations_2.8.scala and rename Annotations_2.9+.scala to 
just Annotations.scala? We can then potentially remove sourceSets setting 
completely.


- Jun Rao


On July 29, 2014, 3:13 p.m., Ivan Lyutov wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/23895/
> ---
> 
> (Updated July 29, 2014, 3:13 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1419
> https://issues.apache.org/jira/browse/KAFKA-1419
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-1419 - cross build for scala 2.11 - dropped scala 2.8 support - minor 
> bug fixes
> 
> 
> KAFKA-1419 - cross build for scala 2.11 - changed 2.11 specific dependency 
> version - updated scala version to 2.11.2 - added getBuffer to 
> ByteBufferMessageSet classes
> 
> 
> Diffs
> -
> 
>   build.gradle a72905df824ba68bed5d5170d18873c23e1782c9 
>   core/src/main/scala/kafka/javaapi/message/ByteBufferMessageSet.scala 
> fecee8d5f7b32f483bb1bfc6a5080d589906f9c4 
>   core/src/main/scala/kafka/message/ByteBufferMessageSet.scala 
> 73401c5ff34d08abce22267aa9c4d86632c6fb74 
>   gradle.properties 4827769a3f8e34f0fe7e783eb58e44d4db04859b 
>   gradle/buildscript.gradle 225e0a82708bc5f390e5e2c1d4d9a0d06f491b95 
>   gradle/wrapper/gradle-wrapper.properties 
> 610282a699afc89a82203ef0e4e71ecc53761039 
>   scala.gradle ebd21b870c0746aade63248344ab65d9b5baf820 
> 
> Diff: https://reviews.apache.org/r/23895/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Ivan Lyutov
> 
>

[jira] [Updated] (KAFKA-1562) kafka-topics.sh alter add partitions resets cleanup.policy

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-1562:
---

   Resolution: Fixed
Fix Version/s: 0.8.2
 Assignee: Jonathan Natkins
   Status: Resolved  (was: Patch Available)

Thanks for the patch. +1 and committed to trunk.

> kafka-topics.sh alter add partitions resets cleanup.policy
> --
>
> Key: KAFKA-1562
> URL: https://issues.apache.org/jira/browse/KAFKA-1562
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Kenny
>Assignee: Jonathan Natkins
> Fix For: 0.8.2
>
> Attachments: KAFKA-1562.patch, KAFKA-1562_2014-07-30_13:18:21.patch, 
> KAFKA-1562_2014-07-30_13:51:25.patch, KAFKA-1562_2014-08-02_11:10:34.patch
>
>
> When partitions are added to an already existing topic the 
> cleanup.policy=compact is not retained.
> {code}
> ./kafka-topics.sh --zookeeper localhost --create --partitions 1 
> --replication-factor 1 --topic KTEST --config cleanup.policy=compact
> ./kafka-topics.sh --zookeeper localhost --describe --topic KTEST
> Topic:KTEST   PartitionCount:1ReplicationFactor:1 
> Configs:cleanup.policy=compact
>   Topic: KTESTPartition: 0Leader: 0   Replicas: 0 Isr: 0
> ./kafka-topics.sh --zookeeper localhost --alter --partitions 3 --topic KTEST 
> --config cleanup.policy=compact
>  ./kafka-topics.sh --zookeeper localhost --describe --topic KTEST
> Topic:KTEST   PartitionCount:3ReplicationFactor:1 Configs:
>   Topic: KTESTPartition: 0Leader: 0   Replicas: 0 Isr: 0
>   Topic: KTESTPartition: 1Leader: 0   Replicas: 0 Isr: 0
>   Topic: KTESTPartition: 2Leader: 0   Replicas: 0 Isr: 0
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

[jira] [Commented] (KAFKA-1019) kafka-preferred-replica-election.sh will fail without clear error message if /brokers/topics/[topic]/partitions does not exist

[Mickael Hemri (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1019?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14084641#comment-14084641
 ] 

Mickael Hemri commented on KAFKA-1019:
--

I tried with zookeeper 3.3.6 and we have the same issue.
To reproduce:

Create a topic named testid
{code}bin/kafka-topics.sh --topic testid --replication-factor 3 --partition 3 
--zookeeper 127.0.0.1:2181/kafka --create
Created topic "testid".{code}

{code}./bin/kafka-topics.sh --topic testid --zookeeper 127.0.0.1:2181/kafka 
--describe
Topic:testidPartitionCount:3ReplicationFactor:3 Configs:
Topic: testid   Partition: 0Leader: 31985   Replicas: 
31985,9920,4580   Isr: 31985,9920,4580
Topic: testid   Partition: 1Leader: 4580Replicas: 
4580,31985,9920   Isr: 4580,31985,9920
Topic: testid   Partition: 2Leader: 9920Replicas: 
9920,4580,31985   Isr: 9920,4580,31985
{code}
Ok great, we have leaders and  /brokers/topics/testid/partitions in zookeeper

Delete testid topic
{code}bin/kafka-run-class.sh kafka.admin.DeleteTopicCommand --topic testid 
--zookeeper 127.0.0.1:2181/kafka
deletion succeeded!
{code}

Create again a topic named testid
{code}bin/kafka-topics.sh --topic testid --replication-factor 3 --partition 3 
--zookeeper 127.0.0.1:2181/kafka --create
Created topic "testid".{code}

Now check:
{code}./bin/kafka-topics.sh --topic testid --zookeeper 127.0.0.1:2181/kafka 
--describe
Topic:testidPartitionCount:3ReplicationFactor:3 Configs:
Topic: testid   Partition: 0Leader: noneReplicas: 
31985,4580,9920   Isr: 
Topic: testid   Partition: 1Leader: noneReplicas: 
4580,9920,31985   Isr: 
Topic: testid   Partition: 2Leader: noneReplicas: 
9920,31985,4580   Isr:{code}

As you can see we have no leader when we create the topic after a deletion. And 
there is no /brokers/topics/testid/partitions in zookeeper
It works again with a different topic name, so it seems that something is not 
properly deleted with DeleteTopicCommand command.

We reproduced it on 3 differents zookeeper chroot: 127.0.0.1:2181/kafka, 
127.0.0.1:2181/kafka2 and 127.0.0.1:2181/kafka3

Thanks

> kafka-preferred-replica-election.sh will fail without clear error message if 
> /brokers/topics/[topic]/partitions does not exist
> --
>
> Key: KAFKA-1019
> URL: https://issues.apache.org/jira/browse/KAFKA-1019
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1
>Reporter: Guozhang Wang
>  Labels: newbie
> Fix For: 0.8.2
>
>
> From Libo Yu:
> I tried to run kafka-preferred-replica-election.sh on our kafka cluster.
> But I got this expection:
> Failed to start preferred replica election
> org.I0Itec.zkclient.exception.ZkNoNodeException: 
> org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = 
> NoNode for /brokers/topics/uattoqaaa.default/partitions
> I checked zookeeper and there is no 
> /brokers/topics/uattoqaaa.default/partitions. All I found is
> /brokers/topics/uattoqaaa.default.



--
This message was sent by Atlassian JIRA
(v6.2#6252)