date:20141112


[ 
https://issues.apache.org/jira/browse/KAFKA-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14207957#comment-14207957
 ] 

Dmytro Kostiuchenko commented on KAFKA-1667:


Updated reviewboard https://reviews.apache.org/r/27634/diff/
 against branch origin/trunk

>  topic-level configuration not validated
> 
>
> Key: KAFKA-1667
> URL: https://issues.apache.org/jira/browse/KAFKA-1667
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Ryan Berdeen
>  Labels: newbie
> Attachments: KAFKA-1667.patch, KAFKA-1667_2014-11-05_19:43:53.patch, 
> KAFKA-1667_2014-11-06_17:10:14.patch, KAFKA-1667_2014-11-07_14:28:14.patch, 
> KAFKA-1667_2014-11-12_12:49:11.patch
>
>
> I was able to set the configuration for a topic to these invalid values:
> {code}
> Topic:topic-config-test  PartitionCount:1ReplicationFactor:2 
> Configs:min.cleanable.dirty.ratio=-30.2,segment.bytes=-1,retention.ms=-12,cleanup.policy=lol
> {code}
> It seems that the values are saved as long as they are the correct type, but 
> are not validated like the corresponding broker-level properties.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1667) topic-level configuration not validated


 [ 
https://issues.apache.org/jira/browse/KAFKA-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmytro Kostiuchenko updated KAFKA-1667:
---
Attachment: KAFKA-1667_2014-11-12_12:49:11.patch

>  topic-level configuration not validated
> 
>
> Key: KAFKA-1667
> URL: https://issues.apache.org/jira/browse/KAFKA-1667
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Ryan Berdeen
>  Labels: newbie
> Attachments: KAFKA-1667.patch, KAFKA-1667_2014-11-05_19:43:53.patch, 
> KAFKA-1667_2014-11-06_17:10:14.patch, KAFKA-1667_2014-11-07_14:28:14.patch, 
> KAFKA-1667_2014-11-12_12:49:11.patch
>
>
> I was able to set the configuration for a topic to these invalid values:
> {code}
> Topic:topic-config-test  PartitionCount:1ReplicationFactor:2 
> Configs:min.cleanable.dirty.ratio=-30.2,segment.bytes=-1,retention.ms=-12,cleanup.policy=lol
> {code}
> It seems that the values are saved as long as they are the correct type, but 
> are not validated like the corresponding broker-level properties.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 27634: Patch for KAFKA-1667

2014-11-12 Thread Dmytro Kostiuchenko


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27634/
---

(Updated Nov. 12, 2014, 11:49 a.m.)


Review request for kafka.


Bugs: KAFKA-1667
https://issues.apache.org/jira/browse/KAFKA-1667


Repository: kafka


Description (updated)
---

KAFKA-1667 Fixed bugs in LogConfig. Added test and documentation


KAFKA-1667 Updated tests to reflect new boolean property parsing logic


KAFKA-1667 renamed methods to match naming convention


KAFKA-1667 Added unit test to cover invalid configuration case


KAFKA-1667 Strict UncleanLeaderElection property parsing


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/common/config/ConfigDef.java 
c4cea2cc072f4db4ce014b63d226431d3766bef1 
  core/src/main/scala/kafka/admin/TopicCommand.scala 
0b2735e7fc42ef9894bef1997b1f06a8ebee5439 
  core/src/main/scala/kafka/log/LogConfig.scala 
e48922a97727dd0b98f3ae630ebb0af3bef2373d 
  core/src/main/scala/kafka/utils/Utils.scala 
23aefb4715b177feae1d2f83e8b910653ea10c5f 
  core/src/test/scala/kafka/log/LogConfigTest.scala PRE-CREATION 
  core/src/test/scala/unit/kafka/integration/UncleanLeaderElectionTest.scala 
f44568cb25edf25db857415119018fd4c9922f61 

Diff: https://reviews.apache.org/r/27634/diff/


Testing
---


Thanks,

Dmytro Kostiuchenko

[jira] [Commented] (KAFKA-1667) topic-level configuration not validated


[ 
https://issues.apache.org/jira/browse/KAFKA-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14207968#comment-14207968
 ] 

Dmytro Kostiuchenko commented on KAFKA-1667:


Can't assign issue to myself. Get an exception when running 
kafka-patch-review.py.

{code}jira.exceptions.JIRAError: HTTP 400: "Field 'assignee' cannot be set. It 
is not on the appropriate screen, or unknown{code}

 Also don't have "Assign to me" button in JIRA.

>  topic-level configuration not validated
> 
>
> Key: KAFKA-1667
> URL: https://issues.apache.org/jira/browse/KAFKA-1667
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Ryan Berdeen
>  Labels: newbie
> Attachments: KAFKA-1667.patch, KAFKA-1667_2014-11-05_19:43:53.patch, 
> KAFKA-1667_2014-11-06_17:10:14.patch, KAFKA-1667_2014-11-07_14:28:14.patch, 
> KAFKA-1667_2014-11-12_12:49:11.patch
>
>
> I was able to set the configuration for a topic to these invalid values:
> {code}
> Topic:topic-config-test  PartitionCount:1ReplicationFactor:2 
> Configs:min.cleanable.dirty.ratio=-30.2,segment.bytes=-1,retention.ms=-12,cleanup.policy=lol
> {code}
> It seems that the values are saved as long as they are the correct type, but 
> are not validated like the corresponding broker-level properties.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1667) topic-level configuration not validated


 [ 
https://issues.apache.org/jira/browse/KAFKA-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmytro Kostiuchenko updated KAFKA-1667:
---
Status: Patch Available  (was: Open)

>  topic-level configuration not validated
> 
>
> Key: KAFKA-1667
> URL: https://issues.apache.org/jira/browse/KAFKA-1667
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Ryan Berdeen
>  Labels: newbie
> Attachments: KAFKA-1667_2014-11-05_19:43:53.patch, 
> KAFKA-1667_2014-11-06_17:10:14.patch, KAFKA-1667_2014-11-07_14:28:14.patch, 
> KAFKA-1667_2014-11-12_12:49:11.patch
>
>
> I was able to set the configuration for a topic to these invalid values:
> {code}
> Topic:topic-config-test  PartitionCount:1ReplicationFactor:2 
> Configs:min.cleanable.dirty.ratio=-30.2,segment.bytes=-1,retention.ms=-12,cleanup.policy=lol
> {code}
> It seems that the values are saved as long as they are the correct type, but 
> are not validated like the corresponding broker-level properties.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1667) topic-level configuration not validated


 [ 
https://issues.apache.org/jira/browse/KAFKA-1667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dmytro Kostiuchenko updated KAFKA-1667:
---
Attachment: (was: KAFKA-1667.patch)

>  topic-level configuration not validated
> 
>
> Key: KAFKA-1667
> URL: https://issues.apache.org/jira/browse/KAFKA-1667
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Ryan Berdeen
>  Labels: newbie
> Attachments: KAFKA-1667_2014-11-05_19:43:53.patch, 
> KAFKA-1667_2014-11-06_17:10:14.patch, KAFKA-1667_2014-11-07_14:28:14.patch, 
> KAFKA-1667_2014-11-12_12:49:11.patch
>
>
> I was able to set the configuration for a topic to these invalid values:
> {code}
> Topic:topic-config-test  PartitionCount:1ReplicationFactor:2 
> Configs:min.cleanable.dirty.ratio=-30.2,segment.bytes=-1,retention.ms=-12,cleanup.policy=lol
> {code}
> It seems that the values are saved as long as they are the correct type, but 
> are not validated like the corresponding broker-level properties.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 27684: Patch for KAFKA-1743

2014-11-12 Thread Manikumar Reddy O



> On Nov. 10, 2014, 7:50 p.m., Jun Rao wrote:
> > core/src/main/scala/kafka/consumer/ConsumerConnector.scala, lines 76-80
> > 
> >
> > We will also need to change the interface in ConsumerConnector from 
> > 
> >   def commitOffsets(retryOnFailure: Boolean = true)
> >   
> > back to 
> > 
> >   def commitOffsets
> >   
> > In ZookeeperConsumerconnector, we can make the following method private
> > 
> > def commitOffsets(retryOnFailure: Boolean = true)
> > 
> > Another question, will scala compiler be confused with 2 methods, one 
> > w/o parenthsis and one with 1 parameter having a default? Could you try 
> > compiling the code on all scala versions?

Currently below classes uses the new method  commitOffsets(true). 

kafka/javaapi/consumer/ZookeeperConsumerConnector.scala
kafka/tools/TestEndToEndLatency.scala

If we are changing the interface,  then we need to change the above classes 
also. 

If we are not fixing this on trunk, then same problem will come in 0.8.3. 
How to handle this? 

2 methods, one w/o parenthsis and one with 1 parameter is getting compiled on 
all scala versions.


- Manikumar Reddy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27684/#review60652
---


On Nov. 8, 2014, 6:20 a.m., Manikumar Reddy O wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27684/
> ---
> 
> (Updated Nov. 8, 2014, 6:20 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1743
> https://issues.apache.org/jira/browse/KAFKA-1743
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> def commitOffsets method added to make ConsumerConnector backward  compatible
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/consumer/ConsumerConnector.scala 
> 07677c1c26768ef9c9032626180d0015f12cb0e0 
> 
> Diff: https://reviews.apache.org/r/27684/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Manikumar Reddy O
> 
>

Re: Review Request 27890: Patch for KAFKA-1764

2014-11-12 Thread Guozhang Wang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27890/#review61001
---


Could you also remove the line in consumer that sends back the shutdown command?

- Guozhang Wang


On Nov. 11, 2014, 10:59 p.m., Jiangjie Qin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27890/
> ---
> 
> (Updated Nov. 11, 2014, 10:59 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1764
> https://issues.apache.org/jira/browse/KAFKA-1764
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> fix for KAFKA-1764
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 
> fbc680fde21b02f11285a4f4b442987356abd17b 
> 
> Diff: https://reviews.apache.org/r/27890/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jiangjie Qin
> 
>

Re: Kafka Command Line & Shell

2014-11-12 Thread Guozhang Wang

Thanks Joe. I will read the wiki page.

On Tue, Nov 11, 2014 at 11:47 PM, Joe Stein  wrote:

> I started writing this up on the wiki
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements
>
> Instead of starting a new thread I figure just continue this one I started.
> I also added another (important) component for centralized management of
> configuration as "global level" much like we have topic level.  These
> global configuration would be overridden (perhaps not all) from the
> server.properties on start (so like in case one broker needs a different
> port, sure).
>
> < controller instead of the current way (via ZK admin path) may expose
> concurrency on the admin requests, which may not be supported yet.
>
> Guozhang, take a look at the diagram how I am thinking of this it would be
> a new handle request that will execute the tools pretty much how they are
> today. My thinking is maybe to-do one at a time (so TopicCommand first I
> think) and have what the TopicCommand is doing happen on server and send
> the RQ/RP to the client but execute on the server. If there is something
> not supported we will of course have to deal with that and implement it for
> sure.  Once we get one working end to end I think adding the rest will be
> (more or less) concise iterations to get it done. I added your concern to
> the wiki under the "gotchas" section.
>
> /***
>  Joe Stein
>  Founder, Principal Consultant
>  Big Data Open Source Security LLC
>  http://www.stealth.ly
>  Twitter: @allthingshadoop 
> /
>
> On Mon, Oct 20, 2014 at 2:15 AM, Guozhang Wang  wrote:
>
> > One concern I have is that using RQ/RP wire protocol to the controller
> > instead of the current way (via ZK admin path) may expose concurrency on
> > the admin requests, which may not be supported yet.
> >
> > Some initial discussion about this is on KAFKA-1305.
> >
> > Guozhang
> >
> > On Sun, Oct 19, 2014 at 1:55 PM, Joe Stein  wrote:
> >
> > > Maybe we should add some AdminMessage RQ/RP wire protocol structure(s)
> > and
> > > let the controller handle it? We could then build the CLI and Shell in
> > the
> > > project both as useful tools and samples for others.
> > >
> > > Making a http interface should be simple after KAFKA-1494 is done which
> > all
> > > client libraries could offer.
> > >
> > > I will update the design tonight/tomorrow and should be able to have
> > > someone starting to work on it this week.
> > >
> > > /***
> > > Joe Stein
> > > Founder, Principal Consultant
> > > Big Data Open Source Security LLC
> > > http://www.stealth.ly
> > > Twitter: @allthingshadoop
> > > /
> > > On Oct 19, 2014 1:21 PM, "Harsha"  wrote:
> > >
> > > > +1 for Web Api
> > > >
> > > > On Sat, Oct 18, 2014, at 11:48 PM, Glen Mazza wrote:
> > > > > Apache Karaf has been doing this for quite a few years, albeit in
> > Java
> > > > > not Scala.  Still, their coding approach to creating a CLI probably
> > > > > captures many lessons learned over that time.
> > > > >
> > > > > Glen
> > > > >
> > > > > On 10/17/2014 08:03 PM, Joe Stein wrote:
> > > > > > Hi, I have been thinking about the ease of use for operations
> with
> > > > Kafka.
> > > > > > We have lots of tools doing a lot of different things and they
> are
> > > all
> > > > kind
> > > > > > of in different places.
> > > > > >
> > > > > > So, what I was thinking is to have a single interface for our
> > tooling
> > > > > > https://issues.apache.org/jira/browse/KAFKA-1694
> > > > > >
> > > > > > This would manifest itself in two ways 1) a command line
> interface
> > 2)
> > > > a repl
> > > > > >
> > > > > > We would have one entry point centrally for all Kafka commands.
> > > > > > kafka  
> > > > > > kafka createTopic --brokerList etc,
> > > > > > kafka reassignPartition --brokerList etc,
> > > > > >
> > > > > > or execute and run the shell
> > > > > >
> > > > > > kafka --brokerList localhost
> > > > > > kafka>use topicName;
> > > > > > kafka>set acl='label';
> > > > > >
> > > > > > I was thinking that all calls would be initialized through
> > > > --brokerList and
> > > > > > the broker can tell the KafkaCommandTool what server to connect
> to
> > > for
> > > > > > MetaData.
> > > > > >
> > > > > > Thoughts? Tomatoes?
> > > > > >
> > > > > > /***
> > > > > >   Joe Stein
> > > > > >   Founder, Principal Consultant
> > > > > >   Big Data Open Source Security LLC
> > > > > >   http://www.stealth.ly
> > > > > >   Twitter: @allthingshadoop <
> > http://www.twitter.com/allthingshadoop>
> > > > > > /
> > > > > >
> > > > >
> > > >
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>



-- 
-- Guozhang

[jira] [Commented] (KAFKA-1737) Document required ZkSerializer for ZkClient used with AdminUtils

2014-11-12 Thread Guozhang Wang (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1737?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14208275#comment-14208275
 ] 

Guozhang Wang commented on KAFKA-1737:
--

Hi Vivek,

Here are my thoughts: since currently we only use a single data format 
(ZkStringSerializer) in Kafka's ZK, we could just enforce it in ZkClient 
construction time; but as you mentioned, people can pass any ZkClient instances 
to the AdminUtils API functions using "new ZkClient" it is a bit hard to 
enforce it programmatically, and instead I was proposing to add a new 
createZkClient function and let people to use it instead of calling "new 
ZkClient" to create new instances. Of course we still need to change the docs 
telling people to do so, not using "new".

> Document required ZkSerializer for ZkClient used with AdminUtils
> 
>
> Key: KAFKA-1737
> URL: https://issues.apache.org/jira/browse/KAFKA-1737
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Affects Versions: 0.8.1.1
>Reporter: Stevo Slavic
>Priority: Minor
>
> {{ZkClient}} instances passed to {{AdminUtils}} calls must have 
> {{kafka.utils.ZKStringSerializer}} set as {{ZkSerializer}}. Otherwise 
> commands executed via {{AdminUtils}} may not be seen/recognizable to broker, 
> producer or consumer. E.g. producer (with auto topic creation turned off) 
> will not be able to send messages to a topic created via {{AdminUtils}}, it 
> will throw {{UnknownTopicOrPartitionException}}.
> Please consider at least documenting this requirement in {{AdminUtils}} 
> scaladoc.
> For more info see [related discussion on Kafka user mailing 
> list|http://mail-archives.apache.org/mod_mbox/kafka-users/201410.mbox/%3CCAAUywg-oihNiXuQRYeS%3D8Z3ymsmEHo6ghLs%3Dru4nbm%2BdHVz6TA%40mail.gmail.com%3E].



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka

2014-11-12 Thread Joe Stein (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14208281#comment-14208281
 ] 

Joe Stein commented on KAFKA-1173:
--

[~ewencp] I am +1 on the virtual box parts to the patch and the updates you 
last made (I really like how you added the vagrant part to the main README, 
nice touch. I am having issue with the EC2 pieces but am pretty convinced it is 
my account how it is setup with VPC so I am going to setup a new account and 
try it again. I may not have a chance to-do that until the weekend FYI.

> Using Vagrant to get up and running with Apache Kafka
> -
>
> Key: KAFKA-1173
> URL: https://issues.apache.org/jira/browse/KAFKA-1173
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Joe Stein
>Assignee: Ewen Cheslack-Postava
> Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, 
> KAFKA-1173_2014-11-11_13:50:55.patch
>
>
> Vagrant has been getting a lot of pickup in the tech communities.  I have 
> found it very useful for development and testing and working with a few 
> clients now using it to help virtualize their environments in repeatable ways.
> Using Vagrant to get up and running.
> For 0.8.0 I have a patch on github https://github.com/stealthly/kafka
> 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/)
> 2) Install Virtual Box 
> [https://www.virtualbox.org/](https://www.virtualbox.org/)
> In the main kafka folder
> 1) ./sbt update
> 2) ./sbt package
> 3) ./sbt assembly-package-dependency
> 4) vagrant up
> once this is done 
> * Zookeeper will be running 192.168.50.5
> * Broker 1 on 192.168.50.10
> * Broker 2 on 192.168.50.20
> * Broker 3 on 192.168.50.30
> When you are all up and running you will be back at a command brompt.  
> If you want you can login to the machines using vagrant shh  but 
> you don't need to.
> You can access the brokers and zookeeper by their IP
> e.g.
> bin/kafka-console-producer.sh --broker-list 
> 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox
> bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox 
> --from-beginning



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-12 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14208284#comment-14208284
 ] 

Vladimir Tretyakov commented on KAFKA-1481:
---

Maybe somebody can answer my last questions? Have to finish with this patch and 
moving forward! Thx.

> Stop using dashes AND underscores as separators in MBean names
> --
>
> Key: KAFKA-1481
> URL: https://issues.apache.org/jira/browse/KAFKA-1481
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Otis Gospodnetic
>Priority: Critical
>  Labels: patch
> Fix For: 0.8.3
>
> Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
> KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
> KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
> KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
> KAFKA-1481_2014-10-31_14-35-43.patch, 
> KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
> KAFKA-1481_2014-11-03_17-02-23.patch, 
> KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
> KAFKA-1481_2014-11-10_21-02-23.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
> alternateLayout2.png, diff-for-alternate-layout1.patch, 
> diff-for-alternate-layout2.patch, originalLayout.png
>
>
> MBeans should not use dashes or underscores as separators because these 
> characters are allowed in hostnames, topics, group and consumer IDs, etc., 
> and these are embedded in MBeans names making it impossible to parse out 
> individual bits from MBeans.
> Perhaps a pipe character should be used to avoid the conflict. 
> This looks like a major blocker because it means nobody can write Kafka 0.8.x 
> monitoring tools unless they are doing it for themselves AND do not use 
> dashes AND do not use underscores.
> See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1684) Implement TLS/SSL authentication


[ 
https://issues.apache.org/jira/browse/KAFKA-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14208321#comment-14208321
 ] 

Jun Rao commented on KAFKA-1684:


Gwen,

Thanks for the comment. Having 3 different ports is probably fine. My point is 
that since adding a port requires inter-broker request format changes (and 
rolling upgrades with this kind of changes is a bit tricky), it would be good 
if we do the request change just once. Perhaps we can work out the needed 
request format changes for both SSL and SASL first.

Regarding finding a good model to mimic, it seems that HDFS supports both 
Kerberos and SSL. Is that a better model to look into?

> Implement TLS/SSL authentication
> 
>
> Key: KAFKA-1684
> URL: https://issues.apache.org/jira/browse/KAFKA-1684
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Affects Versions: 0.9.0
>Reporter: Jay Kreps
>Assignee: Ivan Lyutov
> Attachments: KAFKA-1684.patch
>
>
> Add an SSL port to the configuration and advertise this as part of the 
> metadata request.
> If the SSL port is configured the socket server will need to add a second 
> Acceptor thread to listen on it. Connections accepted on this port will need 
> to go through the SSL handshake prior to being registered with a Processor 
> for request processing.
> SSL requests and responses may need to be wrapped or unwrapped using the 
> SSLEngine that was initialized by the acceptor. This wrapping and unwrapping 
> is very similar to what will need to be done for SASL-based authentication 
> schemes. We should have a uniform interface that covers both of these and we 
> will need to store the instance in the session with the request. The socket 
> server will have to use this object when reading and writing requests. We 
> will need to take care with the FetchRequests as the current 
> FileChannel.transferTo mechanism will be incompatible with wrap/unwrap so we 
> can only use this optimization for unencrypted sockets that don't require 
> userspace translation (wrapping).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 25995: Patch for KAFKA-1650

2014-11-12 Thread Jiangjie Qin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25995/
---

(Updated Nov. 12, 2014, 5:51 p.m.)


Review request for kafka.


Summary (updated)
-

Patch for KAFKA-1650


Bugs: KAFKA-1650 and KAKFA-1650
https://issues.apache.org/jira/browse/KAFKA-1650
https://issues.apache.org/jira/browse/KAKFA-1650


Repository: kafka


Description
---

Addressed Guozhang's comments.


Addressed Guozhang's comments


commit before switch to trunk


commit before rebase


Rebased on trunk, Addressed Guozhang's comments.


Addressed Guozhang's comments on MaxInFlightRequests


Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
mirrormaker-redesign


Diffs (updated)
-

  core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 
fbc680fde21b02f11285a4f4b442987356abd17b 
  core/src/main/scala/kafka/tools/MirrorMaker.scala 
f399105087588946987bbc84e3759935d9498b6a 

Diff: https://reviews.apache.org/r/25995/diff/


Testing
---


Thanks,

Jiangjie Qin

Re: Review Request 25995: Patch for KAKFA-1650

2014-11-12 Thread Jiangjie Qin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/25995/
---

(Updated Nov. 12, 2014, 5:51 p.m.)


Review request for kafka.


Summary (updated)
-

Patch for KAKFA-1650


Bugs: KAFKA-1650 and KAKFA-1650
https://issues.apache.org/jira/browse/KAFKA-1650
https://issues.apache.org/jira/browse/KAKFA-1650


Repository: kafka


Description (updated)
---

Addressed Guozhang's comments.


Addressed Guozhang's comments


commit before switch to trunk


commit before rebase


Rebased on trunk, Addressed Guozhang's comments.


Addressed Guozhang's comments on MaxInFlightRequests


Merge branch 'trunk' of http://git-wip-us.apache.org/repos/asf/kafka into 
mirrormaker-redesign


Diffs (updated)
-

  core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 
fbc680fde21b02f11285a4f4b442987356abd17b 
  core/src/main/scala/kafka/tools/MirrorMaker.scala 
f399105087588946987bbc84e3759935d9498b6a 

Diff: https://reviews.apache.org/r/25995/diff/


Testing
---


Thanks,

Jiangjie Qin

[jira] [Updated] (KAFKA-1650) Mirror Maker could lose data on unclean shutdown.


 [ 
https://issues.apache.org/jira/browse/KAFKA-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-1650:

Attachment: KAFKA-1650_2014-11-12_09:51:30.patch

> Mirror Maker could lose data on unclean shutdown.
> -
>
> Key: KAFKA-1650
> URL: https://issues.apache.org/jira/browse/KAFKA-1650
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Attachments: KAFKA-1650.patch, KAFKA-1650_2014-10-06_10:17:46.patch, 
> KAFKA-1650_2014-11-12_09:51:30.patch
>
>
> Currently if mirror maker got shutdown uncleanly, the data in the data 
> channel and buffer could potentially be lost. With the new producer's 
> callback, this issue could be solved.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1650) Mirror Maker could lose data on unclean shutdown.


[ 
https://issues.apache.org/jira/browse/KAFKA-1650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14208327#comment-14208327
 ] 

Jiangjie Qin commented on KAFKA-1650:
-

Updated reviewboard https://reviews.apache.org/r/25995/diff/
 against branch origin/trunk

> Mirror Maker could lose data on unclean shutdown.
> -
>
> Key: KAFKA-1650
> URL: https://issues.apache.org/jira/browse/KAFKA-1650
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Attachments: KAFKA-1650.patch, KAFKA-1650_2014-10-06_10:17:46.patch, 
> KAFKA-1650_2014-11-12_09:51:30.patch
>
>
> Currently if mirror maker got shutdown uncleanly, the data in the data 
> channel and buffer could potentially be lost. With the new producer's 
> callback, this issue could be solved.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names

2014-11-12 Thread Vladimir Tretyakov (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14208284#comment-14208284
 ] 

Vladimir Tretyakov edited comment on KAFKA-1481 at 11/12/14 5:54 PM:
-

Maybe somebody can answer my last questions? Have to finish with this patch and 
moving forward! Thx.

Will extract Kafka version like "Gwen Shapira
" has suggested in 
http://search-hadoop.com/m/4TaT4xtk36/Programmatic+Kafka+version+detection%252Fextraction&subj=Programmatic+Kafka+version+detection+extraction+
 

{quote}
So it looks like we can use Gradle to add properties to manifest file and
then use getResourceAsStream to read the file and parse it.

The Gradle part would be something like:
jar.manifest {
attributes('Implementation-Title': project.name,
'Implementation-Version': project.version,
'Built-By': System.getProperty('user.name'),
'Built-JDK': System.getProperty('java.version'),
'Built-Host': getHostname(),
'Source-Compatibility': project.sourceCompatibility,
'Target-Compatibility': project.targetCompatibility
)
}

The code part would be:
this.getClass().getClassLoader().getResourceAsStream("/META-INF/MANIFEST.MF")

Does that look like the right approach?
{quote}

What do you think?

What about 65?
{quote}
{quote}


was (Author: vladimir.tretyakov):
Maybe somebody can answer my last questions? Have to finish with this patch and 
moving forward! Thx.

> Stop using dashes AND underscores as separators in MBean names
> --
>
> Key: KAFKA-1481
> URL: https://issues.apache.org/jira/browse/KAFKA-1481
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Otis Gospodnetic
>Priority: Critical
>  Labels: patch
> Fix For: 0.8.3
>
> Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
> KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
> KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
> KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
> KAFKA-1481_2014-10-31_14-35-43.patch, 
> KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
> KAFKA-1481_2014-11-03_17-02-23.patch, 
> KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
> KAFKA-1481_2014-11-10_21-02-23.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
> alternateLayout2.png, diff-for-alternate-layout1.patch, 
> diff-for-alternate-layout2.patch, originalLayout.png
>
>
> MBeans should not use dashes or underscores as separators because these 
> characters are allowed in hostnames, topics, group and consumer IDs, etc., 
> and these are embedded in MBeans names making it impossible to parse out 
> individual bits from MBeans.
> Perhaps a pipe character should be used to avoid the conflict. 
> This looks like a major blocker because it means nobody can write Kafka 0.8.x 
> monitoring tools unless they are doing it for themselves AND do not use 
> dashes AND do not use underscores.
> See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

- Re: Kafka consumer transactional support

2014-11-12 Thread Falabella, Anthony

Hi Jun,

Thanks for taking a look at my issue and also for updating the future release
plan Wiki page.

My use case is to use Kafka as if it were a JMS provider (messaging use case).
I'm currently using Kafka 0.8.1.1 with Java and specifically the Spring
Integration Kafka Inbound Channel Adapter. Internally that adapter uses the
HighLevelConsumer which shields the caller from the internals of offsets. Let's
take the case where a consumer-group reads a number of messages and then is
abruptly terminated before properly processing those messages. In that
scenario upon restart ideally we'd begin reading at the offset we were at prior
to abruptly terminating. If we have "auto.commit.enable=true" upon restart
those messages will be considered already read and will be skipped. Setting
"auto.commit.enable=false" would help in this case but now we'd have to
manually call on the offset manager, requiring the use of the SimpleConsumer.

In my use-case, it's acceptable for some manual intervention to say "reprocess
messaging X thru Y", but to do so would require us to know the exact offset we
had started at prior to the chunk that was read in when the JVM abnormally
terminated. Perhaps I could look at the underlying ExportZkOffsets and
ImportZkOffsets Java classes mentioned in this link, but at the very least I'd
need to log the timestamp just prior to my read to be used in that query per:
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Idon'twantmyconsumer'soffsetstobecommittedautomatically.CanImanuallymanagemyconsumer'soffsets?

It sounds like the ConsumerAPI rewrite mentioned in these links might be
helpful in my situation (potentially targeted for Apr 2015):
https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ConsumerAPI
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design

In the meantime if you have any suggestions for things I might be able to use
to work-around my concern I'd be appreciative. Again I'm on 0.8.1.1 but would
be willing to look at 0.8.2 if it offered anything to help with my use-case.

Thanks
Tony

[jira] [Created] (KAFKA-1767) /admin/reassign_partitions deleted before reassignment completes

2014-11-12 Thread Ryan Berdeen (JIRA)

Ryan Berdeen created KAFKA-1767:
---

 Summary: /admin/reassign_partitions deleted before reassignment 
completes
 Key: KAFKA-1767
 URL: https://issues.apache.org/jira/browse/KAFKA-1767
 Project: Kafka
  Issue Type: Bug
  Components: controller
Affects Versions: 0.8.1.1
Reporter: Ryan Berdeen
Assignee: Neha Narkhede


https://github.com/apache/kafka/blob/0.8.1.1/core/src/main/scala/kafka/controller/KafkaController.scala#L477-L517
 describes the process of reassigning partitions. Specifically,by the time 
{{/admin/reassign_partitions}} is updated, the newly assigned replicas (RAR) 
should be in sync, and the assigned replicas (AR) in ZooKeeper should be 
updated:

{code}
4. Wait until all replicas in RAR are in sync with the leader.
...
10. Update AR in ZK with RAR.
11. Update the /admin/reassign_partitions path in ZK to remove this partition.
{code}

This worked in 0.8.1, but in 0.8.1.1 we observe {{/admin/reassign_partitions}} 
being removed before step 4 has completed.

For example, if we have AR [1,2] and then put [3,4] in 
{{/admin/reassign_partitions}}, the cluster will end up with AR [1,2,3,4] and 
ISR [1,2] when the key is removed. Eventually, the AR will be updated to [3,4].

This means that the {{kafka-reassign-partitions.sh}} tool will accept a new 
batch of reassignments before the current reassignments have finished, and our 
own tool that feeds in reassignments in small batches (see KAFKA-1677) can't 
rely on this key to detect active reassignments.

Although we haven't observed this, it seems likely that if a controller 
resignation happens, the new controller won't know that a reassignment is in 
progress, and the AR will never be updated to the RAR.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14208497#comment-14208497
 ] 

Jun Rao commented on KAFKA-1481:



65. The following are some of the choices that we have.
(1) kafka.server:type=BrokerTopicMetrics,name=AggregateBytesOutPerSec
(2) kafka.server:type=AggregateBrokerTopicMetrics,name=BytesOutPerSec
(3) kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec
(4) kafka.server:type=BrokerTopicMetrics,name=AllTopicsBytesOutPerSec
(5) kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec,allTopics=true
(6) kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSec,topics=aggregate
The following is my take. The issue with (1), (2) and (3) is that it's not 
obvious which dimension is being aggregated upon. I also don't quite like (2) 
since it breaks the convention that type is the class name. If we do go with 
this route, I'd prefer that we explicitly create an AggregateBrokerTopicMetrics 
class instead of sneaking in the prefix in KafkaMetricsGroup. (4), (5) and (6) 
will all make it clear which dimension is being aggregated upon. (4) is a bit 
weird now that we support tags since the main purpose of tags is that we don't 
have to squeeze everything into a single name. So, either (5) and (6) looks 
reasonable to me. Also, I am not sure how jconsole displays mbeans, but the 
key/value pairs in the mbean name are supposed to be unordered.

[~jjkoshy], what's your take?

As for the mbean for the Kafka version, could we do that in a separate jira? 
The approach seems reasonable.

> Stop using dashes AND underscores as separators in MBean names
> --
>
> Key: KAFKA-1481
> URL: https://issues.apache.org/jira/browse/KAFKA-1481
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Otis Gospodnetic
>Priority: Critical
>  Labels: patch
> Fix For: 0.8.3
>
> Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
> KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
> KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
> KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
> KAFKA-1481_2014-10-31_14-35-43.patch, 
> KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
> KAFKA-1481_2014-11-03_17-02-23.patch, 
> KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
> KAFKA-1481_2014-11-10_21-02-23.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
> alternateLayout2.png, diff-for-alternate-layout1.patch, 
> diff-for-alternate-layout2.patch, originalLayout.png
>
>
> MBeans should not use dashes or underscores as separators because these 
> characters are allowed in hostnames, topics, group and consumer IDs, etc., 
> and these are embedded in MBeans names making it impossible to parse out 
> individual bits from MBeans.
> Perhaps a pipe character should be used to avoid the conflict. 
> This looks like a major blocker because it means nobody can write Kafka 0.8.x 
> monitoring tools unless they are doing it for themselves AND do not use 
> dashes AND do not use underscores.
> See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 27735: Patch for KAFKA-1173

2014-11-12 Thread Ewen Cheslack-Postava


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27735/
---

(Updated Nov. 12, 2014, 7:32 p.m.)


Review request for kafka.


Bugs: KAFKA-1173
https://issues.apache.org/jira/browse/KAFKA-1173


Repository: kafka


Description (updated)
---

Add basic EC2 support, cleaner Vagrantfile, README cleanup, etc.


Better naming, hostmanager for routable VM names, vagrant-cachier to reduce 
startup cost, cleanup provisioning scripts, initial support for multiple 
zookeepers, general cleanup.


Don't sync a few directories that aren't actually required on the server.


Add generic worker node support.


Default # of workers should be 0


Add support for Zookeeper clusters.

This requires us to split up allocating VMs and provisioning because Vagrant
will run the provisioner for the first node before all nodes are allocated. This
leaves the first node running Zookeeper with unroutable peer hostnames which it,
for some reason, caches as unroutable. The cluster never properly finishes
forming since the nodes are unable to open connections to nodes booted later
than they were. The simple solution is to make sure all nodes are booted before
starting configuration so we have all the addresses and hostnames available and
routable.

Fix AWS provider commands in Vagrant README.


Addressing Joe's comments.


Add support for EC2 VPC settings.


Update Vagrant README to use --no-parallel when using EC2.

There's an issue that causes Vagrant to hang when running in
parallel. The last message is from vagrant-hostmanager, but it's not
clear if it is the actual cause.


Diffs (updated)
-

  .gitignore 99b32a6770e3da59bc0167d77d45ca339ac3dbbd 
  README.md 9aca90664b2a80a37125775ddbdea06ba6c53644 
  Vagrantfile PRE-CREATION 
  vagrant/README.md PRE-CREATION 
  vagrant/base.sh PRE-CREATION 
  vagrant/broker.sh PRE-CREATION 
  vagrant/zk.sh PRE-CREATION 

Diff: https://reviews.apache.org/r/27735/diff/


Testing
---


Thanks,

Ewen Cheslack-Postava

[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka

2014-11-12 Thread Ewen Cheslack-Postava (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14208512#comment-14208512
 ] 

Ewen Cheslack-Postava commented on KAFKA-1173:
--

Updated reviewboard https://reviews.apache.org/r/27735/diff/
 against branch origin/trunk

> Using Vagrant to get up and running with Apache Kafka
> -
>
> Key: KAFKA-1173
> URL: https://issues.apache.org/jira/browse/KAFKA-1173
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Joe Stein
>Assignee: Ewen Cheslack-Postava
> Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, 
> KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch
>
>
> Vagrant has been getting a lot of pickup in the tech communities.  I have 
> found it very useful for development and testing and working with a few 
> clients now using it to help virtualize their environments in repeatable ways.
> Using Vagrant to get up and running.
> For 0.8.0 I have a patch on github https://github.com/stealthly/kafka
> 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/)
> 2) Install Virtual Box 
> [https://www.virtualbox.org/](https://www.virtualbox.org/)
> In the main kafka folder
> 1) ./sbt update
> 2) ./sbt package
> 3) ./sbt assembly-package-dependency
> 4) vagrant up
> once this is done 
> * Zookeeper will be running 192.168.50.5
> * Broker 1 on 192.168.50.10
> * Broker 2 on 192.168.50.20
> * Broker 3 on 192.168.50.30
> When you are all up and running you will be back at a command brompt.  
> If you want you can login to the machines using vagrant shh  but 
> you don't need to.
> You can access the brokers and zookeeper by their IP
> e.g.
> bin/kafka-console-producer.sh --broker-list 
> 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox
> bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox 
> --from-beginning



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka

2014-11-12 Thread Ewen Cheslack-Postava (JIRA)


 [ 
https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-1173:
-
Attachment: KAFKA-1173_2014-11-12_11:32:09.patch

> Using Vagrant to get up and running with Apache Kafka
> -
>
> Key: KAFKA-1173
> URL: https://issues.apache.org/jira/browse/KAFKA-1173
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Joe Stein
>Assignee: Ewen Cheslack-Postava
> Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, 
> KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch
>
>
> Vagrant has been getting a lot of pickup in the tech communities.  I have 
> found it very useful for development and testing and working with a few 
> clients now using it to help virtualize their environments in repeatable ways.
> Using Vagrant to get up and running.
> For 0.8.0 I have a patch on github https://github.com/stealthly/kafka
> 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/)
> 2) Install Virtual Box 
> [https://www.virtualbox.org/](https://www.virtualbox.org/)
> In the main kafka folder
> 1) ./sbt update
> 2) ./sbt package
> 3) ./sbt assembly-package-dependency
> 4) vagrant up
> once this is done 
> * Zookeeper will be running 192.168.50.5
> * Broker 1 on 192.168.50.10
> * Broker 2 on 192.168.50.20
> * Broker 3 on 192.168.50.30
> When you are all up and running you will be back at a command brompt.  
> If you want you can login to the machines using vagrant shh  but 
> you don't need to.
> You can access the brokers and zookeeper by their IP
> e.g.
> bin/kafka-console-producer.sh --broker-list 
> 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox
> bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox 
> --from-beginning



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1173) Using Vagrant to get up and running with Apache Kafka

2014-11-12 Thread Ewen Cheslack-Postava (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14208515#comment-14208515
 ] 

Ewen Cheslack-Postava commented on KAFKA-1173:
--

[~joestein] To be honest, I'm not too surprised something is coming up with the 
EC2 support. In theory it should be simple, but VPCs introduce a bunch of 
variables, and testing is tricky since some defaults depend on the age of your 
account since that affects whether you have EC2 classic support.

I ran through a test with a VPC and found some issues. I updated the patch, 
including some additional info in the README since setting up under a VPC 
requires slight differences. My testing so far has been in EC2-Classic since 
that's the default for my account. I also put this VPC in a different region to 
make sure that worked. Finally, I've noticed that the default parallel 
provisioning seems to work fine until the very end, when it sometimes seems to 
hang. I couldn't easily track down the cause, so I updated the readme to use 
--no-parallel when using EC2. Not ideal, but it works reliably until we can 
find a better fix.

Hopefully those fixes will clear up the issue you're seeing.

> Using Vagrant to get up and running with Apache Kafka
> -
>
> Key: KAFKA-1173
> URL: https://issues.apache.org/jira/browse/KAFKA-1173
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Joe Stein
>Assignee: Ewen Cheslack-Postava
> Attachments: KAFKA-1173.patch, KAFKA-1173_2013-12-07_12:07:55.patch, 
> KAFKA-1173_2014-11-11_13:50:55.patch, KAFKA-1173_2014-11-12_11:32:09.patch
>
>
> Vagrant has been getting a lot of pickup in the tech communities.  I have 
> found it very useful for development and testing and working with a few 
> clients now using it to help virtualize their environments in repeatable ways.
> Using Vagrant to get up and running.
> For 0.8.0 I have a patch on github https://github.com/stealthly/kafka
> 1) Install Vagrant [http://www.vagrantup.com/](http://www.vagrantup.com/)
> 2) Install Virtual Box 
> [https://www.virtualbox.org/](https://www.virtualbox.org/)
> In the main kafka folder
> 1) ./sbt update
> 2) ./sbt package
> 3) ./sbt assembly-package-dependency
> 4) vagrant up
> once this is done 
> * Zookeeper will be running 192.168.50.5
> * Broker 1 on 192.168.50.10
> * Broker 2 on 192.168.50.20
> * Broker 3 on 192.168.50.30
> When you are all up and running you will be back at a command brompt.  
> If you want you can login to the machines using vagrant shh  but 
> you don't need to.
> You can access the brokers and zookeeper by their IP
> e.g.
> bin/kafka-console-producer.sh --broker-list 
> 192.168.50.10:9092,192.168.50.20:9092,192.168.50.30:9092 --topic sandbox
> bin/kafka-console-consumer.sh --zookeeper 192.168.50.5:2181 --topic sandbox 
> --from-beginning



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Kafka consumer transactional support

2014-11-12 Thread Falabella, Anthony

Hi Jun,

Thanks for taking a look at my issue and also for updating the future release
plan Wiki page.

Thanks
Tony

Re: Kafka consumer transactional support

2014-11-12 Thread Jun Rao

Yes, the new consumer api will solve your probably better. Before that's
ready, another option is to use the commitOffset() api in the high level
consumer. It doesn't take any offset though. So, to prevent message loss
during consumer failure, you will need to make sure all iterated messages
are fully processed before calling commitOffset().

Thanks,

Jun

On Wed, Nov 12, 2014 at 11:35 AM, Falabella, Anthony <
anthony.falabe...@citi.com> wrote:

> Hi Jun,
>
> Thanks for taking a look at my issue and also for updating the future
> release plan Wiki page.
>
> My use case is to use Kafka as if it were a JMS provider (messaging use
> case).
> I'm currently using Kafka  0.8.1.1 with Java and specifically the Spring
> Integration Kafka Inbound Channel Adapter.  Internally that adapter uses
> the HighLevelConsumer which shields the caller from the internals of
> offsets. Let's take the case where a consumer-group reads a number of
> messages and then is abruptly terminated before properly processing those
> messages.  In that scenario upon restart ideally we'd begin reading at the
> offset we were at prior to abruptly terminating.  If we have
> "auto.commit.enable=true" upon restart those messages will be considered
> already read and will be skipped.  Setting "auto.commit.enable=false" would
> help in this case but now we'd have to manually call on the offset manager,
> requiring the use of the SimpleConsumer.
>
> In my use-case, it's acceptable for some manual intervention to say
> "reprocess messaging X thru Y", but to do so would require us to know the
> exact offset we had started at prior to the chunk that was read in when the
> JVM abnormally terminated.  Perhaps I could look at the underlying
> ExportZkOffsets and ImportZkOffsets Java classes mentioned in this link,
> but at the very least I'd need to log the timestamp just prior to my read
> to be used in that query per:
>
> https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-Idon'twantmyconsumer'soffsetstobecommittedautomatically.CanImanuallymanagemyconsumer'soffsets
> ?
>
> It sounds like the ConsumerAPI rewrite mentioned in these links might be
> helpful in my situation (potentially targeted for Apr 2015):
>
> https://cwiki.apache.org/confluence/display/KAFKA/Client+Rewrite#ClientRewrite-ConsumerAPI
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
>
> In the meantime if you have any suggestions for things I might be able to
> use to work-around my concern I'd be appreciative.  Again I'm on 0.8.1.1
> but would be willing to look at 0.8.2 if it offered anything to help with
> my use-case.
>
> Thanks
> Tony
>

Re: Review Request 27890: Patch for KAFKA-1764

2014-11-12 Thread Jiangjie Qin


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27890/
---

(Updated Nov. 12, 2014, 10:05 p.m.)


Review request for kafka.


Bugs: KAFKA-1764
https://issues.apache.org/jira/browse/KAFKA-1764


Repository: kafka


Description (updated)
---

Changed Consumer iterator to stop putting the shutdown message back into 
channel.


Diffs (updated)
-

  core/src/main/scala/kafka/consumer/ConsumerIterator.scala 
ac491b4da2583ef7227c67f5b8bc0fd731d705c3 
  core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 
fbc680fde21b02f11285a4f4b442987356abd17b 

Diff: https://reviews.apache.org/r/27890/diff/


Testing
---


Thanks,

Jiangjie Qin

[jira] [Updated] (KAFKA-1764) ZookeeperConsumerConnector could put multiple shutdownCommand to the same data chunk queue.


 [ 
https://issues.apache.org/jira/browse/KAFKA-1764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-1764:

Attachment: KAFKA-1764_2014-11-12_14:05:35.patch

> ZookeeperConsumerConnector could put multiple shutdownCommand to the same 
> data chunk queue.
> ---
>
> Key: KAFKA-1764
> URL: https://issues.apache.org/jira/browse/KAFKA-1764
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Attachments: KAFKA-1764.patch, KAFKA-1764_2014-11-12_14:05:35.patch
>
>
> In ZookeeperConsumerConnector shutdown(), we could potentially put multiple 
> shutdownCommand into the same data chunk queue, provided the topics are 
> sharing the same data chunk queue in topicThreadIdAndQueues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1764) ZookeeperConsumerConnector could put multiple shutdownCommand to the same data chunk queue.


[ 
https://issues.apache.org/jira/browse/KAFKA-1764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14208788#comment-14208788
 ] 

Jiangjie Qin commented on KAFKA-1764:
-

Updated reviewboard https://reviews.apache.org/r/27890/diff/
 against branch origin/trunk

> ZookeeperConsumerConnector could put multiple shutdownCommand to the same 
> data chunk queue.
> ---
>
> Key: KAFKA-1764
> URL: https://issues.apache.org/jira/browse/KAFKA-1764
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Attachments: KAFKA-1764.patch, KAFKA-1764_2014-11-12_14:05:35.patch
>
>
> In ZookeeperConsumerConnector shutdown(), we could potentially put multiple 
> shutdownCommand into the same data chunk queue, provided the topics are 
> sharing the same data chunk queue in topicThreadIdAndQueues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 27684: Patch for KAFKA-1743

2014-11-12 Thread Jun Rao

> On Nov. 10, 2014, 7:50 p.m., Jun Rao wrote:
> > core/src/main/scala/kafka/consumer/ConsumerConnector.scala, lines 76-80
> > 
> >
> > We will also need to change the interface in ConsumerConnector from 
> > 
> >   def commitOffsets(retryOnFailure: Boolean = true)
> >   
> > back to 
> > 
> >   def commitOffsets
> >   
> > In ZookeeperConsumerconnector, we can make the following method private
> > 
> > def commitOffsets(retryOnFailure: Boolean = true)
> > 
> > Another question, will scala compiler be confused with 2 methods, one 
> > w/o parenthsis and one with 1 parameter having a default? Could you try 
> > compiling the code on all scala versions?
> 
> Manikumar Reddy O wrote:
> Currently below classes uses the new method  commitOffsets(true). 
> 
> kafka/javaapi/consumer/ZookeeperConsumerConnector.scala
> kafka/tools/TestEndToEndLatency.scala
> 
> If we are changing the interface,  then we need to change the above 
> classes 
> also. 
> 
> If we are not fixing this on trunk, then same problem will come in 0.8.3. 
> How to handle this? 
> 
> 2 methods, one w/o parenthsis and one with 1 parameter is getting 
> compiled on 
> all scala versions.

Thanks for the explanation. There is actually a bit of inconsistency introduced 
in this patch. 

In kafka.javaapi.consumer.ZookeeperConsumerConnector, commitOffsets() is 
implemented as the following.
  def commitOffsets() {
underlying.commitOffsets()
  }
This actually calls underlying.commitOffsets(isAutoCommit: Boolean = true) with 
a default value of true. However, ConsumerConnector.commitOffset is implemented 
as the following which sets isAutoCommit to false.
  def commitOffsets { commitOffsets(false) }

So, we should use true in the above.

Another thing that I was thinking is that it's going to be a bit confusing if 
we have the following scala apis.
  def commitOffsets(retryOnFailure: Boolean = true)
  def commitOffsets

So, if you do commitOffset it calls the second one and if you do 
commitOffset(), you actually call the first one. However, the expectation is 
probably the same method will be called in both cases. Would it be better if we 
get rid of the default like the following? Then, it's clear which method will 
be called.
  def commitOffsets(retryOnFailure: Boolean)
  def commitOffsets

- Jun

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27684/#review60652
---

On Nov. 8, 2014, 6:20 a.m., Manikumar Reddy O wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27684/
> ---
> 
> (Updated Nov. 8, 2014, 6:20 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1743
> https://issues.apache.org/jira/browse/KAFKA-1743
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> def commitOffsets method added to make ConsumerConnector backward  compatible
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/consumer/ConsumerConnector.scala 
> 07677c1c26768ef9c9032626180d0015f12cb0e0 
> 
> Diff: https://reviews.apache.org/r/27684/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Manikumar Reddy O
> 
>

[jira] [Commented] (KAFKA-1282) Disconnect idle socket connection in Selector

2014-11-12 Thread nicu marasoiu (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14208810#comment-14208810
 ] 

nicu marasoiu commented on KAFKA-1282:
--

I want, yes, I will add a few tests this week.

> Disconnect idle socket connection in Selector
> -
>
> Key: KAFKA-1282
> URL: https://issues.apache.org/jira/browse/KAFKA-1282
> Project: Kafka
>  Issue Type: Bug
>  Components: producer 
>Affects Versions: 0.8.2
>Reporter: Jun Rao
>Assignee: nicu marasoiu
>  Labels: newbie++
> Fix For: 0.9.0
>
> Attachments: 1282_access-order.patch, 1282_brush.patch, 
> 1282_brushed_up.patch, 
> KAFKA-1282_Disconnect_idle_socket_connection_in_Selector.patch
>
>
> To reduce # socket connections, it would be useful for the new producer to 
> close socket connections that are idle. We can introduce a new producer 
> config for the idle time.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 27890: Patch for KAFKA-1764

2014-11-12 Thread Guozhang Wang


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27890/#review61099
---

Ship it!


Ship It!

- Guozhang Wang


On Nov. 12, 2014, 10:05 p.m., Jiangjie Qin wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27890/
> ---
> 
> (Updated Nov. 12, 2014, 10:05 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1764
> https://issues.apache.org/jira/browse/KAFKA-1764
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Changed Consumer iterator to stop putting the shutdown message back into 
> channel.
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/consumer/ConsumerIterator.scala 
> ac491b4da2583ef7227c67f5b8bc0fd731d705c3 
>   core/src/main/scala/kafka/consumer/ZookeeperConsumerConnector.scala 
> fbc680fde21b02f11285a4f4b442987356abd17b 
> 
> Diff: https://reviews.apache.org/r/27890/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Jiangjie Qin
> 
>

Re: Review Request 27834: Fix KAFKA-1762: add comment on risks using a larger value of max.inflight.requests than 1, in KAFKA-1650 we will add another comment about its risk of data loss

2014-11-12 Thread Joel Koshy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/27834/#review61123
---

Ship it!



clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java


I would actually prefer not mentioning the "default value" comment - (say, 
if we change the default ever).


- Joel Koshy


On Nov. 10, 2014, 10:53 p.m., Guozhang Wang wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/27834/
> ---
> 
> (Updated Nov. 10, 2014, 10:53 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-1762
> https://issues.apache.org/jira/browse/KAFKA-1762
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> dummy
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 9095caf0db1e41a4acb4216fb197626fbd85b806 
> 
> Diff: https://reviews.apache.org/r/27834/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Guozhang Wang
> 
>

[jira] [Commented] (KAFKA-1762) Enforce MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION to 1 in MirrorMaker


[ 
https://issues.apache.org/jira/browse/KAFKA-1762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14208940#comment-14208940
 ] 

Joel Koshy commented on KAFKA-1762:
---

Committed the doc change to trunk

> Enforce MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION to 1 in MirrorMaker
> -
>
> Key: KAFKA-1762
> URL: https://issues.apache.org/jira/browse/KAFKA-1762
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Attachments: KAFKA-1762.patch
>
>
> The new Producer client introduces a config for the max # of inFlight 
> messages. When it is set > 1 on MirrorMaker, however, there is a risk for 
> data loss even with KAFKA-1650 because the offsets recorded in the MM's 
> offset map is no longer continuous.
> Another issue is that when this value is set > 1, there is a risk of message 
> re-ordering in the producer
> Changes:
> 1. Set max # of inFlight messages = 1 in MM
> 2. Leave comments explaining what the risks are of changing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : Kafka-trunk #328

2014-11-12 Thread Apache Jenkins Server

See

Re: Kafka consumer transactional support

2014-11-12 Thread Falabella, Anthony

I didn't realize there was a commitOffset() method on the high level consumer 
(the code is abstracted by the Spring Integration classes).
Yes, this actually suits my needs and I was able to get it to work for my use 
case.
Thank you very much - that was extremely helpful.

In case it's of any use to someone else, here's the solution I came up with.

Spring Configuration file


   
   
   
   
  
 
 
  
   





   









  
   
  
 false
 largest
  
   



   
  
 
  
   





Java Advice Class

public class KafkaConsumerAfterAdvice implements AfterReturningAdvice, 
InitializingBean {
   private KafkaConsumerContext consumerContext;
   private String consumerGroupId;

   public void setConsumerContext(KafkaConsumerContext consumerContext) {
  this.consumerContext = consumerContext;
   }

   public void setConsumerGroupId(String consumerGroupId) {
  this.consumerGroupId = consumerGroupId;
   }

   /**
   * Spring calls this after the bean has be initialized within the 
ApplicationContext.
   */
   @Override
   public void afterPropertiesSet() throws Exception {
  Assert.notNull(consumerContext, "[consumerContext] cannot be 
null");
  Assert.notNull(consumerGroupId, "[consumerGroupId] cannot be 
null");
   }

   @Override
   public void afterReturning(Object returnValue, Method method, Object[] 
args, Object target) throws Throwable {
  // If there were messages then returnValue=true otherwise 
returnValue=false.
  // Only if true do we need to take the hit to commit the offsets.
  if (returnValue.equals(true)) {
 Iterator> 
consumerConfigIterator = consumerContext.getConsumerConfigurations().iterator();
 while (consumerConfigIterator.hasNext()) {
   ConsumerConfiguration consumerConfig 
= consumerConfigIterator.next();
   if 
(consumerGroupId.equals(consumerConfig.getConsumerMetadata().getGroupId())) {
  
consumerConfig.getConsumerConnector().commitOffsets();
   }
 }
  }
   }
}

[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names


[ 
https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209055#comment-14209055
 ] 

Joel Koshy commented on KAFKA-1481:
---

(1) and (4) seem equivalent (i.e., AllTopics vs Aggregate) - or are you saying 
that (4) will be AllTopics or AllBrokers as appropriate?

I'm +0 on (5) for the reason I stated above. i.e., it is odd to see "true" when 
browsing mbeans

I'm +0 on (6) as well as "topics=aggregate" is a bit odd. The field name 
suggests it is a list of topics but it is more like a boolean. Between this and 
(5) I prefer (5).

(3) seems reasonable to me although it is not as clear as having an explicit 
aggregate term in the type. However, I think (1), (2) and (3) do make it clear 
enough what is being aggregated: i.e., bytes-out-per-sec aggregated on topic. I 
actually think "Broker" should not be there since this is a broker-side mbean 
already. i.e., if we had kafka.server:type=TopicMetrics,name=BytesOutPerSec 
(wouldn't it be clear that the dimension of aggregation is across topics?) 
i.e., I think we can just make the dimension clear from the typename.

Likewise, it should be clear (for consumers) that 
FetchRequestAndResponseMetrics is really a broker-level aggregation.


> Stop using dashes AND underscores as separators in MBean names
> --
>
> Key: KAFKA-1481
> URL: https://issues.apache.org/jira/browse/KAFKA-1481
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.1.1
>Reporter: Otis Gospodnetic
>Priority: Critical
>  Labels: patch
> Fix For: 0.8.3
>
> Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, 
> KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, 
> KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, 
> KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, 
> KAFKA-1481_2014-10-31_14-35-43.patch, 
> KAFKA-1481_2014-11-03_16-39-41_doc.patch, 
> KAFKA-1481_2014-11-03_17-02-23.patch, 
> KAFKA-1481_2014-11-10_20-39-41_doc.patch, 
> KAFKA-1481_2014-11-10_21-02-23.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, 
> KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, 
> alternateLayout2.png, diff-for-alternate-layout1.patch, 
> diff-for-alternate-layout2.patch, originalLayout.png
>
>
> MBeans should not use dashes or underscores as separators because these 
> characters are allowed in hostnames, topics, group and consumer IDs, etc., 
> and these are embedded in MBeans names making it impossible to parse out 
> individual bits from MBeans.
> Perhaps a pipe character should be used to avoid the conflict. 
> This looks like a major blocker because it means nobody can write Kafka 0.8.x 
> monitoring tools unless they are doing it for themselves AND do not use 
> dashes AND do not use underscores.
> See: http://search-hadoop.com/m/4TaT4lonIW



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1729) add doc for Kafka-based offset management in 0.8.2


[ 
https://issues.apache.org/jira/browse/KAFKA-1729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209071#comment-14209071
 ] 

Jun Rao commented on KAFKA-1729:


Thanks for the patch. A few comments.

1. We need to make sure that before people start using the Kafka-based offset 
management in production, they set offsets.topic.num.partitions and 
offsets.topic.replication.factor properly for the offset topic since the 
defaults are not suitable for production usage. Could you add that in 
implementation.html? 

2. It seems that issuing manual OffsetCommitRequest is only needed when using 
SimpleConsumer. We can probably make that clear in the wiki.

> add doc for Kafka-based offset management in 0.8.2
> --
>
> Key: KAFKA-1729
> URL: https://issues.apache.org/jira/browse/KAFKA-1729
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jun Rao
>Assignee: Joel Koshy
> Attachments: KAFKA-1782-doc-v1.patch, KAFKA-1782-doc-v2.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1752) add --replace-broker option

2014-11-12 Thread Neha Narkhede (JIRA)


[ 
https://issues.apache.org/jira/browse/KAFKA-1752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14209072#comment-14209072
 ] 

Neha Narkhede commented on KAFKA-1752:
--

bq. So it looks like we actually want --add-broker (and transfer some load to 
it) and --decommission-broker (and transfer its load somewhere)?

Right. 

[~Dmitry Pekar] We would like to avoid adding more and more nuanced options to 
the partition reassignment tool that is already too complex. I would suggest 
taking a step back and arriving at a few simple options that would cover all 
use cases. I think that all we need is a way for users to add and decommission 
brokers and the user's expectation would be that the tool comes up with a 
correct assignment that leads to roughly even distribution of partitions as per 
our replica placement strategy.

> add --replace-broker option
> ---
>
> Key: KAFKA-1752
> URL: https://issues.apache.org/jira/browse/KAFKA-1752
> Project: Kafka
>  Issue Type: Sub-task
>  Components: tools
>Reporter: Dmitry Pekar
>Assignee: Dmitry Pekar
> Fix For: 0.8.3
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1762) Update max-inflight-request doc string


 [ 
https://issues.apache.org/jira/browse/KAFKA-1762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy updated KAFKA-1762:
--
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Update max-inflight-request doc string
> --
>
> Key: KAFKA-1762
> URL: https://issues.apache.org/jira/browse/KAFKA-1762
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Attachments: KAFKA-1762.patch
>
>
> The new Producer client introduces a config for the max # of inFlight 
> messages. When it is set > 1 on MirrorMaker, however, there is a risk for 
> data loss even with KAFKA-1650 because the offsets recorded in the MM's 
> offset map is no longer continuous.
> Another issue is that when this value is set > 1, there is a risk of message 
> re-ordering in the producer
> Changes:
> 1. Set max # of inFlight messages = 1 in MM
> 2. Leave comments explaining what the risks are of changing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1762) Update max-inflight-request doc string


 [ 
https://issues.apache.org/jira/browse/KAFKA-1762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Joel Koshy updated KAFKA-1762:
--
Summary: Update max-inflight-request doc string  (was: Enforce 
MAX_IN_FLIGHT_REQUESTS_PER_CONNECTION to 1 in MirrorMaker)

> Update max-inflight-request doc string
> --
>
> Key: KAFKA-1762
> URL: https://issues.apache.org/jira/browse/KAFKA-1762
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Attachments: KAFKA-1762.patch
>
>
> The new Producer client introduces a config for the max # of inFlight 
> messages. When it is set > 1 on MirrorMaker, however, there is a risk for 
> data loss even with KAFKA-1650 because the offsets recorded in the MM's 
> offset map is no longer continuous.
> Another issue is that when this value is set > 1, there is a risk of message 
> re-ordering in the producer
> Changes:
> 1. Set max # of inFlight messages = 1 in MM
> 2. Leave comments explaining what the risks are of changing



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1555) provide strong consistency with reasonable availability