Re: Maybe 0.8.3 should really be 0.9.0?

2015-09-09 Thread Stevo Slavić 
+1 (non-binding) for 0.9

On Wed, Sep 9, 2015 at 6:41 AM, Jun Rao  wrote:

> +1 for 0.9.
>
> Thanks,
>
> Jun
>
> On Tue, Sep 8, 2015 at 3:04 PM, Ismael Juma  wrote:
>
> > +1 (non-binding) for 0.9.
> >
> > Ismael
> >
> > On Tue, Sep 8, 2015 at 10:19 AM, Gwen Shapira  wrote:
> >
> > > Hi Kafka Fans,
> > >
> > > What do you think of making the next release (the one with security,
> new
> > > consumer, quotas, etc) a 0.9.0 instead of 0.8.3?
> > >
> > > It has lots of new features, and new consumer was pretty much scoped
> for
> > > 0.9.0, so it matches our original roadmap. I feel that so many awesome
> > > features deserve a better release number.
> > >
> > > The downside is mainly some confusion (we refer to 0.8.3 in bunch of
> > > places), and noisy emails from JIRA while we change "fix version" field
> > > everywhere.
> > >
> > > Thoughts?
> > >
> >
>

Re: [VOTE] 0.8.2.2 Candidate 1

2015-09-09 Thread Manikumar Reddy 
+1 (non-binding). verified the artifacts, quick start.

On Wed, Sep 9, 2015 at 2:41 AM, Ashish  wrote:

> +1 (non-binding)
>
> Ran the build, works fine. All test cases passed
>
> On Thu, Sep 3, 2015 at 9:22 AM, Jun Rao  wrote:
> > This is the first candidate for release of Apache Kafka 0.8.2.2. This
> only
> > fixes two critical issues (KAFKA-2189 and KAFKA-2308) related to snappy
> in
> > 0.8.2.1.
> >
> > Release Notes for the 0.8.2.2 release
> >
> https://people.apache.org/~junrao/kafka-0.8.2.2-candidate1/RELEASE_NOTES.html
> >
> > *** Please download, test and vote by Tuesday, Sep 8, 7pm PT
> >
> > Kafka's KEYS file containing PGP keys we use to sign the release:
> > http://kafka.apache.org/KEYS in addition to the md5, sha1
> > and sha2 (SHA256) checksum.
> >
> > * Release artifacts to be voted upon (source and binary):
> > https://people.apache.org/~junrao/kafka-0.8.2.2-candidate1/
> >
> > * Maven artifacts to be voted upon prior to release:
> > https://repository.apache.org/content/groups/staging/
> >
> > * scala-doc
> > https://people.apache.org/~junrao/kafka-0.8.2.2-candidate1/scaladoc/
> >
> > * java-doc
> > https://people.apache.org/~junrao/kafka-0.8.2.2-candidate1/javadoc/
> >
> > * The tag to be voted upon (off the 0.8.2 branch) is the 0.8.2.2 tag
> >
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=d01226cfdcb3d9daad8465234750fa515a1e7e4a
> >
> > /***
> >
> > Thanks,
> >
> > Jun
>
>
>
> --
> thanks
> ashish
>
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>

[jira] [Commented] (KAFKA-2114) Unable to change min.insync.replicas default

2015-09-09 Thread Andrew Olson (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737079#comment-14737079
 ] 

Andrew Olson commented on KAFKA-2114:
-

Yes, that is correct.

> Unable to change min.insync.replicas default
> 
>
> Key: KAFKA-2114
> URL: https://issues.apache.org/jira/browse/KAFKA-2114
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0, 0.8.2.1
>Reporter: Bryan Baugher
>Assignee: Gwen Shapira
> Fix For: 0.8.3
>
> Attachments: KAFKA-2114.patch
>
>
> Following the comment here[1] I was unable to change the min.insync.replicas 
> default value. I tested this by setting up a 3 node cluster, wrote to a topic 
> with a replication factor of 3, using request.required.acks=-1 and setting 
> min.insync.replicas=2 on the broker's server.properties. I then shutdown 2 
> brokers but I was still able to write successfully. Only after running the 
> alter topic command setting min.insync.replicas=2 on the topic did I see 
> write failures.
> [1] - 
> http://mail-archives.apache.org/mod_mbox/kafka-users/201504.mbox/%3CCANZ-JHF71yqKE6%2BKKhWe2EGUJv6R3bTpoJnYck3u1-M35sobgg%40mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2114) Unable to change min.insync.replicas default

2015-09-09 Thread Andrew Olson (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-2114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Andrew Olson updated KAFKA-2114:

Affects Version/s: 0.8.2.0
   0.8.2.1

> Unable to change min.insync.replicas default
> 
>
> Key: KAFKA-2114
> URL: https://issues.apache.org/jira/browse/KAFKA-2114
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0, 0.8.2.1
>Reporter: Bryan Baugher
>Assignee: Gwen Shapira
> Fix For: 0.8.3
>
> Attachments: KAFKA-2114.patch
>
>
> Following the comment here[1] I was unable to change the min.insync.replicas 
> default value. I tested this by setting up a 3 node cluster, wrote to a topic 
> with a replication factor of 3, using request.required.acks=-1 and setting 
> min.insync.replicas=2 on the broker's server.properties. I then shutdown 2 
> brokers but I was still able to write successfully. Only after running the 
> alter topic command setting min.insync.replicas=2 on the topic did I see 
> write failures.
> [1] - 
> http://mail-archives.apache.org/mod_mbox/kafka-users/201504.mbox/%3CCANZ-JHF71yqKE6%2BKKhWe2EGUJv6R3bTpoJnYck3u1-M35sobgg%40mail.gmail.com%3E



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-2526) Console Producer / Consumer's serde config is not working

2015-09-09 Thread Mayuresh Gharat (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-2526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayuresh Gharat reassigned KAFKA-2526:
--

Assignee: Mayuresh Gharat

> Console Producer / Consumer's serde config is not working
> -
>
> Key: KAFKA-2526
> URL: https://issues.apache.org/jira/browse/KAFKA-2526
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Mayuresh Gharat
>  Labels: newbie
> Fix For: 0.9.0
>
>
> Although in the console producer one can specify the key value serializer, 
> they are actually not used since 1) it always serialize the input string as 
> String.getBytes (hence always pre-assume the string serializer) and 2) it is 
> actually only passed into the old producer. The same issues exist in console 
> consumer.
> In addition the configs in the console producer is messy: we have 1) some 
> config values exposed as cmd parameters, and 2) some config values in 
> --producer-property and 3) some in --property.
> It will be great to clean the configs up in both console producer and 
> consumer, and put them into a single --property parameter which could 
> possibly take a file to reading in property values as well, and only leave 
> --new-producer as the other command line parameter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2512: Add version check to broker and cl...

2015-09-09 Thread becketqin 
GitHub user becketqin opened a pull request:

https://github.com/apache/kafka/pull/200

KAFKA-2512: Add version check to broker and clients.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/becketqin/kafka KAFKA-2512

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/200.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #200


commit 486476bd7e270b5f32e35df98d9d3921a665879b
Author: Jiangjie Qin 
Date:   2015-09-08T14:47:56Z

KAFKA-2512: Add version check to broker and clients.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2512) Add version and magic byte verification on broker and clients.

2015-09-09 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-2512?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737234#comment-14737234
 ] 

ASF GitHub Bot commented on KAFKA-2512:
---

GitHub user becketqin opened a pull request:

https://github.com/apache/kafka/pull/200

KAFKA-2512: Add version check to broker and clients.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/becketqin/kafka KAFKA-2512

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/200.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #200


commit 486476bd7e270b5f32e35df98d9d3921a665879b
Author: Jiangjie Qin 
Date:   2015-09-08T14:47:56Z

KAFKA-2512: Add version check to broker and clients.




> Add version and magic byte verification on broker and clients.
> --
>
> Key: KAFKA-2512
> URL: https://issues.apache.org/jira/browse/KAFKA-2512
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>
> Both new and old consumer do not verify the magic byte of a message when it 
> consumes a message. This is needed for protocol upgrade so the consumer can 
> provide correct information to user. Same check is needed on broker side and 
> well.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: Added scripts to automate Vagrant setup...

2015-09-09 Thread granders 
GitHub user granders opened a pull request:

https://github.com/apache/kafka/pull/201

MINOR: Added scripts to automate Vagrant setup for system tests

Updated testing README accordingly.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/confluentinc/kafka 
minor-vagrant-package-script

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/201.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #201


commit b0293d8a761abce049da83501bab6928e6f2bd7f
Author: Geoff Anderson 
Date:   2015-09-09T07:53:59Z

Added tools to simplify Vagrant setup when running system tests, and 
updated testing README accordingly.

commit 26d3d6c0d9620735c15c0caddde68d6719a45bf8
Author: Geoff Anderson 
Date:   2015-09-09T17:29:35Z

Fixed check on Vagrant installation. Also, restore original 
Vagrantfile.local if vagrant up fails during box packaging.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [kafka-clients] [VOTE] 0.8.2.2 Candidate 1

2015-09-09 Thread Joel Koshy 
+1 binding

On Thu, Sep 3, 2015 at 9:22 AM, Jun Rao  wrote:
> This is the first candidate for release of Apache Kafka 0.8.2.2. This only
> fixes two critical issues (KAFKA-2189 and KAFKA-2308) related to snappy in
> 0.8.2.1.
>
> Release Notes for the 0.8.2.2 release
> https://people.apache.org/~junrao/kafka-0.8.2.2-candidate1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Tuesday, Sep 8, 7pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS in addition to the md5, sha1
> and sha2 (SHA256) checksum.
>
> * Release artifacts to be voted upon (source and binary):
> https://people.apache.org/~junrao/kafka-0.8.2.2-candidate1/
>
> * Maven artifacts to be voted upon prior to release:
> https://repository.apache.org/content/groups/staging/
>
> * scala-doc
> https://people.apache.org/~junrao/kafka-0.8.2.2-candidate1/scaladoc/
>
> * java-doc
> https://people.apache.org/~junrao/kafka-0.8.2.2-candidate1/javadoc/
>
> * The tag to be voted upon (off the 0.8.2 branch) is the 0.8.2.2 tag
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=d01226cfdcb3d9daad8465234750fa515a1e7e4a
>
> /***
>
> Thanks,
>
> Jun
>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at http://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAFc58G-O56-Wb29W65fY1KFwA3Dy9Uok%3DpixdfboDD8xQhiMog%40mail.gmail.com.
> For more options, visit https://groups.google.com/d/optout.

Re: [VOTE] 0.8.2.2 Candidate 1

2015-09-09 Thread Jun Rao 
+1 from me too.

Jun

On Thu, Sep 3, 2015 at 9:22 AM, Jun Rao  wrote:

> This is the first candidate for release of Apache Kafka 0.8.2.2. This only
> fixes two critical issues (KAFKA-2189 and KAFKA-2308) related to snappy in
> 0.8.2.1.
>
> Release Notes for the 0.8.2.2 release
>
> https://people.apache.org/~junrao/kafka-0.8.2.2-candidate1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Tuesday, Sep 8, 7pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS in addition to the md5, sha1
> and sha2 (SHA256) checksum.
>
> * Release artifacts to be voted upon (source and binary):
> https://people.apache.org/~junrao/kafka-0.8.2.2-candidate1/
>
> * Maven artifacts to be voted upon prior to release:
> https://repository.apache.org/content/groups/staging/
>
> * scala-doc
> https://people.apache.org/~junrao/kafka-0.8.2.2-candidate1/scaladoc/
>
> * java-doc
> https://people.apache.org/~junrao/kafka-0.8.2.2-candidate1/javadoc/
>
> * The tag to be voted upon (off the 0.8.2 branch) is the 0.8.2.2 tag
>
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=d01226cfdcb3d9daad8465234750fa515a1e7e4a
>
> /***
>
> Thanks,
>
> Jun
>
>

Re: [VOTE] 0.8.2.2 Candidate 1

2015-09-09 Thread Jun Rao 
Thanks everyone for voting.

The following are the results of the votes.

+1 binding = 3 votes
+1 non-binding = 3 votes
-1 = 0 votes
0 = 0 votes

The vote passes.

I will release artifacts to maven central, update the dist svn and download
site. Will send out an announce after that.

Thanks everyone who has contributed to the work in 0.8.2.2!

Jun

On Thu, Sep 3, 2015 at 9:22 AM, Jun Rao  wrote:

> This is the first candidate for release of Apache Kafka 0.8.2.2. This only
> fixes two critical issues (KAFKA-2189 and KAFKA-2308) related to snappy in
> 0.8.2.1.
>
> Release Notes for the 0.8.2.2 release
>
> https://people.apache.org/~junrao/kafka-0.8.2.2-candidate1/RELEASE_NOTES.html
>
> *** Please download, test and vote by Tuesday, Sep 8, 7pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS in addition to the md5, sha1
> and sha2 (SHA256) checksum.
>
> * Release artifacts to be voted upon (source and binary):
> https://people.apache.org/~junrao/kafka-0.8.2.2-candidate1/
>
> * Maven artifacts to be voted upon prior to release:
> https://repository.apache.org/content/groups/staging/
>
> * scala-doc
> https://people.apache.org/~junrao/kafka-0.8.2.2-candidate1/scaladoc/
>
> * java-doc
> https://people.apache.org/~junrao/kafka-0.8.2.2-candidate1/javadoc/
>
> * The tag to be voted upon (off the 0.8.2 branch) is the 0.8.2.2 tag
>
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=d01226cfdcb3d9daad8465234750fa515a1e7e4a
>
> /***
>
> Thanks,
>
> Jun
>
>

[jira] [Resolved] (KAFKA-2523) New consumer doesn't check magic version

2015-09-09 Thread Jay Kreps (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-2523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jay Kreps resolved KAFKA-2523.
--
Resolution: Duplicate

> New consumer doesn't check magic version
> 
>
> Key: KAFKA-2523
> URL: https://issues.apache.org/jira/browse/KAFKA-2523
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jay Kreps
>Priority: Blocker
>
> The new consumer should be checking that messages it reads are in the format 
> it expects (and give a sane error message otherwise) rather than just 
> randomly reading other formats and running out of memory or getting cryptic 
> errors. This should be a one-line fix.
> It is important if we're thinking of changing the message format since you 
> need the proper check in the versions BEFORE the version where you make the 
> change so sane errors occur when the change eventually does happen.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2419) Allow certain Sensors to be garbage collected after inactivity

2015-09-09 Thread Aditya Auradkar (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737497#comment-14737497
 ] 

Aditya Auradkar commented on KAFKA-2419:


[~junrao] I'm working on this now. Shall submit a patch within a few days

> Allow certain Sensors to be garbage collected after inactivity
> --
>
> Key: KAFKA-2419
> URL: https://issues.apache.org/jira/browse/KAFKA-2419
> Project: Kafka
>  Issue Type: New Feature
>Affects Versions: 0.8.3
>Reporter: Aditya Auradkar
>Assignee: Aditya Auradkar
>Priority: Blocker
>  Labels: quotas
> Fix For: 0.8.3
>
>
> Currently, metrics cannot be removed once registered. 
> Implement a feature to remove certain sensors after a certain period of 
> inactivity (perhaps configurable).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2447) Add capability to KafkaLog4jAppender to be able to use SSL

2015-09-09 Thread Ashish K Singh (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737744#comment-14737744
 ] 

Ashish K Singh commented on KAFKA-2447:
---

[~gwenshap] agreed! I have tried this on a cluster with broker's built from 
trunk.

> Add capability to KafkaLog4jAppender to be able to use SSL
> --
>
> Key: KAFKA-2447
> URL: https://issues.apache.org/jira/browse/KAFKA-2447
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> With Kafka supporting SSL, it makes sense to augment KafkaLog4jAppender to be 
> able to use SSL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2419) Allow certain Sensors to be garbage collected after inactivity

2015-09-09 Thread Aditya Auradkar (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737756#comment-14737756
 ] 

Aditya Auradkar commented on KAFKA-2419:


[~junrao][~jjkoshy] We have a couple of ways to solve this problem. Let me know 
which approach you prefer.

1. Generic Time based
- In this approach, we can mark any sensor as "transient". After a certain 
period of inactivity i.e. no record called for x minutes, we can remove the 
sensor and its associated metrics, unregister from all MetricReporters. Here, 
we can build the notion of time based retention into the root Metrics object 
itself i.e. periodically iterate through sensors and remove transient sensors 
that are deemed inactive. Is this a useful addition to the metrics library or 
does it only make sense for Quotas?

2. Time Based but keep time computation in Quota code.
- In this approach, the ClientQuotaManager can keep track of the last record 
time and trigger deleteSensor requests to the Metrics object. The metrics 
library will only understand creation and deletion of sensors.

3. Explicit deletion 
- (First suggested by Joel in 2084) Only delete sensors explicitly. For quotas, 
this involves keeping track of client disconnections. This is tricky to do 
correctly because connections/disconnections are handled in the Selector per 
SocketChannel and we need to start tracking number of connections per unique 
clientId.

> Allow certain Sensors to be garbage collected after inactivity
> --
>
> Key: KAFKA-2419
> URL: https://issues.apache.org/jira/browse/KAFKA-2419
> Project: Kafka
>  Issue Type: New Feature
>Affects Versions: 0.8.3
>Reporter: Aditya Auradkar
>Assignee: Aditya Auradkar
>Priority: Blocker
>  Labels: quotas
> Fix For: 0.8.3
>
>
> Currently, metrics cannot be removed once registered. 
> Implement a feature to remove certain sensors after a certain period of 
> inactivity (perhaps configurable).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1901) Move Kafka version to be generated in code by build (instead of in manifest)

2015-09-09 Thread Guozhang Wang (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737787#comment-14737787
 ] 

Guozhang Wang commented on KAFKA-1901:
--

Just to be more clear, an error is thrown in line 404 complaining there is no 
.git/ref/heads/[BRANCH_NAME] file.

> Move Kafka version to be generated in code by build (instead of in manifest)
> 
>
> Key: KAFKA-1901
> URL: https://issues.apache.org/jira/browse/KAFKA-1901
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0
>Reporter: Jason Rosenberg
>Assignee: Manikumar Reddy
> Fix For: 0.8.3
>
> Attachments: KAFKA-1901.patch, KAFKA-1901_2015-06-26_13:16:29.patch, 
> KAFKA-1901_2015-07-10_16:42:53.patch, KAFKA-1901_2015-07-14_17:59:56.patch, 
> KAFKA-1901_2015-08-09_15:04:39.patch, KAFKA-1901_2015-08-20_12:35:00.patch
>
>
> With 0.8.2 (rc2), I've started seeing this warning in the logs of apps 
> deployed to our staging (both server and client):
> {code}
> 2015-01-23 00:55:25,273  WARN [async-message-sender-0] common.AppInfo$ - 
> Can't read Kafka version from MANIFEST.MF. Possible cause: 
> java.lang.NullPointerException
> {code}
> The issues is that in our deployment, apps are deployed with single 'shaded' 
> jars (e.g. using the maven shade plugin).  This means the MANIFEST.MF file 
> won't have a kafka version.  Instead, suggest the kafka build generate the 
> proper version in code, as part of the build.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1901) Move Kafka version to be generated in code by build (instead of in manifest)

2015-09-09 Thread Guozhang Wang (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-1901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737786#comment-14737786
 ] 

Guozhang Wang commented on KAFKA-1901:
--

[~omkreddy] I ran into an issue with determineCommitId task in build.gradle 
because I packed the refs into .git/packed-refs, and hence .git/refs/heads 
becomes empty. Is it resolvable in build.gradle?

> Move Kafka version to be generated in code by build (instead of in manifest)
> 
>
> Key: KAFKA-1901
> URL: https://issues.apache.org/jira/browse/KAFKA-1901
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0
>Reporter: Jason Rosenberg
>Assignee: Manikumar Reddy
> Fix For: 0.8.3
>
> Attachments: KAFKA-1901.patch, KAFKA-1901_2015-06-26_13:16:29.patch, 
> KAFKA-1901_2015-07-10_16:42:53.patch, KAFKA-1901_2015-07-14_17:59:56.patch, 
> KAFKA-1901_2015-08-09_15:04:39.patch, KAFKA-1901_2015-08-20_12:35:00.patch
>
>
> With 0.8.2 (rc2), I've started seeing this warning in the logs of apps 
> deployed to our staging (both server and client):
> {code}
> 2015-01-23 00:55:25,273  WARN [async-message-sender-0] common.AppInfo$ - 
> Can't read Kafka version from MANIFEST.MF. Possible cause: 
> java.lang.NullPointerException
> {code}
> The issues is that in our deployment, apps are deployed with single 'shaded' 
> jars (e.g. using the maven shade plugin).  This means the MANIFEST.MF file 
> won't have a kafka version.  Instead, suggest the kafka build generate the 
> proper version in code, as part of the build.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2447) Add capability to KafkaLog4jAppender to be able to use SSL

2015-09-09 Thread Gwen Shapira (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-2447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737793#comment-14737793
 ] 

Gwen Shapira commented on KAFKA-2447:
-

I'm a bit uncomfortable having a completely untested component (this applied to 
entire KafkaLog4jAppender, not just the SSL bit).
Does it make sense to add Ducktape-based tests for the appender? it will save 
you the manual testing for next time :)
(This can be a followup patch, since the appender was never covered by tests)

> Add capability to KafkaLog4jAppender to be able to use SSL
> --
>
> Key: KAFKA-2447
> URL: https://issues.apache.org/jira/browse/KAFKA-2447
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ashish K Singh
>Assignee: Ashish K Singh
>
> With Kafka supporting SSL, it makes sense to augment KafkaLog4jAppender to be 
> able to use SSL.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 36858: Patch for KAFKA-2120

2015-09-09 Thread Mayuresh Gharat 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/
---

(Updated Sept. 9, 2015, 11:46 p.m.)


Review request for kafka.


Bugs: KAFKA-2120
https://issues.apache.org/jira/browse/KAFKA-2120


Repository: kafka


Description (updated)
---

Solved compile error


Addressed Jason's comments for Kip-19


Addressed Jun's comments


Addressed Jason's comments about the default values for requestTimeout


checkpoint


Addressed Joel's concerns. Also tried to include Jun's feedback.


Fixed a minor comment


Solved unittest issue


Addressed Jun's comments regarding NetworkClient


Addressed Jun's comments about disconnect() in Selector


changed logging level to debug


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
dc8f0f115bcda893c95d17c0a57be8d14518d034 
  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
049b22eadd5496b70dfcfd9d821f67c62c68a052 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
 9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
804d569498396d431880641041fc9292076452cb 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
06f00a99a73a288df9afa8c1d4abe3580fa968a6 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 4cb1e50d6c4ed55241aeaef1d3af09def5274103 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
  clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
70e74bd6aa629c430b2850ca40c97df0b16e5d75 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
69c93c3adf674b1640534c3d7410fcaafaf2232c 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 5b2e4ffaeab7127648db608c179703b27b577414 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 aa44991777a855f4b7f4f7bf17107c69393ff8ff 
  clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
df1205c935bee9a30a50816dbade64d6014b1ef2 
  clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
  clients/src/test/java/org/apache/kafka/test/MockSelector.java 
f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
da1cff07f7f76dcfa5a805718febcccd4ed5f578 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
1e8b2331486ffe55bfcc0919e48e12aad23b7d3c 
  core/src/main/scala/kafka/server/KafkaServer.scala 
30406ce809caaac56aca1f30c235b35962d55a50 
  core/src/main/scala/kafka/tools/ProducerPerformance.scala 
46a68e97b8bcc8821f21e4220ce9b3acedc5dafe 
  core/src/main/scala/kafka/utils/NetworkClientBlockingOps.scala 
ad10721de844725f27a116611209992cea61b088 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
1198df02ddd7727269e84a751ba99520f6d5584a 
  core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
5b4f2db4607ae6d17696c1140f1a771ce75c80e0 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
09b8444c2add87f0f70dbb182e892977a6b5c243 

Diff: https://reviews.apache.org/r/36858/diff/


Testing
---


Thanks,

Mayuresh Gharat

[jira] [Commented] (KAFKA-2120) Add a request timeout to NetworkClient

2015-09-09 Thread Mayuresh Gharat (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737819#comment-14737819
 ] 

Mayuresh Gharat commented on KAFKA-2120:


Updated reviewboard https://reviews.apache.org/r/36858/diff/
 against branch origin/trunk

> Add a request timeout to NetworkClient
> --
>
> Key: KAFKA-2120
> URL: https://issues.apache.org/jira/browse/KAFKA-2120
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Jiangjie Qin
>Assignee: Mayuresh Gharat
>Priority: Blocker
> Fix For: 0.8.3
>
> Attachments: KAFKA-2120.patch, KAFKA-2120_2015-07-27_15:31:19.patch, 
> KAFKA-2120_2015-07-29_15:57:02.patch, KAFKA-2120_2015-08-10_19:55:18.patch, 
> KAFKA-2120_2015-08-12_10:59:09.patch, KAFKA-2120_2015-09-03_15:12:02.patch, 
> KAFKA-2120_2015-09-04_17:49:01.patch, KAFKA-2120_2015-09-09_16:45:44.patch
>
>
> Currently NetworkClient does not have a timeout setting for requests. So if 
> no response is received for a request due to reasons such as broker is down, 
> the request will never be completed.
> Request timeout will also be used as implicit timeout for some methods such 
> as KafkaProducer.flush() and kafkaProducer.close().
> KIP-19 is created for this public interface change.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 36858: Patch for KAFKA-2120

2015-09-09 Thread Mayuresh Gharat 


> On Sept. 9, 2015, 4:43 a.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/network/Selector.java, lines 
> > 185-190
> > 
> >
> > Do we still need disconnect(id)? It seems that we can just replace the 
> > usage in test with close(id) and remove disconnect from selectable and 
> > KafkaChannel? We have to implement close(id) in MockSelector().

Hi Jun,
Thats a very good suggestion. I had a small orthogonal concern here. If we want 
people to start using NetworkClient instead of BLockingChannel, for example for 
writing a tool for fetching and commiting offsets, it would be great if we can 
provide a disconnect() in NetworkClient that can either call close(id) in 
Selector or disconnect in Selector() (In this case we need to change how 
disconnect() in Selector functions). I can add disconnect API in NetworkClient 
as a separate patch if you feel my concern is valid.


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98136
---


On Sept. 9, 2015, 11:46 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 9, 2015, 11:46 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 049b22eadd5496b70dfcfd9d821f67c62c68a052 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade6

[jira] [Updated] (KAFKA-2120) Add a request timeout to NetworkClient

2015-09-09 Thread Mayuresh Gharat (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayuresh Gharat updated KAFKA-2120:
---
Attachment: KAFKA-2120_2015-09-09_16:45:44.patch

> Add a request timeout to NetworkClient
> --
>
> Key: KAFKA-2120
> URL: https://issues.apache.org/jira/browse/KAFKA-2120
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Jiangjie Qin
>Assignee: Mayuresh Gharat
>Priority: Blocker
> Fix For: 0.8.3
>
> Attachments: KAFKA-2120.patch, KAFKA-2120_2015-07-27_15:31:19.patch, 
> KAFKA-2120_2015-07-29_15:57:02.patch, KAFKA-2120_2015-08-10_19:55:18.patch, 
> KAFKA-2120_2015-08-12_10:59:09.patch, KAFKA-2120_2015-09-03_15:12:02.patch, 
> KAFKA-2120_2015-09-04_17:49:01.patch, KAFKA-2120_2015-09-09_16:45:44.patch
>
>
> Currently NetworkClient does not have a timeout setting for requests. So if 
> no response is received for a request due to reasons such as broker is down, 
> the request will never be completed.
> Request timeout will also be used as implicit timeout for some methods such 
> as KafkaProducer.flush() and kafkaProducer.close().
> KIP-19 is created for this public interface change.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 36858: Patch for KAFKA-2120

2015-09-09 Thread Mayuresh Gharat 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/
---

(Updated Sept. 9, 2015, 11:49 p.m.)


Review request for kafka.


Bugs: KAFKA-2120
https://issues.apache.org/jira/browse/KAFKA-2120


Repository: kafka


Description (updated)
---

Solved compile error


Addressed Jason's comments for Kip-19


Addressed Jun's comments


Addressed Jason's comments about the default values for requestTimeout


checkpoint


Addressed Joel's concerns. Also tried to include Jun's feedback.


Fixed a minor comment


Solved unittest issue


Addressed Jun's comments regarding NetworkClient


Addressed Jun's comments about disconnect() in Selector


Diffs
-

  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
dc8f0f115bcda893c95d17c0a57be8d14518d034 
  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
049b22eadd5496b70dfcfd9d821f67c62c68a052 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
 9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
804d569498396d431880641041fc9292076452cb 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
06f00a99a73a288df9afa8c1d4abe3580fa968a6 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 4cb1e50d6c4ed55241aeaef1d3af09def5274103 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
  clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
70e74bd6aa629c430b2850ca40c97df0b16e5d75 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
69c93c3adf674b1640534c3d7410fcaafaf2232c 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 5b2e4ffaeab7127648db608c179703b27b577414 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 aa44991777a855f4b7f4f7bf17107c69393ff8ff 
  clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
df1205c935bee9a30a50816dbade64d6014b1ef2 
  clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
  clients/src/test/java/org/apache/kafka/test/MockSelector.java 
f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
da1cff07f7f76dcfa5a805718febcccd4ed5f578 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
1e8b2331486ffe55bfcc0919e48e12aad23b7d3c 
  core/src/main/scala/kafka/server/KafkaServer.scala 
30406ce809caaac56aca1f30c235b35962d55a50 
  core/src/main/scala/kafka/tools/ProducerPerformance.scala 
46a68e97b8bcc8821f21e4220ce9b3acedc5dafe 
  core/src/main/scala/kafka/utils/NetworkClientBlockingOps.scala 
ad10721de844725f27a116611209992cea61b088 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
1198df02ddd7727269e84a751ba99520f6d5584a 
  core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
5b4f2db4607ae6d17696c1140f1a771ce75c80e0 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
09b8444c2add87f0f70dbb182e892977a6b5c243 

Diff: https://reviews.apache.org/r/36858/diff/


Testing
---


Thanks,

Mayuresh Gharat

Re: Review Request 36858: Patch for KAFKA-2120

2015-09-09 Thread Ismael Juma 


> On Sept. 9, 2015, 4:43 a.m., Jun Rao wrote:
> > clients/src/main/java/org/apache/kafka/common/network/Selector.java, lines 
> > 185-190
> > 
> >
> > Do we still need disconnect(id)? It seems that we can just replace the 
> > usage in test with close(id) and remove disconnect from selectable and 
> > KafkaChannel? We have to implement close(id) in MockSelector().
> 
> Mayuresh Gharat wrote:
> Hi Jun,
> Thats a very good suggestion. I had a small orthogonal concern here. If 
> we want people to start using NetworkClient instead of BLockingChannel, for 
> example for writing a tool for fetching and commiting offsets, it would be 
> great if we can provide a disconnect() in NetworkClient that can either call 
> close(id) in Selector or disconnect in Selector() (In this case we need to 
> change how disconnect() in Selector functions). I can add disconnect API in 
> NetworkClient as a separate patch if you feel my concern is valid.

NetworkClient has a `close(id)` method which I have used in a number of places 
where I replaced `BlockingChannel` by `NetworkClient`. What would the proposed 
`disconnect` method do differently?


- Ismael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98136
---


On Sept. 9, 2015, 11:49 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 9, 2015, 11:49 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 049b22eadd5496b70dfcfd9d821f67c62c68a052 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apach

[jira] [Commented] (KAFKA-2419) Allow certain Sensors to be garbage collected after inactivity

2015-09-09 Thread Ismael Juma (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737864#comment-14737864
 ] 

Ismael Juma commented on KAFKA-2419:


Hi Aditya. For KAFKA-2440, I need the ability to unregister `KakfaMetric` and 
`Sensor` instances and I've implemented it locally. I'm just cleaning up the PR 
and will submit it today. That only does part of what is being asked here, but 
perhaps it can help (and it would be good to avoid duplicating the work).

> Allow certain Sensors to be garbage collected after inactivity
> --
>
> Key: KAFKA-2419
> URL: https://issues.apache.org/jira/browse/KAFKA-2419
> Project: Kafka
>  Issue Type: New Feature
>Affects Versions: 0.8.3
>Reporter: Aditya Auradkar
>Assignee: Aditya Auradkar
>Priority: Blocker
>  Labels: quotas
> Fix For: 0.8.3
>
>
> Currently, metrics cannot be removed once registered. 
> Implement a feature to remove certain sensors after a certain period of 
> inactivity (perhaps configurable).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2529) Brokers should write current version to log when they first start

2015-09-09 Thread Gwen Shapira (JIRA) 
Gwen Shapira created KAFKA-2529:
---

 Summary: Brokers should write current version to log when they 
first start
 Key: KAFKA-2529
 URL: https://issues.apache.org/jira/browse/KAFKA-2529
 Project: Kafka
  Issue Type: Bug
Reporter: Gwen Shapira


It is currently non-trivial to tell, by looking at log files, which version of 
Kafka is the log from. 

Having this information can be useful in some troubleshooting scenarios. We are 
exposing this via JMX, but since troubleshooting usually involves asking for 
logs, it will be nice if this information will be included there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1215) Rack-Aware replica assignment option

2015-09-09 Thread Aditya Auradkar (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-1215?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737918#comment-14737918
 ] 

Aditya Auradkar commented on KAFKA-1215:


[~allenxwang] Hi Allen. Thanks for the patch. Can you create a KIP to discuss 
the changes being proposed (since this patch adds configs and ZK structures) ? 
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals

We are hoping to leverage this patch within LinkedIn as well.

> Rack-Aware replica assignment option
> 
>
> Key: KAFKA-1215
> URL: https://issues.apache.org/jira/browse/KAFKA-1215
> Project: Kafka
>  Issue Type: Improvement
>  Components: replication
>Affects Versions: 0.8.0
>Reporter: Joris Van Remoortere
>Assignee: Jun Rao
> Fix For: 0.9.0
>
> Attachments: rack_aware_replica_assignment_v1.patch, 
> rack_aware_replica_assignment_v2.patch
>
>
> Adding a rack-id to kafka config. This rack-id can be used during replica 
> assignment by using the max-rack-replication argument in the admin scripts 
> (create topic, etc.). By default the original replication assignment 
> algorithm is used because max-rack-replication defaults to -1. 
> max-rack-replication > -1 is not honored if you are doing manual replica 
> assignment (preffered).
> If this looks good I can add some test cases specific to the rack-aware 
> assignment.
> I can also port this to trunk. We are currently running 0.8.0 in production 
> and need this, so i wrote the patch against that.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 36858: Patch for KAFKA-2120

2015-09-09 Thread Mayuresh Gharat 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98327
---



clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 (line 208)


I am going to upload a new patch for breaking out of this iteration early 
if possible. I am going to test it out and upload a new patch if that works.


- Mayuresh Gharat


On Sept. 9, 2015, 11:49 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 9, 2015, 11:49 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 049b22eadd5496b70dfcfd9d821f67c62c68a052 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
> df1205c935bee9a30a50816dbade64d6014b1ef2 
>   clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
> 3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
>   clients/src/test/java/org/apache/kafka/test/MockSelector.java 
> f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
>   core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
> da1cff07f7f76dcfa5a805718febcccd4ed5f578 
>   core/src/main/scala/kafka/server/KafkaConfig.scala 
> 1e8b2331486ffe55bfcc0919e48e12aad23b7d3c 
>   core/src/main/scala/kafka/server/KafkaServer.scala 
> 30406ce809caaac56aca1f30c235b35962d55a50 
>   core/src/main/scala/kafka/tools/ProducerPerformance.scala 
> 46a68e97b8bcc8821f21e4220ce9b3acedc5dafe 
>   core/src/main/scala/kafka/uti

[jira] [Commented] (KAFKA-2120) Add a request timeout to NetworkClient

2015-09-09 Thread Mayuresh Gharat (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737994#comment-14737994
 ] 

Mayuresh Gharat commented on KAFKA-2120:


Updated reviewboard https://reviews.apache.org/r/36858/diff/
 against branch origin/trunk

> Add a request timeout to NetworkClient
> --
>
> Key: KAFKA-2120
> URL: https://issues.apache.org/jira/browse/KAFKA-2120
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Jiangjie Qin
>Assignee: Mayuresh Gharat
>Priority: Blocker
> Fix For: 0.8.3
>
> Attachments: KAFKA-2120.patch, KAFKA-2120_2015-07-27_15:31:19.patch, 
> KAFKA-2120_2015-07-29_15:57:02.patch, KAFKA-2120_2015-08-10_19:55:18.patch, 
> KAFKA-2120_2015-08-12_10:59:09.patch, KAFKA-2120_2015-09-03_15:12:02.patch, 
> KAFKA-2120_2015-09-04_17:49:01.patch, KAFKA-2120_2015-09-09_16:45:44.patch, 
> KAFKA-2120_2015-09-09_18:56:18.patch
>
>
> Currently NetworkClient does not have a timeout setting for requests. So if 
> no response is received for a request due to reasons such as broker is down, 
> the request will never be completed.
> Request timeout will also be used as implicit timeout for some methods such 
> as KafkaProducer.flush() and kafkaProducer.close().
> KIP-19 is created for this public interface change.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2120) Add a request timeout to NetworkClient

2015-09-09 Thread Mayuresh Gharat (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayuresh Gharat updated KAFKA-2120:
---
Attachment: KAFKA-2120_2015-09-09_18:56:18.patch

> Add a request timeout to NetworkClient
> --
>
> Key: KAFKA-2120
> URL: https://issues.apache.org/jira/browse/KAFKA-2120
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Jiangjie Qin
>Assignee: Mayuresh Gharat
>Priority: Blocker
> Fix For: 0.8.3
>
> Attachments: KAFKA-2120.patch, KAFKA-2120_2015-07-27_15:31:19.patch, 
> KAFKA-2120_2015-07-29_15:57:02.patch, KAFKA-2120_2015-08-10_19:55:18.patch, 
> KAFKA-2120_2015-08-12_10:59:09.patch, KAFKA-2120_2015-09-03_15:12:02.patch, 
> KAFKA-2120_2015-09-04_17:49:01.patch, KAFKA-2120_2015-09-09_16:45:44.patch, 
> KAFKA-2120_2015-09-09_18:56:18.patch
>
>
> Currently NetworkClient does not have a timeout setting for requests. So if 
> no response is received for a request due to reasons such as broker is down, 
> the request will never be completed.
> Request timeout will also be used as implicit timeout for some methods such 
> as KafkaProducer.flush() and kafkaProducer.close().
> KIP-19 is created for this public interface change.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 36858: Patch for KAFKA-2120

2015-09-09 Thread Mayuresh Gharat 

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/
---

(Updated Sept. 10, 2015, 1:56 a.m.)


Review request for kafka.


Bugs: KAFKA-2120
https://issues.apache.org/jira/browse/KAFKA-2120


Repository: kafka


Description (updated)
---

Solved compile error


Addressed Jason's comments for Kip-19


Addressed Jun's comments


Addressed Jason's comments about the default values for requestTimeout


checkpoint


Addressed Joel's concerns. Also tried to include Jun's feedback.


Fixed a minor comment


Solved unittest issue


Addressed Jun's comments regarding NetworkClient


Addressed Jun's comments about disconnect() in Selector


changed logging level to debug


Addressed Joels comments to break out early from the loop while aborting 
expired batches


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
dc8f0f115bcda893c95d17c0a57be8d14518d034 
  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
049b22eadd5496b70dfcfd9d821f67c62c68a052 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
 9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
804d569498396d431880641041fc9292076452cb 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
06f00a99a73a288df9afa8c1d4abe3580fa968a6 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 4cb1e50d6c4ed55241aeaef1d3af09def5274103 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
  clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
70e74bd6aa629c430b2850ca40c97df0b16e5d75 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
69c93c3adf674b1640534c3d7410fcaafaf2232c 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 5b2e4ffaeab7127648db608c179703b27b577414 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 aa44991777a855f4b7f4f7bf17107c69393ff8ff 
  clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
df1205c935bee9a30a50816dbade64d6014b1ef2 
  clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
  clients/src/test/java/org/apache/kafka/test/MockSelector.java 
f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
da1cff07f7f76dcfa5a805718febcccd4ed5f578 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
1e8b2331486ffe55bfcc0919e48e12aad23b7d3c 
  core/src/main/scala/kafka/server/KafkaServer.scala 
30406ce809caaac56aca1f30c235b35962d55a50 
  core/src/main/scala/kafka/tools/ProducerPerformance.scala 
46a68e97b8bcc8821f21e4220ce9b3acedc5dafe 
  core/src/main/scala/kafka/utils/NetworkClientBlockingOps.scala 
ad10721de844725f27a116611209992cea61b088 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
1198df02ddd7727269e84a751ba99520f6d5584a 
  core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
5b4f2db4607ae6d17696c1140f1a771ce75c80e0 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
09b8444c2add87f0f70dbb182e892977a6b5c243 

Diff: https://reviews.apache.org/r/36858/diff/


Testing
---


Thanks,

Mayuresh Gharat

[GitHub] kafka pull request: KAFKA-2373: Add Kafka-backed offset storage fo...

2015-09-09 Thread ewencp 
GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/202

KAFKA-2373: Add Kafka-backed offset storage for Copycat.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
kafka-2373-copycat-distributed-offset

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/202.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #202


commit 35349311ae69acf69a4f845b0064ba0ec0db2149
Author: Ewen Cheslack-Postava 
Date:   2015-09-08T19:14:49Z

KAFKA-2373: Add Kafka-backed offset storage for Copycat.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2373) Copycat distributed offset storage

2015-09-09 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738036#comment-14738036
 ] 

ASF GitHub Bot commented on KAFKA-2373:
---

GitHub user ewencp opened a pull request:

https://github.com/apache/kafka/pull/202

KAFKA-2373: Add Kafka-backed offset storage for Copycat.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ewencp/kafka 
kafka-2373-copycat-distributed-offset

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/202.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #202


commit 35349311ae69acf69a4f845b0064ba0ec0db2149
Author: Ewen Cheslack-Postava 
Date:   2015-09-08T19:14:49Z

KAFKA-2373: Add Kafka-backed offset storage for Copycat.




> Copycat distributed offset storage
> --
>
> Key: KAFKA-2373
> URL: https://issues.apache.org/jira/browse/KAFKA-2373
> Project: Kafka
>  Issue Type: Sub-task
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.8.3
>
>
> Add offset storage for Copycat that works in distributed mode, which likely 
> means storing the data in a Kafka topic. Copycat workers will use this by 
> default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2360) The kafka-consumer-perf-test.sh script help information print useless parameters.

2015-09-09 Thread Bobby Tang (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-2360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738140#comment-14738140
 ] 

Bobby Tang commented on KAFKA-2360:
---

I'm interested in this minor one as a start, but I can't assign JIRA ticket by 
myself as a new user. Is there anyone can help ?

thanks 

-bobby

> The kafka-consumer-perf-test.sh script help information print useless 
> parameters.
> -
>
> Key: KAFKA-2360
> URL: https://issues.apache.org/jira/browse/KAFKA-2360
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.8.2.1
> Environment: Linux
>Reporter: Bo Wang
>Priority: Minor
>  Labels: newbie
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Run kafka-consumer-perf-test.sh --help to show help information,  but found 3 
> parameters useless : 
> --batch-size and --batch-size --messages
> That is producer of parameters.
> bin]# ./kafka-consumer-perf-test.sh --help
> Missing required argument "[topic]"
> Option  Description   
>  
> --  ---   
>  
> --batch-size Number of messages to write in a  
>  
>   single batch. (default: 200)
>  
> --compression-codec   
>   supported codec: NoCompressionCodec (default: 0)
>  
>   as 0, GZIPCompressionCodec as 1,
>  
>   SnappyCompressionCodec as 2,
>  
>   LZ4CompressionCodec as 3>   
>  
> --date-format  The date format to use for formatting 
>  
>   the time field. See java.text.  
>  
>   SimpleDateFormat for options.   
>  
>   (default: -MM-dd HH:mm:ss:SSS)  
>  
> --fetch-size The amount of data to fetch in a  
>  
>   single request. (default: 1048576)  
>  
> --messages The number of messages to send or 
>  
>   consume (default:   
>  
>   9223372036854775807)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2529) Brokers should write current version to log when they first start

2015-09-09 Thread Manikumar Reddy (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-2529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738147#comment-14738147
 ] 

Manikumar Reddy commented on KAFKA-2529:


We are already printing version info. 

{code}
[2015-09-11 10:04:49,145] INFO Kafka version : 0.8.3-SNAPSHOT 
(org.apache.kafka.common.utils.AppInfoParser)
[2015-09-11 10:04:49,146] INFO Kafka commitId : 8c88d198a0bcd029 
(org.apache.kafka.common.utils.AppInfoParser)
{code}

> Brokers should write current version to log when they first start
> -
>
> Key: KAFKA-2529
> URL: https://issues.apache.org/jira/browse/KAFKA-2529
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>
> It is currently non-trivial to tell, by looking at log files, which version 
> of Kafka is the log from. 
> Having this information can be useful in some troubleshooting scenarios. We 
> are exposing this via JMX, but since troubleshooting usually involves asking 
> for logs, it will be nice if this information will be included there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)