[GitHub] kafka pull request: KAFKA-2406[WIP]: Throttle ISR propagation

[becketqin]

GitHub user becketqin opened a pull request:

https://github.com/apache/kafka/pull/114

KAFKA-2406[WIP]: Throttle ISR propagation

This is a follow up patch for KAFKA-2406. Further test to verify if this 
change alone is enough to solve the problem or not. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/becketqin/kafka KAFKA-2406

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/114.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #114


commit 27224779c21781e87353a28d060322bfc2c70be2
Author: Jiangjie Qin 
Date:   2015-08-05T07:09:06Z

KAFKA-2406: Throttle ISR propagation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14654923#comment-14654923
 ] 

ASF GitHub Bot commented on KAFKA-2406:
---

GitHub user becketqin opened a pull request:

https://github.com/apache/kafka/pull/114

KAFKA-2406[WIP]: Throttle ISR propagation

This is a follow up patch for KAFKA-2406. Further test to verify if this 
change alone is enough to solve the problem or not. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/becketqin/kafka KAFKA-2406

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/114.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #114


commit 27224779c21781e87353a28d060322bfc2c70be2
Author: Jiangjie Qin 
Date:   2015-08-05T07:09:06Z

KAFKA-2406: Throttle ISR propagation




> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2351) Brokers are having a problem shutting down correctly

[Stevo Slavic (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14654927#comment-14654927
 ] 

Stevo Slavic commented on KAFKA-2351:
-

Running single instance of Kafka broker from latest trunk I experienced this 
not so successful controlled shutdown:
{noformat}
[2015-08-05 00:19:09,998] INFO [Offset Manager on Broker 0]: Removed 0 expired 
offsets in 0 milliseconds. (kafka.server.OffsetManager)
^C[2015-08-05 00:23:09,144] INFO [Kafka Server 0], shutting down 
(kafka.server.KafkaServer)
[2015-08-05 00:23:09,146] INFO [Kafka Server 0], Starting controlled shutdown 
(kafka.server.KafkaServer)
[2015-08-05 00:23:09,155] ERROR [KafkaApi-0] error when handling request Name: 
ControlledShutdownRequest; Version: 0; CorrelationId: 0; BrokerId: 0 
(kafka.server.KafkaApis)
kafka.common.ControllerMovedException: Controller moved to another broker. 
Aborting controlled shutdown
at 
kafka.controller.KafkaController.shutdownBroker(KafkaController.scala:231)
at 
kafka.server.KafkaApis.handleControlledShutdownRequest(KafkaApis.scala:146)
at kafka.server.KafkaApis.handle(KafkaApis.scala:63)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
[2015-08-05 00:23:09,156] INFO [Kafka Server 0], Remaining partitions to move:  
(kafka.server.KafkaServer)
[2015-08-05 00:23:09,156] INFO [Kafka Server 0], Error code from controller: -1 
(kafka.server.KafkaServer)
[2015-08-05 00:23:14,160] WARN [Kafka Server 0], Retrying controlled shutdown 
after the previous attempt failed... (kafka.server.KafkaServer)
[2015-08-05 00:23:14,166] ERROR [KafkaApi-0] error when handling request Name: 
ControlledShutdownRequest; Version: 0; CorrelationId: 1; BrokerId: 0 
(kafka.server.KafkaApis)
kafka.common.ControllerMovedException: Controller moved to another broker. 
Aborting controlled shutdown
at 
kafka.controller.KafkaController.shutdownBroker(KafkaController.scala:231)
at 
kafka.server.KafkaApis.handleControlledShutdownRequest(KafkaApis.scala:146)
at kafka.server.KafkaApis.handle(KafkaApis.scala:63)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
[2015-08-05 00:23:14,167] INFO [Kafka Server 0], Remaining partitions to move:  
(kafka.server.KafkaServer)
[2015-08-05 00:23:14,167] INFO [Kafka Server 0], Error code from controller: -1 
(kafka.server.KafkaServer)
[2015-08-05 00:23:19,169] WARN [Kafka Server 0], Retrying controlled shutdown 
after the previous attempt failed... (kafka.server.KafkaServer)
[2015-08-05 00:23:19,172] ERROR [KafkaApi-0] error when handling request Name: 
ControlledShutdownRequest; Version: 0; CorrelationId: 2; BrokerId: 0 
(kafka.server.KafkaApis)
kafka.common.ControllerMovedException: Controller moved to another broker. 
Aborting controlled shutdown
at 
kafka.controller.KafkaController.shutdownBroker(KafkaController.scala:231)
at 
kafka.server.KafkaApis.handleControlledShutdownRequest(KafkaApis.scala:146)
at kafka.server.KafkaApis.handle(KafkaApis.scala:63)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
[2015-08-05 00:23:19,173] INFO [Kafka Server 0], Remaining partitions to move:  
(kafka.server.KafkaServer)
[2015-08-05 00:23:19,173] INFO [Kafka Server 0], Error code from controller: -1 
(kafka.server.KafkaServer)
[2015-08-05 00:23:24,176] WARN [Kafka Server 0], Retrying controlled shutdown 
after the previous attempt failed... (kafka.server.KafkaServer)
[2015-08-05 00:23:24,177] WARN [Kafka Server 0], Proceeding to do an unclean 
shutdown as all the controlled shutdown attempts failed 
(kafka.server.KafkaServer)
[2015-08-05 00:23:24,180] INFO [Socket Server on Broker 0], Shutting down 
(kafka.network.SocketServer)
[2015-08-05 00:23:24,189] INFO [Socket Server on Broker 0], Shutdown completed 
(kafka.network.SocketServer)
[2015-08-05 00:23:24,190] INFO [Kafka Request Handler on Broker 0], shutting 
down (kafka.server.KafkaRequestHandlerPool)
[2015-08-05 00:23:24,193] INFO [Kafka Request Handler on Broker 0], shut down 
completely (kafka.server.KafkaRequestHandlerPool)
[2015-08-05 00:23:24,196] INFO [Replica Manager on Broker 0]: Shutting down 
(kafka.server.ReplicaManager)
[2015-08-05 00:23:24,196] INFO [ReplicaFetcherManager on broker 0] shutting 
down (kafka.server.ReplicaFetcherManager)
[2015-08-05 00:23:24,197] INFO [ReplicaFetcherManager on broker 0] shutdown 
completed (kafka.server.ReplicaFetcherManager)
[2015-08-05 00:23:24,197] INFO [ExpirationReaper-0], Shutting down 
(kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2015-08-05 00:23:24,310] INFO [ExpirationReaper-0], Stopped  
(kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2015-08-05 00:23:24,310] INFO [ExpirationR

[jira] [Commented] (KAFKA-1684) Implement TLS/SSL authentication

[Navjot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14655162#comment-14655162
 ] 

Navjot commented on KAFKA-1684:
---

Thanks for your help, We're now able to use the patch available there. Much 
appreciated.

> Implement TLS/SSL authentication
> 
>
> Key: KAFKA-1684
> URL: https://issues.apache.org/jira/browse/KAFKA-1684
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Jay Kreps
>Assignee: Sriharsha Chintalapani
> Fix For: 0.8.3
>
> Attachments: KAFKA-1684.patch, KAFKA-1684.patch
>
>
> Add an SSL port to the configuration and advertise this as part of the 
> metadata request.
> If the SSL port is configured the socket server will need to add a second 
> Acceptor thread to listen on it. Connections accepted on this port will need 
> to go through the SSL handshake prior to being registered with a Processor 
> for request processing.
> SSL requests and responses may need to be wrapped or unwrapped using the 
> SSLEngine that was initialized by the acceptor. This wrapping and unwrapping 
> is very similar to what will need to be done for SASL-based authentication 
> schemes. We should have a uniform interface that covers both of these and we 
> will need to store the instance in the session with the request. The socket 
> server will have to use this object when reading and writing requests. We 
> will need to take care with the FetchRequests as the current 
> FileChannel.transferTo mechanism will be incompatible with wrap/unwrap so we 
> can only use this optimization for unencrypted sockets that don't require 
> userspace translation (wrapping).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1684) Implement TLS/SSL authentication

[Navjot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14655163#comment-14655163
 ] 

Navjot commented on KAFKA-1684:
---

Thanks for your help, We're now able to use the patch available there. Much 
appreciated.

> Implement TLS/SSL authentication
> 
>
> Key: KAFKA-1684
> URL: https://issues.apache.org/jira/browse/KAFKA-1684
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Jay Kreps
>Assignee: Sriharsha Chintalapani
> Fix For: 0.8.3
>
> Attachments: KAFKA-1684.patch, KAFKA-1684.patch
>
>
> Add an SSL port to the configuration and advertise this as part of the 
> metadata request.
> If the SSL port is configured the socket server will need to add a second 
> Acceptor thread to listen on it. Connections accepted on this port will need 
> to go through the SSL handshake prior to being registered with a Processor 
> for request processing.
> SSL requests and responses may need to be wrapped or unwrapped using the 
> SSLEngine that was initialized by the acceptor. This wrapping and unwrapping 
> is very similar to what will need to be done for SASL-based authentication 
> schemes. We should have a uniform interface that covers both of these and we 
> will need to store the instance in the session with the request. The socket 
> server will have to use this object when reading and writing requests. We 
> will need to take care with the FetchRequests as the current 
> FileChannel.transferTo mechanism will be incompatible with wrap/unwrap so we 
> can only use this optimization for unencrypted sockets that don't require 
> userspace translation (wrapping).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1684) Implement TLS/SSL authentication

[Navjot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14655165#comment-14655165
 ] 

Navjot commented on KAFKA-1684:
---

Thanks for your help, We're now able to use the patch available there. Much 
appreciated.

> Implement TLS/SSL authentication
> 
>
> Key: KAFKA-1684
> URL: https://issues.apache.org/jira/browse/KAFKA-1684
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Jay Kreps
>Assignee: Sriharsha Chintalapani
> Fix For: 0.8.3
>
> Attachments: KAFKA-1684.patch, KAFKA-1684.patch
>
>
> Add an SSL port to the configuration and advertise this as part of the 
> metadata request.
> If the SSL port is configured the socket server will need to add a second 
> Acceptor thread to listen on it. Connections accepted on this port will need 
> to go through the SSL handshake prior to being registered with a Processor 
> for request processing.
> SSL requests and responses may need to be wrapped or unwrapped using the 
> SSLEngine that was initialized by the acceptor. This wrapping and unwrapping 
> is very similar to what will need to be done for SASL-based authentication 
> schemes. We should have a uniform interface that covers both of these and we 
> will need to store the instance in the session with the request. The socket 
> server will have to use this object when reading and writing requests. We 
> will need to take care with the FetchRequests as the current 
> FileChannel.transferTo mechanism will be incompatible with wrap/unwrap so we 
> can only use this optimization for unencrypted sockets that don't require 
> userspace translation (wrapping).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (KAFKA-1684) Implement TLS/SSL authentication

[Navjot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navjot updated KAFKA-1684:
--
Comment: was deleted

(was: Thanks for your help, We're now able to use the patch available there. 
Much appreciated.)

> Implement TLS/SSL authentication
> 
>
> Key: KAFKA-1684
> URL: https://issues.apache.org/jira/browse/KAFKA-1684
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Jay Kreps
>Assignee: Sriharsha Chintalapani
> Fix For: 0.8.3
>
> Attachments: KAFKA-1684.patch, KAFKA-1684.patch
>
>
> Add an SSL port to the configuration and advertise this as part of the 
> metadata request.
> If the SSL port is configured the socket server will need to add a second 
> Acceptor thread to listen on it. Connections accepted on this port will need 
> to go through the SSL handshake prior to being registered with a Processor 
> for request processing.
> SSL requests and responses may need to be wrapped or unwrapped using the 
> SSLEngine that was initialized by the acceptor. This wrapping and unwrapping 
> is very similar to what will need to be done for SASL-based authentication 
> schemes. We should have a uniform interface that covers both of these and we 
> will need to store the instance in the session with the request. The socket 
> server will have to use this object when reading and writing requests. We 
> will need to take care with the FetchRequests as the current 
> FileChannel.transferTo mechanism will be incompatible with wrap/unwrap so we 
> can only use this optimization for unencrypted sockets that don't require 
> userspace translation (wrapping).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (KAFKA-1684) Implement TLS/SSL authentication

[Navjot (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Navjot updated KAFKA-1684:
--
Comment: was deleted

(was: Thanks for your help, We're now able to use the patch available there. 
Much appreciated.)

> Implement TLS/SSL authentication
> 
>
> Key: KAFKA-1684
> URL: https://issues.apache.org/jira/browse/KAFKA-1684
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Jay Kreps
>Assignee: Sriharsha Chintalapani
> Fix For: 0.8.3
>
> Attachments: KAFKA-1684.patch, KAFKA-1684.patch
>
>
> Add an SSL port to the configuration and advertise this as part of the 
> metadata request.
> If the SSL port is configured the socket server will need to add a second 
> Acceptor thread to listen on it. Connections accepted on this port will need 
> to go through the SSL handshake prior to being registered with a Processor 
> for request processing.
> SSL requests and responses may need to be wrapped or unwrapped using the 
> SSLEngine that was initialized by the acceptor. This wrapping and unwrapping 
> is very similar to what will need to be done for SASL-based authentication 
> schemes. We should have a uniform interface that covers both of these and we 
> will need to store the instance in the session with the request. The socket 
> server will have to use this object when reading and writing requests. We 
> will need to take care with the FetchRequests as the current 
> FileChannel.transferTo mechanism will be incompatible with wrap/unwrap so we 
> can only use this optimization for unencrypted sockets that don't require 
> userspace translation (wrapping).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Modified the contribution pages in the wiki

[Ismael Juma]

Thanks for making it clearer Gwen. One less item on my to-do list. :) More
below.

On Tue, Aug 4, 2015 at 11:56 PM, Gwen Shapira  wrote:

> There's always a plan!
>
> The contributor page only lists github as a valid contribution method.
>
> Theoretically the committers / reviewers should start asking contributors
> who upload patches to send PRs instead and point them at the contributors
> page (
> https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Code+Changes
> ).
>
> Patches that are "in flight" will continue in their current trajectory
> (i.e. we won't ask anyone in review process to move to github PR).
>
> Once all "in flight" patches are either committed or rejected (expect a
> month or two), we'll remove all trace of the old process from the wiki and
> remove the tool from our trunk.
>
> Sounds good?
>

This sounds good to me. One thing to keep in mind (if I am not mistaken) is
that only you and Guozhang have merged pull requests so far. It would be
good for more committers to put the new system through its paces before we
completely phase out the old one. I am particularly interested in how it
works out for bigger changes; the Copycat PR (
https://github.com/apache/kafka/pull/99) will be a good test, for example.

Best,
Ismael

Re: Modified the contribution pages in the wiki

[Ismael Juma]

Hi Gwen,

I moved some text from
https://cwiki.apache.org/confluence/display/KAFKA/Patch+submission+and+review
to https://cwiki.apache.org/confluence/display/KAFKA/Patch+Review+Tool and
made a few other minor tweaks. I think it's even clearer this way. Please
let me know if you disagree.

Thanks,
Ismael

[jira] [Commented] (KAFKA-2405) KafkaHealthCheck kills the JVM in handleSessionEstablishmentError

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658285#comment-14658285
 ] 

Ismael Juma commented on KAFKA-2405:


We have 43 instances of `System.exit` in `core` and a few instances of 
`Runtime.halt`. Is there a document explaining when it's OK to exit and when it 
is not?

> KafkaHealthCheck kills the JVM in handleSessionEstablishmentError
> -
>
> Key: KAFKA-2405
> URL: https://issues.apache.org/jira/browse/KAFKA-2405
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: jaikiran pai
>Assignee: jaikiran pai
> Fix For: 0.8.3
>
>
> The current code in KafkaHealthCheck in trunk does this:
> {code}
> override def handleSessionEstablishmentError(error: Throwable): Unit = {
>   fatal("Could not establish session with zookeeper", error)
>   System.exit(-1)
> }
> {code}
> thus terminating the JVM. A session establishment error shouldn't cause the 
> JVM to terminate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

JIRA KAFKA-1716

[Chris Barlock]

While I don't have a patch to contribute today, per the "Nag us" note 
here:

http://kafka.apache.org/contributing.html

I'm wondering if we could get some attention to 

https://issues.apache.org/jira/browse/KAFKA-1716

We recently upgraded to Kafka 0.8.2.1 and hit a deadlock every time we 
shutdown our product.  The stacktraces look very much like the ones in 
this JIRA.  This has become a real problem for us.

Thanks!

Chris

Re: Review Request 34641: Patch for KAFKA-2214

[Manikumar Reddy O]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34641/
---

(Updated Aug. 5, 2015, 3:19 p.m.)


Review request for kafka.


Bugs: KAFKA-2214
https://issues.apache.org/jira/browse/KAFKA-2214


Repository: kafka


Description (updated)
---

Addresing ismail juma's comments


Diffs (updated)
-

  core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 
ea345895a52977c25bff57e95e12b8662331d7fe 

Diff: https://reviews.apache.org/r/34641/diff/


Testing
---


Thanks,

Manikumar Reddy O

[jira] [Updated] (KAFKA-2214) kafka-reassign-partitions.sh --verify should return non-zero exit codes when reassignment is not completed yet

[Manikumar Reddy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar Reddy updated KAFKA-2214:
---
Attachment: KAFKA-2214_2015-08-05_20:47:17.patch

> kafka-reassign-partitions.sh --verify should return non-zero exit codes when 
> reassignment is not completed yet
> --
>
> Key: KAFKA-2214
> URL: https://issues.apache.org/jira/browse/KAFKA-2214
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>Affects Versions: 0.8.1.1, 0.8.2.0
>Reporter: Michael Noll
>Assignee: Manikumar Reddy
>Priority: Minor
> Attachments: KAFKA-2214.patch, KAFKA-2214_2015-07-10_21:56:04.patch, 
> KAFKA-2214_2015-07-13_21:10:58.patch, KAFKA-2214_2015-07-14_15:31:12.patch, 
> KAFKA-2214_2015-07-14_15:40:49.patch, KAFKA-2214_2015-08-05_20:47:17.patch
>
>
> h4. Background
> The admin script {{kafka-reassign-partitions.sh}} should integrate better 
> with automation tools such as Ansible, which rely on scripts adhering to Unix 
> best practices such as appropriate exit codes on success/failure.
> h4. Current behavior (incorrect)
> When reassignments are still in progress {{kafka-reassign-partitions.sh}} 
> prints {{ERROR}} messages but returns an exit code of zero, which indicates 
> success.  This behavior makes it a bit cumbersome to integrate the script 
> into automation tools such as Ansible.
> {code}
> $ kafka-reassign-partitions.sh --zookeeper zookeeper1:2181 
> --reassignment-json-file partitions-to-move.json --verify
> Status of partition reassignment:
> ERROR: Assigned replicas (316,324,311) don't match the list of replicas for 
> reassignment (316,324) for partition [mytopic,2]
> Reassignment of partition [mytopic,0] completed successfully
> Reassignment of partition [myothertopic,1] completed successfully
> Reassignment of partition [myothertopic,3] completed successfully
> ...
> $ echo $?
> 0
> # But preferably the exit code in the presence of ERRORs should be, say, 1.
> {code}
> h3. How to improve
> I'd suggest that, using the above as the running example, if there are any 
> {{ERROR}} entries in the output (i.e. if there are any assignments remaining 
> that don't match the desired assignments), then the 
> {{kafka-reassign-partitions.sh}}  should return a non-zero exit code.
> h3. Notes
> In Kafka 0.8.2 the output is a bit different: The ERROR messages are now 
> phrased differently.
> Before:
> {code}
> ERROR: Assigned replicas (316,324,311) don't match the list of replicas for 
> reassignment (316,324) for partition [mytopic,2]
> {code}
> Now:
> {code}
> Reassignment of partition [mytopic,2] is still in progress
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 34641: Patch for KAFKA-2214

[Manikumar Reddy O]

> On July 21, 2015, 1:50 p.m., Ismael Juma wrote:
> >

Thanks for the suggestions.


- Manikumar Reddy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34641/#review92403
---


On Aug. 5, 2015, 3:19 p.m., Manikumar Reddy O wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34641/
> ---
> 
> (Updated Aug. 5, 2015, 3:19 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2214
> https://issues.apache.org/jira/browse/KAFKA-2214
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Addresing ismail juma's comments
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 
> ea345895a52977c25bff57e95e12b8662331d7fe 
> 
> Diff: https://reviews.apache.org/r/34641/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Manikumar Reddy O
> 
>

[jira] [Updated] (KAFKA-2214) kafka-reassign-partitions.sh --verify should return non-zero exit codes when reassignment is not completed yet

[Manikumar Reddy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar Reddy updated KAFKA-2214:
---
Status: Patch Available  (was: In Progress)

> kafka-reassign-partitions.sh --verify should return non-zero exit codes when 
> reassignment is not completed yet
> --
>
> Key: KAFKA-2214
> URL: https://issues.apache.org/jira/browse/KAFKA-2214
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>Affects Versions: 0.8.1.1, 0.8.2.0
>Reporter: Michael Noll
>Assignee: Manikumar Reddy
>Priority: Minor
> Attachments: KAFKA-2214.patch, KAFKA-2214_2015-07-10_21:56:04.patch, 
> KAFKA-2214_2015-07-13_21:10:58.patch, KAFKA-2214_2015-07-14_15:31:12.patch, 
> KAFKA-2214_2015-07-14_15:40:49.patch, KAFKA-2214_2015-08-05_20:47:17.patch
>
>
> h4. Background
> The admin script {{kafka-reassign-partitions.sh}} should integrate better 
> with automation tools such as Ansible, which rely on scripts adhering to Unix 
> best practices such as appropriate exit codes on success/failure.
> h4. Current behavior (incorrect)
> When reassignments are still in progress {{kafka-reassign-partitions.sh}} 
> prints {{ERROR}} messages but returns an exit code of zero, which indicates 
> success.  This behavior makes it a bit cumbersome to integrate the script 
> into automation tools such as Ansible.
> {code}
> $ kafka-reassign-partitions.sh --zookeeper zookeeper1:2181 
> --reassignment-json-file partitions-to-move.json --verify
> Status of partition reassignment:
> ERROR: Assigned replicas (316,324,311) don't match the list of replicas for 
> reassignment (316,324) for partition [mytopic,2]
> Reassignment of partition [mytopic,0] completed successfully
> Reassignment of partition [myothertopic,1] completed successfully
> Reassignment of partition [myothertopic,3] completed successfully
> ...
> $ echo $?
> 0
> # But preferably the exit code in the presence of ERRORs should be, say, 1.
> {code}
> h3. How to improve
> I'd suggest that, using the above as the running example, if there are any 
> {{ERROR}} entries in the output (i.e. if there are any assignments remaining 
> that don't match the desired assignments), then the 
> {{kafka-reassign-partitions.sh}}  should return a non-zero exit code.
> h3. Notes
> In Kafka 0.8.2 the output is a bit different: The ERROR messages are now 
> phrased differently.
> Before:
> {code}
> ERROR: Assigned replicas (316,324,311) don't match the list of replicas for 
> reassignment (316,324) for partition [mytopic,2]
> {code}
> Now:
> {code}
> Reassignment of partition [mytopic,2] is still in progress
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2214) kafka-reassign-partitions.sh --verify should return non-zero exit codes when reassignment is not completed yet

[Manikumar Reddy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658357#comment-14658357
 ] 

Manikumar Reddy commented on KAFKA-2214:


Updated reviewboard https://reviews.apache.org/r/34641/diff/
 against branch origin/trunk

> kafka-reassign-partitions.sh --verify should return non-zero exit codes when 
> reassignment is not completed yet
> --
>
> Key: KAFKA-2214
> URL: https://issues.apache.org/jira/browse/KAFKA-2214
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>Affects Versions: 0.8.1.1, 0.8.2.0
>Reporter: Michael Noll
>Assignee: Manikumar Reddy
>Priority: Minor
> Attachments: KAFKA-2214.patch, KAFKA-2214_2015-07-10_21:56:04.patch, 
> KAFKA-2214_2015-07-13_21:10:58.patch, KAFKA-2214_2015-07-14_15:31:12.patch, 
> KAFKA-2214_2015-07-14_15:40:49.patch, KAFKA-2214_2015-08-05_20:47:17.patch
>
>
> h4. Background
> The admin script {{kafka-reassign-partitions.sh}} should integrate better 
> with automation tools such as Ansible, which rely on scripts adhering to Unix 
> best practices such as appropriate exit codes on success/failure.
> h4. Current behavior (incorrect)
> When reassignments are still in progress {{kafka-reassign-partitions.sh}} 
> prints {{ERROR}} messages but returns an exit code of zero, which indicates 
> success.  This behavior makes it a bit cumbersome to integrate the script 
> into automation tools such as Ansible.
> {code}
> $ kafka-reassign-partitions.sh --zookeeper zookeeper1:2181 
> --reassignment-json-file partitions-to-move.json --verify
> Status of partition reassignment:
> ERROR: Assigned replicas (316,324,311) don't match the list of replicas for 
> reassignment (316,324) for partition [mytopic,2]
> Reassignment of partition [mytopic,0] completed successfully
> Reassignment of partition [myothertopic,1] completed successfully
> Reassignment of partition [myothertopic,3] completed successfully
> ...
> $ echo $?
> 0
> # But preferably the exit code in the presence of ERRORs should be, say, 1.
> {code}
> h3. How to improve
> I'd suggest that, using the above as the running example, if there are any 
> {{ERROR}} entries in the output (i.e. if there are any assignments remaining 
> that don't match the desired assignments), then the 
> {{kafka-reassign-partitions.sh}}  should return a non-zero exit code.
> h3. Notes
> In Kafka 0.8.2 the output is a bit different: The ERROR messages are now 
> phrased differently.
> Before:
> {code}
> ERROR: Assigned replicas (316,324,311) don't match the list of replicas for 
> reassignment (316,324) for partition [mytopic,2]
> {code}
> Now:
> {code}
> Reassignment of partition [mytopic,2] is still in progress
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 34641: Patch for KAFKA-2214

[Manikumar Reddy O]

> On July 17, 2015, 5:03 a.m., Gwen Shapira wrote:
> > Just realized that verifyAssigment runs on a Map of  and 
> > originally printed status for each partition (some can succeed, fail or be 
> > in-progress).
> > 
> > We need to return a single error code. In the existing code, we return 0 if 
> > all partitions were successfull, 1 if one or more failed, 2 if one of more 
> > is in progress. If some are in-progress and some are failed, it seems like 
> > the return value is either 1 or 2, at random (since it depends on order of 
> > foreach on a Map).
> > 
> > I'm not sure thats expected :)
> > 
> > Perhaps we can come up with something better defined?

Actually return value is not random. If some are in-progress and some are 
failed, we are returning failed error code. Re-asginment failed  status is 
given more preference than in-progress error status.


- Manikumar Reddy


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34641/#review92024
---


On Aug. 5, 2015, 3:19 p.m., Manikumar Reddy O wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34641/
> ---
> 
> (Updated Aug. 5, 2015, 3:19 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2214
> https://issues.apache.org/jira/browse/KAFKA-2214
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Addresing ismail juma's comments
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/admin/ReassignPartitionsCommand.scala 
> ea345895a52977c25bff57e95e12b8662331d7fe 
> 
> Diff: https://reviews.apache.org/r/34641/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Manikumar Reddy O
> 
>

[jira] [Commented] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658417#comment-14658417
 ] 

Jun Rao commented on KAFKA-2406:


[~becket_qin], thanks for the patch. Have you tested the patch in a cluster 
with lots of partitions? How is the controlled shutdown time with your patch?



> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Modified the contribution pages in the wiki

[Gwen Shapira]

Completely agree! Thank you for reviewing and improving.





On Wed, Aug 5, 2015 at 7:22 AM, Ismael Juma  wrote:

> Hi Gwen,
>
> I moved some text from
>
> https://cwiki.apache.org/confluence/display/KAFKA/Patch+submission+and+review
> to https://cwiki.apache.org/confluence/display/KAFKA/Patch+Review+Tool and
> made a few other minor tweaks. I think it's even clearer this way. Please
> let me know if you disagree.
>
> Thanks,
> Ismael
>

[jira] [Commented] (KAFKA-2405) KafkaHealthCheck kills the JVM in handleSessionEstablishmentError

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658510#comment-14658510
 ] 

Gwen Shapira commented on KAFKA-2405:
-

It is supported to create KafkaServer by instantiating the class directly 
(without going though Kafka.main). We use it in our tests (I think), and sounds 
like LinkedIn uses this in production.

So System.exit of anything below this level sounds like a problem.

> KafkaHealthCheck kills the JVM in handleSessionEstablishmentError
> -
>
> Key: KAFKA-2405
> URL: https://issues.apache.org/jira/browse/KAFKA-2405
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: jaikiran pai
>Assignee: jaikiran pai
> Fix For: 0.8.3
>
>
> The current code in KafkaHealthCheck in trunk does this:
> {code}
> override def handleSessionEstablishmentError(error: Throwable): Unit = {
>   fatal("Could not establish session with zookeeper", error)
>   System.exit(-1)
> }
> {code}
> thus terminating the JVM. A session establishment error shouldn't cause the 
> JVM to terminate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: JIRA KAFKA-1716

[Jiangjie Qin]

Looking at the stacktrace and I am wondering if it has been solved in
KAFKA-2241.

Thanks,

Jiangjie (Becket) Qin

On Wed, Aug 5, 2015 at 8:00 AM, Chris Barlock  wrote:

> While I don't have a patch to contribute today, per the "Nag us" note
> here:
>
> http://kafka.apache.org/contributing.html
>
> I'm wondering if we could get some attention to
>
> https://issues.apache.org/jira/browse/KAFKA-1716
>
> We recently upgraded to Kafka 0.8.2.1 and hit a deadlock every time we
> shutdown our product.  The stacktraces look very much like the ones in
> this JIRA.  This has become a real problem for us.
>
> Thanks!
>
> Chris
>

Re: New Consumer API and Range Consumption with Fail-over

[Jason Gustafson]

Hey Bhavesh,

I think your use case can be handled with the new consumer API in roughly
the manner I suggested previously. It might be a little easier if we added
the ability to set the end offset for consumption. Perhaps something like
this:

// stop consumption from the partition when offset is reached
void limit(TopicPartition partition, long offset)

My guess is that we'd have a bit of an uphill battle to get this into the
first release, but it may be possible if the use case is common enough. In
any case, I think consuming to the limit offset and manually pausing the
partition is a viable alternative.

As for your question about fail-over, the new consumer provides a similar
capability to the old high-level consumer. Here is a link to the wiki which
describes its design:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design

-Jason

On Tue, Aug 4, 2015 at 12:01 AM, Bhavesh Mistry 
wrote:

> Hi Jason and Kafka Dev Team,
>
>
>
> First of all thanks for responding and I think you got expected behavior
> correctly.
>
>
>
> The use-case is offset range consumption.  We store each minute highest
> offset for each topic per partition.  So if we need to reload or re-consume
> data from yesterday per say 8AM to noon, we would have offset start mapping
> at 8AM and end offset mapping at noon in Time Series Database.
>
>
>
> I was trying to load this use case with New Consumer API.   Do you or Kafka
> Dev team agree with request to either have API that takes in topic and its
> start/end offset for High Level Consumer group  (With older consumer API we
> used Simple consumer before without fail-over).  Also, for each
> range-consumption, there will be different group id  and group id will not
> be reused.  The main purpose is to reload or process past data again (due
> to production bugs or downtime etc occasionally and let main consumer-group
> continue to consume latest records).
>
>
> void subscribe(TopicPartition[] startOffsetPartitions, TopicPartition[]
> endOffsetPartitions)
>
>
>
> or something similar which will allow following:
>
>
>
> 1)   When consumer group already exists (meaning have consumed data and
> committed offset to storage system either Kafka or ZK) ignore start offset
> positions and use committed offset.  If not committed use start Offset
> Partition.
>
> 2)   When partition consumption has reached end Offset for given partition,
> pause is fine or this assigned thread become fail over or wait for
> reassignment.
>
> 3)   When all are Consumer Group is done consuming all partitions offset
> ranges (start to end), gracefully shutdown entire consumer group.
>
> 4)   While consuming records, if one of node or consuming thread goes down
> automatic fail-over to others (Similar to High Level Consumer for OLD
> Consumer API.   I am not sure if there exists High level and/or Simple
> Consumer concept for New API  )
>
>
>
> I hope above explanation clarifies use-case and intended behavior.  Thanks
> for clarifications, and you are correct we need pause(TopicPartition tp),
> resume(TopicPartition tp), and/or API to set to end offset for each
> partition.
>
>
>
> Please do let us know your preference to support above simple use-case.
>
>
> Thanks,
>
>
> Bhavesh
>
> On Thu, Jul 30, 2015 at 1:23 PM, Jason Gustafson 
> wrote:
>
> > Hi Bhavesh,
> >
> > I'm not totally sure I understand the expected behavior, but I think this
> > can work. Instead of seeking to the start of the range before the poll
> > loop, you should probably provide a ConsumerRebalanceCallback to get
> > notifications when group assignment has changed (e.g. when one of your
> > nodes dies). When a new partition is assigned, the callback will be
> invoked
> > by the consumer and you can use it to check if there's a committed
> position
> > in the range or if you need to seek to the beginning of the range. For
> > example:
> >
> > void onPartitionsAssigned(consumer, partitions) {
> >   for (partition : partitions) {
> >  try {
> >offset = consumer.committed(partition)
> >consumer.seek(partition, offset)
> >  } catch (NoOffsetForPartition) {
> >consumer.seek(partition, rangeStart)
> >  }
> >   }
> > }
> >
> > If a failure occurs, then the partitions will be rebalanced across
> > whichever consumers are still active. The case of the entire cluster
> being
> > rebooted is not really different. When the consumers come back, they
> check
> > the committed position and resume where they left off. Does that make
> > sense?
> >
> > After you are finished consuming a partition's range, you can use
> > KafkaConsumer.pause(partition) to prevent further fetches from being
> > initiated while still maintaining the current assignment. The patch to
> add
> > pause() is not in trunk yet, but it probably will be before too long.
> >
> > One potential problem is that you wouldn't be able to reuse the same
> group
> > to consume a different range because of the way it depends on the
> comm

[jira] [Commented] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658531#comment-14658531
 ] 

Jiangjie Qin commented on KAFKA-2406:
-

Hi [~junrao], I'm doing the test right now. This patch only takes care of the 
second part of ISR propagation, which is controller to broker propagation. I 
hope this would solve our current problem. However, I am still not sure if 
creating a zookeeper path for each ISR change is a good way for the broker to 
report ISR change. Creating tens of thousands of ZKpath during cluster startup 
might cause issue.

> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2390) Seek() should take a callback.

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658534#comment-14658534
 ] 

Jason Gustafson commented on KAFKA-2390:


[~becket_qin] I can see where this might be useful for seekToBeginning and 
seekToEnd (which both rely on offset lookups), but it seems unnecessary for 
seek() which just updates the internal fetch position. Can you clarify the 
intended behavior?

> Seek() should take a callback.
> --
>
> Key: KAFKA-2390
> URL: https://issues.apache.org/jira/browse/KAFKA-2390
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>Assignee: Dong Lin
>
> Currently seek is an async call. To have the same interface, seek should also 
> take a callback just like commit().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: New Consumer API and Range Consumption with Fail-over

[Bhavesh Mistry]

Hi Jason,

Thanks for info.  I will implement (by end of next week) what you have
proposed.  If I encounter any issue,  I will let you know.

Indeed, adding new API would be uphill battle.  I did follow email chain
"Re: Kafka Consumer thoughts".

Thanks,

Bhavesh

On Wed, Aug 5, 2015 at 10:03 AM, Jason Gustafson  wrote:

> Hey Bhavesh,
>
> I think your use case can be handled with the new consumer API in roughly
> the manner I suggested previously. It might be a little easier if we added
> the ability to set the end offset for consumption. Perhaps something like
> this:
>
> // stop consumption from the partition when offset is reached
> void limit(TopicPartition partition, long offset)
>
> My guess is that we'd have a bit of an uphill battle to get this into the
> first release, but it may be possible if the use case is common enough. In
> any case, I think consuming to the limit offset and manually pausing the
> partition is a viable alternative.
>
> As for your question about fail-over, the new consumer provides a similar
> capability to the old high-level consumer. Here is a link to the wiki which
> describes its design:
>
> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+0.9+Consumer+Rewrite+Design
>
> -Jason
>
> On Tue, Aug 4, 2015 at 12:01 AM, Bhavesh Mistry <
> mistry.p.bhav...@gmail.com>
> wrote:
>
> > Hi Jason and Kafka Dev Team,
> >
> >
> >
> > First of all thanks for responding and I think you got expected behavior
> > correctly.
> >
> >
> >
> > The use-case is offset range consumption.  We store each minute highest
> > offset for each topic per partition.  So if we need to reload or
> re-consume
> > data from yesterday per say 8AM to noon, we would have offset start
> mapping
> > at 8AM and end offset mapping at noon in Time Series Database.
> >
> >
> >
> > I was trying to load this use case with New Consumer API.   Do you or
> Kafka
> > Dev team agree with request to either have API that takes in topic and
> its
> > start/end offset for High Level Consumer group  (With older consumer API
> we
> > used Simple consumer before without fail-over).  Also, for each
> > range-consumption, there will be different group id  and group id will
> not
> > be reused.  The main purpose is to reload or process past data again (due
> > to production bugs or downtime etc occasionally and let main
> consumer-group
> > continue to consume latest records).
> >
> >
> > void subscribe(TopicPartition[] startOffsetPartitions, TopicPartition[]
> > endOffsetPartitions)
> >
> >
> >
> > or something similar which will allow following:
> >
> >
> >
> > 1)   When consumer group already exists (meaning have consumed data and
> > committed offset to storage system either Kafka or ZK) ignore start
> offset
> > positions and use committed offset.  If not committed use start Offset
> > Partition.
> >
> > 2)   When partition consumption has reached end Offset for given
> partition,
> > pause is fine or this assigned thread become fail over or wait for
> > reassignment.
> >
> > 3)   When all are Consumer Group is done consuming all partitions offset
> > ranges (start to end), gracefully shutdown entire consumer group.
> >
> > 4)   While consuming records, if one of node or consuming thread goes
> down
> > automatic fail-over to others (Similar to High Level Consumer for OLD
> > Consumer API.   I am not sure if there exists High level and/or Simple
> > Consumer concept for New API  )
> >
> >
> >
> > I hope above explanation clarifies use-case and intended behavior.
> Thanks
> > for clarifications, and you are correct we need pause(TopicPartition tp),
> > resume(TopicPartition tp), and/or API to set to end offset for each
> > partition.
> >
> >
> >
> > Please do let us know your preference to support above simple use-case.
> >
> >
> > Thanks,
> >
> >
> > Bhavesh
> >
> > On Thu, Jul 30, 2015 at 1:23 PM, Jason Gustafson 
> > wrote:
> >
> > > Hi Bhavesh,
> > >
> > > I'm not totally sure I understand the expected behavior, but I think
> this
> > > can work. Instead of seeking to the start of the range before the poll
> > > loop, you should probably provide a ConsumerRebalanceCallback to get
> > > notifications when group assignment has changed (e.g. when one of your
> > > nodes dies). When a new partition is assigned, the callback will be
> > invoked
> > > by the consumer and you can use it to check if there's a committed
> > position
> > > in the range or if you need to seek to the beginning of the range. For
> > > example:
> > >
> > > void onPartitionsAssigned(consumer, partitions) {
> > >   for (partition : partitions) {
> > >  try {
> > >offset = consumer.committed(partition)
> > >consumer.seek(partition, offset)
> > >  } catch (NoOffsetForPartition) {
> > >consumer.seek(partition, rangeStart)
> > >  }
> > >   }
> > > }
> > >
> > > If a failure occurs, then the partitions will be rebalanced across
> > > whichever consumers are still active. The case of the entire cluster
> > being
> > > reb

[jira] [Commented] (KAFKA-2390) Seek() should take a callback.

[Dong Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658541#comment-14658541
 ] 

Dong Lin commented on KAFKA-2390:
-

[~hachikuji] Per discussion with [~becket_qin], the motivation is to invoke the 
callback function when the offset to seek is out of range. Does this make sense?

> Seek() should take a callback.
> --
>
> Key: KAFKA-2390
> URL: https://issues.apache.org/jira/browse/KAFKA-2390
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>Assignee: Dong Lin
>
> Currently seek is an async call. To have the same interface, seek should also 
> take a callback just like commit().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2390) Seek() should take a callback.

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658548#comment-14658548
 ] 

Jason Gustafson commented on KAFKA-2390:


[~lindong] Yeah, that makes sense. Perhaps you guys can sketch out the 
interface in the description above?

> Seek() should take a callback.
> --
>
> Key: KAFKA-2390
> URL: https://issues.apache.org/jira/browse/KAFKA-2390
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>Assignee: Dong Lin
>
> Currently seek is an async call. To have the same interface, seek should also 
> take a callback just like commit().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2390) Seek() should take a callback.

[Dong Lin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658550#comment-14658550
 ] 

Dong Lin commented on KAFKA-2390:
-

[~hachikuji] Yeah we definitely should do it. I will update it very soon.

> Seek() should take a callback.
> --
>
> Key: KAFKA-2390
> URL: https://issues.apache.org/jira/browse/KAFKA-2390
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>Assignee: Dong Lin
>
> Currently seek is an async call. To have the same interface, seek should also 
> take a callback just like commit().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2407) Only create a log directory when it will be used

[Grant Henke (JIRA)]

Grant Henke created KAFKA-2407:
--

 Summary: Only create a log directory when it will be used
 Key: KAFKA-2407
 URL: https://issues.apache.org/jira/browse/KAFKA-2407
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.8.2.1
Reporter: Grant Henke
Assignee: Grant Henke
 Fix For: 0.8.3


Currently kafka-run-class.sh will default the $LOG_DIR and create the directory 
regardless of it's use. This can cause permissions issues depending on what 
users are utilizing tools such as kafka-topics.sh.

Further down in the script there is logic to detect whether $KAFKA_LOG4J_OPTS 
is set. If it is not set this is assumed to be a tool call and the script sets 
tools-log4j.properties which only uses the console appender. In this scenario a 
logging directory is not needed. In all other cases $KAFKA_LOG4J_OPTS will be 
set and we can move the $LOG_DIR defaulting & creation logic there. For example 
kafka-server-start.sh sets $KAFKA_LOG4J_OPTS to use its own log4j.properties 
file which respects the $LOG_DIR/kafka.log4j.dir setting. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Modified the contribution pages in the wiki

[Jiangjie Qin]

Hi Gwen,

I just added one sentence saying we would prefer PR for new patches. Let me
know if you think it is too verbose.

Thanks,

Jiangjie (Becket) Qin

On Wed, Aug 5, 2015 at 8:57 AM, Gwen Shapira  wrote:

> Completely agree! Thank you for reviewing and improving.
>
>
>
>
>
> On Wed, Aug 5, 2015 at 7:22 AM, Ismael Juma  wrote:
>
> > Hi Gwen,
> >
> > I moved some text from
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Patch+submission+and+review
> > to https://cwiki.apache.org/confluence/display/KAFKA/Patch+Review+Tool
> and
> > made a few other minor tweaks. I think it's even clearer this way. Please
> > let me know if you disagree.
> >
> > Thanks,
> > Ismael
> >
>

Re: Modified the contribution pages in the wiki

[Gwen Shapira]

The sentence is good, but I think we route new developers to here:
https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Code+Changes

Can you check if this page makes it clear that we take PRs and PRs only?

Gwen

On Wed, Aug 5, 2015 at 10:36 AM, Jiangjie Qin 
wrote:

> Hi Gwen,
>
> I just added one sentence saying we would prefer PR for new patches. Let me
> know if you think it is too verbose.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Wed, Aug 5, 2015 at 8:57 AM, Gwen Shapira  wrote:
>
> > Completely agree! Thank you for reviewing and improving.
> >
> >
> >
> >
> >
> > On Wed, Aug 5, 2015 at 7:22 AM, Ismael Juma  wrote:
> >
> > > Hi Gwen,
> > >
> > > I moved some text from
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Patch+submission+and+review
> > > to https://cwiki.apache.org/confluence/display/KAFKA/Patch+Review+Tool
> > and
> > > made a few other minor tweaks. I think it's even clearer this way.
> Please
> > > let me know if you disagree.
> > >
> > > Thanks,
> > > Ismael
> > >
> >
>

[GitHub] kafka pull request: KAFKA-2407: Only create log directory when it ...

[granthenke]

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/115

KAFKA-2407: Only create log directory when it will be used



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka log-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/115.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #115


commit 49a8dd4218b47978f0210c4e6ec0100aadaf3c21
Author: Grant Henke 
Date:   2015-08-05T17:37:07Z

KAFKA-2407: Only create log directory when it will be used




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2407) Only create a log directory when it will be used

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658594#comment-14658594
 ] 

ASF GitHub Bot commented on KAFKA-2407:
---

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/115

KAFKA-2407: Only create log directory when it will be used



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka log-fix

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/115.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #115


commit 49a8dd4218b47978f0210c4e6ec0100aadaf3c21
Author: Grant Henke 
Date:   2015-08-05T17:37:07Z

KAFKA-2407: Only create log directory when it will be used




> Only create a log directory when it will be used
> 
>
> Key: KAFKA-2407
> URL: https://issues.apache.org/jira/browse/KAFKA-2407
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.8.3
>
>
> Currently kafka-run-class.sh will default the $LOG_DIR and create the 
> directory regardless of it's use. This can cause permissions issues depending 
> on what users are utilizing tools such as kafka-topics.sh.
> Further down in the script there is logic to detect whether $KAFKA_LOG4J_OPTS 
> is set. If it is not set this is assumed to be a tool call and the script 
> sets tools-log4j.properties which only uses the console appender. In this 
> scenario a logging directory is not needed. In all other cases 
> $KAFKA_LOG4J_OPTS will be set and we can move the $LOG_DIR defaulting & 
> creation logic there. For example kafka-server-start.sh sets 
> $KAFKA_LOG4J_OPTS to use its own log4j.properties file which respects the 
> $LOG_DIR/kafka.log4j.dir setting. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2407) Only create a log directory when it will be used

[Grant Henke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Henke updated KAFKA-2407:
---
Status: Patch Available  (was: Open)

> Only create a log directory when it will be used
> 
>
> Key: KAFKA-2407
> URL: https://issues.apache.org/jira/browse/KAFKA-2407
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.8.3
>
>
> Currently kafka-run-class.sh will default the $LOG_DIR and create the 
> directory regardless of it's use. This can cause permissions issues depending 
> on what users are utilizing tools such as kafka-topics.sh.
> Further down in the script there is logic to detect whether $KAFKA_LOG4J_OPTS 
> is set. If it is not set this is assumed to be a tool call and the script 
> sets tools-log4j.properties which only uses the console appender. In this 
> scenario a logging directory is not needed. In all other cases 
> $KAFKA_LOG4J_OPTS will be set and we can move the $LOG_DIR defaulting & 
> creation logic there. For example kafka-server-start.sh sets 
> $KAFKA_LOG4J_OPTS to use its own log4j.properties file which respects the 
> $LOG_DIR/kafka.log4j.dir setting. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Modified the contribution pages in the wiki

[Jiangjie Qin]

Yes, that page is crystal clear that we are using PR only :)

Jiangjie (Becket) Qin

On Wed, Aug 5, 2015 at 10:56 AM, Gwen Shapira  wrote:

> The sentence is good, but I think we route new developers to here:
> https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Code+Changes
>
> Can you check if this page makes it clear that we take PRs and PRs only?
>
> Gwen
>
> On Wed, Aug 5, 2015 at 10:36 AM, Jiangjie Qin 
> wrote:
>
> > Hi Gwen,
> >
> > I just added one sentence saying we would prefer PR for new patches. Let
> me
> > know if you think it is too verbose.
> >
> > Thanks,
> >
> > Jiangjie (Becket) Qin
> >
> > On Wed, Aug 5, 2015 at 8:57 AM, Gwen Shapira  wrote:
> >
> > > Completely agree! Thank you for reviewing and improving.
> > >
> > >
> > >
> > >
> > >
> > > On Wed, Aug 5, 2015 at 7:22 AM, Ismael Juma  wrote:
> > >
> > > > Hi Gwen,
> > > >
> > > > I moved some text from
> > > >
> > > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Patch+submission+and+review
> > > > to
> https://cwiki.apache.org/confluence/display/KAFKA/Patch+Review+Tool
> > > and
> > > > made a few other minor tweaks. I think it's even clearer this way.
> > Please
> > > > let me know if you disagree.
> > > >
> > > > Thanks,
> > > > Ismael
> > > >
> > >
> >
>

[jira] [Updated] (KAFKA-2390) Seek() should take a callback.

[Dong Lin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dong Lin updated KAFKA-2390:

Description: Currently seek is an async call. To have the same interface as 
other calls like commit(), seek() should take a callback. This callback will be 
invoked if the position to seek triggers OFFSET_OUT_OF_RANGE exception from 
broker.  (was: Currently seek is an async call. To have the same interface, 
seek should also take a callback just like commit().)

> Seek() should take a callback.
> --
>
> Key: KAFKA-2390
> URL: https://issues.apache.org/jira/browse/KAFKA-2390
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>Assignee: Dong Lin
>
> Currently seek is an async call. To have the same interface as other calls 
> like commit(), seek() should take a callback. This callback will be invoked 
> if the position to seek triggers OFFSET_OUT_OF_RANGE exception from broker.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2407: Only create log directory when it ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/115


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-2407) Only create a log directory when it will be used

[Gwen Shapira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira updated KAFKA-2407:

Resolution: Fixed
  Reviewer: Gwen Shapira
Status: Resolved  (was: Patch Available)

Pushed to trunk.
Thanks for fixing this annoying issue, [~granthenke]

> Only create a log directory when it will be used
> 
>
> Key: KAFKA-2407
> URL: https://issues.apache.org/jira/browse/KAFKA-2407
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.8.3
>
>
> Currently kafka-run-class.sh will default the $LOG_DIR and create the 
> directory regardless of it's use. This can cause permissions issues depending 
> on what users are utilizing tools such as kafka-topics.sh.
> Further down in the script there is logic to detect whether $KAFKA_LOG4J_OPTS 
> is set. If it is not set this is assumed to be a tool call and the script 
> sets tools-log4j.properties which only uses the console appender. In this 
> scenario a logging directory is not needed. In all other cases 
> $KAFKA_LOG4J_OPTS will be set and we can move the $LOG_DIR defaulting & 
> creation logic there. For example kafka-server-start.sh sets 
> $KAFKA_LOG4J_OPTS to use its own log4j.properties file which respects the 
> $LOG_DIR/kafka.log4j.dir setting. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2407) Only create a log directory when it will be used

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658669#comment-14658669
 ] 

ASF GitHub Bot commented on KAFKA-2407:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/115


> Only create a log directory when it will be used
> 
>
> Key: KAFKA-2407
> URL: https://issues.apache.org/jira/browse/KAFKA-2407
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.8.3
>
>
> Currently kafka-run-class.sh will default the $LOG_DIR and create the 
> directory regardless of it's use. This can cause permissions issues depending 
> on what users are utilizing tools such as kafka-topics.sh.
> Further down in the script there is logic to detect whether $KAFKA_LOG4J_OPTS 
> is set. If it is not set this is assumed to be a tool call and the script 
> sets tools-log4j.properties which only uses the console appender. In this 
> scenario a logging directory is not needed. In all other cases 
> $KAFKA_LOG4J_OPTS will be set and we can move the $LOG_DIR defaulting & 
> creation logic there. For example kafka-server-start.sh sets 
> $KAFKA_LOG4J_OPTS to use its own log4j.properties file which respects the 
> $LOG_DIR/kafka.log4j.dir setting. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2400; expose heartbeat interval in Kafka...

[hachikuji]

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/116

KAFKA-2400; expose heartbeat interval in KafkaConsumer configuration



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-2400

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/116.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #116


commit 3c1b1dd0dc44cd454d02aa7c476825c2ba46
Author: Jason Gustafson 
Date:   2015-08-05T18:52:35Z

KAFKA-2400; expose heartbeat interval in KafkaConsumer configuration




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-2408) (new) system tests: ConsoleConsumerService occasionally fails to register consumed message

[Geoffrey Anderson (JIRA)]

Geoffrey Anderson created KAFKA-2408:


 Summary: (new) system tests: ConsoleConsumerService occasionally 
fails to register consumed message
 Key: KAFKA-2408
 URL: https://issues.apache.org/jira/browse/KAFKA-2408
 Project: Kafka
  Issue Type: Bug
Reporter: Geoffrey Anderson
Assignee: Geoffrey Anderson
 Fix For: 0.8.3



There have been a few spurious failures in ReplicationTest.test_hard_bounce, 
where it was reported that a few of the acked messages were not consumed.

Checking the logs, however, it is clear that they were consumed, but 
ConsoleConsumerService failed to parse.

Lines causing parsing failure looks something like:

779725[2015-08-03 07:25:47,757] ERROR 
[ConsumerFetcherThread-console-consumer-78957_ip-172-31-5-20-1438586715191-249db71c-0-1],
 Error for partition [test_topic,0] to broker 1:class 
kafka.common.NotLeaderForPartitionException 
(kafka.consumer.ConsumerFetcherThread)

(i.e. the consumed message, and a log message appear on the same line)

ConsoleConsumerService simply tries to strip each line of whitespace and parse 
as an integer, which will clearly fail in this case.

Solution should either redirect stderr elsewhere or update parsing to handle 
this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2400) Expose heartbeat frequency in new consumer configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2400?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658686#comment-14658686
 ] 

ASF GitHub Bot commented on KAFKA-2400:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/116

KAFKA-2400; expose heartbeat interval in KafkaConsumer configuration



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-2400

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/116.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #116


commit 3c1b1dd0dc44cd454d02aa7c476825c2ba46
Author: Jason Gustafson 
Date:   2015-08-05T18:52:35Z

KAFKA-2400; expose heartbeat interval in KafkaConsumer configuration




> Expose heartbeat frequency in new consumer configuration
> 
>
> Key: KAFKA-2400
> URL: https://issues.apache.org/jira/browse/KAFKA-2400
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Minor
>
> The consumer coordinator communicates the need to rebalance through responses 
> to heartbeat requests sent from each member of the consumer group. The 
> heartbeat frequency therefore controls how long normal rebalances will take. 
> Currently, the frequency is hard-coded to 3 heartbeats per the configured 
> session timeout, but it would be nice to expose this setting so that the user 
> can control the impact from rebalancing.
> Since the consumer is currently single-threaded and heartbeats are sent in 
> poll(), we cannot guarantee that the heartbeats will actually be sent at the 
> configured frequency. In practice, the user may have to adjust their fetch 
> size to ensure that poll() is called often enough to get the desired 
> heartbeat frequency. For most users, the consumption rate is probably fast 
> enough for this not to matter, but we should make the documentation clear on 
> this point. In any case, we expect that most users will accept the default 
> value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658685#comment-14658685
 ] 

Jun Rao commented on KAFKA-2406:


[~becket_qin], another way to reduce the overhead is to batch on the broker 
side. Every time a broker changes an ISR, it just saves the partition in an 
in-memory map. Periodically, the replica manager can collect all partitions in 
the map and write a single node in the ISR change path. This should reduce the 
number of UpdateMetadataRequests the controller sends to the brokers. It also 
reduces the number of ZK nodes to be deleted and the number of times the ISR 
change watchers are triggered.

Also, could you provide a bit more details on the performance issue? Does that 
issue happen when shutting down the first broker in the cluster or when there 
is another broker just being restarted? It will also be good know where the 
bottleneck is: whether just in the number of UpdateMetadataRequests or in 
processing the watchers as well.

> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2400) Expose heartbeat frequency in new consumer configuration

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-2400:
---
Reviewer: Guozhang Wang
  Status: Patch Available  (was: Open)

> Expose heartbeat frequency in new consumer configuration
> 
>
> Key: KAFKA-2400
> URL: https://issues.apache.org/jira/browse/KAFKA-2400
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Minor
>
> The consumer coordinator communicates the need to rebalance through responses 
> to heartbeat requests sent from each member of the consumer group. The 
> heartbeat frequency therefore controls how long normal rebalances will take. 
> Currently, the frequency is hard-coded to 3 heartbeats per the configured 
> session timeout, but it would be nice to expose this setting so that the user 
> can control the impact from rebalancing.
> Since the consumer is currently single-threaded and heartbeats are sent in 
> poll(), we cannot guarantee that the heartbeats will actually be sent at the 
> configured frequency. In practice, the user may have to adjust their fetch 
> size to ensure that poll() is called often enough to get the desired 
> heartbeat frequency. For most users, the consumption rate is probably fast 
> enough for this not to matter, but we should make the documentation clear on 
> this point. In any case, we expect that most users will accept the default 
> value.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2351) Brokers are having a problem shutting down correctly

[Mayuresh Gharat (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658749#comment-14658749
 ] 

Mayuresh Gharat commented on KAFKA-2351:


[~sslavic] I think this is different. Thanks for reporting this. Let me see if 
I can address this in the current patch. 

> Brokers are having a problem shutting down correctly
> 
>
> Key: KAFKA-2351
> URL: https://issues.apache.org/jira/browse/KAFKA-2351
> Project: Kafka
>  Issue Type: Bug
>Reporter: Mayuresh Gharat
>Assignee: Mayuresh Gharat
> Attachments: KAFKA-2351.patch, KAFKA-2351_2015-07-21_14:58:13.patch, 
> KAFKA-2351_2015-07-23_21:36:52.patch
>
>
> The run() in Acceptor during shutdown might throw an exception that is not 
> caught and it never reaches shutdownComplete due to which the latch is not 
> counted down and the broker will not be able to shutdown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2401: fix transient failure in ProducerS...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/113


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-2401) Fix transient failure of ProducerSendTest.testCloseWithZeroTimeoutFromSenderThread()

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-2401.
--
   Resolution: Fixed
Fix Version/s: 0.8.3

Issue resolved by pull request 113
[https://github.com/apache/kafka/pull/113]

> Fix transient failure of 
> ProducerSendTest.testCloseWithZeroTimeoutFromSenderThread()
> 
>
> Key: KAFKA-2401
> URL: https://issues.apache.org/jira/browse/KAFKA-2401
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.8.3
>
>
> The transient failure can happen because of a race condition of the callback 
> firing order for messages produced to broker 0 and broker 1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2401) Fix transient failure of ProducerSendTest.testCloseWithZeroTimeoutFromSenderThread()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658755#comment-14658755
 ] 

ASF GitHub Bot commented on KAFKA-2401:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/113


> Fix transient failure of 
> ProducerSendTest.testCloseWithZeroTimeoutFromSenderThread()
> 
>
> Key: KAFKA-2401
> URL: https://issues.apache.org/jira/browse/KAFKA-2401
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.8.3
>
>
> The transient failure can happen because of a race condition of the callback 
> firing order for messages produced to broker 0 and broker 1.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658781#comment-14658781
 ] 

Jiangjie Qin commented on KAFKA-2406:
-

[~junrao], I'm still doing the test, here is what I see:
1. When controller was on old version, the controlled shutdown during rolling 
bounce seems to be normal.
2. After controller is running on new version, the controlled shutdown of 
brokers become very slow.
3. There are many zk paths still left in /isr_change_notification after the 
cluster bounce.
4. The first broker shuts down a little bit slower than before, but after that 
the subsequent shutdown takes super long - which is expected.

Because of [1] I was thinking maybe throttling UpdateMetadataRequest would be 
the minimum solution for now, but I have the same concern as you do on the 
number of zk writes and watcher fires.
[3] probably is an indication of zk watcher cannot catch up with the change 
reported by brokers.

Having broker side to batch the changes makes sense. I am thinking about doing 
the following:
1. Broker only update ISR change in a batch periodically by writing partitions 
data to /isr_change_notification/brokerId_IsrChangeEpoch path. so zkWrite is 
bounded to #broker/update_interval.
2. Instead of using zk watcher, controller simply periodically query zookeeper 
and propagate ISR changes.

> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2409) Have KafkaConsumer.committed() return null when there is no committed offset

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-2409:
--

 Summary: Have KafkaConsumer.committed() return null when there is 
no committed offset
 Key: KAFKA-2409
 URL: https://issues.apache.org/jira/browse/KAFKA-2409
 Project: Kafka
  Issue Type: Improvement
Reporter: Jason Gustafson
Priority: Minor


Currently checking whether an offset has been committed requires catching 
NoOffsetForPartitionException. Since this is likely a fairly common case, it is 
more convenient for users just to return null.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2409) Have KafkaConsumer.committed() return null when there is no committed offset

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-2409:
---
Issue Type: Sub-task  (was: Improvement)
Parent: KAFKA-2387

> Have KafkaConsumer.committed() return null when there is no committed offset
> 
>
> Key: KAFKA-2409
> URL: https://issues.apache.org/jira/browse/KAFKA-2409
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jason Gustafson
>Priority: Minor
>
> Currently checking whether an offset has been committed requires catching 
> NoOffsetForPartitionException. Since this is likely a fairly common case, it 
> is more convenient for users just to return null.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[Ashish K Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658909#comment-14658909
 ] 

Ashish K Singh commented on KAFKA-2406:
---

[~junrao], [~becket_qin] it sounds like we are delaying ISR change propagation 
to fix this issue. Quick intimation of ISR change is a good thing, I believe. 
However, what [~becket_qin] ran into is certainly an issue that needs to be 
fixed. I think the issue will surface only when there are lot of changes 
happening in system, like during rolling bounce. Such scenarios are usually 
started by admins. If we just specify a way to turn off or delay ISR 
notifications only during such operations, we will still react fast to ISR 
changes and won;t hit this issue during rolling bounce like scenarios. Thoughts?

> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658922#comment-14658922
 ] 

Jun Rao commented on KAFKA-2406:


[~becket_qin], are you suggesting not using sequential ZK node, but just having 
an isrChangePath per broker and keeping updating it? Not sure how this 
coordinates with the controller.

Also, not sure having the controller periodically read from the IsrChange path 
is better. After the cluster is stable, there shouldn't be any ISR change and 
the periodic ZK read is just pure overhead. If we batch enough, there shouldn't 
be that many ZK watchers fired. Another optimization that we could do is that 
on controller failover, the controller can actually delete all existing 
sequential ZK nodes in isr_change before initialization. The initialization 
will read the latest leader and ISR for all partitions from ZK anyway.

> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658929#comment-14658929
 ] 

Jiangjie Qin commented on KAFKA-2406:
-

[~ashishujjain], the large number of ISR change might not only occur during 
Admin operations, for example, any long GC might cause this issue. Also when a 
broker goes down if zk session timeout is longer than the 
replica.lag.time.max.ms will also cause problem.

I kind of think it is OK to delay the ISR change propagation. From user point 
of view, the ISR from two brokers still can be different even with the current 
approach considering we have a propagation delay.


> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 36548: Patch for KAFKA-2336

[Grant Henke]

> On July 17, 2015, 4:26 a.m., Jiangjie Qin wrote:
> > Looks good to me.
> 
> Jiangjie Qin wrote:
> Actually do we need to talk to Zookeeper every time? Can we read the data 
> from topic metadata cache directly?
> 
> Gwen Shapira wrote:
> Good point, Jiangjie - looks like partitionFor is called on every 
> ConsumerMetadataRequest handling, so some kind of caching will be nice.
> 
> Grant Henke wrote:
> We only talk to Zookeeper once at instance creation via the call to 
> `getOffsetsTopicPartitionCount` and setting the val 
> `offsetsTopicPartitionCount`. The static value is used from then on for every 
> call to `partitionFor`.
> 
> Jiangjie Qin wrote:
> Ah, I see, my bad. Then this patch seems not completely solve the issue, 
> though. Let's say offsets topic is not exist yet. What if two brokers had 
> different offset topic partition number configuration? After they startup and 
> before the offset topic get created in zookeeper, they will have different 
> value for offsetsTopicPartitionCount. Will that cause problem silently?
> 
> Grant Henke wrote:
> I think that sort of issue exists for many of Kafka's configurations, and 
> exists for this configuration without this patch too. I do not aim to solve 
> non-uniform configuration in this patch.
> 
> Jiangjie Qin wrote:
> I agree that configuration mismatch could cause issue for other 
> configurations as well. But having existing problems does not mean we should 
> introduce one more to them. Besides, is it a simpler solution to just read 
> from topic metadata cache?
> 
> Gwen Shapira wrote:
> Was there a resolution here?
> 
> Grant, will reading number of partitions from topic metadata cache solve 
> the edge case that Becket raised?

This is something that I have been thinking about for a little while. I think 
reading from topic metadata would help the scenario Becket raised, though it is 
a very rare edge case and would only occur when both the offset topic does not 
exist yet and the brokers do not have uniform configurations.

I can make the change to use the cache, but the thing is I will either need to 
pass it via the constructors down through ConsumerCoordinator and OffsetManager 
from KafkaApis or through the partitionsFor calls in each. And though its 
sounds mroe simple to use a cache...chaches are never simple. These are the 
trade offs I have been considering.


- Grant


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36548/#review92020
---


On July 16, 2015, 6:04 p.m., Grant Henke wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36548/
> ---
> 
> (Updated July 16, 2015, 6:04 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2336
> https://issues.apache.org/jira/browse/KAFKA-2336
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Fix Scala style
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/server/OffsetManager.scala 
> 47b6ce93da320a565435b4a7916a0c4371143b8a 
> 
> Diff: https://reviews.apache.org/r/36548/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Grant Henke
> 
>

[jira] [Commented] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[Ashish K Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658944#comment-14658944
 ] 

Ashish K Singh commented on KAFKA-2406:
---

Makes sense. Can I suggest that we still keep the batching window duration 
configurable?

> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658968#comment-14658968
 ] 

Jiangjie Qin commented on KAFKA-2406:
-

[~junrao], I was thinking to have a sequential path for each broker before, but 
it probably does not make sense. So a global sequential zk path should be good 
enough.

And I agree that a periodically checking thread might be a bad idea. The main 
concern I have is that what if there are many brokers and they report isr 
change almost at same time, will the watcher be fired for that many times? Will 
that cause problem on controller?

Deleting the zk path on controller migration makes sense and I'm actually 
manually doing it now. The only issue I see here is that when I was trying to 
delete the zk path with > 11 children paths, it takes quite a while. Not 
sure if that will cause issue.

> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658968#comment-14658968
 ] 

Jiangjie Qin edited comment on KAFKA-2406 at 8/5/15 9:24 PM:
-

[~junrao], I was thinking to have a sequential path for each broker before, but 
it probably does not make sense. So a global sequential zk path should be good 
enough.

And I agree that a periodically checking thread might be a bad idea. The main 
concern I have is that what if there are many brokers and they report isr 
change almost at same time, will the watcher be fired for that many times? Will 
that cause problem on controller?

Deleting the zk path on controller migration makes sense and I'm actually 
manually doing it now. The only issue I see here is that when I was trying to 
delete the zk path with > 11 children paths, it takes quite a while. Not 
sure if that will cause issue. But if we throttle the zk path creation, it 
should reduce the paths to be deleted also.


was (Author: becket_qin):
[~junrao], I was thinking to have a sequential path for each broker before, but 
it probably does not make sense. So a global sequential zk path should be good 
enough.

And I agree that a periodically checking thread might be a bad idea. The main 
concern I have is that what if there are many brokers and they report isr 
change almost at same time, will the watcher be fired for that many times? Will 
that cause problem on controller?

Deleting the zk path on controller migration makes sense and I'm actually 
manually doing it now. The only issue I see here is that when I was trying to 
delete the zk path with > 11 children paths, it takes quite a while. Not 
sure if that will cause issue.

> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14658974#comment-14658974
 ] 

Jiangjie Qin commented on KAFKA-2406:
-

I just tried the WIP patch, no luck. I'll try out the solution we just 
discussed and submit patch later today.

> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Issue Comment Deleted] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-2406:

Comment: was deleted

(was: I just tried the WIP patch, no luck. I'll try out the solution we just 
discussed and submit patch later today.)

> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659007#comment-14659007
 ] 

Jun Rao commented on KAFKA-2406:


[~singhashish], yes, ideally we want to propagate isr asap. However, this is 
not critical for the current known use cases. The first use case is for admin 
tools like describing a topic. The second potential use case is for read 
affinity in the consumer, which is just an optimization. For both cases, 
delaying the propagation of isr is ok. Also, isr typically only changes during 
rolling bounces. If you turn off and delay the propagation then, most of the 
isr changes will be delayed anyway.

We can probably have a configurable batching window on the broker side.

> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2393: Correctly Handle InvalidTopicExcep...

[granthenke]

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/117

KAFKA-2393: Correctly Handle InvalidTopicException in KafkaApis.getTo…

…picMetadata()

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka invalid-topic

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/117.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #117


commit 0abda5fffe7cb7cda585941e4909be304ad011f6
Author: Grant Henke 
Date:   2015-08-05T21:45:25Z

KAFKA-2393: Correctly Handle InvalidTopicException in 
KafkaApis.getTopicMetadata()




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2393) Correctly Handle InvalidTopicException in KafkaApis.getTopicMetadata()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659017#comment-14659017
 ] 

ASF GitHub Bot commented on KAFKA-2393:
---

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/117

KAFKA-2393: Correctly Handle InvalidTopicException in KafkaApis.getTo…

…picMetadata()

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka invalid-topic

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/117.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #117


commit 0abda5fffe7cb7cda585941e4909be304ad011f6
Author: Grant Henke 
Date:   2015-08-05T21:45:25Z

KAFKA-2393: Correctly Handle InvalidTopicException in 
KafkaApis.getTopicMetadata()




> Correctly Handle InvalidTopicException in KafkaApis.getTopicMetadata()
> --
>
> Key: KAFKA-2393
> URL: https://issues.apache.org/jira/browse/KAFKA-2393
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Grant Henke
>Assignee: Grant Henke
>
> It seems that in KafkaApis.getTopicMetadata(), we need to handle 
> InvalidTopicException explicitly when calling AdminUtils.createTopic (by 
> returning the corresponding error code for that topic). Otherwise, we may not 
> be able to get the metadata for other valid topics. This seems to be an 
> existing problem, but KAFKA-2337 makes InvalidTopicException more likely to 
> happen. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2393) Correctly Handle InvalidTopicException in KafkaApis.getTopicMetadata()

[Grant Henke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Henke updated KAFKA-2393:
---
Fix Version/s: 0.8.3
   Status: Patch Available  (was: Open)

> Correctly Handle InvalidTopicException in KafkaApis.getTopicMetadata()
> --
>
> Key: KAFKA-2393
> URL: https://issues.apache.org/jira/browse/KAFKA-2393
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.8.3
>
>
> It seems that in KafkaApis.getTopicMetadata(), we need to handle 
> InvalidTopicException explicitly when calling AdminUtils.createTopic (by 
> returning the corresponding error code for that topic). Otherwise, we may not 
> be able to get the metadata for other valid topics. This seems to be an 
> existing problem, but KAFKA-2337 makes InvalidTopicException more likely to 
> happen. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659039#comment-14659039
 ] 

Jun Rao commented on KAFKA-2406:


[~becket_qin], the ZK watcher is a one-time watcher. So, when a watcher is 
triggered, you have to do another read to register the watcher. There could be 
multiple changes in between, but only one more watcher will be triggered. 
However, the brokers may not be writing to ZK at exactly the same time. So, the 
more brokers you have, likely the more times the watcher will be fired. I think 
if we make the batch window configurable, one can adjust this value based on 
the cluster size if needed.

Writes are more expensive than reads in ZK. By batching, we can significantly 
reduce the number of ZK writes for isr changes. The controller still needs to 
read the changed isr from ZK for all affected partitions though. Hopefully 
there's not the main bottleneck.

> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client

[Ashish K Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659065#comment-14659065
 ] 

Ashish K Singh commented on KAFKA-1690:
---

[~harsha_ch] sure, no issues. Also, do we have any JIRA to track inter-broker 
communication over SSL?

> new java producer needs ssl support as a client
> ---
>
> Key: KAFKA-1690
> URL: https://issues.apache.org/jira/browse/KAFKA-1690
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Joe Stein
>Assignee: Sriharsha Chintalapani
> Fix For: 0.8.3
>
> Attachments: KAFKA-1690.patch, KAFKA-1690.patch, 
> KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, 
> KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, 
> KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, 
> KAFKA-1690_2015-05-21_10:37:08.patch, KAFKA-1690_2015-06-03_18:52:29.patch, 
> KAFKA-1690_2015-06-23_13:18:20.patch, KAFKA-1690_2015-07-20_06:10:42.patch, 
> KAFKA-1690_2015-07-20_11:59:57.patch, KAFKA-1690_2015-07-25_12:10:55.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1229) Reload broker config without a restart

[Aditya Auradkar (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659071#comment-14659071
 ] 

Aditya Auradkar commented on KAFKA-1229:


Hey [~vamsi360],

We recently did a pretty detailed discussion on this: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-21+-+Dynamic+Configuration
At that time, we decided not to support reloading broker config via SIGHUP.

Aditya

> Reload broker config without a restart
> --
>
> Key: KAFKA-1229
> URL: https://issues.apache.org/jira/browse/KAFKA-1229
> Project: Kafka
>  Issue Type: Wish
>  Components: config
>Affects Versions: 0.8.0
>Reporter: Carlo Cabanilla
>Priority: Minor
>
> In order to minimize client disruption, ideally you'd be able to reload 
> broker config without having to restart the server. On *nix system the 
> convention is to have the process reread its configuration if it receives a 
> SIGHUP signal.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2410) Implement "Auto Topic Creation" client side and remove support from Broker side

[Grant Henke (JIRA)]

Grant Henke created KAFKA-2410:
--

 Summary: Implement "Auto Topic Creation" client side and remove 
support from Broker side
 Key: KAFKA-2410
 URL: https://issues.apache.org/jira/browse/KAFKA-2410
 Project: Kafka
  Issue Type: Improvement
  Components: clients, core
Affects Versions: 0.8.2.1
Reporter: Grant Henke
Assignee: Grant Henke


Auto topic creation on the broker has caused pain in the past; And today it 
still causes unusual error handling requirements on the client side, added 
complexity in the broker, mixed responsibility of the TopicMetadataRequest, and 
limits configuration of the option to be cluster wide. In the future having it 
broker side will also make features such as authorization very difficult. 

There have been discussions in the past of implementing this feature client 
side. 
[example|https://issues.apache.org/jira/browse/KAFKA-689?focusedCommentId=13548746&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13548746]

This Jira is to track that discussion and implementation once the necessary 
protocol support exists: KAFKA-2229



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 28096: Patch for KAFKA-313

[Ashish Singh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28096/
---

(Updated Aug. 5, 2015, 10:37 p.m.)


Review request for kafka, Gwen Shapira, Jarek Cecho, and Joel Koshy.


Bugs: KAFKA-313
https://issues.apache.org/jira/browse/KAFKA-313


Repository: kafka


Description
---

KAFKA-313: Add JSON/CSV output and looping options to ConsumerGroupCommand


Diffs (updated)
-

  config/server.properties 80ee2fc6e94a114e7710ae4df3f4e2b83e06f080 
  core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala 
f23120ede5f9bf0cfaf795c65c9845f42d8784d0 

Diff: https://reviews.apache.org/r/28096/diff/


Testing
---

Ran ConsumerOffsetChecker with different combinations of --output.format and 
--loop options.


Thanks,

Ashish Singh

Re: Review Request 28096: Patch for KAFKA-313

[Ashish Singh]

> On July 30, 2015, 2:11 p.m., Ismael Juma wrote:
> > core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala, line 45
> > 
> >
> > Do we really need this var or can this be passed as a parameter to the 
> > relevant methods? Avoiding mutable state is good if possible.

It is initialized here with a default value, however gets set just once. 
Moreover, it represents outputFormat for the whole class, I think it is better 
to have its scope for the whole class. Let me know if this still does not make 
sense.


> On July 30, 2015, 2:11 p.m., Ismael Juma wrote:
> > core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala, line 93
> > 
> >
> > In adddition to Gwen's comment, `breakable` is a last resort construct 
> > in Scala. So, if you can avoid using it, it's better.

Removed. Was left over from previous tries.


> On July 30, 2015, 2:11 p.m., Ismael Juma wrote:
> > core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala, line 112
> > 
> >
> > Simpler (and avoid discouraged `return` in Scala):
> > 
> > `numIterations <= 0 || currentIteration < numIterations`

The code has changed a bit, so not relevant anymore.


> On July 30, 2015, 2:11 p.m., Ismael Juma wrote:
> > core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala, line 240
> > 
> >
> > Something along the following seems more readable and concise:
> > 
> > `println(Seq("GROUP", "TOPIC", ...).mkString(", "))`

Good suggestion, thanks!


- Ashish


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28096/#review93559
---


On Aug. 5, 2015, 10:37 p.m., Ashish Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28096/
> ---
> 
> (Updated Aug. 5, 2015, 10:37 p.m.)
> 
> 
> Review request for kafka, Gwen Shapira, Jarek Cecho, and Joel Koshy.
> 
> 
> Bugs: KAFKA-313
> https://issues.apache.org/jira/browse/KAFKA-313
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-313: Add JSON/CSV output and looping options to ConsumerGroupCommand
> 
> 
> Diffs
> -
> 
>   config/server.properties 80ee2fc6e94a114e7710ae4df3f4e2b83e06f080 
>   core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala 
> f23120ede5f9bf0cfaf795c65c9845f42d8784d0 
> 
> Diff: https://reviews.apache.org/r/28096/diff/
> 
> 
> Testing
> ---
> 
> Ran ConsumerOffsetChecker with different combinations of --output.format and 
> --loop options.
> 
> 
> Thanks,
> 
> Ashish Singh
> 
>

[jira] [Commented] (KAFKA-313) Add JSON/CSV output and looping options to ConsumerGroupCommand

[Ashish K Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659105#comment-14659105
 ] 

Ashish K Singh commented on KAFKA-313:
--

Updated reviewboard https://reviews.apache.org/r/28096/
 against branch trunk

> Add JSON/CSV output and looping options to ConsumerGroupCommand
> ---
>
> Key: KAFKA-313
> URL: https://issues.apache.org/jira/browse/KAFKA-313
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Dave DeMaagd
>Assignee: Ashish K Singh
>Priority: Minor
>  Labels: newbie, patch
> Fix For: 0.8.3
>
> Attachments: KAFKA-313-2012032200.diff, KAFKA-313.1.patch, 
> KAFKA-313.patch, KAFKA-313_2015-02-23_18:11:32.patch, 
> KAFKA-313_2015-06-24_11:14:24.patch, KAFKA-313_2015-08-05_15:37:32.patch
>
>
> Adds:
> * '--loop N' - causes the program to loop forever, sleeping for up to N 
> seconds between loops (loop time minus collection time, unless that's less 
> than 0, at which point it will just run again immediately)
> * '--asjson' - display as a JSON string instead of the more human readable 
> output format.
> Neither of the above  depend on each other (you can loop in the human 
> readable output, or do a single shot execution with JSON output).  Existing 
> behavior/output maintained if neither of the above are used.  Diff Attached.
> Impacted files:
> core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 28096: Patch for KAFKA-313

[Ashish Singh]

> On July 29, 2015, 6:35 p.m., Gwen Shapira wrote:
> > core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala, lines 36-39
> > 
> >
> > Shouldn't this be an inner object? since its only visible and used by 
> > ConsumerGroupCommand?

Makes sense :)


> On July 29, 2015, 6:35 p.m., Gwen Shapira wrote:
> > core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala, line 93
> > 
> >
> > This is defined as "breakable", but I don't see where you use break?

There was one, which I removed and forgot to get rid of this. Thanks for 
catching this.


> On July 29, 2015, 6:35 p.m., Gwen Shapira wrote:
> > core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala, lines 109-113
> > 
> >
> > Since its a boolean, we want a name that reflects what we check. 
> > Perhaps "iterationsLeft"? 
> > 
> > it looks like we want to return true if we want to continue iterating - 
> > in this case shouldn't a negative numIterations lead to false?

Changed the func name.

For -ve numIterations, I was thinking that if user has put some invalid value 
then probably we should skip the numIterations check. However, I realize that a 
user intentionally might want to have a -ve value for some reason that I can 
not think of. So changed.


> On July 29, 2015, 6:35 p.m., Gwen Shapira wrote:
> > core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala, lines 237-242
> > 
> >
> > These look identical - copy/paste error?

Not really. There is some difference, None has "%s, %s" format, while CSV has 
"%s,%s" format.


> On July 29, 2015, 6:35 p.m., Gwen Shapira wrote:
> > core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala, line 250
> > 
> >
> > I thought none is tab-delimited, not CSV?

My idea was that I will keep None correspond to exisiting format, which is "%s, 
%s".


> On July 29, 2015, 6:35 p.m., Gwen Shapira wrote:
> > core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala, line 389
> > 
> >
> > If only CSV and JSON are allowed, what is NONE for?

If outputFormat is not specified, it will maintain the existing output format, 
that is what None is for. Makes sense?


> On July 29, 2015, 6:35 p.m., Gwen Shapira wrote:
> > core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala, line 364
> > 
> >
> > Kafka code base usually doesn't use methods as operators. Why are we 
> > doing this here?
> > 
> > Also, why define "ofTypes" here?

Removed ofTypes.


- Ashish


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28096/#review93489
---


On Aug. 5, 2015, 10:37 p.m., Ashish Singh wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28096/
> ---
> 
> (Updated Aug. 5, 2015, 10:37 p.m.)
> 
> 
> Review request for kafka, Gwen Shapira, Jarek Cecho, and Joel Koshy.
> 
> 
> Bugs: KAFKA-313
> https://issues.apache.org/jira/browse/KAFKA-313
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> KAFKA-313: Add JSON/CSV output and looping options to ConsumerGroupCommand
> 
> 
> Diffs
> -
> 
>   config/server.properties 80ee2fc6e94a114e7710ae4df3f4e2b83e06f080 
>   core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala 
> f23120ede5f9bf0cfaf795c65c9845f42d8784d0 
> 
> Diff: https://reviews.apache.org/r/28096/diff/
> 
> 
> Testing
> ---
> 
> Ran ConsumerOffsetChecker with different combinations of --output.format and 
> --loop options.
> 
> 
> Thanks,
> 
> Ashish Singh
> 
>

[jira] [Updated] (KAFKA-313) Add JSON/CSV output and looping options to ConsumerGroupCommand

[Ashish K Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish K Singh updated KAFKA-313:
-
Status: Patch Available  (was: In Progress)

> Add JSON/CSV output and looping options to ConsumerGroupCommand
> ---
>
> Key: KAFKA-313
> URL: https://issues.apache.org/jira/browse/KAFKA-313
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Dave DeMaagd
>Assignee: Ashish K Singh
>Priority: Minor
>  Labels: newbie, patch
> Fix For: 0.8.3
>
> Attachments: KAFKA-313-2012032200.diff, KAFKA-313.1.patch, 
> KAFKA-313.patch, KAFKA-313_2015-02-23_18:11:32.patch, 
> KAFKA-313_2015-06-24_11:14:24.patch, KAFKA-313_2015-08-05_15:37:32.patch
>
>
> Adds:
> * '--loop N' - causes the program to loop forever, sleeping for up to N 
> seconds between loops (loop time minus collection time, unless that's less 
> than 0, at which point it will just run again immediately)
> * '--asjson' - display as a JSON string instead of the more human readable 
> output format.
> Neither of the above  depend on each other (you can loop in the human 
> readable output, or do a single shot execution with JSON output).  Existing 
> behavior/output maintained if neither of the above are used.  Diff Attached.
> Impacted files:
> core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-313) Add JSON/CSV output and looping options to ConsumerGroupCommand

[Ashish K Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish K Singh updated KAFKA-313:
-
Attachment: KAFKA-313_2015-08-05_15:37:32.patch

> Add JSON/CSV output and looping options to ConsumerGroupCommand
> ---
>
> Key: KAFKA-313
> URL: https://issues.apache.org/jira/browse/KAFKA-313
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Dave DeMaagd
>Assignee: Ashish K Singh
>Priority: Minor
>  Labels: newbie, patch
> Fix For: 0.8.3
>
> Attachments: KAFKA-313-2012032200.diff, KAFKA-313.1.patch, 
> KAFKA-313.patch, KAFKA-313_2015-02-23_18:11:32.patch, 
> KAFKA-313_2015-06-24_11:14:24.patch, KAFKA-313_2015-08-05_15:37:32.patch
>
>
> Adds:
> * '--loop N' - causes the program to loop forever, sleeping for up to N 
> seconds between loops (loop time minus collection time, unless that's less 
> than 0, at which point it will just run again immediately)
> * '--asjson' - display as a JSON string instead of the more human readable 
> output format.
> Neither of the above  depend on each other (you can loop in the human 
> readable output, or do a single shot execution with JSON output).  Existing 
> behavior/output maintained if neither of the above are used.  Diff Attached.
> Impacted files:
> core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 28096: Patch for KAFKA-313

[Ashish Singh]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28096/
---

(Updated Aug. 5, 2015, 10:43 p.m.)


Review request for kafka, Gwen Shapira, Jarek Cecho, and Joel Koshy.


Bugs: KAFKA-313
https://issues.apache.org/jira/browse/KAFKA-313


Repository: kafka


Description
---

KAFKA-313: Add JSON/CSV output and looping options to ConsumerGroupCommand


Diffs (updated)
-

  core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala 
f23120ede5f9bf0cfaf795c65c9845f42d8784d0 

Diff: https://reviews.apache.org/r/28096/diff/


Testing
---

Ran ConsumerOffsetChecker with different combinations of --output.format and 
--loop options.


Thanks,

Ashish Singh

[jira] [Updated] (KAFKA-313) Add JSON/CSV output and looping options to ConsumerGroupCommand

[Ashish K Singh (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ashish K Singh updated KAFKA-313:
-
Attachment: KAFKA-313_2015-08-05_15:43:00.patch

> Add JSON/CSV output and looping options to ConsumerGroupCommand
> ---
>
> Key: KAFKA-313
> URL: https://issues.apache.org/jira/browse/KAFKA-313
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Dave DeMaagd
>Assignee: Ashish K Singh
>Priority: Minor
>  Labels: newbie, patch
> Fix For: 0.8.3
>
> Attachments: KAFKA-313-2012032200.diff, KAFKA-313.1.patch, 
> KAFKA-313.patch, KAFKA-313_2015-02-23_18:11:32.patch, 
> KAFKA-313_2015-06-24_11:14:24.patch, KAFKA-313_2015-08-05_15:37:32.patch, 
> KAFKA-313_2015-08-05_15:43:00.patch
>
>
> Adds:
> * '--loop N' - causes the program to loop forever, sleeping for up to N 
> seconds between loops (loop time minus collection time, unless that's less 
> than 0, at which point it will just run again immediately)
> * '--asjson' - display as a JSON string instead of the more human readable 
> output format.
> Neither of the above  depend on each other (you can loop in the human 
> readable output, or do a single shot execution with JSON output).  Existing 
> behavior/output maintained if neither of the above are used.  Diff Attached.
> Impacted files:
> core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-313) Add JSON/CSV output and looping options to ConsumerGroupCommand

[Ashish K Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659110#comment-14659110
 ] 

Ashish K Singh commented on KAFKA-313:
--

Updated reviewboard https://reviews.apache.org/r/28096/
 against branch trunk

> Add JSON/CSV output and looping options to ConsumerGroupCommand
> ---
>
> Key: KAFKA-313
> URL: https://issues.apache.org/jira/browse/KAFKA-313
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Dave DeMaagd
>Assignee: Ashish K Singh
>Priority: Minor
>  Labels: newbie, patch
> Fix For: 0.8.3
>
> Attachments: KAFKA-313-2012032200.diff, KAFKA-313.1.patch, 
> KAFKA-313.patch, KAFKA-313_2015-02-23_18:11:32.patch, 
> KAFKA-313_2015-06-24_11:14:24.patch, KAFKA-313_2015-08-05_15:37:32.patch, 
> KAFKA-313_2015-08-05_15:43:00.patch
>
>
> Adds:
> * '--loop N' - causes the program to loop forever, sleeping for up to N 
> seconds between loops (loop time minus collection time, unless that's less 
> than 0, at which point it will just run again immediately)
> * '--asjson' - display as a JSON string instead of the more human readable 
> output format.
> Neither of the above  depend on each other (you can loop in the human 
> readable output, or do a single shot execution with JSON output).  Existing 
> behavior/output maintained if neither of the above are used.  Diff Attached.
> Impacted files:
> core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2393: Correctly Handle InvalidTopicExcep...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/117


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2393) Correctly Handle InvalidTopicException in KafkaApis.getTopicMetadata()

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659134#comment-14659134
 ] 

ASF GitHub Bot commented on KAFKA-2393:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/117


> Correctly Handle InvalidTopicException in KafkaApis.getTopicMetadata()
> --
>
> Key: KAFKA-2393
> URL: https://issues.apache.org/jira/browse/KAFKA-2393
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.8.3
>
>
> It seems that in KafkaApis.getTopicMetadata(), we need to handle 
> InvalidTopicException explicitly when calling AdminUtils.createTopic (by 
> returning the corresponding error code for that topic). Otherwise, we may not 
> be able to get the metadata for other valid topics. This seems to be an 
> existing problem, but KAFKA-2337 makes InvalidTopicException more likely to 
> happen. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2393) Correctly Handle InvalidTopicException in KafkaApis.getTopicMetadata()

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-2393:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Thanks for the patch. Committed to trunk.

> Correctly Handle InvalidTopicException in KafkaApis.getTopicMetadata()
> --
>
> Key: KAFKA-2393
> URL: https://issues.apache.org/jira/browse/KAFKA-2393
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.8.3
>
>
> It seems that in KafkaApis.getTopicMetadata(), we need to handle 
> InvalidTopicException explicitly when calling AdminUtils.createTopic (by 
> returning the corresponding error code for that topic). Otherwise, we may not 
> be able to get the metadata for other valid topics. This seems to be an 
> existing problem, but KAFKA-2337 makes InvalidTopicException more likely to 
> happen. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659147#comment-14659147
 ] 

Jiangjie Qin commented on KAFKA-2406:
-

[~ashishujjain] [~junrao], sure, we can make the batching window size 
configurable. The related question is that, If we are adding a configuration, 
do you think we need a KIP? I think this is a pretty small change, but since we 
think of config change as API change, so just want to confirm.

> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2397) leave group request

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659158#comment-14659158
 ] 

Guozhang Wang commented on KAFKA-2397:
--

I once considered letting KafkaApis to handle connection closure when I was 
working on purgatory re-design, to purge the requests as mentioned by Jar. The 
difficulties are that at the socket server the connection is not logically tied 
to a client (although in fact it is), while for handling a requests / client 
failure events we need to pass-in a client-id into the API layer. A lot has 
changed in SocketServer since then so I do not know if things have changed so 
that we can infer the client-id (or more specifically consumer-id in this case) 
from SocketServer. 

> leave group request
> ---
>
> Key: KAFKA-2397
> URL: https://issues.apache.org/jira/browse/KAFKA-2397
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Onur Karaman
>Assignee: Onur Karaman
>Priority: Minor
> Fix For: 0.8.3
>
>
> Let's say every consumer in a group has session timeout s. Currently, if a 
> consumer leaves the group, the worst case time to stabilize the group is 2s 
> (s to detect the consumer failure + s for the rebalance window). If a 
> consumer instead can declare they are leaving the group, the worst case time 
> to stabilize the group would just be the s associated with the rebalance 
> window.
> This is a low priority optimization!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: Kafka-trunk #572

[Apache Jenkins Server]

See 

Changes:

[junrao] KAFKA-2393: Correctly Handle InvalidTopicException in 
KafkaApis.getTo���

--
[...truncated 1517 lines...]
kafka.api.test.ProducerCompressionTest > testCompression[0] PASSED

kafka.api.test.ProducerCompressionTest > testCompression[1] PASSED

kafka.api.test.ProducerCompressionTest > testCompression[2] PASSED

kafka.api.test.ProducerCompressionTest > testCompression[3] PASSED

kafka.cluster.BrokerEndPointTest > testEndpointFromURI PASSED

kafka.cluster.BrokerEndPointTest > testSerDe PASSED

kafka.cluster.BrokerEndPointTest > testHashAndEquals PASSED

kafka.cluster.BrokerEndPointTest > testFromOldJSON PASSED

kafka.cluster.BrokerEndPointTest > testFromJSON PASSED

kafka.cluster.BrokerEndPointTest > testBrokerEndpointFromURI PASSED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression PASSED

kafka.integration.UncleanLeaderElectionTest > 
testCleanLeaderElectionDisabledByTopicOverride PASSED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionInvalidTopicOverride PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionEnabled 
PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionDisabled 
PASSED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionEnabledByTopicOverride PASSED

kafka.integration.FetcherTest > testFetcher PASSED

kafka.integration.RollingBounceTest > testRollingBounce PASSED

kafka.integration.MinIsrConfigTest > testDefaultKafkaConfig PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooHigh 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow PASSED

kafka.integration.TopicMetadataTest > testIsrAfterBrokerShutDownAndJoinsBack 
PASSED

kafka.integration.TopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.TopicMetadataTest > testAliveBrokerListWithNoTopics PASSED

kafka.integration.TopicMetadataTest > testTopicMetadataRequest PASSED

kafka.integration.TopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.TopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.TopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.TopicMetadataTest > testAutoCreateTopicWithCollision PASSED

kafka.integration.TopicMetadataTest > testAutoCreateTopic PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.utils.UtilsTest > testReadInt PASSED

kafka.utils.UtilsTest > testCsvList PASSED

kafka.utils.UtilsTest > testCircularIterator PASSED

kafka.utils.UtilsTest > testAbs PASSED

kafka.utils.UtilsTest > testReplaceSuffix PASSED

kafka.utils.UtilsTest > testInLock PASSED

kafka.utils.UtilsTest > testDoublyLinkedList PASSED

kafka.utils.UtilsTest > testCsvMap PASSED

kafka.utils.UtilsTest > testReadBytes PASSED

kafka.utils.UtilsTest > testSwallow PASSED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.utils.SchedulerTest > testRestart PASSED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest > testNonPeriodicTask PASSED

kafka.utils.ByteBoundedBlockingQueueTest > testByteBoundedBlockingQueue PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg PASSED

kafka.utils.CommandLineUtilsTest > testParseSingleArg PASSED

kafka.utils.CommandLineUtilsTest > testParseArgs PASSED

kafka.utils.IteratorTemplateTest >

[jira] [Created] (KAFKA-2411) remove usage of BlockingChannel in the broker

[Jun Rao (JIRA)]

Jun Rao created KAFKA-2411:
--

 Summary: remove usage of BlockingChannel in the broker
 Key: KAFKA-2411
 URL: https://issues.apache.org/jira/browse/KAFKA-2411
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jun Rao


In KAFKA-1690, we are adding the SSL support at Selector. However, there are 
still a few places where we use BlockingChannel for inter-broker communication. 
We need to replace those usage with Selector/NetworkClient to enable 
inter-broker communication over SSL. Specially, BlockingChannel is currently 
used in the following places.
1. ControllerChannelManager: for the controller to propagate metadata to the 
brokers.
2. KafkaServer: for the broker to send controlled shutdown request to the 
controller.
3. AbstractFetcherThread: for the follower to fetch data from the leader 
(through SimpleConsumer).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659174#comment-14659174
 ] 

Jun Rao commented on KAFKA-1690:


[~singhashish], thanks for the reminder. File KAFKA-2411 to track this.

> new java producer needs ssl support as a client
> ---
>
> Key: KAFKA-1690
> URL: https://issues.apache.org/jira/browse/KAFKA-1690
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Joe Stein
>Assignee: Sriharsha Chintalapani
> Fix For: 0.8.3
>
> Attachments: KAFKA-1690.patch, KAFKA-1690.patch, 
> KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, 
> KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, 
> KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, 
> KAFKA-1690_2015-05-21_10:37:08.patch, KAFKA-1690_2015-06-03_18:52:29.patch, 
> KAFKA-1690_2015-06-23_13:18:20.patch, KAFKA-1690_2015-07-20_06:10:42.patch, 
> KAFKA-1690_2015-07-20_11:59:57.patch, KAFKA-1690_2015-07-25_12:10:55.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2411) remove usage of BlockingChannel in the broker

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659184#comment-14659184
 ] 

Jun Rao commented on KAFKA-2411:


For AbstractFetcherThread, we could potentially use NetworkClient. However, we 
need to be a bit careful since the follower fetcher thread doesn't need to 
refresh metadata itself. Instead, the information about the leader is 
propagated from the controller.

> remove usage of BlockingChannel in the broker
> -
>
> Key: KAFKA-2411
> URL: https://issues.apache.org/jira/browse/KAFKA-2411
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Jun Rao
>
> In KAFKA-1690, we are adding the SSL support at Selector. However, there are 
> still a few places where we use BlockingChannel for inter-broker 
> communication. We need to replace those usage with Selector/NetworkClient to 
> enable inter-broker communication over SSL. Specially, BlockingChannel is 
> currently used in the following places.
> 1. ControllerChannelManager: for the controller to propagate metadata to the 
> brokers.
> 2. KafkaServer: for the broker to send controlled shutdown request to the 
> controller.
> 3. AbstractFetcherThread: for the follower to fetch data from the leader 
> (through SimpleConsumer).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2397) leave group request

[Gwen Shapira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659185#comment-14659185
 ] 

Gwen Shapira commented on KAFKA-2397:
-

No, SocketServer may be aware of host:port of client, but not clientID

> leave group request
> ---
>
> Key: KAFKA-2397
> URL: https://issues.apache.org/jira/browse/KAFKA-2397
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Onur Karaman
>Assignee: Onur Karaman
>Priority: Minor
> Fix For: 0.8.3
>
>
> Let's say every consumer in a group has session timeout s. Currently, if a 
> consumer leaves the group, the worst case time to stabilize the group is 2s 
> (s to detect the consumer failure + s for the rebalance window). If a 
> consumer instead can declare they are leaving the group, the worst case time 
> to stabilize the group would just be the s associated with the rebalance 
> window.
> This is a low priority optimization!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1690) new java producer needs ssl support as a client

[Ashish K Singh (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659188#comment-14659188
 ] 

Ashish K Singh commented on KAFKA-1690:
---

Thanks [~junrao]!

> new java producer needs ssl support as a client
> ---
>
> Key: KAFKA-1690
> URL: https://issues.apache.org/jira/browse/KAFKA-1690
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Joe Stein
>Assignee: Sriharsha Chintalapani
> Fix For: 0.8.3
>
> Attachments: KAFKA-1690.patch, KAFKA-1690.patch, 
> KAFKA-1690_2015-05-10_23:20:30.patch, KAFKA-1690_2015-05-10_23:31:42.patch, 
> KAFKA-1690_2015-05-11_16:09:36.patch, KAFKA-1690_2015-05-12_16:20:08.patch, 
> KAFKA-1690_2015-05-15_07:18:21.patch, KAFKA-1690_2015-05-20_14:54:35.patch, 
> KAFKA-1690_2015-05-21_10:37:08.patch, KAFKA-1690_2015-06-03_18:52:29.patch, 
> KAFKA-1690_2015-06-23_13:18:20.patch, KAFKA-1690_2015-07-20_06:10:42.patch, 
> KAFKA-1690_2015-07-20_11:59:57.patch, KAFKA-1690_2015-07-25_12:10:55.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: ConsumerRecords are organized per topic...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/92


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: MINOR: auto.offset.reset docs not in sync with...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/91


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: MINOR: Fixed javadoc for committed return valu...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/89


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (KAFKA-2411) remove usage of BlockingChannel in the broker

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reassigned KAFKA-2411:
--

Assignee: Ismael Juma

> remove usage of BlockingChannel in the broker
> -
>
> Key: KAFKA-2411
> URL: https://issues.apache.org/jira/browse/KAFKA-2411
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Jun Rao
>Assignee: Ismael Juma
>
> In KAFKA-1690, we are adding the SSL support at Selector. However, there are 
> still a few places where we use BlockingChannel for inter-broker 
> communication. We need to replace those usage with Selector/NetworkClient to 
> enable inter-broker communication over SSL. Specially, BlockingChannel is 
> currently used in the following places.
> 1. ControllerChannelManager: for the controller to propagate metadata to the 
> brokers.
> 2. KafkaServer: for the broker to send controlled shutdown request to the 
> controller.
> 3. AbstractFetcherThread: for the follower to fetch data from the leader 
> (through SimpleConsumer).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR - Fix typo in ReplicaVerificationTool ou...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/101


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2406) ISR propagation should be throttled to avoid overwhelming controller.

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659217#comment-14659217
 ] 

Jun Rao commented on KAFKA-2406:


[~becket_qin], we can probably do a quick KIP on the config change so that 
people are aware of it. We probably want to test if this approach works first.

> ISR propagation should be throttled to avoid overwhelming controller.
> -
>
> Key: KAFKA-2406
> URL: https://issues.apache.org/jira/browse/KAFKA-2406
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
>Priority: Blocker
>
> This is a follow up patch for KAFKA-1367.
> We need to throttle the ISR propagation rate to avoid flooding in controller 
> to broker traffic. This might significantly increase time of controlled 
> shutdown or cluster startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 36652: Patch for KAFKA-2351

[Jun Rao]

> On July 24, 2015, 5:01 p.m., Mayuresh Gharat wrote:
> > core/src/main/scala/kafka/network/SocketServer.scala, line 264
> > 
> >
> > Hi Jun,
> > Cleaning up in finally is actually nice. I will make the necessary 
> > change and upload a new patch.
> > 
> > I was looking at the patch for :
> > https://issues.apache.org/jira/browse/KAFKA-2353
> > as per the suggestions on the jira ticket.
> > 
> > Was just curious if we can do the same there as well. We are catching 
> > all the Throwables and allowing the thread to continue processing. Is there 
> > something I am missing here.

Mayuresh,

Yes, saw the changes in KAFKA-2353. We can leave this the way that you did for 
now and revisit the catch throwable issue later.


- Jun


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36652/#review92928
---


On July 24, 2015, 4:36 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36652/
> ---
> 
> (Updated July 24, 2015, 4:36 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2351
> https://issues.apache.org/jira/browse/KAFKA-2351
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Added a try-catch to catch any exceptions thrown by the nioSelector
> 
> 
> Addressed comments on the Jira ticket
> 
> 
> Diffs
> -
> 
>   core/src/main/scala/kafka/network/SocketServer.scala 
> 91319fa010b140cca632e5fa8050509bd2295fc9 
> 
> Diff: https://reviews.apache.org/r/36652/diff/
> 
> 
> Testing
> ---
> 
> 
> Thanks,
> 
> Mayuresh Gharat
> 
>

[GitHub] kafka pull request: KAFKA-2390; Seek() should take a callback

[lindong28]

GitHub user lindong28 opened a pull request:

https://github.com/apache/kafka/pull/118

KAFKA-2390; Seek() should take a callback



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lindong28/kafka KAFKA-2390

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/118.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #118


commit 8a9f88649b8de52b48700821f0e1bc5c51a661f3
Author: Dong Lin 
Date:   2015-08-06T00:05:58Z

KAFKA-2390; Seek() should take a callback




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2351) Brokers are having a problem shutting down correctly

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659235#comment-14659235
 ] 

Jun Rao commented on KAFKA-2351:


[~mgharat], sorry the delay. Just replied on RB.

> Brokers are having a problem shutting down correctly
> 
>
> Key: KAFKA-2351
> URL: https://issues.apache.org/jira/browse/KAFKA-2351
> Project: Kafka
>  Issue Type: Bug
>Reporter: Mayuresh Gharat
>Assignee: Mayuresh Gharat
> Attachments: KAFKA-2351.patch, KAFKA-2351_2015-07-21_14:58:13.patch, 
> KAFKA-2351_2015-07-23_21:36:52.patch
>
>
> The run() in Acceptor during shutdown might throw an exception that is not 
> caught and it never reaches shutdownComplete due to which the latch is not 
> counted down and the broker will not be able to shutdown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2390) Seek() should take a callback.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14659236#comment-14659236
 ] 

ASF GitHub Bot commented on KAFKA-2390:
---

GitHub user lindong28 opened a pull request:

https://github.com/apache/kafka/pull/118

KAFKA-2390; Seek() should take a callback



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/lindong28/kafka KAFKA-2390

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/118.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #118


commit 8a9f88649b8de52b48700821f0e1bc5c51a661f3
Author: Dong Lin 
Date:   2015-08-06T00:05:58Z

KAFKA-2390; Seek() should take a callback




> Seek() should take a callback.
> --
>
> Key: KAFKA-2390
> URL: https://issues.apache.org/jira/browse/KAFKA-2390
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>Assignee: Dong Lin
>
> Currently seek is an async call. To have the same interface as other calls 
> like commit(), seek() should take a callback. This callback will be invoked 
> if the position to seek triggers OFFSET_OUT_OF_RANGE exception from broker.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2412) Documentation bug: Add information for key.serializer and value.serializer to New Producer Config sections

[Jeremy Fields (JIRA)]

Jeremy Fields created KAFKA-2412:


 Summary: Documentation bug: Add information for key.serializer and 
value.serializer to New Producer Config sections
 Key: KAFKA-2412
 URL: https://issues.apache.org/jira/browse/KAFKA-2412
 Project: Kafka
  Issue Type: Bug
Reporter: Jeremy Fields
Priority: Minor


As key.serializer and value.serializer are required options when using the new 
producer, they should be mentioned in the documentation ( here and svn 
http://kafka.apache.org/documentation.html#newproducerconfigs )

Appropriate values for these options exist in javadoc and producer.java 
examples; however, not everyone is reading those, as is the case for anyone 
setting up a producer.config file for mirrormaker.

A sensible default should be suggested, such as
org.apache.kafka.common.serialization.StringSerializer
Or at least a mention of the key.serializer and value.serializer options along 
with a link to javadoc

Thanks



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)