[jira] [Comment Edited] (KAFKA-1577) Exception in ConnectionQuotas while shutting down
[
https://issues.apache.org/jira/browse/KAFKA-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272165#comment-14272165
]
German Borbolla edited comment on KAFKA-1577 at 1/10/15 12:56 AM:
--
This is marked as Fix version for 0.8.2 however I just encountered the same
issue using the latest source from the 0.8.2 branch.
Perhaps it was only committed to trunk? Is there any chance this will be
included in 0.8.2?
Here's the stack trace:
{noformat}
[2015-01-09 15:53:12,486] ERROR Uncaught exception in thread
'kafka-network-thread-9092-4': (kafka.utils.Utils$)
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:322)
at scala.None$.get(Option.scala:320)
at kafka.network.ConnectionQuotas.dec(SocketServer.scala:524)
at kafka.network.AbstractServerThread.close(SocketServer.scala:165)
at kafka.network.AbstractServerThread.close(SocketServer.scala:157)
at kafka.network.Processor.close(SocketServer.scala:374)
at kafka.network.AbstractServerThread.closeAll(SocketServer.scala:180)
at kafka.network.Processor.run(SocketServer.scala:364)
at java.lang.Thread.run(Thread.java:745)
{noformat}
Thanks
was (Author: german.borbolla):
This is marked as Fix version for 0.8.2 however I just encountered the same
issue using the latest source from the 0.8.2 branch.
Perhaps it was only committed to trunk? Is there any chance this will be
included in 0.8.2?
Thanks
Exception in ConnectionQuotas while shutting down
-
Key: KAFKA-1577
URL: https://issues.apache.org/jira/browse/KAFKA-1577
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.2
Reporter: Joel Koshy
Assignee: Sriharsha Chintalapani
Priority: Blocker
Labels: newbie
Fix For: 0.8.2
Attachments: KAFKA-1577.patch, KAFKA-1577.patch,
KAFKA-1577_2014-08-20_19:57:44.patch, KAFKA-1577_2014-08-26_07:33:13.patch,
KAFKA-1577_2014-09-26_19:13:05.patch,
KAFKA-1577_check_counter_before_decrementing.patch
{code}
[2014-08-07 19:38:08,228] ERROR Uncaught exception in thread
'kafka-network-thread-9092-0': (kafka.utils.Utils$)
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:185)
at scala.None$.get(Option.scala:183)
at kafka.network.ConnectionQuotas.dec(SocketServer.scala:471)
at kafka.network.AbstractServerThread.close(SocketServer.scala:158)
at kafka.network.AbstractServerThread.close(SocketServer.scala:150)
at kafka.network.AbstractServerThread.closeAll(SocketServer.scala:171)
at kafka.network.Processor.run(SocketServer.scala:338)
at java.lang.Thread.run(Thread.java:662)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 29756: Patch for KAFKA-1854
On Jan. 9, 2015, 11:07 a.m., Manikumar Reddy O wrote: kafka-patch-review.py, line 96 https://reviews.apache.org/r/29756/diff/3/?file=814216#file814216line96 Did you tested the failure senario? I did not get error message. I think we will get exception only after invoking jira.issue(). Jaikiran Pai wrote: I did test a failure case, yes. Gave an incorrect username and password and that failed during login and did not proceed further. Here's the relevant output (notice that I intentionally use JIRA user name foo): ` Configuring reviewboard url to https://reviews.apache.org Updating your remote branches to pull the latest changes Verifying JIRA connection configurations JIRA user :foo JIRA password : Failed to login to the JIRA instance class 'jira.exceptions.JIRAError' HTTP 401: html head titleUnauthorized (401)/title ` Jaikiran Pai wrote: By the way, if you are testing this, make sure you don't have a jira.ini in its usual place. Else the (correct) credentials in there will be used as usual and you won't be able to replicate a failing scenario. Manikumar Reddy O wrote: Configuring reviewboard url to https://reviews.apache.org Updating your remote branches to pull the latest changes Verifying JIRA connection configurations JIRA user :foo JIRA password : Creating diff against origin/0.8.2 and uploading patch to JIRA KAFKA-1723 I am not getting exception, it going to next line of execution. It is failing at jira.issue(). Version : jira-python 0.16 Jaikiran Pai wrote: That's interesting. Let me see what might be different. Jaikiran Pai wrote: Turns out the version of jira python that I had was 0.31 and behaved differently. I downgraded to 0.16 and was able to replicate what you are seeing. I've updated the patch to fix this and it should now correctly fail if wrong JIRA credentials are specified, even before it attempts to publish to reviewboard. Let me know if this now works for you too. Thanks for testing this! Manikumar Reddy O wrote: Latest patch is working with jira-python 0.16. Some members may be using older versions. We need to find correct API which works across all the jira-python versions. (or) we need to add minimum version requirement. Thank you Manikumar for testing and verifying. I'll see what can be done about the jira-python version issues. - Jaikiran --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/#review67414 --- On Jan. 9, 2015, 12:47 p.m., Jaikiran Pai wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/ --- (Updated Jan. 9, 2015, 12:47 p.m.) Review request for kafka. Bugs: KAFKA-1854 https://issues.apache.org/jira/browse/KAFKA-1854 Repository: kafka Description --- KAFKA-1854 Allow JIRA username and password to be prompted in the absence of a jira.ini file, during patch submission Diffs - kafka-patch-review.py b7f132f9d210b8648859ab8f9c89f30ec128ab38 Diff: https://reviews.apache.org/r/29756/diff/ Testing --- Thanks, Jaikiran Pai
Re: Review Request 29756: Patch for KAFKA-1854
On Jan. 9, 2015, 6:48 p.m., Neha Narkhede wrote: kafka-patch-review.py, line 20 https://reviews.apache.org/r/29756/diff/4/?file=814310#file814310line20 I got the following error on this patch nnarkhed-mn1:kafka nnarkhed$ python kafka-patch-review.py -b trunk -j KAFKA-1854 -d test Configuring reviewboard url to https://reviews.apache.org Updating your remote branches to pull the latest changes Verifying JIRA connection configurations JIRA user :nehanarkhede JIRA password : Failed to login to the JIRA instance type 'exceptions.AttributeError' 'JIRA' object has no attribute 'current_user' Maybe a different version of the jira package we use renamed the user field ? Neha, which version of jira-python are you using? I'll find and check that project's documentation to see which version supports what. - Jaikiran --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/#review67477 --- On Jan. 9, 2015, 12:47 p.m., Jaikiran Pai wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/ --- (Updated Jan. 9, 2015, 12:47 p.m.) Review request for kafka. Bugs: KAFKA-1854 https://issues.apache.org/jira/browse/KAFKA-1854 Repository: kafka Description --- KAFKA-1854 Allow JIRA username and password to be prompted in the absence of a jira.ini file, during patch submission Diffs - kafka-patch-review.py b7f132f9d210b8648859ab8f9c89f30ec128ab38 Diff: https://reviews.apache.org/r/29756/diff/ Testing --- Thanks, Jaikiran Pai
[jira] [Commented] (KAFKA-1577) Exception in ConnectionQuotas while shutting down
[
https://issues.apache.org/jira/browse/KAFKA-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272165#comment-14272165
]
German Borbolla commented on KAFKA-1577:
This is marked as Fix version for 0.8.2 however I just encountered the same
issue using the latest source from the 0.8.2 branch.
Perhaps it was only committed to trunk? Is there any chance this will be
included in 0.8.2?
Thanks
Exception in ConnectionQuotas while shutting down
-
Key: KAFKA-1577
URL: https://issues.apache.org/jira/browse/KAFKA-1577
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.2
Reporter: Joel Koshy
Assignee: Sriharsha Chintalapani
Priority: Blocker
Labels: newbie
Fix For: 0.8.2
Attachments: KAFKA-1577.patch, KAFKA-1577.patch,
KAFKA-1577_2014-08-20_19:57:44.patch, KAFKA-1577_2014-08-26_07:33:13.patch,
KAFKA-1577_2014-09-26_19:13:05.patch,
KAFKA-1577_check_counter_before_decrementing.patch
{code}
[2014-08-07 19:38:08,228] ERROR Uncaught exception in thread
'kafka-network-thread-9092-0': (kafka.utils.Utils$)
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:185)
at scala.None$.get(Option.scala:183)
at kafka.network.ConnectionQuotas.dec(SocketServer.scala:471)
at kafka.network.AbstractServerThread.close(SocketServer.scala:158)
at kafka.network.AbstractServerThread.close(SocketServer.scala:150)
at kafka.network.AbstractServerThread.closeAll(SocketServer.scala:171)
at kafka.network.Processor.run(SocketServer.scala:338)
at java.lang.Thread.run(Thread.java:662)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[DISCUSS] Compatability and KIPs
Hey guys, We had a bit of a compatibility slip-up in 0.8.2 with the offset commit stuff. We caught this one before the final release so it's not too bad. But I do think it kind of points to an area we could do better. One piece of feedback we have gotten from going out and talking to users is that compatibility is really, really important to them. Kafka is getting deployed in big environments where the clients are embedded in lots of applications and any kind of incompatibility is a huge pain for people using it and generally makes upgrade difficult or impossible. In practice what I think this means for development is a lot more pressure to really think about the public interfaces we are making and try our best to get them right. This can be hard sometimes as changes come in patches and it is hard to follow every single rb with enough diligence to know. Compatibility really means a couple things: 1. Protocol changes 2. Binary data format changes 3. Changes in public apis in the clients 4. Configs 5. Metric names 6. Command line tools I think 1-2 are critical. 3 is very important. And 4, 5 and 6 are pretty important but not critical. One thing this implies is that we are really going to have to do a good job of thinking about apis and use cases. You can definitely see a number of places in the old clients and in a couple of the protocols where enough care was not given to thinking things through. Some of those were from long long ago, but we should really try to avoid adding to that set because increasingly we will have to carry around these mistakes for a long time. Here are a few things I thought we could do that might help us get better in this area: 1. Technically we are just in a really bad place with the protocol because it is defined twice--once in the old scala request objects, and once in the new protocol format for the clients. This makes changes massively painful. The good news is that the new request definition DSL was intended to make adding new protocol versions a lot easier and clearer. It will also make it a lot more obvious when the protocol is changed since you will be checking in or reviewing a change to Protocol.java. Getting the server moved over to the new request objects and protocol definition will be a bit of a slog but it will really help here I think. 2. We need to get some testing in place on cross-version compatibility. This is work and no tests here will be perfect, but I suspect with some effort we could catch a lot of things. 3. I was also thinking it might be worth it to get a little bit more formal about the review and discussion process for things which will have impact to these public areas to ensure we end up with something we are happy with. Python has a PIP process (https://www.python.org/dev/peps/pep-0257/) by which major changes are made, and it might be worth it for us to do a similar thing. We have essentially been doing this already--major changes almost always have an associated wiki, but I think just getting a little more rigorous might be good. The idea would be to just call out these wikis as official proposals and do a full Apache discuss/vote thread for these important change. We would use these for big features (security, log compaction, etc) as well as for small changes that introduce or change a public api/config/etc. This is a little heavier weight, but I think it is really just critical that we get these things right and this would be a way to call out this kind of change so that everyone would take the time to look at them. Thoughts? -Jay
[jira] [Commented] (KAFKA-1836) metadata.fetch.timeout.ms set to zero blocks forever
[ https://issues.apache.org/jira/browse/KAFKA-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272298#comment-14272298 ] jaikiran pai commented on KAFKA-1836: - Thanks [~nehanarkhede], I've updated the review board patch to address [~ewencp]'s latest comment. Thanks [~ewencp] for reviewing it. metadata.fetch.timeout.ms set to zero blocks forever Key: KAFKA-1836 URL: https://issues.apache.org/jira/browse/KAFKA-1836 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.2 Reporter: Paul Pearcy Priority: Minor Labels: newbie Fix For: 0.8.3 Attachments: KAFKA-1836-new-patch.patch, KAFKA-1836.patch You can easily work around this by setting the timeout value to 1ms, but 0ms should mean 0ms or at least have the behavior documented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29752: Patch for KAFKA-1836
On Jan. 9, 2015, 5:51 p.m., Ewen Cheslack-Postava wrote: clients/src/main/java/org/apache/kafka/clients/producer/internals/Metadata.java, line 110 https://reviews.apache.org/r/29752/diff/1/?file=813990#file813990line110 This works since the check farther down ensures remainingWaitMs is never 0 unless maxWaitMs is 0, but it's a bit confusing when reading this code why we check maxWaitMs instead of remainingWaitMs. There's no functional difference, but I think checking remainingWaitMs instead would be clearer for someone reading this code. Updated to change this and check the remainingWaitMs. Thanks Ewen. - Jaikiran --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29752/#review67457 --- On Jan. 10, 2015, 2:57 a.m., Jaikiran Pai wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29752/ --- (Updated Jan. 10, 2015, 2:57 a.m.) Review request for kafka. Bugs: KAFKA-1836 https://issues.apache.org/jira/browse/KAFKA-1836 Repository: kafka Description --- KAFKA-1836 Don't block forever if metadata.fetch.timeout.ms is set to 0 Diffs - clients/src/main/java/org/apache/kafka/clients/producer/internals/Metadata.java 1d30f9edd95337f86e632a09fc8f4126a67c238b clients/src/test/java/org/apache/kafka/clients/producer/MetadataTest.java 4547bfcb44be4d72742076d6e93f424b3b22a7a9 Diff: https://reviews.apache.org/r/29752/diff/ Testing --- Thanks, Jaikiran Pai
Re: Review Request 29752: Patch for KAFKA-1836
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29752/ --- (Updated Jan. 10, 2015, 2:57 a.m.) Review request for kafka. Bugs: KAFKA-1836 https://issues.apache.org/jira/browse/KAFKA-1836 Repository: kafka Description --- KAFKA-1836 Don't block forever if metadata.fetch.timeout.ms is set to 0 Diffs (updated) - clients/src/main/java/org/apache/kafka/clients/producer/internals/Metadata.java 1d30f9edd95337f86e632a09fc8f4126a67c238b clients/src/test/java/org/apache/kafka/clients/producer/MetadataTest.java 4547bfcb44be4d72742076d6e93f424b3b22a7a9 Diff: https://reviews.apache.org/r/29752/diff/ Testing --- Thanks, Jaikiran Pai
Re: [DISCUSS] Compatability and KIPs
Hi Ted, https://www.python.org/dev/peps/pep-0257/ works fine for me. Are you sure you aren't having that closing ')' in the URL that you are accessing? -Jaikiran On Saturday 10 January 2015 07:16 AM, Ted Yu wrote: Jay: https://www.python.org/dev/peps/pep-0257 https://www.python.org/dev/peps/pep-0257/) gives me 404. Can you double check ? Cheers
Re: [DISCUSS] Compatability and KIPs
Jay: https://www.python.org/dev/peps/pep-0257 https://www.python.org/dev/peps/pep-0257/) gives me 404. Can you double check ? Cheers
Re: Review Request 29714: Patch for KAFKA-1810
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29714/#review67573 --- core/src/main/scala/kafka/network/SocketServer.scala https://reviews.apache.org/r/29714/#comment111609 This needs to be addressed as I backed out the previous fix for 1512 core/src/main/scala/kafka/server/KafkaServer.scala https://reviews.apache.org/r/29714/#comment111610 Also needs to be fixed to include previous fix for 1512 - Jeff Holoman On Jan. 8, 2015, 7:14 p.m., Jeff Holoman wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29714/ --- (Updated Jan. 8, 2015, 7:14 p.m.) Review request for kafka. Bugs: KAFKA-1810 https://issues.apache.org/jira/browse/KAFKA-1810 Repository: kafka Description --- KAFKA-1810 Diffs - core/src/main/scala/kafka/network/IPFilter.scala PRE-CREATION core/src/main/scala/kafka/network/SocketServer.scala 39b1651b680b2995cedfde95d74c086d9c6219ef core/src/main/scala/kafka/server/KafkaConfig.scala 6e26c5436feb4629d17f199011f3ebb674aa767f core/src/main/scala/kafka/server/KafkaServer.scala 1691ad7fc80ca0b112f68e3ea0cbab265c75b26b core/src/main/scala/kafka/utils/VerifiableProperties.scala 2ffc7f452dc7a1b6a06ca7a509ed49e1ab3d1e68 core/src/test/scala/unit/kafka/network/IPFilterTest.scala PRE-CREATION core/src/test/scala/unit/kafka/network/SocketServerTest.scala 78b431f9c88cca1bc5e430ffd41083d0e92b7e75 core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 2377abe4933e065d037828a214c3a87e1773a8ef Diff: https://reviews.apache.org/r/29714/diff/ Testing --- This code centers around a new class, CIDRRange in IPFilter.scala. The IPFilter class is created and holds two fields, the ruleType (allow|deny|none) and a list of CIDRRange objects. This is used in the Socket Server acceptor thread. The check does an exists on the value in the list if the rule type is allow or deny. On object creation, we pre-calculate the lower and upper range values and store those as a BigInt. The overhead of the check should be fairly minimal as it involves converting the incoming IP Address to a BigInt and then just doing a compare to the low/high values. In writing this review up I realized that I can optimize this further to convert to bigint first and move that conversion out of the range check, which I can address. Testing covers the CIDRRange and IPFilter classes and validation of IPV6, IPV4, and configurations. Additionally the functionality is tested in SocketServerTest. Other changes are just to assist in configuration. I modified the SocketServerTest to use a method for grabbing the Socket server to make the code a bit more concise. One key point is that, if there is an error in configuration, we halt the startup of the broker. The thinking there is that if you screw up security-related configs, you want to know about it right away rather than silently accept connections. (thanks Joe Stein for the input). There are two new exceptions realted to this functionality, one to handle configuration errors, and one to handle blocking the request. Currently the level is set to INFO. Does it make sense to move this to WARN ? Thanks, Jeff Holoman
[jira] [Commented] (KAFKA-1577) Exception in ConnectionQuotas while shutting down
[
https://issues.apache.org/jira/browse/KAFKA-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272309#comment-14272309
]
Sriharsha Chintalapani commented on KAFKA-1577:
---
[~german.borbolla] The patch is in 0.8.2 branch . I'll try to see if I can
reproduce this again.
Exception in ConnectionQuotas while shutting down
-
Key: KAFKA-1577
URL: https://issues.apache.org/jira/browse/KAFKA-1577
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.2
Reporter: Joel Koshy
Assignee: Sriharsha Chintalapani
Priority: Blocker
Labels: newbie
Fix For: 0.8.2
Attachments: KAFKA-1577.patch, KAFKA-1577.patch,
KAFKA-1577_2014-08-20_19:57:44.patch, KAFKA-1577_2014-08-26_07:33:13.patch,
KAFKA-1577_2014-09-26_19:13:05.patch,
KAFKA-1577_check_counter_before_decrementing.patch
{code}
[2014-08-07 19:38:08,228] ERROR Uncaught exception in thread
'kafka-network-thread-9092-0': (kafka.utils.Utils$)
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:185)
at scala.None$.get(Option.scala:183)
at kafka.network.ConnectionQuotas.dec(SocketServer.scala:471)
at kafka.network.AbstractServerThread.close(SocketServer.scala:158)
at kafka.network.AbstractServerThread.close(SocketServer.scala:150)
at kafka.network.AbstractServerThread.closeAll(SocketServer.scala:171)
at kafka.network.Processor.run(SocketServer.scala:338)
at java.lang.Thread.run(Thread.java:662)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1577) Exception in ConnectionQuotas while shutting down
[
https://issues.apache.org/jira/browse/KAFKA-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272313#comment-14272313
]
German Borbolla commented on KAFKA-1577:
[~sriharsha] you're right, this happened twice to me today. In both cases the
broker had something around 3 partitions and controlled shutdown finished
successfully. This is a nine node cluster and each was similarly loaded.
I should be able to get you the logs if it helps to debug.
Exception in ConnectionQuotas while shutting down
-
Key: KAFKA-1577
URL: https://issues.apache.org/jira/browse/KAFKA-1577
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.2
Reporter: Joel Koshy
Assignee: Sriharsha Chintalapani
Priority: Blocker
Labels: newbie
Fix For: 0.8.2
Attachments: KAFKA-1577.patch, KAFKA-1577.patch,
KAFKA-1577_2014-08-20_19:57:44.patch, KAFKA-1577_2014-08-26_07:33:13.patch,
KAFKA-1577_2014-09-26_19:13:05.patch,
KAFKA-1577_check_counter_before_decrementing.patch
{code}
[2014-08-07 19:38:08,228] ERROR Uncaught exception in thread
'kafka-network-thread-9092-0': (kafka.utils.Utils$)
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:185)
at scala.None$.get(Option.scala:183)
at kafka.network.ConnectionQuotas.dec(SocketServer.scala:471)
at kafka.network.AbstractServerThread.close(SocketServer.scala:158)
at kafka.network.AbstractServerThread.close(SocketServer.scala:150)
at kafka.network.AbstractServerThread.closeAll(SocketServer.scala:171)
at kafka.network.Processor.run(SocketServer.scala:338)
at java.lang.Thread.run(Thread.java:662)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (KAFKA-1577) Exception in ConnectionQuotas while shutting down
[
https://issues.apache.org/jira/browse/KAFKA-1577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272320#comment-14272320
]
Sriharsha Chintalapani commented on KAFKA-1577:
---
[~german.borbolla] can you please attach some logs . Thanks.
Exception in ConnectionQuotas while shutting down
-
Key: KAFKA-1577
URL: https://issues.apache.org/jira/browse/KAFKA-1577
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.2
Reporter: Joel Koshy
Assignee: Sriharsha Chintalapani
Priority: Blocker
Labels: newbie
Fix For: 0.8.2
Attachments: KAFKA-1577.patch, KAFKA-1577.patch,
KAFKA-1577_2014-08-20_19:57:44.patch, KAFKA-1577_2014-08-26_07:33:13.patch,
KAFKA-1577_2014-09-26_19:13:05.patch,
KAFKA-1577_check_counter_before_decrementing.patch
{code}
[2014-08-07 19:38:08,228] ERROR Uncaught exception in thread
'kafka-network-thread-9092-0': (kafka.utils.Utils$)
java.util.NoSuchElementException: None.get
at scala.None$.get(Option.scala:185)
at scala.None$.get(Option.scala:183)
at kafka.network.ConnectionQuotas.dec(SocketServer.scala:471)
at kafka.network.AbstractServerThread.close(SocketServer.scala:158)
at kafka.network.AbstractServerThread.close(SocketServer.scala:150)
at kafka.network.AbstractServerThread.closeAll(SocketServer.scala:171)
at kafka.network.Processor.run(SocketServer.scala:338)
at java.lang.Thread.run(Thread.java:662)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (KAFKA-1731) add config/jmx changes in 0.8.2 doc
[ https://issues.apache.org/jira/browse/KAFKA-1731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao resolved KAFKA-1731. Resolution: Fixed The patch is already committed. Also committed the doc change in KAFKA-1481. add config/jmx changes in 0.8.2 doc --- Key: KAFKA-1731 URL: https://issues.apache.org/jira/browse/KAFKA-1731 Project: Kafka Issue Type: Sub-task Reporter: Jun Rao Assignee: Jun Rao Priority: Blocker Fix For: 0.8.2 Attachments: config-jmx_082.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 23702: Patch for KAFKA-1070
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23702/#review67551 --- core/src/main/scala/kafka/server/BrokerMetadataFileHandler.scala https://reviews.apache.org/r/23702/#comment111593 This may not be required since you are writing to the file. It will create it if it doesn't already exist. core/src/main/scala/kafka/server/BrokerMetadataFileHandler.scala https://reviews.apache.org/r/23702/#comment111594 Does java serialization for the Properties object allow you to read just the version field? Basically, if we want to upgrade the version of this file, for some time the server would have to support 2 versions and read the file differently based on the different versions. Also, if you serialize the file this way, is it still human readable? I'd recommend you look at other checkpoint file and follow the same or change those to use the new strategy you pick. core/src/main/scala/kafka/server/BrokerMetadataFileHandler.scala https://reviews.apache.org/r/23702/#comment111591 I think the way you modelled this checkpoint file, which makes it different from all other checkpoints we have, is that the metadata checkpoints handles the multiple log directories itself. So it is not a single broker metadata checkpoint. My only concern with this is that now only this checkpoint is very different from other checkpoints. I'd prefer we maintain some consistency in all our checkpoints. So modeling log directories is fine but then it's desirable if all our checkpoints behaved that way. If you'd prefer changing this checkpoint file to match others, I'd suggest- 1. Change the name to BrokerMetadataCheckpoint. 2. Make it take the File that identifies the metadata checkpoint file in the constructor 3. Change write to not accept the log directories. Just include the BrokerMetadata object. 4. Similarly include no parameters in the read() API - Neha Narkhede On Jan. 2, 2015, 1:39 a.m., Sriharsha Chintalapani wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/23702/ --- (Updated Jan. 2, 2015, 1:39 a.m.) Review request for kafka. Bugs: KAFKA-1070 https://issues.apache.org/jira/browse/KAFKA-1070 Repository: kafka Description --- KAFKA-1070. Auto-assign node id. Diffs - core/src/main/scala/kafka/common/GenerateBrokerIdException.scala PRE-CREATION core/src/main/scala/kafka/common/InconsistentBrokerIdException.scala PRE-CREATION core/src/main/scala/kafka/server/BrokerMetadataFileHandler.scala PRE-CREATION core/src/main/scala/kafka/server/KafkaConfig.scala 6e26c5436feb4629d17f199011f3ebb674aa767f core/src/main/scala/kafka/server/KafkaServer.scala 1bf7d10cef23a77e71eb16bf6d0e68bc4ebe core/src/main/scala/kafka/utils/ZkUtils.scala 56e3e88e0cc6d917b0ffd1254e173295c1c4aabd core/src/test/scala/unit/kafka/server/ServerGenerateBrokerIdTest.scala PRE-CREATION core/src/test/scala/unit/kafka/utils/TestUtils.scala 94d0028d8c4907e747aa8a74a13d719b974c97bf Diff: https://reviews.apache.org/r/23702/diff/ Testing --- Thanks, Sriharsha Chintalapani
[jira] [Commented] (KAFKA-1070) Auto-assign node id
[ https://issues.apache.org/jira/browse/KAFKA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272082#comment-14272082 ] Neha Narkhede commented on KAFKA-1070: -- [~sriharsha] There are still questions around the broker metadata file serialization. I've reviewed the latest patch and left my comments on the rb. Auto-assign node id --- Key: KAFKA-1070 URL: https://issues.apache.org/jira/browse/KAFKA-1070 Project: Kafka Issue Type: Bug Reporter: Jay Kreps Assignee: Sriharsha Chintalapani Labels: usability Fix For: 0.8.3 Attachments: KAFKA-1070.patch, KAFKA-1070_2014-07-19_16:06:13.patch, KAFKA-1070_2014-07-22_11:34:18.patch, KAFKA-1070_2014-07-24_20:58:17.patch, KAFKA-1070_2014-07-24_21:05:33.patch, KAFKA-1070_2014-08-21_10:26:20.patch, KAFKA-1070_2014-11-20_10:50:04.patch, KAFKA-1070_2014-11-25_20:29:37.patch, KAFKA-1070_2015-01-01_17:39:30.patch It would be nice to have Kafka brokers auto-assign node ids rather than having that be a configuration. Having a configuration is irritating because (1) you have to generate a custom config for each broker and (2) even though it is in configuration, changing the node id can cause all kinds of bad things to happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1070) Auto-assign node id
[ https://issues.apache.org/jira/browse/KAFKA-1070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272095#comment-14272095 ] Sriharsha Chintalapani commented on KAFKA-1070: --- Thanks [~nehanarkhede] . I'll send an updated patch. Auto-assign node id --- Key: KAFKA-1070 URL: https://issues.apache.org/jira/browse/KAFKA-1070 Project: Kafka Issue Type: Bug Reporter: Jay Kreps Assignee: Sriharsha Chintalapani Labels: usability Fix For: 0.8.3 Attachments: KAFKA-1070.patch, KAFKA-1070_2014-07-19_16:06:13.patch, KAFKA-1070_2014-07-22_11:34:18.patch, KAFKA-1070_2014-07-24_20:58:17.patch, KAFKA-1070_2014-07-24_21:05:33.patch, KAFKA-1070_2014-08-21_10:26:20.patch, KAFKA-1070_2014-11-20_10:50:04.patch, KAFKA-1070_2014-11-25_20:29:37.patch, KAFKA-1070_2015-01-01_17:39:30.patch It would be nice to have Kafka brokers auto-assign node ids rather than having that be a configuration. Having a configuration is irritating because (1) you have to generate a custom config for each broker and (2) even though it is in configuration, changing the node id can cause all kinds of bad things to happen. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 28769: Patch for KAFKA-1809
On Jan. 7, 2015, 3:01 a.m., Jun Rao wrote: Thanks for the patch. It's a lot work! A few general comments below, in addition to the more detailed comments. 1. Formatting: add a space after comma in function signature and function calls. So instead of def foo(a: A,b: B) use def foo(a: A, b: B) 2. In some of the files, imports can be optimized. 3. Some new files are missing the license header. 4. It seems that the client needs to know the security protocol in addition to the port. Otherwise, it doesn't know which protocol to use to establish the socket connection. So, perhaps the broker.list should now be protocol://host:port? Gwen Shapira wrote: for #4 - we currently have security.protocol parameter for producer and consumer - reason being that: 1. broker.list may specify multiple brokers, but I think we want to keep communication to one protocol per client. We can validate a single protocol in the config, but it is still a bit wierd - asking users to type protocol multiple times when it has to be the same. 2. Zookeeper consumer currently doesn't have broker.list, they have zookeeper list, but it still needs to know the protocol That sounds good. That change is not included in this patch? On Jan. 7, 2015, 3:01 a.m., Jun Rao wrote: core/src/main/scala/kafka/cluster/Broker.scala, lines 36-52 https://reviews.apache.org/r/28769/diff/11/?file=807930#file807930line36 It would be good to document the format for both v1 and v2. Also, would it be cleaner if in v2, we remove the host/port fields and only include the endpoints field? Currently, v2 writes host/port, but doesn't use it itself. So, it's really just for backward compatibility. Since we have the use.new.wire.protocol config, we can let each broker register in ZK using v2 format only after that config is set to true (when all brokers are upgraded to the new binary). Gwen Shapira wrote: I think it makes sense to leave V1 as without endpoints and make V2 only endpoints, no host/port fields. We can use use.new.wire.protocol to choose which we serialize. This means that until we set new.protocol to true, non-plaintext endpoints will exist in config files (and therefore brokers will listen there), but will not exist in ZK. This is in line with our current understanding that until upgrade is finished (and use.new.protocol is set to true), non-plaintext endpoints will not be supported or used. A less expected side-effect: At the time we switch from V1 to V2, createEphemeralPathExpectConflictHandleZKBug will not recognize the existing broker (if ephemeral node is still around) as identical to the new registration (since the new broker will potentially have more endpoints). This means that if the ephemeral node is still around when we switch, we'll get broker already registered exception, instead of looping around until the ephemeral node goes away. I think we are fine with the behavior here, but I wanted to make it explicit. Yes, that typically only happens when ZK session expires, which is rare. On Jan. 7, 2015, 3:01 a.m., Jun Rao wrote: core/src/main/scala/kafka/cluster/EndPoint.scala, line 36 https://reviews.apache.org/r/28769/diff/11/?file=807932#file807932line36 Do we need the dot after -? Gwen Shapira wrote: Yes. Because of the brackets its an actual dot (doesn't match any) and its needed because there's an IP in there, we need to match 1.1.1.1 make sense On Jan. 7, 2015, 3:01 a.m., Jun Rao wrote: core/src/main/scala/kafka/network/SocketServer.scala, line 105 https://reviews.apache.org/r/28769/diff/11/?file=807946#file807946line105 I think each acceptor needs its own set of processors. We probably need to change the doc for num.network.threads to indicate that it's per endpoint. Gwen Shapira wrote: I'm not sure I understand why. Is it because you expect different processor implementations later? You are right. I was thinking that we will implement different types of socket support at the processor level, but that's actually at the socket channel level. - Jun --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28769/#review66944 --- On Jan. 6, 2015, 7:46 p.m., Gwen Shapira wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/28769/ --- (Updated Jan. 6, 2015, 7:46 p.m.) Review request for kafka. Bugs: KAFKA-1809 https://issues.apache.org/jira/browse/KAFKA-1809 Repository: kafka Description --- first commit of refactoring. changed
Re: Review Request 29523: Patch for KAFKA-1723
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29523/#review67446 --- Thanks for the new patch. Looks good. Just a few more minor comments. clients/src/main/java/org/apache/kafka/clients/producer/Producer.java https://reviews.apache.org/r/29523/#comment111471 Since both MetricName and Metric are part of the public api, we need to change the gradle build file to include these two classes in the generated javadoc. clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java https://reviews.apache.org/r/29523/#comment111469 Incorrect param group. clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java https://reviews.apache.org/r/29523/#comment111470 It seems that we pass in the group name here whereas in other places we just define a local constant. It would be good to make this consistent: either the group is passed in or is just defined locally. - Jun Rao On Jan. 9, 2015, 8:56 a.m., Manikumar Reddy O wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29523/ --- (Updated Jan. 9, 2015, 8:56 a.m.) Review request for kafka. Bugs: KAFKA-1723 https://issues.apache.org/jira/browse/KAFKA-1723 Repository: kafka Description --- Standard JMX MBean Naming is implemented;Addresing Jun's comments Diffs - clients/src/main/java/org/apache/kafka/clients/consumer/Consumer.java 1bce50185273dbdbc131fbc9c7f5f3e9c346517a clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 7f8a41c4bf437711685a8271a4d3c83a176dd957 clients/src/main/java/org/apache/kafka/clients/consumer/MockConsumer.java 8cab16c0a0bdb671fea1fc2fc2694247f66cc971 clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 3053f2745c8e5f6e3b75826d3749656f150878db clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 5baa6062bd9ba8a7d38058856ed2d831fae491f0 clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java aa91e1444a49c55870b9a7a32086fa2b04471fba clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java c15485d1af304ef53691d478f113f332fe67af77 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 84a7a07269c51ccc22ebb4ff9797292d07ba778e clients/src/main/java/org/apache/kafka/common/Metric.java b023e8e7c486adf21ed9a554085ab8ad7f3ee038 clients/src/main/java/org/apache/kafka/common/metrics/CompoundStat.java 29185a6a90d0035d650c7e56ce612a0878cb115c clients/src/main/java/org/apache/kafka/common/metrics/JmxReporter.java 3c312011a7ff7e79c277a89048e7e62ebd6078db clients/src/main/java/org/apache/kafka/common/metrics/KafkaMetric.java a7458b50cb16fbb2b31b857d5b359e65258bbf08 clients/src/main/java/org/apache/kafka/common/metrics/MetricName.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java 49be4019ac03835701c49646920766228ac7ffe9 clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 25c1faf2887ea02708c1f5b5f822f5299ed86bd6 clients/src/main/java/org/apache/kafka/common/metrics/stats/Percentile.java 7365ceb39072a6a1ecf533f5a20830ed1f2cfc72 clients/src/main/java/org/apache/kafka/common/metrics/stats/Percentiles.java c70d577ada8c099533d4f4ed2e86d37e0a6e6676 clients/src/main/java/org/apache/kafka/common/network/Selector.java 4dd2cdf773f7eb01a93d7f994383088960303dfc clients/src/test/java/org/apache/kafka/clients/producer/BufferPoolTest.java fe3c13f319d48b89a4f26b6d78c2c3e31cc50d7e clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java 2c9932401d573549c40f16fda8c4e3e11309cb85 clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java ef2ca65cabe97b909f17b62027a1bb06827e88fe clients/src/test/java/org/apache/kafka/common/metrics/JmxReporterTest.java 2f43c49450e1a3d671bd17417dc42941f1858750 clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java 19bea0f1fa1ebf15d86623015ec909b0155e4bd3 clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 5c5e3d40819e41cab7b52a0eeaee5f2e7317b7b3 clients/src/test/java/org/apache/kafka/test/MetricsBench.java 9d98c1148255455fd801043b59b98fed9d0b76b3 Diff: https://reviews.apache.org/r/29523/diff/ Testing --- Thanks, Manikumar Reddy O
[jira] [Updated] (KAFKA-1854) Allow the JIRA username and password to be prompted during patch submission
[ https://issues.apache.org/jira/browse/KAFKA-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jaikiran pai updated KAFKA-1854: Status: Patch Available (was: Open) Patch has been submitted P.S: It looks like (while auto submitting the patch) I can't assign this issue to myself. Perhaps because I'm not in the right JIRA group? Allow the JIRA username and password to be prompted during patch submission --- Key: KAFKA-1854 URL: https://issues.apache.org/jira/browse/KAFKA-1854 Project: Kafka Issue Type: Improvement Reporter: jaikiran pai Attachments: KAFKA-1854.patch, KAFKA-1854_2015-01-09_13:39:23.patch The current patch submission process involves using the kafka-patch-review.py python script which expects a jira.ini file to contain the user's username and password for JIRA authentication. I'm one of those who doesn't like storing passwords in files :) It would be good to (optionally) allow the username/password to be prompted by the patch submission script. I've a patch which I can submit for this enhancement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29756: Patch for KAFKA-1854
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/ --- (Updated Jan. 9, 2015, 8:09 a.m.) Review request for kafka. Bugs: KAFKA-1854 https://issues.apache.org/jira/browse/KAFKA-1854 Repository: kafka Description --- KAFKA-1854 Allow JIRA username and password to be prompted in the absence of a jira.ini file, during patch submission Diffs (updated) - kafka-patch-review.py b7f132f9d210b8648859ab8f9c89f30ec128ab38 Diff: https://reviews.apache.org/r/29756/diff/ Testing --- Thanks, Jaikiran Pai
[jira] [Updated] (KAFKA-1854) Allow the JIRA username and password to be prompted during patch submission
[ https://issues.apache.org/jira/browse/KAFKA-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jaikiran pai updated KAFKA-1854: Attachment: KAFKA-1854_2015-01-09_13:39:23.patch Allow the JIRA username and password to be prompted during patch submission --- Key: KAFKA-1854 URL: https://issues.apache.org/jira/browse/KAFKA-1854 Project: Kafka Issue Type: Improvement Reporter: jaikiran pai Attachments: KAFKA-1854.patch, KAFKA-1854_2015-01-09_13:39:23.patch The current patch submission process involves using the kafka-patch-review.py python script which expects a jira.ini file to contain the user's username and password for JIRA authentication. I'm one of those who doesn't like storing passwords in files :) It would be good to (optionally) allow the username/password to be prompted by the patch submission script. I've a patch which I can submit for this enhancement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1854) Allow the JIRA username and password to be prompted during patch submission
[ https://issues.apache.org/jira/browse/KAFKA-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270735#comment-14270735 ] jaikiran pai commented on KAFKA-1854: - Updated reviewboard https://reviews.apache.org/r/29756/diff/ against branch origin/trunk Allow the JIRA username and password to be prompted during patch submission --- Key: KAFKA-1854 URL: https://issues.apache.org/jira/browse/KAFKA-1854 Project: Kafka Issue Type: Improvement Reporter: jaikiran pai Attachments: KAFKA-1854.patch, KAFKA-1854_2015-01-09_13:39:23.patch The current patch submission process involves using the kafka-patch-review.py python script which expects a jira.ini file to contain the user's username and password for JIRA authentication. I'm one of those who doesn't like storing passwords in files :) It would be good to (optionally) allow the username/password to be prompted by the patch submission script. I've a patch which I can submit for this enhancement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1723) make the metrics name in new producer more standard
[ https://issues.apache.org/jira/browse/KAFKA-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270783#comment-14270783 ] Manikumar Reddy commented on KAFKA-1723: Updated reviewboard https://reviews.apache.org/r/29523/diff/ against branch origin/0.8.2 make the metrics name in new producer more standard --- Key: KAFKA-1723 URL: https://issues.apache.org/jira/browse/KAFKA-1723 Project: Kafka Issue Type: Sub-task Components: clients Affects Versions: 0.8.2 Reporter: Jun Rao Assignee: Manikumar Reddy Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1723.patch, KAFKA-1723_2015-01-08_21:41:13.patch, KAFKA-1723_2015-01-08_22:02:22.patch, KAFKA-1723_2015-01-09_14:24:18.patch The jmx name in the new producer looks like the following: kafka.producer.myclientid:type=mytopic However, this can be ambiguous since we allow . in client id and topic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1723) make the metrics name in new producer more standard
[ https://issues.apache.org/jira/browse/KAFKA-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy updated KAFKA-1723: --- Attachment: KAFKA-1723_2015-01-09_14:24:18.patch make the metrics name in new producer more standard --- Key: KAFKA-1723 URL: https://issues.apache.org/jira/browse/KAFKA-1723 Project: Kafka Issue Type: Sub-task Components: clients Affects Versions: 0.8.2 Reporter: Jun Rao Assignee: Manikumar Reddy Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1723.patch, KAFKA-1723_2015-01-08_21:41:13.patch, KAFKA-1723_2015-01-08_22:02:22.patch, KAFKA-1723_2015-01-09_14:24:18.patch The jmx name in the new producer looks like the following: kafka.producer.myclientid:type=mytopic However, this can be ambiguous since we allow . in client id and topic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29756: Patch for KAFKA-1854
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/ --- (Updated Jan. 9, 2015, 10:13 a.m.) Review request for kafka. Bugs: KAFKA-1854 https://issues.apache.org/jira/browse/KAFKA-1854 Repository: kafka Description --- KAFKA-1854 Allow JIRA username and password to be prompted in the absence of a jira.ini file, during patch submission Diffs (updated) - kafka-patch-review.py b7f132f9d210b8648859ab8f9c89f30ec128ab38 Diff: https://reviews.apache.org/r/29756/diff/ Testing --- Thanks, Jaikiran Pai
[jira] [Commented] (KAFKA-1854) Allow the JIRA username and password to be prompted during patch submission
[ https://issues.apache.org/jira/browse/KAFKA-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270839#comment-14270839 ] jaikiran pai commented on KAFKA-1854: - Updated reviewboard https://reviews.apache.org/r/29756/diff/ against branch origin/trunk Allow the JIRA username and password to be prompted during patch submission --- Key: KAFKA-1854 URL: https://issues.apache.org/jira/browse/KAFKA-1854 Project: Kafka Issue Type: Improvement Reporter: jaikiran pai Attachments: KAFKA-1854.patch, KAFKA-1854_2015-01-09_13:39:23.patch, KAFKA-1854_2015-01-09_15:42:28.patch The current patch submission process involves using the kafka-patch-review.py python script which expects a jira.ini file to contain the user's username and password for JIRA authentication. I'm one of those who doesn't like storing passwords in files :) It would be good to (optionally) allow the username/password to be prompted by the patch submission script. I've a patch which I can submit for this enhancement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1854) Allow the JIRA username and password to be prompted during patch submission
[ https://issues.apache.org/jira/browse/KAFKA-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jaikiran pai updated KAFKA-1854: Attachment: KAFKA-1854_2015-01-09_15:42:28.patch Allow the JIRA username and password to be prompted during patch submission --- Key: KAFKA-1854 URL: https://issues.apache.org/jira/browse/KAFKA-1854 Project: Kafka Issue Type: Improvement Reporter: jaikiran pai Attachments: KAFKA-1854.patch, KAFKA-1854_2015-01-09_13:39:23.patch, KAFKA-1854_2015-01-09_15:42:28.patch The current patch submission process involves using the kafka-patch-review.py python script which expects a jira.ini file to contain the user's username and password for JIRA authentication. I'm one of those who doesn't like storing passwords in files :) It would be good to (optionally) allow the username/password to be prompted by the patch submission script. I've a patch which I can submit for this enhancement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29756: Patch for KAFKA-1854
On Jan. 9, 2015, 7:52 a.m., Jaikiran Pai wrote: This one needs a minor change which I'm going to submit as an update Patch updated and now ready for review. - Jaikiran --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/#review67388 --- On Jan. 9, 2015, 8:09 a.m., Jaikiran Pai wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/ --- (Updated Jan. 9, 2015, 8:09 a.m.) Review request for kafka. Bugs: KAFKA-1854 https://issues.apache.org/jira/browse/KAFKA-1854 Repository: kafka Description --- KAFKA-1854 Allow JIRA username and password to be prompted in the absence of a jira.ini file, during patch submission Diffs - kafka-patch-review.py b7f132f9d210b8648859ab8f9c89f30ec128ab38 Diff: https://reviews.apache.org/r/29756/diff/ Testing --- Thanks, Jaikiran Pai
Re: Review Request 29523: Patch for KAFKA-1723
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29523/ --- (Updated Jan. 9, 2015, 8:56 a.m.) Review request for kafka. Bugs: KAFKA-1723 https://issues.apache.org/jira/browse/KAFKA-1723 Repository: kafka Description (updated) --- Standard JMX MBean Naming is implemented;Addresing Jun's comments Diffs (updated) - clients/src/main/java/org/apache/kafka/clients/consumer/Consumer.java 1bce50185273dbdbc131fbc9c7f5f3e9c346517a clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 7f8a41c4bf437711685a8271a4d3c83a176dd957 clients/src/main/java/org/apache/kafka/clients/consumer/MockConsumer.java 8cab16c0a0bdb671fea1fc2fc2694247f66cc971 clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 3053f2745c8e5f6e3b75826d3749656f150878db clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 5baa6062bd9ba8a7d38058856ed2d831fae491f0 clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java aa91e1444a49c55870b9a7a32086fa2b04471fba clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java c15485d1af304ef53691d478f113f332fe67af77 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 84a7a07269c51ccc22ebb4ff9797292d07ba778e clients/src/main/java/org/apache/kafka/common/Metric.java b023e8e7c486adf21ed9a554085ab8ad7f3ee038 clients/src/main/java/org/apache/kafka/common/metrics/CompoundStat.java 29185a6a90d0035d650c7e56ce612a0878cb115c clients/src/main/java/org/apache/kafka/common/metrics/JmxReporter.java 3c312011a7ff7e79c277a89048e7e62ebd6078db clients/src/main/java/org/apache/kafka/common/metrics/KafkaMetric.java a7458b50cb16fbb2b31b857d5b359e65258bbf08 clients/src/main/java/org/apache/kafka/common/metrics/MetricName.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java 49be4019ac03835701c49646920766228ac7ffe9 clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 25c1faf2887ea02708c1f5b5f822f5299ed86bd6 clients/src/main/java/org/apache/kafka/common/metrics/stats/Percentile.java 7365ceb39072a6a1ecf533f5a20830ed1f2cfc72 clients/src/main/java/org/apache/kafka/common/metrics/stats/Percentiles.java c70d577ada8c099533d4f4ed2e86d37e0a6e6676 clients/src/main/java/org/apache/kafka/common/network/Selector.java 4dd2cdf773f7eb01a93d7f994383088960303dfc clients/src/test/java/org/apache/kafka/clients/producer/BufferPoolTest.java fe3c13f319d48b89a4f26b6d78c2c3e31cc50d7e clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java 2c9932401d573549c40f16fda8c4e3e11309cb85 clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java ef2ca65cabe97b909f17b62027a1bb06827e88fe clients/src/test/java/org/apache/kafka/common/metrics/JmxReporterTest.java 2f43c49450e1a3d671bd17417dc42941f1858750 clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java 19bea0f1fa1ebf15d86623015ec909b0155e4bd3 clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 5c5e3d40819e41cab7b52a0eeaee5f2e7317b7b3 clients/src/test/java/org/apache/kafka/test/MetricsBench.java 9d98c1148255455fd801043b59b98fed9d0b76b3 Diff: https://reviews.apache.org/r/29523/diff/ Testing --- Thanks, Manikumar Reddy O
Re: Review Request 29756: Patch for KAFKA-1854
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/#review67400 --- kafka-patch-review.py https://reviews.apache.org/r/29756/#comment111418 Can we do authentication check at the beginning and fail-fast if the username/password is wrong. Currently it will update the review-board and fails at JIRA update. Is it possible to catch the exception/error, and show the message Invalid JIRA username/password - Manikumar Reddy O On Jan. 9, 2015, 8:09 a.m., Jaikiran Pai wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/ --- (Updated Jan. 9, 2015, 8:09 a.m.) Review request for kafka. Bugs: KAFKA-1854 https://issues.apache.org/jira/browse/KAFKA-1854 Repository: kafka Description --- KAFKA-1854 Allow JIRA username and password to be prompted in the absence of a jira.ini file, during patch submission Diffs - kafka-patch-review.py b7f132f9d210b8648859ab8f9c89f30ec128ab38 Diff: https://reviews.apache.org/r/29756/diff/ Testing --- Thanks, Jaikiran Pai
[jira] [Updated] (KAFKA-1852) OffsetCommitRequest can commit offset on unknown topic
[ https://issues.apache.org/jira/browse/KAFKA-1852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1852: - Reviewer: Joel Koshy OffsetCommitRequest can commit offset on unknown topic -- Key: KAFKA-1852 URL: https://issues.apache.org/jira/browse/KAFKA-1852 Project: Kafka Issue Type: Bug Affects Versions: 0.8.3 Reporter: Jun Rao Currently, we allow an offset to be committed to Kafka, even when the topic/partition for the offset doesn't exist. We probably should disallow that and send an error back in that case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1854) Allow the JIRA username and password to be prompted during patch submission
[ https://issues.apache.org/jira/browse/KAFKA-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1854: - Reviewer: Neha Narkhede Allow the JIRA username and password to be prompted during patch submission --- Key: KAFKA-1854 URL: https://issues.apache.org/jira/browse/KAFKA-1854 Project: Kafka Issue Type: Improvement Reporter: jaikiran pai Attachments: KAFKA-1854.patch, KAFKA-1854_2015-01-09_13:39:23.patch, KAFKA-1854_2015-01-09_15:42:28.patch, KAFKA-1854_2015-01-09_18:16:35.patch The current patch submission process involves using the kafka-patch-review.py python script which expects a jira.ini file to contain the user's username and password for JIRA authentication. I'm one of those who doesn't like storing passwords in files :) It would be good to (optionally) allow the username/password to be prompted by the patch submission script. I've a patch which I can submit for this enhancement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1848) Checking shutdown during each iteration of ZookeeperConsumerConnector
[ https://issues.apache.org/jira/browse/KAFKA-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271714#comment-14271714 ] Aditya Auradkar commented on KAFKA-1848: https://reviews.apache.org/r/29728/ Checking shutdown during each iteration of ZookeeperConsumerConnector - Key: KAFKA-1848 URL: https://issues.apache.org/jira/browse/KAFKA-1848 Project: Kafka Issue Type: Bug Reporter: Aditya Auradkar Assignee: Aditya Auradkar Fix For: 0.9.0 In ZookeeperConsumerConnector the syncedRebalance() method checks the isShuttingDown flag before it triggers a rebalance. However, it does not recheck the same value between successive retries which is possible if the consumer is shutting down. This acquires the rebalanceLock and blocks shutdown from completing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1826) add command to delete all consumer group information for a topic in zookeeper
[ https://issues.apache.org/jira/browse/KAFKA-1826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271733#comment-14271733 ] Neha Narkhede commented on KAFKA-1826: -- [~onurkaraman] The topic tool should have actions related to kafka topics. It is cleaner to not include consumer related actions to the generic topic tool since that is linked to consumers, not necessarily topics. For this new consumer command, you would have options like --group, --delete and --topic to be able to perform actions that relate to consumers. The ConsumerCommand would look very similar to TopicCommand in its structure. The reason I suggested you take the patch in KAFKA-1476 is because it is essentially very close to including this ConsumerCommand. You can take that and add the --delete option. add command to delete all consumer group information for a topic in zookeeper - Key: KAFKA-1826 URL: https://issues.apache.org/jira/browse/KAFKA-1826 Project: Kafka Issue Type: Sub-task Reporter: Onur Karaman Assignee: Onur Karaman Attachments: KAFKA-1826.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1723) make the metrics name in new producer more standard
[
https://issues.apache.org/jira/browse/KAFKA-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271758#comment-14271758
]
Jay Kreps commented on KAFKA-1723:
--
Yeah, I'm not really sure whether to just leave it or to maybe try to replace
the map with a class with some helper methods:
Something like
{code}
class Metrics implements IterableMetric {
public Metric getByFullName(MetricName name);
public ListMetric getByName(String name);
public ListMetric getByNameAndGroup(String name, String group);
}
{code}
make the metrics name in new producer more standard
---
Key: KAFKA-1723
URL: https://issues.apache.org/jira/browse/KAFKA-1723
Project: Kafka
Issue Type: Sub-task
Components: clients
Affects Versions: 0.8.2
Reporter: Jun Rao
Assignee: Manikumar Reddy
Priority: Blocker
Fix For: 0.8.2
Attachments: KAFKA-1723.patch, KAFKA-1723_2015-01-08_21:41:13.patch,
KAFKA-1723_2015-01-08_22:02:22.patch, KAFKA-1723_2015-01-09_14:24:18.patch,
KAFKA-1723_2015-01-09_23:43:22.patch
The jmx name in the new producer looks like the following:
kafka.producer.myclientid:type=mytopic
However, this can be ambiguous since we allow . in client id and topic.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Follow-up On Important Issues for 0.8.2
Bhavesh, 1. I would expect the behavior you're suggesting as well. Any fix for the JIRA should address this, and I raised that point when reviewing the current version of the patch. 2. It's currently marked for 0.8.3 so it probably won't make it into 0.8.2, especially if there isn't a patch for it ready soon. I think that's probably the right choice -- it's a pretty unusual corner case and the current behavior is annoying, but not a critical bug. -Ewen On Thu, Jan 8, 2015 at 10:59 PM, Bhavesh Mistry mistry.p.bhav...@gmail.com wrote: Adding User Community to see if any one knows behavior of Producer for issue #1) and status of 2). Thanks, Bhavesh On Fri, Jan 2, 2015 at 12:37 PM, Bhavesh Mistry mistry.p.bhav...@gmail.com wrote: Hi Kafka Dev Team, I am following-up with you guys regarding New (Java) Producer behavior in event of network or firewall rules. I just wanted to make Java Producer resilient of any network or firewall issues, and does not become single-point of failure in application: 1) Jira Issue https://issues.apache.org/jira/browse/KAFKA-1788 https://issues.apache.org/jira/browse/KAFKA-1788?focusedCommentId=14259235page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14259235 What should be the behavior of the Producer when it can not reach leader broker, but metadata reported broker is leader for that partition (via other broker) ? Should the record-error-rate be counted and Call Back should be called with error or not ? 1) *record-error-rate* metric remain zero despite following firewall rule. In my opinion, it should have called org.apache.kafka.clients.producer.Callback but I did not see that happening either one or two are brokers not reachable. 2) Is jira ticket https://issues.apache.org/jira/browse/KAFKA-1788 will be merged to 0.8.2 ? This will give the ability to close the producer in event of lost connectivity to broker if io thread misbehave (does not end) ? Thanks for your help ! Thanks, Bhavesh -- Thanks, Ewen
[jira] [Commented] (KAFKA-1851) OffsetFetchRequest returns extra partitions when input only contains unknown partitions
[ https://issues.apache.org/jira/browse/KAFKA-1851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271595#comment-14271595 ] Neha Narkhede commented on KAFKA-1851: -- [~junrao] Why is this a blocker? OffsetFetchRequest returns extra partitions when input only contains unknown partitions --- Key: KAFKA-1851 URL: https://issues.apache.org/jira/browse/KAFKA-1851 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Assignee: Jun Rao Priority: Blocker Fix For: 0.8.2 Attachments: kafka-1851.patch When issuing an OffsetFetchRequest with an unknown topic partition, the OffsetFetchResponse unexpectedly returns all partitions in the same consumer group, in addition to the unknown partition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1819) Cleaner gets confused about deleted and re-created topics
[
https://issues.apache.org/jira/browse/KAFKA-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271612#comment-14271612
]
Neha Narkhede edited comment on KAFKA-1819 at 1/9/15 6:14 PM:
--
[~gwenshap] Saw the following unit test failures on the latest patch on 0.8.2
branch and trunk, while trying to check it in -
{code}
kafka.server.ServerShutdownTest testCleanShutdownWithDeleteTopicEnabled FAILED
junit.framework.AssertionFailedError: expected:0 but was:1
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.failNotEquals(Assert.java:277)
at junit.framework.Assert.assertEquals(Assert.java:64)
at junit.framework.Assert.assertEquals(Assert.java:195)
at junit.framework.Assert.assertEquals(Assert.java:201)
at
kafka.server.ServerShutdownTest.verifyNonDaemonThreadsStatus(ServerShutdownTest.scala:145)
at
kafka.server.ServerShutdownTest.testCleanShutdownWithDeleteTopicEnabled(ServerShutdownTest.scala:114)
kafka.server.ServerShutdownTest testCleanShutdownAfterFailedStartup FAILED
junit.framework.AssertionFailedError: expected:0 but was:1
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.failNotEquals(Assert.java:277)
at junit.framework.Assert.assertEquals(Assert.java:64)
at junit.framework.Assert.assertEquals(Assert.java:195)
at junit.framework.Assert.assertEquals(Assert.java:201)
at
kafka.server.ServerShutdownTest.verifyNonDaemonThreadsStatus(ServerShutdownTest.scala:145)
at
kafka.server.ServerShutdownTest.testCleanShutdownAfterFailedStartup(ServerShutdownTest.scala:141)
kafka.server.ServerShutdownTest testCleanShutdown FAILED
junit.framework.AssertionFailedError: expected:0 but was:1
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.failNotEquals(Assert.java:277)
at junit.framework.Assert.assertEquals(Assert.java:64)
at junit.framework.Assert.assertEquals(Assert.java:195)
at junit.framework.Assert.assertEquals(Assert.java:201)
at
kafka.server.ServerShutdownTest.verifyNonDaemonThreadsStatus(ServerShutdownTest.scala:145)
at
kafka.server.ServerShutdownTest.testCleanShutdown(ServerShutdownTest.scala:101)
{code}
was (Author: nehanarkhede):
[~gwenshap] Saw the following unit test failures on the latest patch on 0.8.2
branch, while trying to check it in -
{code}
kafka.server.ServerShutdownTest testCleanShutdownWithDeleteTopicEnabled FAILED
junit.framework.AssertionFailedError: expected:0 but was:1
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.failNotEquals(Assert.java:277)
at junit.framework.Assert.assertEquals(Assert.java:64)
at junit.framework.Assert.assertEquals(Assert.java:195)
at junit.framework.Assert.assertEquals(Assert.java:201)
at
kafka.server.ServerShutdownTest.verifyNonDaemonThreadsStatus(ServerShutdownTest.scala:145)
at
kafka.server.ServerShutdownTest.testCleanShutdownWithDeleteTopicEnabled(ServerShutdownTest.scala:114)
kafka.server.ServerShutdownTest testCleanShutdownAfterFailedStartup FAILED
junit.framework.AssertionFailedError: expected:0 but was:1
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.failNotEquals(Assert.java:277)
at junit.framework.Assert.assertEquals(Assert.java:64)
at junit.framework.Assert.assertEquals(Assert.java:195)
at junit.framework.Assert.assertEquals(Assert.java:201)
at
kafka.server.ServerShutdownTest.verifyNonDaemonThreadsStatus(ServerShutdownTest.scala:145)
at
kafka.server.ServerShutdownTest.testCleanShutdownAfterFailedStartup(ServerShutdownTest.scala:141)
kafka.server.ServerShutdownTest testCleanShutdown FAILED
junit.framework.AssertionFailedError: expected:0 but was:1
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.failNotEquals(Assert.java:277)
at junit.framework.Assert.assertEquals(Assert.java:64)
at junit.framework.Assert.assertEquals(Assert.java:195)
at junit.framework.Assert.assertEquals(Assert.java:201)
at
kafka.server.ServerShutdownTest.verifyNonDaemonThreadsStatus(ServerShutdownTest.scala:145)
at
kafka.server.ServerShutdownTest.testCleanShutdown(ServerShutdownTest.scala:101)
{code}
Cleaner gets confused about deleted and re-created topics
-
Key: KAFKA-1819
URL: https://issues.apache.org/jira/browse/KAFKA-1819
Project: Kafka
Issue Type: Bug
Reporter: Gian Merlino
Assignee: Gwen Shapira
Priority: Blocker
Fix For: 0.8.2
Attachments: KAFKA-1819.patch,
Re: Review Request 29752: Patch for KAFKA-1836
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29752/#review67466 --- Ship it! Looks good other than what Ewen pointed out. - Neha Narkhede On Jan. 9, 2015, 3:59 a.m., Jaikiran Pai wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29752/ --- (Updated Jan. 9, 2015, 3:59 a.m.) Review request for kafka. Bugs: KAFKA-1836 https://issues.apache.org/jira/browse/KAFKA-1836 Repository: kafka Description --- KAFKA-1836 Don't block forever if metadata.fetch.timeout.ms is set to 0 Diffs - clients/src/main/java/org/apache/kafka/clients/producer/internals/Metadata.java 1d30f9edd95337f86e632a09fc8f4126a67c238b clients/src/test/java/org/apache/kafka/clients/producer/MetadataTest.java 4547bfcb44be4d72742076d6e93f424b3b22a7a9 Diff: https://reviews.apache.org/r/29752/diff/ Testing --- Thanks, Jaikiran Pai
[jira] [Updated] (KAFKA-1855) Topic unusable after unsuccessful UpdateMetadataRequest
[ https://issues.apache.org/jira/browse/KAFKA-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1855: - Reviewer: Neha Narkhede Assignee: (was: Neha Narkhede) Topic unusable after unsuccessful UpdateMetadataRequest --- Key: KAFKA-1855 URL: https://issues.apache.org/jira/browse/KAFKA-1855 Project: Kafka Issue Type: Bug Components: controller Affects Versions: 0.8.2 Reporter: Henri Pihkala Sometimes, seemingly randomly, topic creation/initialization might fail with the following lines in controller.log. Other logs show no errors. When this happens, the topic is unusable (gives UnknownTopicOrPartition for all requests). For me this happens 5-10% of the time. Feels like it's more likely to happen if there is time between topic creations. Observed on 0.8.2-beta, have not tried previous versions. [2015-01-09 16:15:27,153] WARN [Controller-0-to-broker-0-send-thread], Controller 0 fails to send a request to broker id:0,host:192.168.10.21,port:9092 (kafka.controller.RequestSendThread) java.io.EOFException: Received -1 when reading from channel, socket has likely been closed. at kafka.utils.Utils$.read(Utils.scala:381) at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54) at kafka.network.Receive$class.readCompletely(Transmission.scala:56) at kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29) at kafka.network.BlockingChannel.receive(BlockingChannel.scala:108) at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:146) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) [2015-01-09 16:15:27,156] ERROR [Controller-0-to-broker-0-send-thread], Controller 0 epoch 6 failed to send request Name:UpdateMetadataRequest;Version:0;Controller:0;ControllerEpoch:6;CorrelationId:48;ClientId:id_0-host_192.168.10.21-port_9092;AliveBrokers:id:0,host:192.168.10.21,port:9092;PartitionState:[40963064-cdd2-4cd1-937a-9827d3ab77ad,0] - (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:6),ReplicationFactor:1),AllReplicas:0) to broker id:0,host:192.168.10.21,port:9092. Reconnecting to broker. (kafka.controller.RequestSendThread) java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:97) at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132) at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29751: Patch for kafka-1851
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29751/#review67476 --- Ship it! Ship It! - Neha Narkhede On Jan. 9, 2015, 2:53 a.m., Jun Rao wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29751/ --- (Updated Jan. 9, 2015, 2:53 a.m.) Review request for kafka. Bugs: kafka-1851 https://issues.apache.org/jira/browse/kafka-1851 Repository: kafka Description --- fix the behavior of OffsetFetchRequest on unknown partition Diffs - core/src/main/scala/kafka/server/KafkaApis.scala 2f009920af37de3cf0a3eb131f2124f4e532c4e4 core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala 8c5364fa97da1be09973c176d1baeb339455d319 Diff: https://reviews.apache.org/r/29751/diff/ Testing --- Thanks, Jun Rao
[jira] [Commented] (KAFKA-1851) OffsetFetchRequest returns extra partitions when input only contains unknown partitions
[ https://issues.apache.org/jira/browse/KAFKA-1851?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271670#comment-14271670 ] Jun Rao commented on KAFKA-1851: Because it doesn't match the spec of the protocol. If you pass in one partition in the request, you are expected to get only one partition in the response. OffsetFetchRequest returns extra partitions when input only contains unknown partitions --- Key: KAFKA-1851 URL: https://issues.apache.org/jira/browse/KAFKA-1851 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Assignee: Jun Rao Priority: Blocker Fix For: 0.8.2 Attachments: kafka-1851.patch When issuing an OffsetFetchRequest with an unknown topic partition, the OffsetFetchResponse unexpectedly returns all partitions in the same consumer group, in addition to the unknown partition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29756: Patch for KAFKA-1854
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/#review67477 --- kafka-patch-review.py https://reviews.apache.org/r/29756/#comment111521 I got the following error on this patch nnarkhed-mn1:kafka nnarkhed$ python kafka-patch-review.py -b trunk -j KAFKA-1854 -d test Configuring reviewboard url to https://reviews.apache.org Updating your remote branches to pull the latest changes Verifying JIRA connection configurations JIRA user :nehanarkhede JIRA password : Failed to login to the JIRA instance type 'exceptions.AttributeError' 'JIRA' object has no attribute 'current_user' Maybe a different version of the jira package we use renamed the user field ? - Neha Narkhede On Jan. 9, 2015, 12:47 p.m., Jaikiran Pai wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/ --- (Updated Jan. 9, 2015, 12:47 p.m.) Review request for kafka. Bugs: KAFKA-1854 https://issues.apache.org/jira/browse/KAFKA-1854 Repository: kafka Description --- KAFKA-1854 Allow JIRA username and password to be prompted in the absence of a jira.ini file, during patch submission Diffs - kafka-patch-review.py b7f132f9d210b8648859ab8f9c89f30ec128ab38 Diff: https://reviews.apache.org/r/29756/diff/ Testing --- Thanks, Jaikiran Pai
[jira] [Commented] (KAFKA-1853) Unsuccessful suffix rename attempt of LogSegment can leak files and also leave the LogSegment in an invalid state
[
https://issues.apache.org/jira/browse/KAFKA-1853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271737#comment-14271737
]
Jay Kreps commented on KAFKA-1853:
--
Hey [~jaikiran] I am not sure about this change. The file channel should be
tied to a file descriptor with one or more file names associated with it. One
confusing thing is that java File isn't a file, just a file name (should really
be called Path). Changing the file name should have no impact on the file
channel at least in my mind. Maybe I am wrong about this?
I think there is potentially a bug in that method in that I think it should be
{code}
def renameTo(f: File): Boolean = {
val success = this.file.renameTo(f)
if(success)
this.file = f
success
}
{code}
Unsuccessful suffix rename attempt of LogSegment can leak files and also
leave the LogSegment in an invalid state
-
Key: KAFKA-1853
URL: https://issues.apache.org/jira/browse/KAFKA-1853
Project: Kafka
Issue Type: Bug
Components: core
Affects Versions: 0.8.1.1
Reporter: jaikiran pai
Fix For: 0.8.3
As noted in this discussion in the user mailing list
http://mail-archives.apache.org/mod_mbox/kafka-users/201501.mbox/%3C54AE3661.8080007%40gmail.com%3E
an unsuccessful attempt at renaming the underlying files of a LogSegment can
lead to file leaks and also leave the LogSegment in an invalid state.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1819) Cleaner gets confused about deleted and re-created topics
[
https://issues.apache.org/jira/browse/KAFKA-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271612#comment-14271612
]
Neha Narkhede edited comment on KAFKA-1819 at 1/9/15 6:06 PM:
--
[~gwenshap] Saw the following unit test failures on the latest patch on 0.8.2
branch, while trying to check it in -
{code}
kafka.server.ServerShutdownTest testCleanShutdownWithDeleteTopicEnabled FAILED
junit.framework.AssertionFailedError: expected:0 but was:1
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.failNotEquals(Assert.java:277)
at junit.framework.Assert.assertEquals(Assert.java:64)
at junit.framework.Assert.assertEquals(Assert.java:195)
at junit.framework.Assert.assertEquals(Assert.java:201)
at
kafka.server.ServerShutdownTest.verifyNonDaemonThreadsStatus(ServerShutdownTest.scala:145)
at
kafka.server.ServerShutdownTest.testCleanShutdownWithDeleteTopicEnabled(ServerShutdownTest.scala:114)
kafka.server.ServerShutdownTest testCleanShutdownAfterFailedStartup FAILED
junit.framework.AssertionFailedError: expected:0 but was:1
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.failNotEquals(Assert.java:277)
at junit.framework.Assert.assertEquals(Assert.java:64)
at junit.framework.Assert.assertEquals(Assert.java:195)
at junit.framework.Assert.assertEquals(Assert.java:201)
at
kafka.server.ServerShutdownTest.verifyNonDaemonThreadsStatus(ServerShutdownTest.scala:145)
at
kafka.server.ServerShutdownTest.testCleanShutdownAfterFailedStartup(ServerShutdownTest.scala:141)
kafka.server.ServerShutdownTest testCleanShutdown FAILED
junit.framework.AssertionFailedError: expected:0 but was:1
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.failNotEquals(Assert.java:277)
at junit.framework.Assert.assertEquals(Assert.java:64)
at junit.framework.Assert.assertEquals(Assert.java:195)
at junit.framework.Assert.assertEquals(Assert.java:201)
at
kafka.server.ServerShutdownTest.verifyNonDaemonThreadsStatus(ServerShutdownTest.scala:145)
at
kafka.server.ServerShutdownTest.testCleanShutdown(ServerShutdownTest.scala:101)
{code}
was (Author: nehanarkhede):
[~gwenshap] Saw the following unit test failures on the latest patch, while
trying to check it in -
{code}
kafka.server.ServerShutdownTest testCleanShutdownWithDeleteTopicEnabled FAILED
junit.framework.AssertionFailedError: expected:0 but was:1
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.failNotEquals(Assert.java:277)
at junit.framework.Assert.assertEquals(Assert.java:64)
at junit.framework.Assert.assertEquals(Assert.java:195)
at junit.framework.Assert.assertEquals(Assert.java:201)
at
kafka.server.ServerShutdownTest.verifyNonDaemonThreadsStatus(ServerShutdownTest.scala:145)
at
kafka.server.ServerShutdownTest.testCleanShutdownWithDeleteTopicEnabled(ServerShutdownTest.scala:114)
kafka.server.ServerShutdownTest testCleanShutdownAfterFailedStartup FAILED
junit.framework.AssertionFailedError: expected:0 but was:1
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.failNotEquals(Assert.java:277)
at junit.framework.Assert.assertEquals(Assert.java:64)
at junit.framework.Assert.assertEquals(Assert.java:195)
at junit.framework.Assert.assertEquals(Assert.java:201)
at
kafka.server.ServerShutdownTest.verifyNonDaemonThreadsStatus(ServerShutdownTest.scala:145)
at
kafka.server.ServerShutdownTest.testCleanShutdownAfterFailedStartup(ServerShutdownTest.scala:141)
kafka.server.ServerShutdownTest testCleanShutdown FAILED
junit.framework.AssertionFailedError: expected:0 but was:1
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.failNotEquals(Assert.java:277)
at junit.framework.Assert.assertEquals(Assert.java:64)
at junit.framework.Assert.assertEquals(Assert.java:195)
at junit.framework.Assert.assertEquals(Assert.java:201)
at
kafka.server.ServerShutdownTest.verifyNonDaemonThreadsStatus(ServerShutdownTest.scala:145)
at
kafka.server.ServerShutdownTest.testCleanShutdown(ServerShutdownTest.scala:101)
{code}
Cleaner gets confused about deleted and re-created topics
-
Key: KAFKA-1819
URL: https://issues.apache.org/jira/browse/KAFKA-1819
Project: Kafka
Issue Type: Bug
Reporter: Gian Merlino
Assignee: Gwen Shapira
Priority: Blocker
Fix For: 0.8.2
Attachments: KAFKA-1819.patch, KAFKA-1819_2014-12-26_13:58:44.patch,
Re: Review Request 29523: Patch for KAFKA-1723
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29523/ --- (Updated Jan. 9, 2015, 6:15 p.m.) Review request for kafka. Bugs: KAFKA-1723 https://issues.apache.org/jira/browse/KAFKA-1723 Repository: kafka Description --- Standard JMX MBean Naming is implemented;Addresing Jun's comments Diffs (updated) - build.gradle ba52288031e2abc70e35e9297a4423dd5025950b clients/src/main/java/org/apache/kafka/clients/consumer/Consumer.java 1bce50185273dbdbc131fbc9c7f5f3e9c346517a clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 7f8a41c4bf437711685a8271a4d3c83a176dd957 clients/src/main/java/org/apache/kafka/clients/consumer/MockConsumer.java 8cab16c0a0bdb671fea1fc2fc2694247f66cc971 clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 3053f2745c8e5f6e3b75826d3749656f150878db clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 5baa6062bd9ba8a7d38058856ed2d831fae491f0 clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java aa91e1444a49c55870b9a7a32086fa2b04471fba clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java c15485d1af304ef53691d478f113f332fe67af77 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 84a7a07269c51ccc22ebb4ff9797292d07ba778e clients/src/main/java/org/apache/kafka/common/Metric.java b023e8e7c486adf21ed9a554085ab8ad7f3ee038 clients/src/main/java/org/apache/kafka/common/metrics/CompoundStat.java 29185a6a90d0035d650c7e56ce612a0878cb115c clients/src/main/java/org/apache/kafka/common/metrics/JmxReporter.java 3c312011a7ff7e79c277a89048e7e62ebd6078db clients/src/main/java/org/apache/kafka/common/metrics/KafkaMetric.java a7458b50cb16fbb2b31b857d5b359e65258bbf08 clients/src/main/java/org/apache/kafka/common/metrics/MetricName.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java 49be4019ac03835701c49646920766228ac7ffe9 clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 25c1faf2887ea02708c1f5b5f822f5299ed86bd6 clients/src/main/java/org/apache/kafka/common/metrics/stats/Percentile.java 7365ceb39072a6a1ecf533f5a20830ed1f2cfc72 clients/src/main/java/org/apache/kafka/common/metrics/stats/Percentiles.java c70d577ada8c099533d4f4ed2e86d37e0a6e6676 clients/src/main/java/org/apache/kafka/common/network/Selector.java 4dd2cdf773f7eb01a93d7f994383088960303dfc clients/src/test/java/org/apache/kafka/clients/producer/BufferPoolTest.java fe3c13f319d48b89a4f26b6d78c2c3e31cc50d7e clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java 2c9932401d573549c40f16fda8c4e3e11309cb85 clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java ef2ca65cabe97b909f17b62027a1bb06827e88fe clients/src/test/java/org/apache/kafka/common/metrics/JmxReporterTest.java 2f43c49450e1a3d671bd17417dc42941f1858750 clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java 19bea0f1fa1ebf15d86623015ec909b0155e4bd3 clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 5c5e3d40819e41cab7b52a0eeaee5f2e7317b7b3 clients/src/test/java/org/apache/kafka/test/MetricsBench.java 9d98c1148255455fd801043b59b98fed9d0b76b3 Diff: https://reviews.apache.org/r/29523/diff/ Testing --- Thanks, Manikumar Reddy O
Re: Review Request 29523: Patch for KAFKA-1723
On Jan. 8, 2015, 9:21 p.m., Jun Rao wrote: clients/src/main/java/org/apache/kafka/common/network/Selector.java, line 419 https://reviews.apache.org/r/29523/diff/20/?file=813232#file813232line419 It's probably better to just reference a ProducerMetrics constant. Selector is a low level utility. It can be used in new producer , consumer or etc. So instead of hard-coding, metricGrpName is passed as an argument On Jan. 8, 2015, 9:21 p.m., Jun Rao wrote: clients/src/main/java/org/apache/kafka/common/network/Selector.java, line 483 https://reviews.apache.org/r/29523/diff/20/?file=813232#file813232line483 Instead of node-+node, it probably can just be node. not sure.. node is a integer value (0,1). I felt node-0, node-1 are more informative, than simple 0, 1. Jconsole MBean Tree view looks like this kafka.producer -- ProducerNodeMetrics -- node-0 -- metric-1 -- node-1 -- metric-1 Is it Ok? If required i can change this? On Jan. 8, 2015, 9:21 p.m., Jun Rao wrote: clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java, line 336 https://reviews.apache.org/r/29523/diff/20/?file=813222#file813222line336 Should this be passed in? Or perhaps we can define this as a constant and reference it here. Sender is internal to producer.. So the group name is hard-coded. - Manikumar Reddy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29523/#review67246 --- On Jan. 9, 2015, 6:15 p.m., Manikumar Reddy O wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29523/ --- (Updated Jan. 9, 2015, 6:15 p.m.) Review request for kafka. Bugs: KAFKA-1723 https://issues.apache.org/jira/browse/KAFKA-1723 Repository: kafka Description --- Standard JMX MBean Naming is implemented;Addresing Jun's comments Diffs - build.gradle ba52288031e2abc70e35e9297a4423dd5025950b clients/src/main/java/org/apache/kafka/clients/consumer/Consumer.java 1bce50185273dbdbc131fbc9c7f5f3e9c346517a clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 7f8a41c4bf437711685a8271a4d3c83a176dd957 clients/src/main/java/org/apache/kafka/clients/consumer/MockConsumer.java 8cab16c0a0bdb671fea1fc2fc2694247f66cc971 clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 3053f2745c8e5f6e3b75826d3749656f150878db clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 5baa6062bd9ba8a7d38058856ed2d831fae491f0 clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java aa91e1444a49c55870b9a7a32086fa2b04471fba clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java c15485d1af304ef53691d478f113f332fe67af77 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 84a7a07269c51ccc22ebb4ff9797292d07ba778e clients/src/main/java/org/apache/kafka/common/Metric.java b023e8e7c486adf21ed9a554085ab8ad7f3ee038 clients/src/main/java/org/apache/kafka/common/metrics/CompoundStat.java 29185a6a90d0035d650c7e56ce612a0878cb115c clients/src/main/java/org/apache/kafka/common/metrics/JmxReporter.java 3c312011a7ff7e79c277a89048e7e62ebd6078db clients/src/main/java/org/apache/kafka/common/metrics/KafkaMetric.java a7458b50cb16fbb2b31b857d5b359e65258bbf08 clients/src/main/java/org/apache/kafka/common/metrics/MetricName.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java 49be4019ac03835701c49646920766228ac7ffe9 clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 25c1faf2887ea02708c1f5b5f822f5299ed86bd6 clients/src/main/java/org/apache/kafka/common/metrics/stats/Percentile.java 7365ceb39072a6a1ecf533f5a20830ed1f2cfc72 clients/src/main/java/org/apache/kafka/common/metrics/stats/Percentiles.java c70d577ada8c099533d4f4ed2e86d37e0a6e6676 clients/src/main/java/org/apache/kafka/common/network/Selector.java 4dd2cdf773f7eb01a93d7f994383088960303dfc clients/src/test/java/org/apache/kafka/clients/producer/BufferPoolTest.java fe3c13f319d48b89a4f26b6d78c2c3e31cc50d7e clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java 2c9932401d573549c40f16fda8c4e3e11309cb85 clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java ef2ca65cabe97b909f17b62027a1bb06827e88fe clients/src/test/java/org/apache/kafka/common/metrics/JmxReporterTest.java 2f43c49450e1a3d671bd17417dc42941f1858750 clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java
Re: Review Request 29523: Patch for KAFKA-1723
On Jan. 8, 2015, 10:28 p.m., Jay Kreps wrote: clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java, line 115 https://reviews.apache.org/r/29523/diff/20/?file=813229#file813229line115 This won't be enough to distinguish the metric, right? Absent the group/tags info... I have implemented toString() method in MetricName. So this will print complete info about the metric name (name , group, description , tags) - Manikumar Reddy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29523/#review67314 --- On Jan. 9, 2015, 6:15 p.m., Manikumar Reddy O wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29523/ --- (Updated Jan. 9, 2015, 6:15 p.m.) Review request for kafka. Bugs: KAFKA-1723 https://issues.apache.org/jira/browse/KAFKA-1723 Repository: kafka Description --- Standard JMX MBean Naming is implemented;Addresing Jun's comments Diffs - build.gradle ba52288031e2abc70e35e9297a4423dd5025950b clients/src/main/java/org/apache/kafka/clients/consumer/Consumer.java 1bce50185273dbdbc131fbc9c7f5f3e9c346517a clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 7f8a41c4bf437711685a8271a4d3c83a176dd957 clients/src/main/java/org/apache/kafka/clients/consumer/MockConsumer.java 8cab16c0a0bdb671fea1fc2fc2694247f66cc971 clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 3053f2745c8e5f6e3b75826d3749656f150878db clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 5baa6062bd9ba8a7d38058856ed2d831fae491f0 clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java aa91e1444a49c55870b9a7a32086fa2b04471fba clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java c15485d1af304ef53691d478f113f332fe67af77 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 84a7a07269c51ccc22ebb4ff9797292d07ba778e clients/src/main/java/org/apache/kafka/common/Metric.java b023e8e7c486adf21ed9a554085ab8ad7f3ee038 clients/src/main/java/org/apache/kafka/common/metrics/CompoundStat.java 29185a6a90d0035d650c7e56ce612a0878cb115c clients/src/main/java/org/apache/kafka/common/metrics/JmxReporter.java 3c312011a7ff7e79c277a89048e7e62ebd6078db clients/src/main/java/org/apache/kafka/common/metrics/KafkaMetric.java a7458b50cb16fbb2b31b857d5b359e65258bbf08 clients/src/main/java/org/apache/kafka/common/metrics/MetricName.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java 49be4019ac03835701c49646920766228ac7ffe9 clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 25c1faf2887ea02708c1f5b5f822f5299ed86bd6 clients/src/main/java/org/apache/kafka/common/metrics/stats/Percentile.java 7365ceb39072a6a1ecf533f5a20830ed1f2cfc72 clients/src/main/java/org/apache/kafka/common/metrics/stats/Percentiles.java c70d577ada8c099533d4f4ed2e86d37e0a6e6676 clients/src/main/java/org/apache/kafka/common/network/Selector.java 4dd2cdf773f7eb01a93d7f994383088960303dfc clients/src/test/java/org/apache/kafka/clients/producer/BufferPoolTest.java fe3c13f319d48b89a4f26b6d78c2c3e31cc50d7e clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java 2c9932401d573549c40f16fda8c4e3e11309cb85 clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java ef2ca65cabe97b909f17b62027a1bb06827e88fe clients/src/test/java/org/apache/kafka/common/metrics/JmxReporterTest.java 2f43c49450e1a3d671bd17417dc42941f1858750 clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java 19bea0f1fa1ebf15d86623015ec909b0155e4bd3 clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 5c5e3d40819e41cab7b52a0eeaee5f2e7317b7b3 clients/src/test/java/org/apache/kafka/test/MetricsBench.java 9d98c1148255455fd801043b59b98fed9d0b76b3 Diff: https://reviews.apache.org/r/29523/diff/ Testing --- Thanks, Manikumar Reddy O
[jira] [Commented] (KAFKA-1723) make the metrics name in new producer more standard
[
https://issues.apache.org/jira/browse/KAFKA-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271663#comment-14271663
]
Manikumar Reddy commented on KAFKA-1723:
1. Submitted new patch with suggested changes.
2. producer-metrics, producer-node-metrics, producer-topic-metrics are used for
group names
Jconsole MBean hierachy will look like this
kafka.producer -- producer-metrics -- producer-1 -- metrics
kafka.producer -- producer-node-metrics -- producer-1 -- node-0 -- metrics
kafka.producer -- producer-topic-metrics -- my-topic -- metrics
{quote}
Another question: How are you supposed to make use of the .metrics() api on the
producer and consumer now?
{quote}
I am still thinking about this. We can implement criteria based metric
retrieval or predicate based retrieval..
make the metrics name in new producer more standard
---
Key: KAFKA-1723
URL: https://issues.apache.org/jira/browse/KAFKA-1723
Project: Kafka
Issue Type: Sub-task
Components: clients
Affects Versions: 0.8.2
Reporter: Jun Rao
Assignee: Manikumar Reddy
Priority: Blocker
Fix For: 0.8.2
Attachments: KAFKA-1723.patch, KAFKA-1723_2015-01-08_21:41:13.patch,
KAFKA-1723_2015-01-08_22:02:22.patch, KAFKA-1723_2015-01-09_14:24:18.patch,
KAFKA-1723_2015-01-09_23:43:22.patch
The jmx name in the new producer looks like the following:
kafka.producer.myclientid:type=mytopic
However, this can be ambiguous since we allow . in client id and topic.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 29756: Patch for KAFKA-1854
On Jan. 9, 2015, 11:07 a.m., Manikumar Reddy O wrote: kafka-patch-review.py, line 96 https://reviews.apache.org/r/29756/diff/3/?file=814216#file814216line96 Did you tested the failure senario? I did not get error message. I think we will get exception only after invoking jira.issue(). Jaikiran Pai wrote: I did test a failure case, yes. Gave an incorrect username and password and that failed during login and did not proceed further. Here's the relevant output (notice that I intentionally use JIRA user name foo): ` Configuring reviewboard url to https://reviews.apache.org Updating your remote branches to pull the latest changes Verifying JIRA connection configurations JIRA user :foo JIRA password : Failed to login to the JIRA instance class 'jira.exceptions.JIRAError' HTTP 401: html head titleUnauthorized (401)/title ` Jaikiran Pai wrote: By the way, if you are testing this, make sure you don't have a jira.ini in its usual place. Else the (correct) credentials in there will be used as usual and you won't be able to replicate a failing scenario. Manikumar Reddy O wrote: Configuring reviewboard url to https://reviews.apache.org Updating your remote branches to pull the latest changes Verifying JIRA connection configurations JIRA user :foo JIRA password : Creating diff against origin/0.8.2 and uploading patch to JIRA KAFKA-1723 I am not getting exception, it going to next line of execution. It is failing at jira.issue(). Version : jira-python 0.16 Jaikiran Pai wrote: That's interesting. Let me see what might be different. Jaikiran Pai wrote: Turns out the version of jira python that I had was 0.31 and behaved differently. I downgraded to 0.16 and was able to replicate what you are seeing. I've updated the patch to fix this and it should now correctly fail if wrong JIRA credentials are specified, even before it attempts to publish to reviewboard. Let me know if this now works for you too. Thanks for testing this! Latest patch is working with jira-python 0.16. Some members may be using older versions. We need to find correct API which works across all the jira-python versions. (or) we need to add minimum version requirement. - Manikumar Reddy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/#review67414 --- On Jan. 9, 2015, 12:47 p.m., Jaikiran Pai wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/ --- (Updated Jan. 9, 2015, 12:47 p.m.) Review request for kafka. Bugs: KAFKA-1854 https://issues.apache.org/jira/browse/KAFKA-1854 Repository: kafka Description --- KAFKA-1854 Allow JIRA username and password to be prompted in the absence of a jira.ini file, during patch submission Diffs - kafka-patch-review.py b7f132f9d210b8648859ab8f9c89f30ec128ab38 Diff: https://reviews.apache.org/r/29756/diff/ Testing --- Thanks, Jaikiran Pai
[jira] [Updated] (KAFKA-1842) New producer/consumer should support configurable connection timeouts
[ https://issues.apache.org/jira/browse/KAFKA-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1842: - Component/s: config New producer/consumer should support configurable connection timeouts - Key: KAFKA-1842 URL: https://issues.apache.org/jira/browse/KAFKA-1842 Project: Kafka Issue Type: Bug Components: clients, config Affects Versions: 0.8.2 Reporter: Ewen Cheslack-Postava Fix For: 0.8.3 During discussion of KAFKA-1642 it became clear that the current connection handling code for the new clients doesn't give enough flexibility in some failure cases. We need to support connection timeouts that are configurable via Kafka configs rather than relying on the underlying TCP stack's default settings. This would give the user control over how aggressively they want to try new servers when trying to fetch metadata (currently dependent on the underlying OS timeouts and some implementation details of NetworkClient.maybeUpdateMetadata and NetworkClient.leastLoadedNode), which is the specific issue that came up in KAFKA-1642. More generally it gives better control over how fast the user sees failures when there are network failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1842) New producer/consumer should support configurable connection timeouts
[ https://issues.apache.org/jira/browse/KAFKA-1842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1842: - Fix Version/s: 0.8.3 New producer/consumer should support configurable connection timeouts - Key: KAFKA-1842 URL: https://issues.apache.org/jira/browse/KAFKA-1842 Project: Kafka Issue Type: Bug Components: clients, config Affects Versions: 0.8.2 Reporter: Ewen Cheslack-Postava Fix For: 0.8.3 During discussion of KAFKA-1642 it became clear that the current connection handling code for the new clients doesn't give enough flexibility in some failure cases. We need to support connection timeouts that are configurable via Kafka configs rather than relying on the underlying TCP stack's default settings. This would give the user control over how aggressively they want to try new servers when trying to fetch metadata (currently dependent on the underlying OS timeouts and some implementation details of NetworkClient.maybeUpdateMetadata and NetworkClient.leastLoadedNode), which is the specific issue that came up in KAFKA-1642. More generally it gives better control over how fast the user sees failures when there are network failures. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1797) add the serializer/deserializer api to the new java client
[ https://issues.apache.org/jira/browse/KAFKA-1797?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271749#comment-14271749 ] Jun Rao commented on KAFKA-1797: Thanks for the reviews. Committed changes suggested by Jay and Manikumar to both 0.8.2 and trunk. add the serializer/deserializer api to the new java client -- Key: KAFKA-1797 URL: https://issues.apache.org/jira/browse/KAFKA-1797 Project: Kafka Issue Type: Improvement Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Assignee: Jun Rao Fix For: 0.8.2 Attachments: kafka-1797.patch, kafka-1797.patch, kafka-1797.patch, kafka-1797.patch, kafka-1797_2014-12-09_18:48:33.patch, kafka-1797_2014-12-15_15:36:24.patch, kafka-1797_2014-12-17_09:47:45.patch Currently, the new java clients take a byte array for both the key and the value. While this api is simple, it pushes the serialization/deserialization logic into the application. This makes it hard to reason about what type of data flows through Kafka and also makes it hard to share an implementation of the serializer/deserializer. For example, to support Avro, the serialization logic could be quite involved since it might need to register the Avro schema in some remote registry and maintain a schema cache locally, etc. Without a serialization api, it's impossible to share such an implementation so that people can easily reuse. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29752: Patch for KAFKA-1836
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29752/#review67457 --- Ship it! One minor nit, but looks good to me. Non-binding +1. clients/src/main/java/org/apache/kafka/clients/producer/internals/Metadata.java https://reviews.apache.org/r/29752/#comment111484 This works since the check farther down ensures remainingWaitMs is never 0 unless maxWaitMs is 0, but it's a bit confusing when reading this code why we check maxWaitMs instead of remainingWaitMs. There's no functional difference, but I think checking remainingWaitMs instead would be clearer for someone reading this code. - Ewen Cheslack-Postava On Jan. 9, 2015, 3:59 a.m., Jaikiran Pai wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29752/ --- (Updated Jan. 9, 2015, 3:59 a.m.) Review request for kafka. Bugs: KAFKA-1836 https://issues.apache.org/jira/browse/KAFKA-1836 Repository: kafka Description --- KAFKA-1836 Don't block forever if metadata.fetch.timeout.ms is set to 0 Diffs - clients/src/main/java/org/apache/kafka/clients/producer/internals/Metadata.java 1d30f9edd95337f86e632a09fc8f4126a67c238b clients/src/test/java/org/apache/kafka/clients/producer/MetadataTest.java 4547bfcb44be4d72742076d6e93f424b3b22a7a9 Diff: https://reviews.apache.org/r/29752/diff/ Testing --- Thanks, Jaikiran Pai
[jira] [Created] (KAFKA-1855) Topic unreadable after unsuccessful UpdateMetadataRequest
Henri Pihkala created KAFKA-1855: Summary: Topic unreadable after unsuccessful UpdateMetadataRequest Key: KAFKA-1855 URL: https://issues.apache.org/jira/browse/KAFKA-1855 Project: Kafka Issue Type: Bug Components: controller Affects Versions: 0.8.2 Reporter: Henri Pihkala Assignee: Neha Narkhede Sometimes, seemingly randomly, topic creation/initialization might fail with the following lines in controller.log. Other logs show no errors. When this happens, the topic is unusable (gives UnknownTopicOrPartition for all requests). For me this happens 5-10% of the time. Feels like it's more likely to happen if there is time between topic creations. Observed on 0.8.2-beta, have not tried previous versions. [2015-01-09 16:15:27,153] WARN [Controller-0-to-broker-0-send-thread], Controller 0 fails to send a request to broker id:0,host:192.168.10.21,port:9092 (kafka.controller.RequestSendThread) java.io.EOFException: Received -1 when reading from channel, socket has likely been closed. at kafka.utils.Utils$.read(Utils.scala:381) at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54) at kafka.network.Receive$class.readCompletely(Transmission.scala:56) at kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29) at kafka.network.BlockingChannel.receive(BlockingChannel.scala:108) at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:146) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) [2015-01-09 16:15:27,156] ERROR [Controller-0-to-broker-0-send-thread], Controller 0 epoch 6 failed to send request Name:UpdateMetadataRequest;Version:0;Controller:0;ControllerEpoch:6;CorrelationId:48;ClientId:id_0-host_192.168.10.21-port_9092;AliveBrokers:id:0,host:192.168.10.21,port:9092;PartitionState:[40963064-cdd2-4cd1-937a-9827d3ab77ad,0] - (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:6),ReplicationFactor:1),AllReplicas:0) to broker id:0,host:192.168.10.21,port:9092. Reconnecting to broker. (kafka.controller.RequestSendThread) java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:97) at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132) at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1855) Topic unusable after unsuccessful UpdateMetadataRequest
[ https://issues.apache.org/jira/browse/KAFKA-1855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Henri Pihkala updated KAFKA-1855: - Summary: Topic unusable after unsuccessful UpdateMetadataRequest (was: Topic unreadable after unsuccessful UpdateMetadataRequest) Topic unusable after unsuccessful UpdateMetadataRequest --- Key: KAFKA-1855 URL: https://issues.apache.org/jira/browse/KAFKA-1855 Project: Kafka Issue Type: Bug Components: controller Affects Versions: 0.8.2 Reporter: Henri Pihkala Assignee: Neha Narkhede Sometimes, seemingly randomly, topic creation/initialization might fail with the following lines in controller.log. Other logs show no errors. When this happens, the topic is unusable (gives UnknownTopicOrPartition for all requests). For me this happens 5-10% of the time. Feels like it's more likely to happen if there is time between topic creations. Observed on 0.8.2-beta, have not tried previous versions. [2015-01-09 16:15:27,153] WARN [Controller-0-to-broker-0-send-thread], Controller 0 fails to send a request to broker id:0,host:192.168.10.21,port:9092 (kafka.controller.RequestSendThread) java.io.EOFException: Received -1 when reading from channel, socket has likely been closed. at kafka.utils.Utils$.read(Utils.scala:381) at kafka.network.BoundedByteBufferReceive.readFrom(BoundedByteBufferReceive.scala:54) at kafka.network.Receive$class.readCompletely(Transmission.scala:56) at kafka.network.BoundedByteBufferReceive.readCompletely(BoundedByteBufferReceive.scala:29) at kafka.network.BlockingChannel.receive(BlockingChannel.scala:108) at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:146) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) [2015-01-09 16:15:27,156] ERROR [Controller-0-to-broker-0-send-thread], Controller 0 epoch 6 failed to send request Name:UpdateMetadataRequest;Version:0;Controller:0;ControllerEpoch:6;CorrelationId:48;ClientId:id_0-host_192.168.10.21-port_9092;AliveBrokers:id:0,host:192.168.10.21,port:9092;PartitionState:[40963064-cdd2-4cd1-937a-9827d3ab77ad,0] - (LeaderAndIsrInfo:(Leader:0,ISR:0,LeaderEpoch:0,ControllerEpoch:6),ReplicationFactor:1),AllReplicas:0) to broker id:0,host:192.168.10.21,port:9092. Reconnecting to broker. (kafka.controller.RequestSendThread) java.nio.channels.ClosedChannelException at kafka.network.BlockingChannel.send(BlockingChannel.scala:97) at kafka.controller.RequestSendThread.liftedTree1$1(ControllerChannelManager.scala:132) at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:131) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29738: Patch for kafka-1797
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29738/#review67458 --- Ship it! Ship It! - Neha Narkhede On Jan. 9, 2015, 1:22 a.m., Jun Rao wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29738/ --- (Updated Jan. 9, 2015, 1:22 a.m.) Review request for kafka. Bugs: kafka-1797 https://issues.apache.org/jira/browse/kafka-1797 Repository: kafka Description --- add null support in string serializer and deserializer Diffs - clients/src/main/java/org/apache/kafka/common/serialization/StringDeserializer.java a3b3700a1e0716643761d7032bd32bce839d3065 clients/src/main/java/org/apache/kafka/common/serialization/StringSerializer.java 02db47f8736988343dd293fc3da03751f78a3b5c clients/src/test/java/org/apache/kafka/common/serialization/SerializationTest.java d550a3137c066abb5e2984ac6245574832929ff8 Diff: https://reviews.apache.org/r/29738/diff/ Testing --- Thanks, Jun Rao
[jira] [Commented] (KAFKA-1819) Cleaner gets confused about deleted and re-created topics
[
https://issues.apache.org/jira/browse/KAFKA-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271612#comment-14271612
]
Neha Narkhede commented on KAFKA-1819:
--
[~gwenshap] Saw the following unit test failures on the latest patch, while
trying to check it in -
{code}
kafka.server.ServerShutdownTest testCleanShutdownWithDeleteTopicEnabled FAILED
junit.framework.AssertionFailedError: expected:0 but was:1
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.failNotEquals(Assert.java:277)
at junit.framework.Assert.assertEquals(Assert.java:64)
at junit.framework.Assert.assertEquals(Assert.java:195)
at junit.framework.Assert.assertEquals(Assert.java:201)
at
kafka.server.ServerShutdownTest.verifyNonDaemonThreadsStatus(ServerShutdownTest.scala:145)
at
kafka.server.ServerShutdownTest.testCleanShutdownWithDeleteTopicEnabled(ServerShutdownTest.scala:114)
kafka.server.ServerShutdownTest testCleanShutdownAfterFailedStartup FAILED
junit.framework.AssertionFailedError: expected:0 but was:1
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.failNotEquals(Assert.java:277)
at junit.framework.Assert.assertEquals(Assert.java:64)
at junit.framework.Assert.assertEquals(Assert.java:195)
at junit.framework.Assert.assertEquals(Assert.java:201)
at
kafka.server.ServerShutdownTest.verifyNonDaemonThreadsStatus(ServerShutdownTest.scala:145)
at
kafka.server.ServerShutdownTest.testCleanShutdownAfterFailedStartup(ServerShutdownTest.scala:141)
kafka.server.ServerShutdownTest testCleanShutdown FAILED
junit.framework.AssertionFailedError: expected:0 but was:1
at junit.framework.Assert.fail(Assert.java:47)
at junit.framework.Assert.failNotEquals(Assert.java:277)
at junit.framework.Assert.assertEquals(Assert.java:64)
at junit.framework.Assert.assertEquals(Assert.java:195)
at junit.framework.Assert.assertEquals(Assert.java:201)
at
kafka.server.ServerShutdownTest.verifyNonDaemonThreadsStatus(ServerShutdownTest.scala:145)
at
kafka.server.ServerShutdownTest.testCleanShutdown(ServerShutdownTest.scala:101)
{code}
Cleaner gets confused about deleted and re-created topics
-
Key: KAFKA-1819
URL: https://issues.apache.org/jira/browse/KAFKA-1819
Project: Kafka
Issue Type: Bug
Reporter: Gian Merlino
Assignee: Gwen Shapira
Priority: Blocker
Fix For: 0.8.2
Attachments: KAFKA-1819.patch, KAFKA-1819_2014-12-26_13:58:44.patch,
KAFKA-1819_2014-12-30_16:01:19.patch
I get an error like this after deleting a compacted topic and re-creating it.
I think it's because the brokers don't remove cleaning checkpoints from the
cleaner-offset-checkpoint file. This is from a build based off commit bd212b7.
java.lang.IllegalArgumentException: requirement failed: Last clean offset is
587607 but segment base offset is 0 for log foo-6.
at scala.Predef$.require(Predef.scala:233)
at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:502)
at kafka.log.Cleaner.clean(LogCleaner.scala:300)
at
kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:214)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:192)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (KAFKA-1786) implement a global configuration feature for brokers
[ https://issues.apache.org/jira/browse/KAFKA-1786?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1786: - Reviewer: Neha Narkhede implement a global configuration feature for brokers Key: KAFKA-1786 URL: https://issues.apache.org/jira/browse/KAFKA-1786 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein Assignee: Andrii Biletskyi Fix For: 0.8.3 Attachments: KAFKA-1786.patch Global level configurations (much like topic level) for brokers are managed by humans and automation systems through server.properties. Some configuration make sense to use default (like it is now) or override from central location (zookeeper for now). We can modify this through the new CLI tool so that every broker can have exact same setting. Some configurations we should allow to be overriden from server.properties (like port) but others we should use the global store as source of truth (e.g. auto topic enable, fetch replica message size, etc). Since most configuration I believe are going to fall into this category we should have the list of server.properties that can override the global config in the code in a list which we can manage... everything else the global takes precedence. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1786) implement a global configuration feature for brokers
[ https://issues.apache.org/jira/browse/KAFKA-1786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271770#comment-14271770 ] Neha Narkhede commented on KAFKA-1786: -- [~abiletskyi], [~charmalloc]. For large changes like this, we should circulate a proposal through the mailing list to get feedback from the other committers as well as the larger community. I'd prefer to have a detailed design proposal on our wiki and discuss that on the mailing list. implement a global configuration feature for brokers Key: KAFKA-1786 URL: https://issues.apache.org/jira/browse/KAFKA-1786 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein Assignee: Andrii Biletskyi Fix For: 0.8.3 Attachments: KAFKA-1786.patch Global level configurations (much like topic level) for brokers are managed by humans and automation systems through server.properties. Some configuration make sense to use default (like it is now) or override from central location (zookeeper for now). We can modify this through the new CLI tool so that every broker can have exact same setting. Some configurations we should allow to be overriden from server.properties (like port) but others we should use the global store as source of truth (e.g. auto topic enable, fetch replica message size, etc). Since most configuration I believe are going to fall into this category we should have the list of server.properties that can override the global config in the code in a list which we can manage... everything else the global takes precedence. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1753) add --decommission-broker option
[ https://issues.apache.org/jira/browse/KAFKA-1753?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271797#comment-14271797 ] Neha Narkhede commented on KAFKA-1753: -- [~joestein], KAFKA-1792 is waiting on a proposal on how the user will interact with the partition reassignment, given that we automated the assignment part. Once we have that, it will be easier to check it in. add --decommission-broker option Key: KAFKA-1753 URL: https://issues.apache.org/jira/browse/KAFKA-1753 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1819) Cleaner gets confused about deleted and re-created topics
[
https://issues.apache.org/jira/browse/KAFKA-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271919#comment-14271919
]
Gwen Shapira commented on KAFKA-1819:
-
I'm having trouble reproducing the errors:
{code}
gshapira-MBP:kafka082 gshapira$ ./gradlew -Dtest.single=ServerShutdownTest
core::test
To honour the JVM settings for this build a new JVM will be forked. Please
consider using the daemon:
http://gradle.org/docs/2.0/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.4
:clients:compileJava UP-TO-DATE
:clients:processResources UP-TO-DATE
:clients:classes UP-TO-DATE
:clients:jar UP-TO-DATE
:core:compileJava UP-TO-DATE
:core:compileScala UP-TO-DATE
:core:processResources UP-TO-DATE
:core:classes UP-TO-DATE
:core:compileTestJava UP-TO-DATE
:core:compileTestScala UP-TO-DATE
:core:processTestResources UP-TO-DATE
:core:testClasses UP-TO-DATE
:core:test
kafka.server.ServerShutdownTest testCleanShutdown PASSED
kafka.server.ServerShutdownTest testCleanShutdownWithDeleteTopicEnabled PASSED
kafka.server.ServerShutdownTest testCleanShutdownAfterFailedStartup PASSED
BUILD SUCCESSFUL
Total time: 8.504 secs
{code}
Cleaner gets confused about deleted and re-created topics
-
Key: KAFKA-1819
URL: https://issues.apache.org/jira/browse/KAFKA-1819
Project: Kafka
Issue Type: Bug
Reporter: Gian Merlino
Assignee: Gwen Shapira
Priority: Blocker
Fix For: 0.8.2
Attachments: KAFKA-1819.patch, KAFKA-1819_2014-12-26_13:58:44.patch,
KAFKA-1819_2014-12-30_16:01:19.patch
I get an error like this after deleting a compacted topic and re-creating it.
I think it's because the brokers don't remove cleaning checkpoints from the
cleaner-offset-checkpoint file. This is from a build based off commit bd212b7.
java.lang.IllegalArgumentException: requirement failed: Last clean offset is
587607 but segment base offset is 0 for log foo-6.
at scala.Predef$.require(Predef.scala:233)
at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:502)
at kafka.log.Cleaner.clean(LogCleaner.scala:300)
at
kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:214)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:192)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Review Request 29523: Patch for KAFKA-1723
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29523/ --- (Updated Jan. 9, 2015, 6:12 p.m.) Review request for kafka. Bugs: KAFKA-1723 https://issues.apache.org/jira/browse/KAFKA-1723 Repository: kafka Description --- Standard JMX MBean Naming is implemented;Addresing Jun's comments Diffs (updated) - build.gradle ba52288031e2abc70e35e9297a4423dd5025950b clients/src/main/java/org/apache/kafka/clients/consumer/Consumer.java 1bce50185273dbdbc131fbc9c7f5f3e9c346517a clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 7f8a41c4bf437711685a8271a4d3c83a176dd957 clients/src/main/java/org/apache/kafka/clients/consumer/MockConsumer.java 8cab16c0a0bdb671fea1fc2fc2694247f66cc971 clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 3053f2745c8e5f6e3b75826d3749656f150878db clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 5baa6062bd9ba8a7d38058856ed2d831fae491f0 clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java aa91e1444a49c55870b9a7a32086fa2b04471fba clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java c15485d1af304ef53691d478f113f332fe67af77 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 84a7a07269c51ccc22ebb4ff9797292d07ba778e clients/src/main/java/org/apache/kafka/common/Metric.java b023e8e7c486adf21ed9a554085ab8ad7f3ee038 clients/src/main/java/org/apache/kafka/common/metrics/CompoundStat.java 29185a6a90d0035d650c7e56ce612a0878cb115c clients/src/main/java/org/apache/kafka/common/metrics/JmxReporter.java 3c312011a7ff7e79c277a89048e7e62ebd6078db clients/src/main/java/org/apache/kafka/common/metrics/KafkaMetric.java a7458b50cb16fbb2b31b857d5b359e65258bbf08 clients/src/main/java/org/apache/kafka/common/metrics/MetricName.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java 49be4019ac03835701c49646920766228ac7ffe9 clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 25c1faf2887ea02708c1f5b5f822f5299ed86bd6 clients/src/main/java/org/apache/kafka/common/metrics/stats/Percentile.java 7365ceb39072a6a1ecf533f5a20830ed1f2cfc72 clients/src/main/java/org/apache/kafka/common/metrics/stats/Percentiles.java c70d577ada8c099533d4f4ed2e86d37e0a6e6676 clients/src/main/java/org/apache/kafka/common/network/Selector.java 4dd2cdf773f7eb01a93d7f994383088960303dfc clients/src/test/java/org/apache/kafka/clients/producer/BufferPoolTest.java fe3c13f319d48b89a4f26b6d78c2c3e31cc50d7e clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java 2c9932401d573549c40f16fda8c4e3e11309cb85 clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java ef2ca65cabe97b909f17b62027a1bb06827e88fe clients/src/test/java/org/apache/kafka/common/metrics/JmxReporterTest.java 2f43c49450e1a3d671bd17417dc42941f1858750 clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java 19bea0f1fa1ebf15d86623015ec909b0155e4bd3 clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 5c5e3d40819e41cab7b52a0eeaee5f2e7317b7b3 clients/src/test/java/org/apache/kafka/test/MetricsBench.java 9d98c1148255455fd801043b59b98fed9d0b76b3 Diff: https://reviews.apache.org/r/29523/diff/ Testing --- Thanks, Manikumar Reddy O
[jira] [Updated] (KAFKA-1723) make the metrics name in new producer more standard
[ https://issues.apache.org/jira/browse/KAFKA-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar Reddy updated KAFKA-1723: --- Attachment: KAFKA-1723_2015-01-09_23:43:22.patch make the metrics name in new producer more standard --- Key: KAFKA-1723 URL: https://issues.apache.org/jira/browse/KAFKA-1723 Project: Kafka Issue Type: Sub-task Components: clients Affects Versions: 0.8.2 Reporter: Jun Rao Assignee: Manikumar Reddy Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1723.patch, KAFKA-1723_2015-01-08_21:41:13.patch, KAFKA-1723_2015-01-08_22:02:22.patch, KAFKA-1723_2015-01-09_14:24:18.patch, KAFKA-1723_2015-01-09_23:43:22.patch The jmx name in the new producer looks like the following: kafka.producer.myclientid:type=mytopic However, this can be ambiguous since we allow . in client id and topic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1723) make the metrics name in new producer more standard
[ https://issues.apache.org/jira/browse/KAFKA-1723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271637#comment-14271637 ] Manikumar Reddy commented on KAFKA-1723: Updated reviewboard https://reviews.apache.org/r/29523/diff/ against branch origin/0.8.2 make the metrics name in new producer more standard --- Key: KAFKA-1723 URL: https://issues.apache.org/jira/browse/KAFKA-1723 Project: Kafka Issue Type: Sub-task Components: clients Affects Versions: 0.8.2 Reporter: Jun Rao Assignee: Manikumar Reddy Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1723.patch, KAFKA-1723_2015-01-08_21:41:13.patch, KAFKA-1723_2015-01-08_22:02:22.patch, KAFKA-1723_2015-01-09_14:24:18.patch, KAFKA-1723_2015-01-09_23:43:22.patch The jmx name in the new producer looks like the following: kafka.producer.myclientid:type=mytopic However, this can be ambiguous since we allow . in client id and topic. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1836) metadata.fetch.timeout.ms set to zero blocks forever
[ https://issues.apache.org/jira/browse/KAFKA-1836?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1836: - Reviewer: Neha Narkhede [~jaikiran] Will help you check this in after [~ewencp]'s comment is addressed. metadata.fetch.timeout.ms set to zero blocks forever Key: KAFKA-1836 URL: https://issues.apache.org/jira/browse/KAFKA-1836 Project: Kafka Issue Type: Bug Components: clients Affects Versions: 0.8.2 Reporter: Paul Pearcy Priority: Minor Labels: newbie Fix For: 0.8.3 Attachments: KAFKA-1836-new-patch.patch, KAFKA-1836.patch You can easily work around this by setting the timeout value to 1ms, but 0ms should mean 0ms or at least have the behavior documented. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1848) Checking shutdown during each iteration of ZookeeperConsumerConnector
[ https://issues.apache.org/jira/browse/KAFKA-1848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1848: - Reviewer: Guozhang Wang Checking shutdown during each iteration of ZookeeperConsumerConnector - Key: KAFKA-1848 URL: https://issues.apache.org/jira/browse/KAFKA-1848 Project: Kafka Issue Type: Bug Reporter: Aditya Auradkar Assignee: Aditya Auradkar Fix For: 0.9.0 In ZookeeperConsumerConnector the syncedRebalance() method checks the isShuttingDown flag before it triggers a rebalance. However, it does not recheck the same value between successive retries which is possible if the consumer is shutting down. This acquires the rebalanceLock and blocks shutdown from completing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29523: Patch for KAFKA-1723
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29523/#review67479 --- Ship it! Thanks for the patch. Checked the jmx and looks good to me. Will wait a bit to see if you come up with a better way of using the metrics through the api. - Jun Rao On Jan. 9, 2015, 6:15 p.m., Manikumar Reddy O wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29523/ --- (Updated Jan. 9, 2015, 6:15 p.m.) Review request for kafka. Bugs: KAFKA-1723 https://issues.apache.org/jira/browse/KAFKA-1723 Repository: kafka Description --- Standard JMX MBean Naming is implemented;Addresing Jun's comments Diffs - build.gradle ba52288031e2abc70e35e9297a4423dd5025950b clients/src/main/java/org/apache/kafka/clients/consumer/Consumer.java 1bce50185273dbdbc131fbc9c7f5f3e9c346517a clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 7f8a41c4bf437711685a8271a4d3c83a176dd957 clients/src/main/java/org/apache/kafka/clients/consumer/MockConsumer.java 8cab16c0a0bdb671fea1fc2fc2694247f66cc971 clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 3053f2745c8e5f6e3b75826d3749656f150878db clients/src/main/java/org/apache/kafka/clients/producer/Producer.java 5baa6062bd9ba8a7d38058856ed2d831fae491f0 clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java aa91e1444a49c55870b9a7a32086fa2b04471fba clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java c15485d1af304ef53691d478f113f332fe67af77 clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 84a7a07269c51ccc22ebb4ff9797292d07ba778e clients/src/main/java/org/apache/kafka/common/Metric.java b023e8e7c486adf21ed9a554085ab8ad7f3ee038 clients/src/main/java/org/apache/kafka/common/metrics/CompoundStat.java 29185a6a90d0035d650c7e56ce612a0878cb115c clients/src/main/java/org/apache/kafka/common/metrics/JmxReporter.java 3c312011a7ff7e79c277a89048e7e62ebd6078db clients/src/main/java/org/apache/kafka/common/metrics/KafkaMetric.java a7458b50cb16fbb2b31b857d5b359e65258bbf08 clients/src/main/java/org/apache/kafka/common/metrics/MetricName.java PRE-CREATION clients/src/main/java/org/apache/kafka/common/metrics/Metrics.java 49be4019ac03835701c49646920766228ac7ffe9 clients/src/main/java/org/apache/kafka/common/metrics/Sensor.java 25c1faf2887ea02708c1f5b5f822f5299ed86bd6 clients/src/main/java/org/apache/kafka/common/metrics/stats/Percentile.java 7365ceb39072a6a1ecf533f5a20830ed1f2cfc72 clients/src/main/java/org/apache/kafka/common/metrics/stats/Percentiles.java c70d577ada8c099533d4f4ed2e86d37e0a6e6676 clients/src/main/java/org/apache/kafka/common/network/Selector.java 4dd2cdf773f7eb01a93d7f994383088960303dfc clients/src/test/java/org/apache/kafka/clients/producer/BufferPoolTest.java fe3c13f319d48b89a4f26b6d78c2c3e31cc50d7e clients/src/test/java/org/apache/kafka/clients/producer/RecordAccumulatorTest.java 2c9932401d573549c40f16fda8c4e3e11309cb85 clients/src/test/java/org/apache/kafka/clients/producer/SenderTest.java ef2ca65cabe97b909f17b62027a1bb06827e88fe clients/src/test/java/org/apache/kafka/common/metrics/JmxReporterTest.java 2f43c49450e1a3d671bd17417dc42941f1858750 clients/src/test/java/org/apache/kafka/common/metrics/MetricsTest.java 19bea0f1fa1ebf15d86623015ec909b0155e4bd3 clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 5c5e3d40819e41cab7b52a0eeaee5f2e7317b7b3 clients/src/test/java/org/apache/kafka/test/MetricsBench.java 9d98c1148255455fd801043b59b98fed9d0b76b3 Diff: https://reviews.apache.org/r/29523/diff/ Testing --- Thanks, Manikumar Reddy O
Re: Review Request 29647: Patch for KAFKA-1697
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29647/#review67489 --- This seems like a small change, but it is not since it changes the behavior of a user facing API. I'd suggest you circulate this proposal through the dev mailing list to solicite feedback since the mailing list has more visibility than reviewboard. - Neha Narkhede On Jan. 7, 2015, 12:20 a.m., Gwen Shapira wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29647/ --- (Updated Jan. 7, 2015, 12:20 a.m.) Review request for kafka. Bugs: KAFKA-1697 https://issues.apache.org/jira/browse/KAFKA-1697 Repository: kafka Description --- removed broker code for handling acks1 and made NotEnoughReplicasAfterAppendException non-retriable Diffs - clients/src/main/java/org/apache/kafka/common/errors/NotEnoughReplicasAfterAppendException.java 75c80a97e43089cb3f924a38f86d67b5a8dd2b89 core/src/main/scala/kafka/cluster/Partition.scala b230e9a1fb1a3161f4c9d164e4890a16eceb2ad4 Diff: https://reviews.apache.org/r/29647/diff/ Testing --- Thanks, Gwen Shapira
[jira] [Updated] (KAFKA-1851) OffsetFetchRequest returns extra partitions when input only contains unknown partitions
[ https://issues.apache.org/jira/browse/KAFKA-1851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1851: --- Resolution: Fixed Status: Resolved (was: Patch Available) Thanks for the review. Committed to both 0.8.2 and trunk. OffsetFetchRequest returns extra partitions when input only contains unknown partitions --- Key: KAFKA-1851 URL: https://issues.apache.org/jira/browse/KAFKA-1851 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.2 Reporter: Jun Rao Assignee: Jun Rao Priority: Blocker Fix For: 0.8.2 Attachments: kafka-1851.patch When issuing an OffsetFetchRequest with an unknown topic partition, the OffsetFetchResponse unexpectedly returns all partitions in the same consumer group, in addition to the unknown partition. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1792: - Reviewer: Neha Narkhede change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments - Key: KAFKA-1792 URL: https://issues.apache.org/jira/browse/KAFKA-1792 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, generate_alg_tests.txt Current implementation produces fair replica distribution between specified list of brokers. Unfortunately, it doesn't take into account current replica assignment. So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth broker id=3, generate will create an assignment config which will redistribute replicas fairly across brokers [0..3] in the same way as those partitions were created from scratch. It will not take into consideration current replica assignment and accordingly will not try to minimize number of replica moves between brokers. As proposed by [~charmalloc] this should be improved. New output of improved --generate algorithm should suite following requirements: - fairness of replica distribution - every broker will have R or R+1 replicas assigned; - minimum of reassignments - number of replica moves between brokers will be minimal; Example. Consider following replica distribution per brokers [0..3] (we just added brokers 2 and 3): - broker - 0, 1, 2, 3 - replicas - 7, 6, 0, 0 The new algorithm will produce following assignment: - broker - 0, 1, 2, 3 - replicas - 4, 3, 3, 3 - moves - -3, -3, +3, +3 It will be fair and number of moves will be 6, which is minimal for specified initial distribution. The scope of this issue is: - design an algorithm matching the above requirements; - implement this algorithm and unit tests; - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1843) Metadata fetch/refresh in new producer should handle all node connection states gracefully
[ https://issues.apache.org/jira/browse/KAFKA-1843?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Neha Narkhede updated KAFKA-1843: - Component/s: producer Metadata fetch/refresh in new producer should handle all node connection states gracefully -- Key: KAFKA-1843 URL: https://issues.apache.org/jira/browse/KAFKA-1843 Project: Kafka Issue Type: Bug Components: clients, producer Affects Versions: 0.8.2 Reporter: Ewen Cheslack-Postava KAFKA-1642 resolved some issues with the handling of broker connection states to avoid high CPU usage, but made the minimal fix rather than the ideal one. The code for handling the metadata fetch is difficult to get right because it has to handle a lot of possible connectivity states and failure modes across all the known nodes. It also needs to correctly integrate with the surrounding event loop, providing correct poll() timeouts to both avoid busy looping and make sure it wakes up and tries new nodes in the face of both connection and request failures. A patch here should address a few issues: 1. Make sure connection timeouts, as implemented in KAFKA-1842, are cleanly integrated. This mostly means that when a connecting node is selected to fetch metadata from, that the code notices that and sets the next timeout based on the connection timeout rather than some other backoff. 2. Rethink the logic and naming of NetworkClient.leastLoadedNode. That method actually takes into account a) the current connectivity of each node, b) whether the node had a recent connection failure, c) the load in terms of in flight requests. It also needs to ensure that different clients don't use the same ordering across multiple calls (which is already addressed in the current code by nodeIndexOffset) and that we always eventually try all nodes in the face of connection failures (which isn't currently handled by leastLoadedNode and probably cannot be without tracking additional state). This method also has to work for new consumer use cases even though it is currently only used by the new producer's metadata fetch. Finally it has to properly handle when other code calls initiateConnect() since the normal path for sending messages also initiates connections. We can already say that there is an order of preference given a single call (as follows), but making this work across multiple calls when some initial choices fail to connect or return metadata *and* connection states may be changing is much more difficult. * Connected, zero in flight requests - the request can be sent immediately * Connecting node - it will hopefully be connected very soon and by definition has no in flight requests * Disconnected - same reasoning as for a connecting node * Connected, 0 in flight requests - we consider any # of in flight requests as a big enough backlog to delay the request a lot. We could use an approach that better accounts for # of in flight requests rather than just turning it into a boolean variable, but that probably introduces much more complexity than it is worth. 3. The most difficult case to handle so far has been when leastLoadedNode returns a disconnected node to maybeUpdateMetadata as its best option. Properly handling the two resulting cases (initiateConnect fails immediately vs. taking some time to possibly establish the connection) is tricky. 4. Consider optimizing for the failure cases. The most common cases are when you already have an active connection and can immediately get the metadata or you need to establish a connection, but the connection and metadata request/response happen very quickly. These common cases are infrequent enough (default every 5 min) that establishing an extra connection isn't a big deal as long as it's eventually cleaned up. The edge cases, like network partitions where some subset of nodes become unreachable for a long period, are harder to reason about but we should be sure we will always be able to gracefully recover from them. KAFKA-1642 enumerated the possible outcomes of a single call to maybeUpdateMetadata. A good fix for this would consider all of those outcomes for repeated calls to -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271791#comment-14271791 ] Neha Narkhede edited comment on KAFKA-1792 at 1/9/15 8:02 PM: -- [~Dmitry Pekar], [~charmalloc] Not sure if you are waiting on me to proceed here. I'm waiting to see a proposal for how the user will interact with partition reassignment given the new way to generate assignment. was (Author: nehanarkhede): [~Dmitry Pekar] Not sure if you are waiting on me to proceed here. I'm waiting to see a proposal for how the user will interact with partition reassignment given the new way to generate assignment. change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments - Key: KAFKA-1792 URL: https://issues.apache.org/jira/browse/KAFKA-1792 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, generate_alg_tests.txt Current implementation produces fair replica distribution between specified list of brokers. Unfortunately, it doesn't take into account current replica assignment. So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth broker id=3, generate will create an assignment config which will redistribute replicas fairly across brokers [0..3] in the same way as those partitions were created from scratch. It will not take into consideration current replica assignment and accordingly will not try to minimize number of replica moves between brokers. As proposed by [~charmalloc] this should be improved. New output of improved --generate algorithm should suite following requirements: - fairness of replica distribution - every broker will have R or R+1 replicas assigned; - minimum of reassignments - number of replica moves between brokers will be minimal; Example. Consider following replica distribution per brokers [0..3] (we just added brokers 2 and 3): - broker - 0, 1, 2, 3 - replicas - 7, 6, 0, 0 The new algorithm will produce following assignment: - broker - 0, 1, 2, 3 - replicas - 4, 3, 3, 3 - moves - -3, -3, +3, +3 It will be fair and number of moves will be 6, which is minimal for specified initial distribution. The scope of this issue is: - design an algorithm matching the above requirements; - implement this algorithm and unit tests; - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29751: Patch for kafka-1851
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29751/#review67499 --- core/src/main/scala/kafka/server/KafkaApis.scala https://reviews.apache.org/r/29751/#comment111536 You could just change getOffsets so it doesn't have this behavior instead of checking the condition here. It looks like this is the only place it's called from so nothing else should be relying on that behavior and it just simplifies the getOffsets code. - Ewen Cheslack-Postava On Jan. 9, 2015, 2:53 a.m., Jun Rao wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29751/ --- (Updated Jan. 9, 2015, 2:53 a.m.) Review request for kafka. Bugs: kafka-1851 https://issues.apache.org/jira/browse/kafka-1851 Repository: kafka Description --- fix the behavior of OffsetFetchRequest on unknown partition Diffs - core/src/main/scala/kafka/server/KafkaApis.scala 2f009920af37de3cf0a3eb131f2124f4e532c4e4 core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala 8c5364fa97da1be09973c176d1baeb339455d319 Diff: https://reviews.apache.org/r/29751/diff/ Testing --- Thanks, Jun Rao
[jira] [Commented] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271791#comment-14271791 ] Neha Narkhede commented on KAFKA-1792: -- [~Dmitry Pekar] Not sure if you are waiting on me to proceed here. I'm waiting to see a proposal for how the user will interact with partition reassignment given the new way to generate assignment. change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments - Key: KAFKA-1792 URL: https://issues.apache.org/jira/browse/KAFKA-1792 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, generate_alg_tests.txt Current implementation produces fair replica distribution between specified list of brokers. Unfortunately, it doesn't take into account current replica assignment. So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth broker id=3, generate will create an assignment config which will redistribute replicas fairly across brokers [0..3] in the same way as those partitions were created from scratch. It will not take into consideration current replica assignment and accordingly will not try to minimize number of replica moves between brokers. As proposed by [~charmalloc] this should be improved. New output of improved --generate algorithm should suite following requirements: - fairness of replica distribution - every broker will have R or R+1 replicas assigned; - minimum of reassignments - number of replica moves between brokers will be minimal; Example. Consider following replica distribution per brokers [0..3] (we just added brokers 2 and 3): - broker - 0, 1, 2, 3 - replicas - 7, 6, 0, 0 The new algorithm will produce following assignment: - broker - 0, 1, 2, 3 - replicas - 4, 3, 3, 3 - moves - -3, -3, +3, +3 It will be fair and number of moves will be 6, which is minimal for specified initial distribution. The scope of this issue is: - design an algorithm matching the above requirements; - implement this algorithm and unit tests; - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1841) OffsetCommitRequest API - timestamp field is not versioned
[ https://issues.apache.org/jira/browse/KAFKA-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271990#comment-14271990 ] Jun Rao commented on KAFKA-1841: Updated reviewboard https://reviews.apache.org/r/29692/diff/ against branch origin/0.8.2 OffsetCommitRequest API - timestamp field is not versioned -- Key: KAFKA-1841 URL: https://issues.apache.org/jira/browse/KAFKA-1841 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.2 Environment: wire-protocol Reporter: Dana Powers Assignee: Jun Rao Priority: Blocker Fix For: 0.8.2 Attachments: kafka-1841.patch, kafka-1841_2015-01-08_15:07:57.patch, kafka-1841_2015-01-09_14:36:50.patch Timestamp field was added to the OffsetCommitRequest wire protocol api for 0.8.2 by KAFKA-1012 . The 0.8.1.1 server does not support the timestamp field, so I think the api version of OffsetCommitRequest should be incremented and checked by the 0.8.2 kafka server before attempting to read a timestamp from the network buffer in OffsetCommitRequest.readFrom (core/src/main/scala/kafka/api/OffsetCommitRequest.scala) It looks like a subsequent patch (KAFKA-1462) added another api change to support a new constructor w/ params generationId and consumerId, calling that version 1, and a pending patch (KAFKA-1634) adds retentionMs as another field, while possibly removing timestamp altogether, calling this version 2. So the fix here is not straightforward enough for me to submit a patch. This could possibly be merged into KAFKA-1634, but opening as a separate Issue because I believe the lack of versioning in the current trunk should block 0.8.2 release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29692: Patch for kafka-1841
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29692/ --- (Updated Jan. 9, 2015, 10:36 p.m.) Review request for kafka. Bugs: kafka-1841 https://issues.apache.org/jira/browse/kafka-1841 Repository: kafka Description (updated) --- rebased Diffs (updated) - clients/src/main/java/org/apache/kafka/common/protocol/Protocol.java 7517b879866fc5dad5f8d8ad30636da8bbe7784a clients/src/main/java/org/apache/kafka/common/requests/OffsetCommitRequest.java 3ee5cbad55ce836fd04bb954dcf6ef2f9bc3288f core/src/main/scala/kafka/api/OffsetCommitRequest.scala 861a6cf11dc6b6431fcbbe9de00c74a122f204bd core/src/main/scala/kafka/api/OffsetFetchRequest.scala c7604b9cdeb8f46507f04ed7d35fcb3d5f150713 core/src/main/scala/kafka/common/OffsetMetadataAndError.scala 4cabffeacea09a49913505db19a96a55d58c0909 core/src/main/scala/kafka/server/KafkaApis.scala 9a61fcba3b5eeb295676b3ef720c719ef5244642 core/src/test/scala/unit/kafka/api/RequestResponseSerializationTest.scala cd16ced5465d098be7a60498326b2a98c248f343 Diff: https://reviews.apache.org/r/29692/diff/ Testing --- Thanks, Jun Rao
[jira] [Commented] (KAFKA-1819) Cleaner gets confused about deleted and re-created topics
[ https://issues.apache.org/jira/browse/KAFKA-1819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272031#comment-14272031 ] Gwen Shapira commented on KAFKA-1819: - Looking more into this, I'm not even sure if it can be related. testCleanShutdownWithDeleteTopicEnabled does enable topic deletion, but it never creates or deletes topics - just startup and shutdown. So the code-path for deleting topics (and our changes) don't seem to execute here. Still trying to figure out what's going on. Cleaner gets confused about deleted and re-created topics - Key: KAFKA-1819 URL: https://issues.apache.org/jira/browse/KAFKA-1819 Project: Kafka Issue Type: Bug Reporter: Gian Merlino Assignee: Gwen Shapira Priority: Blocker Fix For: 0.8.2 Attachments: KAFKA-1819.patch, KAFKA-1819_2014-12-26_13:58:44.patch, KAFKA-1819_2014-12-30_16:01:19.patch I get an error like this after deleting a compacted topic and re-creating it. I think it's because the brokers don't remove cleaning checkpoints from the cleaner-offset-checkpoint file. This is from a build based off commit bd212b7. java.lang.IllegalArgumentException: requirement failed: Last clean offset is 587607 but segment base offset is 0 for log foo-6. at scala.Predef$.require(Predef.scala:233) at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:502) at kafka.log.Cleaner.clean(LogCleaner.scala:300) at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:214) at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:192) at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1841) OffsetCommitRequest API - timestamp field is not versioned
[ https://issues.apache.org/jira/browse/KAFKA-1841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-1841: --- Attachment: kafka-1841_2015-01-09_14:36:50.patch OffsetCommitRequest API - timestamp field is not versioned -- Key: KAFKA-1841 URL: https://issues.apache.org/jira/browse/KAFKA-1841 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.2 Environment: wire-protocol Reporter: Dana Powers Assignee: Jun Rao Priority: Blocker Fix For: 0.8.2 Attachments: kafka-1841.patch, kafka-1841_2015-01-08_15:07:57.patch, kafka-1841_2015-01-09_14:36:50.patch Timestamp field was added to the OffsetCommitRequest wire protocol api for 0.8.2 by KAFKA-1012 . The 0.8.1.1 server does not support the timestamp field, so I think the api version of OffsetCommitRequest should be incremented and checked by the 0.8.2 kafka server before attempting to read a timestamp from the network buffer in OffsetCommitRequest.readFrom (core/src/main/scala/kafka/api/OffsetCommitRequest.scala) It looks like a subsequent patch (KAFKA-1462) added another api change to support a new constructor w/ params generationId and consumerId, calling that version 1, and a pending patch (KAFKA-1634) adds retentionMs as another field, while possibly removing timestamp altogether, calling this version 2. So the fix here is not straightforward enough for me to submit a patch. This could possibly be merged into KAFKA-1634, but opening as a separate Issue because I believe the lack of versioning in the current trunk should block 0.8.2 release. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272039#comment-14272039 ] Neha Narkhede commented on KAFKA-1792: -- [~joestein] No problem at all. I was catching up on all the JIRAs I was reviewing when I came across this. Just want to make sure nothing is blocked on me :) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments - Key: KAFKA-1792 URL: https://issues.apache.org/jira/browse/KAFKA-1792 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, generate_alg_tests.txt Current implementation produces fair replica distribution between specified list of brokers. Unfortunately, it doesn't take into account current replica assignment. So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth broker id=3, generate will create an assignment config which will redistribute replicas fairly across brokers [0..3] in the same way as those partitions were created from scratch. It will not take into consideration current replica assignment and accordingly will not try to minimize number of replica moves between brokers. As proposed by [~charmalloc] this should be improved. New output of improved --generate algorithm should suite following requirements: - fairness of replica distribution - every broker will have R or R+1 replicas assigned; - minimum of reassignments - number of replica moves between brokers will be minimal; Example. Consider following replica distribution per brokers [0..3] (we just added brokers 2 and 3): - broker - 0, 1, 2, 3 - replicas - 7, 6, 0, 0 The new algorithm will produce following assignment: - broker - 0, 1, 2, 3 - replicas - 4, 3, 3, 3 - moves - -3, -3, +3, +3 It will be fair and number of moves will be 6, which is minimal for specified initial distribution. The scope of this issue is: - design an algorithm matching the above requirements; - implement this algorithm and unit tests; - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1481) Stop using dashes AND underscores as separators in MBean names
[ https://issues.apache.org/jira/browse/KAFKA-1481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14272057#comment-14272057 ] Jun Rao commented on KAFKA-1481: Also committed the doc change to 0.8.2 documentation. Stop using dashes AND underscores as separators in MBean names -- Key: KAFKA-1481 URL: https://issues.apache.org/jira/browse/KAFKA-1481 Project: Kafka Issue Type: Bug Components: core Affects Versions: 0.8.1.1 Reporter: Otis Gospodnetic Assignee: Vladimir Tretyakov Priority: Critical Labels: patch Fix For: 0.8.2, 0.8.3 Attachments: KAFKA-1481_2014-06-06_13-06-35.patch, KAFKA-1481_2014-10-13_18-23-35.patch, KAFKA-1481_2014-10-14_21-53-35.patch, KAFKA-1481_2014-10-15_10-23-35.patch, KAFKA-1481_2014-10-20_23-14-35.patch, KAFKA-1481_2014-10-21_09-14-35.patch, KAFKA-1481_2014-10-30_21-35-43.patch, KAFKA-1481_2014-10-31_14-35-43.patch, KAFKA-1481_2014-11-03_16-39-41_doc.patch, KAFKA-1481_2014-11-03_17-02-23.patch, KAFKA-1481_2014-11-10_20-39-41_doc.patch, KAFKA-1481_2014-11-10_21-02-23.patch, KAFKA-1481_2014-11-14_16-33-03.patch, KAFKA-1481_2014-11-14_16-39-41_doc.patch, KAFKA-1481_2014-11-17_14-33-03.patch, KAFKA-1481_2014-11-19_16-03-03_trunk.patch, KAFKA-1481_IDEA_IDE_2014-10-14_21-53-35.patch, KAFKA-1481_IDEA_IDE_2014-10-15_10-23-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_20-14-35.patch, KAFKA-1481_IDEA_IDE_2014-10-20_23-14-35.patch, alternateLayout1.png, alternateLayout2.png, diff-for-alternate-layout1.patch, diff-for-alternate-layout2.patch, logflushes.png, originalLayout.png MBeans should not use dashes or underscores as separators because these characters are allowed in hostnames, topics, group and consumer IDs, etc., and these are embedded in MBeans names making it impossible to parse out individual bits from MBeans. Perhaps a pipe character should be used to avoid the conflict. This looks like a major blocker because it means nobody can write Kafka 0.8.x monitoring tools unless they are doing it for themselves AND do not use dashes AND do not use underscores. See: http://search-hadoop.com/m/4TaT4lonIW -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1786) implement a global configuration feature for brokers
[ https://issues.apache.org/jira/browse/KAFKA-1786?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271871#comment-14271871 ] Joe Stein commented on KAFKA-1786: -- Hey [~nehanarkhede] I had sent this out on the mailing list a while back as part of the CLI tool changes (parent ticket) http://search-hadoop.com/m/4TaT4CJpj11 also with confluence page too https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Command+Line+and+Related+Improvements implement a global configuration feature for brokers Key: KAFKA-1786 URL: https://issues.apache.org/jira/browse/KAFKA-1786 Project: Kafka Issue Type: Sub-task Reporter: Joe Stein Assignee: Andrii Biletskyi Fix For: 0.8.3 Attachments: KAFKA-1786.patch Global level configurations (much like topic level) for brokers are managed by humans and automation systems through server.properties. Some configuration make sense to use default (like it is now) or override from central location (zookeeper for now). We can modify this through the new CLI tool so that every broker can have exact same setting. Some configurations we should allow to be overriden from server.properties (like port) but others we should use the global store as source of truth (e.g. auto topic enable, fetch replica message size, etc). Since most configuration I believe are going to fall into this category we should have the list of server.properties that can override the global config in the code in a list which we can manage... everything else the global takes precedence. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (KAFKA-1792) change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments
[ https://issues.apache.org/jira/browse/KAFKA-1792?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271872#comment-14271872 ] Joe Stein commented on KAFKA-1792: -- [~nehanarkhede] Dmitry has been out on vacation and should be able to pick this back up once he returns, sorry for radio silence but it was due to that change behavior of --generate to produce assignment config with fair replica distribution and minimal number of reassignments - Key: KAFKA-1792 URL: https://issues.apache.org/jira/browse/KAFKA-1792 Project: Kafka Issue Type: Sub-task Components: tools Reporter: Dmitry Pekar Assignee: Dmitry Pekar Fix For: 0.8.3 Attachments: KAFKA-1792.patch, KAFKA-1792_2014-12-03_19:24:56.patch, KAFKA-1792_2014-12-08_13:42:43.patch, KAFKA-1792_2014-12-19_16:48:12.patch, generate_alg_tests.txt Current implementation produces fair replica distribution between specified list of brokers. Unfortunately, it doesn't take into account current replica assignment. So if we have, for instance, 3 brokers id=[0..2] and are going to add fourth broker id=3, generate will create an assignment config which will redistribute replicas fairly across brokers [0..3] in the same way as those partitions were created from scratch. It will not take into consideration current replica assignment and accordingly will not try to minimize number of replica moves between brokers. As proposed by [~charmalloc] this should be improved. New output of improved --generate algorithm should suite following requirements: - fairness of replica distribution - every broker will have R or R+1 replicas assigned; - minimum of reassignments - number of replica moves between brokers will be minimal; Example. Consider following replica distribution per brokers [0..3] (we just added brokers 2 and 3): - broker - 0, 1, 2, 3 - replicas - 7, 6, 0, 0 The new algorithm will produce following assignment: - broker - 0, 1, 2, 3 - replicas - 4, 3, 3, 3 - moves - -3, -3, +3, +3 It will be fair and number of moves will be 6, which is minimal for specified initial distribution. The scope of this issue is: - design an algorithm matching the above requirements; - implement this algorithm and unit tests; - test it manually using different initial assignments; -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29751: Patch for kafka-1851
On Jan. 9, 2015, 8:01 p.m., Ewen Cheslack-Postava wrote: core/src/main/scala/kafka/server/KafkaApis.scala, line 512 https://reviews.apache.org/r/29751/diff/1/?file=813957#file813957line512 You could just change getOffsets so it doesn't have this behavior instead of checking the condition here. It looks like this is the only place it's called from so nothing else should be relying on that behavior and it just simplifies the getOffsets code. Yes, I wasn't sure if there will be a use case to get the offsets of all partitions in the same consumer group. - Jun --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29751/#review67499 --- On Jan. 9, 2015, 2:53 a.m., Jun Rao wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29751/ --- (Updated Jan. 9, 2015, 2:53 a.m.) Review request for kafka. Bugs: kafka-1851 https://issues.apache.org/jira/browse/kafka-1851 Repository: kafka Description --- fix the behavior of OffsetFetchRequest on unknown partition Diffs - core/src/main/scala/kafka/server/KafkaApis.scala 2f009920af37de3cf0a3eb131f2124f4e532c4e4 core/src/test/scala/unit/kafka/server/OffsetCommitTest.scala 8c5364fa97da1be09973c176d1baeb339455d319 Diff: https://reviews.apache.org/r/29751/diff/ Testing --- Thanks, Jun Rao
[jira] [Commented] (KAFKA-1852) OffsetCommitRequest can commit offset on unknown topic
[ https://issues.apache.org/jira/browse/KAFKA-1852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14271884#comment-14271884 ] Jun Rao commented on KAFKA-1852: Another weird thing this causes is that we can commit an offset to a non-existing topic, but will get an UnknownTopic error when fetching the offset. OffsetCommitRequest can commit offset on unknown topic -- Key: KAFKA-1852 URL: https://issues.apache.org/jira/browse/KAFKA-1852 Project: Kafka Issue Type: Bug Affects Versions: 0.8.3 Reporter: Jun Rao Currently, we allow an offset to be committed to Kafka, even when the topic/partition for the offset doesn't exist. We probably should disallow that and send an error back in that case. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29756: Patch for KAFKA-1854
On Jan. 9, 2015, 9:44 a.m., Manikumar Reddy O wrote: kafka-patch-review.py, line 134 https://reviews.apache.org/r/29756/diff/2/?file=814144#file814144line134 Can we do authentication check at the beginning and fail-fast if the username/password is wrong. Currently it will update the review-board and fails at JIRA update. Is it possible to catch the exception/error, and show the message Invalid JIRA username/password I've updated the patch to verify the JIRA login before publishing/updating the reviewboard. If the JIRA login fails, the patch submission tool now prints out the inability to login to JIRA and exits the tool. - Jaikiran --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/#review67400 --- On Jan. 9, 2015, 10:13 a.m., Jaikiran Pai wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/ --- (Updated Jan. 9, 2015, 10:13 a.m.) Review request for kafka. Bugs: KAFKA-1854 https://issues.apache.org/jira/browse/KAFKA-1854 Repository: kafka Description --- KAFKA-1854 Allow JIRA username and password to be prompted in the absence of a jira.ini file, during patch submission Diffs - kafka-patch-review.py b7f132f9d210b8648859ab8f9c89f30ec128ab38 Diff: https://reviews.apache.org/r/29756/diff/ Testing --- Thanks, Jaikiran Pai
Re: Review Request 29756: Patch for KAFKA-1854
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/ --- (Updated Jan. 9, 2015, 12:47 p.m.) Review request for kafka. Bugs: KAFKA-1854 https://issues.apache.org/jira/browse/KAFKA-1854 Repository: kafka Description --- KAFKA-1854 Allow JIRA username and password to be prompted in the absence of a jira.ini file, during patch submission Diffs (updated) - kafka-patch-review.py b7f132f9d210b8648859ab8f9c89f30ec128ab38 Diff: https://reviews.apache.org/r/29756/diff/ Testing --- Thanks, Jaikiran Pai
[jira] [Commented] (KAFKA-1854) Allow the JIRA username and password to be prompted during patch submission
[ https://issues.apache.org/jira/browse/KAFKA-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14270968#comment-14270968 ] jaikiran pai commented on KAFKA-1854: - Updated reviewboard https://reviews.apache.org/r/29756/diff/ against branch origin/trunk Allow the JIRA username and password to be prompted during patch submission --- Key: KAFKA-1854 URL: https://issues.apache.org/jira/browse/KAFKA-1854 Project: Kafka Issue Type: Improvement Reporter: jaikiran pai Attachments: KAFKA-1854.patch, KAFKA-1854_2015-01-09_13:39:23.patch, KAFKA-1854_2015-01-09_15:42:28.patch, KAFKA-1854_2015-01-09_18:16:35.patch The current patch submission process involves using the kafka-patch-review.py python script which expects a jira.ini file to contain the user's username and password for JIRA authentication. I'm one of those who doesn't like storing passwords in files :) It would be good to (optionally) allow the username/password to be prompted by the patch submission script. I've a patch which I can submit for this enhancement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (KAFKA-1854) Allow the JIRA username and password to be prompted during patch submission
[ https://issues.apache.org/jira/browse/KAFKA-1854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jaikiran pai updated KAFKA-1854: Attachment: KAFKA-1854_2015-01-09_18:16:35.patch Allow the JIRA username and password to be prompted during patch submission --- Key: KAFKA-1854 URL: https://issues.apache.org/jira/browse/KAFKA-1854 Project: Kafka Issue Type: Improvement Reporter: jaikiran pai Attachments: KAFKA-1854.patch, KAFKA-1854_2015-01-09_13:39:23.patch, KAFKA-1854_2015-01-09_15:42:28.patch, KAFKA-1854_2015-01-09_18:16:35.patch The current patch submission process involves using the kafka-patch-review.py python script which expects a jira.ini file to contain the user's username and password for JIRA authentication. I'm one of those who doesn't like storing passwords in files :) It would be good to (optionally) allow the username/password to be prompted by the patch submission script. I've a patch which I can submit for this enhancement. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Review Request 29756: Patch for KAFKA-1854
On Jan. 9, 2015, 11:07 a.m., Manikumar Reddy O wrote: kafka-patch-review.py, line 96 https://reviews.apache.org/r/29756/diff/3/?file=814216#file814216line96 Did you tested the failure senario? I did not get error message. I think we will get exception only after invoking jira.issue(). Jaikiran Pai wrote: I did test a failure case, yes. Gave an incorrect username and password and that failed during login and did not proceed further. Here's the relevant output (notice that I intentionally use JIRA user name foo): ` Configuring reviewboard url to https://reviews.apache.org Updating your remote branches to pull the latest changes Verifying JIRA connection configurations JIRA user :foo JIRA password : Failed to login to the JIRA instance class 'jira.exceptions.JIRAError' HTTP 401: html head titleUnauthorized (401)/title ` Jaikiran Pai wrote: By the way, if you are testing this, make sure you don't have a jira.ini in its usual place. Else the (correct) credentials in there will be used as usual and you won't be able to replicate a failing scenario. Manikumar Reddy O wrote: Configuring reviewboard url to https://reviews.apache.org Updating your remote branches to pull the latest changes Verifying JIRA connection configurations JIRA user :foo JIRA password : Creating diff against origin/0.8.2 and uploading patch to JIRA KAFKA-1723 I am not getting exception, it going to next line of execution. It is failing at jira.issue(). Version : jira-python 0.16 Jaikiran Pai wrote: That's interesting. Let me see what might be different. Turns out the version of jira python that I had was 0.31 and behaved differently. I downgraded to 0.16 and was able to replicate what you are seeing. I've updated the patch to fix this and it should now correctly fail if wrong JIRA credentials are specified, even before it attempts to publish to reviewboard. Let me know if this now works for you too. Thanks for testing this! - Jaikiran --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/#review67414 --- On Jan. 9, 2015, 12:47 p.m., Jaikiran Pai wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/ --- (Updated Jan. 9, 2015, 12:47 p.m.) Review request for kafka. Bugs: KAFKA-1854 https://issues.apache.org/jira/browse/KAFKA-1854 Repository: kafka Description --- KAFKA-1854 Allow JIRA username and password to be prompted in the absence of a jira.ini file, during patch submission Diffs - kafka-patch-review.py b7f132f9d210b8648859ab8f9c89f30ec128ab38 Diff: https://reviews.apache.org/r/29756/diff/ Testing --- Thanks, Jaikiran Pai
Re: Review Request 29756: Patch for KAFKA-1854
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/#review67414 --- kafka-patch-review.py https://reviews.apache.org/r/29756/#comment111442 Did you tested the failure senario? I did not get error message. I think we will get exception only after invoking jira.issue(). - Manikumar Reddy O On Jan. 9, 2015, 10:13 a.m., Jaikiran Pai wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/ --- (Updated Jan. 9, 2015, 10:13 a.m.) Review request for kafka. Bugs: KAFKA-1854 https://issues.apache.org/jira/browse/KAFKA-1854 Repository: kafka Description --- KAFKA-1854 Allow JIRA username and password to be prompted in the absence of a jira.ini file, during patch submission Diffs - kafka-patch-review.py b7f132f9d210b8648859ab8f9c89f30ec128ab38 Diff: https://reviews.apache.org/r/29756/diff/ Testing --- Thanks, Jaikiran Pai
Re: Review Request 29756: Patch for KAFKA-1854
On Jan. 9, 2015, 11:07 a.m., Manikumar Reddy O wrote: kafka-patch-review.py, line 96 https://reviews.apache.org/r/29756/diff/3/?file=814216#file814216line96 Did you tested the failure senario? I did not get error message. I think we will get exception only after invoking jira.issue(). I did test a failure case, yes. Gave an incorrect username and password and that failed during login and did not proceed further. Here's the relevant output (notice that I intentionally use JIRA user name foo): ` Configuring reviewboard url to https://reviews.apache.org Updating your remote branches to pull the latest changes Verifying JIRA connection configurations JIRA user :foo JIRA password : Failed to login to the JIRA instance class 'jira.exceptions.JIRAError' HTTP 401: html head titleUnauthorized (401)/title ` - Jaikiran --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/#review67414 --- On Jan. 9, 2015, 10:13 a.m., Jaikiran Pai wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/ --- (Updated Jan. 9, 2015, 10:13 a.m.) Review request for kafka. Bugs: KAFKA-1854 https://issues.apache.org/jira/browse/KAFKA-1854 Repository: kafka Description --- KAFKA-1854 Allow JIRA username and password to be prompted in the absence of a jira.ini file, during patch submission Diffs - kafka-patch-review.py b7f132f9d210b8648859ab8f9c89f30ec128ab38 Diff: https://reviews.apache.org/r/29756/diff/ Testing --- Thanks, Jaikiran Pai
Re: Review Request 29756: Patch for KAFKA-1854
On Jan. 9, 2015, 11:07 a.m., Manikumar Reddy O wrote: kafka-patch-review.py, line 96 https://reviews.apache.org/r/29756/diff/3/?file=814216#file814216line96 Did you tested the failure senario? I did not get error message. I think we will get exception only after invoking jira.issue(). Jaikiran Pai wrote: I did test a failure case, yes. Gave an incorrect username and password and that failed during login and did not proceed further. Here's the relevant output (notice that I intentionally use JIRA user name foo): ` Configuring reviewboard url to https://reviews.apache.org Updating your remote branches to pull the latest changes Verifying JIRA connection configurations JIRA user :foo JIRA password : Failed to login to the JIRA instance class 'jira.exceptions.JIRAError' HTTP 401: html head titleUnauthorized (401)/title ` By the way, if you are testing this, make sure you don't have a jira.ini in its usual place. Else the (correct) credentials in there will be used as usual and you won't be able to replicate a failing scenario. - Jaikiran --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/#review67414 --- On Jan. 9, 2015, 10:13 a.m., Jaikiran Pai wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/ --- (Updated Jan. 9, 2015, 10:13 a.m.) Review request for kafka. Bugs: KAFKA-1854 https://issues.apache.org/jira/browse/KAFKA-1854 Repository: kafka Description --- KAFKA-1854 Allow JIRA username and password to be prompted in the absence of a jira.ini file, during patch submission Diffs - kafka-patch-review.py b7f132f9d210b8648859ab8f9c89f30ec128ab38 Diff: https://reviews.apache.org/r/29756/diff/ Testing --- Thanks, Jaikiran Pai
Re: Review Request 29756: Patch for KAFKA-1854
On Jan. 9, 2015, 11:07 a.m., Manikumar Reddy O wrote: kafka-patch-review.py, line 96 https://reviews.apache.org/r/29756/diff/3/?file=814216#file814216line96 Did you tested the failure senario? I did not get error message. I think we will get exception only after invoking jira.issue(). Jaikiran Pai wrote: I did test a failure case, yes. Gave an incorrect username and password and that failed during login and did not proceed further. Here's the relevant output (notice that I intentionally use JIRA user name foo): ` Configuring reviewboard url to https://reviews.apache.org Updating your remote branches to pull the latest changes Verifying JIRA connection configurations JIRA user :foo JIRA password : Failed to login to the JIRA instance class 'jira.exceptions.JIRAError' HTTP 401: html head titleUnauthorized (401)/title ` Jaikiran Pai wrote: By the way, if you are testing this, make sure you don't have a jira.ini in its usual place. Else the (correct) credentials in there will be used as usual and you won't be able to replicate a failing scenario. Configuring reviewboard url to https://reviews.apache.org Updating your remote branches to pull the latest changes Verifying JIRA connection configurations JIRA user :foo JIRA password : Creating diff against origin/0.8.2 and uploading patch to JIRA KAFKA-1723 I am not getting exception, it going to next line of execution. It is failing at jira.issue(). Version : jira-python 0.16 - Manikumar Reddy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/#review67414 --- On Jan. 9, 2015, 10:13 a.m., Jaikiran Pai wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/ --- (Updated Jan. 9, 2015, 10:13 a.m.) Review request for kafka. Bugs: KAFKA-1854 https://issues.apache.org/jira/browse/KAFKA-1854 Repository: kafka Description --- KAFKA-1854 Allow JIRA username and password to be prompted in the absence of a jira.ini file, during patch submission Diffs - kafka-patch-review.py b7f132f9d210b8648859ab8f9c89f30ec128ab38 Diff: https://reviews.apache.org/r/29756/diff/ Testing --- Thanks, Jaikiran Pai
Re: Review Request 29756: Patch for KAFKA-1854
On Jan. 9, 2015, 11:07 a.m., Manikumar Reddy O wrote: kafka-patch-review.py, line 96 https://reviews.apache.org/r/29756/diff/3/?file=814216#file814216line96 Did you tested the failure senario? I did not get error message. I think we will get exception only after invoking jira.issue(). Jaikiran Pai wrote: I did test a failure case, yes. Gave an incorrect username and password and that failed during login and did not proceed further. Here's the relevant output (notice that I intentionally use JIRA user name foo): ` Configuring reviewboard url to https://reviews.apache.org Updating your remote branches to pull the latest changes Verifying JIRA connection configurations JIRA user :foo JIRA password : Failed to login to the JIRA instance class 'jira.exceptions.JIRAError' HTTP 401: html head titleUnauthorized (401)/title ` Jaikiran Pai wrote: By the way, if you are testing this, make sure you don't have a jira.ini in its usual place. Else the (correct) credentials in there will be used as usual and you won't be able to replicate a failing scenario. Manikumar Reddy O wrote: Configuring reviewboard url to https://reviews.apache.org Updating your remote branches to pull the latest changes Verifying JIRA connection configurations JIRA user :foo JIRA password : Creating diff against origin/0.8.2 and uploading patch to JIRA KAFKA-1723 I am not getting exception, it going to next line of execution. It is failing at jira.issue(). Version : jira-python 0.16 That's interesting. Let me see what might be different. - Jaikiran --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/#review67414 --- On Jan. 9, 2015, 10:13 a.m., Jaikiran Pai wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/29756/ --- (Updated Jan. 9, 2015, 10:13 a.m.) Review request for kafka. Bugs: KAFKA-1854 https://issues.apache.org/jira/browse/KAFKA-1854 Repository: kafka Description --- KAFKA-1854 Allow JIRA username and password to be prompted in the absence of a jira.ini file, during patch submission Diffs - kafka-patch-review.py b7f132f9d210b8648859ab8f9c89f30ec128ab38 Diff: https://reviews.apache.org/r/29756/diff/ Testing --- Thanks, Jaikiran Pai
