Re: Review Request 36858: Patch for KAFKA-2120

[Ismael Juma]

> On Sept. 10, 2015, 6:19 p.m., Ismael Juma wrote:
> > clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java, line 
> > 137
> > 
> >
> > Out of curiosity, why are we using `LinkedList` here? Generally 
> > `ArrayList` is a better default, but maybe there's a good reason to deviate 
> > in this case.
> 
> Mayuresh Gharat wrote:
> LinkedList add method gives O(1) performance while ArrayList gives O(n) 
> in worst case. Since we are just adding and not doing random accesses 
> LinkedList would be better in this case.

This analysis is a bit incomplete: ArrayList has amortised O(1) append and 
simply O(1) append if you know the expected size. Furthermore, it is much more 
cache-friendly and memory efficient than LinkedList. These factors mean that 
it's almost always a better data structure than LinkedList in modern hardware.


- Ismael


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98438
---


On Sept. 11, 2015, 4:39 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 11, 2015, 4:39 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 049b22eadd5496b70dfcfd9d821f67c62c68a052 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
>   clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
> 69c93c3adf674b1640534c3d7410fcaafaf2232c 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
>  2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
>  5b2e4ffaeab7127648db608c179703b27b577414 
>   
> clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
>  aa44991777a855f4b7f4f7bf17107c69393ff8ff 
>   clients/src/test/java/o

[jira] [Created] (KAFKA-2534) SSLTransportLayer does not handle buffer overflow correctly

[Rajini Sivaram (JIRA)]

Rajini Sivaram created KAFKA-2534:
-

 Summary: SSLTransportLayer does not handle buffer overflow 
correctly
 Key: KAFKA-2534
 URL: https://issues.apache.org/jira/browse/KAFKA-2534
 Project: Kafka
  Issue Type: Bug
  Components: network
Affects Versions: 0.8.3
Reporter: Rajini Sivaram
Assignee: Rajini Sivaram
 Fix For: 0.8.3


There are a couple of issues with the handling of buffer overflow in 
{{SSLTransportLayer}}.
# {{netWriteBuffer}} is flipped immediately after {{wrap()}}, leaving the data 
ready for writing onto the socket channel. If {{netWriteBuffer}} is expanded 
because {{wrap()}} returns BUFFER_OVERFLOW, the expanded buffer needs to be 
un-flipped before expansion and flipped afterwards to leave the resulting 
buffer in the same state. The current implementation does not do this and hence 
the expanded buffer is not as expected.
# If {{handshakeUnwrap()}} returns a BUFFER_OVERFLOW because application buffer 
needs to be expanded, the current implementation expands the buffer and returns 
from the {{handshake()}} call. Since handshake data was already read from the 
network, this can result in the handshake never completing if no more data is 
received on the channel. {{handshakeUnwrap()}} should be invoked after 
application buffer expansion to process data that has already arrived.

I will submit a PR with fixes for these along with unit tests which demonstrate 
these issues. I am not sure it is easy to trigger buffer overflow/underflow in 
a system test. Since these code paths are currently untested, unit tests which 
mock these scenarios may be useful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2534: Fixes and unit tests for SSLTransp...

[rajinisivaram]

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/205

KAFKA-2534: Fixes and unit tests for SSLTransportLayer buffer overflow

Unit tests which mock buffer overflow and underflow in the SSL transport 
layer and fixes for the couple of issues in buffer overflow handling described 
in the JIRA.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-2534

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/205.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #205


commit 260b0bba5b906e96f3acc637ae1ece1c3cc5ec83
Author: Rajini Sivaram 
Date:   2015-09-11T10:03:29Z

KAFKA-2534: Fixes and unit tests for SSLTransportLayer buffer overflow




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2534) SSLTransportLayer does not handle buffer overflow correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740540#comment-14740540
 ] 

ASF GitHub Bot commented on KAFKA-2534:
---

GitHub user rajinisivaram opened a pull request:

https://github.com/apache/kafka/pull/205

KAFKA-2534: Fixes and unit tests for SSLTransportLayer buffer overflow

Unit tests which mock buffer overflow and underflow in the SSL transport 
layer and fixes for the couple of issues in buffer overflow handling described 
in the JIRA.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rajinisivaram/kafka KAFKA-2534

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/205.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #205


commit 260b0bba5b906e96f3acc637ae1ece1c3cc5ec83
Author: Rajini Sivaram 
Date:   2015-09-11T10:03:29Z

KAFKA-2534: Fixes and unit tests for SSLTransportLayer buffer overflow




> SSLTransportLayer does not handle buffer overflow correctly
> ---
>
> Key: KAFKA-2534
> URL: https://issues.apache.org/jira/browse/KAFKA-2534
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 0.8.3
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.8.3
>
>
> There are a couple of issues with the handling of buffer overflow in 
> {{SSLTransportLayer}}.
> # {{netWriteBuffer}} is flipped immediately after {{wrap()}}, leaving the 
> data ready for writing onto the socket channel. If {{netWriteBuffer}} is 
> expanded because {{wrap()}} returns BUFFER_OVERFLOW, the expanded buffer 
> needs to be un-flipped before expansion and flipped afterwards to leave the 
> resulting buffer in the same state. The current implementation does not do 
> this and hence the expanded buffer is not as expected.
> # If {{handshakeUnwrap()}} returns a BUFFER_OVERFLOW because application 
> buffer needs to be expanded, the current implementation expands the buffer 
> and returns from the {{handshake()}} call. Since handshake data was already 
> read from the network, this can result in the handshake never completing if 
> no more data is received on the channel. {{handshakeUnwrap()}} should be 
> invoked after application buffer expansion to process data that has already 
> arrived.
> I will submit a PR with fixes for these along with unit tests which 
> demonstrate these issues. I am not sure it is easy to trigger buffer 
> overflow/underflow in a system test. Since these code paths are currently 
> untested, unit tests which mock these scenarios may be useful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2534) SSLTransportLayer does not handle buffer overflow correctly

[Rajini Sivaram (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Rajini Sivaram updated KAFKA-2534:
--
Reviewer: Sriharsha Chintalapani
  Status: Patch Available  (was: Open)

> SSLTransportLayer does not handle buffer overflow correctly
> ---
>
> Key: KAFKA-2534
> URL: https://issues.apache.org/jira/browse/KAFKA-2534
> Project: Kafka
>  Issue Type: Bug
>  Components: network
>Affects Versions: 0.8.3
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.8.3
>
>
> There are a couple of issues with the handling of buffer overflow in 
> {{SSLTransportLayer}}.
> # {{netWriteBuffer}} is flipped immediately after {{wrap()}}, leaving the 
> data ready for writing onto the socket channel. If {{netWriteBuffer}} is 
> expanded because {{wrap()}} returns BUFFER_OVERFLOW, the expanded buffer 
> needs to be un-flipped before expansion and flipped afterwards to leave the 
> resulting buffer in the same state. The current implementation does not do 
> this and hence the expanded buffer is not as expected.
> # If {{handshakeUnwrap()}} returns a BUFFER_OVERFLOW because application 
> buffer needs to be expanded, the current implementation expands the buffer 
> and returns from the {{handshake()}} call. Since handshake data was already 
> read from the network, this can result in the handshake never completing if 
> no more data is received on the channel. {{handshakeUnwrap()}} should be 
> invoked after application buffer expansion to process data that has already 
> arrived.
> I will submit a PR with fixes for these along with unit tests which 
> demonstrate these issues. I am not sure it is easy to trigger buffer 
> overflow/underflow in a system test. Since these code paths are currently 
> untested, unit tests which mock these scenarios may be useful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: let kafka support "commit offset after consumi...

[wyzssw]

GitHub user wyzssw opened a pull request:

https://github.com/apache/kafka/pull/206

let kafka support "commit offset after consuming"

Kafka Of Original Version do not support "commit offset after 
consuming",when kafka.consumer.ConsumerIterator[K, V].next() return a 
MessageAndMetadata,which consumed offset is already being set,and commit thread 
maybe commit this offset before "consuming process finished",when jvm restart 
or being down，this msg will not be consumed next time

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apache/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/206.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #206


commit b7439c8081465f8764e88326b3fa7b320f99f130
Author: Sriharsha Chintalapani 
Date:   2015-03-12T20:56:52Z

kafka-1461; Replica fetcher thread does not implement any back-off 
behavior; patched by Sriharsha Chintalapani; reviewed by Jun Rao

commit c41c7b40b63ecd668c727a897f29e276a1c5adf7
Author: Jiangjie Qin 
Date:   2015-03-13T22:06:10Z

KAFKA-1997; Refactor MirrorMaker based on KIP-3; reviewed by Joel Koshy and 
Guozhang Wang

commit 1caaf6db400df7e37b7f0416bb83ab451018a5c8
Author: Guozhang Wang 
Date:   2015-03-13T22:17:08Z

KAFKA-1863; Add docs for possible thrown exception in Callback; reviewed by 
Jiangjie Qin

commit 57fddbddc823b0946fda6e82dfba4ab8b5fd
Author: Jiangjie Qin 
Date:   2015-03-17T21:22:01Z

KAFKA-1997; Follow-up to add the shutdown hook before starting the 
consumers; reviewed by Guozhang Wang

commit 82789e75199fdc1cae115c5c2eadfd0f1ece4d0d
Author: Jiangjie Qin 
Date:   2015-03-18T21:58:11Z

KAFKA-1997; Hopefully last follow-up fix to get messageHandlerArgs right

commit b2c833aa41cb9a7a6232781b273402042e021607
Author: Guozhang Wang 
Date:   2015-03-19T22:52:54Z

KAFKA-1910; Fix two bugs on MemoryRecords and KafkaConsumer; reviewed by 
Onur Karaman

commit c62cff355936eea2a96e3b9678c761334fce8ee2
Author: Guozhang Wang 
Date:   2015-03-19T23:10:23Z

trivial fix on coding style

commit 991195416e0c179d2d2a79891d0214244c287618
Author: Jiangjie Qin 
Date:   2015-03-24T22:48:46Z

KAFKA-2042; Update topic list of the metadata regardless of cluster 
information; reviewed by Guozhang Wang

commit 4e0da4965b83b25debdec4d55e075c18be436fa8
Author: Tao Xiao 
Date:   2015-03-25T03:56:22Z

KAFKA-2048; Change lock synchronized to inLock() for partitionMapCond; 
reviewed by Guozhang Wang

commit eb2100876b70d1e0d53a88c2513ca3a4ac7a1c1e
Author: Joe Stein 
Date:   2015-03-25T05:57:42Z

KAFKA-1856 Add PreCommit Patch Testing patch by Ashish K Singh reviewed by 
Gwen Shapira

commit a74688de46c23dd34ad964372fbed05c1f7f4b3e
Author: Yasuhiro Matsuda 
Date:   2015-03-25T20:08:38Z

kafka-527; Compression support does numerous byte copies; patched by 
Yasuhiro Matsuda; reviewed by Guozhang Wang and Jun Rao

commit 5b42b538eb46203f7fd308cb3d3f27dde98840b8
Author: Jiangjie Qin 
Date:   2015-03-25T21:01:19Z

KAFKA-2047; Move the stream creation into concurrent mirror maker threads; 
reviewed by Guozhang Wang

commit d2f50fc3886896bc569fea7fb308036008b89f94
Author: Guozhang Wang 
Date:   2015-03-26T22:43:18Z

KAFKA-527; Use in-place decompression enabled inner iterator to replace old 
decompress function; reviewed by Joel Koshy and Jun Rao

commit c5df2a8e3acca1e2c905fa6b78e73e09b1dd0cd7
Author: Guozhang Wang 
Date:   2015-03-27T00:16:33Z

KAFKA-1634; Bump up Offset Commit Request to v2 to add global retention and 
remove per-partition commit timestamp; reviewed by Joel Koshy and Jun Rao

commit d8fe98efee5a44ae12c1e3484fa20f89b0f30054
Author: Gwen Shapira 
Date:   2015-03-28T15:39:48Z

kafka-2044; Support requests and responses from o.a.k.common in KafkaApis; 
patched by Gwen Shapira; reviewed by Jun Rao

commit 66c6f9b1c436187635674f82a9e9940d8f5331f2
Author: Ismael Juma 
Date:   2015-04-01T20:26:45Z

kafka-2039; Update Scala to 2.10.5 and 2.11.6; patched by Ismael Juma; 
reviewed by Jun Rao

commit 619d78eb521185f365a002e5f1987b75d3f35192
Author: Yasuhiro Matsuda 
Date:   2015-04-01T23:14:48Z

kafka-2013; benchmark test for the purgatory; patched by Yasuhiro Matsuda; 
reviewed by Jun Rao

commit ad722531daafadbcb27b0c0db0b9fcbb68b25b28
Author: Ted Malaska 
Date:   2015-04-03T01:20:54Z

kafka-2016; RollingBounceTest takes long; patched by Ted Malaska; reviewed 
by Jun Rao

commit 48f997047228c327f91e8f848142b4607366fc3e
Author: Ted Malaska 
Date:   2015-04-03T18:43:52Z

KAFKA-1961 Prevent deletion of _consumer_offsets topic; reviewed by Neha 
Narkhede, Gwen Shapira and Jun Rao

commit 15b93a410a8e988e09dc147c6d5250cbbe328b26
Author: Aditya Auradkar 
Date:   2015-04-04T01:39:27Z

KAFKA-1546; Automate replica lag tuning; reviewed by Joel Koshy, Ne

[GitHub] kafka pull request: support commit offset after consumed

[wyzssw]

GitHub user wyzssw opened a pull request:

https://github.com/apache/kafka/pull/207

support commit offset after consumed

Kafka Of Original Version do not support "commit offset after 
consuming",when kafka.consumer.ConsumerIterator[K, V].next() return a 
MessageAndMetadata,which consumed offset is already being set,and commit thread 
maybe commit this offset before "consuming process finished",when jvm restart 
or being down，this msg will not be consumed next time

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wyzssw/kafka 0.8.2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/207.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #207


commit 642f0fd3755830b4d2fdeb38bfa689676d13e7a5
Author: wanghongfeng 
Date:   2015-09-11T13:38:32Z

support commit offset after consumed




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Reopened] (KAFKA-2300) Error in controller log when broker tries to rejoin cluster

[Flavio Junqueira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Flavio Junqueira reopened KAFKA-2300:
-

I'm observing sometimes that the same controller gets elected, but it is not 
restarting with a clean state, which causes the test case committed in this 
patch to fail. We need to make sure that the controller starts with a clean 
state for brokerRequestBatch to fix this problem.

> Error in controller log when broker tries to rejoin cluster
> ---
>
> Key: KAFKA-2300
> URL: https://issues.apache.org/jira/browse/KAFKA-2300
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Johnny Brown
>Assignee: Flavio Junqueira
> Fix For: 0.8.3
>
> Attachments: KAFKA-2300-controller-logs.tar.gz, 
> KAFKA-2300-repro.patch, KAFKA-2300.patch, KAFKA-2300.patch
>
>
> Hello Kafka folks,
> We are having an issue where a broker attempts to join the cluster after 
> being restarted, but is never added to the ISR for its assigned partitions. 
> This is a three-node cluster, and the controller is broker 2.
> When broker 1 starts, we see the following message in broker 2's 
> controller.log.
> {{
> [2015-06-23 13:57:16,535] ERROR [BrokerChangeListener on Controller 2]: Error 
> while handling broker changes 
> (kafka.controller.ReplicaStateMachine$BrokerChangeListener)
> java.lang.IllegalStateException: Controller to broker state change requests 
> batch is not empty while creating a new one. Some UpdateMetadata state 
> changes Map(2 -> Map([prod-sver-end,1] -> 
> (LeaderAndIsrInfo:(Leader:-2,ISR:1,LeaderEpoch:0,ControllerEpoch:165),ReplicationFactor:1),AllReplicas:1)),
>  1 -> Map([prod-sver-end,1] -> 
> (LeaderAndIsrInfo:(Leader:-2,ISR:1,LeaderEpoch:0,ControllerEpoch:165),ReplicationFactor:1),AllReplicas:1)),
>  3 -> Map([prod-sver-end,1] -> 
> (LeaderAndIsrInfo:(Leader:-2,ISR:1,LeaderEpoch:0,ControllerEpoch:165),ReplicationFactor:1),AllReplicas:1)))
>  might be lost 
>   at 
> kafka.controller.ControllerBrokerRequestBatch.newBatch(ControllerChannelManager.scala:202)
>   at 
> kafka.controller.KafkaController.sendUpdateMetadataRequest(KafkaController.scala:974)
>   at 
> kafka.controller.KafkaController.onBrokerStartup(KafkaController.scala:399)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ReplicaStateMachine.scala:371)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:359)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:359)
>   at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:358)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:357)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:357)
>   at kafka.utils.Utils$.inLock(Utils.scala:535)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildChange(ReplicaStateMachine.scala:356)
>   at org.I0Itec.zkclient.ZkClient$7.run(ZkClient.java:568)
>   at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> }}
> {{prod-sver-end}} is a topic we previously deleted. It seems some remnant of 
> it persists in the controller's memory, causing an exception which interrupts 
> the state change triggered by the broker startup.
> Has anyone seen something like this? Any idea what's happening here? Any 
> information would be greatly appreciated.
> Thanks,
> Johnny



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 36858: Patch for KAFKA-2120

[Jason Gustafson]

> On Sept. 11, 2015, 1:44 a.m., Jason Gustafson wrote:
> > clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java, line 
> > 140
> > 
> >
> > This line is puzzling me a little bit. Wouldn't we want to use the 
> > oldest request for this check. If we use the last sent request, then we 
> > might miss older requests that had already timed out, which would tend to 
> > delay timeout detection. Maybe I'm missing something?
> 
> Mayuresh Gharat wrote:
> The inflight request are sent one after another. The reason we use 
> lastSent() is because if we know that if the last Sent is timed out, it means 
> that all older requests should have been timed out.

Ok, so the guarantee this provides is that each request will wait at least the 
timeout to be fulfilled, but some could wait longer. This treats the timeout 
setting as a minimum. The downside is that it will take longer to detect a node 
failure since we always have to wait for the latest request to timeout (which 
will delay the timeout for older requests). Is that right? Alternatively, if 
you used the oldest request, the timeout would be treated as a maximum: no 
request would wait longer than the timeout, but some could be cancelled sooner. 
In general, this would lead to earlier failure detection. I'm not sure I see 
the benefit of using the minimum approach, but it might be worthwhile 
documenting the tradeoff if you haven't already done so somewhere.


- Jason


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98526
---


On Sept. 11, 2015, 4:39 a.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 11, 2015, 4:39 a.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 049b22eadd5496b70dfcfd9d821f67c62c68a052 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 19ef6ebead3dc97d4912985e35b6b8d17503cb0e 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
>  4cb1e50d6c4ed55241aeaef1d3af09def5274103 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
>  a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
>  06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
>   
> clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
> d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
>   clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
> 70e74bd6aa629c430b2850ca40c97df0b16e5d75 
>   clients/src/main/java/org/apache/kafka/common/network/Selector.java 
> 4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
>   clients/src/test/java/org/apache/kafka/clients/MockClient.java 
> e5815f56bdf8e2

[jira] [Created] (KAFKA-2535) When removing a partition from an out of sync broker, it should not need to join the ISR first

[Sean Fellows (JIRA)]

Sean Fellows created KAFKA-2535:
---

 Summary: When removing a partition from an out of sync broker, it 
should not need to join the ISR first
 Key: KAFKA-2535
 URL: https://issues.apache.org/jira/browse/KAFKA-2535
 Project: Kafka
  Issue Type: Improvement
  Components: replication
Affects Versions: 0.8.2.0
Reporter: Sean Fellows
Assignee: Neha Narkhede
Priority: Minor


I'm not 100% sure this improvement would be legitimate, but I will describe the 
issue I have been seeing and hopefully folks can advise.

To reproduce:
1. Start with a broker that is out of sync for a particular partition.
2. Issue a reassignment that moves that partition off of that broker.

Observed behavior:
The broker continues replicating until it joins the ISR.

Desired behavior:
It seems to me that this broker should stop replication immediately. If this 
broker is able to replicate from somewhere, it means the ISR is not empty, so 
it should be safe to delete this partition's data from this broker.

The reason I am interested in this is that I would like to be able to reduce 
disk usage on a broker by removing partitions from it. But if it has fallen 
behind on replication it continues to need *more* disk space until it has 
caught up. But it is just going to delete this data anyway, so it seems like 
this optimization should be possible.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2276) Initial patch for KIP-25

[Geoff Anderson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741211#comment-14741211
 ] 

Geoff Anderson commented on KAFKA-2276:
---

Hey [~jkreps] - if we're ok with adding Kafka core as a dependency of the tools 
jar, then no we don't need both.
I suspect this logic was copy-pasted into clients (to avoid dragging in core 
dependency there), and then apparently I recreated the original class when I 
created the tools jar :)

> Initial patch for KIP-25
> 
>
> Key: KAFKA-2276
> URL: https://issues.apache.org/jira/browse/KAFKA-2276
> Project: Kafka
>  Issue Type: Bug
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
> Fix For: 0.8.3
>
>
> Submit initial patch for KIP-25 
> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-25+-+System+test+improvements)
> This patch should contain a few Service classes and a few tests which can 
> serve as examples 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2276) Initial patch for KIP-25

[Jay Kreps (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2276?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741225#comment-14741225
 ] 

Jay Kreps commented on KAFKA-2276:
--

[~granders] This is common code so it should go under clients/o.a.k.common, I 
think. That won't change any of the dependencies. Cut-and-paste isn't the right 
answer.

> Initial patch for KIP-25
> 
>
> Key: KAFKA-2276
> URL: https://issues.apache.org/jira/browse/KAFKA-2276
> Project: Kafka
>  Issue Type: Bug
>Reporter: Geoff Anderson
>Assignee: Geoff Anderson
> Fix For: 0.8.3
>
>
> Submit initial patch for KIP-25 
> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-25+-+System+test+improvements)
> This patch should contain a few Service classes and a few tests which can 
> serve as examples 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-31 - Message format change proposal

[Jiangjie Qin]

Ewen and Jay,

They way I see the LogAppendTime is another format of "offset". It serves
the following purpose:
1. Locate messages not only by position, but also by time. The difference
from offset is timestamp is not unique for all messags.
2. Allow broker to manage messages based on time, e.g. retention, rolling
3. Provide convenience for user to search message not only by offset, but
also by timestamp.

For purpose (2) we don't need per message server timestamp. We only need
per log segment server timestamp and propagate it among brokers.

For (1) and (3), we need per message timestamp. Then the question is
whether we should use CreateTime or LogAppendTime?

I completely agree that an application timestamp is very useful for many
use cases. But it seems to me that having Kafka to understand and maintain
application timestamp is a bit over demanding. So I think there is value to
pass on CreateTime for application convenience, but I am not sure it can
replace LogAppendTime. Managing out-of-order CreateTime is equivalent to
allowing producer to send their own offset and ask broker to manage the
offset for them, It is going to be very hard to maintain and could create
huge performance/functional issue because of complicated logic.

About whether we should expose LogAppendTime to broker, I agree that server
timestamp is internal to broker, but isn't offset also an internal concept?
Arguably it's not provided by producer so consumer application logic does
not have to know offset. But user needs to know offset because they need to
know "where is the message" in the log. LogAppendTime provides the answer
of "When was the message appended" to the log. So personally I think it is
reasonable to expose the LogAppendTime to consumers.

I can see some use cases of exposing the LogAppendTime, to name some:
1. Let's say broker has 7 days of log retention, some application wants to
reprocess the data in past 3 days. User can simply provide the timestamp
and start consume.
2. User can easily know lag by time.
3. Cross cluster fail over. This is a more complicated use case, there are
two goals: 1) Not lose message; and 2) do not reconsume tons of messages.
Only knowing offset of cluster A won't help with finding fail over point in
cluster B  because an offset of a cluster means nothing to another cluster.
Timestamp however is a good cross cluster reference in this case.

Thanks,

Jiangjie (Becket) Qin

On Thu, Sep 10, 2015 at 9:28 PM, Ewen Cheslack-Postava 
wrote:

> Re: MM preserving timestamps: Yes, this was how I interpreted the point in
> the KIP and I only raised the issue because it restricts the usefulness of
> timestamps anytime MM is involved. I agree it's not a deal breaker, but I
> wanted to understand exact impact of the change. Some users seem to want to
> be able to seek by application-defined timestamps (despite the many obvious
> issues involved), and the proposal clearly would not support that unless
> the timestamps submitted with the produce requests were respected. If we
> ignore client submitted timestamps, then we probably want to try to hide
> the timestamps as much as possible in any public interface (e.g. never
> shows up in any public consumer APIs), but expose it just enough to be
> useful for operational purposes.
>
> Sorry if my devil's advocate position / attempt to map the design space led
> to some confusion!
>
> -Ewen
>
>
> On Thu, Sep 10, 2015 at 5:48 PM, Jay Kreps  wrote:
>
> > Ah, I see, I think I misunderstood about MM, it was called out in the
> > proposal and I thought you were saying you'd retain the timestamp but I
> > think you're calling out that you're not. In that case you do have the
> > opposite problem, right? When you add mirroring for a topic all that data
> > will have a timestamp of now and retention won't be right. Not a blocker
> > but a bit of a gotcha.
> >
> > -Jay
> >
> >
> >
> > On Thu, Sep 10, 2015 at 5:40 PM, Joel Koshy  wrote:
> >
> > > > Don't you see all the same issues you see with client-defined
> > timestamp's
> > > > if you let mm control the timestamp as you were proposing? That means
> > > time
> > >
> > > Actually I don't think that was in the proposal (or was it?). i.e., I
> > > think it was always supposed to be controlled by the broker (and not
> > > MM).
> > >
> > > > Also, Joel, can you just confirm that you guys have talked through
> the
> > > > whole timestamp thing with the Samza folks at LI? The reason I ask
> > about
> > > > this is that Samza and Kafka Streams (KIP-28) are both trying to rely
> > on
> > >
> > > We have not. This is a good point - we will follow-up.
> > >
> > > > WRT your idea of a FollowerFetchRequestI had thought of a similar
> idea
> > > > where we use the leader's timestamps to approximately set the
> > follower's
> > > > timestamps. I had thought of just adding a partition metadata request
> > > that
> > > > would subsume the current offset/time lookup and could be used by the
> > > > follower to try to approximately keep

[jira] [Updated] (KAFKA-2477) Replicas spuriously deleting all segments in partition

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin updated KAFKA-2477:

Status: Patch Available  (was: Open)

> Replicas spuriously deleting all segments in partition
> --
>
> Key: KAFKA-2477
> URL: https://issues.apache.org/jira/browse/KAFKA-2477
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Håkon Hitland
>Assignee: Jiangjie Qin
> Attachments: kafka_log.txt, kafka_log_trace.txt
>
>
> We're seeing some strange behaviour in brokers: a replica will sometimes 
> schedule all segments in a partition for deletion, and then immediately start 
> replicating them back, triggering our check for under-replicating topics.
> This happens on average a couple of times a week, for different brokers and 
> topics.
> We have per-topic retention.ms and retention.bytes configuration, the topics 
> where we've seen this happen are hitting the size limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2477: Fix a race condition between log a...

[becketqin]

GitHub user becketqin reopened a pull request:

https://github.com/apache/kafka/pull/204

KAFKA-2477: Fix a race condition between log append and fetch that causes 
OffsetOutOfRangeException.

Tried two fixes. I prefer the second approach because it saves an 
additional offset search.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/becketqin/kafka KAFKA-2477

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/204.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #204


commit e7610fb69a4007ae661a768635e930355c8caa76
Author: Jiangjie Qin 
Date:   2015-09-11T02:17:12Z

KAFKA-2477: Fix a race condition between log append and fetch that causes 
OffsetOutOfRangeException

commit 45364d76e756fc6075924b3a07651b7fbbcc391a
Author: Jiangjie Qin 
Date:   2015-09-11T03:06:35Z

A second fix that avoids an addition offset search




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-2477: Fix a race condition between log a...

[becketqin]

Github user becketqin closed the pull request at:

https://github.com/apache/kafka/pull/204


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2477) Replicas spuriously deleting all segments in partition

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741275#comment-14741275
 ] 

ASF GitHub Bot commented on KAFKA-2477:
---

GitHub user becketqin reopened a pull request:

https://github.com/apache/kafka/pull/204

KAFKA-2477: Fix a race condition between log append and fetch that causes 
OffsetOutOfRangeException.

Tried two fixes. I prefer the second approach because it saves an 
additional offset search.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/becketqin/kafka KAFKA-2477

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/204.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #204


commit e7610fb69a4007ae661a768635e930355c8caa76
Author: Jiangjie Qin 
Date:   2015-09-11T02:17:12Z

KAFKA-2477: Fix a race condition between log append and fetch that causes 
OffsetOutOfRangeException

commit 45364d76e756fc6075924b3a07651b7fbbcc391a
Author: Jiangjie Qin 
Date:   2015-09-11T03:06:35Z

A second fix that avoids an addition offset search




> Replicas spuriously deleting all segments in partition
> --
>
> Key: KAFKA-2477
> URL: https://issues.apache.org/jira/browse/KAFKA-2477
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Håkon Hitland
>Assignee: Jiangjie Qin
> Attachments: kafka_log.txt, kafka_log_trace.txt
>
>
> We're seeing some strange behaviour in brokers: a replica will sometimes 
> schedule all segments in a partition for deletion, and then immediately start 
> replicating them back, triggering our check for under-replicating topics.
> This happens on average a couple of times a week, for different brokers and 
> topics.
> We have per-topic retention.ms and retention.bytes configuration, the topics 
> where we've seen this happen are hitting the size limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2477) Replicas spuriously deleting all segments in partition

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741274#comment-14741274
 ] 

ASF GitHub Bot commented on KAFKA-2477:
---

Github user becketqin closed the pull request at:

https://github.com/apache/kafka/pull/204


> Replicas spuriously deleting all segments in partition
> --
>
> Key: KAFKA-2477
> URL: https://issues.apache.org/jira/browse/KAFKA-2477
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Håkon Hitland
>Assignee: Jiangjie Qin
> Attachments: kafka_log.txt, kafka_log_trace.txt
>
>
> We're seeing some strange behaviour in brokers: a replica will sometimes 
> schedule all segments in a partition for deletion, and then immediately start 
> replicating them back, triggering our check for under-replicating topics.
> This happens on average a couple of times a week, for different brokers and 
> topics.
> We have per-topic retention.ms and retention.bytes configuration, the topics 
> where we've seen this happen are hitting the size limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-2529) Brokers should write current version to log when they first start

[Gwen Shapira (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Gwen Shapira resolved KAFKA-2529.
-
Resolution: Fixed

Sorry, missed the jira that fixed this.

> Brokers should write current version to log when they first start
> -
>
> Key: KAFKA-2529
> URL: https://issues.apache.org/jira/browse/KAFKA-2529
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>
> It is currently non-trivial to tell, by looking at log files, which version 
> of Kafka is the log from. 
> Having this information can be useful in some troubleshooting scenarios. We 
> are exposing this via JMX, but since troubleshooting usually involves asking 
> for logs, it will be nice if this information will be included there.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2536) topics tool should allow users to alter topic configuration

[Gwen Shapira (JIRA)]

Gwen Shapira created KAFKA-2536:
---

 Summary: topics tool should allow users to alter topic 
configuration
 Key: KAFKA-2536
 URL: https://issues.apache.org/jira/browse/KAFKA-2536
 Project: Kafka
  Issue Type: Bug
Reporter: Gwen Shapira


When we added dynamic config, we added a kafka-config tool (which can be used 
to maintain configs for non-topic entities), and remove the capability from 
kafka-topic tool.

Removing the capability from kafka-topic is:
1. Breaks backward compatibility in our most essential tools. This has 
significant impact on usability.
2. Kinda confusing that --create --config works but --alter --config does not. 

I suggest fixing this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1929) Convert core kafka module to use the errors in org.apache.kafka.common.errors

[Grant Henke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Henke updated KAFKA-1929:
---
Assignee: (was: Grant Henke)

> Convert core kafka module to use the errors in org.apache.kafka.common.errors
> -
>
> Key: KAFKA-1929
> URL: https://issues.apache.org/jira/browse/KAFKA-1929
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jay Kreps
> Attachments: KAFKA-1929.patch
>
>
> With the introduction of the common package there are now a lot of errors 
> duplicated in both the common package and in the server. We should refactor 
> the server code (but not the scala clients) to switch over to the exceptions 
> in common.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2229) Phase 1: Requests and KafkaApis

[Grant Henke (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741307#comment-14741307
 ] 

Grant Henke commented on KAFKA-2229:


[~abiletskyi] Are you still working on this? If not, perhaps I can take it? 
Otherwise I am happy to help in anyway I can.

> Phase 1: Requests and KafkaApis
> ---
>
> Key: KAFKA-2229
> URL: https://issues.apache.org/jira/browse/KAFKA-2229
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Andrii Biletskyi
>Assignee: Andrii Biletskyi
> Attachments: KAFKA-2229.patch, KAFKA-2229_2015-06-30_16:59:17.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1929) Convert core kafka module to use the errors in org.apache.kafka.common.errors

[Grant Henke (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741299#comment-14741299
 ] 

Grant Henke commented on KAFKA-1929:


[~hachikuji], feel free to pick this up

> Convert core kafka module to use the errors in org.apache.kafka.common.errors
> -
>
> Key: KAFKA-1929
> URL: https://issues.apache.org/jira/browse/KAFKA-1929
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jay Kreps
>Assignee: Grant Henke
> Attachments: KAFKA-1929.patch
>
>
> With the introduction of the common package there are now a lot of errors 
> duplicated in both the common package and in the server. We should refactor 
> the server code (but not the scala clients) to switch over to the exceptions 
> in common.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-31 - Message format change proposal

[Joel Koshy]

Jay had mentioned the scenario of mirror-maker bootstrap which would
effectively reset the logAppendTimestamps for the bootstrapped data.
If we don't include logAppendTimestamps in each message there is a
similar scenario when rebuilding indexes during recovery. So it seems
it may be worth adding that timestamp to messages. The drawback to
that is exposing a server-side concept in the protocol (although we
already do that with offsets). logAppendTimestamp really should be
decided by the broker so I think the first scenario may have to be
written off as a gotcha, but the second may be worth addressing (by
adding it to the message format).

The other point that Jay raised which needs to be addressed (since we
require monotically increasing timestamps in the index) in the
proposal is changing time on the server (I'm a little less concerned
about NTP clock skews than a user explicitly changing the server's
time - i.e., big clock skews). We would at least want to "set back"
all the existing timestamps to guarantee non-decreasing timestamps
with future messages. I'm not sure at this point how best to handle
that, but we could perhaps have a epoch/base-time (or time-correction)
stored in the log directories and base all log index timestamps off
that base-time (or corrected). So if at any time you determine that
time has changed backwards you can adjust that base-time without
having to fix up all the entries. Without knowing the exact diff
between the previous clock and new clock we cannot adjust the times
exactly, but we can at least ensure increasing timestamps.

On Fri, Sep 11, 2015 at 10:52 AM, Jiangjie Qin
 wrote:
> Ewen and Jay,
>
> They way I see the LogAppendTime is another format of "offset". It serves
> the following purpose:
> 1. Locate messages not only by position, but also by time. The difference
> from offset is timestamp is not unique for all messags.
> 2. Allow broker to manage messages based on time, e.g. retention, rolling
> 3. Provide convenience for user to search message not only by offset, but
> also by timestamp.
>
> For purpose (2) we don't need per message server timestamp. We only need
> per log segment server timestamp and propagate it among brokers.
>
> For (1) and (3), we need per message timestamp. Then the question is
> whether we should use CreateTime or LogAppendTime?
>
> I completely agree that an application timestamp is very useful for many
> use cases. But it seems to me that having Kafka to understand and maintain
> application timestamp is a bit over demanding. So I think there is value to
> pass on CreateTime for application convenience, but I am not sure it can
> replace LogAppendTime. Managing out-of-order CreateTime is equivalent to
> allowing producer to send their own offset and ask broker to manage the
> offset for them, It is going to be very hard to maintain and could create
> huge performance/functional issue because of complicated logic.
>
> About whether we should expose LogAppendTime to broker, I agree that server
> timestamp is internal to broker, but isn't offset also an internal concept?
> Arguably it's not provided by producer so consumer application logic does
> not have to know offset. But user needs to know offset because they need to
> know "where is the message" in the log. LogAppendTime provides the answer
> of "When was the message appended" to the log. So personally I think it is
> reasonable to expose the LogAppendTime to consumers.
>
> I can see some use cases of exposing the LogAppendTime, to name some:
> 1. Let's say broker has 7 days of log retention, some application wants to
> reprocess the data in past 3 days. User can simply provide the timestamp
> and start consume.
> 2. User can easily know lag by time.
> 3. Cross cluster fail over. This is a more complicated use case, there are
> two goals: 1) Not lose message; and 2) do not reconsume tons of messages.
> Only knowing offset of cluster A won't help with finding fail over point in
> cluster B  because an offset of a cluster means nothing to another cluster.
> Timestamp however is a good cross cluster reference in this case.
>
> Thanks,
>
> Jiangjie (Becket) Qin
>
> On Thu, Sep 10, 2015 at 9:28 PM, Ewen Cheslack-Postava 
> wrote:
>
>> Re: MM preserving timestamps: Yes, this was how I interpreted the point in
>> the KIP and I only raised the issue because it restricts the usefulness of
>> timestamps anytime MM is involved. I agree it's not a deal breaker, but I
>> wanted to understand exact impact of the change. Some users seem to want to
>> be able to seek by application-defined timestamps (despite the many obvious
>> issues involved), and the proposal clearly would not support that unless
>> the timestamps submitted with the produce requests were respected. If we
>> ignore client submitted timestamps, then we probably want to try to hide
>> the timestamps as much as possible in any public interface (e.g. never
>> shows up in any public consumer APIs), but expose it just enough to be
>> usef

[jira] [Created] (KAFKA-2537) Mirrormaker defaults to localhost with no sanity checks

[Evan Huus (JIRA)]

Evan Huus created KAFKA-2537:


 Summary: Mirrormaker defaults to localhost with no sanity checks
 Key: KAFKA-2537
 URL: https://issues.apache.org/jira/browse/KAFKA-2537
 Project: Kafka
  Issue Type: Bug
  Components: consumer, replication, zkclient
Affects Versions: 0.8.2.0
Reporter: Evan Huus
Assignee: Neha Narkhede


Short version: Like many other tools, mirror-maker's consumer defaults to using 
the localhost zookeeper instance when no specific zookeeper source is 
specified. It shouldn't do this. MM should also have a sanity check that the 
source and destination clusters are different.


Long version: We run multiple clusters, all using mirrormaker to replicate to 
the master cluster. The kafka, zookeeper, and mirrormaker instances all run on 
the same nodes in the master cluster since the hardware can more than handle 
the load. We were doing some zookeeper maintenance on one of our remote 
clusters recently which accidentally caused our configuration manager (chef) to 
generate empty zkConnect strings for some mirrormaker instances. These 
instances defaulted to localhost and started mirroring from the master cluster 
back to itself, an infinite replication loop that caused all sorts of havok.

We were able to recover gracefully and we've added additional safe-guards on 
our end, but mirror-maker is at least partially at fault here as well. There is 
no reason for it to treat an empty string as anything but an error - especially 
not localhost, which is typically the target cluster, not the source. 
Additionally, it should be trivial and very useful for mirrormaker to verify it 
is not consuming and producing from the same cluster; I can think of no 
legitimate use case for this kind of cycle.

If you need any clarification or additional information, please let me know.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2373) Copycat distributed offset storage

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2373?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-2373:
-
Status: Patch Available  (was: Open)

> Copycat distributed offset storage
> --
>
> Key: KAFKA-2373
> URL: https://issues.apache.org/jira/browse/KAFKA-2373
> Project: Kafka
>  Issue Type: Sub-task
>  Components: copycat
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.8.3
>
>
> Add offset storage for Copycat that works in distributed mode, which likely 
> means storing the data in a Kafka topic. Copycat workers will use this by 
> default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/
---

(Updated Sept. 11, 2015, 9:54 p.m.)


Review request for kafka.


Bugs: KAFKA-2120
https://issues.apache.org/jira/browse/KAFKA-2120


Repository: kafka


Description (updated)
---

Solved compile error


Addressed Jason's comments for Kip-19


Addressed Jun's comments


Addressed Jason's comments about the default values for requestTimeout


checkpoint


Addressed Joel's concerns. Also tried to include Jun's feedback.


Fixed a minor comment


Solved unittest issue


Addressed Jun's comments regarding NetworkClient


Addressed Jun's comments about disconnect() in Selector


changed logging level to debug


Addressed Joels comments to break out early from the loop while aborting 
expired batches


Addressed Jun's comments


Addressed Jason's concern about iterating over timeout request in 
getNodesWithTimedOutRequest()


Diffs (updated)
-

  clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
dc8f0f115bcda893c95d17c0a57be8d14518d034 
  clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
  clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
  clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
  clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
1302f359791b657b6f7c1ca1bd419ded9b01c67d 
  clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
  clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
5763bac6cfe667bfbabc5f160f35fb85f9b158e3 
  
clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
 9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
  clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
804d569498396d431880641041fc9292076452cb 
  clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
06f00a99a73a288df9afa8c1d4abe3580fa968a6 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/BufferPool.java
 4cb1e50d6c4ed55241aeaef1d3af09def5274103 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordAccumulator.java
 a152bd7697dca55609a9ec4cfe0a82c10595fbc3 
  
clients/src/main/java/org/apache/kafka/clients/producer/internals/RecordBatch.java
 06182db1c3a5da85648199b4c0c98b80ea7c6c0c 
  clients/src/main/java/org/apache/kafka/clients/producer/internals/Sender.java 
d2e64f7cd8bf56e433a210905b2874f71eee9ea0 
  clients/src/main/java/org/apache/kafka/common/network/Selectable.java 
70e74bd6aa629c430b2850ca40c97df0b16e5d75 
  clients/src/main/java/org/apache/kafka/common/network/Selector.java 
4aa5cbb86ce6e1bf8f6769147ee2a6452c855c74 
  clients/src/test/java/org/apache/kafka/clients/MockClient.java 
e5815f56bdf8e2d980f2bc36b831ed234c0ac781 
  clients/src/test/java/org/apache/kafka/clients/NetworkClientTest.java 
69c93c3adf674b1640534c3d7410fcaafaf2232c 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/BufferPoolTest.java
 2c693824fa53db1e38766b8c66a0ef42ef9d0f3a 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/RecordAccumulatorTest.java
 5b2e4ffaeab7127648db608c179703b27b577414 
  
clients/src/test/java/org/apache/kafka/clients/producer/internals/SenderTest.java
 aa44991777a855f4b7f4f7bf17107c69393ff8ff 
  clients/src/test/java/org/apache/kafka/common/network/SSLSelectorTest.java 
df1205c935bee9a30a50816dbade64d6014b1ef2 
  clients/src/test/java/org/apache/kafka/common/network/SelectorTest.java 
3a684d98b05cadfb25c6f7f9a038ef1f6697edbf 
  clients/src/test/java/org/apache/kafka/test/MockSelector.java 
f83fd9b794a3bd191121a22bcb40fd6ec31d83b5 
  core/src/main/scala/kafka/controller/ControllerChannelManager.scala 
da1cff07f7f76dcfa5a805718febcccd4ed5f578 
  core/src/main/scala/kafka/server/KafkaConfig.scala 
1e8b2331486ffe55bfcc0919e48e12aad23b7d3c 
  core/src/main/scala/kafka/server/KafkaServer.scala 
30406ce809caaac56aca1f30c235b35962d55a50 
  core/src/main/scala/kafka/tools/ProducerPerformance.scala 
46a68e97b8bcc8821f21e4220ce9b3acedc5dafe 
  core/src/main/scala/kafka/utils/NetworkClientBlockingOps.scala 
ad10721de844725f27a116611209992cea61b088 
  core/src/test/scala/integration/kafka/api/ProducerFailureHandlingTest.scala 
1198df02ddd7727269e84a751ba99520f6d5584a 
  core/src/test/scala/unit/kafka/server/KafkaConfigTest.scala 
5b4f2db4607ae6d17696c1140f1a771ce75c80e0 
  core/src/test/scala/unit/kafka/utils/TestUtils.scala 
09b8444c2add87f0f70dbb182e892977a6b5c243 

Diff: https://reviews.apache.org/r/36858/diff/


Testing
---


Thanks,

Mayuresh Gharat

[jira] [Updated] (KAFKA-2120) Add a request timeout to NetworkClient

[Mayuresh Gharat (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mayuresh Gharat updated KAFKA-2120:
---
Attachment: KAFKA-2120_2015-09-11_14:54:15.patch

> Add a request timeout to NetworkClient
> --
>
> Key: KAFKA-2120
> URL: https://issues.apache.org/jira/browse/KAFKA-2120
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Jiangjie Qin
>Assignee: Mayuresh Gharat
>Priority: Blocker
> Fix For: 0.8.3
>
> Attachments: KAFKA-2120.patch, KAFKA-2120_2015-07-27_15:31:19.patch, 
> KAFKA-2120_2015-07-29_15:57:02.patch, KAFKA-2120_2015-08-10_19:55:18.patch, 
> KAFKA-2120_2015-08-12_10:59:09.patch, KAFKA-2120_2015-09-03_15:12:02.patch, 
> KAFKA-2120_2015-09-04_17:49:01.patch, KAFKA-2120_2015-09-09_16:45:44.patch, 
> KAFKA-2120_2015-09-09_18:56:18.patch, KAFKA-2120_2015-09-10_21:38:55.patch, 
> KAFKA-2120_2015-09-11_14:54:15.patch
>
>
> Currently NetworkClient does not have a timeout setting for requests. So if 
> no response is received for a request due to reasons such as broker is down, 
> the request will never be completed.
> Request timeout will also be used as implicit timeout for some methods such 
> as KafkaProducer.flush() and kafkaProducer.close().
> KIP-19 is created for this public interface change.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2120) Add a request timeout to NetworkClient

[Mayuresh Gharat (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741594#comment-14741594
 ] 

Mayuresh Gharat commented on KAFKA-2120:


Updated reviewboard https://reviews.apache.org/r/36858/diff/
 against branch origin/trunk

> Add a request timeout to NetworkClient
> --
>
> Key: KAFKA-2120
> URL: https://issues.apache.org/jira/browse/KAFKA-2120
> Project: Kafka
>  Issue Type: New Feature
>Reporter: Jiangjie Qin
>Assignee: Mayuresh Gharat
>Priority: Blocker
> Fix For: 0.8.3
>
> Attachments: KAFKA-2120.patch, KAFKA-2120_2015-07-27_15:31:19.patch, 
> KAFKA-2120_2015-07-29_15:57:02.patch, KAFKA-2120_2015-08-10_19:55:18.patch, 
> KAFKA-2120_2015-08-12_10:59:09.patch, KAFKA-2120_2015-09-03_15:12:02.patch, 
> KAFKA-2120_2015-09-04_17:49:01.patch, KAFKA-2120_2015-09-09_16:45:44.patch, 
> KAFKA-2120_2015-09-09_18:56:18.patch, KAFKA-2120_2015-09-10_21:38:55.patch, 
> KAFKA-2120_2015-09-11_14:54:15.patch
>
>
> Currently NetworkClient does not have a timeout setting for requests. So if 
> no response is received for a request due to reasons such as broker is down, 
> the request will never be completed.
> Request timeout will also be used as implicit timeout for some methods such 
> as KafkaProducer.flush() and kafkaProducer.close().
> KIP-19 is created for this public interface change.
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-19+-+Add+a+request+timeout+to+NetworkClient



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Review Request 36858: Patch for KAFKA-2120

[Mayuresh Gharat]

> On Sept. 11, 2015, 1:44 a.m., Jason Gustafson wrote:
> > clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java, line 
> > 140
> > 
> >
> > This line is puzzling me a little bit. Wouldn't we want to use the 
> > oldest request for this check. If we use the last sent request, then we 
> > might miss older requests that had already timed out, which would tend to 
> > delay timeout detection. Maybe I'm missing something?
> 
> Mayuresh Gharat wrote:
> The inflight request are sent one after another. The reason we use 
> lastSent() is because if we know that if the last Sent is timed out, it means 
> that all older requests should have been timed out.
> 
> Jason Gustafson wrote:
> Ok, so the guarantee this provides is that each request will wait at 
> least the timeout to be fulfilled, but some could wait longer. This treats 
> the timeout setting as a minimum. The downside is that it will take longer to 
> detect a node failure since we always have to wait for the latest request to 
> timeout (which will delay the timeout for older requests). Is that right? 
> Alternatively, if you used the oldest request, the timeout would be treated 
> as a maximum: no request would wait longer than the timeout, but some could 
> be cancelled sooner. In general, this would lead to earlier failure 
> detection. I'm not sure I see the benefit of using the minimum approach, but 
> it might be worthwhile documenting the tradeoff if you haven't already done 
> so somewhere.

Hi Jason,

Now I understand what you are saying. Thats an excellent catch. Although the 
time diference between the subsequent request will be minimum and my current 
patch will detect the timeout and do metadata update, I completely agree with 
you : We should be peeking the tail element from the queue rather than the head 
element. Will upload a new patch for it. Thanks a lot for this.


Actually thinking further we should be doing the same thing when we are 
iterating over completedSends() in NetworkClient.handleCompletedSends(). 
Although it works the current way, its kind of confusing if we don't keep the 
iteration consistent. What do you think on this?


- Mayuresh


---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/36858/#review98526
---


On Sept. 11, 2015, 9:54 p.m., Mayuresh Gharat wrote:
> 
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/36858/
> ---
> 
> (Updated Sept. 11, 2015, 9:54 p.m.)
> 
> 
> Review request for kafka.
> 
> 
> Bugs: KAFKA-2120
> https://issues.apache.org/jira/browse/KAFKA-2120
> 
> 
> Repository: kafka
> 
> 
> Description
> ---
> 
> Solved compile error
> 
> 
> Addressed Jason's comments for Kip-19
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's comments about the default values for requestTimeout
> 
> 
> checkpoint
> 
> 
> Addressed Joel's concerns. Also tried to include Jun's feedback.
> 
> 
> Fixed a minor comment
> 
> 
> Solved unittest issue
> 
> 
> Addressed Jun's comments regarding NetworkClient
> 
> 
> Addressed Jun's comments about disconnect() in Selector
> 
> 
> changed logging level to debug
> 
> 
> Addressed Joels comments to break out early from the loop while aborting 
> expired batches
> 
> 
> Addressed Jun's comments
> 
> 
> Addressed Jason's concern about iterating over timeout request in 
> getNodesWithTimedOutRequest()
> 
> 
> Diffs
> -
> 
>   clients/src/main/java/org/apache/kafka/clients/ClientRequest.java 
> dc8f0f115bcda893c95d17c0a57be8d14518d034 
>   clients/src/main/java/org/apache/kafka/clients/CommonClientConfigs.java 
> 7d24c6f5dd2b63b96584f3aa8922a1d048dc1ae4 
>   clients/src/main/java/org/apache/kafka/clients/InFlightRequests.java 
> 15d00d4e484bb5d51a9ae6857ed6e024a2cc1820 
>   clients/src/main/java/org/apache/kafka/clients/KafkaClient.java 
> f46c0d9b5eb73887c62a0e09c96e9d8c964c709d 
>   clients/src/main/java/org/apache/kafka/clients/NetworkClient.java 
> 1302f359791b657b6f7c1ca1bd419ded9b01c67d 
>   clients/src/main/java/org/apache/kafka/clients/consumer/ConsumerConfig.java 
> b9a2d4e2bc565f0ee72b27791afe5c894af262f1 
>   clients/src/main/java/org/apache/kafka/clients/consumer/KafkaConsumer.java 
> 5763bac6cfe667bfbabc5f160f35fb85f9b158e3 
>   
> clients/src/main/java/org/apache/kafka/clients/consumer/internals/ConsumerNetworkClient.java
>  9517d9d0cd480d5ba1d12f1fde7963e60528d2f8 
>   clients/src/main/java/org/apache/kafka/clients/producer/KafkaProducer.java 
> 804d569498396d431880641041fc9292076452cb 
>   clients/src/main/java/org/apache/kafka/clients/producer/ProducerConfig.java 
> 06f00a99a73a288df9afa8c1d4abe3580fa968a6 
>   
> clients/src/main/jav

[jira] [Commented] (KAFKA-2389) CommitType seems not necessary in commit().

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741654#comment-14741654
 ] 

ASF GitHub Bot commented on KAFKA-2389:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/134


> CommitType seems not necessary in commit().
> ---
>
> Key: KAFKA-2389
> URL: https://issues.apache.org/jira/browse/KAFKA-2389
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.8.3
>
>
> The CommitType does not seem to be necessary in for commit(), it can be 
> inferred from whether user passed in a callback or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2389) CommitType seems not necessary in commit().

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2389:
-
   Resolution: Fixed
Fix Version/s: 0.8.3
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 134
[https://github.com/apache/kafka/pull/134]

> CommitType seems not necessary in commit().
> ---
>
> Key: KAFKA-2389
> URL: https://issues.apache.org/jira/browse/KAFKA-2389
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.8.3
>
>
> The CommitType does not seem to be necessary in for commit(), it can be 
> inferred from whether user passed in a callback or not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2389: remove commit type from new consum...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/134


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-2278; JmxTool should support querying al...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/181


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2278) JmxTool should support querying all objects when object-name is omitted

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741675#comment-14741675
 ] 

ASF GitHub Bot commented on KAFKA-2278:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/181


> JmxTool should support querying all objects when object-name is omitted
> ---
>
> Key: KAFKA-2278
> URL: https://issues.apache.org/jira/browse/KAFKA-2278
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Dong Lin
>Assignee: Dong Lin
> Fix For: 0.9.0
>
> Attachments: KAFKA-2278.patch
>
>
> One usecase of JmxTool is to allow user to query all existing objects and 
> attributes, such that the developer can quickly lookup the right object name 
> and attribute name in the output. Indeed, existing API of JmxTool allows user 
> to do this when object-name is not specified in argument.
> However, current implementation of JmxTool doesn't support this operation due 
> to the way it is implemented. This is partly due to the fact that some JMX 
> object, such as java.lang:type=MemoryPool, doesn't have value for all its 
> attributes.
> The patch submitted here will fix the problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2278) JmxTool should support querying all objects when object-name is omitted

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2278:
-
   Resolution: Fixed
Fix Version/s: 0.9.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 181
[https://github.com/apache/kafka/pull/181]

> JmxTool should support querying all objects when object-name is omitted
> ---
>
> Key: KAFKA-2278
> URL: https://issues.apache.org/jira/browse/KAFKA-2278
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Dong Lin
>Assignee: Dong Lin
> Fix For: 0.9.0
>
> Attachments: KAFKA-2278.patch
>
>
> One usecase of JmxTool is to allow user to query all existing objects and 
> attributes, such that the developer can quickly lookup the right object name 
> and attribute name in the output. Indeed, existing API of JmxTool allows user 
> to do this when object-name is not specified in argument.
> However, current implementation of JmxTool doesn't support this operation due 
> to the way it is implemented. This is partly due to the fact that some JMX 
> object, such as java.lang:type=MemoryPool, doesn't have value for all its 
> attributes.
> The patch submitted here will fix the problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2440) Use `NetworkClient` instead of `SimpleConsumer` to fetch data from replica

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741699#comment-14741699
 ] 

ASF GitHub Bot commented on KAFKA-2440:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/194


> Use `NetworkClient` instead of `SimpleConsumer` to fetch data from replica
> --
>
> Key: KAFKA-2440
> URL: https://issues.apache.org/jira/browse/KAFKA-2440
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>Priority: Blocker
> Fix For: 0.8.3
>
>
> This is necessary for SSL/TLS support for inter-broker communication as 
> `SimpleConsumer` will not be updated to support SSL/TLS.
> As explained by [~junrao] in KAFKA-2411: we need to be a bit careful since 
> the follower fetcher thread doesn't need to refresh metadata itself. Instead, 
> the information about the leader is propagated from the controller.
> This work was originally described in KAFKA-2411, which was then updated to 
> be more narrowly focused on replacing `BlockingChannel` with `Selector`.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2440) Use `NetworkClient` instead of `SimpleConsumer` to fetch data from replica

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-2440:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 194
[https://github.com/apache/kafka/pull/194]

> Use `NetworkClient` instead of `SimpleConsumer` to fetch data from replica
> --
>
> Key: KAFKA-2440
> URL: https://issues.apache.org/jira/browse/KAFKA-2440
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Ismael Juma
>Assignee: Ismael Juma
>Priority: Blocker
> Fix For: 0.8.3
>
>
> This is necessary for SSL/TLS support for inter-broker communication as 
> `SimpleConsumer` will not be updated to support SSL/TLS.
> As explained by [~junrao] in KAFKA-2411: we need to be a bit careful since 
> the follower fetcher thread doesn't need to refresh metadata itself. Instead, 
> the information about the leader is propagated from the controller.
> This work was originally described in KAFKA-2411, which was then updated to 
> be more narrowly focused on replacing `BlockingChannel` with `Selector`.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2440; Use `NetworkClient` instead of `Si...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/194


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-2278) JmxTool should support querying all objects when object-name is omitted

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-2278:
---
Fix Version/s: (was: 0.9.0)
   0.8.3

> JmxTool should support querying all objects when object-name is omitted
> ---
>
> Key: KAFKA-2278
> URL: https://issues.apache.org/jira/browse/KAFKA-2278
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Dong Lin
>Assignee: Dong Lin
> Fix For: 0.8.3
>
> Attachments: KAFKA-2278.patch
>
>
> One usecase of JmxTool is to allow user to query all existing objects and 
> attributes, such that the developer can quickly lookup the right object name 
> and attribute name in the output. Indeed, existing API of JmxTool allows user 
> to do this when object-name is not specified in argument.
> However, current implementation of JmxTool doesn't support this operation due 
> to the way it is implemented. This is partly due to the fact that some JMX 
> object, such as java.lang:type=MemoryPool, doesn't have value for all its 
> attributes.
> The patch submitted here will fix the problem.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Kafka KIP meeting Sep 14 at 9:30am PST

[Jun Rao]

Hi, Everyone,

Since most people from Confluent will be out Tuesday, we will have a Kafka
KIP meeting on Sep. 14 (Monday) at 9:30am PST. If you plan to attend but
haven't received an invite, please let me know. The following is the agenda.

Agenda:
1. KIP-28: Add a processor client
2. KIP-31: Move to relative offsets in compressed message sets

Thanks,

Jun

Build failed in Jenkins: Kafka-trunk #620

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-2389: remove commit type from new consumer.

[wangguoz] KAFKA-2278: JmxTool should support querying all objects when object 
name is omitted

[junrao] KAFKA-2440; Use `NetworkClient` instead of `SimpleConsumer` to fetch 
data from replica

--
[...truncated 1703 lines...]
kafka.utils.UtilsTest > testCircularIterator PASSED

kafka.utils.UtilsTest > testReadBytes PASSED

kafka.utils.UtilsTest > testCsvList PASSED

kafka.utils.UtilsTest > testReadInt PASSED

kafka.utils.UtilsTest > testCsvMap PASSED

kafka.utils.UtilsTest > testInLock PASSED

kafka.utils.UtilsTest > testSwallow PASSED

kafka.integration.SslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.api.ConsumerTest > testUnsubscribeTopic PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[0] PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicAlreadyMarkedAsDeleted PASSED

kafka.admin.AdminTest > testShutdownBroker PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.admin.AdminTest > testTopicCreationWithCollision PASSED

kafka.admin.AdminTest > testTopicCreationInZK PASSED

kafka.admin.AddPartitionsTest > testReplicaPlacement PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.api.QuotasTest > testThrottledProducerConsumer PASSED

kafka.utils.JsonTest > testJsonEncoding PASSED

kafka.api.ConsumerTest > testListTopics PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.admin.DeleteTopicTest > testPartitionReassignmentDuringDeleteTopic PASSED

kafka.log.LogTest > testCorruptLog PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.log.LogTest > testLogRecoversToCorrectOffset PASSED

kafka.log.LogTest > testReopenThenTruncate PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingPartition PASSED

kafka.log.LogTest > testParseTopicPartitionNameForEmptyName PASSED

kafka.admin.DeleteTopicTest > testDeleteNonExistingTopic PASSED

kafka.log.LogTest > testOpenDeletesObsoleteFiles PASSED

kafka.log.LogTest > testSizeBasedLogRoll PASSED

kafka.log.LogTest > testTimeBasedLogRollJitter PASSED

kafka.log.LogTest > testParseTopicPartitionName PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.log.LogTest > testTruncateTo PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.log.LogTest > testCleanShutdownFile PASSED

kafka.api.ConsumerTest > testExpandingTopicSubscriptions PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionEnabled 
PASSED

kafka.api.ProducerFailureHandlingTest > testCannotSendToInternalTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.admin.DeleteTopicTest > testRecreateTopicAfterDeletion PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.api.ProducerFailureHandlingTest > testTooLargeRecordWithAckOne PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.api.ConsumerTest > testPatternUnsubscription PASSED

kafka.api.ConsumerBounceTest > testConsumptionWithBrokerFailures PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg PASSED

kafka.utils.CommandLineUtilsTest > testParseSingleArg PASSED

kafka.utils.CommandLineUtilsTest > testParseArgs PASSED

kafka.admin.DeleteTopicTest > testAddPartitionDuringDeleteTopic PASSED

kafka.api.ProducerFailureHandlingTest > testWrongBrokerList PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.api.ConsumerTest > testGroupConsumption PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicWithAllAliveReplicas PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.api.ProducerFailureHandlingTest > testNotEnoughReplicas PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[1] PASSED

kafka.integration.PlaintextTopicMetadataTest > testTopicMetadataRequest PASSED

kafka.api.ConsumerTest > testPartitionsFor PASSED

kafka.admin.DeleteTopicTest > testDeleteTopicDuringAddPartition PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.common.ConfigTest > testInvalidGroupIds PASSED

kafka.common.ConfigTest > testInvalidClientIds PASSED

kafka.integration.MinIsrConfigTest > testDefaultKafkaConfig PASSED

kafka.api.ProducerFailureHandlingTest > testNoResponse PASSED

kafka.integration.UncleanLeaderElectionTest > 
testCleanLeaderElectionDisabledByTopicOverride PASSED

kafka.api.ProducerFailureHandlingTest

Kafka cluster cannot start anymore after unexpected shutdown

[Qi Xu]

Hi,
We're running the Trunk version of Kafka (for its SSL feature) and recently
I'm trying to enable the kafka manager with it.
After enabling that,  I find out some machine's Kafka Server is dead.
Looking at the server.log, it has the following logs.
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:19:41,310] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:19:44,766] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:19:48,567] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:19:52,112] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:19:56,307] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space


And then I tried to start Kafka on these dead machines, but find the errors
below:
[2015-09-12 00:43:44,878] WARN Found an corrupted index file,
/datadrive/kafka-logs/userlogs-139/.index, deleting and
rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,883] WARN Found an corrupted index file,
/datadrive/kafka-logs/userlogs-166/.index, deleting and
rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,888] WARN Found an corrupted index file,
/datadrive/kafka-logs/exoactivitylogs-114/.index,
deleting and rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,895] WARN Found an corrupted index file,
/datadrive/kafka-logs/exoactivitylogs-15/.index,
deleting and rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,900] WARN Found an corrupted index file,
/datadrive/kafka-logs/exoactivitylogs-178/.index,
deleting and rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,906] WARN Found an corrupted index file,
/datadrive/kafka-logs/__consumer_offsets-4/.index,
deleting and rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,912] WARN Found an corrupted index file,
/datadrive/kafka-logs/userlogs-147/.index, deleting and
rebuilding index... (kafka.log.Log)
[2015-09-12 00:43:44,917] WARN Found an corrupted index file,
/datadrive/kafka-logs/exoactivitylogs-95/.index,
deleting and rebuilding index... (kafka.log.Log)
Exception in thread "kafka-log-cleaner-thread-0"
java.lang.OutOfMemoryError: Java heap space

Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "main-SendThread(10.1.130.12:2181)"
[2015-09-12 00:44:39,466] WARN Session 0x14f84cab5240046 for server
10.1.130.12/10.1.130.12:2181, unexpected error, closing socket connection
and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2015-09-12 00:44:45,169] ERROR from main-SendThread(10.1.130.12:2181)
(org.apache.zookeeper.ClientCnxn)
[2015-09-12 00:45:39,466] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:39,467] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:39,732] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:41,299] ERROR Uncaught exception in scheduled task
'highwatermark-checkpoint' (kafka.utils.KafkaScheduler)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:49,070] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:49,702] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:49,912] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 00:45:50,140] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space

>From zookeeper, I can see that node appear in /brokers/ids, but disappear
when the error above happens.

Do you know what's the problem? Any hint is very appreciated.

Thanks,
Qi

Re: Kafka cluster cannot start anymore after unexpected shutdown

[Qi Xu]

And I tried to clean up the whole kafka-logs folder, and then starts the
kafka server again.
It will the following errors:
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in
[jar:file:/usr/share/kafka/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in
[jar:file:/usr/share/kafka/tools/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
[2015-09-12 00:58:32,203] WARN No meta.properties file under dir
/datadrive/kafka-logs/meta.properties
(kafka.server.BrokerMetadataCheckpoint)
Exception in thread "kafka-log-cleaner-thread-0"
java.lang.OutOfMemoryError: Java heap space

Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-33"

Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-51"
log4j:ERROR No output stream or file set for the appender named
[kafkaAppender].

Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-56"
[2015-09-12 00:59:54,546] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-33': (org.apache.kafka.common.utils.Utils)
[2015-09-12 01:00:34,261] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-51': (org.apache.kafka.common.utils.Utils)
[2015-09-12 01:00:51,108] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-56': (org.apache.kafka.common.utils.Utils)
[2015-09-12 01:00:48,867] ERROR Processor got uncaught exception.
(kafka.network.Processor)
[2015-09-12 01:00:48,867] ERROR Processor got uncaught exception.
(kafka.network.Processor)
[2015-09-12 01:00:49,121] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:00:57,643] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:00:54,028] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-40': (org.apache.kafka.common.utils.Utils)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:00:53,656] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:00:53,656] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-58': (org.apache.kafka.common.utils.Utils)
java.lang.OutOfMemoryError: Java heap space

Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "main-SendThread(10.1.130.12:2181)"

Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-46"

Exception: java.lang.OutOfMemoryError thrown from the
UncaughtExceptionHandler in thread "kafka-network-thread-3-SSL-61"
[2015-09-12 01:00:53,043] WARN Session 0x24f84cb22a90029 for server
10.1.130.12/10.1.130.12:2181, unexpected error, closing socket connection
and attempting reconnect (org.apache.zookeeper.ClientCnxn)
[2015-09-12 01:01:22,419] ERROR from main-SendThread(10.1.130.12:2181)
(org.apache.zookeeper.ClientCnxn)
[2015-09-12 01:01:22,051] ERROR Processor got uncaught exception.
(kafka.network.Processor)
[2015-09-12 01:01:23,489] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-46': (org.apache.kafka.common.utils.Utils)
[2015-09-12 01:01:21,313] ERROR Processor got uncaught exception.
(kafka.network.Processor)
[2015-09-12 01:01:24,438] ERROR Uncaught exception in thread
'kafka-network-thread-3-SSL-61': (org.apache.kafka.common.utils.Utils)
[2015-09-12 01:01:18,075] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:01:02,539] ERROR Uncaught exception in scheduled task
'kafka-recovery-point-checkpoint' (kafka.utils.KafkaScheduler)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:01:34,023] ERROR Processor got uncaught exception.
(kafka.network.Processor)
java.lang.OutOfMemoryError: Java heap space
[2015-09-12 01:01:35,676] ERROR Uncaught exception in scheduled task
'kafka-recovery-point-checkpoint' (kafka.utils.KafkaScheduler)


And the broker does not appear in zookeeper.
Any idea?

Thanks,
Qi

On Fri, Sep 11, 2015 at 5:53 PM, Qi Xu  wrote:

> Hi,
> We're running the Trunk version of Kafka (for its SSL feature) and
> recently I'm trying to enable the kafka manager with it.
> After enabling that,  I find out some machine's Kafka Server is dead.
> Looking at the server.log, it has the following logs.
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:19:41,310] ERROR Processor got uncaught exception.
> (kafka.network.Processor)
> java.lang.OutOfMemoryError: Java heap space
> [2015-09-12 00:19:44,766] ERROR Processor got uncaught exception.
> (kafka.n

[jira] [Commented] (KAFKA-2530) metrics for old replica fetcher thread need to be deregistered

[Otis Gospodnetic (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741840#comment-14741840
 ] 

Otis Gospodnetic commented on KAFKA-2530:
-

Is that, by any chance, related to 
http://search-hadoop.com/m/uyzND1XVyK12UNtd32/kafka+orphaned&subj=Consumer+lag+lies+orphaned+offsets+
 ?

> metrics for old replica fetcher thread need to be deregistered
> --
>
> Key: KAFKA-2530
> URL: https://issues.apache.org/jira/browse/KAFKA-2530
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2.1
>Reporter: Jun Rao
>
> Currently, the lag metrics in the replica fetcher has the following format 
> where the leader broker id is included in the clientId tag.
> clientId="ReplicaFetcherThread-0-101",partition="0",topic="test",mbean_property_type="FetcherLagMetrics",Value="262"
> There are a couple of issues. (1) When the replica changes from a follower to 
> a leader, we will need to set the lag to 0 or deregister the metric. (2) 
> Similarly, when the follower switch to another leader, we should deregister 
> the metric or clear the value. Also, we probably should remove the leader 
> broker id from the clientId tag. That way, the metric name doesn't change 
> when the follower switches leaders.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-2538) Compilation in trunk is failing due to https://github.com/apache/kafka/commit/845514d62329be8382e6d02b8041fc858718d534

[Parth Brahmbhatt (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Parth Brahmbhatt reassigned KAFKA-2538:
---

Assignee: Parth Brahmbhatt

> Compilation in trunk is failing due to 
> https://github.com/apache/kafka/commit/845514d62329be8382e6d02b8041fc858718d534
> --
>
> Key: KAFKA-2538
> URL: https://issues.apache.org/jira/browse/KAFKA-2538
> Project: Kafka
>  Issue Type: Bug
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
>Priority: Blocker
>
> Getting 
> /Users/pbrahmbhatt/repo/kafka/core/src/main/scala/kafka/tools/EndToEndLatency.scala:82:
>  value commit is not a member of 
> org.apache.kafka.clients.consumer.KafkaConsumer[Array[Byte],Array[Byte]]
>   consumer.commit(CommitType.SYNC)
>^
> Which I believe was missed when committing KAFKA-2389 which replaces all 
> occurrences of commit(mode) with commit(Sync/Async). This is resulting in 
> other PRS reporting as bad by jenkins like 
> https://github.com/apache/kafka/pull/195 where 2 failures were reported by 
> jenkins https://builds.apache.org/job/kafka-trunk-git-pr/410/ and 
> https://builds.apache.org/job/kafka-trunk-git-pr/411/ 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-2538) Compilation in trunk is failing due to https://github.com/apache/kafka/commit/845514d62329be8382e6d02b8041fc858718d534

[Parth Brahmbhatt (JIRA)]

Parth Brahmbhatt created KAFKA-2538:
---

 Summary: Compilation in trunk is failing due to 
https://github.com/apache/kafka/commit/845514d62329be8382e6d02b8041fc858718d534
 Key: KAFKA-2538
 URL: https://issues.apache.org/jira/browse/KAFKA-2538
 Project: Kafka
  Issue Type: Bug
Reporter: Parth Brahmbhatt
Priority: Blocker


Getting 
/Users/pbrahmbhatt/repo/kafka/core/src/main/scala/kafka/tools/EndToEndLatency.scala:82:
 value commit is not a member of 
org.apache.kafka.clients.consumer.KafkaConsumer[Array[Byte],Array[Byte]]
  consumer.commit(CommitType.SYNC)
   ^

Which I believe was missed when committing KAFKA-2389 which replaces all 
occurrences of commit(mode) with commit(Sync/Async). This is resulting in other 
PRS reporting as bad by jenkins like https://github.com/apache/kafka/pull/195 
where 2 failures were reported by jenkins 
https://builds.apache.org/job/kafka-trunk-git-pr/410/ and 
https://builds.apache.org/job/kafka-trunk-git-pr/411/ 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2538: Fixing a compilation error in trun...

[Parth-Brahmbhatt]

GitHub user Parth-Brahmbhatt opened a pull request:

https://github.com/apache/kafka/pull/208

KAFKA-2538: Fixing a compilation error in trunk.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Parth-Brahmbhatt/kafka KAFKA-2538

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/208.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #208


commit 24b4e01c251c51e0896887dbf65d92d101750b45
Author: Parth Brahmbhatt 
Date:   2015-09-12T02:33:08Z

KAFKA-2538: Fixing a compilation error in trunk.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2538) Compilation in trunk is failing due to https://github.com/apache/kafka/commit/845514d62329be8382e6d02b8041fc858718d534

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741860#comment-14741860
 ] 

ASF GitHub Bot commented on KAFKA-2538:
---

GitHub user Parth-Brahmbhatt opened a pull request:

https://github.com/apache/kafka/pull/208

KAFKA-2538: Fixing a compilation error in trunk.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/Parth-Brahmbhatt/kafka KAFKA-2538

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/208.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #208


commit 24b4e01c251c51e0896887dbf65d92d101750b45
Author: Parth Brahmbhatt 
Date:   2015-09-12T02:33:08Z

KAFKA-2538: Fixing a compilation error in trunk.




> Compilation in trunk is failing due to 
> https://github.com/apache/kafka/commit/845514d62329be8382e6d02b8041fc858718d534
> --
>
> Key: KAFKA-2538
> URL: https://issues.apache.org/jira/browse/KAFKA-2538
> Project: Kafka
>  Issue Type: Bug
>Reporter: Parth Brahmbhatt
>Assignee: Parth Brahmbhatt
>Priority: Blocker
>
> Getting 
> /Users/pbrahmbhatt/repo/kafka/core/src/main/scala/kafka/tools/EndToEndLatency.scala:82:
>  value commit is not a member of 
> org.apache.kafka.clients.consumer.KafkaConsumer[Array[Byte],Array[Byte]]
>   consumer.commit(CommitType.SYNC)
>^
> Which I believe was missed when committing KAFKA-2389 which replaces all 
> occurrences of commit(mode) with commit(Sync/Async). This is resulting in 
> other PRS reporting as bad by jenkins like 
> https://github.com/apache/kafka/pull/195 where 2 failures were reported by 
> jenkins https://builds.apache.org/job/kafka-trunk-git-pr/410/ and 
> https://builds.apache.org/job/kafka-trunk-git-pr/411/ 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-1679) JmxTool outputs nothing if any mbean attributes can't be retrieved

[Manikumar Reddy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manikumar Reddy resolved KAFKA-1679.

   Resolution: Fixed
Fix Version/s: 0.8.3

This got fixed in KAFKA-2278

> JmxTool outputs nothing if any mbean attributes can't be retrieved
> --
>
> Key: KAFKA-1679
> URL: https://issues.apache.org/jira/browse/KAFKA-1679
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Ryan Berdeen
>Assignee: Jonathan Rafalski
>Priority: Minor
>  Labels: newbie
> Fix For: 0.8.3
>
>
> JmxTool counts the number of attributes for all MBeans and if the number of 
> attributes retrieved does not equal this number, nothing is printed.
> Several {{java.lang:type=MemoryPool}} MBeans have unsupported attributes (see 
> HADOOP-8027, for example), so running JmxTool with no arguments fails to 
> fetch these metrics and outputs nothing while continuing to run.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)