[jira] [Commented] (KAFKA-2632) Move fetchable check from fetchedRecords to fetch response handler

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952176#comment-14952176
 ] 

ASF GitHub Bot commented on KAFKA-2632:
---

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/295

KAFKA-2632: move fetchable check ahead in handleFetchResponse



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka K2632

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/295.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #295


commit 6ef55eeb2363b46e99d8630121c8772b4193234a
Author: Guozhang Wang 
Date:   2015-10-11T06:41:18Z

KAFKA-2632.v1




> Move fetchable check from fetchedRecords to fetch response handler
> --
>
> Key: KAFKA-2632
> URL: https://issues.apache.org/jira/browse/KAFKA-2632
> Project: Kafka
>  Issue Type: Sub-task
>  Components: consumer
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.0.0
>
>
> Since we check if the partition is fetchable or not in fetchedRecords(), 
> there is a bug when the partition is paused during there is in-flight fetch 
> request, it will not be ignored in fetch response handler but after that, in 
> fetchedRecords(), causing the fetcher to update the fetched position already; 
> later no fetch requests will ever be sent to the broker for this partition 
> since consumed != fetched.
> The proposed fix is to move this check from fetchedRecords to 
> handleFetchResponse in Fetcher.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2632: move fetchable check ahead in hand...

[guozhangwang]

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/295

KAFKA-2632: move fetchable check ahead in handleFetchResponse



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka K2632

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/295.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #295


commit 6ef55eeb2363b46e99d8630121c8772b4193234a
Author: Guozhang Wang 
Date:   2015-10-11T06:41:18Z

KAFKA-2632.v1




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-2632) Move fetchable check from fetchedRecords to fetch response handler

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-2632:


 Summary: Move fetchable check from fetchedRecords to fetch 
response handler
 Key: KAFKA-2632
 URL: https://issues.apache.org/jira/browse/KAFKA-2632
 Project: Kafka
  Issue Type: Sub-task
Reporter: Guozhang Wang
Assignee: Guozhang Wang
 Fix For: 0.9.0.0


Since we check if the partition is fetchable or not in fetchedRecords(), there 
is a bug when the partition is paused during there is in-flight fetch request, 
it will not be ignored in fetch response handler but after that, in 
fetchedRecords(), causing the fetcher to update the fetched position already; 
later no fetch requests will ever be sent to the broker for this partition 
since consumed != fetched.

The proposed fix is to move this check from fetchedRecords to 
handleFetchResponse in Fetcher.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #15

[Apache Jenkins Server]

See 

Changes:

[junrao] MINOR: Use the correct processor id in the processor thread name

--
[...truncated 861 lines...]
public MockProducer(boolean autoComplete, Serializer keySerializer, 
Serializer valueSerializer) {
   ^
:39:
 warning: no @return
public int partition(String topic, Object key, byte[] keyBytes, Object 
value, byte[] valueBytes, Cluster cluster);
   ^
:71:
 warning: no @return
public String topic() {
  ^
:78:
 warning: no @return
public K key() {
 ^
:92:
 warning: no @return
public Integer partition() {
   ^
:44:
 warning: no @return
public long offset() {
^
:51:
 warning: no @return
public String topic() {
  ^
:58:
 warning: no @return
public int partition() {
   ^

76 warnings
:kafka-trunk-jdk8:log4j-appender:compileJavawarning: [options] bootstrap class 
path not set in conjunction with -source 1.7
1 warning

:kafka-trunk-jdk8:log4j-appender:processResources UP-TO-DATE
:kafka-trunk-jdk8:log4j-appender:classes
:kafka-trunk-jdk8:log4j-appender:jar
:kafka-trunk-jdk8:core:compileJava UP-TO-DATE
:kafka-trunk-jdk8:core:compileScalaJava HotSpot(TM) 64-Bit Server VM warning: 
ignoring option MaxPermSize=512m; support was removed in 8.0

:78:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.

org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP
 ^
:36:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
 commitTimestamp: Long = 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP,

  ^
:37:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
 expireTimestamp: Long = 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP) {

  ^
:259:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
if (offsetAndMetadata.commitTimestamp == 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP)

  ^
:277:
 a pure expression does nothing in statement position; you may be omitting 
necessary parentheses
ControllerStats.uncleanLeaderElectionRate
^
:278:
 a pure expression does nothing in statement position; you may be omitting 
necessary parentheses
ControllerStats.leaderElectionTimer
^
:380:
 value DEFAULT_TIMESTAMP in object OffsetCommitRequest is deprecated: see 
corresponding Javadoc for more information.
if (value.expireTimestamp == 
org.apache.kafka.common.requests.OffsetCommitRequest.DEFAULT_TIMESTAMP)

Jenkins build is back to normal : kafka-trunk-jdk7 #672

[Apache Jenkins Server]

See 

[GitHub] kafka pull request: MINOR: Use the correct processor id in the pro...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/294


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2477) Replicas spuriously deleting all segments in partition

[Chinmay Soman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952128#comment-14952128
 ] 

Chinmay Soman commented on KAFKA-2477:
--

Btw, this doesn't happen everywhere but is definitely seen in our biggest 
cluster (with way more partitions / node). Maybe it has something to do with 
scale ?

> Replicas spuriously deleting all segments in partition
> --
>
> Key: KAFKA-2477
> URL: https://issues.apache.org/jira/browse/KAFKA-2477
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Håkon Hitland
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.0
>
> Attachments: Screen Shot 2015-10-10 at 6.54.44 PM.png, kafka_log.txt, 
> kafka_log_trace.txt
>
>
> We're seeing some strange behaviour in brokers: a replica will sometimes 
> schedule all segments in a partition for deletion, and then immediately start 
> replicating them back, triggering our check for under-replicating topics.
> This happens on average a couple of times a week, for different brokers and 
> topics.
> We have per-topic retention.ms and retention.bytes configuration, the topics 
> where we've seen this happen are hitting the size limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-2477) Replicas spuriously deleting all segments in partition

[Chinmay Soman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952127#comment-14952127
 ] 

Chinmay Soman edited comment on KAFKA-2477 at 10/11/15 1:49 AM:


[~ewencp] I'm attaching the screenshot of Max lag observed for the different 
brokers which describes the behaviour. 

Also here's the pertinent log:


[2015-10-10 22:17:17,759] 3171793337 [kafka-request-handler-0] WARN  
kafka.server.ReplicaManager  - [Replica Manager on Broker 70]: Error when 
processing fetch request for partition [...topic...,62] offset 91963211 from 
follower with correlation id 176614372. Possible cause: Request for offset 
91963211 but we only have log segments in the range 55923986 to 91963210.

[2015-10-10 22:17:17,759] 3171793337 [kafka-request-handler-4] WARN  
kafka.server.ReplicaManager  - [Replica Manager on Broker 70]: Error when 
processing fetch request for partition [...topic...,62] offset 91963211 from 
follower with correlation id 152788081. Possible cause: Request for offset 
91963211 but we only have log segments in the range 55923986 to 91963210.

[2015-10-10 22:17:20,256] 3171795834 [kafka-scheduler-4] INFO  
kafka.cluster.Partition  - Partition [...topic...,62] on broker 70: Shrinking 
ISR for partition [hp.event.user.driver_app.experiment,62] from 70,69,71 to 70




was (Author: cpsoman):
Max lag observed for the different brokers

> Replicas spuriously deleting all segments in partition
> --
>
> Key: KAFKA-2477
> URL: https://issues.apache.org/jira/browse/KAFKA-2477
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Håkon Hitland
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.0
>
> Attachments: Screen Shot 2015-10-10 at 6.54.44 PM.png, kafka_log.txt, 
> kafka_log_trace.txt
>
>
> We're seeing some strange behaviour in brokers: a replica will sometimes 
> schedule all segments in a partition for deletion, and then immediately start 
> replicating them back, triggering our check for under-replicating topics.
> This happens on average a couple of times a week, for different brokers and 
> topics.
> We have per-topic retention.ms and retention.bytes configuration, the topics 
> where we've seen this happen are hitting the size limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2477) Replicas spuriously deleting all segments in partition

[Chinmay Soman (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Chinmay Soman updated KAFKA-2477:
-
Attachment: Screen Shot 2015-10-10 at 6.54.44 PM.png

Max lag observed for the different brokers

> Replicas spuriously deleting all segments in partition
> --
>
> Key: KAFKA-2477
> URL: https://issues.apache.org/jira/browse/KAFKA-2477
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Håkon Hitland
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.0
>
> Attachments: Screen Shot 2015-10-10 at 6.54.44 PM.png, kafka_log.txt, 
> kafka_log_trace.txt
>
>
> We're seeing some strange behaviour in brokers: a replica will sometimes 
> schedule all segments in a partition for deletion, and then immediately start 
> replicating them back, triggering our check for under-replicating topics.
> This happens on average a couple of times a week, for different brokers and 
> topics.
> We have per-topic retention.ms and retention.bytes configuration, the topics 
> where we've seen this happen are hitting the size limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2477) Replicas spuriously deleting all segments in partition

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952120#comment-14952120
 ] 

Ewen Cheslack-Postava commented on KAFKA-2477:
--

[~cpsoman] Interesting, the only other org I had heard this affecting was 
seeing it at approximately the same frequency as the original report -- maybe 
once or twice a week. It seemed random, so the events could sometimes clump 
together (e.g. see two in one day), but the average rate was pretty low. It was 
an annoyance rather than a serious issue.

The patch does seem to apply trivially to 0.8.2.2.

> Replicas spuriously deleting all segments in partition
> --
>
> Key: KAFKA-2477
> URL: https://issues.apache.org/jira/browse/KAFKA-2477
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Håkon Hitland
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.0
>
> Attachments: kafka_log.txt, kafka_log_trace.txt
>
>
> We're seeing some strange behaviour in brokers: a replica will sometimes 
> schedule all segments in a partition for deletion, and then immediately start 
> replicating them back, triggering our check for under-replicating topics.
> This happens on average a couple of times a week, for different brokers and 
> topics.
> We have per-topic retention.ms and retention.bytes configuration, the topics 
> where we've seen this happen are hitting the size limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2477) Replicas spuriously deleting all segments in partition

[Chinmay Soman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952113#comment-14952113
 ] 

Chinmay Soman commented on KAFKA-2477:
--

We're hitting this issue quite often (every 15-20 mins) and is a problem since 
its eating up the already scarce disk / Network resource. At the moment we're 
running 0.8.2.0. Is there any plan to backport this patch given the severity of 
the issue ?

> Replicas spuriously deleting all segments in partition
> --
>
> Key: KAFKA-2477
> URL: https://issues.apache.org/jira/browse/KAFKA-2477
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
>Reporter: Håkon Hitland
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.0
>
> Attachments: kafka_log.txt, kafka_log_trace.txt
>
>
> We're seeing some strange behaviour in brokers: a replica will sometimes 
> schedule all segments in a partition for deletion, and then immediately start 
> replicating them back, triggering our check for under-replicating topics.
> This happens on average a couple of times a week, for different brokers and 
> topics.
> We have per-topic retention.ms and retention.bytes configuration, the topics 
> where we've seen this happen are hitting the size limit.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: Use the correct processor id in the pro...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/294

MINOR: Use the correct processor id in the processor thread name



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka fix-processor-thread-name

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/294.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #294


commit 385f850c55d4fce3dd6fb3528b1737d80d8428b6
Author: Ismael Juma 
Date:   2015-10-10T23:48:17Z

MINOR: Use the correct processor id in the processor thread name




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk8 #14

[Apache Jenkins Server]

See 

Changes:

[junrao] KAFKA-2614; No more clients can connect after

--
[...truncated 3238 lines...]

kafka.api.ConsumerTest > testSeek PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] PASSED

kafka.integration.SslTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.api.SSLConsumerTest > testSimpleConsumption PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.api.ConsumerTest > testUnsubscribeTopic PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.SslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.api.ConsumerTest > testAutoCommitOnClose PASSED

kafka.integration.SslTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.api.SSLConsumerTest > testAutoOffsetReset PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.SslTopicMetadataTest > testTopicMetadataRequest PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.utils.ByteBoundedBlockingQueueTest > testByteBoundedBlockingQueue PASSED

kafka.coordinator.PartitionAssignorTest > 
testRoundRobinAssignorOneConsumerNonexistentTopic PASSED

kafka.coordinator.PartitionAssignorTest > 
testRangeAssignorOnlyAssignsPartitionsFromSubscribedTopics PASSED

kafka.coordinator.PartitionAssignorTest > testRangeAssignorOneConsumerNoTopic 
PASSED

kafka.coordinator.PartitionAssignorTest > 
testRangeAssignorOneConsumerMultipleTopics PASSED

kafka.coordinator.PartitionAssignorTest > 
testRoundRobinAssignorOnlyAssignsPartitionsFromSubscribedTopics PASSED

kafka.coordinator.PartitionAssignorTest > 
testRoundRobinAssignorTwoConsumersOneTopicOnePartition PASSED

kafka.coordinator.PartitionAssignorTest > 
testRangeAssignorTwoConsumersOneTopicTwoPartitions PASSED

kafka.coordinator.PartitionAssignorTest > 
testRoundRobinAssignorTwoConsumersOneTopicTwoPartitions PASSED

kafka.coordinator.PartitionAssignorTest > 
testRoundRobinAssignorMultipleConsumersMixedTopics PASSED

kafka.coordinator.PartitionAssignorTest > 
testRangeAssignorTwoConsumersOneTopicOnePartition PASSED

kafka.coordinator.PartitionAssignorTest > 
testRangeAssignorMultipleConsumersMixedTopics PASSED

kafka.coordinator.PartitionAssignorTest > 
testRoundRobinAssignorOneConsumerNoTopic PASSED

kafka.coordinator.PartitionAssignorTest > 
testRoundRobinAssignorOneConsumerOneTopic PASSED

kafka.coordinator.PartitionAssignorTest > testRangeAssignorOneConsumerOneTopic 
PASSED

kafka.coordinator.PartitionAssignorTest > 
testRangeAssignorOneConsumerNonexistentTopic PASSED

kafka.coordinator.PartitionAssignorTest > 
testRoundRobinAssignorOneConsumerMultipleTopics PASSED

kafka.coordinator.PartitionAssignorTest > 
testRoundRobinAssignorTwoConsumersTwoTopicsSixPartitions PASSED

kafka.coordinator.PartitionAssignorTest > 
testRangeAssignorTwoConsumersTwoTopicsSixPartitions PASSED

kafka.api.ConsumerTest > testListTopics PASSED

kafka.integration.SslTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.api.ConsumerTest > testExpandingTopicSubscriptions PASSED

kafka.utils.timer.TimerTest > testTaskExpiration PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.api.ConsumerTest > testPatternUnsubscription PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.api.ConsumerTest > testGroupConsumption PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.api.ConsumerTest > testPartitionsFor PASSED

kafka.integration.PlaintextTopicMetadataTes

[jira] [Commented] (KAFKA-2515) handle oversized messages properly in new consumer

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952094#comment-14952094
 ] 

Jun Rao commented on KAFKA-2515:


The following is how you can reproduce this. 

1. Create a new topic test3, allowing a larger message size.
kafka-topics --zookeeper localhost:2181 --create --topic test3 --partition 
1  --replication-factor 1 --config max.message.bytes=130

2. Add one oversize message. 
kafka-run-class org.apache.kafka.clients.tools.ProducerPerformance test3 1 
120 1000 bootstrap.servers=localhost:9090,localhost:9091,localhost:9092 
max.request.size=200

3. Consume messages on topic test3.
kafka-console-consumer --from-beginning --topic test3 --zookeeper 
localhost:2181



> handle oversized messages properly in new consumer
> --
>
> Key: KAFKA-2515
> URL: https://issues.apache.org/jira/browse/KAFKA-2515
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients
>Reporter: Jun Rao
>Assignee: Onur Karaman
> Fix For: 0.9.0.0
>
>
> When there is an oversized message in the broker, it seems that the new 
> consumer just silently gets stuck. We should at least log an error when this 
> happens.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #671

[Apache Jenkins Server]

See 

Changes:

[junrao] KAFKA-2614; No more clients can connect after

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu-2 (docker Ubuntu ubuntu) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 1265d7cb7f56b5f5aad940250dc705a9a060a6f0 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 1265d7cb7f56b5f5aad940250dc705a9a060a6f0
 > git rev-list c67ca65889721e3af7527ab26df49a9fb4db87ef # timeout=10
Setting 
GRADLE_2_1_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1
Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7
[kafka-trunk-jdk7] $ /bin/bash -xe /tmp/hudson5728415922883432356.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.1/userguide/gradle_daemon.html.

FAILURE: Build failed with an exception.

* What went wrong:
Unable to start the daemon process.
This problem might be caused by incorrect configuration of the daemon.
For example, an unrecognized jvm option is used.
Please refer to the user guide chapter on the daemon at 
http://gradle.org/docs/2.1/userguide/gradle_daemon.html
Please read below process output to find out more:
---
22:34:57.445 [main] DEBUG o.g.l.daemon.bootstrap.DaemonMain - Assuming the 
daemon was started with following jvm opts: [-XX:MaxPermSize=512m, -Xss2m, 
-Xmx1024m, -Dfile.encoding=ISO-8859-1, -Duser.country=US, -Duser.language=en, 
-Duser.variant]
22:34:58.162 [main] DEBUG o.g.l.daemon.server.DaemonServices - Creating daemon 
context with opts: [-XX:MaxPermSize=512m, -Xss2m, -Xmx1024m, 
-Dfile.encoding=ISO-8859-1, -Duser.country=US, -Duser.language=en, 
-Duser.variant]
22:34:58.225 [INFO] [org.gradle.launcher.daemon.server.Daemon] start() called 
on daemon - 
DefaultDaemonContext[uid=b429d51e-fef0-45f2-b23c-2ba86992998a,javaHome=/home/jenkins/tools/java/jdk1.7.0_25-32,daemonRegistryDir=/home/jenkins/.gradle/daemon,pid=27366,idleTimeout=12,daemonOpts=-XX:MaxPermSize=512m,-Xss2m,-Xmx1024m,-Dfile.encoding=ISO-8859-1,-Duser.country=US,-Duser.language=en,-Duser.variant]
22:34:58.232 [DEBUG] [org.gradle.launcher.daemon.server.DaemonStateCoordinator] 
updating lastActivityAt to 1444516498232
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x9292e654, pid=27366, tid=4137122624
#
# JRE version: 7.0_25-b15
# Java VM: Java HotSpot(TM) Server VM (23.25-b01 mixed mode linux-x86 )
# Problematic frame:
# C  [libnet.so+0x14654]  _fini+0x1d0c
#
# Failed to write core dump. Core dumps have been disabled. To enable core 
dumping, try "ulimit -c unlimited" before starting Java again
#
# An error report file with more information is saved as:
# /home/jenkins/.gradle/daemon/2.1/hs_err_pid27366.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
# The crash happened outside the Java Virtual Machine in native code.
# See problematic frame for where to report the bug.
#


* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
GRADLE_2_1_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1
Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7
ERROR: Publisher 'Publish JUnit test result report' failed: Test reports were 
found but none of them are new. Did tests run? 
For example, 

 is 5 days 2 hr old

Setting 
GRADLE_2_1_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.1
Setting JDK_1_7_LATEST__HOME=/home/jenkins/tools/java/latest1.7

[GitHub] kafka pull request: KAFKA-2614; No more clients can connect after ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/288


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2614) No more clients can connect after `TooManyConnectionsException` threshold (max.connections.per.ip) is reached

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14952069#comment-14952069
 ] 

ASF GitHub Bot commented on KAFKA-2614:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/288


> No more clients can connect after `TooManyConnectionsException` threshold 
> (max.connections.per.ip) is reached
> -
>
> Key: KAFKA-2614
> URL: https://issues.apache.org/jira/browse/KAFKA-2614
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2.1
> Environment: Debian Jessie
>Reporter: Stephen Chu
>Assignee: Ismael Juma
>Priority: Critical
> Fix For: 0.9.0.0
>
>
> It seems no more clients can connect to Kafka after `max.connections.per.ip` 
> is reached, even if previous clients were already disconnected.
> Using 0.8.3 (9c936b18), upon starting a fresh Kafka server that is configured 
> with (max.connections.per.ip = 24), I noticed that I can cause the server to 
> hit the error case of {{INFO Rejected connection from /0:0:0:0:0:0:0:1, 
> address already has the configured maximum of 24 connections.}} very quickly, 
> by simply looping through a bunch of simple clients against the server:
> {noformat}
> #! /bin/bash
> for i in {1..30}; do
> # either:
> nc -vz 127.0.0.1 9092;
> # or:
> ( telnet 127.0.0.1 9092; ) &
> done
> # if using telnet, kill all connected jobs now via:
> kill %{2..31}
> {noformat}
> The problem seems to be that the counter for such short-lived client 
> connections aren't properly decrementing when using the 
> `max.connections.per.ip` feature.
> Turning on DEBUG logs, I cannot see the log lines "Closing connection from 
> xxx" on [this 
> line|https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/network/SocketServer.scala#L164]
>  from the first few still-under-threshold short-lived connections, but starts 
> showing *after* I hit the limit per that config.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2614) No more clients can connect after `TooManyConnectionsException` threshold (max.connections.per.ip) is reached

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-2614:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 288
[https://github.com/apache/kafka/pull/288]

> No more clients can connect after `TooManyConnectionsException` threshold 
> (max.connections.per.ip) is reached
> -
>
> Key: KAFKA-2614
> URL: https://issues.apache.org/jira/browse/KAFKA-2614
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2.1
> Environment: Debian Jessie
>Reporter: Stephen Chu
>Assignee: Ismael Juma
>Priority: Critical
> Fix For: 0.9.0.0
>
>
> It seems no more clients can connect to Kafka after `max.connections.per.ip` 
> is reached, even if previous clients were already disconnected.
> Using 0.8.3 (9c936b18), upon starting a fresh Kafka server that is configured 
> with (max.connections.per.ip = 24), I noticed that I can cause the server to 
> hit the error case of {{INFO Rejected connection from /0:0:0:0:0:0:0:1, 
> address already has the configured maximum of 24 connections.}} very quickly, 
> by simply looping through a bunch of simple clients against the server:
> {noformat}
> #! /bin/bash
> for i in {1..30}; do
> # either:
> nc -vz 127.0.0.1 9092;
> # or:
> ( telnet 127.0.0.1 9092; ) &
> done
> # if using telnet, kill all connected jobs now via:
> kill %{2..31}
> {noformat}
> The problem seems to be that the counter for such short-lived client 
> connections aren't properly decrementing when using the 
> `max.connections.per.ip` feature.
> Turning on DEBUG logs, I cannot see the log lines "Closing connection from 
> xxx" on [this 
> line|https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/network/SocketServer.scala#L164]
>  from the first few still-under-threshold short-lived connections, but starts 
> showing *after* I hit the limit per that config.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Confluent Clarification

[Ewen Cheslack-Postava]

Dinesh,

As Grant mentioned, Confluent's mailing list is a better place to ask
questions about the Confluent Platform.

The version of Kafka (and Zookeeper) match the Apache versions -- currently
the CP 1.0.1 release packages the Apache Kafka 0.8.2.2 release directly. In
the future, there may be cases where they diverge, but the motivation for
that is just to get critical fixes out faster than it sometimes takes to
get an Apache release rolled out (which requires preparation of the release
candidate, a voting period, etc.). All the patches/bug fixes will be in
Apache Kafka as well.

-Ewen

On Fri, Oct 9, 2015 at 6:54 AM, Grant Henke  wrote:

> Hi Dinesh,
>
> You may get a response here, as many members of the confluent team read
> this list regularly. However, you may want to check the Confluent platform
> mailing list to see if your question has been answered before.
>
> https://groups.google.com/forum/#!forum/confluent-platform
>
> Thanks,
> Grant
>
>
> On Fri, Oct 9, 2015 at 8:11 AM, Dinesh J  wrote:
>
> > Hi All,
> >
> > We are Planning to use Confluent Rest API in our messaging system since
> we
> > don't find any good Dot Net Client.
> >
> > Is the Kafka and ZooKeeper  delivered by Confluent Zip is same as Apache
> > Kafka? Or Changes made in Apache Kafka?
> >
> > Any one please Confirm this?
> >
> >
> > Thanks,
> > Dinesh
> >
>
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>



-- 
Thanks,
Ewen

[jira] [Created] (KAFKA-2631) Expose data volume via JMX

[Eno Thereska (JIRA)]

Eno Thereska created KAFKA-2631:
---

 Summary: Expose data volume via JMX
 Key: KAFKA-2631
 URL: https://issues.apache.org/jira/browse/KAFKA-2631
 Project: Kafka
  Issue Type: Improvement
  Components: core
Affects Versions: 0.8.2.1
Reporter: Eno Thereska
Assignee: Eno Thereska
 Fix For: 0.8.2.1


It would be good to expose the volume of data stored on a broker. This must 
include only the volume created by a client and must not include the volume of 
data created due to replication. In practice this means only a leader needs to 
keep track of the volume of the data received by a producer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-37 - Add namespaces in Kafka

[Magnus Edenhill]

Good write-up Ashish.

Looking at one of the rejected alternatives got me thinking:
"Modify request/ response formats to take namespace specifically.

Solves the issue of delimiting string required in proposed approach and the
issue with existing regex consumers.
This definitely is the cleanest approach. However, will require lots of API
and protocol changes. This is listed in rejected alternatives, but we
should totally consider this as an option."


I think we could actually achieve this without any protocol change at all
by moving the namespace token to the common request header's ClientId field.
E.g.: .. RequestHeader { ..., ClientId = "myclient@mynamespace", ... }

That would provide the desired per-request namespacing and in theory the
existing clients dont even need to be updated as long as the clientId is
configurable (which it should be).


My two cents,
Magnus


2015-10-10 3:32 GMT+02:00 Ashish Singh :

> Hey Guys,
>
> I just created KIP-37 for adding namespaces to Kafka.
>
> KIP-37
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-37+-+Add+Namespaces+to+Kafka
> >
> tracks the proposal.
>
> The idea is to make Kafka support multi-tenancy via namespaces.
>
> Feedback and comments are welcome.
> ​
> --
>
> Regards,
> Ashish
>