[jira] [Commented] (KAFKA-2977) Transient Failure in kafka.log.LogCleanerIntegrationTest.cleanerTest

2015-12-13 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15055005#comment-15055005
 ] 

ASF GitHub Bot commented on KAFKA-2977:
---

GitHub user ZoneMayor opened a pull request:

https://github.com/apache/kafka/pull/671

KAFKA-2977: Transient Failure in 
kafka.log.LogCleanerIntegrationTest.cleanerTest

Hi @guozhangwang 
-
Code is as below:
val appends = writeDups(numKeys = 100, numDups = 3, log, 
CompressionCodec.getCompressionCodec(compressionCodec))
cleaner.startup()
val firstDirty = log.activeSegment.baseOffset
cleaner.awaitCleaned("log", 0, firstDirty)

val appends2 = appends ++ writeDups(numKeys = 100, numDups = 3, log, 
CompressionCodec.getCompressionCodec(compressionCodec))
val firstDirty2 = log.activeSegment.baseOffset
cleaner.awaitCleaned("log", 0, firstDirty2)
--
log cleaner and writeDups are two different threads;
log cleaner do cleaning every 15s, timeout in "cleaner.awaitCleaned" is 60s;
there is a filtering condition for a log to be chosen to become a cleaning 
target: cleanableRatio> 0.5(configured log.cleaner.min.cleanable.ratio) by 
default;
It may happen that, during "val appends2 = appends ++ writeDups(numKeys = 
100, numDups = 3, log, 
CompressionCodec.getCompressionCodec(compressionCodec))", log is also 
undergoing a cleaning process; 
Since the segment size configured in this test is quite small: 100, there 
is possibility that before the end of 'writeDups', some 'dirty segment' of the 
log is already cleaned;
With tiny dirty part left,  cleanableRatio> 0.5 cannot be satisfied;
thus firstDirty2>lastCleaned2, which leads to this test failed;

Does it make sense?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ZoneMayor/kafka trunk-KAFKA-2977

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/671.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #671


commit 95374147a28208d4850f6e73f714bf418935fc2d
Author: ZoneMayor 
Date:   2015-11-27T03:49:34Z

Merge pull request #1 from apache/trunk

merge

commit cec5b48b651a7efd3900cfa3c1fd0ab1eeeaa3ec
Author: ZoneMayor 
Date:   2015-12-01T10:44:02Z

Merge pull request #2 from apache/trunk

2015-12-1

commit a119d547bf1741625ce0627073c7909992a20f15
Author: ZoneMayor 
Date:   2015-12-04T13:42:27Z

Merge pull request #3 from apache/trunk

2015-12-04#KAFKA-2893

commit b767a8dff85fc71c75d4cf5178c3f6f03ff81bfc
Author: ZoneMayor 
Date:   2015-12-09T10:42:30Z

Merge pull request #5 from apache/trunk

2015-12-9

commit 0070c2d71d06ee8baa1cddb3451cd5af6c6b1d4a
Author: ZoneMayor 
Date:   2015-12-11T14:50:30Z

Merge pull request #8 from apache/trunk

2015-12-11

commit 09908ac646d4c84f854dad63b8c99213b74a7063
Author: ZoneMayor 
Date:   2015-12-13T14:17:19Z

Merge pull request #9 from apache/trunk

2015-12-13

commit ff1e68bb7101d12624c189174ef1dceb21ed9798
Author: jinxing 
Date:   2015-12-13T14:31:34Z

KAFKA-2054: Transient Failure in 
kafka.log.LogCleanerIntegrationTest.cleanerTest

commit 6321ab6599cb7a981fac2a4eea64a5f2ea805dd6
Author: jinxing 
Date:   2015-12-13T14:36:11Z

removed unnecessary maven repo




> Transient Failure in kafka.log.LogCleanerIntegrationTest.cleanerTest
> 
>
> Key: KAFKA-2977
> URL: https://issues.apache.org/jira/browse/KAFKA-2977
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: jin xing
>
> {code}
> java.lang.AssertionError: log cleaner should have processed up to offset 599
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at 
> kafka.log.LogCleanerIntegrationTest.cleanerTest(LogCleanerIntegrationTest.scala:76)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
>

[jira] [Created] (KAFKA-2986) Consumer group doesn't lend itself well for slow consumers with varying message size

2015-12-13 Thread Jens Rantil (JIRA) 
Jens Rantil created KAFKA-2986:
--

 Summary: Consumer group doesn't lend itself well for slow 
consumers with varying message size
 Key: KAFKA-2986
 URL: https://issues.apache.org/jira/browse/KAFKA-2986
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 0.9.0.0
 Environment: Java consumer API 0.9.0.0
Reporter: Jens Rantil
Assignee: Neha Narkhede


I sent a related post to the Kafka mailing list, but haven't received any 
response: 
http://mail-archives.apache.org/mod_mbox/kafka-users/201512.mbox/%3CCAL%2BArfWNfkpymkNDuf6UJ06CJJ63XC1bPHeT4TSYXKjSsOpu-Q%40mail.gmail.com%3E
 So far, I think this is a design issue in Kafka so I'm taking the liberty of 
creating an issue.

*Use case:*
 - Slow consumtion. Maybe around 20 seconds per record.
 - Large variation in message size: Serialized tasks are in the range of ~300 
bytes up to ~3 MB.
 - Consumtion latency (20 seconds) is independent of message size.

*Code example:*
{noformat}
while (isRunning()) {
  ConsumerRecords records = consumer.poll(100);
  for (final ConsumerRecord record : records) {
// Handle record...
  }
}
{noformat}

*Problem:* Kafka doesn't have any issues with large messages (as long as you 
bump some configuration flags). However, the problem is two-fold:
- KafkaConsumer#poll is the only call that sends healthchecks.
- There is no limit as to how many messages KafkaConsumer#poll will return. The 
limit is only set to the total number of bytes to be prefetched. This is 
problematic for varying message sizes as the session timeout becomes extremelly 
hard to tune:
-- delay until next KafkaConsumer#poll call is proportional to the number of 
records returned by previous KafkaConsumer#poll call.
-- KafkaConsumer#poll will return many small records or just a few larger 
records. For many small messages the risk is very large of the session timeout 
to kick in. Raising the session timeout in the order of magnitudes required to 
handle the smaller messages increases the latency until a dead consumer is 
discovered a thousand fold.

*Proposed fixes:* I do not claim to be a Kafka expert, but two ideas are to 
either
 - allow add `KafkaConsumer#healthy` call to let the broker know we are still 
processing records; or
 - add an upper number of message limit to `KafkaConsumer#poll`. I am thinking 
of something like `KafkaConsumer#poll(timeout, nMaxMessages)`.

*Workaround:* Have different topics for different message sizes. Makes tuning 
of partition prefetch easier.

*Questions:* Should Kafka be able to handle this case? Maybe I am using the 
wrong tool for this and Kafka is simply designed for high-throughput/low 
latency?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2977) Transient Failure in kafka.log.LogCleanerIntegrationTest.cleanerTest

2015-12-13 Thread jin xing (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-2977?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

jin xing updated KAFKA-2977:

   Labels: newbie  (was: )
 Reviewer: Guozhang Wang
Fix Version/s: 0.9.0.1
Affects Version/s: 0.9.0.0
   Status: Patch Available  (was: Open)

> Transient Failure in kafka.log.LogCleanerIntegrationTest.cleanerTest
> 
>
> Key: KAFKA-2977
> URL: https://issues.apache.org/jira/browse/KAFKA-2977
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Guozhang Wang
>Assignee: jin xing
>  Labels: newbie
> Fix For: 0.9.0.1
>
>
> {code}
> java.lang.AssertionError: log cleaner should have processed up to offset 599
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at 
> kafka.log.LogCleanerIntegrationTest.cleanerTest(LogCleanerIntegrationTest.scala:76)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at org.junit.runners.Suite.runChild(Suite.java:128)
>   at org.junit.runners.Suite.runChild(Suite.java:27)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
>   at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106)
>   at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
>

[jira] [Commented] (KAFKA-2875) Class path contains multiple SLF4J bindings warnings when using scripts under bin

2015-12-13 Thread jin xing (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15055009#comment-15055009
 ] 

jin xing commented on KAFKA-2875:
-

hi [~ijuma]
the classpath parameter is passed from parent script to kafka-run-class.sh;
I specified the classpath in kafka-topics.sh, kafka-console-consumer.sh, 
kafka-console-producer.sh;
If my way of fixing is appropriate, may be I also need to modify some other 
'parent script', which calling kafka-run-class.sh;
please give some comment when you have time : )


> Class path contains multiple SLF4J bindings warnings when using scripts under 
> bin
> -
>
> Key: KAFKA-2875
> URL: https://issues.apache.org/jira/browse/KAFKA-2875
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.9.0.0
>Reporter: Ismael Juma
>Assignee: jin xing
>Priority: Minor
>  Labels: patch
> Fix For: 0.9.0.1
>
>
> This adds a lot of noise when running the scripts, see example when running 
> kafka-console-producer.sh:
> {code}
> ~/D/s/kafka-0.9.0.0-src ❯❯❯ ./bin/kafka-console-producer.sh --topic topic 
> --broker-list localhost:9092 ⏎
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/tools/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/api/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/runtime/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/file/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/json/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-2986) Consumer group doesn't lend itself well for slow consumers with varying message size

2015-12-13 Thread Jens Rantil (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jens Rantil updated KAFKA-2986:
---
Description: 
I sent a related post to the Kafka mailing list, but haven't received any 
response: 
http://mail-archives.apache.org/mod_mbox/kafka-users/201512.mbox/%3CCAL%2BArfWNfkpymkNDuf6UJ06CJJ63XC1bPHeT4TSYXKjSsOpu-Q%40mail.gmail.com%3E
 So far, I think this is a design issue in Kafka so I'm taking the liberty of 
creating an issue.

*Use case:*
 - Slow consumtion. Maybe around 20 seconds per record.
 - Large variation in message size: Serialized tasks are in the range of ~300 
bytes up to ~3 MB.
 - Consumtion latency (20 seconds) is independent of message size.

*Code example:*
{noformat}
while (isRunning()) {
  ConsumerRecords records = consumer.poll(100);
  for (final ConsumerRecord record : records) {
// Handle record...
  }
}
{noformat}

*Problem:* Kafka doesn't have any issues with large messages (as long as you 
bump some configuration flags). However, the problem is two-fold:
- KafkaConsumer#poll is the only call that sends healthchecks.
- There is no limit as to how many messages KafkaConsumer#poll will return. The 
limit is only set to the total number of bytes to be prefetched. This is 
problematic for varying message sizes as the session timeout becomes extremelly 
hard to tune:
-- delay until next KafkaConsumer#poll call is proportional to the number of 
records returned by previous KafkaConsumer#poll call.
-- KafkaConsumer#poll will return many small records or just a few larger 
records. For many small messages the risk is very large of the session timeout 
to kick in. Raising the session timeout in the order of magnitudes required to 
handle the smaller messages increases the latency until a dead consumer is 
discovered a thousand fold.

*Proposed fixes:* I do not claim to be a Kafka expert, but two ideas are to 
either
 - allow add `KafkaConsumer#healthy` call to let the broker know we are still 
processing records; or
 - add an upper number of message limit to `KafkaConsumer#poll`. I am thinking 
of something like `KafkaConsumer#poll(timeout, nMaxMessages)`. This could 
obviously be set a configuration property instead. To avoid the broker having 
to look at the messages it sends, I suggest the KafkaConsumer decides how many 
messages it returns from poll.

*Workarounds:*
 - Have different topics for different message sizes. Makes tuning of partition 
prefetch easier.
 - Use another tool :)

*Questions:* Should Kafka be able to handle this case? Maybe I am using the 
wrong tool for this and Kafka is simply designed for high-throughput/low 
latency?

  was:
I sent a related post to the Kafka mailing list, but haven't received any 
response: 
http://mail-archives.apache.org/mod_mbox/kafka-users/201512.mbox/%3CCAL%2BArfWNfkpymkNDuf6UJ06CJJ63XC1bPHeT4TSYXKjSsOpu-Q%40mail.gmail.com%3E
 So far, I think this is a design issue in Kafka so I'm taking the liberty of 
creating an issue.

*Use case:*
 - Slow consumtion. Maybe around 20 seconds per record.
 - Large variation in message size: Serialized tasks are in the range of ~300 
bytes up to ~3 MB.
 - Consumtion latency (20 seconds) is independent of message size.

*Code example:*
{noformat}
while (isRunning()) {
  ConsumerRecords records = consumer.poll(100);
  for (final ConsumerRecord record : records) {
// Handle record...
  }
}
{noformat}

*Problem:* Kafka doesn't have any issues with large messages (as long as you 
bump some configuration flags). However, the problem is two-fold:
- KafkaConsumer#poll is the only call that sends healthchecks.
- There is no limit as to how many messages KafkaConsumer#poll will return. The 
limit is only set to the total number of bytes to be prefetched. This is 
problematic for varying message sizes as the session timeout becomes extremelly 
hard to tune:
-- delay until next KafkaConsumer#poll call is proportional to the number of 
records returned by previous KafkaConsumer#poll call.
-- KafkaConsumer#poll will return many small records or just a few larger 
records. For many small messages the risk is very large of the session timeout 
to kick in. Raising the session timeout in the order of magnitudes required to 
handle the smaller messages increases the latency until a dead consumer is 
discovered a thousand fold.

*Proposed fixes:* I do not claim to be a Kafka expert, but two ideas are to 
either
 - allow add `KafkaConsumer#healthy` call to let the broker know we are still 
processing records; or
 - add an upper number of message limit to `KafkaConsumer#poll`. I am thinking 
of something like `KafkaConsumer#poll(timeout, nMaxMessages)`.

*Workarounds:*
 - Have different topics for different message sizes. Makes tuning of partition 
prefetch easier.
 - Use another tool :)

*Questions:* Should Kafka be able to handle this case? Maybe I am

[GitHub] kafka pull request: KAFKA-2977: Transient Failure in kafka.log.Log...

2015-12-13 Thread ZoneMayor 
GitHub user ZoneMayor opened a pull request:

https://github.com/apache/kafka/pull/671

KAFKA-2977: Transient Failure in 
kafka.log.LogCleanerIntegrationTest.cleanerTest

Hi @guozhangwang 
-
Code is as below:
val appends = writeDups(numKeys = 100, numDups = 3, log, 
CompressionCodec.getCompressionCodec(compressionCodec))
cleaner.startup()
val firstDirty = log.activeSegment.baseOffset
cleaner.awaitCleaned("log", 0, firstDirty)

val appends2 = appends ++ writeDups(numKeys = 100, numDups = 3, log, 
CompressionCodec.getCompressionCodec(compressionCodec))
val firstDirty2 = log.activeSegment.baseOffset
cleaner.awaitCleaned("log", 0, firstDirty2)
--
log cleaner and writeDups are two different threads;
log cleaner do cleaning every 15s, timeout in "cleaner.awaitCleaned" is 60s;
there is a filtering condition for a log to be chosen to become a cleaning 
target: cleanableRatio> 0.5(configured log.cleaner.min.cleanable.ratio) by 
default;
It may happen that, during "val appends2 = appends ++ writeDups(numKeys = 
100, numDups = 3, log, 
CompressionCodec.getCompressionCodec(compressionCodec))", log is also 
undergoing a cleaning process; 
Since the segment size configured in this test is quite small: 100, there 
is possibility that before the end of 'writeDups', some 'dirty segment' of the 
log is already cleaned;
With tiny dirty part left,  cleanableRatio> 0.5 cannot be satisfied;
thus firstDirty2>lastCleaned2, which leads to this test failed;

Does it make sense?

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ZoneMayor/kafka trunk-KAFKA-2977

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/671.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #671


commit 95374147a28208d4850f6e73f714bf418935fc2d
Author: ZoneMayor 
Date:   2015-11-27T03:49:34Z

Merge pull request #1 from apache/trunk

merge

commit cec5b48b651a7efd3900cfa3c1fd0ab1eeeaa3ec
Author: ZoneMayor 
Date:   2015-12-01T10:44:02Z

Merge pull request #2 from apache/trunk

2015-12-1

commit a119d547bf1741625ce0627073c7909992a20f15
Author: ZoneMayor 
Date:   2015-12-04T13:42:27Z

Merge pull request #3 from apache/trunk

2015-12-04#KAFKA-2893

commit b767a8dff85fc71c75d4cf5178c3f6f03ff81bfc
Author: ZoneMayor 
Date:   2015-12-09T10:42:30Z

Merge pull request #5 from apache/trunk

2015-12-9

commit 0070c2d71d06ee8baa1cddb3451cd5af6c6b1d4a
Author: ZoneMayor 
Date:   2015-12-11T14:50:30Z

Merge pull request #8 from apache/trunk

2015-12-11

commit 09908ac646d4c84f854dad63b8c99213b74a7063
Author: ZoneMayor 
Date:   2015-12-13T14:17:19Z

Merge pull request #9 from apache/trunk

2015-12-13

commit ff1e68bb7101d12624c189174ef1dceb21ed9798
Author: jinxing 
Date:   2015-12-13T14:31:34Z

KAFKA-2054: Transient Failure in 
kafka.log.LogCleanerIntegrationTest.cleanerTest

commit 6321ab6599cb7a981fac2a4eea64a5f2ea805dd6
Author: jinxing 
Date:   2015-12-13T14:36:11Z

removed unnecessary maven repo




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-2986) Consumer group doesn't lend itself well for slow consumers with varying message size

2015-12-13 Thread Jens Rantil (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-2986?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jens Rantil updated KAFKA-2986:
---
Description: 
I sent a related post to the Kafka mailing list, but haven't received any 
response: 
http://mail-archives.apache.org/mod_mbox/kafka-users/201512.mbox/%3CCAL%2BArfWNfkpymkNDuf6UJ06CJJ63XC1bPHeT4TSYXKjSsOpu-Q%40mail.gmail.com%3E
 So far, I think this is a design issue in Kafka so I'm taking the liberty of 
creating an issue.

*Use case:*
 - Slow consumtion. Maybe around 20 seconds per record.
 - Large variation in message size: Serialized tasks are in the range of ~300 
bytes up to ~3 MB.
 - Consumtion latency (20 seconds) is independent of message size.

*Code example:*
{noformat}
while (isRunning()) {
  ConsumerRecords records = consumer.poll(100);
  for (final ConsumerRecord record : records) {
// Handle record...
  }
}
{noformat}

*Problem:* Kafka doesn't have any issues with large messages (as long as you 
bump some configuration flags). However, the problem is two-fold:
- KafkaConsumer#poll is the only call that sends healthchecks.
- There is no limit as to how many messages KafkaConsumer#poll will return. The 
limit is only set to the total number of bytes to be prefetched. This is 
problematic for varying message sizes as the session timeout becomes extremelly 
hard to tune:
-- delay until next KafkaConsumer#poll call is proportional to the number of 
records returned by previous KafkaConsumer#poll call.
-- KafkaConsumer#poll will return many small records or just a few larger 
records. For many small messages the risk is very large of the session timeout 
to kick in. Raising the session timeout in the order of magnitudes required to 
handle the smaller messages increases the latency until a dead consumer is 
discovered a thousand fold.

*Proposed fixes:* I do not claim to be a Kafka expert, but two ideas are to 
either
 - allow add `KafkaConsumer#healthy` call to let the broker know we are still 
processing records; or
 - add an upper number of message limit to `KafkaConsumer#poll`. I am thinking 
of something like `KafkaConsumer#poll(timeout, nMaxMessages)`.

*Workarounds:*
 - Have different topics for different message sizes. Makes tuning of partition 
prefetch easier.
 - Use another tool :)

*Questions:* Should Kafka be able to handle this case? Maybe I am using the 
wrong tool for this and Kafka is simply designed for high-throughput/low 
latency?

  was:
I sent a related post to the Kafka mailing list, but haven't received any 
response: 
http://mail-archives.apache.org/mod_mbox/kafka-users/201512.mbox/%3CCAL%2BArfWNfkpymkNDuf6UJ06CJJ63XC1bPHeT4TSYXKjSsOpu-Q%40mail.gmail.com%3E
 So far, I think this is a design issue in Kafka so I'm taking the liberty of 
creating an issue.

*Use case:*
 - Slow consumtion. Maybe around 20 seconds per record.
 - Large variation in message size: Serialized tasks are in the range of ~300 
bytes up to ~3 MB.
 - Consumtion latency (20 seconds) is independent of message size.

*Code example:*
{noformat}
while (isRunning()) {
  ConsumerRecords records = consumer.poll(100);
  for (final ConsumerRecord record : records) {
// Handle record...
  }
}
{noformat}

*Problem:* Kafka doesn't have any issues with large messages (as long as you 
bump some configuration flags). However, the problem is two-fold:
- KafkaConsumer#poll is the only call that sends healthchecks.
- There is no limit as to how many messages KafkaConsumer#poll will return. The 
limit is only set to the total number of bytes to be prefetched. This is 
problematic for varying message sizes as the session timeout becomes extremelly 
hard to tune:
-- delay until next KafkaConsumer#poll call is proportional to the number of 
records returned by previous KafkaConsumer#poll call.
-- KafkaConsumer#poll will return many small records or just a few larger 
records. For many small messages the risk is very large of the session timeout 
to kick in. Raising the session timeout in the order of magnitudes required to 
handle the smaller messages increases the latency until a dead consumer is 
discovered a thousand fold.

*Proposed fixes:* I do not claim to be a Kafka expert, but two ideas are to 
either
 - allow add `KafkaConsumer#healthy` call to let the broker know we are still 
processing records; or
 - add an upper number of message limit to `KafkaConsumer#poll`. I am thinking 
of something like `KafkaConsumer#poll(timeout, nMaxMessages)`.

*Workaround:* Have different topics for different message sizes. Makes tuning 
of partition prefetch easier.

*Questions:* Should Kafka be able to handle this case? Maybe I am using the 
wrong tool for this and Kafka is simply designed for high-throughput/low 
latency?


> Consumer group doesn't lend itself well for slow consumers with varying 
> message size
>

[GitHub] kafka pull request: KAFKA-2837: fix transient failure of kafka.api...

2015-12-13 Thread asfgit 
Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/648


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-2837) FAILING TEST: kafka.api.ProducerBounceTest > testBrokerFailure

2015-12-13 Thread Guozhang Wang (JIRA) 

 [ 
https://issues.apache.org/jira/browse/KAFKA-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-2837:
-
   Resolution: Fixed
Fix Version/s: (was: 0.9.0.1)
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 648
[https://github.com/apache/kafka/pull/648]

> FAILING TEST: kafka.api.ProducerBounceTest > testBrokerFailure 
> ---
>
> Key: KAFKA-2837
> URL: https://issues.apache.org/jira/browse/KAFKA-2837
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Gwen Shapira
>Assignee: jin xing
>  Labels: newbie
> Fix For: 0.9.1.0
>
>
> {code}
> java.lang.AssertionError
>   at org.junit.Assert.fail(Assert.java:86)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at org.junit.Assert.assertTrue(Assert.java:52)
>   at 
> kafka.api.ProducerBounceTest.testBrokerFailure(ProducerBounceTest.scala:117)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
>   at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106)
>   at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
>   at 
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
>   at 
> org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
>

[jira] [Commented] (KAFKA-2837) FAILING TEST: kafka.api.ProducerBounceTest > testBrokerFailure

2015-12-13 Thread ASF GitHub Bot (JIRA) 

[ 
https://issues.apache.org/jira/browse/KAFKA-2837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15055111#comment-15055111
 ] 

ASF GitHub Bot commented on KAFKA-2837:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/648


> FAILING TEST: kafka.api.ProducerBounceTest > testBrokerFailure 
> ---
>
> Key: KAFKA-2837
> URL: https://issues.apache.org/jira/browse/KAFKA-2837
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Gwen Shapira
>Assignee: jin xing
>  Labels: newbie
> Fix For: 0.9.1.0
>
>
> {code}
> java.lang.AssertionError
>   at org.junit.Assert.fail(Assert.java:86)
>   at org.junit.Assert.assertTrue(Assert.java:41)
>   at org.junit.Assert.assertTrue(Assert.java:52)
>   at 
> kafka.api.ProducerBounceTest.testBrokerFailure(ProducerBounceTest.scala:117)
>   at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>   at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>   at 
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>   at 
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>   at 
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
>   at 
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
>   at 
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
>   at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
>   at 
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
>   at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
>   at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
>   at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
>   at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
>   at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
>   at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.runTestClass(JUnitTestClassExecuter.java:105)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassExecuter.execute(JUnitTestClassExecuter.java:56)
>   at 
> org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor.processTestClass(JUnitTestClassProcessor.java:64)
>   at 
> org.gradle.api.internal.tasks.testing.SuiteTestClassProcessor.processTestClass(SuiteTestClassProcessor.java:50)
>   at sun.reflect.GeneratedMethodAccessor12.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.messaging.dispatch.ContextClassLoaderDispatch.dispatch(ContextClassLoaderDispatch.java:32)
>   at 
> org.gradle.messaging.dispatch.ProxyDispatchAdapter$DispatchingInvocationHandler.invoke(ProxyDispatchAdapter.java:93)
>   at com.sun.proxy.$Proxy2.processTestClass(Unknown Source)
>   at 
> org.gradle.api.internal.tasks.testing.worker.TestWorker.processTestClass(TestWorker.java:106)
>   at sun.reflect.GeneratedMethodAccessor11.invoke(Unknown Source)
>   at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>   at java.lang.reflect.Method.invoke(Method.java:606)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:35)
>   at 
> org.gradle.messaging.dispatch.ReflectionDispatch.dispatch(ReflectionDispatch.java:24)
>   at 
> org.gradle.messaging.remote.internal.hub.MessageHub$Handler.run(MessageHub.java:360)
>   at 
> org.gradle.internal.concurrent.ExecutorPolicy$CatchAndRecordFailures.onExecute(ExecutorPolicy.java:54)
>   at 
> org.gradle.internal.concurrent.StoppableExecutorImpl$1.run(StoppableExecutorImpl.java:40)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>   at

Build failed in Jenkins: kafka-trunk-jdk8 #233

2015-12-13 Thread Apache Jenkins Server 
See 

Changes:

[wangguoz] KAFKA-2837: Fix transient failure of

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on ubuntu3 (Ubuntu ubuntu legacy-ubuntu) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 3fed57909c16f807715f305152a5034aeb1a5532 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 3fed57909c16f807715f305152a5034aeb1a5532
 > git rev-list fd8af25d0ae354e962850c9d6b5991056613aa2d # timeout=10
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson7338685278644817722.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.4-rc-2/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:downloadWrapper

BUILD SUCCESSFUL

Total time: 11.676 secs
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson1035934456504493608.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew -Dorg.gradle.project.maxParallelForks=1 clean jarAll testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.9/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:clean UP-TO-DATE
:clients:clean UP-TO-DATE
:connect:clean UP-TO-DATE
:core:clean UP-TO-DATE
:examples:clean UP-TO-DATE
:log4j-appender:clean UP-TO-DATE
:streams:clean UP-TO-DATE
:tools:clean UP-TO-DATE
:connect:api:clean UP-TO-DATE
:connect:file:clean UP-TO-DATE
:connect:json:clean UP-TO-DATE
:connect:runtime:clean UP-TO-DATE
:jar_core_2_10
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk8:clients:compileJava
:jar_core_2_10 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Could not read entry ':clients:compileJava' from cache taskArtifacts.bin 
(/x1/jenkins/jenkins-slave/workspace/kafka-trunk-jdk8/.gradle/2.9/taskArtifacts/taskArtifacts.bin).
> java.io.UTFDataFormatException (no error message)

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 9.274 secs
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
ERROR: Publisher 'Publish JUnit test result report' failed: No test report 
files were found. Configuration error?
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2