[jira] [Commented] (KAFKA-1801) Remove non-functional variable definition in log4j.properties

[Lee Hanxue (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129946#comment-15129946
 ] 

Lee Hanxue commented on KAFKA-1801:
---

Removing the {{kafka.logs.dir}} variable definition does not make sense. It 
will break build for kafka 0.9.0.0. 

There are 5 instances of {{${kafka.logs.dir}}} in the {{log4j.properties}} 
file. This variable is now undefined. Either add in the definition again, or 
remove all references to {{kafka.logs.dir}} in the {{log4j.properties}} file.

I don't see why there is a need to remove any custom variables even though 
there are system defined properties. Users can use system defined properties to 
set their preferred value, but let's keep a sane default ?

> Remove non-functional variable definition in log4j.properties
> -
>
> Key: KAFKA-1801
> URL: https://issues.apache.org/jira/browse/KAFKA-1801
> Project: Kafka
>  Issue Type: Improvement
>  Components: log
>Affects Versions: 0.8.2.0
>Reporter: Raman Gupta
>Assignee: Raman Gupta
>Priority: Trivial
>  Labels: easyfix, patch
> Fix For: 0.9.0.0
>
>   Original Estimate: 5m
>  Remaining Estimate: 5m
>
> In log4j.properties, a property kafka.logs.dir is defined. However, modifying 
> this property has no effect because log4j will always use the system property 
> defined in kafka-run-class.sh before using the locally defined property in 
> log4j.properties. Therefore, its probably less confusing to simply remove 
> this property from here.
> See 
> http://logging.apache.org/log4j/1.2/apidocs/org/apache/log4j/PropertyConfigurator.html
>  for the property search order (system property first, locally defined 
> property second).
> An alternative solution: remove the system property from kafka-run-class.sh 
> and keep the one here.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3170) Default value of fetch_min_bytes in new consumer is 1024 while doc says it is 1

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129795#comment-15129795
 ] 

Guozhang Wang commented on KAFKA-3170:
--

Yes, thanks for reminding :)

> Default value of fetch_min_bytes in new consumer is 1024 while doc says it is 
> 1
> ---
>
> Key: KAFKA-3170
> URL: https://issues.apache.org/jira/browse/KAFKA-3170
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.9.0.1
>
>
> FETCH_MIN_BYTES_DOC says:
> {quote}
> The minimum amount of data the server should return for a fetch request. If 
> insufficient data is available the request will wait for that much data to 
> accumulate before answering the request. The default setting of 1 byte means 
> that fetch requests are answered as soon as a single byte of data is 
> available or the fetch request times out waiting for data to arrive. Setting 
> this to something greater than 1 will cause the server to wait for larger 
> amounts of data to accumulate which can improve server throughput a bit at 
> the cost of some additional latency.
> {quote}
> But the default value is actually set to 1024. Either the doc or the value 
> needs to be changed. Perhaps 1 is a better default?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3170) Default value of fetch_min_bytes in new consumer is 1024 while doc says it is 1

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3170:
-
Fix Version/s: 0.9.0.1

> Default value of fetch_min_bytes in new consumer is 1024 while doc says it is 
> 1
> ---
>
> Key: KAFKA-3170
> URL: https://issues.apache.org/jira/browse/KAFKA-3170
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.9.0.1
>
>
> FETCH_MIN_BYTES_DOC says:
> {quote}
> The minimum amount of data the server should return for a fetch request. If 
> insufficient data is available the request will wait for that much data to 
> accumulate before answering the request. The default setting of 1 byte means 
> that fetch requests are answered as soon as a single byte of data is 
> available or the fetch request times out waiting for data to arrive. Setting 
> this to something greater than 1 will cause the server to wait for larger 
> amounts of data to accumulate which can improve server throughput a bit at 
> the cost of some additional latency.
> {quote}
> But the default value is actually set to 1024. Either the doc or the value 
> needs to be changed. Perhaps 1 is a better default?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3170) Default value of fetch_min_bytes in new consumer is 1024 while doc says it is 1

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3170:
-
Resolution: Fixed
Status: Resolved  (was: Patch Available)

> Default value of fetch_min_bytes in new consumer is 1024 while doc says it is 
> 1
> ---
>
> Key: KAFKA-3170
> URL: https://issues.apache.org/jira/browse/KAFKA-3170
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.9.0.1
>
>
> FETCH_MIN_BYTES_DOC says:
> {quote}
> The minimum amount of data the server should return for a fetch request. If 
> insufficient data is available the request will wait for that much data to 
> accumulate before answering the request. The default setting of 1 byte means 
> that fetch requests are answered as soon as a single byte of data is 
> available or the fetch request times out waiting for data to arrive. Setting 
> this to something greater than 1 will cause the server to wait for larger 
> amounts of data to accumulate which can improve server throughput a bit at 
> the cost of some additional latency.
> {quote}
> But the default value is actually set to 1024. Either the doc or the value 
> needs to be changed. Perhaps 1 is a better default?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2875) Class path contains multiple SLF4J bindings warnings when using scripts under bin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129791#comment-15129791
 ] 

ASF GitHub Bot commented on KAFKA-2875:
---

GitHub user ZoneMayor reopened a pull request:

https://github.com/apache/kafka/pull/693

KAFKA-2875:  remove slf4j multi binding warnings when running form source 
distribution

hi @ijuma I reopened this pr again (sorry for my inexperience using github);
I think I did much deduplication for the script;
Please have a look when you have time  : - )

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ZoneMayor/kafka trunk-KAFKA-2875

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/693.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #693


commit 34240b52e1b70aa172b65155f6042243d838b420
Author: ZoneMayor 
Date:   2015-12-18T07:22:20Z

Merge pull request #12 from apache/trunk

2015-12-18

commit ffedf6fd04280e89978531fd73e7fe37a4d9bbed
Author: jinxing 
Date:   2015-12-18T07:24:14Z

KAFKA-2875 Class path contains multiple SLF4J bindings warnings when using 
scripts under bin

commit 52d02f333e86d06cfa8fff5facd18999b3db6d83
Author: ZoneMayor 
Date:   2015-12-30T03:08:08Z

Merge pull request #13 from apache/trunk

2015-12-30

commit d56be0b9e0849660c07d656c6019f9cc2f17ae55
Author: ZoneMayor 
Date:   2016-01-10T09:24:06Z

Merge pull request #14 from apache/trunk

2016-1-10

commit a937ad38ac90b90a57a1969bdd8ce06d6faaaeb1
Author: jinxing 
Date:   2016-01-10T10:28:18Z

Merge branch 'trunk-KAFKA-2875' of https://github.com/ZoneMayor/kafka into 
trunk-KAFKA-2875

commit 83b2bcca237ba9445360bbfcb05a0de82c36274f
Author: jinxing 
Date:   2016-01-10T12:39:20Z

KAFKA-2875: wip

commit 6e6f2c20c5730253d8e818c2dc1e5e741a05ac08
Author: jinxing 
Date:   2016-01-10T14:53:28Z

KAFKA-2875: Classpath contains multiple SLF4J bindings warnings when using 
scripts under bin

commit fbd380659727d991dff242be33cc6a3bb78f4861
Author: ZoneMayor 
Date:   2016-01-28T06:28:25Z

Merge pull request #15 from apache/trunk

2016-01-28

commit f21aa55ed68907376d5b0924e228875530cc1046
Author: jinxing 
Date:   2016-01-28T07:10:30Z

KAFKA-2875: remove slf4j multi binding warnings when running form source 
distribution (merge to trunk and resolve conflict)

commit 51fcc408302ebb0c4adaf2a4d0e6647cc469c6a0
Author: jinxing 
Date:   2016-01-28T07:43:52Z

added a new line

commit 8a6cbad74ca4f07a4c70c1d522b604d58e4917c6
Author: jinxing 
Date:   2016-02-01T08:49:06Z

KAFKA-2875: create deduplicated dependant-libs and use symlink to construct 
classpath

commit 153a1177c943e76c9c8457c47244ec59ea91d6fc
Author: jinxing 
Date:   2016-02-01T09:42:37Z

small fix

commit 1d283120bd7c3b90928090c4d22376d4ac05c4d5
Author: jinxing 
Date:   2016-02-01T10:09:46Z

KAFKA-2875: modify classpath in windows bat

commit 29c1797ae4f3ba47445e45049c8fc0fc2e1609f4
Author: jinxing 
Date:   2016-02-01T10:13:20Z

mod server.properties for test

commit a1993e5ca2908862340113ce965bd7fdc5020bab
Author: jinxing 
Date:   2016-02-01T12:44:22Z

KAFKA-2875: small fix

commit e523bd2ce91e03e38c20413aef3c48998fc3c263
Author: jinxing 
Date:   2016-02-01T16:12:27Z

KAFKA-2875: small fix

commit fb27bdeba925e6833ca9bc9feb1d6d3cf55c5aaf
Author: jinxing 
Date:   2016-02-02T09:44:32Z

KAFKA-2875: replace PROJECT_NAMES with PROJECT_NAME, use the central 
deduplicated libs if PROJECT_NAME not specified

commit f8ba5a920a0db8654c0776ad8449b167689c0eb4
Author: jinxing 
Date:   2016-02-02T09:48:00Z

small fix




> Class path contains multiple SLF4J bindings warnings when using scripts under 
> bin
> -
>
> Key: KAFKA-2875
> URL: https://issues.apache.org/jira/browse/KAFKA-2875
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.9.0.0
>Reporter: Ismael Juma
>Assignee: jin xing
>Priority: Minor
>  Labels: patch
> Fix For: 0.9.1.0
>
>
> This adds a lot of noise when running the scripts, see example when running 
> kafka-console-producer.sh:
> {code}
> ~/D/s/kafka-0.9.0.0-src ❯❯❯ ./bin/kafka-console-producer.sh --topic topic 
> --broker-list localhost:9092 ⏎
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala

[jira] [Commented] (KAFKA-2875) Class path contains multiple SLF4J bindings warnings when using scripts under bin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129790#comment-15129790
 ] 

ASF GitHub Bot commented on KAFKA-2875:
---

Github user ZoneMayor closed the pull request at:

https://github.com/apache/kafka/pull/693


> Class path contains multiple SLF4J bindings warnings when using scripts under 
> bin
> -
>
> Key: KAFKA-2875
> URL: https://issues.apache.org/jira/browse/KAFKA-2875
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.9.0.0
>Reporter: Ismael Juma
>Assignee: jin xing
>Priority: Minor
>  Labels: patch
> Fix For: 0.9.1.0
>
>
> This adds a lot of noise when running the scripts, see example when running 
> kafka-console-producer.sh:
> {code}
> ~/D/s/kafka-0.9.0.0-src ❯❯❯ ./bin/kafka-console-producer.sh --topic topic 
> --broker-list localhost:9092 ⏎
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/tools/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/api/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/runtime/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/file/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/json/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2875: remove slf4j multi binding warning...

[ZoneMayor]

Github user ZoneMayor closed the pull request at:

https://github.com/apache/kafka/pull/693


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-2875: remove slf4j multi binding warning...

[ZoneMayor]

GitHub user ZoneMayor reopened a pull request:

https://github.com/apache/kafka/pull/693

KAFKA-2875:  remove slf4j multi binding warnings when running form source 
distribution

hi @ijuma I reopened this pr again (sorry for my inexperience using github);
I think I did much deduplication for the script;
Please have a look when you have time  : - )

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ZoneMayor/kafka trunk-KAFKA-2875

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/693.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #693


commit 34240b52e1b70aa172b65155f6042243d838b420
Author: ZoneMayor 
Date:   2015-12-18T07:22:20Z

Merge pull request #12 from apache/trunk

2015-12-18

commit ffedf6fd04280e89978531fd73e7fe37a4d9bbed
Author: jinxing 
Date:   2015-12-18T07:24:14Z

KAFKA-2875 Class path contains multiple SLF4J bindings warnings when using 
scripts under bin

commit 52d02f333e86d06cfa8fff5facd18999b3db6d83
Author: ZoneMayor 
Date:   2015-12-30T03:08:08Z

Merge pull request #13 from apache/trunk

2015-12-30

commit d56be0b9e0849660c07d656c6019f9cc2f17ae55
Author: ZoneMayor 
Date:   2016-01-10T09:24:06Z

Merge pull request #14 from apache/trunk

2016-1-10

commit a937ad38ac90b90a57a1969bdd8ce06d6faaaeb1
Author: jinxing 
Date:   2016-01-10T10:28:18Z

Merge branch 'trunk-KAFKA-2875' of https://github.com/ZoneMayor/kafka into 
trunk-KAFKA-2875

commit 83b2bcca237ba9445360bbfcb05a0de82c36274f
Author: jinxing 
Date:   2016-01-10T12:39:20Z

KAFKA-2875: wip

commit 6e6f2c20c5730253d8e818c2dc1e5e741a05ac08
Author: jinxing 
Date:   2016-01-10T14:53:28Z

KAFKA-2875: Classpath contains multiple SLF4J bindings warnings when using 
scripts under bin

commit fbd380659727d991dff242be33cc6a3bb78f4861
Author: ZoneMayor 
Date:   2016-01-28T06:28:25Z

Merge pull request #15 from apache/trunk

2016-01-28

commit f21aa55ed68907376d5b0924e228875530cc1046
Author: jinxing 
Date:   2016-01-28T07:10:30Z

KAFKA-2875: remove slf4j multi binding warnings when running form source 
distribution (merge to trunk and resolve conflict)

commit 51fcc408302ebb0c4adaf2a4d0e6647cc469c6a0
Author: jinxing 
Date:   2016-01-28T07:43:52Z

added a new line

commit 8a6cbad74ca4f07a4c70c1d522b604d58e4917c6
Author: jinxing 
Date:   2016-02-01T08:49:06Z

KAFKA-2875: create deduplicated dependant-libs and use symlink to construct 
classpath

commit 153a1177c943e76c9c8457c47244ec59ea91d6fc
Author: jinxing 
Date:   2016-02-01T09:42:37Z

small fix

commit 1d283120bd7c3b90928090c4d22376d4ac05c4d5
Author: jinxing 
Date:   2016-02-01T10:09:46Z

KAFKA-2875: modify classpath in windows bat

commit 29c1797ae4f3ba47445e45049c8fc0fc2e1609f4
Author: jinxing 
Date:   2016-02-01T10:13:20Z

mod server.properties for test

commit a1993e5ca2908862340113ce965bd7fdc5020bab
Author: jinxing 
Date:   2016-02-01T12:44:22Z

KAFKA-2875: small fix

commit e523bd2ce91e03e38c20413aef3c48998fc3c263
Author: jinxing 
Date:   2016-02-01T16:12:27Z

KAFKA-2875: small fix

commit fb27bdeba925e6833ca9bc9feb1d6d3cf55c5aaf
Author: jinxing 
Date:   2016-02-02T09:44:32Z

KAFKA-2875: replace PROJECT_NAMES with PROJECT_NAME, use the central 
deduplicated libs if PROJECT_NAME not specified

commit f8ba5a920a0db8654c0776ad8449b167689c0eb4
Author: jinxing 
Date:   2016-02-02T09:48:00Z

small fix




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

RE: [jira] [Commented] (KAFKA-2143) Replicas get ahead of leader and fail

[chenlax]

i meet the same issues,the error log,
Error when processing fetch request for partition [To_S3_comm_V0_10,2]
offset 456234794 from follower with correlation id 254117341. Possible cause: 
Request for offset 456234794 but we only have log segmen
ts in the range 432322850 to 456234793. (kafka.server.ReplicaManager)

and i find in the handleOffsetOutOfRange,it just check (leaderEndOffset < 
replica.logEndOffset.messageOffset),if not, will delete the all log

INFO Scheduling log segment 432322850 for log To_S3_comm_V0_10-2 for deletion. 
(kafka.log.Log)
...
INFO Deleting segment 434379909 from log To_S3_comm_V0_10-2. (kafka.log.Log)

i think,it must add check  (log.logEndOffset < leaderStartOffset) when 
leaderEndOffset not small than replica.logEndOffset.messageOffset.


Thanks，
Lax


> Date: Fri, 4 Sep 2015 00:41:47 +
> From: j...@apache.org
> To: dev@kafka.apache.org
> Subject: [jira] [Commented] (KAFKA-2143) Replicas get ahead of leader and fail
> 
> 
> [ 
> https://issues.apache.org/jira/browse/KAFKA-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14730109#comment-14730109
>  ] 
> 
> Jun Rao commented on KAFKA-2143:
> 
> 
> [~becket_qin], since before step (3), both A and B are in ISR, the last 
> committed offset in A can't be larger than 3000. So, in step (3), if A 
> becomes a follower, it has to first truncate its log to last committed offset 
> before fetching. So, at that point, A's fetch offset can't be larger than 
> 3000 and therefore won't be out of range.
> 
> The following is a alternative scenario that can cause this.
> 
> 1) Broker A (leader) receives messages to 5000
> 2) Broker B (follower) receives messages to 3000 (it is still in ISR because 
> of replica.lag.max.messages)
> 3) For some reason, B is dropped out of ISR.
> 4) Broker A (the only one in ISR) commits messages to 5000.
> 5) For some reason, Broker A is considered dead and Broker B is live.
> 6) Broker B is selected as the new leader (unclean leader election) and is 
> the only one in ISR.
> 7) Broker A is considered live again and starts fetching from 5000 (last 
> committed offset) and gets OffsetOutOfRangeException.
> 8) In the mean time, B receives more messages to offset 6000.
> 9) Broker A tries to handle OffsetOutOfRangeException and finds out leader 
> B's log end offset is now larger than its log end offset and truncates all 
> its log.
> 
> Your patch reduces the amount of the data that Broker A needs to replicate in 
> step 9, which is probably fine. However, we probably should first verify if 
> this is indeed what's happening since it seems that it should happen rarely. 
> Also, KAFKA-2477 reports a similar issue w/o any leadership change. So, may 
> be there is something else that can cause this.
> 
> > Replicas get ahead of leader and fail
> > -
> >
> > Key: KAFKA-2143
> > URL: https://issues.apache.org/jira/browse/KAFKA-2143
> > Project: Kafka
> >  Issue Type: Bug
> >  Components: replication
> >Affects Versions: 0.8.2.1
> >Reporter: Evan Huus
> >Assignee: Jiangjie Qin
> >
> > On a cluster of 6 nodes, we recently saw a case where a single 
> > under-replicated partition suddenly appeared, replication lag spiked, and 
> > network IO spiked. The cluster appeared to recover eventually on its own,
> > Looking at the logs, the thing which failed was partition 7 of the topic 
> > {{background_queue}}. It had an ISR of 1,4,3 and its leader at the time was 
> > 3. Here are the interesting log lines:
> > On node 3 (the leader):
> > {noformat}
> > [2015-04-23 16:50:05,879] ERROR [Replica Manager on Broker 3]: Error when 
> > processing fetch request for partition [background_queue,7] offset 
> > 3722949957 from follower with correlation id 148185816. Possible cause: 
> > Request for offset 3722949957 but we only have log segments in the range 
> > 3648049863 to 3722949955. (kafka.server.ReplicaManager)
> > [2015-04-23 16:50:05,879] ERROR [Replica Manager on Broker 3]: Error when 
> > processing fetch request for partition [background_queue,7] offset 
> > 3722949957 from follower with correlation id 156007054. Possible cause: 
> > Request for offset 3722949957 but we only have log segments in the range 
> > 3648049863 to 3722949955. (kafka.server.ReplicaManager)
> > [2015-04-23 16:50:13,960] INFO Partition [background_queue,7] on broker 3: 
> > Shrinking ISR for partition [background_queue,7] from 1,4,3 to 3 
> > (kafka.cluster.Partition)
> > {noformat}
> > Note that both replicas suddenly asked for an offset *ahead* of the 
> > available offsets.
> > And on nodes 1 and 4 (the replicas) many occurrences of the following:
> > {noformat}
> > [2015-04-23 16:50:05,935] INFO Scheduling log segment 3648049863 for log 
> > background_queue-7 for deletion. (kafka.log.Log) (edited)
> > {noformat}
> > Bas

Build failed in Jenkins: kafka-trunk-jdk8 #340

[Apache Jenkins Server]

See 

Changes:

[me] KAFKA-3195; Transient test failure in OffsetCheckpointTest.testReadWrite

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H11 (docker Ubuntu ubuntu yahoo-not-h2) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision e8343e67e1ca74c4c0ebd994b9811b9fed7aa386 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f e8343e67e1ca74c4c0ebd994b9811b9fed7aa386
 > git rev-list fd10f6dff11c74ab4b12229305ab78fb694d5ea9 # timeout=10
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson2483085432114157766.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.4-rc-2/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:downloadWrapper

BUILD SUCCESSFUL

Total time: 29.336 secs
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson1825640557904132334.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew -Dorg.gradle.project.maxParallelForks=1 clean jarAll testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.10/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:clean UP-TO-DATE
:clients:clean UP-TO-DATE
:connect:clean UP-TO-DATE
:core:clean UP-TO-DATE
:examples:clean UP-TO-DATE
:log4j-appender:clean UP-TO-DATE
:streams:clean UP-TO-DATE
:tools:clean UP-TO-DATE
:connect:api:clean UP-TO-DATE
:connect:file:clean UP-TO-DATE
:connect:json:clean UP-TO-DATE
:connect:runtime:clean UP-TO-DATE
:streams:examples:clean UP-TO-DATE
:jar_core_2_10
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk8:clients:compileJava
:jar_core_2_10 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Failed to capture snapshot of input files for task 'compileJava' during 
up-to-date check.  See stacktrace for details.
> Could not add entry 
> '/home/jenkins/.gradle/caches/modules-2/files-2.1/net.jpountz.lz4/lz4/1.3.0/c708bb2590c0652a642236ef45d9f99ff842a2ce/lz4-1.3.0.jar'
>  to cache fileHashes.bin 
> (

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 33.855 secs
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
ERROR: Publisher 'Publish JUnit test result report' failed: No test report 
files were found. Configuration error?
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2

[jira] [Commented] (KAFKA-2875) Class path contains multiple SLF4J bindings warnings when using scripts under bin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129723#comment-15129723
 ] 

ASF GitHub Bot commented on KAFKA-2875:
---

GitHub user ZoneMayor reopened a pull request:

https://github.com/apache/kafka/pull/693

KAFKA-2875:  remove slf4j multi binding warnings when running form source 
distribution

hi @ijuma I reopened this pr again (sorry for my inexperience using github);
I think I did much deduplication for the script;
Please have a look when you have time  : - )

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ZoneMayor/kafka trunk-KAFKA-2875

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/693.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #693


commit 34240b52e1b70aa172b65155f6042243d838b420
Author: ZoneMayor 
Date:   2015-12-18T07:22:20Z

Merge pull request #12 from apache/trunk

2015-12-18

commit ffedf6fd04280e89978531fd73e7fe37a4d9bbed
Author: jinxing 
Date:   2015-12-18T07:24:14Z

KAFKA-2875 Class path contains multiple SLF4J bindings warnings when using 
scripts under bin

commit 52d02f333e86d06cfa8fff5facd18999b3db6d83
Author: ZoneMayor 
Date:   2015-12-30T03:08:08Z

Merge pull request #13 from apache/trunk

2015-12-30

commit d56be0b9e0849660c07d656c6019f9cc2f17ae55
Author: ZoneMayor 
Date:   2016-01-10T09:24:06Z

Merge pull request #14 from apache/trunk

2016-1-10

commit a937ad38ac90b90a57a1969bdd8ce06d6faaaeb1
Author: jinxing 
Date:   2016-01-10T10:28:18Z

Merge branch 'trunk-KAFKA-2875' of https://github.com/ZoneMayor/kafka into 
trunk-KAFKA-2875

commit 83b2bcca237ba9445360bbfcb05a0de82c36274f
Author: jinxing 
Date:   2016-01-10T12:39:20Z

KAFKA-2875: wip

commit 6e6f2c20c5730253d8e818c2dc1e5e741a05ac08
Author: jinxing 
Date:   2016-01-10T14:53:28Z

KAFKA-2875: Classpath contains multiple SLF4J bindings warnings when using 
scripts under bin

commit fbd380659727d991dff242be33cc6a3bb78f4861
Author: ZoneMayor 
Date:   2016-01-28T06:28:25Z

Merge pull request #15 from apache/trunk

2016-01-28

commit f21aa55ed68907376d5b0924e228875530cc1046
Author: jinxing 
Date:   2016-01-28T07:10:30Z

KAFKA-2875: remove slf4j multi binding warnings when running form source 
distribution (merge to trunk and resolve conflict)

commit 51fcc408302ebb0c4adaf2a4d0e6647cc469c6a0
Author: jinxing 
Date:   2016-01-28T07:43:52Z

added a new line

commit 8a6cbad74ca4f07a4c70c1d522b604d58e4917c6
Author: jinxing 
Date:   2016-02-01T08:49:06Z

KAFKA-2875: create deduplicated dependant-libs and use symlink to construct 
classpath

commit 153a1177c943e76c9c8457c47244ec59ea91d6fc
Author: jinxing 
Date:   2016-02-01T09:42:37Z

small fix

commit 1d283120bd7c3b90928090c4d22376d4ac05c4d5
Author: jinxing 
Date:   2016-02-01T10:09:46Z

KAFKA-2875: modify classpath in windows bat

commit 29c1797ae4f3ba47445e45049c8fc0fc2e1609f4
Author: jinxing 
Date:   2016-02-01T10:13:20Z

mod server.properties for test

commit a1993e5ca2908862340113ce965bd7fdc5020bab
Author: jinxing 
Date:   2016-02-01T12:44:22Z

KAFKA-2875: small fix

commit e523bd2ce91e03e38c20413aef3c48998fc3c263
Author: jinxing 
Date:   2016-02-01T16:12:27Z

KAFKA-2875: small fix

commit fb27bdeba925e6833ca9bc9feb1d6d3cf55c5aaf
Author: jinxing 
Date:   2016-02-02T09:44:32Z

KAFKA-2875: replace PROJECT_NAMES with PROJECT_NAME, use the central 
deduplicated libs if PROJECT_NAME not specified

commit f8ba5a920a0db8654c0776ad8449b167689c0eb4
Author: jinxing 
Date:   2016-02-02T09:48:00Z

small fix




> Class path contains multiple SLF4J bindings warnings when using scripts under 
> bin
> -
>
> Key: KAFKA-2875
> URL: https://issues.apache.org/jira/browse/KAFKA-2875
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.9.0.0
>Reporter: Ismael Juma
>Assignee: jin xing
>Priority: Minor
>  Labels: patch
> Fix For: 0.9.1.0
>
>
> This adds a lot of noise when running the scripts, see example when running 
> kafka-console-producer.sh:
> {code}
> ~/D/s/kafka-0.9.0.0-src ❯❯❯ ./bin/kafka-console-producer.sh --topic topic 
> --broker-list localhost:9092 ⏎
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala

[GitHub] kafka pull request: KAFKA-2875: remove slf4j multi binding warning...

[ZoneMayor]

Github user ZoneMayor closed the pull request at:

https://github.com/apache/kafka/pull/693


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2875) Class path contains multiple SLF4J bindings warnings when using scripts under bin

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129722#comment-15129722
 ] 

ASF GitHub Bot commented on KAFKA-2875:
---

Github user ZoneMayor closed the pull request at:

https://github.com/apache/kafka/pull/693


> Class path contains multiple SLF4J bindings warnings when using scripts under 
> bin
> -
>
> Key: KAFKA-2875
> URL: https://issues.apache.org/jira/browse/KAFKA-2875
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Affects Versions: 0.9.0.0
>Reporter: Ismael Juma
>Assignee: jin xing
>Priority: Minor
>  Labels: patch
> Fix For: 0.9.1.0
>
>
> This adds a lot of noise when running the scripts, see example when running 
> kafka-console-producer.sh:
> {code}
> ~/D/s/kafka-0.9.0.0-src ❯❯❯ ./bin/kafka-console-producer.sh --topic topic 
> --broker-list localhost:9092 ⏎
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.10.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/core/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/tools/build/dependant-libs-2.10.5/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/api/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/runtime/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/file/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/Users/ijuma/Downloads/scala-releases/kafka-0.9.0.0-src/connect/json/build/dependant-libs/slf4j-log4j12-1.7.6.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2875: remove slf4j multi binding warning...

[ZoneMayor]

GitHub user ZoneMayor reopened a pull request:

https://github.com/apache/kafka/pull/693

KAFKA-2875:  remove slf4j multi binding warnings when running form source 
distribution

hi @ijuma I reopened this pr again (sorry for my inexperience using github);
I think I did much deduplication for the script;
Please have a look when you have time  : - )

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ZoneMayor/kafka trunk-KAFKA-2875

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/693.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #693


commit 34240b52e1b70aa172b65155f6042243d838b420
Author: ZoneMayor 
Date:   2015-12-18T07:22:20Z

Merge pull request #12 from apache/trunk

2015-12-18

commit ffedf6fd04280e89978531fd73e7fe37a4d9bbed
Author: jinxing 
Date:   2015-12-18T07:24:14Z

KAFKA-2875 Class path contains multiple SLF4J bindings warnings when using 
scripts under bin

commit 52d02f333e86d06cfa8fff5facd18999b3db6d83
Author: ZoneMayor 
Date:   2015-12-30T03:08:08Z

Merge pull request #13 from apache/trunk

2015-12-30

commit d56be0b9e0849660c07d656c6019f9cc2f17ae55
Author: ZoneMayor 
Date:   2016-01-10T09:24:06Z

Merge pull request #14 from apache/trunk

2016-1-10

commit a937ad38ac90b90a57a1969bdd8ce06d6faaaeb1
Author: jinxing 
Date:   2016-01-10T10:28:18Z

Merge branch 'trunk-KAFKA-2875' of https://github.com/ZoneMayor/kafka into 
trunk-KAFKA-2875

commit 83b2bcca237ba9445360bbfcb05a0de82c36274f
Author: jinxing 
Date:   2016-01-10T12:39:20Z

KAFKA-2875: wip

commit 6e6f2c20c5730253d8e818c2dc1e5e741a05ac08
Author: jinxing 
Date:   2016-01-10T14:53:28Z

KAFKA-2875: Classpath contains multiple SLF4J bindings warnings when using 
scripts under bin

commit fbd380659727d991dff242be33cc6a3bb78f4861
Author: ZoneMayor 
Date:   2016-01-28T06:28:25Z

Merge pull request #15 from apache/trunk

2016-01-28

commit f21aa55ed68907376d5b0924e228875530cc1046
Author: jinxing 
Date:   2016-01-28T07:10:30Z

KAFKA-2875: remove slf4j multi binding warnings when running form source 
distribution (merge to trunk and resolve conflict)

commit 51fcc408302ebb0c4adaf2a4d0e6647cc469c6a0
Author: jinxing 
Date:   2016-01-28T07:43:52Z

added a new line

commit 8a6cbad74ca4f07a4c70c1d522b604d58e4917c6
Author: jinxing 
Date:   2016-02-01T08:49:06Z

KAFKA-2875: create deduplicated dependant-libs and use symlink to construct 
classpath

commit 153a1177c943e76c9c8457c47244ec59ea91d6fc
Author: jinxing 
Date:   2016-02-01T09:42:37Z

small fix

commit 1d283120bd7c3b90928090c4d22376d4ac05c4d5
Author: jinxing 
Date:   2016-02-01T10:09:46Z

KAFKA-2875: modify classpath in windows bat

commit 29c1797ae4f3ba47445e45049c8fc0fc2e1609f4
Author: jinxing 
Date:   2016-02-01T10:13:20Z

mod server.properties for test

commit a1993e5ca2908862340113ce965bd7fdc5020bab
Author: jinxing 
Date:   2016-02-01T12:44:22Z

KAFKA-2875: small fix

commit e523bd2ce91e03e38c20413aef3c48998fc3c263
Author: jinxing 
Date:   2016-02-01T16:12:27Z

KAFKA-2875: small fix

commit fb27bdeba925e6833ca9bc9feb1d6d3cf55c5aaf
Author: jinxing 
Date:   2016-02-02T09:44:32Z

KAFKA-2875: replace PROJECT_NAMES with PROJECT_NAME, use the central 
deduplicated libs if PROJECT_NAME not specified

commit f8ba5a920a0db8654c0776ad8449b167689c0eb4
Author: jinxing 
Date:   2016-02-02T09:48:00Z

small fix




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3170) Default value of fetch_min_bytes in new consumer is 1024 while doc says it is 1

[Grant Henke (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129719#comment-15129719
 ] 

Grant Henke commented on KAFKA-3170:


[~guozhang] Should this be marked resolved for 0.9.0.1?

> Default value of fetch_min_bytes in new consumer is 1024 while doc says it is 
> 1
> ---
>
> Key: KAFKA-3170
> URL: https://issues.apache.org/jira/browse/KAFKA-3170
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
>
> FETCH_MIN_BYTES_DOC says:
> {quote}
> The minimum amount of data the server should return for a fetch request. If 
> insufficient data is available the request will wait for that much data to 
> accumulate before answering the request. The default setting of 1 byte means 
> that fetch requests are answered as soon as a single byte of data is 
> available or the fetch request times out waiting for data to arrive. Setting 
> this to something greater than 1 will cause the server to wait for larger 
> amounts of data to accumulate which can improve server throughput a bit at 
> the cost of some additional latency.
> {quote}
> But the default value is actually set to 1024. Either the doc or the value 
> needs to be changed. Perhaps 1 is a better default?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #339

[Apache Jenkins Server]

See 

Changes:

[me] MINOR: Increment ducktape dependency

--
[...truncated 6631 lines...]
org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testPutConnectorTaskConfigs PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testPutConnectorTaskConfigsConnectorNotFound PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testPutConnectorConfig PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testCreateConnector PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testGetConnector PASSED

org.apache.kafka.connect.util.ShutdownableThreadTest > testGracefulShutdown 
PASSED

org.apache.kafka.connect.util.ShutdownableThreadTest > testForcibleShutdown 
PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testStartStop PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testConsumerError PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testProducerError PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testReloadOnStart PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testSendAndReadToEnd PASSED

org.apache.kafka.connect.storage.KafkaOffsetBackingStoreTest > testMissingTopic 
PASSED

org.apache.kafka.connect.storage.KafkaOffsetBackingStoreTest > testSetFailure 
PASSED

org.apache.kafka.connect.storage.KafkaOffsetBackingStoreTest > testGetSet PASSED

org.apache.kafka.connect.storage.KafkaOffsetBackingStoreTest > testStartStop 
PASSED

org.apache.kafka.connect.storage.KafkaOffsetBackingStoreTest > 
testReloadOnStart PASSED

org.apache.kafka.connect.storage.KafkaConfigStorageTest > testRestore PASSED

org.apache.kafka.connect.storage.KafkaConfigStorageTest > 
testPutTaskConfigsDoesNotResolveAllInconsistencies PASSED

org.apache.kafka.connect.storage.KafkaConfigStorageTest > testStartStop PASSED

org.apache.kafka.connect.storage.KafkaConfigStorageTest > 
testPutConnectorConfig PASSED

org.apache.kafka.connect.storage.KafkaConfigStorageTest > testPutTaskConfigs 
PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > testNoOffsetsToFlush 
PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > 
testWriteNullKeyFlush PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > testWriteFlush PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > testAlreadyFlushing 
PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > 
testFlushFailureReplacesOffsets PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > 
testCancelBeforeAwaitFlush PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > 
testCancelAfterAwaitFlush PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > 
testWriteNullValueFlush PASSED

org.apache.kafka.connect.storage.FileOffsetBackingStoreTest > testSaveRestore 
PASSED

org.apache.kafka.connect.storage.FileOffsetBackingStoreTest > testGetSet PASSED
:testAll

BUILD SUCCESSFUL

Total time: 1 hrs 2 mins 31.546 secs
+ ./gradlew --stacktrace docsJarAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.10/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:docsJar_2_10
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk8:clients:compileJava UP-TO-DATE
:kafka-trunk-jdk8:clients:processResources UP-TO-DATE
:kafka-trunk-jdk8:clients:classes UP-TO-DATE
:kafka-trunk-jdk8:clients:determineCommitId UP-TO-DATE
:kafka-trunk-jdk8:clients:createVersionFile
:kafka-trunk-jdk8:clients:jar UP-TO-DATE
:kafka-trunk-jdk8:clients:javadoc
:docsJar_2_10 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Could not add entry 
'
 to cache fileHashes.bin 
(
> Corrupted FreeListBlock 678094 found in cache 
> '

* Try:
Run with --info or --debug option to get more log output.

* Exception is:
org.gradle.api.UncheckedIOException: Could not add entry 
'
 to cache fileHashes.bin 
(
at 
org.gradle.cache.internal.btree.BTreePersistentIndexedCache.put(BTreePersistentIndexedCache.java:155)
at 
org.gradle.cache.internal.DefaultMultiProcessSafePersistentIndexedCache$2.run(DefaultMultiProcessSafePersistentIndexedCache.java:51)
at 
org.gradle.cache.internal.DefaultFileLockManager$DefaultFileLock.doWriteAction(DefaultFileLockManager.java:173)
at 
org.gra

[jira] [Assigned] (KAFKA-3044) Consumer.poll doesnot return messages when poll interval is less

[Praveen Devarao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Praveen Devarao reassigned KAFKA-3044:
--

Assignee: Praveen Devarao  (was: Jason Gustafson)

> Consumer.poll doesnot return messages when poll interval is less
> 
>
> Key: KAFKA-3044
> URL: https://issues.apache.org/jira/browse/KAFKA-3044
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Praveen Devarao
>Assignee: Praveen Devarao
> Fix For: 0.9.1.0
>
>
> When seeking to particular position in consumer and starting poll with 
> timeout param 0 the consumer does not come back with data though there is 
> data published via a producer already. If the timeout is increased slowly in 
> chunks of 100ms then at 700ms value the consumer returns back the record on 
> first call to poll.
> Docs 
> [http://kafka.apache.org/090/javadoc/org/apache/kafka/clients/consumer/KafkaConsumer.html#poll(long)]
>  for poll reads if timeout is 0 then data will be returned immediately but 
> the behaviour seen is that data is not returned.
> The test code I am using can be found here 
> https://gist.github.com/praveend/013dcab01ebb8c7e2f2d
> I have created a topic with data published as below and then running the test 
> program [ConsumerPollTest.java]
> $ bin/kafka-topics.sh --create --zookeeper localhost:2181 
> --replication-factor 1 --partitions 1 --topic mytopic
> $ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic mytopic
> Hello
> Hai
> bye
> $ java ConsumerPollTest
> I have published this 3 lines of data to kafka only oncelater on I just 
> use the above program with different poll interval
> Let me know if I am missing anything and interpreting it wrongly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3189) Kafka server returns UnknownServerException for inherited exceptions

[Grant Henke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Henke updated KAFKA-3189:
---
Status: Patch Available  (was: Open)

> Kafka server returns UnknownServerException for inherited exceptions
> 
>
> Key: KAFKA-3189
> URL: https://issues.apache.org/jira/browse/KAFKA-3189
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jiangjie Qin
>Assignee: Grant Henke
> Fix For: 0.9.1.0
>
>
> This issue was introduced in KAFKA-2929. The problem is that we are using 
> o.a.k.common.protocol.Errors.forException() while some exceptions thrown by 
> the broker are still using old scala exception. This cause 
> Errors.forException() always return UnknownServerException.
> InvalidMessageException is inherited from CorruptRecordException. But it 
> seems Errors.forException() needs the exception class to be the exact class, 
> so it does not map the subclass InvalidMessageException to the correct error 
> code. Instead it returns -1 which is UnknownServerException.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3189) Kafka server returns UnknownServerException for inherited exceptions

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129701#comment-15129701
 ] 

ASF GitHub Bot commented on KAFKA-3189:
---

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/856

KAFKA-3189: Kafka server returns UnknownServerException for inherited…

… exceptions

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka inherited-errors

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/856.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #856


commit 4de1cce990d3df283f9688d4b91d7a6582b55853
Author: Grant Henke 
Date:   2016-02-03T03:10:30Z

KAFKA-3189: Kafka server returns UnknownServerException for inherited 
exceptions




> Kafka server returns UnknownServerException for inherited exceptions
> 
>
> Key: KAFKA-3189
> URL: https://issues.apache.org/jira/browse/KAFKA-3189
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jiangjie Qin
>Assignee: Grant Henke
> Fix For: 0.9.1.0
>
>
> This issue was introduced in KAFKA-2929. The problem is that we are using 
> o.a.k.common.protocol.Errors.forException() while some exceptions thrown by 
> the broker are still using old scala exception. This cause 
> Errors.forException() always return UnknownServerException.
> InvalidMessageException is inherited from CorruptRecordException. But it 
> seems Errors.forException() needs the exception class to be the exact class, 
> so it does not map the subclass InvalidMessageException to the correct error 
> code. Instead it returns -1 which is UnknownServerException.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3189: Kafka server returns UnknownServer...

[granthenke]

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/856

KAFKA-3189: Kafka server returns UnknownServerException for inherited…

… exceptions

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka inherited-errors

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/856.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #856


commit 4de1cce990d3df283f9688d4b91d7a6582b55853
Author: Grant Henke 
Date:   2016-02-03T03:10:30Z

KAFKA-3189: Kafka server returns UnknownServerException for inherited 
exceptions




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3189) Kafka server returns UnknownServerException for inherited exceptions

[Grant Henke (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129685#comment-15129685
 ] 

Grant Henke commented on KAFKA-3189:


[~becket_qin] [~ijuma] I updated the jira description to match the exact 
scenario. I will open a PR shortly.

> Kafka server returns UnknownServerException for inherited exceptions
> 
>
> Key: KAFKA-3189
> URL: https://issues.apache.org/jira/browse/KAFKA-3189
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jiangjie Qin
>Assignee: Grant Henke
> Fix For: 0.9.1.0
>
>
> This issue was introduced in KAFKA-2929. The problem is that we are using 
> o.a.k.common.protocol.Errors.forException() while some exceptions thrown by 
> the broker are still using old scala exception. This cause 
> Errors.forException() always return UnknownServerException.
> InvalidMessageException is inherited from CorruptRecordException. But it 
> seems Errors.forException() needs the exception class to be the exact class, 
> so it does not map the subclass InvalidMessageException to the correct error 
> code. Instead it returns -1 which is UnknownServerException.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3189) Kafka server returns UnknownServerException for inherited exceptions

[Grant Henke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Henke updated KAFKA-3189:
---
Description: 
This issue was introduced in KAFKA-2929. The problem is that we are using 
o.a.k.common.protocol.Errors.forException() while some exceptions thrown by the 
broker are still using old scala exception. This cause Errors.forException() 
always return UnknownServerException.

InvalidMessageException is inherited from CorruptRecordException. But it seems 
Errors.forException() needs the exception class to be the exact class, so it 
does not map the subclass InvalidMessageException to the correct error code. 
Instead it returns -1 which is UnknownServerException.

  was:
This issue was introduced in KAFKA-2929. The problem is that we are using 
o.a.k.common.protocol.Errors.forException() while all exceptions thrown by the 
broker are still using old scala exception. This cause Errors.forException() 
always return UnknownServerException.

We should either switch back to Errors.forException() to ErrorMapping.codeFor() 
or migrate all the exceptions to Java exception.

I prefer the latter option since it seems to be a simple class replacement.


> Kafka server returns UnknownServerException for inherited exceptions
> 
>
> Key: KAFKA-3189
> URL: https://issues.apache.org/jira/browse/KAFKA-3189
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jiangjie Qin
>Assignee: Grant Henke
> Fix For: 0.9.1.0
>
>
> This issue was introduced in KAFKA-2929. The problem is that we are using 
> o.a.k.common.protocol.Errors.forException() while some exceptions thrown by 
> the broker are still using old scala exception. This cause 
> Errors.forException() always return UnknownServerException.
> InvalidMessageException is inherited from CorruptRecordException. But it 
> seems Errors.forException() needs the exception class to be the exact class, 
> so it does not map the subclass InvalidMessageException to the correct error 
> code. Instead it returns -1 which is UnknownServerException.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3189) Kafka server returns UnknownServerException for inherited exceptions

[Grant Henke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Henke updated KAFKA-3189:
---
Summary: Kafka server returns UnknownServerException for inherited 
exceptions  (was: Kafka server always return UnknownServerException.)

> Kafka server returns UnknownServerException for inherited exceptions
> 
>
> Key: KAFKA-3189
> URL: https://issues.apache.org/jira/browse/KAFKA-3189
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Jiangjie Qin
>Assignee: Grant Henke
> Fix For: 0.9.1.0
>
>
> This issue was introduced in KAFKA-2929. The problem is that we are using 
> o.a.k.common.protocol.Errors.forException() while all exceptions thrown by 
> the broker are still using old scala exception. This cause 
> Errors.forException() always return UnknownServerException.
> We should either switch back to Errors.forException() to 
> ErrorMapping.codeFor() or migrate all the exceptions to Java exception.
> I prefer the latter option since it seems to be a simple class replacement.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3195) Transient test failure in OffsetCheckpointTest.testReadWrite

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-3195:
-
   Resolution: Fixed
Fix Version/s: 0.9.1.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 855
[https://github.com/apache/kafka/pull/855]

> Transient test failure in OffsetCheckpointTest.testReadWrite
> 
>
> Key: KAFKA-3195
> URL: https://issues.apache.org/jira/browse/KAFKA-3195
> Project: Kafka
>  Issue Type: Bug
>  Components: kafka streams
>Reporter: Ewen Cheslack-Postava
>Assignee: Ismael Juma
> Fix For: 0.9.1.0
>
>
> It looks like its probably an issue with parallel tests trying to access the 
> same fixed path, where one test deletes the file. Saw this on 
> 86a9036a7b03c8ae07d014c25a5eedc315544139.
> {quote}
> org.apache.kafka.streams.state.internals.OffsetCheckpointTest > testReadWrite 
> FAILED
> java.io.FileNotFoundException: 
> /tmp/kafka-streams/offset_checkpoint.test.tmp (No such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:171)
> at 
> org.apache.kafka.streams.state.internals.OffsetCheckpoint.write(OffsetCheckpoint.java:68)
> at 
> org.apache.kafka.streams.state.internals.OffsetCheckpointTest.testReadWrite(OffsetCheckpointTest.java:48)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3195) Transient test failure in OffsetCheckpointTest.testReadWrite

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129646#comment-15129646
 ] 

ASF GitHub Bot commented on KAFKA-3195:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/855


> Transient test failure in OffsetCheckpointTest.testReadWrite
> 
>
> Key: KAFKA-3195
> URL: https://issues.apache.org/jira/browse/KAFKA-3195
> Project: Kafka
>  Issue Type: Bug
>  Components: kafka streams
>Reporter: Ewen Cheslack-Postava
>Assignee: Ismael Juma
> Fix For: 0.9.1.0
>
>
> It looks like its probably an issue with parallel tests trying to access the 
> same fixed path, where one test deletes the file. Saw this on 
> 86a9036a7b03c8ae07d014c25a5eedc315544139.
> {quote}
> org.apache.kafka.streams.state.internals.OffsetCheckpointTest > testReadWrite 
> FAILED
> java.io.FileNotFoundException: 
> /tmp/kafka-streams/offset_checkpoint.test.tmp (No such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:171)
> at 
> org.apache.kafka.streams.state.internals.OffsetCheckpoint.write(OffsetCheckpoint.java:68)
> at 
> org.apache.kafka.streams.state.internals.OffsetCheckpointTest.testReadWrite(OffsetCheckpointTest.java:48)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3174) Re-evaluate the CRC32 class performance.

[Jiangjie Qin (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129644#comment-15129644
 ] 

Jiangjie Qin commented on KAFKA-3174:
-

[~ijuma] Thanks for the clarification. Yes, I agree we'd better run a 
parameterized test to see what is the performance difference between current 
Crc32 and Java CRC32. But I might be busy in the next a few days. So please 
feel free to take this ticket if you are interested. Thanks.

> Re-evaluate the CRC32 class performance.
> 
>
> Key: KAFKA-3174
> URL: https://issues.apache.org/jira/browse/KAFKA-3174
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.9.1.0
>
>
> We used org.apache.kafka.common.utils.CRC32 in clients because it has better 
> performance than java.util.zip.CRC32 in Java 1.6.
> In a recent test I ran it looks in Java 1.8 the CRC32 class is 2x as fast as 
> the Crc32 class we are using now. We may want to re-evaluate the performance 
> of Crc32 class and see it makes sense to simply use java CRC32 instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3195; Transient test failure in OffsetCh...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/855


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (KAFKA-3197) Producer can send message out of order even when in flight request is set to 1.

[Jiangjie Qin (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jiangjie Qin reassigned KAFKA-3197:
---

Assignee: Jiangjie Qin  (was: Jun Rao)

> Producer can send message out of order even when in flight request is set to 
> 1.
> ---
>
> Key: KAFKA-3197
> URL: https://issues.apache.org/jira/browse/KAFKA-3197
> Project: Kafka
>  Issue Type: Bug
>  Components: clients, producer 
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> The issue we saw is following:
> 1. Producer send message 0 to topic-partition-0 on broker A. The in-flight 
> request to broker A is 1.
> 2. The request is somehow lost
> 3. Producer refreshed its topic metadata and found leader of 
> topic-partition-0 migrated from broker A to broker B.
> 4. Because there is no in-flight request to broker B. All the subsequent 
> messages to topic-partition-0 in the record accumulator are sent to broker B.
> 5. Later on when the request in step (1) times out, message 0 will be retried 
> and sent to broker B. At this point, all the later messages has already been 
> sent, so we have re-order.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[DISCUSS] KIP-46: Self Healing

[Aditya Auradkar]

Hey everyone,

I just created a kip to discuss automated replica reassignment when we lose
a broker in the cluster.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-46%3A+Self+Healing+Kafka

Any feedback is welcome.

Thanks,
Aditya

[jira] [Created] (KAFKA-3197) Producer can send message out of order even when in flight request is set to 1.

[Jiangjie Qin (JIRA)]

Jiangjie Qin created KAFKA-3197:
---

 Summary: Producer can send message out of order even when in 
flight request is set to 1.
 Key: KAFKA-3197
 URL: https://issues.apache.org/jira/browse/KAFKA-3197
 Project: Kafka
  Issue Type: Bug
  Components: clients, producer 
Affects Versions: 0.9.0.0
Reporter: Jiangjie Qin
Assignee: Jun Rao
 Fix For: 0.9.0.1


The issue we saw is following:

1. Producer send message 0 to topic-partition-0 on broker A. The in-flight 
request to broker A is 1.
2. The request is somehow lost
3. Producer refreshed its topic metadata and found leader of topic-partition-0 
migrated from broker A to broker B.
4. Because there is no in-flight request to broker B. All the subsequent 
messages to topic-partition-0 in the record accumulator are sent to broker B.
5. Later on when the request in step (1) times out, message 0 will be retried 
and sent to broker B. At this point, all the later messages has already been 
sent, so we have re-order.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: Increment ducktape dependency

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/852


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3157) Mirror maker doesn't commit offset with new consumer when there is no more messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129589#comment-15129589
 ] 

ASF GitHub Bot commented on KAFKA-3157:
---

GitHub user xiaotao183 reopened a pull request:

https://github.com/apache/kafka/pull/821

KAFKA-3157 - Mirror maker doesn't commit offset

Mirror maker doesn't commit offset with new consumer enabled when data 
volume is low. This is caused by infinite loop in ```receive()``` which would 
never jump out of loop if no data coming

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xiaotao183/kafka KAFKA-3157

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/821.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #821


commit c3221c0d0391080889ab899f8570b629561a8529
Author: Tao Xiao 
Date:   2016-01-28T07:42:27Z

KAFKA-3157 - Mirror maker doesn't commit offset with new consumer enabled 
if data volume is low

commit f674012525cc17b6d68e18ccf8819f9e84936ef9
Author: Tao Xiao 
Date:   2016-02-02T10:58:09Z

Reuse NewShinyConsumer receive to make receive behavior in mirror maker 
consistent across the board

commit 18fe1a60ddcc2b75d357f664bdedb5cebfb5bb3a
Author: Tao Xiao 
Date:   2016-02-02T11:12:58Z

add parentheses to next

commit 3f193b04ab9fa6f2abaf80b07c259cbf8fb5dba2
Author: Tao Xiao 
Date:   2016-02-02T11:14:31Z

avoid whitespace in empty line




> Mirror maker doesn't commit offset with new consumer when there is no more 
> messages
> ---
>
> Key: KAFKA-3157
> URL: https://issues.apache.org/jira/browse/KAFKA-3157
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: TAO XIAO
>Assignee: TAO XIAO
> Fix For: 0.9.0.1
>
>
> Mirror maker doesn't commit messages with new consumer enabled when messages 
> are sent to source within the time range that is smaller than commit interval 
> and no more messages are sent afterwards.
> The steps to reproduce:
> 1. Start mirror maker.
> 2. The default commit interval is 1 min. Send message for less than 1 min for 
> example 10 seconds.
> 3. Don't send more messages.
> 4. Check committed offset with group command. The lag remains unchanged for 
> ever even though the messages have been sent to destination.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3157) Mirror maker doesn't commit offset with new consumer when there is no more messages

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129588#comment-15129588
 ] 

ASF GitHub Bot commented on KAFKA-3157:
---

Github user xiaotao183 closed the pull request at:

https://github.com/apache/kafka/pull/821


> Mirror maker doesn't commit offset with new consumer when there is no more 
> messages
> ---
>
> Key: KAFKA-3157
> URL: https://issues.apache.org/jira/browse/KAFKA-3157
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.0
>Reporter: TAO XIAO
>Assignee: TAO XIAO
> Fix For: 0.9.0.1
>
>
> Mirror maker doesn't commit messages with new consumer enabled when messages 
> are sent to source within the time range that is smaller than commit interval 
> and no more messages are sent afterwards.
> The steps to reproduce:
> 1. Start mirror maker.
> 2. The default commit interval is 1 min. Send message for less than 1 min for 
> example 10 seconds.
> 3. Don't send more messages.
> 4. Check committed offset with group command. The lag remains unchanged for 
> ever even though the messages have been sent to destination.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3157 - Mirror maker doesn't commit offse...

[xiaotao183]

GitHub user xiaotao183 reopened a pull request:

https://github.com/apache/kafka/pull/821

KAFKA-3157 - Mirror maker doesn't commit offset

Mirror maker doesn't commit offset with new consumer enabled when data 
volume is low. This is caused by infinite loop in ```receive()``` which would 
never jump out of loop if no data coming

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/xiaotao183/kafka KAFKA-3157

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/821.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #821


commit c3221c0d0391080889ab899f8570b629561a8529
Author: Tao Xiao 
Date:   2016-01-28T07:42:27Z

KAFKA-3157 - Mirror maker doesn't commit offset with new consumer enabled 
if data volume is low

commit f674012525cc17b6d68e18ccf8819f9e84936ef9
Author: Tao Xiao 
Date:   2016-02-02T10:58:09Z

Reuse NewShinyConsumer receive to make receive behavior in mirror maker 
consistent across the board

commit 18fe1a60ddcc2b75d357f664bdedb5cebfb5bb3a
Author: Tao Xiao 
Date:   2016-02-02T11:12:58Z

add parentheses to next

commit 3f193b04ab9fa6f2abaf80b07c259cbf8fb5dba2
Author: Tao Xiao 
Date:   2016-02-02T11:14:31Z

avoid whitespace in empty line




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-3157 - Mirror maker doesn't commit offse...

[xiaotao183]

Github user xiaotao183 closed the pull request at:

https://github.com/apache/kafka/pull/821


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3195) Transient test failure in OffsetCheckpointTest.testReadWrite

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3195:
---
Reviewer: Guozhang Wang
  Status: Patch Available  (was: Open)

> Transient test failure in OffsetCheckpointTest.testReadWrite
> 
>
> Key: KAFKA-3195
> URL: https://issues.apache.org/jira/browse/KAFKA-3195
> Project: Kafka
>  Issue Type: Bug
>  Components: kafka streams
>Reporter: Ewen Cheslack-Postava
>Assignee: Ismael Juma
>
> It looks like its probably an issue with parallel tests trying to access the 
> same fixed path, where one test deletes the file. Saw this on 
> 86a9036a7b03c8ae07d014c25a5eedc315544139.
> {quote}
> org.apache.kafka.streams.state.internals.OffsetCheckpointTest > testReadWrite 
> FAILED
> java.io.FileNotFoundException: 
> /tmp/kafka-streams/offset_checkpoint.test.tmp (No such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:171)
> at 
> org.apache.kafka.streams.state.internals.OffsetCheckpoint.write(OffsetCheckpoint.java:68)
> at 
> org.apache.kafka.streams.state.internals.OffsetCheckpointTest.testReadWrite(OffsetCheckpointTest.java:48)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3195; Transient test failure in OffsetCh...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/855

KAFKA-3195; Transient test failure in OffsetCheckpointTest.testReadWrite



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-3195-offset-checkpoint-test-transient-failure

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/855.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #855


commit 4db54d599de7461aa15d717714747bc03d9857bc
Author: Ismael Juma 
Date:   2016-02-03T01:28:43Z

Use temp file in `OffsetCheckpointTest`




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3195) Transient test failure in OffsetCheckpointTest.testReadWrite

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129554#comment-15129554
 ] 

ASF GitHub Bot commented on KAFKA-3195:
---

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/855

KAFKA-3195; Transient test failure in OffsetCheckpointTest.testReadWrite



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-3195-offset-checkpoint-test-transient-failure

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/855.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #855


commit 4db54d599de7461aa15d717714747bc03d9857bc
Author: Ismael Juma 
Date:   2016-02-03T01:28:43Z

Use temp file in `OffsetCheckpointTest`




> Transient test failure in OffsetCheckpointTest.testReadWrite
> 
>
> Key: KAFKA-3195
> URL: https://issues.apache.org/jira/browse/KAFKA-3195
> Project: Kafka
>  Issue Type: Bug
>  Components: kafka streams
>Reporter: Ewen Cheslack-Postava
>Assignee: Ismael Juma
>
> It looks like its probably an issue with parallel tests trying to access the 
> same fixed path, where one test deletes the file. Saw this on 
> 86a9036a7b03c8ae07d014c25a5eedc315544139.
> {quote}
> org.apache.kafka.streams.state.internals.OffsetCheckpointTest > testReadWrite 
> FAILED
> java.io.FileNotFoundException: 
> /tmp/kafka-streams/offset_checkpoint.test.tmp (No such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:171)
> at 
> org.apache.kafka.streams.state.internals.OffsetCheckpoint.write(OffsetCheckpoint.java:68)
> at 
> org.apache.kafka.streams.state.internals.OffsetCheckpointTest.testReadWrite(OffsetCheckpointTest.java:48)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3178) Expose a method in AdminUtils to manually truncate a specific partition to a particular offset

[Vahid Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129547#comment-15129547
 ] 

Vahid Hashemian commented on KAFKA-3178:


[~wwarshaw] Thanks for letting me know. I won't work on it then.

> Expose a method in AdminUtils to manually truncate a specific partition to a 
> particular offset
> --
>
> Key: KAFKA-3178
> URL: https://issues.apache.org/jira/browse/KAFKA-3178
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Bill Warshaw
>Assignee: Vahid Hashemian
>  Labels: kafka
>
> h3. Description
> One of Kafka's officially-described use cases is a distributed commit log 
> (http://kafka.apache.org/documentation.html#uses_commitlog).  In this case, 
> for a distributed service that needed a commit log, there would be a topic 
> with a single partition to guarantee log order.  This service would use the 
> commit log to re-sync failed nodes.  Kafka is generally an excellent fit for 
> such a system, but it does not expose an adequate mechanism for log cleanup 
> in such a case.  The built-in log cleanup mechanisms are based on time / size 
> thresholds, which doesn't work well with a commit log; data can only be 
> deleted from a commit log when the client application determines that it is 
> no longer needed.  Here we propose a new API exposed to clients through 
> AdminUtils that will delete all messages before a certain offset from a 
> specific partition.
> h3. Rejected Alternatives
> - Manually setting / resetting time intervals for log retention configs to 
> periodically flush messages from the logs from before a certain time period.  
> Doing this involves several asynchronous processes, none of which provide any 
> hooks to know when they are actually complete.
> - Rolling a new topic each time we want to cleanup the log.  This is the best 
> existing approach, but is not ideal.  All incoming writes would be paused 
> while waiting for a new topic to be created.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3178) Expose a method in AdminUtils to manually truncate a specific partition to a particular offset

[Bill Warshaw (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129513#comment-15129513
 ] 

Bill Warshaw commented on KAFKA-3178:
-

[~vahid] fyi this ticket may end up closing; we are exploring using the message 
timestamp functionality from KIP-32 to implement a timestamp-based deletion 
policy, which will effectively do the same thing as this proposal, and be 
simpler to implement

> Expose a method in AdminUtils to manually truncate a specific partition to a 
> particular offset
> --
>
> Key: KAFKA-3178
> URL: https://issues.apache.org/jira/browse/KAFKA-3178
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Bill Warshaw
>Assignee: Vahid Hashemian
>  Labels: kafka
>
> h3. Description
> One of Kafka's officially-described use cases is a distributed commit log 
> (http://kafka.apache.org/documentation.html#uses_commitlog).  In this case, 
> for a distributed service that needed a commit log, there would be a topic 
> with a single partition to guarantee log order.  This service would use the 
> commit log to re-sync failed nodes.  Kafka is generally an excellent fit for 
> such a system, but it does not expose an adequate mechanism for log cleanup 
> in such a case.  The built-in log cleanup mechanisms are based on time / size 
> thresholds, which doesn't work well with a commit log; data can only be 
> deleted from a commit log when the client application determines that it is 
> no longer needed.  Here we propose a new API exposed to clients through 
> AdminUtils that will delete all messages before a certain offset from a 
> specific partition.
> h3. Rejected Alternatives
> - Manually setting / resetting time intervals for log retention configs to 
> periodically flush messages from the logs from before a certain time period.  
> Doing this involves several asynchronous processes, none of which provide any 
> hooks to know when they are actually complete.
> - Rolling a new topic each time we want to cleanup the log.  This is the best 
> existing approach, but is not ideal.  All incoming writes would be paused 
> while waiting for a new topic to be created.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3162) KIP-42: Add Producer and Consumer Interceptors

[Anna Povzner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Povzner updated KAFKA-3162:

Status: Patch Available  (was: In Progress)

> KIP-42: Add Producer and Consumer Interceptors
> --
>
> Key: KAFKA-3162
> URL: https://issues.apache.org/jira/browse/KAFKA-3162
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Anna Povzner
>Assignee: Anna Povzner
>
> This JIRA is for main part of KIP-42 implementation, which includes:
> 1. Add ProducerInterceptor interface and call its callbacks from appropriate 
> places in Kafka Producer.
> 2. Add ConsumerInterceptor interface and call its callbacks from appropriate 
> places in Kafka Consumer.
> 3. Add unit tests for interceptor changes
> 4. Add integration test for both mutable consumer and producer interceptors 
> (running at the same time).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3129) Console Producer/Consumer Issue

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129499#comment-15129499
 ] 

Jason Gustafson commented on KAFKA-3129:


I haven't been able to reproduce this. It might be helpful to see some producer 
logs. Can you you turn the level up to DEBUG and attach the log from one of the 
failed cases?

> Console Producer/Consumer Issue
> ---
>
> Key: KAFKA-3129
> URL: https://issues.apache.org/jira/browse/KAFKA-3129
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, producer 
>Affects Versions: 0.9.0.0
>Reporter: Vahid Hashemian
>Assignee: Neha Narkhede
> Attachments: kafka-3129.mov
>
>
> I have been running a simple test case in which I have a text file 
> {{messages.txt}} with 1,000,000 lines (lines contain numbers from 1 to 
> 1,000,000 in ascending order). I run the console consumer like this:
> {{$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test}}
> Topic {{test}} is on 1 partition with a replication factor of 1.
> Then I run the console producer like this:
> {{$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test < 
> messages.txt}}
> Then the console starts receiving the messages. And about half the times it 
> goes all the way to 1,000,000. But, in other cases, it stops short, usually 
> at 999,735.
> I tried running another console consumer on another machine and both 
> consumers behave the same way. I can't see anything related to this in the 
> logs.
> I also ran the same experiment with a similar file of 10,000 lines, and am 
> getting a similar behavior. When the consumer does not receive all the 10,000 
> messages it usually stops at 9,864.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3162) KIP-42: Add Producer and Consumer Interceptors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129500#comment-15129500
 ] 

ASF GitHub Bot commented on KAFKA-3162:
---

GitHub user apovzner opened a pull request:

https://github.com/apache/kafka/pull/854

KAFKA-3162: Added producer and consumer interceptors

This is the most of the KIP-42: Producer and consumer interceptor. (Except 
exposing CRC and record sizes to the interceptor, which is coming as a separate 
PR; tracked by KAFKA-3196).

This PR includes:
1. Add ProducerInterceptor interface and call its callbacks from 
appropriate places in Kafka Producer.
2. Add ConsumerInterceptor interface and call its callbacks from 
appropriate places in Kafka Consumer.
3. Add unit tests for interceptor changes
4. Add integration test for both mutable consumer and producer interceptors.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apovzner/kafka kip42

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/854.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #854


commit d851ab68fadfff4f80318251cdcb4caf2097e161
Author: Anna Povzner 
Date:   2016-02-03T00:40:55Z

KAFKA-3162 Added producer and consumer interceptors




> KIP-42: Add Producer and Consumer Interceptors
> --
>
> Key: KAFKA-3162
> URL: https://issues.apache.org/jira/browse/KAFKA-3162
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Anna Povzner
>Assignee: Anna Povzner
>
> This JIRA is for main part of KIP-42 implementation, which includes:
> 1. Add ProducerInterceptor interface and call its callbacks from appropriate 
> places in Kafka Producer.
> 2. Add ConsumerInterceptor interface and call its callbacks from appropriate 
> places in Kafka Consumer.
> 3. Add unit tests for interceptor changes
> 4. Add integration test for both mutable consumer and producer interceptors 
> (running at the same time).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3162: Added producer and consumer interc...

[apovzner]

GitHub user apovzner opened a pull request:

https://github.com/apache/kafka/pull/854

KAFKA-3162: Added producer and consumer interceptors

This is the most of the KIP-42: Producer and consumer interceptor. (Except 
exposing CRC and record sizes to the interceptor, which is coming as a separate 
PR; tracked by KAFKA-3196).

This PR includes:
1. Add ProducerInterceptor interface and call its callbacks from 
appropriate places in Kafka Producer.
2. Add ConsumerInterceptor interface and call its callbacks from 
appropriate places in Kafka Consumer.
3. Add unit tests for interceptor changes
4. Add integration test for both mutable consumer and producer interceptors.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apovzner/kafka kip42

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/854.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #854


commit d851ab68fadfff4f80318251cdcb4caf2097e161
Author: Anna Povzner 
Date:   2016-02-03T00:40:55Z

KAFKA-3162 Added producer and consumer interceptors




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3162) KIP-42: Add Producer and Consumer Interceptors

[Anna Povzner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Povzner updated KAFKA-3162:

Issue Type: Improvement  (was: Bug)

> KIP-42: Add Producer and Consumer Interceptors
> --
>
> Key: KAFKA-3162
> URL: https://issues.apache.org/jira/browse/KAFKA-3162
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Anna Povzner
>Assignee: Anna Povzner
>
> This JIRA is for main part of KIP-42 implementation, which includes:
> 1. Add ProducerInterceptor interface and call its callbacks from appropriate 
> places in Kafka Producer.
> 2. Add ConsumerInterceptor interface and call its callbacks from appropriate 
> places in Kafka Consumer.
> 3. Add unit tests for interceptor changes
> 4. Add integration test for both mutable consumer and producer interceptors 
> (running at the same time).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3196) KIP-42 (part 2): add record size and CRC to RecordMetadata and ConsumerRecords

[Anna Povzner (JIRA)]

Anna Povzner created KAFKA-3196:
---

 Summary: KIP-42 (part 2): add record size and CRC to 
RecordMetadata and ConsumerRecords
 Key: KAFKA-3196
 URL: https://issues.apache.org/jira/browse/KAFKA-3196
 Project: Kafka
  Issue Type: Improvement
Reporter: Anna Povzner
Assignee: Anna Povzner


This is the second (smaller) part of KIP-42, which includes: Add record size 
and CRC to RecordMetadata and ConsumerRecord.

See details in KIP-42 wiki: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors
 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3162) KIP-42: Add Producer and Consumer Interceptors

[Anna Povzner (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Anna Povzner updated KAFKA-3162:

Description: 
This JIRA is for main part of KIP-42 implementation, which includes:

1. Add ProducerInterceptor interface and call its callbacks from appropriate 
places in Kafka Producer.
2. Add ConsumerInterceptor interface and call its callbacks from appropriate 
places in Kafka Consumer.
3. Add unit tests for interceptor changes
4. Add integration test for both mutable consumer and producer interceptors 
(running at the same time).

  was:
This JIRA is for KIP-42 implementation, which includes:

1. Add ProducerInterceptor interface and call its callbacks from appropriate 
places in Kafka Producer.
2. Add ConsumerInterceptor interface and call its callbacks from appropriate 
places in Kafka Consumer.
3. Add unit tests for interceptor changes
4. Add integration test for both mutable consumer and producer interceptors 
(running at the same time).
5. Add record size and CRC to RecordMetadata and ConsumerRecord.

See details in KIP-42 wiki: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-42%3A+Add+Producer+and+Consumer+Interceptors
 


> KIP-42: Add Producer and Consumer Interceptors
> --
>
> Key: KAFKA-3162
> URL: https://issues.apache.org/jira/browse/KAFKA-3162
> Project: Kafka
>  Issue Type: Bug
>Reporter: Anna Povzner
>Assignee: Anna Povzner
>
> This JIRA is for main part of KIP-42 implementation, which includes:
> 1. Add ProducerInterceptor interface and call its callbacks from appropriate 
> places in Kafka Producer.
> 2. Add ConsumerInterceptor interface and call its callbacks from appropriate 
> places in Kafka Consumer.
> 3. Add unit tests for interceptor changes
> 4. Add integration test for both mutable consumer and producer interceptors 
> (running at the same time).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: some more Javadocs

[guozhangwang]

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/853

MINOR: some more Javadocs



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka KJavaDoc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/853.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #853


commit 1335f72afcbd962c303208031b64e6caae1833d9
Author: Guozhang Wang 
Date:   2016-02-03T00:56:48Z

java docs




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3129) Console Producer/Consumer Issue

[Vahid Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129449#comment-15129449
 ] 

Vahid Hashemian commented on KAFKA-3129:


As I mentioned, in the input file I have numbers from 1 to 10,000.
Sometimes when I run the producer all 10,000 of them are produced (I verify 
this by running a consumer and through GetOffsetShell).
But sometimes, messages from 1 to 9864 are produced and then it stops there. 
Usually 9864 is the magic number, but occasionally I see a different number.
Always messages from the end of the file are not produced when the issue occurs.

> Console Producer/Consumer Issue
> ---
>
> Key: KAFKA-3129
> URL: https://issues.apache.org/jira/browse/KAFKA-3129
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, producer 
>Affects Versions: 0.9.0.0
>Reporter: Vahid Hashemian
>Assignee: Neha Narkhede
> Attachments: kafka-3129.mov
>
>
> I have been running a simple test case in which I have a text file 
> {{messages.txt}} with 1,000,000 lines (lines contain numbers from 1 to 
> 1,000,000 in ascending order). I run the console consumer like this:
> {{$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test}}
> Topic {{test}} is on 1 partition with a replication factor of 1.
> Then I run the console producer like this:
> {{$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test < 
> messages.txt}}
> Then the console starts receiving the messages. And about half the times it 
> goes all the way to 1,000,000. But, in other cases, it stops short, usually 
> at 999,735.
> I tried running another console consumer on another machine and both 
> consumers behave the same way. I can't see anything related to this in the 
> logs.
> I also ran the same experiment with a similar file of 10,000 lines, and am 
> getting a similar behavior. When the consumer does not receive all the 10,000 
> messages it usually stops at 9,864.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3195) Transient test failure in OffsetCheckpointTest.testReadWrite

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129438#comment-15129438
 ] 

Ismael Juma commented on KAFKA-3195:


Sure.

> Transient test failure in OffsetCheckpointTest.testReadWrite
> 
>
> Key: KAFKA-3195
> URL: https://issues.apache.org/jira/browse/KAFKA-3195
> Project: Kafka
>  Issue Type: Bug
>  Components: kafka streams
>Reporter: Ewen Cheslack-Postava
>Assignee: Guozhang Wang
>
> It looks like its probably an issue with parallel tests trying to access the 
> same fixed path, where one test deletes the file. Saw this on 
> 86a9036a7b03c8ae07d014c25a5eedc315544139.
> {quote}
> org.apache.kafka.streams.state.internals.OffsetCheckpointTest > testReadWrite 
> FAILED
> java.io.FileNotFoundException: 
> /tmp/kafka-streams/offset_checkpoint.test.tmp (No such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:171)
> at 
> org.apache.kafka.streams.state.internals.OffsetCheckpoint.write(OffsetCheckpoint.java:68)
> at 
> org.apache.kafka.streams.state.internals.OffsetCheckpointTest.testReadWrite(OffsetCheckpointTest.java:48)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-3195) Transient test failure in OffsetCheckpointTest.testReadWrite

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reassigned KAFKA-3195:
--

Assignee: Ismael Juma  (was: Guozhang Wang)

> Transient test failure in OffsetCheckpointTest.testReadWrite
> 
>
> Key: KAFKA-3195
> URL: https://issues.apache.org/jira/browse/KAFKA-3195
> Project: Kafka
>  Issue Type: Bug
>  Components: kafka streams
>Reporter: Ewen Cheslack-Postava
>Assignee: Ismael Juma
>
> It looks like its probably an issue with parallel tests trying to access the 
> same fixed path, where one test deletes the file. Saw this on 
> 86a9036a7b03c8ae07d014c25a5eedc315544139.
> {quote}
> org.apache.kafka.streams.state.internals.OffsetCheckpointTest > testReadWrite 
> FAILED
> java.io.FileNotFoundException: 
> /tmp/kafka-streams/offset_checkpoint.test.tmp (No such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:171)
> at 
> org.apache.kafka.streams.state.internals.OffsetCheckpoint.write(OffsetCheckpoint.java:68)
> at 
> org.apache.kafka.streams.state.internals.OffsetCheckpointTest.testReadWrite(OffsetCheckpointTest.java:48)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3129) Console Producer/Consumer Issue

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129427#comment-15129427
 ] 

Jason Gustafson commented on KAFKA-3129:


Are the missing messages always the last ones written? That definitely would 
suggest that the application is shutting down before all data gets pushed. I'll 
see if I can reproduce this locally.

> Console Producer/Consumer Issue
> ---
>
> Key: KAFKA-3129
> URL: https://issues.apache.org/jira/browse/KAFKA-3129
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, producer 
>Affects Versions: 0.9.0.0
>Reporter: Vahid Hashemian
>Assignee: Neha Narkhede
> Attachments: kafka-3129.mov
>
>
> I have been running a simple test case in which I have a text file 
> {{messages.txt}} with 1,000,000 lines (lines contain numbers from 1 to 
> 1,000,000 in ascending order). I run the console consumer like this:
> {{$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test}}
> Topic {{test}} is on 1 partition with a replication factor of 1.
> Then I run the console producer like this:
> {{$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test < 
> messages.txt}}
> Then the console starts receiving the messages. And about half the times it 
> goes all the way to 1,000,000. But, in other cases, it stops short, usually 
> at 999,735.
> I tried running another console consumer on another machine and both 
> consumers behave the same way. I can't see anything related to this in the 
> logs.
> I also ran the same experiment with a similar file of 10,000 lines, and am 
> getting a similar behavior. When the consumer does not receive all the 10,000 
> messages it usually stops at 9,864.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3195) Transient test failure in OffsetCheckpointTest.testReadWrite

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129426#comment-15129426
 ] 

Guozhang Wang commented on KAFKA-3195:
--

Thanks [~ewencp]. [~ijuma] Do you want to provide a PR? Then I can help merging 
it otherwise I need to ping Ewen again :)

> Transient test failure in OffsetCheckpointTest.testReadWrite
> 
>
> Key: KAFKA-3195
> URL: https://issues.apache.org/jira/browse/KAFKA-3195
> Project: Kafka
>  Issue Type: Bug
>  Components: kafka streams
>Reporter: Ewen Cheslack-Postava
>Assignee: Guozhang Wang
>
> It looks like its probably an issue with parallel tests trying to access the 
> same fixed path, where one test deletes the file. Saw this on 
> 86a9036a7b03c8ae07d014c25a5eedc315544139.
> {quote}
> org.apache.kafka.streams.state.internals.OffsetCheckpointTest > testReadWrite 
> FAILED
> java.io.FileNotFoundException: 
> /tmp/kafka-streams/offset_checkpoint.test.tmp (No such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:171)
> at 
> org.apache.kafka.streams.state.internals.OffsetCheckpoint.write(OffsetCheckpoint.java:68)
> at 
> org.apache.kafka.streams.state.internals.OffsetCheckpointTest.testReadWrite(OffsetCheckpointTest.java:48)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3129) Console Producer/Consumer Issue

[Vahid Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129419#comment-15129419
 ] 

Vahid Hashemian commented on KAFKA-3129:


Thanks [~hachikuji] for the suggestion. I tried it as you described but still 
saw the issue occur.

> Console Producer/Consumer Issue
> ---
>
> Key: KAFKA-3129
> URL: https://issues.apache.org/jira/browse/KAFKA-3129
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, producer 
>Affects Versions: 0.9.0.0
>Reporter: Vahid Hashemian
>Assignee: Neha Narkhede
> Attachments: kafka-3129.mov
>
>
> I have been running a simple test case in which I have a text file 
> {{messages.txt}} with 1,000,000 lines (lines contain numbers from 1 to 
> 1,000,000 in ascending order). I run the console consumer like this:
> {{$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test}}
> Topic {{test}} is on 1 partition with a replication factor of 1.
> Then I run the console producer like this:
> {{$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test < 
> messages.txt}}
> Then the console starts receiving the messages. And about half the times it 
> goes all the way to 1,000,000. But, in other cases, it stops short, usually 
> at 999,735.
> I tried running another console consumer on another machine and both 
> consumers behave the same way. I can't see anything related to this in the 
> logs.
> I also ran the same experiment with a similar file of 10,000 lines, and am 
> getting a similar behavior. When the consumer does not receive all the 10,000 
> messages it usually stops at 9,864.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-3178) Expose a method in AdminUtils to manually truncate a specific partition to a particular offset

[Vahid Hashemian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian reassigned KAFKA-3178:
--

Assignee: Vahid Hashemian

> Expose a method in AdminUtils to manually truncate a specific partition to a 
> particular offset
> --
>
> Key: KAFKA-3178
> URL: https://issues.apache.org/jira/browse/KAFKA-3178
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Bill Warshaw
>Assignee: Vahid Hashemian
>  Labels: kafka
>
> h3. Description
> One of Kafka's officially-described use cases is a distributed commit log 
> (http://kafka.apache.org/documentation.html#uses_commitlog).  In this case, 
> for a distributed service that needed a commit log, there would be a topic 
> with a single partition to guarantee log order.  This service would use the 
> commit log to re-sync failed nodes.  Kafka is generally an excellent fit for 
> such a system, but it does not expose an adequate mechanism for log cleanup 
> in such a case.  The built-in log cleanup mechanisms are based on time / size 
> thresholds, which doesn't work well with a commit log; data can only be 
> deleted from a commit log when the client application determines that it is 
> no longer needed.  Here we propose a new API exposed to clients through 
> AdminUtils that will delete all messages before a certain offset from a 
> specific partition.
> h3. Rejected Alternatives
> - Manually setting / resetting time intervals for log retention configs to 
> periodically flush messages from the logs from before a certain time period.  
> Doing this involves several asynchronous processes, none of which provide any 
> hooks to know when they are actually complete.
> - Rolling a new topic each time we want to cleanup the log.  This is the best 
> existing approach, but is not ideal.  All incoming writes would be paused 
> while waiting for a new topic to be created.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: Increment ducktape dependency

[granders]

GitHub user granders opened a pull request:

https://github.com/apache/kafka/pull/852

MINOR: Increment ducktape dependency

Pin kafka system tests to a newer version of ducktape.

Ran in branch builder; only one preexisting (transient) failure:

http://confluent-kafka-branch-builder-system-test-results.s3-us-west-2.amazonaws.com/2016-02-01--001.1454333721--confluentinc--increment-ducktape-dependency--a40f474/report.html

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/confluentinc/kafka 
increment-ducktape-dependency

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/852.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #852


commit 49ddd6a038bf12fb0eee453b05d86d0f60c25519
Author: Geoff Anderson 
Date:   2016-01-31T23:53:34Z

For system tests require ducktape 0.3.9

commit a40f474dd35ad89f5826c9a28681c5c7e8a5724c
Author: Geoff Anderson 
Date:   2016-02-01T07:13:58Z

Skip ducktape 0.3.9




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3195) Transient test failure in OffsetCheckpointTest.testReadWrite

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129387#comment-15129387
 ] 

Ismael Juma commented on KAFKA-3195:


There is:

File f = new File("/tmp/kafka-streams/offset_checkpoint.test");

The test should probably use `TestUtils.tempFile`.

> Transient test failure in OffsetCheckpointTest.testReadWrite
> 
>
> Key: KAFKA-3195
> URL: https://issues.apache.org/jira/browse/KAFKA-3195
> Project: Kafka
>  Issue Type: Bug
>  Components: kafka streams
>Reporter: Ewen Cheslack-Postava
>Assignee: Guozhang Wang
>
> It looks like its probably an issue with parallel tests trying to access the 
> same fixed path, where one test deletes the file. Saw this on 
> 86a9036a7b03c8ae07d014c25a5eedc315544139.
> {quote}
> org.apache.kafka.streams.state.internals.OffsetCheckpointTest > testReadWrite 
> FAILED
> java.io.FileNotFoundException: 
> /tmp/kafka-streams/offset_checkpoint.test.tmp (No such file or directory)
> at java.io.FileOutputStream.open(Native Method)
> at java.io.FileOutputStream.(FileOutputStream.java:221)
> at java.io.FileOutputStream.(FileOutputStream.java:171)
> at 
> org.apache.kafka.streams.state.internals.OffsetCheckpoint.write(OffsetCheckpoint.java:68)
> at 
> org.apache.kafka.streams.state.internals.OffsetCheckpointTest.testReadWrite(OffsetCheckpointTest.java:48)
> {quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3129) Console Producer/Consumer Issue

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129385#comment-15129385
 ] 

Jason Gustafson commented on KAFKA-3129:


Ah, my mistake. It actually is being closed in a shutdown hook. It still might 
be helpful to test using sync mode though.

> Console Producer/Consumer Issue
> ---
>
> Key: KAFKA-3129
> URL: https://issues.apache.org/jira/browse/KAFKA-3129
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, producer 
>Affects Versions: 0.9.0.0
>Reporter: Vahid Hashemian
>Assignee: Neha Narkhede
> Attachments: kafka-3129.mov
>
>
> I have been running a simple test case in which I have a text file 
> {{messages.txt}} with 1,000,000 lines (lines contain numbers from 1 to 
> 1,000,000 in ascending order). I run the console consumer like this:
> {{$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test}}
> Topic {{test}} is on 1 partition with a replication factor of 1.
> Then I run the console producer like this:
> {{$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test < 
> messages.txt}}
> Then the console starts receiving the messages. And about half the times it 
> goes all the way to 1,000,000. But, in other cases, it stops short, usually 
> at 999,735.
> I tried running another console consumer on another machine and both 
> consumers behave the same way. I can't see anything related to this in the 
> logs.
> I also ran the same experiment with a similar file of 10,000 lines, and am 
> getting a similar behavior. When the consumer does not receive all the 10,000 
> messages it usually stops at 9,864.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3129) Console Producer/Consumer Issue

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129382#comment-15129382
 ] 

Jason Gustafson commented on KAFKA-3129:


[~vahid] Are you using sync or async mode (I think async is the default)? I 
noticed that the console producer doesn't close the producer when it finishes, 
so it seems possible that some messages haven't been sent at the time the 
application shuts down. If you're not using sync already, you can enable it by 
adding '--sync' to the console producer's arguments.

> Console Producer/Consumer Issue
> ---
>
> Key: KAFKA-3129
> URL: https://issues.apache.org/jira/browse/KAFKA-3129
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, producer 
>Affects Versions: 0.9.0.0
>Reporter: Vahid Hashemian
>Assignee: Neha Narkhede
> Attachments: kafka-3129.mov
>
>
> I have been running a simple test case in which I have a text file 
> {{messages.txt}} with 1,000,000 lines (lines contain numbers from 1 to 
> 1,000,000 in ascending order). I run the console consumer like this:
> {{$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test}}
> Topic {{test}} is on 1 partition with a replication factor of 1.
> Then I run the console producer like this:
> {{$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test < 
> messages.txt}}
> Then the console starts receiving the messages. And about half the times it 
> goes all the way to 1,000,000. But, in other cases, it stops short, usually 
> at 999,735.
> I tried running another console consumer on another machine and both 
> consumers behave the same way. I can't see anything related to this in the 
> logs.
> I also ran the same experiment with a similar file of 10,000 lines, and am 
> getting a similar behavior. When the consumer does not receive all the 10,000 
> messages it usually stops at 9,864.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3174) Re-evaluate the CRC32 class performance.

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3174:
---
Fix Version/s: (was: 0.9.0.1)
   0.9.1.0

> Re-evaluate the CRC32 class performance.
> 
>
> Key: KAFKA-3174
> URL: https://issues.apache.org/jira/browse/KAFKA-3174
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.9.1.0
>
>
> We used org.apache.kafka.common.utils.CRC32 in clients because it has better 
> performance than java.util.zip.CRC32 in Java 1.6.
> In a recent test I ran it looks in Java 1.8 the CRC32 class is 2x as fast as 
> the Crc32 class we are using now. We may want to re-evaluate the performance 
> of Crc32 class and see it makes sense to simply use java CRC32 instead.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3195) Transient test failure in OffsetCheckpointTest.testReadWrite

[Ewen Cheslack-Postava (JIRA)]

Ewen Cheslack-Postava created KAFKA-3195:


 Summary: Transient test failure in 
OffsetCheckpointTest.testReadWrite
 Key: KAFKA-3195
 URL: https://issues.apache.org/jira/browse/KAFKA-3195
 Project: Kafka
  Issue Type: Bug
  Components: kafka streams
Reporter: Ewen Cheslack-Postava
Assignee: Guozhang Wang


It looks like its probably an issue with parallel tests trying to access the 
same fixed path, where one test deletes the file. Saw this on 
86a9036a7b03c8ae07d014c25a5eedc315544139.

{quote}
org.apache.kafka.streams.state.internals.OffsetCheckpointTest > testReadWrite 
FAILED
java.io.FileNotFoundException: 
/tmp/kafka-streams/offset_checkpoint.test.tmp (No such file or directory)
at java.io.FileOutputStream.open(Native Method)
at java.io.FileOutputStream.(FileOutputStream.java:221)
at java.io.FileOutputStream.(FileOutputStream.java:171)
at 
org.apache.kafka.streams.state.internals.OffsetCheckpoint.write(OffsetCheckpoint.java:68)
at 
org.apache.kafka.streams.state.internals.OffsetCheckpointTest.testReadWrite(OffsetCheckpointTest.java:48)
{quote}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3177) Kafka consumer can hang when position() is called on a non-existing partition.

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129355#comment-15129355
 ] 

Jason Gustafson commented on KAFKA-3177:


[~guozhang] I think it's useful to separate failures into two categories: 1) 
ephemeral failures which should eventually resolve themselves, and 2) permanent 
failures. Blocking seems justifiable (even if undesirable) for ephemeral 
failures, but we ought to raise an exception if we don't expect the error to 
resolve "soon." I'd classify this type of failure as permanent since we can't 
really depend on the number of partitions being increased. And it almost 
certainly indicates a bug in the user's code if they're trying to assign 
partitions which don't exist. I think we'll have to handle cases like this 
individually as they come up.

> Kafka consumer can hang when position() is called on a non-existing partition.
> --
>
> Key: KAFKA-3177
> URL: https://issues.apache.org/jira/browse/KAFKA-3177
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Jiangjie Qin
> Fix For: 0.9.0.1
>
>
> This can be easily reproduced as following:
> {code}
> {
> ...
> consumer.assign(SomeNonExsitingTopicParition);
> consumer.position();
> ...
> }
> {code}
> It seems when position is called we will try to do the following:
> 1. Fetch committed offsets.
> 2. If there is no committed offsets, try to reset offset using reset 
> strategy. in sendListOffsetRequest(), if the consumer does not know the 
> TopicPartition, it will refresh its metadata and retry. In this case, because 
> the partition does not exist, we fall in to the infinite loop of refreshing 
> topic metadata.
> Another orthogonal issue is that if the topic in the above code piece does 
> not exist, position() call will actually create the topic due to the fact 
> that currently topic metadata request could automatically create the topic. 
> This is a known separate issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: 0.9.0

[stumped2]

Github user stumped2 closed the pull request at:

https://github.com/apache/kafka/pull/848


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-1545) java.net.InetAddress.getLocalHost in KafkaHealthcheck.register may fail on some irregular hostnames

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129303#comment-15129303
 ] 

ASF GitHub Bot commented on KAFKA-1545:
---

Github user stumped2 closed the pull request at:

https://github.com/apache/kafka/pull/48


> java.net.InetAddress.getLocalHost in KafkaHealthcheck.register may fail on 
> some irregular hostnames
> ---
>
> Key: KAFKA-1545
> URL: https://issues.apache.org/jira/browse/KAFKA-1545
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.1.1
>Reporter: Guozhang Wang
>Assignee: Rekha Joshi
>  Labels: newbie
> Fix For: 0.10.0.0
>
>
> For example:
> kafka.server.LogOffsetTest > testGetOffsetsForUnknownTopic FAILED
> java.net.UnknownHostException: guwang-mn2: guwang-mn2: nodename nor 
> servname provided, or not known
> at java.net.InetAddress.getLocalHost(InetAddress.java:1473)
> at kafka.server.KafkaHealthcheck.register(KafkaHealthcheck.scala:59)
> at kafka.server.KafkaHealthcheck.startup(KafkaHealthcheck.scala:45)
> at kafka.server.KafkaServer.startup(KafkaServer.scala:121)
> at kafka.utils.TestUtils$.createServer(TestUtils.scala:130)
> at kafka.server.LogOffsetTest.setUp(LogOffsetTest.scala:53)
> Caused by:
> java.net.UnknownHostException: guwang-mn2: nodename nor servname 
> provided, or not known
> at java.net.Inet6AddressImpl.lookupAllHostAddr(Native Method)
> at java.net.InetAddress$1.lookupAllHostAddr(InetAddress.java:901)
> at 
> java.net.InetAddress.getAddressesFromNameService(InetAddress.java:1293)
> at java.net.InetAddress.getLocalHost(InetAddress.java:1469)
> ... 5 more



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-1545: KafkaHealthcheck.register failure

[stumped2]

Github user stumped2 closed the pull request at:

https://github.com/apache/kafka/pull/48


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: 0.8.2

[stumped2]

Github user stumped2 closed the pull request at:

https://github.com/apache/kafka/pull/796


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-1695) Authenticate connection to Zookeeper

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129296#comment-15129296
 ] 

ASF GitHub Bot commented on KAFKA-1695:
---

Github user stumped2 closed the pull request at:

https://github.com/apache/kafka/pull/93


> Authenticate connection to Zookeeper
> 
>
> Key: KAFKA-1695
> URL: https://issues.apache.org/jira/browse/KAFKA-1695
> Project: Kafka
>  Issue Type: Sub-task
>  Components: security
>Reporter: Jay Kreps
>Assignee: Parth Brahmbhatt
> Fix For: 0.9.0.0
>
>
> We need to make it possible to secure the Zookeeper cluster Kafka is using. 
> This would make use of the normal authentication ZooKeeper provides. 
> ZooKeeper supports a variety of authentication mechanisms so we will need to 
> figure out what has to be passed in to the zookeeper client.
> The intention is that when the current round of client work is done it should 
> be possible to run without clients needing access to Zookeeper so all we need 
> here is to make it so that only the Kafka cluster is able to read and write 
> to the Kafka znodes  (we shouldn't need to set any kind of acl on a per-znode 
> basis).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2356) Support retrieving partitions of ConsumerRecords

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129299#comment-15129299
 ] 

ASF GitHub Bot commented on KAFKA-2356:
---

Github user stumped2 closed the pull request at:

https://github.com/apache/kafka/pull/95


> Support retrieving partitions of ConsumerRecords
> 
>
> Key: KAFKA-2356
> URL: https://issues.apache.org/jira/browse/KAFKA-2356
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Stevo Slavic
>Assignee: Stevo Slavic
>Priority: Trivial
>  Labels: newbie
>
> In new consumer on trunk, ConsumerRecords has method to retrieve records for 
> given TopicPartition, but there is no method to retrieve TopicPartition's 
> included/available in ConsumerRecords. Please have it supported.
> Method could be something like:
> {noformat}
> /**
>  * Get partitions of records returned by a {@link Consumer#poll(long)} 
> operation
> */
> public Set partitions() {
> return Collections.unmodifiableSet(this.records.keySet());
> }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: MINOR: MockClient's disconnect() method has tw...

[stumped2]

Github user stumped2 closed the pull request at:

https://github.com/apache/kafka/pull/183


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-1369) snappy version update 1.1.x

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129293#comment-15129293
 ] 

ASF GitHub Bot commented on KAFKA-1369:
---

Github user stumped2 closed the pull request at:

https://github.com/apache/kafka/pull/30


> snappy version update 1.1.x
> ---
>
> Key: KAFKA-1369
> URL: https://issues.apache.org/jira/browse/KAFKA-1369
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.0, 0.8.1.1
> Environment: Red Hat Enterprise Linux Server release 5.8 (Tikanga)
> - x64 
>Reporter: thinker0
>Assignee: Roger Hoover
>Priority: Minor
> Fix For: 0.8.2.0
>
> Attachments: patch.diff
>
>
> https://github.com/xerial/snappy-java/issues/38 issue
> snappy version 1.1.x
> {code}
> org.xerial.snappy.SnappyError: [FAILED_TO_LOAD_NATIVE_LIBRARY] null
> at org.xerial.snappy.SnappyLoader.load(SnappyLoader.java:239)
> at org.xerial.snappy.Snappy.(Snappy.java:48)
> at 
> org.xerial.snappy.SnappyInputStream.hasNextChunk(SnappyInputStream.java:351)
> at org.xerial.snappy.SnappyInputStream.rawRead(SnappyInputStream.java:159)
> at org.xerial.snappy.SnappyInputStream.read(SnappyInputStream.java:142)
> at java.io.InputStream.read(InputStream.java:101)
> at 
> kafka.message.ByteBufferMessageSet$$anonfun$decompress$1.apply$mcI$sp(ByteBufferMessageSet.scala:68)
> at 
> kafka.message.ByteBufferMessageSet$$anonfun$decompress$1.apply(ByteBufferMessageSet.scala:68)
> at 
> kafka.message.ByteBufferMessageSet$$anonfun$decompress$1.apply(ByteBufferMessageSet.scala:68)
> at scala.collection.immutable.Stream$.continually(Stream.scala:1129)
> at 
> kafka.message.ByteBufferMessageSet$.decompress(ByteBufferMessageSet.scala:68)
> at 
> kafka.message.ByteBufferMessageSet$$anon$1.makeNextOuter(ByteBufferMessageSet.scala:178)
> at 
> kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:191)
> at 
> kafka.message.ByteBufferMessageSet$$anon$1.makeNext(ByteBufferMessageSet.scala:145)
> at 
> kafka.utils.IteratorTemplate.maybeComputeNext(IteratorTemplate.scala:66)
> at kafka.utils.IteratorTemplate.hasNext(IteratorTemplate.scala:58)
> {code}
> {code}
> /tmp] ldd snappy-1.0.5-libsnappyjava.so
> ./snappy-1.0.5-libsnappyjava.so: /usr/lib64/libstdc++.so.6: version 
> `GLIBCXX_3.4.9' not found (required by ./snappy-1.0.5-libsnappyjava.so)
> ./snappy-1.0.5-libsnappyjava.so: /usr/lib64/libstdc++.so.6: version 
> `GLIBCXX_3.4.11' not found (required by ./snappy-1.0.5-libsnappyjava.so)
>   linux-vdso.so.1 =>  (0x7fff81dfc000)
>   libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x2b554b43)
>   libm.so.6 => /lib64/libm.so.6 (0x2b554b731000)
>   libc.so.6 => /lib64/libc.so.6 (0x2b554b9b4000)
>   libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x2b554bd0c000)
>   /lib64/ld-linux-x86-64.so.2 (0x0033e2a0)
> {code}
> {code}
> /tmp] ldd snappy-1.1.1M1-be6ba593-9ac7-488e-953e-ba5fd9530ee1-libsnappyjava.so
> ldd: warning: you do not have execution permission for 
> `./snappy-1.1.1M1-be6ba593-9ac7-488e-953e-ba5fd9530ee1-libsnappyjava.so'
>   linux-vdso.so.1 =>  (0x7fff1c132000)
>   libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x2b9548319000)
>   libm.so.6 => /lib64/libm.so.6 (0x2b954861a000)
>   libc.so.6 => /lib64/libc.so.6 (0x2b954889d000)
>   libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x2b9548bf5000)
>   /lib64/ld-linux-x86-64.so.2 (0x0033e2a0)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-1695: Adding zookeeper authentication. Z...

[stumped2]

Github user stumped2 closed the pull request at:

https://github.com/apache/kafka/pull/93


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-1344) Kafka-console-producer.sh support snappy compression

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129292#comment-15129292
 ] 

ASF GitHub Bot commented on KAFKA-1344:
---

Github user stumped2 closed the pull request at:

https://github.com/apache/kafka/pull/20


> Kafka-console-producer.sh support snappy compression
> 
>
> Key: KAFKA-1344
> URL: https://issues.apache.org/jira/browse/KAFKA-1344
> Project: Kafka
>  Issue Type: Improvement
>Reporter: aio
>Assignee: Ivan Lyutov
>  Labels: tools, usability
> Fix For: 0.8.2.0
>
> Attachments: KAFKA-1344_2014-03-31_21:13:57.patch, 
> KAFKA-1344_2014-04-01_19:50:43.patch, KAFKA-1344_2014-04-02_20:31:27.patch
>
>
> Wish kafka-console-producer.sh support snappy compression.
> Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-2356 Added support for retrieving partit...

[stumped2]

Github user stumped2 closed the pull request at:

https://github.com/apache/kafka/pull/95


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-1834) No Response when handle LeaderAndIsrRequest some case

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1834?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129295#comment-15129295
 ] 

ASF GitHub Bot commented on KAFKA-1834:
---

Github user stumped2 closed the pull request at:

https://github.com/apache/kafka/pull/40


> No Response when handle LeaderAndIsrRequest some case
> -
>
> Key: KAFKA-1834
> URL: https://issues.apache.org/jira/browse/KAFKA-1834
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.0
>Reporter: xiajun
>  Labels: easyfix
> Attachments: KAFKA-1834.patch
>
>
> When a replica become leader or follower, if this broker no exist in assigned 
> replicas, there are no response.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-1369 snappy-1.1.x update (RHEL4,5,6)

[stumped2]

Github user stumped2 closed the pull request at:

https://github.com/apache/kafka/pull/30


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: KAFKA-1834: No Response when handle LeaderAndI...

[stumped2]

Github user stumped2 closed the pull request at:

https://github.com/apache/kafka/pull/40


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: [KAFKA-1344] - Added compression codec option ...

[stumped2]

Github user stumped2 closed the pull request at:

https://github.com/apache/kafka/pull/20


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request: Switch to using scala 2.9.2

[stumped2]

Github user stumped2 closed the pull request at:

https://github.com/apache/kafka/pull/1


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-826) Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129290#comment-15129290
 ] 

ASF GitHub Bot commented on KAFKA-826:
--

Github user stumped2 closed the pull request at:

https://github.com/apache/kafka/pull/3


> Make Kafka 0.8 depend on metrics 2.2.0 instead of 3.x
> -
>
> Key: KAFKA-826
> URL: https://issues.apache.org/jira/browse/KAFKA-826
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.0
>Reporter: Neha Narkhede
>Assignee: Dragos Manolescu
>Priority: Blocker
>  Labels: build, kafka-0.8, metrics
> Fix For: 0.8.0
>
> Attachments: kafka-fix-for-826-complete.patch, 
> kafka-fix-for-826-take2.patch, kafka-fix-for-826.patch
>
>
> In order to mavenize Kafka 0.8, we have to depend on metrics 2.2.0 since 
> metrics 3.x is a huge change as well as not an officially supported release.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: zkclient and scalatest library updates

[stumped2]

Github user stumped2 closed the pull request at:

https://github.com/apache/kafka/pull/3


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3159) Kafka consumer 0.9.0.0 client poll is very CPU intensive under certain conditions

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129287#comment-15129287
 ] 

Jason Gustafson commented on KAFKA-3159:


[~ra...@signalfx.com] I think we'll probably need some more information to 
investigate this further. It appears that the EOFExceptions were only symptoms 
of some other problem which is causing high CPU utilization. It might be 
helpful to see some of the logs so we know what the consumer was doing during 
that time. Can you turn on TRACE level logging and attach a sample to this 
ticket?

> Kafka consumer 0.9.0.0  client poll is very CPU intensive under certain 
> conditions
> --
>
> Key: KAFKA-3159
> URL: https://issues.apache.org/jira/browse/KAFKA-3159
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0
> Environment: Linux, Oracle JVM 8.
>Reporter: Rajiv Kurian
>Assignee: Jason Gustafson
> Fix For: 0.9.0.1
>
> Attachments: Memory-profile-patched-client.png, Screen Shot 
> 2016-02-01 at 11.09.32 AM.png
>
>
> We are using the new kafka consumer with the following config (as logged by 
> kafka)
> metric.reporters = []
> metadata.max.age.ms = 30
> value.deserializer = class 
> org.apache.kafka.common.serialization.ByteArrayDeserializer
> group.id = myGroup.id
> partition.assignment.strategy = 
> [org.apache.kafka.clients.consumer.RangeAssignor]
> reconnect.backoff.ms = 50
> sasl.kerberos.ticket.renew.window.factor = 0.8
> max.partition.fetch.bytes = 2097152
> bootstrap.servers = [myBrokerList]
> retry.backoff.ms = 100
> sasl.kerberos.kinit.cmd = /usr/bin/kinit
> sasl.kerberos.service.name = null
> sasl.kerberos.ticket.renew.jitter = 0.05
> ssl.keystore.type = JKS
> ssl.trustmanager.algorithm = PKIX
> enable.auto.commit = false
> ssl.key.password = null
> fetch.max.wait.ms = 1000
> sasl.kerberos.min.time.before.relogin = 6
> connections.max.idle.ms = 54
> ssl.truststore.password = null
> session.timeout.ms = 3
> metrics.num.samples = 2
> client.id = 
> ssl.endpoint.identification.algorithm = null
> key.deserializer = class sf.kafka.VoidDeserializer
> ssl.protocol = TLS
> check.crcs = true
> request.timeout.ms = 4
> ssl.provider = null
> ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
> ssl.keystore.location = null
> heartbeat.interval.ms = 3000
> auto.commit.interval.ms = 5000
> receive.buffer.bytes = 32768
> ssl.cipher.suites = null
> ssl.truststore.type = JKS
> security.protocol = PLAINTEXT
> ssl.truststore.location = null
> ssl.keystore.password = null
> ssl.keymanager.algorithm = SunX509
> metrics.sample.window.ms = 3
> fetch.min.bytes = 512
> send.buffer.bytes = 131072
> auto.offset.reset = earliest
> We use the consumer.assign() feature to assign a list of partitions and call 
> poll in a loop.  We have the following setup:
> 1. The messages have no key and we use the byte array deserializer to get 
> byte arrays from the config.
> 2. The messages themselves are on an average about 75 bytes. We get this 
> number by dividing the Kafka broker bytes-in metric by the messages-in metric.
> 3. Each consumer is assigned about 64 partitions of the same topic spread 
> across three brokers.
> 4. We get very few messages per second maybe around 1-2 messages across all 
> partitions on a client right now.
> 5. We have no compression on the topic.
> Our run loop looks something like this
> while (isRunning()) {
> ConsumerRecords records = null;
> try {
> // Here timeout is about 10 seconds, so it is pretty big.
> records = consumer.poll(timeout);
> } catch (Exception e) {
>// This never hits for us
> logger.error("Exception polling Kafka ", e);
> records = null;
> }
> if (records != null) {
> for (ConsumerRecord record : records) {
>// The handler puts the byte array on a very fast ring buffer 
> so it barely takes any time.
> handler.handleMessage(ByteBuffer.wrap(record.value()));
> }
> }
> }
> With this setup our performance has taken a horrendous hit as soon as we 
> started this one thread that just polls Kafka in a loop.
> I profiled the application using Java Mission Control and have a few insights.
> 1. There doesn't seem to be a single hotspot. The consumer just ends up using 
> a lot of CPU for handi

[jira] [Updated] (KAFKA-3194) Validate security.inter.broker.protocol against the advertised.listeners protocols

[Grant Henke (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Grant Henke updated KAFKA-3194:
---
Status: Patch Available  (was: Open)

> Validate security.inter.broker.protocol against the advertised.listeners 
> protocols
> --
>
> Key: KAFKA-3194
> URL: https://issues.apache.org/jira/browse/KAFKA-3194
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.9.0.1
>
>
> When testing Kafka I found that Kafka can run in a very unhealthy state due 
> to a misconfigured security.inter.broker.protocol. There are errors in the 
> log such (shown below) but it would be better to prevent startup with a clear 
> error message in this scenario.
> Sample error in the server logs:
> {code}
> ERROR kafka.controller.ReplicaStateMachine$BrokerChangeListener: 
> [BrokerChangeListener on Controller 71]: Error while handling broker changes
> kafka.common.BrokerEndPointNotAvailableException: End point PLAINTEXT not 
> found for broker 69
>   at kafka.cluster.Broker.getBrokerEndPoint(Broker.scala:141)
>   at 
> kafka.controller.ControllerChannelManager.kafka$controller$ControllerChannelManager$$addNewBroker(ControllerChannelManager.scala:88)
>   at 
> kafka.controller.ControllerChannelManager.addBroker(ControllerChannelManager.scala:73)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$4.apply(ReplicaStateMachine.scala:372)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$4.apply(ReplicaStateMachine.scala:372)
>   at scala.collection.immutable.Set$Set1.foreach(Set.scala:79)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ReplicaStateMachine.scala:372)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:359)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:359)
>   at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:358)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:357)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:357)
>   at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:259)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildChange(ReplicaStateMachine.scala:356)
>   at org.I0Itec.zkclient.ZkClient$10.run(ZkClient.java:842)
>   at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3194) Validate security.inter.broker.protocol against the advertised.listeners protocols

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129225#comment-15129225
 ] 

ASF GitHub Bot commented on KAFKA-3194:
---

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/851

KAFKA-3194: Validate security.inter.broker.protocol against the adver…

…tised.listeners protocols

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka verify-protocol

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/851.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #851


commit 918a391a2fc3cf825bfbad73ae3ab923a307f4e4
Author: Grant Henke 
Date:   2016-02-02T22:38:58Z

KAFKA-3194: Validate security.inter.broker.protocol against the 
advertised.listeners protocols




> Validate security.inter.broker.protocol against the advertised.listeners 
> protocols
> --
>
> Key: KAFKA-3194
> URL: https://issues.apache.org/jira/browse/KAFKA-3194
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.9.0.0
>Reporter: Grant Henke
>Assignee: Grant Henke
> Fix For: 0.9.0.1
>
>
> When testing Kafka I found that Kafka can run in a very unhealthy state due 
> to a misconfigured security.inter.broker.protocol. There are errors in the 
> log such (shown below) but it would be better to prevent startup with a clear 
> error message in this scenario.
> Sample error in the server logs:
> {code}
> ERROR kafka.controller.ReplicaStateMachine$BrokerChangeListener: 
> [BrokerChangeListener on Controller 71]: Error while handling broker changes
> kafka.common.BrokerEndPointNotAvailableException: End point PLAINTEXT not 
> found for broker 69
>   at kafka.cluster.Broker.getBrokerEndPoint(Broker.scala:141)
>   at 
> kafka.controller.ControllerChannelManager.kafka$controller$ControllerChannelManager$$addNewBroker(ControllerChannelManager.scala:88)
>   at 
> kafka.controller.ControllerChannelManager.addBroker(ControllerChannelManager.scala:73)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$4.apply(ReplicaStateMachine.scala:372)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$4.apply(ReplicaStateMachine.scala:372)
>   at scala.collection.immutable.Set$Set1.foreach(Set.scala:79)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ReplicaStateMachine.scala:372)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:359)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:359)
>   at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:358)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:357)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:357)
>   at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:259)
>   at 
> kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildChange(ReplicaStateMachine.scala:356)
>   at org.I0Itec.zkclient.ZkClient$10.run(ZkClient.java:842)
>   at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3194: Validate security.inter.broker.pro...

[granthenke]

GitHub user granthenke opened a pull request:

https://github.com/apache/kafka/pull/851

KAFKA-3194: Validate security.inter.broker.protocol against the adver…

…tised.listeners protocols

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/granthenke/kafka verify-protocol

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/851.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #851


commit 918a391a2fc3cf825bfbad73ae3ab923a307f4e4
Author: Grant Henke 
Date:   2016-02-02T22:38:58Z

KAFKA-3194: Validate security.inter.broker.protocol against the 
advertised.listeners protocols




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3121) KStream DSL API Improvement

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-3121:
-
Description: 
>From some collected feedbacks, here is a list of potential improvements that 
>we want to make:

1. Remove AggregateSupplier for aggregate, and provide more built-in 
aggregations. [DONE]

2. Join to return KeyValue<>.

3. "Windows" class syntax-sugers. [DONE]

4. Add print() to KTable / KStream.

5. flatMap / flatMapValues to return arrays in addition to Iterable.

6. make the API function names aligned with Java 8+, e.g. filterOut -> filterNot

7. collapse process() and transform() in KStream.

8. validate Streaming configs and allow passing properties to KafkaStreaming.  
[DONE]

9. Rename KafkaStreaming to Streams.  [DONE]

Also move some of the state package into internals, and create a new top-level 
common folder with KeyValue / etc in it.

  was:
>From some collected feedbacks, here is a list of potential improvements that 
>we want to make:

1. Remove AggregateSupplier for aggregate, and provide more built-in 
aggregations.

2. Join to return KeyValue<>.

3. "Windows" class syntax-sugers.

4. Add print() to KTable / KStream.

5. flatMap / flatMapValues to return arrays in addition to Iterable.

6. make the API function names aligned with Java 8+, e.g. filterOut -> filterNot

7. collapse process() and transform() in KStream.

8. validate Streaming configs and allow passing properties to KafkaStreaming.  
[DONE]

9. Rename KafkaStreaming to Streams.  [DONE]

Also move some of the state package into internals, and create a new top-level 
common folder with KeyValue / etc in it.


> KStream DSL API Improvement
> ---
>
> Key: KAFKA-3121
> URL: https://issues.apache.org/jira/browse/KAFKA-3121
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.1.0
>
>
> From some collected feedbacks, here is a list of potential improvements that 
> we want to make:
> 1. Remove AggregateSupplier for aggregate, and provide more built-in 
> aggregations. [DONE]
> 2. Join to return KeyValue<>.
> 3. "Windows" class syntax-sugers. [DONE]
> 4. Add print() to KTable / KStream.
> 5. flatMap / flatMapValues to return arrays in addition to Iterable.
> 6. make the API function names aligned with Java 8+, e.g. filterOut -> 
> filterNot
> 7. collapse process() and transform() in KStream.
> 8. validate Streaming configs and allow passing properties to KafkaStreaming. 
>  [DONE]
> 9. Rename KafkaStreaming to Streams.  [DONE]
> Also move some of the state package into internals, and create a new 
> top-level common folder with KeyValue / etc in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3194) Validate security.inter.broker.protocol against the advertised.listeners protocols

[Grant Henke (JIRA)]

Grant Henke created KAFKA-3194:
--

 Summary: Validate security.inter.broker.protocol against the 
advertised.listeners protocols
 Key: KAFKA-3194
 URL: https://issues.apache.org/jira/browse/KAFKA-3194
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.9.0.0
Reporter: Grant Henke
Assignee: Grant Henke
 Fix For: 0.9.0.1


When testing Kafka I found that Kafka can run in a very unhealthy state due to 
a misconfigured security.inter.broker.protocol. There are errors in the log 
such (shown below) but it would be better to prevent startup with a clear error 
message in this scenario.

Sample error in the server logs:
{code}
ERROR kafka.controller.ReplicaStateMachine$BrokerChangeListener: 
[BrokerChangeListener on Controller 71]: Error while handling broker changes
kafka.common.BrokerEndPointNotAvailableException: End point PLAINTEXT not found 
for broker 69
at kafka.cluster.Broker.getBrokerEndPoint(Broker.scala:141)
at 
kafka.controller.ControllerChannelManager.kafka$controller$ControllerChannelManager$$addNewBroker(ControllerChannelManager.scala:88)
at 
kafka.controller.ControllerChannelManager.addBroker(ControllerChannelManager.scala:73)
at 
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$4.apply(ReplicaStateMachine.scala:372)
at 
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1$$anonfun$apply$mcV$sp$4.apply(ReplicaStateMachine.scala:372)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:79)
at 
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply$mcV$sp(ReplicaStateMachine.scala:372)
at 
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:359)
at 
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1$$anonfun$apply$mcV$sp$1.apply(ReplicaStateMachine.scala:359)
at kafka.metrics.KafkaTimer.time(KafkaTimer.scala:33)
at 
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply$mcV$sp(ReplicaStateMachine.scala:358)
at 
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:357)
at 
kafka.controller.ReplicaStateMachine$BrokerChangeListener$$anonfun$handleChildChange$1.apply(ReplicaStateMachine.scala:357)
at kafka.utils.CoreUtils$.inLock(CoreUtils.scala:259)
at 
kafka.controller.ReplicaStateMachine$BrokerChangeListener.handleChildChange(ReplicaStateMachine.scala:356)
at org.I0Itec.zkclient.ZkClient$10.run(ZkClient.java:842)
at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3188) Add integration test for KIP-31 and KIP-32

[Anna Povzner (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129105#comment-15129105
 ] 

Anna Povzner commented on KAFKA-3188:
-

[~becket_qin] Do you mind making separate JIRAs for each of the three tests? It 
will be much easier to review as separate PRs and also would be easy to share 
workload if one person cannot take on all three tests. Thanks!

> Add integration test for KIP-31 and KIP-32
> --
>
> Key: KAFKA-3188
> URL: https://issues.apache.org/jira/browse/KAFKA-3188
> Project: Kafka
>  Issue Type: Sub-task
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
> Fix For: 0.10.0.0
>
>
> The integration test should cover the followings:
> 1. Compatibility test.
> 2. Upgrade test
> 3. Changing message format type on the fly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-trunk-jdk7 #1011

[Apache Jenkins Server]

See 

[jira] [Updated] (KAFKA-3129) Console Producer/Consumer Issue

[Vahid Hashemian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian updated KAFKA-3129:
---
Summary: Console Producer/Consumer Issue  (was: Potential Console 
Producer/Consumer Issue)

> Console Producer/Consumer Issue
> ---
>
> Key: KAFKA-3129
> URL: https://issues.apache.org/jira/browse/KAFKA-3129
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, producer 
>Affects Versions: 0.9.0.0
>Reporter: Vahid Hashemian
>Assignee: Neha Narkhede
> Attachments: kafka-3129.mov
>
>
> I have been running a simple test case in which I have a text file 
> {{messages.txt}} with 1,000,000 lines (lines contain numbers from 1 to 
> 1,000,000 in ascending order). I run the console consumer like this:
> {{$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test}}
> Topic {{test}} is on 1 partition with a replication factor of 1.
> Then I run the console producer like this:
> {{$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test < 
> messages.txt}}
> Then the console starts receiving the messages. And about half the times it 
> goes all the way to 1,000,000. But, in other cases, it stops short, usually 
> at 999,735.
> I tried running another console consumer on another machine and both 
> consumers behave the same way. I can't see anything related to this in the 
> logs.
> I also ran the same experiment with a similar file of 10,000 lines, and am 
> getting a similar behavior. When the consumer does not receive all the 10,000 
> messages it usually stops at 9,864.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-3129) Potential Console Producer/Consumer Issue

[Vahid Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129021#comment-15129021
 ] 

Vahid Hashemian edited comment on KAFKA-3129 at 2/2/16 9:00 PM:


Further evidence that this is a producer issue:

I ran the producer 6 times with the 1 message file, and on the 6th try I 
noticed the consumer came back with 9864 (instead of 1).
Then I ran this command:

{{bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 
localhost:9092 --topic test --time -1}}

And it returned this:

{{test:0:59864}}


was (Author: vahid):
Further evidence that this is a producer issue:

I ran the producer 6 times with the 1 message file, and on the 6th try I 
noticed the consumer came back with 9864 (instead of 1).
Then I ran this command:

{{bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 
localhost:9092 --topic test --time -1}}

And it returned this:

{{topic-p1-r1:0:59864}}

> Potential Console Producer/Consumer Issue
> -
>
> Key: KAFKA-3129
> URL: https://issues.apache.org/jira/browse/KAFKA-3129
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, producer 
>Affects Versions: 0.9.0.0
>Reporter: Vahid Hashemian
>Assignee: Neha Narkhede
> Attachments: kafka-3129.mov
>
>
> I have been running a simple test case in which I have a text file 
> {{messages.txt}} with 1,000,000 lines (lines contain numbers from 1 to 
> 1,000,000 in ascending order). I run the console consumer like this:
> {{$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test}}
> Topic {{test}} is on 1 partition with a replication factor of 1.
> Then I run the console producer like this:
> {{$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test < 
> messages.txt}}
> Then the console starts receiving the messages. And about half the times it 
> goes all the way to 1,000,000. But, in other cases, it stops short, usually 
> at 999,735.
> I tried running another console consumer on another machine and both 
> consumers behave the same way. I can't see anything related to this in the 
> logs.
> I also ran the same experiment with a similar file of 10,000 lines, and am 
> getting a similar behavior. When the consumer does not receive all the 10,000 
> messages it usually stops at 9,864.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3129) Potential Console Producer/Consumer Issue

[Vahid Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15129021#comment-15129021
 ] 

Vahid Hashemian commented on KAFKA-3129:


Further evidence that this is a producer issue:

I ran the producer 6 times with the 1 message file, and on the 6th try I 
noticed the consumer came back with 9864 (instead of 1).
Then I ran this command:

{{bin/kafka-run-class.sh kafka.tools.GetOffsetShell --broker-list 
localhost:9092 --topic test --time -1}}

And it returned this:

{{topic-p1-r1:0:59864}}

> Potential Console Producer/Consumer Issue
> -
>
> Key: KAFKA-3129
> URL: https://issues.apache.org/jira/browse/KAFKA-3129
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, producer 
>Affects Versions: 0.9.0.0
>Reporter: Vahid Hashemian
>Assignee: Neha Narkhede
> Attachments: kafka-3129.mov
>
>
> I have been running a simple test case in which I have a text file 
> {{messages.txt}} with 1,000,000 lines (lines contain numbers from 1 to 
> 1,000,000 in ascending order). I run the console consumer like this:
> {{$ bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test}}
> Topic {{test}} is on 1 partition with a replication factor of 1.
> Then I run the console producer like this:
> {{$ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test < 
> messages.txt}}
> Then the console starts receiving the messages. And about half the times it 
> goes all the way to 1,000,000. But, in other cases, it stops short, usually 
> at 999,735.
> I tried running another console consumer on another machine and both 
> consumers behave the same way. I can't see anything related to this in the 
> logs.
> I also ran the same experiment with a similar file of 10,000 lines, and am 
> getting a similar behavior. When the consumer does not receive all the 10,000 
> messages it usually stops at 9,864.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #338

[Apache Jenkins Server]

See 

Changes:

[me] KAFKA-3121: Refactor KStream Aggregate to be Lambda-able.

--
Started by an SCM change
[EnvInject] - Loading node environment variables.
Building remotely on H11 (docker Ubuntu ubuntu yahoo-not-h2) in workspace 

 > git rev-parse --is-inside-work-tree # timeout=10
Fetching changes from the remote Git repository
 > git config remote.origin.url 
 > https://git-wip-us.apache.org/repos/asf/kafka.git # timeout=10
Fetching upstream changes from https://git-wip-us.apache.org/repos/asf/kafka.git
 > git --version # timeout=10
 > git -c core.askpass=true fetch --tags --progress 
 > https://git-wip-us.apache.org/repos/asf/kafka.git 
 > +refs/heads/*:refs/remotes/origin/*
 > git rev-parse refs/remotes/origin/trunk^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/trunk^{commit} # timeout=10
Checking out Revision 95174337c2f6cda90e213e5c3a73fc89854f42a7 
(refs/remotes/origin/trunk)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 95174337c2f6cda90e213e5c3a73fc89854f42a7
 > git rev-list 8189f9d58002ec0092737741bf6c74eebab4dc73 # timeout=10
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson3514870082524780613.sh
+ 
/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2/bin/gradle
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
http://gradle.org/docs/2.4-rc-2/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:downloadWrapper

BUILD SUCCESSFUL

Total time: 40.098 secs
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
[kafka-trunk-jdk8] $ /bin/bash -xe /tmp/hudson5480334025250517057.sh
+ export GRADLE_OPTS=-Xmx1024m
+ GRADLE_OPTS=-Xmx1024m
+ ./gradlew -Dorg.gradle.project.maxParallelForks=1 clean jarAll testAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.10/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:clean UP-TO-DATE
:clients:clean UP-TO-DATE
:connect:clean UP-TO-DATE
:core:clean UP-TO-DATE
:examples:clean UP-TO-DATE
:log4j-appender:clean UP-TO-DATE
:streams:clean UP-TO-DATE
:tools:clean UP-TO-DATE
:connect:api:clean UP-TO-DATE
:connect:file:clean UP-TO-DATE
:connect:json:clean UP-TO-DATE
:connect:runtime:clean UP-TO-DATE
:streams:examples:clean UP-TO-DATE
:jar_core_2_10
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk8:clients:compileJava
:jar_core_2_10 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Failed to capture snapshot of input files for task 'compileJava' during 
up-to-date check.  See stacktrace for details.
> Could not add entry 
> '/home/jenkins/.gradle/caches/modules-2/files-2.1/net.jpountz.lz4/lz4/1.3.0/c708bb2590c0652a642236ef45d9f99ff842a2ce/lz4-1.3.0.jar'
>  to cache fileHashes.bin 
> (

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug 
option to get more log output.

BUILD FAILED

Total time: 33.507 secs
Build step 'Execute shell' marked build as failure
Recording test results
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2
ERROR: Publisher 'Publish JUnit test result report' failed: No test report 
files were found. Configuration error?
Setting 
JDK1_8_0_45_HOME=/home/jenkins/jenkins-slave/tools/hudson.model.JDK/jdk1.8.0_45
Setting 
GRADLE_2_4_RC_2_HOME=/home/jenkins/jenkins-slave/tools/hudson.plugins.gradle.GradleInstallation/Gradle_2.4-rc-2

Build failed in Jenkins: kafka-trunk-jdk8 #337

[Apache Jenkins Server]

See 

Changes:

[wangguoz] MINOR: some javadocs for kstream public api

--
[...truncated 6689 lines...]
org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testGetConnectorTaskConfigs PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testGetConnectorTaskConfigsConnectorNotFound PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testPutConnectorTaskConfigs PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testPutConnectorTaskConfigsConnectorNotFound PASSED

org.apache.kafka.connect.runtime.rest.resources.ConnectorsResourceTest > 
testCreateConnector PASSED

org.apache.kafka.connect.util.ShutdownableThreadTest > testGracefulShutdown 
PASSED

org.apache.kafka.connect.util.ShutdownableThreadTest > testForcibleShutdown 
PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testSendAndReadToEnd PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testConsumerError PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testProducerError PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testStartStop PASSED

org.apache.kafka.connect.util.KafkaBasedLogTest > testReloadOnStart PASSED

org.apache.kafka.connect.storage.KafkaOffsetBackingStoreTest > testStartStop 
PASSED

org.apache.kafka.connect.storage.KafkaOffsetBackingStoreTest > testMissingTopic 
PASSED

org.apache.kafka.connect.storage.KafkaOffsetBackingStoreTest > testGetSet PASSED

org.apache.kafka.connect.storage.KafkaOffsetBackingStoreTest > testSetFailure 
PASSED

org.apache.kafka.connect.storage.KafkaOffsetBackingStoreTest > 
testReloadOnStart PASSED

org.apache.kafka.connect.storage.KafkaConfigStorageTest > testStartStop PASSED

org.apache.kafka.connect.storage.KafkaConfigStorageTest > 
testPutConnectorConfig PASSED

org.apache.kafka.connect.storage.KafkaConfigStorageTest > testPutTaskConfigs 
PASSED

org.apache.kafka.connect.storage.KafkaConfigStorageTest > 
testPutTaskConfigsDoesNotResolveAllInconsistencies PASSED

org.apache.kafka.connect.storage.KafkaConfigStorageTest > testRestore PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > 
testWriteNullValueFlush PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > 
testWriteNullKeyFlush PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > testNoOffsetsToFlush 
PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > 
testFlushFailureReplacesOffsets PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > testAlreadyFlushing 
PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > 
testCancelBeforeAwaitFlush PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > 
testCancelAfterAwaitFlush PASSED

org.apache.kafka.connect.storage.OffsetStorageWriterTest > testWriteFlush PASSED

org.apache.kafka.connect.storage.FileOffsetBackingStoreTest > testSaveRestore 
PASSED

org.apache.kafka.connect.storage.FileOffsetBackingStoreTest > testGetSet PASSED
:testAll

BUILD SUCCESSFUL

Total time: 1 hrs 0 mins 57.098 secs
+ ./gradlew --stacktrace docsJarAll
To honour the JVM settings for this build a new JVM will be forked. Please 
consider using the daemon: 
https://docs.gradle.org/2.10/userguide/gradle_daemon.html.
Building project 'core' with Scala version 2.10.6
:docsJar_2_10
Building project 'core' with Scala version 2.10.6
:kafka-trunk-jdk8:clients:compileJava UP-TO-DATE
:kafka-trunk-jdk8:clients:processResources UP-TO-DATE
:kafka-trunk-jdk8:clients:classes UP-TO-DATE
:kafka-trunk-jdk8:clients:determineCommitId UP-TO-DATE
:kafka-trunk-jdk8:clients:createVersionFile
:kafka-trunk-jdk8:clients:jar UP-TO-DATE
:kafka-trunk-jdk8:clients:javadoc
:docsJar_2_10 FAILED

FAILURE: Build failed with an exception.

* What went wrong:
Could not add entry 
'
 to cache fileHashes.bin 
(
> Corrupted FreeListBlock 676622 found in cache 
> '

* Try:
Run with --info or --debug option to get more log output.

* Exception is:
org.gradle.api.UncheckedIOException: Could not add entry 
'
 to cache fileHashes.bin 
(
at 
org.gradle.cache.internal.btree.BTreePersistentIndexedCache.put(BTreePersistentIndexedCache.java:155)
at 
org.gradle.cache.internal.DefaultMultiProcessSafePersistentIndexedCache$2.run(DefaultMultiProcessSafePersistentIndexedCache.java:51)
at 
org.gradle.cache.internal.DefaultFileLockManager$DefaultFileLock.doWriteAction(Defaul

[jira] [Resolved] (KAFKA-3121) KStream DSL API Improvement

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-3121.
--
   Resolution: Fixed
Fix Version/s: 0.9.1.0

Issue resolved by pull request 839
[https://github.com/apache/kafka/pull/839]

> KStream DSL API Improvement
> ---
>
> Key: KAFKA-3121
> URL: https://issues.apache.org/jira/browse/KAFKA-3121
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.1.0
>
>
> From some collected feedbacks, here is a list of potential improvements that 
> we want to make:
> 1. Remove AggregateSupplier for aggregate, and provide more built-in 
> aggregations.
> 2. Join to return KeyValue<>.
> 3. "Windows" class syntax-sugers.
> 4. Add print() to KTable / KStream.
> 5. flatMap / flatMapValues to return arrays in addition to Iterable.
> 6. make the API function names aligned with Java 8+, e.g. filterOut -> 
> filterNot
> 7. collapse process() and transform() in KStream.
> 8. validate Streaming configs and allow passing properties to KafkaStreaming. 
>  [DONE]
> 9. Rename KafkaStreaming to Streams.  [DONE]
> Also move some of the state package into internals, and create a new 
> top-level common folder with KeyValue / etc in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3121) KStream DSL API Improvement

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15128915#comment-15128915
 ] 

ASF GitHub Bot commented on KAFKA-3121:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/839


> KStream DSL API Improvement
> ---
>
> Key: KAFKA-3121
> URL: https://issues.apache.org/jira/browse/KAFKA-3121
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.9.1.0
>
>
> From some collected feedbacks, here is a list of potential improvements that 
> we want to make:
> 1. Remove AggregateSupplier for aggregate, and provide more built-in 
> aggregations.
> 2. Join to return KeyValue<>.
> 3. "Windows" class syntax-sugers.
> 4. Add print() to KTable / KStream.
> 5. flatMap / flatMapValues to return arrays in addition to Iterable.
> 6. make the API function names aligned with Java 8+, e.g. filterOut -> 
> filterNot
> 7. collapse process() and transform() in KStream.
> 8. validate Streaming configs and allow passing properties to KafkaStreaming. 
>  [DONE]
> 9. Rename KafkaStreaming to Streams.  [DONE]
> Also move some of the state package into internals, and create a new 
> top-level common folder with KeyValue / etc in it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request: KAFKA-3121: Refactor KStream Aggregate to be L...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/839


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : kafka_0.9.0_jdk7 #108

[Apache Jenkins Server]

See 

[jira] [Commented] (KAFKA-3055) JsonConverter mangles schema during serialization (fromConnectData)

[Saravanan Tirugnanum (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15128891#comment-15128891
 ] 

Saravanan Tirugnanum commented on KAFKA-3055:
-

Hi Kishore

I am facing this same problem and trying to see if we can handle without
updating the patch fix done already. Can you please let me know how to
disable cache as you mentioned this works fine when cache is disabled.

regards
Saravanan


> JsonConverter mangles schema during serialization (fromConnectData)
> ---
>
> Key: KAFKA-3055
> URL: https://issues.apache.org/jira/browse/KAFKA-3055
> Project: Kafka
>  Issue Type: Bug
>  Components: copycat
>Affects Versions: 0.9.0.0
>Reporter: Kishore Senji
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.9.0.1, 0.9.1.0
>
>
> Test case is here: 
> https://github.com/ksenji/kafka-connect-test/tree/master/src/test/java/org/apache/kafka/connect/json
> If Caching is disabled, it behaves correctly and JsonConverterWithNoCacheTest 
> runs successfully. Otherwise the test JsonConverterTest fails.
> The reason is that the JsonConverter has a bug where it mangles the schema as 
> it assigns all String fields with the same name (and similar for all Int32 
> fields)
> This is how the schema & payload gets serialized for the Person Struct (with 
> caching disabled):
> {code}
> {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"firstName"},{"type":"string","optional":false,"field":"lastName"},{"type":"string","optional":false,"field":"email"},{"type":"int32","optional":false,"field":"age"},{"type":"int32","optional":false,"field":"weightInKgs"}],"optional":false,"name":"Person"},"payload":{"firstName":"Eric","lastName":"Cartman","email":"eric.cart...@southpark.com","age":10,"weightInKgs":40}}
> {code}
> where as when caching is enabled the same Struct gets serialized as (with 
> caching enabled) :
> {code}
> {"schema":{"type":"struct","fields":[{"type":"string","optional":false,"field":"email"},{"type":"string","optional":false,"field":"email"},{"type":"string","optional":false,"field":"email"},{"type":"int32","optional":false,"field":"weightInKgs"},{"type":"int32","optional":false,"field":"weightInKgs"}],"optional":false,"name":"Person"},"payload":{"firstName":"Eric","lastName":"Cartman","email":"eric.cart...@southpark.com","age":10,"weightInKgs":40}}
> {code}
> As we can see all String fields became "email" and all int32 fields became 
> "weightInKgs". 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (KAFKA-3191) Improve offset committing JavaDoc in KafkaConsumer

[Adam Kunicki (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3191?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-3191 started by Adam Kunicki.
---
> Improve offset committing JavaDoc in KafkaConsumer
> --
>
> Key: KAFKA-3191
> URL: https://issues.apache.org/jira/browse/KAFKA-3191
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Adam Kunicki
>Assignee: Adam Kunicki
>Priority: Minor
>  Labels: documentation
>
> http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0.9-consumer-client
>  includes some great examples of how to manage offset commits in Kafka 0.9.
> It is currently unclear from the JavaDoc that the following is expected:
> {code}
> consumer.commitSync(Collections.singletonMap(record.partition(), new
> OffsetAndMetadata(record.offset() + 1)));
> {code}
> The committed offset should always be the offset of the next message that
> your application will read.
> This information should be incorporated into the existing JavaDoc to make it 
> clearer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3191) Improve offset committing JavaDoc in KafkaConsumer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15128856#comment-15128856
 ] 

ASF GitHub Bot commented on KAFKA-3191:
---

GitHub user kunickiaj opened a pull request:

https://github.com/apache/kafka/pull/850

KAFKA-3191: Improve offset committing JavaDoc in KafkaConsumer

Added an example clarifying the correct way to use explicit offsets with 
commitSync().

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kunickiaj/kafka KAFKA-3191

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/850.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #850


commit 72705b19bee5c4ac0ef1428229c261bf6ed38490
Author: Adam Kunicki 
Date:   2016-02-02T19:28:54Z

KAFKA-3191: Improve offset committing JavaDoc in KafkaConsumer




> Improve offset committing JavaDoc in KafkaConsumer
> --
>
> Key: KAFKA-3191
> URL: https://issues.apache.org/jira/browse/KAFKA-3191
> Project: Kafka
>  Issue Type: Improvement
>  Components: consumer
>Affects Versions: 0.9.0.0
>Reporter: Adam Kunicki
>Assignee: Adam Kunicki
>Priority: Minor
>  Labels: documentation
>
> http://www.confluent.io/blog/tutorial-getting-started-with-the-new-apache-kafka-0.9-consumer-client
>  includes some great examples of how to manage offset commits in Kafka 0.9.
> It is currently unclear from the JavaDoc that the following is expected:
> {code}
> consumer.commitSync(Collections.singletonMap(record.partition(), new
> OffsetAndMetadata(record.offset() + 1)));
> {code}
> The committed offset should always be the offset of the next message that
> your application will read.
> This information should be incorporated into the existing JavaDoc to make it 
> clearer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)