Re: [DISCUSS] KIP-78: Cluster Id

2016-08-31 Thread Ismael Juma
Thanks for the feedback Guozhang. Comment inline. On Wed, Aug 31, 2016 at 2:49 AM, Guozhang Wang wrote: > About logging / debugging with the cluster id: I think the random UUID > itself may not be very helpful for human-readable debugging information, > and we'd better use the cluster name menti

[jira] [Closed] (KAFKA-4097) "Server not found in kerberos database" issue while starting a Kafka server in a secured mode

2016-08-31 Thread Syam (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Syam closed KAFKA-4097. --- > "Server not found in kerberos database" issue while starting a Kafka server > in a secured mode > -

[jira] [Commented] (KAFKA-3587) LogCleaner fails due to incorrect offset map computation on a replica

2016-08-31 Thread Zaiming Shi (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3587?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15451637#comment-15451637 ] Zaiming Shi commented on KAFKA-3587: any plan to port the fix back to 0.9 ? > LogClea

Re: Newbie JIRAs

2016-08-31 Thread Matthias J. Sax
Hi and welcome to Kafka community! You can browse the Jira board for tickets with labels "newbie" or "beginner": Like this: https://issues.apache.org/jira/browse/KAFKA-3827?jql=project%20%3D%20KAFKA%20AND%20status%20in%20%28Open%2C%20Reopened%29%20AND%20labels%20in%20%28newbie%2C%20%22newbie%2B%2

[jira] [Created] (KAFKA-4104) Queryable state metadata is sometimes invalid

2016-08-31 Thread Eno Thereska (JIRA)
Eno Thereska created KAFKA-4104: --- Summary: Queryable state metadata is sometimes invalid Key: KAFKA-4104 URL: https://issues.apache.org/jira/browse/KAFKA-4104 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-4105) Queryable state tests for concurrency and rebalancing

2016-08-31 Thread Eno Thereska (JIRA)
Eno Thereska created KAFKA-4105: --- Summary: Queryable state tests for concurrency and rebalancing Key: KAFKA-4105 URL: https://issues.apache.org/jira/browse/KAFKA-4105 Project: Kafka Issue Type:

[jira] [Updated] (KAFKA-4104) Queryable state metadata is sometimes invalid

2016-08-31 Thread Eno Thereska (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eno Thereska updated KAFKA-4104: Assignee: Damian Guy > Queryable state metadata is sometimes invalid > -

[GitHub] kafka pull request #1804: KAFKA-4104: Queryable state metadata is sometimes ...

2016-08-31 Thread dguy
GitHub user dguy opened a pull request: https://github.com/apache/kafka/pull/1804 KAFKA-4104: Queryable state metadata is sometimes invalid If the thread or process is not the coordinator the Cluster instance in StreamPartitionAssignor will always be null. This builds an instance of

[jira] [Commented] (KAFKA-4104) Queryable state metadata is sometimes invalid

2016-08-31 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15452031#comment-15452031 ] ASF GitHub Bot commented on KAFKA-4104: --- GitHub user dguy opened a pull request:

[jira] [Work started] (KAFKA-4104) Queryable state metadata is sometimes invalid

2016-08-31 Thread Damian Guy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-4104 started by Damian Guy. - > Queryable state metadata is sometimes invalid > -

[jira] [Work started] (KAFKA-3595) Add capability to specify replication compact option for stream store

2016-08-31 Thread Damian Guy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3595 started by Damian Guy. - > Add capability to specify replication compact option for stream store > -

[jira] [Updated] (KAFKA-3595) Add capability to specify replication compact option for stream store

2016-08-31 Thread Damian Guy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3595?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy updated KAFKA-3595: -- Status: Patch Available (was: In Progress) > Add capability to specify replication compact option for s

[jira] [Work started] (KAFKA-4069) Forward records in context of cache flushing/eviction

2016-08-31 Thread Damian Guy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-4069 started by Damian Guy. - > Forward records in context of cache flushing/eviction > -

[GitHub] kafka pull request #1805: MINOR: onCompletion: metadata valid only if no exc...

2016-08-31 Thread ybyzek
GitHub user ybyzek opened a pull request: https://github.com/apache/kafka/pull/1805 MINOR: onCompletion: metadata valid only if no exception Modifies example in doc change You can merge this pull request into a Git repository by running: $ git pull https://github.com/ybyzek/kaf

[jira] [Updated] (KAFKA-4104) Queryable state metadata is sometimes invalid

2016-08-31 Thread Damian Guy (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Damian Guy updated KAFKA-4104: -- Status: Patch Available (was: In Progress) > Queryable state metadata is sometimes invalid > --

[jira] [Created] (KAFKA-4106) Consumer / add configure method to PartitionAssignor interface

2016-08-31 Thread Florian Hussonnois (JIRA)
Florian Hussonnois created KAFKA-4106: - Summary: Consumer / add configure method to PartitionAssignor interface Key: KAFKA-4106 URL: https://issues.apache.org/jira/browse/KAFKA-4106 Project: Kafka

Re: [DISCUSS] KIP-78: Cluster Id

2016-08-31 Thread Jun Rao
Ismael, Thanks for the proposal. It looks good overall. Just a comment below. We are adding the following interface in ClusterListener and Cluster includes all brokers' endpoint and the metadata of topic partitions. void onClusterUpdate(Cluster cluster); On the broker side, will that method be

[jira] [Commented] (KAFKA-4105) Queryable state tests for concurrency and rebalancing

2016-08-31 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4105?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15452590#comment-15452590 ] ASF GitHub Bot commented on KAFKA-4105: --- GitHub user enothereska opened a pull reque

[GitHub] kafka pull request #1806: KAFKA-4105: Queryable state tests

2016-08-31 Thread enothereska
GitHub user enothereska opened a pull request: https://github.com/apache/kafka/pull/1806 KAFKA-4105: Queryable state tests You can merge this pull request into a Git repository by running: $ git pull https://github.com/enothereska/kafka queryable-state-tests Alternatively you

Re: [VOTE] KIP-77: Improve Kafka Streams Join Semantics

2016-08-31 Thread Michael Noll
+1 (non-binding) On Wed, Aug 31, 2016 at 6:18 AM, Neha Narkhede wrote: > +1 (binding) > > On Tue, Aug 30, 2016 at 5:33 PM Ewen Cheslack-Postava > wrote: > > > +1 (binding) > > > > I think the major gap I notice in the PR is a lack of docs updates to > > notify people of the change. Given these

Re: [VOTE] KIP-63: Unify store and downstream caching in streams

2016-08-31 Thread Michael Noll
+1 (non-binding) On Fri, Aug 26, 2016 at 3:10 PM, Bill Bejeck wrote: > +1 > > On Thu, Aug 25, 2016 at 11:45 AM, Matthias J. Sax > wrote: > > > +1 > > > > On 08/25/2016 04:22 PM, Damian Guy wrote: > > > +1 > > > > > > On Thu, 25 Aug 2016 at 11:57 Eno Thereska > > wrote: > > > > > >> Hi folks, >

Re: [VOTE] KIP-77: Improve Kafka Streams Join Semantics

2016-08-31 Thread Jim Jagielski
+1 > On Aug 29, 2016, at 5:50 AM, Matthias J. Sax wrote: > > I’d like to initiate the voting process for KIP-77: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-77%3A+Improve+Kafka+Streams+Join+Semantics > > -Matthias > >

Re: Kafka KIP meeting Aug 30 at 11:00am PST

2016-08-31 Thread Jim Jagielski
Just a reminder that these kinds of meetings are, by their very nature, synchronous and disenfranchise those members of the community unable to attend. It is great that discussions and videos are brought back to the list BUT they should not be "this is what has been decided" or "this is what we ar

[jira] [Updated] (KAFKA-3799) Turn on endpoint validation in SSL system tests

2016-08-31 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-3799: - Resolution: Fixed Fix Version/s: 0.10.1.0 Status: Resolved (was:

Re: [VOTE] KIP-63: Unify store and downstream caching in streams

2016-08-31 Thread Jim Jagielski
+1 > On Aug 25, 2016, at 6:57 AM, Eno Thereska wrote: > > Hi folks, > > We'd like to start the vote for KIP-63. At this point the Wiki addresses > all previous questions and we believe the PoC is feature-complete. > > Thanks > Eno

[jira] [Commented] (KAFKA-3799) Turn on endpoint validation in SSL system tests

2016-08-31 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15452666#comment-15452666 ] ASF GitHub Bot commented on KAFKA-3799: --- Github user asfgit closed the pull request

[GitHub] kafka pull request #1483: KAFKA-3799: Enable SSL endpoint validation in syst...

2016-08-31 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1483 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

Re: Kafka KIP meeting Aug 30 at 11:00am PST

2016-08-31 Thread Ismael Juma
Hi Jim, Thanks for the feedback. More inline. On Wed, Aug 31, 2016 at 5:19 PM, Jim Jagielski wrote: > Just a reminder that these kinds of meetings are, by their very nature, > synchronous and disenfranchise those members of the community unable to > attend. > > It is great that discussions and

Re: Kafka KIP meeting Aug 30 at 11:00am PST

2016-08-31 Thread Jim Jagielski
As I said, just a general reminder... It is good, I think, to have them every so often to reinforce things. Sometimes it is easy to get "lazy" about it :) > On Aug 31, 2016, at 12:34 PM, Ismael Juma wrote: > > Hi Jim, > > Thanks for the feedback. More inline. > > On Wed, Aug 31, 2016 at 5:19

[GitHub] kafka pull request #1807: KAFKA-4103: Fix regression in DumpLogSegments offs...

2016-08-31 Thread hachikuji
GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/1807 KAFKA-4103: Fix regression in DumpLogSegments offsets decoder You can merge this pull request into a Git repository by running: $ git pull https://github.com/hachikuji/kafka KAFKA-4103 Alte

[jira] [Commented] (KAFKA-4103) DumpLogSegments cannot print data from offsets topic

2016-08-31 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15452869#comment-15452869 ] ASF GitHub Bot commented on KAFKA-4103: --- GitHub user hachikuji opened a pull request

[jira] [Commented] (KAFKA-4106) Consumer / add configure method to PartitionAssignor interface

2016-08-31 Thread Jason Gustafson (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15452913#comment-15452913 ] Jason Gustafson commented on KAFKA-4106: [~fhussonnois] I think you might be able

Re: Kafka KIP meeting Aug 30 at 11:00am PST

2016-08-31 Thread Ismael Juma
OK, thanks. Just making sure. :) Ismael On 31 Aug 2016 5:47 pm, "Jim Jagielski" wrote: > As I said, just a general reminder... > > It is good, I think, to have them every so often to reinforce things. > Sometimes it is easy to get "lazy" about it :) > > > On Aug 31, 2016, at 12:34 PM, Ismael Ju

Re: Question regarding Producer and Duplicates

2016-08-31 Thread Guozhang Wang
Hi Florian, The broker will accept a batch of records as a whole or reject them as a whole unless it encounters an IOException while trying to append the messages, which will be treated as a fatal error anyways. Duplicates usually happen when the whole batch is accepted but the ack was not delive

Re: [VOTE] KIP-33 - Add a time based log index

2016-08-31 Thread Guozhang Wang
Some of the streams integration tests also encounters this issue where the timestamps we are using in the test are very small (e.g. 1,2,3...) which cause the log to roll upon each append, and the old segment gets deleted very soon. Arguably this can be resolved to enforce LogAppendTime configuratio

[GitHub] kafka pull request #1805: MINOR: onCompletion: metadata valid only if no exc...

2016-08-31 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1805 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Commented] (KAFKA-3144) report members with no assigned partitions in ConsumerGroupCommand

2016-08-31 Thread Vahid Hashemian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453096#comment-15453096 ] Vahid Hashemian commented on KAFKA-3144: [~junrao] [~alexlod] Thanks for the feedb

Kafka consumers unable to process message

2016-08-31 Thread Ghosh, Achintya (Contractor)
Hi there, Kafka consumer gets stuck at consumer.poll() method if my current datacenter is down and replicated messages are in remote datacenter. How to solve that issue? Thanks Achintya

[jira] [Comment Edited] (KAFKA-4106) Consumer / add configure method to PartitionAssignor interface

2016-08-31 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453098#comment-15453098 ] Guozhang Wang edited comment on KAFKA-4106 at 8/31/16 7:11 PM: -

[jira] [Commented] (KAFKA-4106) Consumer / add configure method to PartitionAssignor interface

2016-08-31 Thread Guozhang Wang (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4106?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453098#comment-15453098 ] Guozhang Wang commented on KAFKA-4106: -- [~fhussonnois] {PartitionAssignor} extends {C

[jira] [Commented] (KAFKA-3410) Unclean leader election and "Halting because log truncation is not allowed"

2016-08-31 Thread James Cheng (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453103#comment-15453103 ] James Cheng commented on KAFKA-3410: [~maysamyabandeh] Oh right, yeah, I didn't expect

[jira] [Updated] (KAFKA-4033) KIP-70: Revise Partition Assignment Semantics on New Consumer's Subscription Change

2016-08-31 Thread Vahid Hashemian (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vahid Hashemian updated KAFKA-4033: --- Status: Patch Available (was: In Progress) > KIP-70: Revise Partition Assignment Semantics on

[jira] [Created] (KAFKA-4107) Support offset reset capability in Kafka Connect

2016-08-31 Thread Jason Gustafson (JIRA)
Jason Gustafson created KAFKA-4107: -- Summary: Support offset reset capability in Kafka Connect Key: KAFKA-4107 URL: https://issues.apache.org/jira/browse/KAFKA-4107 Project: Kafka Issue Type

Re: Kafka consumers unable to process message

2016-08-31 Thread Jason Gustafson
Hi Achintya, We have a JIRA for this problem: https://issues. apache.org/jira/browse/KAFKA-3834. Do you expect the client to raise an exception in this case or do you just want to keep it from blocking indefinitely? If the latter, you could escape the poll from another thread using wakeup(). Than

Build failed in Jenkins: kafka-trunk-jdk8 #854

2016-08-31 Thread Apache Jenkins Server
See Changes: [wangguoz] MINOR: onCompletion: metadata valid only if no exception -- [...truncated 3469 lines...] kafka.api.AuthorizerIntegrationTest > testCommitWithNoTopicAccess PASSED kafka.ap

[jira] [Assigned] (KAFKA-3807) OffsetValidationTest - transient failure on test_broker_rolling_bounce

2016-08-31 Thread Jason Gustafson (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson reassigned KAFKA-3807: -- Assignee: Jason Gustafson > OffsetValidationTest - transient failure on test_broker_rol

[jira] [Work started] (KAFKA-3807) OffsetValidationTest - transient failure on test_broker_rolling_bounce

2016-08-31 Thread Jason Gustafson (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-3807?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on KAFKA-3807 started by Jason Gustafson. -- > OffsetValidationTest - transient failure on test_broker_rolling_bounce > --

Re: [DISCUSS] KIP-79 - ListOffsetRequest v1 and offsetForTime() method in new consumer.

2016-08-31 Thread Mayuresh Gharat
Hi Becket, Thanks for the write up. I had few minor comments : 1) The wiki says ListOffsetRequest v0 still behaves in the old way. ---> It would be good to include what it means although that's not the part of this KIP. You can add a note what V0 means like : (Kafka allows querying offsets of mes

[GitHub] kafka pull request #1807: KAFKA-4103: Fix regression in DumpLogSegments offs...

2016-08-31 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1807 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is enab

[jira] [Commented] (KAFKA-4103) DumpLogSegments cannot print data from offsets topic

2016-08-31 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15453298#comment-15453298 ] ASF GitHub Bot commented on KAFKA-4103: --- Github user asfgit closed the pull request

[jira] [Resolved] (KAFKA-4103) DumpLogSegments cannot print data from offsets topic

2016-08-31 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-4103. -- Resolution: Fixed Issue resolved by pull request 1807 [https://github.com/apache

[jira] [Created] (KAFKA-4108) Improve DumpLogSegments offsets-decoder output format

2016-08-31 Thread Jason Gustafson (JIRA)
Jason Gustafson created KAFKA-4108: -- Summary: Improve DumpLogSegments offsets-decoder output format Key: KAFKA-4108 URL: https://issues.apache.org/jira/browse/KAFKA-4108 Project: Kafka Issue

Re: [VOTE] KIP-33 - Add a time based log index

2016-08-31 Thread Becket Qin
Thanks for the feedback, Guozhang. I'll update the KIP wiki and submit a patch if no one have concerns on this. On Wed, Aug 31, 2016 at 11:59 AM, Guozhang Wang wrote: > Some of the streams integration tests also encounters this issue where the > timestamps we are using in the test are very small

Re: Kafka consumers unable to process message

2016-08-31 Thread Jim Jagielski
Yeah, let's figure out the "best" action to take... Looks like something I'd like to get a handle on. > On Aug 31, 2016, at 4:05 PM, Jason Gustafson wrote: > > Hi Achintya, > > We have a JIRA for this problem: https://issues. > apache.org/jira/browse/KAFKA-3834. Do you expect the client to rai

Re: [DISCUSS] KIP-79 - ListOffsetRequest v1 and offsetForTime() method in new consumer.

2016-08-31 Thread Becket Qin
Hi Mayuresh, Thanks for the comments. 1) I added the link to the Kafka protocol guide. 2) True that for LogAppendTime the message timestamp in different cluster will not be the same. I updated the wiki. But notice that even the LogAppendTime on different clusters are still comparable given a rea

[GitHub] kafka pull request #1808: MINOR: changes embedded broker time to MockTime

2016-08-31 Thread mjsax
GitHub user mjsax opened a pull request: https://github.com/apache/kafka/pull/1808 MINOR: changes embedded broker time to MockTime You can merge this pull request into a Git repository by running: $ git pull https://github.com/mjsax/kafka mockTime Alternatively you can review

Build failed in Jenkins: kafka-trunk-jdk8 #855

2016-08-31 Thread Apache Jenkins Server
See Changes: [me] KAFKA-4103: Fix regression in DumpLogSegments offsets decoder -- [...truncated 12258 lines...] org.apache.kafka.streams.processor.internals.StreamTaskTest > testProcessOrder PA

Re: Kafka consumers unable to process message

2016-08-31 Thread Jason Gustafson
Kafka clients have tended to make broker retries transparent to the user. There's been discussion on various JIRAs about what we should do when all the known brokers become unreachable. One option is to revert to the configured bootstrap broker list, which is nice if you've configured a vip for boo

RE: Kafka consumers unable to process message

2016-08-31 Thread Ghosh, Achintya (Contractor)
Hi Jason, Thanks for your response. I know that is a known issue and I resolved it calling wakeup method by another thread. But here my problem is different, let me explain , it's very basic I created one cluster with 6 nodes( 3 from one datacenter and 3 from another(remote) datacenter and kep

[jira] [Created] (KAFKA-4109) kafka client send msg exception

2016-08-31 Thread frank (JIRA)
frank created KAFKA-4109: Summary: kafka client send msg exception Key: KAFKA-4109 URL: https://issues.apache.org/jira/browse/KAFKA-4109 Project: Kafka Issue Type: Bug Components: producer

[jira] [Updated] (KAFKA-4109) kafka client send msg exception

2016-08-31 Thread frank (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] frank updated KAFKA-4109: - Environment: java8 kafka cluster was:java8 > kafka client send msg exception > --

Re: Kafka consumers unable to process message

2016-08-31 Thread Jason Gustafson
Hi Achintya, Just to clarify, you did not take down either of the zookeepers in this test, right? Having only two zookeepers in the ensemble would mean that if either one of them failed, zookeeper wouldn't be able to reach quorum. I'm not entirely sure why this would happen. One possibility is th

RE: Kafka consumers unable to process message

2016-08-31 Thread Ghosh, Achintya (Contractor)
Hi Jason, No, I didn't bring down any zookeeper server. Even I tried with 3 zookeeper server one as an 'Observer' but the same issue. Here is the server log from one of the node of my other datacenter: [2016-09-01 01:25:19,221] INFO Truncating log TEST3-0 to offset 0. (kafka.log.Log) [2016-09-

Re: Kafka consumers unable to process message

2016-08-31 Thread Jason Gustafson
The exceptions show one of the replica fetcher threads on the broker failing which makes perfect sense since some of the partitions were bound to have leaders in the failed datacenter. I'd actually like to see the consumer logs at DEBUG level if possible. Thanks, Jason On Wed, Aug 31, 2016 at 7:4

RE: Kafka consumers unable to process message

2016-08-31 Thread Ghosh, Achintya (Contractor)
I'm trying get the consumer logs and will send you. So it means it can happen even my local datacenter too. Still I'm not understanding if 3 nodes are up and message already replicated why it's trying to fetch the data from failed node. Can you please explain bit details how it works. Thanks for

[jira] [Updated] (KAFKA-4099) Change the time based log rolling to only based on the message timestamp.

2016-08-31 Thread Jiangjie Qin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-4099: Summary: Change the time based log rolling to only based on the message timestamp. (was: Change the

[jira] [Updated] (KAFKA-4099) Change the time based log rolling to only based on the message timestamp.

2016-08-31 Thread Jiangjie Qin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-4099: Description: This is an issue introduced in KAFKA-3163. When partition relocation occurs, the newly

[jira] [Updated] (KAFKA-4099) Change the time based log rolling to only based on the message timestamp.

2016-08-31 Thread Jiangjie Qin (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin updated KAFKA-4099: Description: This is an issue introduced in KAFKA-3163. When partition relocation occurs, the newly

[GitHub] kafka pull request #1809: KAFKA-4099: Fix the potential frequent log rolling

2016-08-31 Thread becketqin
GitHub user becketqin opened a pull request: https://github.com/apache/kafka/pull/1809 KAFKA-4099: Fix the potential frequent log rolling You can merge this pull request into a Git repository by running: $ git pull https://github.com/becketqin/kafka KAFKA-4099 Alternatively y

[jira] [Commented] (KAFKA-4099) Change the time based log rolling to only based on the message timestamp.

2016-08-31 Thread ASF GitHub Bot (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-4099?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15454198#comment-15454198 ] ASF GitHub Bot commented on KAFKA-4099: --- GitHub user becketqin opened a pull request

Re: Kafka consumers unable to process message

2016-08-31 Thread Jason Gustafson
The replicas are always trying to fetch new data from the partition leader. When the leader fails, any in-flight fetches will fail and cause errors such as the ones you saw in the broker log. Eventually the replicas will discover the new leader and begin fetching again. And of course one of the rep

Message sent ordering guarantees

2016-08-31 Thread 郭旭
Hi Kafka Experts, (Sorry to send this question to DEV group, but it seems that I can not find related document in user manual.) For official document ,I can find message sent guarantee as below. For *sync producer*, I think it is true but sync sent are very slow.(about 408 message per second if a