[jira] [Commented] (KAFKA-3853) Report offsets for empty groups in ConsumerGroupCommand

[Vahid Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15760443#comment-15760443
 ] 

Vahid Hashemian commented on KAFKA-3853:


[~jeffwidman] Yes, it did pass. But, unfortunately, an issue came up that 
require an update to the KIP and another round of discussion and voting. You 
can read more about it 
[here|https://www.mail-archive.com/dev@kafka.apache.org/msg62357.html] and 
[here|https://www.mail-archive.com/dev@kafka.apache.org/msg62429.html]. I'll be 
updating the KIP in the next couple of days, and restart the discussion and 
then voting. Sorry for the inconvenience.

> Report offsets for empty groups in ConsumerGroupCommand
> ---
>
> Key: KAFKA-3853
> URL: https://issues.apache.org/jira/browse/KAFKA-3853
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Vahid Hashemian
> Fix For: 0.10.2.0
>
>
> We ought to be able to display offsets for groups which either have no active 
> members or which are not using group management. The owner column can be left 
> empty or set to "N/A". If a group is active, I'm not sure it would make sense 
> to report all offsets, in particular when partitions are unassigned, but if 
> it seems problematic to do so, we could enable the behavior with a flag (e.g. 
> --include-unassigned).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-98: Exactly Once Delivery and Transactional Messaging

[Becket Qin]

@Jason

Yes, second thought on the number of messages included, the offset delta
will probably be sufficient. The use case I encounter before for number of
messages in a message set is an embedded mirror maker on the destination
broker side which fetches message directly from the source cluster. Ideally
the destination cluster only needs to check CRC and assign the offsets
because all the message verification has been done by the source cluster,
but due to the lack of the number of messages in the message set, we have
to decompress the message set to increment offsets correctly. By knowing
the number of the messages in the message set, we can avoid doing that. The
offset delta will also help. It's just then the offsets may have holes for
log compacted topics, but that may be fine.

@Apurva

I am not sure if it is true that the consumer will either deliver all the
message for the entire transaction or none of them from one poll() call. If
we allow the transactions to be across partitions, unless the consumer
consumes from all the partitions involved in a transactions, it seems
impossible for it to deliver *all* the messages in a transaction, right? A
weaker guarantee is we will deliver all or none of the messages that belong
to the same transaction in ONE partition, but this would be different from
the guarantee from the producer side.

My two cents on Radai's sideways partition design:
1. If we consider the producer side behavior as doing a two phase commit
which including the committing the consumer offsets, it is a little awkward
that we allow uncommitted message goes into the main log and rely on the
consumer to filter out. So semantic wise I think it would be better if we
can avoid this. Radai's suggestion is actually intuitive because if the
brokers do not want to expose uncommitted transactions to the consumer, the
brokers have to buffer it.

2. Regarding the efficiency. I think may be it worth looking at the
efficiency cost v.s benefit. The efficiency includes both server side
efficiency and consumer side efficiency.

Regarding the server side efficiency, the current proposal would probably
have better efficiency regardless of whether something goes wrong. Radai's
suggestion would put more burden on the server side. If nothing goes wrong
we always pay the cost of having double copy of the transactional messages
and do not get the semantic benefit. But if something goes wrong, the
efficiency cost we pay we get us a better semantic.

For the consumer side efficiency, because there is no need to buffer the
uncommitted messages. The current proposal may have to potentially buffer
uncommitted messages so it would be less efficient than Radai's suggestion
when a transaction aborts. When everything goes well, both design seems
having the similar performance. However, it depends on whether we are
willing to loosen the consumer side transaction guarantee that I mentioned
earlier to Apurva.

Currently the biggest pressure on the consumer side is that it has to
buffer incomplete transactions. There are two reasons for it,
A. A transaction may be aborted so we cannot expose the messages to the
users.
B. We want to return all or none of the messages in a transaction in ONE
partition.

While reason A is mandatory, I think reason B may be discussable. Radai's
design actually removes reason A because there is no uncommitted messages
exposed to the consumers. This may potentially give us a chance to
significantly improve consumer side efficiency in normal cases. It again
depends on the use case, i.e. whether user can process a transaction
progressively (message by message) or it has to be buffered and returned
all together. If in most cases, users can process the transactions message
by message (most stream processing tasks probably can do so), then with
Radai's proposal we don't need to buffer the transactions for the users
anymore, which is a big difference. For the latter case, the consumer may
have to buffer the incomplete transactions otherwise we are just throwing
the burden onto the users.

Thanks,

Jiangjie (Becket) Qin

On Fri, Dec 16, 2016 at 4:56 PM, Jay Kreps  wrote:

> Yeah good point. I relent!
>
> -jay
>
> On Fri, Dec 16, 2016 at 1:46 PM Jason Gustafson 
> wrote:
>
> > Jay/Ismael,
> >
> >
> >
> > I agree that lazy initialization of metadata seems unavoidable. Ideally,
> we
> >
> > could follow the same pattern for transactions, but remember that in the
> >
> > consumer+producer use case, the initialization needs to be completed
> prior
> >
> > to setting the consumer's position. Otherwise we risk reading stale
> >
> > offsets. But it would be pretty awkward if you have to begin a
> transaction
> >
> > first to ensure that your consumer can read the right offset from the
> >
> > consumer, right? It's a bit easier to explain that you should always call
> >
> > `producer.init()` prior to initializing the consumer. Users would
> probably
> >
> > get this right 

[GitHub] kafka pull request #2274: Implement topic config for internal topics

[sjmittal]

GitHub user sjmittal opened a pull request:

https://github.com/apache/kafka/pull/2274

Implement topic config for internal topics



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/sjmittal/kafka sm

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2274.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2274


commit 4af9190ed012283b9210c3b419fcd5bde0b4e748
Author: Sachin Mittal 
Date:   2016-12-19T06:13:55Z

Implement topic config for internal topics




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3853) Report offsets for empty groups in ConsumerGroupCommand

[Jeff Widman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3853?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15759588#comment-15759588
 ] 

Jeff Widman commented on KAFKA-3853:


Did the KIP get the missing vote required to pass?

> Report offsets for empty groups in ConsumerGroupCommand
> ---
>
> Key: KAFKA-3853
> URL: https://issues.apache.org/jira/browse/KAFKA-3853
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jason Gustafson
>Assignee: Vahid Hashemian
> Fix For: 0.10.2.0
>
>
> We ought to be able to display offsets for groups which either have no active 
> members or which are not using group management. The owner column can be left 
> empty or set to "N/A". If a group is active, I'm not sure it would make sense 
> to report all offsets, in particular when partitions are unassigned, but if 
> it seems problematic to do so, we could enable the behavior with a flag (e.g. 
> --include-unassigned).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-82 - Add Record Headers

[Matthias J. Sax]

The details about headers for control messages are still to define. But
yes, the idea is to have some common default behavior that clients would
need to implement.

The point is, that "regular headers" add meta data to regular messages.
Thus, those messages will be returned to the user via .poll(). And after
the message is received the user can check if meta data is present and
read it.

For control messages, we do not want those to pop up via .poll() as
those are no regular messages. A client would need to opt-in to see
those messages (either via poll() or maybe a callback). Thus, we need
some special (standardized) header IDs that indicate control messages
that should not be returned to the user via poll() by default.


-Matthias

On 12/17/16 9:37 PM, Roger Hoover wrote:
> Matthias,
> 
> Thanks for your input.  I'm +1 on control messages as they seem to be the
> simplest way to implement watermarks (
> https://www.oreilly.com/ideas/the-world-beyond-batch-streaming-102), a
> feature that would add a lot of value to Kafka Streams IMHO.
> 
> Your argument that the control-message indicator belongs in the client-only
> section of the record format make sense.  Just to make sure I understand,
> are you suggesting that control messages would be indicated by a standard
> reserved header?  By standard, I mean that ALL Kafka consumers would know
> to handle these messages differently (possibly just ignoring them).  This
> would need to be added to the specification of the consumer protocol so
> that all Kafka clients implement it, right?  I think it's a good idea but
> just checking.
> 
> Cheers,
> 
> Roger
> 
> 
> On Wed, Dec 14, 2016 at 9:51 AM, Matthias J. Sax 
> wrote:
> 
>> Yes and no. I did overload the term "control message".
>>
>> EOS control messages are for client-broker communication and thus never
>> exposed to any application. And I think this is a good design because
>> broker needs to understand those control messages. Thus, this should be
>> a protocol change.
>>
>> The type of control messages I have in mind are for client-client
>> (application-application) communication and the broker is agnostic to
>> them. Thus, it should not be a protocol change.
>>
>>
>> -Matthias
>>
>>
>>
>> On 12/14/16 9:42 AM, radai wrote:
>>> arent control messages getting pushed as their own top level protocol
>>> change (and a fairly massive one) for the transactions KIP ?
>>>
>>> On Tue, Dec 13, 2016 at 5:54 PM, Matthias J. Sax 
>>> wrote:
>>>
 Hi,

 I want to add a completely new angle to this discussion. For this, I
 want to propose an extension for the headers feature that enables new
 uses cases -- and those new use cases might convince people to support
 headers (of course including the larger scoped proposal).

 Extended Proposal:

 Allow messages with a certain header key to be special "control
 messages" (w/ o w/o payload) that are not exposed to an application via
 .poll().

 Thus, a consumer client would automatically skip over those messages. If
 an application knows about embedded control messages, it can "sing up"
 to those messages by the consumer client and either get a callback or
 the consumer auto-drop for this messages gets disabled (allowing to
 consumer those messages via poll()).

 (The details need further considerations/discussion. I just want to
 sketch the main idea.)

 Usage:

 There is a shared topic (ie, used by multiple applications) and a
 producer application wants to embed a special message in the topic for a
 dedicated consumer application. Because only one application will
 understand this message, it cannot be a regular message as this would
 break all applications that do not understand this message. The producer
 application would set a special metadata key and no consumer application
 would see this control message by default because they did not enable
 their consumer client to return this message in poll() (and the client
 would just drop this message with special metadata key). Only the single
 application that should receive this message, will subscribe to this
 message on its consumer client and process it.


 Concrete Use Case: Kafka Streams

 In Kafka Streams, we would like to propagate "control messages" from
 subtopology to subtopology. There are multiple scenarios for which this
 would be useful. For example, currently we do not guarantee a
 "consistent shutdown" of an application. By this, I mean that input
 records might not be completely processed by the whole topology because
 the application shutdown happens "in between" and an intermediate result
 topic gets "stock" in an intermediate topic. Thus, a user would see an
 committed offset of the source topic of the application, but no
 corresponding result record in the output 

[jira] [Commented] (KAFKA-4555) Using Hamcrest for easy intent expression in tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15759489#comment-15759489
 ] 

ASF GitHub Bot commented on KAFKA-4555:
---

GitHub user rekhajoshm opened a pull request:

https://github.com/apache/kafka/pull/2273

KAFKA-4555: Using Hamcrest for expressive intent in tests

- Adding hamcrest in gradle files
- Using hamcrest in couple of tests - SourceTaskOffsetCommitterTest, 
MetadataTest

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rekhajoshm/kafka KAFKA-4555

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2273.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2273


commit c9a66992b1095616f87c5748f210b973ebc7eb01
Author: Rekha Joshi 
Date:   2016-05-26T17:48:37Z

Merge pull request #2 from apache/trunk

Apache Kafka trunk pull

commit 8d7fb005cb132440e7768a5b74257d2598642e0f
Author: Rekha Joshi 
Date:   2016-05-30T21:37:43Z

Merge pull request #3 from apache/trunk

Apache Kafka trunk pull

commit fbef9a8fb1411282fbadec46955691c3e7ba2578
Author: Rekha Joshi 
Date:   2016-06-04T23:58:02Z

Merge pull request #4 from apache/trunk

Apache Kafka trunk pull

commit 172db701bf9affda1304b684921260d1cd36ae9e
Author: Rekha Joshi 
Date:   2016-06-06T22:10:31Z

Merge pull request #6 from apache/trunk

Apache Kafka trunk pull

commit 9d18d93745cf2bc9b0ab4bb9b25d9a31196ef918
Author: Rekha Joshi 
Date:   2016-06-07T19:36:45Z

Merge pull request #7 from apache/trunk

Apache trunk pull

commit 882faea01f28aef1977f4ced6567833bcf736840
Author: Rekha Joshi 
Date:   2016-06-13T20:01:43Z

Merge pull request #8 from confluentinc/trunk

Apache kafka trunk pull

commit 851315d39c0c308d79b9575546822aa932c46a09
Author: Rekha Joshi 
Date:   2016-06-27T17:34:54Z

Merge pull request #9 from apache/trunk

Merge Apache kafka trunk

commit 613f07c2b4193302c82a5d6eaa1e53e4b87bfbc1
Author: Rekha Joshi 
Date:   2016-07-09T17:03:45Z

Merge pull request #11 from apache/trunk

Merge Apache kafka trunk

commit 150e46e462cc192fb869e633f6d9ab681e7b83f9
Author: Rekha Joshi 
Date:   2016-08-02T19:44:09Z

Merge pull request #12 from apache/trunk

Apache Kafka trunk pull

commit 46b4868faed40ec83f87a089cf0db83b31bb2d8a
Author: Rekha Joshi 
Date:   2016-12-07T05:19:35Z

Merge pull request #13 from apache/trunk

Apache Kafka trunk pull

commit c2e44b8a935fb6e0ad9df399cdc5c5ee3e1b287a
Author: Joshi 
Date:   2016-12-18T21:09:00Z

KAFKA-4555: Using Hamcrest in tests

commit 5101c1ec065c6ec210b3aa87b3e2122d529e323a
Author: Joshi 
Date:   2016-12-18T21:11:16Z

KAFKA-4555: Using Hamcrest in tests




> Using Hamcrest for easy intent expression in tests
> --
>
> Key: KAFKA-4555
> URL: https://issues.apache.org/jira/browse/KAFKA-4555
> Project: Kafka
>  Issue Type: Test
>  Components: unit tests
>Affects Versions: 0.9.0.1
>Reporter: Rekha Joshi
>Assignee: Rekha Joshi
>Priority: Minor
>
> Using Hamcrest for easy intent expression in tests



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2273: KAFKA-4555: Using Hamcrest for expressive intent i...

[rekhajoshm]

GitHub user rekhajoshm opened a pull request:

https://github.com/apache/kafka/pull/2273

KAFKA-4555: Using Hamcrest for expressive intent in tests

- Adding hamcrest in gradle files
- Using hamcrest in couple of tests - SourceTaskOffsetCommitterTest, 
MetadataTest

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rekhajoshm/kafka KAFKA-4555

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2273.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2273


commit c9a66992b1095616f87c5748f210b973ebc7eb01
Author: Rekha Joshi 
Date:   2016-05-26T17:48:37Z

Merge pull request #2 from apache/trunk

Apache Kafka trunk pull

commit 8d7fb005cb132440e7768a5b74257d2598642e0f
Author: Rekha Joshi 
Date:   2016-05-30T21:37:43Z

Merge pull request #3 from apache/trunk

Apache Kafka trunk pull

commit fbef9a8fb1411282fbadec46955691c3e7ba2578
Author: Rekha Joshi 
Date:   2016-06-04T23:58:02Z

Merge pull request #4 from apache/trunk

Apache Kafka trunk pull

commit 172db701bf9affda1304b684921260d1cd36ae9e
Author: Rekha Joshi 
Date:   2016-06-06T22:10:31Z

Merge pull request #6 from apache/trunk

Apache Kafka trunk pull

commit 9d18d93745cf2bc9b0ab4bb9b25d9a31196ef918
Author: Rekha Joshi 
Date:   2016-06-07T19:36:45Z

Merge pull request #7 from apache/trunk

Apache trunk pull

commit 882faea01f28aef1977f4ced6567833bcf736840
Author: Rekha Joshi 
Date:   2016-06-13T20:01:43Z

Merge pull request #8 from confluentinc/trunk

Apache kafka trunk pull

commit 851315d39c0c308d79b9575546822aa932c46a09
Author: Rekha Joshi 
Date:   2016-06-27T17:34:54Z

Merge pull request #9 from apache/trunk

Merge Apache kafka trunk

commit 613f07c2b4193302c82a5d6eaa1e53e4b87bfbc1
Author: Rekha Joshi 
Date:   2016-07-09T17:03:45Z

Merge pull request #11 from apache/trunk

Merge Apache kafka trunk

commit 150e46e462cc192fb869e633f6d9ab681e7b83f9
Author: Rekha Joshi 
Date:   2016-08-02T19:44:09Z

Merge pull request #12 from apache/trunk

Apache Kafka trunk pull

commit 46b4868faed40ec83f87a089cf0db83b31bb2d8a
Author: Rekha Joshi 
Date:   2016-12-07T05:19:35Z

Merge pull request #13 from apache/trunk

Apache Kafka trunk pull

commit c2e44b8a935fb6e0ad9df399cdc5c5ee3e1b287a
Author: Joshi 
Date:   2016-12-18T21:09:00Z

KAFKA-4555: Using Hamcrest in tests

commit 5101c1ec065c6ec210b3aa87b3e2122d529e323a
Author: Joshi 
Date:   2016-12-18T21:11:16Z

KAFKA-4555: Using Hamcrest in tests




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-4555) Using Hamcrest for easy intent expression in tests

[Rekha Joshi (JIRA)]

Rekha Joshi created KAFKA-4555:
--

 Summary: Using Hamcrest for easy intent expression in tests
 Key: KAFKA-4555
 URL: https://issues.apache.org/jira/browse/KAFKA-4555
 Project: Kafka
  Issue Type: Test
  Components: unit tests
Affects Versions: 0.9.0.1
Reporter: Rekha Joshi
Assignee: Rekha Joshi
Priority: Minor


Using Hamcrest for easy intent expression in tests



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4527) Transient failure of ConnectDistributedTest.test_pause_and_resume_sink where paused connector produces messages

[Roger Hoover (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15759269#comment-15759269
 ] 

Roger Hoover commented on KAFKA-4527:
-

Happened again: 

http://confluent-systest.s3-website-us-west-2.amazonaws.com/confluent-kafka-system-test-results/?prefix=2016-12-18--001.1482053747--apache--trunk--d6b0b52/

> Transient failure of ConnectDistributedTest.test_pause_and_resume_sink where 
> paused connector produces messages
> ---
>
> Key: KAFKA-4527
> URL: https://issues.apache.org/jira/browse/KAFKA-4527
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect, system tests
>Reporter: Ewen Cheslack-Postava
>Assignee: Shikhar Bhushan
>  Labels: system-test-failure, system-tests
> Fix For: 0.10.2.0
>
>
> {quote}
> 
> test_id:
> kafkatest.tests.connect.connect_distributed_test.ConnectDistributedTest.test_pause_and_resume_sink
> status: FAIL
> run time:   40.164 seconds
> Paused sink connector should not consume any messages
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py",
>  line 123, in run
> data = self.run_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.6.0-py2.7.egg/ducktape/tests/runner_client.py",
>  line 176, in run_test
> return self.test_context.function(self.test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/connect/connect_distributed_test.py",
>  line 257, in test_pause_and_resume_sink
> assert num_messages == len(self.sink.received_messages()), "Paused sink 
> connector should not consume any messages"
> AssertionError: Paused sink connector should not consume any messages
> {quote}
> See one case here: 
> http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2016-12-12--001.1481535295--apache--trunk--62e043a/report.html
>  but it has also happened before, e.g. 
> http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2016-12-06--001.1481017508--apache--trunk--34aa538/report.html
> Thinking about the test, one simple possibility is that our approach to get 
> the number of messages produced/consumed during the test is flawed -- I think 
> we may not account for additional buffering between the connectors and the 
> process reading their output to determine what they have produced. However, 
> that's just a theory -- the minimal checking on the logs that I did didn't 
> reveal anything obviously wrong.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3808) Transient failure in ReplicaVerificationToolTest

[Roger Hoover (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15759266#comment-15759266
 ] 

Roger Hoover commented on KAFKA-3808:
-

Happened again:  
http://confluent-systest.s3-website-us-west-2.amazonaws.com/confluent-kafka-system-test-results/?prefix=2016-12-18--001.1482053747--apache--trunk--d6b0b52/



> Transient failure in ReplicaVerificationToolTest
> 
>
> Key: KAFKA-3808
> URL: https://issues.apache.org/jira/browse/KAFKA-3808
> Project: Kafka
>  Issue Type: Bug
>  Components: system tests
>Reporter: Geoff Anderson
>
> {code}
> test_id:
> 2016-05-29--001.kafkatest.tests.tools.replica_verification_test.ReplicaVerificationToolTest.test_replica_lags
> status: FAIL
> run time:   1 minute 9.231 seconds
> Timed out waiting to reach non-zero number of replica lags.
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/tools/replica_verification_test.py",
>  line 88, in test_replica_lags
> err_msg="Timed out waiting to reach non-zero number of replica lags.")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Timed out waiting to reach non-zero number of replica lags.
> {code}
> http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2016-05-29--001.1464540508--apache--trunk--404b696/report.html



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4166) TestMirrorMakerService.test_bounce transient system test failure

[Roger Hoover (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15759264#comment-15759264
 ] 

Roger Hoover commented on KAFKA-4166:
-

Similar failure: 
http://confluent-systest.s3-website-us-west-2.amazonaws.com/confluent-kafka-system-test-results/?prefix=2016-12-18--001.1482053747--apache--trunk--d6b0b52/

> TestMirrorMakerService.test_bounce transient system test failure
> 
>
> Key: KAFKA-4166
> URL: https://issues.apache.org/jira/browse/KAFKA-4166
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
>Assignee: Jason Gustafson
>  Labels: transient-system-test-failure
>
> We've only seen one failure so far and it's a timeout error so it could be an 
> environment issue. Filing it here so that we can track it in case there are 
> additional failures:
> {code}
> Module: kafkatest.tests.core.mirror_maker_test
> Class:  TestMirrorMakerService
> Method: test_bounce
> Arguments:
> {
>   "clean_shutdown": true,
>   "new_consumer": true,
>   "security_protocol": "SASL_SSL"
> }
> {code}
>  
> {code}
> test_id:
> 2016-09-12--001.kafkatest.tests.core.mirror_maker_test.TestMirrorMakerService.test_bounce.clean_shutdown=True.security_protocol=SASL_SSL.new_consumer=True
> status: FAIL
> run time:   3 minutes 30.354 seconds
> 
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape/mark/_mark.py",
>  line 331, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/core/mirror_maker_test.py",
>  line 178, in test_bounce
> self.run_produce_consume_validate(core_test_action=lambda: 
> self.bounce(clean_shutdown=clean_shutdown))
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/produce_consume_validate.py",
>  line 79, in run_produce_consume_validate
> raise e
> TimeoutError
> {code}
>  
> http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2016-09-12--001.1473700895--apache--trunk--a7ab9cb/TestMirrorMakerService/test_bounce/clean_shutdown=True.security_protocol=SASL_SSL.new_consumer=True.tgz



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4526) Transient failure in ThrottlingTest.test_throttled_reassignment

[Roger Hoover (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15759260#comment-15759260
 ] 

Roger Hoover commented on KAFKA-4526:
-

Thanks, [~apurva]

> Transient failure in ThrottlingTest.test_throttled_reassignment
> ---
>
> Key: KAFKA-4526
> URL: https://issues.apache.org/jira/browse/KAFKA-4526
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ewen Cheslack-Postava
>Assignee: Apurva Mehta
>  Labels: system-test-failure, system-tests
> Fix For: 0.10.2.0
>
>
> This test is seeing transient failures sometimes
> {quote}
> Module: kafkatest.tests.core.throttling_test
> Class:  ThrottlingTest
> Method: test_throttled_reassignment
> Arguments:
> {
>   "bounce_brokers": false
> }
> {quote}
> This happens with both bounce_brokers = true and false. Fails with
> {quote}
> AssertionError: 1646 acked message did not make it to the Consumer. They are: 
> 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19...plus 
> 1626 more. Total Acked: 174799, Total Consumed: 173153. We validated that the 
> first 1000 of these missing messages correctly made it into Kafka's data 
> files. This suggests they were lost on their way to the consumer.
> {quote}
> See 
> http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2016-12-12--001.1481535295--apache--trunk--62e043a/report.html
>  for an example.
> Note that there are a number of similar bug reports for different tests: 
> https://issues.apache.org/jira/issues/?jql=text%20~%20%22acked%20message%20did%20not%20make%20it%20to%20the%20Consumer%22%20and%20project%20%3D%20Kafka
>  I am wondering if we have a wrong ack setting somewhere that we should be 
> specifying as acks=all but is only defaulting to 0?
> It also seems interesting that the missing messages in these recent failures 
> seem to always start at 0...



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: log.flush.interval.messages setting of Kafka 0.9.0.0

[Json Tu]

Would be grateful to hear opinions from experts out there. Thanks in advance


> 在 2016年12月16日，下午6:17，Json Tu  写道：
> 
> Hi all,
>   we have a cluster of 0.9.0.0 with 3 nodes, we have a topic with 3 
> replicas, and send it with ack -1, our sending latency is avg 7ms. I prepare 
> to optimize performance of cluster through adjusting some params.
> we find our brokers has set config item as below,
>   log.flush.interval.messages=1
> and other relevant parameter is default, and I find the default value of 
> log.flush.interval.messages is LONG.MAX_VALUE, because of setting this config 
> will flush intiative that may affect performace . I wonder can I cancel this 
> config  item’s setting, and use default value.
> 
>   I think use default value may have two drawback as below.
>   1.recovery checkpoint can not be updated，so when load 
> segments，it will scan from begin to end.
>   2.it may lose data when leader partition’s broker’s vm is 
> restart，but I think 3 replicas can remedy this drawback if the network 
> between them is good.
> 
>   any suggestions? thank you

[jira] [Resolved] (KAFKA-3841) MirrorMaker topic renaming

[Ning Zhang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ning Zhang resolved KAFKA-3841.
---
Resolution: Won't Fix

> MirrorMaker topic renaming
> --
>
> Key: KAFKA-3841
> URL: https://issues.apache.org/jira/browse/KAFKA-3841
> Project: Kafka
>  Issue Type: New Feature
>  Components: tools
>Affects Versions: 0.10.0.0
>Reporter: Ning Zhang
>
> Our organization (walmart.com) has been a Kafka user since some years back 
> and MirrorMaker has been a convenient tool to bring our Kafka data from one 
> Kafka cluster to another cluster.
> In many our use cases, the mirrored topic from the source cluster may not 
> want to have the same name in the target cluster. This could be a valid 
> scenario when the same topic name already exists on the target cluster, or we 
> want to append the name of the data center to the topic name in the target 
> cluster, such as "grocery_items_mirror_sunnyvale", to explicitly identify the 
> source (e.g. sunnyvale) and nature (e.g. mirroring) of the topic.
> We have implemented the MirrorMaker topic renaming feature internally which 
> has been used for production over a couple of years. While keeping our 
> internal Kafka fork with the above "renaming" branch across version upgrade 
> does not cost us too much labor, we think it may be meaningful to contribute 
> back to the community so that potentially many people may have the similar 
> expectation and could benefit from this feature.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4187) Adding a flag to prefix topics with mirror maker

[James Cheng (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4187?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

James Cheng resolved KAFKA-4187.

Resolution: Resolved

[~m...@vrischmann.me] was able to use MirrorMakerMessageHandler to solve this. 
Marking as Resolved.


> Adding a flag to prefix topics with mirror maker
> 
>
> Key: KAFKA-4187
> URL: https://issues.apache.org/jira/browse/KAFKA-4187
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Affects Versions: 0.8.2.1, 0.9.0.1, 0.10.0.0, 0.10.0.1
>Reporter: Vincent Rischmann
>Priority: Minor
>
> So I have a setup where I need to mirror our production cluster to our 
> preproduction cluster, but can't use the original topic names.
> I've patched mirror maker to allow me to define a prefix for each topic and I 
> basically prefix everything with mirror_. I'm wondering if there's interest 
> for this feature upstream ?
> I have a patch available for Kafka 0.9.0.1 (what I'm using) and from what 
> I've seen it should apply well to Kafka 0.10.0.X too.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)