[jira] [Commented] (KAFKA-5269) TransactionBounceTest occasionally fails due to partition errors

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018245#comment-16018245
 ] 

ASF GitHub Bot commented on KAFKA-5269:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3094


> TransactionBounceTest occasionally fails due to partition errors
> 
>
> Key: KAFKA-5269
> URL: https://issues.apache.org/jira/browse/KAFKA-5269
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
>Priority: Blocker
>  Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> The test sometimes encounters a partition level error 
> `UNKNOWN_TOPIC_OR_PARTITION` for the output topic. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5269) TransactionBounceTest occasionally fails due to partition errors

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-5269:
---
   Resolution: Fixed
Fix Version/s: 0.11.0.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 3094
[https://github.com/apache/kafka/pull/3094]

> TransactionBounceTest occasionally fails due to partition errors
> 
>
> Key: KAFKA-5269
> URL: https://issues.apache.org/jira/browse/KAFKA-5269
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
>Priority: Blocker
>  Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> The test sometimes encounters a partition level error 
> `UNKNOWN_TOPIC_OR_PARTITION` for the output topic. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3094: KAFKA-5269: Correct handling of UNKNOWN_TOPIC_OR_P...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3094


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : kafka-trunk-jdk8 #1562

[Apache Jenkins Server]

See 

[jira] [Updated] (KAFKA-5296) Unable to write to some partitions of newly created topic in 10.2

[Abhisek Saikia (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5296?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Abhisek Saikia updated KAFKA-5296:
--
Description: 
We are using kafka 10.2 and the cluster was running fine for a month with 50 
topics and now we are having issue in producing message by creating new topics. 
The create topic command is successful but producers are throwing error while 
writing to some partitions. 

Error in producer-
java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for 
[topic1]-8: 30039 ms has passed since batch creation plus linger time
at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:70)
 ~[kafka-clients-0.10.2.0.jar:na]
at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:57)
 ~[kafka-clients-0.10.2.0.jar:na]
at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
 ~[kafka-clients-0.10.2.0.jar:na]

On the broker side, I don't see any topic-parition folder getting created for 
the broker who is the leader for the partition. 


While using 0.8 client, the write used to hang while it starts writing to the 
partition having this issue. With 10.2 it resolved the the producer hang issue
 

  was:
We are using kafka 10.2 and the cluster was running fine for a month wiht 50 
topics and now we are having issue in producing message by creating new topics. 
The create topic command is successful but producers are throwing error while 
writing to some partitions. 

Error in producer-
java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for 
[topic1]-8: 30039 ms has passed since batch creation plus linger time
at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:70)
 ~[kafka-clients-0.10.2.0.jar:na]
at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:57)
 ~[kafka-clients-0.10.2.0.jar:na]
at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
 ~[kafka-clients-0.10.2.0.jar:na]

On the broker side, I don't see any topic-parition folder getting created for 
the broker who is the leader for the partition. 


While using 0.8 client, the write used to hang while it starts writing to the 
partition having this issue. With 10.2 it resolved the the producer hang issue
 


> Unable to write to some partitions of newly created topic in 10.2
> -
>
> Key: KAFKA-5296
> URL: https://issues.apache.org/jira/browse/KAFKA-5296
> Project: Kafka
>  Issue Type: Bug
>Reporter: Abhisek Saikia
>
> We are using kafka 10.2 and the cluster was running fine for a month with 50 
> topics and now we are having issue in producing message by creating new 
> topics. The create topic command is successful but producers are throwing 
> error while writing to some partitions. 
> Error in producer-
> java.util.concurrent.ExecutionException: 
> org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for 
> [topic1]-8: 30039 ms has passed since batch creation plus linger time
>   at 
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:70)
>  ~[kafka-clients-0.10.2.0.jar:na]
>   at 
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:57)
>  ~[kafka-clients-0.10.2.0.jar:na]
>   at 
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
>  ~[kafka-clients-0.10.2.0.jar:na]
> On the broker side, I don't see any topic-parition folder getting created for 
> the broker who is the leader for the partition. 
> While using 0.8 client, the write used to hang while it starts writing to the 
> partition having this issue. With 10.2 it resolved the the producer hang issue
>  



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5296) Unable to write to some partitions of newly created topic in 10.2

[Abhisek Saikia (JIRA)]

Abhisek Saikia created KAFKA-5296:
-

 Summary: Unable to write to some partitions of newly created topic 
in 10.2
 Key: KAFKA-5296
 URL: https://issues.apache.org/jira/browse/KAFKA-5296
 Project: Kafka
  Issue Type: Bug
Reporter: Abhisek Saikia


We are using kafka 10.2 and the cluster was running fine for a month wiht 50 
topics and now we are having issue in producing message by creating new topics. 
The create topic command is successful but producers are throwing error while 
writing to some partitions. 

Error in producer-
java.util.concurrent.ExecutionException: 
org.apache.kafka.common.errors.TimeoutException: Expiring 1 record(s) for 
[topic1]-8: 30039 ms has passed since batch creation plus linger time
at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:70)
 ~[kafka-clients-0.10.2.0.jar:na]
at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:57)
 ~[kafka-clients-0.10.2.0.jar:na]
at 
org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
 ~[kafka-clients-0.10.2.0.jar:na]

On the broker side, I don't see any topic-parition folder getting created for 
the broker who is the leader for the partition. 


While using 0.8 client, the write used to hang while it starts writing to the 
partition having this issue. With 10.2 it resolved the the producer hang issue
 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5247) Consumer GroupCoordinator should continue to materialize committed offsets in offset order even for transactional offset commits

[Apurva Mehta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apurva Mehta updated KAFKA-5247:

Status: Patch Available  (was: Open)

> Consumer GroupCoordinator should continue to materialize committed offsets in 
> offset order even for transactional offset commits
> 
>
> Key: KAFKA-5247
> URL: https://issues.apache.org/jira/browse/KAFKA-5247
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
>Priority: Blocker
>  Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> In the TxnOffsetCommit patch, we thought it was ok for the group coordinator 
> to use "transaction order" semantics when updating the cache, but we weren't 
> thinking about the log cleaner.
> The log cleaner uses offset order when cleaning which means that the key with 
> the largest offset always wins. So if we use transaction order when 
> dynamically updating the cache, we will get different results from when we're 
> loading the cache (even if the loading logic also uses transaction order).
> The fix should be straightforward: we need to remember the offset in the 
> offsets topic of the offset that we cache. Then we only update it if the new 
> entry has a higher offset.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5247) Consumer GroupCoordinator should continue to materialize committed offsets in offset order even for transactional offset commits

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018186#comment-16018186
 ] 

ASF GitHub Bot commented on KAFKA-5247:
---

GitHub user apurvam opened a pull request:

https://github.com/apache/kafka/pull/3108

KAFKA-5247: Materialize committed offsets in offset order

With this patch, offset commits are always materialized according to the 
order of the commit records in the offsets topic. 

Before this patch, transactional offset commits were materialized in 
transaction order. However, the log cleaner will always preserve the record 
with the greatest offset. This meant that if there was a mix of offset commits 
from a consumer and a transactional producer, then it we would switch from 
transactional order to offset order after cleaning, resulting in an 
inconsistent state.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apurvam/kafka 
KAFKA-5247-materialize-committed-offsets-in-offset-order

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3108.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3108


commit f6efd565023d440c6d15091609442ff61ad6f85a
Author: Apurva Mehta 
Date:   2017-05-19T18:31:48Z

KAFKA-5247 materialize offset commits in offset order

Updated the GroupMetadata to keep track of the offset in the
__consumer_offsets topic for the commit record for a given offset commit.
We only update the offsets cache when a given offset is committed if the
offset of the commit record in the offsets topic is greater than the offset
of the existing materialized offset.

This way, if we have a mix of transactional and non transactional offset
commits for the same group, we will always materialize the offset
commtis in offset order.

commit 20ee45422130f197791600891a9872826d510ca7
Author: Apurva Mehta 
Date:   2017-05-19T22:35:27Z

Update the return values of the GroupMetadata.remove* methods

commit 2fd79d1680711cdd746233dfbeaea957e65e67d8
Author: Apurva Mehta 
Date:   2017-05-19T23:49:08Z

Minor cleanups and added unit tests

commit 7e5f2820809d9a085333e1fa97efd13207e5a4e0
Author: Apurva Mehta 
Date:   2017-05-20T00:02:13Z

Remove erroneous comment




> Consumer GroupCoordinator should continue to materialize committed offsets in 
> offset order even for transactional offset commits
> 
>
> Key: KAFKA-5247
> URL: https://issues.apache.org/jira/browse/KAFKA-5247
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
>Priority: Blocker
>  Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> In the TxnOffsetCommit patch, we thought it was ok for the group coordinator 
> to use "transaction order" semantics when updating the cache, but we weren't 
> thinking about the log cleaner.
> The log cleaner uses offset order when cleaning which means that the key with 
> the largest offset always wins. So if we use transaction order when 
> dynamically updating the cache, we will get different results from when we're 
> loading the cache (even if the loading logic also uses transaction order).
> The fix should be straightforward: we need to remember the offset in the 
> offsets topic of the offset that we cache. Then we only update it if the new 
> entry has a higher offset.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3108: KAFKA-5247: Materialize committed offsets in offse...

[apurvam]

GitHub user apurvam opened a pull request:

https://github.com/apache/kafka/pull/3108

KAFKA-5247: Materialize committed offsets in offset order

With this patch, offset commits are always materialized according to the 
order of the commit records in the offsets topic. 

Before this patch, transactional offset commits were materialized in 
transaction order. However, the log cleaner will always preserve the record 
with the greatest offset. This meant that if there was a mix of offset commits 
from a consumer and a transactional producer, then it we would switch from 
transactional order to offset order after cleaning, resulting in an 
inconsistent state.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apurvam/kafka 
KAFKA-5247-materialize-committed-offsets-in-offset-order

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3108.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3108


commit f6efd565023d440c6d15091609442ff61ad6f85a
Author: Apurva Mehta 
Date:   2017-05-19T18:31:48Z

KAFKA-5247 materialize offset commits in offset order

Updated the GroupMetadata to keep track of the offset in the
__consumer_offsets topic for the commit record for a given offset commit.
We only update the offsets cache when a given offset is committed if the
offset of the commit record in the offsets topic is greater than the offset
of the existing materialized offset.

This way, if we have a mix of transactional and non transactional offset
commits for the same group, we will always materialize the offset
commtis in offset order.

commit 20ee45422130f197791600891a9872826d510ca7
Author: Apurva Mehta 
Date:   2017-05-19T22:35:27Z

Update the return values of the GroupMetadata.remove* methods

commit 2fd79d1680711cdd746233dfbeaea957e65e67d8
Author: Apurva Mehta 
Date:   2017-05-19T23:49:08Z

Minor cleanups and added unit tests

commit 7e5f2820809d9a085333e1fa97efd13207e5a4e0
Author: Apurva Mehta 
Date:   2017-05-20T00:02:13Z

Remove erroneous comment




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3107: MINOR: improve descriptions of Streams reset tool ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3107


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Reg: [VOTE] KIP 157 - Add consumer config options to streams reset tool

[Matthias J. Sax]

Couple of follow ups.

The reset tool uses AdminClient, ZkUtils, and a KafkaConsumer
internally. Thus, I am wondering if we need the possibility to specify
configs for all of them?

The original JIRA reported, that the reset tool does not work for a
secured cluster, and thus, I doubt that consumer properties are
sufficient for resolve this.

Maybe some people that are more familiar with Kafka security can help
out here. I personally have only limited knowledge about this topic.


-Matthias



On 5/19/17 11:09 AM, BigData dev wrote:
> Thanks for the info, Matthias.
> 
> Regards,
> Bharat
> 
> 
> On Fri, May 19, 2017 at 10:25 AM, Matthias J. Sax 
> wrote:
> 
>> KIP-157 cannot be included in 0.11.0.0 anymore. KIP freeze date deadline
>> is strict.
>>
>> -Matthias
>>
>> On 5/19/17 10:15 AM, BigData dev wrote:
>>> Hi Matthias,
>>> I will start a new KIP for Kafka tools options to be a standard across
>> all
>>> tools shortly. But I think the KIP 157 for Kafka Streams, should be
>> needed
>>> for 0.11.0.0 release, (KIP freeze date is already over, but I think this
>> is
>>> minor code change in tools to add option to streams reset tool) as
>> without
>>> this consumer config options, it will not be possible to use the tool in
>> a
>>> secured environment. Please let me know your thoughts on this. If it
>> needs
>>> to be moved to next release, I will work on this as part of KIP 14.
>>>
>>> Thanks,
>>> Bharat
>>>
>>>
>>> On Fri, May 19, 2017 at 10:10 AM, Matthias J. Sax >>
>>> wrote:
>>>
 I double checked with Matthew Warhaftig (the original author of KIP-14)
 and he has not interest to continue the KIP atm.

 Thus, Bharat can continue the work on KIP-14. I think it would be best,
 to start a new DISCUSS thread after you update KIP-14.

 Thanks for your contributions!


 -Matthias


 On 5/17/17 12:56 PM, BigData dev wrote:
> Hi,
> When I was trying to find more info, there is already a proposed KIP
>> for
> this
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
 14+-+Tools+Standardization
>
>
> Thanks,
> Bharat
>
> On Wed, May 17, 2017 at 12:38 PM, BigData dev >>
> wrote:
>
>> Hi Ewen, Matthias,
>> For common configuration across all the tools, I will work on that as
 part
>> of other KIP by looking into all Kafka tools.
>>
>>
>> Thanks,
>> Bharat
>>
>>
>> On Wed, May 17, 2017 at 9:40 AM, Matthias J. Sax <
>> matth...@confluent.io
>
>> wrote:
>>
>>> +1
>>>
>>> I also second Ewen comment -- standardizing the common supported
>>> parameters over all tools would be great!
>>>
>>>
>>> -Matthias
>>>
>>> On 5/17/17 12:57 AM, Damian Guy wrote:
 +1

 On Wed, 17 May 2017 at 05:40 Ewen Cheslack-Postava <
>> e...@confluent.io
>
 wrote:

> +1 (binding)
>
> I mentioned this in the PR that triggered this:
>
>> KIP is accurate, though this is one of those things that we should
> probably get a KIP for a standard set of config options across all
>>> tools so
> additions like this can just fall under the umbrella of that KIP...
>
> I think it would be great if someone wrote up a small KIP providing
>>> some
> standardized settings that we could get future additions
 automatically
> umbrella'd under, e.g. no need to do a KIP if just adding a
>>> consumer.config
> or consumer-property config conforming to existing expectations for
>>> other
> tools. We could also standardize on a few other settings names that
 are
> inconsistent across different tools and set out a clear path
>> forward
>>> for
> future tools.
>
> I think I still have at least one open PR from when I first started
 on
>>> the
> project where I was trying to clean up some command line stuff to
>> be
>>> more
> consistent. This has been an issue for many years now...
>
> -Ewen
>
>
>
> On Tue, May 16, 2017 at 1:12 AM, Eno Thereska <
 eno.there...@gmail.com>
> wrote:
>
>> +1 thanks.
>>
>> Eno
>>> On 16 May 2017, at 04:20, BigData dev 
>>> wrote:
>>>
>>> Hi All,
>>> Given the simple and non-controversial nature of the KIP, I would
>>> like
> to
>>> start the voting process for KIP-157: Add consumer config options
 to
>>> streams reset tool
>>>
>>> *https://cwiki.apache.org/confluence/display/KAFKA/KIP+
>> 157+-+Add+consumer+config+options+to+streams+reset+tool
>>> > 157+-+Add+consumer+config+options+to+streams+reset+tool>*
>>>
>>>
>>> The vote will run for a minim

[jira] [Commented] (KAFKA-5063) Flaky ResetIntegrationTest.testReprocessingFromScratchAfterResetWithIntermediateUserTopic

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018135#comment-16018135
 ] 

Matthias J. Sax commented on KAFKA-5063:


[~damianguy] The error you report is a different stack trace, and it seems to 
have a different root cause. Maybe we should have a different Jira for this? 
Also, can you point to the build when this happened? The tool returned with 
error exit code and the exception should be in the logs:
{noformat}
 } catch (final Throwable e) {
exitCode = EXIT_CODE_ERROR;
System.err.println("ERROR: " + e.getMessage());
}
return extiCode;
{noformat}

Right now it's unclear what went wrong.

> Flaky 
> ResetIntegrationTest.testReprocessingFromScratchAfterResetWithIntermediateUserTopic
> -
>
> Key: KAFKA-5063
> URL: https://issues.apache.org/jira/browse/KAFKA-5063
> Project: Kafka
>  Issue Type: Bug
>  Components: streams, unit tests
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
> Fix For: 0.11.0.0
>
>
> {noformat}
> org.apache.kafka.streams.integration.ResetIntegrationTest > 
> testReprocessingFromScratchAfterResetWithIntermediateUserTopic FAILED
> java.lang.AssertionError: 
> Expected: <[KeyValue(2983939126775, 1), KeyValue(2983939126815, 1), 
> KeyValue(2983939126835, 1), KeyValue(2983939126795, 1), 
> KeyValue(2983939126815, 1), KeyValue(2983939126855, 1), 
> KeyValue(2983939126835, 1), KeyValue(2983939126795, 1), 
> KeyValue(2983939126815, 2), KeyValue(2983939126835, 2), 
> KeyValue(2983939126855, 1), KeyValue(2983939126875, 1), 
> KeyValue(2983939126835, 2), KeyValue(2983939126855, 2), 
> KeyValue(2983939126895, 1), KeyValue(2983939126875, 1), 
> KeyValue(2983939126855, 2), KeyValue(2983939126875, 2), 
> KeyValue(2983939126895, 1), KeyValue(2983939126915, 1), 
> KeyValue(2983939126875, 2), KeyValue(2983939126895, 2), 
> KeyValue(2983939126915, 1), KeyValue(2983939126935, 1), 
> KeyValue(2983939126875, 3), KeyValue(2983939126895, 2), 
> KeyValue(2983939126915, 2), KeyValue(2983939126935, 1), 
> KeyValue(2983939126875, 3), KeyValue(2983939126895, 3), 
> KeyValue(2983939126915, 2), KeyValue(2983939126935, 2), 
> KeyValue(2983939126875, 4), KeyValue(2983939126895, 3), 
> KeyValue(2983939126915, 3), KeyValue(2983939126935, 2), 
> KeyValue(2983939126875, 4), KeyValue(2983939126895, 4), 
> KeyValue(2983939126915, 3), KeyValue(2983939126935, 3)]>
>  but: was <[KeyValue(2983939126775, 1), KeyValue(2983939126815, 1), 
> KeyValue(2983939126835, 1), KeyValue(2983939126795, 1), 
> KeyValue(2983939126815, 1), KeyValue(2983939126855, 1), 
> KeyValue(2983939126835, 1), KeyValue(2983939126795, 1), 
> KeyValue(2983939126815, 2), KeyValue(2983939126835, 2), 
> KeyValue(2983939126855, 1), KeyValue(2983939126875, 1), 
> KeyValue(2983939126835, 2), KeyValue(2983939126855, 2), 
> KeyValue(2983939126895, 1), KeyValue(2983939126875, 1), 
> KeyValue(2983939126855, 2), KeyValue(2983939126875, 2), 
> KeyValue(2983939126895, 1), KeyValue(2983939126915, 1), 
> KeyValue(2983939126875, 2), KeyValue(2983939126895, 2), 
> KeyValue(2983939126915, 1), KeyValue(2983939126935, 1), 
> KeyValue(2983939126875, 3), KeyValue(2983939126895, 2), 
> KeyValue(2983939126915, 2), KeyValue(2983939126935, 1), 
> KeyValue(2983939126875, 3), KeyValue(2983939126895, 3), 
> KeyValue(2983939126915, 2), KeyValue(2983939126935, 2), 
> KeyValue(2983939126875, 4)]>
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:8)
> at 
> org.apache.kafka.streams.integration.ResetIntegrationTest.testReprocessingFromScratchAfterResetWithIntermediateUserTopic(ResetIntegrationTest.java:190)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5269) TransactionBounceTest occasionally fails due to partition errors

[Apurva Mehta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apurva Mehta updated KAFKA-5269:

Status: Patch Available  (was: Open)

> TransactionBounceTest occasionally fails due to partition errors
> 
>
> Key: KAFKA-5269
> URL: https://issues.apache.org/jira/browse/KAFKA-5269
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
>Priority: Blocker
>  Labels: exactly-once
>
> The test sometimes encounters a partition level error 
> `UNKNOWN_TOPIC_OR_PARTITION` for the output topic. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5266) Follow-up improvements for consumer offset reset tool (KIP-122)

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018106#comment-16018106
 ] 

Jason Gustafson commented on KAFKA-5266:


[~jeqo] Maybe it could be "--to-current," which sort of fits with the 
"--to-offset" option? On 4., filtering makes sense, but there will always be 
some case where some error pops up when we actually try to collect the offsets 
we want to commit. My point is just that we should raise a friendly error to 
the user when that happens. So all of those paths where we currently pass 
through null in {{prepareOffsetsToReset}} should raise some exception instead 
with a nice message.

> Follow-up improvements for consumer offset reset tool (KIP-122)
> ---
>
> Key: KAFKA-5266
> URL: https://issues.apache.org/jira/browse/KAFKA-5266
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Jason Gustafson
>Assignee: Jorge Quilcate
> Fix For: 0.11.0.0
>
>
> 1. We should try to ensure that offsets are in range for the topic partition. 
> We currently only verify this for the shift option.
> 2. If you provide a CSV file, you shouldn't need to specify one of the 
> --all-topics or --topic options.
> 3. We currently support a "reset to current offsets" option if none of the 
> supported reset options are provided. This seems kind of useless. Perhaps we 
> should just enforce that one of the reset options is provided.
> 4. The command fails with an NPE if we cannot find one of the offsets we are 
> trying to reset. It would be better to raise an exception with a friendlier 
> message.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (KAFKA-5266) Follow-up improvements for consumer offset reset tool (KIP-122)

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018106#comment-16018106
 ] 

Jason Gustafson edited comment on KAFKA-5266 at 5/19/17 10:14 PM:
--

[~jeqo] Maybe it could be {{--to-current}}, which sort of fits with the 
{{--to-offset}} option? On 4., filtering makes sense, but there will always be 
some case where some error pops up when we actually try to collect the offsets 
we want to commit. My point is just that we should raise a friendly error to 
the user when that happens. So all of those paths where we currently pass 
through null in {{prepareOffsetsToReset}} should raise some exception instead 
with a nice message.


was (Author: hachikuji):
[~jeqo] Maybe it could be "--to-current," which sort of fits with the 
"--to-offset" option? On 4., filtering makes sense, but there will always be 
some case where some error pops up when we actually try to collect the offsets 
we want to commit. My point is just that we should raise a friendly error to 
the user when that happens. So all of those paths where we currently pass 
through null in {{prepareOffsetsToReset}} should raise some exception instead 
with a nice message.

> Follow-up improvements for consumer offset reset tool (KIP-122)
> ---
>
> Key: KAFKA-5266
> URL: https://issues.apache.org/jira/browse/KAFKA-5266
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Jason Gustafson
>Assignee: Jorge Quilcate
> Fix For: 0.11.0.0
>
>
> 1. We should try to ensure that offsets are in range for the topic partition. 
> We currently only verify this for the shift option.
> 2. If you provide a CSV file, you shouldn't need to specify one of the 
> --all-topics or --topic options.
> 3. We currently support a "reset to current offsets" option if none of the 
> supported reset options are provided. This seems kind of useless. Perhaps we 
> should just enforce that one of the reset options is provided.
> 4. The command fails with an NPE if we cannot find one of the offsets we are 
> trying to reset. It would be better to raise an exception with a friendlier 
> message.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5259) TransactionalId authorization should imply ProducerId authorization

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-5259:
---
Status: Patch Available  (was: In Progress)

> TransactionalId authorization should imply ProducerId authorization
> ---
>
> Key: KAFKA-5259
> URL: https://issues.apache.org/jira/browse/KAFKA-5259
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Blocker
>  Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> There is not much point to only authorizing a transactionalId: without 
> producerId authorization, a principal cannot actually write any transactional 
> data. So we may as well make ProducerId authorization implicit if a 
> transactionalId is authorized. 
> There are also a couple cases that we missed in the initial authorization 
> patch which we may as well handle here.
> 1. FindCoordinatorRequest should authorize by transactionalId
> 2. TxnOffsetCommitRequest should also authorize by transactionalId. Currently 
> this field is not included in the request type but it probably should be 
> since then writing any transactional data requires authorization to some 
> transactionalId.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Work started] (KAFKA-5259) TransactionalId authorization should imply ProducerId authorization

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5259?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-5259 started by Jason Gustafson.
--
> TransactionalId authorization should imply ProducerId authorization
> ---
>
> Key: KAFKA-5259
> URL: https://issues.apache.org/jira/browse/KAFKA-5259
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Blocker
>  Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> There is not much point to only authorizing a transactionalId: without 
> producerId authorization, a principal cannot actually write any transactional 
> data. So we may as well make ProducerId authorization implicit if a 
> transactionalId is authorized. 
> There are also a couple cases that we missed in the initial authorization 
> patch which we may as well handle here.
> 1. FindCoordinatorRequest should authorize by transactionalId
> 2. TxnOffsetCommitRequest should also authorize by transactionalId. Currently 
> this field is not included in the request type but it probably should be 
> since then writing any transactional data requires authorization to some 
> transactionalId.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3107: MINOR: improve descriptions of Streams reset tool ...

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/3107

MINOR: improve descriptions of Streams reset tool options



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka minor-reset-tool-options

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3107.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3107


commit 117b59fbb6097212dfa1519dd46b3c2950805a42
Author: Matthias J. Sax 
Date:   2017-05-19T21:52:43Z

MINOR: improve descriptions of Streams reset tool options




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4785) Records from internal repartitioning topics should always use RecordMetadataTimestampExtractor

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018071#comment-16018071
 ] 

ASF GitHub Bot commented on KAFKA-4785:
---

GitHub user jeyhunkarimov opened a pull request:

https://github.com/apache/kafka/pull/3106

KAFKA-4785: Records from internal repartitioning topics should always use 
RecordMetadataTimestampExtractor



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jeyhunkarimov/kafka KAFKA-4785

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3106.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3106


commit 98278baae63638efc2b750b1d3f12e936d845f18
Author: Jeyhun Karimov 
Date:   2017-05-19T21:39:50Z

Records from internal repartitioning topics should always use 
RecordMetadataTimestampExtractor




> Records from internal repartitioning topics should always use 
> RecordMetadataTimestampExtractor
> --
>
> Key: KAFKA-4785
> URL: https://issues.apache.org/jira/browse/KAFKA-4785
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Matthias J. Sax
>Assignee: Jeyhun Karimov
>
> Users can specify what timestamp extractor should be used to decode the 
> timestamp of input topic records. As long as RecordMetadataTimestamp or 
> WallclockTime is use this is fine. 
> However, for custom timestamp extractors it might be invalid to apply this 
> custom extractor to records received from internal repartitioning topics. The 
> reason is that Streams sets the current "stream time" as record metadata 
> timestamp explicitly before writing to intermediate repartitioning topics 
> because this timestamp should be use by downstream subtopologies. A custom 
> timestamp extractor might return something different breaking this assumption.
> Thus, for reading data from intermediate repartitioning topic, the configured 
> timestamp extractor should not be used, but the record's metadata timestamp 
> should be extracted as record timestamp.
> In order to leverage the same behavior for intermediate user topic (ie, used 
> in {{through()}})  we can leverage KAFKA-4144 and internally set an extractor 
> for those "intermediate sources" that returns the record's metadata timestamp 
> in order to overwrite the global extractor from {{StreamsConfig}} (ie, set 
> {{FailOnInvalidTimestampExtractor}}).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3106: KAFKA-4785: Records from internal repartitioning t...

[jeyhunkarimov]

GitHub user jeyhunkarimov opened a pull request:

https://github.com/apache/kafka/pull/3106

KAFKA-4785: Records from internal repartitioning topics should always use 
RecordMetadataTimestampExtractor



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jeyhunkarimov/kafka KAFKA-4785

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3106.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3106


commit 98278baae63638efc2b750b1d3f12e936d845f18
Author: Jeyhun Karimov 
Date:   2017-05-19T21:39:50Z

Records from internal repartitioning topics should always use 
RecordMetadataTimestampExtractor




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-5191) Autogenerate Consumer Fetcher metrics

[James Cheng (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5191?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018069#comment-16018069
 ] 

James Cheng commented on KAFKA-5191:


Pinging [~ijuma] [~hachikuji] [~benstopford] for a code review. If there is 
someone else we should be reviewing this instead, please let me know.

> Autogenerate Consumer Fetcher metrics
> -
>
> Key: KAFKA-5191
> URL: https://issues.apache.org/jira/browse/KAFKA-5191
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: James Cheng
>Assignee: James Cheng
> Attachments: docs_now_include_partition_level_metrics.png, 
> generated_fetcher_docs.png, generated_fetcher_docs_with_alternate_css.png, 
> generated_fetcher_docs_with_css.png
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: KTable aggregate corruption on poorly timed exception

[Matthias J. Sax]

Hi Mathieu,

Thanks for reporting this!

This is definitely a bug and it must get fixed for at-last-once
processing, too. Exactly-once is not required to avoid the bug.

Can you open a Jira? I think, I know already how to fix it.


-Matthias


On 5/19/17 8:53 AM, Mathieu Fenniak wrote:
> Whoops, I said I'd put the specific exception at the bottom of the
> e-mail.  It probably isn't the important part of this thread, but
> might suggest when this situation can occur.  Also of note, this is
> occurring on Kafka Streams 0.10.2.1.
> 
> 
> 20:56:07.061 [StreamThread-3] ERROR o.a.k.s.p.internals.StreamThread -
> stream-thread [StreamThread-3] Failed to commit StreamTask 0_4 state:
> org.apache.kafka.streams.errors.StreamsException: task [0_4] exception
> caught when producing
> at 
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.checkForException(RecordCollectorImpl.java:121)
> at 
> org.apache.kafka.streams.processor.internals.RecordCollectorImpl.flush(RecordCollectorImpl.java:129)
> at 
> org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamTask.java:76)
> at 
> org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:188)
> at 
> org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:280)
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.commitOne(StreamThread.java:807)
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.commitAll(StreamThread.java:794)
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.maybeCommit(StreamThread.java:769)
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:647)
> at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:361)
> Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to
> update metadata after 6 ms.
> 
> On Fri, May 19, 2017 at 9:47 AM, Mathieu Fenniak
>  wrote:
>> Hi Kafka devs,
>>
>> This morning I observed a specific Kafka Streams aggregation that
>> ended up with an incorrect computed output after a Kafka Streams
>> thread crashed with an unhandled exception.
>>
>> The topology is pretty simple -- a single KTable source, group by a
>> field in the value, aggregate that adds up another field, output to a
>> topic.
>>
>> Here's the sequence of events that appears to have occurred:
>>
>> 1. A new record (record A) is received by the source KTable, and put
>> in the KTable RocksDB state store.
>>
>> 2. While processing record A, an exception happens preventing
>> producing to Kafka. (specific exception at end of e-mail).
>>
>> 3. The stream thread throws an unhandled exception and stops.
>>
>> 4. The state stores are closed and flushed.  Record A is now in the
>> local state store.
>>
>> 5. The consumer group rebalances.
>>
>> 6. A different thread, in the same process, on the same host, picks up the 
>> task.
>>
>> 7. New thread initializes its state store for the KTable, but it's on
>> the same host as the original thread, so it still contains the k/v for
>> record A.
>>
>> 8. New thread resumes consuming at the last committed offset, which is
>> before record A.
>>
>> 9. When processing record A, the new thread reads the value that was
>> written to the state store in step #1 by record A's key.
>>
>> 10. The repartition map receives a Change with both an oldValue and a
>> newValue, and forwards a Change(null, v) and Change(v, null)
>>
>> 11. The aggregation ends up both subtracting and adding the value of
>> record A, resulting in an incorrect output.
>>
>> As a result of this sequence, my aggregate output went from a value of
>> 0, to negative (subtracting record A), to 0.  And stayed there.
>>
>> Does this seem like a feasible series of events?  Is this a bug in KS,
>> or, is it behavior that maybe can't be improved without exactly-once?
>> I'd think the best behavior would be for writes to the RocksDB state
>> store to be transactional and only commit when the producer commits,
>> but, there's a lot of overhead involved in that.
>>
>> Mathieu



signature.asc
Description: OpenPGP digital signature

[jira] [Comment Edited] (KAFKA-5266) Follow-up improvements for consumer offset reset tool (KIP-122)

[Jorge Quilcate (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018004#comment-16018004
 ] 

Jorge Quilcate edited comment on KAFKA-5266 at 5/19/17 9:04 PM:


[~hachikuji] Thanks!

For case 3, maybe `now` could be less verbose but I like `current` also. I'd 
avoid adding `*-offsets` to the option name because it is kind of implicit from 
the main `reset-offsets`. But definitely it is better having an explicit 
option.  Any preference between `now` and `current`? 

For 4, this will be possible only when you set the topic partition manually, 
right? What if we filter the topicPartitions from allTopics, so if you put some 
unassigned topic partition you will get an empty list of topicPartitions from 
#getPartitionsToReset. Check it here: 
https://github.com/jeqo/kafka/blob/f1a4c5236984de348d4928450c5c0ad5112b6012/core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala#L548
 


was (Author: jeqo):
[~hachikuji] Thanks!

For case 3, maybe `--now` could be less verbose but I like `--current` also. 
I'd avoid adding `*-offsets` to the option name because it is kind of implicit 
from the main `reset-offsets`. But definitely it is better having an explicit 
option.  Any preference between `--now` and `--current`? 

For 4, this will be possible only when you set the topic partition manually, 
right? What if we filter the topicPartitions from allTopics, so if you put some 
unassigned topic partition you will get an empty list of topicPartitions from 
#getPartitionsToReset. Check it here: 
https://github.com/jeqo/kafka/blob/f1a4c5236984de348d4928450c5c0ad5112b6012/core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala#L548
 

> Follow-up improvements for consumer offset reset tool (KIP-122)
> ---
>
> Key: KAFKA-5266
> URL: https://issues.apache.org/jira/browse/KAFKA-5266
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Jason Gustafson
>Assignee: Jorge Quilcate
> Fix For: 0.11.0.0
>
>
> 1. We should try to ensure that offsets are in range for the topic partition. 
> We currently only verify this for the shift option.
> 2. If you provide a CSV file, you shouldn't need to specify one of the 
> --all-topics or --topic options.
> 3. We currently support a "reset to current offsets" option if none of the 
> supported reset options are provided. This seems kind of useless. Perhaps we 
> should just enforce that one of the reset options is provided.
> 4. The command fails with an NPE if we cannot find one of the offsets we are 
> trying to reset. It would be better to raise an exception with a friendlier 
> message.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Work started] (KAFKA-5266) Follow-up improvements for consumer offset reset tool (KIP-122)

[Jorge Quilcate (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-5266 started by Jorge Quilcate.
-
> Follow-up improvements for consumer offset reset tool (KIP-122)
> ---
>
> Key: KAFKA-5266
> URL: https://issues.apache.org/jira/browse/KAFKA-5266
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Jason Gustafson
>Assignee: Jorge Quilcate
> Fix For: 0.11.0.0
>
>
> 1. We should try to ensure that offsets are in range for the topic partition. 
> We currently only verify this for the shift option.
> 2. If you provide a CSV file, you shouldn't need to specify one of the 
> --all-topics or --topic options.
> 3. We currently support a "reset to current offsets" option if none of the 
> supported reset options are provided. This seems kind of useless. Perhaps we 
> should just enforce that one of the reset options is provided.
> 4. The command fails with an NPE if we cannot find one of the offsets we are 
> trying to reset. It would be better to raise an exception with a friendlier 
> message.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5266) Follow-up improvements for consumer offset reset tool (KIP-122)

[Jorge Quilcate (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018004#comment-16018004
 ] 

Jorge Quilcate commented on KAFKA-5266:
---

[~hachikuji] Thanks!

For case 3, maybe `--now` could be less verbose but I like `--current` also. 
I'd avoid adding `*-offsets` to the option name because it is kind of implicit 
from the main `reset-offsets`. But definitely it is better having an explicit 
option.  Any preference between `--now` and `--current`? 

For 4, this will be possible only when you set the topic partition manually, 
right? What if we filter the topicPartitions from allTopics, so if you put some 
unassigned topic partition you will get an empty list of topicPartitions from 
#getPartitionsToReset. Check it here: 
https://github.com/jeqo/kafka/blob/f1a4c5236984de348d4928450c5c0ad5112b6012/core/src/main/scala/kafka/admin/ConsumerGroupCommand.scala#L548
 

> Follow-up improvements for consumer offset reset tool (KIP-122)
> ---
>
> Key: KAFKA-5266
> URL: https://issues.apache.org/jira/browse/KAFKA-5266
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Jason Gustafson
>Assignee: Jorge Quilcate
> Fix For: 0.11.0.0
>
>
> 1. We should try to ensure that offsets are in range for the topic partition. 
> We currently only verify this for the shift option.
> 2. If you provide a CSV file, you shouldn't need to specify one of the 
> --all-topics or --topic options.
> 3. We currently support a "reset to current offsets" option if none of the 
> supported reset options are provided. This seems kind of useless. Perhaps we 
> should just enforce that one of the reset options is provided.
> 4. The command fails with an NPE if we cannot find one of the offsets we are 
> trying to reset. It would be better to raise an exception with a friendlier 
> message.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

[Matthias J. Sax]

With the current KIP, using ValueMapper and ValueMapperWithKey
interfaces, RichFunction seems to be an independent add-on. To fix the
original issue to allow key access, RichFunctions are not required IMHO.

I initially put the RichFunction idea on the table, because I was hoping
to get a uniform API. And I think, is was good to consider them --
however, the discussion showed that they are not necessary for key
access. Thus, it seems to be better to handle RichFunctions in an own
KIP. The ProcessorContext/RecordContext issues seems to be a main
challenge for this. And introducing RichFunctions with parameter-less
init() method, seem not to help too much. We would get an "intermediate"
API that we want to change anyway later on...

As you put already much effort into RichFunction, feel free do push this
further and start a new KIP (we can do this even in parallel) -- we
don't want to slow you down :) But it make the discussion and code
review easier, if we separate both IMHO.


-Matthias


On 5/19/17 2:25 AM, Jeyhun Karimov wrote:
> Hi Damian,
> 
> Thanks for your comments. I think providing to users *interface* rather
> than *abstract class* should be preferred (Matthias also raised this issue
> ), anyway I changed the corresponding parts of KIP.
> 
> Regarding with passing additional contextual information, I think it is a
> tradeoff,
> 1) first, we fix the context parameter for *init() *method in another PR
> and solve Rich functions afterwards
> 2) first, we fix the requested issues on jira ([1-3]) with providing
> (not-complete) Rich functions and integrate the context parameters to this
> afterwards (like improvement)
> 
> To me, the second approach seems more incremental. However you are right,
> the names might confuse the users.
> 
> 
> 
> [1] https://issues.apache.org/jira/browse/KAFKA-4218
> [2] https://issues.apache.org/jira/browse/KAFKA-4726
> [3] https://issues.apache.org/jira/browse/KAFKA-3745
> 
> 
> Cheers,
> Jeyhun
> 
> 
> On Fri, May 19, 2017 at 10:42 AM Damian Guy  wrote:
> 
>> Hi,
>>
>> I see you've removed the `ProcessorContext` from the RichFunction which is
>> good, but why is it a `RichFunction`? I'd have expected it to pass some
>> additional contextual information, i.e., the `RecordContext` that contains
>> just the topic, partition, timestamp, offset.  I'm ok with it not passing
>> this contextual information, but is the naming incorrect? I'm not sure,
>> tbh. I'm wondering if we should drop `RichFunctions` until we can do it
>> properly with the correct context?
>>
>> Also, i don't like the abstract classes: RichValueMapper, RichValueJoiner,
>> RichInitializer etc. Why can't they not just be interfaces? Generally we
>> should provide users with Intefaces to code against, not classes.
>>
>> Thanks,
>> Damian
>>
>> On Fri, 19 May 2017 at 00:50 Jeyhun Karimov  wrote:
>>
>>> Hi,
>>>
>>> Thanks. I initiated the PR as well, to get a better overview.
>>>
>>> The only reason that I used abstract class instead of interface for Rich
>>> functions is that in future if we will have some AbstractRichFunction
>>> abstract classes,
>>> we can easily extend:
>>>
>>> public abstract class RichValueMapper  implements
>>> ValueMapperWithKey, RichFunction *extends
>> AbstractRichFunction*{
>>> }
>>>  With interfaces we are only limited to interfaces for inheritance.
>>>
>>> However, I think we can easily change it (from abstract class ->
>> interface)
>>> if you think interface is a better fit.
>>>
>>>
>>> Cheers,
>>> Jeyhun
>>>
>>>
>>> On Fri, May 19, 2017 at 1:00 AM Matthias J. Sax 
>>> wrote:
>>>
 Thanks for the update and explanations. The KIP is quite improved now
>> --
 great job!

 One more question: Why are RichValueMapper etc abstract classes and not
 interfaces?


 Overall, I like the KIP a lot!


 -Matthias


 On 5/16/17 7:03 AM, Jeyhun Karimov wrote:
> Hi,
>
> Thanks for your comments.
>
> I think supporting Lambdas for `withKey` and `AbstractRichFunction`
>> don't go together, as Lambdas are only supported for interfaces
>> AFAIK.
>
>
> Maybe I misunderstood your comment.
> *withKey* and and *withOnlyValue* are interfaces. So they don't have
 direct
> relation with *AbstractRichFunction*.
> *withKey* and and *withOnlyValue* interfaces have only one  method ,
>> so
 we
> can use lambdas.
> Where does the *AbstractRichFunction* comes to play? Inside Rich
 functions.
> And we use Rich functions in 2 places:
>
> 1. User doesn't use rich functions. Just regular *withKey* and and
> *withOnlyValue* interfaces(both support lambdas) . We get those
 interfaces
> and wrap into Rich function while building the topology, and send to
> Processor.
> 2. User does use rich functions (Rich functions implement *withKey*
> interface). As a result no lamdas here by definition. In this case,
>>> while
> building the topology we do a type check if

[jira] [Commented] (KAFKA-5294) PlainSaslServerFactory should allow a null Map in getMechanismNames

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017935#comment-16017935
 ] 

ASF GitHub Bot commented on KAFKA-5294:
---

GitHub user mimaison opened a pull request:

https://github.com/apache/kafka/pull/3105

KAFKA-5294: PlainSaslServerFactory should allow a null Map in getMech…

…anismNames

If props is null, use POLICY_NOPLAINTEXT default value: false

As far as I can tell, none of the other classes implementing 
SaslServerFactory use the properties Map

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mimaison/kafka KAFKA-5294

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3105.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3105


commit 432c48c3f49ccc3090646671bdba82a2ef05b839
Author: Mickael Maison 
Date:   2017-05-19T19:38:13Z

KAFKA-5294: PlainSaslServerFactory should allow a null Map in 
getMechanismNames

If props is null, use POLICY_NOPLAINTEXT default value: false




> PlainSaslServerFactory should allow a null Map in getMechanismNames
> ---
>
> Key: KAFKA-5294
> URL: https://issues.apache.org/jira/browse/KAFKA-5294
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.1
>Reporter: Bryan Bende
> Fix For: 0.11.0.0
>
>
> PlainSaslServerFactory will get a NullPointerException if 
> getMechanismNames(Map props) is called with a null props map.
> The Java docs for getMechanismNames from the base class 
> javax.security.sasl.SaslServerFactory say that the map can be null:
> * @param props The possibly null set of properties used to specify the
>  * security policy of the SASL mechanisms.
> A scenario where this causes a problem is when attempting to use a Kafka 
> consumer/producer with the PlainLoginModule in the same JVM as a Hadoop 
> client with Kerberos...
> The Hadoop client purposely calls getMechanismNames with a null map, so if 
> you have already create a Kafka consumer/producer with PlainLoginModule then 
> the PlainSaslServerFactory will be registered in the list of providers in 
> java.security.Security, and then NPEs when the Hadoop client calls 
> getMechanismNames on each provider with a null map.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3105: KAFKA-5294: PlainSaslServerFactory should allow a ...

[mimaison]

GitHub user mimaison opened a pull request:

https://github.com/apache/kafka/pull/3105

KAFKA-5294: PlainSaslServerFactory should allow a null Map in getMech…

…anismNames

If props is null, use POLICY_NOPLAINTEXT default value: false

As far as I can tell, none of the other classes implementing 
SaslServerFactory use the properties Map

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mimaison/kafka KAFKA-5294

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3105.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3105


commit 432c48c3f49ccc3090646671bdba82a2ef05b839
Author: Mickael Maison 
Date:   2017-05-19T19:38:13Z

KAFKA-5294: PlainSaslServerFactory should allow a null Map in 
getMechanismNames

If props is null, use POLICY_NOPLAINTEXT default value: false




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : kafka-trunk-jdk7 #2234

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk8 #1561

[Apache Jenkins Server]

See 


Changes:

[jason] KAFKA-4714; TimestampConverter transformation (KIP-66)

--
[...truncated 883.70 KB...]
kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testTopicAcl PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testSuperUserHasAccess STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testSuperUserHasAccess PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testDenyTakesPrecedence STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testDenyTakesPrecedence PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFoundOverride STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testNoAclFoundOverride PASSED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyModificationOfResourceAcls STARTED

kafka.security.auth.SimpleAclAuthorizerTest > 
testHighConcurrencyModificationOfResourceAcls PASSED

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache STARTED

kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete STARTED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooHigh 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooHigh 
PASSED

kafka.integration.TopicMetadataTest > testIsrAfterBrokerShutDownAndJoinsBack 
STARTED

kafka.integration.TopicMetadataTest > testIsrAfterBrokerShutDownAndJoinsBack 
FAILED
java.lang.AssertionError: Topic metadata is not correctly updated for 
broker kafka.server.KafkaServer@6f405f35.
Expected ISR: List(BrokerEndPoint(0,localhost,49525), 
BrokerEndPoint(1,localhost,51890))
Actual ISR  : 
at kafka.utils.TestUtils$.fail(TestUtils.scala:329)
at kafka.utils.TestUtils$.waitUntilTrue(TestUtils.scala:839)
at 
kafka.integration.TopicMetadataTest$$anonfun$checkIsr$1.apply(TopicMetadataTest.scala:194)
at 
kafka.integration.TopicMetadataTest$$anonfun$checkIsr$1.apply(TopicMetadataTest.scala:192)
at scala.collection.immutable.List.foreach(List.scala:392)
at 
kafka.integration.TopicMetadataTest.checkIsr(TopicMetadataTest.scala:192)
at 
kafka.integration.TopicMetadataTest.testIsrAfterBrokerShutDownAndJoinsBack(TopicMetadataTest.scala:234)

kafka.integration.TopicMetadataTest > testAutoCreateTopicWithCollision STARTED

kafka.integration.TopicMetadataTest > testAutoCreateTopicWithCollision PASSED

kafka.integration.TopicMetadataTest > testAliveBrokerListWithNoTopics STARTED

kafka.integration.TopicMetadataTest > testAliveBrokerListWithNoTopics PASSED

kafka.integration.TopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.TopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.TopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.TopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.TopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.TopicMetadat

[jira] [Commented] (KAFKA-3821) Allow Kafka Connect source tasks to produce offset without writing to topics

[kyle k (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017884#comment-16017884
 ] 

kyle k commented on KAFKA-3821:
---

As a potential solution why can't the {{OffsetStorageWriter}}'s {{void 
offset(Map partition, Map offset)}} function be made 
available on the {{SourceTaskContext}}? This seems to me as if it would work 
out of the box... or am I missing something? 

Additionally offsets need to be written in {{WorkerSourceTask}} after 
{{transformationChain.apply}} if it returns {{null}} indicating that the record 
should be filtered.

> Allow Kafka Connect source tasks to produce offset without writing to topics
> 
>
> Key: KAFKA-3821
> URL: https://issues.apache.org/jira/browse/KAFKA-3821
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Affects Versions: 0.9.0.1
>Reporter: Randall Hauch
>  Labels: needs-kip
>
> Provide a way for a {{SourceTask}} implementation to record a new offset for 
> a given partition without necessarily writing a source record to a topic.
> Consider a connector task that uses the same offset when producing an unknown 
> number of {{SourceRecord}} objects (e.g., it is taking a snapshot of a 
> database). Once the task completes those records, the connector wants to 
> update the offsets (e.g., the snapshot is complete) but has no more records 
> to be written to a topic. With this change, the task could simply supply an 
> updated offset.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5266) Follow-up improvements for consumer offset reset tool (KIP-122)

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017870#comment-16017870
 ] 

Jason Gustafson commented on KAFKA-5266:


[~jeqo] Thanks for explaining the case for 3. I guess that makes sense. Would 
it make sense to give this behavior an explicit option (e.g. 
--current-offsets)? For 4., an easy way to reproduce it is set the offset for a 
non-existing topic partition. You'll get something like this:
{code}
Error: Executing consumer group command failed due to null
{code}

> Follow-up improvements for consumer offset reset tool (KIP-122)
> ---
>
> Key: KAFKA-5266
> URL: https://issues.apache.org/jira/browse/KAFKA-5266
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Jason Gustafson
>Assignee: Jorge Quilcate
> Fix For: 0.11.0.0
>
>
> 1. We should try to ensure that offsets are in range for the topic partition. 
> We currently only verify this for the shift option.
> 2. If you provide a CSV file, you shouldn't need to specify one of the 
> --all-topics or --topic options.
> 3. We currently support a "reset to current offsets" option if none of the 
> supported reset options are provided. This seems kind of useless. Perhaps we 
> should just enforce that one of the reset options is provided.
> 4. The command fails with an NPE if we cannot find one of the offsets we are 
> trying to reset. It would be better to raise an exception with a friendlier 
> message.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3104: MINOR: improve EmbeddedKafkaCluster test utility f...

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/3104

MINOR: improve EmbeddedKafkaCluster test utility for deleting topics



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka 
minor-improve-embedded-kafka-cluster

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3104.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3104


commit 9797ca5225d2c67c33c0b18e6ef0f5480c7ab37a
Author: Matthias J. Sax 
Date:   2017-05-19T17:59:21Z

MINOR: improve EmbeddedKafkaCluster test utility for deleting topics




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4714) Implement remaining KIP-66 SMTs

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017806#comment-16017806
 ] 

ASF GitHub Bot commented on KAFKA-4714:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3065


> Implement remaining KIP-66 SMTs
> ---
>
> Key: KAFKA-4714
> URL: https://issues.apache.org/jira/browse/KAFKA-4714
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Ewen Cheslack-Postava
>Assignee: Ewen Cheslack-Postava
> Fix For: 0.11.0.0
>
>
> Three didn't make it for the 0.10.2.0 release: Flatten, Cast, and 
> TimestampConverter.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3065: KAFKA-4714: TimestampConverter transformation (KIP...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3065


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Reg: [VOTE] KIP 157 - Add consumer config options to streams reset tool

[BigData dev]

Thanks for the info, Matthias.

Regards,
Bharat


On Fri, May 19, 2017 at 10:25 AM, Matthias J. Sax 
wrote:

> KIP-157 cannot be included in 0.11.0.0 anymore. KIP freeze date deadline
> is strict.
>
> -Matthias
>
> On 5/19/17 10:15 AM, BigData dev wrote:
> > Hi Matthias,
> > I will start a new KIP for Kafka tools options to be a standard across
> all
> > tools shortly. But I think the KIP 157 for Kafka Streams, should be
> needed
> > for 0.11.0.0 release, (KIP freeze date is already over, but I think this
> is
> > minor code change in tools to add option to streams reset tool) as
> without
> > this consumer config options, it will not be possible to use the tool in
> a
> > secured environment. Please let me know your thoughts on this. If it
> needs
> > to be moved to next release, I will work on this as part of KIP 14.
> >
> > Thanks,
> > Bharat
> >
> >
> > On Fri, May 19, 2017 at 10:10 AM, Matthias J. Sax  >
> > wrote:
> >
> >> I double checked with Matthew Warhaftig (the original author of KIP-14)
> >> and he has not interest to continue the KIP atm.
> >>
> >> Thus, Bharat can continue the work on KIP-14. I think it would be best,
> >> to start a new DISCUSS thread after you update KIP-14.
> >>
> >> Thanks for your contributions!
> >>
> >>
> >> -Matthias
> >>
> >>
> >> On 5/17/17 12:56 PM, BigData dev wrote:
> >>> Hi,
> >>> When I was trying to find more info, there is already a proposed KIP
> for
> >>> this
> >>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> 14+-+Tools+Standardization
> >>>
> >>>
> >>> Thanks,
> >>> Bharat
> >>>
> >>> On Wed, May 17, 2017 at 12:38 PM, BigData dev  >
> >>> wrote:
> >>>
>  Hi Ewen, Matthias,
>  For common configuration across all the tools, I will work on that as
> >> part
>  of other KIP by looking into all Kafka tools.
> 
> 
>  Thanks,
>  Bharat
> 
> 
>  On Wed, May 17, 2017 at 9:40 AM, Matthias J. Sax <
> matth...@confluent.io
> >>>
>  wrote:
> 
> > +1
> >
> > I also second Ewen comment -- standardizing the common supported
> > parameters over all tools would be great!
> >
> >
> > -Matthias
> >
> > On 5/17/17 12:57 AM, Damian Guy wrote:
> >> +1
> >>
> >> On Wed, 17 May 2017 at 05:40 Ewen Cheslack-Postava <
> e...@confluent.io
> >>>
> >> wrote:
> >>
> >>> +1 (binding)
> >>>
> >>> I mentioned this in the PR that triggered this:
> >>>
>  KIP is accurate, though this is one of those things that we should
> >>> probably get a KIP for a standard set of config options across all
> > tools so
> >>> additions like this can just fall under the umbrella of that KIP...
> >>>
> >>> I think it would be great if someone wrote up a small KIP providing
> > some
> >>> standardized settings that we could get future additions
> >> automatically
> >>> umbrella'd under, e.g. no need to do a KIP if just adding a
> > consumer.config
> >>> or consumer-property config conforming to existing expectations for
> > other
> >>> tools. We could also standardize on a few other settings names that
> >> are
> >>> inconsistent across different tools and set out a clear path
> forward
> > for
> >>> future tools.
> >>>
> >>> I think I still have at least one open PR from when I first started
> >> on
> > the
> >>> project where I was trying to clean up some command line stuff to
> be
> > more
> >>> consistent. This has been an issue for many years now...
> >>>
> >>> -Ewen
> >>>
> >>>
> >>>
> >>> On Tue, May 16, 2017 at 1:12 AM, Eno Thereska <
> >> eno.there...@gmail.com>
> >>> wrote:
> >>>
>  +1 thanks.
> 
>  Eno
> > On 16 May 2017, at 04:20, BigData dev 
> > wrote:
> >
> > Hi All,
> > Given the simple and non-controversial nature of the KIP, I would
> > like
> >>> to
> > start the voting process for KIP-157: Add consumer config options
> >> to
> > streams reset tool
> >
> > *https://cwiki.apache.org/confluence/display/KAFKA/KIP+
>  157+-+Add+consumer+config+options+to+streams+reset+tool
> >   157+-+Add+consumer+config+options+to+streams+reset+tool>*
> >
> >
> > The vote will run for a minimum of 72 hours.
> >
> > Thanks,
> >
> > Bharat
> 
> 
> >>>
> >>
> >
> >
> 
> >>>
> >>
> >>
> >
>
>

[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)

[Randall Hauch (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017701#comment-16017701
 ] 

Randall Hauch commented on KAFKA-5275:
--

[~colinmccabe] wrote:

{quote}
We could add an option to {{CreateTopicsOptions}} that suppresses 
{{TopicExistsException}} in the results. I guess the question is, does your 
code want to at least log something when a topic already exists rather than 
getting created, since the configuration options will potentially be 
different...
{quote}

Kafka Connect does log which topics were created and which were found. That'd 
be trivial if {{CreateTopicResults}} had methods to return the names of the 
newly-created and existing topics -- perhaps something like:

{code:java}
KafkaFuture> createdTopicNames() {...}
KafkaFuture> existingTopicNames() {...}
{code}

[~xvrl] wrote:
{quote}
However, we should also define constants for every topic configuration option 
that exists.
{quote}

Yes, and ideally the NewTopic builder might even have some convenience methods 
that saved having to use some of them.

> Review and potentially tweak AdminClient API for the initial release (KIP-117)
> --
>
> Key: KAFKA-5275
> URL: https://issues.apache.org/jira/browse/KAFKA-5275
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.11.0.0
>
>
> Once all the pieces are in, we should take a pass and ensure that the APIs 
> work well together and that they are consistent.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Reg: [VOTE] KIP 157 - Add consumer config options to streams reset tool

[Matthias J. Sax]

KIP-157 cannot be included in 0.11.0.0 anymore. KIP freeze date deadline
is strict.

-Matthias

On 5/19/17 10:15 AM, BigData dev wrote:
> Hi Matthias,
> I will start a new KIP for Kafka tools options to be a standard across all
> tools shortly. But I think the KIP 157 for Kafka Streams, should be needed
> for 0.11.0.0 release, (KIP freeze date is already over, but I think this is
> minor code change in tools to add option to streams reset tool) as without
> this consumer config options, it will not be possible to use the tool in a
> secured environment. Please let me know your thoughts on this. If it needs
> to be moved to next release, I will work on this as part of KIP 14.
> 
> Thanks,
> Bharat
> 
> 
> On Fri, May 19, 2017 at 10:10 AM, Matthias J. Sax 
> wrote:
> 
>> I double checked with Matthew Warhaftig (the original author of KIP-14)
>> and he has not interest to continue the KIP atm.
>>
>> Thus, Bharat can continue the work on KIP-14. I think it would be best,
>> to start a new DISCUSS thread after you update KIP-14.
>>
>> Thanks for your contributions!
>>
>>
>> -Matthias
>>
>>
>> On 5/17/17 12:56 PM, BigData dev wrote:
>>> Hi,
>>> When I was trying to find more info, there is already a proposed KIP for
>>> this
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 14+-+Tools+Standardization
>>>
>>>
>>> Thanks,
>>> Bharat
>>>
>>> On Wed, May 17, 2017 at 12:38 PM, BigData dev 
>>> wrote:
>>>
 Hi Ewen, Matthias,
 For common configuration across all the tools, I will work on that as
>> part
 of other KIP by looking into all Kafka tools.


 Thanks,
 Bharat


 On Wed, May 17, 2017 at 9:40 AM, Matthias J. Sax >>
 wrote:

> +1
>
> I also second Ewen comment -- standardizing the common supported
> parameters over all tools would be great!
>
>
> -Matthias
>
> On 5/17/17 12:57 AM, Damian Guy wrote:
>> +1
>>
>> On Wed, 17 May 2017 at 05:40 Ewen Cheslack-Postava >>
>> wrote:
>>
>>> +1 (binding)
>>>
>>> I mentioned this in the PR that triggered this:
>>>
 KIP is accurate, though this is one of those things that we should
>>> probably get a KIP for a standard set of config options across all
> tools so
>>> additions like this can just fall under the umbrella of that KIP...
>>>
>>> I think it would be great if someone wrote up a small KIP providing
> some
>>> standardized settings that we could get future additions
>> automatically
>>> umbrella'd under, e.g. no need to do a KIP if just adding a
> consumer.config
>>> or consumer-property config conforming to existing expectations for
> other
>>> tools. We could also standardize on a few other settings names that
>> are
>>> inconsistent across different tools and set out a clear path forward
> for
>>> future tools.
>>>
>>> I think I still have at least one open PR from when I first started
>> on
> the
>>> project where I was trying to clean up some command line stuff to be
> more
>>> consistent. This has been an issue for many years now...
>>>
>>> -Ewen
>>>
>>>
>>>
>>> On Tue, May 16, 2017 at 1:12 AM, Eno Thereska <
>> eno.there...@gmail.com>
>>> wrote:
>>>
 +1 thanks.

 Eno
> On 16 May 2017, at 04:20, BigData dev 
> wrote:
>
> Hi All,
> Given the simple and non-controversial nature of the KIP, I would
> like
>>> to
> start the voting process for KIP-157: Add consumer config options
>> to
> streams reset tool
>
> *https://cwiki.apache.org/confluence/display/KAFKA/KIP+
 157+-+Add+consumer+config+options+to+streams+reset+tool
> *
>
>
> The vote will run for a minimum of 72 hours.
>
> Thanks,
>
> Bharat


>>>
>>
>
>

>>>
>>
>>
> 



signature.asc
Description: OpenPGP digital signature

[jira] [Updated] (KAFKA-4660) Improve test coverage KafkaStreams

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-4660:
---
Status: Patch Available  (was: Open)

> Improve test coverage KafkaStreams
> --
>
> Key: KAFKA-4660
> URL: https://issues.apache.org/jira/browse/KAFKA-4660
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Damian Guy
>Priority: Minor
> Fix For: 0.11.0.0
>
>
> {{toString}} is used to print the topology, so probably should have a unit 
> test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5117) Kafka Connect REST endpoints reveal Password typed values

[Thomas Holmes (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017689#comment-16017689
 ] 

Thomas Holmes commented on KAFKA-5117:
--

I had some code that seems to work but I'm not very happy with it. I'll try to 
clean it up a bit and get a PR made so the approach/impact can be discussed.

One main thing I was concerned about was whether or not the connectors use 
these endpoints to share task configuration. I was seeing some instability when 
running my code change so I am unsure if it was related or not.  I didn't get a 
chance to fully diagnose the situation, though.

> Kafka Connect REST endpoints reveal Password typed values
> -
>
> Key: KAFKA-5117
> URL: https://issues.apache.org/jira/browse/KAFKA-5117
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Thomas Holmes
>
> A Kafka Connect connector can specify ConfigDef keys as type of Password. 
> This type was added to prevent logging the values (instead "[hidden]" is 
> logged).
> This change does not apply to the values returned by executing a GET on 
> {{connectors/\{connector-name\}}} and 
> {{connectors/\{connector-name\}/config}}. This creates an easily accessible 
> way for an attacker who has infiltrated your network to gain access to 
> potential secrets that should not be available.
> I have started on a code change that addresses this issue by parsing the 
> config values through the ConfigDef for the connector and returning their 
> output instead (which leads to the masking of Password typed configs as 
> [hidden]).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Reg: [VOTE] KIP 157 - Add consumer config options to streams reset tool

[BigData dev]

Hi Matthias,
I will start a new KIP for Kafka tools options to be a standard across all
tools shortly. But I think the KIP 157 for Kafka Streams, should be needed
for 0.11.0.0 release, (KIP freeze date is already over, but I think this is
minor code change in tools to add option to streams reset tool) as without
this consumer config options, it will not be possible to use the tool in a
secured environment. Please let me know your thoughts on this. If it needs
to be moved to next release, I will work on this as part of KIP 14.

Thanks,
Bharat


On Fri, May 19, 2017 at 10:10 AM, Matthias J. Sax 
wrote:

> I double checked with Matthew Warhaftig (the original author of KIP-14)
> and he has not interest to continue the KIP atm.
>
> Thus, Bharat can continue the work on KIP-14. I think it would be best,
> to start a new DISCUSS thread after you update KIP-14.
>
> Thanks for your contributions!
>
>
> -Matthias
>
>
> On 5/17/17 12:56 PM, BigData dev wrote:
> > Hi,
> > When I was trying to find more info, there is already a proposed KIP for
> > this
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 14+-+Tools+Standardization
> >
> >
> > Thanks,
> > Bharat
> >
> > On Wed, May 17, 2017 at 12:38 PM, BigData dev 
> > wrote:
> >
> >> Hi Ewen, Matthias,
> >> For common configuration across all the tools, I will work on that as
> part
> >> of other KIP by looking into all Kafka tools.
> >>
> >>
> >> Thanks,
> >> Bharat
> >>
> >>
> >> On Wed, May 17, 2017 at 9:40 AM, Matthias J. Sax  >
> >> wrote:
> >>
> >>> +1
> >>>
> >>> I also second Ewen comment -- standardizing the common supported
> >>> parameters over all tools would be great!
> >>>
> >>>
> >>> -Matthias
> >>>
> >>> On 5/17/17 12:57 AM, Damian Guy wrote:
>  +1
> 
>  On Wed, 17 May 2017 at 05:40 Ewen Cheslack-Postava  >
>  wrote:
> 
> > +1 (binding)
> >
> > I mentioned this in the PR that triggered this:
> >
> >> KIP is accurate, though this is one of those things that we should
> > probably get a KIP for a standard set of config options across all
> >>> tools so
> > additions like this can just fall under the umbrella of that KIP...
> >
> > I think it would be great if someone wrote up a small KIP providing
> >>> some
> > standardized settings that we could get future additions
> automatically
> > umbrella'd under, e.g. no need to do a KIP if just adding a
> >>> consumer.config
> > or consumer-property config conforming to existing expectations for
> >>> other
> > tools. We could also standardize on a few other settings names that
> are
> > inconsistent across different tools and set out a clear path forward
> >>> for
> > future tools.
> >
> > I think I still have at least one open PR from when I first started
> on
> >>> the
> > project where I was trying to clean up some command line stuff to be
> >>> more
> > consistent. This has been an issue for many years now...
> >
> > -Ewen
> >
> >
> >
> > On Tue, May 16, 2017 at 1:12 AM, Eno Thereska <
> eno.there...@gmail.com>
> > wrote:
> >
> >> +1 thanks.
> >>
> >> Eno
> >>> On 16 May 2017, at 04:20, BigData dev 
> >>> wrote:
> >>>
> >>> Hi All,
> >>> Given the simple and non-controversial nature of the KIP, I would
> >>> like
> > to
> >>> start the voting process for KIP-157: Add consumer config options
> to
> >>> streams reset tool
> >>>
> >>> *https://cwiki.apache.org/confluence/display/KAFKA/KIP+
> >> 157+-+Add+consumer+config+options+to+streams+reset+tool
> >>>  >> 157+-+Add+consumer+config+options+to+streams+reset+tool>*
> >>>
> >>>
> >>> The vote will run for a minimum of 72 hours.
> >>>
> >>> Thanks,
> >>>
> >>> Bharat
> >>
> >>
> >
> 
> >>>
> >>>
> >>
> >
>
>

Re: Reg: [VOTE] KIP 157 - Add consumer config options to streams reset tool

[Matthias J. Sax]

I double checked with Matthew Warhaftig (the original author of KIP-14)
and he has not interest to continue the KIP atm.

Thus, Bharat can continue the work on KIP-14. I think it would be best,
to start a new DISCUSS thread after you update KIP-14.

Thanks for your contributions!


-Matthias


On 5/17/17 12:56 PM, BigData dev wrote:
> Hi,
> When I was trying to find more info, there is already a proposed KIP for
> this
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-14+-+Tools+Standardization
> 
> 
> Thanks,
> Bharat
> 
> On Wed, May 17, 2017 at 12:38 PM, BigData dev 
> wrote:
> 
>> Hi Ewen, Matthias,
>> For common configuration across all the tools, I will work on that as part
>> of other KIP by looking into all Kafka tools.
>>
>>
>> Thanks,
>> Bharat
>>
>>
>> On Wed, May 17, 2017 at 9:40 AM, Matthias J. Sax 
>> wrote:
>>
>>> +1
>>>
>>> I also second Ewen comment -- standardizing the common supported
>>> parameters over all tools would be great!
>>>
>>>
>>> -Matthias
>>>
>>> On 5/17/17 12:57 AM, Damian Guy wrote:
 +1

 On Wed, 17 May 2017 at 05:40 Ewen Cheslack-Postava 
 wrote:

> +1 (binding)
>
> I mentioned this in the PR that triggered this:
>
>> KIP is accurate, though this is one of those things that we should
> probably get a KIP for a standard set of config options across all
>>> tools so
> additions like this can just fall under the umbrella of that KIP...
>
> I think it would be great if someone wrote up a small KIP providing
>>> some
> standardized settings that we could get future additions automatically
> umbrella'd under, e.g. no need to do a KIP if just adding a
>>> consumer.config
> or consumer-property config conforming to existing expectations for
>>> other
> tools. We could also standardize on a few other settings names that are
> inconsistent across different tools and set out a clear path forward
>>> for
> future tools.
>
> I think I still have at least one open PR from when I first started on
>>> the
> project where I was trying to clean up some command line stuff to be
>>> more
> consistent. This has been an issue for many years now...
>
> -Ewen
>
>
>
> On Tue, May 16, 2017 at 1:12 AM, Eno Thereska 
> wrote:
>
>> +1 thanks.
>>
>> Eno
>>> On 16 May 2017, at 04:20, BigData dev 
>>> wrote:
>>>
>>> Hi All,
>>> Given the simple and non-controversial nature of the KIP, I would
>>> like
> to
>>> start the voting process for KIP-157: Add consumer config options to
>>> streams reset tool
>>>
>>> *https://cwiki.apache.org/confluence/display/KAFKA/KIP+
>> 157+-+Add+consumer+config+options+to+streams+reset+tool
>>> > 157+-+Add+consumer+config+options+to+streams+reset+tool>*
>>>
>>>
>>> The vote will run for a minimum of 72 hours.
>>>
>>> Thanks,
>>>
>>> Bharat
>>
>>
>

>>>
>>>
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

[Kyle Winkelman]

I like your point about not handling other cases such as count and reduce.

I think that reduce may not make sense because reduce assumes that the
input values are the same as the output values. With cogroup there may be
multiple different input types and then your output type cant be multiple
different things. In the case where you have all matching value types you
can do KStreamBuilder#merge followed by the reduce.

As for count I think it is possible to call count on all the individual
grouped streams and then do joins. Otherwise we could maybe make a special
call in groupedstream for this case. Because in this case we dont need to
do type checking on the values. It could be similar to the current count
methods but accept a var args of additonal grouped streams as well and make
sure they have a key type of K.

The way I have put the kip together is to ensure that we do type checking.
I don't see a way we could group them all first and then make a call to
count, reduce, or aggregate because with aggregate they would need to pass
a list of aggregators and we would have no way of type checking that they
match the grouped streams.

Thanks,
Kyle

On May 19, 2017 11:42 AM, "Xavier Léauté"  wrote:

> Sorry to jump on this thread so late. I agree this is a very useful
> addition and wanted to provide an additional use-case and some more
> comments.
>
> This is actually a very common analytics use-case in the ad-tech industry.
> The typical setup will have an auction stream, an impression stream, and a
> click stream. Those three streams need to be combined to compute aggregate
> statistics (e.g. impression statistics, and click-through rates), since
> most of the attributes of interest are only present the auction stream.
>
> A simple way to do this is to co-group all the streams by the auction key,
> and process updates to the co-group as events for each stream come in,
> keeping only one value from each stream before sending downstream for
> further processing / aggregation.
>
> One could view the result of that co-group operation as a "KTable" with
> multiple values per key. The key being the grouping key, and the values
> consisting of one value per stream.
>
> What I like about Kyle's approach is that allows elegant co-grouping of
> multiple streams without having to worry about the number of streams, and
> avoids dealing with Tuple types or other generic interfaces that could get
> messy if we wanted to preserve all the value types in the resulting
> co-grouped stream.
>
> My only concern is that we only allow the cogroup + aggregate combined
> operation. This forces the user to build their own tuple serialization
> format if they want to preserve the individual input stream values as a
> group. It also deviates quite a bit from our approach in KGroupedStream
> which offers other operations, such as count and reduce, which should also
> be applicable to a co-grouped stream.
>
> Overall I still think this is a really useful addition, but I feel we
> haven't spend much time trying to explore alternative DSLs that could maybe
> generalize better or match our existing syntax more closely.
>
> On Tue, May 9, 2017 at 8:08 AM Kyle Winkelman 
> wrote:
>
> > Eno, is there anyone else that is an expert in the kafka streams realm
> that
> > I should reach out to for input?
> >
> > I believe Damian Guy is still planning on reviewing this more in depth
> so I
> > will wait for his inputs before continuing.
> >
> > On May 9, 2017 7:30 AM, "Eno Thereska"  wrote:
> >
> > > Thanks Kyle, good arguments.
> > >
> > > Eno
> > >
> > > > On May 7, 2017, at 5:06 PM, Kyle Winkelman  >
> > > wrote:
> > > >
> > > > *- minor: could you add an exact example (similar to what Jay’s
> example
> > > is,
> > > > or like your Spark/Pig pointers had) to make this super concrete?*
> > > > I have added a more concrete example to the KIP.
> > > >
> > > > *- my main concern is that we’re exposing this optimization to the
> DSL.
> > > In
> > > > an ideal world, an optimizer would take the existing DSL and do the
> > right
> > > > thing under the covers (create just one state store, arrange the
> nodes
> > > > etc). The original DSL had a bunch of small, composable pieces
> (group,
> > > > aggregate, join) that this proposal groups together. I’d like to hear
> > > your
> > > > thoughts on whether it’s possible to do this optimization with the
> > > current
> > > > DSL, at the topology builder level.*
> > > > You would have to make a lot of checks to understand if it is even
> > > possible
> > > > to make this optimization:
> > > > 1. Make sure they are all KTableKTableOuterJoins
> > > > 2. None of the intermediate KTables are used for anything else.
> > > > 3. None of the intermediate stores are used. (This may be impossible
> > > > especially if they use KafkaStreams#store after the topology has
> > already
> > > > been built.)
> > > > You would then need to make decisions during the optimization:
> > > > 1. Your new initializer would the compos

[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)

[JIRA]

[ 
https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017659#comment-16017659
 ] 

Xavier Léauté commented on KAFKA-5275:
--

To echo Randall's point about having a topic config builder, I completely 
agree. However, we should also define constants for every topic configuration 
option that exists. Currently the only way to get those is to pull in the 
kafka-server jar and access things like  {{LogConfig.SegmentMsProp()}} directly

I believe it would be useful to keep the ability to set arbitrary string 
configuration to be forward-compatible and not require someone to upgrade their 
client just to set a new topic option.

> Review and potentially tweak AdminClient API for the initial release (KIP-117)
> --
>
> Key: KAFKA-5275
> URL: https://issues.apache.org/jira/browse/KAFKA-5275
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.11.0.0
>
>
> Once all the pieces are in, we should take a pass and ensure that the APIs 
> work well together and that they are consistent.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)

[JIRA]

[ 
https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017646#comment-16017646
 ] 

Xavier Léauté commented on KAFKA-5275:
--

Speaking of {{TopicExistsException}} it would also be nice to document the 
possible failure modes of the various api methods, so one doesn't have to find 
out by trial and error. It's also not immediately clear which recoverable 
errors the admin client retries automatically for the user vs. which ones need 
to be handled by the user.

> Review and potentially tweak AdminClient API for the initial release (KIP-117)
> --
>
> Key: KAFKA-5275
> URL: https://issues.apache.org/jira/browse/KAFKA-5275
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.11.0.0
>
>
> Once all the pieces are in, we should take a pass and ensure that the APIs 
> work well together and that they are consistent.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5291) AdminClient should not trigger auto creation of topics

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017637#comment-16017637
 ] 

Ismael Juma commented on KAFKA-5291:


cc [~cmccabe]

> AdminClient should not trigger auto creation of topics
> --
>
> Key: KAFKA-5291
> URL: https://issues.apache.org/jira/browse/KAFKA-5291
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.11.0.0
>
>
> Feedback from people who tried the AdminClient is that auto topic creation 
> during describe is very confusing. We expected that, but we also thought that 
> there wasn't much that could be done.
> However, there's a simple solution (suggested by [~hachikuji]): make it 
> possible to disallow auto topic creation when sending the metadata request. 
> We can keep the existing behaviour for the producer and consumer (until we 
> move to client-side auto topic creation) while doing the right thing for 
> AdminClient (which is new).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)

[Colin P. McCabe (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017629#comment-16017629
 ] 

Colin P. McCabe edited comment on KAFKA-5275 at 5/19/17 4:37 PM:
-

bq. It'd be great to simplify the creation of the NewTopic objects. As part of 
the PR for KIP-154 / KAFKA-4667 I added a builder for NewTopic that has methods 
to set replication factor, num partitions, the cleanup policy (currently 
compacted only), min ISRs, etc. It's currently an implementation detail of 
Kafka Connect, but it'd be great if that could move down to the AdminClient.

Yeah, I think it makes sense to consider adding this to {{AdminClient}}.

[~xvrl] also proposed adding a comment to the {{AdminClient}} JavaDoc 
discussing the existence of the {{AdminClientConfig}} class, which might not be 
clear for first-time users.


was (Author: cmccabe):
bq. It'd be great to simplify the creation of the NewTopic objects. As part of 
the PR for KIP-154 / KAFKA-4667 I added a builder for NewTopic that has methods 
to set replication factor, num partitions, the cleanup policy (currently 
compacted only), min ISRs, etc. It's currently an implementation detail of 
Kafka Connect, but it'd be great if that could move down to the AdminClient.

Yeah, I think it makes sense to consider adding this to `AdminClient`.

[~xvrl] also proposed adding a comment to the {{AdminClient}} JavaDoc 
discussing the existence of the {{AdminClientConfig}} class, which might not be 
clear for first-time users.

> Review and potentially tweak AdminClient API for the initial release (KIP-117)
> --
>
> Key: KAFKA-5275
> URL: https://issues.apache.org/jira/browse/KAFKA-5275
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.11.0.0
>
>
> Once all the pieces are in, we should take a pass and ensure that the APIs 
> work well together and that they are consistent.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)

[Colin P. McCabe (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017629#comment-16017629
 ] 

Colin P. McCabe commented on KAFKA-5275:


bq. It'd be great to simplify the creation of the NewTopic objects. As part of 
the PR for KIP-154 / KAFKA-4667 I added a builder for NewTopic that has methods 
to set replication factor, num partitions, the cleanup policy (currently 
compacted only), min ISRs, etc. It's currently an implementation detail of 
Kafka Connect, but it'd be great if that could move down to the AdminClient.

Yeah, I think it makes sense to consider adding this to `AdminClient`.

[~xvrl] also proposed adding a comment to the {{AdminClient}} JavaDoc 
discussing the existence of the {{AdminClientConfig}} class, which might not be 
clear for first-time users.

> Review and potentially tweak AdminClient API for the initial release (KIP-117)
> --
>
> Key: KAFKA-5275
> URL: https://issues.apache.org/jira/browse/KAFKA-5275
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.11.0.0
>
>
> Once all the pieces are in, we should take a pass and ensure that the APIs 
> work well together and that they are consistent.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5275) Review and potentially tweak AdminClient API for the initial release (KIP-117)

[Colin P. McCabe (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017626#comment-16017626
 ] 

Colin P. McCabe commented on KAFKA-5275:


bq. [~rhauch]: Also, a common use case for creating topics is going to be 
"create new topic(s) if they don't already exist". The current AdminClient can 
do this, but it requires a fair amount of error handling, especially when 
trying to create multiple topics at once.

We could add an option to {{CreateTopicsOptions}} that suppresses 
{{TopicExistsException}} in the results.  I guess the question is, does your 
code want to at least log something when a topic already exists rather than 
getting created, since the configuration options will potentially be 
different...

bq. it would be nice to have a default describeTopics() that doesn't need a 
list of topics and just returns the description for all topics, right now i 
have to do 
adminClient.describeTopics(a.listTopics().names().get()).all().get(); :/

The intention here was to encourage people to fetch a few topics at a time, 
since we don't currently have very scalable ways of fetching all topics.  
KIP-142 will help here.

Of course, sometimes you really do need to fetch every single topic in the 
cluster.  We probably should have some nicer way of doing this.  Perhaps it 
could do batching in the AdminClient.

> Review and potentially tweak AdminClient API for the initial release (KIP-117)
> --
>
> Key: KAFKA-5275
> URL: https://issues.apache.org/jira/browse/KAFKA-5275
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.11.0.0
>
>
> Once all the pieces are in, we should take a pass and ensure that the APIs 
> work well together and that they are consistent.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

KTable aggregate corruption on poorly timed exception

[Mathieu Fenniak]

Hi Kafka devs,

This morning I observed a specific Kafka Streams aggregation that
ended up with an incorrect computed output after a Kafka Streams
thread crashed with an unhandled exception.

The topology is pretty simple -- a single KTable source, group by a
field in the value, aggregate that adds up another field, output to a
topic.

Here's the sequence of events that appears to have occurred:

1. A new record (record A) is received by the source KTable, and put
in the KTable RocksDB state store.

2. While processing record A, an exception happens preventing
producing to Kafka. (specific exception at end of e-mail).

3. The stream thread throws an unhandled exception and stops.

4. The state stores are closed and flushed.  Record A is now in the
local state store.

5. The consumer group rebalances.

6. A different thread, in the same process, on the same host, picks up the task.

7. New thread initializes its state store for the KTable, but it's on
the same host as the original thread, so it still contains the k/v for
record A.

8. New thread resumes consuming at the last committed offset, which is
before record A.

9. When processing record A, the new thread reads the value that was
written to the state store in step #1 by record A's key.

10. The repartition map receives a Change with both an oldValue and a
newValue, and forwards a Change(null, v) and Change(v, null)

11. The aggregation ends up both subtracting and adding the value of
record A, resulting in an incorrect output.

As a result of this sequence, my aggregate output went from a value of
0, to negative (subtracting record A), to 0.  And stayed there.

Does this seem like a feasible series of events?  Is this a bug in KS,
or, is it behavior that maybe can't be improved without exactly-once?
I'd think the best behavior would be for writes to the RocksDB state
store to be transactional and only commit when the producer commits,
but, there's a lot of overhead involved in that.

Mathieu

Re: KTable aggregate corruption on poorly timed exception

[Mathieu Fenniak]

Whoops, I said I'd put the specific exception at the bottom of the
e-mail.  It probably isn't the important part of this thread, but
might suggest when this situation can occur.  Also of note, this is
occurring on Kafka Streams 0.10.2.1.


20:56:07.061 [StreamThread-3] ERROR o.a.k.s.p.internals.StreamThread -
stream-thread [StreamThread-3] Failed to commit StreamTask 0_4 state:
org.apache.kafka.streams.errors.StreamsException: task [0_4] exception
caught when producing
at 
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.checkForException(RecordCollectorImpl.java:121)
at 
org.apache.kafka.streams.processor.internals.RecordCollectorImpl.flush(RecordCollectorImpl.java:129)
at 
org.apache.kafka.streams.processor.internals.StreamTask$1.run(StreamTask.java:76)
at 
org.apache.kafka.streams.processor.internals.StreamsMetricsImpl.measureLatencyNs(StreamsMetricsImpl.java:188)
at 
org.apache.kafka.streams.processor.internals.StreamTask.commit(StreamTask.java:280)
at 
org.apache.kafka.streams.processor.internals.StreamThread.commitOne(StreamThread.java:807)
at 
org.apache.kafka.streams.processor.internals.StreamThread.commitAll(StreamThread.java:794)
at 
org.apache.kafka.streams.processor.internals.StreamThread.maybeCommit(StreamThread.java:769)
at 
org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:647)
at 
org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:361)
Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to
update metadata after 6 ms.

On Fri, May 19, 2017 at 9:47 AM, Mathieu Fenniak
 wrote:
> Hi Kafka devs,
>
> This morning I observed a specific Kafka Streams aggregation that
> ended up with an incorrect computed output after a Kafka Streams
> thread crashed with an unhandled exception.
>
> The topology is pretty simple -- a single KTable source, group by a
> field in the value, aggregate that adds up another field, output to a
> topic.
>
> Here's the sequence of events that appears to have occurred:
>
> 1. A new record (record A) is received by the source KTable, and put
> in the KTable RocksDB state store.
>
> 2. While processing record A, an exception happens preventing
> producing to Kafka. (specific exception at end of e-mail).
>
> 3. The stream thread throws an unhandled exception and stops.
>
> 4. The state stores are closed and flushed.  Record A is now in the
> local state store.
>
> 5. The consumer group rebalances.
>
> 6. A different thread, in the same process, on the same host, picks up the 
> task.
>
> 7. New thread initializes its state store for the KTable, but it's on
> the same host as the original thread, so it still contains the k/v for
> record A.
>
> 8. New thread resumes consuming at the last committed offset, which is
> before record A.
>
> 9. When processing record A, the new thread reads the value that was
> written to the state store in step #1 by record A's key.
>
> 10. The repartition map receives a Change with both an oldValue and a
> newValue, and forwards a Change(null, v) and Change(v, null)
>
> 11. The aggregation ends up both subtracting and adding the value of
> record A, resulting in an incorrect output.
>
> As a result of this sequence, my aggregate output went from a value of
> 0, to negative (subtracting record A), to 0.  And stayed there.
>
> Does this seem like a feasible series of events?  Is this a bug in KS,
> or, is it behavior that maybe can't be improved without exactly-once?
> I'd think the best behavior would be for writes to the RocksDB state
> store to be transactional and only commit when the producer commits,
> but, there's a lot of overhead involved in that.
>
> Mathieu

[jira] [Commented] (KAFKA-4785) Records from internal repartitioning topics should always use RecordMetadataTimestampExtractor

[Jeyhun Karimov (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017575#comment-16017575
 ] 

Jeyhun Karimov commented on KAFKA-4785:
---

[~mjsax] Thanks for reminding. I am on it

> Records from internal repartitioning topics should always use 
> RecordMetadataTimestampExtractor
> --
>
> Key: KAFKA-4785
> URL: https://issues.apache.org/jira/browse/KAFKA-4785
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Matthias J. Sax
>Assignee: Jeyhun Karimov
>
> Users can specify what timestamp extractor should be used to decode the 
> timestamp of input topic records. As long as RecordMetadataTimestamp or 
> WallclockTime is use this is fine. 
> However, for custom timestamp extractors it might be invalid to apply this 
> custom extractor to records received from internal repartitioning topics. The 
> reason is that Streams sets the current "stream time" as record metadata 
> timestamp explicitly before writing to intermediate repartitioning topics 
> because this timestamp should be use by downstream subtopologies. A custom 
> timestamp extractor might return something different breaking this assumption.
> Thus, for reading data from intermediate repartitioning topic, the configured 
> timestamp extractor should not be used, but the record's metadata timestamp 
> should be extracted as record timestamp.
> In order to leverage the same behavior for intermediate user topic (ie, used 
> in {{through()}})  we can leverage KAFKA-4144 and internally set an extractor 
> for those "intermediate sources" that returns the record's metadata timestamp 
> in order to overwrite the global extractor from {{StreamsConfig}} (ie, set 
> {{FailOnInvalidTimestampExtractor}}).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [Vote] KIP-150 - Kafka-Streams Cogroup

[Xavier Léauté]

Hi Kyle, I left a few more comments in the discussion thread, if you
wouldn't mind taking a look

On Fri, May 19, 2017 at 5:31 AM Kyle Winkelman 
wrote:

> Hello all,
>
> I would like to start the vote on KIP-150.
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+Kafka-Streams+Cogroup
>
> Thanks,
> Kyle
>

Re: [DISCUSS] KIP-150 - Kafka-Streams Cogroup

[Xavier Léauté]

Sorry to jump on this thread so late. I agree this is a very useful
addition and wanted to provide an additional use-case and some more
comments.

This is actually a very common analytics use-case in the ad-tech industry.
The typical setup will have an auction stream, an impression stream, and a
click stream. Those three streams need to be combined to compute aggregate
statistics (e.g. impression statistics, and click-through rates), since
most of the attributes of interest are only present the auction stream.

A simple way to do this is to co-group all the streams by the auction key,
and process updates to the co-group as events for each stream come in,
keeping only one value from each stream before sending downstream for
further processing / aggregation.

One could view the result of that co-group operation as a "KTable" with
multiple values per key. The key being the grouping key, and the values
consisting of one value per stream.

What I like about Kyle's approach is that allows elegant co-grouping of
multiple streams without having to worry about the number of streams, and
avoids dealing with Tuple types or other generic interfaces that could get
messy if we wanted to preserve all the value types in the resulting
co-grouped stream.

My only concern is that we only allow the cogroup + aggregate combined
operation. This forces the user to build their own tuple serialization
format if they want to preserve the individual input stream values as a
group. It also deviates quite a bit from our approach in KGroupedStream
which offers other operations, such as count and reduce, which should also
be applicable to a co-grouped stream.

Overall I still think this is a really useful addition, but I feel we
haven't spend much time trying to explore alternative DSLs that could maybe
generalize better or match our existing syntax more closely.

On Tue, May 9, 2017 at 8:08 AM Kyle Winkelman 
wrote:

> Eno, is there anyone else that is an expert in the kafka streams realm that
> I should reach out to for input?
>
> I believe Damian Guy is still planning on reviewing this more in depth so I
> will wait for his inputs before continuing.
>
> On May 9, 2017 7:30 AM, "Eno Thereska"  wrote:
>
> > Thanks Kyle, good arguments.
> >
> > Eno
> >
> > > On May 7, 2017, at 5:06 PM, Kyle Winkelman 
> > wrote:
> > >
> > > *- minor: could you add an exact example (similar to what Jay’s example
> > is,
> > > or like your Spark/Pig pointers had) to make this super concrete?*
> > > I have added a more concrete example to the KIP.
> > >
> > > *- my main concern is that we’re exposing this optimization to the DSL.
> > In
> > > an ideal world, an optimizer would take the existing DSL and do the
> right
> > > thing under the covers (create just one state store, arrange the nodes
> > > etc). The original DSL had a bunch of small, composable pieces (group,
> > > aggregate, join) that this proposal groups together. I’d like to hear
> > your
> > > thoughts on whether it’s possible to do this optimization with the
> > current
> > > DSL, at the topology builder level.*
> > > You would have to make a lot of checks to understand if it is even
> > possible
> > > to make this optimization:
> > > 1. Make sure they are all KTableKTableOuterJoins
> > > 2. None of the intermediate KTables are used for anything else.
> > > 3. None of the intermediate stores are used. (This may be impossible
> > > especially if they use KafkaStreams#store after the topology has
> already
> > > been built.)
> > > You would then need to make decisions during the optimization:
> > > 1. Your new initializer would the composite of all the individual
> > > initializers and the valueJoiners.
> > > 2. I am having a hard time thinking about how you would turn the
> > > aggregators and valueJoiners into an aggregator that would work on the
> > > final object, but this may be possible.
> > > 3. Which state store would you use? The ones declared would be for the
> > > aggregate values. None of the declared ones would be guaranteed to hold
> > the
> > > final object. This would mean you must created a new state store and
> not
> > > created any of the declared ones.
> > >
> > > The main argument I have against it is even if it could be done I don't
> > > know that we would want to have this be an optimization in the
> background
> > > because the user would still be required to think about all of the
> > > intermediate values that they shouldn't need to worry about if they
> only
> > > care about the final object.
> > >
> > > In my opinion cogroup is a common enough case that it should be part of
> > the
> > > composable pieces (group, aggregate, join) because we want to allow
> > people
> > > to join more than 2 or more streams in an easy way. Right now I don't
> > think
> > > we give them ways of handling this use case easily.
> > >
> > > *-I think there will be scope for several such optimizations in the
> > future
> > > and perhaps at some point we need to think about decoupling the 

[GitHub] kafka pull request #3103: KAFKA-5128: check inter broker and message format ...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/3103

KAFKA-5128: check inter broker and message format versions in transactional 
methods [WIP]

Add check in `KafkaApis` that the inter broker protocol and message format 
versions are at least `KAFKA_0_11_0_IV0`, i.e., supporting transactions 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka kafka-5128

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3103.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3103


commit 75846aee5a945ea13c08856fc12ffc4ff62f7e45
Author: Damian Guy 
Date:   2017-05-19T15:36:22Z

check api version in transactional methods




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-5128) TransactionCoordinator - Check inter broker protocol and message format and raise errors if incompatible

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5128?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017561#comment-16017561
 ] 

ASF GitHub Bot commented on KAFKA-5128:
---

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/3103

KAFKA-5128: check inter broker and message format versions in transactional 
methods [WIP]

Add check in `KafkaApis` that the inter broker protocol and message format 
versions are at least `KAFKA_0_11_0_IV0`, i.e., supporting transactions 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka kafka-5128

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3103.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3103


commit 75846aee5a945ea13c08856fc12ffc4ff62f7e45
Author: Damian Guy 
Date:   2017-05-19T15:36:22Z

check api version in transactional methods




> TransactionCoordinator - Check inter broker protocol and message format and 
> raise errors if incompatible
> 
>
> Key: KAFKA-5128
> URL: https://issues.apache.org/jira/browse/KAFKA-5128
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Damian Guy
>Assignee: Damian Guy
>Priority: Blocker
>
> In {{TransactionStateManager}} and {{InterBrokerSendThread}} we need to check 
> if the message format supports transactions and error out if it doesn't



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5278) kafka-console-consumer: `--value-deserializer` is not working but `--property value.deserializer` does

[Vahid Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017556#comment-16017556
 ] 

Vahid Hashemian commented on KAFKA-5278:


Looks like the issue reported here is a subset of 
[KAFKA-3982|https://issues.apache.org/jira/browse/KAFKA-3982].
Plus, the correct syntax for providing additional properties directly in the 
command line is {{--consumer-propert}}, not {{--property}}.

> kafka-console-consumer: `--value-deserializer` is not working but `--property 
> value.deserializer` does
> --
>
> Key: KAFKA-5278
> URL: https://issues.apache.org/jira/browse/KAFKA-5278
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.2.1
>Reporter: Yeva Byzek
>Assignee: huxi
>Priority: Minor
>
> kafka-console-consumer: {{--value-deserializer}} is not working but 
> {{--property value.deserializer}} is working
> 1. Does not work
> {noformat}
> $ kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning 
> --topic TEST1  --value-deserializer 
> org.apache.kafka.common.serialization.LongDeserializer
> [2017-05-18 13:09:41,745] ERROR Error processing message, terminating 
> consumer process:  (kafka.tools.ConsoleConsumer$)
> java.lang.ClassCastException: java.lang.Long cannot be cast to [B
>   at kafka.consumer.NewShinyConsumer.receive(BaseConsumer.scala:100)
>   at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:120)
>   at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:75)
>   at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:50)
>   at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
> Processed a total of 0 messages
> {noformat}
> 2. Does work
> {noformat}
> $ kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning 
> --topic TEST1  --property 
> value.deserializer=org.apache.kafka.common.serialization.LongDeserializer
> 1000
> 2500
> 2000
> 5500
> 8000
> {noformat}
> Without either, the output is
> {noformat}
> $ kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning 
> --topic TEST1
> ?
>   ?
> ?
> |
> @
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (KAFKA-5278) kafka-console-consumer: `--value-deserializer` is not working but `--property value.deserializer` does

[Vahid Hashemian (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017556#comment-16017556
 ] 

Vahid Hashemian edited comment on KAFKA-5278 at 5/19/17 3:38 PM:
-

Looks like the issue reported here is a subset of 
[KAFKA-3982|https://issues.apache.org/jira/browse/KAFKA-3982].
Plus, the correct syntax for providing additional properties directly in the 
command line is {{\-\-consumer-propert}}, not {{\-\-property}}.


was (Author: vahid):
Looks like the issue reported here is a subset of 
[KAFKA-3982|https://issues.apache.org/jira/browse/KAFKA-3982].
Plus, the correct syntax for providing additional properties directly in the 
command line is {{--consumer-propert}}, not {{--property}}.

> kafka-console-consumer: `--value-deserializer` is not working but `--property 
> value.deserializer` does
> --
>
> Key: KAFKA-5278
> URL: https://issues.apache.org/jira/browse/KAFKA-5278
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.2.1
>Reporter: Yeva Byzek
>Assignee: huxi
>Priority: Minor
>
> kafka-console-consumer: {{--value-deserializer}} is not working but 
> {{--property value.deserializer}} is working
> 1. Does not work
> {noformat}
> $ kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning 
> --topic TEST1  --value-deserializer 
> org.apache.kafka.common.serialization.LongDeserializer
> [2017-05-18 13:09:41,745] ERROR Error processing message, terminating 
> consumer process:  (kafka.tools.ConsoleConsumer$)
> java.lang.ClassCastException: java.lang.Long cannot be cast to [B
>   at kafka.consumer.NewShinyConsumer.receive(BaseConsumer.scala:100)
>   at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:120)
>   at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:75)
>   at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:50)
>   at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
> Processed a total of 0 messages
> {noformat}
> 2. Does work
> {noformat}
> $ kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning 
> --topic TEST1  --property 
> value.deserializer=org.apache.kafka.common.serialization.LongDeserializer
> 1000
> 2500
> 2000
> 5500
> 8000
> {noformat}
> Without either, the output is
> {noformat}
> $ kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning 
> --topic TEST1
> ?
>   ?
> ?
> |
> @
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5294) PlainSaslServerFactory should allow a null Map in getMechanismNames

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5294:
---
Fix Version/s: 0.11.0.0

> PlainSaslServerFactory should allow a null Map in getMechanismNames
> ---
>
> Key: KAFKA-5294
> URL: https://issues.apache.org/jira/browse/KAFKA-5294
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.1
>Reporter: Bryan Bende
>Priority: Minor
> Fix For: 0.11.0.0
>
>
> PlainSaslServerFactory will get a NullPointerException if 
> getMechanismNames(Map props) is called with a null props map.
> The Java docs for getMechanismNames from the base class 
> javax.security.sasl.SaslServerFactory say that the map can be null:
> * @param props The possibly null set of properties used to specify the
>  * security policy of the SASL mechanisms.
> A scenario where this causes a problem is when attempting to use a Kafka 
> consumer/producer with the PlainLoginModule in the same JVM as a Hadoop 
> client with Kerberos...
> The Hadoop client purposely calls getMechanismNames with a null map, so if 
> you have already create a Kafka consumer/producer with PlainLoginModule then 
> the PlainSaslServerFactory will be registered in the list of providers in 
> java.security.Security, and then NPEs when the Hadoop client calls 
> getMechanismNames on each provider with a null map.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5294) PlainSaslServerFactory should allow a null Map in getMechanismNames

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017505#comment-16017505
 ] 

Ismael Juma commented on KAFKA-5294:


Thanks for the report. This is a simple fix, so we should do it. [~rsivaram], 
maybe you can submit a fix?

> PlainSaslServerFactory should allow a null Map in getMechanismNames
> ---
>
> Key: KAFKA-5294
> URL: https://issues.apache.org/jira/browse/KAFKA-5294
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.1
>Reporter: Bryan Bende
> Fix For: 0.11.0.0
>
>
> PlainSaslServerFactory will get a NullPointerException if 
> getMechanismNames(Map props) is called with a null props map.
> The Java docs for getMechanismNames from the base class 
> javax.security.sasl.SaslServerFactory say that the map can be null:
> * @param props The possibly null set of properties used to specify the
>  * security policy of the SASL mechanisms.
> A scenario where this causes a problem is when attempting to use a Kafka 
> consumer/producer with the PlainLoginModule in the same JVM as a Hadoop 
> client with Kerberos...
> The Hadoop client purposely calls getMechanismNames with a null map, so if 
> you have already create a Kafka consumer/producer with PlainLoginModule then 
> the PlainSaslServerFactory will be registered in the list of providers in 
> java.security.Security, and then NPEs when the Hadoop client calls 
> getMechanismNames on each provider with a null map.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5294) PlainSaslServerFactory should allow a null Map in getMechanismNames

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5294:
---
Priority: Major  (was: Minor)

> PlainSaslServerFactory should allow a null Map in getMechanismNames
> ---
>
> Key: KAFKA-5294
> URL: https://issues.apache.org/jira/browse/KAFKA-5294
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.2.1
>Reporter: Bryan Bende
> Fix For: 0.11.0.0
>
>
> PlainSaslServerFactory will get a NullPointerException if 
> getMechanismNames(Map props) is called with a null props map.
> The Java docs for getMechanismNames from the base class 
> javax.security.sasl.SaslServerFactory say that the map can be null:
> * @param props The possibly null set of properties used to specify the
>  * security policy of the SASL mechanisms.
> A scenario where this causes a problem is when attempting to use a Kafka 
> consumer/producer with the PlainLoginModule in the same JVM as a Hadoop 
> client with Kerberos...
> The Hadoop client purposely calls getMechanismNames with a null map, so if 
> you have already create a Kafka consumer/producer with PlainLoginModule then 
> the PlainSaslServerFactory will be registered in the list of providers in 
> java.security.Security, and then NPEs when the Hadoop client calls 
> getMechanismNames on each provider with a null map.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5294) PlainSaslServerFactory should allow a null Map in getMechanismNames

[Bryan Bende (JIRA)]

Bryan Bende created KAFKA-5294:
--

 Summary: PlainSaslServerFactory should allow a null Map in 
getMechanismNames
 Key: KAFKA-5294
 URL: https://issues.apache.org/jira/browse/KAFKA-5294
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.10.2.1
Reporter: Bryan Bende
Priority: Minor


PlainSaslServerFactory will get a NullPointerException if 
getMechanismNames(Map props) is called with a null props map.

The Java docs for getMechanismNames from the base class 
javax.security.sasl.SaslServerFactory say that the map can be null:

* @param props The possibly null set of properties used to specify the
 * security policy of the SASL mechanisms.

A scenario where this causes a problem is when attempting to use a Kafka 
consumer/producer with the PlainLoginModule in the same JVM as a Hadoop client 
with Kerberos...

The Hadoop client purposely calls getMechanismNames with a null map, so if you 
have already create a Kafka consumer/producer with PlainLoginModule then the 
PlainSaslServerFactory will be registered in the list of providers in 
java.security.Security, and then NPEs when the Hadoop client calls 
getMechanismNames on each provider with a null map.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5295) Allow Kafka Connect source connectors to specify topic-specific settings for new topics

[Randall Hauch (JIRA)]

Randall Hauch created KAFKA-5295:


 Summary: Allow Kafka Connect source connectors to specify 
topic-specific settings for new topics
 Key: KAFKA-5295
 URL: https://issues.apache.org/jira/browse/KAFKA-5295
 Project: Kafka
  Issue Type: Improvement
  Components: KafkaConnect
Affects Versions: 0.10.2.1
Reporter: Randall Hauch
Assignee: Randall Hauch


As of 0.11.0.0, Kafka Connect will be able to automatically create its internal 
topics using the new AdminClient (see KAFKA-4667). However, it still relies 
upon the broker auto-creating topics to which source connector records are 
written.

Kafka Connect should use the AdminClient to explicitly create the new topics 
before writing the first source record to a new topic, and it should allow the 
connector the opportunity to customize the topic-specific settings for those 
new topics. As such, it will require a change in the public API (configs and/or 
framework) for source connectors and thus will require a KIP.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5266) Follow-up improvements for consumer offset reset tool (KIP-122)

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017489#comment-16017489
 ] 

ASF GitHub Bot commented on KAFKA-5266:
---

GitHub user jeqo opened a pull request:

https://github.com/apache/kafka/pull/3102

Implement Improvements

Implement improvements defined here: 
https://issues.apache.org/jira/browse/KAFKA-5266

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jeqo/kafka feature/KAFKA-5266

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3102.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3102


commit f0219737139760a455baf54bce6b1fa06789b255
Author: Jorge Quilcate Otoya 
Date:   2017-05-19T14:40:15Z

implement improvements (1,2,3)




> Follow-up improvements for consumer offset reset tool (KIP-122)
> ---
>
> Key: KAFKA-5266
> URL: https://issues.apache.org/jira/browse/KAFKA-5266
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Jason Gustafson
>Assignee: Jorge Quilcate
> Fix For: 0.11.0.0
>
>
> 1. We should try to ensure that offsets are in range for the topic partition. 
> We currently only verify this for the shift option.
> 2. If you provide a CSV file, you shouldn't need to specify one of the 
> --all-topics or --topic options.
> 3. We currently support a "reset to current offsets" option if none of the 
> supported reset options are provided. This seems kind of useless. Perhaps we 
> should just enforce that one of the reset options is provided.
> 4. The command fails with an NPE if we cannot find one of the offsets we are 
> trying to reset. It would be better to raise an exception with a friendlier 
> message.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3102: Implement Improvements

[jeqo]

GitHub user jeqo opened a pull request:

https://github.com/apache/kafka/pull/3102

Implement Improvements

Implement improvements defined here: 
https://issues.apache.org/jira/browse/KAFKA-5266

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/jeqo/kafka feature/KAFKA-5266

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3102.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3102


commit f0219737139760a455baf54bce6b1fa06789b255
Author: Jorge Quilcate Otoya 
Date:   2017-05-19T14:40:15Z

implement improvements (1,2,3)




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Bug Submit - Critical

[J Pai]

Fixing the subject line in my previous reply. Not sure what happened there.

-Jaikiran
On 19-May-2017, at 8:06 PM, J Pai  wrote:

This sounds like https://issues.apache.org/jira/browse/KAFKA-5232. It’s been 
fixed a few days back in upstream and will be available in next releases.

Having said that, you note something about split() method which wasn’t 
applicable here. Which version of Kafka are you on? Can you paste the complete 
exception stacktrace that you are running into?

-Jaikiran
On 19-May-2017, at 7:57 PM, Apache Jenkins Server  
wrote:

Hello,

I have no idea how to submit a bug : amongst all the help given to
participate to the project, submitting a bug is the hardest and least
documented, neither is easy to do. So I hope this email will reach the
proper hands on your side :-)

The bug is easy to reproduce:

1 - Create a topic with dots inside the name (eg: test.test2.test3)

2 - Mark the topic for deletion

3 - Stop the server (using the stop service kafka command, which is an ugly
kill -9 in behind...)

4 - Restart the server: it will say that it cannot find the topic to
delete, because it seems to look for an extension but instead finds a topic
name part.

This is a simble mistake: the string is splitted on the dots, and instead
of taking the last element of the list returned by the split() it seems to
look for the first element + 1 (hence thinking topic is test and its
extension is test2.test3.log). So the server crashes and is impossible to
restore/restart until you perform a manual deletion.

A fix and a cleaner shutdown method which will prevent creating "corrupted
index files" is welcomed :-D

Thank awesome guys

Have a good day

-- 

Fabien KERBOUCI

M. +33 (0) 6 75 45 56 62 | T. +33 (0) 1 85 08 88 83

6, rue Antonin Raynaud - 92300 Levallois-Perret

fkerbo...@opensense.fr | www.opensense.fr

[jira] [Commented] (KAFKA-5266) Follow-up improvements for consumer offset reset tool (KIP-122)

[Jorge Quilcate (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5266?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017475#comment-16017475
 ] 

Jorge Quilcate commented on KAFKA-5266:
---

[~hachikuji] some comments about the improvements:

1. Agree, it will be fixed for the following cases: `resetToOffset`, 
`resetFromFile` and `resetShiftBy`; as in the other cases, offsets are gathered 
from the log, not by input.
2. Agree.
3. This option was kept to support the following case: export current offsets, 
and use it later to go back to an specific point in time. I think this scenario 
is usable enough to maintain it.
4. This looks OK, but I'm not sure if I get where this could happen. Could you 
point where this is possible? 

> Follow-up improvements for consumer offset reset tool (KIP-122)
> ---
>
> Key: KAFKA-5266
> URL: https://issues.apache.org/jira/browse/KAFKA-5266
> Project: Kafka
>  Issue Type: Bug
>  Components: tools
>Reporter: Jason Gustafson
>Assignee: Jorge Quilcate
> Fix For: 0.11.0.0
>
>
> 1. We should try to ensure that offsets are in range for the topic partition. 
> We currently only verify this for the shift option.
> 2. If you provide a CSV file, you shouldn't need to specify one of the 
> --all-topics or --topic options.
> 3. We currently support a "reset to current offsets" option if none of the 
> supported reset options are provided. This seems kind of useless. Perhaps we 
> should just enforce that one of the reset options is provided.
> 4. The command fails with an NPE if we cannot find one of the offsets we are 
> trying to reset. It would be better to raise an exception with a friendlier 
> message.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Jenkins build is back to normal : kafka-0.11.0-jdk7 #3

[J Pai]

This sounds like https://issues.apache.org/jira/browse/KAFKA-5232. It’s been 
fixed a few days back in upstream and will be available in next releases.

Having said that, you note something about split() method which wasn’t 
applicable here. Which version of Kafka are you on? Can you paste the complete 
exception stacktrace that you are running into?

-Jaikiran
On 19-May-2017, at 7:57 PM, Apache Jenkins Server  
wrote:

Hello,

I have no idea how to submit a bug : amongst all the help given to
participate to the project, submitting a bug is the hardest and least
documented, neither is easy to do. So I hope this email will reach the
proper hands on your side :-)

The bug is easy to reproduce:

1 - Create a topic with dots inside the name (eg: test.test2.test3)

2 - Mark the topic for deletion

3 - Stop the server (using the stop service kafka command, which is an ugly
kill -9 in behind...)

4 - Restart the server: it will say that it cannot find the topic to
delete, because it seems to look for an extension but instead finds a topic
name part.

This is a simble mistake: the string is splitted on the dots, and instead
of taking the last element of the list returned by the split() it seems to
look for the first element + 1 (hence thinking topic is test and its
extension is test2.test3.log). So the server crashes and is impossible to
restore/restart until you perform a manual deletion.

A fix and a cleaner shutdown method which will prevent creating "corrupted
index files" is welcomed :-D

Thank awesome guys

Have a good day

-- 

Fabien KERBOUCI

M. +33 (0) 6 75 45 56 62 | T. +33 (0) 1 85 08 88 83

6, rue Antonin Raynaud - 92300 Levallois-Perret

fkerbo...@opensense.fr | www.opensense.fr

Re: Bug Submit - Critical

[Ismael Juma]

Hi Fabien,

This has already been fixed, see:

https://issues.apache.org/jira/browse/KAFKA-5232
https://github.com/apache/kafka/commit/f56bbb6510df8c12db3ad075e2f6c78dd0092d60
https://github.com/apache/kafka/commit/1ca2b1aacc2994e441c1a949310129d3e78532b5

To file issues in the future, please go to:

https://issues.apache.org/jira/browse/KAFKA

Thanks,
Ismael

On Fri, May 19, 2017 at 3:17 PM, Fabien Kerbouci 
wrote:

> Hello,
>
> I have no idea how to submit a bug : amongst all the help given to
> participate to the project, submitting a bug is the hardest and least
> documented, neither is easy to do. So I hope this email will reach the
> proper hands on your side :-)
>
> The bug is easy to reproduce:
>
> 1 - Create a topic with dots inside the name (eg: test.test2.test3)
>
> 2 - Mark the topic for deletion
>
> 3 - Stop the server (using the stop service kafka command, which is an ugly
> kill -9 in behind...)
>
> 4 - Restart the server: it will say that it cannot find the topic to
> delete, because it seems to look for an extension but instead finds a topic
> name part.
>
> This is a simble mistake: the string is splitted on the dots, and instead
> of taking the last element of the list returned by the split() it seems to
> look for the first element + 1 (hence thinking topic is test and its
> extension is test2.test3.log). So the server crashes and is impossible to
> restore/restart until you perform a manual deletion.
>
> A fix and a cleaner shutdown method which will prevent creating "corrupted
> index files" is welcomed :-D
>
> Thank awesome guys
>
> Have a good day
>
> --
>
> Fabien KERBOUCI
>
> M. +33 (0) 6 75 45 56 62 | T. +33 (0) 1 85 08 88 83
>
> 6, rue Antonin Raynaud - 92300 Levallois-Perret
>
> fkerbo...@opensense.fr | www.opensense.fr
>

Bug Submit - Critical

[Fabien Kerbouci]

Hello,

I have no idea how to submit a bug : amongst all the help given to
participate to the project, submitting a bug is the hardest and least
documented, neither is easy to do. So I hope this email will reach the
proper hands on your side :-)

The bug is easy to reproduce:

1 - Create a topic with dots inside the name (eg: test.test2.test3)

2 - Mark the topic for deletion

3 - Stop the server (using the stop service kafka command, which is an ugly
kill -9 in behind...)

4 - Restart the server: it will say that it cannot find the topic to
delete, because it seems to look for an extension but instead finds a topic
name part.

This is a simble mistake: the string is splitted on the dots, and instead
of taking the last element of the list returned by the split() it seems to
look for the first element + 1 (hence thinking topic is test and its
extension is test2.test3.log). So the server crashes and is impossible to
restore/restart until you perform a manual deletion.

A fix and a cleaner shutdown method which will prevent creating "corrupted
index files" is welcomed :-D

Thank awesome guys

Have a good day

-- 

Fabien KERBOUCI

M. +33 (0) 6 75 45 56 62 | T. +33 (0) 1 85 08 88 83

6, rue Antonin Raynaud - 92300 Levallois-Perret

fkerbo...@opensense.fr | www.opensense.fr

Jenkins build is back to normal : kafka-0.11.0-jdk7 #3

[Apache Jenkins Server]

See 

[jira] [Updated] (KAFKA-5279) TransactionCoordinator must expire transactionalIds

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy updated KAFKA-5279:
--
Status: Patch Available  (was: Open)

> TransactionCoordinator must expire transactionalIds
> ---
>
> Key: KAFKA-5279
> URL: https://issues.apache.org/jira/browse/KAFKA-5279
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Apurva Mehta
>Assignee: Damian Guy
>Priority: Blocker
>  Labels: exactly-once
>
> Currently transactionalIds are not expired anywhere, so we accumulate forever.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5279) TransactionCoordinator must expire transactionalIds

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017447#comment-16017447
 ] 

ASF GitHub Bot commented on KAFKA-5279:
---

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/3101

KAFKA-5279: TransactionCoordinator must expire transactionalIds

remove transactions that have not been updated for at least 
`transactional.id.expiration.ms`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka kafka-5279

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3101.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3101






> TransactionCoordinator must expire transactionalIds
> ---
>
> Key: KAFKA-5279
> URL: https://issues.apache.org/jira/browse/KAFKA-5279
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Apurva Mehta
>Assignee: Damian Guy
>Priority: Blocker
>  Labels: exactly-once
>
> Currently transactionalIds are not expired anywhere, so we accumulate forever.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3101: KAFKA-5279: TransactionCoordinator must expire tra...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/3101

KAFKA-5279: TransactionCoordinator must expire transactionalIds

remove transactions that have not been updated for at least 
`transactional.id.expiration.ms`

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka kafka-5279

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3101.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3101






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk7 #2233

[Apache Jenkins Server]

See 


Changes:

[ismael] KAFKA-3878; Support exponential backoff policy via 
reconnect.backoff.max

--
[...truncated 886.37 KB...]
kafka.controller.ControllerIntegrationTest > testPartitionReassignment PASSED

kafka.controller.ControllerIntegrationTest > testTopicPartitionExpansion STARTED

kafka.controller.ControllerIntegrationTest > testTopicPartitionExpansion PASSED

kafka.controller.ControllerIntegrationTest > 
testControllerMoveIncrementsControllerEpoch STARTED

kafka.controller.ControllerIntegrationTest > 
testControllerMoveIncrementsControllerEpoch PASSED

kafka.controller.ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionEnabled STARTED

kafka.controller.ControllerIntegrationTest > 
testLeaderAndIsrWhenEntireIsrOfflineAndUncleanLeaderElectionEnabled PASSED

kafka.controller.ControllerIntegrationTest > testEmptyCluster STARTED

kafka.controller.ControllerIntegrationTest > testEmptyCluster PASSED

kafka.controller.ControllerIntegrationTest > testPreferredReplicaLeaderElection 
STARTED

kafka.controller.ControllerIntegrationTest > testPreferredReplicaLeaderElection 
PASSED

kafka.controller.ControllerFailoverTest > testMetadataUpdate STARTED

kafka.controller.ControllerFailoverTest > testMetadataUpdate SKIPPED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic STARTED

kafka.javaapi.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytesWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > 
testIteratorIsConsistentWithCompression PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testIteratorIsConsistent PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEqualsWithCompression 
PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testWrittenEqualsRead PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testEquals PASSED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes STARTED

kafka.javaapi.message.ByteBufferMessageSetTest > testSizeInBytes PASSED

kafka.network.SocketServerTest > testGracefulClose STARTED

kafka.network.SocketServerTest > testGracefulClose PASSED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
STARTED

kafka.network.SocketServerTest > testClientDisconnectionUpdatesRequestMetrics 
PASSED

kafka.network.SocketServerTest > testProcessorMetricsTags STARTED

kafka.network.SocketServerTest > testProcessorMetricsTags PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIp STARTED

kafka.network.SocketServerTest > testMaxConnectionsPerIp PASSED

kafka.network.SocketServerTest > 
testBrokerSendAfterChannelClosedUpdatesRequestMetrics STARTED

kafka.network.SocketServerTest > 
testBrokerSendAfterChannelClosedUpdatesRequestMetrics PASSED

kafka.network.SocketServerTest > simpleRequest STARTED

kafka.network.SocketServerTest > simpleRequest PASSED

kafka.network.SocketServerTest > testMetricCollectionAfterShutdown STARTED

kafka.network.SocketServerTest > testMetricCollectionAfterShutdown PASSED

kafka.network.SocketServerTest > testSessionPrincipal STARTED

kafka.network.SocketServerTest > testSessionPrincipal PASSED

kafka.network.SocketServerTest > testMaxConnectionsPerIpOverrides STARTED

kafka.network.SocketServerTest > testMaxConnectionsPerIpOverrides PASSED

kafka.network.SocketServerTest > testSocketsCloseOnShutdown STARTED

kafka.network.SocketServerTest > testSocketsCloseOnShutdown PASSED

kafka.network.SocketServerTest > testSslSocketServer STARTED

kafka.network.SocketServerTest > testSslSocketServer PASSED

kafka.network.SocketServerTest > tooBigRequestIsRejected STARTED

kafka.network.SocketServerTest > tooBigRequestIsRejected PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.

[jira] [Assigned] (KAFKA-5245) KStream builder should capture serdes

[anugrah (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

anugrah reassigned KAFKA-5245:
--

Assignee: anugrah

> KStream builder should capture serdes 
> --
>
> Key: KAFKA-5245
> URL: https://issues.apache.org/jira/browse/KAFKA-5245
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.2.0, 0.10.2.1
>Reporter: Yeva Byzek
>Assignee: anugrah
>Priority: Minor
>  Labels: beginner, newbie
>
> Even if one specifies a serdes in `builder.stream`, later a call to 
> `groupByKey` may require the serdes again if it differs from the configured 
> streams app serdes. The preferred behavior is that if no serdes is provided 
> to `groupByKey`, it should use whatever was provided in `builder.stream` and 
> not what was in the app.
> From the current docs:
> “When to set explicit serdes: Variants of groupByKey exist to override the 
> configured default serdes of your application, which you must do if the key 
> and/or value types of the resulting KGroupedStream do not match the 
> configured default serdes.”



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5245) KStream builder should capture serdes

[anugrah (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017431#comment-16017431
 ] 

anugrah commented on KAFKA-5245:


sure. I will start on this.

> KStream builder should capture serdes 
> --
>
> Key: KAFKA-5245
> URL: https://issues.apache.org/jira/browse/KAFKA-5245
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.2.0, 0.10.2.1
>Reporter: Yeva Byzek
>Priority: Minor
>  Labels: beginner, newbie
>
> Even if one specifies a serdes in `builder.stream`, later a call to 
> `groupByKey` may require the serdes again if it differs from the configured 
> streams app serdes. The preferred behavior is that if no serdes is provided 
> to `groupByKey`, it should use whatever was provided in `builder.stream` and 
> not what was in the app.
> From the current docs:
> “When to set explicit serdes: Variants of groupByKey exist to override the 
> configured default serdes of your application, which you must do if the key 
> and/or value types of the resulting KGroupedStream do not match the 
> configured default serdes.”



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5293) Do not apply exponential backoff if users have overridden reconnect.backoff.ms

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5293:
---
Description: 
The PR for KAFKA-3878 implemented KIP-144 with one exception: it automatically 
enables exponential backoff for the producer and consumer even if 
reconnect.backoff.ms is set by users. The KIP stated that this would not be the 
case.

As part of this JIRA, we should also add a few unit tests for connectionDelay 
and perhaps consider enabling exponential backoff for Connect and Streams as 
well.

  was:
The PR for KAFKA-3878 implemented KIP-144 with one exception: it automatically 
enables exponential backoff for the producer and consumer even if 
reconnect.backoff.ms is set by users. The KIP stated that this would not be the 
case.

As part of this JIRA, we should also add a few unit tests for connectionDelay.


> Do not apply exponential backoff if users have overridden reconnect.backoff.ms
> --
>
> Key: KAFKA-5293
> URL: https://issues.apache.org/jira/browse/KAFKA-5293
> Project: Kafka
>  Issue Type: Bug
>Reporter: Colin P. McCabe
>Assignee: Ismael Juma
>Priority: Critical
> Fix For: 0.11.0.0
>
>
> The PR for KAFKA-3878 implemented KIP-144 with one exception: it 
> automatically enables exponential backoff for the producer and consumer even 
> if reconnect.backoff.ms is set by users. The KIP stated that this would not 
> be the case.
> As part of this JIRA, we should also add a few unit tests for connectionDelay 
> and perhaps consider enabling exponential backoff for Connect and Streams as 
> well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5293) Do not apply exponential backoff if users have overridden reconnect.backoff.ms

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5293:
---
Issue Type: Bug  (was: Improvement)

> Do not apply exponential backoff if users have overridden reconnect.backoff.ms
> --
>
> Key: KAFKA-5293
> URL: https://issues.apache.org/jira/browse/KAFKA-5293
> Project: Kafka
>  Issue Type: Bug
>Reporter: Colin P. McCabe
>Assignee: Ismael Juma
>Priority: Critical
> Fix For: 0.11.0.0
>
>
> The PR for KAFKA-3878 implemented KIP-144 with one exception: it 
> automatically enables exponential backoff for the producer and consumer even 
> if reconnect.backoff.ms is set by users. The KIP stated that this would not 
> be the case.
> As part of this JIRA, we should also add a few unit tests for connectionDelay.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5293) Do not apply exponential backoff if users have overridden reconnect.backoff.ms

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-5293:
--

 Summary: Do not apply exponential backoff if users have overridden 
reconnect.backoff.ms
 Key: KAFKA-5293
 URL: https://issues.apache.org/jira/browse/KAFKA-5293
 Project: Kafka
  Issue Type: Improvement
Reporter: Colin P. McCabe
Assignee: Ismael Juma
Priority: Critical
 Fix For: 0.11.0.0


The PR for KAFKA-3878 implemented KIP-144 with one exception: it automatically 
enables exponential backoff for the producer and consumer even if 
reconnect.backoff.ms is set by users. The KIP stated that this would not be the 
case.

As part of this JIRA, we should also add a few unit tests for connectionDelay.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-3878) Exponential backoff for broker reconnect attempts (KIP-144)

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3878:
---
Summary: Exponential backoff for broker reconnect attempts (KIP-144)  (was: 
Exponential backoff for broker reconnect attempts)

> Exponential backoff for broker reconnect attempts (KIP-144)
> ---
>
> Key: KAFKA-3878
> URL: https://issues.apache.org/jira/browse/KAFKA-3878
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, network
>Reporter: Dana Powers
>Assignee: Dana Powers
>  Labels: kip
> Fix For: 0.11.0.0
>
>
> The client currently uses a constant backoff policy, configured via 
> 'reconnect.backoff.ms' . To reduce network load during longer broker outages, 
> it would be useful to support an optional exponentially increasing backoff 
> policy.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-3878) Exponential backoff for broker reconnect attempts

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017372#comment-16017372
 ] 

ASF GitHub Bot commented on KAFKA-3878:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1523


> Exponential backoff for broker reconnect attempts
> -
>
> Key: KAFKA-3878
> URL: https://issues.apache.org/jira/browse/KAFKA-3878
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, network
>Reporter: Dana Powers
>Assignee: Dana Powers
>  Labels: kip
> Fix For: 0.11.0.0
>
>
> The client currently uses a constant backoff policy, configured via 
> 'reconnect.backoff.ms' . To reduce network load during longer broker outages, 
> it would be useful to support an optional exponentially increasing backoff 
> policy.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #1523: KAFKA-3878: Support exponential backoff policy via...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1523


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3878) Exponential backoff for broker reconnect attempts

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3878:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 1523
[https://github.com/apache/kafka/pull/1523]

> Exponential backoff for broker reconnect attempts
> -
>
> Key: KAFKA-3878
> URL: https://issues.apache.org/jira/browse/KAFKA-3878
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients, network
>Reporter: Dana Powers
>Assignee: Dana Powers
>  Labels: kip
> Fix For: 0.11.0.0
>
>
> The client currently uses a constant backoff policy, configured via 
> 'reconnect.backoff.ms' . To reduce network load during longer broker outages, 
> it would be useful to support an optional exponentially increasing backoff 
> policy.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-5154) Kafka Streams throws NPE during rebalance

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy reassigned KAFKA-5154:
-

Assignee: Damian Guy  (was: Matthias J. Sax)

> Kafka Streams throws NPE during rebalance
> -
>
> Key: KAFKA-5154
> URL: https://issues.apache.org/jira/browse/KAFKA-5154
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Lukas Gemela
>Assignee: Damian Guy
> Attachments: 5154_problem.log, clio_afa596e9b809.gz, clio_reduced.gz, 
> clio.txt.gz
>
>
> please see attached log, Kafka streams throws NullPointerException during 
> rebalance, which is caught by our custom exception handler
> {noformat}
> 2017-04-30T17:44:17,675 INFO  kafka-coordinator-heartbeat-thread | hades 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead()
>  @618 - Marking the coordinator 10.210.200.144:9092 (id: 2147483644 rack: 
> null) dead for group hades
> 2017-04-30T17:44:27,395 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess() 
> @573 - Discovered coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) 
> for group hades.
> 2017-04-30T17:44:27,941 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare()
>  @393 - Revoking previously assigned partitions [poseidonIncidentFeed-27, 
> poseidonIncidentFeed-29, poseidonIncidentFeed-30, poseidonIncidentFeed-18] 
> for group hades
> 2017-04-30T17:44:27,947 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
>  @407 - (Re-)joining group hades
> 2017-04-30T17:44:48,468 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
>  @407 - (Re-)joining group hades
> 2017-04-30T17:44:53,628 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
>  @407 - (Re-)joining group hades
> 2017-04-30T17:45:09,587 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
>  @407 - (Re-)joining group hades
> 2017-04-30T17:45:11,961 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess() 
> @375 - Successfully joined group hades with generation 99
> 2017-04-30T17:45:13,126 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete()
>  @252 - Setting newly assigned partitions [poseidonIncidentFeed-11, 
> poseidonIncidentFeed-27, poseidonIncidentFeed-25, poseidonIncidentFeed-29, 
> poseidonIncidentFeed-19, poseidonIncidentFeed-18] for group hades
> 2017-04-30T17:46:37,254 INFO  kafka-coordinator-heartbeat-thread | hades 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead()
>  @618 - Marking the coordinator 10.210.200.144:9092 (id: 2147483644 rack: 
> null) dead for group hades
> 2017-04-30T18:04:25,993 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess() 
> @573 - Discovered coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) 
> for group hades.
> 2017-04-30T18:04:29,401 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare()
>  @393 - Revoking previously assigned partitions [poseidonIncidentFeed-11, 
> poseidonIncidentFeed-27, poseidonIncidentFeed-25, poseidonIncidentFeed-29, 
> poseidonIncidentFeed-19, poseidonIncidentFeed-18] for group hades
> 2017-04-30T18:05:10,877 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
>  @407 - (Re-)joining group hades
> 2017-05-01T00:01:55,707 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead()
>  @618 - Marking the coordinator 10.210.200.144:9092 (id: 2147483644 rack: 
> null) dead for group hades
> 2017-05-01T00:01:59,027 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess() 
> @573 - Discovered coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) 
> for group hades.
> 2017-05-01T00:01:59,031 ERROR StreamThread-1 
> org.apache.kafka.streams.processor.internals.StreamThread.run() @376 - 
> stream-thread [StreamThread-1] Streams application error during processing:
>  java.lang.NullPointerException
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:619)
>  ~[kafka-streams-0.10.2.0.jar!/:?]
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
>  [kafka-streams-0.10.2.0.jar!/:?]
> 2017-05-01T00:02:00,038 INFO  StreamThread-1 
> org.apache.kafka.clients.producer.KafkaProducer.close() 

Re: [VOTE] KIP-117: Add a public AdminClient API for Kafka admin operations

[Ismael Juma]

Hi all,

Feedback from people who tried the AdminClient is that auto topic creation
during describe is unexpected and confusing. This is consistent with the
reaction of most people when they learn that MetadataRequest can cause
topics to be created. We had assumed that we'd tackle this issue for all
the clients as part of deprecation of server-side auto topic creation in
favour of client-side auto-topic creation.

However, it would be better to do the right thing for the AdminClient from
the start. Users will be less confused and we won't have to deal with
compatibility concerns. Jason suggested a simple solution: make it possible
to disallow auto topic creation when sending the metadata request. The
AdminClient would take advantage of this now (i.e. 0.11.0.0) while the
producer and consumer would retain the existing behaviour. In a subsequent
release, we'll work out the details of how to move away from server-side
auto topic creation for the producer and consumer (taking into account the
compatibility impact).

Because of the protocol change, this solution would only help in cases
where the AdminClient is describing topics from a 0.11.0.0 or newer broker.

I submitted a PR for this and it's small and straightforward:

https://github.com/apache/kafka/pull/3098

Thoughts?

Ismael

On Sat, Mar 25, 2017 at 1:25 AM, Colin McCabe  wrote:

> With binding +1 votes from Gwen Shapira, Sriram Subramanian, and Grant
> Henke, and a non-binding vote from Dong Lin, the vote passes.  There
> were no +0 or -1 votes.  As mentioned earlier, the interface will be
> unstable at first and we will continue to evolve it.
>
> thanks,
> Colin McCabe
>
>
> On Wed, Mar 22, 2017, at 10:21, Colin McCabe wrote:
> > On Fri, Mar 17, 2017, at 10:50, Jun Rao wrote:
> > > Hi, Colin,
> > >
> > > Thanks for the KIP. Looks good overall. A few comments below.
> > >
> > > 1. Sometimes we return
> > > CompletableFuture>
> > > and some other times we return
> > > Map>
> > > , which doesn't seem consistent. Is that intentional?
> >
> > Yes, this is intentional.  We got feedback from some people that they
> > wanted a single future that would fail if anything failed.  Other people
> > wanted to be able to detect failures on individual elements of a batch.
> > This API lets us have both (you just choose which future you want to
> > wait on).
> >
> > >
> > > 2. We support batching in CreateTopic/DeleteTopic/ListTopic, but not
> in
> > > DescribeTopic. Should we add batching in DescribeTopic to make it
> > > consistent?
> >
> > Good idea.  Let's add batching to DescribeTopic(s).
> >
> > > Also, both ListTopic and DescribeTopic seem to return
> > > TopicDescription. Could we just consolidate the two by just keeping
> > > DescribeTopic?
> >
> > Sorry, that was a typo.  ListTopics is supposed to return TopicListing,
> > which tells you only the name of the topic and whether it is internal.
> > The idea is that later we will add another RPC which allows us to fetch
> > just this information, and not the other topic fields. That way, we can
> > be more efficient.  The idea is that ListTopics is like readdir()/ls and
> > DescribeTopics is like stat().  Getting detailed information about
> > 1,000s of topics could be quite a resource hog for cluster management
> > systems in a big cluster.
> >
> > >
> > > 3. listNodes: At the request protocol level, we can get things like
> > > clusterId and controller broker id. Both are useful info from an admin
> > > perspective, but are not exposed through the api. Perhaps we can
> > > generalize
> > > listNodes to sth like describeCluster so that we can return those
> > > additional info as well?
> >
> > Yeah, let's change listNodes -> describeCluster.
> >
> > >
> > > 4. Configurations: To support security, we will need to include all
> > > properties related to SSL and SASL.
> >
> > Yeah
> >
> > best,
> > Colin
> >
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Thu, Mar 16, 2017 at 11:59 PM, Colin McCabe 
> > > wrote:
> > >
> > > > Hi all,
> > > >
> > > > It seems like people agree with the basic direction of the proposal
> and
> > > > the API, including the operations that are included, the async and
> > > > batching support, and the mechanisms for extending it in the
> future.  If
> > > > there's no more votes, I'd like to close the vote and start progress
> on
> > > > this.
> > > >
> > > > I think the API should be unstable for a while (at least until the
> 0.11
> > > > release is made), so we can consider ways to improve it.  A few have
> > > > been suggested here: removing or adding functions, renaming things a
> > > > bit, or using request objects instead of options objects.  I think
> once
> > > > people try out the API a bit, it will be easier to evaluate these.
> > > >
> > > > best,
> > > > Colin
> > > >
> > > >
> > > > On Tue, Mar 14, 2017, at 10:12, Dong Lin wrote:
> > > > > +1
> > > > >
> > > > > On Tue, Mar 14, 2017 at 8:50 AM, Grant Henke 
> > > > wrote:
> > > > >
> > > > > > +1
> > > > >

[jira] [Created] (KAFKA-5292) Authorization tests for AdminClient

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-5292:
--

 Summary: Authorization tests for AdminClient
 Key: KAFKA-5292
 URL: https://issues.apache.org/jira/browse/KAFKA-5292
 Project: Kafka
  Issue Type: Sub-task
Reporter: Ismael Juma
Assignee: Colin P. McCabe
Priority: Critical
 Fix For: 0.11.0.0


AuthorizerIntegrationTest includes protocol, consumer and producer tests. We 
should add tests for the AdminClient as well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[Vote] KIP-150 - Kafka-Streams Cogroup

[Kyle Winkelman]

Hello all,

I would like to start the vote on KIP-150.
https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+Kafka-Streams+Cogroup

Thanks,
Kyle

[jira] [Commented] (KAFKA-5278) kafka-console-consumer: `--value-deserializer` is not working but `--property value.deserializer` does

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017280#comment-16017280
 ] 

ASF GitHub Bot commented on KAFKA-5278:
---

GitHub user amethystic opened a pull request:

https://github.com/apache/kafka/pull/3100

KAFKA-5278: Have console-consumer support `--value-deserializer`

In the original implementation, console-consumer fails to honor 
`--value-deserializer` config.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amethystic/kafka KAFKA-5278

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3100.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3100


commit 34de8be03217645dbf1c4111fc495d62dcc59028
Author: amethystic 
Date:   2017-05-19T11:39:08Z

KAFKA-5278: Have console-consumer support `--value-deserializer`




> kafka-console-consumer: `--value-deserializer` is not working but `--property 
> value.deserializer` does
> --
>
> Key: KAFKA-5278
> URL: https://issues.apache.org/jira/browse/KAFKA-5278
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.2.1
>Reporter: Yeva Byzek
>Assignee: huxi
>Priority: Minor
>
> kafka-console-consumer: {{--value-deserializer}} is not working but 
> {{--property value.deserializer}} is working
> 1. Does not work
> {noformat}
> $ kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning 
> --topic TEST1  --value-deserializer 
> org.apache.kafka.common.serialization.LongDeserializer
> [2017-05-18 13:09:41,745] ERROR Error processing message, terminating 
> consumer process:  (kafka.tools.ConsoleConsumer$)
> java.lang.ClassCastException: java.lang.Long cannot be cast to [B
>   at kafka.consumer.NewShinyConsumer.receive(BaseConsumer.scala:100)
>   at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:120)
>   at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:75)
>   at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:50)
>   at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
> Processed a total of 0 messages
> {noformat}
> 2. Does work
> {noformat}
> $ kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning 
> --topic TEST1  --property 
> value.deserializer=org.apache.kafka.common.serialization.LongDeserializer
> 1000
> 2500
> 2000
> 5500
> 8000
> {noformat}
> Without either, the output is
> {noformat}
> $ kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning 
> --topic TEST1
> ?
>   ?
> ?
> |
> @
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3100: KAFKA-5278: Have console-consumer support `--value...

[amethystic]

GitHub user amethystic opened a pull request:

https://github.com/apache/kafka/pull/3100

KAFKA-5278: Have console-consumer support `--value-deserializer`

In the original implementation, console-consumer fails to honor 
`--value-deserializer` config.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/amethystic/kafka KAFKA-5278

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3100.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3100


commit 34de8be03217645dbf1c4111fc495d62dcc59028
Author: amethystic 
Date:   2017-05-19T11:39:08Z

KAFKA-5278: Have console-consumer support `--value-deserializer`




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (KAFKA-5278) kafka-console-consumer: `--value-deserializer` is not working but `--property value.deserializer` does

[huxi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

huxi reassigned KAFKA-5278:
---

Assignee: huxi

> kafka-console-consumer: `--value-deserializer` is not working but `--property 
> value.deserializer` does
> --
>
> Key: KAFKA-5278
> URL: https://issues.apache.org/jira/browse/KAFKA-5278
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.2.1
>Reporter: Yeva Byzek
>Assignee: huxi
>Priority: Minor
>
> kafka-console-consumer: {{--value-deserializer}} is not working but 
> {{--property value.deserializer}} is working
> 1. Does not work
> {noformat}
> $ kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning 
> --topic TEST1  --value-deserializer 
> org.apache.kafka.common.serialization.LongDeserializer
> [2017-05-18 13:09:41,745] ERROR Error processing message, terminating 
> consumer process:  (kafka.tools.ConsoleConsumer$)
> java.lang.ClassCastException: java.lang.Long cannot be cast to [B
>   at kafka.consumer.NewShinyConsumer.receive(BaseConsumer.scala:100)
>   at kafka.tools.ConsoleConsumer$.process(ConsoleConsumer.scala:120)
>   at kafka.tools.ConsoleConsumer$.run(ConsoleConsumer.scala:75)
>   at kafka.tools.ConsoleConsumer$.main(ConsoleConsumer.scala:50)
>   at kafka.tools.ConsoleConsumer.main(ConsoleConsumer.scala)
> Processed a total of 0 messages
> {noformat}
> 2. Does work
> {noformat}
> $ kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning 
> --topic TEST1  --property 
> value.deserializer=org.apache.kafka.common.serialization.LongDeserializer
> 1000
> 2500
> 2000
> 5500
> 8000
> {noformat}
> Without either, the output is
> {noformat}
> $ kafka-console-consumer --bootstrap-server localhost:9092 --from-beginning 
> --topic TEST1
> ?
>   ?
> ?
> |
> @
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : kafka-trunk-jdk8 #1559

[Apache Jenkins Server]

See 

[jira] [Commented] (KAFKA-4660) Improve test coverage KafkaStreams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4660?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017250#comment-16017250
 ] 

ASF GitHub Bot commented on KAFKA-4660:
---

GitHub user umesh9794 opened a pull request:

https://github.com/apache/kafka/pull/3099

KAFKA-4660 : Improve test coverage KafkaStreams

@dguy , @mjsax Please review the PR and let me know your comments. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/umesh9794/kafka mylocal

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3099.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3099


commit ed31c1c2f7f8d9b1ef675d80433154025fd2a639
Author: umesh chaudhary 
Date:   2017-05-18T12:18:02Z

Rebase: KAFKA-5171 : TC should not accept empty string transactional id

commit d0a98cfe49841a99433657a068c83b718833898d
Author: umesh chaudhary 
Date:   2017-05-19T04:14:03Z

Merge branch 'trunk' of https://github.com/apache/kafka into mylocal

commit ecb189739487704daa51afc2346758d940bee910
Author: umesh chaudhary 
Date:   2017-05-19T11:06:28Z

KAFKA-4660 : Improve test coverage KafkaStreams




> Improve test coverage KafkaStreams
> --
>
> Key: KAFKA-4660
> URL: https://issues.apache.org/jira/browse/KAFKA-4660
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Reporter: Damian Guy
>Priority: Minor
> Fix For: 0.11.0.0
>
>
> {{toString}} is used to print the topology, so probably should have a unit 
> test.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3099: KAFKA-4660 : Improve test coverage KafkaStreams

[umesh9794]

GitHub user umesh9794 opened a pull request:

https://github.com/apache/kafka/pull/3099

KAFKA-4660 : Improve test coverage KafkaStreams

@dguy , @mjsax Please review the PR and let me know your comments. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/umesh9794/kafka mylocal

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3099.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3099


commit ed31c1c2f7f8d9b1ef675d80433154025fd2a639
Author: umesh chaudhary 
Date:   2017-05-18T12:18:02Z

Rebase: KAFKA-5171 : TC should not accept empty string transactional id

commit d0a98cfe49841a99433657a068c83b718833898d
Author: umesh chaudhary 
Date:   2017-05-19T04:14:03Z

Merge branch 'trunk' of https://github.com/apache/kafka into mylocal

commit ecb189739487704daa51afc2346758d940bee910
Author: umesh chaudhary 
Date:   2017-05-19T11:06:28Z

KAFKA-4660 : Improve test coverage KafkaStreams




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-5291) AdminClient should not trigger auto creation of topics

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5291:
---
Status: Patch Available  (was: Open)

> AdminClient should not trigger auto creation of topics
> --
>
> Key: KAFKA-5291
> URL: https://issues.apache.org/jira/browse/KAFKA-5291
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.11.0.0
>
>
> Feedback from people who tried the AdminClient is that auto topic creation 
> during describe is very confusing. We expected that, but we also thought that 
> there wasn't much that could be done.
> However, there's a simple solution (suggested by [~hachikuji]): make it 
> possible to disallow auto topic creation when sending the metadata request. 
> We can keep the existing behaviour for the producer and consumer (until we 
> move to client-side auto topic creation) while doing the right thing for 
> AdminClient (which is new).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5291) AdminClient should not trigger auto creation of topics

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5291?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017222#comment-16017222
 ] 

ASF GitHub Bot commented on KAFKA-5291:
---

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/3098

KAFKA-5291: AdminClient should not trigger auto creation of topics

- Added a boolean allow_auto_topic_creation to MetadataRequest.
I didn’t bump the version a second time since we did it once for this
release already, but this needs to be verified.

- Set it to false in the new AdminClient and StreamsKafkaClient
(which exists for the purpose of creating topics manually); set it
to true everywhere else for now. Other clients will eventually
rely on client-side auto topic creation, but that’s not there yet.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-5291-admin-client-no-auto-topic-creation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3098.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3098






> AdminClient should not trigger auto creation of topics
> --
>
> Key: KAFKA-5291
> URL: https://issues.apache.org/jira/browse/KAFKA-5291
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.11.0.0
>
>
> Feedback from people who tried the AdminClient is that auto topic creation 
> during describe is very confusing. We expected that, but we also thought that 
> there wasn't much that could be done.
> However, there's a simple solution (suggested by [~hachikuji]): make it 
> possible to disallow auto topic creation when sending the metadata request. 
> We can keep the existing behaviour for the producer and consumer (until we 
> move to client-side auto topic creation) while doing the right thing for 
> AdminClient (which is new).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3098: KAFKA-5291: AdminClient should not trigger auto cr...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/3098

KAFKA-5291: AdminClient should not trigger auto creation of topics

- Added a boolean allow_auto_topic_creation to MetadataRequest.
I didn’t bump the version a second time since we did it once for this
release already, but this needs to be verified.

- Set it to false in the new AdminClient and StreamsKafkaClient
(which exists for the purpose of creating topics manually); set it
to true everywhere else for now. Other clients will eventually
rely on client-side auto topic creation, but that’s not there yet.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-5291-admin-client-no-auto-topic-creation

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3098.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3098






---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-5289) One StopReplicaRequest will caused two Responses

[Ma Tianchi (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ma Tianchi updated KAFKA-5289:
--
Attachment: KAFKA-5289.patch

just catch it

> One StopReplicaRequest will caused two Responses
> 
>
> Key: KAFKA-5289
> URL: https://issues.apache.org/jira/browse/KAFKA-5289
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.10.0.0
>Reporter: xuzq
>Assignee: Ismael Juma
>Priority: Critical
> Fix For: 0.11.0.0
>
> Attachments: handleStopReplicaRequest.png, KAFKA-5289.patch
>
>
> After discussed with my friend markTC，we find a bug.
> One StopReplicaRequest will caused two Responses.
> At core/src/main/scala/kafka/server/KafkaApi.class 175 and 176 lines.
> When an exception caused at 
> 'replicaManager.replicaFetcherManager.shutdownIdleFetcherThreads()', 
> will also return two responses.
> one is at 175 lines 'requestChannel.sendResponse(new 
> RequestChannel.Response(request, new ResponseSend(request.connectionId, 
> responseHeader, response)))' and another at 111 lines 
> 'requestChannel.sendResponse(new Response(request, new 
> ResponseSend(request.connectionId, respHeader, response)))'.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4830) Augment KStream.print() to allow users pass in extra parameters in the printed string

[james chien (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4830?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017189#comment-16017189
 ] 

james chien commented on KAFKA-4830:


[~mjsax] Okay, I will study about how to write it :)

> Augment KStream.print() to allow users pass in extra parameters in the 
> printed string
> -
>
> Key: KAFKA-4830
> URL: https://issues.apache.org/jira/browse/KAFKA-4830
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: james chien
>  Labels: needs-kip, newbie
>
> Today {{KStream.print}} use the hard-coded result string as:
> {code}
> "[" + this.streamName + "]: " + keyToPrint + " , " + valueToPrint
> {code}
> And some users are asking to augment this so that they can customize the 
> output string as {{KStream.print(KeyValueMapper)}} :
> {code}
> "[" + this.streamName + "]: " + mapper.apply(keyToPrint, valueToPrint)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5134) Replace zkClient.getChildren method with zkUtils.getChildren

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017167#comment-16017167
 ] 

ASF GitHub Bot commented on KAFKA-5134:
---

GitHub user baluchicken opened a pull request:

https://github.com/apache/kafka/pull/3097

KAFKA-5134 Replace zkClient.getChildren method with zkUtils.getChildren

@ijuma plz review. I also created a version where I didn't refactor the  
getPartitionAssignmentForTopics and the getConsumersPerTopic methods, but I 
thought maybe it is a good idea to do and move the logic (where there is no ZK 
connection) to the ZkUtils Object.
About the two new methods, I wanted to make them private and visible only 
for testing, but because the tests where I use them are not in the same package 
I couldn't do it. 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/baluchicken/kafka-1 KAFKA-5134-2

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3097.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3097






> Replace zkClient.getChildren method with zkUtils.getChildren
> 
>
> Key: KAFKA-5134
> URL: https://issues.apache.org/jira/browse/KAFKA-5134
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Reporter: Balint Molnar
>Assignee: Balint Molnar
>
> Refactor related classes to use the zkUtils.getChildren method intead of the 
> zkClient variant.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)