Re: [DISCUSS]: KIP-161: streams record processing exception handlers

[Matthias J. Sax]

First a meta comment. KIP discussion should take place on the dev list
-- if user list is cc'ed please make sure to reply to both lists. Thanks.

Thanks for making the scope of the KIP clear. Makes a lot of sense to
focus on deserialization exceptions for now.

With regard to corrupted state stores, would it make sense to fail a
task and wipe out the store to repair it via recreation from the
changelog? That's of course a quite advance pattern, but I want to bring
it up to design the first step in a way such that we can get there (if
we think it's a reasonable idea).

I also want to comment about fail fast vs making progress. I think that
fail-fast must not always be the best option. The scenario I have in
mind is like this: you got a bunch of producers that feed the Streams
input topic. Most producers work find, but maybe one producer miss
behaves and the data it writes is corrupted. You might not even be able
to recover this lost data at any point -- thus, there is no reason to
stop processing but you just skip over those records. Of course, you
need to fix the root cause, and thus you need to alert (either via logs
of the exception handler directly) and you need to start to investigate
to find the bad producer, shut it down and fix it.

Here the dead letter queue comes into place. From my understanding, the
purpose of this feature is solely enable post debugging. I don't think
those record would be fed back at any point in time (so I don't see any
ordering issue -- a skipped record, with this regard, is just "fully
processed"). Thus, the dead letter queue should actually encode the
original records metadata (topic, partition offset etc) to enable such
debugging. I guess, this might also be possible if you just log the bad
records, but it would be harder to access (you first must find the
Streams instance that did write the log and extract the information from
there). Reading it from topic is much simpler.

I also want to mention the following. Assume you have such a topic with
some bad records and some good records. If we always fail-fast, it's
going to be super hard to process the good data. You would need to write
an extra app that copied the data into a new topic filtering out the bad
records (or apply the map() workaround withing stream). So I don't think
that failing fast is most likely the best option in production is
necessarily, true.

Or do you think there are scenarios, for which you can recover the
corrupted records successfully? And even if this is possible, it might
be a case for reprocessing instead of failing the whole application?
Also, if you think you can "repair" a corrupted record, should the
handler allow to return a "fixed" record? This would solve the ordering
problem.



-Matthias




On 5/30/17 1:47 AM, Michael Noll wrote:
> Thanks for your work on this KIP, Eno -- much appreciated!
> 
> - I think it would help to improve the KIP by adding an end-to-end code
> example that demonstrates, with the DSL and with the Processor API, how the
> user would write a simple application that would then be augmented with the
> proposed KIP changes to handle exceptions.  It should also become much
> clearer then that e.g. the KIP would lead to different code paths for the
> happy case and any failure scenarios.
> 
> - Do we have sufficient information available to make informed decisions on
> what to do next?  For example, do we know in which part of the topology the
> record failed? `ConsumerRecord` gives us access to topic, partition,
> offset, timestamp, etc., but what about topology-related information (e.g.
> what is the associated state store, if any)?
> 
> - Only partly on-topic for the scope of this KIP, but this is about the
> bigger picture: This KIP would give users the option to send corrupted
> records to dead letter queue (quarantine topic).  But, what pattern would
> we advocate to process such a dead letter queue then, e.g. how to allow for
> retries with backoff ("If the first record in the dead letter queue fails
> again, then try the second record for the time being and go back to the
> first record at a later time").  Jay and Jan already alluded to ordering
> problems that will be caused by dead letter queues. As I said, retries
> might be out of scope but perhaps the implications should be considered if
> possible?
> 
> Also, I wrote the text below before reaching the point in the conversation
> that this KIP's scope will be limited to exceptions in the category of
> poison pills / deserialization errors.  But since Jay brought up user code
> errors again, I decided to include it again.
> 
> snip
> A meta comment: I am not sure about this split between the code for the
> happy path (e.g. map/filter/... in the DSL) from the failure path (using
> exception handlers).  In Scala, for example, we can do:
> 
> scala> val computation = scala.util.Try(1 / 0)
> computation: scala.util.Try[Int] =
> Failure(java.lang.Ari

[jira] [Updated] (KAFKA-5283) Update clients and server code to make sure that epoch and sequence numbers wrap around

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-5283:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 3183
[https://github.com/apache/kafka/pull/3183]

> Update clients and server code to make sure that epoch and sequence numbers 
> wrap around
> ---
>
> Key: KAFKA-5283
> URL: https://issues.apache.org/jira/browse/KAFKA-5283
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Apurva Mehta
>Assignee: Jason Gustafson
>Priority: Blocker
>  Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> The design doc mentions that the epoch and sequence numbers will wrap around. 
> However, the current client and server code (on the producer, in the 
> `ProducerIdMapping` class, in the transaction coordinator) does not do this.
> Once all the pieces are in place we should go through and make sure that the 
> handling of sequence numbers and epoch is consistent across the board. Would 
> be good to add a system or integration test for this as well, if possible.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5283) Update clients and server code to make sure that epoch and sequence numbers wrap around

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034227#comment-16034227
 ] 

ASF GitHub Bot commented on KAFKA-5283:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3183


> Update clients and server code to make sure that epoch and sequence numbers 
> wrap around
> ---
>
> Key: KAFKA-5283
> URL: https://issues.apache.org/jira/browse/KAFKA-5283
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Apurva Mehta
>Assignee: Jason Gustafson
>Priority: Blocker
>  Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> The design doc mentions that the epoch and sequence numbers will wrap around. 
> However, the current client and server code (on the producer, in the 
> `ProducerIdMapping` class, in the transaction coordinator) does not do this.
> Once all the pieces are in place we should go through and make sure that the 
> handling of sequence numbers and epoch is consistent across the board. Would 
> be good to add a system or integration test for this as well, if possible.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3183: KAFKA-5283: Handle producer epoch/sequence overflo...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3183


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Assigned] (KAFKA-4370) CorruptRecordException when ProducerRecord constructed without key nor partition and send

[Mariam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4370?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mariam John reassigned KAFKA-4370:
--

Assignee: Mariam John

> CorruptRecordException when ProducerRecord constructed without key nor 
> partition and send
> -
>
> Key: KAFKA-4370
> URL: https://issues.apache.org/jira/browse/KAFKA-4370
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.10.1.0
>Reporter: Lars Pfannenschmidt
>Assignee: Mariam John
>Priority: Trivial
>
> According to the JavaDoc of ProducerRecord it should be possible to send 
> messages without a key:
> {quote}
> If neither key nor partition is present a partition will be assigned in a 
> round-robin fashion.
> {quote}
> {code:title=SomeProducer.java|borderStyle=solid}
> ProducerRecord record = new ProducerRecord<>(topic, 
> "somemessage");
> return this.producer.send(record).get();
> {code}
> Unfortunately an Exception is thrown:
> {code}
> java.util.concurrent.ExecutionException: 
> org.apache.kafka.common.errors.CorruptRecordException: This message has 
> failed its CRC checksum, exceeds the valid size, or is otherwise corrupt.
>   at 
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.valueOrError(FutureRecordMetadata.java:65)
>   at 
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:52)
>   at 
> org.apache.kafka.clients.producer.internals.FutureRecordMetadata.get(FutureRecordMetadata.java:25)
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-147: Add missing type parameters to StateStoreSupplier factories and KGroupedStream/Table methods

[Matthias J. Sax]

Thanks for the update Michal.

I did skip over the PR. Looks good to me, as far as I can tell. Maybe
Damian, Xavier, or Ismael can comment on this. Would be good to get
confirmation that the change is backward compatible.


-Matthias


On 5/27/17 11:11 AM, Michal Borowiecki wrote:
> Hi all,
> 
> I've updated the KIP to reflect the proposed backwards-compatible approach:
> 
> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=69408481
> 
> 
> Given the vast area of APIs affected, I think the PR is easier to read
> than the code excerpts in the KIP itself:
> https://github.com/apache/kafka/pull/2992/files
> 
> Thanks,
> Michał
> 
> On 07/05/17 10:16, Eno Thereska wrote:
>> I like this KIP in general and I agree it’s needed. Perhaps Damian can 
>> comment on the session store issue?
>>
>> Thanks
>> Eno
>>> On May 6, 2017, at 10:32 PM, Michal Borowiecki 
>>>  wrote:
>>>
>>> Hi Matthias,
>>>
>>> Agreed. I tried your proposal and indeed it would work.
>>>
>>> However, I think to maintain full backward compatibility we would also need 
>>> to deprecate Stores.create() and leave it unchanged, while providing a new 
>>> method that returns the more strongly typed Factories.
>>>
>>> ( This is because PersistentWindowFactory and PersistentSessionFactory 
>>> cannot extend the existing PersistentKeyValueFactory interface, since their 
>>> build() methods will be returning TypedStateStoreSupplier>> V>> and TypedStateStoreSupplier> respectively, which are 
>>> NOT subclasses of TypedStateStoreSupplier>. I do not 
>>> see another way around it. Admittedly, my type covariance skills are 
>>> rudimentary. Does anyone see a better way around this? )
>>>
>>> Since create() takes only the store name as argument, and I don't see what 
>>> we could overload it with, the new method would need to have a different 
>>> name. 
>>>
>>> Alternatively, since create(String) is the only method in Stores, we could 
>>> deprecate the entire class and provide a new one. That would be my 
>>> preference. Any ideas what to call it?
>>>
>>>
>>>
>>> All comments and suggestions appreciated.
>>>
>>>
>>>
>>> Cheers,
>>>
>>> Michał
>>>
>>>
>>> On 04/05/17 21:48, Matthias J. Sax wrote:
 I had a quick look into this.

 With regard to backward compatibility, I think it would be required do
 introduce a new type `TypesStateStoreSupplier` (that extends
 `StateStoreSupplier`) and to overload all methods that take a
 `StateStoreSupplier` that accept the new type instead of the current one.

 This would allow `.build` to return a `TypedStateStoreSupplier` and
 thus, would not break any code. As least if I did not miss anything with
 regard to some magic of type inference using generics (I am not an
 expert in this field).


 -Matthias

 On 5/4/17 11:32 AM, Matthias J. Sax wrote:
> Did not have time to have a look. But backward compatibility is a must
> from my point of view.
>
> -Matthias
>
>
> On 5/4/17 12:56 AM, Michal Borowiecki wrote:
>> Hello,
>>
>> I've updated the KIP with missing information.
>>
>> I would especially appreciate some comments on the compatibility aspects
>> of this as the proposed change is not fully backwards-compatible.
>>
>> In the absence of comments I shall call for a vote in the next few days.
>>
>> Thanks,
>>
>> Michal
>>
>>
>> On 30/04/17 23:11, Michal Borowiecki wrote:
>>> Hi community!
>>>
>>> I have just drafted KIP-147: Add missing type parameters to
>>> StateStoreSupplier factories and KGroupedStream/Table methods
>>> 
>>>  
>>> 
>>>
>>> Please let me know if this a step in the right direction.
>>>
>>> All comments welcome.
>>>
>>> Thanks,
>>> Michal
>>> -- 
>>> Signature
>>>   Michal 
>>> Borowiecki
>>> Senior Software Engineer L4
>>> T:  +44 208 742 1600
>>>
>>> 
>>> +44 203 249 8448
>>>
>>> 
>>>  
>>> E:  michal.borowie...@openbet.com 
>>> 
>>> W:  www.openbet.com  
>>>  
>>>
>>> 
>>> OpenBet Ltd
>>>
>>> Chiswick Park Building 9
>>>
>>> 566 Chiswick High Rd
>>>
>>> London
>>>
>>> W4 5XT
>>>
>>> UK
>>>
>>> 
>>>  
>>> 
>>>
>>> This message is confidential and intended only for the addressee. If
>>> you have received this message in error, please

Jenkins build is back to normal : kafka-trunk-jdk8 #1637

[Apache Jenkins Server]

See 

[jira] [Updated] (KAFKA-5322) Resolve AddPartitions response error code inconsistency

[Apurva Mehta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apurva Mehta updated KAFKA-5322:

Status: Patch Available  (was: In Progress)

> Resolve AddPartitions response error code inconsistency
> ---
>
> Key: KAFKA-5322
> URL: https://issues.apache.org/jira/browse/KAFKA-5322
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Apurva Mehta
>Priority: Blocker
>  Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> The AddPartitions request does not support partial failures. Either all 
> partitions are successfully added to the transaction or none of them are. 
> Currently we return a separate error code for each partition that was added 
> to the transaction, which begs the question of what error code to return if 
> we have not actually encountered a fatal partition-level error for some 
> partition. For example, suppose we send AddPartitions with partitions A and 
> B. If A is not authorized, we will not even attempt to add B to the 
> transaction, but what error code should we use. The current solution is to 
> only include partition A and its error code in the response, but this is a 
> little inconsistent with most other request types. Alternatives that have 
> been proposed:
> 1. Instead of a partition-level error, use one global error. We can add a 
> global error message to return friendlier details to the user about which 
> partition had a fault. The downside is that we would have to parse the 
> message contents if we wanted to do any partition-specific handling. We could 
> not easily fill the set of topics in {{TopicAuthorizationException}} for 
> example.
> 2. We can add a new error code to indicate that the broker did not even 
> attempt to add the partition to the transaction. For example: 
> OPERATION_NOT_ATTEMPTED or something like that. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5322) Resolve AddPartitions response error code inconsistency

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5322?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034177#comment-16034177
 ] 

ASF GitHub Bot commented on KAFKA-5322:
---

GitHub user apurvam opened a pull request:

https://github.com/apache/kafka/pull/3204

KAFKA-5322:  Add an `OPERATION_NOT_ATTEMPTED` error code

In the `AddPartitionsToTxn` request handling, if even one partition fails 
authorization checks, the entire request is essentially failed. However, the 
`AddPartitionsToTxnResponse` today will only contain the error codes for the 
topics which failed authorization. It will have no error code for the topics 
which succeeded, making it inconsistent with other APIs.

This patch adds a new error code `OPERATION_NOT_ATTEMPTED` which is 
returned for the successful partitions to indicate that they were not added to 
the transaction.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apurvam/kafka 
KAFKA-5322-add-operation-not-attempted-for-add-partitions

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3204.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3204


commit f454d495df20764105a42fe5e66351683e1a23ac
Author: Apurva Mehta 
Date:   2017-06-02T04:27:30Z

WIP

commit 764d8dc713ffde5703d58eb2a485567c6552f557
Author: Apurva Mehta 
Date:   2017-06-02T04:58:02Z

Update comments and client code

commit 7651210a7156a54376ff8a827b9220f60ff0ff20
Author: Apurva Mehta 
Date:   2017-06-02T05:38:49Z

Add test, fix checkstyle




> Resolve AddPartitions response error code inconsistency
> ---
>
> Key: KAFKA-5322
> URL: https://issues.apache.org/jira/browse/KAFKA-5322
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Apurva Mehta
>Priority: Blocker
>  Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> The AddPartitions request does not support partial failures. Either all 
> partitions are successfully added to the transaction or none of them are. 
> Currently we return a separate error code for each partition that was added 
> to the transaction, which begs the question of what error code to return if 
> we have not actually encountered a fatal partition-level error for some 
> partition. For example, suppose we send AddPartitions with partitions A and 
> B. If A is not authorized, we will not even attempt to add B to the 
> transaction, but what error code should we use. The current solution is to 
> only include partition A and its error code in the response, but this is a 
> little inconsistent with most other request types. Alternatives that have 
> been proposed:
> 1. Instead of a partition-level error, use one global error. We can add a 
> global error message to return friendlier details to the user about which 
> partition had a fault. The downside is that we would have to parse the 
> message contents if we wanted to do any partition-specific handling. We could 
> not easily fill the set of topics in {{TopicAuthorizationException}} for 
> example.
> 2. We can add a new error code to indicate that the broker did not even 
> attempt to add the partition to the transaction. For example: 
> OPERATION_NOT_ATTEMPTED or something like that. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3204: KAFKA-5322: Add an `OPERATION_NOT_ATTEMPTED` error...

[apurvam]

GitHub user apurvam opened a pull request:

https://github.com/apache/kafka/pull/3204

KAFKA-5322:  Add an `OPERATION_NOT_ATTEMPTED` error code

In the `AddPartitionsToTxn` request handling, if even one partition fails 
authorization checks, the entire request is essentially failed. However, the 
`AddPartitionsToTxnResponse` today will only contain the error codes for the 
topics which failed authorization. It will have no error code for the topics 
which succeeded, making it inconsistent with other APIs.

This patch adds a new error code `OPERATION_NOT_ATTEMPTED` which is 
returned for the successful partitions to indicate that they were not added to 
the transaction.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apurvam/kafka 
KAFKA-5322-add-operation-not-attempted-for-add-partitions

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3204.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3204


commit f454d495df20764105a42fe5e66351683e1a23ac
Author: Apurva Mehta 
Date:   2017-06-02T04:27:30Z

WIP

commit 764d8dc713ffde5703d58eb2a485567c6552f557
Author: Apurva Mehta 
Date:   2017-06-02T04:58:02Z

Update comments and client code

commit 7651210a7156a54376ff8a827b9220f60ff0ff20
Author: Apurva Mehta 
Date:   2017-06-02T05:38:49Z

Add test, fix checkstyle




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-165: Extend Interactive Queries for return latest update timestamp per key

[Matthias J. Sax]

Thanks for the KIP Jeyhun.

Some comments:
 - why do you only consider get() and not range() and all() ?
 - we cannot have a second get() (this would be ambiguous) but need
another name like getWithTs() (or something better)
 - what use case do you have in mind for getKeyTs() ? Would a single new
method returning KeyContext not be sufficient?
 - for backward compatibility, we will also need a new interface and
cannot just extend the existing one



-Matthias

On 5/29/17 4:55 PM, Jeyhun Karimov wrote:
> Dear community,
> 
> I want to share KIP-165 [1] based on issue KAFKA-4304 [2].
> I would like to get your comments.
> 
> [1]
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-165%3A+Extend+Interactive+Queries+for+return+latest+update+timestamp+per+key
> [2] https://issues.apache.org/jira/browse/KAFKA-4304
> 
> Cheers,
> Jeyhun
> 



signature.asc
Description: OpenPGP digital signature

Re: KIP-160 - Augment KStream.print() to allow users pass in extra parameters in the printed string

[Matthias J. Sax]

Thanks for the KIP.

Two comments:
 - I think we should include #writeAsText()
 - I am not sure if we should use

> "[" + this.streamName + "]: " + mapper.apply(keyToPrint, valueToPrint)

in case a mapper is provided. This still dictates a fixed prefix a user
might not want to have (what contradicts or at least limits the scope of
this new functionality). Considering he current discussion of KIP-159, a
user would be able to access the stream name within the provided mapper
and add it if they wish anyway, and thus, I don't think we should force
this format.



-Matthias



On 5/30/17 1:38 PM, Guozhang Wang wrote:
> Overall +1. One comment about the wiki itself:
> 
> Could you replace the general description of "Argument KStream.print() which
> is KStream.print(KeyValueMapper)" with the actual added
> overloaded functions in the wiki page?
> 
> 
> Guozhang
> 
> On Mon, May 22, 2017 at 12:21 AM, James Chain 
> wrote:
> 
>> Hi All,
>>
>> I want to start this KIP to argument KStream.print().
>> This vote is already started.
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 160+-+Augment+KStream.print%28%29+to+allow+users+pass+in+
>> extra+parameters+in+the+printed+string
>>
>> Thanks,
>>
>> James Chien
>>
> 
> 
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

[Matthias J. Sax]

I agree with Jeyhun. As already mention, the overall API improvement
ideas are overlapping and/or contradicting each other. For this reason,
not all ideas can be accomplished and some Jira might just be closed as
"won't fix".

For this reason, we try to do those KIP discussion with are large scope
to get an overall picture to converge to an overall consisted API.


@Jeyhun: about the overloads. Yes, we might get more overload. It might
be sufficient though, to do a single xxxWithContext() overload that will
provide key+value+context. Otherwise, if might get too messy having
ValueMapper, ValueMapperWithKey, ValueMapperWithContext,
ValueMapperWithKeyWithContext.

On the other hand, we also have the "builder pattern" idea as an API
change and this might mitigate the overload problem. Not for simple
function like map/flatMap etc but for joins and aggregations.


On the other hand, as I mentioned in an older email, I am personally
fine to break the pure functional interface, and add

  - interface WithRecordContext with method `open(RecordContext)` (or
`init(...)`, or any better name) -- but not `close()`)

  - interface ValueMapperWithRecordContext extends ValueMapper,
WithRecordContext

This would allow us to avoid any overload. Of course, we don't get a
"pure function" interface and also sacrifices Lambdas.



I am personally a little bit undecided what the better option might be.
Curious to hear what other think about this trade off.


-Matthias


On 6/1/17 6:13 PM, Jeyhun Karimov wrote:
> Hi Guozhang,
> 
> It subsumes partially. Initially the idea was to support RichFunctions as a
> separate interface. Throughout the discussion, however, we considered maybe
> overloading the related methods (with RecodContext param) is better
> approach than providing a separate RichFunction interface.
> 
> Cheers,
> Jeyhun
> 
> On Fri, Jun 2, 2017 at 2:27 AM Guozhang Wang  wrote:
> 
>> Does this KIP subsume this ticket as well?
>> https://issues.apache.org/jira/browse/KAFKA-4125
>>
>> On Sat, May 20, 2017 at 9:05 AM, Jeyhun Karimov 
>> wrote:
>>
>>> Dear community,
>>>
>>> As we discussed in KIP-149 [DISCUSS] thread [1], I would like to initiate
>>> KIP for rich functions (interfaces) [2].
>>> I would like to get your comments.
>>>
>>>
>>> [1]
>>> http://search-hadoop.com/m/Kafka/uyzND1PMjdk2CslH12?subj=
>>> Re+DISCUSS+KIP+149+Enabling+key+access+in+ValueTransformer+ValueMapper+
>>> and+ValueJoiner
>>> [2]
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> 159%3A+Introducing+Rich+functions+to+Streams
>>>
>>>
>>> Cheers,
>>> Jeyhun
>>> --
>>> -Cheers
>>>
>>> Jeyhun
>>>
>>
>>
>>
>> --
>> -- Guozhang
>>



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-167: Add a restoreAll method to StateRestoreCallback

[Matthias J. Sax]

Thanks for the KIP Bill.

With regard to backward compatibility, we should not change the current
interface, but add a new interface that extends the current one.

If we are going to add "begin" and "after", we might also consider to
add some intermediate call backs. This would allow an application to
monitor the restoring progress. We could consider a call back each x
records or each x seconds. This callback would not contain a record, but
only offset meta data -- like start offset, current offset, end offset.

WDYT?


-Matthias


On 6/1/17 6:17 PM, Bill Bejeck wrote:
> Sure thing. I'll update the KIP.
> 
> Thanks,
> Bill
> 
> On Thu, Jun 1, 2017 at 6:20 PM, Guozhang Wang  wrote:
> 
>> There are also some request to add "begin" and "after" callbacks in the
>> restoration func:
>>
>> https://issues.apache.org/jira/browse/KAFKA-4322
>>
>> Could we piggy back them into the same KIP?
>>
>>
>> Guozhang
>>
>> On Thu, Jun 1, 2017 at 2:04 PM, Bill Bejeck  wrote:
>>
>>> All,
>>>
>>> I'd like to start the discussion for adding bulk add functionality when
>>> restoring a state store.  The KIP can be found here:
>>>
>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>> 167%3A+Add+a+restoreAll+method+to+StateRestoreCallback
>>>
>>> Thanks,
>>> Bill
>>>
>>
>>
>>
>> --
>> -- Guozhang
>>
> 



signature.asc
Description: OpenPGP digital signature

[jira] [Updated] (KAFKA-5363) Add restoreAll functionality to StateRestoreCallback

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-5363:
---
Description: 
Add a new method {{restoreAll(List> records)}} to the 
{{StateRestoreCallback}} to enable bulk writing to the underlying state store 
vs individual {{restore(byte[] key, byte[] value)}} resulting in quicker 
restore times.

KIP: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-167%3A+Add+a+restoreAll+method+to+StateRestoreCallback

  was:Add a new method {{restoreAll(List> records)}} 
to the {{StateRestoreCallback}} to enable bulk writing to the underlying state 
store vs individual {{restore(byte[] key, byte[] value)}} resulting in quicker 
restore times.


> Add restoreAll functionality to StateRestoreCallback
> 
>
> Key: KAFKA-5363
> URL: https://issues.apache.org/jira/browse/KAFKA-5363
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Bill Bejeck
>Assignee: Bill Bejeck
>  Labels: kip
> Fix For: 0.11.1.0
>
>
> Add a new method {{restoreAll(List> records)}} to 
> the {{StateRestoreCallback}} to enable bulk writing to the underlying state 
> store vs individual {{restore(byte[] key, byte[] value)}} resulting in 
> quicker restore times.
> KIP: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-167%3A+Add+a+restoreAll+method+to+StateRestoreCallback



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5363) Add restoreAll functionality to StateRestoreCallback

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-5363:
---
Labels: kip  (was: )

> Add restoreAll functionality to StateRestoreCallback
> 
>
> Key: KAFKA-5363
> URL: https://issues.apache.org/jira/browse/KAFKA-5363
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Bill Bejeck
>Assignee: Bill Bejeck
>  Labels: kip
> Fix For: 0.11.1.0
>
>
> Add a new method {{restoreAll(List> records)}} to 
> the {{StateRestoreCallback}} to enable bulk writing to the underlying state 
> store vs individual {{restore(byte[] key, byte[] value)}} resulting in 
> quicker restore times.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Reg: [VOTE] KIP 157 - Add consumer config options to streams reset tool

[Matthias J. Sax]

+1

Thanks for updating the KIP!

-Matthias

On 6/1/17 6:18 PM, Bill Bejeck wrote:
> +1
> 
> Thanks,
> Bill
> 
> On Thu, Jun 1, 2017 at 7:45 PM, Guozhang Wang  wrote:
> 
>> +1 again. Thanks.
>>
>> On Tue, May 30, 2017 at 1:46 PM, BigData dev 
>> wrote:
>>
>>> Hi All,
>>> Updated the KIP, as the consumer configurations are required for both
>> Admin
>>> Client and Consumer in Stream reset tool. Updated the KIP to use
>>> command-config option, similar to other tools like
>> kafka-consumer-groups.sh
>>>
>>>
>>> *https://cwiki.apache.org/confluence/display/KAFKA/KIP+
>>> 157+-+Add+consumer+config+options+to+streams+reset+tool
>>> >> 157+-+Add+consumer+config+options+to+streams+reset+tool>*
>>>
>>>
>>> So, starting the voting process again for further inputs.
>>>
>>> This vote will run for a minimum of 72 hours.
>>>
>>> Thanks,
>>>
>>> Bharat
>>>
>>>
>>>
>>> On Tue, May 30, 2017 at 1:18 PM, Guozhang Wang 
>> wrote:
>>>
 +1. Thanks!

 On Tue, May 16, 2017 at 1:12 AM, Eno Thereska 
 wrote:

> +1 thanks.
>
> Eno
>> On 16 May 2017, at 04:20, BigData dev 
>>> wrote:
>>
>> Hi All,
>> Given the simple and non-controversial nature of the KIP, I would
>>> like
 to
>> start the voting process for KIP-157: Add consumer config options
>> to
>> streams reset tool
>>
>> *https://cwiki.apache.org/confluence/display/KAFKA/KIP+157+-
> +Add+consumer+config+options+to+streams+reset+tool
>>  +Add+consumer+config+options+to+streams+reset+tool>*
>>
>>
>> The vote will run for a minimum of 72 hours.
>>
>> Thanks,
>>
>> Bharat
>
>


 --
 -- Guozhang

>>>
>>
>>
>>
>> --
>> -- Guozhang
>>
> 



signature.asc
Description: OpenPGP digital signature

[jira] [Assigned] (KAFKA-5359) Exceptions from RequestFuture lack parts of the stack trace

[Vahid Hashemian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian reassigned KAFKA-5359:
--

Assignee: Vahid Hashemian

> Exceptions from RequestFuture lack parts of the stack trace
> ---
>
> Key: KAFKA-5359
> URL: https://issues.apache.org/jira/browse/KAFKA-5359
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Reporter: Magnus Reftel
>Assignee: Vahid Hashemian
>Priority: Minor
>
> When an exception occurs within a task that reports its result using a 
> RequestFuture, that exception is stored in a field on the RequestFuture using 
> the {{raise}} method. In many places in the code where such futures are 
> completed, that exception is then thrown directly using {{throw 
> future.exception();}} (see e.g. 
> [Fetcher.getTopicMetadata|https://github.com/apache/kafka/blob/aebba89a2b9b5ea6a7cab2599555232ef3fe21ad/clients/src/main/java/org/apache/kafka/clients/consumer/internals/Fetcher.java#L316]).
> This means that the exception that ends up in client code only has stack 
> traces related to the original exception, but nothing leading up to the 
> completion of the future. The client therefore gets no indication of what was 
> going on in the client code - only that it somehow ended up in the Kafka 
> libraries, and that a task failed at some point.
> One solution to this is to use the exceptions from the future as causes for 
> chained exceptions, so that the client gets a stack trace that shows what the 
> client was doing, in addition to getting the stack traces for the exception 
> in the task.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : kafka-trunk-jdk7 #2320

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk8 #1636

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: Add upgrade notes for KAFKA-2358

--
[...truncated 4.06 MB...]
kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete STARTED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooHigh 
PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooHigh 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooHigh 
PASSED

kafka.integration.TopicMetadataTest > testIsrAfterBrokerShutDownAndJoinsBack 
STARTED

kafka.integration.TopicMetadataTest > testIsrAfterBrokerShutDownAndJoinsBack 
PASSED

kafka.integration.TopicMetadataTest > testAutoCreateTopicWithCollision STARTED

kafka.integration.TopicMetadataTest > testAutoCreateTopicWithCollision PASSED

kafka.integration.TopicMetadataTest > testAliveBrokerListWithNoTopics STARTED

kafka.integration.TopicMetadataTest > testAliveBrokerListWithNoTopics PASSED

kafka.integration.TopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.TopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.TopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.TopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.TopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.TopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.TopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.TopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.TopicMetadataTest > testAutoCreateTopicWithInvalidReplication 
STARTED

kafka.integration.TopicMetadataTest > testAutoCreateTopicWithInvalidReplication 
PASSED

kafka.integration.TopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.TopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.FetcherTest > testFetcher STARTED

kafka.integration.FetcherTest > testFetcher PASSED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionEnabled 
STARTED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionEnabled 
PASSED

kafka.integration.UncleanLeaderElectionTest > 
testCleanLeaderElectionDisabledByTopicOverride STARTED

kafka.integration.UncleanLeaderElectionTest > 
testCleanLeaderElectionDisabledByTopicOverride SKIPPED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionDisabled 
STARTED

kafka.integration.UncleanLeaderElectionTest > testUncleanLeaderElectionDisabled 
SKIPPED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionInvalidTopicOverride STARTED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionInvalidTopicOverride PASSED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionEnabledByTopicOverride STARTED

kafka.integration.UncleanLeaderElectionTest > 
testUncleanLeaderElectionEnabledByTopicOverride PASSED

kafka.integration.MinIsrConfigTest > testDefaultKafkaConfig STARTED

kafka.integration.MinIsrConfigTest > testDefaultKafkaConfig PASSED

unit.kafka.server.KafkaApisTest > 
shouldRespond

[jira] [Commented] (KAFKA-2289) KafkaProducer logs erroneous warning on startup

[Bharat Viswanadham (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034055#comment-16034055
 ] 

Bharat Viswanadham commented on KAFKA-2289:
---

[~hgschmie] Would you like to continue to work on this?
If not, I will provide a patch based on latest trunk code.

> KafkaProducer logs erroneous warning on startup
> ---
>
> Key: KAFKA-2289
> URL: https://issues.apache.org/jira/browse/KAFKA-2289
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.8.2.1
>Reporter: Henning Schmiedehausen
>Priority: Trivial
>
> When creating a new KafkaProducer using the 
> KafkaProducer(KafkaConfig, Serializer, Serializer) constructor, Kafka 
> will list the following lines, which are harmless but are still at WARN level:
> WARN  [2015-06-19 23:13:56,557] 
> org.apache.kafka.clients.producer.ProducerConfig: The configuration 
> value.serializer = class  was supplied but isn't a known config.
> WARN  [2015-06-19 23:13:56,557] 
> org.apache.kafka.clients.producer.ProducerConfig: The configuration 
> key.serializer = class  was supplied but isn't a known config.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: kafka-trunk-jdk7 #2319

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: Add upgrade notes for KAFKA-2358

--
[...truncated 2.32 MB...]

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldNotMakeStoreAvailableUntilAllStoresAvailable PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
queryOnRebalance PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
concurrentAccesses PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldAllowToQueryAfterThreadDied STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldAllowToQueryAfterThreadDied PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryStateWithZeroSizedCache STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryStateWithZeroSizedCache PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryMapValuesAfterFilterState STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryMapValuesAfterFilterState PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryFilterState STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryFilterState PASSED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryStateWithNonZeroSizedCache STARTED

org.apache.kafka.streams.integration.QueryableStateIntegrationTest > 
shouldBeAbleToQueryStateWithNonZeroSizedCache PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerLeftJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterOuterJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerOuterJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldInnerInnerJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftOuterJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterLeftJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoin STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftLeftJoin PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldOuterInnerJoinQueryable PASSED

org.apache.kafka.streams.integration.KTableKTableJoinIntegrationTest > 
shouldLeftInnerJoinQueryable STARTED

org.apache.kafka.streams.integration.KTableKTableJoin

[jira] [Commented] (KAFKA-1595) Remove deprecated and slower scala JSON parser from kafka.consumer.TopicCount

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034042#comment-16034042
 ] 

Ismael Juma commented on KAFKA-1595:


[~onurkaraman], I had trouble getting people to review it last time. But since 
we now have concrete numbers showing the improvement, it should be easier to 
motivate the change. I can rebase and fix the conflicts some time next week.

> Remove deprecated and slower scala JSON parser from kafka.consumer.TopicCount
> -
>
> Key: KAFKA-1595
> URL: https://issues.apache.org/jira/browse/KAFKA-1595
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.1.1
>Reporter: Jagbir
>Assignee: Ismael Juma
>  Labels: newbie
>
> The following issue is created as a follow up suggested by Jun Rao
> in a kafka news group message with the Subject
> "Blocking Recursive parsing from 
> kafka.consumer.TopicCount$.constructTopicCount"
> SUMMARY:
> An issue was detected in a typical cluster of 3 kafka instances backed
> by 3 zookeeper instances (kafka version 0.8.1.1, scala version 2.10.3,
> java version 1.7.0_65). On consumer end, when consumers get recycled,
> there is a troubling JSON parsing recursion which takes a busy lock and
> blocks consumers thread pool.
> In 0.8.1.1 scala client library ZookeeperConsumerConnector.scala:355 takes
> a global lock (0xd3a7e1d0) during the rebalance, and fires an
> expensive JSON parsing, while keeping the other consumers from shutting
> down, see, e.g,
> at 
> kafka.consumer.ZookeeperConsumerConnector.shutdown(ZookeeperConsumerConnector.scala:161)
> The deep recursive JSON parsing should be deprecated in favor
> of a better JSON parser, see, e.g,
> http://engineering.ooyala.com/blog/comparing-scala-json-libraries?
> DETAILS:
> The first dump is for a recursive blocking thread holding the lock for 
> 0xd3a7e1d0
> and the subsequent dump is for a waiting thread.
> (Please grep for 0xd3a7e1d0 to see the locked object.)
> Â 
> -8<-
> "Sa863f22b1e5hjh6788991800900b34545c_profile-a-prod1-s-140789080845312-c397945e8_watcher_executor"
> prio=10 tid=0x7f24dc285800 nid=0xda9 runnable [0x7f249e40b000]
> java.lang.Thread.State: RUNNABLE
> at 
> scala.util.parsing.combinator.Parsers$$anonfun$rep1$1.p$7(Parsers.scala:722)
> at 
> scala.util.parsing.combinator.Parsers$$anonfun$rep1$1.continue$1(Parsers.scala:726)
> at 
> scala.util.parsing.combinator.Parsers$$anonfun$rep1$1.apply(Parsers.scala:737)
> at 
> scala.util.parsing.combinator.Parsers$$anonfun$rep1$1.apply(Parsers.scala:721)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Success.flatMapWithNext(Parsers.scala:142)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$flatMap$1.apply(Parsers.scala:239)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$flatMap$1.apply(Parsers.scala:239)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$flatMap$1.apply(Parsers.scala:239)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$flatMap$1.apply(Parsers.scala:239)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$map$1.apply(Parsers.scala:242)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at 
> scala.util.parsing.combinator.Parsers$Parser$$anonfun$append$1.apply(Parsers.scala:254)
> at scala.util.parsing.combinator.Parsers$$anon$3.apply(Parsers.scala:222)
> at 
> scala.util.parsing.combinator.Parsers

[GitHub] kafka pull request #3185: MINOR: Logging/debugging improvements for transact...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3185


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #83: KAFKA-1595; Remove deprecated and slower scala JSON ...

[ijuma]

GitHub user ijuma reopened a pull request:

https://github.com/apache/kafka/pull/83

KAFKA-1595; Remove deprecated and slower scala JSON parser

Tested that we only use Jackson methods introduced in 2.0 in the main 
codebase by compiling it with the older version locally. We use a constructor 
introduced in 2.4 in one test, but I didn't remove it as it seemed harmless. 
The reasoning for this is explained in the mailing list thread:

http://search-hadoop.com/m/uyzND1FWbWw1qUbWe


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-1595-remove-deprecated-json-parser-jackson

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/83.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #83


commit bdceb1080ef78258c81bb3b8405be1f5e07a12b1
Author: Ismael Juma 
Date:   2015-07-17T10:43:22Z

Update to JUnit 4.12.

It includes `assertNotEquals`, which is used in a subsequent
commit.

commit 117cd1db07798d5d8ff021fa2a1569a301d775e6
Author: Ismael Juma 
Date:   2015-07-17T11:03:00Z

Introduce `testJsonParse`

Simple test that shows existing behaviour.

commit 26b0525df2c04b0ffd7718b1ef31b94ce12e486c
Author: Ismael Juma 
Date:   2015-07-17T11:05:00Z

KAFKA-1595; Remove deprecated and slower scala JSON parser from 
kafka.consumer.TopicCount

A thin wrapper over Jackson's Tree Model API is used as the replacement. 
This wrapper
increases safety while providing a simple, but powerful API through the 
usage of the
`DecodeJson` type class. Even though this has a maintenance cost, it makes 
the API
much more convenient from Scala. A number of tests were added to verify the
behaviour of this wrapper.

The Scala module for Jackson doesn't provide any help for our current 
usage, so we don't
depend on it.

An attempt has been made to maintain the existing behaviour regarding when 
exceptions
are thrown. There are a number of cases where `JsonMappingException` will 
be thrown
instead of `ClassCastException`, however. It is expected that users would 
not try to catch
`ClassCastException`.

commit 8a952c31866933fd769840221293ba9c95fed162
Author: Ismael Juma 
Date:   2015-07-17T11:06:00Z

Minor clean-ups in `Json.encode`




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-1595) Remove deprecated and slower scala JSON parser from kafka.consumer.TopicCount

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1595?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034038#comment-16034038
 ] 

ASF GitHub Bot commented on KAFKA-1595:
---

GitHub user ijuma reopened a pull request:

https://github.com/apache/kafka/pull/83

KAFKA-1595; Remove deprecated and slower scala JSON parser

Tested that we only use Jackson methods introduced in 2.0 in the main 
codebase by compiling it with the older version locally. We use a constructor 
introduced in 2.4 in one test, but I didn't remove it as it seemed harmless. 
The reasoning for this is explained in the mailing list thread:

http://search-hadoop.com/m/uyzND1FWbWw1qUbWe


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-1595-remove-deprecated-json-parser-jackson

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/83.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #83


commit bdceb1080ef78258c81bb3b8405be1f5e07a12b1
Author: Ismael Juma 
Date:   2015-07-17T10:43:22Z

Update to JUnit 4.12.

It includes `assertNotEquals`, which is used in a subsequent
commit.

commit 117cd1db07798d5d8ff021fa2a1569a301d775e6
Author: Ismael Juma 
Date:   2015-07-17T11:03:00Z

Introduce `testJsonParse`

Simple test that shows existing behaviour.

commit 26b0525df2c04b0ffd7718b1ef31b94ce12e486c
Author: Ismael Juma 
Date:   2015-07-17T11:05:00Z

KAFKA-1595; Remove deprecated and slower scala JSON parser from 
kafka.consumer.TopicCount

A thin wrapper over Jackson's Tree Model API is used as the replacement. 
This wrapper
increases safety while providing a simple, but powerful API through the 
usage of the
`DecodeJson` type class. Even though this has a maintenance cost, it makes 
the API
much more convenient from Scala. A number of tests were added to verify the
behaviour of this wrapper.

The Scala module for Jackson doesn't provide any help for our current 
usage, so we don't
depend on it.

An attempt has been made to maintain the existing behaviour regarding when 
exceptions
are thrown. There are a number of cases where `JsonMappingException` will 
be thrown
instead of `ClassCastException`, however. It is expected that users would 
not try to catch
`ClassCastException`.

commit 8a952c31866933fd769840221293ba9c95fed162
Author: Ismael Juma 
Date:   2015-07-17T11:06:00Z

Minor clean-ups in `Json.encode`




> Remove deprecated and slower scala JSON parser from kafka.consumer.TopicCount
> -
>
> Key: KAFKA-1595
> URL: https://issues.apache.org/jira/browse/KAFKA-1595
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.1.1
>Reporter: Jagbir
>Assignee: Ismael Juma
>  Labels: newbie
>
> The following issue is created as a follow up suggested by Jun Rao
> in a kafka news group message with the Subject
> "Blocking Recursive parsing from 
> kafka.consumer.TopicCount$.constructTopicCount"
> SUMMARY:
> An issue was detected in a typical cluster of 3 kafka instances backed
> by 3 zookeeper instances (kafka version 0.8.1.1, scala version 2.10.3,
> java version 1.7.0_65). On consumer end, when consumers get recycled,
> there is a troubling JSON parsing recursion which takes a busy lock and
> blocks consumers thread pool.
> In 0.8.1.1 scala client library ZookeeperConsumerConnector.scala:355 takes
> a global lock (0xd3a7e1d0) during the rebalance, and fires an
> expensive JSON parsing, while keeping the other consumers from shutting
> down, see, e.g,
> at 
> kafka.consumer.ZookeeperConsumerConnector.shutdown(ZookeeperConsumerConnector.scala:161)
> The deep recursive JSON parsing should be deprecated in favor
> of a better JSON parser, see, e.g,
> http://engineering.ooyala.com/blog/comparing-scala-json-libraries?
> DETAILS:
> The first dump is for a recursive blocking thread holding the lock for 
> 0xd3a7e1d0
> and the subsequent dump is for a waiting thread.
> (Please grep for 0xd3a7e1d0 to see the locked object.)
> Â 
> -8<-
> "Sa863f22b1e5hjh6788991800900b34545c_profile-a-prod1-s-140789080845312-c397945e8_watcher_executor"
> prio=10 tid=0x7f24dc285800 nid=0xda9 runnable [0x7f249e40b000]
> java.lang.Thread.State: RUNNABLE
> at 
> scala.util.parsing.combinator.Parsers$$anonfun$rep1$1.p$7(Parsers.scala:722)
> at 
> scala.util.parsing.combinator.Parsers$$anonfun$rep1$1.continue$1(Parsers.scala:726)
> at 
> scala.util.parsing.combinator.Parsers$$anonfun$rep1$1.apply(P

[jira] [Commented] (KAFKA-5319) Add a tool to make cluster replica and leader balance

[Ma Tianchi (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034037#comment-16034037
 ] 

Ma Tianchi commented on KAFKA-5319:
---

I create a KIP about it. Comment 
welcome.(https://cwiki.apache.org/confluence/display/KAFKA/KIP-166+-+Add+a+tool+to+make+amounts+of+replicas+and+leaders+on+brokers+balanced)

> Add a tool to make cluster replica and leader balance
> -
>
> Key: KAFKA-5319
> URL: https://issues.apache.org/jira/browse/KAFKA-5319
> Project: Kafka
>  Issue Type: New Feature
>  Components: admin
>Affects Versions: 0.10.2.1
>Reporter: Ma Tianchi
>Assignee: Ma Tianchi
>  Labels: patch
> Attachments: KAFKA-5319.patch
>
>
> When a new broker is added to cluster,there is not any topics in the new 
> broker.When we use console command to create a topic without 
> 'replicaAssignmentOpt',Kafka use AdminUtils.assignReplicasToBrokers to get 
> replicaAssignment.Even though it is balance at the creating time if the 
> cluster never change,with more and more brokers added to cluster the replica 
> balanced will become not well. We also can use 'kafka-reassign-partitions.sh' 
> to balance ,but the big amount of topics make it not easy.And at the topic 
> created time , Kafka choose a  PreferredReplicaLeader which be put at the 
> first position of  the AR to make leader balance.But the balance will be 
> destroyed when cluster changed.Using  'kafka-reassign-partitions.sh' to make 
> partition reassignment may be also destroy the leader balance ,because user 
> can change the AR of the partition . It may be not balanced , but Kafka 
> believe cluster leader balance is well with every leaders is the first on at 
> AR.
> So we create a tool to make the number of replicas and number of leaders on 
> every brokers is balanced.It uses a algorithm to get a balanced replica and 
> leader  reassignment,then uses ReassignPartitionsCommand to real balance the 
> cluster.
> It can be used to make balance when cluster added brokers or cluster is not 
> balanced .It does not deal with moving replicas of a dead broker to an alive 
> broker,it only makes replicas and leaders on alive brokers is balanced.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5332) When resize the index file, maybe caused the content disappear

[xuzq (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034005#comment-16034005
 ] 

xuzq commented on KAFKA-5332:
-

[~junrao] when you are idle, please check this bug, maybe it will make KAFAK 
stronger. Thank you.

> When resize the index file, maybe caused the content disappear
> --
>
> Key: KAFKA-5332
> URL: https://issues.apache.org/jira/browse/KAFKA-5332
> Project: Kafka
>  Issue Type: Bug
>  Components: core, log
>Affects Versions: 0.10.2.1
>Reporter: xuzq
>
> When resize the index file, maybe caused the content disappear.
> When the kafka server is running, someone removed the index file on the disk 
> manually, if at this point, the function AbstractIndex.Resize(newSize: Int) 
> is triggered, it will create a new .index file which the size is 
> roundedNewSize, but the content is empty. 
> After this, the contents of mmap is empty. 
> When looking for specific offset corresponds to position, it also return 
> zero. The index file to locate the position does not provide any convenience.
> I think if the ".index" is not exist, we should copy the contents from old 
> "mmap" to new "mmap" to avoid the "empty file".



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (KAFKA-5332) When resize the index file, maybe caused the content disappear

[xuzq (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034005#comment-16034005
 ] 

xuzq edited comment on KAFKA-5332 at 6/2/17 1:36 AM:
-

[~junrao] when you have free time, please check this bug, maybe it will make 
KAFKA stronger. Thank you.


was (Author: xuzq_zander):
[~junrao] when you have free time, please check this bug, maybe it will make 
KAFAK stronger. Thank you.

> When resize the index file, maybe caused the content disappear
> --
>
> Key: KAFKA-5332
> URL: https://issues.apache.org/jira/browse/KAFKA-5332
> Project: Kafka
>  Issue Type: Bug
>  Components: core, log
>Affects Versions: 0.10.2.1
>Reporter: xuzq
>
> When resize the index file, maybe caused the content disappear.
> When the kafka server is running, someone removed the index file on the disk 
> manually, if at this point, the function AbstractIndex.Resize(newSize: Int) 
> is triggered, it will create a new .index file which the size is 
> roundedNewSize, but the content is empty. 
> After this, the contents of mmap is empty. 
> When looking for specific offset corresponds to position, it also return 
> zero. The index file to locate the position does not provide any convenience.
> I think if the ".index" is not exist, we should copy the contents from old 
> "mmap" to new "mmap" to avoid the "empty file".



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (KAFKA-5332) When resize the index file, maybe caused the content disappear

[xuzq (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5332?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034005#comment-16034005
 ] 

xuzq edited comment on KAFKA-5332 at 6/2/17 1:34 AM:
-

[~junrao] when you have free time, please check this bug, maybe it will make 
KAFAK stronger. Thank you.


was (Author: xuzq_zander):
[~junrao] when you are idle, please check this bug, maybe it will make KAFAK 
stronger. Thank you.

> When resize the index file, maybe caused the content disappear
> --
>
> Key: KAFKA-5332
> URL: https://issues.apache.org/jira/browse/KAFKA-5332
> Project: Kafka
>  Issue Type: Bug
>  Components: core, log
>Affects Versions: 0.10.2.1
>Reporter: xuzq
>
> When resize the index file, maybe caused the content disappear.
> When the kafka server is running, someone removed the index file on the disk 
> manually, if at this point, the function AbstractIndex.Resize(newSize: Int) 
> is triggered, it will create a new .index file which the size is 
> roundedNewSize, but the content is empty. 
> After this, the contents of mmap is empty. 
> When looking for specific offset corresponds to position, it also return 
> zero. The index file to locate the position does not provide any convenience.
> I think if the ".index" is not exist, we should copy the contents from old 
> "mmap" to new "mmap" to avoid the "empty file".



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (KAFKA-2358) Cluster collection returning methods should never return null

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-2358?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-2358.

Resolution: Fixed

> Cluster collection returning methods should never return null
> -
>
> Key: KAFKA-2358
> URL: https://issues.apache.org/jira/browse/KAFKA-2358
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Stevo Slavic
>Assignee: Stevo Slavic
>Priority: Minor
> Fix For: 0.11.0.0
>
>
> {{KafkaConsumer.partitionsFor}} method by it's signature returns a 
> {{List}}. Problem is that in case (metadata for) topic does 
> not exist, current implementation will return null, which is considered a bad 
> practice - instead of null it should return empty list.
> Root cause is that the Cluster collection returning methods are returning 
> null.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-2358) Cluster collection returning methods should never return null

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034001#comment-16034001
 ] 

ASF GitHub Bot commented on KAFKA-2358:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3200


> Cluster collection returning methods should never return null
> -
>
> Key: KAFKA-2358
> URL: https://issues.apache.org/jira/browse/KAFKA-2358
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Stevo Slavic
>Assignee: Stevo Slavic
>Priority: Minor
> Fix For: 0.11.0.0
>
>
> {{KafkaConsumer.partitionsFor}} method by it's signature returns a 
> {{List}}. Problem is that in case (metadata for) topic does 
> not exist, current implementation will return null, which is considered a bad 
> practice - instead of null it should return empty list.
> Root cause is that the Cluster collection returning methods are returning 
> null.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3200: MINOR: Add upgrade notes for KAFKA-2358

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3200


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: Reg: [VOTE] KIP 157 - Add consumer config options to streams reset tool

[Bill Bejeck]

+1

Thanks,
Bill

On Thu, Jun 1, 2017 at 7:45 PM, Guozhang Wang  wrote:

> +1 again. Thanks.
>
> On Tue, May 30, 2017 at 1:46 PM, BigData dev 
> wrote:
>
> > Hi All,
> > Updated the KIP, as the consumer configurations are required for both
> Admin
> > Client and Consumer in Stream reset tool. Updated the KIP to use
> > command-config option, similar to other tools like
> kafka-consumer-groups.sh
> >
> >
> > *https://cwiki.apache.org/confluence/display/KAFKA/KIP+
> > 157+-+Add+consumer+config+options+to+streams+reset+tool
> >  > 157+-+Add+consumer+config+options+to+streams+reset+tool>*
> >
> >
> > So, starting the voting process again for further inputs.
> >
> > This vote will run for a minimum of 72 hours.
> >
> > Thanks,
> >
> > Bharat
> >
> >
> >
> > On Tue, May 30, 2017 at 1:18 PM, Guozhang Wang 
> wrote:
> >
> > > +1. Thanks!
> > >
> > > On Tue, May 16, 2017 at 1:12 AM, Eno Thereska 
> > > wrote:
> > >
> > > > +1 thanks.
> > > >
> > > > Eno
> > > > > On 16 May 2017, at 04:20, BigData dev 
> > wrote:
> > > > >
> > > > > Hi All,
> > > > > Given the simple and non-controversial nature of the KIP, I would
> > like
> > > to
> > > > > start the voting process for KIP-157: Add consumer config options
> to
> > > > > streams reset tool
> > > > >
> > > > > *https://cwiki.apache.org/confluence/display/KAFKA/KIP+157+-
> > > > +Add+consumer+config+options+to+streams+reset+tool
> > > > >  > > > +Add+consumer+config+options+to+streams+reset+tool>*
> > > > >
> > > > >
> > > > > The vote will run for a minimum of 72 hours.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Bharat
> > > >
> > > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
>
>
> --
> -- Guozhang
>

Re: [DISCUSS] KIP-167: Add a restoreAll method to StateRestoreCallback

[Bill Bejeck]

Sure thing. I'll update the KIP.

Thanks,
Bill

On Thu, Jun 1, 2017 at 6:20 PM, Guozhang Wang  wrote:

> There are also some request to add "begin" and "after" callbacks in the
> restoration func:
>
> https://issues.apache.org/jira/browse/KAFKA-4322
>
> Could we piggy back them into the same KIP?
>
>
> Guozhang
>
> On Thu, Jun 1, 2017 at 2:04 PM, Bill Bejeck  wrote:
>
> > All,
> >
> > I'd like to start the discussion for adding bulk add functionality when
> > restoring a state store.  The KIP can be found here:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 167%3A+Add+a+restoreAll+method+to+StateRestoreCallback
> >
> > Thanks,
> > Bill
> >
>
>
>
> --
> -- Guozhang
>

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

[Jeyhun Karimov]

Hi Guozhang,

It subsumes partially. Initially the idea was to support RichFunctions as a
separate interface. Throughout the discussion, however, we considered maybe
overloading the related methods (with RecodContext param) is better
approach than providing a separate RichFunction interface.

Cheers,
Jeyhun

On Fri, Jun 2, 2017 at 2:27 AM Guozhang Wang  wrote:

> Does this KIP subsume this ticket as well?
> https://issues.apache.org/jira/browse/KAFKA-4125
>
> On Sat, May 20, 2017 at 9:05 AM, Jeyhun Karimov 
> wrote:
>
> > Dear community,
> >
> > As we discussed in KIP-149 [DISCUSS] thread [1], I would like to initiate
> > KIP for rich functions (interfaces) [2].
> > I would like to get your comments.
> >
> >
> > [1]
> > http://search-hadoop.com/m/Kafka/uyzND1PMjdk2CslH12?subj=
> > Re+DISCUSS+KIP+149+Enabling+key+access+in+ValueTransformer+ValueMapper+
> > and+ValueJoiner
> > [2]
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 159%3A+Introducing+Rich+functions+to+Streams
> >
> >
> > Cheers,
> > Jeyhun
> > --
> > -Cheers
> >
> > Jeyhun
> >
>
>
>
> --
> -- Guozhang
>
-- 
-Cheers

Jeyhun

[GitHub] kafka pull request #3203: KAFKA-5365 [WIP]: Fix regression in compressed mes...

[hachikuji]

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/3203

KAFKA-5365 [WIP]: Fix regression in compressed message iteration affecting 
magic v0 and v1



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-5365

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3203.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3203


commit 0a67684c9d895be1dc7a883e753b5c42c662e819
Author: Jason Gustafson 
Date:   2017-06-02T00:52:52Z

KAFKA-5365 [WIP]: Fix regression in compressed message iteration affecting 
magic v0 and v1




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-5365) Fix regression in compressed message iteration affecting magic v0 and v1

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033989#comment-16033989
 ] 

ASF GitHub Bot commented on KAFKA-5365:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/3203

KAFKA-5365 [WIP]: Fix regression in compressed message iteration affecting 
magic v0 and v1



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-5365

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3203.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3203


commit 0a67684c9d895be1dc7a883e753b5c42c662e819
Author: Jason Gustafson 
Date:   2017-06-02T00:52:52Z

KAFKA-5365 [WIP]: Fix regression in compressed message iteration affecting 
magic v0 and v1




> Fix regression in compressed message iteration affecting magic v0 and v1
> 
>
> Key: KAFKA-5365
> URL: https://issues.apache.org/jira/browse/KAFKA-5365
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>Priority: Blocker
> Fix For: 0.11.0.0
>
>
> We added a shortcut to break iteration over compressed message sets for v0 
> and v1 if the inner offset matches the last offset in the wrapper. 
> Unfortunately this breaks older clients which may use offset 0 in the wrapper 
> record in records sent in produce requests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: kafka-trunk-jdk8 #1635

[Apache Jenkins Server]

See 


Changes:

[wangguoz] MINOR: update Streams docs for KIP-123

--
[...truncated 907.87 KB...]
kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestWithUnsupportedVersion STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestWithUnsupportedVersion PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestBeforeSaslHandshakeRequest STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestBeforeSaslHandshakeRequest PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestAfterSaslHandshakeRequest STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestAfterSaslHandshakeRequest PASSED

kafka.server.ServerGenerateClusterIdTest > 
testAutoGenerateClusterIdForKafkaClusterParallel STARTED

kafka.server.ServerGenerateClusterIdTest > 
testAutoGenerateClusterIdForKafkaClusterParallel PASSED

kafka.server.ServerGenerateClusterIdTest > testAutoGenerateClusterId STARTED

kafka.server.ServerGenerateClusterIdTest > testAutoGenerateClusterId PASSED

kafka.server.ServerGenerateClusterIdTest > 
testAutoGenerateClusterIdForKafkaClusterSequential STARTED

kafka.server.ServerGenerateClusterIdTest > 
testAutoGenerateClusterIdForKafkaClusterSequential PASSED

kafka.server.ApiVersionsTest > testApiVersions STARTED

kafka.server.ApiVersionsTest > testApiVersions PASSED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK STARTED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK PASSED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot STARTED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot PASSED

kafka.server.ServerStartupTest > testConflictBrokerStartupWithSamePort STARTED

kafka.server.ServerStartupTest > testConflictBrokerStartupWithSamePort PASSED

kafka.server.ServerStartupTest > testConflictBrokerRegistration STARTED

kafka.server.ServerStartupTest > testConflictBrokerRegistration PASSED

kafka.server.ServerStartupTest > testBrokerSelfAware STARTED

kafka.server.ServerStartupTest > testBrokerSelfAware PASSED

kafka.server.SessionExpireListenerTest > testSessionExpireListenerMetrics 
STARTED

kafka.server.SessionExpireListenerTest > testSessionExpireListenerMetrics PASSED

kafka.server.ProduceRequestTest > testSimpleProduceRequest STARTED

kafka.server.ProduceRequestTest > testSimpleProduceRequest PASSED

kafka.server.ProduceRequestTest > testCorruptLz4ProduceRequest STARTED

kafka.server.ProduceRequestTest > testCorruptLz4ProduceRequest PASSED

kafka.server.ReplicaFetcherThreadFatalErrorTest > 
testFatalErrorInProcessFetchRequest STARTED

kafka.server.ReplicaFetcherThreadFatalErrorTest > 
testFatalErrorInProcessFetchRequest PASSED

kafka.server.ReplicaFetcherThreadFatalErrorTest > testFatalErrorInAddPartitions 
STARTED

kafka.server.ReplicaFetcherThreadFatalErrorTest > testFatalErrorInAddPartitions 
PASSED

kafka.server.DeleteTopicsRequestTest > testValidDeleteTopicRequests STARTED

kafka.server.DeleteTopicsRequestTest > testValidDeleteTopicRequests PASSED

kafka.server.DeleteTopicsRequestTest > testErrorDeleteTopicRequests STARTED

kafka.server.DeleteTopicsRequestTest > testErrorDeleteTopicRequests PASSED

kafka.server.DeleteTopicsRequestTest > testNotController STARTED

kafka.server.DeleteTopicsRequestTest > testNotController PASSED

kafka.server.SimpleFetchTest > testReadFromLog STARTED

kafka.server.SimpleFetchTest > testReadFromLog PASSED

kafka.common.ZkNodeChangeNotificationListenerTest > testProcessNotification 
STARTED

kafka.common.ZkNodeChangeNotificationListenerTest > testProcessNotification 
PASSED

kafka.common.ConfigTest > testInvalidGroupIds STARTED

kafka.common.ConfigTest > testInvalidGroupIds PASSED

kafka.common.ConfigTest > testInvalidClientIds STARTED

kafka.common.ConfigTest > testInvalidClientIds PASSED

kafka.common.InterBrokerSendThreadTest > 
shouldCreateClientRequestAndSendWhenNodeIsReady STARTED

kafka.common.InterBrokerSendThreadTest > 
shouldCreateClientRequestAndSendWhenNodeIsReady PASSED

kafka.common.InterBrokerSendThreadTest > 
shouldCallCompletionHandlerWithDisconnectedResponseWhenNodeNotReady STARTED

kafka.common.InterBrokerSendThreadTest > 
shouldCallCompletionHandlerWithDisconnectedResponseWhenNodeNotReady PASSED

kafka.common.InterBrokerSendThreadTest > shouldNotSendAnythingWhenNoRequests 
STARTED

kafka.common.InterBrokerSendThreadTest > shouldNotSendAnythingWhenNoRequests 
PASSED

unit.kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsShiftByHigherThanLatest STARTED

unit.kafka.admin.ResetConsumerGroupOffsetTest > 
testResetOffsetsShiftByHigherTh

[jira] [Work started] (KAFKA-5322) Resolve AddPartitions response error code inconsistency

[Apurva Mehta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-5322 started by Apurva Mehta.
---
> Resolve AddPartitions response error code inconsistency
> ---
>
> Key: KAFKA-5322
> URL: https://issues.apache.org/jira/browse/KAFKA-5322
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Apurva Mehta
>Priority: Blocker
>  Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> The AddPartitions request does not support partial failures. Either all 
> partitions are successfully added to the transaction or none of them are. 
> Currently we return a separate error code for each partition that was added 
> to the transaction, which begs the question of what error code to return if 
> we have not actually encountered a fatal partition-level error for some 
> partition. For example, suppose we send AddPartitions with partitions A and 
> B. If A is not authorized, we will not even attempt to add B to the 
> transaction, but what error code should we use. The current solution is to 
> only include partition A and its error code in the response, but this is a 
> little inconsistent with most other request types. Alternatives that have 
> been proposed:
> 1. Instead of a partition-level error, use one global error. We can add a 
> global error message to return friendlier details to the user about which 
> partition had a fault. The downside is that we would have to parse the 
> message contents if we wanted to do any partition-specific handling. We could 
> not easily fill the set of topics in {{TopicAuthorizationException}} for 
> example.
> 2. We can add a new error code to indicate that the broker did not even 
> attempt to add the partition to the transaction. For example: 
> OPERATION_NOT_ATTEMPTED or something like that. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

[Guozhang Wang]

Does this KIP subsume this ticket as well?
https://issues.apache.org/jira/browse/KAFKA-4125

On Sat, May 20, 2017 at 9:05 AM, Jeyhun Karimov 
wrote:

> Dear community,
>
> As we discussed in KIP-149 [DISCUSS] thread [1], I would like to initiate
> KIP for rich functions (interfaces) [2].
> I would like to get your comments.
>
>
> [1]
> http://search-hadoop.com/m/Kafka/uyzND1PMjdk2CslH12?subj=
> Re+DISCUSS+KIP+149+Enabling+key+access+in+ValueTransformer+ValueMapper+
> and+ValueJoiner
> [2]
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 159%3A+Introducing+Rich+functions+to+Streams
>
>
> Cheers,
> Jeyhun
> --
> -Cheers
>
> Jeyhun
>



-- 
-- Guozhang

[jira] [Updated] (KAFKA-5364) Producer attempts to send transactional messages before adding partitions to transaction

[Apurva Mehta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Apurva Mehta updated KAFKA-5364:

Status: Patch Available  (was: In Progress)

> Producer attempts to send transactional messages before adding partitions to 
> transaction
> 
>
> Key: KAFKA-5364
> URL: https://issues.apache.org/jira/browse/KAFKA-5364
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Affects Versions: 0.11.0.0
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
>Priority: Blocker
>  Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> Due to a race condition between the sender thread and the producer.send(), 
> the following is possible: 
> # In KakfaProducer.doSend(), we add partitions to the transaction and then do 
> accumulator.append. 
> # In Sender.run(), we check whether there are transactional request. If there 
> are, we send them and wait for the response. 
> # If there aren't we drain the accumulator queue and send the produce 
> requests.
> # The problem is that the sequence step 2, 1, 3 is entire possible. This 
> means that we won't send the 'AddPartitions' request but yet try to send the 
> produce data. Which results in a fatal error and requires the producer to 
> close. 
> The solution is that in the accumulator.drain, we should check again if there 
> are pending add partitions requests, and if so, don't drain anything.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5364) Producer attempts to send transactional messages before adding partitions to transaction

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033956#comment-16033956
 ] 

ASF GitHub Bot commented on KAFKA-5364:
---

GitHub user apurvam opened a pull request:

https://github.com/apache/kafka/pull/3202

KAFKA-5364: Don't fail producer if drained partition is not yet in 
transaction

Due to the async nature of the producer, it is possible to attempt to drain 
a messages whose partition hasn't been added to the transaction yet. Before 
this patch, we considered this a fatal error. However, it is only in error if 
the partition isn't in the queue to be sent to the coordinator. 

This patch updates the logic so that we only fail the producer if the 
partition would never be added to the transaction. If the partition of the 
batch is yet to be added, we will simply wait for the partition to be added to 
the transaction before sending the batch to the broker.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apurvam/kafka 
KAFKA-5364-ensure-partitions-added-to-txn-before-send

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3202.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3202


commit 7dd2f7f9c3c59c80ed0cac4965df819a8c7ddcbb
Author: Apurva Mehta 
Date:   2017-06-02T00:18:08Z

Fix logic in ensurePartitionAddedToTransaction to account for partitions 
which would be added to the transaction.




> Producer attempts to send transactional messages before adding partitions to 
> transaction
> 
>
> Key: KAFKA-5364
> URL: https://issues.apache.org/jira/browse/KAFKA-5364
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Affects Versions: 0.11.0.0
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
>Priority: Blocker
>  Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> Due to a race condition between the sender thread and the producer.send(), 
> the following is possible: 
> # In KakfaProducer.doSend(), we add partitions to the transaction and then do 
> accumulator.append. 
> # In Sender.run(), we check whether there are transactional request. If there 
> are, we send them and wait for the response. 
> # If there aren't we drain the accumulator queue and send the produce 
> requests.
> # The problem is that the sequence step 2, 1, 3 is entire possible. This 
> means that we won't send the 'AddPartitions' request but yet try to send the 
> produce data. Which results in a fatal error and requires the producer to 
> close. 
> The solution is that in the accumulator.drain, we should check again if there 
> are pending add partitions requests, and if so, don't drain anything.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3202: KAFKA-5364: Don't fail producer if drained partiti...

[apurvam]

GitHub user apurvam opened a pull request:

https://github.com/apache/kafka/pull/3202

KAFKA-5364: Don't fail producer if drained partition is not yet in 
transaction

Due to the async nature of the producer, it is possible to attempt to drain 
a messages whose partition hasn't been added to the transaction yet. Before 
this patch, we considered this a fatal error. However, it is only in error if 
the partition isn't in the queue to be sent to the coordinator. 

This patch updates the logic so that we only fail the producer if the 
partition would never be added to the transaction. If the partition of the 
batch is yet to be added, we will simply wait for the partition to be added to 
the transaction before sending the batch to the broker.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/apurvam/kafka 
KAFKA-5364-ensure-partitions-added-to-txn-before-send

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3202.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3202


commit 7dd2f7f9c3c59c80ed0cac4965df819a8c7ddcbb
Author: Apurva Mehta 
Date:   2017-06-02T00:18:08Z

Fix logic in ensurePartitionAddedToTransaction to account for partitions 
which would be added to the transaction.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Work started] (KAFKA-5364) Producer attempts to send transactional messages before adding partitions to transaction

[Apurva Mehta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-5364 started by Apurva Mehta.
---
> Producer attempts to send transactional messages before adding partitions to 
> transaction
> 
>
> Key: KAFKA-5364
> URL: https://issues.apache.org/jira/browse/KAFKA-5364
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Affects Versions: 0.11.0.0
>Reporter: Apurva Mehta
>Assignee: Apurva Mehta
>Priority: Blocker
>  Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> Due to a race condition between the sender thread and the producer.send(), 
> the following is possible: 
> # In KakfaProducer.doSend(), we add partitions to the transaction and then do 
> accumulator.append. 
> # In Sender.run(), we check whether there are transactional request. If there 
> are, we send them and wait for the response. 
> # If there aren't we drain the accumulator queue and send the produce 
> requests.
> # The problem is that the sequence step 2, 1, 3 is entire possible. This 
> means that we won't send the 'AddPartitions' request but yet try to send the 
> produce data. Which results in a fatal error and requires the producer to 
> close. 
> The solution is that in the accumulator.drain, we should check again if there 
> are pending add partitions requests, and if so, don't drain anything.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: Reg: [VOTE] KIP 157 - Add consumer config options to streams reset tool

[Guozhang Wang]

+1 again. Thanks.

On Tue, May 30, 2017 at 1:46 PM, BigData dev 
wrote:

> Hi All,
> Updated the KIP, as the consumer configurations are required for both Admin
> Client and Consumer in Stream reset tool. Updated the KIP to use
> command-config option, similar to other tools like kafka-consumer-groups.sh
>
>
> *https://cwiki.apache.org/confluence/display/KAFKA/KIP+
> 157+-+Add+consumer+config+options+to+streams+reset+tool
>  157+-+Add+consumer+config+options+to+streams+reset+tool>*
>
>
> So, starting the voting process again for further inputs.
>
> This vote will run for a minimum of 72 hours.
>
> Thanks,
>
> Bharat
>
>
>
> On Tue, May 30, 2017 at 1:18 PM, Guozhang Wang  wrote:
>
> > +1. Thanks!
> >
> > On Tue, May 16, 2017 at 1:12 AM, Eno Thereska 
> > wrote:
> >
> > > +1 thanks.
> > >
> > > Eno
> > > > On 16 May 2017, at 04:20, BigData dev 
> wrote:
> > > >
> > > > Hi All,
> > > > Given the simple and non-controversial nature of the KIP, I would
> like
> > to
> > > > start the voting process for KIP-157: Add consumer config options to
> > > > streams reset tool
> > > >
> > > > *https://cwiki.apache.org/confluence/display/KAFKA/KIP+157+-
> > > +Add+consumer+config+options+to+streams+reset+tool
> > > >  > > +Add+consumer+config+options+to+streams+reset+tool>*
> > > >
> > > >
> > > > The vote will run for a minimum of 72 hours.
> > > >
> > > > Thanks,
> > > >
> > > > Bharat
> > >
> > >
> >
> >
> > --
> > -- Guozhang
> >
>



-- 
-- Guozhang

[jira] [Created] (KAFKA-5365) Fix regression in compressed message iteration affecting magic v0 and v1

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-5365:
--

 Summary: Fix regression in compressed message iteration affecting 
magic v0 and v1
 Key: KAFKA-5365
 URL: https://issues.apache.org/jira/browse/KAFKA-5365
 Project: Kafka
  Issue Type: Sub-task
Reporter: Jason Gustafson
Assignee: Jason Gustafson
Priority: Blocker
 Fix For: 0.11.0.0


We added a shortcut to break iteration over compressed message sets for v0 and 
v1 if the inner offset matches the last offset in the wrapper. Unfortunately 
this breaks older clients which may use offset 0 in the wrapper record in 
records sent in produce requests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5362) Add EOS system tests for Streams API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033909#comment-16033909
 ] 

ASF GitHub Bot commented on KAFKA-5362:
---

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/3201

KAFKA-5362: Add EOS system tests for Streams API



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka 
kafka-5362-add-eos-system-tests-for-streams-api

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3201.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3201


commit 22abf4f18d009b286ed7bb84ca89959c5a829cef
Author: Matthias J. Sax 
Date:   2017-05-30T19:14:55Z

KAFKA-5362: Add EOS system tests for Streams API

commit 321c6a2c640e8284ce016cdbd52173dc2620ee5e
Author: Matthias J. Sax 
Date:   2017-06-01T22:20:20Z

cleanup




> Add EOS system tests for Streams API
> 
>
> Key: KAFKA-5362
> URL: https://issues.apache.org/jira/browse/KAFKA-5362
> Project: Kafka
>  Issue Type: Bug
>  Components: streams, system tests
>Affects Versions: 0.11.0.0
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Blocker
>  Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> We need to add more system tests for Streams API with exactly-once enabled.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3201: KAFKA-5362: Add EOS system tests for Streams API

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/3201

KAFKA-5362: Add EOS system tests for Streams API



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka 
kafka-5362-add-eos-system-tests-for-streams-api

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3201.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3201


commit 22abf4f18d009b286ed7bb84ca89959c5a829cef
Author: Matthias J. Sax 
Date:   2017-05-30T19:14:55Z

KAFKA-5362: Add EOS system tests for Streams API

commit 321c6a2c640e8284ce016cdbd52173dc2620ee5e
Author: Matthias J. Sax 
Date:   2017-06-01T22:20:20Z

cleanup




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka-site pull request #48: Filled in og meta tags

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka-site/pull/48


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka-site issue #48: Filled in og meta tags

[guozhangwang]

Github user guozhangwang commented on the issue:

https://github.com/apache/kafka-site/pull/48
  
LGTM. Merged to asf-site.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka-site issue #59: Add auto syntax highlighter

[guozhangwang]

Github user guozhangwang commented on the issue:

https://github.com/apache/kafka-site/pull/59
  
LGTM. Merged to asf-site.


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka-site pull request #59: Add auto syntax highlighter

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka-site/pull/59


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2358) Cluster collection returning methods should never return null

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033836#comment-16033836
 ] 

ASF GitHub Bot commented on KAFKA-2358:
---

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/3200

MINOR: Add upgrade notes for KAFKA-2358



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka 
KMinor-KAFKA-2358-upgrade-notes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3200.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3200


commit 13394329c03eee647d1d194972b10108666ed6e1
Author: Guozhang Wang 
Date:   2017-06-01T22:40:02Z

upgrade notes for KAFKA-2358




> Cluster collection returning methods should never return null
> -
>
> Key: KAFKA-2358
> URL: https://issues.apache.org/jira/browse/KAFKA-2358
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.9.0.0
>Reporter: Stevo Slavic
>Assignee: Stevo Slavic
>Priority: Minor
> Fix For: 0.11.0.0
>
>
> {{KafkaConsumer.partitionsFor}} method by it's signature returns a 
> {{List}}. Problem is that in case (metadata for) topic does 
> not exist, current implementation will return null, which is considered a bad 
> practice - instead of null it should return empty list.
> Root cause is that the Cluster collection returning methods are returning 
> null.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3200: MINOR: Add upgrade notes for KAFKA-2358

[guozhangwang]

GitHub user guozhangwang opened a pull request:

https://github.com/apache/kafka/pull/3200

MINOR: Add upgrade notes for KAFKA-2358



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/guozhangwang/kafka 
KMinor-KAFKA-2358-upgrade-notes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3200.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3200


commit 13394329c03eee647d1d194972b10108666ed6e1
Author: Guozhang Wang 
Date:   2017-06-01T22:40:02Z

upgrade notes for KAFKA-2358




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3194: MINOR: update Streams docs for KIP-123

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3194


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-5364) Producer attempts to send transactional messages before adding partitions to transaction

[Apurva Mehta (JIRA)]

Apurva Mehta created KAFKA-5364:
---

 Summary: Producer attempts to send transactional messages before 
adding partitions to transaction
 Key: KAFKA-5364
 URL: https://issues.apache.org/jira/browse/KAFKA-5364
 Project: Kafka
  Issue Type: Sub-task
Affects Versions: 0.11.0.0
Reporter: Apurva Mehta
Assignee: Apurva Mehta
Priority: Blocker
 Fix For: 0.11.0.0


Due to a race condition between the sender thread and the producer.send(), the 
following is possible: 

# In KakfaProducer.doSend(), we add partitions to the transaction and then do 
accumulator.append. 
# In Sender.run(), we check whether there are transactional request. If there 
are, we send them and wait for the response. 
# If there aren't we drain the accumulator queue and send the produce requests.
# The problem is that the sequence step 2, 1, 3 is entire possible. This means 
that we won't send the 'AddPartitions' request but yet try to send the produce 
data. Which results in a fatal error and requires the producer to close. 

The solution is that in the accumulator.drain, we should check again if there 
are pending add partitions requests, and if so, don't drain anything.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-167: Add a restoreAll method to StateRestoreCallback

[Guozhang Wang]

There are also some request to add "begin" and "after" callbacks in the
restoration func:

https://issues.apache.org/jira/browse/KAFKA-4322

Could we piggy back them into the same KIP?


Guozhang

On Thu, Jun 1, 2017 at 2:04 PM, Bill Bejeck  wrote:

> All,
>
> I'd like to start the discussion for adding bulk add functionality when
> restoring a state store.  The KIP can be found here:
>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 167%3A+Add+a+restoreAll+method+to+StateRestoreCallback
>
> Thanks,
> Bill
>



-- 
-- Guozhang

[jira] [Commented] (KAFKA-5311) Support ExtendedDeserializer in Kafka Streams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033793#comment-16033793
 ] 

ASF GitHub Bot commented on KAFKA-5311:
---

GitHub user subnova opened a pull request:

https://github.com/apache/kafka/pull/3199

KAFKA-5311: Support ExtendedDeserializer in Kafka Streams.

The contribution is my original work and I license the work to the project 
under the project's open source license.

This is a clone of the previous PR #3125 but now against trunk.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/subnova/kafka streams-extendeddeserializer

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3199.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3199


commit 32a75c6a3a072aceff88a87e381ad3e4b46d04f7
Author: Dale Peakall 
Date:   2017-05-23T10:40:32Z

KAFKA-5311: Support ExtendedDeserializer in Kafka Streams.

Author: Dale Peakall 




> Support ExtendedDeserializer in Kafka Streams
> -
>
> Key: KAFKA-5311
> URL: https://issues.apache.org/jira/browse/KAFKA-5311
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.11.0.0
>Reporter: Dale Peakall
>Assignee: Dale Peakall
> Fix For: 0.11.0.0
>
>
> KIP-82 introduced the concept of message headers and introduced an 
> ExtendedDeserializer interface that allowed a Deserializer to access those 
> message headers.
> Change Kafka Streams to support the use of ExtendedDeserializer to provide 
> compatibility with Kafka Clients that use the new header functionality.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3199: KAFKA-5311: Support ExtendedDeserializer in Kafka ...

[subnova]

GitHub user subnova opened a pull request:

https://github.com/apache/kafka/pull/3199

KAFKA-5311: Support ExtendedDeserializer in Kafka Streams.

The contribution is my original work and I license the work to the project 
under the project's open source license.

This is a clone of the previous PR #3125 but now against trunk.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/subnova/kafka streams-extendeddeserializer

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3199.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3199


commit 32a75c6a3a072aceff88a87e381ad3e4b46d04f7
Author: Dale Peakall 
Date:   2017-05-23T10:40:32Z

KAFKA-5311: Support ExtendedDeserializer in Kafka Streams.

Author: Dale Peakall 




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[DISCUSS] KIP-167: Add a restoreAll method to StateRestoreCallback

[Bill Bejeck]

All,

I'd like to start the discussion for adding bulk add functionality when
restoring a state store.  The KIP can be found here:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-167%3A+Add+a+restoreAll+method+to+StateRestoreCallback

Thanks,
Bill

[jira] [Updated] (KAFKA-5164) SetSchemaMetadata does not replace the schemas in structs correctly

[Randall Hauch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Randall Hauch updated KAFKA-5164:
-
Status: Patch Available  (was: Open)

> SetSchemaMetadata does not replace the schemas in structs correctly
> ---
>
> Key: KAFKA-5164
> URL: https://issues.apache.org/jira/browse/KAFKA-5164
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Randall Hauch
>
> In SetSchemaMetadataTest we verify that the name and version of the schema in 
> the record have been replaced:
> https://github.com/apache/kafka/blob/trunk/connect/transforms/src/test/java/org/apache/kafka/connect/transforms/SetSchemaMetadataTest.java#L62
> However, in the case of Structs, the schema will be attached to both the 
> record and the Struct itself. So we correctly rebuild the Record:
> https://github.com/apache/kafka/blob/trunk/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/SetSchemaMetadata.java#L77
> https://github.com/apache/kafka/blob/trunk/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/SetSchemaMetadata.java#L104
> https://github.com/apache/kafka/blob/trunk/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/SetSchemaMetadata.java#L119
> But if the key or value are a struct, they will still contain the old schema 
> embedded in the struct.
> Ultimately this can lead to validations in other code failing (even for very 
> simple changes like adjusting the name of a schema):
> {code}
> (org.apache.kafka.connect.runtime.WorkerTask:141)
> org.apache.kafka.connect.errors.DataException: Mismatching struct schema
> at io.confluent.connect.avro.AvroData.fromConnectData(AvroData.java:471)
> at io.confluent.connect.avro.AvroData.fromConnectData(AvroData.java:295)
> at 
> io.confluent.connect.avro.AvroConverter.fromConnectData(AvroConverter.java:73)
> at 
> org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:196)
> at 
> org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:167)
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:139)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:182)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> The solution to this is probably to check whether we're dealing with a Struct 
> when we use a new schema and potentially copy/reallocate it.
> This particular issue would only appear if we don't modify the data, so I 
> think SetSchemaMetadata is currently the only transformation that would have 
> the issue.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5164) SetSchemaMetadata does not replace the schemas in structs correctly

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033687#comment-16033687
 ] 

ASF GitHub Bot commented on KAFKA-5164:
---

GitHub user rhauch opened a pull request:

https://github.com/apache/kafka/pull/3198

KAFKA-5164 Ensure SetSchemaMetadata updates key or value when Schema changes

When the `SetSchemaMetadata` SMT is used to change the name and/or version 
of the key or value’s schema, any references to the old schema in the key or 
value must be changed to reference the new schema. Only keys or values that are 
`Struct` have such references, and so currently only these are adjusted.

This is based on `trunk` since the fix is expected to be targeted to the 
0.11.1 release.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rhauch/kafka kafka-5164

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3198.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3198


commit 68bf68b3df065437318e2a87a6d1181b1205a806
Author: Randall Hauch 
Date:   2017-06-01T20:40:01Z

KAFKA-5164 Ensure SetSchemaMetadata updates key or value when Schema changes

When the `SetSchemaMetadata` SMT is used to change the name and/or version 
of the key or value’s schema, any references to the old schema in the key or 
value must be changed to reference the new schema. Only keys or values that are 
`Struct` have such references, and so currently only these are adjusted.




> SetSchemaMetadata does not replace the schemas in structs correctly
> ---
>
> Key: KAFKA-5164
> URL: https://issues.apache.org/jira/browse/KAFKA-5164
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.1
>Reporter: Ewen Cheslack-Postava
>
> In SetSchemaMetadataTest we verify that the name and version of the schema in 
> the record have been replaced:
> https://github.com/apache/kafka/blob/trunk/connect/transforms/src/test/java/org/apache/kafka/connect/transforms/SetSchemaMetadataTest.java#L62
> However, in the case of Structs, the schema will be attached to both the 
> record and the Struct itself. So we correctly rebuild the Record:
> https://github.com/apache/kafka/blob/trunk/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/SetSchemaMetadata.java#L77
> https://github.com/apache/kafka/blob/trunk/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/SetSchemaMetadata.java#L104
> https://github.com/apache/kafka/blob/trunk/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/SetSchemaMetadata.java#L119
> But if the key or value are a struct, they will still contain the old schema 
> embedded in the struct.
> Ultimately this can lead to validations in other code failing (even for very 
> simple changes like adjusting the name of a schema):
> {code}
> (org.apache.kafka.connect.runtime.WorkerTask:141)
> org.apache.kafka.connect.errors.DataException: Mismatching struct schema
> at io.confluent.connect.avro.AvroData.fromConnectData(AvroData.java:471)
> at io.confluent.connect.avro.AvroData.fromConnectData(AvroData.java:295)
> at 
> io.confluent.connect.avro.AvroConverter.fromConnectData(AvroConverter.java:73)
> at 
> org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:196)
> at 
> org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:167)
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:139)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:182)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> The solution to this is probably to check whether we're dealing with a Struct 
> when we use a new schema and potentially copy/reallocate it.
> This particular issue would only appear if we don't modify the data, so I 
> think SetSchemaMetadata is currently the only transformation that would have 
> the issue.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-5164) SetSchemaMetadata does not replace the schemas in structs correctly

[Randall Hauch (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Randall Hauch reassigned KAFKA-5164:


Assignee: Randall Hauch

> SetSchemaMetadata does not replace the schemas in structs correctly
> ---
>
> Key: KAFKA-5164
> URL: https://issues.apache.org/jira/browse/KAFKA-5164
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Randall Hauch
>
> In SetSchemaMetadataTest we verify that the name and version of the schema in 
> the record have been replaced:
> https://github.com/apache/kafka/blob/trunk/connect/transforms/src/test/java/org/apache/kafka/connect/transforms/SetSchemaMetadataTest.java#L62
> However, in the case of Structs, the schema will be attached to both the 
> record and the Struct itself. So we correctly rebuild the Record:
> https://github.com/apache/kafka/blob/trunk/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/SetSchemaMetadata.java#L77
> https://github.com/apache/kafka/blob/trunk/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/SetSchemaMetadata.java#L104
> https://github.com/apache/kafka/blob/trunk/connect/transforms/src/main/java/org/apache/kafka/connect/transforms/SetSchemaMetadata.java#L119
> But if the key or value are a struct, they will still contain the old schema 
> embedded in the struct.
> Ultimately this can lead to validations in other code failing (even for very 
> simple changes like adjusting the name of a schema):
> {code}
> (org.apache.kafka.connect.runtime.WorkerTask:141)
> org.apache.kafka.connect.errors.DataException: Mismatching struct schema
> at io.confluent.connect.avro.AvroData.fromConnectData(AvroData.java:471)
> at io.confluent.connect.avro.AvroData.fromConnectData(AvroData.java:295)
> at 
> io.confluent.connect.avro.AvroConverter.fromConnectData(AvroConverter.java:73)
> at 
> org.apache.kafka.connect.runtime.WorkerSourceTask.sendRecords(WorkerSourceTask.java:196)
> at 
> org.apache.kafka.connect.runtime.WorkerSourceTask.execute(WorkerSourceTask.java:167)
> at org.apache.kafka.connect.runtime.WorkerTask.doRun(WorkerTask.java:139)
> at org.apache.kafka.connect.runtime.WorkerTask.run(WorkerTask.java:182)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> The solution to this is probably to check whether we're dealing with a Struct 
> when we use a new schema and potentially copy/reallocate it.
> This particular issue would only appear if we don't modify the data, so I 
> think SetSchemaMetadata is currently the only transformation that would have 
> the issue.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3198: KAFKA-5164 Ensure SetSchemaMetadata updates key or...

[rhauch]

GitHub user rhauch opened a pull request:

https://github.com/apache/kafka/pull/3198

KAFKA-5164 Ensure SetSchemaMetadata updates key or value when Schema changes

When the `SetSchemaMetadata` SMT is used to change the name and/or version 
of the key or value’s schema, any references to the old schema in the key or 
value must be changed to reference the new schema. Only keys or values that are 
`Struct` have such references, and so currently only these are adjusted.

This is based on `trunk` since the fix is expected to be targeted to the 
0.11.1 release.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rhauch/kafka kafka-5164

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3198.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3198


commit 68bf68b3df065437318e2a87a6d1181b1205a806
Author: Randall Hauch 
Date:   2017-06-01T20:40:01Z

KAFKA-5164 Ensure SetSchemaMetadata updates key or value when Schema changes

When the `SetSchemaMetadata` SMT is used to change the name and/or version 
of the key or value’s schema, any references to the old schema in the key or 
value must be changed to reference the new schema. Only keys or values that are 
`Struct` have such references, and so currently only these are adjusted.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-5345) Some socket connections not closed after restart of Kafka Streams

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033679#comment-16033679
 ] 

ASF GitHub Bot commented on KAFKA-5345:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3195


> Some socket connections not closed after restart of Kafka Streams
> -
>
> Key: KAFKA-5345
> URL: https://issues.apache.org/jira/browse/KAFKA-5345
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0, 0.10.2.1
> Environment: MacOs 10.12.5 and Ubuntu 14.04
>Reporter: Jeroen van Wilgenburg
>Assignee: Rajini Sivaram
> Fix For: 0.11.0.0
>
>
> We ran into a problem that resulted in a "Too many open files" exception 
> because some sockets are not closed after a restart.
> This problem only occurs with version {{0.10.2.1}} and {{0.10.2.0}}. 
> {{0.10.1.1}} and {{0.10.1.0}} both work as expected.
> I used the same version for the server and client.
> I used https://github.com/kohsuke/file-leak-detector to display the open file 
> descriptors. The culprit was :
> {noformat}
> #146 socket channel by thread:pool-2-thread-1 on Mon May 29 11:20:47 CEST 2017
>   at sun.nio.ch.SocketChannelImpl.(SocketChannelImpl.java:108)
>   at 
> sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)
>   at java.nio.channels.SocketChannel.open(SocketChannel.java:145)
>   at org.apache.kafka.common.network.Selector.connect(Selector.java:168)
>   at 
> org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:629)
>   at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:186)
>   at 
> org.apache.kafka.streams.processor.internals.StreamsKafkaClient.ensureOneNodeIsReady(StreamsKafkaClient.java:195)
>   at 
> org.apache.kafka.streams.processor.internals.StreamsKafkaClient.getAnyReadyBrokerId(StreamsKafkaClient.java:233)
>   at 
> org.apache.kafka.streams.processor.internals.StreamsKafkaClient.checkBrokerCompatibility(StreamsKafkaClient.java:300)
>   at 
> org.apache.kafka.streams.KafkaStreams.checkBrokerVersionCompatibility(KafkaStreams.java:401)
>   at org.apache.kafka.streams.KafkaStreams.start(KafkaStreams.java:425)
> {noformat}
>   
>   
> I could narrow the problem down to a reproducable example below (the only 
> dependency is 
> {{org.apache.kafka:kafka-streams:jar:0.10.2.1}}). 
> *IMPORTANT*: You have to run this code in the Intellij IDEA debugger with a 
> special breakpoint to see it fail. 
> See the comments on the socketChannels variable on how to add this 
> breakpoint. 
> When you run this code you will see the number of open SocketChannels 
> increase (only on version 0.10.2.x).
>   
> {code:title=App.java}
> import org.apache.kafka.common.serialization.Serdes;
> import org.apache.kafka.streams.KafkaStreams;
> import org.apache.kafka.streams.StreamsConfig;
> import org.apache.kafka.streams.kstream.KStream;
> import org.apache.kafka.streams.kstream.KStreamBuilder;
> import java.nio.channels.SocketChannel;
> import java.nio.channels.spi.AbstractInterruptibleChannel;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Properties;
> import java.util.concurrent.Executors;
> import java.util.concurrent.ScheduledExecutorService;
> import java.util.concurrent.TimeUnit;
> import java.util.stream.Collectors;
> public class App {
> private static KafkaStreams streams;
> private static String brokerList;
> // Fill socketChannels with entries on line 'Socket socket = 
> socketChannel.socket();' (line number 170  on 0.10.2.1)
> // of org.apache.kafka.common.network.Selector: Add breakpoint, right 
> click on breakpoint.
> // - Uncheck 'Suspend'
> // - Check 'Evaluate and log' and fill text field with (without quotes) 
> 'App.socketChannels.add(socketChannel)'
> private static final List socketChannels = new 
> ArrayList<>();
> public static void main(String[] args) {
> brokerList = args[0];
> init();
> ScheduledExecutorService scheduledThreadPool = 
> Executors.newScheduledThreadPool(1);
> Runnable command = () -> {
> streams.close();
> System.out.println("Open socketChannels: " + 
> socketChannels.stream()
> .filter(AbstractInterruptibleChannel::isOpen)
> .collect(Collectors.toList()).size());
> init();
> };
> scheduledThreadPool.scheduleWithFixedDelay(command, 1L, 2000, 
> TimeUnit.MILLISECONDS);
> }
> private static void init() {
> Properties streamsConfiguration = new Properties();
> streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, 
> "JeroenApp");
>

[jira] [Resolved] (KAFKA-5345) Some socket connections not closed after restart of Kafka Streams

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-5345.

   Resolution: Fixed
Fix Version/s: 0.11.0.0

Issue resolved by pull request 3195
[https://github.com/apache/kafka/pull/3195]

> Some socket connections not closed after restart of Kafka Streams
> -
>
> Key: KAFKA-5345
> URL: https://issues.apache.org/jira/browse/KAFKA-5345
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0, 0.10.2.1
> Environment: MacOs 10.12.5 and Ubuntu 14.04
>Reporter: Jeroen van Wilgenburg
>Assignee: Rajini Sivaram
> Fix For: 0.11.0.0
>
>
> We ran into a problem that resulted in a "Too many open files" exception 
> because some sockets are not closed after a restart.
> This problem only occurs with version {{0.10.2.1}} and {{0.10.2.0}}. 
> {{0.10.1.1}} and {{0.10.1.0}} both work as expected.
> I used the same version for the server and client.
> I used https://github.com/kohsuke/file-leak-detector to display the open file 
> descriptors. The culprit was :
> {noformat}
> #146 socket channel by thread:pool-2-thread-1 on Mon May 29 11:20:47 CEST 2017
>   at sun.nio.ch.SocketChannelImpl.(SocketChannelImpl.java:108)
>   at 
> sun.nio.ch.SelectorProviderImpl.openSocketChannel(SelectorProviderImpl.java:60)
>   at java.nio.channels.SocketChannel.open(SocketChannel.java:145)
>   at org.apache.kafka.common.network.Selector.connect(Selector.java:168)
>   at 
> org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:629)
>   at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:186)
>   at 
> org.apache.kafka.streams.processor.internals.StreamsKafkaClient.ensureOneNodeIsReady(StreamsKafkaClient.java:195)
>   at 
> org.apache.kafka.streams.processor.internals.StreamsKafkaClient.getAnyReadyBrokerId(StreamsKafkaClient.java:233)
>   at 
> org.apache.kafka.streams.processor.internals.StreamsKafkaClient.checkBrokerCompatibility(StreamsKafkaClient.java:300)
>   at 
> org.apache.kafka.streams.KafkaStreams.checkBrokerVersionCompatibility(KafkaStreams.java:401)
>   at org.apache.kafka.streams.KafkaStreams.start(KafkaStreams.java:425)
> {noformat}
>   
>   
> I could narrow the problem down to a reproducable example below (the only 
> dependency is 
> {{org.apache.kafka:kafka-streams:jar:0.10.2.1}}). 
> *IMPORTANT*: You have to run this code in the Intellij IDEA debugger with a 
> special breakpoint to see it fail. 
> See the comments on the socketChannels variable on how to add this 
> breakpoint. 
> When you run this code you will see the number of open SocketChannels 
> increase (only on version 0.10.2.x).
>   
> {code:title=App.java}
> import org.apache.kafka.common.serialization.Serdes;
> import org.apache.kafka.streams.KafkaStreams;
> import org.apache.kafka.streams.StreamsConfig;
> import org.apache.kafka.streams.kstream.KStream;
> import org.apache.kafka.streams.kstream.KStreamBuilder;
> import java.nio.channels.SocketChannel;
> import java.nio.channels.spi.AbstractInterruptibleChannel;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Properties;
> import java.util.concurrent.Executors;
> import java.util.concurrent.ScheduledExecutorService;
> import java.util.concurrent.TimeUnit;
> import java.util.stream.Collectors;
> public class App {
> private static KafkaStreams streams;
> private static String brokerList;
> // Fill socketChannels with entries on line 'Socket socket = 
> socketChannel.socket();' (line number 170  on 0.10.2.1)
> // of org.apache.kafka.common.network.Selector: Add breakpoint, right 
> click on breakpoint.
> // - Uncheck 'Suspend'
> // - Check 'Evaluate and log' and fill text field with (without quotes) 
> 'App.socketChannels.add(socketChannel)'
> private static final List socketChannels = new 
> ArrayList<>();
> public static void main(String[] args) {
> brokerList = args[0];
> init();
> ScheduledExecutorService scheduledThreadPool = 
> Executors.newScheduledThreadPool(1);
> Runnable command = () -> {
> streams.close();
> System.out.println("Open socketChannels: " + 
> socketChannels.stream()
> .filter(AbstractInterruptibleChannel::isOpen)
> .collect(Collectors.toList()).size());
> init();
> };
> scheduledThreadPool.scheduleWithFixedDelay(command, 1L, 2000, 
> TimeUnit.MILLISECONDS);
> }
> private static void init() {
> Properties streamsConfiguration = new Properties();
> streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, 
> "JeroenApp");
> streamsConfiguration.p

[GitHub] kafka pull request #3195: KAFKA-5345: Close KafkaClient when streams client ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3195


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-5355) Broker returns messages beyond "latest stable offset" to transactional consumer in read_committed mode

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-5355:
---
Description: 
This issue is exposed by the new Streams EOS integration test.

Streams has two tasks (ie, two producers with {{pid}} 0 and 2000) both writing 
to output topic {{output}} with one partition (replication factor 1).

The test uses an transactional consumer with {{group.id=readCommitted}} to read 
the data from {{output}} topic. When it read the data, each producer has 
committed 10 records (one producer write messages with {{key=0}} and the other 
with {{key=1}}). Furthermore, each producer has an open transaction and 5 
uncommitted records written.

The test fails, as we expect to see 10 records per key, but we get 15 for key=1:
{noformat}
java.lang.AssertionError: 
Expected: <[KeyValue(1, 0), KeyValue(1, 1), KeyValue(1, 3), KeyValue(1, 6), 
KeyValue(1, 10), KeyValue(1, 15), KeyValue(1, 21), KeyValue(1, 28), KeyValue(1, 
36), KeyValue(1, 45)]>
 but: was <[KeyValue(1, 0), KeyValue(1, 1), KeyValue(1, 3), KeyValue(1, 6), 
KeyValue(1, 10), KeyValue(1, 15), KeyValue(1, 21), KeyValue(1, 28), KeyValue(1, 
36), KeyValue(1, 45), KeyValue(1, 55), KeyValue(1, 66), KeyValue(1, 78), 
KeyValue(1, 91), KeyValue(1, 105)]>
{noformat}

Dumping the segment shows, that there are two commit markers (one for each 
producer) for the first 10 messages written. Furthermore, there are 5 pending 
records. Thus, "latest stable offset" should be 21 (20 messages plus 2 commit 
markers) and not data should be returned beyond this offset.

Dumped Log Segment {{output-0}}
{noformat}
Starting offset: 0
baseOffset: 0 lastOffset: 9 baseSequence: 0 lastSequence: 9 producerId: 0 
producerEpoch: 6 partitionLeaderEpoch: 0 isTransactional: true position: 0 
CreateTime: 1496255947332 isvalid: true size: 291 magic: 2 compresscodec: NONE 
crc: 600535135
baseOffset: 10 lastOffset: 10 baseSequence: -1 lastSequence: -1 producerId: 0 
producerEpoch: 6 partitionLeaderEpoch: 0 isTransactional: true position: 291 
CreateTime: 1496256005429 isvalid: true size: 78 magic: 2 compresscodec: NONE 
crc: 3458060752
baseOffset: 11 lastOffset: 20 baseSequence: 0 lastSequence: 9 producerId: 2000 
producerEpoch: 2 partitionLeaderEpoch: 0 isTransactional: true position: 369 
CreateTime: 1496255947322 isvalid: true size: 291 magic: 2 compresscodec: NONE 
crc: 3392915713
baseOffset: 21 lastOffset: 25 baseSequence: 10 lastSequence: 14 producerId: 0 
producerEpoch: 6 partitionLeaderEpoch: 0 isTransactional: true position: 660 
CreateTime: 1496255947342 isvalid: true size: 176 magic: 2 compresscodec: NONE 
crc: 3513911368
baseOffset: 26 lastOffset: 26 baseSequence: -1 lastSequence: -1 producerId: 
2000 producerEpoch: 2 partitionLeaderEpoch: 0 isTransactional: true position: 
836 CreateTime: 1496256011784 isvalid: true size: 78 magic: 2 compresscodec: 
NONE crc: 1619151485
{noformat}

Dump with {{--deep-iteration}}
{noformat}
Starting offset: 0
offset: 0 position: 0 CreateTime: 1496255947323 isvalid: true keysize: 8 
valuesize: 8 magic: 2 compresscodec: NONE crc: 600535135 sequence: 0 
headerKeys: [] key: 1 payload: 0
offset: 1 position: 0 CreateTime: 1496255947324 isvalid: true keysize: 8 
valuesize: 8 magic: 2 compresscodec: NONE crc: 600535135 sequence: 1 
headerKeys: [] key: 1 payload: 1
offset: 2 position: 0 CreateTime: 1496255947325 isvalid: true keysize: 8 
valuesize: 8 magic: 2 compresscodec: NONE crc: 600535135 sequence: 2 
headerKeys: [] key: 1 payload: 3
offset: 3 position: 0 CreateTime: 1496255947326 isvalid: true keysize: 8 
valuesize: 8 magic: 2 compresscodec: NONE crc: 600535135 sequence: 3 
headerKeys: [] key: 1 payload: 6
offset: 4 position: 0 CreateTime: 1496255947327 isvalid: true keysize: 8 
valuesize: 8 magic: 2 compresscodec: NONE crc: 600535135 sequence: 4 
headerKeys: [] key: 1 payload: 10
offset: 5 position: 0 CreateTime: 1496255947328 isvalid: true keysize: 8 
valuesize: 8 magic: 2 compresscodec: NONE crc: 600535135 sequence: 5 
headerKeys: [] key: 1 payload: 15
offset: 6 position: 0 CreateTime: 1496255947329 isvalid: true keysize: 8 
valuesize: 8 magic: 2 compresscodec: NONE crc: 600535135 sequence: 6 
headerKeys: [] key: 1 payload: 21
offset: 7 position: 0 CreateTime: 1496255947330 isvalid: true keysize: 8 
valuesize: 8 magic: 2 compresscodec: NONE crc: 600535135 sequence: 7 
headerKeys: [] key: 1 payload: 28
offset: 8 position: 0 CreateTime: 1496255947331 isvalid: true keysize: 8 
valuesize: 8 magic: 2 compresscodec: NONE crc: 600535135 sequence: 8 
headerKeys: [] key: 1 payload: 36
offset: 9 position: 0 CreateTime: 1496255947332 isvalid: true keysize: 8 
valuesize: 8 magic: 2 compresscodec: NONE crc: 600535135 sequence: 9 
headerKeys: [] key: 1 payload: 45
offset: 10 position: 291 CreateTime: 1496256005429 isvalid: true keysize: 4 
valuesize: 6 magic: 2 compresscodec: NONE crc: 3458060752 sequence: -1 
headerKeys: [] endTxnMarke

[jira] [Commented] (KAFKA-5355) Broker returns messages beyond "latest stable offset" to transactional consumer in read_committed mode

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033666#comment-16033666
 ] 

Matthias J. Sax commented on KAFKA-5355:


The Streams test revealing this issue is 
{{EosIntegrationTest#shouldNotViolateEosIfOneTaskFails}} -- if this get's 
fixed, the test should get enabled again. The second disabled test 
{{#shouldNotViolateEosIfOneTaskFailsWithState}} might also be enabled (We need 
to tested if it got fixed, too, though. It's unclear if it hit the same or 
another issue atm).

> Broker returns messages beyond "latest stable offset" to transactional 
> consumer in read_committed mode
> --
>
> Key: KAFKA-5355
> URL: https://issues.apache.org/jira/browse/KAFKA-5355
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Affects Versions: 0.11.0.0
>Reporter: Matthias J. Sax
>Assignee: Jason Gustafson
>Priority: Blocker
>  Labels: exactly-once
> Fix For: 0.11.0.0
>
> Attachments: test.log
>
>
> This issue is exposed by the new (not yet committed) Streams EOS integration 
> test.
> Streams has two tasks (ie, two producers with {{pid}} 0 and 2000) both 
> writing to output topic {{output}} with one partition (replication factor 1).
> The test uses an transactional consumer with {{group.id=readCommitted}} to 
> read the data from {{output}} topic. When it read the data, each producer has 
> committed 10 records (one producer write messages with {{key=0}} and the 
> other with {{key=1}}). Furthermore, each producer has an open transaction and 
> 5 uncommitted records written.
> The test fails, as we expect to see 10 records per key, but we get 15 for 
> key=1:
> {noformat}
> java.lang.AssertionError: 
> Expected: <[KeyValue(1, 0), KeyValue(1, 1), KeyValue(1, 3), KeyValue(1, 6), 
> KeyValue(1, 10), KeyValue(1, 15), KeyValue(1, 21), KeyValue(1, 28), 
> KeyValue(1, 36), KeyValue(1, 45)]>
>  but: was <[KeyValue(1, 0), KeyValue(1, 1), KeyValue(1, 3), KeyValue(1, 
> 6), KeyValue(1, 10), KeyValue(1, 15), KeyValue(1, 21), KeyValue(1, 28), 
> KeyValue(1, 36), KeyValue(1, 45), KeyValue(1, 55), KeyValue(1, 66), 
> KeyValue(1, 78), KeyValue(1, 91), KeyValue(1, 105)]>
> {noformat}
> Dumping the segment shows, that there are two commit markers (one for each 
> producer) for the first 10 messages written. Furthermore, there are 5 pending 
> records. Thus, "latest stable offset" should be 21 (20 messages plus 2 commit 
> markers) and not data should be returned beyond this offset.
> Dumped Log Segment {{output-0}}
> {noformat}
> Starting offset: 0
> baseOffset: 0 lastOffset: 9 baseSequence: 0 lastSequence: 9 producerId: 0 
> producerEpoch: 6 partitionLeaderEpoch: 0 isTransactional: true position: 0 
> CreateTime: 1496255947332 isvalid: true size: 291 magic: 2 compresscodec: 
> NONE crc: 600535135
> baseOffset: 10 lastOffset: 10 baseSequence: -1 lastSequence: -1 producerId: 0 
> producerEpoch: 6 partitionLeaderEpoch: 0 isTransactional: true position: 291 
> CreateTime: 1496256005429 isvalid: true size: 78 magic: 2 compresscodec: NONE 
> crc: 3458060752
> baseOffset: 11 lastOffset: 20 baseSequence: 0 lastSequence: 9 producerId: 
> 2000 producerEpoch: 2 partitionLeaderEpoch: 0 isTransactional: true position: 
> 369 CreateTime: 1496255947322 isvalid: true size: 291 magic: 2 compresscodec: 
> NONE crc: 3392915713
> baseOffset: 21 lastOffset: 25 baseSequence: 10 lastSequence: 14 producerId: 0 
> producerEpoch: 6 partitionLeaderEpoch: 0 isTransactional: true position: 660 
> CreateTime: 1496255947342 isvalid: true size: 176 magic: 2 compresscodec: 
> NONE crc: 3513911368
> baseOffset: 26 lastOffset: 26 baseSequence: -1 lastSequence: -1 producerId: 
> 2000 producerEpoch: 2 partitionLeaderEpoch: 0 isTransactional: true position: 
> 836 CreateTime: 1496256011784 isvalid: true size: 78 magic: 2 compresscodec: 
> NONE crc: 1619151485
> {noformat}
> Dump with {{--deep-iteration}}
> {noformat}
> Starting offset: 0
> offset: 0 position: 0 CreateTime: 1496255947323 isvalid: true keysize: 8 
> valuesize: 8 magic: 2 compresscodec: NONE crc: 600535135 sequence: 0 
> headerKeys: [] key: 1 payload: 0
> offset: 1 position: 0 CreateTime: 1496255947324 isvalid: true keysize: 8 
> valuesize: 8 magic: 2 compresscodec: NONE crc: 600535135 sequence: 1 
> headerKeys: [] key: 1 payload: 1
> offset: 2 position: 0 CreateTime: 1496255947325 isvalid: true keysize: 8 
> valuesize: 8 magic: 2 compresscodec: NONE crc: 600535135 sequence: 2 
> headerKeys: [] key: 1 payload: 3
> offset: 3 position: 0 CreateTime: 1496255947326 isvalid: true keysize: 8 
> valuesize: 8 magic: 2 compresscodec: NONE crc: 600535135 sequence: 3 
> headerKeys: [] key: 1 payload: 6
> offset: 4 position: 0 CreateTime: 1496

[jira] [Commented] (KAFKA-5229) Reflections logs excessive warnings when scanning classpaths

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033657#comment-16033657
 ] 

ASF GitHub Bot commented on KAFKA-5229:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3072


> Reflections logs excessive warnings when scanning classpaths
> 
>
> Key: KAFKA-5229
> URL: https://issues.apache.org/jira/browse/KAFKA-5229
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1, 0.10.2.0, 
> 0.10.2.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Bharat Viswanadham
>  Labels: newbie
> Fix For: 0.11.0.0, 0.11.1.0
>
>
> We use Reflections to scan the classpath for available plugins (connectors, 
> converters, transformations), but when doing so Reflections tends to generate 
> a lot of log noise like this:
> {code}
> [2017-05-12 14:59:48,224] WARN could not get type for name 
> org.jboss.netty.channel.SimpleChannelHandler from any class loader 
> (org.reflections.Reflections:396)
> org.reflections.ReflectionsException: could not get type for name 
> org.jboss.netty.channel.SimpleChannelHandler
>   at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
>   at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
>   at org.reflections.Reflections.(Reflections.java:126)
>   at 
> org.apache.kafka.connect.runtime.PluginDiscovery.scanClasspathForPlugins(PluginDiscovery.java:68)
>   at 
> org.apache.kafka.connect.runtime.AbstractHerder$1.run(AbstractHerder.java:391)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException: 
> org.jboss.netty.channel.SimpleChannelHandler
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>   at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
>   ... 5 more
> {code}
> Despite being benign, these warnings worry users, especially first time users.
> We should either a) see if we can get Reflections to turn off these specific 
> warnings via some config or b) make Reflections only log at > WARN by default 
> in our log4j config. (b) is probably safe since we should only be seeing 
> these at startup and I don't think I've seen any actual issue logged at WARN.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3072: KAFKA-5229:Reflections logs excessive warnings whe...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3072


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-5229) Reflections logs excessive warnings when scanning classpaths

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5229?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava updated KAFKA-5229:
-
   Resolution: Fixed
Fix Version/s: 0.11.1.0
   0.11.0.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 3072
[https://github.com/apache/kafka/pull/3072]

> Reflections logs excessive warnings when scanning classpaths
> 
>
> Key: KAFKA-5229
> URL: https://issues.apache.org/jira/browse/KAFKA-5229
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.0.0, 0.10.0.1, 0.10.1.0, 0.10.1.1, 0.10.2.0, 
> 0.10.2.1
>Reporter: Ewen Cheslack-Postava
>Assignee: Bharat Viswanadham
>  Labels: newbie
> Fix For: 0.11.0.0, 0.11.1.0
>
>
> We use Reflections to scan the classpath for available plugins (connectors, 
> converters, transformations), but when doing so Reflections tends to generate 
> a lot of log noise like this:
> {code}
> [2017-05-12 14:59:48,224] WARN could not get type for name 
> org.jboss.netty.channel.SimpleChannelHandler from any class loader 
> (org.reflections.Reflections:396)
> org.reflections.ReflectionsException: could not get type for name 
> org.jboss.netty.channel.SimpleChannelHandler
>   at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:390)
>   at org.reflections.Reflections.expandSuperTypes(Reflections.java:381)
>   at org.reflections.Reflections.(Reflections.java:126)
>   at 
> org.apache.kafka.connect.runtime.PluginDiscovery.scanClasspathForPlugins(PluginDiscovery.java:68)
>   at 
> org.apache.kafka.connect.runtime.AbstractHerder$1.run(AbstractHerder.java:391)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.ClassNotFoundException: 
> org.jboss.netty.channel.SimpleChannelHandler
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>   at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>   at java.security.AccessController.doPrivileged(Native Method)
>   at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>   at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>   at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>   at org.reflections.ReflectionUtils.forName(ReflectionUtils.java:388)
>   ... 5 more
> {code}
> Despite being benign, these warnings worry users, especially first time users.
> We should either a) see if we can get Reflections to turn off these specific 
> warnings via some config or b) make Reflections only log at > WARN by default 
> in our log4j config. (b) is probably safe since we should only be seeing 
> these at startup and I don't think I've seen any actual issue logged at WARN.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3197: MINOR: Set log level for org.reflections to ERROR.

[kkonstantine]

Github user kkonstantine closed the pull request at:

https://github.com/apache/kafka/pull/3197


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3189: MINOR: add upgrade not for group.initial.rebalance...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3189


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-5363) Add restoreAll functionality to StateRestoreCallback

[Bill Bejeck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Bejeck updated KAFKA-5363:
---
Description: Add a new method {{restoreAll(List> 
records)}} to the {{StateRestoreCallback}} to enable bulk writing to the 
underlying state store vs individual {{restore(byte[] key, byte[] value)}} 
resulting in quicker restore times.  (was: Add a new method 
{{restoreAll(KeyValue records)}} to the 
{{StateRestoreCallback}} to enable bulk writing to the underlying state store 
vs individual {{restore(byte[] key, byte[] value)}} resulting in quicker 
restore times.)

> Add restoreAll functionality to StateRestoreCallback
> 
>
> Key: KAFKA-5363
> URL: https://issues.apache.org/jira/browse/KAFKA-5363
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Bill Bejeck
>Assignee: Bill Bejeck
> Fix For: 0.11.1.0
>
>
> Add a new method {{restoreAll(List> records)}} to 
> the {{StateRestoreCallback}} to enable bulk writing to the underlying state 
> store vs individual {{restore(byte[] key, byte[] value)}} resulting in 
> quicker restore times.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5154) Kafka Streams throws NPE during rebalance

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033636#comment-16033636
 ] 

Guozhang Wang commented on KAFKA-5154:
--

[~Lukas Gemela] We have found the root cause, which is is a bug in the consumer 
client but streams client exposed it. It has been fixed in trunk and will be 
included in the 0.11.0.0 RC0 releasing this week. You can also apply the patch 
above and try it out. Thanks again for reporting this!

> Kafka Streams throws NPE during rebalance
> -
>
> Key: KAFKA-5154
> URL: https://issues.apache.org/jira/browse/KAFKA-5154
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Lukas Gemela
>Assignee: Damian Guy
> Attachments: 5154_problem.log, clio_afa596e9b809.gz, clio_reduced.gz, 
> clio.txt.gz
>
>
> please see attached log, Kafka streams throws NullPointerException during 
> rebalance, which is caught by our custom exception handler
> {noformat}
> 2017-04-30T17:44:17,675 INFO  kafka-coordinator-heartbeat-thread | hades 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead()
>  @618 - Marking the coordinator 10.210.200.144:9092 (id: 2147483644 rack: 
> null) dead for group hades
> 2017-04-30T17:44:27,395 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess() 
> @573 - Discovered coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) 
> for group hades.
> 2017-04-30T17:44:27,941 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare()
>  @393 - Revoking previously assigned partitions [poseidonIncidentFeed-27, 
> poseidonIncidentFeed-29, poseidonIncidentFeed-30, poseidonIncidentFeed-18] 
> for group hades
> 2017-04-30T17:44:27,947 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
>  @407 - (Re-)joining group hades
> 2017-04-30T17:44:48,468 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
>  @407 - (Re-)joining group hades
> 2017-04-30T17:44:53,628 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
>  @407 - (Re-)joining group hades
> 2017-04-30T17:45:09,587 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
>  @407 - (Re-)joining group hades
> 2017-04-30T17:45:11,961 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess() 
> @375 - Successfully joined group hades with generation 99
> 2017-04-30T17:45:13,126 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete()
>  @252 - Setting newly assigned partitions [poseidonIncidentFeed-11, 
> poseidonIncidentFeed-27, poseidonIncidentFeed-25, poseidonIncidentFeed-29, 
> poseidonIncidentFeed-19, poseidonIncidentFeed-18] for group hades
> 2017-04-30T17:46:37,254 INFO  kafka-coordinator-heartbeat-thread | hades 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead()
>  @618 - Marking the coordinator 10.210.200.144:9092 (id: 2147483644 rack: 
> null) dead for group hades
> 2017-04-30T18:04:25,993 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess() 
> @573 - Discovered coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) 
> for group hades.
> 2017-04-30T18:04:29,401 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare()
>  @393 - Revoking previously assigned partitions [poseidonIncidentFeed-11, 
> poseidonIncidentFeed-27, poseidonIncidentFeed-25, poseidonIncidentFeed-29, 
> poseidonIncidentFeed-19, poseidonIncidentFeed-18] for group hades
> 2017-04-30T18:05:10,877 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
>  @407 - (Re-)joining group hades
> 2017-05-01T00:01:55,707 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead()
>  @618 - Marking the coordinator 10.210.200.144:9092 (id: 2147483644 rack: 
> null) dead for group hades
> 2017-05-01T00:01:59,027 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess() 
> @573 - Discovered coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) 
> for group hades.
> 2017-05-01T00:01:59,031 ERROR StreamThread-1 
> org.apache.kafka.streams.processor.internals.StreamThread.run() @376 - 
> stream-thread [StreamThread-1] Streams application error during processing:
>  java.lang.NullPointerException
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop

[jira] [Commented] (KAFKA-5154) Kafka Streams throws NPE during rebalance

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033633#comment-16033633
 ] 

ASF GitHub Bot commented on KAFKA-5154:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3181


> Kafka Streams throws NPE during rebalance
> -
>
> Key: KAFKA-5154
> URL: https://issues.apache.org/jira/browse/KAFKA-5154
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Lukas Gemela
>Assignee: Damian Guy
> Attachments: 5154_problem.log, clio_afa596e9b809.gz, clio_reduced.gz, 
> clio.txt.gz
>
>
> please see attached log, Kafka streams throws NullPointerException during 
> rebalance, which is caught by our custom exception handler
> {noformat}
> 2017-04-30T17:44:17,675 INFO  kafka-coordinator-heartbeat-thread | hades 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead()
>  @618 - Marking the coordinator 10.210.200.144:9092 (id: 2147483644 rack: 
> null) dead for group hades
> 2017-04-30T17:44:27,395 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess() 
> @573 - Discovered coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) 
> for group hades.
> 2017-04-30T17:44:27,941 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare()
>  @393 - Revoking previously assigned partitions [poseidonIncidentFeed-27, 
> poseidonIncidentFeed-29, poseidonIncidentFeed-30, poseidonIncidentFeed-18] 
> for group hades
> 2017-04-30T17:44:27,947 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
>  @407 - (Re-)joining group hades
> 2017-04-30T17:44:48,468 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
>  @407 - (Re-)joining group hades
> 2017-04-30T17:44:53,628 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
>  @407 - (Re-)joining group hades
> 2017-04-30T17:45:09,587 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
>  @407 - (Re-)joining group hades
> 2017-04-30T17:45:11,961 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess() 
> @375 - Successfully joined group hades with generation 99
> 2017-04-30T17:45:13,126 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete()
>  @252 - Setting newly assigned partitions [poseidonIncidentFeed-11, 
> poseidonIncidentFeed-27, poseidonIncidentFeed-25, poseidonIncidentFeed-29, 
> poseidonIncidentFeed-19, poseidonIncidentFeed-18] for group hades
> 2017-04-30T17:46:37,254 INFO  kafka-coordinator-heartbeat-thread | hades 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead()
>  @618 - Marking the coordinator 10.210.200.144:9092 (id: 2147483644 rack: 
> null) dead for group hades
> 2017-04-30T18:04:25,993 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess() 
> @573 - Discovered coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) 
> for group hades.
> 2017-04-30T18:04:29,401 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinPrepare()
>  @393 - Revoking previously assigned partitions [poseidonIncidentFeed-11, 
> poseidonIncidentFeed-27, poseidonIncidentFeed-25, poseidonIncidentFeed-29, 
> poseidonIncidentFeed-19, poseidonIncidentFeed-18] for group hades
> 2017-04-30T18:05:10,877 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.sendJoinGroupRequest()
>  @407 - (Re-)joining group hades
> 2017-05-01T00:01:55,707 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.coordinatorDead()
>  @618 - Marking the coordinator 10.210.200.144:9092 (id: 2147483644 rack: 
> null) dead for group hades
> 2017-05-01T00:01:59,027 INFO  StreamThread-1 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.onSuccess() 
> @573 - Discovered coordinator 10.210.200.144:9092 (id: 2147483644 rack: null) 
> for group hades.
> 2017-05-01T00:01:59,031 ERROR StreamThread-1 
> org.apache.kafka.streams.processor.internals.StreamThread.run() @376 - 
> stream-thread [StreamThread-1] Streams application error during processing:
>  java.lang.NullPointerException
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:619)
>  ~[kafka-streams-0.10.2.0.jar!/:?]
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.run(StreamThread.java:368)
>  [kafka-streams-0.10.2.0.jar!/:?]
>

[GitHub] kafka pull request #3197: MINOR: Set log level for org.reflections to ERROR.

[kkonstantine]

GitHub user kkonstantine opened a pull request:

https://github.com/apache/kafka/pull/3197

MINOR: Set log level for org.reflections to ERROR.



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/kkonstantine/kafka 
MINOR-Set-log-level-for-org-reflections-to-error

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3197.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3197


commit 24ec8bbea3f800de3e264dfdd5734c9b699ed25a
Author: Konstantine Karantasis 
Date:   2017-06-01T20:32:48Z

MINOR: Set log level for org.reflections to ERROR.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #3181: KAFKA-5154: Consumer fetches from revoked partitio...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3181


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-5363) Add restoreAll functionality to StateRestoreCallback

[Bill Bejeck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Bejeck updated KAFKA-5363:
---
Description: Add a new method {{restoreAll(KeyValue 
records)}} to the {{StateRestoreCallback}} to enable bulk writing to the 
underlying state store vs individual {{restore(byte[] key, byte[] value)}} 
resulting in quicker restore times.  (was: Add a new method 
{{restoreAll(KeyValue records)}} to the 
{{StateRestoreCallback}} to enable bulk writing to the underlying state store 
vs individual {{put(byte[] key, byte[] value)}} resulting in quicker restore 
times.)

> Add restoreAll functionality to StateRestoreCallback
> 
>
> Key: KAFKA-5363
> URL: https://issues.apache.org/jira/browse/KAFKA-5363
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Bill Bejeck
>Assignee: Bill Bejeck
> Fix For: 0.11.1.0
>
>
> Add a new method {{restoreAll(KeyValue records)}} to the 
> {{StateRestoreCallback}} to enable bulk writing to the underlying state store 
> vs individual {{restore(byte[] key, byte[] value)}} resulting in quicker 
> restore times.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5293) Do not apply exponential backoff if users have overridden reconnect.backoff.ms

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033631#comment-16033631
 ] 

ASF GitHub Bot commented on KAFKA-5293:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3174


> Do not apply exponential backoff if users have overridden reconnect.backoff.ms
> --
>
> Key: KAFKA-5293
> URL: https://issues.apache.org/jira/browse/KAFKA-5293
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
>Assignee: Colin P. McCabe
>Priority: Critical
> Fix For: 0.11.0.0
>
>
> The PR for KAFKA-3878 implemented KIP-144 with one exception: it 
> automatically enables exponential backoff for the producer and consumer even 
> if reconnect.backoff.ms is set by users. The KIP stated that this would not 
> be the case.
> As part of this JIRA, we should also add a few unit tests for connectionDelay 
> and perhaps consider enabling exponential backoff for Connect and Streams as 
> well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3174: KAFKA-5293. Do not apply exponential backoff if us...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3174


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-5293) Do not apply exponential backoff if users have overridden reconnect.backoff.ms

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5293:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 3174
[https://github.com/apache/kafka/pull/3174]

> Do not apply exponential backoff if users have overridden reconnect.backoff.ms
> --
>
> Key: KAFKA-5293
> URL: https://issues.apache.org/jira/browse/KAFKA-5293
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
>Assignee: Colin P. McCabe
>Priority: Critical
> Fix For: 0.11.0.0
>
>
> The PR for KAFKA-3878 implemented KIP-144 with one exception: it 
> automatically enables exponential backoff for the producer and consumer even 
> if reconnect.backoff.ms is set by users. The KIP stated that this would not 
> be the case.
> As part of this JIRA, we should also add a few unit tests for connectionDelay 
> and perhaps consider enabling exponential backoff for Connect and Streams as 
> well.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5361) Add EOS integration tests for Streams API

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033613#comment-16033613
 ] 

ASF GitHub Bot commented on KAFKA-5361:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3193


> Add EOS integration tests for Streams API
> -
>
> Key: KAFKA-5361
> URL: https://issues.apache.org/jira/browse/KAFKA-5361
> Project: Kafka
>  Issue Type: Bug
>  Components: streams, unit tests
>Affects Versions: 0.11.0.0
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Blocker
>  Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> We need to add more integration tests for Streams API with exactly-once 
> enabled.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (KAFKA-5361) Add EOS integration tests for Streams API

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-5361.
--
Resolution: Fixed

Issue resolved by pull request 3193
[https://github.com/apache/kafka/pull/3193]

> Add EOS integration tests for Streams API
> -
>
> Key: KAFKA-5361
> URL: https://issues.apache.org/jira/browse/KAFKA-5361
> Project: Kafka
>  Issue Type: Bug
>  Components: streams, unit tests
>Affects Versions: 0.11.0.0
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
>Priority: Blocker
>  Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> We need to add more integration tests for Streams API with exactly-once 
> enabled.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3193: KAFKA-5361: Add EOS integration tests for Streams ...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/3193


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-5363) Add restoreAll functionality to StateRestoreCallback

[Bill Bejeck (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5363?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bill Bejeck updated KAFKA-5363:
---
Description: Add a new method {{restoreAll(KeyValue 
records)}} to the {{StateRestoreCallback}} to enable bulk writing to the 
underlying state store vs individual {{put(byte[] key, byte[] value)}} 
resulting in quicker restore times.  (was: Add a new method 
`restoreAll(KeyValue records)` to the `StateRestoreCallback` to 
enable bulk writing to the underlying state store vs individual `put(byte[] 
key, byte[] value)` resulting in quicker restore times.)

> Add restoreAll functionality to StateRestoreCallback
> 
>
> Key: KAFKA-5363
> URL: https://issues.apache.org/jira/browse/KAFKA-5363
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Bill Bejeck
>Assignee: Bill Bejeck
> Fix For: 0.11.1.0
>
>
> Add a new method {{restoreAll(KeyValue records)}} to the 
> {{StateRestoreCallback}} to enable bulk writing to the underlying state store 
> vs individual {{put(byte[] key, byte[] value)}} resulting in quicker restore 
> times.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5363) Add restoreAll functionality to StateRestoreCallback

[Bill Bejeck (JIRA)]

Bill Bejeck created KAFKA-5363:
--

 Summary: Add restoreAll functionality to StateRestoreCallback
 Key: KAFKA-5363
 URL: https://issues.apache.org/jira/browse/KAFKA-5363
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Reporter: Bill Bejeck
Assignee: Bill Bejeck
 Fix For: 0.11.1.0


Add a new method `restoreAll(KeyValue records)` to the 
`StateRestoreCallback` to enable bulk writing to the underlying state store vs 
individual `put(byte[] key, byte[] value)` resulting in quicker restore times.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5330) Use per-task converters in Connect

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033578#comment-16033578
 ] 

ASF GitHub Bot commented on KAFKA-5330:
---

GitHub user tbcdns opened a pull request:

https://github.com/apache/kafka/pull/3196

KAFKA-5330: Use per-task converters

Instead of sharing the same converter instance within the worker, use a 
converter per task.

More details:
- https://github.com/confluentinc/schema-registry/issues/514
- https://issues.apache.org/jira/browse/KAFKA-5330

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tbcdns/kafka KAFKA-5330

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3196.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3196


commit 43f38a5a4e0a7eeff233afe90a7893f252c6edac
Author: Thibaud Chardonnens 
Date:   2017-06-01T19:38:16Z

KAFKA-5330: Use per-task converters




> Use per-task converters in Connect
> --
>
> Key: KAFKA-5330
> URL: https://issues.apache.org/jira/browse/KAFKA-5330
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Affects Versions: 0.11.0.0
>Reporter: Ewen Cheslack-Postava
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Because Connect started with a worker-wide model of data formats, we 
> currently allocate a single Converter per worker and only allocate an 
> independent one when the user overrides the converter.
> This can lead to performance problems when the worker-level default converter 
> is used by a large number of tasks because converters need to be threadsafe 
> to support this model and they may spend a lot of time just on 
> synchronization.
> We could, instead, simply allocate one converter per task. There is some 
> overhead involved, but generally it shouldn't be that large. For example, 
> Confluent's Avro converters will each have their own schema cache and have to 
> make their on calls to the schema registry API, but these are relatively 
> small, likely inconsequential compared to any normal overhead we would 
> already have for creating and managing each task. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #3196: KAFKA-5330: Use per-task converters

[tbcdns]

GitHub user tbcdns opened a pull request:

https://github.com/apache/kafka/pull/3196

KAFKA-5330: Use per-task converters

Instead of sharing the same converter instance within the worker, use a 
converter per task.

More details:
- https://github.com/confluentinc/schema-registry/issues/514
- https://issues.apache.org/jira/browse/KAFKA-5330

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/tbcdns/kafka KAFKA-5330

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/3196.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #3196


commit 43f38a5a4e0a7eeff233afe90a7893f252c6edac
Author: Thibaud Chardonnens 
Date:   2017-06-01T19:38:16Z

KAFKA-5330: Use per-task converters




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-5204) Connect needs to validate Connector type during instantiation

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5204?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5204:
---
Fix Version/s: (was: 0.11.0.0)
   0.11.1.0

> Connect needs to validate Connector type during instantiation
> -
>
> Key: KAFKA-5204
> URL: https://issues.apache.org/jira/browse/KAFKA-5204
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.2.0
>Reporter: Konstantine Karantasis
>Assignee: Konstantine Karantasis
> Fix For: 0.11.1.0
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> Currently Connect will accept to instantiate connectors that extend the 
> {{Connector}} abstract class but not one of its subclasses, 
> {{SourceConnector}} or {{SinkConnector}}. 
> However, in distributed mode as well as in REST, Connect assumes in a few 
> places that there are only two types of connectors, sinks or sources. Based 
> on this assumption it checks the type dynamically, and if it is not a sink it 
> treats it as a source (by constructing the corresponding configs). 
> A connector that implements only the {{Connector}} abstract class does not 
> fit into this classification. Therefore a validation needs to take place 
> early, during the instantiation of the {{Connector}} object. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (KAFKA-5054) ChangeLoggingKeyValueByteStore delete and putIfAbsent should be synchronized

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033577#comment-16033577
 ] 

Ismael Juma edited comment on KAFKA-5054 at 6/1/17 8:00 PM:


This seems important and simple, [~guozhang], can you please check?


was (Author: ijuma):
This seems important and simple, @guozhangwang, can you please check?

> ChangeLoggingKeyValueByteStore delete and putIfAbsent should be synchronized
> 
>
> Key: KAFKA-5054
> URL: https://issues.apache.org/jira/browse/KAFKA-5054
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 0.11.0.0
>
>
> {{putIfAbsent}} and {{delete}} should be synchronized as they involve at 
> least 2 operations on the underlying store and may result in inconsistent 
> results if someone were to query via IQ



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5054) ChangeLoggingKeyValueByteStore delete and putIfAbsent should be synchronized

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5054?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5054:
---
Priority: Critical  (was: Major)

> ChangeLoggingKeyValueByteStore delete and putIfAbsent should be synchronized
> 
>
> Key: KAFKA-5054
> URL: https://issues.apache.org/jira/browse/KAFKA-5054
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Damian Guy
>Assignee: Damian Guy
>Priority: Critical
> Fix For: 0.11.0.0
>
>
> {{putIfAbsent}} and {{delete}} should be synchronized as they involve at 
> least 2 operations on the underlying store and may result in inconsistent 
> results if someone were to query via IQ



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4928) Add integration test for DumpLogSegments

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4928?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4928:
---
Fix Version/s: (was: 0.11.0.0)
   0.11.1.0

> Add integration test for DumpLogSegments
> 
>
> Key: KAFKA-4928
> URL: https://issues.apache.org/jira/browse/KAFKA-4928
> Project: Kafka
>  Issue Type: Test
>Reporter: Ismael Juma
>Assignee: Armin Braun
>  Labels: newbie
> Fix For: 0.11.1.0
>
>
> DumpLogSegments is an important tool to analyse log files, but we have no 
> JUnit tests for it. It would be good to have some tests that verify that the 
> output is sane for a populated log.
> Our system tests call DumpLogSegments, but we should be able to detect 
> regressions via the JUnit test suite.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5018) LogCleaner tests to verify behaviour of message format v2

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5018:
---
Fix Version/s: (was: 0.11.0.0)
   0.11.1.0

> LogCleaner tests to verify behaviour of message format v2
> -
>
> Key: KAFKA-5018
> URL: https://issues.apache.org/jira/browse/KAFKA-5018
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Ismael Juma
>Assignee: Armin Braun
> Fix For: 0.11.1.0
>
>
> It would be good to add LogCleaner tests to verify the behaviour of fields 
> like baseOffset after compaction.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4850) RocksDb cannot use Bloom Filters

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4850:
---
Fix Version/s: (was: 0.11.0.0)
   0.11.1.0

> RocksDb cannot use Bloom Filters
> 
>
> Key: KAFKA-4850
> URL: https://issues.apache.org/jira/browse/KAFKA-4850
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Eno Thereska
>Assignee: Bharat Viswanadham
> Fix For: 0.11.1.0
>
>
> Bloom Filters would speed up RocksDb lookups. However they currently do not 
> work in RocksDb 5.0.2. This has been fixed in trunk, but we'll have to wait 
> until that is released and tested. 
> Then we can add the line in RocksDbStore.java in openDb:
> tableConfig.setFilter(new BloomFilter(10));



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5054) ChangeLoggingKeyValueByteStore delete and putIfAbsent should be synchronized

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5054?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033577#comment-16033577
 ] 

Ismael Juma commented on KAFKA-5054:


This seems important and simple, @guozhangwang, can you please check?

> ChangeLoggingKeyValueByteStore delete and putIfAbsent should be synchronized
> 
>
> Key: KAFKA-5054
> URL: https://issues.apache.org/jira/browse/KAFKA-5054
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 0.11.0.0
>
>
> {{putIfAbsent}} and {{delete}} should be synchronized as they involve at 
> least 2 operations on the underlying store and may result in inconsistent 
> results if someone were to query via IQ



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4785) Records from internal repartitioning topics should always use RecordMetadataTimestampExtractor

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4785:
---
Fix Version/s: (was: 0.11.0.0)
   0.11.1.0

> Records from internal repartitioning topics should always use 
> RecordMetadataTimestampExtractor
> --
>
> Key: KAFKA-4785
> URL: https://issues.apache.org/jira/browse/KAFKA-4785
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Matthias J. Sax
>Assignee: Jeyhun Karimov
> Fix For: 0.11.1.0
>
>
> Users can specify what timestamp extractor should be used to decode the 
> timestamp of input topic records. As long as RecordMetadataTimestamp or 
> WallclockTime is use this is fine. 
> However, for custom timestamp extractors it might be invalid to apply this 
> custom extractor to records received from internal repartitioning topics. The 
> reason is that Streams sets the current "stream time" as record metadata 
> timestamp explicitly before writing to intermediate repartitioning topics 
> because this timestamp should be use by downstream subtopologies. A custom 
> timestamp extractor might return something different breaking this assumption.
> Thus, for reading data from intermediate repartitioning topic, the configured 
> timestamp extractor should not be used, but the record's metadata timestamp 
> should be extracted as record timestamp.
> In order to leverage the same behavior for intermediate user topic (ie, used 
> in {{through()}})  we can leverage KAFKA-4144 and internally set an extractor 
> for those "intermediate sources" that returns the record's metadata timestamp 
> in order to overwrite the global extractor from {{StreamsConfig}} (ie, set 
> {{FailOnInvalidTimestampExtractor}}).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-4665) Inconsistent handling of non-existing topics in offset fetch handling

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-4665:
---
Fix Version/s: (was: 0.11.0.0)
   0.11.1.0

> Inconsistent handling of non-existing topics in offset fetch handling
> -
>
> Key: KAFKA-4665
> URL: https://issues.apache.org/jira/browse/KAFKA-4665
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer, core
>Reporter: Jason Gustafson
>Assignee: Vahid Hashemian
> Fix For: 0.11.1.0
>
>
> For version 0 of the offset fetch API, the broker returns 
> UNKNOWN_TOPIC_OR_PARTITION for any topics/partitions which do not exist at 
> the time of fetching. In later versions, we skip this check. We do, however, 
> continue to return UNKNOWN_TOPIC_OR_PARTITION for authorization errors (i.e. 
> if the principal does not have Describe access to the corresponding topic). 
> We should probably make this behavior consistent across versions.
> Note also that currently the consumer raises {{KafkaException}} when it 
> encounters an UNKNOWN_TOPIC_OR_PARTITION error in the offset fetch response, 
> which is inconsistent with how we usually handle this error. This probably 
> doesn't cause any problems currently only because of the inconsistency 
> mentioned in the first paragraph above.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)