[GitHub] kafka pull request #2952: MINOR: fix some c/p error that was causing describ...

[norwood]

GitHub user norwood opened a pull request:

https://github.com/apache/kafka/pull/2952

MINOR: fix some c/p error that was causing describe to delete

@ijuma @cmccabe 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/norwood/kafka describe-not-delete

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2952.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2952


commit 916dbb1eac9f3d89cb1f94302bc8fb911f3eba72
Author: dan norwood 
Date:   2017-05-02T06:43:04Z

fix some c/p error that was causing describe to delete




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2953: handle 0 futures in all()

[norwood]

GitHub user norwood opened a pull request:

https://github.com/apache/kafka/pull/2953

handle 0 futures in all()

if we pass in 0 futures to an AllOfAdapter, we should complete immediately

@ijuma @cmccabe 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/norwood/kafka handle-all-of-0

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2953.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2953


commit 5ca7e69ec75569c44226bb8d9863d7cd88714523
Author: dan norwood 
Date:   2017-05-02T06:44:25Z

handle 0 futures in all()

if we pass in 0 futures to an AllOfAdapter, we should complete immediately




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-5145) Remove task close() call from closeNonAssignedSuspendedTasks method

[Narendra Kumar (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Narendra Kumar updated KAFKA-5145:
--
Reviewer: Guozhang Wang

> Remove task close() call from closeNonAssignedSuspendedTasks method
> ---
>
> Key: KAFKA-5145
> URL: https://issues.apache.org/jira/browse/KAFKA-5145
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.0
>Reporter: Narendra Kumar
>  Labels: newbie
> Attachments: BugTest.java, DebugTransformer.java, logs.txt
>
>
> While rebalancing ProcessorNode.close() can be called twice, once  from 
> StreamThread.suspendTasksAndState() and once from  
> StreamThread.closeNonAssignedSuspendedTasks(). If ProcessorNode.close() 
> throws some exception because of calling close() multiple times( i.e. 
> IllegalStateException from  some KafkaConsumer instance being used by some 
> processor for some lookup), we fail to close the task's state manager ( i.e. 
> call to task.closeStateManager(true); fails).  After rebalance, if the same 
> task id is launched on same application instance but in different thread then 
> the task get stuck because it fails to get lock to the task's state directory.
> Since processor close() is already called from 
> StreamThread.suspendTasksAndState() we don't need to call again from 
> StreamThread.closeNonAssignedSuspendedTasks().



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5152) Kafka Streams keeps restoring state after shutdown is initiated during startup

[JIRA]

Xavier Léauté created KAFKA-5152:


 Summary: Kafka Streams keeps restoring state after shutdown is 
initiated during startup
 Key: KAFKA-5152
 URL: https://issues.apache.org/jira/browse/KAFKA-5152
 Project: Kafka
  Issue Type: Bug
  Components: streams
Affects Versions: 0.10.2.1
Reporter: Xavier Léauté


If streams shutdown is initiated during state restore (e.g. an uncaught 
exception is thrown) streams will not shut down until all stores are first 
finished restoring.

As restore progresses, stream threads appear to be taken out of service as part 
of the shutdown sequence, causing rebalancing of tasks. This compounds the 
problem by slowing down the restore process even further, since the remaining 
threads now have to also restore the reassigned tasks before they can shut down.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-146: Classloading Isolation in Connect

[Konstantine Karantasis]

Gwen, Randall thank you for your very insightful observations. I'm glad you
find this first draft to be an adequate platform for discussion.

I'll attempt replying to your comments in order.

Gwen, I also debated exactly the same two options: a) interpreting absence
of module path as a user's intention to turn off isolation and b)
explicitly using an additional boolean property. A few reasons why I went
with b) in this first draft are:
1) As Randall mentions, to leave the option of using a default value open.
If not immediately in the first version of isolation, maybe in the future.
2) I didn't like the implicit character of the choice of interpreting an
empty string as a clear intention to turn isolation off by the user. Half
the time could be just that users forget to set a location, although they'd
like to use class loading isolation.
3) There's a slim possibility that in rare occasions a user might want to
avoid even the slightest increase in memory consumption due to class
loading duplication. I admit this should be very rare, but given the other
concerns and that we would really like to keep the isolation implementation
simple, the option to turn off this feature by using only one additional
config property might not seem too excessive. At least at the start of this
discussion.
4) Debugging during development might be simpler in some cases.
5) Finally, as you mention, this could allow for smoother upgrades.

Randall, regarding your comments:
1) To keep its focus narrow, this KIP, as well as the first implementation
of isolation in Connect, assume filesystem based discovery. With careful
implementation, transitioning to discovery schemes that support broader
URIs I believe should be easy in the future.
2) The example you give makes a good point. However I'm inclined to say
that such cases should be addressed more as exceptions rather than as being
the common case. Therefore, I wouldn't see all dependencies imported by the
framework as required to be filtered out, because in that case we lose the
advantage of isolation between the framework and the connectors (and we are
left only with isolation between connectors).
3) I tried to abstract implementation details in this the KIP, but you are
right. Even though filtering here is mainly used semantically rather than
literally, it gives an implementation hint that we could avoid.
4) In the same spirit as in 3) I believe we should reserve enough
flexibility to the implementation to discover and load classes, when they
appear in multiple locations under the general module location.

Thanks again! Let me know what you think.
Konstantine


On Mon, May 1, 2017 at 10:12 AM, Randall Hauch  wrote:

> Very nice work, Konstantine. Conflicting dependencies of connectors is
> indeed a big issue that makes it hard to manage installed connectors.
>
> I do like Gwen's idea about removing the 'module.isolation.enabled'
> property. However, I would have anticipated always using classpath
> isolation for *only* those components registered under the module path and
> not really for anything else already on the normal classpath. So, people
> could continue to place custom connector JARs onto the classpath, though
> this would become deprecated in favor of installing custom connector JARs /
> modules via the module path. This keeps configuration simple, gives people
> time to migrate, but let's people that need classpath isolation get it to
> install a variety of connectors each with their dependencies that
> potentially conflict with other components.
>
> The challenge is whether there should be a default for 'module.path'.
> Ideally there would be so that users know where they can install their
> connectors. However, I suspect that this might be difficult to do unless it
> can make use of system properties such as "${kafka.home}" so that relative
> directories can be specified.
>
> A few other questions/comments:
>
> 1) Does the KIP have to specify how are components / modules installed,
> discovered, or recognized by Kafka Connect? Or perhaps the KIP needs to
> just specify the semantics of the file system module path (e.g., the
> directories below those specified in the module path are to be unique and
> identify an installed component).
>
> 2) Will the module classloader filtering also have to exclude Kafka Connect
> dependencies? The only one that I can think of is the SLF4J API, which
> can't be loaded from the module's classloader if the connector is to send
> its log messages to the same logging system.
>
> 3) Rather than specify filtering, would be it a bit more flexible to simply
> say that the implementation will need to ensure that Java, Kafka Connect,
> and other third party APIs (e.g., SLF4J API) will not be loaded from the
> module classloaders? It'd be better to avoid specifying how it will be
> done, just in case the implementation needs to evolve or use a different
> technique (e.g., load the Java and public Kafka Connect APIs via one
> classloader that is re

[jira] [Commented] (KAFKA-4772) Exploit #peek to implement #print() and other methods

[james chien (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15992298#comment-15992298
 ] 

james chien commented on KAFKA-4772:


okay, times to resolve this :) :)

> Exploit #peek to implement #print() and other methods
> -
>
> Key: KAFKA-4772
> URL: https://issues.apache.org/jira/browse/KAFKA-4772
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: james chien
>Priority: Minor
>  Labels: beginner, newbie
>
> From: https://github.com/apache/kafka/pull/2493#pullrequestreview-22157555
> Things that I can think of:
> - print / writeAsTest can be a special impl of peek; KStreamPrint etc can be 
> removed.
> - consider collapse KStreamPeek with KStreamForeach with a flag parameter 
> indicating if the acted key-value pair should still be forwarded.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4772) Exploit #peek to implement #print() and other methods

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15992286#comment-15992286
 ] 

Matthias J. Sax commented on KAFKA-4772:


Please open a new PR. :)

> Exploit #peek to implement #print() and other methods
> -
>
> Key: KAFKA-4772
> URL: https://issues.apache.org/jira/browse/KAFKA-4772
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: james chien
>Priority: Minor
>  Labels: beginner, newbie
>
> From: https://github.com/apache/kafka/pull/2493#pullrequestreview-22157555
> Things that I can think of:
> - print / writeAsTest can be a special impl of peek; KStreamPrint etc can be 
> removed.
> - consider collapse KStreamPeek with KStreamForeach with a flag parameter 
> indicating if the acted key-value pair should still be forwarded.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: kafka-trunk-jdk8 #1468

[Apache Jenkins Server]

See 


Changes:

[wangguoz] KAFKA-5059 Follow-up: Remove broken locking and fix 
handleAddPartitions

[ismael] MINOR: Some cleanups in the transactional producer

[wangguoz] KAFKA-5136: move coordinatorEpoch from WriteTxnMarkerRequest to

[wangguoz] HOTFIX: fix unit tests for KAFKA-5136

--
[...truncated 841.47 KB...]

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete STARTED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffse

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

[Jay Kreps]

Hey Matthias,

Yeah, I think we mostly agree. I think if we think the new way is a lot
better, then I agree keeping it deprecated for a long while then removing
is a viable path. To me this case just seemed like a minor thing so the 1+
years of having a deprecated way in common use and a new way and the
resulting confusion just seems unnecessary, especially since this is so
much in the common path of the most basic kafka usage. My experience with
other apis like MapReduce or Google Collections that did this was pretty
negative.

Big time agreement that doing both is not the right thing to do. People
aren't looking for 4 different ways of creating a ProducerRecord, that just
generates confusion.

-Jay

On Mon, May 1, 2017 at 10:34 AM, Matthias J. Sax 
wrote:

> Hi,
>
> I am personally not a big fan of providing two APIs to do the same
> thing. If we believe that one API is better than the other, we should
> indicate this by deprecating the old API IMHO.
>
> Just my two cents.
>
>
> -Matthias
>
>
> On 4/30/17 11:05 PM, Michael Pearce wrote:
> > See
> >
> > https://hc.apache.org/httpcomponents-client-ga/tutorial/html/fluent.html
> >
> > Doesn't cause much issue over there where you have a fluent api wrapper
> for those who like that style, and the original more verbose api.
> >
> > Surely it would be better than removing a way of doing things that
> everyone has got used to and built their code around ala constructors
> approach. And simply provide a wrapper to provide a per field way of doing
> things.
> >
> >
> >
> >
> >
> > Sent using OWA for iPhone
> > 
> > From: Stephane Maarek 
> > Sent: Monday, May 1, 2017 6:37:44 AM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface
> >
> > I’m not sure how people would feel about having two distinct methods to
> build the same object?
> > An API wrapper may be useful, but it doesn’t bring opinion about how one
> should program, that’s just driven by the docs.
> > I’m okay with that, but we need concensus
> >
> >
> > On 1/5/17, 6:08 am, "Michael Pearce"  wrote:
> >
> > Why not, instead of deprecating or removing whats there, as noted,
> its a point of preference, think about something that could wrap the
> existing, but provide an api that for you is cleaner?
> >
> > e.g. here's a sample idea building on a fluent api way. (this wraps
> the producer and producer records so no changes needed)
> >
> > https://gist.github.com/michaelandrepearce/
> de0f5ad4aa7d39d243781741c58c293e
> >
> > In future as new items further add to Producer Record, they just
> become new methods in the fluent API, as it builds the ProducerRecord using
> the most exhaustive constructor.
> >
> >
> >
> > 
> > From: Matthias J. Sax 
> > Sent: Saturday, April 29, 2017 6:52 PM
> > To: dev@kafka.apache.org
> > Subject: Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface
> >
> > I understand that we cannot just break stuff (btw: also not for
> > Streams!). But deprecating does not break anything, so I don't think
> > it's a big deal to change the API as long as we keep the old API as
> > deprecated.
> >
> >
> > -Matthias
> >
> > On 4/29/17 9:28 AM, Jay Kreps wrote:
> > > Hey Matthias,
> > >
> > > Yeah I agree, I'm not against change as a general thing! I also
> think if
> > > you look back on the last two years, we completely rewrote the
> producer and
> > > consumer APIs, reworked the binary protocol many times over, and
> added the
> > > connector and stream processing apis, both major new additions. So
> I don't
> > > think we're in too much danger of stagnating!
> > >
> > > My two cents was just around breaking compatibility for trivial
> changes
> > > like constructor => builder. I think this only applies to the
> producer,
> > > consumer, and connect apis which are heavily embedded in hundreds
> of
> > > ecosystem components that depend on them. This is different from
> direct
> > > usage. If we break the streams api it is really no big deal---apps
> just
> > > need to rebuild when they upgrade, not the end of the world at
> all. However
> > > because many intermediate things depend on the Kafka producer you
> can cause
> > > these weird situations where your app depends on two third party
> things
> > > that use Kafka and each requires different, incompatible versions.
> We did
> > > this a lot in earlier versions of Kafka and it was the cause of
> much angst
> > > (and an ingrained general reluctance to upgrade) from our users.
> > >
> > > I still think we may have to break things, i just don't think we
> should do
> > > it for things like builders vs direct constructors which i think
> are kind
> > > of a debatable matter of taste.
> > >
> > > -Jay
> > >
> > >
> > >
> > > On Mon, Apr 24, 2017 

[jira] [Assigned] (KAFKA-5055) Kafka Streams skipped-records-rate sensor producing nonzero values even when FailOnInvalidTimestamp is used as extractor

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang reassigned KAFKA-5055:


Assignee: Davor Poldrugo  (was: Guozhang Wang)

> Kafka Streams skipped-records-rate sensor producing nonzero values even when 
> FailOnInvalidTimestamp is used as extractor
> 
>
> Key: KAFKA-5055
> URL: https://issues.apache.org/jira/browse/KAFKA-5055
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Nikki Thean
>Assignee: Davor Poldrugo
>
> According to the code and the documentation for this metric, the only reason 
> for a skipped record is an invalid timestamp, except that a) I am reading 
> from a topic that is populated solely by Kafka Connect and b) I am using 
> `FailOnInvalidTimestamp` as the timestamp extractor.
> Either I'm missing something in the documentation (i.e. another reason for 
> skipped records) or there is a bug in the code that calculates this metric.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5055) Kafka Streams skipped-records-rate sensor producing nonzero values even when FailOnInvalidTimestamp is used as extractor

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991978#comment-15991978
 ] 

Guozhang Wang commented on KAFKA-5055:
--

[~dpoldrugo] Thanks! Added.

> Kafka Streams skipped-records-rate sensor producing nonzero values even when 
> FailOnInvalidTimestamp is used as extractor
> 
>
> Key: KAFKA-5055
> URL: https://issues.apache.org/jira/browse/KAFKA-5055
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Nikki Thean
>Assignee: Guozhang Wang
>
> According to the code and the documentation for this metric, the only reason 
> for a skipped record is an invalid timestamp, except that a) I am reading 
> from a topic that is populated solely by Kafka Connect and b) I am using 
> `FailOnInvalidTimestamp` as the timestamp extractor.
> Either I'm missing something in the documentation (i.e. another reason for 
> skipped records) or there is a bug in the code that calculates this metric.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (KAFKA-4772) Exploit #peek to implement #print() and other methods

[james chien (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991968#comment-15991968
 ] 

james chien edited comment on KAFKA-4772 at 5/2/17 1:54 AM:


Great !! 
I am a newbie, what should I do now? (keep going this comment 
https://github.com/apache/kafka/pull/2704#issuecomment-294661203 or make a new 
PR)
(I am very excited now :) )


was (Author: james.c):
Great !! 
what should I do now? (keep going this comment 
https://github.com/apache/kafka/pull/2704#issuecomment-294661203 or make a new 
PR)
(I am very excited now :) )

> Exploit #peek to implement #print() and other methods
> -
>
> Key: KAFKA-4772
> URL: https://issues.apache.org/jira/browse/KAFKA-4772
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: james chien
>Priority: Minor
>  Labels: beginner, newbie
>
> From: https://github.com/apache/kafka/pull/2493#pullrequestreview-22157555
> Things that I can think of:
> - print / writeAsTest can be a special impl of peek; KStreamPrint etc can be 
> removed.
> - consider collapse KStreamPeek with KStreamForeach with a flag parameter 
> indicating if the acted key-value pair should still be forwarded.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : kafka-trunk-jdk7 #2134

[Apache Jenkins Server]

See 

[jira] [Commented] (KAFKA-4772) Exploit #peek to implement #print() and other methods

[james chien (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991968#comment-15991968
 ] 

james chien commented on KAFKA-4772:


Great !! 
what should I do now? (keep going this comment 
https://github.com/apache/kafka/pull/2704#issuecomment-294661203 or make a new 
PR)
(I am very excited now :) )

> Exploit #peek to implement #print() and other methods
> -
>
> Key: KAFKA-4772
> URL: https://issues.apache.org/jira/browse/KAFKA-4772
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: james chien
>Priority: Minor
>  Labels: beginner, newbie
>
> From: https://github.com/apache/kafka/pull/2493#pullrequestreview-22157555
> Things that I can think of:
> - print / writeAsTest can be a special impl of peek; KStreamPrint etc can be 
> removed.
> - consider collapse KStreamPeek with KStreamForeach with a flag parameter 
> indicating if the acted key-value pair should still be forwarded.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Throttle replicas based on client-id

[Archie]

Hi All,
I am pretty new to kafka development and I am trying to limit
the amount of traffic that a client can send/produce to the kafka cluster.

 Using quotas on client-id, I was able to limit the amount of
traffic that can be produced by a client to a value very close to the
quota. Now I also want to make sure that the replicated traffic also
follows the same limit.

 For example, I have 2 clients: client-1 and client-2, and 3
machines: mac-1, mac-2 and mac-3. Both the clients have a master replica at
mac-1 and both have one slave replica each in mac-2 and mac-3 (Replication
factor = 3). I have set quota of client-1 as 50 MBps and client-2 as 40
MBps. The client-quotas were able to ensure that data is written into
master at the specified rate ( approximately 50 and 40 MBps respectively).
I tested it by running producers of both clients simultaneously, setting
ack as 1 (means ack when master receives) and using the script
bin/kafka-producer-perf-test.sh.

  But when I ran the same script with ack as -1 (ack from all
replicas), both my client's performance drops down to 30 MBps. My first
question is shouldn't kafka still allow producers to get throughput of 50
and 40 MBps respectively even in presence of replicas?

 I then thought may be this is happening because the replica
traffic does not have the same quota. Now I've been exploring the kafka
source code and found out that there is already something called
replication quota but I believe it does throttling only during partition
movement.

 I've looked at ReplicationQuotaManager.scala ,
ClientQuotaManager.scala files but I am not sure how to implement this
(throttle replica traffic traffic based on client-id). Does anyone have any
ideas/suggestions on how this can be done in the existing kafka code?




Thanks,
Archie

Re: [DISCUSS] KIP-143: Controller Health Metrics

[Jun Rao]

Hi, Joel,

1. Yes, the long term goal is to migrate the metrics on the broker to
kafka-metrics. Since many people are using Yammer reporters, we probably
need to support a few popular ones in kafka-metrics before migrating. Until
that happens, we probably want to stick with the Yammer metrics on the
server side unless we depend on features from kafka-metrics (e.g, quota).

2. Thanks for Onur, we now have moved to a single threaded model. So, none
of the event in metric #1 will overlap.

3. Metric #6 just track the rate/latency each time the controller is called
to initiate or resume the processing of a reassginment request.

4. Metrics #2 and #3. The issue with relying on metric #1 is that the
latter is sensitive to the frequency of metric collection. For example, if
the starting of the controller takes 30 secs and the metric is only
collected once a minute, one may not know the latency with just metric #1,
but will know the latency with metrics #2 and #3. Are you concerned about
the memory overhead of histograms? It doesn't seem that a couple of more
histograms will hurt.

5. Metric #9. Agreed. After KAFKA-5028, this will be reflected in the
remoteTimeMs of the controlled shutdown request.

6. Metrics #4 and #5. They are actually a bit different. The local time of
createTopic/deleteTopic just includes the time to add/delete the topic path
in ZK. The remote time includes the time that the controller processes the
request plus the time for the metadata to be propagated back to the
controller. So, knowing just the portion of the time spent in the
controller can still be useful.

Hi, Isamel,

Thanks the for proposal. A couple of more comments.,

10. It would be useful to add a new metrics for the controller queue size.
kafka.controller:type=ControllerStats,name=QueueSize

11. It would also be useful to know how long an event is waiting in the
controller queue before being processing. Perhaps, we can add a histogram
metric like the following.
kafka.controller:type=ControllerStats,name=QueueTimeMs

Jun

On Thu, Apr 27, 2017 at 11:39 AM, Joel Koshy  wrote:

> Thanks for the KIP - couple of comments:
> - Do you intend to actually use yammer metrics? or use kafka-metrics and
> split the timer into an explicit rate and time? I think long term we ought
> to move off yammer and use kafka-metrics only. Actually either is fine, but
> we should ideally use only one in the long term - and I thought the plan
> was to use kafka-metrics.
> - metric #9 appears to be redundant since we already have per-API request
> rate and time metrics.
> - Same for metric #4, #5 (as there are request stats for
> DeleteTopicRequest - although it is possible for users to trigger deletes
> via ZK)
> - metric #2, #3 are potentially useful, but a bit overkill for a
> histogram. Alternative is to stick to last known value, but that doesn't
> play well with alerts if a high value isn't reset/decayed. Perhaps metric
> #1 would be sufficient to gauge slow start/resignation transitions.
> - metric #1 - some of the states may actually overlap
> - I don't actually understand the semantics of metric #6. Is it rate of
> partition reassignment triggers? does the number of partitions matter?
>
> Joel
>
> On Thu, Apr 27, 2017 at 8:04 AM, Tom Crayford 
> wrote:
>
>> Ismael,
>>
>> Great, that sounds lovely.
>>
>> I'd like a `Timer` (using yammer metrics parlance) over how long it took
>> to
>> process the event, so we can get at p99 and max times spent processing
>> things. Maybe we could even do a log at warning level if event processing
>> takes over some timeout?
>>
>> Thanks
>>
>> Tom
>>
>> On Thu, Apr 27, 2017 at 3:59 PM, Ismael Juma  wrote:
>>
>> > Hi Tom,
>> >
>> > Yes, the plan is to merge KAFKA-5028 first and then use a lock-free
>> > approach for the new  metrics. I considered mentioning that in the KIP
>> > given KAFKA-5120, but didn't in the end. I'll add it to make it clear.
>> >
>> > Regarding locks, they are removed by KAFKA-5028, as you say. So, if I
>> > understand correctly, you are suggesting an event processing rate metric
>> > with event type as a tag? Onur and Jun, what do you think?
>> >
>> > Ismael
>> >
>> > On Thu, Apr 27, 2017 at 3:47 PM, Tom Crayford 
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > We (Heroku) are very excited about this KIP, as we've struggled a bit
>> > with
>> > > controller stability recently. Having these additional metrics would
>> be
>> > > wonderful.
>> > >
>> > > I'd like to ensure polling these metrics *doesn't* hold any locks etc,
>> > > because, as noted in https://issues.apache.org/jira/browse/KAFKA-5120
>> ,
>> > > that
>> > > lock can be held for quite some time. This may become not an issue as
>> of
>> > > KAFKA-5028 though.
>> > >
>> > > Lastly, I'd love to see some metrics around how long the controller
>> > spends
>> > > inside its lock. We've been tracking an issue (
>> > > https://issues.apache.org/jira/browse/KAFKA-5116) where it can hold
>> the
>> > > lock for many, many minutes in a zk client list

Build failed in Jenkins: kafka-trunk-jdk8 #1467

[Apache Jenkins Server]

See 


Changes:

[ismael] KAFKA-3265; Add a public AdminClient API in Java (KIP-117)

--
[...truncated 1.65 MB...]
kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete STARTED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
STARTED

kafka.integ

[jira] [Commented] (KAFKA-4667) Connect should create internal topics

[Colin P. McCabe (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991939#comment-15991939
 ] 

Colin P. McCabe commented on KAFKA-4667:


The Admin client has been merged :)

KIP-133, which is still under discussion, adds a method of getting broker and 
topic configs through Admin client.

> Connect should create internal topics
> -
>
> Key: KAFKA-4667
> URL: https://issues.apache.org/jira/browse/KAFKA-4667
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Emanuele Cesena
>Priority: Critical
> Fix For: 0.11.0.0
>
>
> I'm reporting this as an issue but in fact it requires more investigation 
> (which unfortunately I'm not able to perform at this time).
> Repro steps:
> - configure Kafka for consistency, for example:
> default.replication.factor=3
> min.insync.replicas=2
> unclean.leader.election.enable=false
> - run Connect for the first time, which should create its internal topics
> I believe these topics are created with the broker's default, in particular:
> min.insync.replicas=2
> unclean.leader.election.enable=false
> but connect doesn't produce with acks=all, which in turn may cause the 
> cluster to go in a bad state (see, e.g., 
> https://issues.apache.org/jira/browse/KAFKA-4666).
> Solution would be to force availability mode, i.e. force:
> unclean.leader.election.enable=true
> when creating the connect topics, or viceversa detect availability vs 
> consistency mode and turn acks=all if needed.
> I assume the same happens with other kafka-based services such as streams.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5151) Refactor TransactionCoordinator in-memory structure and error handling logic

[Guozhang Wang (JIRA)]

Guozhang Wang created KAFKA-5151:


 Summary: Refactor TransactionCoordinator in-memory structure and 
error handling logic
 Key: KAFKA-5151
 URL: https://issues.apache.org/jira/browse/KAFKA-5151
 Project: Kafka
  Issue Type: Sub-task
Reporter: Guozhang Wang
Assignee: Guozhang Wang


Current status:

1. we are having two types of threads: request handling thread for any client 
requests as well as controller requests for `immigration` and `emigration`, and 
the marker sender thread for draining queued marker entries and handle 
responses. They maintain different in-memory cache structures like the 
`txnMetadataCache`, and the `pendingTxnMap` which are storing the same info, 
and they access some of the shared structures concurrently, like the markers 
queue and the markerPurgatory.

2. we are having one queue per broker today, and due to the emigration purpose 
we probably are having one queue per brokerId + TxnLogPartitionId + 
DataPartitionId, which would result in a lot of queues to handle.

This ticket is for collapsing some of these structures and simplify the access 
of them from concurrent threads.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5136) Move coordinatorEpoch from WriteTxnMarkerRequest to TxnMarkerEntry

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991873#comment-15991873
 ] 

ASF GitHub Bot commented on KAFKA-5136:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2925


> Move coordinatorEpoch from WriteTxnMarkerRequest to TxnMarkerEntry
> --
>
> Key: KAFKA-5136
> URL: https://issues.apache.org/jira/browse/KAFKA-5136
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 0.11.0.0
>
>
> It makes more sense for the coordinatorEpoc to be on the TxnMarkerEntry 
> rather than the WriteTxnMarkerRequest. It will generate fewer requests per 
> broker



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (KAFKA-5136) Move coordinatorEpoch from WriteTxnMarkerRequest to TxnMarkerEntry

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-5136.
--
   Resolution: Fixed
Fix Version/s: 0.11.0.0

Issue resolved by pull request 2925
[https://github.com/apache/kafka/pull/2925]

> Move coordinatorEpoch from WriteTxnMarkerRequest to TxnMarkerEntry
> --
>
> Key: KAFKA-5136
> URL: https://issues.apache.org/jira/browse/KAFKA-5136
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Reporter: Damian Guy
>Assignee: Damian Guy
> Fix For: 0.11.0.0
>
>
> It makes more sense for the coordinatorEpoc to be on the TxnMarkerEntry 
> rather than the WriteTxnMarkerRequest. It will generate fewer requests per 
> broker



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2925: KAFKA-5136: move coordinatorEpoch from WriteTxnMar...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2925


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2933: MINOR: Some cleanup in the transactional producer

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2933


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk7 #2133

[Apache Jenkins Server]

See 


Changes:

[ismael] KAFKA-3265; Add a public AdminClient API in Java (KIP-117)

--
[...truncated 846.82 KB...]
kafka.server.DynamicConfigChangeTest > testDefaultClientIdQuotaConfigChange 
PASSED

kafka.server.DynamicConfigChangeTest > testQuotaInitialization STARTED

kafka.server.DynamicConfigChangeTest > testQuotaInitialization PASSED

kafka.server.DynamicConfigChangeTest > testUserQuotaConfigChange STARTED

kafka.server.DynamicConfigChangeTest > testUserQuotaConfigChange PASSED

kafka.server.DynamicConfigChangeTest > testClientIdQuotaConfigChange STARTED

kafka.server.DynamicConfigChangeTest > testClientIdQuotaConfigChange PASSED

kafka.server.DynamicConfigChangeTest > testUserClientIdQuotaChange STARTED

kafka.server.DynamicConfigChangeTest > testUserClientIdQuotaChange PASSED

kafka.server.DynamicConfigChangeTest > shouldParseReplicationQuotaProperties 
STARTED

kafka.server.DynamicConfigChangeTest > shouldParseReplicationQuotaProperties 
PASSED

kafka.server.DynamicConfigChangeTest > 
shouldParseRegardlessOfWhitespaceAroundValues STARTED

kafka.server.DynamicConfigChangeTest > 
shouldParseRegardlessOfWhitespaceAroundValues PASSED

kafka.server.DynamicConfigChangeTest > testDefaultUserQuotaConfigChange STARTED

kafka.server.DynamicConfigChangeTest > testDefaultUserQuotaConfigChange PASSED

kafka.server.DynamicConfigChangeTest > shouldParseReplicationQuotaReset STARTED

kafka.server.DynamicConfigChangeTest > shouldParseReplicationQuotaReset PASSED

kafka.server.DynamicConfigChangeTest > testDefaultUserClientIdQuotaConfigChange 
STARTED

kafka.server.DynamicConfigChangeTest > testDefaultUserClientIdQuotaConfigChange 
PASSED

kafka.server.DynamicConfigChangeTest > testConfigChangeOnNonExistingTopic 
STARTED

kafka.server.DynamicConfigChangeTest > testConfigChangeOnNonExistingTopic PASSED

kafka.server.DynamicConfigChangeTest > testConfigChange STARTED

kafka.server.DynamicConfigChangeTest > testConfigChange PASSED

kafka.server.ServerGenerateBrokerIdTest > testGetSequenceIdMethod STARTED

kafka.server.ServerGenerateBrokerIdTest > testGetSequenceIdMethod PASSED

kafka.server.ServerGenerateBrokerIdTest > testBrokerMetadataOnIdCollision 
STARTED

kafka.server.ServerGenerateBrokerIdTest > testBrokerMetadataOnIdCollision PASSED

kafka.server.ServerGenerateBrokerIdTest > testAutoGenerateBrokerId STARTED

kafka.server.ServerGenerateBrokerIdTest > testAutoGenerateBrokerId PASSED

kafka.server.ServerGenerateBrokerIdTest > testMultipleLogDirsMetaProps STARTED

kafka.server.ServerGenerateBrokerIdTest > testMultipleLogDirsMetaProps PASSED

kafka.server.ServerGenerateBrokerIdTest > testDisableGeneratedBrokerId STARTED

kafka.server.ServerGenerateBrokerIdTest > testDisableGeneratedBrokerId PASSED

kafka.server.ServerGenerateBrokerIdTest > testUserConfigAndGeneratedBrokerId 
STARTED

kafka.server.ServerGenerateBrokerIdTest > testUserConfigAndGeneratedBrokerId 
PASSED

kafka.server.ServerGenerateBrokerIdTest > 
testConsistentBrokerIdFromUserConfigAndMetaProps STARTED

kafka.server.ServerGenerateBrokerIdTest > 
testConsistentBrokerIdFromUserConfigAndMetaProps PASSED

kafka.server.AdvertiseBrokerTest > testBrokerAdvertiseHostNameAndPortToZK 
STARTED

kafka.server.AdvertiseBrokerTest > testBrokerAdvertiseHostNameAndPortToZK PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testValidCreateTopicsRequests 
PASSED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
STARTED

kafka.server.CreateTopicsRequestWithPolicyTest > testErrorCreateTopicsRequests 
PASSED

kafka.server.SaslPlaintextReplicaFetchTest > testReplicaFetcherThread STARTED

kafka.server.SaslPlaintextReplicaFetchTest > testReplicaFetcherThread PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestWithUnsupportedVersion STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestWithUnsupportedVersion PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestBeforeSaslHandshakeRequest STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestBeforeSaslHandshakeRequest PASSED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestAfterSaslHandshakeRequest STARTED

kafka.server.SaslApiVersionsRequestTest > 
testApiVersionsRequestAfterSaslHandshakeRequest PASSED

kafka.server.ServerGenerateClusterIdTest > 
testAutoGenerateClusterIdForKafkaClusterParallel STARTED

kafka.server.ServerGenerateClusterIdTest > 
testAutoGenerateClusterIdForKafkaClusterParallel PASSED

kafka.server.ServerGenerateClusterIdTest > testAutoGenerateClusterId STARTED

kafka.server.ServerGenerateClusterIdTest > testAutoGenerateClusterId PASSED

kafka.server.ServerGenerateClusterIdTest > 
testAutoGenerateClusterIdForKafkaClusterSequential STARTED

kafka.server.ServerGenerateClu

[jira] [Updated] (KAFKA-5150) LZ4 decompression is 4-5x slower than Snappy on small batches / messages

[JIRA]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5150?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Xavier Léauté updated KAFKA-5150:
-
Affects Version/s: (was: 0.10.2.0)
   0.10.2.1
   0.11.0.0
   0.8.2.2
   0.9.0.1

> LZ4 decompression is 4-5x slower than Snappy on small batches / messages
> 
>
> Key: KAFKA-5150
> URL: https://issues.apache.org/jira/browse/KAFKA-5150
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.8.2.2, 0.9.0.1, 0.11.0.0, 0.10.2.1
>Reporter: Xavier Léauté
>Assignee: Xavier Léauté
>
> I benchmarked RecordsIteratorDeepRecordsIterator instantiation on small batch 
> sizes with small messages after observing some performance bottlenecks in the 
> consumer. 
> For batch sizes of 1 with messages of 100 bytes, LZ4 heavily underperforms 
> compared to Snappy (see benchmark below). Most of our time is currently spent 
> allocating memory blocks in KafkaLZ4BlockInputStream, due to the fact that we 
> default to larger 64kB block sizes. Some quick testing shows we could improve 
> performance by almost an order of magnitude for small batches and messages if 
> we re-used buffers between instantiations of the input stream.
> [Benchmark 
> Code|https://github.com/xvrl/kafka/blob/small-batch-lz4-benchmark/clients/src/test/java/org/apache/kafka/common/record/DeepRecordsIteratorBenchmark.java#L86]
> {code}
> Benchmark  (compressionType)  
> (messageSize)   Mode  Cnt   Score   Error  Units
> DeepRecordsIteratorBenchmark.measureSingleMessageLZ4  
>   100  thrpt   20   84802.279 ±  1983.847  ops/s
> DeepRecordsIteratorBenchmark.measureSingleMessage SNAPPY  
>   100  thrpt   20  407585.747 ±  9877.073  ops/s
> DeepRecordsIteratorBenchmark.measureSingleMessage   NONE  
>   100  thrpt   20  579141.634 ± 18482.093  ops/s
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5059) Implement Transactional Coordinator

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991810#comment-15991810
 ] 

ASF GitHub Bot commented on KAFKA-5059:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2934


> Implement Transactional Coordinator
> ---
>
> Key: KAFKA-5059
> URL: https://issues.apache.org/jira/browse/KAFKA-5059
> Project: Kafka
>  Issue Type: New Feature
>  Components: core
>Reporter: Damian Guy
>Assignee: Damian Guy
>
> This covers the implementation of the transaction coordinator to support 
> transactions, as described in KIP-98: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2934: KAFKA-5059: [Follow Up] remove broken locking. Fix...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2934


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-5150) LZ4 decompression is 4-5x slower than Snappy on small batches / messages

[JIRA]

Xavier Léauté created KAFKA-5150:


 Summary: LZ4 decompression is 4-5x slower than Snappy on small 
batches / messages
 Key: KAFKA-5150
 URL: https://issues.apache.org/jira/browse/KAFKA-5150
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 0.10.2.0
Reporter: Xavier Léauté
Assignee: Xavier Léauté


I benchmarked RecordsIteratorDeepRecordsIterator instantiation on small batch 
sizes with small messages after observing some performance bottlenecks in the 
consumer. 

For batch sizes of 1 with messages of 100 bytes, LZ4 heavily underperforms 
compared to Snappy (see benchmark below). Most of our time is currently spent 
allocating memory blocks in KafkaLZ4BlockInputStream, due to the fact that we 
default to larger 64kB block sizes. Some quick testing shows we could improve 
performance by almost an order of magnitude for small batches and messages if 
we re-used buffers between instantiations of the input stream.

[Benchmark 
Code|https://github.com/xvrl/kafka/blob/small-batch-lz4-benchmark/clients/src/test/java/org/apache/kafka/common/record/DeepRecordsIteratorBenchmark.java#L86]
{code}
Benchmark  (compressionType)  
(messageSize)   Mode  Cnt   Score   Error  Units
DeepRecordsIteratorBenchmark.measureSingleMessageLZ4
100  thrpt   20   84802.279 ±  1983.847  ops/s
DeepRecordsIteratorBenchmark.measureSingleMessage SNAPPY
100  thrpt   20  407585.747 ±  9877.073  ops/s
DeepRecordsIteratorBenchmark.measureSingleMessage   NONE
100  thrpt   20  579141.634 ± 18482.093  ops/s
{code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

[Jeyhun Karimov]

Thanks for comments.

The concerns makes sense. Although we can guard for immutable keys in
current implementation (with few changes), I didn't consider backward
compatibility.

In this case 2 solutions come to my mind. In both cases, user accesses the
key in Object type, as passing extra type parameter will break
backwards-compatibility.  So user has to cast to actual key type.

1. Firstly, We can overload apply method with 2 argument (key and value)
and force key to be *final*. By doing this,  I think we can address both
backward-compatibility and guarding against key change.

2. Secondly, we can create class KeyAccess like:

public class KeyAccess {
Object key;
public void beforeApply(final Object key) {
this.key = key;
}
public Object getKey() {
return key;
}
}

We can extend *ValueMapper, ValueJoiner* and *ValueTransformer* from
*KeyAccess*. Inside processor (for example *KTableMapValuesProcessor*)
before calling *mapper.apply(value)* we can set the *key* by
*mapper.beforeApply(key)*. As a result, user can use *getKey()* to access
the key inside *apply(value)* method.


Cheers,
Jeyhun




On Mon, May 1, 2017 at 7:24 PM Matthias J. Sax 
wrote:

> Jeyhun,
>
> thanks a lot for the KIP!
>
> I think there are two issues we need to address:
>
> (1) The KIP does not consider backward compatibility. Users did complain
> about this in past releases already, and as the user base grows, we
> should not break backward compatibility in future releases anymore.
> Thus, we should think of a better way to allow key access.
>
> Mathieu's comment goes into the same direction
>
> >> On the other hand, the number of compile failures that would need to be
> >> fixed from this change is unfortunate. :-)
>
> (2) Another concern is, that there is no guard to prevent user code to
> modify the key. This might corrupt partitioning if users do alter the
> key (accidentally -- or users are just not aware that they are not
> allowed to modify the provided key object) and thus break the
> application. (This was the original motivation to not provide the key in
> the first place -- it's guards against modification.)
>
>
> -Matthias
>
>
>
> On 5/1/17 6:31 AM, Mathieu Fenniak wrote:
> > Hi Jeyhun,
> >
> > I just want to add my voice that, I too, have wished for access to the
> > record key during a mapValues or similar operation.
> >
> > On the other hand, the number of compile failures that would need to be
> > fixed from this change is unfortunate. :-)  But at least it would all be
> a
> > pretty clear and easy change.
> >
> > Mathieu
> >
> >
> > On Mon, May 1, 2017 at 6:55 AM, Jeyhun Karimov 
> wrote:
> >
> >> Dear community,
> >>
> >> I want to share KIP-149 [1] based on issues KAFKA-4218 [2], KAFKA-4726
> [3],
> >> KAFKA-3745 [4]. The related PR can be found at [5].
> >> I would like to get your comments.
> >>
> >> [1]
> >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> >> 149%3A+Enabling+key+access+in+ValueTransformer%2C+
> >> ValueMapper%2C+and+ValueJoiner
> >> [2] https://issues.apache.org/jira/browse/KAFKA-4218
> >> [3] https://issues.apache.org/jira/browse/KAFKA-4726
> >> [4] https://issues.apache.org/jira/browse/KAFKA-3745
> >> [5] https://github.com/apache/kafka/pull/2946
> >>
> >>
> >> Cheers,
> >> Jeyhun
> >>
> >>
> >>
> >> --
> >> -Cheers
> >>
> >> Jeyhun
> >>
> >
>
> --
-Cheers

Jeyhun

[jira] [Updated] (KAFKA-5126) Implement KIP-98 transactional methods in the MockProducer

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-5126:
---
Status: Patch Available  (was: In Progress)

> Implement KIP-98 transactional methods in the MockProducer
> --
>
> Key: KAFKA-5126
> URL: https://issues.apache.org/jira/browse/KAFKA-5126
> Project: Kafka
>  Issue Type: Test
>Reporter: Apurva Mehta
>Assignee: Matthias J. Sax
>
> The initial code for the transactional producer leaves the implementation of 
> `initTransactions`, `beginTransaction`, `sendOffsetsToTransaction`, 
> `commitTransaction`, and `abortTransaction` empty in the MockProducer. We 
> need have some implementation there so that our mocks stay healthy.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5126) Implement KIP-98 transactional methods in the MockProducer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991779#comment-15991779
 ] 

ASF GitHub Bot commented on KAFKA-5126:
---

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/2951

KAFKA-5126: Implement KIP-98 transactional methods in the MockProducer



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka 
kafka-5126-add-transactions-to-mock-producer

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2951.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2951


commit 0ee76ea931d09a592091fffe621fb897cfd12fda
Author: Matthias J. Sax 
Date:   2017-05-01T22:20:11Z

KAFKA-5126: Implement KIP-98 transactional methods in the MockProducer




> Implement KIP-98 transactional methods in the MockProducer
> --
>
> Key: KAFKA-5126
> URL: https://issues.apache.org/jira/browse/KAFKA-5126
> Project: Kafka
>  Issue Type: Test
>Reporter: Apurva Mehta
>Assignee: Matthias J. Sax
>
> The initial code for the transactional producer leaves the implementation of 
> `initTransactions`, `beginTransaction`, `sendOffsetsToTransaction`, 
> `commitTransaction`, and `abortTransaction` empty in the MockProducer. We 
> need have some implementation there so that our mocks stay healthy.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2951: KAFKA-5126: Implement KIP-98 transactional methods...

[mjsax]

GitHub user mjsax opened a pull request:

https://github.com/apache/kafka/pull/2951

KAFKA-5126: Implement KIP-98 transactional methods in the MockProducer



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/mjsax/kafka 
kafka-5126-add-transactions-to-mock-producer

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2951.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2951


commit 0ee76ea931d09a592091fffe621fb897cfd12fda
Author: Matthias J. Sax 
Date:   2017-05-01T22:20:11Z

KAFKA-5126: Implement KIP-98 transactional methods in the MockProducer




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-3265) Create Java Admin Client

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-3265.

Resolution: Fixed

> Create Java Admin Client
> 
>
> Key: KAFKA-3265
> URL: https://issues.apache.org/jira/browse/KAFKA-3265
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Grant Henke
>Assignee: Colin P. McCabe
> Fix For: 0.11.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2472: KAFKA-3265: Create Java Admin Client

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2472


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3265) Create Java Admin Client

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991774#comment-15991774
 ] 

ASF GitHub Bot commented on KAFKA-3265:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2472


> Create Java Admin Client
> 
>
> Key: KAFKA-3265
> URL: https://issues.apache.org/jira/browse/KAFKA-3265
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Grant Henke
>Assignee: Colin P. McCabe
> Fix For: 0.11.0.0
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : kafka-trunk-jdk8 #1466

[Apache Jenkins Server]

See 

[jira] [Comment Edited] (KAFKA-4667) Connect should create internal topics

[Randall Hauch (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991697#comment-15991697
 ] 

Randall Hauch edited comment on KAFKA-4667 at 5/1/17 10:47 PM:
---

The AdminClient pull request is being added as part of [PR 
2472|https://github.com/apache/kafka/pull/2472], though it looks like it's very 
close to being merged.

As mentioned above, the most strongly consistent and durable topics require 
per-topic configuration with:

* replication factor of {{min(3, nodeCount)}}
* number of partitions to 1
* {{min.insync.replicas=min(round(replicationFactor/2),nodeCount)}}
* {{unclean.leader.election.enable=false}}
* {{cleanup.policy=compact}}

Note that since we can't specify a replication factor or 
{{min.insync.replicas}} greater than the number of nodes in a cluster, we need 
to first determine the cluster size, {{nodeCount}}, and calculate the desired 
replication factor and {{min.insync.replicas}} value.

Kafka Connect currently creates 3 internal topics for each connector: one for 
the connector configuration(s), one for the offsets, and one for status. Should 
all 3 of these internal topics (per connector) use these same settings, or 
might the status topic not need to be so conservatively defined? Secondly, 
should a connector configuration be allowed to override these or any of the 
other [topic configuration properties| 
https://kafka.apache.org/documentation/#topic-config]?

Finally, the AdminClient in [PR 2472|https://github.com/apache/kafka/pull/2472] 
will allow us to get the standard metadata for each topic partition, meaning we 
can get the partition count, ISRs, and replica nodes. So we could output a 
warning if the replication factor is smaller than the preferred value as 
calculated above, but can we do much else? For example, can we even read the 
topic's cleanup policy, whether unclean leader election is enabled, or other 
topic properties, and if so it doesn't appear we can alter a topic 
configuration.


was (Author: rhauch):
The AdminClient pull request is being added as part of [PR 
2472|https://github.com/apache/kafka/pull/2472], though it looks like it's very 
close to being merged.

As mentioned above, the most strongly consistent and durable topics require 
per-topic configuration with:

* replication factor of {{min(3, nodeCount)}}
* number of partitions to 1
* {{min.insync.replicas=min(round(replicationFactor/2),nodeCount)}}
* {{unclean.leader.election.enable=false}}
* {{cleanup.policy=compact}}

Note that since we can't specify a replication factor or 
{{min.insync.replicas}} greater than the number of nodes in a cluster, we need 
to first determine the cluster size, {{nodeCount}}.

Kafka Connect currently creates 3 internal topics for each connector: one for 
the connector configuration(s), one for the offsets, and one for status. Should 
all 3 of these internal topics (per connector) use these same settings, or 
might the status topic not need to be so conservatively defined? Secondly, 
should a connector configuration be allowed to override these or any of the 
other [topic configuration properties| 
https://kafka.apache.org/documentation/#topic-config]?

Finally, the AdminClient in [PR 2472|https://github.com/apache/kafka/pull/2472] 
will allow us to get the standard metadata for each topic partition, meaning we 
can get the partition count, ISRs, and replica nodes. So we could output a 
warning if the replication factor is smaller than the preferred value as 
calculated above, but can we do much else? For example, can we even read the 
topic's cleanup policy, whether unclean leader election is enabled, or other 
topic properties, and if so it doesn't appear we can alter a topic 
configuration.

> Connect should create internal topics
> -
>
> Key: KAFKA-4667
> URL: https://issues.apache.org/jira/browse/KAFKA-4667
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Emanuele Cesena
>Priority: Critical
> Fix For: 0.11.0.0
>
>
> I'm reporting this as an issue but in fact it requires more investigation 
> (which unfortunately I'm not able to perform at this time).
> Repro steps:
> - configure Kafka for consistency, for example:
> default.replication.factor=3
> min.insync.replicas=2
> unclean.leader.election.enable=false
> - run Connect for the first time, which should create its internal topics
> I believe these topics are created with the broker's default, in particular:
> min.insync.replicas=2
> unclean.leader.election.enable=false
> but connect doesn't produce with acks=all, which in turn may cause the 
> cluster to go in a bad state (see, e.g., 
> https://issues.apache.org/jira/browse/KAFKA-4666).
> Solution would be to force availability mode, i.e. force:
> unclean.leader.election.

[jira] [Commented] (KAFKA-4667) Connect should create internal topics

[Randall Hauch (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991697#comment-15991697
 ] 

Randall Hauch commented on KAFKA-4667:
--

The AdminClient pull request is being added as part of [PR 
2472|https://github.com/apache/kafka/pull/2472], though it looks like it's very 
close to being merged.

As mentioned above, the most strongly consistent and durable topics require 
per-topic configuration with:

* replication factor of {{min(3, nodeCount)}}
* number of partitions to 1
* {{min.insync.replicas=min(round(replicationFactor/2),nodeCount)}}
* {{unclean.leader.election.enable=false}}
* {{cleanup.policy=compact}}

Note that since we can't specify a replication factor or 
{{min.insync.replicas}} greater than the number of nodes in a cluster, we need 
to first determine the cluster size, {{nodeCount}}.

Kafka Connect currently creates 3 internal topics for each connector: one for 
the connector configuration(s), one for the offsets, and one for status. Should 
all 3 of these internal topics (per connector) use these same settings, or 
might the status topic not need to be so conservatively defined? Secondly, 
should a connector configuration be allowed to override these or any of the 
other [topic configuration properties| 
https://kafka.apache.org/documentation/#topic-config]?

Finally, the AdminClient in [PR 2472|https://github.com/apache/kafka/pull/2472] 
will allow us to get the standard metadata for each topic partition, meaning we 
can get the partition count, ISRs, and replica nodes. So we could output a 
warning if the replication factor is smaller than the preferred value as 
calculated above, but can we do much else? For example, can we even read the 
topic's cleanup policy, whether unclean leader election is enabled, or other 
topic properties, and if so it doesn't appear we can alter a topic 
configuration.

> Connect should create internal topics
> -
>
> Key: KAFKA-4667
> URL: https://issues.apache.org/jira/browse/KAFKA-4667
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Reporter: Emanuele Cesena
>Priority: Critical
> Fix For: 0.11.0.0
>
>
> I'm reporting this as an issue but in fact it requires more investigation 
> (which unfortunately I'm not able to perform at this time).
> Repro steps:
> - configure Kafka for consistency, for example:
> default.replication.factor=3
> min.insync.replicas=2
> unclean.leader.election.enable=false
> - run Connect for the first time, which should create its internal topics
> I believe these topics are created with the broker's default, in particular:
> min.insync.replicas=2
> unclean.leader.election.enable=false
> but connect doesn't produce with acks=all, which in turn may cause the 
> cluster to go in a bad state (see, e.g., 
> https://issues.apache.org/jira/browse/KAFKA-4666).
> Solution would be to force availability mode, i.e. force:
> unclean.leader.election.enable=true
> when creating the connect topics, or viceversa detect availability vs 
> consistency mode and turn acks=all if needed.
> I assume the same happens with other kafka-based services such as streams.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Jenkins build is back to normal : kafka-trunk-jdk7 #2132

[Apache Jenkins Server]

See 

[jira] [Updated] (KAFKA-5148) Add configurable compression block size to the broker

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5148?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-5148:
---
Description: Similar to the discussion in KAFKA-3704, we should consider a 
configurable compression block size on the broker side. This especially 
considering the change in block size from 32KB to 1KB in the 0.10.2 release.  
(was: Similar to the discussion in KAFKA-3704, we should consider a 
configurable compression block size on the broker side. This especially 
considering the change in block size from 32KB to 1KB in the 0.10 release.)

> Add configurable compression block size to the broker
> -
>
> Key: KAFKA-5148
> URL: https://issues.apache.org/jira/browse/KAFKA-5148
> Project: Kafka
>  Issue Type: Improvement
>  Components: compression
>Reporter: Dustin Cote
> Fix For: 0.10.2.0
>
>
> Similar to the discussion in KAFKA-3704, we should consider a configurable 
> compression block size on the broker side. This especially considering the 
> change in block size from 32KB to 1KB in the 0.10.2 release.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4669) KafkaProducer.flush hangs when NetworkClient.handleCompletedReceives throws exception

[Shannon Carey (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4669?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991488#comment-15991488
 ] 

Shannon Carey commented on KAFKA-4669:
--

We encountered this exception with our 0.9.0.1 client too, but it was preceded 
by another exception:

{code}
2017-05-01 10:38:02:866 thread=kafka-producer-network-thread | producer-1, 
level=ERROR, logger=org.apache.kafka.clients.producer.internals.Sender, , 
message="Uncaught error in kafka producer I/O thread: 
"org.apache.kafka.common.protocol.types.SchemaException: Error reading field 
'brokers': Error reading field 'host': java.nio.BufferUnderflowException
at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:71)
at 
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:439)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:265)
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128)
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]

2017-05-01 10:38:04:847 thread=kafka-producer-network-thread | producer-1, 
level=ERROR, logger=org.apache.kafka.clients.producer.internals.Sender, , me
ssage="Uncaught error in kafka producer I/O thread: "
java.lang.IllegalStateException: Correlation id for response (3623413) does not 
match request (3623406)
at 
org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:477)
at 
org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:440)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:265)
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128)
at java.lang.Thread.run(Thread.java:745) [na:1.8.0_51]
{code}

It seems like we will need to use metrics in order to detect this situation and 
react by restarting the application, assuming there's no workaround.

The producer should probably be fixed so that any unhandled exceptions that 
occur in the Kafka internal threads result in an exception thrown to the user 
code so that user code can react to it instead of silently ceasing to work. Or, 
if it's a transient problem, the thread should recover.

> KafkaProducer.flush hangs when NetworkClient.handleCompletedReceives throws 
> exception
> -
>
> Key: KAFKA-4669
> URL: https://issues.apache.org/jira/browse/KAFKA-4669
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.1
>Reporter: Cheng Ju
>Priority: Critical
>  Labels: reliability
> Fix For: 0.11.0.0
>
>
> There is no try catch in NetworkClient.handleCompletedReceives.  If an 
> exception is thrown after inFlightRequests.completeNext(source), then the 
> corresponding RecordBatch's done will never get called, and 
> KafkaProducer.flush will hang on this RecordBatch.
> I've checked 0.10 code and think this bug does exist in 0.10 versions.
> A real case.  First a correlateId not match exception happens:
> 13 Jan 2017 17:08:24,059 ERROR [kafka-producer-network-thread | producer-21] 
> (org.apache.kafka.clients.producer.internals.Sender.run:130)  - Uncaught 
> error in kafka producer I/O thread: 
> java.lang.IllegalStateException: Correlation id for response (703766) does 
> not match request (703764)
>   at 
> org.apache.kafka.clients.NetworkClient.correlate(NetworkClient.java:477)
>   at 
> org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:440)
>   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:265)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:216)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:128)
>   at java.lang.Thread.run(Thread.java:745)
> Then jstack shows the thread is hanging on:
>   at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
>   at 
> org.apache.kafka.clients.producer.internals.ProduceRequestResult.await(ProduceRequestResult.java:57)
>   at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.awaitFlushCompletion(RecordAccumulator.java:425)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.flush(KafkaProducer.java:544)
>   at org.apache.flume.sink.kafka.KafkaSink.process(KafkaSink.java:224)
>   at 
> org.apache.flume.sink.DefaultSinkProcessor.process(DefaultSinkProcessor.java:67)
>   at org.apache.flume.SinkRunner$PollingRunner.run(SinkRunner.java:145)
>   at java.lang.Thread.run(Thread.java:745)
> client code 



--
This message was sen

[DISCUSS] KIP-150 - Kafka-Streams Cogroup

[Kyle Winkelman]

Hello all,

I have created KIP-150 to facilitate discussion about adding cogroup to the
streams DSL.

Please find the KIP here:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-150+-+Kafka-Streams+Cogroup

Please find my initial implementation here:
https://github.com/KyleWinkelman/kafka

Thanks,
Kyle Winkelman

Build failed in Jenkins: kafka-trunk-jdk7 #2131

[Apache Jenkins Server]

See 


Changes:

[junrao] KAFKA-4954; Request handler utilization quotas

--
[...truncated 822.64 KB...]
java.lang.OutOfMemoryError: unable to create new native thread

java.lang.NullPointerException

kafka.server.ControlledShutdownLeaderSelectorTest > testSelectLeader STARTED

kafka.server.ControlledShutdownLeaderSelectorTest > testSelectLeader PASSED

kafka.server.HighwatermarkPersistenceTest > 
testHighWatermarkPersistenceMultiplePartitions STARTED

kafka.server.HighwatermarkPersistenceTest > 
testHighWatermarkPersistenceMultiplePartitions FAILED
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at kafka.server.ClientQuotaManager.start(ClientQuotaManager.scala:153)
at kafka.server.ClientQuotaManager.(ClientQuotaManager.scala:151)
at kafka.server.QuotaFactory$.instantiate(QuotaFactory.scala:51)
at 
kafka.server.HighwatermarkPersistenceTest.testHighWatermarkPersistenceMultiplePartitions(HighwatermarkPersistenceTest.scala:108)

kafka.server.HighwatermarkPersistenceTest > 
testHighWatermarkPersistenceSinglePartition STARTED

kafka.server.HighwatermarkPersistenceTest > 
testHighWatermarkPersistenceSinglePartition FAILED
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at kafka.server.ClientQuotaManager.start(ClientQuotaManager.scala:153)
at kafka.server.ClientQuotaManager.(ClientQuotaManager.scala:151)
at kafka.server.QuotaFactory$.instantiate(QuotaFactory.scala:51)
at 
kafka.server.HighwatermarkPersistenceTest.testHighWatermarkPersistenceSinglePartition(HighwatermarkPersistenceTest.scala:63)

kafka.server.FetchRequestTest > testBrokerRespectsPartitionsOrderAndSizeLimits 
STARTED

kafka.server.FetchRequestTest > testBrokerRespectsPartitionsOrderAndSizeLimits 
FAILED
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at 
org.apache.zookeeper.server.ZooKeeperServer.setupRequestProcessors(ZooKeeperServer.java:431)
at 
org.apache.zookeeper.server.ZooKeeperServer.startup(ZooKeeperServer.java:419)
at 
org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:119)
at kafka.zk.EmbeddedZookeeper.(EmbeddedZookeeper.scala:36)
at kafka.zk.ZooKeeperTestHarness.setUp(ZooKeeperTestHarness.scala:44)
at 
kafka.integration.KafkaServerTestHarness.setUp(KafkaServerTestHarness.scala:83)
at kafka.server.FetchRequestTest.setUp(FetchRequestTest.scala:46)

java.lang.NullPointerException
at kafka.server.FetchRequestTest.tearDown(FetchRequestTest.scala:52)

kafka.server.FetchRequestTest > testFetchRequestV2WithOversizedMessage STARTED

kafka.server.FetchRequestTest > testFetchRequestV2WithOversizedMessage FAILED
java.lang.OutOfMemoryError: unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:714)
at 
org.apache.zookeeper.server.NIOServerCnxnFactory.start(NIOServerCnxnFactory.java:109)
at 
org.apache.zookeeper.server.NIOServerCnxnFactory.startup(NIOServerCnxnFactory.java:116)
at kafka.zk.EmbeddedZookeeper.(EmbeddedZookeeper.scala:36)
at kafka.zk.ZooKeeperTestHarness.setUp(ZooKeeperTestHarness.scala:44)
at 
kafka.integration.KafkaServerTestHarness.setUp(KafkaServerTestHarness.scala:83)
at kafka.server.FetchRequestTest.setUp(FetchRequestTest.scala:46)

java.lang.NullPointerException
at kafka.server.FetchRequestTest.tearDown(FetchRequestTest.scala:52)

kafka.server.MetadataRequestTest > testReplicaDownResponse STARTED

kafka.server.MetadataRequestTest > testReplicaDownResponse FAILED
java.lang.OutOfMemoryError: unable to create new native thread

kafka.server.MetadataRequestTest > testRack STARTED

kafka.server.MetadataRequestTest > testRack FAILED
java.lang.OutOfMemoryError: unable to create new native thread

kafka.server.MetadataRequestTest > testIsInternal STARTED

kafka.server.MetadataRequestTest > testIsInternal FAILED
java.lang.OutOfMemoryError: unable to create new native thread

kafka.server.MetadataRequestTest > testControllerId STARTED

kafka.server.MetadataRequestTest > testControllerId FAILED
java.lang.OutOfMemoryError: unable to create new native thread

kafka.server.MetadataRequestTest > testAllTopicsRequest STARTED

kafka.server.MetadataRequestTest > testAllTopicsRequest FAILED
java.lang.OutOfMemoryError: unable to create new native thread

kafka.server.MetadataRequestTest > testClusterIdIsValid STARTED

kafka.server.MetadataRequestTe

[jira] [Commented] (KAFKA-3877) Gradle compiler daemon exits with non-zero exit code while running LogOffsetTest

[Colin P. McCabe (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991444#comment-15991444
 ] 

Colin P. McCabe commented on KAFKA-3877:


As [~original-brownbear] commented, this is almost certainly an OOM error 
caused by the Jenkins environment not having enough free memory.  I've seen 
this locally in low-memory situations as well.

> Gradle compiler daemon exits with non-zero exit code while running 
> LogOffsetTest
> 
>
> Key: KAFKA-3877
> URL: https://issues.apache.org/jira/browse/KAFKA-3877
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ismael Juma
>  Labels: transient-unit-test-failure
>
> This happened in a recent build:
> {code}
> kafka.server.LogOffsetTest > testGetOffsetsBeforeNow STARTED
> :kafka-trunk-jdk8:core:test FAILED
> :test_core_2_11 FAILED
> FAILURE: Build failed with an exception.
> * What went wrong:
> Execution failed for task ':test_core_2_11'.
> > Process 'Gradle Compiler Daemon 1' finished with non-zero exit value 137
> {code}
> https://builds.apache.org/job/kafka-trunk-jdk8/702



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5112) Trunk compatibility tests should test against 0.10.2

[Colin P. McCabe (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991418#comment-15991418
 ] 

Colin P. McCabe commented on KAFKA-5112:


This seems to be a duplicate of https://github.com/apache/kafka/pull/2701.  
Reassigning to [~ijuma]

> Trunk compatibility tests should test against 0.10.2
> 
>
> Key: KAFKA-5112
> URL: https://issues.apache.org/jira/browse/KAFKA-5112
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Colin P. McCabe
>Assignee: Ismael Juma
>
> Now that 0.10.2 has been released, our trunk compatibility tests should test 
> against it.  This will ensure that 0.11 clients are backwards compatible with 
> 0.10.2 brokers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-5112) Trunk compatibility tests should test against 0.10.2

[Colin P. McCabe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5112?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin P. McCabe reassigned KAFKA-5112:
--

Assignee: Ismael Juma  (was: Colin P. McCabe)

> Trunk compatibility tests should test against 0.10.2
> 
>
> Key: KAFKA-5112
> URL: https://issues.apache.org/jira/browse/KAFKA-5112
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Colin P. McCabe
>Assignee: Ismael Juma
>
> Now that 0.10.2 has been released, our trunk compatibility tests should test 
> against it.  This will ensure that 0.11 clients are backwards compatible with 
> 0.10.2 brokers.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Resolved] (KAFKA-4899) Fix findbugs warnings in kafka-core

[Colin P. McCabe (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Colin P. McCabe resolved KAFKA-4899.

   Resolution: Fixed
Fix Version/s: 0.11.0.0

> Fix findbugs warnings in kafka-core
> ---
>
> Key: KAFKA-4899
> URL: https://issues.apache.org/jira/browse/KAFKA-4899
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Colin P. McCabe
>Assignee: Colin P. McCabe
> Fix For: 0.11.0.0
>
>
> Fix findbugs warnings in kafka-core



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5091) ReassignPartitionsCommand should protect against empty replica list assignment

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991354#comment-15991354
 ] 

ASF GitHub Bot commented on KAFKA-5091:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2904


> ReassignPartitionsCommand should protect against empty replica list 
> assignment 
> ---
>
> Key: KAFKA-5091
> URL: https://issues.apache.org/jira/browse/KAFKA-5091
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Reporter: Ryan P
>Assignee: huxi
> Fix For: 0.11.0.0
>
>
> Currently it is possible to lower a topics replication factor to 0 through 
> the use of the kafka-reassign-partitions command. 
> i.e. 
> cat increase-replication-factor.json
>   {"version":1,
>   "partitions":[{"topic":"foo","partition":0,"replicas":[]}]}
> kafka-reassign-partitions --zookeeper localhost:2181 --reassignment-json-file 
> increase-replication-factor.json --execute
> Topic:testPartitionCount:1ReplicationFactor:0 Configs:
>   Topic: foo  Partition: 0Leader: -1  Replicas:   Isr:
> I for one can't think of a reason why this is something someone would do 
> intentionally. That said I think it's worth validating that at least 1 
> replica remains within the replica list prior to executing the partition 
> reassignment. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2904: KAFKA-5091: ReassignPartitionsCommand should...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2904


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-5091) ReassignPartitionsCommand should protect against empty replica list assignment

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-5091.

   Resolution: Fixed
Fix Version/s: 0.11.0.0

Issue resolved by pull request 2904
[https://github.com/apache/kafka/pull/2904]

> ReassignPartitionsCommand should protect against empty replica list 
> assignment 
> ---
>
> Key: KAFKA-5091
> URL: https://issues.apache.org/jira/browse/KAFKA-5091
> Project: Kafka
>  Issue Type: Improvement
>  Components: tools
>Reporter: Ryan P
>Assignee: huxi
> Fix For: 0.11.0.0
>
>
> Currently it is possible to lower a topics replication factor to 0 through 
> the use of the kafka-reassign-partitions command. 
> i.e. 
> cat increase-replication-factor.json
>   {"version":1,
>   "partitions":[{"topic":"foo","partition":0,"replicas":[]}]}
> kafka-reassign-partitions --zookeeper localhost:2181 --reassignment-json-file 
> increase-replication-factor.json --execute
> Topic:testPartitionCount:1ReplicationFactor:0 Configs:
>   Topic: foo  Partition: 0Leader: -1  Replicas:   Isr:
> I for one can't think of a reason why this is something someone would do 
> intentionally. That said I think it's worth validating that at least 1 
> replica remains within the replica list prior to executing the partition 
> reassignment. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5149) New producer hardcodes key and value serializers to ByteArray

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5149?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991317#comment-15991317
 ] 

Jun Rao commented on KAFKA-5149:


This is similar to what's reported in 
https://issues.apache.org/jira/browse/KAFKA-2526. So, the functionality is 
there through a customized line-reader. 

> New producer hardcodes key and value serializers to ByteArray
> -
>
> Key: KAFKA-5149
> URL: https://issues.apache.org/jira/browse/KAFKA-5149
> Project: Kafka
>  Issue Type: Bug
>  Components: producer , tools
>Affects Versions: 0.10.2.0
>Reporter: Yeva Byzek
>Assignee: Bharat Viswanadham
>  Labels: newbie
>
> New producer hardcodes the serializers:
> {noformat}
>   props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, 
> "org.apache.kafka.common.serialization.ByteArraySerializer")
> props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, 
> "org.apache.kafka.common.serialization.ByteArraySerializer")
> {noformat}
> And thus cannot be overridden from the commandline argument 
> {{--key-serializer}} or {{--value-serializer}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

[Michael André Pearce]

If it's a choice of either or. I would vote keep as is. At least then people 
can write their own api wrappers easily with not many lines of code, like the 
one supplied.

Sent from my iPhone

> On 1 May 2017, at 18:34, Matthias J. Sax  wrote:
> 
> Hi,
> 
> I am personally not a big fan of providing two APIs to do the same
> thing. If we believe that one API is better than the other, we should
> indicate this by deprecating the old API IMHO.
> 
> Just my two cents.
> 
> 
> -Matthias
> 
> 
>> On 4/30/17 11:05 PM, Michael Pearce wrote:
>> See
>> 
>> https://hc.apache.org/httpcomponents-client-ga/tutorial/html/fluent.html
>> 
>> Doesn't cause much issue over there where you have a fluent api wrapper for 
>> those who like that style, and the original more verbose api.
>> 
>> Surely it would be better than removing a way of doing things that everyone 
>> has got used to and built their code around ala constructors approach. And 
>> simply provide a wrapper to provide a per field way of doing things.
>> 
>> 
>> 
>> 
>> 
>> Sent using OWA for iPhone
>> 
>> From: Stephane Maarek 
>> Sent: Monday, May 1, 2017 6:37:44 AM
>> To: dev@kafka.apache.org
>> Subject: Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface
>> 
>> I’m not sure how people would feel about having two distinct methods to 
>> build the same object?
>> An API wrapper may be useful, but it doesn’t bring opinion about how one 
>> should program, that’s just driven by the docs.
>> I’m okay with that, but we need concensus
>> 
>> 
>> On 1/5/17, 6:08 am, "Michael Pearce"  wrote:
>> 
>>Why not, instead of deprecating or removing whats there, as noted, its a 
>> point of preference, think about something that could wrap the existing, but 
>> provide an api that for you is cleaner?
>> 
>>e.g. here's a sample idea building on a fluent api way. (this wraps the 
>> producer and producer records so no changes needed)
>> 
>>
>> https://gist.github.com/michaelandrepearce/de0f5ad4aa7d39d243781741c58c293e
>> 
>>In future as new items further add to Producer Record, they just become 
>> new methods in the fluent API, as it builds the ProducerRecord using the 
>> most exhaustive constructor.
>> 
>> 
>> 
>>
>>From: Matthias J. Sax 
>>Sent: Saturday, April 29, 2017 6:52 PM
>>To: dev@kafka.apache.org
>>Subject: Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface
>> 
>>I understand that we cannot just break stuff (btw: also not for
>>Streams!). But deprecating does not break anything, so I don't think
>>it's a big deal to change the API as long as we keep the old API as
>>deprecated.
>> 
>> 
>>-Matthias
>> 
>>>On 4/29/17 9:28 AM, Jay Kreps wrote:
>>> Hey Matthias,
>>> 
>>> Yeah I agree, I'm not against change as a general thing! I also think if
>>> you look back on the last two years, we completely rewrote the producer and
>>> consumer APIs, reworked the binary protocol many times over, and added the
>>> connector and stream processing apis, both major new additions. So I don't
>>> think we're in too much danger of stagnating!
>>> 
>>> My two cents was just around breaking compatibility for trivial changes
>>> like constructor => builder. I think this only applies to the producer,
>>> consumer, and connect apis which are heavily embedded in hundreds of
>>> ecosystem components that depend on them. This is different from direct
>>> usage. If we break the streams api it is really no big deal---apps just
>>> need to rebuild when they upgrade, not the end of the world at all. However
>>> because many intermediate things depend on the Kafka producer you can cause
>>> these weird situations where your app depends on two third party things
>>> that use Kafka and each requires different, incompatible versions. We did
>>> this a lot in earlier versions of Kafka and it was the cause of much angst
>>> (and an ingrained general reluctance to upgrade) from our users.
>>> 
>>> I still think we may have to break things, i just don't think we should do
>>> it for things like builders vs direct constructors which i think are kind
>>> of a debatable matter of taste.
>>> 
>>> -Jay
>>> 
>>> 
>>> 
>>> On Mon, Apr 24, 2017 at 9:40 AM, Matthias J. Sax 
>>> wrote:
>>> 
 Hey Jay,
 
 I understand your concern, and for sure, we will need to keep the
 current constructors deprecated for a long time (ie, many years).
 
 But if we don't make the move, we will not be able to improve. And I
 think warnings about using deprecated APIs is an acceptable price to
 pay. And the API improvements will help new people who adopt Kafka to
 get started more easily.
 
 Otherwise Kafka might end up as many other enterprise software with a
 lots of old stuff that is kept forever because nobody has the guts to
 improve/change it.
 
 Of course, we can still improve the docs of the deprecated constructors,
 too

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

[Michael Pearce]

Hi Gwen,

Then intent here was to allow tools that perform similar role to mirror makers 
of replicating the messaging from one cluster to another.  Eg like mirror make 
should just be taking and transferring the headers as is.

We don't actually use this inside our company, so not exposing this isn't an 
issue for us. Just believe there are companies like confluent who have tools 
like replicator that do.

And as good citizens think we should complete the work and expose the headers 
same as in the record to at least allow them to replicate the messages as is. 
Note Steph seems to want it.

Cheers
Mike

Sent using OWA for iPhone

From: Gwen Shapira 
Sent: Monday, May 1, 2017 2:36:34 PM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

Hi,

I'm excited to see the community expanding Connect in this direction!
Headers + Transforms == Fun message routing.

I like how clean the proposal is, but I'm concerned that it kinda deviates
from how Connect handles data elsewhere.
Unlike Kafka, Connect doesn't look at all data as byte-arrays, we have
converters that take data in specific formats (JSON, Avro) and turns it
into Connect data types (defined in the data api). I think it will be more
consistent for connector developers to also get headers as some kind of
structured or semi-structured data (and to expand the converters to handle
header conversions as well).
This will allow for Connect's separation of concerns - Connector developers
don't worry about data formats (because they get the internal connect
objects) and Converters do all the data format work.

Another thing, in my experience, APIs work better if they are put into use
almost immediately - so difficulties in using the APIs are immediately
surfaced. Are you planning any connectors that will use this feature (not
necessarily in Kafka, just in general)? Or perhaps we can think of a way to
expand Kafka's file connectors so they'll use headers somehow (can't think
of anything, but maybe?).

Gwen

On Sat, Apr 29, 2017 at 12:12 AM, Michael Pearce 
wrote:

> Hi All,
>
> Now KIP-82 is committed I would like to discuss extending the work to
> expose it in Kafka Connect, its primary focus being so connectors that may
> do similar tasks as MirrorMakers, either Kafka->Kafka or JMS-Kafka would be
> able to replicate the headers.
> It would be ideal but not mandatory for this to go in 0.11 release so is
> available on day one of headers being available.
>
> Please find the KIP here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 145+-+Expose+Record+Headers+in+Kafka+Connect
>
> Please find an initial implementation as a PR here:
> https://github.com/apache/kafka/pull/2942
>
> Kind Regards
> Mike
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>



--
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter  | blog

The information contained in this email is strictly confidential and for the 
use of the addressee only, unless otherwise indicated. If you are not the 
intended recipient, please do not read, copy, use or disclose to others this 
message or any attachment. Please also notify the sender by replying to this 
email or by telephone (+44(020 7896 0011) and then delete the email and any 
copies of it. Opinions, conclusion (etc) that do not relate to the official 
business of this company shall be understood as neither given nor endorsed by 
it. IG is a trading name of IG Markets Limited (a company registered in England 
and Wales, company number 04008957) and IG Index Limited (a company registered 
in England and Wales, company number 01190902). Registered address at Cannon 
Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited 
(register number 195355) and IG Index Limited (register number 114059) are 
authorised and regulated by the Financial Conduct Authority.

[jira] [Work started] (KAFKA-5149) New producer hardcodes key and value serializers to ByteArray

[Bharat Viswanadham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-5149 started by Bharat Viswanadham.
-
> New producer hardcodes key and value serializers to ByteArray
> -
>
> Key: KAFKA-5149
> URL: https://issues.apache.org/jira/browse/KAFKA-5149
> Project: Kafka
>  Issue Type: Bug
>  Components: producer , tools
>Affects Versions: 0.10.2.0
>Reporter: Yeva Byzek
>Assignee: Bharat Viswanadham
>  Labels: newbie
>
> New producer hardcodes the serializers:
> {noformat}
>   props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, 
> "org.apache.kafka.common.serialization.ByteArraySerializer")
> props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, 
> "org.apache.kafka.common.serialization.ByteArraySerializer")
> {noformat}
> And thus cannot be overridden from the commandline argument 
> {{--key-serializer}} or {{--value-serializer}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-5149) New producer hardcodes key and value serializers to ByteArray

[Bharat Viswanadham (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Bharat Viswanadham reassigned KAFKA-5149:
-

Assignee: Bharat Viswanadham

> New producer hardcodes key and value serializers to ByteArray
> -
>
> Key: KAFKA-5149
> URL: https://issues.apache.org/jira/browse/KAFKA-5149
> Project: Kafka
>  Issue Type: Bug
>  Components: producer , tools
>Affects Versions: 0.10.2.0
>Reporter: Yeva Byzek
>Assignee: Bharat Viswanadham
>  Labels: newbie
>
> New producer hardcodes the serializers:
> {noformat}
>   props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, 
> "org.apache.kafka.common.serialization.ByteArraySerializer")
> props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, 
> "org.apache.kafka.common.serialization.ByteArraySerializer")
> {noformat}
> And thus cannot be overridden from the commandline argument 
> {{--key-serializer}} or {{--value-serializer}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Build failed in Jenkins: kafka-trunk-jdk8 #1465

[Apache Jenkins Server]

See 


Changes:

[junrao] KAFKA-4954; Request handler utilization quotas

--
[...truncated 1.64 MB...]

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithCollision PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokerListWithNoTopics PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testGetAllTopicMetadata 
PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata 
STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.SaslPlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.PrimitiveApiTest > testMultiProduce STARTED

kafka.integration.PrimitiveApiTest > testMultiProduce PASSED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch STARTED

kafka.integration.PrimitiveApiTest > testDefaultEncoderProducerAndFetch PASSED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize 
STARTED

kafka.integration.PrimitiveApiTest > testFetchRequestCanProperlySerialize PASSED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests STARTED

kafka.integration.PrimitiveApiTest > testPipelinedProduceRequests PASSED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch STARTED

kafka.integration.PrimitiveApiTest > testProduceAndMultiFetch PASSED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression STARTED

kafka.integration.PrimitiveApiTest > 
testDefaultEncoderProducerAndFetchWithCompression PASSED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic STARTED

kafka.integration.PrimitiveApiTest > testConsumerEmptyTopic PASSED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest STARTED

kafka.integration.PrimitiveApiTest > testEmptyFetchRequest PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testIsrAfterBrokerShutDownAndJoinsBack PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopicWithCollision 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
STARTED

kafka.integration.PlaintextTopicMetadataTest > testAliveBrokerListWithNoTopics 
PASSED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic STARTED

kafka.integration.PlaintextTopicMetadataTest > testAutoCreateTopic PASSED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testGetAllTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterNewBrokerStartup PASSED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata STARTED

kafka.integration.PlaintextTopicMetadataTest > testBasicTopicMetadata PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAutoCreateTopicWithInvalidReplication PASSED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown STARTED

kafka.integration.PlaintextTopicMetadataTest > 
testAliveBrokersListWithNoTopicsAfterABrokerShutdown PASSED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete STARTED

kafka.integration.MetricsDuringTopicCreationDeletionTest > 
testMetricsDuringTopicCreateDelete PASSED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow 
STARTED

kafka.integration.AutoOffsetResetTest > testResetToLatestWhenOffsetTooLow PASSED

kafka.integration.AutoOffsetResetTest > testResetToEarliestWhenOffsetTooLow 
STARTED

kafka.integration.Auto

[jira] [Updated] (KAFKA-5149) New producer hardcodes key and value serializers to ByteArray

[Yeva Byzek (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Yeva Byzek updated KAFKA-5149:
--
Labels: newbie  (was: )

> New producer hardcodes key and value serializers to ByteArray
> -
>
> Key: KAFKA-5149
> URL: https://issues.apache.org/jira/browse/KAFKA-5149
> Project: Kafka
>  Issue Type: Bug
>  Components: producer , tools
>Affects Versions: 0.10.2.0
>Reporter: Yeva Byzek
>  Labels: newbie
>
> New producer hardcodes the serializers:
> {noformat}
>   props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, 
> "org.apache.kafka.common.serialization.ByteArraySerializer")
> props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, 
> "org.apache.kafka.common.serialization.ByteArraySerializer")
> {noformat}
> And thus cannot be overridden from the commandline argument 
> {{--key-serializer}} or {{--value-serializer}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5149) New producer hardcodes key and value serializers to ByteArray

[Yeva Byzek (JIRA)]

Yeva Byzek created KAFKA-5149:
-

 Summary: New producer hardcodes key and value serializers to 
ByteArray
 Key: KAFKA-5149
 URL: https://issues.apache.org/jira/browse/KAFKA-5149
 Project: Kafka
  Issue Type: Bug
  Components: producer , tools
Affects Versions: 0.10.2.0
Reporter: Yeva Byzek


New producer hardcodes the serializers:

{noformat}
  props.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, 
"org.apache.kafka.common.serialization.ByteArraySerializer")
props.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, 
"org.apache.kafka.common.serialization.ByteArraySerializer")
{noformat}

And thus cannot be overridden from the commandline argument 
{{--key-serializer}} or {{--value-serializer}}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5148) Add configurable compression block size to the broker

[Dustin Cote (JIRA)]

Dustin Cote created KAFKA-5148:
--

 Summary: Add configurable compression block size to the broker
 Key: KAFKA-5148
 URL: https://issues.apache.org/jira/browse/KAFKA-5148
 Project: Kafka
  Issue Type: Improvement
  Components: compression
Reporter: Dustin Cote
 Fix For: 0.10.2.0


Similar to the discussion in KAFKA-3704, we should consider a configurable 
compression block size on the broker side. This especially considering the 
change in block size from 32KB to 1KB in the 0.10 release.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5147) KafkaProducer's TransactionManager needs a review on synchronization

[Apurva Mehta (JIRA)]

Apurva Mehta created KAFKA-5147:
---

 Summary: KafkaProducer's TransactionManager needs a review on 
synchronization
 Key: KAFKA-5147
 URL: https://issues.apache.org/jira/browse/KAFKA-5147
 Project: Kafka
  Issue Type: Bug
Reporter: Apurva Mehta
Assignee: Apurva Mehta


Currently, the completion handlers are not synchronized, though they access 
shared state like `partitionsInTransaction`, and `lastError`, 
`pendingPartitionsToBeaddedToTransaction`, etc. 

We should either make the collections concurrent or synchronize the handlers. 

In general, we need to review this code to ensure that the synchronization is 
correct. 



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Work started] (KAFKA-5126) Implement KIP-98 transactional methods in the MockProducer

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-5126 started by Matthias J. Sax.
--
> Implement KIP-98 transactional methods in the MockProducer
> --
>
> Key: KAFKA-5126
> URL: https://issues.apache.org/jira/browse/KAFKA-5126
> Project: Kafka
>  Issue Type: Test
>Reporter: Apurva Mehta
>Assignee: Matthias J. Sax
>
> The initial code for the transactional producer leaves the implementation of 
> `initTransactions`, `beginTransaction`, `sendOffsetsToTransaction`, 
> `commitTransaction`, and `abortTransaction` empty in the MockProducer. We 
> need have some implementation there so that our mocks stay healthy.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Assigned] (KAFKA-5126) Implement KIP-98 transactional methods in the MockProducer

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax reassigned KAFKA-5126:
--

Assignee: Matthias J. Sax  (was: Apurva Mehta)

> Implement KIP-98 transactional methods in the MockProducer
> --
>
> Key: KAFKA-5126
> URL: https://issues.apache.org/jira/browse/KAFKA-5126
> Project: Kafka
>  Issue Type: Test
>Reporter: Apurva Mehta
>Assignee: Matthias J. Sax
>
> The initial code for the transactional producer leaves the implementation of 
> `initTransactions`, `beginTransaction`, `sendOffsetsToTransaction`, 
> `commitTransaction`, and `abortTransaction` empty in the MockProducer. We 
> need have some implementation there so that our mocks stay healthy.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5055) Kafka Streams skipped-records-rate sensor producing nonzero values even when FailOnInvalidTimestamp is used as extractor

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-5055:
---
Status: Patch Available  (was: Open)

> Kafka Streams skipped-records-rate sensor producing nonzero values even when 
> FailOnInvalidTimestamp is used as extractor
> 
>
> Key: KAFKA-5055
> URL: https://issues.apache.org/jira/browse/KAFKA-5055
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Nikki Thean
>Assignee: Guozhang Wang
>
> According to the code and the documentation for this metric, the only reason 
> for a skipped record is an invalid timestamp, except that a) I am reading 
> from a topic that is populated solely by Kafka Connect and b) I am using 
> `FailOnInvalidTimestamp` as the timestamp extractor.
> Either I'm missing something in the documentation (i.e. another reason for 
> skipped records) or there is a bug in the code that calculates this metric.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface

[Matthias J. Sax]

Hi,

I am personally not a big fan of providing two APIs to do the same
thing. If we believe that one API is better than the other, we should
indicate this by deprecating the old API IMHO.

Just my two cents.


-Matthias


On 4/30/17 11:05 PM, Michael Pearce wrote:
> See
> 
> https://hc.apache.org/httpcomponents-client-ga/tutorial/html/fluent.html
> 
> Doesn't cause much issue over there where you have a fluent api wrapper for 
> those who like that style, and the original more verbose api.
> 
> Surely it would be better than removing a way of doing things that everyone 
> has got used to and built their code around ala constructors approach. And 
> simply provide a wrapper to provide a per field way of doing things.
> 
> 
> 
> 
> 
> Sent using OWA for iPhone
> 
> From: Stephane Maarek 
> Sent: Monday, May 1, 2017 6:37:44 AM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface
> 
> I’m not sure how people would feel about having two distinct methods to build 
> the same object?
> An API wrapper may be useful, but it doesn’t bring opinion about how one 
> should program, that’s just driven by the docs.
> I’m okay with that, but we need concensus
> 
> 
> On 1/5/17, 6:08 am, "Michael Pearce"  wrote:
> 
> Why not, instead of deprecating or removing whats there, as noted, its a 
> point of preference, think about something that could wrap the existing, but 
> provide an api that for you is cleaner?
> 
> e.g. here's a sample idea building on a fluent api way. (this wraps the 
> producer and producer records so no changes needed)
> 
> 
> https://gist.github.com/michaelandrepearce/de0f5ad4aa7d39d243781741c58c293e
> 
> In future as new items further add to Producer Record, they just become 
> new methods in the fluent API, as it builds the ProducerRecord using the most 
> exhaustive constructor.
> 
> 
> 
> 
> From: Matthias J. Sax 
> Sent: Saturday, April 29, 2017 6:52 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP 141 - ProducerRecordBuilder Interface
> 
> I understand that we cannot just break stuff (btw: also not for
> Streams!). But deprecating does not break anything, so I don't think
> it's a big deal to change the API as long as we keep the old API as
> deprecated.
> 
> 
> -Matthias
> 
> On 4/29/17 9:28 AM, Jay Kreps wrote:
> > Hey Matthias,
> >
> > Yeah I agree, I'm not against change as a general thing! I also think if
> > you look back on the last two years, we completely rewrote the producer 
> and
> > consumer APIs, reworked the binary protocol many times over, and added 
> the
> > connector and stream processing apis, both major new additions. So I 
> don't
> > think we're in too much danger of stagnating!
> >
> > My two cents was just around breaking compatibility for trivial changes
> > like constructor => builder. I think this only applies to the producer,
> > consumer, and connect apis which are heavily embedded in hundreds of
> > ecosystem components that depend on them. This is different from direct
> > usage. If we break the streams api it is really no big deal---apps just
> > need to rebuild when they upgrade, not the end of the world at all. 
> However
> > because many intermediate things depend on the Kafka producer you can 
> cause
> > these weird situations where your app depends on two third party things
> > that use Kafka and each requires different, incompatible versions. We 
> did
> > this a lot in earlier versions of Kafka and it was the cause of much 
> angst
> > (and an ingrained general reluctance to upgrade) from our users.
> >
> > I still think we may have to break things, i just don't think we should 
> do
> > it for things like builders vs direct constructors which i think are 
> kind
> > of a debatable matter of taste.
> >
> > -Jay
> >
> >
> >
> > On Mon, Apr 24, 2017 at 9:40 AM, Matthias J. Sax 
> > wrote:
> >
> >> Hey Jay,
> >>
> >> I understand your concern, and for sure, we will need to keep the
> >> current constructors deprecated for a long time (ie, many years).
> >>
> >> But if we don't make the move, we will not be able to improve. And I
> >> think warnings about using deprecated APIs is an acceptable price to
> >> pay. And the API improvements will help new people who adopt Kafka to
> >> get started more easily.
> >>
> >> Otherwise Kafka might end up as many other enterprise software with a
> >> lots of old stuff that is kept forever because nobody has the guts to
> >> improve/change it.
> >>
> >> Of course, we can still improve the docs of the deprecated 
> constructors,
> >> too.
> >>
> >> Just my two cents.
> >>
> >>
> >> -Matthias
> >>
> >> On 4/23/17 3:37 PM, Ja

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

[Matthias J. Sax]

Jeyhun,

thanks a lot for the KIP!

I think there are two issues we need to address:

(1) The KIP does not consider backward compatibility. Users did complain
about this in past releases already, and as the user base grows, we
should not break backward compatibility in future releases anymore.
Thus, we should think of a better way to allow key access.

Mathieu's comment goes into the same direction

>> On the other hand, the number of compile failures that would need to be
>> fixed from this change is unfortunate. :-)  

(2) Another concern is, that there is no guard to prevent user code to
modify the key. This might corrupt partitioning if users do alter the
key (accidentally -- or users are just not aware that they are not
allowed to modify the provided key object) and thus break the
application. (This was the original motivation to not provide the key in
the first place -- it's guards against modification.)


-Matthias



On 5/1/17 6:31 AM, Mathieu Fenniak wrote:
> Hi Jeyhun,
> 
> I just want to add my voice that, I too, have wished for access to the
> record key during a mapValues or similar operation.
> 
> On the other hand, the number of compile failures that would need to be
> fixed from this change is unfortunate. :-)  But at least it would all be a
> pretty clear and easy change.
> 
> Mathieu
> 
> 
> On Mon, May 1, 2017 at 6:55 AM, Jeyhun Karimov  wrote:
> 
>> Dear community,
>>
>> I want to share KIP-149 [1] based on issues KAFKA-4218 [2], KAFKA-4726 [3],
>> KAFKA-3745 [4]. The related PR can be found at [5].
>> I would like to get your comments.
>>
>> [1]
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>> 149%3A+Enabling+key+access+in+ValueTransformer%2C+
>> ValueMapper%2C+and+ValueJoiner
>> [2] https://issues.apache.org/jira/browse/KAFKA-4218
>> [3] https://issues.apache.org/jira/browse/KAFKA-4726
>> [4] https://issues.apache.org/jira/browse/KAFKA-3745
>> [5] https://github.com/apache/kafka/pull/2946
>>
>>
>> Cheers,
>> Jeyhun
>>
>>
>>
>> --
>> -Cheers
>>
>> Jeyhun
>>
> 



signature.asc
Description: OpenPGP digital signature

Re: [DISCUSS] KIP-145: Classloading Isolation in Connect

[Colin McCabe]

+1 (non-binding)

This would be a nice improvement.

C.

On Fri, Apr 28, 2017, at 11:03, Konstantine Karantasis wrote:
> Hi everyone,
> 
> we aim to address dependency conflicts in Kafka Connect soon by applying
> class loading isolation.
> 
> Feel free to take a look at KIP-145 here:
> 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-145+Classloading+Isolation+in+Connect
> 
> 
> which describes minimal required changes in the public interfaces and the
> general implementation approach.
> 
> This is a much wanted feature for Kafka Connect. Your feedback is highly
> appreciated.
> 
> -Konstantine

Re: [DISCUSS] KIP-144: Exponential backoff for broker reconnect attempts

[Colin McCabe]

Thanks for the KIP, Ismael & Dana!  This could be pretty important for
avoiding congestion collapse when there are a lot of clients.

It seems like a good idea to keep the "ms" suffix, like we have with
"reconnect.backoff.ms".  So maybe we should use
"reconnect.backoff.max.ms"?  In general unitless timeouts can be the
source of a lot of confusion (is it seconds, milliseconds, etc.?)

It's good that the KIP inject random delays (jitter) into the timeout. 
As per Gwen's point, does it make sense to put an upper bound on the
jitter, though?  If someone sets reconnect.backoff.max to 5 minutes,
they probably would be a little surprised to find it doing three retries
after 100 ms in a row (as it could under the current scheme.)  Maybe a
maximum jitter configuration would help address that, and make the
behavior a little more intuitive.

best,
Colin


On Thu, Apr 27, 2017, at 09:39, Gwen Shapira wrote:
> This is a great suggestion. I like how we just do it by default instead
> of
> making it a choice users need to figure out.
> Avoiding connection storms is great.
> 
> One concern. If I understand the formula for effective maximum backoff
> correctly, then with default maximum of 1000ms and default backoff of
> 100ms, the effective maximum backoff will be 450ms rather than 1000ms.
> This
> isn't exactly intuitive.
> I'm wondering if it makes more sense to allow "one last doubling" which
> may
> bring us slightly over the maximum, but much closer to it. I.e. have the
> effective maximum be in [max.backoff - backoff, max.backoff + backoff]
> range rather than half that. Does that make sense?
> 
> Gwen
> 
> On Thu, Apr 27, 2017 at 9:06 AM, Ismael Juma  wrote:
> 
> > Hi all,
> >
> > Dana Powers posted a PR a while back for exponential backoff for broker
> > reconnect attempts. Because it adds a config, a KIP is required and Dana
> > seems to be busy so I posted it:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 144%3A+Exponential+backoff+for+broker+reconnect+attempts
> >
> > Please take a look. Your feedback is appreciated.
> >
> > Thanks,
> > Ismael
> >
> 
> 
> 
> -- 
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter  | blog
> 

Re: Kafka-Streams: Cogroup

[Guozhang Wang]

Kyle,

What's your apache id? I can grant you the permission.


Guozhang


On Sat, Apr 29, 2017 at 7:33 AM, Kyle Winkelman 
wrote:

> I don't seem to have permission. When logged in I can neither edit the
> main page nor create an additional KIP.
>
> Thanks,
> Kyle
>
> On Thu, Apr 27, 2017 at 12:35 PM, Eno Thereska 
> wrote:
>
>> Hi Kyle,
>>
>> I believe Guozhang has now given you permission to edit the KIP wiki at
>> https://cwiki.apache.org/confluence/display/KAFKA/Kafka+
>> Improvement+Proposals. Could you see if you can add this there?
>>
>> Many thanks
>> Eno
>>
>> On Wed, Apr 26, 2017 at 6:00 PM, Kyle Winkelman > > wrote:
>>
>>> Thank you for your reply.
>>>
>>> I have attached my first attempt at writing a KIP and I was wondering if
>>> you could review it and share your thoughts.
>>>
>>> Going forward I would like to create this KIP. I was wondering whom I
>>> should ask to get the necessary permissions on the wiki. Username:
>>> winkelman.kyle
>>>
>>>
>>>
>>> On Fri, Apr 21, 2017 at 3:15 PM, Eno Thereska 
>>> wrote:
>>>
 Hi Kyle,

 Sorry for the delay in replying. I think it's worth doing a KIP for
 this one. One super helpful thing with KIPs is to list a few more scenarios
 that would benefit from this approach. In particular it seems the main
 benefit is from reducing the number of state stores. Does this necessarily
 reduce the number of IOs to the stores (number of puts/gets), or the extra
 space overheads with multiple stores. Quantifying that a bit would help.

 To answer your original questions:

 >The problem I am having with this approach is understanding if there
 is a race condition. Obviously the source topics would be copartitioned.
 But would it be multithreaded and possibly cause one of the processors to
 grab patient 1 at the same time a different processor has grabbed patient 
 1?


 I don't think there will be a problem here. A processor cannot be
 accessed by multiple threads in Kafka Streams.


 >My understanding is that for each partition there would be a single
 complete set of processors and a new incoming record would go completely
 through the processor topology from a source node to a sink node before the
 next one is sent through. Is this correct?

 This is mostly true, however if caching is enabled (for dedupping, see
 KIP-63), then a record may reside in a cache before going to the sink.
 Meanwhile another record can come in. So multiple records can be in the
 topology at the same time.

 Thanks
 Eno





 On Fri, Apr 14, 2017 at 8:16 PM, Kyle Winkelman <
 winkelman.k...@gmail.com> wrote:

> Eno,
> Thanks for the response. The figure was just a restatement of my
> questions. I have made an attempt at a low level processor and it appears
> to work but it isn't very pretty and was hoping for something at the
> streams api level.
>
> I have written some code to show an example of how I see the Cogroup
> working in kafka.
>
> First the KGroupedStream would have a cogroup method that takes the
> initializer and the aggregator for that specific KGroupedStream. This 
> would
> return a KCogroupedStream that has 2 methods one to add more
> KGroupedStream, Aggregator pairs and one to complete the construction and
> return a KTable.
>
> builder.stream("topic").groupByKey ().cogroup(Initializer,
> Aggregator, aggValueSerde, storeName).cogroup(groupedStream1,
> Aggregator1).cogroup(groupedStream2, Aggregator2).aggregate();
>
> Behind the scenes we create a KStreamAggregate for each
> KGroupedStream, Aggregator pair. Then a final pass through processor to
> pass on the aggregate values. This gives us a KTable backed by a single
> store that is used in all of the processors.
>
> Please let me know if this is something you think would add value to
> kafka streams. And I will try to create a KIP to foster more 
> communication.
>
> You can take a look at what I have. I think it's missing a fair amount
> but it's a good start. I took the doAggregate method in KGroupedStream as
> my starting point and expanded on it for multiple streams:
> https://github.com/KyleWinkelman/kafka/tree/cogroup
>
>

>>>
>>
>


-- 
-- Guozhang

Re: [DISCUSS] KIP-146: Classloading Isolation in Connect

[Randall Hauch]

Very nice work, Konstantine. Conflicting dependencies of connectors is
indeed a big issue that makes it hard to manage installed connectors.

I do like Gwen's idea about removing the 'module.isolation.enabled'
property. However, I would have anticipated always using classpath
isolation for *only* those components registered under the module path and
not really for anything else already on the normal classpath. So, people
could continue to place custom connector JARs onto the classpath, though
this would become deprecated in favor of installing custom connector JARs /
modules via the module path. This keeps configuration simple, gives people
time to migrate, but let's people that need classpath isolation get it to
install a variety of connectors each with their dependencies that
potentially conflict with other components.

The challenge is whether there should be a default for 'module.path'.
Ideally there would be so that users know where they can install their
connectors. However, I suspect that this might be difficult to do unless it
can make use of system properties such as "${kafka.home}" so that relative
directories can be specified.

A few other questions/comments:

1) Does the KIP have to specify how are components / modules installed,
discovered, or recognized by Kafka Connect? Or perhaps the KIP needs to
just specify the semantics of the file system module path (e.g., the
directories below those specified in the module path are to be unique and
identify an installed component).

2) Will the module classloader filtering also have to exclude Kafka Connect
dependencies? The only one that I can think of is the SLF4J API, which
can't be loaded from the module's classloader if the connector is to send
its log messages to the same logging system.

3) Rather than specify filtering, would be it a bit more flexible to simply
say that the implementation will need to ensure that Java, Kafka Connect,
and other third party APIs (e.g., SLF4J API) will not be loaded from the
module classloaders? It'd be better to avoid specifying how it will be
done, just in case the implementation needs to evolve or use a different
technique (e.g., load the Java and public Kafka Connect APIs via one
classloader that is reused and that always appears before the module
classloader, while Kafka Connect implementation JARs appear after the
component's classloader.

4) Perhaps to address #2 and #3 above, perhaps the KIP could explicitly
specify the classloader order for a deployed connector. For example,
'java', 'kafka-connect-apis', 'connector-module', 'smt-module-1', ...,
'kafka-connect-impls', where 'connector-module' is the classloader for the
(first) module where the connector is found, 'smt-module-1' is the
classloader for the (first) module where the first SMT class is found (if
specified and found in a separate module), 'smt-module-2' is the
classloader  Might also need to say that the KIP does not specify how
the implementation will pick the module if a specified class if found in
more than one module.

Thoughts?

Randall

On Mon, May 1, 2017 at 6:43 AM, Gwen Shapira  wrote:

> Hi Konstantine,
>
> Thank you so much for driving this! The connector classpath mess is driving
> me nuts (or worse, driving me to use Docker).
>
> I like the proposal for micro-benchmarks to test the context switching
> overhead.
>
> I have a difficult time figuring out the module.isolation.enabled.
> Especially with a default to false. I can't think of a reason that anyone
> will not want classpath isolation. "No! I want my connectors to mess up
> each other's dependencies" said no one ever.
>
> So it looks like this is mostly for upgrade purpose? Because the initial
> upgrade will not have the module.path set and therefore classpath isolation
> will simply not work by default?
>
> In that case, why don't we simply use the existence of non-empty
> module.path as an indicator of whether isolation should work or not? seem
> simpler and intuitive to me.
>
> Thanks!
>
> Gwen
>
>
>
>
>
> On Sat, Apr 29, 2017 at 9:16 AM, Konstantine Karantasis <
> konstant...@confluent.io> wrote:
>
> > * Because of KIP number collision, please disregard my previous KIP
> > announcement and use this thread for discussion instead *
> >
> >
> > Hi everyone,
> >
> > we aim to address dependency conflicts in Kafka Connect soon by applying
> > class loading isolation.
> >
> > Feel free to take a look at KIP-146 here:
> >
> > *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > 146+-+Classloading+Isolation+in+Connect
> >  > 146+-+Classloading+Isolation+in+Connect>*
> >
> > which describes minimal required changes to public interfaces and the
> > general implementation approach.
> >
> > This is a much wanted feature for Kafka Connect. Your feedback is highly
> > appreciated.
> >
> > -Konstantine
> >
>
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter 

Re: Edit Permissions for KIP Wiki

[Jun Rao]

Hi, Kyle,

Thanks for your interest. Just gave the permission to the wiki.

Jun

On Mon, May 1, 2017 at 4:43 AM, Kyle Winkelman 
wrote:

> Hello,
>
> I was hoping to get the ability to edit the wiki so I can post a KIP.
>
> Username: winkelman.kyle
>
> Thanks,
> Kyle
>

Re: Support for Kafka Consumer over SASL_SSL

[Michal Borowiecki]

Hi Nixon,

kafka.consumer.Consumer is the old consumer.
When the documentation is referring to the new producer and consumer, it 
means those in the org.apache.kafka.clients.producer and 
org.apache.kafka.clients.consumer packages respectively.

They are packaged in the kafka-clients-0.10.x.y.jar

Hope that helps,
Michał

On 01/05/17 14:04, Nixon Rodrigues wrote:

   Hi Kafka dev team,


I am using  kafka_2.11-0.10.0.0.jar api for kafka consumer,

i am facing issue while consuming from topics over SASL_SSL,
getting below exception

Consumer.createJavaConsumerConnector(new
kafka.consumer.ConsumerConfig(consumerProperties))


As per documentation found on
http://docs.confluent.io/2.0.0/kafka/sasl.html  , it is said that

"SASL authentication is only supported for the new Kafka producer
and consumer, the older API is not supported."

Can anybody confirm me which kafka version onward SASl_SSL is
supported, any input on this is appreciated.


2017-04-26 16:06:36,655 WARN  -
[node11.openstacklocal-1493222790718-b862352a-leader-finder-thread:] ~
Fetching topic metadata with correlation id 18 for topics [Set(TEST)]
from broker [BrokerEndPoint(1001,node11.openstacklocal,6667)] failed
(Logging$class:89)
java.io.EOFException
at 
org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:99)
at kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:140)
at kafka.network.BlockingChannel.receive(BlockingChannel.scala:131)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:84)
at 
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:81)
at kafka.producer.SyncProducer.send(SyncProducer.scala:126)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:96)
at 
kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:67)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
2017-04-26 16:06:36,655 WARN  -
[node11.openstacklocal-1493222790718-b862352a-leader-finder-thread:] ~
[node11.openstacklocal-1493222790718-b862352a-leader-finder-thread],
Failed to find leader for Set([TEST,0]) (Logging$class:89)
kafka.common.KafkaException: fetching topic metadata for topics
[Set(TEST)] from broker
[ArrayBuffer(BrokerEndPoint(1001,node11.openstacklocal,6667))] failed
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:73)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:96)
at 
kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:67)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Caused by: java.io.EOFException
at 
org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:99)
at kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:140)
at kafka.network.BlockingChannel.receive(BlockingChannel.scala:131)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:84)
at 
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:81)
at kafka.producer.SyncProducer.send(SyncProducer.scala:126)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)


Nixon



--
Signature
 Michal Borowiecki
Senior Software Engineer L4
T:  +44 208 742 1600


+44 203 249 8448



E:  michal.borowie...@openbet.com
W:  www.openbet.com 


OpenBet Ltd

Chiswick Park Building 9

566 Chiswick High Rd

London

W4 5XT

UK




This message is confidential and intended only for the addressee. If you 
have received this message in error, please immediately notify the 
postmas...@openbet.com  and delete it 
from your system as well as any copies. The content of e-mails as well 
as traffic data may be monitored by OpenBet for employment and security 
purposes. To protect the environment please do not print this e-mail 
unless necessary. OpenBet Ltd. Registered Office: Chiswick Park Building 
9, 566 Chiswick High Road, London, W4 5XT, United Kingdom. A company 
registered in England and Wales. Registered no. 3134634. VAT no. 
GB927523612

[jira] [Commented] (KAFKA-4954) Implementation and unit tests for request handler thread utilization quota

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15991013#comment-15991013
 ] 

ASF GitHub Bot commented on KAFKA-4954:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2744


> Implementation and unit tests for request handler thread utilization quota
> --
>
> Key: KAFKA-4954
> URL: https://issues.apache.org/jira/browse/KAFKA-4954
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.11.0.0
>
>
> Implement request quotas as described in 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-124+-+Request+rate+quotas.
> Network handler thread utilization will be handled in a separate sub-task.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2744: KAFKA-4954: Request handler utilization quotas

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2744


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-4954) Implementation and unit tests for request handler thread utilization quota

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao updated KAFKA-4954:
---
Resolution: Fixed
Status: Resolved  (was: Patch Available)

Issue resolved by pull request 2744
[https://github.com/apache/kafka/pull/2744]

> Implementation and unit tests for request handler thread utilization quota
> --
>
> Key: KAFKA-4954
> URL: https://issues.apache.org/jira/browse/KAFKA-4954
> Project: Kafka
>  Issue Type: Sub-task
>  Components: core
>Reporter: Rajini Sivaram
>Assignee: Rajini Sivaram
> Fix For: 0.11.0.0
>
>
> Implement request quotas as described in 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-124+-+Request+rate+quotas.
> Network handler thread utilization will be handled in a separate sub-task.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5144) MinTimestampTracker uses confusing variable names

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990952#comment-15990952
 ] 

Matthias J. Sax commented on KAFKA-5144:


Makes sense. Thanks for your contribution!

> MinTimestampTracker uses confusing variable names
> -
>
> Key: KAFKA-5144
> URL: https://issues.apache.org/jira/browse/KAFKA-5144
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Michal Borowiecki
>Assignee: Michal Borowiecki
>Priority: Trivial
> Fix For: 0.11.0.0
>
>
> --When adding elements MinTimestampTracker removes all existing elements 
> greater than the added element.--
> --Perhaps I've missed something and this is intended behaviour but I can't 
> find any evidence for that in comments or tests.--
> {{MinTimestampTracker}} maintains a list of elements ascending by timestamp, 
> but calls the list var {{descendingSubsequence}} -- it also get the largest 
> element at the end and stores it in a var called {{minElem}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5144) MinTimestampTracker uses confusing variable names

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-5144:
---
Description: 
--When adding elements MinTimestampTracker removes all existing elements 
greater than the added element.--
--Perhaps I've missed something and this is intended behaviour but I can't find 
any evidence for that in comments or tests.--

{{MinTimestampTracker}} maintains a list of elements ascending by timestamp, 
but calls the list var {{descendingSubsequence}} -- it also get the largest 
element at the end and stores it in a var called {{minElem}}.

  was:
--When adding elements MinTimestampTracker removes all existing elements 
greater than the added element.
Perhaps I've missed something and this is intended behaviour but I can't find 
any evidence for that in comments or tests.--

{{MinTimestampTracker}} maintains a list of elements ascending by timestamp, 
but call the list var {{descendingSubsequence}} -- it also get the largest 
element at the end and stores in a var called {{minElem}}.


> MinTimestampTracker uses confusing variable names
> -
>
> Key: KAFKA-5144
> URL: https://issues.apache.org/jira/browse/KAFKA-5144
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Michal Borowiecki
>Assignee: Michal Borowiecki
>Priority: Trivial
> Fix For: 0.11.0.0
>
>
> --When adding elements MinTimestampTracker removes all existing elements 
> greater than the added element.--
> --Perhaps I've missed something and this is intended behaviour but I can't 
> find any evidence for that in comments or tests.--
> {{MinTimestampTracker}} maintains a list of elements ascending by timestamp, 
> but calls the list var {{descendingSubsequence}} -- it also get the largest 
> element at the end and stores it in a var called {{minElem}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5144) MinTimestampTracker uses confusing variable names

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-5144:
---
Description: 
--When adding elements MinTimestampTracker removes all existing elements 
greater than the added element.
Perhaps I've missed something and this is intended behaviour but I can't find 
any evidence for that in comments or tests.--

{{MinTimestampTracker}} maintains a list of elements ascending by timestamp, 
but call the list var {{descendingSubsequence}} -- it also get the largest 
element at the end and stores in a var called {{minElem}}.

  was:
When adding elements MinTimestampTracker removes all existing elements greater 
than the added element.
Perhaps I've missed something and this is intended behaviour but I can't find 
any evidence for that in comments or tests.


> MinTimestampTracker uses confusing variable names
> -
>
> Key: KAFKA-5144
> URL: https://issues.apache.org/jira/browse/KAFKA-5144
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Michal Borowiecki
>Assignee: Michal Borowiecki
>Priority: Trivial
> Fix For: 0.11.0.0
>
>
> --When adding elements MinTimestampTracker removes all existing elements 
> greater than the added element.
> Perhaps I've missed something and this is intended behaviour but I can't find 
> any evidence for that in comments or tests.--
> {{MinTimestampTracker}} maintains a list of elements ascending by timestamp, 
> but call the list var {{descendingSubsequence}} -- it also get the largest 
> element at the end and stores in a var called {{minElem}}.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5144) MinTimestampTracker does not correctly add timestamps lower than the current max

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-5144:
---
Fix Version/s: 0.11.0.0
   Status: Patch Available  (was: Open)

> MinTimestampTracker does not correctly add timestamps lower than the current 
> max
> 
>
> Key: KAFKA-5144
> URL: https://issues.apache.org/jira/browse/KAFKA-5144
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Michal Borowiecki
>Assignee: Michal Borowiecki
> Fix For: 0.11.0.0
>
>
> When adding elements MinTimestampTracker removes all existing elements 
> greater than the added element.
> Perhaps I've missed something and this is intended behaviour but I can't find 
> any evidence for that in comments or tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5144) MinTimestampTracker does not correctly add timestamps lower than the current max

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-5144:
---
Priority: Trivial  (was: Major)

> MinTimestampTracker does not correctly add timestamps lower than the current 
> max
> 
>
> Key: KAFKA-5144
> URL: https://issues.apache.org/jira/browse/KAFKA-5144
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Michal Borowiecki
>Assignee: Michal Borowiecki
>Priority: Trivial
> Fix For: 0.11.0.0
>
>
> When adding elements MinTimestampTracker removes all existing elements 
> greater than the added element.
> Perhaps I've missed something and this is intended behaviour but I can't find 
> any evidence for that in comments or tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5144) MinTimestampTracker uses confusing variable names

[Matthias J. Sax (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Matthias J. Sax updated KAFKA-5144:
---
Summary: MinTimestampTracker uses confusing variable names  (was: 
MinTimestampTracker does not correctly add timestamps lower than the current 
max)

> MinTimestampTracker uses confusing variable names
> -
>
> Key: KAFKA-5144
> URL: https://issues.apache.org/jira/browse/KAFKA-5144
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Michal Borowiecki
>Assignee: Michal Borowiecki
>Priority: Trivial
> Fix For: 0.11.0.0
>
>
> When adding elements MinTimestampTracker removes all existing elements 
> greater than the added element.
> Perhaps I've missed something and this is intended behaviour but I can't find 
> any evidence for that in comments or tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5096) Only log invalid user configs and overwrite with correct one

[Mariam John (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Mariam John updated KAFKA-5096:
---
Reviewer: Matthias J. Sax
  Status: Patch Available  (was: Open)

> Only log invalid user configs and overwrite with correct one
> 
>
> Key: KAFKA-5096
> URL: https://issues.apache.org/jira/browse/KAFKA-5096
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Mariam John
>Priority: Minor
>  Labels: beginner, newbie
>
> Streams does not allow to overwrite some config parameters (eg, 
> {{enable.auto.commit}}) Currently, we throw an exception, but this is 
> actually not required, as Streams can just ignore/overwrite the user provided 
> value.
> Thus, instead of throwing, we should just log a WARN message and overwrite 
> the config with the values that suits Streams. (atm it's only one parameter 
> {{enable.auto.commit}}), but with exactly-once it's going to be more (cf. 
> KAFKA-4923). Thus, the scope of this ticket depends when it will be 
> implemented (ie, before or after KAFKA-4923).
> This ticket should also include JavaDoc updates that explain what parameters 
> cannot be specified by the user.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5096) Only log invalid user configs and overwrite with correct one

[Mariam John (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990902#comment-15990902
 ] 

Mariam John commented on KAFKA-5096:


[~mjsax] This is ready for review. All the tests passed. Appreciate your time 
and feedback.

> Only log invalid user configs and overwrite with correct one
> 
>
> Key: KAFKA-5096
> URL: https://issues.apache.org/jira/browse/KAFKA-5096
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Mariam John
>Priority: Minor
>  Labels: beginner, newbie
>
> Streams does not allow to overwrite some config parameters (eg, 
> {{enable.auto.commit}}) Currently, we throw an exception, but this is 
> actually not required, as Streams can just ignore/overwrite the user provided 
> value.
> Thus, instead of throwing, we should just log a WARN message and overwrite 
> the config with the values that suits Streams. (atm it's only one parameter 
> {{enable.auto.commit}}), but with exactly-once it's going to be more (cf. 
> KAFKA-4923). Thus, the scope of this ticket depends when it will be 
> implemented (ie, before or after KAFKA-4923).
> This ticket should also include JavaDoc updates that explain what parameters 
> cannot be specified by the user.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Re: [DISCUSS] KIP-146: Classloading Isolation in Connect

[Gwen Shapira]

Hi Konstantine,

Thank you so much for driving this! The connector classpath mess is driving
me nuts (or worse, driving me to use Docker).

I like the proposal for micro-benchmarks to test the context switching
overhead.

I have a difficult time figuring out the module.isolation.enabled.
Especially with a default to false. I can't think of a reason that anyone
will not want classpath isolation. "No! I want my connectors to mess up
each other's dependencies" said no one ever.

So it looks like this is mostly for upgrade purpose? Because the initial
upgrade will not have the module.path set and therefore classpath isolation
will simply not work by default?

In that case, why don't we simply use the existence of non-empty
module.path as an indicator of whether isolation should work or not? seem
simpler and intuitive to me.

Thanks!

Gwen





On Sat, Apr 29, 2017 at 9:16 AM, Konstantine Karantasis <
konstant...@confluent.io> wrote:

> * Because of KIP number collision, please disregard my previous KIP
> announcement and use this thread for discussion instead *
>
>
> Hi everyone,
>
> we aim to address dependency conflicts in Kafka Connect soon by applying
> class loading isolation.
>
> Feel free to take a look at KIP-146 here:
>
> *https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 146+-+Classloading+Isolation+in+Connect
>  146+-+Classloading+Isolation+in+Connect>*
>
> which describes minimal required changes to public interfaces and the
> general implementation approach.
>
> This is a much wanted feature for Kafka Connect. Your feedback is highly
> appreciated.
>
> -Konstantine
>



-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter  | blog

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

[Gwen Shapira]

Hi,

I'm excited to see the community expanding Connect in this direction!
Headers + Transforms == Fun message routing.

I like how clean the proposal is, but I'm concerned that it kinda deviates
from how Connect handles data elsewhere.
Unlike Kafka, Connect doesn't look at all data as byte-arrays, we have
converters that take data in specific formats (JSON, Avro) and turns it
into Connect data types (defined in the data api). I think it will be more
consistent for connector developers to also get headers as some kind of
structured or semi-structured data (and to expand the converters to handle
header conversions as well).
This will allow for Connect's separation of concerns - Connector developers
don't worry about data formats (because they get the internal connect
objects) and Converters do all the data format work.

Another thing, in my experience, APIs work better if they are put into use
almost immediately - so difficulties in using the APIs are immediately
surfaced. Are you planning any connectors that will use this feature (not
necessarily in Kafka, just in general)? Or perhaps we can think of a way to
expand Kafka's file connectors so they'll use headers somehow (can't think
of anything, but maybe?).

Gwen

On Sat, Apr 29, 2017 at 12:12 AM, Michael Pearce 
wrote:

> Hi All,
>
> Now KIP-82 is committed I would like to discuss extending the work to
> expose it in Kafka Connect, its primary focus being so connectors that may
> do similar tasks as MirrorMakers, either Kafka->Kafka or JMS-Kafka would be
> able to replicate the headers.
> It would be ideal but not mandatory for this to go in 0.11 release so is
> available on day one of headers being available.
>
> Please find the KIP here:
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 145+-+Expose+Record+Headers+in+Kafka+Connect
>
> Please find an initial implementation as a PR here:
> https://github.com/apache/kafka/pull/2942
>
> Kind Regards
> Mike
> The information contained in this email is strictly confidential and for
> the use of the addressee only, unless otherwise indicated. If you are not
> the intended recipient, please do not read, copy, use or disclose to others
> this message or any attachment. Please also notify the sender by replying
> to this email or by telephone (+44(020 7896 0011) and then delete the email
> and any copies of it. Opinions, conclusion (etc) that do not relate to the
> official business of this company shall be understood as neither given nor
> endorsed by it. IG is a trading name of IG Markets Limited (a company
> registered in England and Wales, company number 04008957) and IG Index
> Limited (a company registered in England and Wales, company number
> 01190902). Registered address at Cannon Bridge House, 25 Dowgate Hill,
> London EC4R 2YA. Both IG Markets Limited (register number 195355) and IG
> Index Limited (register number 114059) are authorised and regulated by the
> Financial Conduct Authority.
>



-- 
*Gwen Shapira*
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter  | blog

Re: [DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

[Mathieu Fenniak]

Hi Jeyhun,

I just want to add my voice that, I too, have wished for access to the
record key during a mapValues or similar operation.

On the other hand, the number of compile failures that would need to be
fixed from this change is unfortunate. :-)  But at least it would all be a
pretty clear and easy change.

Mathieu


On Mon, May 1, 2017 at 6:55 AM, Jeyhun Karimov  wrote:

> Dear community,
>
> I want to share KIP-149 [1] based on issues KAFKA-4218 [2], KAFKA-4726 [3],
> KAFKA-3745 [4]. The related PR can be found at [5].
> I would like to get your comments.
>
> [1]
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> 149%3A+Enabling+key+access+in+ValueTransformer%2C+
> ValueMapper%2C+and+ValueJoiner
> [2] https://issues.apache.org/jira/browse/KAFKA-4218
> [3] https://issues.apache.org/jira/browse/KAFKA-4726
> [4] https://issues.apache.org/jira/browse/KAFKA-3745
> [5] https://github.com/apache/kafka/pull/2946
>
>
> Cheers,
> Jeyhun
>
>
>
> --
> -Cheers
>
> Jeyhun
>

Support for Kafka Consumer over SASL_SSL

[Nixon Rodrigues]

  Hi Kafka dev team,


   I am using  kafka_2.11-0.10.0.0.jar api for kafka consumer,

   i am facing issue while consuming from topics over SASL_SSL,
getting below exception

   Consumer.createJavaConsumerConnector(new
kafka.consumer.ConsumerConfig(consumerProperties))


   As per documentation found on
http://docs.confluent.io/2.0.0/kafka/sasl.html  , it is said that

   "SASL authentication is only supported for the new Kafka producer
and consumer, the older API is not supported."

   Can anybody confirm me which kafka version onward SASl_SSL is
supported, any input on this is appreciated.


2017-04-26 16:06:36,655 WARN  -
[node11.openstacklocal-1493222790718-b862352a-leader-finder-thread:] ~
Fetching topic metadata with correlation id 18 for topics [Set(TEST)]
from broker [BrokerEndPoint(1001,node11.openstacklocal,6667)] failed
(Logging$class:89)
java.io.EOFException
at 
org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:99)
at kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:140)
at kafka.network.BlockingChannel.receive(BlockingChannel.scala:131)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:84)
at 
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:81)
at kafka.producer.SyncProducer.send(SyncProducer.scala:126)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:96)
at 
kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:67)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
2017-04-26 16:06:36,655 WARN  -
[node11.openstacklocal-1493222790718-b862352a-leader-finder-thread:] ~
[node11.openstacklocal-1493222790718-b862352a-leader-finder-thread],
Failed to find leader for Set([TEST,0]) (Logging$class:89)
kafka.common.KafkaException: fetching topic metadata for topics
[Set(TEST)] from broker
[ArrayBuffer(BrokerEndPoint(1001,node11.openstacklocal,6667))] failed
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:73)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:96)
at 
kafka.consumer.ConsumerFetcherManager$LeaderFinderThread.doWork(ConsumerFetcherManager.scala:67)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
Caused by: java.io.EOFException
at 
org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:99)
at kafka.network.BlockingChannel.readCompletely(BlockingChannel.scala:140)
at kafka.network.BlockingChannel.receive(BlockingChannel.scala:131)
at kafka.producer.SyncProducer.liftedTree1$1(SyncProducer.scala:84)
at 
kafka.producer.SyncProducer.kafka$producer$SyncProducer$$doSend(SyncProducer.scala:81)
at kafka.producer.SyncProducer.send(SyncProducer.scala:126)
at kafka.client.ClientUtils$.fetchTopicMetadata(ClientUtils.scala:59)


Nixon

[DISCUSS]: KIP-149: Enabling key access in ValueTransformer, ValueMapper, and ValueJoiner

[Jeyhun Karimov]

Dear community,

I want to share KIP-149 [1] based on issues KAFKA-4218 [2], KAFKA-4726 [3],
KAFKA-3745 [4]. The related PR can be found at [5].
I would like to get your comments.

[1]
https://cwiki.apache.org/confluence/display/KAFKA/KIP-149%3A+Enabling+key+access+in+ValueTransformer%2C+ValueMapper%2C+and+ValueJoiner
[2] https://issues.apache.org/jira/browse/KAFKA-4218
[3] https://issues.apache.org/jira/browse/KAFKA-4726
[4] https://issues.apache.org/jira/browse/KAFKA-3745
[5] https://github.com/apache/kafka/pull/2946


Cheers,
Jeyhun



-- 
-Cheers

Jeyhun

Edit Permissions for KIP Wiki

[Kyle Winkelman]

Hello,

I was hoping to get the ability to edit the wiki so I can post a KIP.

Username: winkelman.kyle

Thanks,
Kyle

[jira] [Updated] (KAFKA-5146) Kafka Streams: remove compile dependency on connect-json

[Michael Noll (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Noll updated KAFKA-5146:

Description: 
We currently have a compile-dependency on `connect-json`:

{code}

  org.apache.kafka
  connect-json
  0.10.2.0
  compile
  
{code}

The snippet above is from the generated POM of Kafka Streams as of 0.10.2.0 
release.

AFAICT the only reason for that is because the *examples* showcase some JSON 
processing, but that’s it.

First and foremost, we should remove the connect-json dependency, and also 
figure out a way to set up / structure the examples so we that we can continue 
showcasing JSON support.  Alternatively, we could consider removing the JSON 
example (but I don't like that, personally).

  was:
We currently have a compile-dependency on `connect-json`:

{code}

org.apache.kafka
connect-json
0.10.2.0
compile

{code}

The snippet above is from the generated POM of Kafka Streams as of 0.10.2.0 
release.

AFAICT the only reason for that is because the *examples* showcase some JSON 
processing, but that’s it.

First and foremost, we should remove the connect-json dependency, and also 
figure out a way to set up / structure the examples so we that we can continue 
showcasing JSON support.  Alternatively, we could consider removing the JSON 
example (but I don't like that, personally).


> Kafka Streams: remove compile dependency on connect-json
> 
>
> Key: KAFKA-5146
> URL: https://issues.apache.org/jira/browse/KAFKA-5146
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0, 0.10.2.0, 0.10.2.1
>Reporter: Michael Noll
>Priority: Minor
>
> We currently have a compile-dependency on `connect-json`:
> {code}
> 
>   org.apache.kafka
>   connect-json
>   0.10.2.0
>   compile
>   
> {code}
> The snippet above is from the generated POM of Kafka Streams as of 0.10.2.0 
> release.
> AFAICT the only reason for that is because the *examples* showcase some JSON 
> processing, but that’s it.
> First and foremost, we should remove the connect-json dependency, and also 
> figure out a way to set up / structure the examples so we that we can 
> continue showcasing JSON support.  Alternatively, we could consider removing 
> the JSON example (but I don't like that, personally).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5146) Kafka Streams: remove compile dependency on connect-json

[Michael Noll (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Noll updated KAFKA-5146:

Affects Version/s: 0.10.2.1
   0.10.1.0
   0.10.2.0

> Kafka Streams: remove compile dependency on connect-json
> 
>
> Key: KAFKA-5146
> URL: https://issues.apache.org/jira/browse/KAFKA-5146
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0, 0.10.2.0, 0.10.2.1
>Reporter: Michael Noll
>Priority: Minor
>
> We currently have a compile-dependency on `connect-json`:
> {code}
> 
>   org.apache.kafka
>   connect-json
>   0.10.2.0
>   compile
>   
> {code}
> The snippet above is from the generated POM of Kafka Streams as of 0.10.2.0 
> release.
> AFAICT the only reason for that is because the Kafka Streams *examples* 
> showcase some JSON processing, but that’s it.
> First and foremost, we should remove the connect-json dependency, and also 
> figure out a way to set up / structure the examples so we that we can 
> continue showcasing JSON support.  Alternatively, we could consider removing 
> the JSON example (but I don't like that, personally).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5146) Kafka Streams: remove compile dependency on connect-json

[Michael Noll (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Noll updated KAFKA-5146:

Description: 
We currently have a compile-dependency on `connect-json`:

{code}

org.apache.kafka
connect-json
0.10.2.0
compile

{code}

The snippet above is from the generated POM of Kafka Streams as of 0.10.2.0 
release.

AFAICT the only reason for that is because the *examples* showcase some JSON 
processing, but that’s it.

First and foremost, we should remove the connect-json dependency, and also 
figure out a way to set up / structure the examples so we that we can continue 
showcasing JSON support.  Alternatively, we could consider removing the JSON 
example (but I don't like that, personally).

  was:
We currently have a compile-dependency on `connect-json`:

```
org.apache.kafka
connect-json
0.10.2.0
compile

```
The snippet above is from the generated POM of Kafka Streams as of 0.10.2.0 
release.

AFAICT the only reason for that is because the *examples* showcase some JSON 
processing, but that’s it.

First and foremost, we should remove the connect-json dependency, and also 
figure out a way to set up / structure the examples so we that we can continue 
showcasing JSON support.  Alternatively, we could consider removing the JSON 
example (but I don't like that, personally).


> Kafka Streams: remove compile dependency on connect-json
> 
>
> Key: KAFKA-5146
> URL: https://issues.apache.org/jira/browse/KAFKA-5146
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0, 0.10.2.0, 0.10.2.1
>Reporter: Michael Noll
>Priority: Minor
>
> We currently have a compile-dependency on `connect-json`:
> {code}
> 
> org.apache.kafka
> connect-json
> 0.10.2.0
> compile
> 
> {code}
> The snippet above is from the generated POM of Kafka Streams as of 0.10.2.0 
> release.
> AFAICT the only reason for that is because the *examples* showcase some JSON 
> processing, but that’s it.
> First and foremost, we should remove the connect-json dependency, and also 
> figure out a way to set up / structure the examples so we that we can 
> continue showcasing JSON support.  Alternatively, we could consider removing 
> the JSON example (but I don't like that, personally).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Updated] (KAFKA-5146) Kafka Streams: remove compile dependency on connect-json

[Michael Noll (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-5146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Michael Noll updated KAFKA-5146:

Description: 
We currently have a compile-dependency on `connect-json`:

{code}

  org.apache.kafka
  connect-json
  0.10.2.0
  compile
  
{code}

The snippet above is from the generated POM of Kafka Streams as of 0.10.2.0 
release.

AFAICT the only reason for that is because the Kafka Streams *examples* 
showcase some JSON processing, but that’s it.

First and foremost, we should remove the connect-json dependency, and also 
figure out a way to set up / structure the examples so we that we can continue 
showcasing JSON support.  Alternatively, we could consider removing the JSON 
example (but I don't like that, personally).

  was:
We currently have a compile-dependency on `connect-json`:

{code}

  org.apache.kafka
  connect-json
  0.10.2.0
  compile
  
{code}

The snippet above is from the generated POM of Kafka Streams as of 0.10.2.0 
release.

AFAICT the only reason for that is because the *examples* showcase some JSON 
processing, but that’s it.

First and foremost, we should remove the connect-json dependency, and also 
figure out a way to set up / structure the examples so we that we can continue 
showcasing JSON support.  Alternatively, we could consider removing the JSON 
example (but I don't like that, personally).


> Kafka Streams: remove compile dependency on connect-json
> 
>
> Key: KAFKA-5146
> URL: https://issues.apache.org/jira/browse/KAFKA-5146
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.1.0, 0.10.2.0, 0.10.2.1
>Reporter: Michael Noll
>Priority: Minor
>
> We currently have a compile-dependency on `connect-json`:
> {code}
> 
>   org.apache.kafka
>   connect-json
>   0.10.2.0
>   compile
>   
> {code}
> The snippet above is from the generated POM of Kafka Streams as of 0.10.2.0 
> release.
> AFAICT the only reason for that is because the Kafka Streams *examples* 
> showcase some JSON processing, but that’s it.
> First and foremost, we should remove the connect-json dependency, and also 
> figure out a way to set up / structure the examples so we that we can 
> continue showcasing JSON support.  Alternatively, we could consider removing 
> the JSON example (but I don't like that, personally).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Created] (KAFKA-5146) Kafka Streams: remove compile dependency on connect-json

[Michael Noll (JIRA)]

Michael Noll created KAFKA-5146:
---

 Summary: Kafka Streams: remove compile dependency on connect-json
 Key: KAFKA-5146
 URL: https://issues.apache.org/jira/browse/KAFKA-5146
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Michael Noll
Priority: Minor


We currently have a compile-dependency on `connect-json`:

```
org.apache.kafka
connect-json
0.10.2.0
compile

```
The snippet above is from the generated POM of Kafka Streams as of 0.10.2.0 
release.

AFAICT the only reason for that is because the *examples* showcase some JSON 
processing, but that’s it.

First and foremost, we should remove the connect-json dependency, and also 
figure out a way to set up / structure the examples so we that we can continue 
showcasing JSON support.  Alternatively, we could consider removing the JSON 
example (but I don't like that, personally).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5144) MinTimestampTracker does not correctly add timestamps lower than the current max

[Michal Borowiecki (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990683#comment-15990683
 ] 

Michal Borowiecki commented on KAFKA-5144:
--

I understand now, thanks a lot for the thorough explanation!
Closed #2947 with the invalid tests.
Leaving #2948 open as I think it is still of value.

> MinTimestampTracker does not correctly add timestamps lower than the current 
> max
> 
>
> Key: KAFKA-5144
> URL: https://issues.apache.org/jira/browse/KAFKA-5144
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Michal Borowiecki
>Assignee: Michal Borowiecki
>
> When adding elements MinTimestampTracker removes all existing elements 
> greater than the added element.
> Perhaps I've missed something and this is intended behaviour but I can't find 
> any evidence for that in comments or tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-5144) MinTimestampTracker does not correctly add timestamps lower than the current max

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990682#comment-15990682
 ] 

ASF GitHub Bot commented on KAFKA-5144:
---

Github user mihbor closed the pull request at:

https://github.com/apache/kafka/pull/2947


> MinTimestampTracker does not correctly add timestamps lower than the current 
> max
> 
>
> Key: KAFKA-5144
> URL: https://issues.apache.org/jira/browse/KAFKA-5144
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Michal Borowiecki
>Assignee: Michal Borowiecki
>
> When adding elements MinTimestampTracker removes all existing elements 
> greater than the added element.
> Perhaps I've missed something and this is intended behaviour but I can't find 
> any evidence for that in comments or tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[GitHub] kafka pull request #2947: KAFKA-5144 added 2 test cases

[mihbor]

Github user mihbor closed the pull request at:

https://github.com/apache/kafka/pull/2947


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (KAFKA-5144) MinTimestampTracker does not correctly add timestamps lower than the current max

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990643#comment-15990643
 ] 

Matthias J. Sax edited comment on KAFKA-5144 at 5/1/17 7:30 AM:


--After digging a little deeper, partition time get's advanced as {{5, 10, 12, 
12, 20, 20}} -- thus the initial example is correct. At the same time, the last 
example is also correct, however, this new partition time (that is ready one 
step ahead) will only be used _after_ the record got processed. Thus, time is 
advanced correctly.--

Updated my comment above accordingly.


was (Author: mjsax):
After digging a little deeper, partition time get's advanced as {{5, 10, 12, 
12, 20, 20}} -- thus the initial example is correct. At the same time, the last 
example is also correct, however, this new partition time (that is ready one 
step ahead) will only be used _after_ the record got processed. Thus, time is 
advanced correctly.

> MinTimestampTracker does not correctly add timestamps lower than the current 
> max
> 
>
> Key: KAFKA-5144
> URL: https://issues.apache.org/jira/browse/KAFKA-5144
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Michal Borowiecki
>Assignee: Michal Borowiecki
>
> When adding elements MinTimestampTracker removes all existing elements 
> greater than the added element.
> Perhaps I've missed something and this is intended behaviour but I can't find 
> any evidence for that in comments or tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Comment Edited] (KAFKA-5144) MinTimestampTracker does not correctly add timestamps lower than the current max

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990619#comment-15990619
 ] 

Matthias J. Sax edited comment on KAFKA-5144 at 5/1/17 7:29 AM:


It is intended behavior. {{MinTimestampTracker}} tracks the minimum timestamp 
for non-processed records in the buffer. The undocumented usage pattern is, 
that `addElement()` and `removeElement()` are (must be) called in the same 
order. Your tests don't follow this pattern. The logic is about tracking the 
current partition time as minimum of whatever is in the buffer (also 
considering out-of-order records) (cf. KAFKA-3514)

Example: We get a batch of records with ts {{5, 10, 15, 12, 20}} and add them 
consecutively to the timestamp tracker. Thus, when we process those record one 
by one, is the following steps:

 - process {{5}}, queue {{10, 15, 12, 20}}, partition time {{5}}
 - process {{10}}, queue {{15, 12, 20}}, partition time {{10}}
 - process {{15}}, queue {{12, 20}}, partition time {{12}} (!! current minimum 
is {{12}}, not {{15}} !!)
 - process {{12}}, queue {{20}}, partition time {{12}}
 - process {{20}}, queue empty, partition time {{20}}

The tracker will have the following states when adding the records one by one 
(this happens before processing begins):
 - add {{5}}: {{5}}
 - add {{10}}: {{5, 10}}
 - add {{15}}: {{5, 10, 15}}
 - add {{12}}: {{5, 10, 12}} (!! this the the behavior that is not a bug !!)
 - add {{20}}: {{5, 10, 12, 20}}

During processing, we {{poll}} the head record from the queue, call 
{{removeElement}} on the tracker afterwards. Thus, we get (not we start with 
tracker state {{5, 10, 12, 20}}):
 - poll from queue {{5}}, tracker after remove {{10, 12, 20}}
 - poll from queue {{10}}, tracker after remove {{12, 20}}
 - poll from queue {{15}}, tracker after remove {{12, 20}} (!! as we call 
{{removeElement(15)}} we keep {{12}}, this allows us to use {{12}} for two 
records)
 - poll from queue {{12}}, tracker after remove {{20}}
 - poll from queue {{20}}, tracker after remove empty

Note that the call to {{timeTracker.get()}} ({{RecordQueue}} L124) happens 
after {{timeTracker.removeElement(elem)}} ({{RecordQueue}} L120). This is also 
correct, as we advance "partition time" in this step, but only use it _after_ 
the current record got fully processed.

 - process {{5}}, queue {{10, 15, 12, 20}}, tracker after remove {{10, 12, 
20}}, next partition time {{10}}
 - process {{10}}, queue {{15, 12, 20}}, tracker after remove {{12, 20}}, next 
partition time {{12}}
 - process {{15}}, queue {{12, 20}}, tracker after remove {{12, 20}}, next 
partition time {{12}}
 - process {{12}}, queue {{20}}, tracker after remove {{20}},  next partition 
time {{20}}
 - process {{20}}, queue empty, tracker after remove empty, next partition time 
{{20}} (from {{lastKnownTime}})



was (Author: mjsax):
It is intended behavior. {{MinTimestampTracker}} tracks the minimum timestamp 
for non-processed records in the buffer. The undocumented usage pattern is, 
that `addElement()` and `removeElement()` are (must be) called in the same 
order. Your tests don't follow this pattern. The logic is about tracking the 
current partition time as minimum of whatever is in the buffer (also 
considering out-of-order records) (cf. KAFKA-3514)

Example: We get a batch of records with ts {{5, 10, 15, 12, 20}} and add them 
consecutively to the timestamp tracker. Thus, when we process those record one 
by one, is the following steps (note, I am not sure if "partition time" in this 
example is correct; cf. below):

 - process {{5}}, queue {{10, 15, 12, 20}}, partition time {{5}}
 - process {{10}}, queue {{15, 12, 20}}, partition time {{10}}
 - process {{15}}, queue {{12, 20}}, partition time {{12}} (!! current minimum 
is {{12}}, not {{15}} !!)
 - process {{12}}, queue {{20}}, partition time {{12}}
 - process {{20}}, queue empty, partition time {{20}}

The tracker will have the following states when adding the records one by one 
(this happens before processing begins):
 - add {{5}}: {{5}}
 - add {{10}}: {{5, 10}}
 - add {{15}}: {{5, 10, 15}}
 - add {{12}}: {{5, 10, 12}} (!! this the the behavior that is not a bug !!)
 - add {{20}}: {{5, 10, 12, 20}}

During processing, we {{poll}} the head record from the queue, call 
{{removeElement}} on the tracker afterwards. Thus, we get (not we start with 
tracker state {{5, 10, 12, 20}}):
 - poll from queue {{5}}, tracker after remove {{10, 12, 20}}
 - poll from queue {{10}}, tracker after remove {{12, 20}}
 - poll from queue {{15}}, tracker after remove {{12, 20}} (!! as we call 
{{removeElement(15)}} we keep {{12}}, this allows us to use {{12}} for two 
records)
 - poll from queue {{12}}, tracker after remove {{20}}
 - poll from queue {{20}}, tracker after remove empty

However, I am not sure if we actually advance "partition time" as indented -- 
we might want to call {{timeTra

[jira] [Commented] (KAFKA-5144) MinTimestampTracker does not correctly add timestamps lower than the current max

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-5144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15990643#comment-15990643
 ] 

Matthias J. Sax commented on KAFKA-5144:


After digging a little deeper, partition time get's advanced as {{5, 10, 12, 
12, 20, 20}} -- thus the initial example is correct. At the same time, the last 
example is also correct, however, this new partition time (that is ready one 
step ahead) will only be used _after_ the record got processed. Thus, time is 
advanced correctly.

> MinTimestampTracker does not correctly add timestamps lower than the current 
> max
> 
>
> Key: KAFKA-5144
> URL: https://issues.apache.org/jira/browse/KAFKA-5144
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.2.1
>Reporter: Michal Borowiecki
>Assignee: Michal Borowiecki
>
> When adding elements MinTimestampTracker removes all existing elements 
> greater than the added element.
> Perhaps I've missed something and this is intended behaviour but I can't find 
> any evidence for that in comments or tests.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)