Build failed in Jenkins: kafka-trunk-jdk8 #2922

See Changes: [jason] MINOR: Make JAAS configurable via template variables in system tests [wangguoz] MINOR: Fix streams Scala foreach recursive call (#5539) [mjsax] MINOR: restructure Windows to favor

Jenkins build is back to normal : kafka-trunk-jdk10 #432

See

Request for contributor permissions

JIRA ID: tony80720 Cwiki ID: Yu Tien Wang Thanks in advance! Tony --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

Request for contributor permissions

JIRA ID: tony80720 Cwiki ID: 王又田 Thanks in advance! Tony --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

hi

Tony --- This email has been checked for viruses by Avast antivirus software. https://www.avast.com/antivirus

Build failed in Jenkins: kafka-trunk-jdk10 #431

See Changes: [jason] KAFKA-7225; Corrected system tests by generating external properties [jason] MINOR: Make JAAS configurable via template variables in system tests --

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Hello Boyang, Thanks for the updated proposal, a few questions: 1. Where will "change-group-timeout" be communicated to the broker? Will that be a new field in the JoinGroupRequest, or are we going to piggy-back on the existing session-timeout field (assuming that the original value will not be

Re: [DISCUSS] KIP-363: Make FunctionConversions private

+1. On Thu, Aug 23, 2018 at 12:47 PM, Ted Yu wrote: > +1 > > In the Motivation section, you can quote the comment from pull request so > that reader doesn't have to click through. > > Cheers > > On Thu, Aug 23, 2018 at 12:13 PM Joan Goyeau wrote: > > > Hi, > > > > As pointed out in this

Re: [DISCUSS] KIP-280: Enhanced log compaction

Hi, Luis, Thanks for the reply. A few more comments below. 1. About the topic level configuration. It seems that it's useful for the new configs to be at the topic level. Currently, the following configs related to compaction are already at the topic level. min.cleanable.dirty.ratio

Re: [VOTE] KIP-346 - Improve LogCleaner behavior on error

Hi, Stan, Thanks for the KIP. Looks good to me overall. Just one comment below. uncleanable-partitions-count is per logDir, but uncleanable-bytes is not. Should we make them consistent? Jun On Wed, Aug 22, 2018 at 4:15 AM, Stanislav Kozlovski wrote: > Hi everybody, > > @Jason - I've updated

Build failed in Jenkins: kafka-trunk-jdk10 #430

See Changes: [wangguoz] MINOR: Return correct instance of SessionWindowSerde (#5546) -- [...truncated 1.53 MB...]

Re: [VOTE] KIP-291: Have separate queues for control requests and data requests

Hi, Lucas, Sorry for the delay. The new proposal looks good to me overall. A few minor comments below. 1. It's possible that listener.name.for.controller is set, but set to the same value as inter.broker.listener.name. In that case, should we have a single network thread and the request handling

Metadata Update Corner Case

While updating a small Kafka client application to use Kafka 1.1.1 and the KafkaProducer (rather than the deprecated Producer) a maintenance related unit test started failing. The scenario is out of the ordinary, but, the resulting problem may still be of interest: * Two Kafka nodes *

[jira] [Created] (KAFKA-7335) Store clusterId locally to ensure broker joins the right cluster

Jason Gustafson created KAFKA-7335: -- Summary: Store clusterId locally to ensure broker joins the right cluster Key: KAFKA-7335 URL: https://issues.apache.org/jira/browse/KAFKA-7335 Project: Kafka

Jenkins build is back to normal : kafka-trunk-jdk8 #2920

See

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

To clarify, what I am suggesting is to only remove the default implementation for these methods. So users would be required to implement serialize(topic, data) and deserialize(topic, data). -Jason On Thu, Aug 23, 2018 at 1:48 PM, Jason Gustafson wrote: > Hey Viktor, > > Thinking about it a

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

Hey Viktor, Thinking about it a little more, I wonder if we should just not provide a default method for serialize(topic, data) and deserialize(topic, data). Implementing these methods is a trivial burden for users and it feels like there's no good solution which allows both methods to have

Re: Build failed in Jenkins: kafka-trunk-jdk10 #429

I ran streams unit tests as of commit 4156ea0a9bcca67d209fd3b43d2268c9abd5a0b5 . All tests passed locally. FYI On Thu, Aug 23, 2018 at 12:23 PM Joan Goyeau wrote: > I'm looking into this one. > > On Thu, 23 Aug 2018 at 20:19 Apache Jenkins Server < > jenk...@builds.apache.org> wrote: > > >

Re: [DISCUSS] KIP-363: Make FunctionConversions private

+1 In the Motivation section, you can quote the comment from pull request so that reader doesn't have to click through. Cheers On Thu, Aug 23, 2018 at 12:13 PM Joan Goyeau wrote: > Hi, > > As pointed out in this comment #5539 (comment) >

Re: [DISCUSS] KIP-357: Add support to list ACLs per principal

+1 (binding) Thanks, Harsha On Wed, Aug 22, 2018, at 9:15 AM, Manikumar wrote: > Hi Viktor, > We already have a method in Authorizer interface to get acls for a given > principal. > We will use this method to fetch acls and filter the results for requested > Resources. > Authorizer { >def

[jira] [Created] (KAFKA-7334) Suggest changing config for state.dir in case of FileNotFoundException

Ted Yu created KAFKA-7334: - Summary: Suggest changing config for state.dir in case of FileNotFoundException Key: KAFKA-7334 URL: https://issues.apache.org/jira/browse/KAFKA-7334 Project: Kafka

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

Hi Ismael, Regarding the deprecation of the 2 parameter method: should we do this with the Serializer interface as well? I've updated the "Rejected Alternatives" with a few. I've added this circular reference one too but actually there's a way (pretty heavyweight) by adding a guard class that

Re: Build failed in Jenkins: kafka-trunk-jdk10 #429

I'm looking into this one. On Thu, 23 Aug 2018 at 20:19 Apache Jenkins Server < jenk...@builds.apache.org> wrote: > See < > https://builds.apache.org/job/kafka-trunk-jdk10/429/display/redirect?page=changes > > > > Changes: > > [wangguoz] KAFKA-7316: Fix Streams Scala filter recursive call #5538

Build failed in Jenkins: kafka-trunk-jdk10 #429

See Changes: [wangguoz] KAFKA-7316: Fix Streams Scala filter recursive call #5538 -- [...truncated 1.98 MB...]

[DISCUSS] KIP-363: Make FunctionConversions private

Hi, As pointed out in this comment #5539 (comment) the object FunctionConversions is only of internal use and therefore should be private to the lib only so that we can do changes without going through KIP like this one. KIP:

[jira] [Created] (KAFKA-7333) Protocol changes for KIP-320

Jason Gustafson created KAFKA-7333: -- Summary: Protocol changes for KIP-320 Key: KAFKA-7333 URL: https://issues.apache.org/jira/browse/KAFKA-7333 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-7332) Improve error message when trying to produce message without key for compacted topic

Patrik Kleindl created KAFKA-7332: - Summary: Improve error message when trying to produce message without key for compacted topic Key: KAFKA-7332 URL: https://issues.apache.org/jira/browse/KAFKA-7332

[jira] [Created] (KAFKA-7331) Kafka does not detect broker loss in the event of a network partition within the cluster

Kevin Li created KAFKA-7331: --- Summary: Kafka does not detect broker loss in the event of a network partition within the cluster Key: KAFKA-7331 URL: https://issues.apache.org/jira/browse/KAFKA-7331

Re: [VOTE] KIP-328: Ability to suppress updates for KTables

It seems nobody has any objections against the change. That's for the KIP improvement. I'll go ahead and merge the PR. -Matthias On 8/21/18 2:44 PM, John Roesler wrote: > Hello again, all, > > I belatedly had a better idea for adding grace period to the Windows class > hierarchy (TimeWindows,

Build failed in Jenkins: kafka-trunk-jdk8 #2919

See Changes: [ismael] MINOR: Eliminate warnings from KafkaProducerTest (#5548) -- [...truncated 870.50 KB...] kafka.security.auth.ZkAuthorizationTest >

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Yeah, the reason why we want to deprecate the auto create functionality is that it happens when a metadata request is done instead of when a write operation happens. So, there's no reason to differentiate between the two. Ismael On Thu, Aug 23, 2018 at 8:16 AM Andrew Otto wrote: > Ah, I just

[jira] [Resolved] (KAFKA-6309) add support for getting topic defaults from AdminClient

[ https://issues.apache.org/jira/browse/KAFKA-6309?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-6309. -- Resolution: Fixed Closing as per above comment. We can use "describeConfigs(brokerResource, new

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Ah, I just realized that as proposed this is only for the Java consumer client, correct? Would it be possible to make this a broker config, like the current one? Something like: auto.create.topics.enable=true # allow both producer and consumer to create auto.create.topics.enable=consumer #

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Yup :) On Thu, Aug 23, 2018 at 11:04 AM Ismael Juma wrote: > Andrew, one question: you are relying on auto topic creation for the > producer and that's why you can't just disable it? > > On Thu, Aug 23, 2018 at 8:01 AM Ismael Juma wrote: > > > Thanks for sharing Andrew! > > > > Ismael > > > >

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Andrew, one question: you are relying on auto topic creation for the producer and that's why you can't just disable it? On Thu, Aug 23, 2018 at 8:01 AM Ismael Juma wrote: > Thanks for sharing Andrew! > > Ismael > > On Thu, Aug 23, 2018 at 7:57 AM Andrew Otto wrote: > >> We recently had a

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Thanks for sharing Andrew! Ismael On Thu, Aug 23, 2018 at 7:57 AM Andrew Otto wrote: > We recently had a pretty serious Kafka outage > < > https://wikitech.wikimedia.org/wiki/Incident_documentation/20180711-kafka-eqiad#Summary > > > caused by a bug in one of our consumers that caused it to

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

We recently had a pretty serious Kafka outage caused by a bug in one of our consumers that caused it to create new topics in an infinite loop AKA a topic bomb! Having consumers restricted from creating

Re: [DISCUSS] KIP-358: Migrate Streams API to Duration instead of long ms times

Hello, Mathias. Thanks for your feedback. > Thus, it might make sense to keep old and just add new ones? As far as I understand, we will keep old methods anyway to prevent public API backward compatibility. I agree with you, methods that used internally shouldn't be deprecated. > End users

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

Also, we may consider deprecating the deserialize method that does not take headers. Yes, it's a convenience, but it also adds confusion. Ismael On Thu, Aug 23, 2018 at 6:48 AM Ismael Juma wrote: > I think the KIP needs the rejected alternatives section to have more > detail. For example,

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

I think the KIP needs the rejected alternatives section to have more detail. For example, another option would be something like the following, which works great as long as one overrides one of the methods, but pretty bad if one doesn't. :) default T deserialize(String topic, byte[] data) {

Re: Current Kafka Steams and KSQL Performance Metrics / Benchmarks?

Thanks Guozhang! I am asking primarily because I have seen Flink & Spark Streaming users boasting of millions of records / second being processed and was interested to learn where Kafka Streams / KSQL stands. This would also help a lot in capability planning for teams looking to use Kafka

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

Hi Jason, Thanks for the feedback. 1. I chose to return null here because according to the documentation it may return null data, therefore the users of this methods are perpared for getting a null. Thinking of it though it may be better to throw an exception by default because it'd indicate a

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Generally, I think positive configs (`allow` instead of `suppress`) are easier to understand. Ismael On Wed, Aug 22, 2018 at 11:05 PM Matthias J. Sax wrote: > Thanks for the summary! > > We might want to add a diagram/table to the docs when we add this > feature (with whatever config name we

Re: [VOTE] KIP-349 Priorities for Source Topics

also: https://cwiki.apache.org/confluence/display/KAFKA/KIP-353%3A+Improve+Kafka+Streams+Timestamp+Synchronization On 20.08.2018 15:01, Thomas Becker wrote: I agree with Jan. A strategy interface for choosing processing order is nice, and would hopefully be a step towards getting this in

[jira] [Resolved] (KAFKA-6314) Add a tool to delete kafka based consumer offsets for a given group

[ https://issues.apache.org/jira/browse/KAFKA-6314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-6314. -- Resolution: Duplicate Resolving as duplicate of KAFKA-6275 > Add a tool to delete kafka based

[jira] [Resolved] (KAFKA-1921) Delete offsets for a group with kafka offset storage

[ https://issues.apache.org/jira/browse/KAFKA-1921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-1921. -- Resolution: Duplicate Resolving as duplicate of KAFKA-6275 > Delete offsets for a group with kafka

[jira] [Created] (KAFKA-7330) Kakfa 0.10.2.1 producer close method issue

vinay created KAFKA-7330: Summary: Kakfa 0.10.2.1 producer close method issue Key: KAFKA-7330 URL: https://issues.apache.org/jira/browse/KAFKA-7330 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

Thanks for the summary! We might want to add a diagram/table to the docs when we add this feature (with whatever config name we choose) to explain how broker config `auto.create.topics.enable` and the consumer config work together. I think both options are equally easy to understand. "allow"