[DISCUSS] KIP-424: Allow suppression of intermediate events based on wall clock time

Hi all, I just published KIP-424: Allow suppression of intermediate events based on wall clock time https://cwiki.apache.org/confluence/display/KAFKA/KIP-424%3A+Allow+suppression+of+intermediate+events+based+on+wall+clock+time I am eager to hear your feedback and concerns. Thanks John Roesler

Re: [DISCUSS] KIP-390: Add producer option to adjust compression level

Hello. Do you have any idea on Becket's Idea of new config format (example below)? ``` compression.config="gzip.compression.level=5, lz4.compression.level=17, zstd.compression.level=22" ``` It requires some additional KIP for supporting new config format (map), but it can significantly simplify

RE: Why is enable.auto.commit=true the default value for consumer?

Indeed, but this is what the documentation says you must do with auto-commit. I say it is a user error if you don't. Regardless, I think this is a fairly common misconception, so it would not hurt to debunk it explicitly in the documentation. -Original Message- From: Colin McCabe

Re: Why is enable.auto.commit=true the default value for consumer?

Hi Clement, You are assuming that the client application is single-threaded-- or at least processes all the records before polling for more. This may or may not be the case. But that is a fair point-- in this case, auto-commit would be safe. best, Colin On Tue, Jan 29, 2019, at 16:23,

RE: Why is enable.auto.commit=true the default value for consumer?

I had the same impression at some point but this is not how auto-commit works. Auto-commit can only commit when the application comes back to poll and if it decides to commit at that time, it will only commit the previous batch. In your example, the app might come back and have to re-execute all

Build failed in Jenkins: kafka-trunk-jdk8 #3354

See Changes: [github] MINOR: Update usage of deprecated API (#6146) -- [...truncated 2.28 MB...] org.apache.kafka.connect.json.JsonConverterTest > stringHeaderToConnect

Build failed in Jenkins: kafka-trunk-jdk11 #251

See Changes: [github] MINOR: Update usage of deprecated API (#6146) -- [...truncated 2.28 MB...] org.apache.kafka.streams.test.OutputVerifierTest >

RE: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message Consumption Across Partitions in KafkaConsumer

I beg to differ... The original KIP intended to consume messages fairly and the current implementation can be improved with the patch that can benefit the community. I believe that fair consumption is an important characteristic at the consumer side. But anyhow, looks like this is not received

Re: Why is enable.auto.commit=true the default value for consumer?

This is a very fair question. I suspect the answer is that people feel that automatic offset commit is easier to get started with, even though it has the error handling shortcomings you mention. It would be very hard to change the default now, since it would break such a huge amount of code.

Re: [EXTERNAL] - Re: [DISCUSS] KIP-387: Fair Message Consumption Across Partitions in KafkaConsumer

On Mon, Jan 28, 2019, at 06:57, ChienHsing Wu wrote: > So... Does this non-response mean I should drop this topic after almost > one month, folks? Hi ChienHsing, I would recommend dropping it, since I don't see a lot of uses for it. Maybe there is something I missed, though. See my responses

Why is enable.auto.commit=true the default value for consumer?

As the question indicates. Should this not be default false? I think this is a bit nefarious to someone launching their application into production without testing it extensively around failure modes. I can see a scenario where a consumer polls for events, processes them, produces to output

Re: [VOTE] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

Thanks, Tom! Great work. best, Colin On Mon, Jan 28, 2019, at 04:33, Tom Bentley wrote: > Hi Folks, > > It took a while, but the work for KIP-183 has now been merged. My thanks to > everyone involved. > > A few details changed between what was voted on and what ultimately got > merged. I've

[jira] [Created] (KAFKA-7884) Docs for message.format.version and log.message.format.version show invalid (corrupt?) "valid values"

James Cheng created KAFKA-7884: -- Summary: Docs for message.format.version and log.message.format.version show invalid (corrupt?) "valid values" Key: KAFKA-7884 URL: https://issues.apache.org/jira/browse/KAFKA-7884

[jira] [Created] (KAFKA-7883) Add schema.namespace support to SetSchemaMetadata SMT in Kafka Connect

Jérémy Thulliez created KAFKA-7883: -- Summary: Add schema.namespace support to SetSchemaMetadata SMT in Kafka Connect Key: KAFKA-7883 URL: https://issues.apache.org/jira/browse/KAFKA-7883 Project:

[jira] [Created] (KAFKA-7882) StateStores are frequently closed during the 'transform' method

Mateusz Owczarek created KAFKA-7882: --- Summary: StateStores are frequently closed during the 'transform' method Key: KAFKA-7882 URL: https://issues.apache.org/jira/browse/KAFKA-7882 Project: Kafka

[jira] [Resolved] (KAFKA-7881) Inter.broker.procol.version is incorrect for Rolling Upgrade

[ https://issues.apache.org/jira/browse/KAFKA-7881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] M. Manna resolved KAFKA-7881. - Resolution: Cannot Reproduce Assignee: M. Manna Duplicate jars were present - a delete and copy of

[jira] [Created] (KAFKA-7881) Inter.broker.procol.version is incorrect for Rolling Upgrade

M. Manna created KAFKA-7881: --- Summary: Inter.broker.procol.version is incorrect for Rolling Upgrade Key: KAFKA-7881 URL: https://issues.apache.org/jira/browse/KAFKA-7881 Project: Kafka Issue

Re: Rolling Upgrade failed after bumping version to 2.1 from 1.1

By investigating journalctl logs - I see that it says version 2.1 is not a valid version? So what is a valid version, does anyone know? Thanks, On Tue, 29 Jan 2019 at 11:37, M. Manna wrote: > Hello, > > I have recently upgraded to 2.12-2.1.0 from 2.11-1.1.0. After bumping >

Rolling Upgrade failed after bumping version to 2.1 from 1.1

Hello, I have recently upgraded to 2.12-2.1.0 from 2.11-1.1.0. After bumping inter.broker.protocol.version to 2.1 (and doing 2 bleed-and-restart from 1.1), my services are all failing to start. Slightly clueless at this stage regarding what to do other than reverting. I followed the steps in