Re: Re: [DISCUSS] KIP-905: Broker interceptors

FWIW, this would be very useful for the Wikimedia Foundation's Event Platform. We have some requirements for our event stream producers, and not having to re-implement this logic in multiple programming languages and

Re: [DISCUSS] KIP-932: Queues for Kafka

Wow looks very cool. One Q. In the Future Work session: > The concept can be extended to give key-based ordering so that partial ordering and fine-grained sharing can be achieved at the same time. I think this will be a pretty important feature needed to make shared consumer groups useful.

Re: Re: [DISCUSS] Cluster Linking / Cross-Cluster Replication - Call for Interest & Co-Authors

w0x>to solve this problem, but something built into Kafka itself that still allows for distinct Kafka clusters would be really great! -Andrew Otto On Sat, May 20, 2023 at 7:42 AM hudeqi <16120...@bjtu.edu.cn> wrote: > Hi, I am also very excited to see this discussion, because I als

Re: [DISCUSS] KIP-879: Multi-level Rack Awareness

FWIW, the Wikimedia Foundation would find this change really helpful. We are going to soon experiment with a stretched Kafka cluster, and it would be nice to be able to target datacenter AND racks for replica placement. On Thu, Dec 8, 2022 at 3:37 AM ziming deng wrote: > Hi Viktor, > > As far

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

> Stretch clusters are possible with the KRaft architecture. For example, if you had a cluster with nodes in us-west-1, us-west-2 and us-central-1, you could put a KRaft controller node in each region. This is similar to how with ZK you'd put a ZK node in each region. Ah ha! TIL about

Re: [DISCUSS] KIP-833: Mark KRaft as Production Ready

> Deprecating ZK Mode and > Removing Zookeeper Mode I'm excited about KRaft, but quick Q. I'm researching Kafka 'stretch' cluster deployments, and as far as I can tell stretch clusters require Zookeeper to function properly, is this correct? If so, we might want to solve that before Deprecating

Re: [jira] [Created] (KAFKA-10415) Provide an officially supported Node.js client

FWIW, Wikimedia uses node-rdkafka from Blizzard, and is pretty happy with it. I think Blizzard is trying to find external supports and maintainers though, so the project may not be well supported there for much longer. On Tue, Aug 18, 2020 at 10:32

GitBox emails for kafka-site going to dev@kafka.apache.org

Is this intentional? If not, can someone disable this feature? dev@ is getting emails for every change to kafka-site on github. Thank you! On Mon, Aug 10, 2020 at 11:31 AM GitBox wrote: > > rhauch opened a new pull request #286: > URL: https://github.com/apache/kafka-site/pull/286 > > >

Re: [DISCUSS] KIP-651 - Support PEM format for SSL certificates and private key

. - Andrew Otto On Mon, Aug 3, 2020 at 7:29 AM Rajini Sivaram wrote: > Hi all, > > I have submitted KIP-651 to support PEM format for SSL key and trust > stores: > >- > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-651+-+Support+PEM+format+for+SSL+cert

Re: [DISCUSS] Kafka 3.0

> Before we deprecate server side auto topic creation, we should have client side auto topic creation for the producer: Deprecating sounds fine, but before disabling it, it might be worthwhile to wait long enough for non Java clients to catch up to this too. :) On Mon, May 11, 2020 at 4:45 PM

Re: Feature request for 'prefixless' topic mirroring

rcin's where folks want to do this not just for migration to MM2 reasons. 'Prefixless' as Marcin suggests is pretty descriptive. :) -Andrew Otto SRE, Wikimedia Foundation On Mon, Mar 16, 2020 at 1:23 PM Ryanne Dolan wrote: > Marcin, I just created this jira per your suggestion

Re: [DISCUSS] KIP-527: Add NothingSerde to Serdes

s > > > > Any additional feeback on this KIP? > > Should I start a vote? > > > > В Пт, 20/09/2019 в 08:52 +0300, Nikolay Izhikov пишет: > >> Hello, Andrew. > >> > >> OK, if nobody mind, let's change it to Null. > >> > >> В Чт,

Re: [DISCUSS] KIP-527: Add NothingSerde to Serdes

NullSerdes seems more descriptive, but up to you! :) On Thu, Sep 19, 2019 at 1:37 PM Nikolay Izhikov wrote: > Hello, Andrew. > > Seems, usage null or nothing is matter of taste. I dont mind if we call it > NullSerde > > чт, 19 сент. 2019 г., 20:28 Andrew Otto : > &

Re: [DISCUSS] KIP-527: Add NothingSerde to Serdes

Why 'NothingSerdes' instead of 'NullSerdes'? On Thu, Sep 19, 2019 at 1:10 PM Nikolay Izhikov wrote: > All, > > I'd like to start a discussion for adding a NothingSerde to Serdes. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-527%3A+Add+NothingSerde+to+Serdes > > Your comments and

Re: Kafka Connect Schema from JSON

In case this is helpful, I wrote a (WIP) version that does what you say, but using JSONSchema instead of avro schema. https://github.com/ottomata/kafka-connect-jsonschema On Thu, Aug 29, 2019 at 12:48 PM Josef Hak wrote: > Hi, > please is it possible to create kafka connect Schema using json,

Re: [DISCUSS] KIP-481: SerDe Improvements for Connect Decimal type in JSON

This is a bit orthogonal, but in JsonSchemaConverter I use JSONSchemas to indicate whether a JSON number should be deserialized as an integer or a decimal

Re: [VOTE] KIP-382 MirrorMaker 2.0

+1 This looks like a huge project! Wikimedia would be very excited to have this. Thanks! On Thu, Dec 20, 2018 at 9:52 AM Ryanne Dolan wrote: > Hey y'all, please vote to adopt KIP-382 by replying +1 to this thread. > > For your reference, here are the highlights of the proposal: > > - Leverages

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

Wikimedia currently implements 'master <-> master' replication by manually prefixing topics with datacenter names, and then configuring MirrorMaker to only replicate topics that begin with a DC name to another. While having topics named with topological details is manageable, I wouldn't say it is

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

gt; can become the group coordinator. > > > > On 16.10.2018 15:04, Andrew Otto wrote: > >> I would generally say a LAN is better than a WAN for doing group > >> coordinaton > > > > For sure, but a LAN is better than a WAN for producing messages too. If > >

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

> I would generally say a LAN is better than a WAN for doing group > coordinaton For sure, but a LAN is better than a WAN for producing messages too. If there is network congestion during network production, messages will be dropped. With MirrorMaker currently, you can either skip these dropped

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

FWIW, I’d find this feature useful. On Mon, Sep 24, 2018 at 2:42 PM Randall Hauch wrote: > Ryanne, > > If your connector is already using the AdminClient, then you as the > developer have a choice of switching to the new Connect-based functionality > or keeping the existing use of the

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

, 2018 at 11:08 AM Andrew Otto wrote: > Yup :) > > On Thu, Aug 23, 2018 at 11:04 AM Ismael Juma wrote: > >> Andrew, one question: you are relying on auto topic creation for the >> producer and that's why you can't just disable it? >> >> On Thu, Aug 23

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

ndrew! > > > > Ismael > > > > On Thu, Aug 23, 2018 at 7:57 AM Andrew Otto wrote: > > > >> We recently had a pretty serious Kafka outage > >> < > >> > https://wikitech.wikimedia.org/wiki/Incident_documentation/20180711-kafka-eqiad#Summ

Re: [DISCUSS] KIP-361: Add Consumer Configuration to Disable Auto Topic Creation

We recently had a pretty serious Kafka outage caused by a bug in one of our consumers that caused it to create new topics in an infinite loop AKA a topic bomb! Having consumers restricted from creating

[DISCUSS] KIP-301 Schema Inferencing in JsonConverters

this is similar to how Confluent’s AvroConverter works, but that is Confluent Schema Registry specific.) Anyway, just a thought. - Andrew Otto Senior Systems Engineer Wikimedia Foundation

Re: Kafka mirror maker help

Hiya, Saravanan, I saw you emailed my colleague Alex about WMF’s old debian packaging. I’ll reply here. We now use Confluent’s Kafka debian packaging which does not (or did not?) ship with init scripts. We don’t use Sys V init.d scripts anymore either, but use systemd instead. Our systemd

Re: offsetsForTimes API performance

Speaking of, has there been any talk of combining those two requests into a single API call? I’d assume that offsetForTimes + consumer seek is probably the most common use case of offsetForTimes. Maybe a round trip could be avoided if the broker could just auto-assign the consumer to the offset

Re: [VOTE] Add REST Server to Apache Kafka

-1 for http kafka client in core Although a read only management interface, perhaps via http, sounds kinda useful for things like health checks as mentioned. On Wed, Oct 26, 2016 at 2:00 PM, Zakee wrote: > -1 > > Thanks. > > On Oct 25, 2016, at 2:16 PM, Harsha

[jira] [Commented] (KAFKA-3802) log mtimes reset on broker restart

[ https://issues.apache.org/jira/browse/KAFKA-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319321#comment-15319321 ] Andrew Otto commented on KAFKA-3802: I am running Kafka 0.9.01 on Debian Jessie, with 12 log.dirs

[jira] [Commented] (KAFKA-3802) log mtimes reset on broker restart

[ https://issues.apache.org/jira/browse/KAFKA-3802?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15319225#comment-15319225 ] Andrew Otto commented on KAFKA-3802: Possibly related to KAFKA-1379 > log mtimes reset on bro

[jira] [Created] (KAFKA-3802) log mtimes reset on broker restart

Andrew Otto created KAFKA-3802: -- Summary: log mtimes reset on broker restart Key: KAFKA-3802 URL: https://issues.apache.org/jira/browse/KAFKA-3802 Project: Kafka Issue Type: Bug Affects

[jira] [Commented] (KAFKA-2143) Replicas get ahead of leader and fail

[ https://issues.apache.org/jira/browse/KAFKA-2143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15056672#comment-15056672 ] Andrew Otto commented on KAFKA-2143: I'd like to add that Wikimedia is experiencing this as well

Re: [DISCUSSION] Kafka 0.8.2.2 release?

I agree: keep it simple :) The latest stable version of Kafka right now has a critical bug in it. Fixing that would be good enough. 0.8.2.2 should probably just a maintenance/bugfix release. On Aug 18, 2015, at 14:29, Edward Ribeiro edward.ribe...@gmail.com wrote: I sort of follow

[jira] [Commented] (KAFKA-2189) Snappy compression of message batches less efficient in 0.8.2.1

[ https://issues.apache.org/jira/browse/KAFKA-2189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14695589#comment-14695589 ] Andrew Otto commented on KAFKA-2189: Hi all, The Wikimedia Foundation had a serious

Re: Announcing the Confluent Platform built on Apache Kafka

Wow, .deb packages. I love you. On Feb 25, 2015, at 14:48, Joseph Lawson jlaw...@roomkey.com wrote: This is really awesome stuff. It's great to see y'all growing! Thank you and congratulations! From: Neha Narkhede n...@confluent.io Sent:

[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14141002#comment-14141002 ] Andrew Otto commented on KAFKA-1367: This happens to me as well. See: https

[jira] [Commented] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14141017#comment-14141017 ] Andrew Otto commented on KAFKA-1367: I just updated the librdkafka issue, pasting

[jira] [Comment Edited] (KAFKA-1367) Broker topic metadata not kept in sync with ZooKeeper

[ https://issues.apache.org/jira/browse/KAFKA-1367?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14141017#comment-14141017 ] Andrew Otto edited comment on KAFKA-1367 at 9/19/14 6:15 PM

[jira] [Created] (KAFKA-1564) Replace barrage of bin/*.sh with single cli wrapper

Andrew Otto created KAFKA-1564: -- Summary: Replace barrage of bin/*.sh with single cli wrapper Key: KAFKA-1564 URL: https://issues.apache.org/jira/browse/KAFKA-1564 Project: Kafka Issue Type

[jira] [Commented] (KAFKA-1564) Replace barrage of bin/*.sh with single cli wrapper

[ https://issues.apache.org/jira/browse/KAFKA-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081253#comment-14081253 ] Andrew Otto commented on KAFKA-1564: Ha, actually, don't quote me on the Yelp is using

[jira] [Updated] (KAFKA-1564) Replace barrage of bin/*.sh with single cli wrapper

[ https://issues.apache.org/jira/browse/KAFKA-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Otto updated KAFKA-1564: --- Description: Kafka ships with a bunch of shell wrapper scripts in bin/. These are very small shell

[jira] [Commented] (KAFKA-1564) Replace barrage of bin/*.sh with single cli wrapper

[ https://issues.apache.org/jira/browse/KAFKA-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14081259#comment-14081259 ] Andrew Otto commented on KAFKA-1564: The difficult part might be making the CLASSPATH

[jira] [Created] (KAFKA-1123) Broker IPv6 addresses parsed incorrectly

Andrew Otto created KAFKA-1123: -- Summary: Broker IPv6 addresses parsed incorrectly Key: KAFKA-1123 URL: https://issues.apache.org/jira/browse/KAFKA-1123 Project: Kafka Issue Type: Bug

[jira] [Commented] (KAFKA-1046) Added support for Scala 2.10 builds while maintaining compatibility with 2.8.x

[ https://issues.apache.org/jira/browse/KAFKA-1046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13769853#comment-13769853 ] Andrew Otto commented on KAFKA-1046: Hm, I seem to be getting the same compilation

Re: Kafka/Hadoop consumers and producers

Andrew, I'm about to dive into figuring out how to use Camus without Avro. Perhaps we should join forces? (Be warned thought! My java fu is low at the moment. :) ). -Ao On Aug 12, 2013, at 11:20 PM, Andrew Psaltis andrew.psal...@webtrends.com wrote: Kam, I am perfectly fine if you pick

Re: Kafka/Hadoop consumers and producers

- very clean and simple. What installs all the kafka dependencies under /usr/share/java? From: Andrew Otto o...@wikimedia.org To: Kam Kasravi kamkasr...@yahoo.com Cc: dev@kafka.apache.org dev@kafka.apache.org; Ken Goodhope kengoodh...@gmail.com; Andrew Psaltis psaltis.and...@gmail.com

Re: Kafka/Hadoop consumers and producers

support. From: Andrew Otto o...@wikimedia.org To: Kam Kasravi kamkasr...@yahoo.com Cc: dev@kafka.apache.org dev@kafka.apache.org; Ken Goodhope kengoodh...@gmail.com; Andrew Psaltis psaltis.and...@gmail.com; dibyendu.bhattacha...@pearson.com dibyendu.bhattacha...@pearson.com; camus_

Re: Kafka/Hadoop consumers and producers

We've done a bit of work over at Wikimedia to debianize Kafka and make it behave like a regular service. https://github.com/wikimedia/operations-debs-kafka/blob/debian/debian Most relevant, Ken, is an init script for Kafka: