Re: [VOTE] KIP-714: Client Metrics and Observability
+1 As a member of a team which operates several Kafka clusters, I am unequipped when it comes to troubleshooting issues with project teams that did not understand the importance of configuring client-side monitoring. Kafka represents a fraction of their work and they don't have enough experience, time or interest in trying to understand the meaning behind every metric. I stand 100% behind what Colin stated back in June in the Discuss thread : > Magnus and I explained a few times the reasons why it does matter. Within > most organizations, there are usually several teams using clients, which > are separate from the team which maintains the Kafka cluster. The Kafka > team has the Kafka experts, which makes it the best place to centralize > collecting and analyzing Kafka metrics. Thanks for this KIP. Le mer. 26 janv. 2022 à 16:01, rifer...@riferrei.com a écrit : > +1 > > I think this KIP solves a problem that has been around for some time with > Kafka deployments, which is the ability to assess the current state of a > Kafka architecture but looking at the whole picture. I also share other > folks' concerns regarding adding runtime dependencies to the clients; this > may be problematic for large deployments. Still, I think it is worth > refactoring. > > IMHO, it is a fair trade-off. > > — Ricardo > > > On Jan 26, 2022, at 9:34 AM, Magnus Edenhill wrote: > > > > Hi all, > > > > it's been a while and there's been some more discussions of the KIP which > > have been > > addressed on the KIP page. > > > > I think it's a good time to revive this vote thread and get things > moving. > > > > We're currently at +3 (non-binding) and -1 (non-binding) votes. > > > > Regards, > > Magnus > > > > > > Den mån 1 nov. 2021 kl 21:19 skrev J Rivers : > > > >> +1 > >> > >> Thank you for the KIP! > >> > >> Our organization runs kafka at large scale in a multi-tenant > configuration. > >> We actually have many other enterprises connecting up to our system to > >> retrieve stream data. These feeds vary greatly in volume and velocity. > The > >> peak rates are a multiplicative factor of the nominal. There is extreme > >> skew in our datasets in a number of ways. > >> > >> We don't have time to work with every new internal/external client to > tune > >> their feeds. They need to be able to take one of the many kafka clients > and > >> go off to the races. > >> > >> Being able to retrieve client metrics would be invaluable here as it's > hard > >> and time consuming to communicate out of the enterprise walls. > >> > >> This KIP is important to us to expand the use of our datasets internally > >> and outside the borders of the enterprise. Our clients like the > performance > >> and data safeties related to the kafka connection. The observability has > >> been a problem... > >> > >> Jonathan Rivers > >> jrivers...@gmail.com > >> > >> > >> > >> > >> On Mon, Oct 18, 2021 at 11:56 PM Ryanne Dolan > >> wrote: > >> > >>> -1 > >>> > >>> Ryanne > >>> > >>> On Mon, Oct 18, 2021, 4:30 AM Magnus Edenhill > >> wrote: > >>> > Hi all, > > I'd like to start a vote on KIP-714. > https://cwiki.apache.org/confluence/x/2xRRCg > > Discussion thread: > https://www.mail-archive.com/dev@kafka.apache.org/msg119000.html > > Thanks, > Magnus > > >>> > >> > >
Re: [DISCUSS] Apache Kafka 3.2.0 release
Hello Bruno, Can we also add KIP-808 to the plan ? https://cwiki.apache.org/confluence/display/KAFKA/KIP-808%3A+Add+support+for+different+unix+precisions+in+TimestampConverter+SMT Thank you, Julien Le mar. 15 févr. 2022 à 19:01, Mickael Maison a écrit : > Hi Bruno, > > Thanks for publishing the release plan! > Can we add KIP-769 to the plan? > > Thanks, > Mickael > > On Tue, Feb 15, 2022 at 12:37 PM Bruno Cadonna wrote: > > > > Hi all, > > > > I published a release plan for the Apache Kafka 3.2.0 release here: > > https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+3.2.0 > > > > KIP Freeze: 2 March 2022 > > Feature Freeze: 16 March 2022 > > Code Freeze:30 March 2022 > > > > At least two weeks of stabilization will follow Code Freeze. > > > > Please let me know if should add or remove KIPs from the plan or if you > > have any other objections. > > > > Best, > > Bruno > > > > > > On 04.02.22 16:03, Bruno Cadonna wrote: > > > Hi, > > > > > > I'd like to volunteer to be the release manager for our next > > > feature release, 3.2.0. If there are no objections, I'll send > > > out the release plan soon. > > > > > > Best, > > > Bruno >
Re: [VOTE] KIP-808: Add support for unix epoch precision in TimestampConverter SMT
Hi all, With 3 binding votes (Tom, Mickael, John) and 1 non-binding vote (Donjin), the KIP passes. Thanks, Julien Le ven. 11 févr. 2022 à 20:41, John Roesler a écrit : > Thanks, Julien! > > I’m +1 (binding) > > -John > > On Fri, Feb 11, 2022, at 08:42, Dongjin Lee wrote: > > +1 (non-binding) > > > > Thanks for the KIP! > > > > Thanks, > > Dongjin > > > > On Fri, Feb 11, 2022, 10:16 PM Julien Chanaud > > wrote: > > > >> Bumping this vote. > >> > >> We have 2 binding votes so far. > >> > >> The associated PR is already implemented: > >> https://github.com/apache/kafka/pull/11575 > >> Let me know if you have any feedback. > >> > >> Thanks, > >> Julien > >> > >> PS: Have a look at the KIP-769 voting thread as well :) > >> > >> Le mer. 26 janv. 2022 à 12:46, Mickael Maison > a > >> écrit : > >> > >> > +1 (binding) > >> > > >> > Thanks Julien for the KIP! > >> > > >> > On Wed, Jan 26, 2022 at 12:05 PM Tom Bentley > >> wrote: > >> > > > >> > > Hi Julien, > >> > > > >> > > Thanks again for this KIP. +1 (binding). > >> > > > >> > > Kind regards, > >> > > > >> > > Tom > >> > > > >> > > On Tue, 18 Jan 2022 at 08:15, Julien Chanaud < > chanaud.jul...@gmail.com > >> > > >> > > wrote: > >> > > > >> > > > Hi everyone, > >> > > > > >> > > > I'd like to start a vote for KIP-808: Add support for unix epoch > >> > precision > >> > > > in TimestampConverter SMT > >> > > > > >> > > > https://cwiki.apache.org/confluence/x/GJuqCw > >> > > > > >> > > > Thanks for your help, > >> > > > > >> > > > Julien > >> > > > > >> > > >> >
Re: [VOTE] KIP-808: Add support for unix epoch precision in TimestampConverter SMT
Bumping this vote. We have 2 binding votes so far. The associated PR is already implemented: https://github.com/apache/kafka/pull/11575 Let me know if you have any feedback. Thanks, Julien PS: Have a look at the KIP-769 voting thread as well :) Le mer. 26 janv. 2022 à 12:46, Mickael Maison a écrit : > +1 (binding) > > Thanks Julien for the KIP! > > On Wed, Jan 26, 2022 at 12:05 PM Tom Bentley wrote: > > > > Hi Julien, > > > > Thanks again for this KIP. +1 (binding). > > > > Kind regards, > > > > Tom > > > > On Tue, 18 Jan 2022 at 08:15, Julien Chanaud > > wrote: > > > > > Hi everyone, > > > > > > I'd like to start a vote for KIP-808: Add support for unix epoch > precision > > > in TimestampConverter SMT > > > > > > https://cwiki.apache.org/confluence/x/GJuqCw > > > > > > Thanks for your help, > > > > > > Julien > > > >
Re: [VOTE] KIP-769: Connect APIs to list all connector plugins and retrieve their configuration definitions
Hi Michael, I'm very interested in seeing this feature implemented. +1 (non-binding) Julien Le mar. 1 févr. 2022 à 08:36, Tom Bentley a écrit : > Hi Mickael, > > +1 (binding), thanks! > > Tom > > On Wed, 26 Jan 2022 at 09:42, Viktor Somogyi-Vass > wrote: > > > Hi Michael, > > > > +1 (non-binding) from me. > > > > Viktor > > > > On Thu, Jan 13, 2022 at 11:32 AM Mickael Maison < > mickael.mai...@gmail.com> > > wrote: > > > > > Bumping this vote. > > > > > > We have 2 non-binding votes so far. Please take a look and let me know > > > if you have any feedback. > > > > > > Thanks, > > > Mickael > > > > > > On Mon, Dec 13, 2021 at 10:50 PM Ryanne Dolan > > > wrote: > > > > > > > > +1 (non-binding) > > > > > > > > Ryanne > > > > > > > > On Mon, Dec 13, 2021, 4:18 AM Mickael Maison < > mickael.mai...@gmail.com > > > > > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > I'd like to start a vote on KIP-769 which proposes adding new > > > > > endpoints to the Connect REST API to list all connectors plugins > and > > > > > retrieve their configurations. > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-769%3A+Connect+APIs+to+list+all+connector+plugins+and+retrieve+their+configuration+definitions > > > > > > > > > > Please take a look and let me know if you have any feedback. > > > > > > > > > > Thanks, > > > > > Mickael > > > > > > > > > > >
Re: [DISCUSS] KIP-808: Add support for unix epoch precision in TimestampConverter SMT
Hi Tom, Thanks for your review. My original intent was to use the naming available in TimeUnit doc : https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/TimeUnit.html This led me to follow their logic : "toMicros", "toNanos", etc. By looking around the kafka code to see if there was some sort of consistency, ms/millis/milliseconds are present at multiple places but millis seems to be the least present wording. I've modified the KIP accordingly to use the full naming (seconds, milliseconds, ...) as per your suggestion. I've also added a note on precision loss but I'm not sure my phrasing is clear enough. Please let me know, Julien Le ven. 21 janv. 2022 à 13:20, Tom Bentley a écrit : > Hi Julien, > > Thanks! A couple of other points sprang to mind: > > 1. seconds is a unit, but millis etc are really just prefixes. I can > imagine what my old physics teacher would have to say about mixing these > concepts :-). I would have preferred to use the abbreviations s, ms, µs and > ns (ms in particular would be consistent with the unit names used in > configs), but I'm quite sure many people would struggle to know how to type > µ. Maybe 'us' is an acceptable alternative (it's a pretty common > convention), or perhaps we should use 'seconds', 'milliseconds', > 'microseconds' and 'nanoseconds' as the allowed values, even though they're > a bit long. > > 2. I think the truncation and sub-milliseconds behaviour when converting > to/from java.util.Date etc. should be part of the documentation of the > config. > > Apart from these minor points this is looking good to me. > > Kind regards, > > Tom > > On Wed, 19 Jan 2022 at 22:06, Julien Chanaud > wrote: > > > Hi Tom and thanks for your review, > > > > I agree and I have renamed the field name to unix.precision. > > I've replaced all references to the word "epoch" which I mistakenly used > to > > describe measurements throughout the KIP. > > I have also modified the KIP name as well. > > Before : Add support for unix epoch precision in TimestampConverter SMT > > Now : Add support for different unix precisions in TimestampConverter SMT > > > > Let me know what you think, > > > > Julien > > > > > > Le mer. 19 janv. 2022 à 19:27, Tom Bentley a > écrit : > > > > > Hi Julien, > > > > > > I wonder if the name epoch.precision is a little confusing. An epoch > is a > > > point in time chosen as a origin for a particular calendar system. As > > such > > > it doesn't have a precision. It's only measurements from this point in > > time > > > which have a precision. In the unix case, precisions of seconds, ms, µs > > and > > > ns seem to make the most sense. So I wonder if the name should be > > > unix.precision instead. That also makes it clearer that it only applies > > in > > > the type=unix case. Wdyt? > > > > > > Kind regards, > > > > > > Tom > > > > > > On Thu, 13 Jan 2022 at 13:42, Julien Chanaud > > > > wrote: > > > > > > > Hi Mickael, > > > > > > > > Thank you very much for your feedback and direction. > > > > > > > > I have added the documentation to the "Public Interfaces" chapter > > > > (formatted in a table as I've seen in other KIPs) and I'll put this > > > > KIP to a vote next week as per your suggestion. > > > > > > > > Regards, > > > > > > > > Julien > > > > > > > > Le mer. 12 janv. 2022 à 18:19, Mickael Maison > > > > a écrit : > > > > > > > > > > Hi Julien, > > > > > > > > > > Thanks for the KIP. I looks like a useful improvement to the > > > > > TimestampConverter SMT. > > > > > > > > > > I'd suggest adding the documentation for the new setting to the > KIP. > > > > > I've had to go check your PR to fully understand how you want to > use > > > > > it, both for input and output. > > > > > Apart from that, if you don't get any further feedback, feel free > to > > > > > start a vote. > > > > > > > > > > Thanks, > > > > > Mickael > > > > > > > > > > On Tue, Dec 21, 2021 at 2:19 PM Julien Chanaud < > > > chanaud.jul...@gmail.com> > > > > wrote: > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > Bumping this KIP discussion. > > > > > > It's a small change, entirely backward compatible and I'd love > your > > > > > > feedback on it. > > > > > > Thanks, > > > > > > Julien > > > > > > > > > > > > > > > > > > Le jeu. 9 déc. 2021 à 21:56, Julien Chanaud < > > > chanaud.jul...@gmail.com> > > > > a écrit : > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > I would like to start a discussion for KIP-808 > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-808%3A+Add+support+for+unix+epoch+precision+in+TimestampConverter+SMT > > > > > > > > > > > > > > This seems like a simple change but I suspect there are several > > > > things to consider, most notably regarding the java.util.Date object, > > > which > > > > is at the heart of the conversions. > > > > > > > > > > > > > > Let me know what you think. > > > > > > > > > > > > > > Julien > > > > > > > > > > > > > > > > > > > > >
Re: [DISCUSS] KIP-808: Add support for unix epoch precision in TimestampConverter SMT
Hi Tom and thanks for your review, I agree and I have renamed the field name to unix.precision. I've replaced all references to the word "epoch" which I mistakenly used to describe measurements throughout the KIP. I have also modified the KIP name as well. Before : Add support for unix epoch precision in TimestampConverter SMT Now : Add support for different unix precisions in TimestampConverter SMT Let me know what you think, Julien Le mer. 19 janv. 2022 à 19:27, Tom Bentley a écrit : > Hi Julien, > > I wonder if the name epoch.precision is a little confusing. An epoch is a > point in time chosen as a origin for a particular calendar system. As such > it doesn't have a precision. It's only measurements from this point in time > which have a precision. In the unix case, precisions of seconds, ms, µs and > ns seem to make the most sense. So I wonder if the name should be > unix.precision instead. That also makes it clearer that it only applies in > the type=unix case. Wdyt? > > Kind regards, > > Tom > > On Thu, 13 Jan 2022 at 13:42, Julien Chanaud > wrote: > > > Hi Mickael, > > > > Thank you very much for your feedback and direction. > > > > I have added the documentation to the "Public Interfaces" chapter > > (formatted in a table as I've seen in other KIPs) and I'll put this > > KIP to a vote next week as per your suggestion. > > > > Regards, > > > > Julien > > > > Le mer. 12 janv. 2022 à 18:19, Mickael Maison > > a écrit : > > > > > > Hi Julien, > > > > > > Thanks for the KIP. I looks like a useful improvement to the > > > TimestampConverter SMT. > > > > > > I'd suggest adding the documentation for the new setting to the KIP. > > > I've had to go check your PR to fully understand how you want to use > > > it, both for input and output. > > > Apart from that, if you don't get any further feedback, feel free to > > > start a vote. > > > > > > Thanks, > > > Mickael > > > > > > On Tue, Dec 21, 2021 at 2:19 PM Julien Chanaud < > chanaud.jul...@gmail.com> > > wrote: > > > > > > > > Hi everyone, > > > > > > > > Bumping this KIP discussion. > > > > It's a small change, entirely backward compatible and I'd love your > > > > feedback on it. > > > > Thanks, > > > > Julien > > > > > > > > > > > > Le jeu. 9 déc. 2021 à 21:56, Julien Chanaud < > chanaud.jul...@gmail.com> > > a écrit : > > > > > > > > > > Hi everyone, > > > > > > > > > > I would like to start a discussion for KIP-808 > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-808%3A+Add+support+for+unix+epoch+precision+in+TimestampConverter+SMT > > > > > > > > > > This seems like a simple change but I suspect there are several > > things to consider, most notably regarding the java.util.Date object, > which > > is at the heart of the conversions. > > > > > > > > > > Let me know what you think. > > > > > > > > > > Julien > > > > > > > > > >
[VOTE] KIP-808: Add support for unix epoch precision in TimestampConverter SMT
Hi everyone, I'd like to start a vote for KIP-808: Add support for unix epoch precision in TimestampConverter SMT https://cwiki.apache.org/confluence/x/GJuqCw Thanks for your help, Julien
Re: [DISCUSS] KIP-808: Add support for unix epoch precision in TimestampConverter SMT
Hi Mickael, Thank you very much for your feedback and direction. I have added the documentation to the "Public Interfaces" chapter (formatted in a table as I've seen in other KIPs) and I'll put this KIP to a vote next week as per your suggestion. Regards, Julien Le mer. 12 janv. 2022 à 18:19, Mickael Maison a écrit : > > Hi Julien, > > Thanks for the KIP. I looks like a useful improvement to the > TimestampConverter SMT. > > I'd suggest adding the documentation for the new setting to the KIP. > I've had to go check your PR to fully understand how you want to use > it, both for input and output. > Apart from that, if you don't get any further feedback, feel free to > start a vote. > > Thanks, > Mickael > > On Tue, Dec 21, 2021 at 2:19 PM Julien Chanaud > wrote: > > > > Hi everyone, > > > > Bumping this KIP discussion. > > It's a small change, entirely backward compatible and I'd love your > > feedback on it. > > Thanks, > > Julien > > > > > > Le jeu. 9 déc. 2021 à 21:56, Julien Chanaud a > > écrit : > > > > > > Hi everyone, > > > > > > I would like to start a discussion for KIP-808 > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-808%3A+Add+support+for+unix+epoch+precision+in+TimestampConverter+SMT > > > > > > This seems like a simple change but I suspect there are several things to > > > consider, most notably regarding the java.util.Date object, which is at > > > the heart of the conversions. > > > > > > Let me know what you think. > > > > > > Julien > > >
Re: [DISCUSS] KIP-808: Add support for unix epoch precision in TimestampConverter SMT
Hi everyone, Bumping this KIP discussion. It's a small change, entirely backward compatible and I'd love your feedback on it. Thanks, Julien Le jeu. 9 déc. 2021 à 21:56, Julien Chanaud a écrit : > > Hi everyone, > > I would like to start a discussion for KIP-808 > https://cwiki.apache.org/confluence/display/KAFKA/KIP-808%3A+Add+support+for+unix+epoch+precision+in+TimestampConverter+SMT > > This seems like a simple change but I suspect there are several things to > consider, most notably regarding the java.util.Date object, which is at the > heart of the conversions. > > Let me know what you think. > > Julien >
[DISCUSS] KIP-808: Add support for unix epoch precision in TimestampConverter SMT
Hi everyone, I would like to start a discussion for KIP-808 https://cwiki.apache.org/confluence/display/KAFKA/KIP-808%3A+Add+support+for+unix+epoch+precision+in+TimestampConverter+SMT This seems like a simple change but I suspect there are several things to consider, most notably regarding the java.util.Date object, which is at the heart of the conversions. Let me know what you think. Julien
Permission to create KIP
Hello, I would like to be able to contribute to Apache Kafka and create a KIP. wiki id: twobeeb jira id: Twobeeb Thanks, Julien
[jira] [Created] (KAFKA-13511) Update TimestampConverter SMT to support unix epoch as millis, micros, and seconds
Julien Chanaud created KAFKA-13511: -- Summary: Update TimestampConverter SMT to support unix epoch as millis, micros, and seconds Key: KAFKA-13511 URL: https://issues.apache.org/jira/browse/KAFKA-13511 Project: Kafka Issue Type: Improvement Components: KafkaConnect Reporter: Julien Chanaud Currently, the SMT TimestampConverter can convert Timestamp from either source String, Long or any target Date to String, Long or Date. The problem is that Long source or target is required to be epoch in milliseconds. In many cases, epoch is represented with different precisions. This leads to several Jira tickets : * KAFKA-12364 * KAFKA-10561 I propose to add a new config to TimestampConverter called "epoch.precision" which defaults to "millis" so as to not impact existing code, and allows for more precisions : seconds, millis, micros. "transforms": "TimestampConverter", "transforms.TimestampConverter.type": "org.apache.kafka.connect.transforms.TimestampConverter$Value","transforms.TimestampConverter.field": "event_date" *"transforms.TimestampConverter.epoch.precision": "micros"* "transforms.TimestampConverter.target.type": "Timestamp" -- This message was sent by Atlassian Jira (v8.20.1#820001)
