Re: [DISCUSS] KIP-1000: List Client Metrics Configuration Resources

Let's take this moment to celebrate the creation of KIP-1000! On Tue, Nov 7, 2023 at 3:49 PM Andrew Schofield < andrew_schofield_j...@outlook.com> wrote: > Hi, > I would like to start discussion of a small KIP which fills a gap in the > administration of client metrics configuration. > > >

Re: [ANNOUNCE] New PMC chair: Mickael Maison

Thank you for all your hard work Jun - that's a decade-long legacy! And congratulations to you Mickael! On Fri, Apr 21, 2023 at 11:20 AM Josep Prat wrote: > Thanks Jun for your work as Chair all these years! > Congratulations Mickael! > > Best, > > ——— > Josep Prat > > Aiven Deutschland GmbH >

Re: [ANNOUNCE] New Kafka PMC Member: A. Sophie Blee-Goldman

Congratulations Sophie! I’m glad to see you made as a PMC member! Well earned. > On Aug 1, 2022, at 6:42 PM, Guozhang Wang wrote: > > Hi everyone, > > I'd like to introduce our new Kafka PMC member, Sophie. She has been a > committer since Oct. 2020 and has been contributing to the community

Re: Support regarding the connection of Redshift to Kafka

Hi Israa I suspect you could use the JDBC Source Connector to connect to Redshift. You may also need to play around with the Redshift JDBC driver to get it working. : https://github.com/aws/amazon-redshift-jdbc-driver For clarity, I haven't done this myself but I suspect this would be a

Re: newbie: PR 11955 review

ple. > > I restarted the builds. > > Best, > Bruno > > On 31.03.22 15:51, Adam Bellemare wrote: > > It could be an intermittent error. I think there is a way to issue a > retest > > via a comment in github ("retest this") but I am not sure if that's the > >

Re: newbie: PR 11955 review

It could be an intermittent error. I think there is a way to issue a retest via a comment in github ("retest this") but I am not sure if that's the precise command, nor if you will have the ability to kick it off. On Thu, Mar 31, 2022 at 9:12 AM Rajani Karuturi wrote: > Hi All, > I am new

Re: Gigantic list of Avro spec issues

Hi Askar This is certainly an extensive list. I wanted to email you just to let you know that *someone* has seen it. Though I can't speak to all of it, I do have a few of my own impressions: > How it is happened that such good format has so bad spec? How it is happened that *best* format for

Re: [VOTE] KIP-775: Custom partitioners in foreign key joins

+1 (non-binding) Glad to see this in here :) On Tue, Sep 28, 2021 at 5:11 PM Bill Bejeck wrote: > +1 (binding) > > On Tue, Sep 28, 2021 at 12:59 PM Matthias J. Sax wrote: > > > +1 (binding) > > > > On 9/28/21 8:29 AM, Victoria Xia wrote: > > > Hi all, > > > > > > I'd like to start a vote for

Re: Kafka Issue

Hi folks You’ll need to host the images externally and send a link. The email server doesn’t send images due to the sheer number of recipients. Try using something like imgur Adam > On Aug 19, 2021, at 2:18 AM, Roel Angeles > wrote: > >  > Hi Luke, > > Resending the screenshot. > > >

Re: [ANNOUNCE] New Kafka PMC Member: Konstantine Karantasis

Congratulations Konstantine!!! On Tue, Jun 22, 2021 at 10:34 AM Jeremy Custenborder < jcustenbor...@gmail.com> wrote: > NICE work buddy! > > On Tue, Jun 22, 2021 at 9:33 AM Rankesh Kumar > wrote: > > > > Yay! Congratulations, KK! > > > > Best regards, > > Rankesh Kumar > > Partner Solutions

Re: [kafka-clients] Re: [ANNOUNCE] Apache Kafka 2.8.0

Wow, 2.8! I remember 0.8.2.0 :) Thanks for all the work everyone, and thank you John for putting together the release! On Mon, Apr 19, 2021 at 3:48 PM Israel Ekpo wrote: > This is fantastic news! > > Thanks everyone for contributing and thanks John for managing the release. > > On Mon, Apr 19,

Re: [ANNOUNCE] New Kafka PMC Member: Randall Hauch

Congratulations Randall ! > On Apr 17, 2021, at 6:24 PM, Randall Hauch wrote: > > Thanks for the kind responses, everyone! > > Best regards, > > Randall > >> On Sat, Apr 17, 2021 at 4:00 PM Guozhang Wang wrote: >> >> Congratulations Randall ! Well deserved. >> >> Guozhang >> >>> On Fri,

Re: [ANNOUNCE] New Kafka PMC Member: Bill Bejeck

Congratulations Bill!! > On Apr 17, 2021, at 5:20 PM, Kowshik Prakasam > wrote: > > Congrats Bill! > > > Cheers, > Kowshik > >> On Mon, Apr 12, 2021, 11:15 AM Randall Hauch wrote: >> >> Congratulations, Bill! >> >>> On Mon, Apr 12, 2021 at 11:02 AM Guozhang Wang wrote: >>> >>>

Re: Known issues with time stepping

Did you check the JIRAs? On Mon, Apr 5, 2021 at 6:58 PM Tirtha Chatterjee < tirtha.p.chatter...@gmail.com> wrote: > Hi team > > Are there any known issues with Kafka that can happen because of the system > time getting stepped forward or backward in one shot? This can happen > because of NTP

Re: [DISCUSS] Please review 2.8.0 blog post

Read it all. It looks good to me in terms of structure and content. I am not sufficiently up to date on all the features that are otherwise included in 2.8.0, but the ones listed seem very prominent! On Thu, Apr 1, 2021 at 4:39 PM John Roesler wrote: > Hello all, > > In the steady march

[jira] [Created] (KAFKA-12323) Record timestamps not populated in event

Adam Bellemare created KAFKA-12323: -- Summary: Record timestamps not populated in event Key: KAFKA-12323 URL: https://issues.apache.org/jira/browse/KAFKA-12323 Project: Kafka Issue Type: Bug

Re: Why many "Load Bug xxx" JIRA bug by Tim?

If we do look to enable Captchas, I think it would be important that we avoid corporate offerings (eg: Google's). On Thu, Jan 7, 2021 at 12:12 PM Govinda Sakhare wrote: > Hi, > > If it is possible, we should configure/enable Captcha to prevent automated > spamming attacks. > > Thanks > Govinda

Serious Java Consumer performance issues / expectations vs. librdkafka

Hi I am trying to use a plain Java consumer (over SSL) to consume a very large amount of historic data (20+TB across 20+ partitions). Consumption performance is very low when fully parallelized. We are seeing about* 200k rec/s* with java consumer versus *950k rec/s* with librdkafka We are seeing

Re: virtual KIP meeting for KIP-405

Hello I am interested in attending, mostly just to listen and observe. Thanks ! > On Aug 20, 2020, at 6:20 PM, Jun Rao wrote: > > Hi, everyone, > > We plan to have weekly virtual meetings for KIP-405 to discuss progress and > outstanding issues, starting from this coming Tuesday at 9am PT.

Re: [VOTE] KIP-657: Add Customized Kafka Streams Logo

I prefer Design B, but given that I missed the discussion thread, I think it would be better without the Otter obscuring any part of the Kafka logo. On Thu, Aug 13, 2020 at 6:31 PM Boyang Chen wrote: > Hello everyone, > > I would like to start a vote thread for KIP-657: > >

Re: [ANNOUNCE] New Kafka PMC Member: John Roesler

Congratulations John! You have been an excellent help to me and many others. I am pleased to see this! > On Aug 10, 2020, at 5:54 PM, Bill Bejeck wrote: > > Congrats! > >> On Mon, Aug 10, 2020 at 4:52 PM Guozhang Wang wrote: >> >> Congratulations! >> >>> On Mon, Aug 10, 2020 at 1:11 PM

Re: [VOTE] KIP-614: Add Prefix Scan support for State Stores

LGTM +1 non-binding On Sun, Jul 19, 2020 at 4:13 AM Sagar wrote: > Hi All, > > Bumping this thread to see if there are any feedbacks. > > Thanks! > Sagar. > > On Tue, Jul 14, 2020 at 9:49 AM John Roesler wrote: > > > Thanks for the KIP, Sagar! > > > > I’m +1 (binding) > > > > -John > > > > On

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

My 2 cents - I agree with Colin. I think that it's important that the metadata not grow unbounded without being delegated to external storage. Indefinite long-term storage of entity data in Kafka can result in extremely large datasets where the vast majority of data is stored in the external

Re: Feedback: Print schemaId using bin/kafka-dump-log.sh

Hi Mohanraj While I see the usefulness of your suggestion, the main issue is that you're using the Confluent schema registry's conventions and hardwiring them into Kafka core. Given that Confluent's standards are not part of Kafka's official standards, I do not think you will get approval to

Re: [ANNOUNCE] New committer: Boyang Chen

Just adding my congratulations, Boyang! Thank you for all your contributions and effort! On Tue, Jun 23, 2020 at 9:14 PM Kowshik Prakasam wrote: > Congrats, Boyang! :) > > > Cheers, > Kowshik > > On Tue, Jun 23, 2020 at 8:43 AM Aparnesh Gaurav > wrote: > > > Congrats Boyang. > > > > On Tue, 23

Re: [DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

ng out the brokers to perhaps less trustworthy sources? Adam On Sun, May 10, 2020 at 7:52 PM Adam Bellemare wrote: > @Ryanne > > Seems that could still get us per-topic keys (vs encrypting the entire > > volume), which would be my main requirement. > > Agreed, I thin

Re: [DISCUSS] KIP-614: Add Prefix Scan support for State Stores

comparator in JNI it is very expensive and the cost of > JNI overwhelms its benefits. If you've got some ideas around it please lmk > as well. > > Guozhang > > > > > On Tue, May 12, 2020 at 6:26 AM Adam Bellemare > wrote: > > > Hi Sagar > > > >

Re: [DISCUSS] KIP-614: Add Prefix Scan support for State Stores

Hi Sagar I implemented a very similar interface for KIP-213, the foreign-key joiner. We pulled it out of the final implementation and instead used RocksDB range instead. You can see the particular code where we use RocksDB.range(...) to get the same iterator result.

Re: [DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

the > newly > > encrypted ones. > > > > Seems that could still get us per-topic keys (vs encrypting the entire > > volume), which would be my main requirement. > > > > Not really "end-to-end", but combined with TLS or something, seems > > reas

Re: [DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

Hi All I typed up a number of replies which I have below, but I have one major overriding question: Is there a reason we aren't implementing encryption-at-rest almost exactly the same way that most relational databases do? ie: https://wiki.postgresql.org/wiki/Transparent_Data_Encryption I ask

[jira] [Resolved] (KAFKA-9732) Kafka Foreign-Key Joiner has unexpected default value used when a table is created via a stream+groupByKey+reduce

[ https://issues.apache.org/jira/browse/KAFKA-9732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Bellemare resolved KAFKA-9732. --- Resolution: Not A Problem Issue was with reporter's usage of the API. > Kafka Foreign-

[jira] [Created] (KAFKA-9732) Kafka Foreign-Key Joiner has unexpected default value used when a table is created via a stream+groupByKey+reduce

Adam Bellemare created KAFKA-9732: - Summary: Kafka Foreign-Key Joiner has unexpected default value used when a table is created via a stream+groupByKey+reduce Key: KAFKA-9732 URL: https://issues.apache.org/jira

Re: Can I ask protocol questions here?

Hi Chris I think it’s fine to ask it here. I’m not aware of any rules against it. Adam > On Mar 9, 2020, at 10:43 AM, Chris Done wrote: > > Hi all, > > I'm writing a Kafka client at the protocol level and was wondering whether > here, or the users@ mailing list was more appropriate for

Re: Mistake in official documentation ?

Screenshot didn't arrive for me - you may need to host it in an image site (ie: imgur), not sure if the mailing list will allow it to be attached. It could very well be a mistake. highlight it and send it to dev@kafka.apache.org and if it is, we'll make a ticket and address it. On Thu, Feb 6,

Re: Is there a way to auto scale topic partitions in kafka?

There is no way to automatically scale it, but you could write a script to increase the partition count using the command line tools, and trigger it on certain metrics. One thing to consider is that any *keyed* events would need to be rewritten to topics that have their partition count increased.

Re: [RESULTS] [VOTE] Release Kafka version 2.4.0

or -1 votes. >> >> +1 votes >> PMC Members: >> * Gwen Shapira >> * Jun Rao >> * Guozhang Wang >> >> Committers: >> * Mickael Maison >> >> Community: >> * Adam Bellemare >> * Israel Ekpo >> >> 0 vote

Re: [VOTE] 2.4.0 RC4

- All PGP signatures are good - All md5, sha1sums and sha512sums pass Initial test results: 1310 tests completed, 2 failed, 17 skipped > Task :core:integrationTest FAILED The failed tests: SaslSslAdminClientIntegrationTest. testElectPreferredLeaders SslAdminClientIntegrationTest.

Re: [VOTE] 2.4.0 RC1

n with --scan to get full > insights. > > * Get more help at https://help.gradle.org > > BUILD FAILED in 699ms > > On Tue, Nov 26, 2019 at 1:31 PM Manikumar > wrote: > > > Hi All, > > > > Please download, test and vote the RC1 in order to provide qual

Re: [VOTE] 2.4.0 RC1

Hello, Ran Eric's test script: $ git clone https://github.com/elalonde/kafka $ ./kafka/bin/verify-kafka-rc.sh 2.4.0 https://home.apache.org/~manikumar/kafka-2.4.0-rc1 - All PGP signatures are good - All md5, sha1sums and sha512sums pass - Had

Re: [ANNOUNCE] New committer: John Roesler

Congratulations John, and thanks for all your help on KIP-213! > On Nov 12, 2019, at 6:24 PM, Bill Bejeck wrote: > > Congratulations John! > > On Tue, Nov 12, 2019 at 6:20 PM Matthias J. Sax > wrote: > >> Congrats John! >> >> >>> On 11/12/19 2:52 PM, Boyang Chen wrote: >>> Great work John!

Re: [VOTE] - KIP-213 (new vote) - Simplified and revised.

Awesome. Thanks John for fixing this! On Thu, Oct 17, 2019 at 3:07 PM John Roesler wrote: > Hello all, > > While writing some new test cases for foreign key joins (as accepted > in KIP-213), I realized that there was an oversight in the review > process: we only proposed to add join methods

Re: server-side message filter

I do not think this will be implemented in the server. Why not simply create a consumer that filters as required and outputs to a new topic? > On Aug 28, 2019, at 7:48 AM, Xiangyuan LI wrote: > > Hi, > I want to know whether kafka has any plan to filter message on > server-side.this can

Re: ACL for group creation?

> hard to do in a compatible way. > > cheers, > Colin > > > On Wed, Aug 21, 2019, at 12:05, Adam Bellemare wrote: > > +users mailing list > > > > David, > > > > I don't think I really understand your email. Are you saying that this > can > &

Re: ACL for group creation?

application from > creating new groups. (READ Acl on Group resource with a specific name). > > Best, > David > > On Mon, Aug 19, 2019 at 9:01 PM Adam Bellemare > wrote: > > > Hi All > > > > I am looking through the Confluent docs and core Kafka docs and d

ACL for group creation?

Hi All I am looking through the Confluent docs and core Kafka docs and don't see an ACL for group creation: https://docs.confluent.io/current/kafka/authorization.html#acl-format and https://kafka.apache.org/documentation/#security_authz My scenario is simple: We use the consumer group as the

Re: KIP-382 + Kafka Streams Question

there is no ordering guarantee across > different topics in the same stream, so materialization could be > inconsistent between the two clusters if, say, the same users's email was > changed to different values at the same millisecond in both clusters. This > may or may not be a probl

Re: KIP-382 + Kafka Streams Question

usters, as you'd get two > emails. So you could run it in one cluster and use offset translation to > migrate during failover. The send-email app is stateless, so you just need > to translate and reset offsets (there is no internal state to rebuild). > > 2) you want to use the table ch

Re: KIP-382 + Kafka Streams Question

> Streams would need to start back at offset 0 and re-process everything in > order to rebuild state (which you don't want to do during a disaster, > especially!). The same logic applies to using Streams with MM2. You _could_ > failover by resetting the app and rebuilding all the missing s

KIP-382 + Kafka Streams Question

Hi Ryanne I have a quick question for you about Active+Active replication and Kafka Streams. First, does your org /do you use Kafka Streams? If not then I think this conversation can end here. ;) Secondly, and for the broader Kafka Dev group - what happens if I want to use Active+Active

Re: Nag!! KAFKA-8629 - Some feedback wanted

Hey Andy Any follow up to this? I think this is an important feature and would love to see it move forward. I think the main thing now is just getting a better handle on the limitations as questioned by John to get a better idea of what the terrain looks like. From there we can spec out the

Re: Stopping All Tasks When a New Connector Added

Hi Luying Would you be willing to make a PR to address this? It seems that you have already done most of the work. Thanks Adam On Thu, Jul 18, 2019 at 11:00 PM Liu Luying wrote: > Hi all, > I have noticed that Kafka Connect 2.3.0 will stop all existing tasks and > then start all the tasks,

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

dam On Thu, Jul 11, 2019 at 9:57 AM Jan Filipiak wrote: > > > On 10.07.2019 06:25, Adam Bellemare wrote: > > In my experience (obviously empirical) it seems that many people just > want > > the ability to join on foreign keys for the sake of handling all the > >

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

ntically ok to emit, it > really doesn't seem worthwhile to take on all the extra complexity > proposed earlier in this conversation. > > A much simpler solution, if someone is really struggling with extra > tombstones, would be just be to slap an LRU cache on the result table >

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

think of one myself. With that being said, I am far more inclined to actually then support just option #2 above and only have LEFT joins, forgoing INNER completely since it would not be a true inner join. Adam On Thu, Jul 4, 2019 at 8:50 AM Adam Bellemare wrote: > Hi Matthias > >

Re: Nag!! KAFKA-8629 - Some feedback wanted

Hi Andy Can you elaborate on two things for clarity’s sake? 1) Are there any alternatives we can use for JMX instead of commenting it out? 2) Are there any other limitations with Graal VM that may impede future development? The second one is more difficult to answer I suspect because it

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

d introducing unintuitive behavior. > > > Would be good to get input from others and how they judge the impact. I > think it might be still worth to explore how complex S-1 would be. If we > think it's too complex it might be a good argument to just accept the > unnecessary t

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

is quite bad (in very old > releases we had a similar issue and fixed it). However, I am also not > 100% sure if the solutions I came up with are good enough to justify > them. (Personally, I slightly tend to prefer S-2 because I think that > the additional store is less of an issue

Re: Synchronized consumption + processing based on timestamps?

nly achieve this, if > > 1) you don't use two clients, but only one client that reads both > partitions > > or > > 2) let both clients exchange data about their time progress > > > -Matthias > > > On 7/2/19 6:01 PM, Adam Bellemare wrote: > > Hi All > &g

Synchronized consumption + processing based on timestamps?

Hi All The use-case is pretty simple. Lets say we have a history of events with the following: key=userId, value = (timestamp, productId) and we want to remap it to (just as we would with an internal topic): key=productId, value=(original_timestamp, userId) Now, say I have 30 days of backlog,

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

nore the events as it would ruin the delivery guarantees. At this point it seems to me that they would have to do a full streams reset for that applicationId. Am I incorrect in this? Adam On Fri, Jun 28, 2019 at 9:19 AM Adam Bellemare wrote: > Hi Matthias > > Yes, thanks for the questions

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

ke an Enum in Java-land and a byte on the wire > > is a good choice. > > > > Agreed on the right and full outer joins, it doesn't seem necessary > > right now, although I am happy to see the left join "join" the party, > > since as you said, we were so close

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

le)", and > inputs like: > LHS: > "a": 1 > "b": 2 > > RHS: > 1: true > 3: false > > a full outer join would produce: > "a": (1, true) > "b": (2, null) > null(3): (null, false) > > which can be correctly upd

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

pgrade completely to 2.4 before deploying a FK join. > > Just want to make sure we anticipate these issues in case it affects > the design at all. > > Thanks, > -John > > On Wed, Jun 26, 2019 at 2:38 PM Adam Bellemare > wrote: > > > > Sigh... Forgot the link: >

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

Sigh... Forgot the link: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=74684836=78=74 I'll update it when I validate that there are no issues with removing the SubscriptionResponseWrapper boolean. On Wed, Jun 26, 2019 at 3:37 PM Adam Bellemare wrote: > >Mayb

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

load is small, and they are trivial to handle > downstream. If users want to, they can materialize the join result to > suppress any extra tombstones, so there's a way out. > > Thanks for the awesome idea. It's better than what I was thinking. > -john > > On Wed, Jun 26, 2019 at

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

Thanks John. I'm looking forward to any feedback on this. In the meantime I will work on the unit tests to ensure that we have well-defined and readable coverage. At the moment I cannot see a way around emitting (k,null) whenever we emit an event that lacks a matching foreign key on the RHS,

Re: [RESULT] [VOTE] 2.3.0 RC3

Thanks for the hard work Colin, and everyone else who helped get this out! On Mon, Jun 24, 2019 at 1:38 PM Colin McCabe wrote: > Hi all, > > This vote passes with 6 +1 votes (3 of which are binding) and no 0 or -1 > votes. Thanks to everyone who voted. > > +1 votes > PMC Members: > * Ismael

Re: [DISCUSS] KIP-360: Improve handling of unknown producer

Ach - Sorry. I meant Jason. I had just read a John Roesler email. On Mon, Mar 25, 2019 at 5:21 PM Adam Bellemare wrote: > Hi John > > What is the status of this KIP? > > My teammates and I are running into the "UNKNOWN_PRODUCER_ID" error on > 2.1.1 for a multitude o

Re: [DISCUSS] KIP-360: Improve handling of unknown producer

Hi John What is the status of this KIP? My teammates and I are running into the "UNKNOWN_PRODUCER_ID" error on 2.1.1 for a multitude of our internal topics, and I suspect that a proper fix is needed. Adam On Mon, Jan 7, 2019 at 7:42 PM Guozhang Wang wrote: > Thanks Jason. The proposed

Re: Question on performance data for Kafka vs NATS

One more thing to note: You are looking at regular, base NATS. On its own, it is not a direct 1-1 comparison to Kafka because it lacks things like data retention, clustering and replication. Instead, you would want to compare it to NATS-Streaming, (

Re: KIP-213- [DISCUSS] - Three follow-up discussion points - topic partitioning, serializers, hashers

n hashing, but 128-bit murmur3 sounds reasonable > > to me > > > > > > > > Btw: I think we can have this discussion on the PR -- no need to concern > > the mailing list (it's a lot of people that are subscribed). > > > > > > > > -Matthi

KIP-213- [DISCUSS] - Three follow-up discussion points - topic partitioning, serializers, hashers

Hey folks I have been implementing the KIP as outlined in https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+Support+non-key+joining+in+KTable, and I have run into a few points to consider that we did not include in the original. *1) Do all input topics need to have the same partitions or

Re: [DISCUSS] KIP-437: Custom replacement for MaskField SMT

re effort, so it is better to do > it as separate issue. > > Kind Regards, > Valeria > > пт, 15 мар. 2019 г. в 17:35, Adam Bellemare : > > > Hi Valeria > > > > Thanks for the KIP. I admit my knowledge on Kafka Connect transforms is a > > bit rusty,

Re: [DISCUSS] KIP-437: Custom replacement for MaskField SMT

Hi Valeria Thanks for the KIP. I admit my knowledge on Kafka Connect transforms is a bit rusty, however - Is there any other way to currently achieve this same functionality outlined in your KIP using existing transforms? Thanks On Thu, Mar 14, 2019 at 12:05 PM Valeria Vasylieva <

Re: [VOTE] - KIP-213 (new vote) - Simplified and revised.

ding the parent page) with the newest > status as well? Thanks. > > > Guozhang > > On Mon, Mar 11, 2019 at 6:12 PM Adam Bellemare > wrote: > > > Thanks to everyone, including John, Guozhang, Matthias and Jan for all > the > > help! > > > >

Re: [VOTE] - KIP-213 (new vote) - Simplified and revised.

gt; +1 (binding) > > > > -Matthias > >> On 2/15/19 11:27 AM, Adam Bellemare wrote: >> Hi Bill >> >> Now that you are a committer, does your vote add a +1 to binding? Can you >> recast it if you believe this is a sound decision? I am eager

Re: KIP-213 - Scalable/Usable Foreign-Key KTable joins - Rebooted.

reams aggressively deletes messages from the reparation > > topics once it knows they are handled. Thus, we don't need to try and > cater > > to the log compactor. > > > > Thanks, > > -John > > > > On Mon, Mar 11, 2019 at 9:10 AM Adam Bellemare > &g

Re: KIP-213 - Scalable/Usable Foreign-Key KTable joins - Rebooted.

For the sake of expediency, I updated the KIP with what I believe we have discussed. https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+Support+non-key+joining+in+KTable#KIP-213Supportnon-keyjoininginKTable-Tombstones On Mon, Mar 11, 2019 at 8:20 AM Adam Bellemare wrote: > My o

Re: KIP-213 - Scalable/Usable Foreign-Key KTable joins - Rebooted.

t; Would this pose some sort of verbosity problem in the internal topics, > >> especially if we have to rebuild state off of them? > > I don't see an issue atm. Can you elaborate how this relates to rebuild > state? > > > -Matthias > > On 3/10/19 12:25 PM, Adam B

Re: KIP-213 - Scalable/Usable Foreign-Key KTable joins - Rebooted.

this as "evidence" that we're better off > > eliminating those duplicates from the start. Guozhang's proposal seems > like > > a lightweight solution to the problem, so FWIW, I'm in favor. > > > > Thanks, > > -John > > > > On Fri, Mar 8, 20

Re: KIP-213 - Scalable/Usable Foreign-Key KTable joins - Rebooted.

ack from the right hand side does not match with the left hand > side any more, i.e. we will only send: > > (k1, (f-k1, v2-v3)) > > to down streams once. > > WDYT? > > > Guozhang > > > On Wed, Mar 6, 2019 at 7:58 AM Adam Bellemare > wrote: > > > Ah yes, I re

Re: KIP-213 - Scalable/Usable Foreign-Key KTable joins - Rebooted.

Ah yes, I recall it all now. That answers that question as to why I had caching disabled. I can certainly re-enable it since I believe the main concern was simply about reconciling those two iterators. A lack of knowledge there on my part. Thank you John for weighing in - we certainly both do

Re: KIP-213 - Scalable/Usable Foreign-Key KTable joins - Rebooted.

Hi Matthias Thank you for the feedback! I appreciate your well thought-out questions. I have tried to answer and comment on everything that I know below. *> Q) For the materialized combined-key store, why do we need to disable> caching? And why do we need to flush the store?* This is an

Re: [VOTE] 2.2.0 RC0

Downloaded, compiled and passed all tests successfully. Ran quickstart (https://kafka.apache.org/quickstart) up to step 6 without issue. (+1 non-binding). Adam On Mon, Feb 25, 2019 at 9:19 PM Matthias J. Sax wrote: > @Stephane > > Thanks! You are right (I copied the list from an older

Re: Statestore restoration & scaling questions - possible KIP as well.

is complete. Thanks for your help - I appreciate you taking the time to reply. Adam On Wed, Feb 20, 2019 at 8:38 PM Guozhang Wang wrote: > Hello Adam, > > Sorry for being late replying on this thread, I've put my comments inlined > below. > > On Sun, Feb 3, 2019 at 7:34 AM Adam

Re: [VOTE] - KIP-213 (new vote) - Simplified and revised.

think, he needs to re-cast his vote. > > -Matthias > >> On 2/15/19 5:49 AM, Adam Bellemare wrote: >> Hi all >> >> Since Bill is now a committer, the vote is changed to 3 binding and 3 >> non-binding (unless I am somehow mistaken - please let me kno

Re: [VOTE] - KIP-213 (new vote) - Simplified and revised.

Hi all Since Bill is now a committer, the vote is changed to 3 binding and 3 non-binding (unless I am somehow mistaken - please let me know!). In this case, I believe the vote passes. Thanks Adam > On Jan 24, 2019, at 7:28 PM, Adam Bellemare wrote: > > Bumping this vote becaus

Re: [ANNOUNCE] New Committer: Bill Bejeck

Great work Bill! Well deserved! > On Feb 14, 2019, at 3:55 AM, Edoardo Comar wrote: > > Well done Bill! > -- > > Edoardo Comar > > IBM Event Streams > IBM UK Ltd, Hursley Park, SO21 2JN > > > > > From: Rajini Sivaram > To: dev >

Re: Statestore restoration & scaling questions - possible KIP as well.

Bump - hoping someone has some insight. Alternately, redirection to a more suitable forum. Thanks On Sun, Feb 3, 2019 at 10:25 AM Adam Bellemare wrote: > Hey Folks > > I have a few questions around the operations of stateful processing while > scaling nodes up/down, and a

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

Hi Harsha Agreed with Eno: "However I'd argue that this KIP is about strengthening the role of Kafka as a long-term storage service" - Put the focus on the single source of truth being from Kafka, with clients not needing to source from multiple data sources. Also, clarify if the segment is

Statestore restoration & scaling questions - possible KIP as well.

out requiring any pauses for rebuilding state. I am interested in kicking this off as a KIP if so, but would appreciate any JIRAs or related KIPs to read up on prior to digging into this. Thanks Adam Bellemare

Re: Why is enable.auto.commit=true the default value for consumer?

the previous > batch. > > In your example, the app might come back and have to re-execute all > > the records in the uncommitted batch but it will never skip over > > unprocessed records. > > > > -Original Message- > > From: Adam Bellemare [mailto:adam.bellem

Why is enable.auto.commit=true the default value for consumer?

As the question indicates. Should this not be default false? I think this is a bit nefarious to someone launching their application into production without testing it extensively around failure modes. I can see a scenario where a consumer polls for events, processes them, produces to output

Re: [VOTE] - KIP-213 (new vote) - Simplified and revised.

Bumping this vote because I don't want it to languish. It is very unlikely to go into 2.2 at this point, but I would like to avoid resurrecting a dead thread in 30 days time. On Tue, Jan 15, 2019 at 5:07 PM Adam Bellemare wrote: > All good Matthias. If it doesn’t get in for 2.2 I’ll just

Re: [VOTE] - KIP-213 (new vote) - Simplified and revised.

romise. Even if > we make the voting deadline, it might also be tight to review the PR, as > it seems to be big and complicated. > > I'll try my very best to get it into 2.2... > > > -Matthias > >> On 1/15/19 3:27 AM, Adam Bellemare wrote: >> If I can get one

Re: [VOTE] - KIP-213 (new vote) - Simplified and revised.

o and am very excited > > about it. It would be a huge help on the project I'm working on. > > > > On Fri, Jan 11, 2019 at 9:05 AM Adam Bellemare > > > wrote: > > > >> Thanks all - > >> > >> So far that's +2 Binding, +2 non-bindi

Re: [VOTE] - KIP-213 (new vote) - Simplified and revised.

as well. > > > > > > Thanks for sticking with this, Adam! > > > -John > > > > > > On Wed, Jan 9, 2019 at 6:22 PM Guozhang Wang > wrote: > > > > > > > Hello Adam, > > > > > > > > I'm +1 on the current proposal

Re: [VOTE] KIP-349 Priorities for Source Topics

s we should consider some kind of API to do > pluggable buffering on the client side. > > In any case, this needs more discussion. We need to be clear and definite > about what use cases we want to solve, and the tradeoffs we're making to > solve them. For now, I have to reiterat

Re: [VOTE] KIP-349 Priorities for Source Topics

Looks good to me then! +1 non-binding > On Jan 10, 2019, at 1:22 PM, Afshartous, Nick wrote: > > > Hi Adam, > > > This change is only intended for the basic consumer API. > > > Cheers, > > -- > >Nick > > >

[VOTE] - KIP-213 (new vote) - Simplified and revised.

helped put it into a much better state. I would appreciate any feedback or votes. https://cwiki.apache.org/confluence/display/KAFKA/KIP-213+Support+non-key+joining+in+KTable Thank you Adam Bellemare

  1   2   >