Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

Hi John Good thinking with regards to upgrade path between versions regarding over-the-wire instructions in SubscriptionWrapper. At this point in time I can't think of any new wire message instructions, but I would appreciate as many eyes on it as possible. I have just included the LEFT join in

[jira] [Created] (KAFKA-8605) Warn users when they have same connector in their plugin-path more than once

Cyrus Vafadari created KAFKA-8605: - Summary: Warn users when they have same connector in their plugin-path more than once Key: KAFKA-8605 URL: https://issues.apache.org/jira/browse/KAFKA-8605

Jenkins build is back to normal : kafka-2.2-jdk8 #143

See

Re: [DISCUSS] KIP-480 : Sticky Partitioner

Stanislav, Thank you for looking at my KIP! I did discuss with Colin about whether the null vs. Optional types and we did not come to a strong conclusion either way. I'd be happy to change it if it makes the logic more clear. Thanks, Justine On Wed, Jun 26, 2019 at 2:46 PM Stanislav Kozlovski

Re: [DISCUSS] KIP-480 : Sticky Partitioner

Hey Justine, Thanks for the KIP! I am impressed by the performance results linked in the KIP and I like the data-driven approach. This looks like a great improvement. I had one minor question regarding the public interface `repartitionOnNewBatch` where we return null in the case of no change

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

Hi Viktor, Good point. Sorry, I should have read the KIP more closely. It would be good to change the title of the mail thread to reflect the new title of the KIP, "Internal Partition Reassignment Batching." I do think there will be some interaction with KIP-455 here. One example is that

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

On Wed, Jun 26, 2019, at 12:02, Jason Gustafson wrote: > Hi Colin, > > Responses below and another question: > > > I guess the thought process here is that most reassignment tools want to > > know about all the reassignments that are going on. If you don't know all > > the pending

[DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

Hi all, I would like to start a discussion for KIP-482: https://cwiki.apache.org/confluence/display/KAFKA/KIP-482%3A+The+Kafka+Protocol+should+Support+Optional+Fields cheers, Colin

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

Thanks, Adam! One unrelated thought that has just now occurred to me is that (unlike the equi-joins we currently have), this join logic is potentially spread over multiple Streams instances, which in general means that the instances may be running different versions of Kafka Streams. This means

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

Sigh... Forgot the link: https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=74684836=78=74 I'll update it when I validate that there are no issues with removing the SubscriptionResponseWrapper boolean. On Wed, Jun 26, 2019 at 3:37 PM Adam Bellemare wrote: > >Maybe just

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

>Maybe just call it as (k, leftval, null) or (k, null, rightval)? Done. > if you update the KIP, you might want to send a new "diff link" to this thread Here it is: > Looking closely at the proposal, can you explain more about the propagateIfNull field in SubscriptionResponseWrapper? It sort of

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

Hi Colin, Responses below and another question: I guess the thought process here is that most reassignment tools want to > know about all the reassignments that are going on. If you don't know all > the pending reassignments, then it's hard to say whether adding a new one > is a good idea, or

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

I think the "scenario trace" is very nice, but has one point that I found confusing: You indicate a retraction in the join output as (k,null) and a join result as (k, leftval, rightval), but confusingly, you also write a join result as (k, JoinResult) when one side is null. Maybe just call it as

Re: [DISCUSS] KIP-447: Producer scalability for exactly once semantics

2. The reason we did not expose generation.id from KafkaConsumer public APIs directly is to abstract this notion from users (since it is an implementation detail of the rebalance protocol itself, e.g. if user calls consumer.assign() they do not need to invoke ConsumerCoordinator and no need to be

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

On Tue, Jun 25, 2019, at 18:37, Jason Gustafson wrote: > Hi Colin, > > Took another pass on the KIP. Looks good overall. A few questions below: > > 1. I wasn't clear why `currentReplicas` is an optional field. Wouldn't we > always have a current set of replicas? Good point. When I wrote that I

Re: [DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

Thanks John. I'm looking forward to any feedback on this. In the meantime I will work on the unit tests to ensure that we have well-defined and readable coverage. At the moment I cannot see a way around emitting (k,null) whenever we emit an event that lacks a matching foreign key on the RHS,

[DISCUSS] KIP-213: Second follow-up on Foreign Key Joins

Hi Adam, Thanks for the proposed revision to your KIP (https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=74684836=77=74) in response to the concern pointed out during code review (https://github.com/apache/kafka/pull/5527#issuecomment-505137962) We should have a brief

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

Thanks for the update, Daniyar! In addition to specifying the config interface, can you also specify the Java interface? Namely, if I need to pass an instance of this serde in to the DSL directly, as in Produced, Materialized, etc., what constructor(s) would I have available? Likewise with the

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

Hey, Finally made updates to the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-466%3A+Add+support+for+List%3CT%3E+serialization+and+deserialization Sorry for

Re: [DISCUSS] KIP-477: Add PATCH method for connector config in Connect REST API

Ivan, I looked at adding PATCH a while ago as well. I decided not to pursue the idea for a few reasons: 1) PATCH is still racy. For example, if you want to add a topic to the "topics" property, you still need to read, modify, and write the existing value. To handle this, you'd need to support

Re: [jira] [Created] (KAFKA-8604) kafka log dir was marked as offline because of deleting segments of __consumer_offsets failed

This is a known issue for Windows. See KAFKA-6188. Thanks, On Wed, 26 Jun 2019 at 14:46, songyingshuan (JIRA) wrote: > songyingshuan created KAFKA-8604: > > > Summary: kafka log dir was marked as offline because of > deleting segments of

Re: [VOTE] KIP-476: Add Java AdminClient interface

+1 (non-binding) Thanks. Ryanne On Tue, Jun 25, 2019 at 10:21 PM Satish Duggana wrote: > +1 (non-binding) > > On Wed, Jun 26, 2019 at 8:37 AM Satish Duggana > wrote: > > > > +1 Matthias/Andy. > > IMHO, interface is about the contract, it should not have/expose any > > implementation. I am

[jira] [Created] (KAFKA-8604) kafka log dir was marked as offline because of deleting segments of __consumer_offsets failed

songyingshuan created KAFKA-8604: Summary: kafka log dir was marked as offline because of deleting segments of __consumer_offsets failed Key: KAFKA-8604 URL: https://issues.apache.org/jira/browse/KAFKA-8604

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

Hey Colin, I think there's some confusion here so I might change the name of this. So KIP-435 is about the internal batching of reassignments (so purely a controller change) and not about client side APIs. As per this moment these kind of improvements are listed on KIP-455's future work section

Build failed in Jenkins: kafka-trunk-jdk8 #3753

See Changes: [bill] MINOR: Fix for typos in processor-api.html (#6961) [bbejeck] Minor: code enhancment (#6999) -- [...truncated 5.09 MB...]

[jira] [Resolved] (KAFKA-8596) Kafka topic pre-creation error message needs to be passed to application as an exception

[ https://issues.apache.org/jira/browse/KAFKA-8596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gurudatt Kulkarni resolved KAFKA-8596. -- Resolution: Duplicate > Kafka topic pre-creation error message needs to be passed to