Hey All,
I had an offline discussion with Yash on this and while so far there didn't
seem to be a pressing need to introduce the delete offsets mechanism via
the updateOffsets method, Yash had brought up an interesting point. Point
being that if we don't introduce the deletion of offsets
Hi Sagar,
> The size of offsets topic can be controlled by
> setting appropriate topic retention values and
> that is a standard practice in Kafka
Kafka Connect enforces the `cleanup.policy` configuration for the offsets
topic to be "compact" only (references - [1], [2]), so the topic retention
Hey Yash,
Thanks for your further comments. Here are my responses:
1) Deleting offsets via updateOffsets.
Hmm, I am not sure this is really necessary to be part of the KIP at this
point, and we can always add it later on if needed. I say this for the
following reasons:
- The size of
Hi Sagar,
Thanks for the updates and apologies for the delayed response.
> Hmm the question is how do you qualify a
> partition as stale or old? Let's say a connector
> has implemented updateOffsets and for a certain
> partition for which no records are received then it
> will update it's
Hey Yash,
Thanks for your comments.
1) Hmm the question is how do you qualify a partition as stale or old?
Let's say a connector has implemented updateOffsets and for a certain
partition for which no records are received then it will update it's
offsets. So technically that offset can't be
Hi Sagar,
Thanks for the updates. I had a few more follow up questions:
> I have added that a better way of doing that would be
> via KIP-875. Also, I didn't want to include any mechamisms
> for users to meddle with the offsets topic. Allowing tombstone
> records via this method would be akin to
Hey All,
Please let me know how the KIP looks now. Is it at a stage where I can
start with the Voting phase? Of course I am still open to
feedback/suggestions but planning to start the Vote for it.
Thanks!
Sagar.
On Tue, Jul 11, 2023 at 10:00 PM Sagar wrote:
> Hi Yash/Chris,
>
> Thanks for
Hi Yash/Chris,
Thanks for the feedback! I have updated the KIP with the suggestions
provided. I would also update the PR with the suggestions.
Also, I was hoping that this could make it to the 3.6 release given that it
would benefit source connectors which have some of the problems listed in
the
Hi Sagar,
Thanks for updating the KIP! The latest draft seems simpler and more
focused, which I think is a win for users and developers alike. Here are my
thoughts on the current draft:
1. (Nit) Can we move the "Public Interfaces" section before the "Proposed
Changes" section? It's nice to have
Hi Sagar,
Thanks for your continued work on this KIP! Here are my thoughts on your
updated proposal:
1) In the proposed changes section where you talk about modifying the
offsets, could you please clarify that tasks shouldn't modify the offsets
map that is passed as an argument? Currently, the
Hi All,
I have created this PR: https://github.com/apache/kafka/pull/13899 which
implements the approach outlined in the latest version of the KIP. I
thought I could use this to validate the approach based on my understanding
while the KIP itself gets reviewed. I can always change the
Hey All,
Bumping this discussion thread again to see how the modified KIP looks
like.
Thanks!
Sagar.
On Mon, May 29, 2023 at 8:12 PM Sagar wrote:
> Hi,
>
> Bumping this thread again for further reviews.
>
> Thanks!
> Sagar.
>
> On Fri, May 12, 2023 at 3:38 PM Sagar wrote:
>
>> Hi All,
>>
>>
Hi,
Bumping this thread again for further reviews.
Thanks!
Sagar.
On Fri, May 12, 2023 at 3:38 PM Sagar wrote:
> Hi All,
>
> Thanks for the comments/reviews. I have updated the KIP
>
Hi All,
Thanks for the comments/reviews. I have updated the KIP
https://cwiki.apache.org/confluence/display/KAFKA/KIP-910%3A+Update+Source+offsets+for+Source+Connectors+without+producing+records
with a newer approach which shelves the need for an explicit topic.
Please review again and let me
Hi Sagar,
Thanks for the KIP! I have a few questions and comments:
1) I agree with Chris' point about the separation of a connector heartbeat
mechanism and allowing source connectors to generate offsets without
producing data. What is the purpose of the heartbeat topic here and are
there any
One thing I forgot to mention in my previous email was that the reason I
chose to include the opt-in behaviour via configs was that the users of the
connector know their workload patterns. If the workload is such that the
connector would receive regular valid updates then there’s ideally no need
Hi Chris,
Thanks for following up on the response. Sharing my thoughts further:
If we want to add support for connectors to emit offsets without
> accompanying source records, we could (and IMO should) do that without
> requiring users to manually enable that feature by adjusting worker or
>
Hi Sagar,
I'm sorry, I'm still not convinced that this design solves the problem(s)
it sets out to solve in the best way possible. I tried to highlight this in
my last email:
> In general, it seems like we're trying to solve two completely different
problems with this single KIP: adding
Hi Chris,
Thanks for your detailed feedback!
nits: I have taken care of them now. Thanks for pointing those out.
non-nits:
6) It seems (based on both the KIP and discussion on KAFKA-3821) that the
> only use case for being able to emit offsets without also emitting source
> records that's been
Hi Sagar,
Thanks for the KIP! I have some thoughts.
Nits:
1. Shouldn't KAFKA-3821 [1] be linked as the Jira ticket on the KIP? Or is
there a different ticket that should be associated with it?
2. The current state is listed as "Draft". Considering it's been brought up
for discussion, maybe the
Hi John,
Thanks for taking. look at the KIP!
The point about stream time not advancing in case of infrequent updates is
an interesting one. I can imagine if the upstream producer to a Kafka
Streams application is a Source Connector which isn't sending records
frequently(due to the nature of the
Thanks for the KIP, Sagar!
At first glance, this seems like a very useful feature.
A common pain point in Streams is when upstream producers don't send regular
updates and stream time cannot advance. This causes stream-time-driven
operations to appear to hang, like time windows not closing,
Hi All,
Bumping the thread again.
Sagar.
On Fri, Mar 10, 2023 at 4:42 PM Sagar wrote:
> Hi All,
>
> Bumping this discussion thread again.
>
> Thanks!
> Sagar.
>
> On Thu, Mar 2, 2023 at 3:44 PM Sagar wrote:
>
>> Hi All,
>>
>> I wanted to create a discussion thread for KIP-910:
>>
>>
>>
Hi All,
Bumping this discussion thread again.
Thanks!
Sagar.
On Thu, Mar 2, 2023 at 3:44 PM Sagar wrote:
> Hi All,
>
> I wanted to create a discussion thread for KIP-910:
>
>
>
Hi All,
I wanted to create a discussion thread for KIP-910:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-910%3A+Update+Source+offsets+for+Source+Connectors+without+producing+records
Thanks!
Sagar.
25 matches
Mail list logo
