Handling the "changelog" topic for the buffer in the same way as
repartition topics makes sense. Thanks for clearing that up!
- Victoria
On Tue, Jun 6, 2023 at 10:22 AM Walker Carlson
wrote:
> Good Point Victoria. I just removed the compacted topic mention from the
> KIP. I agree with Burno
Good Point Victoria. I just removed the compacted topic mention from the
KIP. I agree with Burno about using a normal topic and deleting records
that have been processed.
On Tue, Jun 6, 2023 at 2:28 AM Bruno Cadonna wrote:
> Hi,
>
> another idea that came to my mind. Instead of using a
Hi,
another idea that came to my mind. Instead of using a compacted topic,
the buffer could use a non-compacted topic and regularly delete records
before a given offset as Streams does for repartition topics.
Best,
Bruno
On 05.06.23 21:48, Bruno Cadonna wrote:
Hi Victoria,
that is a good
Hi Victoria,
that is a good point!
I think, the topic needs to be a compacted topic to be able to get rid
of records that are evicted from the buffer. So the key might be
something with the key, the timestamp, and a sequence number to
distinguish between records with the same key and same
Hi Walker,
Thanks for the latest updates! The KIP looks great. Just one question about
the changelog topic for the join buffer: The KIP says "When a failure
occurs the buffer will try to recover from an OffsetCheckpoint if possible.
If not it will reload the buffer from a compacted change-log
Hi Walker,
Thanks once more for the updates to the KIP!
Do you also plan to expose metrics for the buffer?
Best,
Bruno
On 02.06.23 17:16, Walker Carlson wrote:
Hello Bruno,
I think this covers your questions. Let me know what you think
2.
We can use a changelog topic. I think we can treat
Hello Bruno,
I think this covers your questions. Let me know what you think
2.
We can use a changelog topic. I think we can treat it like any other store
and recover in the usual manner. Also implementation is on disk
3.
The description is in the public interfaces description. I will copy it
Hi Walker,
Thanks for the updates!
2.
It is still not clear to me how a failure is handled. I do not
understand what you mean by "recover from an OffsetCheckpoint".
My understanding is that the buffer needs to be replicated into its own
Kafka topic. The input topic is not enough. The offset
Hey Bruno thanks for the feedback.
1)
I will add this to the kip, but stream time only advances as the when the
buffer receives a new record.
2)
You are correct, I will add a failure section on to the kip. Since the
records wont change in the buffer from when they are read from the topic
they
Hi Walker,
thanks for the KIP!
Here my feedback:
1.
It is still not clear to me when stream time for the buffer advances.
What is the event that let the stream time advance? In the discussion, I
do not understand what you mean by "The segment store already has an
observed stream time, we
Thanks for all the additional comments. I will either address them here or
update the kip accordingly.
I mentioned a follow kip to add extra features before and in the responses.
I will try to briefly summarize what options and optimizations I plan to
include. If a concern is not covered in this
Walker,
thanks for the updates. The KIP itself reads fine (of course Victoria
made good comments about some phrases), but there is a couple of things
from your latest reply I don't understand, and that I still think need
some more discussions.
Lukas, asked about in-memory option and
Hi Walker,
thanks for your responses. That makes sense. I guess there is always
the option to make the implementation more configurable later on, if
users request it. Also thanks for the clarifications. From my side,
the KIP is good to go.
Cheers,
Lucas
On Wed, May 24, 2023 at 11:54 PM Victoria
Thanks for the updates, Walker! Looks great, though I do have a couple
questions about the latest updates:
1. The new example says that without stream-side buffering, "ex" and
"fy" are possible join results. How could those join results happen? The
example versioned table suggests that
Hey all,
Thanks for the comments, they gave me a lot to think about. I'll try to
address them all inorder. I have made some updates to the kip related to
them, but I mention where below.
Lucas
Good idea about the example. I added a simple one.
1) I have thought about including options for the
Thanks for the KIP! Also some question/comments from my side:
10) Notation: you use the term "late data" but I think you mean
out-of-order. We reserve the term "late" to records that arrive after
grace period passed, and thus, "late == out-of-order data that is dropped".
20) "There is only
Cool KIP, Walker! Thanks for sharing this proposal.
A few clarifications:
1. Is the order that records exit the buffer in necessarily the same as the
order that records enter the buffer in, or no? Based on the description in
the KIP, it sounds like the answer is no, i.e., records will exit the
HI Walker,
thanks for the KIP! We definitely need this. I have two questions:
- Have you considered allowing the customization of the underlying
buffer implementation? As I can see, `StreamJoined` lets you customize
the underlying store via a `WindowStoreSupplier`. Would it make sense
for
Hello everybody,
I have a stream proposal to improve the stream table join by adding a grace
period and buffer to the stream side of the join to allow processing in
timestamp order matching the recent improvements of the versioned tables.
Please take a look here
19 matches
Mail list logo
