Hi Christo,
Thanks for the explanation.
I think it would be good if you could add that into the KIP.
Otherwise, LGTM.
Thank you.
Luke
On Mon, May 13, 2024 at 11:55 PM Christo Lolov
wrote:
> Heya!
>
> re Kamal - Okay, I believe I understand what you mean and I agree. I have
> made the
Heya!
re Kamal - Okay, I believe I understand what you mean and I agree. I have
made the following change
```
During tiered storage disablement, when RemoteLogManager#stopPartition() is
called:
- Tasks scheduled for the topic-partitions in the
RemoteStorageCopierThreadPool will be
Hi Christo,
> 1. I am not certain I follow the question. From DISABLED you can only go
to
ENABLED regardless of whether your cluster is backed by Zookeeper or KRaft.
Am I misunderstanding your point?
Yes, you're right.
> 4. I was thinking that if there is a mismatch we will just fail accepting
Christo,
IMO, we can converge the implementation for both the policies
and remove the remote-log-segments in async fashion.
The only difference between the "delete" and "retain" policies is
that, for "delete" we move the log-start-offset to match with the
local-log-start-offset and then the
Heya!
re: Luke
1. I am not certain I follow the question. From DISABLED you can only go to
ENABLED regardless of whether your cluster is backed by Zookeeper or KRaft.
Am I misunderstanding your point?
2. Apologies, this was a leftover from previous versions. I have updated
the Zookeeper
Hi Christo,
Thanks for the update!
For both the policies "retain" and "delete", can we maintain the same
approach to delete the segments async?
> If the disablement policy is set to delete, the Log start offset (LSO) is
updated to match the Local Log Start Offset and the remote log is deleted
Also, I think using `stopReplicas` request is a good idea because it won't
cause any problems while migrating to KRaft mode.
The stopReplicas request is one of the request that KRaft controller will
send to ZK brokers during migration.
Thanks.
Luke
On Fri, May 3, 2024 at 11:48 AM Luke Chen
Hi Christo,
Thanks for the update.
Questions:
1. For this
"The possible state transition from DISABLED state is to the ENABLED."
I think it only applies for KRaft mode. In ZK mode, the possible state is
"DISABLING", right?
2. For this:
"If the cluster is using Zookeeper as the control plane,
Heya all!
I have updated KIP-950. A list of what I have updated is:
* Explicitly state that Zookeeper-backed clusters will have ENABLED ->
DISABLING -> DISABLED while KRaft-backed clusters will only have ENABLED ->
DISABLED
* Added two configurations for the new thread pools and explained where
Heya Doguscan,
I believe that the state of the world after this KIP will be the following:
For Zookeeper-backed clusters there will be 3 states: ENABLED, DISABLING
and DISABLED. We want this because Zookeeper-backed clusters will await a
confirmation from the brokers that they have indeed
+1 let's not introduce a new api and mark it immediately as deprecated :)
On your second comment Luke, one thing we need to clarify is when do we
consider remote storage to be DISABLED for a topic?
Particularly, what is the state when the remote storage is being deleted in
case of
Hi Christo,
> I would then opt for moving information from DisableRemoteTopic
within the StopReplicas API which will then disappear in KRaft world as it
is already scheduled for deprecation. What do you think?
Sounds good to me.
Thanks.
Luke
On Tue, Apr 9, 2024 at 6:46 PM Christo Lolov wrote:
Heya Luke!
I thought a bit more about it and I reached the same conclusion as you for
2 as a follow-up from 1. In other words, in KRaft world I don't think the
controller needs to wait for acknowledgements for the brokers. All we care
about is that the leader (who is responsible for
Hi Christo, Doguscan,
Can you please address the comments in this mail thread and update the
KIP accordingly?
Thanks,
Satish.
On Wed, 3 Apr 2024 at 12:30, Luke Chen wrote:
>
> Hi Christo,
>
> 1. I agree with Doguscan that in KRaft mode, the controller won't send RPCs
> to the brokers (except in
Hi Christo,
1. I agree with Doguscan that in KRaft mode, the controller won't send RPCs
to the brokers (except in the migration path).
So, I think we could adopt the similar way we did to `AlterReplicaLogDirs` (
KIP-858
Heya everyone!
re: Doguscan
I believe the answer to 101 needs a bit more discussion. As far as I know,
tiered storage today has methods to update a metadata of a segment to say
"hey, I would like this deleted", but actual deletion is left to plugin
implementations (or any background cleaners).
Hi Satish, I will try to answer as much as I can and the others could chime
in with further details.
*101. For remote.log.disable.policy=delete: Does it delete the remote log
data immediately and the data in remote storage will not be taken into
account by any replica? That means
Hi,
Thanks for the KIP! Overall the KIP looks good and covered most of the
items.
1. Could you explain how the brokers will handle the DisableRemoteTopic API
request?
2. Who will initiate the controller interaction sequence? Does the
controller listens for
topic config updates and initiate the
Thanks Mehari, Divij, Christo etal for the KIP.
I had an initial review of the KIP and left the below comments.
101. For remote.log.disable.policy=delete:
Does it delete the remote log data immediately and the data in remote
storage will not be taken into account by any replica? That means
Hi Christo and Luke,
I think the KRaft section of the KIP requires slight improvement. The
metadata propagation in KRaft is handled by the RAFT layer instead of
sending Controller -> Broker RPCs. In fact, KIP-631 deprecated these RPCs
Heya,
I do have some time to put into this, but to be honest I am still after
reviews of the KIP itself :)
After the latest changes it ought to be detailing both a Zookeeper approach
and a KRaft approach.
Do you have any thoughts on how it could be improved or should I start a
voting thread?
Hi Christo,
Any update with this KIP?
If you don't have time to complete it, I can collaborate with you to work
on it.
Thanks.
Luke
On Wed, Jan 17, 2024 at 11:38 PM Satish Duggana
wrote:
> Hi Christo,
> Thanks for volunteering to contribute to the KIP discussion. I suggest
> considering this
Hi Christo,
Thanks for volunteering to contribute to the KIP discussion. I suggest
considering this KIP for both ZK and KRaft as it will be helpful for
this feature to be available in 3.8.0 running with ZK clusters.
Thanks,
Satish.
On Wed, 17 Jan 2024 at 19:04, Christo Lolov wrote:
>
> Hello!
>
Hello!
I volunteer to get this KIP moving forward and implemented in Apache Kafka
3.8.
I have caught up with Mehari offline and we have agreed that given Apache
Kafka 4.0 being around the corner we would like to propose this feature
only for KRaft clusters.
Any and all reviews and comments are
Hi everyone, any progress on the status of this KIP? Overall looks good to
me but I wonder whether we still need to support it for Zookeeper mode
given that it will be deprecated in the next 3 months.
On 2023/07/21 20:16:46 "Beyene, Mehari" wrote:
> Hi everyone,
> I would like to start a
Hi everyone,
I would like to start a discussion on KIP-950: Tiered Storage Disablement
(https://cwiki.apache.org/confluence/display/KAFKA/KIP-950%3A++Tiered+Storage+Disablement).
This KIP proposes adding the ability to disable and re-enable tiered storage on
a topic.
Thanks,
Mehari
26 matches
Mail list logo
