Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey all, After discussion with Becket in the https://github.com/apache/kafka/pull/3621, I have updated the KIP-113 to make the following minor changes to the protocol: 1) Renamed DescribeDirsRequest (and DescribeDirsResponse) to DescribeLogDirsRequest (and DescribeLogDirsResponse). This change

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Tom, The inter-broker-with-log-dirs case can be split into inter-broker-w/o-log-dirs and log-dirs change per broker. KIP-113 proposes to do the split in the tool. I am not sure if we really need to persist log-dirs changes in ZK. During the discussion in KIP-113, we realized that there is

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi Jun, The inter-broker movement case has two subcases: 1. Where no log dir is supplied. This corresponds to the existing kafka-reassign-partitions script. This just needs the appropriate JSON to be written to the reassignment znode. 2. Where the log dir is supplied. This is covered in KIP-113

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Tom, One approach is to have a PartitionReassignmentRequest that only deals with inter broker data movement (i.e, w/o any log dirs in the request). The request is directed to any broker, which then just writes the reassignment json to ZK. There is a separate AlterReplicaDirRequest that only

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi Jun and Dong, Thanks for your replies... On 10 August 2017 at 20:43, Dong Lin wrote: > This is a very good idea. I have updated the KIP-113 so that > DescribeDirResponse returns lag instead of LEO. Excellent! On Thu, Aug 10, 2017 at 10:21 AM, Jun Rao

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, This is a very good idea. I have updated the KIP-113 so that DescribeDirResponse returns lag instead of LEO. If the replica is not a temporary replica, then lag = max(0, HW - LEO). Otherwise, lag = primary Replica's LEO - temporary Replica's LEO. Thanks! Dong On Thu, Aug 10, 2017 at

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Tom, Dong, A couple of comments on that. 1. I think we can unify the reporting of lags. Basically, the lag will be reported on every replica (temporary or permanent), not just at the leader replica. If it's permanent, lag is max(0, HW - LEO) as it is now. Otherwise, lag is (LEO of permanent

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

I've spent some time thinking about KIP-179 and KIP-113, the proposed algorithms and APIs, and trying to weigh the pros and cons of various alternative options. I think Dong's reasons for the algorithm for inter-broker move in KIP-113 make a lot of sense. I don't think it would be at all simple

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi Dong and Jun, It seems that KIP-179 does not explicitly specify the definition of this > lag. Given that the definition of "caught up" is "is the replica in the ISR?", I found the code in Partition.maybeExpandIsr() which decides whether a replica should be added to the to the ISR and it uses

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, I have been thinking about whether it is better to return lag (i.e. HW - LEO) instead of LEO. Note that the lag in the DescribeDirsResponse may be negative if LEO > HW. It will almost always be negative for leader and in-sync replicas. Note that we can not calculate lag as max(0, HW -

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, Yes, the lag in a replica is calculated as the difference of LEO of the replica and the HW. So, as long as a replica is in sync, the lag is always 0. So, I was suggesting to return lag instead of LEO in DescribeDirsResponse for each replica. I am not sure if we need to return HW

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, It just came to me that you may be assuming that folower_lag = HW - follower_LEO. If that is the case, then we need to have new request/response to retrieve this lag since the DescribeDirsResponse doesn't even include HW. It seems that KIP-179 does not explicitly specify the definition

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, If I understand you right, you are suggesting that, in the case when there is continuous incoming traffic, the approach in the KIP-179 will report lag as 0 whereas the approach using DescribeDirsRequest will report lag as non-zero. But I think the approach in KIP-179 will also report

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, As for whether to return LEO or lag, my point was the following. What you are concerned about is that an in-sync replica could become out of sync again. However, the more common case is that once a replica is caught up, it will stay in sync afterwards. In that case, once the

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi Dong and Jun, Thanks for your responses! Jun's interpretation of how AlterTopicsRequest could be sent to any broker is indeed what I meant. Since the data has to get persisted in ZK anyway, it doesn't really matter whether we send it to the controller (it will will have to write it to the

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, Thanks for the comment! Yes, it should work. The tool can send request to any broker and broker can just write the reassignment znode. My previous intuition is that it may be better to only send this request to controller. But I don't have good reasons for this restriction. My

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, I think Tom was suggesting to have the AlterTopicsRequest sent to any broker, which just writes the reassignment json to ZK. The controller will pick up the reassignment and act on it as usual. This should work, right? Having a separate AlterTopicsRequest and AlterReplicaDirRequest

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Tom, Thanks for the quick reply. Please see my comment inline. On Tue, Aug 8, 2017 at 11:06 AM, Tom Bentley wrote: > Hi Dong, > > Replies inline, as usual > > > As I originally envisaged it, KIP-179's support for reassigning > partitions > > > > would have

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi Dong, Replies inline, as usual > As I originally envisaged it, KIP-179's support for reassigning partitions > > would have more-or-less taken the logic currently in the > > ReassignPartitionsCommand (that is, writing JSON to the > > ZkUtils.ReassignPartitionsPath) > > and put it behind a

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Thanks for your reply. Yes, my original idea is that user can continue to collect the static information for reassignment as they are doing now. It is the status quo. I agree it can be beneficial to have a tool in Kafka to collect other information that may be needed for reassignment so that user

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi Dong, Thanks for your reply. Yeah I agree with you that the total disk capacity can be useful > particularly if it is different across brokers but it is probably of > limited use in most cases. I also expect that most users would have their > own customized tool across to determine the new

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

> > > > Also, how do you think things would work in the context of KIP-179? Would > > the tool still invoke these requests or would it be done by the broker > > receiving the alterTopics/reassignPartitions protocol call? > > > > My gut feel is that the tool will still invoke these requests. But I

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Tom, Yeah I agree with you that the total disk capacity can be useful particularly if it is different across brokers but it is probably of limited use in most cases. I also expect that most users would have their own customized tool across to determine the new partition reassignment after

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi Dong, The reason I thought this would be useful is it seems likely to me that people will want to write tools to help them generate allocations. If, as you say, all the brokers and all the disks are the same size, then it's not too difficult to tell the tool the size of the disk. But if

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Ismael, Thanks much for your comments. Please see my reply inline. On Mon, Aug 7, 2017 at 5:28 AM, Ismael Juma wrote: > Hi Dong, > > Thanks for the explanation. Comments inline. > > On Fri, Aug 4, 2017 at 6:47 PM, Dong Lin wrote: > > > 1. Yes it has

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Tom, Good question. We have actually considered having DescribeDirsResponse provide the capacity of each disk as well. This was not included because we believe Kafka cluster admin will always configure all brokers with same number of disks of the same size. This is because it is generally

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi Dong, Thanks for the explanation. Comments inline. On Fri, Aug 4, 2017 at 6:47 PM, Dong Lin wrote: > 1. Yes it has been considered. Here are the reasons why we don't do it > through controller. > > - There can be use-cases where we only want to rebalance the load of log

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi Dong, Your comments on KIP-179 prompted me to look at KIP-113, and I have a question: AFAICS the DescribeDirsResponse (via ReplicaInfo) can be used to get the size of a partition on a disk, but I don't see a mechanism for knowing the total capacity of a disk (and/or the free capacity of a

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Ismael, Thanks for the comments! Here are my answers: 1. Yes it has been considered. Here are the reasons why we don't do it through controller. - There can be use-cases where we only want to rebalance the load of log directories on a given broker. It seems unnecessary to go through

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Thanks Dong. I have a few initial questions, sorry if I it has been discussed and I missed it. 1. The KIP suggests that the reassignment tool is responsible for sending the ChangeReplicaDirRequests to the relevant brokers. I had imagined that this would be done by the Controller, like the rest of

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi all, I realized that we need new API in AdminClient in order to use the new request/response added in KIP-113. Since this is required by KIP-113, I choose to add the new interface in this KIP instead of creating a new KIP. The documentation of the new API in AdminClient can be found here

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi all, I have made a minor change to the DescribeDirsRequest so that user can choose to query the status for a specific list of partitions. This is a bit more fine-granular than the previous format that allows user to query the status for a specific list of topics. I realized that querying the

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

The protocol change has been updated in KIP-113 . On Wed, Jul 12, 2017 at 10:44 AM, Dong Lin wrote: > Hi all, > > I have made a minor change to the

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Colin, Thanks for the suggestion. We have actually considered this and list this as the first future work in KIP-112 . The two advantages that you mentioned are exactly the motivation for this feature.

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Has anyone considered a scheme for sharding topic data across multiple disks? For example, if you sharded topics across 3 disks, and you had 10 disks, you could pick a different set of 3 disks for each topic. If you distribute them randomly then you have 10 choose 3 = 120 different combinations.

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Just a few comments on this. 1. One of the issues with using RAID 0 is that a single disk failure causes a hard failure of the broker. Hard failure increases the unavailability window for all the partitions on the failed broker, which includes the failure detection time (tied to ZK session

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Sriram, I think there is one way to explain why the ability to move replica between disks can save space. Let's say the load is distributed to disks independent of the broker. Sooner or later, the load imbalance will exceed a threshold and we will need to rebalance load across disks. Now our

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Sriram, Thanks for raising these concerns. Let me answer these questions below: - The benefit of those additional complexity to move the data stored on a disk within the broker is to avoid network bandwidth usage. Creating replica on another broker is less efficient than creating replica on

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Dong, Thanks for the explanation. I don't think anyone is denying that we should rebalance at the disk level. I think it is important to restore the disk and not wait for disk replacement. There are also other benefits of doing that which is that you don't need to opt for hot swap racks that

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jay, Sriram, Great point. If I understand you right, you are suggesting that we can simply use RAID-0 so that the load can be evenly distributed across disks. And even though a disk failure will bring down the enter broker, the reduced availability as compared to using KIP-112 and KIP-113

RE: [DISCUSS] KIP-113: Support replicas movement between log directories

of labor on-prem - salaries dev ops personnel - M -Original Message- From: Jay Kreps [mailto:j...@confluent.io] Sent: Wednesday, June 07, 2017 2:27 AM To: dev@kafka.apache.org Subject: Re: [DISCUSS] KIP-113: Support replicas movement between log directories I think

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

I think Ram's point is that in place failure is pretty complicated, and this is meant to be a cost saving feature, we should construct an argument for it grounded in data. Assume an annual failure rate of 1% (reasonable, but data is available online), and assume it takes 3 days to get the drive

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Sriram, Thanks for taking time to review the KIP. Please see below my answers to your questions: >1. Could you pick a hardware/Kafka configuration and go over what is the >average disk/partition repair/restore time that we are targeting for a >typical JBOD setup? We currently don't have

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Thanks Jun! Hi all, Thanks for all the comments. I am going to open the voting thread if there is no further concern with the KIP. Dong On Thu, Mar 30, 2017 at 3:19 PM, Jun Rao wrote: > Hi, Dong, > > I don't have further concerns. If there are no more comments from other >

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, I don't have further concerns. If there are no more comments from other people, we can start the vote. Thanks, Jun On Thu, Mar 30, 2017 at 10:59 AM, Dong Lin wrote: > Hey Jun, > > Thanks much for the comment! Do you think we start vote for KIP-112 and > KIP-113

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, Thanks much for the comment! Do you think we start vote for KIP-112 and KIP-113 if there is no further concern? Dong On Thu, Mar 30, 2017 at 10:40 AM, Jun Rao wrote: > Hi, Dong, > > Ok, so it seems that in solution (2), if the tool exits successfully, then > we

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, Ok, so it seems that in solution (2), if the tool exits successfully, then we know for sure that all replicas will be in the right log dirs. Solution (1) doesn't guarantee that. That seems better and we can go with your current solution then. Thanks, Jun On Fri, Mar 24, 2017 at 4:28

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, No.. the current approach describe in the KIP (see here ) also

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, We are just comparing whether it's better for the reassignment tool to send ChangeReplicaDirRequest (1) before or (2) after writing the reassignment path in ZK . In the case when all brokers are alive when the reassignment tool is run, (1) guarantees 100% that the new replicas will be

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, Thanks much for the response! I agree with you that if multiple replicas are created in the wrong directory, we may waste resource if either replicaMoveThread number is low or intra.broker.throttled.rate is slow. Then the question is whether the suggested approach increases the chance of

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, 11.2 I think there are a few reasons why the cross disk movement may not catch up if the replicas are created in the wrong log dirs to start with. (a) There could be more replica fetcher threads than the disk movement threads. (b) intra.broker.throttled.rate may be configured lower than

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, Thanks for the explanation. Please see below my thoughts. 10. I see. So you are concerned with the potential implementation complexity which I wasn't aware of. I think it is OK not to do log cleaning on the .move log since there can be only one such log in each directory. I have updated

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, 10. I was mainly concerned about the additional complexity needed to support log cleaning in the .move log. For example, LogToClean is keyed off TopicPartition. To be able to support cleaning different instances of the same partition, we need additional logic. I am not how much

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, Thanks for the response! It seems that we have only two remaining issues. Please see my reply below. On Mon, Mar 20, 2017 at 7:45 AM, Jun Rao wrote: > Hi, Dong, > > Thanks for the update. A few replies inlined below. > > On Thu, Mar 16, 2017 at 12:28 AM, Dong Lin

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, Thanks for the update. A few replies inlined below. On Thu, Mar 16, 2017 at 12:28 AM, Dong Lin wrote: > Hey Jun, > > Thanks for your comment! Please see my reply below. > > On Wed, Mar 15, 2017 at 9:45 PM, Jun Rao wrote: > > > Hi, Dong, > > >

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, After thinking about 14 more, I think your solution is reasonable. I have updated the KIP to specify that the number of ReplicaMoveThread defaults to # log dirs. Thanks! Dong On Thu, Mar 16, 2017 at 12:28 AM, Dong Lin wrote: > Hey Jun, > > Thanks for your

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, Thanks for your comment! Please see my reply below. On Wed, Mar 15, 2017 at 9:45 PM, Jun Rao wrote: > Hi, Dong, > > Thanks for the reply. > > 10. Could you comment on that? > Sorry, I missed that comment. Good point. I think the log segments in topicPartition.move

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, Thanks for the reply. 10. Could you comment on that? 11.2 "I am concerned that the ChangeReplicaDirRequest would be lost if broker restarts after it sends ChangeReplicaDirResponse but before it receives LeaderAndIsrRequest." In that case, the reassignment tool could detect that

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, Thanks much for your detailed comments. Please see my reply below. On Mon, Mar 13, 2017 at 9:09 AM, Jun Rao wrote: > Hi, Dong, > > Thanks for the updated KIP. Some more comments below. > > 10. For the .move log, do we perform any segment deletion (based on >

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, Thanks for the updated KIP. Some more comments below. 10. For the .move log, do we perform any segment deletion (based on retention) or log cleaning (if a compacted topic)? Or do we only enable that after the swap? 11. kafka-reassign-partitions.sh 11.1 If all reassigned replicas are

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

I just made one correction in the KIP. If broker receives ChangeReplicaDirRequest and the replica hasn't been created there, the broker will respond ReplicaNotAvailableException. The kafka-reassignemnt-partitions.sh will need to re-send ChangeReplicaDirRequest in this case in order to wait for

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, Thanks for your comments! I have updated the KIP to address your comments. Please see my reply inline. Can you let me know if the latest KIP has addressed your comments? On Wed, Mar 8, 2017 at 9:56 PM, Jun Rao wrote: > Hi, Dong, > > Thanks for the reply. > > 1.3 So

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, Thanks for the reply. 1.3 So the thread gets the lock, checks if caught up and releases the lock if not? Then, in the case when there is continuous incoming data, the thread may never get a chance to swap. One way to address this is when the thread is getting really close in catching

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, Thanks for the detailed explanation. I will use the separate thread pool to move replica between log directories. I will let you know when the KIP has been updated to use a separate thread pool. Here is my response to your other questions: 1.3 My idea is that the ReplicaMoveThread that

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, Thanks for the updated KIP. A few more comments below. 1.1 and 1.2: I am still not sure there is enough benefit of reusing ReplicaFetchThread to move data across disks. (a) A big part of ReplicaFetchThread is to deal with issuing and tracking fetch requests. So, it doesn't feel that we

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Jun, Thanks for all the comments! Please see my answer below. I have updated the KIP to address most of the questions and make the KIP easier to understand. Thanks, Dong On Thu, Mar 2, 2017 at 9:35 AM, Jun Rao wrote: > Hi, Dong, > > Thanks for the KIP. A few comments

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, Dong, Thanks for the KIP. A few comments below. 1. For moving data across directories 1.1 I am not sure why we want to use ReplicaFetcherThread to move data around in the leader. ReplicaFetchThread fetches data from socket. For moving data locally, it seems that we want to avoid the socket

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

24.01.2017, 22:03, "Dong Lin" : > Hey Alexey, > > Thanks. I think we agreed that the suggested solution doesn't work in > general for kafka users. To answer your questions: > > 1. I agree we need quota to rate limit replica movement when a broker is > moving a "leader"

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Alexey, Thanks. I think we agreed that the suggested solution doesn't work in general for kafka users. To answer your questions: 1. I agree we need quota to rate limit replica movement when a broker is moving a "leader" replica. I will come up with solution, probably re-use the config of

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

23.01.2017, 22:11, "Dong Lin" : > Thanks. Please see my comment inline. > > On Mon, Jan 23, 2017 at 6:45 AM, Alexey Ozeritsky > wrote: > >>  13.01.2017, 22:29, "Dong Lin" : >>  > Hey Alexey, >>  > >>  > Thanks for your review and

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Thanks. Please see my comment inline. On Mon, Jan 23, 2017 at 6:45 AM, Alexey Ozeritsky wrote: > > > 13.01.2017, 22:29, "Dong Lin" : > > Hey Alexey, > > > > Thanks for your review and the alternative approach. Here is my > > understanding of your

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

13.01.2017, 22:29, "Dong Lin" : > Hey Alexey, > > Thanks for your review and the alternative approach. Here is my > understanding of your patch. kafka's background threads are used to move > data between replicas. When data movement is triggered, the log will be > rolled and

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hey Alexey, Thanks for your review and the alternative approach. Here is my understanding of your patch. kafka's background threads are used to move data between replicas. When data movement is triggered, the log will be rolled and the new logs will be put in the new directory, and background

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

Hi, We have the similar solution that have been working in production since 2014. You can see it here: https://github.com/resetius/kafka/commit/20658593e246d2184906879defa2e763c4d413fb The idea is very simple 1. Disk balancer runs in a separate thread inside scheduler pool. 2. It does not touch

[DISCUSS] KIP-113: Support replicas movement between log directories

Hi all, We created KIP-113: Support replicas movement between log directories. Please find the KIP wiki in the link *https://cwiki.apache.org/confluence/display/KAFKA/KIP-113%3A+Support+replicas+movement+between+log+directories