Hi Cong,
Some of use cases I have in mind are more around validation that an
operation was successful.
- Let's say you trigger a reassignment to even out disk usage on some
brokers. Your tool will submit the reassignment and wait for
completion using the AlterPartitionReassignments and
ListPartitionReassignments APIs respectively. Once it is complete you
may want to check the new disk usage on your brokers to validate the
reassignment did what you wanted. In that case you'd only call
DescribeLogDirs once.
- A similar example is when resizing volumes. You typically will query
repetitively the API that handles the volume resize and at the end
make a single call to DescribeLogDirs to validate brokers indeed
picked up the updated volumes.
Thanks,
Mickael
On Fri, Jun 3, 2022 at 9:08 PM Cong Ding <cd...@confluent.io.invalid> wrote:
>
> Thank you, Mickael. One more question: are you imaging these
> tooling/automation to call this API at a very low frequency? since
> high-frequency calls to this API are prohibitively expensive. Can you
> give some examples of low-frequency call use cases? I can think of
> some high-frequency call use cases which are valid in this case, but I
> had a hard time coming up with low-frequency call use cases.
>
> The one you give in the KIP is validating whether disk resize
> operations have been completed. However, this looks like a
> high-frequency call use case to me because we need to keep monitoring
> disk usage before and after resizing.
>
> Cong
>
> On Fri, Jun 3, 2022 at 5:22 AM Mickael Maison <mickael.mai...@gmail.com>
> wrote:
> >
> > Hi Cong,
> >
> > Maybe some people can do without this KIP.
> > But in many cases, especially around tooling and automation, it's
> > useful to be able to retrieve disk utilization values via the Kafka
> > API rather than interfacing with a metrics system.
> >
> > Does that clarify the motivation?
> >
> > Thanks,
> > Mickael
> >
> > On Wed, Jun 1, 2022 at 7:10 PM Cong Ding <cd...@confluent.io.invalid> wrote:
> > >
> > > Thanks for the explanation. I think the question is that if we have disk
> > > utilization in our environment, what is the use case for KIP-827? The disk
> > > utilization in our environment can already do the job. Is there anything I
> > > missed?
> > >
> > > Thanks,
> > > Cong
> > >
> > > On Tue, May 31, 2022 at 2:57 AM Mickael Maison <mickael.mai...@gmail.com>
> > > wrote:
> > >
> > > > Hi Cong,
> > > >
> > > > Kafka does not expose disk utilization metrics. This is something you
> > > > need to provide in your environment. You definitively should have a
> > > > mechanism for exposing metrics from your Kafka broker hosts and you
> > > > should absolutely monitor disk usage and have appropriate alerts.
> > > >
> > > > Thanks,
> > > > Mickael
> > > >
> > > > On Thu, May 26, 2022 at 7:34 PM Jun Rao <j...@confluent.io.invalid>
> > > > wrote:
> > > > >
> > > > > Hi, Igor,
> > > > >
> > > > > Thanks for the reply.
> > > > >
> > > > > I agree that this KIP could be useful for improving the tool for
> > > > > moving
> > > > > data across disks. It would be useful to clarify on the main
> > > > > motivation
> > > > of
> > > > > the KIP. Also, DescribeLogDirsResponse already includes the size of
> > > > > each
> > > > > partition on a disk. So, it seems that UsableBytes is redundant since
> > > > it's
> > > > > derivable.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jun
> > > > >
> > > > > On Thu, May 26, 2022 at 3:30 AM Igor Soarez <i...@soarez.me> wrote:
> > > > >
> > > > > > Hi,
> > > > > >
> > > > > > This can also be quite useful to make better use of existing
> > > > functionality
> > > > > > in the Kafka API — moving replicas between log directories via
> > > > > > ALTER_REPLICA_LOG_DIRS. If usable space information is also
> > > > > > available
> > > > the
> > > > > > caller can make better decisions using the same API. It means a more
> > > > > > consistent way of interacting with Kafka to manage replicas
> > > > > > locations
> > > > > > within a broker without having to correlate Kafka metrics with
> > > > information
> > > > > > from the Kafka API.
> > > > > >
> > > > > > --
> > > > > > Igor
> > > > > >
> > > > > > On Wed, May 25, 2022, at 8:16 PM, Jun Rao wrote:
> > > > > > > Hi, Mickael,
> > > > > > >
> > > > > > > Thanks for the KIP. Since this is mostly for monitoring and
> > > > alerting,
> > > > > > > could we expose them as metrics instead of as part of the API? We
> > > > already
> > > > > > > have a size metric per log. Perhaps we could extend that to add
> > > > > > used/total
> > > > > > > metrics per disk?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > Jun
> > > > > > >
> > > > > > > On Thu, May 19, 2022 at 10:21 PM Raman Verma
> > > > <rve...@confluent.io.invalid
> > > > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Hello Mikael,
> > > > > > >>
> > > > > > >> Thanks for the KIP.
> > > > > > >>
> > > > > > >> I see that the API response contains some information about each
> > > > > > partition.
> > > > > > >> ```
> > > > > > >> { "name": "PartitionSize", "type": "int64", "versions": "0+",
> > > > > > >> "about": "The size of the log segments in this partition in
> > > > bytes." }
> > > > > > >> ```
> > > > > > >> Can this be summed up to provide a used space in a `log.dir`
> > > > > > >> This will also be specific to a `log.dir` (for the case where
> > > > multiple
> > > > > > >> log.dir are hosted on the same underlying device)
> > > > > > >>
> > > > > > >> On Thu, May 19, 2022 at 10:21 AM Cong Ding
> > > > <cd...@confluent.io.invalid>
> > > > > > >> wrote:
> > > > > > >> >
> > > > > > >> > Hey Mickael,
> > > > > > >> >
> > > > > > >> > Great KIP!
> > > > > > >> >
> > > > > > >> > I have one question:
> > > > > > >> >
> > > > > > >> > You mentioned "DescribeLogDirs is usually a low volume API.
> > > > > > >> > This
> > > > > > change
> > > > > > >> > should not
> > > > > > >> > significantly affect the latency of this API." and "That would
> > > > allow
> > > > > > to
> > > > > > >> > easily validate whether disk operations (like a resize), or
> > > > > > >> > topic
> > > > > > >> deletion
> > > > > > >> > (log deletion only happen after a short delay) have
> > > > > > >> > completed." I
> > > > > > wonder
> > > > > > >> if
> > > > > > >> > there is an existing metric/API that can allow administrators
> > > > > > >> > to
> > > > > > >> determine
> > > > > > >> > whether we need to resize? If administrators use this API to
> > > > determine
> > > > > > >> > whether we need a resize, would this API become a high-volume
> > > > API? I
> > > > > > >> > understand we don't want this API to be a high-volume one
> > > > > > >> > because
> > > > the
> > > > > > API
> > > > > > >> > is already costly by returning `"name": "Topics"`.
> > > > > > >> >
> > > > > > >> > Cong
> > > > > > >> >
> > > > > > >> > On Thu, Apr 7, 2022 at 2:17 AM Mickael Maison <
> > > > > > mickael.mai...@gmail.com>
> > > > > > >> > wrote:
> > > > > > >> >
> > > > > > >> > > Hi,
> > > > > > >> > >
> > > > > > >> > > I wrote a small KIP to expose the total and usable space of
> > > > logdirs
> > > > > > >> > > via the DescribeLogDirs API:
> > > > > > >> > >
> > > > > > >> > >
> > > > > > >>
> > > > > >
> > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-827%3A+Expose+logdirs+total+and+usable+space+via+Kafka+API
> > > > > > >> > >
> > > > > > >> > > Please take a look and let me know if you have any feedback.
> > > > > > >> > >
> > > > > > >> > > Thanks,
> > > > > > >> > > Mickael
> > > > > > >> > >
> > > > > > >>
> > > > > > >>
> > > > > > >>
> > > > > > >> --
> > > > > > >> Best Regards,
> > > > > > >> Raman Verma
> > > > > > >>
> > > > > >
> > > >
