Previously when doing research on this topic, I saw that the metrics-server documentation says:"*Metrics Server is not meant for non-autoscaling purposes. For example, don't use it to forward metrics to monitoring solutions, or as a source of monitoring solution metrics. In such cases please collect metrics from Kubelet /metrics/resource endpoint directly*." But the Kubelet APIs <https://github.com/kubernetes/kubernetes/blob/v1.21.5/pkg/kubelet/server/server.go#L236>that the statement refers to are not documented, meaning they are hidden APIs that can change or be deprecated at any future Kubernetes release. Integrating with these APIs doesn't sound promising. But besides Kubelet, the actual utilization info of workloads is not readily available anywhere else. We'll need to explore other ideas.
On Tue, Dec 21, 2021 at 12:51 PM Weiwei Yang <w...@apache.org> wrote: > Thank you Bowen to raise this up, this is an interesting topic. Bear with > me this long reply : ) > > Like Wilfred mentioned, YK doesn't know about the actual used resources in > terms of CPU and memory for each pod, or application, at least not today. I > understand the requirements about tracking this info in order to give users > some feedback or even recommendations on how to tune their jobs more > properly. It would be good to have something in our view as "Allocated" vs > "Used" for each app/queue. We could further introduce some penalties if > people keep over-requesting resources. > > However, most likely we will need to do this outside of YK. The major > reason is all data YK is consuming are from api-server, backed by etcd. Non > of such metrics will be stored in etcd, as per design in metrics-server > <https://github.com/kubernetes-sigs/metrics-server>. Second, YK doesn't > have any per-node agent running that we can facilitate to collect actual > resource usages, we still need to leverage a 3rd party tool to do so. Maybe > we can do some integration with metrics-server, aggregating app/queue used > info from those fragmented metrics, and then plug that into our > yunikorn-web UI. We have the flexibility to do this I believe, which could > be an option. > > On Mon, Dec 20, 2021 at 10:28 PM Wilfred Spiegelenburg < > wilfr...@apache.org> > wrote: > > > Hi Bowen, > > > > Maybe a strange question but what do you consider "actually > > used" resources? Anything the scheduler sees is used. The scheduler has > no > > information on what the container really occupies: it asked for 100GB but > > it only allocated 50GB etc. If you need that YuniKorn cannot help you. If > > it is just a looking at allocation over time YuniKorn is capable of > giving > > you the information. > > > > Second point to make is that normally applications do not provide any > > information on what they expect to use before they use it. Let's take a > > Spark application. The driver creates pods as it needs new executors. The > > Spark config drives those requests and the limitations. The scheduler > only > > sees the pods that are really requested. It does not know, and should not > > know, if that is limited by what is configured or that the job uses only > > part or more than what is configured. > > > > The only time the scheduler would have any idea about a "maximum" is > when a > > gang request is made. For gang scheduling we can track if the gang > > request is completely used or not. We could add metrics for it on an > > application. We can also track the number of containers allocated for an > > application or queue, the time from start to finish for containers etc. > We > > could even track the maximum resource allocation for an application or a > > queue over a time interval. Prometheus should give us a number of > > possibilities, we just need to hook them into the scheduling cycle. > > > > As far as I know we currently do not have application metrics but that > can > > always be added. Some queue metrics are there already. I think one of > those > > is what you are looking for to fill a number of the gaps that you see. I > > have added YUNIKORN-829 as a subtask to YUNIKORN-720 [1] which is already > > referencing a number of metrics to improve. With the release of v0.12.1 I > > moved that jira to v1.0.0. A major improvement to the metrics would be a > > nice addition for the v1.0.0. > > > > I would not see anything that is blocking enhancing metrics: it is a part > > that can be improved without a major impact on other functionality. We do > > need to make sure that we measure the impact on performance and memory > > usage. > > > > Wilfred > > > > [1] https://issues.apache.org/jira/browse/YUNIKORN-720 > > > > On Tue, 21 Dec 2021 at 16:18, Bowen Li <b...@apache.org> wrote: > > > > > Hi community, > > > > > > Reviving https://issues.apache.org/jira/browse/YUNIKORN-829 . We are > > > running Spark on YuniKorn, and have a requirement to provide more > > > observability of *actual* resource usage for our customers, data > > > engineers/scientists who wrote Spark jobs who may not have deep > expertise > > > in Spark job optimization. > > > > > > - requirement: > > > > > > - have actual resource usage metrics at both job level and queue level > > (YK > > > already have requested resource usage metrics) > > > > > > - key use case: > > > > > > - as indicators of job optimization for ICs like data > > engineers/scientists, > > > to show users how much resources they requested v.s. how much resources > > > their jobs actually used > > > > > > - as indicator for managers on their team's resource utilization. In > our > > > setup or a typical YK setup, each customer team has their own YuniKorn > > > queue in a shared, multi tenant environment. Managers of the team would > > > want high level (queue) metrics rather than low level (job) ones > > > > > > Currently we haven't found a good product on the market to do this, so > > > would be great if YuniKorn can support it. Would like your input here > on > > > feasibility (seems feasible according Weiwei's comment in Jira), > > priority, > > > and timeline/complexity of the projects. > > > > > > Thanks, > > > Bowen > > > > > >
