[GitHub] [hudi] danny0405 commented on pull request #9118: [HUDI-2141] Support flink write metrics
danny0405 commented on PR #9118: URL: https://github.com/apache/hudi/pull/9118#issuecomment-1644912724 > > I kind of think 2 kinds of metrics are more important than the write metrics: > > > > 1. the streaming reader latest instant and max completion time; > > 2. the compaction progress, that is the total pending compaction plans and the current compaction instant > > @danny0405 Hi Danny, thanks for your review! I will update the write metrics document ASAP. And I'd like to add some metrics to indicate the current progress of compaction like all pending log files count & size, compaction IO, compaction execution time, etc. However, I'm not very familiar with Flink stream read path, we could finish that part in the future iteration. In `StreamReadMonitoringFunction` we will bookkeep the latest consumed instant for each round of read, it can be defined as a metric. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9118: [HUDI-2141] Support flink write metrics
danny0405 commented on PR #9118: URL: https://github.com/apache/hudi/pull/9118#issuecomment-1643685014 I kind of think 2 kinds of metrics are more important than the write metrics: 1. the streaming reader latest instant and max completion time; 2. the compaction progress, that is the total pending compaction plans and the current compaction instant -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [hudi] danny0405 commented on pull request #9118: [HUDI-2141] Support flink write metrics
danny0405 commented on PR #9118: URL: https://github.com/apache/hudi/pull/9118#issuecomment-1628084953 Thanks for the contribution @stream2000 , can we add list a table of metrics we support here, illustrating the usage of each metrics, and then we are clear whether it deserves to introduce the metrics. Another metrics I got from feedback is the compaction progress metrics. Do we have plan to support that? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@hudi.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
