wypoon commented on pull request #4395: URL: https://github.com/apache/iceberg/pull/4395#issuecomment-1082594615
@kbendick thank you for the perspective. As @RussellSpitzer has pointed out, custom task metrics do not appear to be the right mechanism for getting a correct count for number of data files read, or number of delete files applied while reading data files. I think we need to wait for Spark support for driver-side custom metrics to implement those properly. In the meantime, I was thinking of renaming the custom metric that I have here "number of file splits read". I'll update the PR very soon. For other metrics that are actually computed at task time, such as the one Russell suggested, let me look into that. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
