RatulDawar commented on issue #21719: URL: https://github.com/apache/datafusion/issues/21719#issuecomment-4299442947
@Dandandan so I have been trying certain things for past 2 days and wanted to share here so that you can review my approach. I wanted to first try profilers like [tokio-console](https://github.com/tokio-rs/console), which I tried setting up locally but was unable to stream datafusion metrics to it. So I switched to logging thead swtich events and ran a short LLM generated experiment to simulate the same. [Here](https://github.com/RatulDawar/datafusion/pull/2) are the details so you can take a brief look. If the logging approach looks correct I can bundle datafusion with something similar to this and run benchmarks to get these metrics per benchmark. Let me know that you thing about here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
