karuppayya commented on PR #4309: URL: https://github.com/apache/datafusion-comet/pull/4309#issuecomment-4521960402
I guess the JNI call to `getCredentialsForPath` runs through the Comet tokio runtime, which is sized at `spark.executor.cores` worker threads. The call seems to be synchronous. Since the call duration is non-deterministic and entirely controlled by the vendor's implementation, this can potentially block unrelated work on the runtime today and as the system grows. Do you think this is an issue? That said, the current shape is a reasonable starting point . We can keeping refining in subsqeuent PRs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
