akshatb1 edited a comment on pull request #33135: URL: https://github.com/apache/spark/pull/33135#issuecomment-875250761
Thanks @tgravescs for looking into this. Please find my responses below. > Can you give more details about exactly what requires changing here? We have a custom implementation of setting up tokens for Yarn applications. This involves invoking some native libraries to generate different kinds of tokens and these needs to be set in AM Container launch context. Similarly, there are some changes required in Yarn configuration for running on multi-tenant Federated Yarn cluster. For this purpose, we are adding a new client similar to yarn/Client.scala along with a bunch of helper utility classes related to token handling. > If its a different yarn Client, if its way different it should just be a separate resource manager perhaps. If its something smaller (which it doesn't sound like from the description) perhaps something else needs to be more flexible. We are adding a new folder under /resource-managers/yarn such as /resource-managers/yarn/customyarn. This way it can leverage all existing internal classes and we can add custom implementation for Client in this new folder along with other utilities. > Another question - the ApplicationMaster doesn't need modified as well? ApplicationMaster does not change since only changes required are in how application submission is done. > These are all internal Spark classes as well, so making a config for it without some public or developer interface doesn't make a lot of sense. Will it be a good idea to make the private methods/fields in Client.scala to make protected to allow developers to extend the Client? Currently, it can not be extended since many methods are private. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
