LuciferYang commented on pull request #31517: URL: https://github.com/apache/spark/pull/31517#issuecomment-897430503
> @LuciferYang Caffeine is definitely an amazing and popular project. But when we decide whether we should add a library , especially a popular library, to Spark, we also need to think about the impact to the downstream libraries and Spark applications. As Caffeine is pretty popular, many downstream libraries and Spark users probably are using it. If Spark adds it to its classpath, it will force them to use the same Caffeine version as Spark. If someone would like to use a different Caffeine version, the work would be non trivial since they need to shade their Caffeine version. Can we solve this problem by `shade + relocation`, just like now Spark uses `Guava` and `Jetty` now. If we accept it, I can do this work. > Do you have any examples that we can get Caffeine's benefits from hot path, so that we can use that to make the tradeoff? For example, `FileStatusCache` on driver side of a resident process(like thrift-server). Of course, the improvement will not be very significant because Guava Cache is not bad. Overall, I think both are OK. If need to revert the code, please tell me.I am not entangled and opposed to this. I can also be responsible for completing this work :) I just want to mention it again. As I said [above](https://github.com/apache/spark/pull/31517#issuecomment-896504990), we are still using Guava 14.0.1, which is a 2013 version. Some fixed about Guava Cache and other Guava components are still hidden in Spark, and I found many JIRA want to upgrade it which seems to have not been completed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org
