GitHub user mbutrovich edited a comment on the discussion: Support Spark 3.3/JDK8
> @andygrove @mbutrovich one more question, for "shift resources particularly > for 3.5 and 4.0 support", the "resource" here means "developing/developer > resource" right? If that is case, if I'd like to support spark 3.3 related > developing/reviewing staff(After a glance at #1529, seems the main code is > not too much), is it possible to add it back to Comet master? The resource is not only developers/reviewers, but also CI resources. Our CI matrix is already quite large, and running Spark's test suites for 3.4, 3.5. and 4.0 is typically the bottleneck. The difference in Spark behavior between 3.3 and 4.0 (which we'd like to start treating as a primary target for Comet) is also such that maintaining that many shims and conditional codepaths is concerning to me. Lastly, would you require JDK8 support or are you just asking about 3.3 with JDK 11+? AFAIK Comet cannot support JDK8 anymore as Arrow Java has dropped support for it, and we'd have to backport critical behavior and bug fixes to an older fork of Arrow Java, which seems like a non-starter to me. GitHub link: https://github.com/apache/datafusion-comet/discussions/3043#discussioncomment-15445990 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
