comphead commented on issue #17045: URL: https://github.com/apache/datafusion/issues/17045#issuecomment-3156645933
> > Thanks [@alamb](https://github.com/alamb) this is true, the entire `datafusion-spark` crate is more niche I would say. I still think it can be a subproject under datafusion, `datafusion-spark` or similar. > > Currently to ensure that Spark builtin functions in crate corresponds to Spark it is required to run tests in local Spark to verify the correctness, WDYT if we can introduce some CI or script to run tests using Spark inside the scope of this crate? > > I suspect it it would be not a good idea to add an integration test here that actually runs spark, unless it can be integrated with an existing runner (similarly to the sqllogictests runner for postgres) 🤔 Exactly, the postgres runner was an inspiration, planning to investigate if Spark runner can be done in a similar way. Ideally to have a separate CI which triggers only if `datafusion-spark` crate changed. But stepping back and look at that again the submodule more reminds a separate subproject(its not a core, niche usage, separate CI, test runner). We can keep it evolving in DF repo but later it would probably make sense to move it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
