Re: [PR] build: Switch back to released version of DataFusion and arrow-rs after Arrow Java 16 is released [datafusion-comet]

advancedxy commented on PR #403: URL: https://github.com/apache/datafusion-comet/pull/403#issuecomment-2115424225 > We can hold Comet release after the new releases of arrow-rs and DataFusion. Thanks for the clarification. I second this. -- This is an automated message from the Apa

Re: [PR] build: Switch back to released version of DataFusion and arrow-rs after Arrow Java 16 is released [datafusion-comet]

viirya commented on PR #403: URL: https://github.com/apache/datafusion-comet/pull/403#issuecomment-2115419825 arrow-rs and DataFusion have fast release cycle. We can hold Comet release after the new releases of arrow-rs and DataFusion. -- This is an automated message from the Apache Git S

Re: [PR] build: Switch back to released version of DataFusion and arrow-rs after Arrow Java 16 is released [datafusion-comet]

advancedxy commented on PR #403: URL: https://github.com/apache/datafusion-comet/pull/403#issuecomment-2115404901 > We only need to wait for new release of arrow-rs and DataFusion. Do we have an estimate on when new releases of arrow-rs and DataFusion will be available? I'm asking bec

Re: [PR] build: Switch back to released version of DataFusion and arrow-rs after Arrow Java 16 is released [datafusion-comet]

viirya commented on PR #403: URL: https://github.com/apache/datafusion-comet/pull/403#issuecomment-2109035531 I have verified the fix https://github.com/apache/arrow-rs/issues/5756 can work by cherry-picking it in my forked branch of arrow-rs. -- This is an automated message from the Apac

Re: [PR] build: Switch back to released version of DataFusion and arrow-rs after Arrow Java 16 is released [datafusion-comet]

viirya commented on PR #403: URL: https://github.com/apache/datafusion-comet/pull/403#issuecomment-2106677456 I fixed the issue at arrow-rs in the PR https://github.com/apache/arrow-rs/issues/5756. We can continue this PR after the fix is released. Keep this as draft for now. -

Re: [PR] build: Switch back to released version of DataFusion and arrow-rs after Arrow Java 16 is released [datafusion-comet]

viirya commented on PR #403: URL: https://github.com/apache/datafusion-comet/pull/403#issuecomment-2103477288 Ah, I found that I made a mistake in the Java Arrow PR that it doesn't initiate the offset buffer well. Proposed another issue at Java Arrow https://github.com/apache/arrow/issues/4

Re: [PR] build: Switch back to released version of DataFusion and arrow-rs after Arrow Java 16 is released [datafusion-comet]

viirya commented on PR #403: URL: https://github.com/apache/datafusion-comet/pull/403#issuecomment-2103282710 Hmm, the error is actually different: ``` Cause: org.apache.comet.CometNativeException: Fail to process Arrow array with reason C Data interface error: The external buffe

Re: [PR] build: Switch back to released version of DataFusion and arrow-rs after Arrow Java 16 is released [datafusion-comet]

viirya commented on PR #403: URL: https://github.com/apache/datafusion-comet/pull/403#issuecomment-2101797372 Besides, the Java Arrow bus is supported to be fixed in Arrow Java 16. But I still see it happens in a few pipelines: ``` - multiple column distinct count *** FAILED *** (3

Re: [PR] build: Switch back to released version of DataFusion and arrow-rs after Arrow Java 16 is released [datafusion-comet]

viirya commented on PR #403: URL: https://github.com/apache/datafusion-comet/pull/403#issuecomment-2101795378 The fix (https://github.com/apache/datafusion/pull/10094) is not released yet. So the following error still happens in CI: ``` CometAggregateSuite: - count with aggregat

Re: [PR] build: Switch back to released version of DataFusion and arrow-rs after Arrow Java 16 is released [datafusion-comet]

viirya commented on PR #403: URL: https://github.com/apache/datafusion-comet/pull/403#issuecomment-2101108193 Note that this is blocked by #250. We need to verify Java Arrow after #250 is merged. -- This is an automated message from the Apache Git Service. To respond to the message, pleas

[PR] build: Switch back to released version of DataFusion and arrow-rs after Arrow Java 16 is released [datafusion-comet]

viirya opened a new pull request, #403: URL: https://github.com/apache/datafusion-comet/pull/403 ## Which issue does this PR close? Closes #248. ## Rationale for this change ## What changes are included in this PR? ## How are these changes t