mbutrovich commented on PR #1387: URL: https://github.com/apache/datafusion-comet/pull/1387#issuecomment-2654633256
For context: when we were working in the experimental branch and added the `SchemaAdapter`, we were calling into cast.rs for type conversions. However, we started making changes in cast.rs to get more tests to pass for the experimental native scans. Not wanting to commingle logic and accidentally break cast behavior, we duplicated cast.rs and created parquet_support.rs to start making changes in. I now realize that duplicating cast.rs might have been a mistake, and have removed most of that logic from parquet_support.rs. This PR now defaults to DataFusion's cast logic unless we need specialized logic as in `cast_struct_to_struct`. This change does not introduce any new failures for the experimental native scans, but that doesn't mean this approach is bug free. @andygrove is working on some fuzz testing that will help us have more confidence in the experimental native scans in the future. I propose we move forward with this design, and bring over (or call into) cast.rs logic as we discover issues with this design. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
