GitHub user ianthetechie edited a discussion: Difficulty reconciling table option types for listing vs direct usage
When adding a data source to the context, I think the current story around types is a bit clunky, but I wanted to start a discussion to make sure that 1) I'm not missing something, and 2) this isn't just me hitting it a a bad time in the middle of a refactor ;) I tried to search for similar issues but couldn't find anything, so here I am! What I think is clunky: When writing some code like `ctx.register_parquet`, I need to pass a type `ParquetReadOptions<'_>`. However when I want to create a `ListingTable`, there is no way to pass these on. The closest I could do is to is create a `ParquetFormat` and (maybe a method call, like `with_options` later) specify some `ParquetOptions` which are similar but not the same. It looks like there is a `TryFrom` impl for `ParquetWriterOptions`. There is a similar story for CSV, though a few details differ. It looks like there is also a family of types like `CsvOptions` that get used in the file format APIs. Are these eventually going to be some common subset between reader and writer options which could be used for setting listing table options and the `register_X` (where `X` is a format) paths alike? Or am I just thinking about this wrong? GitHub link: https://github.com/apache/datafusion/discussions/16763 ---- This is an automatically sent email for github@datafusion.apache.org. To unsubscribe, please send an email to: github-unsubscr...@datafusion.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
