jayshrivastava commented on issue #20096:
URL: https://github.com/apache/datafusion/issues/20096#issuecomment-3834894128
Sure, feel free to close out the issue if you think that makes sense.
One follow up - do you know how split points are determined? I would expect
files to not be split up unless the `FileSource::file_type` was well known (ex.
vortext, parquet, csv). My file format is none of those.
I'm also using a `FileScanConfig` wrapper which does not support
`repartitioned`.
```
struct Wrapper<T> {
inner: T,
partitioning: Partitioning,
}
impl<T> DataSource for PartitionedFileScanConfig<T>
where
T: DataSource + 'static,
{
delegate! {
to self.inner {
fn open(
&self,
partition: usize,
context: Arc<TaskContext>,
) -> Result<SendableRecordBatchStream>;
fn eq_properties(&self) -> EquivalenceProperties;
fn scheduling_type(&self) -> SchedulingType;
fn partition_statistics(&self, partition: Option<usize>) ->
Result<Statistics>;
fn with_fetch(&self, limit: Option<usize>) -> Option<Arc<dyn
DataSource>>;
fn fetch(&self) -> Option<usize>;
fn metrics(&self) -> ExecutionPlanMetricsSet;
fn try_swapping_with_projection(
&self,
projection: &ProjectionExprs
) -> Result<Option<Arc<dyn DataSource>>>;
fn try_pushdown_filters(
&self,
filters: Vec<Arc<dyn PhysicalExpr>>,
config: &ConfigOptions,
) -> Result<FilterPushdownPropagation<Arc<dyn DataSource>>>;
}
}
fn repartitioned(
&self,
_target_partitions: usize,
_repartition_file_min_size: usize,
_output_ordering: Option<LexOrdering>,
) -> Result<Option<Arc<dyn DataSource>>> {
Ok(None)
}
fn output_partitioning(&self) -> Partitioning {
self.partitioning.clone()
}
fn as_any(&self) -> &dyn Any {
// Delegate to inner so downcasting to FileScanConfig works
self.inner.as_any()
}
fn fmt_as(&self, t: DisplayFormatType, f: &mut Formatter) -> FmtResult {
self.inner.fmt_as(t, f)
}
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
