JingsongLi commented on code in PR #6147:
URL: https://github.com/apache/paimon/pull/6147#discussion_r2300149516
##########
paimon-format/src/main/java/org/apache/paimon/format/csv/CsvFileFormat.java:
##########
@@ -52,7 +52,23 @@ public CsvFileFormat(FormatContext context) {
@Override
public FormatReaderFactory createReaderFactory(
RowType projectedRowType, @Nullable List<Predicate> filters) {
- return new CsvReaderFactory(projectedRowType, options);
+ throw new UnsupportedOperationException(
+ "CSV format support projection push down must with all row
type");
+ }
+
+ /**
+ * Create a {@link FormatReaderFactory} with support for different read
and projected types.
+ * This enables better projection pushdown for CSV format by reading the
full file schema and
+ * projecting to the requested columns.
+ *
+ * @param rowReadType The full schema type of the data in the CSV file
+ * @param projectedRowType The projected schema type for columns to return
+ * @param filters A list of filters in conjunctive form for filtering on a
best-effort basis
+ * @return A reader factory configured for projection pushdown
+ */
+ public FormatReaderFactory createReaderFactory(
+ RowType rowReadType, RowType projectedRowType, @Nullable
List<Predicate> filters) {
Review Comment:
Maybe just modify `FileFormat.createReaderFactory`, one method:
```
public abstract FormatReaderFactory createReaderFactory(
RowType tableRowType, RowType projectedRowType, @Nullable
List<Predicate> filters);
```
You can create a separate PR to do this refactor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
