rangareddy commented on issue #17128: URL: https://github.com/apache/hudi/issues/17128#issuecomment-5366099498
This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-9674). **Findings: routed to the performance validation pass. Worth splitting first.** Unusually for a performance ticket, the description lists concrete and independently actionable items rather than a general complaint: 1. Make `isSplitable` in the `FileFormat` interface depend on whether the path is base-file-only, so large files parallelise - always for MOR read-optimized queries, and in some MOR real-time cases. 2. Gate returning columnar batches in Spark on the same base-file-only condition, so MOR read-optimized can use it. 3. Explore vectorized read support. Each of those is a separate change with its own measurable outcome and its own risk profile - item 1 affects split planning, item 2 affects the columnar path, item 3 is exploratory. Tracking them as one audit makes it hard to land any of them, and hard to attribute a regression if one causes it. Recommend splitting into one ticket per item, each carrying its own before/after measurement on a stated dataset and hardware. Items 1 and 2 share the base-file-only predicate, so they could reasonably be one change. Keeping this open. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
