strawgate opened a new issue, #25663: URL: https://github.com/apache/datafusion/issues/25663
When adapting expressions to an older physical schema, a missing nullable column is expected and is replaced with a null literal. The current adapter calls to check for that column. On a miss, Arrow formats an error containing all field names, which the adapter immediately discards. This makes schema adaptation increasingly expensive as physical schemas grow.\n\nUse the non-error-producing field lookup for this expected miss and retain the existing behavior for present columns and non-nullable missing columns. A small benchmark covering present and missing columns in 16- and 128-field schemas would make the cost visible. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
