yurmix opened a new issue, #18596: URL: https://github.com/apache/druid/issues/18596
### Description The [Delta Kernel API](https://delta-docs-incubator.netlify.app/delta-kernel/) accepts Hadoop config via the `Engine` interface. However, `DeltaInputSource` hides this and instantiates the engine with an empty `Configuration()`. It is proposed to add a hadoopProperties JSON field to the Delta Lake input source. Example: ```json "inputSource": { "type": "delta", "tablePath": "s3a://bucket/path/to/table", "hadoopProperties": { "fs.s3a.access.key": "${AWS_ACCESS_KEY_ID}", "fs.s3a.secret.key": "${AWS_SECRET_ACCESS_KEY}", "fs.s3a.session.token": "${AWS_SESSION_TOKEN}", "fs.s3a.endpoint": "s3.amazonaws.com" } } ``` ### Motivation Allow providing Hadoop properties such as S3A credentials, endpoints, and similar options at the ingestion spec level rather than globally. This enables ingesting from different environments and adding configuration specific to the ingestion task. Example use case: support reading Delta tables from S3 with temporary AWS STS credentials without relying on global configuration. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
