AnishMahto commented on PR #56578: URL: https://github.com/apache/spark/pull/56578#issuecomment-4863130626
That's a good question, the biggest difficulty with (b) would be how do you reliably identify whether a path is really just an identifier or actually a file path, so I would probably prefer (a). But also, let me back up for a second. There's two orthogonal problems I'm realizing we need to solve: 1. For identifier based reads using `.load()`, we need to solve the eager analysis problem + we need to register the input in the dataflow graph, which could be either an internal or external read. 2. For path based reads using `.load()`, eager analysis doesn't matter. Path based reads must sldo always be to tables external to the pipeline, which we maybe we should track within the dataflow graph just for consistency reasons, but it doesn't actually affect topological execution order. Anyway I think to solve both problems (delay eager analysis and register identifier reads in the dataflow graph), we'll have to: 1. Delay analysis for all `.load()` branches in `SparkConnectPlanner` when inside a pipelines context. By delaying analysis, an unanalyzed `spark.load()` should always produce an `UnresolvedDataSource` logical plan. Notably still not an `UnresolvedRelation`, since `UnresolvedRelation` doesn't represent things like `format` that `UnresolvedDataSource` does. 2. In SDP flow analysis, in addition to inspecting `UnresolvedRelation` for identifiers and registering them in the dataflow graph, we'll also have to do the same for `UnresolvedDataSource`. This is actually similar to what you're already doing, but we'll be extracting identifiers from the canonical unresolved logical plan for data sources, rather than the analyzed plan. For step (2) what's not 100% clear to me yet and will require further thought is whether we can really use `readStreamInput`/`readBatchInput` as the drop in substitution for `UnresolvedDataSource` nodes, the way that we do for `UnresolvedRelation`. Tangentially in reading the existing `readStreamInput`/`readBatchInput` paths for `UnresolvedRelation`, I'm realizing there might be a bug in respecting read options on internal dataset reads. But that's unrelated to this discussion. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
