PDGGK opened a new pull request, #2826: URL: https://github.com/apache/datafusion-ballista/pull/2826
# Which issue does this PR close? Closes #2817. # Rationale for this change `result_schema` in `flight_sql.rs` takes the schema of the first stage with no `output_links`. The adaptive graph never fills in `output_links`, so every stage matches, and `stages()` is a `HashMap`, so the schema can come from an intermediate stage. Through Flight SQL against an in-process scheduler and executor, `SELECT name, min(id) FROM people GROUP BY name` can promise the partial aggregate's column name: ``` FlightInfo schema: name: Utf8, min(people.id)[value]: Int64 schema of the data: name: Utf8, min(people.id): Int64 ``` # What changes are included in this PR? - `ExecutionGraph::final_stage_ids()`. By default it returns the stages with no `output_links`. The adaptive graph overrides it with the stages its `output_locations` point at, which is what #2753 used for `intermediate_stage_ids()`, and that method now calls it. - `result_schema` reads the schema of that stage. If the final stage is not known yet, it plans the query again, as it already does when the graph is gone. Tests: - `the_advertised_schema_of_a_multi_stage_query_matches_the_data` runs the query above through Flight SQL 16 times, because the result depends on `HashMap` order. Without the `flight_sql.rs` change it fails with the schemas above. - The `job_success` tests check that `final_stage_ids()` is empty until the job succeeds and then matches the output locations. With the change, the 7 Flight SQL tests and the 613 scheduler lib tests pass, and `cargo fmt` and `cargo clippy --all-targets --features flight-sql -- -D warnings` are clean for the scheduler. # Are there any user-facing changes? A Flight SQL client now gets the final stage's schema for adaptive jobs. `final_stage_ids()` is a new trait method with a default implementation, so no public API changes in a breaking way. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
