andygrove opened a new pull request, #1613:
URL: https://github.com/apache/datafusion-ballista/pull/1613
# Which issue does this PR close?
Closes #.
# Rationale for this change
`BallistaSessionContext` accepts a `cluster_config` dict that is intended to
be forwarded to the scheduler-side session (e.g.
`ballista.shuffle.sort_based.enabled`). The constructor also applies every
entry to the local DataFusion `SessionConfig` so that DataFusion-namespaced
keys (e.g. `datafusion.execution.target_partitions`) are honored during local
table registration and planning.
The problem is that the local `SessionConfig` built inside
`BallistaSessionContext` does not have the `BallistaConfig` extension
registered, so any `ballista.*` key blows up with:
```
PanicException: Configuration("Could not find config namespace \"ballista\"")
```
This makes the documented `cluster_config` parameter effectively unusable
for any Ballista-specific tuning from Python.
# What changes are included in this PR?
- `python/python/ballista/extension.py`: skip `ballista.*` keys in the
local-apply loop. The full dict is still stored on the context and forwarded to
the scheduler via `create_ballista_data_frame`, so scheduler-side behavior is
unchanged.
- `python/python/tests/test_context.py`: new
`test_cluster_config_accepts_ballista_namespaced_keys` covering both a
DataFusion key and a Ballista key in the same `cluster_config` and asserting
that context construction and a trivial query both succeed.
# Are there any user-facing changes?
Yes, but only in the sense that a previously panicking call now works. No
public API shape changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
