andygrove commented on code in PR #2478:
URL:
https://github.com/apache/datafusion-ballista/pull/2478#discussion_r4077017450
##########
python/README.md:
##########
@@ -62,10 +62,17 @@ df : DataFrame = ctx.sql("select * from t limit 5")
df.show()
```
+Session and Ballista settings are set with the `cluster_config` argument:
+
+```python
+ctx = BallistaSessionContext(
+ "df://localhost:50050",
+ cluster_config={"datafusion.execution.target_partitions": "256"},
+)
+```
+
Known limitations and inefficiencies of the current approach:
-- The client's `SessionConfig` is not propagated to Ballista.
Review Comment:
Small thing here. The `cluster_config` addition is great, but I think the
"client's `SessionConfig` is not propagated" bullet is still accurate. A
`SessionConfig` passed as `config=` only configures the local context, and
`create_ballista_data_frame` starts from a fresh `new_with_ballista()` config
and applies just `cluster_config`. Could we keep that limitation and point
people at `cluster_config` instead? Something like "A `SessionConfig` passed as
`config` is used locally only. Use `cluster_config` for settings that should
reach the cluster."
##########
docs/source/user-guide/tuning-guide.md:
##########
@@ -256,8 +256,8 @@ let session_config = SessionConfig::new_with_ballista()
When AQE is enabled, the scheduler builds the stage DAG incrementally. As each
shuffle stage completes, the planner re-optimizes the remaining plan and emits
-the next set of runnable stages. Two adaptive optimizations are currently
-implemented:
+the next set of runnable stages. The following adaptive optimizations are
+currently implemented:
- **Join reordering.** Uses runtime row counts from completed stages so the
Review Comment:
Nit: architecture.md now calls this "Join selection", but the tuning guide
still has "Join reordering". Worth aligning the two while we're here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
