andygrove commented on code in PR #2619:
URL:
https://github.com/apache/datafusion-ballista/pull/2619#discussion_r4407217634
##########
python/README.md:
##########
@@ -81,7 +81,9 @@ Known limitations and inefficiencies of the current approach:
- A `SessionConfig` passed as `config` is used locally only. Use
`cluster_config` for settings that
should reach the cluster.
- Anything requiring custom
`datafusion_proto::logical_plan::LogicalExtensionCodec`.
-- No support for `UDF` as DataFusion Python does not serialise them.
+- Python UDFs (scalar, aggregate and window) are serialised with the plan
using `cloudpickle`, so
Review Comment:
No need to hold this PR for it, I filed #2687 to add the trust note and the
Python-started-cluster caveat as a follow-up.
##########
python/src/cluster.rs:
##########
@@ -42,7 +45,15 @@ impl PyScheduler {
#[pyo3(signature = (bind_host=None, bind_port=None))]
#[new]
pub fn new(_py: Python, bind_host: Option<String>, bind_port: Option<u16>)
-> Self {
- let mut config = SchedulerConfig::default();
+ let mut config = SchedulerConfig {
+ override_logical_codec: Some(Arc::new(
Review Comment:
Filed #2688 for this one.
##########
python/src/ffi_planner_workaround.rs:
##########
@@ -182,11 +189,23 @@ unsafe extern "C" fn create_physical_plan(
}
}
+/// Family prefixes of inline Python UDF payloads written by
+/// [`PythonLogicalCodec`], mirroring datafusion-python's `pub(crate)`
constants.
+const PY_SCALAR_UDF_FAMILY: &[u8] = b"DFPYUDF";
Review Comment:
I went ahead and opened apache/datafusion-python#1817 for this. Once it
lands we can drop the copies here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
