milenkovicm commented on code in PR #2252:
URL:
https://github.com/apache/datafusion-ballista/pull/2252#discussion_r3738733636
##########
python/python/ballista/extension.py:
##########
Review Comment:
Can we remove this once we get FFI planner ?
##########
python/python/ballista/extension.py:
##########
@@ -66,7 +68,12 @@ def method_wrapper(*args, **kwargs):
address = args[0].address
session_id = args[0].session_id
df = func(*args, **kwargs)
- return DistributedDataFrame(df, session_id, address)
+ return DistributedDataFrame(
Review Comment:
DistributedDataFrame, should be removed with FFI, should it ?
##########
python/examples/getting_started.ipynb:
##########
@@ -77,10 +82,25 @@
"# ctx = BallistaSessionContext(\"df://your-scheduler:50050\")\n",
"\n",
"host, port = setup_test_cluster()\n",
- "ctx = BallistaSessionContext(f\"df://{host}:{port}\")\n",
+ "address = f\"df://{host}:{port}\"\n",
+ "\n",
+ "ballista_config = ballista_datafusion_config_defaults()\n",
+ "ctx = SessionContext(SessionConfig(ballista_config)).with_extensions(\n",
Review Comment:
Can we use ballista context to hide next few lines and simplify session
context creation ?
##########
ballista/core/src/serde/mod.rs:
##########
@@ -349,22 +358,98 @@ impl LogicalExtensionCodec for
BallistaLogicalExtensionCodec {
.encode(buf)
.map_err(|e| DataFusionError::Internal(e.to_string()))
}
+
+ fn try_decode_udf(
+ &self,
+ name: &str,
+ buf: &[u8],
+ ) -> Result<Arc<datafusion::logical_expr::ScalarUDF>> {
+ self.default_codec.try_decode_udf(name, buf)
Review Comment:
Do we need to do something extra to get python udf serde?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
