[
https://issues.apache.org/jira/browse/DATAFU-155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17810299#comment-17810299
]
Eyal Allweil edited comment on DATAFU-155 at 1/24/24 9:53 AM:
--------------------------------------------------------------
I'm embarrassed at how long this has slipped under our radar .... anyways,
looks like compilation for this is broken now (in Spark 3.1.x). But maybe
fixing it isn't too hard ...
{code:java}
... git/datafu/datafu-spark/src/main/scala/datafu/spark/SparkUDFs.scala:149:
type mismatch;
found : Seq[Boolean]
required:
Seq[Option[org.apache.spark.sql.catalyst.encoders.ExpressionEncoder[_]]]
ScalaUDF(func, dataType, e, inputsNullSafe, inputTypes)
^
... git/datafu/datafu-spark/src/main/scala/datafu/spark/SparkUDFs.scala:149:
type mismatch;
found : Seq[org.apache.spark.sql.types.DataType]
required: Option[org.apache.spark.sql.catalyst.encoders.ExpressionEncoder[_]]
ScalaUDF(func, dataType, e, inputsNullSafe, inputTypes)
^
two errors found{code}
was (Author: eyal):
Looks like compilation for this is broken now (in Spark 3.1.x). But maybe
fixing it isn't too hard ...
{code:java}
... git/datafu/datafu-spark/src/main/scala/datafu/spark/SparkUDFs.scala:149:
type mismatch;
found : Seq[Boolean]
required:
Seq[Option[org.apache.spark.sql.catalyst.encoders.ExpressionEncoder[_]]]
ScalaUDF(func, dataType, e, inputsNullSafe, inputTypes)
^
... git/datafu/datafu-spark/src/main/scala/datafu/spark/SparkUDFs.scala:149:
type mismatch;
found : Seq[org.apache.spark.sql.types.DataType]
required: Option[org.apache.spark.sql.catalyst.encoders.ExpressionEncoder[_]]
ScalaUDF(func, dataType, e, inputsNullSafe, inputTypes)
^
two errors found{code}
> Ease Scala-Python API expoure
> -----------------------------
>
> Key: DATAFU-155
> URL: https://issues.apache.org/jira/browse/DATAFU-155
> Project: DataFu
> Issue Type: Bug
> Reporter: Ohad Raviv
> Priority: Minor
> Fix For: 2.1.0
>
>
> As one of the key features in datafu-spark is the integration between Scala
> and Python code, we should add a helper method to more easily expose Scala
> APIs in python.
> currently, the wrapper methods need to deal with some non-trivial conversions
> that py4j doesn't handle well as it is oriented to python-java conversions.
> so we want to extend it to cope with Scala collections, Option, etc. to save
> some boilerplate code in both the Scala side and Python side.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
