https://spark.apache.org/docs/3.3.2/sql-ref-functions-udf-aggregate.html
I'm trying to run this example on Databricks, and it fails with the stacktrace
below. It's literally a copy-paste from the example, what am I missing?
Job aborted due to stage failure: Task not serializable:
java.io.NotSerializableException: org.apache.spark.sql.TypedColumn
Serialization stack: - object not serializable (class:
org.apache.spark.sql.TypedColumn, value:
myaverage(knownnotnull(assertnotnull(input[0,
$line867a2458f6df413bb3920f7fc2dbb8c381.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$Average,
true])).sum AS sum, knownnotnull(assertnotnull(input[0,
$line867a2458f6df413bb3920f7fc2dbb8c381.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$Average,
true])).count AS count, newInstance(class
$line867a2458f6df413bb3920f7fc2dbb8c381.$read$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$$iw$Average),
boundreference()) AS average_salary)