Github user HyukjinKwon commented on a diff in the pull request:
https://github.com/apache/spark/pull/20163#discussion_r161363630
--- Diff:
sql/core/src/main/scala/org/apache/spark/sql/execution/python/EvaluatePython.scala
---
@@ -144,6 +145,7 @@ object EvaluatePython {
}
case StringType => (obj: Any) => nullSafeConvert(obj) {
+ case _: Calendar => null
case _ => UTF8String.fromString(obj.toString)
--- End diff --
BTW, seems there is another hole when we actually do the internal
conversion with unexpected types:
```python
>>> from pyspark.sql.functions import udf
>>> f = udf(lambda x: x, "date")
>>> spark.range(1).select(f("id")).show()
```
```
org.apache.spark.api.python.PythonException: Traceback (most recent call
last):
File "./python/pyspark/worker.py", line 229, in main
process()
File "./python/pyspark/worker.py", line 224, in process
serializer.dump_stream(func(split_index, iterator), outfile)
File "./python/pyspark/worker.py", line 149, in <lambda>
func = lambda _, it: map(mapper, it)
File "<string>", line 1, in <lambda>
File "./python/pyspark/worker.py", line 72, in <lambda>
return lambda *a: toInternal(f(*a))
File "/.../pyspark/sql/types.py", line 175, in toInternal
return d.toordinal() - self.EPOCH_ORDINAL
AttributeError: 'int' object has no attribute 'toordinal'
```
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
