itholic commented on PR #40280:
URL: https://github.com/apache/spark/pull/40280#issuecomment-1455565581
Thanks @panbingkun for the nice fix!
Btw, think I found another `createDataFrame` bug which is not working
properly with non-nullable schema as below:
```python
>>> from pyspark.sql.types import *
>>> schema_false = StructType([StructField("id", IntegerType(), False)])
>>> spark.createDataFrame([[1]], schema=schema_false)
Traceback (most recent call last):
...
pyspark.errors.exceptions.connect.AnalysisException:
[NULLABLE_COLUMN_OR_FIELD] Column or field `id` is nullable while it's required
to be non-nullable.
```
whereas working find with nullable schema as below:
```python
>>> schema_true = StructType([StructField("id", IntegerType(), True)])
>>> spark.createDataFrame([[1]], schema=schema_true)
DataFrame[id: int]
```
Do you have any idea what might be causing this? Could you take a look at it
if you're interested in? I have filed an issue at SPARK-42679.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
