[ https://issues.apache.org/jira/browse/SPARK-37039?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Yikun Jiang updated SPARK-37039: -------------------------------- Description: np.nan series.astype(bool) should be True, rather than Fasle: https://github.com/apache/spark/blob/46bcef7472edd40c23afd9ac74cffe13c6a608ad/python/pyspark/pandas/data_type_ops/base.py#L147 >>> pd.Series([1, 2, np.nan], dtype=float).astype(bool) >>> pd.Series([1, 2, np.nan], dtype=str).astype(bool) >>> pd.Series([datetime.date(1994, 1, 31), datetime.date(1994, 2, 1), np.nan]) 0 True 1 True 2 True dtype: bool But in pyspark, it is: 0 True 1 True 2 False dtype: bool was: np.nan series.astype(bool) should be True, rather than Fasle: https://github.com/apache/spark/blob/46bcef7472edd40c23afd9ac74cffe13c6a608ad/python/pyspark/pandas/data_type_ops/base.py#L147 > np.nan series.astype(bool) should be True > ----------------------------------------- > > Key: SPARK-37039 > URL: https://issues.apache.org/jira/browse/SPARK-37039 > Project: Spark > Issue Type: Sub-task > Components: PySpark > Affects Versions: 3.3.0 > Reporter: Yikun Jiang > Priority: Major > > np.nan series.astype(bool) should be True, rather than Fasle: > https://github.com/apache/spark/blob/46bcef7472edd40c23afd9ac74cffe13c6a608ad/python/pyspark/pandas/data_type_ops/base.py#L147 > >>> pd.Series([1, 2, np.nan], dtype=float).astype(bool) > >>> pd.Series([1, 2, np.nan], dtype=str).astype(bool) > >>> pd.Series([datetime.date(1994, 1, 31), datetime.date(1994, 2, 1), np.nan]) > 0 True > 1 True > 2 True > dtype: bool > But in pyspark, it is: > 0 True > 1 True > 2 False > dtype: bool -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org