xinrong-databricks commented on a change in pull request #32821:
URL: https://github.com/apache/spark/pull/32821#discussion_r650119142
##########
File path: python/pyspark/pandas/data_type_ops/base.py
##########
@@ -206,3 +209,10 @@ def restore(self, col: pd.Series) -> pd.Series:
def prepare(self, col: pd.Series) -> pd.Series:
"""Prepare column when from_pandas."""
return col.replace({np.nan: None})
+
+ def isnull(self, series):
Review comment:
Shall we add type annotations?
##########
File path: python/pyspark/pandas/data_type_ops/base.py
##########
@@ -206,3 +209,10 @@ def restore(self, col: pd.Series) -> pd.Series:
def prepare(self, col: pd.Series) -> pd.Series:
"""Prepare column when from_pandas."""
return col.replace({np.nan: None})
+
+ def isnull(self, series):
Review comment:
The `series` parameter should be of `Union["Series",
"Index"]`(`IndexOpsMixin`) type, shall we rename it more generally, for
example, `index_ops`?
It would be wonderful the parameter type hints can be added as well.
##########
File path: python/pyspark/pandas/data_type_ops/num_ops.py
##########
@@ -344,3 +344,20 @@ def rfloordiv(left, right):
right = transform_boolean_operand_to_numeric(right,
left.spark.data_type)
return numpy_column_op(rfloordiv)(left, right)
+
+ def isnull(self, series) -> Union["Series", "Index"]:
+ return series._with_new_scol(series.spark.column.isNull() |
F.isnan(series.spark.column))
+
+
+class DecimalOps(FractionalOps):
+ """
+ The class for decimal operations of pandas-on-Spark objects with spark
type:
+ DecimalType.
+ """
+
+ @property
+ def pretty_name(self) -> str:
+ return "decimal"
+
+ def isnull(self, series) -> Union["Series", "Index"]:
Review comment:
ditto
##########
File path: python/pyspark/pandas/data_type_ops/num_ops.py
##########
@@ -344,3 +344,20 @@ def rfloordiv(left, right):
right = transform_boolean_operand_to_numeric(right,
left.spark.data_type)
return numpy_column_op(rfloordiv)(left, right)
+
+ def isnull(self, series) -> Union["Series", "Index"]:
Review comment:
ditto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
