HyukjinKwon commented on a change in pull request #32853:
URL: https://github.com/apache/spark/pull/32853#discussion_r650452589
##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -1992,6 +1992,16 @@ def difference(self, other: "Index", sort:
Optional[bool] = None) -> "Index":
"""
from pyspark.pandas.indexes.multi import MultiIndex
+ if isinstance(other, Index) and not isinstance(self, type(other)):
+ if isinstance(self, MultiIndex):
+ # In case `self` is MultiIndex and `other` is Index,
+ # return MultiIndex without its names.
+ return self.rename([None] * len(self))
+ elif isinstance(self, Index):
+ # In case `self` is Index and `other` is MultiIndex,
+ # return Index without its name.
+ return self.rename(None)
+
if not is_list_like(other):
Review comment:
Can we don't depend on the pandas comparison? `is_list_like` in pandas
is not designed to handle pandas-on-Spark instances, and I don't know if it's
safe to use it against pandas-on-Spark instances.
##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -1992,6 +1992,16 @@ def difference(self, other: "Index", sort:
Optional[bool] = None) -> "Index":
"""
from pyspark.pandas.indexes.multi import MultiIndex
+ if isinstance(other, Index) and not isinstance(self, type(other)):
Review comment:
can you extract this condition and define it in a variable? It's
difficult to follow the if-elses here.
##########
File path: python/pyspark/pandas/indexes/base.py
##########
@@ -2000,11 +2010,16 @@ def difference(self, other: "Index", sort:
Optional[bool] = None) -> "Index":
sort
)
)
- # Handling MultiIndex
+ # Handling MultiIndex when `other` is list-like objects.
if isinstance(self, MultiIndex) and not isinstance(other, MultiIndex):
Review comment:
can you extract this condition and define it in a variable? It's
difficult to follow the if-elses here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
