BryanCutler commented on a change in pull request #28957:
URL: https://github.com/apache/spark/pull/28957#discussion_r453407345
##########
File path: python/pyspark/sql/types.py
##########
@@ -1487,36 +1451,14 @@ class Row(tuple):
True
"""
- # Remove after Python < 3.6 dropped, see SPARK-29748
Review comment:
Yup, this looks good. I noticed you already fixed up the test cases that
if affects, so that's great!
##########
File path: python/pyspark/sql/tests/test_pandas_grouped_map.py
##########
@@ -139,9 +134,9 @@ def test_supported_types(self):
result3 = df.groupby('id').apply(udf3).sort('id').toPandas()
expected3 = expected1
- assert_frame_equal(expected1, result1,
check_column_type=_check_column_type)
- assert_frame_equal(expected2, result2,
check_column_type=_check_column_type)
- assert_frame_equal(expected3, result3,
check_column_type=_check_column_type)
+ assert_frame_equal(expected1, result1, check_column_type=True)
Review comment:
I think the default is `True`, so not needed but no big deal to leave it.
##########
File path: python/pyspark/sql/pandas/serializers.py
##########
@@ -180,7 +173,7 @@ def create_array(s, t):
if len(s) == 0 and len(s.columns) == 0:
arrs_names = [(pa.array([], type=field.type), field.name)
for field in t]
# Assign result columns by schema name if user labeled with
strings
- elif self._assign_cols_by_name and any(isinstance(name,
basestring)
+ elif self._assign_cols_by_name and any(isinstance(name, str)
Review comment:
We might want to think about removing this as an option as a followup.
It was mostly added because dataframe constructed with python < 3.6 could not
guarantee the order of columns, but now it should match the given schema.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org
For additional commands, e-mail: reviews-h...@spark.apache.org
