Github user AlexanderKoryagin commented on the issue: https://github.com/apache/spark/pull/22568 @HyukjinKwon Just noticed strange behavior, that can be reproduced with a test below placed under `python.pyspark.sql.tests.GroupedMapPandasUDFTests`. Is it a bug or some feature? ```python def test_supported_types_array(self): from pyspark.sql.functions import pandas_udf, PandasUDFType schema = StructType([ StructField('id', IntegerType()), StructField('array', ArrayType(IntegerType())) ]) df = self.spark.createDataFrame( [[1, [1, 2, 3]]], schema=schema ) udf1 = pandas_udf( lambda pdf: pdf.assign(array=pdf.array * 2), schema, PandasUDFType.GROUPED_MAP ) result1 = df.groupby('id').apply(udf1).sort('id').toPandas() expected1 = df.toPandas().groupby('id').apply(udf1.func).reset_index(drop=True) self.assertPandasEqual(expected1, result1) ``` Here is output: ``` python/pyspark/sql/tests.py:244: in assertPandasEqual self.assertTrue(expected.equals(result), msg=msg) E AssertionError: DataFrames are not equal: E E Expected: E id array E 0 1 [1, 2, 3, 1, 2, 3] E id int32 E array object E dtype: object E E Result: E id array E 0 1 [2, 4, 6] E id int32 E array object E dtype: object ``` You can see that behavior of `array=pdf.array * 2` different for `result1` and `expected1`: ``` result1 = df.groupby('id').apply(udf1).sort('id').toPandas() >> [2, 4, 6] expected1 = df.toPandas().groupby('id').apply(udf1.func).reset_index(drop=True) >> [1, 2, 3, 1, 2, 3] ``` Default Python behavior is: ``` [1, 2, 3] * 2 >> [1, 2, 3, 1, 2, 3] ``` Thanks.
--- --------------------------------------------------------------------- To unsubscribe, e-mail: reviews-unsubscr...@spark.apache.org For additional commands, e-mail: reviews-h...@spark.apache.org