Spenserrrr commented on code in PR #57939: URL: https://github.com/apache/spark/pull/57939#discussion_r3765116805
########## python/pyspark/tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py: ########## @@ -0,0 +1,340 @@ +# +# Licensed to the Apache Software Foundation (ASF) under one or more +# contributor license agreements. See the NOTICE file distributed with +# this work for additional information regarding copyright ownership. +# The ASF licenses this file to You under the Apache License, Version 2.0 +# (the "License"); you may not use this file except in compliance with +# the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# + +""" +Tests for PyArrow Table.to_pandas() using golden file comparison. + +Unlike Array/ChunkedArray.to_pandas() (which returns a Series and is covered by +test_pyarrow_arrow_to_pandas_{default,non_default}.py), Table.to_pandas() returns a +DataFrame. Its per-column conversion matches the Array tests, so this file pins the +genuinely Table-specific behavior instead: multi-column DataFrame assembly and the +empty-table edges (0 columns / 0 rows). Spark calls Table.to_pandas() at +python/pyspark/sql/pandas/conversion.py:255 (the 0-column path) and, in Spark Connect, +at python/pyspark/sql/connect/client/core.py:1423 (a bare whole-Table conversion). + +## Golden File Cell Format + +Each cell uses the value@type format: +- pyarrow Table: "{col: [val1, val2, None], ...}@Table[name: type, ...]" +- pandas DataFrame: "{col: [values], ...}@Dataframe[name dtype, ...]" +- Error: "ERR@ExceptionClassName" + +Values are formatted via tolist() for stable, Python-native representation. + +## Regenerating Golden Files + +Set SPARK_GENERATE_GOLDEN_FILES=1 before running: + + SPARK_GENERATE_GOLDEN_FILES=1 python -m pytest \\ + python/pyspark/tests/upstream/pyarrow/test_pyarrow_table_to_pandas.py +""" + +import datetime +import unittest + +from pyspark.loose_version import LooseVersion +from pyspark.testing.utils import ( + have_pyarrow, + have_pandas, + pyarrow_requirement_message, + pandas_requirement_message, +) +from pyspark.testing.goldenutils import GoldenFileTestMixin + +if have_pandas: + import pandas as pd +if have_pyarrow: + import pyarrow as pa + + +class _PyArrowTableToPandasTestBase(GoldenFileTestMixin, unittest.TestCase): + """ + Shared machinery for pa.Table.to_pandas() golden file tests. + + Holds the conversion helper and the source-table inventory, split into group + methods that these and the (temporal-flag) tests reuse. Defines no ``test_*`` of + its own. + """ + + @staticmethod + def _repr_dataframe(df) -> str: + """ + Format a pandas DataFrame result as a golden-file cell: per-column tolist() + for a stable, Python-native representation, mirroring goldenutils' + ``repr_pandas_series_value`` for the Series case. + + This deliberately does NOT go through ``repr_value``/``repr_pandas_value`` + (which use ``to_json``). With the default date_as_object=True a far-future + date (year 9999) comes back as an object column of Python ``datetime.date`` + objects, and ``DataFrame.to_json`` overflows converting each to an epoch + nanosecond int (OverflowError); tolist() returns the objects as-is, so + nothing overflows. + """ + body = str({name: col.tolist() for name, col in df.items()}).replace("\n", " ") + schema = ", ".join(f"{t} {d.name}" for t, d in df.dtypes.items()) + return f"{body}@Dataframe[{schema}]" Review Comment: Thanks for the advice @Yicong-Huang! I'd like to do this in another PR, and I want to explain the reason behind it. The reason why I use a local helper instead of modifying the shared `repr_pandas_value` is that it is already in use by another file. `repr_value` dispatches DataFrame to `repr_pandas_value`, and in `test_pandas_udf_return_type`, it runs a DataFrame through `repr_value` to build a golden column header. Thus, I added a local `tolist()` helper here. I actually tried changing r`epr_pandas_value` first, and it broke test_pandas_udf_return_type's goldens. Also, the reason a different formatter is needed at all is that to_json() isn't safe for the temporal cases these Table tests cover. It raises OverflowError on out-of-nanosecond-range dates (the year-9999 rows, which are object-dtype datetime.date under the default date_as_object=True), and on pandas 2 it also misreads non-ns datetime64 units. tolist() avoids both. To promote it into GoldenFileTestMixin cleanly, I plan to fix `repr_pandas_value`, regenerate test_pandas_udf_return_type's goldens, and switch these Table tests to call repr_value directly. Does this sound reasonable, or do you have any suggestions on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
