[ https://issues.apache.org/jira/browse/SPARK-42024?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ruifeng Zheng resolved SPARK-42024. ----------------------------------- Resolution: Resolved > createDataFrame should corse types of string float to float > ----------------------------------------------------------- > > Key: SPARK-42024 > URL: https://issues.apache.org/jira/browse/SPARK-42024 > Project: Spark > Issue Type: Sub-task > Components: Connect > Affects Versions: 3.4.0 > Reporter: Hyukjin Kwon > Priority: Major > > {code} > pyspark/sql/tests/test_types.py:245 > (TypesParityTests.test_infer_schema_upcast_float_to_string) > self = <pyspark.sql.tests.connect.test_parity_types.TypesParityTests > testMethod=test_infer_schema_upcast_float_to_string> > def test_infer_schema_upcast_float_to_string(self): > > df = self.spark.createDataFrame([[1.33, 1], ["2.1", 1]], schema=["a", > > "b"]) > ../test_types.py:247: > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ > ../../connect/session.py:282: in createDataFrame > _table = pa.Table.from_pylist([dict(zip(_cols, list(item))) for item in > _data]) > pyarrow/table.pxi:3700: in pyarrow.lib.Table.from_pylist > ??? > pyarrow/table.pxi:5221: in pyarrow.lib._from_pylist > ??? > pyarrow/table.pxi:3575: in pyarrow.lib.Table.from_arrays > ??? > pyarrow/table.pxi:1383: in pyarrow.lib._sanitize_arrays > ??? > pyarrow/table.pxi:1364: in pyarrow.lib._schema_from_arrays > ??? > pyarrow/array.pxi:320: in pyarrow.lib.array > ??? > pyarrow/array.pxi:39: in pyarrow.lib._sequence_to_array > ??? > pyarrow/error.pxi:144: in pyarrow.lib.pyarrow_internal_check_status > ??? > _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ > _ > > ??? > E pyarrow.lib.ArrowInvalid: Could not convert '2.1' with type str: tried to > convert to double > pyarrow/error.pxi:100: ArrowInvalid > {code} -- This message was sent by Atlassian Jira (v8.20.10#820010) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org