Yaoxuan Wu created SPARK-61763:
----------------------------------
Summary: ORC and Hive SerDe writes keep only one entry of a MAP
with DATE keys (DaysWritable does not override compareTo / equals / hashCode)
Key: SPARK-61763
URL: https://issues.apache.org/jira/browse/SPARK-61763
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.1, 4.2.0, 4.0.0
Reporter: Yaoxuan Wu
Writing a MAP whose keys are DATEs to ORC, or into a Hive SerDe table, keeps
only one entry per map. No error is raised; the other entries are silently lost
on write.
{code:python}
import tempfile
from pyspark.sql import SparkSession
spark = SparkSession.builder.enableHiveSupport().getOrCreate()
q = "SELECT map(DATE'2003-01-13', 6, DATE'2011-07-19', -4) AS m"
p = tempfile.mkdtemp()
spark.sql(q).write.mode("overwrite").orc(p)
print(spark.read.orc(p).collect())
# expected: [Row(m={datetime.date(2003, 1, 13): 6, datetime.date(2011, 7, 19):
-4})]
# actual: [Row(m={datetime.date(2003, 1, 13): -4})]
spark.sql("SET spark.sql.hive.convertMetastoreParquet=false")
spark.sql("SET spark.sql.hive.convertInsertingUnpartitionedTable=false")
spark.sql("CREATE TABLE t (m MAP<DATE, INT>) STORED AS PARQUET")
spark.sql(f"INSERT INTO t {q}")
print(spark.table("t").collect())
# actual: [Row(m={datetime.date(2003, 1, 13): -4})]
{code}
Same result on 3.5.8, 4.0.0, 4.1.1 and 4.2.0:
* Native ORC data source: the written file holds a single entry (pyarrow reads
the same thing), so the reader settings don't matter.
* Hive SerDe tables STORED AS ORC, PARQUET and TEXTFILE: same.
* Native Parquet is correct. DATE map values, TIMESTAMP map keys and DATE
arrays are correct too.
* The surviving entry is the first key with the last value.
Cause (master d3f4a73a):
1. Dates are written as {{DaysWritable}}
(sql/core/.../datasources/DaysWritable.scala), a subclass of Hive's
{{DateWritable}}. It keeps its own {{gregorianDays}} / {{julianDays}} fields
and overrides {{getDays}}, {{get}}, {{set}}, {{write}} and {{readFields}}.
2. It overrides neither {{compareTo}}, {{equals}} nor {{hashCode}}. In
hive-storage-api 2.8.1 these read the private field
{{DateWritable.daysSinceEpoch}}, which {{DaysWritable}} never sets, so it is
always 0. As a result, all {{DaysWritable}} instances compare equal.
3. {{OrcSerializer}} (MapType case) converts each key with {{reuseObj = false}}
and calls {{put}} on an {{OrcMap}}, which is a {{TreeMap}}. Every key compares
equal to the first one, so each put only replaces that entry's value. The Hive
SerDe path ({{HiveInspectors}}, which builds a {{java.util.HashMap}} of
writables) collapses the same way through {{hashCode}} / {{equals}}.
SPARK-32602 was the same missing-field problem for {{toString}} (dates written
to Hive tables as 1970-01-01), and {{get(boolean)}} was overridden for it. The
comparison methods were left out. A possible fix is to override {{compareTo}},
{{equals}} and {{hashCode}} in {{DaysWritable}} based on {{julianDays}} (the
value that is written), or to keep the parent's field in sync in {{set}} and
the constructors.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]