Yaoxuan Wu created SPARK-61763:
----------------------------------

             Summary: ORC and Hive SerDe writes keep only one entry of a MAP 
with DATE keys (DaysWritable does not override compareTo / equals / hashCode)
                 Key: SPARK-61763
                 URL: https://issues.apache.org/jira/browse/SPARK-61763
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 4.1.1, 4.2.0, 4.0.0
            Reporter: Yaoxuan Wu


Writing a MAP whose keys are DATEs to ORC, or into a Hive SerDe table, keeps 
only one entry per map. No error is raised; the other entries are silently lost 
on write.

{code:python}
import tempfile
from pyspark.sql import SparkSession

spark = SparkSession.builder.enableHiveSupport().getOrCreate()
q = "SELECT map(DATE'2003-01-13', 6, DATE'2011-07-19', -4) AS m"

p = tempfile.mkdtemp()
spark.sql(q).write.mode("overwrite").orc(p)
print(spark.read.orc(p).collect())
# expected: [Row(m={datetime.date(2003, 1, 13): 6, datetime.date(2011, 7, 19): 
-4})]
# actual:   [Row(m={datetime.date(2003, 1, 13): -4})]

spark.sql("SET spark.sql.hive.convertMetastoreParquet=false")
spark.sql("SET spark.sql.hive.convertInsertingUnpartitionedTable=false")
spark.sql("CREATE TABLE t (m MAP<DATE, INT>) STORED AS PARQUET")
spark.sql(f"INSERT INTO t {q}")
print(spark.table("t").collect())
# actual:   [Row(m={datetime.date(2003, 1, 13): -4})]
{code}

Same result on 3.5.8, 4.0.0, 4.1.1 and 4.2.0:
* Native ORC data source: the written file holds a single entry (pyarrow reads 
the same thing), so the reader settings don't matter.
* Hive SerDe tables STORED AS ORC, PARQUET and TEXTFILE: same.
* Native Parquet is correct. DATE map values, TIMESTAMP map keys and DATE 
arrays are correct too.
* The surviving entry is the first key with the last value.

Cause (master d3f4a73a):
1. Dates are written as {{DaysWritable}} 
(sql/core/.../datasources/DaysWritable.scala), a subclass of Hive's 
{{DateWritable}}. It keeps its own {{gregorianDays}} / {{julianDays}} fields 
and overrides {{getDays}}, {{get}}, {{set}}, {{write}} and {{readFields}}.
2. It overrides neither {{compareTo}}, {{equals}} nor {{hashCode}}. In 
hive-storage-api 2.8.1 these read the private field 
{{DateWritable.daysSinceEpoch}}, which {{DaysWritable}} never sets, so it is 
always 0. As a result, all {{DaysWritable}} instances compare equal.
3. {{OrcSerializer}} (MapType case) converts each key with {{reuseObj = false}} 
and calls {{put}} on an {{OrcMap}}, which is a {{TreeMap}}. Every key compares 
equal to the first one, so each put only replaces that entry's value. The Hive 
SerDe path ({{HiveInspectors}}, which builds a {{java.util.HashMap}} of 
writables) collapses the same way through {{hashCode}} / {{equals}}.

SPARK-32602 was the same missing-field problem for {{toString}} (dates written 
to Hive tables as 1970-01-01), and {{get(boolean)}} was overridden for it. The 
comparison methods were left out. A possible fix is to override {{compareTo}}, 
{{equals}} and {{hashCode}} in {{DaysWritable}} based on {{julianDays}} (the 
value that is written), or to keep the parent's field in sync in {{set}} and 
the constructors.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to