[
https://issues.apache.org/jira/browse/SPARK-19256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300591#comment-17300591
]
Cheng Su commented on SPARK-19256:
----------------------------------
[~zsxwing] - sorry the design doc was years old, and I can create a short new
one if needed. Note here this Jira is supporting writing hive bucketed table,
but not reading. The reason is we don't want to introduce hash function change
to `HashPartitioning` right now, and would like to do this in a more general
way with data source v2.
So
> What happens if using an old Spark version to read this new format? Does it
>treat it as a non-bucketed table, or read it a a bucketed table but may return
>different results?
The read behavior is not changed. Spark will read the Hive/Trino bucketed table
as non-bucketed table. This has no problem for data correctness as bucketing is
just an optimization.
> How does Spark know which bucket format when reading an existing table, and
>how does it decide whether to use the old format or the new format to read and
>write?
For new format (Hive murmur3hash), Hive is adding a table property
"bucketing_version"=2 to metastore for this kind of new bucketed table, and
Trino followed that. In the future if we want to read old/new Hive bucketed
table with bucketing enabled, we can decide based on this property field.
For writing new format or old format, default behavior can be depending on Hive
version used during runtime for Spark query. For Hive 3.x.y, writing Hive
bucketed table with Hive murmur3hash, and for Hive 2.x.y and 1.x.y, writing
Hive bucketed table with Hive hivehash. We can also add a Hive config for user
to control this behavior if needed.
> Hive bucketing write support
> ----------------------------
>
> Key: SPARK-19256
> URL: https://issues.apache.org/jira/browse/SPARK-19256
> Project: Spark
> Issue Type: Umbrella
> Components: SQL
> Affects Versions: 2.1.0, 2.2.0, 2.3.0, 2.4.0, 3.0.0, 3.1.0
> Reporter: Tejas Patil
> Priority: Minor
>
> Update (2020 by Cheng Su):
> We use this JIRA to track progress for Hive bucketing write support in Spark.
> The goal is for Spark to write Hive bucketed table, to be compatible with
> other compute engines (Hive and Presto).
>
> Current status for Hive bucketed table in Spark:
> Not support for reading Hive bucketed table: read bucketed table as
> non-bucketed table.
> Wrong behavior for writing Hive ORC and Parquet bucketed table: write
> orc/parquet bucketed table as non-bucketed table (code path:
> InsertIntoHadoopFsRelationCommand -> FileFormatWriter).
> Do not allow for writing Hive non-ORC/Parquet bucketed table: throw exception
> by default if writing non-orc/parquet bucketed table (code path:
> InsertIntoHiveTable), and exception can be disabled by setting config
> `hive.enforce.bucketing`=false and `hive.enforce.sorting`=false, which will
> write as non-bucketed table.
>
> Current status for Hive bucketed table in Hive:
> Hive 3.0.0 and after: support writing bucketed table with Hive murmur3hash
> (https://issues.apache.org/jira/browse/HIVE-18910).
> Hive 1.x.y and 2.x.y: support writing bucketed table with Hive hivehash.
> Hive on Tez: support zero and multiple files per bucket
> (https://issues.apache.org/jira/browse/HIVE-14014). And more code pointer on
> read path -
> [https://github.com/apache/hive/blob/master/ql/src/java/org/apache/hadoop/hive/ql/optimizer/metainfo/annotation/OpTraitsRulesProcFactory.java#L183-L212]
> .
>
> Current status for Hive bucketed table in Presto (take presto-sql here):
> Support writing bucketed table with Hive murmur3hash and hivehash
> ([https://github.com/prestosql/presto/pull/1697]).
> Support zero and multiple files per bucket
> ([https://github.com/prestosql/presto/pull/822]).
>
> TLDR is to achieve Hive bucketed table compatibility across Spark, Presto and
> Hive. Here with this JIRA, we need to add support writing Hive bucketed table
> with Hive murmur3hash (for Hive 3.x.y) and hivehash (for Hive 1.x.y and
> 2.x.y).
>
> To allow Spark efficiently read Hive bucketed table, this needs more radical
> change and we decide to wait until data source v2 supports bucketing, and do
> the read path on data source v2. Read path will not covered by this JIRA.
>
> Original description (2017 by Tejas Patil):
> JIRA to track design discussions and tasks related to Hive bucketing support
> in Spark.
> Proposal :
> [https://docs.google.com/document/d/1a8IDh23RAkrkg9YYAeO51F4aGO8-xAlupKwdshve2fc/edit?usp=sharing]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
