[
https://issues.apache.org/jira/browse/SPARK-52772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wenchen Fan resolved SPARK-52772.
---------------------------------
Fix Version/s: 4.1.0
Resolution: Fixed
Issue resolved by pull request 51466
[https://github.com/apache/spark/pull/51466]
> Inconsistent table attribute handling during updates
> ----------------------------------------------------
>
> Key: SPARK-52772
> URL: https://issues.apache.org/jira/browse/SPARK-52772
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: Kevin Beyer
> Assignee: Kevin Beyer
> Priority: Minor
> Labels: pull-request-available
> Fix For: 4.1.0
>
>
> TableOutputResolver is inconsistent in copying the table column metadata to
> the query attributes. For example, it does not copy for struct fields but
> does copy for primitives or when casting is required. Additionally, even
> when the table metadata is copied to the query attributes, rewrites sometimes
> undo the desired metadata and leak source table metadata that gets confused
> for target metadata. Another problem with leaking source metadata is if the
> CHAR_VARCHAR_TYPE_STRING_METADATA_KEY is propagated, then the write operation
> will remain unresolved, or worse flip from resolved to unresolved.
> As one example, JacksonGenerator#writeFields depends on the query attribute
> metadata to detect if a target column has a default value. It uses this
> information to decide if a null is written explicitly or if the field is not
> written to the json at all.
> Passing this table information via the source query attributes is not the
> best design. Ideally we should change all the writers to pick up table
> configuration from the table directly. However, there are many third-party
> connectors that may rely on this behavior. So for now, we will simply make
> the query output attributes match the target column's names and metadata in
> all cases.
> Getting the compiler to honor our required metadata is harder than it should
> be. Many rewrites are careless with the effects they have on resulting
> metadata. This is especially true when the metadata is empty. We need to
> fix removeRedundantAlias and trimNonTopLevelAliases to preserve empty
> metadata instead of changing to inherited metadata. Project.doCanonicalize
> also needs to consider the effective metadata instead of the alias
> inheritance instructions.
> While attempting to fix this issue, many things were attempted to get only
> the target table column metadata to flow, but many subtle bugs were
> encountered or discovered. Examples of things that get the target column
> metadata to flow initially, but then rewrites undo it (in
> org.apache.spark.sql.catalyst.analysis.TableOutputResolver.applyColumnMetadata):
> {code:java}
> expr match {
> // This does not work because if
> // expr = Alias(cast(attr, attr.dataType), attr.name)()
> // expr.metadata is empty, but after the no-op cast is removed by a rewrite
> // expr = Alias(attr, attr.name)()
> // Now expr.metadata propagates source metadata.
> // We don't want to rely on inherited metadata, so we only keep this
> pattern when the metadata is
> // not derived.
> case expr: NamedExpression if expr.name == attr.name && expr.metadata ==
> requiredMetadata =>
> expr
> // This doesn't should work because RemoveNoopOperators discards our
> requiredMetadata.
> // It's also risky to reuse the attribute with different state.
> // So we don't use this pattern.
> case a: Attribute =>
> a.withName(attr.name).withMetadata(requiredMetadata)
> // This doesn't work because RemoveRedundantAliases discards the alias if
> the
> // derived alias.metadata is empty, even though attr.metadata is not empty.
> // This is a silly way to request empty metadata anyway; we use the direct
> way below instead.
> case attr: Attribute if requiredMetadata.isEmpty =>
> Alias(attr, attr.name)(nonInheritableMetadataKeys = attr.metadata.keys)
> // This doesn't work because CleanupAliases discards
> explicitMetadata=Some(empty)
> // We fixed the rewrites to make this work.
> case expr =>
> Alias(expr, attr.name)(explicitMetadata = Some(requiredMetadata)) {code}
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
