Eric Marnadi created SPARK-56453:
------------------------------------
Summary: Open Add ResolveEventTimeWatermark to
HiveSessionStateBuilder
Key: SPARK-56453
URL: https://issues.apache.org/jira/browse/SPARK-56453
Project: Spark
Issue Type: Bug
Components: Structured Streaming
Affects Versions: 4.2.0
Reporter: Eric Marnadi
* {*}Add {{ResolveEventTimeWatermark}} to
{{HiveSessionStateBuilder.extendedResolutionRules}}{*}: SPARK-53687 introduced
the {{UnresolvedEventTimeWatermark}} / {{ResolveEventTimeWatermark}} pattern
for the SQL WATERMARK clause but only added the rule to
{{{}BaseSessionStateBuilder{}}}. {{HiveSessionStateBuilder}} — used in all
Databricks production environments (Hive catalog is the default) — was never
updated. Without the rule, {{UnresolvedEventTimeWatermark}} (which has {{output
= Nil}} per the {{UnresolvedNode}} contract) persists through analysis,
blocking column resolution when WATERMARK is used with an alias in JOIN
conditions.
* *Add {{WatermarkColumnResolutionSuite}}* in both {{sql/core}}
(BaseSessionStateBuilder) and {{sql/hive}} (HiveSessionStateBuilder) to cover
STREAM + WATERMARK + alias column resolution through the production analyzer.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
