Fabian Hueske created FLINK-40541:
-------------------------------------
Summary: LATERAL SNAPSHOT join rejects a valid build-side
watermark when it is declared on a column hidden by the column-expansion
strategy
Key: FLINK-40541
URL: https://issues.apache.org/jira/browse/FLINK-40541
Project: Flink
Issue Type: Bug
Components: Table SQL / Planner
Reporter: Fabian Hueske
Assignee: Fabian Hueske
*Description*
The {{LATERAL SNAPSHOT}} join requires the build-side input to declare exactly
one watermark (row-time attribute), which drives the operator's LOAD phase. The
planner detects this watermark by counting row-time indicator fields in the row
type of the projected {{TABLE}} argument.
When the watermark is declared on a column that the configured column-expansion
strategy hides from {{SELECT *}} — e.g. a {{VIRTUAL}} metadata column combined
with {{TableConfigOptions.TABLE_COLUMN_EXPANSION_STRATEGY =
EXCLUDE_DEFAULT_VIRTUAL_METADATA_COLUMNS}} — the {{TABLE t}} argument expands
to a projection that drops the row-time attribute. The row-time count is then
{{0}}, and the query is wrongly rejected with:
{code}
org.apache.flink.table.api.ValidationException: LATERAL SNAPSHOT requires a
watermark on the build-side input.
{code}
even though the table does declare a watermark.
*How to reproduce*
{code:sql}
SET 'table.column-expansion-strategy' =
'EXCLUDE_DEFAULT_VIRTUAL_METADATA_COLUMNS';
CREATE TABLE build_side (
bk STRING,
bv INT,
rt TIMESTAMP_LTZ(3) METADATA VIRTUAL,
WATERMARK FOR rt AS rt
) WITH (...);
SELECT * FROM probe
JOIN LATERAL SNAPSHOT(
input => TABLE build_side,
load_completed_condition => 'user_time',
load_completed_time => CAST(TIMESTAMP '2026-07-01 00:00:00' AS
TIMESTAMP_LTZ(3))
) AS s ON probe.pk = s.bk;
{code}
Explicitly re-projecting the hidden column (e.g. {{SELECT *, rt FROM
build_side}} in a preceding CTE/view) works around the problem, confirming that
the watermark itself is present and only the visibility of the row-time column
is at fault.
*Proposed fix*
Detect the build-side row-time attribute from the watermark below any
projection that hides it, rather than from a visible time-attribute column, and
re-append the row-time attribute from below the hiding projection so it reaches
the operator. Run the watermark validation on the rewritten build side instead
of the raw {{TABLE}} argument. The existing top-level {{Calc}} wrapper projects
the re-appended column away, so the user-visible output is unchanged.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
