[jira] [Commented] (FLINK-7179) ProjectableTableSource interface doesn't compatible with BoundedOutOfOrdernessTimestampExtractor
[
https://issues.apache.org/jira/browse/FLINK-7179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16106389#comment-16106389
]
Fabian Hueske commented on FLINK-7179:
--
I think a {{ProjectableTableSource}} knows that is provides a virtual column
time indicator column. It should be able to identify that the field index
points to a virtual attribute.
So, I think it should be possible to handle this case correctly, but I agree
that this is not easy and requires knowledge of how things work internally.
We are currently discussing to change the handling and representation of time
indicators. In a nutshell, we are thinking of treating them as regular Row
fields and not as virtual columns anymore. See this
[thread|https://lists.apache.org/thread.html/735d55f9022df8ff73566a9f1553e14be94f8443986ad46559b35869@%3Cdev.flink.apache.org%3E]
for some background info.
I am not sure what we can do about the situation right now or if it is worth to
invest a lot of time into it if the time indicator representation is changed
anyway.
What do you think [~suez1224], [~hpeter]?
> ProjectableTableSource interface doesn't compatible with
> BoundedOutOfOrdernessTimestampExtractor
>
>
> Key: FLINK-7179
> URL: https://issues.apache.org/jira/browse/FLINK-7179
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.3.1
>Reporter: Zhenqiu Huang
>Assignee: Zhenqiu Huang
>
> In the implementation of window of stream sql,
> BoundedOutOfOrdernessTimestampExtractor is designed to extract row time from
> each row. It assumes the ts field is in the data stream by default. On the
> other hand, ProjectableTableSource is designed to help projection push down.
> If there is no row time related field in a query, the extractor can't
> function well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-7179) ProjectableTableSource interface doesn't compatible with BoundedOutOfOrdernessTimestampExtractor
[
https://issues.apache.org/jira/browse/FLINK-7179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16103533#comment-16103533
]
Shuyi Chen commented on FLINK-7179:
---
[~fhueske] I think the problem is in PushProjectIntoTableSourceScanRule in
streaming mode, when there is rowtime involved. An example logical plan is as
follows:
FlinkLogicalCalc.LOGICAL(input=rel#44:Subset#0.LOGICAL,expr#0..5={inputs},first=$t0,rowtime=$t5,id=$t1)
$t5 here refers to a virtual column of TimeIndicatorType, which does not exist
in the physical table directly. Therefore, it will generate a
ArrayIndexOutOfBoundException when ProjectableTableSource.projectFields is
called since there is only 5 columns in the physical table, and $t5 tries to
refer to the 6th column.
This problem only exist in Streaming. For batch, we can use a physical column
of SqlTimeType in the physical table in the window function, so it's fine.
What do you think how we can resolve it?
> ProjectableTableSource interface doesn't compatible with
> BoundedOutOfOrdernessTimestampExtractor
>
>
> Key: FLINK-7179
> URL: https://issues.apache.org/jira/browse/FLINK-7179
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.3.1
>Reporter: Zhenqiu Huang
>Assignee: Zhenqiu Huang
>
> In the implementation of window of stream sql,
> BoundedOutOfOrdernessTimestampExtractor is designed to extract row time from
> each row. It assumes the ts field is in the data stream by default. On the
> other hand, ProjectableTableSource is designed to help projection push down.
> If there is no row time related field in a query, the extractor can't
> function well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (FLINK-7179) ProjectableTableSource interface doesn't compatible with BoundedOutOfOrdernessTimestampExtractor
[
https://issues.apache.org/jira/browse/FLINK-7179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16087056#comment-16087056
]
Fabian Hueske commented on FLINK-7179:
--
Can you describe the issue in more detail?
The {{ProjectableTableSource}} receives the fields that it needs to emit.
A {{TableSource}} knows from which field the timestamps must be extracted can
read this field in addition to the selected fields.
Then it can apply the apply the {{TimestampExtractor}} and also possible remove
the timestamp field if it is not requested by the {{ProjectableTableSource}}.
This is a bit cumbersome, but it should be possible.
IMO, it makes sense to think about how this could be improved.
> ProjectableTableSource interface doesn't compatible with
> BoundedOutOfOrdernessTimestampExtractor
>
>
> Key: FLINK-7179
> URL: https://issues.apache.org/jira/browse/FLINK-7179
> Project: Flink
> Issue Type: Bug
> Components: Table API & SQL
>Affects Versions: 1.3.1
>Reporter: Zhenqiu Huang
>
> In the implementation of window of stream sql,
> BoundedOutOfOrdernessTimestampExtractor is designed to extract row time from
> each row. It assumes the ts field is in the data stream by default. On the
> other hand, ProjectableTableSource is designed to help projection push down.
> If there is no row time related field in a query, the extractor can't
> function well.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
