[jira] [Updated] (FLINK-19881) Optimize temporal join with upsert-Source(upsert-kafka)
[ https://issues.apache.org/jira/browse/FLINK-19881?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-19881: --- Parent: (was: FLINK-16824) Issue Type: Improvement (was: Sub-task) > Optimize temporal join with upsert-Source(upsert-kafka) > --- > > Key: FLINK-19881 > URL: https://issues.apache.org/jira/browse/FLINK-19881 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Runtime >Reporter: Leonard Xu >Priority: Major > Fix For: 1.14.0 > > > Currently upsert-kafka will do normalize in a physical node named > `ChangelogNormalize`, the normalization will do a deduplicate using state and > produce `UPDATE_AFTER`, `DELETE` changelog. We do same thing In the state of > temporal join operator, we can merge them to one as an optimization if the > query contains temporal join an upsert-kafka. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16824) FLIP-132 Temporal Table DDL and Temporal Table Join
[ https://issues.apache.org/jira/browse/FLINK-16824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17327084#comment-17327084 ] Leonard Xu commented on FLINK-16824: Yes, I'll close this. Thanks [~knaufk] > FLIP-132 Temporal Table DDL and Temporal Table Join > --- > > Key: FLINK-16824 > URL: https://issues.apache.org/jira/browse/FLINK-16824 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Reporter: Konstantin Knauf >Assignee: Leonard Xu >Priority: Major > > Currently, a Temporal Table Function can only be created via the Table API or > indirectly via the configuration file of the SQL Client. > It would be great, if this was also possible in pure SQL via a DDL statement. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22356) Filesystem/Hive partition file is not committed when watermark is applied on rowtime of TIMESTAMP_LTZ type
[
https://issues.apache.org/jira/browse/FLINK-22356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17326572#comment-17326572
]
Leonard Xu commented on FLINK-22356:
I create an initial PR based on above comment, I'll add tests if we can reach
an agreement or we have another idea to fix this issue.
> Filesystem/Hive partition file is not committed when watermark is applied on
> rowtime of TIMESTAMP_LTZ type
> --
>
> Key: FLINK-22356
> URL: https://issues.apache.org/jira/browse/FLINK-22356
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hive, Table SQL /
> API
>Reporter: Jark Wu
>Assignee: Leonard Xu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.13.0, 1.14.0
>
>
> {code:sql}
> set execution.checkpointing.interval = 10s;
> set table.local-time-zone = 'Asia/Shanghai';
> create table mysource (
> ms bigint,
> ts as to_timestamp_ltz(ms, 3),
> watermark for ts as ts - interval '0.001' second
> ) with (
> 'connector' = 'socket',
> 'format' = 'json',
> 'hostname' = '127.0.0.1',
> 'port' = ''
> );
> CREATE TABLE fs_table2 (
> ms bigint,
> dt STRING,
> `hour` STRING,
> `mm` string
> ) PARTITIONED BY (dt, `hour`, `mm`) WITH (
> 'connector'='filesystem',
> 'path'='/Users/wuchong/Downloads/fs_table2',
> 'format'='csv',
> 'sink.partition-commit.delay'='1min',
> 'sink.partition-commit.policy.kind'='success-file',
> 'sink.rolling-policy.rollover-interval' = '30s',
> 'sink.rolling-policy.check-interval' = '30s',
> 'sink.partition-commit.trigger'='partition-time',
> 'partition.time-extractor.timestamp-pattern' = '$dt $hour:$mm:00'
> );
> insert into fs_table2
> SELECT ms,
> DATE_FORMAT(ts, '-MM-dd'), DATE_FORMAT(ts, 'HH'), DATE_FORMAT(ts, 'mm')
> FROM mysource;
> {code}
> Enther some data in socket:
> {code}
> > nc -lk
> {"ms": 161883960}
> {"ms": 1618839600123}
> {"ms": 1618839600456}
> {"ms": 1618839600789}
> {"ms": 161883966}
> {"ms": 1618839660123}
> {"ms": 1618839660456}
> {"ms": 1618839660789}
> {"ms": 161883972}
> {"ms": 161883978}
> {"ms": 161883984}
> {"ms": 161883990}
> {"ms": 161883996}
> {"ms": 161884002}
> {code}
> However, all the files are not committed (not {{_SUCCESS}} file):
> {code}
> ➜ hour=21 tree
> .
> ├── mm=40
> │ └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-0
> ├── mm=41
> │ └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-1
> ├── mm=42
> │ └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-2
> ├── mm=43
> │ └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-3
> ├── mm=44
> │ └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-4
> ├── mm=45
> │ └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-5
> ├── mm=46
> │ └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-6
> └── mm=47
> └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-7
> 8 directories, 8 files
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22356) Filesystem/Hive partition file is not committed when watermark is applied on rowtime of TIMESTAMP_LTZ type
[
https://issues.apache.org/jira/browse/FLINK-22356?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17326561#comment-17326561
]
Leonard Xu commented on FLINK-22356:
The root cause of this issue is the the extracted partition time is a TIMESTAMP
literal from hive partition field, the `PartitionTimeCommitter` use watermark
to compare the TIMESTAMP value, if the input rowtime field of TableSink is
TIMESTAMP_LTZ, we need to shift the TIMESTAMP literal and then compare with
watermark.
But currently the rowtime field Datatype information is unreachable for
TableSink, what I can image is introducing a method in DynamicTableSink.Context
{code:java}
public interface DynamicTableSink {
ChangelogMode getChangelogMode(ChangelogMode requestedMode);
SinkRuntimeProvider getSinkRuntimeProvider(Context context);
DynamicTableSink copy();
interface Context {
boolean isBounded();
/**
* Returns the DataType of the input time attribute field.
*
* This information can be used to compare a table partition value
with watermark in
* partition commit scenario.
*/
Optional getTimeAttributeDataType();
{code}
Do you have any suggestion? [~twalthr] [~jark]
> Filesystem/Hive partition file is not committed when watermark is applied on
> rowtime of TIMESTAMP_LTZ type
> --
>
> Key: FLINK-22356
> URL: https://issues.apache.org/jira/browse/FLINK-22356
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem, Connectors / Hive, Table SQL /
> API
>Reporter: Jark Wu
>Assignee: Leonard Xu
>Priority: Critical
> Fix For: 1.13.0, 1.14.0
>
>
> {code:sql}
> set execution.checkpointing.interval = 10s;
> set table.local-time-zone = 'Asia/Shanghai';
> create table mysource (
> ms bigint,
> ts as to_timestamp_ltz(ms, 3),
> watermark for ts as ts - interval '0.001' second
> ) with (
> 'connector' = 'socket',
> 'format' = 'json',
> 'hostname' = '127.0.0.1',
> 'port' = ''
> );
> CREATE TABLE fs_table2 (
> ms bigint,
> dt STRING,
> `hour` STRING,
> `mm` string
> ) PARTITIONED BY (dt, `hour`, `mm`) WITH (
> 'connector'='filesystem',
> 'path'='/Users/wuchong/Downloads/fs_table2',
> 'format'='csv',
> 'sink.partition-commit.delay'='1min',
> 'sink.partition-commit.policy.kind'='success-file',
> 'sink.rolling-policy.rollover-interval' = '30s',
> 'sink.rolling-policy.check-interval' = '30s',
> 'sink.partition-commit.trigger'='partition-time',
> 'partition.time-extractor.timestamp-pattern' = '$dt $hour:$mm:00'
> );
> insert into fs_table2
> SELECT ms,
> DATE_FORMAT(ts, '-MM-dd'), DATE_FORMAT(ts, 'HH'), DATE_FORMAT(ts, 'mm')
> FROM mysource;
> {code}
> Enther some data in socket:
> {code}
> > nc -lk
> {"ms": 161883960}
> {"ms": 1618839600123}
> {"ms": 1618839600456}
> {"ms": 1618839600789}
> {"ms": 161883966}
> {"ms": 1618839660123}
> {"ms": 1618839660456}
> {"ms": 1618839660789}
> {"ms": 161883972}
> {"ms": 161883978}
> {"ms": 161883984}
> {"ms": 161883990}
> {"ms": 161883996}
> {"ms": 161884002}
> {code}
> However, all the files are not committed (not {{_SUCCESS}} file):
> {code}
> ➜ hour=21 tree
> .
> ├── mm=40
> │ └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-0
> ├── mm=41
> │ └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-1
> ├── mm=42
> │ └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-2
> ├── mm=43
> │ └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-3
> ├── mm=44
> │ └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-4
> ├── mm=45
> │ └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-5
> ├── mm=46
> │ └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-6
> └── mm=47
> └── part-cf06c6da-d301-4623-832c-9e0f356f6fb4-0-7
> 8 directories, 8 files
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22122) StreamingKafkaITCase Fail due to TestTimedOutException
[
https://issues.apache.org/jira/browse/FLINK-22122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17326208#comment-17326208
]
Leonard Xu commented on FLINK-22122:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16871=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529=27500
> StreamingKafkaITCase Fail due to TestTimedOutException
> --
>
> Key: FLINK-22122
> URL: https://issues.apache.org/jira/browse/FLINK-22122
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.13.0
>Reporter: Guowei Ma
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16059=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529=27391
> {code:java}
> Apr 06 08:34:01 org.junit.runners.model.TestTimedOutException: test timed out
> after 3 minutes
> Apr 06 08:34:01 at java.lang.Object.wait(Native Method)
> Apr 06 08:34:01 at java.lang.Object.wait(Object.java:502)
> Apr 06 08:34:01 at java.lang.UNIXProcess.waitFor(UNIXProcess.java:395)
> Apr 06 08:34:01 at
> org.apache.flink.tests.util.flink.FlinkDistribution.submitJob(FlinkDistribution.java:194)
> Apr 06 08:34:01 at
> org.apache.flink.tests.util.flink.LocalStandaloneFlinkResource$StandaloneClusterController.submitJob(LocalStandaloneFlinkResource.java:200)
> Apr 06 08:34:01 at
> org.apache.flink.tests.util.kafka.StreamingKafkaITCase.testKafka(StreamingKafkaITCase.java:109)
> Apr 06 08:34:01 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Apr 06 08:34:01 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Apr 06 08:34:01 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Apr 06 08:34:01 at java.lang.reflect.Method.invoke(Method.java:498)
> Apr 06 08:34:01 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> Apr 06 08:34:01 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Apr 06 08:34:01 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> Apr 06 08:34:01 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Apr 06 08:34:01 at
> org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:48)
> Apr 06 08:34:01 at
> org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:48)
> Apr 06 08:34:01 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> Apr 06 08:34:01 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> Apr 06 08:34:01 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Apr 06 08:34:01 at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22378) Type mismatch when declaring SOURCE_WATERMARK on TIMESTAMP_LTZ column
[
https://issues.apache.org/jira/browse/FLINK-22378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325909#comment-17325909
]
Leonard Xu commented on FLINK-22378:
I think the return type of SOURCE_WATERMARK() should keep same with the input
time field, this can ensure the consistency. IIRC, this function is plan to
used in source like Hive, maybe we need to make the function to accept one arg
of time attribute field, I think we need some discussion here.
> Type mismatch when declaring SOURCE_WATERMARK on TIMESTAMP_LTZ column
> -
>
> Key: FLINK-22378
> URL: https://issues.apache.org/jira/browse/FLINK-22378
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Timo Walther
>Priority: Major
>
> The following schema cannot be resolved currently:
> {code}
> Schema.newBuilder()
> .columnByMetadata("rowtime", DataTypes.TIMESTAMP_LTZ(3))
> .watermark("rowtime", "SOURCE_WATERMARK()")
> .build()
> {code}
> It leads to:
> {code}
> The watermark output type TIMESTAMP(3) is different from input time filed
> type TIMESTAMP_LTZ(3).
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22354) Failed to define watermark on computed column of CURRENT_TIMESTAMP and LOCALTIMESTAMP
[
https://issues.apache.org/jira/browse/FLINK-22354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325900#comment-17325900
]
Leonard Xu commented on FLINK-22354:
Thanks [~twalthr] , I'll open the PR to fix current behavior ASSP.
> Failed to define watermark on computed column of CURRENT_TIMESTAMP and
> LOCALTIMESTAMP
> -
>
> Key: FLINK-22354
> URL: https://issues.apache.org/jira/browse/FLINK-22354
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Jark Wu
>Assignee: Leonard Xu
>Priority: Blocker
> Fix For: 1.13.0, 1.14.0
>
>
> It is very common to define watermarks on {{localtimestamp}} and
> {{current_tiemstamp}} to support *ingestion time*. However, the following
> DDLs failed in v1.13. I also tested the following DDLs can pass in v1.12.1,
> so this is a regression.
> The root cause may be introduced by FLINK-21435 which adds a [strict check to
> only allow precision =
> 3|https://github.com/apache/flink/blob/f51168041512b0473decabb2088c1ff4fa4f34bc/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/DefaultSchemaResolver.java#L225]
> (precision = 0 should also can be time attribute), however the precision of
> return type of {{current_timestamp}} and {{localtimestamp}} are 0 (another
> long-lived bug). We should fix them both.
> {code:sql}
> Flink SQL> create table datagen_source (
> > order_number BIGINT,
> > priceint,
> > buyerstring,
> > ts as current_timestamp,
> > proctime as proctime(),
> > watermark for ts as ts - interval '0.001' second
> > ) with (
> > 'connector' = 'datagen',
> > 'rows-per-second' = '1'
> > );
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.api.ValidationException: Invalid data type of time
> field for watermark definition. The field must be of type TIMESTAMP(3) or
> TIMESTAMP_LTZ(3).
> Flink SQL> create table datagen_source (
> > order_number BIGINT,
> > priceint,
> > buyerstring,
> > ts as cast(current_timestamp as timestamp_ltz(3)),
> > proctime as proctime(),
> > watermark for ts as ts - interval '0.001' second
> > ) with (
> > 'connector' = 'datagen',
> > 'rows-per-second' = '1'
> > );
> [INFO] Execute statement succeed.
> Flink SQL> create table datagen_source2 (
> > order_number BIGINT,
> > priceint,
> > buyerstring,
> > ts as localtimestamp,
> > proctime as proctime(),
> > watermark for ts as ts - interval '0.001' second
> > ) with (
> > 'connector' = 'datagen',
> > 'rows-per-second' = '1'
> > );
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.api.ValidationException: Invalid data type of time
> field for watermark definition. The field must be of type TIMESTAMP(3) or
> TIMESTAMP_LTZ(3).
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-22354) Failed to define watermark on computed column of CURRENT_TIMESTAMP and LOCALTIMESTAMP
[
https://issues.apache.org/jira/browse/FLINK-22354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325835#comment-17325835
]
Leonard Xu edited comment on FLINK-22354 at 4/20/21, 2:31 PM:
--
I used the new function stack and its very useful to add some new functions,
but for this case maybe it's not a good solution, calcite only supports the
function `LOCALTIMETIMESTAMP` and doesn't support `LOCALTIMESTAMP()`, and the
new function stack doesn't supports `CURRENT_ROW_TIMESTAMP` style function
that's why I finally work around with `CURRENT_ROW_TIMESTAMP()`(see
https://issues.apache.org/jira/browse/CALCITE-4545). The configurable time
function should be achieved, but I think it isn't current time point, and this
should be a new feature? So I tend to just fix the wrong behavior in this issue
and introduce configurable time function in the future. WDYT? [~twalthr]
was (Author: leonard xu):
I used the new function stack and its very useful to add some new functions,
but for this case maybe it's not a good solution, calcite only supports the
function `LOCALTIMETIMESTAMP` and doesn't supports `LOCALTIMESTAMP()`, and the
new function stack doesn't supports `CURRENT_ROW_TIMESTAMP` style function
that's why I finally work around with `CURRENT_ROW_TIMESTAMP()`(see
https://issues.apache.org/jira/browse/CALCITE-4545). The configurable time
function should be achieved, but I think it isn't current time point, and this
should be a new feature? So I tend to just fix the wrong behavior in this issue
and introduce configurable time function in the future. WDYT? [~twalthr]
> Failed to define watermark on computed column of CURRENT_TIMESTAMP and
> LOCALTIMESTAMP
> -
>
> Key: FLINK-22354
> URL: https://issues.apache.org/jira/browse/FLINK-22354
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Jark Wu
>Assignee: Leonard Xu
>Priority: Blocker
> Fix For: 1.13.0, 1.14.0
>
>
> It is very common to define watermarks on {{localtimestamp}} and
> {{current_tiemstamp}} to support *ingestion time*. However, the following
> DDLs failed in v1.13. I also tested the following DDLs can pass in v1.12.1,
> so this is a regression.
> The root cause may be introduced by FLINK-21435 which adds a [strict check to
> only allow precision =
> 3|https://github.com/apache/flink/blob/f51168041512b0473decabb2088c1ff4fa4f34bc/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/DefaultSchemaResolver.java#L225]
> (precision = 0 should also can be time attribute), however the precision of
> return type of {{current_timestamp}} and {{localtimestamp}} are 0 (another
> long-lived bug). We should fix them both.
> {code:sql}
> Flink SQL> create table datagen_source (
> > order_number BIGINT,
> > priceint,
> > buyerstring,
> > ts as current_timestamp,
> > proctime as proctime(),
> > watermark for ts as ts - interval '0.001' second
> > ) with (
> > 'connector' = 'datagen',
> > 'rows-per-second' = '1'
> > );
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.api.ValidationException: Invalid data type of time
> field for watermark definition. The field must be of type TIMESTAMP(3) or
> TIMESTAMP_LTZ(3).
> Flink SQL> create table datagen_source (
> > order_number BIGINT,
> > priceint,
> > buyerstring,
> > ts as cast(current_timestamp as timestamp_ltz(3)),
> > proctime as proctime(),
> > watermark for ts as ts - interval '0.001' second
> > ) with (
> > 'connector' = 'datagen',
> > 'rows-per-second' = '1'
> > );
> [INFO] Execute statement succeed.
> Flink SQL> create table datagen_source2 (
> > order_number BIGINT,
> > priceint,
> > buyerstring,
> > ts as localtimestamp,
> > proctime as proctime(),
> > watermark for ts as ts - interval '0.001' second
> > ) with (
> > 'connector' = 'datagen',
> > 'rows-per-second' = '1'
> > );
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.api.ValidationException: Invalid data type of time
> field for watermark definition. The field must be of type TIMESTAMP(3) or
> TIMESTAMP_LTZ(3).
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22354) Failed to define watermark on computed column of CURRENT_TIMESTAMP and LOCALTIMESTAMP
[
https://issues.apache.org/jira/browse/FLINK-22354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325835#comment-17325835
]
Leonard Xu commented on FLINK-22354:
I used the new function stack and its very useful to add some new functions,
but for this case maybe it's not a good solution, calcite only supports the
function `LOCALTIMETIMESTAMP` and doesn't supports `LOCALTIMESTAMP()`, and the
new function stack doesn't supports `CURRENT_ROW_TIMESTAMP` style function
that's why I finally work around with `CURRENT_ROW_TIMESTAMP()`(see
https://issues.apache.org/jira/browse/CALCITE-4545). The configurable time
function should be achieved, but I think it isn't current time point, and this
should be a new feature? So I tend to just fix the wrong behavior in this issue
and introduce configurable time function in the future. WDYT? [~twalthr]
> Failed to define watermark on computed column of CURRENT_TIMESTAMP and
> LOCALTIMESTAMP
> -
>
> Key: FLINK-22354
> URL: https://issues.apache.org/jira/browse/FLINK-22354
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Jark Wu
>Assignee: Leonard Xu
>Priority: Blocker
> Fix For: 1.13.0, 1.14.0
>
>
> It is very common to define watermarks on {{localtimestamp}} and
> {{current_tiemstamp}} to support *ingestion time*. However, the following
> DDLs failed in v1.13. I also tested the following DDLs can pass in v1.12.1,
> so this is a regression.
> The root cause may be introduced by FLINK-21435 which adds a [strict check to
> only allow precision =
> 3|https://github.com/apache/flink/blob/f51168041512b0473decabb2088c1ff4fa4f34bc/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/DefaultSchemaResolver.java#L225]
> (precision = 0 should also can be time attribute), however the precision of
> return type of {{current_timestamp}} and {{localtimestamp}} are 0 (another
> long-lived bug). We should fix them both.
> {code:sql}
> Flink SQL> create table datagen_source (
> > order_number BIGINT,
> > priceint,
> > buyerstring,
> > ts as current_timestamp,
> > proctime as proctime(),
> > watermark for ts as ts - interval '0.001' second
> > ) with (
> > 'connector' = 'datagen',
> > 'rows-per-second' = '1'
> > );
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.api.ValidationException: Invalid data type of time
> field for watermark definition. The field must be of type TIMESTAMP(3) or
> TIMESTAMP_LTZ(3).
> Flink SQL> create table datagen_source (
> > order_number BIGINT,
> > priceint,
> > buyerstring,
> > ts as cast(current_timestamp as timestamp_ltz(3)),
> > proctime as proctime(),
> > watermark for ts as ts - interval '0.001' second
> > ) with (
> > 'connector' = 'datagen',
> > 'rows-per-second' = '1'
> > );
> [INFO] Execute statement succeed.
> Flink SQL> create table datagen_source2 (
> > order_number BIGINT,
> > priceint,
> > buyerstring,
> > ts as localtimestamp,
> > proctime as proctime(),
> > watermark for ts as ts - interval '0.001' second
> > ) with (
> > 'connector' = 'datagen',
> > 'rows-per-second' = '1'
> > );
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.api.ValidationException: Invalid data type of time
> field for watermark definition. The field must be of type TIMESTAMP(3) or
> TIMESTAMP_LTZ(3).
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-22354) Failed to define watermark on computed column of CURRENT_TIMESTAMP and LOCALTIMESTAMP
[
https://issues.apache.org/jira/browse/FLINK-22354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325819#comment-17325819
]
Leonard Xu edited comment on FLINK-22354 at 4/20/21, 2:09 PM:
--
[~twalthr] MySQL is 0, Oracle and postgresql is 6, Snowflake is 9, I think in
big data area, milliseconds is more used, and our watermark alway support to
milliseconds precision. In the past, many users wrongly using these functions
but they can still work because the real value returned(code-gened) is in
millisecond and the rowtime checks isn't strict to 3, it's strange to
connector/UDF developer the underlying data value is in millisecond but the
DataType precision is 0, that's why I want to fix the wrong behavior.
I think the configurable precision is right thing and we should improve it in
the future. In the FLIP we didn't discuss the precision but focus on the Type
and behavior, and I think it's not a part with the FLIP, should be an existed
wrong behavior from my side.
was (Author: leonard xu):
[~twalthr] MySQL is 0, Oracle and postgresql is 6, Snowflake is 9, I think in
big data area, milliseconds is more used, and our watermark alway support to
milliseconds precision. In the past, many users wrongly using these functions
but they can still work because the real value returned(code-gened) is in
millisecond and the rowtime checks isn't strict to 3, it's strange to
connector/UDF developer the underlying data value is in millisecond but the
DataType precision is 0, that's why I want to fix the wrong behavior.
I think the configurable precision is right thing and we should improve it in
the future.
> Failed to define watermark on computed column of CURRENT_TIMESTAMP and
> LOCALTIMESTAMP
> -
>
> Key: FLINK-22354
> URL: https://issues.apache.org/jira/browse/FLINK-22354
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Jark Wu
>Assignee: Leonard Xu
>Priority: Blocker
> Fix For: 1.13.0, 1.14.0
>
>
> It is very common to define watermarks on {{localtimestamp}} and
> {{current_tiemstamp}} to support *ingestion time*. However, the following
> DDLs failed in v1.13. I also tested the following DDLs can pass in v1.12.1,
> so this is a regression.
> The root cause may be introduced by FLINK-21435 which adds a [strict check to
> only allow precision =
> 3|https://github.com/apache/flink/blob/f51168041512b0473decabb2088c1ff4fa4f34bc/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/DefaultSchemaResolver.java#L225]
> (precision = 0 should also can be time attribute), however the precision of
> return type of {{current_timestamp}} and {{localtimestamp}} are 0 (another
> long-lived bug). We should fix them both.
> {code:sql}
> Flink SQL> create table datagen_source (
> > order_number BIGINT,
> > priceint,
> > buyerstring,
> > ts as current_timestamp,
> > proctime as proctime(),
> > watermark for ts as ts - interval '0.001' second
> > ) with (
> > 'connector' = 'datagen',
> > 'rows-per-second' = '1'
> > );
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.api.ValidationException: Invalid data type of time
> field for watermark definition. The field must be of type TIMESTAMP(3) or
> TIMESTAMP_LTZ(3).
> Flink SQL> create table datagen_source (
> > order_number BIGINT,
> > priceint,
> > buyerstring,
> > ts as cast(current_timestamp as timestamp_ltz(3)),
> > proctime as proctime(),
> > watermark for ts as ts - interval '0.001' second
> > ) with (
> > 'connector' = 'datagen',
> > 'rows-per-second' = '1'
> > );
> [INFO] Execute statement succeed.
> Flink SQL> create table datagen_source2 (
> > order_number BIGINT,
> > priceint,
> > buyerstring,
> > ts as localtimestamp,
> > proctime as proctime(),
> > watermark for ts as ts - interval '0.001' second
> > ) with (
> > 'connector' = 'datagen',
> > 'rows-per-second' = '1'
> > );
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.api.ValidationException: Invalid data type of time
> field for watermark definition. The field must be of type TIMESTAMP(3) or
> TIMESTAMP_LTZ(3).
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22354) Failed to define watermark on computed column of CURRENT_TIMESTAMP and LOCALTIMESTAMP
[
https://issues.apache.org/jira/browse/FLINK-22354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325819#comment-17325819
]
Leonard Xu commented on FLINK-22354:
[~twalthr] MySQL is 0, Oracle and postgresql is 6, Snowflake is 9, I think in
big data area, milliseconds is more used, and our watermark alway support to
milliseconds precision. In the past, many users wrongly using these functions
but they can still work because the real value returned(code-gened) is in
millisecond and the rowtime checks isn't strict to 3, it's strange to
connector/UDF developer the underlying data value is in millisecond but the
DataType precision is 0, that's why I want to fix the wrong behavior.
I think the configurable precision is right thing and we should improve it in
the future.
> Failed to define watermark on computed column of CURRENT_TIMESTAMP and
> LOCALTIMESTAMP
> -
>
> Key: FLINK-22354
> URL: https://issues.apache.org/jira/browse/FLINK-22354
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Jark Wu
>Assignee: Leonard Xu
>Priority: Blocker
> Fix For: 1.13.0, 1.14.0
>
>
> It is very common to define watermarks on {{localtimestamp}} and
> {{current_tiemstamp}} to support *ingestion time*. However, the following
> DDLs failed in v1.13. I also tested the following DDLs can pass in v1.12.1,
> so this is a regression.
> The root cause may be introduced by FLINK-21435 which adds a [strict check to
> only allow precision =
> 3|https://github.com/apache/flink/blob/f51168041512b0473decabb2088c1ff4fa4f34bc/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/DefaultSchemaResolver.java#L225]
> (precision = 0 should also can be time attribute), however the precision of
> return type of {{current_timestamp}} and {{localtimestamp}} are 0 (another
> long-lived bug). We should fix them both.
> {code:sql}
> Flink SQL> create table datagen_source (
> > order_number BIGINT,
> > priceint,
> > buyerstring,
> > ts as current_timestamp,
> > proctime as proctime(),
> > watermark for ts as ts - interval '0.001' second
> > ) with (
> > 'connector' = 'datagen',
> > 'rows-per-second' = '1'
> > );
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.api.ValidationException: Invalid data type of time
> field for watermark definition. The field must be of type TIMESTAMP(3) or
> TIMESTAMP_LTZ(3).
> Flink SQL> create table datagen_source (
> > order_number BIGINT,
> > priceint,
> > buyerstring,
> > ts as cast(current_timestamp as timestamp_ltz(3)),
> > proctime as proctime(),
> > watermark for ts as ts - interval '0.001' second
> > ) with (
> > 'connector' = 'datagen',
> > 'rows-per-second' = '1'
> > );
> [INFO] Execute statement succeed.
> Flink SQL> create table datagen_source2 (
> > order_number BIGINT,
> > priceint,
> > buyerstring,
> > ts as localtimestamp,
> > proctime as proctime(),
> > watermark for ts as ts - interval '0.001' second
> > ) with (
> > 'connector' = 'datagen',
> > 'rows-per-second' = '1'
> > );
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.api.ValidationException: Invalid data type of time
> field for watermark definition. The field must be of type TIMESTAMP(3) or
> TIMESTAMP_LTZ(3).
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-22374) ConfigOptionsDocsCompletenessITCase fails on traivs
[
https://issues.apache.org/jira/browse/FLINK-22374?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu closed FLINK-22374.
--
Resolution: Invalid
> ConfigOptionsDocsCompletenessITCase fails on traivs
> ---
>
> Key: FLINK-22374
> URL: https://issues.apache.org/jira/browse/FLINK-22374
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Affects Versions: 1.13.0
>Reporter: Leonard Xu
>Priority: Major
>
>
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.128
> s <<< FAILURE! - in
> org.apache.flink.docs.configuration.ConfigOptionsDocsCompletenessITCase
> Apr 20 12:27:32 [ERROR]
> testCompleteness(org.apache.flink.docs.configuration.ConfigOptionsDocsCompletenessITCase)
> Time elapsed: 1.043 s <<< FAILURE!
> Apr 20 12:27:32 java.lang.AssertionError:
> Apr 20 12:27:32 Documentation is outdated, please regenerate it according to
> the instructions in flink-docs/README.md.
> Apr 20 12:27:32 Problems:
> Apr 20 12:27:32 Documentation of table.local-time-zone in TableConfigOptions
> is outdated. Expected: default=("default") description=(The local time zone
> defines current session time zone id. It is used when converting to/from
> codeTIMESTAMP WITH LOCAL TIME ZONE/code. Internally,
> timestamps with local time zone are always represented in the UTC time zone.
> However, when converting to data types that don't include a time zone (e.g.
> TIMESTAMP, TIME, or simply STRING), the session time zone is used during
> conversion. The input of option is either a full name such as
> "America/Los_Angeles", or a custom timezone id such as "GMT-8:00".).
> Apr 20 12:27:32 Documented option table.local-time-zone does not exist.
> Apr 20 12:27:32 at org.junit.Assert.fail(Assert.java:88)
> Apr 20 12:27:32 at
> org.apache.flink.docs.configuration.ConfigOptionsDocsCompletenessITCase.compareDocumentedAndExistingOptions(ConfigOptionsDocsCompletenessITCase.java:220)
>
> Apr 20 12:27:32 at
> org.apache.flink.docs.configuration.ConfigOptionsDocsCompletenessITCase.testCompleteness(ConfigOptionsDocsCompletenessITCase.java:76)
>
> Apr 20 12:27:32 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Apr 20 12:27:32 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Apr 20 12:27:32 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>
> Apr 20 12:27:32 at java.lang.reflect.Method.invoke(Method.java:498)
> Apr 20 12:27:32 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
>
> Apr 20 12:27:32 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
>
> Apr 20 12:27:32 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16870=logs=fc5181b0-e452-5c8f-68de-1097947f6483=62110053-334f-5295-a0ab-80dd7e2babbf=34561
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-22374) ConfigOptionsDocsCompletenessITCase fails on traivs
Leonard Xu created FLINK-22374:
--
Summary: ConfigOptionsDocsCompletenessITCase fails on traivs
Key: FLINK-22374
URL: https://issues.apache.org/jira/browse/FLINK-22374
Project: Flink
Issue Type: Bug
Components: Tests
Affects Versions: 1.13.0
Reporter: Leonard Xu
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.128 s
<<< FAILURE! - in
org.apache.flink.docs.configuration.ConfigOptionsDocsCompletenessITCase
Apr 20 12:27:32 [ERROR]
testCompleteness(org.apache.flink.docs.configuration.ConfigOptionsDocsCompletenessITCase)
Time elapsed: 1.043 s <<< FAILURE!
Apr 20 12:27:32 java.lang.AssertionError:
Apr 20 12:27:32 Documentation is outdated, please regenerate it according to
the instructions in flink-docs/README.md.
Apr 20 12:27:32 Problems:
Apr 20 12:27:32 Documentation of table.local-time-zone in TableConfigOptions is
outdated. Expected: default=("default") description=(The local time zone
defines current session time zone id. It is used when converting to/from
codeTIMESTAMP WITH LOCAL TIME ZONE/code. Internally, timestamps
with local time zone are always represented in the UTC time zone. However, when
converting to data types that don't include a time zone (e.g. TIMESTAMP, TIME,
or simply STRING), the session time zone is used during conversion. The input
of option is either a full name such as "America/Los_Angeles", or a custom
timezone id such as "GMT-8:00".).
Apr 20 12:27:32 Documented option table.local-time-zone does not exist.
Apr 20 12:27:32 at org.junit.Assert.fail(Assert.java:88)
Apr 20 12:27:32 at
org.apache.flink.docs.configuration.ConfigOptionsDocsCompletenessITCase.compareDocumentedAndExistingOptions(ConfigOptionsDocsCompletenessITCase.java:220)
Apr 20 12:27:32 at
org.apache.flink.docs.configuration.ConfigOptionsDocsCompletenessITCase.testCompleteness(ConfigOptionsDocsCompletenessITCase.java:76)
Apr 20 12:27:32 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
Apr 20 12:27:32 at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Apr 20 12:27:32 at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Apr 20 12:27:32 at java.lang.reflect.Method.invoke(Method.java:498)
Apr 20 12:27:32 at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
Apr 20 12:27:32 at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
Apr 20 12:27:32 at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16870=logs=fc5181b0-e452-5c8f-68de-1097947f6483=62110053-334f-5295-a0ab-80dd7e2babbf=34561
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22354) Failed to define watermark on computed column of CURRENT_TIMESTAMP and LOCALTIMESTAMP
[
https://issues.apache.org/jira/browse/FLINK-22354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325753#comment-17325753
]
Leonard Xu commented on FLINK-22354:
There're are some user feedback from the user mail list that users are using
{{CURRENT_TIMESTAMP, or NOW() as rowtime, I think the backward compatibility is
Jark's concern. The recommended way should be CURRENT_ROW_TIMESTAMP() and
PROCTIME() as Timo described.}}
I also found some function precision is 0 like CURRENT_TIMESTAMP but the
underlying data is in milliseconds, I'd like to fix them to 3 at the same time.
> Failed to define watermark on computed column of CURRENT_TIMESTAMP and
> LOCALTIMESTAMP
> -
>
> Key: FLINK-22354
> URL: https://issues.apache.org/jira/browse/FLINK-22354
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Reporter: Jark Wu
>Assignee: Leonard Xu
>Priority: Blocker
> Fix For: 1.13.0, 1.14.0
>
>
> It is very common to define watermarks on {{localtimestamp}} and
> {{current_tiemstamp}} to support *ingestion time*. However, the following
> DDLs failed in v1.13. I also tested the following DDLs can pass in v1.12.1,
> so this is a regression.
> The root cause may be introduced by FLINK-21435 which adds a [strict check to
> only allow precision =
> 3|https://github.com/apache/flink/blob/f51168041512b0473decabb2088c1ff4fa4f34bc/flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/DefaultSchemaResolver.java#L225]
> (precision = 0 should also can be time attribute), however the precision of
> return type of {{current_timestamp}} and {{localtimestamp}} are 0 (another
> long-lived bug). We should fix them both.
> {code:sql}
> Flink SQL> create table datagen_source (
> > order_number BIGINT,
> > priceint,
> > buyerstring,
> > ts as current_timestamp,
> > proctime as proctime(),
> > watermark for ts as ts - interval '0.001' second
> > ) with (
> > 'connector' = 'datagen',
> > 'rows-per-second' = '1'
> > );
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.api.ValidationException: Invalid data type of time
> field for watermark definition. The field must be of type TIMESTAMP(3) or
> TIMESTAMP_LTZ(3).
> Flink SQL> create table datagen_source (
> > order_number BIGINT,
> > priceint,
> > buyerstring,
> > ts as cast(current_timestamp as timestamp_ltz(3)),
> > proctime as proctime(),
> > watermark for ts as ts - interval '0.001' second
> > ) with (
> > 'connector' = 'datagen',
> > 'rows-per-second' = '1'
> > );
> [INFO] Execute statement succeed.
> Flink SQL> create table datagen_source2 (
> > order_number BIGINT,
> > priceint,
> > buyerstring,
> > ts as localtimestamp,
> > proctime as proctime(),
> > watermark for ts as ts - interval '0.001' second
> > ) with (
> > 'connector' = 'datagen',
> > 'rows-per-second' = '1'
> > );
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.api.ValidationException: Invalid data type of time
> field for watermark definition. The field must be of type TIMESTAMP(3) or
> TIMESTAMP_LTZ(3).
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (FLINK-22160) Test Window TVF based aggregation and TopN
[
https://issues.apache.org/jira/browse/FLINK-22160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu resolved FLINK-22160.
Resolution: Resolved
> Test Window TVF based aggregation and TopN
> --
>
> Key: FLINK-22160
> URL: https://issues.apache.org/jira/browse/FLINK-22160
> Project: Flink
> Issue Type: Test
> Components: Table SQL / API
>Reporter: Jark Wu
>Assignee: Leonard Xu
>Priority: Blocker
> Labels: release-testing
> Fix For: 1.13.0
>
>
> In FLINK-19604
> ([FLIP-145|https://cwiki.apache.org/confluence/display/FLINK/FLIP-145%3A+Support+SQL+windowing+table-valued+function]),
> we introduced a new syntax to express Window Aggregate and Window TopN. For
> Window Aggregate, we have also introduced a new window kind: cumulate
> windows.
> The scope of this task is to make sure:
> 1. The old window aggergate syntax ({{GROUP BY TUMBLE(...)}}) can be rewrite
> using the new syntax, and get the same results. Note, session window is not
> supported yet in the new syntax.
> 2. Verify the new CUMULATE window works as expect
> 3. Verify the new Window TopN workss as expect
> 4. Failure and recovery and rescale case: results are still correct.
> 5. Window emitting: window should be fired once watermark advances window end
> (we can manually generate source data with monotonically and slowly
> increasing timestamp)
> 6. The feature is well-documented
> Note: the documentation for this feature is still going on (FLINK-22159), for
> testing the feature, we can use the FLIP documentation as an instruction for
> now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22160) Test Window TVF based aggregation and TopN
[
https://issues.apache.org/jira/browse/FLINK-22160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17325479#comment-17325479
]
Leonard Xu commented on FLINK-22160:
* Tested TUMBLE/HOP/CUMULATE TVF in Sql-Client, the result is as expected
* Tested TUMBLE/HOP/CUMULATE TVF using TIMESTAMP_LTZ rowtime in Sql-Client,
the result is as expected
* Tested PROCTIME TUMBLE/HOP/CUMULATE TVF in different timezone in
Sql-Client, the result is as expected
* Tested rowtime/proctime TopN on TUMBLE TVF, the result is as expected
* Tested Failure and recovery by Killing one TM, the job is recovery and run
continue as expected.
1. Found the session window table value function parse ERROR, due to the
session window is unsupported, I think we can ignore it,
{code:java}
Flink SQL> SELECT bidder, window_start, window_end, SUM(price)
> FROM TABLE(
> SESSION(TABLE BidSession PARTITION BY bidder, DESCRIPTOR(bidtime),
> DESCRIPTOR(bidder), INTERVAL '5' MINUTES)
> GROUP BY bidder, window_start, window_end;
[ERROR] Could not execute SQL statement. Reason:
org.apache.flink.sql.parser.impl.ParseException: Encountered "PARTITION" at
line 3, column 30.
Was expecting one of:
Was expecting one of:
"EXCEPT" ...
"FETCH" ...
"INTERSECT" ...
"LIMIT" ...
"OFFSET" ...
"ORDER" ...
"MINUS" ...
"UNION" ...
{code}
2. Found the doc PR has some minor improvement, I left comments on the doc PR.
The detail tests can refer here:
{code:java}
In FLINK-19604 (FLIP-145), we introduced a new syntax to express Window
Aggregate and Window TopN. For Window Aggregate, we have also introduced a new
window kind: cumulate windows.The scope of this task is to make sure:1. The old
window aggergate syntax (GROUP BY TUMBLE(...)) can be rewrite using the new
syntax, and get the same results. Note, session window is not supported yet in
the new syntax.
2. Verify the new CUMULATE window works as expect
3. Verify the new Window TopN workss as expect
4. Failure and recovery and rescale case: results are still correct.
5. Window emitting: window should be fired once watermark advances window end
(we can manually generate source data with monotonically and slowly increasing
timestamp)
6. The feature is well-documentedNote: the documentation for this feature is
still going on (FLINK-22159), for testing the feature, we can use the FLIP
documentation as an instruction for now.
## Test ROWTIME WINDOW TVF
### Test TUMBLE WINDOWcreate table Bid (
bidtime TIMESTAMP(3),
bidtime_ltz TIMESTAMP_LTZ(3),
price DOUBLE,
item STRING,
proctime AS PROCTIME(),
watermark for bidtime as bidtime - interval '10' MINUTES
) WITH (
'connector' = 'socket',
'hostname' = '127.0.0.1',
'port' = '',
'format' = 'csv'
);2020-04-20 08:07:00,2020-04-20 08:07:00Z,2,A
2020-04-20 08:11:00,2020-04-20 08:11:00Z,3,B
2020-04-20 08:05:00,2020-04-20 08:05:00Z,4,C
2020-04-20 08:09:00,2020-04-20 08:09:00Z,5,D
2020-04-20 08:13:00,2020-04-20 08:13:00Z,1,E
2020-04-20 08:17:00,2020-04-20 08:17:00Z,6,F
2020-04-20 08:30:01,2020-04-20 08:17:00Z,6,G ADD note flink doesn't support
individual window table-valued function now.
SELECT * FROM TABLE(
TUMBLE(TABLE Bid, DESCRIPTOR(bidtime), INTERVAL '10' MINUTES));
Flink SQL> SELECT window_start, window_end, SUM(price)
> FROM TABLE(
> TUMBLE(TABLE Bid, DESCRIPTOR(bidtime), INTERVAL '10' MINUTES))
> GROUP BY window_start, window_end;
++-+-++
| op |window_start | window_end |
EXPR$2 |
++-+-++
| +I | 2020-04-20 08:00:00.000 | 2020-04-20 08:10:00.000 |
11.0 |
| +I | 2020-04-20 08:10:00.000 | 2020-04-20 08:20:00.000 |
10.0 |
2020-04-20 08:07:00,2020-04-20 08:07:00Z,2,A
2020-04-20 08:11:00,2020-04-20 08:11:00Z,3,B
2020-04-20 08:05:00,2020-04-20 08:05:00Z,4,C
2020-04-20 08:09:00,2020-04-20 08:09:00Z,5,D
2020-04-20 08:13:00,2020-04-20 08:13:00Z,1,E
2020-04-20 08:17:00,2020-04-20 08:17:00Z,6,F
2020-04-20 08:35:01,2020-04-20 08:17:00Z,6,G
### Test HOP WINDOWFlink SQL> SELECT window_start, window_end, SUM(price)
> FROM TABLE(
> HOP(TABLE Bid, DESCRIPTOR(bidtime), INTERVAL '5' MINUTES, INTERVAL '10'
> MINUTES))
> GROUP BY window_start, window_end;
++-+-++
| op |window_start | window_end |
EXPR$2 |
++-+-++
| +I | 2020-04-20 08:00:00.000 | 2020-04-20 08:10:00.000 |
11.0 |
| +I | 2020-04-20 08:05:00.000 | 2020-04-20 08:15:00.000 |
15.0 |
| +I | 2020-04-20 08:10:00.000 |
[jira] [Commented] (FLINK-21716) Support higher precision for Data Type TIME(p)
[ https://issues.apache.org/jira/browse/FLINK-21716?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17323900#comment-17323900 ] Leonard Xu commented on FLINK-21716: I'll work for this. > Support higher precision for Data Type TIME(p) > --- > > Key: FLINK-21716 > URL: https://issues.apache.org/jira/browse/FLINK-21716 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Table SQL / Runtime >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > Labels: stale-assigned > > Due to the historical reason, we only support TIME(3) yet, we can support > higher precision eg. TIME(9). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21431) UpsertKafkaTableITCase.testTemporalJoin hang
[
https://issues.apache.org/jira/browse/FLINK-21431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17322760#comment-17322760
]
Leonard Xu commented on FLINK-21431:
[~maguowei] I'll create a PR for 1.12 branch
> UpsertKafkaTableITCase.testTemporalJoin hang
>
>
> Key: FLINK-21431
> URL: https://issues.apache.org/jira/browse/FLINK-21431
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.2, 1.13.0
>Reporter: Guowei Ma
>Assignee: Leonard Xu
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.13.0
>
>
> This case hangs almost 3 hours:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13543=logs=ce8f3cc3-c1ea-5281-f5eb-df9ebd24947f=f266c805-9429-58ed-2f9e-482e7b82f58b
> {code:java}
> Test testTemporalJoin[format =
> csv](org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaTableITCase)
> is running.
>
> 23:08:43,259 [ main] INFO
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl [] -
> Creating topic users_csv 23:08:45,303 [ main] WARN
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer [] - Property
> [transaction.timeout.ms] not specified. Setting it to 360 ms 23:08:45,430
> [ChangelogNormalize(key=[user_id]) -> Calc(select=[user_id, user_name,
> region, CAST(modification_time) AS timestamp]) -> Sink:
> Sink(table=[default_catalog.default_database.users_csv], fields=[user_id,
> user_name, region, timestamp]) (1/1)#0] WARN
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer [] - Using
> AT_LEAST_ONCE semantic, but checkpointing is not enabled. Switching to NONE
> semantic. 23:08:45,438 [ChangelogNormalize(key=[user_id]) ->
> Calc(select=[user_id, user_name, region, CAST(modification_time) AS
> timestamp]) -> Sink: Sink(table=[default_catalog.default_database.users_csv],
> fields=[user_id, user_name, region, timestamp]) (1/1)#0] INFO
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer [] - Starting
> FlinkKafkaInternalProducer (1/1) to produce into default topic users_csv
> 23:08:45,791 [Source: TableSourceScan(table=[[default_catalog,
> default_database, users_csv, watermark=[CAST($3):TIMESTAMP(3)]]],
> fields=[user_id, user_name, region, timestamp]) (1/1)#0] INFO
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase [] -
> Consumer subtask 0 has no restore state. 23:08:45,810 [Source:
> TableSourceScan(table=[[default_catalog, default_database, users_csv,
> watermark=[CAST($3):TIMESTAMP(3)]]], fields=[user_id, user_name, region,
> timestamp]) (1/1)#0] INFO
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase [] -
> Consumer subtask 0 will start reading the following 2 partitions from the
> earliest offsets: [KafkaTopicPartition{topic='users_csv', partition=1},
> KafkaTopicPartition{topic='users_csv', partition=0}] 23:08:45,825 [Legacy
> Source Thread - Source: TableSourceScan(table=[[default_catalog,
> default_database, users_csv, watermark=[CAST($3):TIMESTAMP(3)]]],
> fields=[user_id, user_name, region, timestamp]) (1/1)#0] INFO
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase [] -
> Consumer subtask 0 creating fetcher with offsets
> {KafkaTopicPartition{topic='users_csv', partition=1}=-915623761775,
> KafkaTopicPartition{topic='users_csv', partition=0}=-915623761775}.
> ##[error]The operation was canceled.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21431) UpsertKafkaTableITCase.testTemporalJoin hang
[
https://issues.apache.org/jira/browse/FLINK-21431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated FLINK-21431:
---
Fix Version/s: 1.12.3
> UpsertKafkaTableITCase.testTemporalJoin hang
>
>
> Key: FLINK-21431
> URL: https://issues.apache.org/jira/browse/FLINK-21431
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.12.2, 1.13.0
>Reporter: Guowei Ma
>Assignee: Leonard Xu
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.13.0, 1.12.3
>
>
> This case hangs almost 3 hours:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13543=logs=ce8f3cc3-c1ea-5281-f5eb-df9ebd24947f=f266c805-9429-58ed-2f9e-482e7b82f58b
> {code:java}
> Test testTemporalJoin[format =
> csv](org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaTableITCase)
> is running.
>
> 23:08:43,259 [ main] INFO
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl [] -
> Creating topic users_csv 23:08:45,303 [ main] WARN
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer [] - Property
> [transaction.timeout.ms] not specified. Setting it to 360 ms 23:08:45,430
> [ChangelogNormalize(key=[user_id]) -> Calc(select=[user_id, user_name,
> region, CAST(modification_time) AS timestamp]) -> Sink:
> Sink(table=[default_catalog.default_database.users_csv], fields=[user_id,
> user_name, region, timestamp]) (1/1)#0] WARN
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer [] - Using
> AT_LEAST_ONCE semantic, but checkpointing is not enabled. Switching to NONE
> semantic. 23:08:45,438 [ChangelogNormalize(key=[user_id]) ->
> Calc(select=[user_id, user_name, region, CAST(modification_time) AS
> timestamp]) -> Sink: Sink(table=[default_catalog.default_database.users_csv],
> fields=[user_id, user_name, region, timestamp]) (1/1)#0] INFO
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer [] - Starting
> FlinkKafkaInternalProducer (1/1) to produce into default topic users_csv
> 23:08:45,791 [Source: TableSourceScan(table=[[default_catalog,
> default_database, users_csv, watermark=[CAST($3):TIMESTAMP(3)]]],
> fields=[user_id, user_name, region, timestamp]) (1/1)#0] INFO
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase [] -
> Consumer subtask 0 has no restore state. 23:08:45,810 [Source:
> TableSourceScan(table=[[default_catalog, default_database, users_csv,
> watermark=[CAST($3):TIMESTAMP(3)]]], fields=[user_id, user_name, region,
> timestamp]) (1/1)#0] INFO
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase [] -
> Consumer subtask 0 will start reading the following 2 partitions from the
> earliest offsets: [KafkaTopicPartition{topic='users_csv', partition=1},
> KafkaTopicPartition{topic='users_csv', partition=0}] 23:08:45,825 [Legacy
> Source Thread - Source: TableSourceScan(table=[[default_catalog,
> default_database, users_csv, watermark=[CAST($3):TIMESTAMP(3)]]],
> fields=[user_id, user_name, region, timestamp]) (1/1)#0] INFO
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase [] -
> Consumer subtask 0 creating fetcher with offsets
> {KafkaTopicPartition{topic='users_csv', partition=1}=-915623761775,
> KafkaTopicPartition{topic='users_csv', partition=0}=-915623761775}.
> ##[error]The operation was canceled.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20254) HiveTableSourceITCase.testStreamPartitionReadByCreateTime times out
[
https://issues.apache.org/jira/browse/FLINK-20254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated FLINK-20254:
---
Description:
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=9808=logs=fc5181b0-e452-5c8f-68de-1097947f6483=62110053-334f-5295-a0ab-80dd7e2babbf]
{code:java}
2020-11-19T10:34:23.5591765Z [ERROR] Tests run: 18, Failures: 0, Errors: 1,
Skipped: 0, Time elapsed: 192.243 s <<< FAILURE! - in
org.apache.flink.connectors.hive.HiveTableSourceITCase
2020-11-19T10:34:23.5593193Z [ERROR]
testStreamPartitionReadByCreateTime(org.apache.flink.connectors.hive.HiveTableSourceITCase)
Time elapsed: 120.075 s <<< ERROR!
2020-11-19T10:34:23.5593929Z org.junit.runners.model.TestTimedOutException:
test timed out after 12 milliseconds
2020-11-19T10:34:23.5594321Zat java.lang.Thread.sleep(Native Method)
2020-11-19T10:34:23.5594777Zat
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sleepBeforeRetry(CollectResultFetcher.java:231)
2020-11-19T10:34:23.5595378Zat
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:119)
2020-11-19T10:34:23.5596001Zat
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:103)
2020-11-19T10:34:23.5596610Zat
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:77)
2020-11-19T10:34:23.5597218Zat
org.apache.flink.table.planner.sinks.SelectTableSinkBase$RowIteratorWrapper.hasNext(SelectTableSinkBase.java:115)
2020-11-19T10:34:23.5597811Zat
org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:355)
2020-11-19T10:34:23.5598555Zat
org.apache.flink.connectors.hive.HiveTableSourceITCase.fetchRows(HiveTableSourceITCase.java:653)
2020-11-19T10:34:23.5599407Zat
org.apache.flink.connectors.hive.HiveTableSourceITCase.testStreamPartitionReadByCreateTime(HiveTableSourceITCase.java:594)
2020-11-19T10:34:23.5599982Zat
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-11-19T10:34:23.5600393Zat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-11-19T10:34:23.5600865Zat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-11-19T10:34:23.5601300Zat
java.lang.reflect.Method.invoke(Method.java:498)
2020-11-19T10:34:23.5601713Zat
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2020-11-19T10:34:23.5602211Zat
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2020-11-19T10:34:23.5602688Zat
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2020-11-19T10:34:23.5603181Zat
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2020-11-19T10:34:23.5603753Zat
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
2020-11-19T10:34:23.5604308Zat
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
2020-11-19T10:34:23.5604780Zat
java.util.concurrent.FutureTask.run(FutureTask.java:266)
2020-11-19T10:34:23.5605114Zat java.lang.Thread.run(Thread.java:748)
2020-11-19T10:34:23.5605299Z
2020-11-19T10:34:24.4180149Z [INFO] Running
org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase
{code}
I've spent some time to debug this case in local env, but unfortunately I
didn't find the root cause. I think this is the same case with FLINK-22129,
FLINK-22100, but after the debug, these failed tests in *HiveTableSourceITCase*
should not exist bugs, and the root cause may be related to FLINK-21996.
Given some debug results:
* 1. Run HiveTableSourceITCase.testStreamPartitionReadByCreateTime times out
repeatedly in IDE, it mostly failed at about the 36th round
* 2. The thread usually hangs on
{code:java}
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:119){code}
{code:java}
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sendRequest(CollectResultFetcher.java:155){code}
I try to add timeout for CompletableFutures in CollectResultFetcher, but looks
it doesn't help
* 3. When the thread stuck, the SourceReaderBase#addSplits may receive split
or may not(I reproduced multiple rounds)， but before the test failed, there's a
checkpoint ERRORR which may be related to the case.
{code:java}
INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Running
initialization on master for job collect (3572e9cf6c0829ebc9f76327388d).
INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - No
checkpoint found during restore.
INFO
[jira] [Updated] (FLINK-22179) Manual test time function changes and new time attributes support
[ https://issues.apache.org/jira/browse/FLINK-22179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-22179: --- Description: * 1. Test time functions return expected value in both batch and streaming mode, we can use Sql-client to test. - LOCALTIME - LOCALTIMESTAMP - CURRENT_DATE - CURRENT_TIME - CURRENT_TIMESTAMP - NOW() - PROCTIME() * 2. Test PROCTIME attribute works well in different case, eg(`interval join`, `temporal join(lookup)`, `window`), and check sink.partition-commit.trigger that used proctime is as expected in Hive/FileSystem sink * 3. Test ROWTIME attribute that use TIMESTAMP_LTZ, cover `interval join`, `temporal join`, `window` scenarios * 4. Test the `window` result is as expected in DaylightSaving timezone was: * 1. Test time functions return expected value in both batch and streaming mode, we can use Sql-client to test. - LOCALTIME - LOCALTIMESTAMP - CURRENT_DATE - CURRENT_TIME - CURRENT_TIMESTAMP - NOW() - PROCTIME() * 2. Test PROCTIME attribute works well in different case, eg(`interval join`, `temporal join(lookup)`, `window`), and check sink.partition-commit.trigger that used proctime is as expected in Hive/FileSystem sink * 3. Test ROWTIME attribute that use TIMESTAMP_LTZ, cover `interval join`, `temporal join`, `window` scenario > Manual test time function changes and new time attributes support > - > > Key: FLINK-22179 > URL: https://issues.apache.org/jira/browse/FLINK-22179 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.13.0 >Reporter: Leonard Xu >Assignee: Jark Wu >Priority: Blocker > Labels: release-testing > Fix For: 1.13.0 > > > * 1. Test time functions return expected value in both batch and streaming > mode, we can use Sql-client to test. > - LOCALTIME > - LOCALTIMESTAMP > - CURRENT_DATE > - CURRENT_TIME > - CURRENT_TIMESTAMP > - NOW() > - PROCTIME() > * 2. Test PROCTIME attribute works well in different case, eg(`interval > join`, `temporal join(lookup)`, `window`), and check > sink.partition-commit.trigger that used proctime is as expected in > Hive/FileSystem sink > * 3. Test ROWTIME attribute that use TIMESTAMP_LTZ, cover `interval join`, > `temporal join`, `window` scenarios > * 4. Test the `window` result is as expected in DaylightSaving timezone -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-22179) Manual test time function changes and new time attributes support
[ https://issues.apache.org/jira/browse/FLINK-22179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-22179: --- Description: * 1. Test time functions return expected value in both batch and streaming mode, we can use Sql-client to test. - LOCALTIME - LOCALTIMESTAMP - CURRENT_DATE - CURRENT_TIME - CURRENT_TIMESTAMP - NOW() - PROCTIME() * 2. Test PROCTIME attribute works well in different case, eg(`interval join`, `temporal join(lookup)`, `window`), and check sink.partition-commit.trigger that used proctime is as expected in Hive/FileSystem sink * 3. Test ROWTIME attribute that use TIMESTAMP_LTZ, cover `interval join`, `temporal join`, `window` scenario > Manual test time function changes and new time attributes support > - > > Key: FLINK-22179 > URL: https://issues.apache.org/jira/browse/FLINK-22179 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.13.0 >Reporter: Leonard Xu >Assignee: Jark Wu >Priority: Blocker > Labels: release-testing > Fix For: 1.13.0 > > > * 1. Test time functions return expected value in both batch and streaming > mode, we can use Sql-client to test. > - LOCALTIME > - LOCALTIMESTAMP > - CURRENT_DATE > - CURRENT_TIME > - CURRENT_TIMESTAMP > - NOW() > - PROCTIME() > * 2. Test PROCTIME attribute works well in different case, eg(`interval > join`, `temporal join(lookup)`, `window`), and check > sink.partition-commit.trigger that used proctime is as expected in > Hive/FileSystem sink > * 3. Test ROWTIME attribute that use TIMESTAMP_LTZ, cover `interval join`, > `temporal join`, `window` scenario -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20254) HiveTableSourceITCase.testStreamPartitionReadByCreateTime times out
[
https://issues.apache.org/jira/browse/FLINK-20254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17321008#comment-17321008
]
Leonard Xu commented on FLINK-20254:
{quote}
Is this the same issue?
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=15915=logs=fc5181b0-e452-5c8f-68de-1097947f6483=62110053-334f-5295-a0ab-80dd7e2babbf
It's a different unit test but in the same file, with very similar name and the
same symptom
[ERROR] Errors:
[ERROR]
HiveTableSourceITCase.testStreamPartitionReadByPartitionTime:703->fetchRows:712
» TestTimedOut
{quote}
I think these case are same issue and the timeout reason should not relate to
hive tests.
> HiveTableSourceITCase.testStreamPartitionReadByCreateTime times out
> ---
>
> Key: FLINK-20254
> URL: https://issues.apache.org/jira/browse/FLINK-20254
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.12.0, 1.13.0
>Reporter: Robert Metzger
>Assignee: Leonard Xu
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.13.0
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=9808=logs=fc5181b0-e452-5c8f-68de-1097947f6483=62110053-334f-5295-a0ab-80dd7e2babbf]
> {code:java}
> 2020-11-19T10:34:23.5591765Z [ERROR] Tests run: 18, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 192.243 s <<< FAILURE! - in
> org.apache.flink.connectors.hive.HiveTableSourceITCase
> 2020-11-19T10:34:23.5593193Z [ERROR]
> testStreamPartitionReadByCreateTime(org.apache.flink.connectors.hive.HiveTableSourceITCase)
> Time elapsed: 120.075 s <<< ERROR!
> 2020-11-19T10:34:23.5593929Z org.junit.runners.model.TestTimedOutException:
> test timed out after 12 milliseconds
> 2020-11-19T10:34:23.5594321Z at java.lang.Thread.sleep(Native Method)
> 2020-11-19T10:34:23.5594777Z at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sleepBeforeRetry(CollectResultFetcher.java:231)
> 2020-11-19T10:34:23.5595378Z at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:119)
> 2020-11-19T10:34:23.5596001Z at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:103)
> 2020-11-19T10:34:23.5596610Z at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:77)
> 2020-11-19T10:34:23.5597218Z at
> org.apache.flink.table.planner.sinks.SelectTableSinkBase$RowIteratorWrapper.hasNext(SelectTableSinkBase.java:115)
> 2020-11-19T10:34:23.5597811Z at
> org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:355)
> 2020-11-19T10:34:23.5598555Z at
> org.apache.flink.connectors.hive.HiveTableSourceITCase.fetchRows(HiveTableSourceITCase.java:653)
> 2020-11-19T10:34:23.5599407Z at
> org.apache.flink.connectors.hive.HiveTableSourceITCase.testStreamPartitionReadByCreateTime(HiveTableSourceITCase.java:594)
> 2020-11-19T10:34:23.5599982Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-11-19T10:34:23.5600393Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-11-19T10:34:23.5600865Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-11-19T10:34:23.5601300Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-11-19T10:34:23.5601713Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-11-19T10:34:23.5602211Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-11-19T10:34:23.5602688Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-11-19T10:34:23.5603181Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-11-19T10:34:23.5603753Z at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> 2020-11-19T10:34:23.5604308Z at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> 2020-11-19T10:34:23.5604780Z at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2020-11-19T10:34:23.5605114Z at java.lang.Thread.run(Thread.java:748)
> 2020-11-19T10:34:23.5605299Z
> 2020-11-19T10:34:24.4180149Z [INFO] Running
> org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase
> {code}
>
> I've spent some time to debug this case in local env, but unfortunately I
> didn't find the root cause. I think this is the same case with FLINK-22129,
> FLINK-22100, but after the debug, Ithe failed test in
[jira] [Updated] (FLINK-20254) HiveTableSourceITCase.testStreamPartitionReadByCreateTime times out
[
https://issues.apache.org/jira/browse/FLINK-20254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated FLINK-20254:
---
Description:
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=9808=logs=fc5181b0-e452-5c8f-68de-1097947f6483=62110053-334f-5295-a0ab-80dd7e2babbf]
{code:java}
2020-11-19T10:34:23.5591765Z [ERROR] Tests run: 18, Failures: 0, Errors: 1,
Skipped: 0, Time elapsed: 192.243 s <<< FAILURE! - in
org.apache.flink.connectors.hive.HiveTableSourceITCase
2020-11-19T10:34:23.5593193Z [ERROR]
testStreamPartitionReadByCreateTime(org.apache.flink.connectors.hive.HiveTableSourceITCase)
Time elapsed: 120.075 s <<< ERROR!
2020-11-19T10:34:23.5593929Z org.junit.runners.model.TestTimedOutException:
test timed out after 12 milliseconds
2020-11-19T10:34:23.5594321Zat java.lang.Thread.sleep(Native Method)
2020-11-19T10:34:23.5594777Zat
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sleepBeforeRetry(CollectResultFetcher.java:231)
2020-11-19T10:34:23.5595378Zat
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:119)
2020-11-19T10:34:23.5596001Zat
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:103)
2020-11-19T10:34:23.5596610Zat
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:77)
2020-11-19T10:34:23.5597218Zat
org.apache.flink.table.planner.sinks.SelectTableSinkBase$RowIteratorWrapper.hasNext(SelectTableSinkBase.java:115)
2020-11-19T10:34:23.5597811Zat
org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:355)
2020-11-19T10:34:23.5598555Zat
org.apache.flink.connectors.hive.HiveTableSourceITCase.fetchRows(HiveTableSourceITCase.java:653)
2020-11-19T10:34:23.5599407Zat
org.apache.flink.connectors.hive.HiveTableSourceITCase.testStreamPartitionReadByCreateTime(HiveTableSourceITCase.java:594)
2020-11-19T10:34:23.5599982Zat
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-11-19T10:34:23.5600393Zat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-11-19T10:34:23.5600865Zat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-11-19T10:34:23.5601300Zat
java.lang.reflect.Method.invoke(Method.java:498)
2020-11-19T10:34:23.5601713Zat
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2020-11-19T10:34:23.5602211Zat
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2020-11-19T10:34:23.5602688Zat
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2020-11-19T10:34:23.5603181Zat
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2020-11-19T10:34:23.5603753Zat
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
2020-11-19T10:34:23.5604308Zat
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
2020-11-19T10:34:23.5604780Zat
java.util.concurrent.FutureTask.run(FutureTask.java:266)
2020-11-19T10:34:23.5605114Zat java.lang.Thread.run(Thread.java:748)
2020-11-19T10:34:23.5605299Z
2020-11-19T10:34:24.4180149Z [INFO] Running
org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase
{code}
I've spent some time to debug this case in local env, but unfortunately I
didn't find the root cause. I think this is the same case with FLINK-22129,
FLINK-22100, but after the debug, Ithe failed test in *HiveTableSourceITCase*
should not exist bugs, and the root cause may be related to FLINK-21996.
Given some debug results:
* 1. Run HiveTableSourceITCase.testStreamPartitionReadByCreateTime times out
repeatedly in IDE, it mostly failed in the aboud the 36th round
* 2. The thread usually hang on
{code:java}
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:119){code}
{code:java}
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sendRequest(CollectResultFetcher.java:155){code}
I try to add timeout for CompletableFutures in CollectResultFetcher, but looks
it doesn't help
* 3. When the thread stuck, the SourceReaderBase#addSplits may recieve split
or may not(I reproduced multiple round)， but before the test failed, there're a
checkpoint ERRORR which may related to the case.
{code:java}
INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Running
initialization on master for job collect (3572e9cf6c0829ebc9f76327388d).
INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - No
checkpoint found during restore.
INFO
[jira] [Updated] (FLINK-20254) HiveTableSourceITCase.testStreamPartitionReadByCreateTime times out
[
https://issues.apache.org/jira/browse/FLINK-20254?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated FLINK-20254:
---
Description:
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=9808=logs=fc5181b0-e452-5c8f-68de-1097947f6483=62110053-334f-5295-a0ab-80dd7e2babbf]
{code:java}
2020-11-19T10:34:23.5591765Z [ERROR] Tests run: 18, Failures: 0, Errors: 1,
Skipped: 0, Time elapsed: 192.243 s <<< FAILURE! - in
org.apache.flink.connectors.hive.HiveTableSourceITCase
2020-11-19T10:34:23.5593193Z [ERROR]
testStreamPartitionReadByCreateTime(org.apache.flink.connectors.hive.HiveTableSourceITCase)
Time elapsed: 120.075 s <<< ERROR!
2020-11-19T10:34:23.5593929Z org.junit.runners.model.TestTimedOutException:
test timed out after 12 milliseconds
2020-11-19T10:34:23.5594321Zat java.lang.Thread.sleep(Native Method)
2020-11-19T10:34:23.5594777Zat
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sleepBeforeRetry(CollectResultFetcher.java:231)
2020-11-19T10:34:23.5595378Zat
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:119)
2020-11-19T10:34:23.5596001Zat
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:103)
2020-11-19T10:34:23.5596610Zat
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:77)
2020-11-19T10:34:23.5597218Zat
org.apache.flink.table.planner.sinks.SelectTableSinkBase$RowIteratorWrapper.hasNext(SelectTableSinkBase.java:115)
2020-11-19T10:34:23.5597811Zat
org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:355)
2020-11-19T10:34:23.5598555Zat
org.apache.flink.connectors.hive.HiveTableSourceITCase.fetchRows(HiveTableSourceITCase.java:653)
2020-11-19T10:34:23.5599407Zat
org.apache.flink.connectors.hive.HiveTableSourceITCase.testStreamPartitionReadByCreateTime(HiveTableSourceITCase.java:594)
2020-11-19T10:34:23.5599982Zat
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
2020-11-19T10:34:23.5600393Zat
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
2020-11-19T10:34:23.5600865Zat
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
2020-11-19T10:34:23.5601300Zat
java.lang.reflect.Method.invoke(Method.java:498)
2020-11-19T10:34:23.5601713Zat
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
2020-11-19T10:34:23.5602211Zat
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
2020-11-19T10:34:23.5602688Zat
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
2020-11-19T10:34:23.5603181Zat
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
2020-11-19T10:34:23.5603753Zat
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
2020-11-19T10:34:23.5604308Zat
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
2020-11-19T10:34:23.5604780Zat
java.util.concurrent.FutureTask.run(FutureTask.java:266)
2020-11-19T10:34:23.5605114Zat java.lang.Thread.run(Thread.java:748)
2020-11-19T10:34:23.5605299Z
2020-11-19T10:34:24.4180149Z [INFO] Running
org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase
{code}
I've spent some time to debug this case in local env, but unfortunately I
didn't find the root cause. I think this is the same case with FLINK-22129,
FLINK-22100, but after the debug, I think the Hiv, and the root cause may be
related to FLINK-21996.
Given some debug results:
* 1. Run HiveTableSourceITCase.testStreamPartitionReadByCreateTime times out
repeatedly in IDE, it mostly failed in the aboud the 36th round
* 2. The thread usually hang on
{code:java}
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:119){code}
{code:java}
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sendRequest(CollectResultFetcher.java:155){code}
I try to add timeout for CompletableFutures in CollectResultFetcher, but looks
it doesn't help
* 3. When the thread stuck, the SourceReaderBase#addSplits may recieve split
or may not(I reproduced multiple round)， but before the test failed, there're a
checkpoint ERRORR which may related to the case.
{code:java}
INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Running
initialization on master for job collect (3572e9cf6c0829ebc9f76327388d).
INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator[] - No
checkpoint found during restore.
INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Using
failover strategy
[jira] [Commented] (FLINK-21345) NullPointerException LogicalCorrelateToJoinFromTemporalTableFunctionRule.scala:157
[
https://issues.apache.org/jira/browse/FLINK-21345?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17320094#comment-17320094
]
Leonard Xu commented on FLINK-21345:
Hi [~zicat] Do you have time to update this PR?
> NullPointerException
> LogicalCorrelateToJoinFromTemporalTableFunctionRule.scala:157
> --
>
> Key: FLINK-21345
> URL: https://issues.apache.org/jira/browse/FLINK-21345
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.12.1
> Environment: Planner: BlinkPlanner
> Flink Version: 1.12.1_2.11
> Java Version: 1.8
> OS: mac os
>Reporter: lynn1.zhang
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.13.0
>
> Attachments: image-2021-02-10-16-00-45-553.png
>
>
> First Step: Create 2 Source Tables as below:
> {code:java}
> CREATE TABLE test_streaming(
> vid BIGINT,
> ts BIGINT,
> proc AS proctime()
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'test-streaming',
> 'properties.bootstrap.servers' = '127.0.0.1:9092',
> 'scan.startup.mode' = 'latest-offset',
> 'format' = 'json'
> );
> CREATE TABLE test_streaming2(
> vid BIGINT,
> ts BIGINT,
> proc AS proctime()
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'test-streaming2',
> 'properties.bootstrap.servers' = '127.0.0.1:9092',
> 'scan.startup.mode' = 'latest-offset',
> 'format' = 'json'
> );
> {code}
> Second Step: Create a TEMPORARY Table Function, function name:dim, key:vid,
> timestamp:proctime()
> Third Step: test_streaming union all test_streaming2 join dim like below:
> {code:java}
> SELECT r.vid,d.name,timestamp_from_long(r.ts)
> FROM (
> SELECT * FROM test_streaming UNION ALL SELECT * FROM test_streaming2
> ) AS r,
> LATERAL TABLE (dim(r.proc)) AS d
> WHERE r.vid = d.vid;
> {code}
> Exception Detail: (if only use test-streaming or test-streaming2 join
> temporary table function, the program run ok)
> {code:java}
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.flink.table.planner.plan.rules.logical.LogicalCorrelateToJoinFromTemporalTableFunctionRule.getRelOptSchema(LogicalCorrelateToJoinFromTemporalTableFunctionRule.scala:157)
> at
> org.apache.flink.table.planner.plan.rules.logical.LogicalCorrelateToJoinFromTemporalTableFunctionRule.onMatch(LogicalCorrelateToJoinFromTemporalTableFunctionRule.scala:99)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:243)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202)
> at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkHepProgram.optimize(FlinkHepProgram.scala:69)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkHepRuleSetProgram.optimize(FlinkHepRuleSetProgram.scala:87)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram$$anonfun$optimize$1$$anonfun$apply$1.apply(FlinkGroupProgram.scala:63)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram$$anonfun$optimize$1$$anonfun$apply$1.apply(FlinkGroupProgram.scala:60)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:155)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:155)
> at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:155)
> at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram$$anonfun$optimize$1.apply(FlinkGroupProgram.scala:60)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram$$anonfun$optimize$1.apply(FlinkGroupProgram.scala:55)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:155)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:155)
> at
[jira] [Commented] (FLINK-22181) SourceNAryInputChainingITCase hangs on azure
[
https://issues.apache.org/jira/browse/FLINK-22181?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17319806#comment-17319806
]
Leonard Xu commented on FLINK-22181:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16398=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=5360d54c-8d94-5d85-304e-a89267eb785a=10600
> SourceNAryInputChainingITCase hangs on azure
>
>
> Key: FLINK-22181
> URL: https://issues.apache.org/jira/browse/FLINK-22181
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.13.0
>Reporter: Dawid Wysakowicz
>Priority: Major
> Labels: test-stability
> Fix For: 1.13.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16277=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2=11607
> {code}
> 2021-04-09T11:37:50.3331246Z "main" #1 prio=5 os_prio=0
> tid=0x7fae7000b800 nid=0x253e sleeping[0x7fae76c65000]
> 2021-04-09T11:37:50.3331592Zjava.lang.Thread.State: TIMED_WAITING
> (sleeping)
> 2021-04-09T11:37:50.3331883Z at java.lang.Thread.sleep(Native Method)
> 2021-04-09T11:37:50.3332313Z at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sleepBeforeRetry(CollectResultFetcher.java:229)
> 2021-04-09T11:37:50.3332976Z at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:111)
> 2021-04-09T11:37:50.577Z at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
> 2021-04-09T11:37:50.3334166Z at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> 2021-04-09T11:37:50.3334714Z at
> org.apache.flink.streaming.api.datastream.DataStreamUtils.collectBoundedStream(DataStreamUtils.java:106)
> 2021-04-09T11:37:50.3335356Z at
> org.apache.flink.test.streaming.runtime.SourceNAryInputChainingITCase.testMixedInputsWithMultipleUnionsExecution(SourceNAryInputChainingITCase.java:140)
> 2021-04-09T11:37:50.3335874Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2021-04-09T11:37:50.3336264Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2021-04-09T11:37:50.3336730Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2021-04-09T11:37:50.3337136Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2021-04-09T11:37:50.3337590Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2021-04-09T11:37:50.3338064Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2021-04-09T11:37:50.3338517Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2021-04-09T11:37:50.3339139Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2021-04-09T11:37:50.3339575Z at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2021-04-09T11:37:50.3339991Z at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2021-04-09T11:37:50.3340357Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2021-04-09T11:37:50.3340709Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2021-04-09T11:37:50.3341134Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2021-04-09T11:37:50.3341649Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2021-04-09T11:37:50.3342066Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2021-04-09T11:37:50.3342541Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2021-04-09T11:37:50.3342931Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2021-04-09T11:37:50.3343341Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2021-04-09T11:37:50.3343748Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2021-04-09T11:37:50.3344144Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2021-04-09T11:37:50.3344561Z at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2021-04-09T11:37:50.3344932Z at
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2021-04-09T11:37:50.3345295Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2021-04-09T11:37:50.3345710Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2021-04-09T11:37:50.3346177Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
>
[jira] [Comment Edited] (FLINK-21516) UpsertKafkaTableITCase.testTemporalJoin fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-21516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17319460#comment-17319460
]
Leonard Xu edited comment on FLINK-21516 at 4/12/21, 1:41 PM:
--
Yes, the same root cause with FLINK-21431, I've raised a PR
https://github.com/apache/flink/pull/15578 for these hang cases, I close this
ticket [~trohrmann].
was (Author: leonard xu):
Yes, the same root cause with FLINK-21431, I've raised a PR for FLINK-21431 ,
I close this ticket [~trohrmann].
> UpsertKafkaTableITCase.testTemporalJoin fails on AZP
>
>
> Key: FLINK-21516
> URL: https://issues.apache.org/jira/browse/FLINK-21516
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.13.0
>Reporter: Till Rohrmann
>Assignee: Leonard Xu
>Priority: Critical
> Labels: test-stability
> Fix For: 1.13.0
>
>
> The test case {{UpsertKafkaTableITCase.testTemporalJoin}} fails on AZP
> because it cannot find the job.
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13828=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5
> {code}
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could not find
> Flink job (3a751208d186784670fcf385ab9a23d1)
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
> at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestUtils.lambda$collectRows$0(KafkaTableTestUtils.java:60)
> ... 47 more
> Caused by: org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could
> not find Flink job (3a751208d186784670fcf385ab9a23d1)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$cancelJob$8(Dispatcher.java:539)
> at java.util.Optional.orElseGet(Optional.java:267)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.cancelJob(Dispatcher.java:536)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212)
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> at akka.actor.ActorCell.invoke(ActorCell.scala:561)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> at akka.dispatch.Mailbox.run(Mailbox.scala:225)
> at
[jira] [Resolved] (FLINK-21516) UpsertKafkaTableITCase.testTemporalJoin fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-21516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu resolved FLINK-21516.
Resolution: Duplicate
> UpsertKafkaTableITCase.testTemporalJoin fails on AZP
>
>
> Key: FLINK-21516
> URL: https://issues.apache.org/jira/browse/FLINK-21516
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.13.0
>Reporter: Till Rohrmann
>Assignee: Leonard Xu
>Priority: Critical
> Labels: test-stability
> Fix For: 1.13.0
>
>
> The test case {{UpsertKafkaTableITCase.testTemporalJoin}} fails on AZP
> because it cannot find the job.
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13828=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5
> {code}
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could not find
> Flink job (3a751208d186784670fcf385ab9a23d1)
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
> at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestUtils.lambda$collectRows$0(KafkaTableTestUtils.java:60)
> ... 47 more
> Caused by: org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could
> not find Flink job (3a751208d186784670fcf385ab9a23d1)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$cancelJob$8(Dispatcher.java:539)
> at java.util.Optional.orElseGet(Optional.java:267)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.cancelJob(Dispatcher.java:536)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212)
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> at akka.actor.ActorCell.invoke(ActorCell.scala:561)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> at akka.dispatch.Mailbox.run(Mailbox.scala:225)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at akka.dispatch.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
> at
>
[jira] [Commented] (FLINK-21516) UpsertKafkaTableITCase.testTemporalJoin fails on AZP
[
https://issues.apache.org/jira/browse/FLINK-21516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17319460#comment-17319460
]
Leonard Xu commented on FLINK-21516:
Yes, the same root cause with FLINK-21431, I've raised a PR for FLINK-21431 ,
I close this ticket [~trohrmann].
> UpsertKafkaTableITCase.testTemporalJoin fails on AZP
>
>
> Key: FLINK-21516
> URL: https://issues.apache.org/jira/browse/FLINK-21516
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.13.0
>Reporter: Till Rohrmann
>Assignee: Leonard Xu
>Priority: Critical
> Labels: test-stability
> Fix For: 1.13.0
>
>
> The test case {{UpsertKafkaTableITCase.testTemporalJoin}} fails on AZP
> because it cannot find the job.
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13828=logs=c5f0071e-1851-543e-9a45-9ac140befc32=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5
> {code}
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418)
> Caused by: java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could not find
> Flink job (3a751208d186784670fcf385ab9a23d1)
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
> at
> org.apache.flink.streaming.connectors.kafka.table.KafkaTableTestUtils.lambda$collectRows$0(KafkaTableTestUtils.java:60)
> ... 47 more
> Caused by: org.apache.flink.runtime.messages.FlinkJobNotFoundException: Could
> not find Flink job (3a751208d186784670fcf385ab9a23d1)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.lambda$cancelJob$8(Dispatcher.java:539)
> at java.util.Optional.orElseGet(Optional.java:267)
> at
> org.apache.flink.runtime.dispatcher.Dispatcher.cancelJob(Dispatcher.java:536)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcInvocation(AkkaRpcActor.java:305)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:212)
> at
> org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77)
> at
> org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:158)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
> at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
> at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
> at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
> at akka.actor.Actor$class.aroundReceive(Actor.scala:517)
> at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225)
> at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592)
> at akka.actor.ActorCell.invoke(ActorCell.scala:561)
> at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258)
> at akka.dispatch.Mailbox.run(Mailbox.scala:225)
> at akka.dispatch.Mailbox.exec(Mailbox.scala:235)
> at akka.dispatch.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
> at
> akka.dispatch.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
> at
[jira] [Updated] (FLINK-18206) The timestamp is displayed incorrectly
[ https://issues.apache.org/jira/browse/FLINK-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-18206: --- Priority: Critical (was: Blocker) > The timestamp is displayed incorrectly > --- > > Key: FLINK-18206 > URL: https://issues.apache.org/jira/browse/FLINK-18206 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.10.1 >Reporter: JasonLee >Assignee: Leonard Xu >Priority: Critical > Fix For: 1.13.0 > > Attachments: image-2021-04-09-12-11-29-049.png > > > I am using the latest Flink version. When I run a scrolling window SQL in SQL > client, the time stamp of the printed result will not be correct > > The results are as follows > > + jason 49 2 2020-06-09T07:59:40 2020-06-09T07:59:45 > + jason 50 2 2020-06-09T07:59:45 2020-06-09T07:59:50 > + jason 50 2 2020-06-09T07:59:50 2020-06-09T07:59:55 > + jason 50 2 2020-06-09T07:59:55 2020-06-09T08:00 > + jason 49 2 2020-06-09T08:00 2020-06-09T08:00:05 > + jason 50 2 2020-06-09T08:00:05 2020-06-09T08:00:10 > + jason 50 2 2020-06-09T08:00:10 2020-06-09T08:00:15 > + jason 50 2 2020-06-09T08:00:15 2020-06-09T08:00:20 > + jason 49 2 2020-06-09T08:00:20 2020-06-09T08:00:25 > + jason 50 2 2020-06-09T08:00:25 2020-06-09T08:00:30 > + jason 50 2 2020-06-09T08:00:30 2020-06-09T08:00:35 > + jason 49 2 2020-06-09T08:00:35 2020-06-09T08:00:40 > + jason 51 2 2020-06-09T08:00:40 2020-06-09T08:00:45 > + jason 50 2 2020-06-09T08:00:45 2020-06-09T08:00:50 > + jason 49 2 2020-06-09T08:00:50 2020-06-09T08:00:55 > + jason 50 2 2020-06-09T08:00:55 2020-06-09T08:01 > + jason 50 2 2020-06-09T08:01 2020-06-09T08:01:05 > + jason 51 2 2020-06-09T08:01:05 2020-06-09T08:01:10 > + jason 49 2 2020-06-09T08:01:10 2020-06-09T08:01:15 > + jason 46 2 2020-06-09T08:01:15 2020-06-09T08:01:20 > + jason 54 2 2020-06-09T08:01:20 2020-06-09T08:01:25 > + jason 50 2 2020-06-09T08:01:25 2020-06-09T08:01:30 > + jason 49 2 2020-06-09T08:01:30 2020-06-09T08:01:35 > + jason 50 2 2020-06-09T08:01:35 2020-06-09T08:01:40 > + jason 50 2 2020-06-09T08:01:40 2020-06-09T08:01:45 > + jason 50 2 2020-06-09T08:01:45 2020-06-09T08:01:50 > + jason 49 2 2020-06-09T08:01:50 2020-06-09T08:01:55 > + jason 50 2 2020-06-09T08:01:55 2020-06-09T08:02 > + jason 49 2 2020-06-09T08:02 2020-06-09T08:02:05 > + jason 51 2 2020-06-09T08:02:05 2020-06-09T08:02:10 > + jason 49 2 2020-06-09T08:02:10 2020-06-09T08:02:15 > + jason 50 2 2020-06-09T08:02:15 2020-06-09T08:02:20 > + jason 50 2 2020-06-09T08:02:20 2020-06-09T08:02:25 > + jason 50 2 2020-06-09T08:02:25 2020-06-09T08:02:30 > + jason 50 2 2020-06-09T08:02:30 2020-06-09T08:02:35 > + jason 50 2 2020-06-09T08:02:35 2020-06-09T08:02:40 > + jason 49 2 2020-06-09T08:02:40 2020-06-09T08:02:45 > + jason 50 2 2020-06-09T08:02:45 2020-06-09T08:02:50 > + jason 50 2 2020-06-09T08:02:50 2020-06-09T08:02:55 > + jason 50 2 2020-06-09T08:02:55 2020-06-09T08:03 > + jason 49 2 2020-06-09T08:03 2020-06-09T08:03:05 > + jason 51 2 2020-06-09T08:03:05 2020-06-09T08:03:10 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22179) Manual test time function changes and new time attributes support
Leonard Xu created FLINK-22179: -- Summary: Manual test time function changes and new time attributes support Key: FLINK-22179 URL: https://issues.apache.org/jira/browse/FLINK-22179 Project: Flink Issue Type: Sub-task Components: Table SQL / API Affects Versions: 1.13.0 Reporter: Leonard Xu Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22177) Add docs for consistent time functions and compatibility specification
Leonard Xu created FLINK-22177: -- Summary: Add docs for consistent time functions and compatibility specification Key: FLINK-22177 URL: https://issues.apache.org/jira/browse/FLINK-22177 Project: Flink Issue Type: Sub-task Components: Documentation, Table SQL / API Reporter: Leonard Xu Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21519) SQLClientHBaseITCase hangs on azure
[
https://issues.apache.org/jira/browse/FLINK-21519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317792#comment-17317792
]
Leonard Xu commented on FLINK-21519:
HBase has no official docker image and thus we can not use testcontainer :( ,
maybe we should find a better way to download source for e2e tests.
> SQLClientHBaseITCase hangs on azure
> ---
>
> Key: FLINK-21519
> URL: https://issues.apache.org/jira/browse/FLINK-21519
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase, Table SQL / Client
>Affects Versions: 1.13.0
>Reporter: Dawid Wysakowicz
>Assignee: Jane Chan
>Priority: Critical
> Labels: test-stability
>
> https://dev.azure.com/wysakowiczdawid/Flink/_build/results?buildId=707=logs=ae4f8708-9994-57d3-c2d7-b892156e7812=9401bf33-03c4-5a24-83fe-e51d75db73ef
> {code}
> Feb 26 13:58:15 [INFO] --- maven-surefire-plugin:2.22.1:test
> (end-to-end-tests) @ flink-end-to-end-tests-hbase ---
> Feb 26 13:58:15 [INFO]
> Feb 26 13:58:15 [INFO] ---
> Feb 26 13:58:15 [INFO] T E S T S
> Feb 26 13:58:15 [INFO] ---
> Feb 26 13:58:16 [INFO] Running
> org.apache.flink.tests.util.hbase.SQLClientHBaseITCase
> ==
> === WARNING: This E2E Run will time out in the next few minutes. Starting to
> upload the log output ===
> ==
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22122) StreamingKafkaITCase Fail due to TestTimedOutException
[
https://issues.apache.org/jira/browse/FLINK-22122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317765#comment-17317765
]
Leonard Xu commented on FLINK-22122:
another instance
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16241=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> StreamingKafkaITCase Fail due to TestTimedOutException
> --
>
> Key: FLINK-22122
> URL: https://issues.apache.org/jira/browse/FLINK-22122
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.13.0
>Reporter: Guowei Ma
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=16059=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529=27391
> {code:java}
> Apr 06 08:34:01 org.junit.runners.model.TestTimedOutException: test timed out
> after 3 minutes
> Apr 06 08:34:01 at java.lang.Object.wait(Native Method)
> Apr 06 08:34:01 at java.lang.Object.wait(Object.java:502)
> Apr 06 08:34:01 at java.lang.UNIXProcess.waitFor(UNIXProcess.java:395)
> Apr 06 08:34:01 at
> org.apache.flink.tests.util.flink.FlinkDistribution.submitJob(FlinkDistribution.java:194)
> Apr 06 08:34:01 at
> org.apache.flink.tests.util.flink.LocalStandaloneFlinkResource$StandaloneClusterController.submitJob(LocalStandaloneFlinkResource.java:200)
> Apr 06 08:34:01 at
> org.apache.flink.tests.util.kafka.StreamingKafkaITCase.testKafka(StreamingKafkaITCase.java:109)
> Apr 06 08:34:01 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Apr 06 08:34:01 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Apr 06 08:34:01 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Apr 06 08:34:01 at java.lang.reflect.Method.invoke(Method.java:498)
> Apr 06 08:34:01 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> Apr 06 08:34:01 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Apr 06 08:34:01 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> Apr 06 08:34:01 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Apr 06 08:34:01 at
> org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:48)
> Apr 06 08:34:01 at
> org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:48)
> Apr 06 08:34:01 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> Apr 06 08:34:01 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> Apr 06 08:34:01 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Apr 06 08:34:01 at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21345) NullPointerException LogicalCorrelateToJoinFromTemporalTableFunctionRule.scala:157
[
https://issues.apache.org/jira/browse/FLINK-21345?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated FLINK-21345:
---
Fix Version/s: 1.13.0
> NullPointerException
> LogicalCorrelateToJoinFromTemporalTableFunctionRule.scala:157
> --
>
> Key: FLINK-21345
> URL: https://issues.apache.org/jira/browse/FLINK-21345
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.12.1
> Environment: Planner: BlinkPlanner
> Flink Version: 1.12.1_2.11
> Java Version: 1.8
> OS: mac os
>Reporter: lynn1.zhang
>Priority: Minor
> Labels: pull-request-available
> Fix For: 1.13.0
>
> Attachments: image-2021-02-10-16-00-45-553.png
>
>
> First Step: Create 2 Source Tables as below:
> {code:java}
> CREATE TABLE test_streaming(
> vid BIGINT,
> ts BIGINT,
> proc AS proctime()
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'test-streaming',
> 'properties.bootstrap.servers' = '127.0.0.1:9092',
> 'scan.startup.mode' = 'latest-offset',
> 'format' = 'json'
> );
> CREATE TABLE test_streaming2(
> vid BIGINT,
> ts BIGINT,
> proc AS proctime()
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'test-streaming2',
> 'properties.bootstrap.servers' = '127.0.0.1:9092',
> 'scan.startup.mode' = 'latest-offset',
> 'format' = 'json'
> );
> {code}
> Second Step: Create a TEMPORARY Table Function, function name:dim, key:vid,
> timestamp:proctime()
> Third Step: test_streaming union all test_streaming2 join dim like below:
> {code:java}
> SELECT r.vid,d.name,timestamp_from_long(r.ts)
> FROM (
> SELECT * FROM test_streaming UNION ALL SELECT * FROM test_streaming2
> ) AS r,
> LATERAL TABLE (dim(r.proc)) AS d
> WHERE r.vid = d.vid;
> {code}
> Exception Detail: (if only use test-streaming or test-streaming2 join
> temporary table function, the program run ok)
> {code:java}
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.flink.table.planner.plan.rules.logical.LogicalCorrelateToJoinFromTemporalTableFunctionRule.getRelOptSchema(LogicalCorrelateToJoinFromTemporalTableFunctionRule.scala:157)
> at
> org.apache.flink.table.planner.plan.rules.logical.LogicalCorrelateToJoinFromTemporalTableFunctionRule.onMatch(LogicalCorrelateToJoinFromTemporalTableFunctionRule.scala:99)
> at
> org.apache.calcite.plan.AbstractRelOptPlanner.fireRule(AbstractRelOptPlanner.java:333)
> at org.apache.calcite.plan.hep.HepPlanner.applyRule(HepPlanner.java:542)
> at
> org.apache.calcite.plan.hep.HepPlanner.applyRules(HepPlanner.java:407)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeInstruction(HepPlanner.java:243)
> at
> org.apache.calcite.plan.hep.HepInstruction$RuleInstance.execute(HepInstruction.java:127)
> at
> org.apache.calcite.plan.hep.HepPlanner.executeProgram(HepPlanner.java:202)
> at
> org.apache.calcite.plan.hep.HepPlanner.findBestExp(HepPlanner.java:189)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkHepProgram.optimize(FlinkHepProgram.scala:69)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkHepRuleSetProgram.optimize(FlinkHepRuleSetProgram.scala:87)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram$$anonfun$optimize$1$$anonfun$apply$1.apply(FlinkGroupProgram.scala:63)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram$$anonfun$optimize$1$$anonfun$apply$1.apply(FlinkGroupProgram.scala:60)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:155)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:155)
> at scala.collection.Iterator$class.foreach(Iterator.scala:742)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1194)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:155)
> at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram$$anonfun$optimize$1.apply(FlinkGroupProgram.scala:60)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram$$anonfun$optimize$1.apply(FlinkGroupProgram.scala:55)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:155)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:155)
> at scala.collection.immutable.Range.foreach(Range.scala:166)
> at
>
[jira] [Comment Edited] (FLINK-18206) The timestamp is displayed incorrectly
[ https://issues.apache.org/jira/browse/FLINK-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317637#comment-17317637 ] Leonard Xu edited comment on FLINK-18206 at 4/9/21, 4:14 AM: - we should update the TIMESTAMP_LTZ display, the right display should be 2021-04-09 11:56:30 !image-2021-04-09-12-11-29-049.png! was (Author: leonard xu): update the TIMESTAMP_LTZ display, the right display should be 2021-04-01 11:56:30 !image-2021-04-09-12-11-29-049.png! > The timestamp is displayed incorrectly > --- > > Key: FLINK-18206 > URL: https://issues.apache.org/jira/browse/FLINK-18206 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.10.1 >Reporter: JasonLee >Assignee: Leonard Xu >Priority: Blocker > Fix For: 1.13.0 > > Attachments: image-2021-04-09-12-11-29-049.png > > > I am using the latest Flink version. When I run a scrolling window SQL in SQL > client, the time stamp of the printed result will not be correct > > The results are as follows > > + jason 49 2 2020-06-09T07:59:40 2020-06-09T07:59:45 > + jason 50 2 2020-06-09T07:59:45 2020-06-09T07:59:50 > + jason 50 2 2020-06-09T07:59:50 2020-06-09T07:59:55 > + jason 50 2 2020-06-09T07:59:55 2020-06-09T08:00 > + jason 49 2 2020-06-09T08:00 2020-06-09T08:00:05 > + jason 50 2 2020-06-09T08:00:05 2020-06-09T08:00:10 > + jason 50 2 2020-06-09T08:00:10 2020-06-09T08:00:15 > + jason 50 2 2020-06-09T08:00:15 2020-06-09T08:00:20 > + jason 49 2 2020-06-09T08:00:20 2020-06-09T08:00:25 > + jason 50 2 2020-06-09T08:00:25 2020-06-09T08:00:30 > + jason 50 2 2020-06-09T08:00:30 2020-06-09T08:00:35 > + jason 49 2 2020-06-09T08:00:35 2020-06-09T08:00:40 > + jason 51 2 2020-06-09T08:00:40 2020-06-09T08:00:45 > + jason 50 2 2020-06-09T08:00:45 2020-06-09T08:00:50 > + jason 49 2 2020-06-09T08:00:50 2020-06-09T08:00:55 > + jason 50 2 2020-06-09T08:00:55 2020-06-09T08:01 > + jason 50 2 2020-06-09T08:01 2020-06-09T08:01:05 > + jason 51 2 2020-06-09T08:01:05 2020-06-09T08:01:10 > + jason 49 2 2020-06-09T08:01:10 2020-06-09T08:01:15 > + jason 46 2 2020-06-09T08:01:15 2020-06-09T08:01:20 > + jason 54 2 2020-06-09T08:01:20 2020-06-09T08:01:25 > + jason 50 2 2020-06-09T08:01:25 2020-06-09T08:01:30 > + jason 49 2 2020-06-09T08:01:30 2020-06-09T08:01:35 > + jason 50 2 2020-06-09T08:01:35 2020-06-09T08:01:40 > + jason 50 2 2020-06-09T08:01:40 2020-06-09T08:01:45 > + jason 50 2 2020-06-09T08:01:45 2020-06-09T08:01:50 > + jason 49 2 2020-06-09T08:01:50 2020-06-09T08:01:55 > + jason 50 2 2020-06-09T08:01:55 2020-06-09T08:02 > + jason 49 2 2020-06-09T08:02 2020-06-09T08:02:05 > + jason 51 2 2020-06-09T08:02:05 2020-06-09T08:02:10 > + jason 49 2 2020-06-09T08:02:10 2020-06-09T08:02:15 > + jason 50 2 2020-06-09T08:02:15 2020-06-09T08:02:20 > + jason 50 2 2020-06-09T08:02:20 2020-06-09T08:02:25 > + jason 50 2 2020-06-09T08:02:25 2020-06-09T08:02:30 > + jason 50 2 2020-06-09T08:02:30 2020-06-09T08:02:35 > + jason 50 2 2020-06-09T08:02:35 2020-06-09T08:02:40 > + jason 49 2 2020-06-09T08:02:40 2020-06-09T08:02:45 > + jason 50 2 2020-06-09T08:02:45 2020-06-09T08:02:50 > + jason 50 2 2020-06-09T08:02:50 2020-06-09T08:02:55 > + jason 50 2 2020-06-09T08:02:55 2020-06-09T08:03 > + jason 49 2 2020-06-09T08:03 2020-06-09T08:03:05 > + jason 51 2 2020-06-09T08:03:05 2020-06-09T08:03:10 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18206) The timestamp is displayed incorrectly
[ https://issues.apache.org/jira/browse/FLINK-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17317637#comment-17317637 ] Leonard Xu commented on FLINK-18206: update the TIMESTAMP_LTZ display, the right display should be 2021-04-01 11:56:30 !image-2021-04-09-12-11-29-049.png! > The timestamp is displayed incorrectly > --- > > Key: FLINK-18206 > URL: https://issues.apache.org/jira/browse/FLINK-18206 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.10.1 >Reporter: JasonLee >Assignee: Leonard Xu >Priority: Blocker > Fix For: 1.13.0 > > Attachments: image-2021-04-09-12-11-29-049.png > > > I am using the latest Flink version. When I run a scrolling window SQL in SQL > client, the time stamp of the printed result will not be correct > > The results are as follows > > + jason 49 2 2020-06-09T07:59:40 2020-06-09T07:59:45 > + jason 50 2 2020-06-09T07:59:45 2020-06-09T07:59:50 > + jason 50 2 2020-06-09T07:59:50 2020-06-09T07:59:55 > + jason 50 2 2020-06-09T07:59:55 2020-06-09T08:00 > + jason 49 2 2020-06-09T08:00 2020-06-09T08:00:05 > + jason 50 2 2020-06-09T08:00:05 2020-06-09T08:00:10 > + jason 50 2 2020-06-09T08:00:10 2020-06-09T08:00:15 > + jason 50 2 2020-06-09T08:00:15 2020-06-09T08:00:20 > + jason 49 2 2020-06-09T08:00:20 2020-06-09T08:00:25 > + jason 50 2 2020-06-09T08:00:25 2020-06-09T08:00:30 > + jason 50 2 2020-06-09T08:00:30 2020-06-09T08:00:35 > + jason 49 2 2020-06-09T08:00:35 2020-06-09T08:00:40 > + jason 51 2 2020-06-09T08:00:40 2020-06-09T08:00:45 > + jason 50 2 2020-06-09T08:00:45 2020-06-09T08:00:50 > + jason 49 2 2020-06-09T08:00:50 2020-06-09T08:00:55 > + jason 50 2 2020-06-09T08:00:55 2020-06-09T08:01 > + jason 50 2 2020-06-09T08:01 2020-06-09T08:01:05 > + jason 51 2 2020-06-09T08:01:05 2020-06-09T08:01:10 > + jason 49 2 2020-06-09T08:01:10 2020-06-09T08:01:15 > + jason 46 2 2020-06-09T08:01:15 2020-06-09T08:01:20 > + jason 54 2 2020-06-09T08:01:20 2020-06-09T08:01:25 > + jason 50 2 2020-06-09T08:01:25 2020-06-09T08:01:30 > + jason 49 2 2020-06-09T08:01:30 2020-06-09T08:01:35 > + jason 50 2 2020-06-09T08:01:35 2020-06-09T08:01:40 > + jason 50 2 2020-06-09T08:01:40 2020-06-09T08:01:45 > + jason 50 2 2020-06-09T08:01:45 2020-06-09T08:01:50 > + jason 49 2 2020-06-09T08:01:50 2020-06-09T08:01:55 > + jason 50 2 2020-06-09T08:01:55 2020-06-09T08:02 > + jason 49 2 2020-06-09T08:02 2020-06-09T08:02:05 > + jason 51 2 2020-06-09T08:02:05 2020-06-09T08:02:10 > + jason 49 2 2020-06-09T08:02:10 2020-06-09T08:02:15 > + jason 50 2 2020-06-09T08:02:15 2020-06-09T08:02:20 > + jason 50 2 2020-06-09T08:02:20 2020-06-09T08:02:25 > + jason 50 2 2020-06-09T08:02:25 2020-06-09T08:02:30 > + jason 50 2 2020-06-09T08:02:30 2020-06-09T08:02:35 > + jason 50 2 2020-06-09T08:02:35 2020-06-09T08:02:40 > + jason 49 2 2020-06-09T08:02:40 2020-06-09T08:02:45 > + jason 50 2 2020-06-09T08:02:45 2020-06-09T08:02:50 > + jason 50 2 2020-06-09T08:02:50 2020-06-09T08:02:55 > + jason 50 2 2020-06-09T08:02:55 2020-06-09T08:03 > + jason 49 2 2020-06-09T08:03 2020-06-09T08:03:05 > + jason 51 2 2020-06-09T08:03:05 2020-06-09T08:03:10 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18206) The timestamp is displayed incorrectly
[ https://issues.apache.org/jira/browse/FLINK-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-18206: --- Attachment: image-2021-04-09-12-11-29-049.png > The timestamp is displayed incorrectly > --- > > Key: FLINK-18206 > URL: https://issues.apache.org/jira/browse/FLINK-18206 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.10.1 >Reporter: JasonLee >Assignee: Leonard Xu >Priority: Blocker > Fix For: 1.13.0 > > Attachments: image-2021-04-09-12-11-29-049.png > > > I am using the latest Flink version. When I run a scrolling window SQL in SQL > client, the time stamp of the printed result will not be correct > > The results are as follows > > + jason 49 2 2020-06-09T07:59:40 2020-06-09T07:59:45 > + jason 50 2 2020-06-09T07:59:45 2020-06-09T07:59:50 > + jason 50 2 2020-06-09T07:59:50 2020-06-09T07:59:55 > + jason 50 2 2020-06-09T07:59:55 2020-06-09T08:00 > + jason 49 2 2020-06-09T08:00 2020-06-09T08:00:05 > + jason 50 2 2020-06-09T08:00:05 2020-06-09T08:00:10 > + jason 50 2 2020-06-09T08:00:10 2020-06-09T08:00:15 > + jason 50 2 2020-06-09T08:00:15 2020-06-09T08:00:20 > + jason 49 2 2020-06-09T08:00:20 2020-06-09T08:00:25 > + jason 50 2 2020-06-09T08:00:25 2020-06-09T08:00:30 > + jason 50 2 2020-06-09T08:00:30 2020-06-09T08:00:35 > + jason 49 2 2020-06-09T08:00:35 2020-06-09T08:00:40 > + jason 51 2 2020-06-09T08:00:40 2020-06-09T08:00:45 > + jason 50 2 2020-06-09T08:00:45 2020-06-09T08:00:50 > + jason 49 2 2020-06-09T08:00:50 2020-06-09T08:00:55 > + jason 50 2 2020-06-09T08:00:55 2020-06-09T08:01 > + jason 50 2 2020-06-09T08:01 2020-06-09T08:01:05 > + jason 51 2 2020-06-09T08:01:05 2020-06-09T08:01:10 > + jason 49 2 2020-06-09T08:01:10 2020-06-09T08:01:15 > + jason 46 2 2020-06-09T08:01:15 2020-06-09T08:01:20 > + jason 54 2 2020-06-09T08:01:20 2020-06-09T08:01:25 > + jason 50 2 2020-06-09T08:01:25 2020-06-09T08:01:30 > + jason 49 2 2020-06-09T08:01:30 2020-06-09T08:01:35 > + jason 50 2 2020-06-09T08:01:35 2020-06-09T08:01:40 > + jason 50 2 2020-06-09T08:01:40 2020-06-09T08:01:45 > + jason 50 2 2020-06-09T08:01:45 2020-06-09T08:01:50 > + jason 49 2 2020-06-09T08:01:50 2020-06-09T08:01:55 > + jason 50 2 2020-06-09T08:01:55 2020-06-09T08:02 > + jason 49 2 2020-06-09T08:02 2020-06-09T08:02:05 > + jason 51 2 2020-06-09T08:02:05 2020-06-09T08:02:10 > + jason 49 2 2020-06-09T08:02:10 2020-06-09T08:02:15 > + jason 50 2 2020-06-09T08:02:15 2020-06-09T08:02:20 > + jason 50 2 2020-06-09T08:02:20 2020-06-09T08:02:25 > + jason 50 2 2020-06-09T08:02:25 2020-06-09T08:02:30 > + jason 50 2 2020-06-09T08:02:30 2020-06-09T08:02:35 > + jason 50 2 2020-06-09T08:02:35 2020-06-09T08:02:40 > + jason 49 2 2020-06-09T08:02:40 2020-06-09T08:02:45 > + jason 50 2 2020-06-09T08:02:45 2020-06-09T08:02:50 > + jason 50 2 2020-06-09T08:02:50 2020-06-09T08:02:55 > + jason 50 2 2020-06-09T08:02:55 2020-06-09T08:03 > + jason 49 2 2020-06-09T08:03 2020-06-09T08:03:05 > + jason 51 2 2020-06-09T08:03:05 2020-06-09T08:03:10 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-20105) Temporal Table does not work when Kafka is used as the versioned side (planner PK problem)
[
https://issues.apache.org/jira/browse/FLINK-20105?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu closed FLINK-20105.
--
Fix Version/s: (was: 1.13.0)
Resolution: Not A Problem
After discussion, this should not a problem, feel free to reopen
> Temporal Table does not work when Kafka is used as the versioned side
> (planner PK problem)
> --
>
> Key: FLINK-20105
> URL: https://issues.apache.org/jira/browse/FLINK-20105
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / Planner
>Affects Versions: 1.12.0
>Reporter: Benoît Paris
>Assignee: Leonard Xu
>Priority: Minor
> Attachments: flink-test-temporal-table.zip
>
>
>
> This is probably an early bug report, that I'm making before 0.12 is out.
> In 1.12-SNAPSHOT, doing this:
>
> {code:java}
> INSERT INTO output_table
> SELECT
> o.f_sequence * r.f_int,
> o.f_random_str
> FROM datagen_1 AS o
> LEFT JOIN input_2 FOR SYSTEM_TIME AS OF o.ts r
> ON o.f_random = r.f_int_pk{code}
>
> works when input_2 is build with datagen, but fails when data comes from
> kafka; yielding the following error that comes from planner code:
>
> {code:java}
> Type INT NOT NULL of table field 'f_int_pk' does not match with the physical
> type INT of the 'f_int_pk' field of the TableSource return type.{code}
>
> Included is code for a complete reproduction, with:
> * docker-compose file for ZooKeeper and Kafka (latest)
> * pom.xml
> * OK_TempTableSQLTestDatagen.java: it works with the datagen
> * KO_TempTableSQLTestKafka.java: fails with Kafka
> * KO_TempTableSQLTestKafkaNoPK.java: I tried to have no PK, it fails
> * KO_TempTableSQLTestKafkaNull.java: I tried with the PK being nullable, it
> fails
> * KO_TempTableSQLTestKafkaNullif.java: I tried with using the PK in a
> NULLIF(pk, '') as [advertised
> here|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Inserting-nullable-data-into-NOT-NULL-columns-td34198.html],
> but if fails (loses PK powers)
> I just can't think of a workaround. I even tried to GROUP BY on the PK.
> From memory, the Temporal Table Function suffers from a similar problem; My
> usual workaround being to do a
> COALESCE(problematic_temp_table_function_primary_key, null), but it fails
> here as well (interestingly, it does not fail because of losing PK powers,
> but because of the NOT NULL planner difference).
> It seems like the same problem of having transformations of the same field
> being NULL and NOT NULL between planner transformations.
>
>
> This isssue is probably related to the ongoing FLIP-132 Temporal Table DDL
> and Temporal Table Join developments.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-22063) Lookup Join outputs wrong results for some scenario
[
https://issues.apache.org/jira/browse/FLINK-22063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17316816#comment-17316816
]
Leonard Xu commented on FLINK-22063:
Hi, [~icshuo] Do you have any progress?
> Lookup Join outputs wrong results for some scenario
> ---
>
> Key: FLINK-22063
> URL: https://issues.apache.org/jira/browse/FLINK-22063
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.12.2
>Reporter: Shuo Cheng
>Assignee: Shuo Cheng
>Priority: Critical
> Fix For: 1.13.0
>
>
> Reproduce the bug as following:
> In LookupJoinITCase, given the sql
> {code:sql}
> SELECT
> T.id, T.len, D.id, T.content, D.name
> FROM src AS T JOIN user_table for system_time as of T.proctime AS D
> ON T.id = D.id and cast(T.len as bigint) = D.id
> {code}
> the following execution plan is generated:
> {code:java}
> LegacySink(name=[DataStreamTableSink], fields=[id, len, id0, content, name])
> +- Calc(select=[id, len, id0, content, name])
>+- LookupJoin(table=[**], joinType=[InnerJoin], async=[false],
> lookup=[id=len0], select=[id, len, content, len0, id, name])
> +- Calc(select=[id, len, content, CAST(len) AS len0])
> +- TableSourceScan(table=[[**]], fields=[id, len, content])
> {code}
> As we can see, the condition `T.id = D.id` is lost, so a wrong result may be
> produced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-21557) An error was reported when temp join hive.
[ https://issues.apache.org/jira/browse/FLINK-21557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu closed FLINK-21557. -- Resolution: Duplicate > An error was reported when temp join hive. > -- > > Key: FLINK-21557 > URL: https://issues.apache.org/jira/browse/FLINK-21557 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.12.0 >Reporter: HideOnBush >Assignee: Leonard Xu >Priority: Major > Fix For: 1.13.0 > > Attachments: image-2021-03-02-18-38-43-789.png > > > An error was reported when temp join hive.。 > Caused by: java.lang.NullPointerException > at > org.apache.flink.table.data.DecimalDataUtils.doubleValue(DecimalDataUtils.java:48) > at > org.apache.flink.table.data.DecimalDataUtils.castToDouble(DecimalDataUtils.java:193) > at StreamExecCalc$3741.processElement(Unknown Source) > at > org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71) > !image-2021-03-02-18-38-43-789.png|thumbnail! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-14036) function log(f0,f1) in Table API do not support decimal type
[
https://issues.apache.org/jira/browse/FLINK-14036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu closed FLINK-14036.
--
Resolution: Won't Fix
> function log(f0,f1) in Table API do not support decimal type
> ---
>
> Key: FLINK-14036
> URL: https://issues.apache.org/jira/browse/FLINK-14036
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API, Table SQL / Planner
>Affects Versions: 1.9.0
>Reporter: Leonard Xu
>Priority: Major
>
> function log(f0,f1) in Table API module do not support decimal type, but it
> works in Table SQL module. The following code will run fail:
>
> {code:java}
> testTableApi(
> 'f0.log(f1),
> "log(f0,f1)",
> "2.0")
> override def testData: Row = {
> val testData = new Row(2)
> testData.setField(0, BigDecimal("3").bigDecimal)
> testData.setField(1, 9)
> testData
> }
>
> override def typeInfo: RowTypeInfo = {
> new RowTypeInfo(
> /* 0 */ fromLogicalTypeToTypeInfo(DECIMAL(1, 0)),
> /* 1 */ Types.INT
> )
> }{code}
>
> The real cause is that the return type of *log()* function must be Double
> type，planner will cast all oprands' type to Double Type before function
> execution, however *org.apache.flink.table.planner.typeutils.TypeCoercion*
> can not yet cast Decimal type to Double type。
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-19074) Materialize timeindicator in the right input of temporal join
[ https://issues.apache.org/jira/browse/FLINK-19074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu closed FLINK-19074. -- Resolution: Implemented > Materialize timeindicator in the right input of temporal join > - > > Key: FLINK-19074 > URL: https://issues.apache.org/jira/browse/FLINK-19074 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Reporter: Leonard Xu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-19075) Infer changelog trait for temporal join node
[ https://issues.apache.org/jira/browse/FLINK-19075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu closed FLINK-19075. -- Resolution: Not A Problem > Infer changelog trait for temporal join node > - > > Key: FLINK-19075 > URL: https://issues.apache.org/jira/browse/FLINK-19075 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Reporter: Leonard Xu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22074) testRequirementCheckOnlyTriggeredOnce(org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest) failed
[ https://issues.apache.org/jira/browse/FLINK-22074?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17312513#comment-17312513 ] Leonard Xu commented on FLINK-22074: failed link https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=15895=logs=a57e0635-3fad-5b08-57c7-a4142d7d6fa9=5360d54c-8d94-5d85-304e-a89267eb785a=6000 > testRequirementCheckOnlyTriggeredOnce(org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest) > failed > - > > Key: FLINK-22074 > URL: https://issues.apache.org/jira/browse/FLINK-22074 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.13.0 >Reporter: Leonard Xu >Priority: Major > Fix For: 1.13.0 > > > [ERROR] Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: > 1.654 s <<< FAILURE! - in > org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest > > [ERROR] > testRequirementCheckOnlyTriggeredOnce(org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest) > Time elapsed: 0.059 s <<< FAILURE! > java.lang.AssertionError: Expected to fail with a timeout. > at org.junit.Assert.fail(Assert.java:88) > at > org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTestBase.assertFutureNotComplete(FineGrainedSlotManagerTestBase.java:126) > > at > org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest$10.lambda$new$3(FineGrainedSlotManagerTest.java:605) > > at > org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTestBase$Context.runTest(FineGrainedSlotManagerTestBase.java:197) > > at > org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest$10.(FineGrainedSlotManagerTest.java:581) > > at > org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest.testRequirementCheckOnlyTriggeredOnce(FineGrainedSlotManagerTest.java:565) > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) > > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) > > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > > at > org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) > at org.junit.rules.RunRules.evaluate(RunRules.java:20) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) > > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) > > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) > at org.junit.runners.ParentRunner.run(ParentRunner.java:363) > at > org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) > > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) > > at > org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) > > at > org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) > > at > org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) > > at > org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) > > at > org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) > at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-22074) testRequirementCheckOnlyTriggeredOnce(org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest) failed
Leonard Xu created FLINK-22074: -- Summary: testRequirementCheckOnlyTriggeredOnce(org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest) failed Key: FLINK-22074 URL: https://issues.apache.org/jira/browse/FLINK-22074 Project: Flink Issue Type: Bug Components: Tests Affects Versions: 1.13.0 Reporter: Leonard Xu Fix For: 1.13.0 [ERROR] Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.654 s <<< FAILURE! - in org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest [ERROR] testRequirementCheckOnlyTriggeredOnce(org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest) Time elapsed: 0.059 s <<< FAILURE! java.lang.AssertionError: Expected to fail with a timeout. at org.junit.Assert.fail(Assert.java:88) at org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTestBase.assertFutureNotComplete(FineGrainedSlotManagerTestBase.java:126) at org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest$10.lambda$new$3(FineGrainedSlotManagerTest.java:605) at org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTestBase$Context.runTest(FineGrainedSlotManagerTestBase.java:197) at org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest$10.(FineGrainedSlotManagerTest.java:581) at org.apache.flink.runtime.resourcemanager.slotmanager.FineGrainedSlotManagerTest.testRequirementCheckOnlyTriggeredOnce(FineGrainedSlotManagerTest.java:565) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:498) at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50) at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47) at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) at org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45) at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55) at org.junit.rules.RunRules.evaluate(RunRules.java:20) at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78) at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57) at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at org.junit.runners.ParentRunner.run(ParentRunner.java:363) at org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365) at org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273) at org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238) at org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159) at org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384) at org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345) at org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:126) at org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:418) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-22005) SQL Client end-to-end test (Old planner) Elasticsearch (v7.5.1)
[ https://issues.apache.org/jira/browse/FLINK-22005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17310391#comment-17310391 ] Leonard Xu commented on FLINK-22005: [~maguowei] Could you rebase to latest master, this issue has been fixed in [https://github.com/apache/flink/pull/15394#event-4516849115] > SQL Client end-to-end test (Old planner) Elasticsearch (v7.5.1) > > > Key: FLINK-22005 > URL: https://issues.apache.org/jira/browse/FLINK-22005 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.13.0 >Reporter: Guowei Ma >Priority: Major > Labels: test-stability > > The test fail because of Waiting for Elasticsearch records indefinitely. > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=15583=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529=19826 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21713) Correct function CURRENT_TIMESTAMP/CURRENT_TIME/CURRENT_DATE/NOW/LOCALTIME/LOCALTIMESTAMP
[ https://issues.apache.org/jira/browse/FLINK-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-21713: --- Environment: * 1.The CURRENT_TIMESTAMP function returned timestamp of UTC+0 timezone with TIMESTAMP type, now it's corrected to return timestamp of local time zone with TIMESTAMP_LTZ type. * 2. Time function LOCALTIME, LOCALTIMESTAMP, CURRENT_DATE, CURRENT_TIME, CURRENT_TIMESTAMP, NOW() were evaluated for per record no matter the job is stream or batch. Now, it is corrected to evaluate for per record in stream job, evaluate once at query-start for batch job. was: * 1.The CURRENT_TIMESTAMP function returned timestamp of UTC+0 timezone with TIMESTAMP type, now it's corrected to return timestamp of local time zone with TIMESTAMP_LTZ type. * 2. Time function LOCALTIME, LOCALTIMESTAMP, CURRENT_DATE, CURRENT_TIME, CURRENT_TIMESTAMP, NOW() was evaluated for per record no matter the job is stream or batch. Now, it is corrected to evaluate for per record in stream job, evaluate once at query-start for batch job. > Correct function > CURRENT_TIMESTAMP/CURRENT_TIME/CURRENT_DATE/NOW/LOCALTIME/LOCALTIMESTAMP > - > > Key: FLINK-21713 > URL: https://issues.apache.org/jira/browse/FLINK-21713 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API > Environment: * 1.The CURRENT_TIMESTAMP function returned timestamp of > UTC+0 timezone with TIMESTAMP type, now it's corrected to return timestamp of > local time zone with TIMESTAMP_LTZ type. > * 2. Time function LOCALTIME, LOCALTIMESTAMP, CURRENT_DATE, CURRENT_TIME, > CURRENT_TIMESTAMP, NOW() were evaluated for per record no matter the job is > stream or batch. Now, it is corrected to evaluate for per record in stream > job, evaluate once at query-start for batch job. >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > 1. fix the return value type as well the return value for following time > functions > > |*function*|*existed problem*|*current behavior*|*proposed changes*| > |CURRENT_DATE|returns UTC date, but user expects current date in session time > zone|return type: DATE > #session timezone: UTC > 2020-12-28 > #session timezone: UTC+8 > 2020-12-28| return current date in session time zone, the return type should > be *DATE* > #session timezone: UTC > 2020-12-28 > #session timezone: UTC+8 > 2020-12-29| > |CURRENT_TIME|returns UTC time, but user expects current time in session time > zone|return type: TIME > #session timezone: UTC > 23:52:52 > #session timezone: UTC+8 > 23:52:52|return current time in session time zone, the return type should be > *TIME* > #session timezone: UTC > 23:52:52 > #session timezone: UTC+8 > 07:52:52| > |CURRENT_TIMESTAMP|returns UTC timestamp, but user expects current timestamp > in session time zone| > return type: TIMESTAMP > #session timezone: UTC > 2020-12-28 23:52:52 > #session timezone: UTC+8 > 2020-12-28 23:52:52|return current timestamp in session time zone, the return > type should be > *TIMESTAMP WITH LOCAL TIME ZONE* > #session timezone: UTC > 2020-12-28 23:52:52 > #session timezone: UTC+8 > 2020-12-29 07:52:52| > |NOW()|returns UTC timestamp, but user expects current timestamp in session > time zone| > return type: TIMESTAMP > #session timezone: UTC > 2020-12-28 23:52:52 > #session timezone: UTC+8 > 2020-12-28 23:52:52|return current timestamp in session time zone, the return > type should be > *TIMESTAMP WITH LOCAL TIME ZONE* > #session timezone: UTC > 2020-12-28 23:52:52 > #session timezone: UTC+8 > 2020-12-29 07:52:52| > > 2. The following functions' evaluation is depend on execution mode > * LOCALTIME > * LOCALTIMESTAMP > * CURRENT_DATE > * CURRENT_TIME > * CURRENT_TIMESTAMP > * NOW() > Flink evaluates above time function values according to execution mode, i.e. > Flink evaluates time function value for row level in Streaming mode, > evaluates the time function value at query start for batch mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21713) Correct function CURRENT_TIMESTAMP/CURRENT_TIME/CURRENT_DATE/NOW/LOCALTIME/LOCALTIMESTAMP
[ https://issues.apache.org/jira/browse/FLINK-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-21713: --- Release Note: Environment: * 1.The CURRENT_TIMESTAMP function returned timestamp of UTC+0 timezone with TIMESTAMP type, now it's corrected to return timestamp of local time zone with TIMESTAMP_LTZ type. * 2. Time function LOCALTIME, LOCALTIMESTAMP, CURRENT_DATE, CURRENT_TIME, CURRENT_TIMESTAMP, NOW() was evaluated for per record no matter the job is stream or batch. Now, it is corrected to evaluate for per record in stream job, evaluate once at query-start for batch job. > Correct function > CURRENT_TIMESTAMP/CURRENT_TIME/CURRENT_DATE/NOW/LOCALTIME/LOCALTIMESTAMP > - > > Key: FLINK-21713 > URL: https://issues.apache.org/jira/browse/FLINK-21713 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API > Environment: * 1.The CURRENT_TIMESTAMP function returned timestamp of > UTC+0 timezone with TIMESTAMP type, now it's corrected to return timestamp of > local time zone with TIMESTAMP_LTZ type. > * 2. Time function LOCALTIME, LOCALTIMESTAMP, CURRENT_DATE, CURRENT_TIME, > CURRENT_TIMESTAMP, NOW() was evaluated for per record no matter the job is > stream or batch. Now, it is corrected to evaluate for per record in stream > job, evaluate once at query-start for batch job. >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > 1. fix the return value type as well the return value for following time > functions > > |*function*|*existed problem*|*current behavior*|*proposed changes*| > |CURRENT_DATE|returns UTC date, but user expects current date in session time > zone|return type: DATE > #session timezone: UTC > 2020-12-28 > #session timezone: UTC+8 > 2020-12-28| return current date in session time zone, the return type should > be *DATE* > #session timezone: UTC > 2020-12-28 > #session timezone: UTC+8 > 2020-12-29| > |CURRENT_TIME|returns UTC time, but user expects current time in session time > zone|return type: TIME > #session timezone: UTC > 23:52:52 > #session timezone: UTC+8 > 23:52:52|return current time in session time zone, the return type should be > *TIME* > #session timezone: UTC > 23:52:52 > #session timezone: UTC+8 > 07:52:52| > |CURRENT_TIMESTAMP|returns UTC timestamp, but user expects current timestamp > in session time zone| > return type: TIMESTAMP > #session timezone: UTC > 2020-12-28 23:52:52 > #session timezone: UTC+8 > 2020-12-28 23:52:52|return current timestamp in session time zone, the return > type should be > *TIMESTAMP WITH LOCAL TIME ZONE* > #session timezone: UTC > 2020-12-28 23:52:52 > #session timezone: UTC+8 > 2020-12-29 07:52:52| > |NOW()|returns UTC timestamp, but user expects current timestamp in session > time zone| > return type: TIMESTAMP > #session timezone: UTC > 2020-12-28 23:52:52 > #session timezone: UTC+8 > 2020-12-28 23:52:52|return current timestamp in session time zone, the return > type should be > *TIMESTAMP WITH LOCAL TIME ZONE* > #session timezone: UTC > 2020-12-28 23:52:52 > #session timezone: UTC+8 > 2020-12-29 07:52:52| > > 2. The following functions' evaluation is depend on execution mode > * LOCALTIME > * LOCALTIMESTAMP > * CURRENT_DATE > * CURRENT_TIME > * CURRENT_TIMESTAMP > * NOW() > Flink evaluates above time function values according to execution mode, i.e. > Flink evaluates time function value for row level in Streaming mode, > evaluates the time function value at query start for batch mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21623) Introduce CURRENT_ROW_TIMESTAMP() function
[ https://issues.apache.org/jira/browse/FLINK-21623?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-21623: --- Summary: Introduce CURRENT_ROW_TIMESTAMP() function (was: Introduce CURRENT_ROW_TIMESTAMP function) > Introduce CURRENT_ROW_TIMESTAMP() function > -- > > Key: FLINK-21623 > URL: https://issues.apache.org/jira/browse/FLINK-21623 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Table SQL / Planner >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21713) Correct function CURRENT_TIMESTAMP/CURRENT_TIME/CURRENT_DATE/NOW/LOCALTIME/LOCALTIMESTAMP
[ https://issues.apache.org/jira/browse/FLINK-21713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309467#comment-17309467 ] Leonard Xu commented on FLINK-21713: Okay, i'll write a note for this > Correct function > CURRENT_TIMESTAMP/CURRENT_TIME/CURRENT_DATE/NOW/LOCALTIME/LOCALTIMESTAMP > - > > Key: FLINK-21713 > URL: https://issues.apache.org/jira/browse/FLINK-21713 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > 1. fix the return value type as well the return value for following time > functions > > |*function*|*existed problem*|*current behavior*|*proposed changes*| > |CURRENT_DATE|returns UTC date, but user expects current date in session time > zone|return type: DATE > #session timezone: UTC > 2020-12-28 > #session timezone: UTC+8 > 2020-12-28| return current date in session time zone, the return type should > be *DATE* > #session timezone: UTC > 2020-12-28 > #session timezone: UTC+8 > 2020-12-29| > |CURRENT_TIME|returns UTC time, but user expects current time in session time > zone|return type: TIME > #session timezone: UTC > 23:52:52 > #session timezone: UTC+8 > 23:52:52|return current time in session time zone, the return type should be > *TIME* > #session timezone: UTC > 23:52:52 > #session timezone: UTC+8 > 07:52:52| > |CURRENT_TIMESTAMP|returns UTC timestamp, but user expects current timestamp > in session time zone| > return type: TIMESTAMP > #session timezone: UTC > 2020-12-28 23:52:52 > #session timezone: UTC+8 > 2020-12-28 23:52:52|return current timestamp in session time zone, the return > type should be > *TIMESTAMP WITH LOCAL TIME ZONE* > #session timezone: UTC > 2020-12-28 23:52:52 > #session timezone: UTC+8 > 2020-12-29 07:52:52| > |NOW()|returns UTC timestamp, but user expects current timestamp in session > time zone| > return type: TIMESTAMP > #session timezone: UTC > 2020-12-28 23:52:52 > #session timezone: UTC+8 > 2020-12-28 23:52:52|return current timestamp in session time zone, the return > type should be > *TIMESTAMP WITH LOCAL TIME ZONE* > #session timezone: UTC > 2020-12-28 23:52:52 > #session timezone: UTC+8 > 2020-12-29 07:52:52| > > 2. The following functions' evaluation is depend on execution mode > * LOCALTIME > * LOCALTIMESTAMP > * CURRENT_DATE > * CURRENT_TIME > * CURRENT_TIMESTAMP > * NOW() > Flink evaluates above time function values according to execution mode, i.e. > Flink evaluates time function value for row level in Streaming mode, > evaluates the time function value at query start for batch mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-21978) Disable cast conversion between Numeric type and TIMESTAMP_LTZ type
[ https://issues.apache.org/jira/browse/FLINK-21978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309437#comment-17309437 ] Leonard Xu edited comment on FLINK-21978 at 3/26/21, 1:49 PM: -- [~twalthr], Snowflake supports cast conversion between INT/BIGINT and TIMESTAMP_LTZ, both INT value and BIGINT value are treated as seconds, and Oracle doesn't support the cast conversion between INT/BIGINT and TIMESTAMP WITH LOCAL TIME ZONE. The conservative approach looks better base on this. was (Author: leonard xu): [~twalthr], Snowflake supports cast conversion between INT/BIGINT and TIMESTAMP, both INT value and BIGINT value are treated as seconds, and Oracle doesn't support the cast conversion between INT/BIGINT and TIMESTAMP. The conservative approach looks better base on this. > Disable cast conversion between Numeric type and TIMESTAMP_LTZ type > --- > > Key: FLINK-21978 > URL: https://issues.apache.org/jira/browse/FLINK-21978 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.13.0 >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > Labels: pull-request-available > > Currently we has supported the cast conversion between Numeric type and > TIMESTAMP_LTZ type, we suppose the numeric value e.g `Long type 1000L` as > epoch seconds and then cast to TIMESTAMP_LTZ, but the java.lang.Long is a > conversion class of `LocalZonedTimestampType` and treats as milliseconds. > To avoid the inconsistency, we should disable it and encourage user to use > `TO_TIMESTAMP_LTZ(numeric, precisoon)` function. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21978) Disable cast conversion between Numeric type and TIMESTAMP_LTZ type
[ https://issues.apache.org/jira/browse/FLINK-21978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309437#comment-17309437 ] Leonard Xu commented on FLINK-21978: [~twalthr], Snowflake supports cast conversion between INT/BIGINT and TIMESTAMP, both INT value and BIGINT value are treated as seconds, and Oracle doesn't support the cast conversion between INT/BIGINT and TIMESTAMP. The conservative approach looks better base on this. > Disable cast conversion between Numeric type and TIMESTAMP_LTZ type > --- > > Key: FLINK-21978 > URL: https://issues.apache.org/jira/browse/FLINK-21978 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.13.0 >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > Labels: pull-request-available > > Currently we has supported the cast conversion between Numeric type and > TIMESTAMP_LTZ type, we suppose the numeric value e.g `Long type 1000L` as > epoch seconds and then cast to TIMESTAMP_LTZ, but the java.lang.Long is a > conversion class of `LocalZonedTimestampType` and treats as milliseconds. > To avoid the inconsistency, we should disable it and encourage user to use > `TO_TIMESTAMP_LTZ(numeric, precisoon)` function. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21978) Disable cast conversion between Numeric type and TIMESTAMP_LTZ type
[ https://issues.apache.org/jira/browse/FLINK-21978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309326#comment-17309326 ] Leonard Xu commented on FLINK-21978: [~twalthr] Besides conversion class may cause confusion, another confusion source is that the `FROM_UNIXTIME(numeric [,foramtString])` treats numeric value in seconds. And we invested some main DB vendors like snowflake (TO_TIMESTAMP_LTZ,TO_TIMESTAMP_NTZ, FROM_UNIXTIME), oracle(FROM_UNIXTIME), most of them treat BIGINT as seconds > Disable cast conversion between Numeric type and TIMESTAMP_LTZ type > --- > > Key: FLINK-21978 > URL: https://issues.apache.org/jira/browse/FLINK-21978 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.13.0 >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > Labels: pull-request-available > > Currently we has supported the cast conversion between Numeric type and > TIMESTAMP_LTZ type, we suppose the numeric value e.g `Long type 1000L` as > epoch seconds and then cast to TIMESTAMP_LTZ, but the java.lang.Long is a > conversion class of `LocalZonedTimestampType` and treats as milliseconds. > To avoid the inconsistency, we should disable it and encourage user to use > `TO_TIMESTAMP_LTZ(numeric, precisoon)` function. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19705) Avro Confluent Schema Registry nightly end-to-end test: curl: (56) Recv failure: Connection reset by peer
[
https://issues.apache.org/jira/browse/FLINK-19705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17309189#comment-17309189
]
Leonard Xu commented on FLINK-19705:
another instance
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=15494=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529=17342
> Avro Confluent Schema Registry nightly end-to-end test: curl: (56) Recv
> failure: Connection reset by peer
> -
>
> Key: FLINK-19705
> URL: https://issues.apache.org/jira/browse/FLINK-19705
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7826=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> 2020-10-19T06:08:30.2781766Z Downloading confluent from
> http://packages.confluent.io/archive/5.0/confluent-oss-5.0.0-2.11.tar.gz
> 2020-10-19T06:08:30.2849354Z % Total% Received % Xferd Average Speed
> TimeTime Time Current
> 2020-10-19T06:08:30.2850032Z Dload Upload
> Total SpentLeft Speed
> 2020-10-19T06:08:30.2850669Z
> 2020-10-19T06:08:30.3457235Z 0 00 00 0 0 0
> --:--:-- --:--:-- --:--:-- 0
> 2020-10-19T06:08:31.3295810Z 0 00 00 0 0 0
> --:--:-- --:--:-- --:--:-- 0
> 2020-10-19T06:08:32.3309796Z 22 296M 22 67.1M0 0 64.1M 0
> 0:00:04 0:00:01 0:00:03 64.1M
> 2020-10-19T06:08:33.3293737Z 47 296M 47 141M0 0 69.1M 0
> 0:00:04 0:00:02 0:00:02 69.1M
> 2020-10-19T06:08:33.8972228Z 78 296M 78 232M0 0 76.1M 0
> 0:00:03 0:00:03 --:--:-- 76.1M
> 2020-10-19T06:08:33.8973866Z 97 296M 97 290M0 0 80.3M 0
> 0:00:03 0:00:03 --:--:-- 80.3M
> 2020-10-19T06:08:33.8974595Z curl: (56) Recv failure: Connection reset by peer
> 2020-10-19T06:08:33.9000256Z Stopping job timeout watchdog (with pid=111423)
> 2020-10-19T06:08:33.9007709Z
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/kafka-common.sh: line
> 209:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-26263816822/confluent-5.0.0/bin/schema-registry-stop:
> No such file or directory
> 2020-10-19T06:08:33.9010767Z [FAIL] Test script contains errors.
> 2020-10-19T06:08:33.9021624Z Checking for errors...
> 2020-10-19T06:08:33.9231098Z No errors in log files.
> 2020-10-19T06:08:33.9231651Z Checking for exceptions...
> 2020-10-19T06:08:33.9470201Z No exceptions in log files.
> 2020-10-19T06:08:33.9471568Z Checking for non-empty .out files...
> 2020-10-19T06:08:33.9489148Z grep:
> /home/vsts/work/1/s/flink-dist/target/flink-1.12-SNAPSHOT-bin/flink-1.12-SNAPSHOT/log/*.out:
> No such file or directory
> 2020-10-19T06:08:33.9493300Z No non-empty .out files.
> 2020-10-19T06:08:33.9493708Z
> 2020-10-19T06:08:33.9494586Z [FAIL] 'Avro Confluent Schema Registry nightly
> end-to-end test' failed after 0 minutes and 7 seconds! Test exited with exit
> code 1
> 2020-10-19T06:08:33.9495050Z
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21984) Change precision argument from optional to required in TO_TIMESTAMP_LTZ(numeric, precision)
Leonard Xu created FLINK-21984: -- Summary: Change precision argument from optional to required in TO_TIMESTAMP_LTZ(numeric, precision) Key: FLINK-21984 URL: https://issues.apache.org/jira/browse/FLINK-21984 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Affects Versions: 1.13.0 Reporter: Leonard Xu To avoid the confusing the numeric value is in second or millisecond, we'd better set the precision argument to required. The background is that `LocalZonedTimestampType` always treats `Integer` conversion class as second, treats `Long` as millisecond. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21978) Disable cast conversion between Numeric type and TIMESTAMP_LTZ type
[ https://issues.apache.org/jira/browse/FLINK-21978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-21978: --- Parent: FLINK-21617 Issue Type: Sub-task (was: Task) > Disable cast conversion between Numeric type and TIMESTAMP_LTZ type > --- > > Key: FLINK-21978 > URL: https://issues.apache.org/jira/browse/FLINK-21978 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.13.0 >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > Labels: pull-request-available > > Currently we has supported the cast conversion between Numeric type and > TIMESTAMP_LTZ type, we suppose the numeric value e.g `Long type 1000L` as > epoch seconds and then cast to TIMESTAMP_LTZ, but the java.lang.Long is a > conversion class of `LocalZonedTimestampType` and treats as milliseconds. > To avoid the inconsistency, we should disable it and encourage user to use > `TO_TIMESTAMP_LTZ(numeric, precisoon)` function. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21978) Disable cast conversion between Numeric type and TIMESTAMP_LTZ type
Leonard Xu created FLINK-21978: -- Summary: Disable cast conversion between Numeric type and TIMESTAMP_LTZ type Key: FLINK-21978 URL: https://issues.apache.org/jira/browse/FLINK-21978 Project: Flink Issue Type: Task Components: Table SQL / Planner Affects Versions: 1.13.0 Reporter: Leonard Xu Currently we has supported the cast conversion between Numeric type and TIMESTAMP_LTZ type, we suppose the numeric value e.g `Long type 1000L` as epoch seconds and then cast to TIMESTAMP_LTZ, but the java.lang.Long is a conversion class of `LocalZonedTimestampType` and treats as milliseconds. To avoid the inconsistency, we should disable it and encourage user to use `TO_TIMESTAMP_LTZ(numeric, precisoon)` function. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21715) Support implicit cast conversion between timestamp and timestamp_ltz
[ https://issues.apache.org/jira/browse/FLINK-21715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-21715: --- Description: The existed UDFs using `LocalDateTime` or `java.sql.Timestamp` as parameters type, if the data type of parameter column in SQL changed from TIMESTAMP to TIMESTAMP_LTZ, these UDFs must be rewritten, because TIMESTAMP_LTZ data type does not support `LocalDateTime` or `java.sql.Timestamp` as conversion class. The first approach is to support the two conversion classes for TIMESTAMP_LTZ data type, to obtain the correct result, the local time zone information is necessary when conversion happens between SQL internal data structure `TimestampData` and external conversion class `LocalDateTime` or `java.sql.Timestamp`. In this approach, after a e2e POC, we found that there are more changes than we thought before. It is difficult to cover with tests and it is easy to introduce bugs. Thus, to resolve the UDF compatibility issue, we consider support the implicit cast conversion between timestamp and timestamp_ltz after some offline discuss with [~jark] and [~ykt836], This way is clean and lightweight way to resolve the UDF compatibility issue and doesn't change any public interface as well. BTW the implicit cast conversion is supported in Oracle[1] [1]https://docs.oracle.com/cd/B19306_01/server.102/b14225/ch4datetime.htm was: The existed UDFs using `LocalDateTime` or `java.sql.Timestamp` as parameters type, if the data type of parameter column in SQL changed from TIMESTAMP to TIMESTAMP_LTZ, these UDFs must be rewritten, because TIMESTAMP_LTZ data type does not support `LocalDateTime` or `java.sql.Timestamp` as conversion class. The first approach is to support the two conversion classes for TIMESTAMP_LTZ data type, to obtain the correct result, the local time zone information is necessary when conversion happens between SQL internal data structure `TimestampData` and external conversion class `LocalDateTime` or `java.sql.Timestamp`. In this approach, I made a POC, I found that there are more changes than we thought before. It is difficult to cover with tests and it is easy to introduce bugs. Thus, to resolve the UDF compatibility issue, we consider support the > Support implicit cast conversion between timestamp and timestamp_ltz > - > > Key: FLINK-21715 > URL: https://issues.apache.org/jira/browse/FLINK-21715 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Table SQL / Runtime >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > > The existed UDFs using `LocalDateTime` or `java.sql.Timestamp` as parameters > type, if the data type of parameter column in SQL changed from TIMESTAMP to > TIMESTAMP_LTZ, these UDFs must be rewritten, because TIMESTAMP_LTZ data type > does not support `LocalDateTime` or `java.sql.Timestamp` as conversion class. > > The first approach is to support the two conversion classes for > TIMESTAMP_LTZ data type, to obtain the correct result, the local time zone > information is necessary when conversion happens between SQL internal data > structure `TimestampData` and external conversion class `LocalDateTime` or > `java.sql.Timestamp`. > In this approach, after a e2e POC, we found that there are more changes than > we thought before. It is difficult to cover with tests and it is easy to > introduce bugs. > > Thus, to resolve the UDF compatibility issue, we consider support the > implicit cast conversion between timestamp and timestamp_ltz after some > offline discuss with [~jark] and [~ykt836], > This way is clean and lightweight way to resolve the UDF compatibility issue > and doesn't change any public interface as well. BTW the implicit cast > conversion is supported in Oracle[1] > > [1]https://docs.oracle.com/cd/B19306_01/server.102/b14225/ch4datetime.htm > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21715) Support implicit cast conversion between timestamp and timestamp_ltz
[ https://issues.apache.org/jira/browse/FLINK-21715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-21715: --- Description: The existed UDFs using `LocalDateTime` or `java.sql.Timestamp` as parameters type, if the data type of parameter column in SQL changed from TIMESTAMP to TIMESTAMP_LTZ, these UDFs must be rewritten, because TIMESTAMP_LTZ data type does not support `LocalDateTime` or `java.sql.Timestamp` as conversion class. The first approach is to support the two conversion classes for TIMESTAMP_LTZ data type, to obtain the correct result, the local time zone information is necessary when conversion happens between SQL internal data structure `TimestampData` and external conversion class `LocalDateTime` or `java.sql.Timestamp`. In this approach, I made a POC, I found that there are more changes than we thought before. It is difficult to cover with tests and it is easy to introduce bugs. Thus, to resolve the UDF compatibility issue, we consider support the > Support implicit cast conversion between timestamp and timestamp_ltz > - > > Key: FLINK-21715 > URL: https://issues.apache.org/jira/browse/FLINK-21715 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Table SQL / Runtime >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > > The existed UDFs using `LocalDateTime` or `java.sql.Timestamp` as parameters > type, if the data type of parameter column in SQL changed from TIMESTAMP to > TIMESTAMP_LTZ, these UDFs must be rewritten, because TIMESTAMP_LTZ data type > does not support `LocalDateTime` or `java.sql.Timestamp` as conversion class. > The first approach is to support the two conversion classes for > TIMESTAMP_LTZ data type, to obtain the correct result, the local time zone > information is necessary when conversion happens between SQL internal data > structure `TimestampData` and external conversion class `LocalDateTime` or > `java.sql.Timestamp`. > In this approach, I made a POC, I found that there are more changes than we > thought before. It is difficult to cover with tests and it is easy to > introduce bugs. > Thus, to resolve the UDF compatibility issue, we consider support the -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21715) Support implicit cast conversion between timestamp and timestamp_ltz
[ https://issues.apache.org/jira/browse/FLINK-21715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-21715: --- Summary: Support implicit cast conversion between timestamp and timestamp_ltz (was: Introduce more conversion classes for LocalZonedTimestampType(TIMESTAMP_LTZ)) > Support implicit cast conversion between timestamp and timestamp_ltz > - > > Key: FLINK-21715 > URL: https://issues.apache.org/jira/browse/FLINK-21715 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Table SQL / Runtime >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21947) Support TIMESTAMP_LTZ type in CSV format
Leonard Xu created FLINK-21947: -- Summary: Support TIMESTAMP_LTZ type in CSV format Key: FLINK-21947 URL: https://issues.apache.org/jira/browse/FLINK-21947 Project: Flink Issue Type: Sub-task Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table SQL / Ecosystem Affects Versions: 1.13.0 Reporter: Leonard Xu Fix For: 1.13.0 Currently CSV format does not support TIMESTAMP_LTZ yet, and the code exists some bug, we should support the TIMESTAMP_LTZ type properly and correct the timestamp conversion. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20254) HiveTableSourceITCase.testStreamPartitionReadByCreateTime times out
[
https://issues.apache.org/jira/browse/FLINK-20254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17306915#comment-17306915
]
Leonard Xu commented on FLINK-20254:
I think this is another bug in `collecting result` after some investigation, I
can reproduce this by run some tests which used `TableResult.collect()` many
time until failed
cc [~TsReaper]
{code:java}
05:08:37,507 [main] WARN
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher [] - An
exception occurs when fetching query
resultsjava.util.concurrent.ExecutionException:
org.apache.flink.runtime.dispatcher.UnavailableDispatcherOperationException:
Unable to get JobMasterGateway for initializing job. The requested operation is
not available while the JobManager is initializing.at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
~[?:1.8.0_282]at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
~[?:1.8.0_282]at
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sendRequest(CollectResultFetcher.java:155)
~[flink-streaming-java_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]at
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:126)
[flink-streaming-java_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]at
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
[flink-streaming-java_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]at
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
[flink-streaming-java_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]at
org.apache.flink.table.planner.sinks.SelectTableSinkBase$RowIteratorWrapper.hasNext(SelectTableSinkBase.java:117)
[flink-table-planner-blink_2.11-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]at
org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:350)
[flink-table-api-java-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]at
java.util.Iterator.forEachRemaining(Iterator.java:115) [?:1.8.0_282]at
org.apache.flink.util.CollectionUtil.iteratorToList(CollectionUtil.java:109)
[flink-core-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]at
org.apache.flink.connectors.hive.HiveTableSourceITCase.testReadPartitionTable(HiveTableSourceITCase.java:182)
[test-classes/:?]at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method) ~[?:1.8.0_282]at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_282]at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.8.0_282]at java.lang.reflect.Method.invoke(Method.java:498)
~[?:1.8.0_282]at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
[junit-4.12.jar:4.12]at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
[junit-4.12.jar:4.12]at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
[junit-4.12.jar:4.12]at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
[junit-4.12.jar:4.12]at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
[junit-4.12.jar:4.12]at
org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
[flink-test-utils-junit-1.13-SNAPSHOT.jar:1.13-SNAPSHOT]at
org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
[junit-4.12.jar:4.12]at org.junit.rules.RunRules.evaluate(RunRules.java:20)
[junit-4.12.jar:4.12]at
org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
[junit-4.12.jar:4.12]at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
[junit-4.12.jar:4.12]at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
[junit-4.12.jar:4.12]at
org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
[junit-4.12.jar:4.12]at
org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
[junit-4.12.jar:4.12]at
org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
[junit-4.12.jar:4.12]at
org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
[junit-4.12.jar:4.12]at
org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
[junit-4.12.jar:4.12]at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
[junit-4.12.jar:4.12]at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
[junit-4.12.jar:4.12]at
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
[junit-4.12.jar:4.12]at
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
[junit-4.12.jar:4.12]at org.junit.rules.RunRules.evaluate(RunRules.java:20)
[jira] [Updated] (FLINK-21917) Add back missing zh docs
[ https://issues.apache.org/jira/browse/FLINK-21917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-21917: --- Description: We lost some Chinese doc pages when migrate doc website from Jekyll to Hugo in ([https://github.com/apache/flink/pull/14903]), such as `jdbc zh`, `kafka zh`. we should add them back. was: We lost some Chinese doc pages when migrate from legacy doc website to current one in ([https://github.com/apache/flink/pull/14903]), such as `jdbc zh`, `kafka zh`. we should add them back. > Add back missing zh docs > > > Key: FLINK-21917 > URL: https://issues.apache.org/jira/browse/FLINK-21917 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.13.0 >Reporter: Leonard Xu >Priority: Major > Fix For: 1.13.0 > > > We lost some Chinese doc pages when migrate doc website from Jekyll to Hugo > in ([https://github.com/apache/flink/pull/14903]), such as `jdbc zh`, `kafka > zh`. > we should add them back. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21917) Add back missing zh docs
[ https://issues.apache.org/jira/browse/FLINK-21917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-21917: --- Description: We lost some Chinese doc pages when migrate from legacy doc website to current one in ([https://github.com/apache/flink/pull/14903]), such as `jdbc zh`, `kafka zh`. we should add them back. was: We lost some Chinese doc pages are lost when migrate from legacy doc website to current one in (https://github.com/apache/flink/pull/14903), such as `jdbc zh`, `kafka zh`. we should add them back. > Add back missing zh docs > > > Key: FLINK-21917 > URL: https://issues.apache.org/jira/browse/FLINK-21917 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.13.0 >Reporter: Leonard Xu >Priority: Major > Fix For: 1.13.0 > > > We lost some Chinese doc pages when migrate from legacy doc website to > current one in ([https://github.com/apache/flink/pull/14903]), such as `jdbc > zh`, `kafka zh`. > we should add them back. > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21917) Add back missing zh docs
Leonard Xu created FLINK-21917: -- Summary: Add back missing zh docs Key: FLINK-21917 URL: https://issues.apache.org/jira/browse/FLINK-21917 Project: Flink Issue Type: Bug Components: Documentation Affects Versions: 1.13.0 Reporter: Leonard Xu Fix For: 1.13.0 We lost some Chinese doc pages are lost when migrate from legacy doc website to current one in (https://github.com/apache/flink/pull/14903), such as `jdbc zh`, `kafka zh`. we should add them back. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21871) Support watermark for Hive and Filesystem streaming source
[
https://issues.apache.org/jira/browse/FLINK-21871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17304803#comment-17304803
]
Leonard Xu commented on FLINK-21871:
Thanks [~jark] for driving this, the design looks well
> Support watermark for Hive and Filesystem streaming source
> --
>
> Key: FLINK-21871
> URL: https://issues.apache.org/jira/browse/FLINK-21871
> Project: Flink
> Issue Type: New Feature
> Components: Connectors / FileSystem, Connectors / Hive, Table SQL /
> API
>Reporter: Jark Wu
>Assignee: Jark Wu
>Priority: Major
> Fix For: 1.13.0
>
>
> Hive and Filesystem already support streaming source. However, they doesn't
> support watermark on the source. That means users can't leverage the
> streaming source to perform the Flink powerful streaming analysis, e.g.
> window aggregate, interval join, and so on.
> In order to make more Hive users can leverage Flink to perform streaming
> analysis, and also cooperate with the new optimized window-TVF operations
> (FLIP-145), we need to support watermark for Hive and Filesystem.
> h2. How to emit watermark in Hive and Filesystem
> Factual data in Hive are usually partitioned by date time, e.g.
> {{pt_day=2021-03-19, pt_hour=10}}. In this case, when the data of partition
> {{pt_day=2021-03-19, pt_hour=10}} are emitted, we should be able to know all
> the data before {{2021-03-19 11:00:00}} have been arrived, so we can emit a
> watermark value of {{2021-03-19 11:00:00}}. We call this partition watermark.
> The partition watermark is much better than record watermark (extract
> watermark from record, e.g. {{ts - INTERVAL '1' MINUTE}}). Because in above
> example, if we are using partition watermark, the window of [10:00, 11:00)
> will be triggered when pt_hour=10 is finished. However, if we are using
> record watermark, the window of [10:00, 11:00) will be triggered when
> pt_hour=11 is arrived, that will make the pipeline have one more partition
> dely.
> Therefore, we firstly focus on support partition watermark for Hive and
> Filesystem.
> h2. Example
> In order to support such watermarks, we propose using the following DDL to
> define a Hive table with watermark defined:
> {code:sql}
> -- using hive dialect
> CREATE TABLE hive_table (
> x int,
> y string,
> z int,
> rowtime timestamp,
> WATERMARK FOR rowtime AS SOURCE_WATERMARK
> ) PARTITIONED BY (pt_day string, pt_hour string)
> TBLPROPERTIES (
> 'streaming-source.enable'='true',
> 'streaming-source.monitor-interval'='1s',
> 'partition.time-extractor.timestamp-pattern'='$pt_day $pt_hour:00:00',
> 'streaming-source.partition-interval'='1h'
> );
> -- window aggregate on the hive table
> SELECT window_start, window_end, COUNT(*), MAX(y), SUM(z)
> FROM TABLE(
>TUMBLE(TABLE hive_table, DESCRIPTOR(ts), INTERVAL '1' HOUR))
> GROUP BY window_start, window_end;
> {code}
> For filesystem connector, the DDL can be:
> {code:sql}
> CREATE TABLE fs_table (
> x int,
> y string,
> z int,
> ts TIMESTAMP(3),
> pt_day string,
> pt_hour string,
> WATERMARK FOR ts AS SOURCE_WATERMARK
> ) PARTITIONED BY (pt_day, pt_hour)
> WITH (
> 'connector' = 'filesystem',
> 'path' = '/path/to/file',
> 'format' = 'parquet',
> 'streaming-source.enable'='true',
> 'streaming-source.monitor-interval'='1s',
> 'partition.time-extractor.timestamp-pattern'='$pt_day $pt_hour:00:00',
> 'streaming-source.partition-interval'='1h'
> );
> {code}
> I will explain the new function/configuration.
> h2. SOURCE_WATERMARK built-in function
> FLIP-66[1] proposed {{SYSTEM_WATERMARK}} function for watermarks preserved in
> underlying source system.
> However, the SYSTEM prefix sounds like a Flink system generated value, but
> actually, this is a SOURCE system generated value.
> So I propose to use {{SOURCE_WATERMARK}} intead, this also keeps the concept
> align with the API of
> {{org.apache.flink.table.descriptors.Rowtime#watermarksFromSource}}.
> h2. Table Options for Watermark
> - {{partition.time-extractor.timestamp-pattern}}: this option already exists.
> This is used to extract/convert partition value to a timestamp value.
> - {{streaming-source.partition-interval}}: this is a new option. It indicates
> the minimal time interval of the partitions. It's used to calculate the
> correct watermark when a partition is finished. The watermark =
> partition-timestamp + time-inteval.
> h2. How to support watermark for existing Hive tables
> We all know that we can't create a new table for an existing Hive table. So
> we should support altering existing Hive table to add the watermark
> inforamtion.
> This can be supported by the new ALTER TABLE syntax proposed in FLINK-21634.
> Because watermark, computed
[jira] [Commented] (FLINK-21833) TemporalRowTimeJoinOperator State Leak Although configure idle.state.retention.time
[ https://issues.apache.org/jira/browse/FLINK-21833?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17303059#comment-17303059 ] Leonard Xu commented on FLINK-21833: [~zicat] Thanks for the report, I'll help review this PR > TemporalRowTimeJoinOperator State Leak Although configure > idle.state.retention.time > --- > > Key: FLINK-21833 > URL: https://issues.apache.org/jira/browse/FLINK-21833 > Project: Flink > Issue Type: Bug > Components: Table SQL / Runtime >Affects Versions: 1.12.2 >Reporter: lynn1.zhang >Priority: Major > Labels: pull-request-available > Attachments: image-2021-03-17-11-06-21-768.png > > > Use TemporalRowTimeJoinOperator feature will lead to unlimited data > expansion, although configure idle.state.retention.time > I have found the bug, and fixed it. > !image-2021-03-17-11-06-21-768.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21622) Introduce function TO_TIMESTAMP_LTZ(numeric [, precision])
[ https://issues.apache.org/jira/browse/FLINK-21622?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17302620#comment-17302620 ] Leonard Xu commented on FLINK-21622: Thanks for you information [~twalthr], I'll consider the new way (y) > Introduce function TO_TIMESTAMP_LTZ(numeric [, precision]) > --- > > Key: FLINK-21622 > URL: https://issues.apache.org/jira/browse/FLINK-21622 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Table SQL / Planner >Reporter: Leonard Xu >Assignee: Leonard Xu >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21820) JDBC connector shouldn't read all rows in per statement by default
Leonard Xu created FLINK-21820: -- Summary: JDBC connector shouldn't read all rows in per statement by default Key: FLINK-21820 URL: https://issues.apache.org/jira/browse/FLINK-21820 Project: Flink Issue Type: Improvement Components: Connectors / JDBC Reporter: Leonard Xu The default value for JDBC option 'scan.fetch-size' is 0 which means read all rows in statement, this may lead to OOM or IO timeout. We'd better set a reasonable value as default value. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21716) Support higher precision for Data Type TIME(p)
[ https://issues.apache.org/jira/browse/FLINK-21716?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-21716: --- Component/s: Table SQL / Runtime Table SQL / API > Support higher precision for Data Type TIME(p) > --- > > Key: FLINK-21716 > URL: https://issues.apache.org/jira/browse/FLINK-21716 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Table SQL / Runtime >Reporter: Leonard Xu >Priority: Major > > Due to the historical reason, we only support TIME(3) yet, we can support > higher precision eg. TIME(9). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21716) Support higher precision for Data Type TIME(p)
Leonard Xu created FLINK-21716: -- Summary: Support higher precision for Data Type TIME(p) Key: FLINK-21716 URL: https://issues.apache.org/jira/browse/FLINK-21716 Project: Flink Issue Type: Sub-task Reporter: Leonard Xu Due to the historical reason, we only support TIME(3) yet, we can support higher precision eg. TIME(9). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21715) Introduce more conversion classes for LocalZonedTimestampType(TIMESTAMP_LTZ)
Leonard Xu created FLINK-21715: -- Summary: Introduce more conversion classes for LocalZonedTimestampType(TIMESTAMP_LTZ) Key: FLINK-21715 URL: https://issues.apache.org/jira/browse/FLINK-21715 Project: Flink Issue Type: Sub-task Components: Table SQL / API, Table SQL / Runtime Reporter: Leonard Xu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21714) Use TIMESTAMP_LTZ as return type for function PROCTIME()
Leonard Xu created FLINK-21714: -- Summary: Use TIMESTAMP_LTZ as return type for function PROCTIME() Key: FLINK-21714 URL: https://issues.apache.org/jira/browse/FLINK-21714 Project: Flink Issue Type: Sub-task Components: Table SQL / API Reporter: Leonard Xu When users use a PROCTIME() in SQL, the return value of PROCTIME() has a timezone offset with the wall-clock time in users' local time zone, users need to add their local time zone offset manually to get expected local timestamp(e.g: Users in Germany need to +1h to get expected local timestamp). This issue to correct PROCTIME() function |*function*|*existed problem*|*current behavior*|*proposed changes*| |PROCTIME()|returns UTC timestamp, but user expects current timestamp in session time zone|return type: TIMESTAMP *PROCTIME* #session timezone: UTC 2020-12-28 23:52:52 #session timezone: UTC+8 2020-12-28 23:52:52|return current timestamp in session time zone for PROCTIME(), the return type should be *TIMESTAMP WITH LOCAL TIME ZONE *PROCTIME** #session timezone: UTC 2020-12-28 23:52:52 #session timezone: UTC+8 2020-12-29 07:52:52| -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21713) Correct function CURRENT_TIMESTAMP/CURRENT_TIME/CURRENT_DATE/NOW/LOCALTIME/LOCALTIMESTAMP
Leonard Xu created FLINK-21713: -- Summary: Correct function CURRENT_TIMESTAMP/CURRENT_TIME/CURRENT_DATE/NOW/LOCALTIME/LOCALTIMESTAMP Key: FLINK-21713 URL: https://issues.apache.org/jira/browse/FLINK-21713 Project: Flink Issue Type: Sub-task Components: Table SQL / API Reporter: Leonard Xu 1. fix the return value type as well the return value for following time functions |*function*|*existed problem*|*current behavior*|*proposed changes*| |CURRENT_DATE|returns UTC date, but user expects current date in session time zone|return type: DATE #session timezone: UTC 2020-12-28 #session timezone: UTC+8 2020-12-28| return current date in session time zone, the return type should be *DATE* #session timezone: UTC 2020-12-28 #session timezone: UTC+8 2020-12-29| |CURRENT_TIME|returns UTC time, but user expects current time in session time zone|return type: TIME #session timezone: UTC 23:52:52 #session timezone: UTC+8 23:52:52|return current time in session time zone, the return type should be *TIME* #session timezone: UTC 23:52:52 #session timezone: UTC+8 07:52:52| |CURRENT_TIMESTAMP|returns UTC timestamp, but user expects current timestamp in session time zone| return type: TIMESTAMP #session timezone: UTC 2020-12-28 23:52:52 #session timezone: UTC+8 2020-12-28 23:52:52|return current timestamp in session time zone, the return type should be *TIMESTAMP WITH LOCAL TIME ZONE* #session timezone: UTC 2020-12-28 23:52:52 #session timezone: UTC+8 2020-12-29 07:52:52| |NOW()|returns UTC timestamp, but user expects current timestamp in session time zone| return type: TIMESTAMP #session timezone: UTC 2020-12-28 23:52:52 #session timezone: UTC+8 2020-12-28 23:52:52|return current timestamp in session time zone, the return type should be *TIMESTAMP WITH LOCAL TIME ZONE* #session timezone: UTC 2020-12-28 23:52:52 #session timezone: UTC+8 2020-12-29 07:52:52| 2. The following functions' evaluation is depend on execution mode * LOCALTIME * LOCALTIMESTAMP * CURRENT_DATE * CURRENT_TIME * CURRENT_TIMESTAMP * NOW() Flink evaluates above time function values according to execution mode, i.e. Flink evaluates time function value for row level in Streaming mode, evaluates the time function value at query start for batch mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21698) Disable problematic cast conversion between NUMERIC type and TIMESTAMP type
Leonard Xu created FLINK-21698: -- Summary: Disable problematic cast conversion between NUMERIC type and TIMESTAMP type Key: FLINK-21698 URL: https://issues.apache.org/jira/browse/FLINK-21698 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Reporter: Leonard Xu The cast conversion between NUMERIC type and TIMESTAMP type is problematic , we should disable it, NUMERIC type and TIMESTAMP_LTZ type cast conversion is valid. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20387) Support column of TIMESTAMP WITH LOCAL ZONE TIME type as rowtime attribute
[
https://issues.apache.org/jira/browse/FLINK-20387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated FLINK-20387:
---
Parent: FLINK-21617
Issue Type: Sub-task (was: New Feature)
> Support column of TIMESTAMP WITH LOCAL ZONE TIME type as rowtime attribute
> --
>
> Key: FLINK-20387
> URL: https://issues.apache.org/jira/browse/FLINK-20387
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API, Table SQL / Planner
>Reporter: Jark Wu
>Priority: Major
>
> Currently, only {{TIMESTAMP}} type can be used as rowtime attribute. Would be
> better to support {{TIMESTAMP WITH LOCAL ZONE TIME}} as well.
> As a workaround, users can cast the TIMESTAMP WITH LOCAL ZONE TIME into
> TIMESTAMP, {{CAST(ts AS TIMESTAMP)}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21624) Correct FLOOR/CEIL (TIMESTAMP/TIMESTAMP_LTZ/DATE TO WEEK) functions
[ https://issues.apache.org/jira/browse/FLINK-21624?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-21624: --- Description: While using a simple query such as this SELECT `ts`, FLOOR(`ts` TO WEEK) as `week_start`, CEIL(`ts` TO WEEK) as `week_end` FROM some_table I get some weird results like these: 2021-03-01T00:00| 2021-02-25T00:00| 2021-03-04T00:00 Which is obviously wrong since March 1st is on Monday, February 25th is Thursday as well as March 04th. I've tried different combinations of timezone configurations and with both timestamps and dates, with the same results. Is there anything obviously wrong in that query? Is there any configuration to keep in mind for the start of week day? from user ML: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-SQL-FLOORing-OR-CEILing-a-DATE-or-TIMESTAMP-to-WEEK-uses-Thursdays-as-week-start-td41838.html > Correct FLOOR/CEIL (TIMESTAMP/TIMESTAMP_LTZ/DATE TO WEEK) functions > --- > > Key: FLINK-21624 > URL: https://issues.apache.org/jira/browse/FLINK-21624 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Table SQL / Planner >Reporter: Leonard Xu >Priority: Major > > While using a simple query such as this > > SELECT > `ts`, > FLOOR(`ts` TO WEEK) as `week_start`, > CEIL(`ts` TO WEEK) as `week_end` > FROM some_table > > I get some weird results like these: > > 2021-03-01T00:00| 2021-02-25T00:00| 2021-03-04T00:00 > > Which is obviously wrong since March 1st is on Monday, February 25th is > Thursday as well as March 04th. > > I've tried different combinations of timezone configurations and with both > timestamps and dates, with the same results. > > Is there anything obviously wrong in that query? Is there any configuration > to keep in mind for the start of week day? > > from user ML: > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-SQL-FLOORing-OR-CEILing-a-DATE-or-TIMESTAMP-to-WEEK-uses-Thursdays-as-week-start-td41838.html -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21624) Correct FLOOR/CEIL (TIMESTAMP/TIMESTAMP_LTZ/DATE TO WEEK) functions
Leonard Xu created FLINK-21624: -- Summary: Correct FLOOR/CEIL (TIMESTAMP/TIMESTAMP_LTZ/DATE TO WEEK) functions Key: FLINK-21624 URL: https://issues.apache.org/jira/browse/FLINK-21624 Project: Flink Issue Type: Sub-task Components: Table SQL / API, Table SQL / Planner Reporter: Leonard Xu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21623) Introduce CURRENT_ROW_TIMESTAMP function
Leonard Xu created FLINK-21623: -- Summary: Introduce CURRENT_ROW_TIMESTAMP function Key: FLINK-21623 URL: https://issues.apache.org/jira/browse/FLINK-21623 Project: Flink Issue Type: Sub-task Components: Table SQL / API, Table SQL / Planner Reporter: Leonard Xu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21620) Support abbreviation TIMESTAMP_LTZ FOR TIMESTAMP WITH LOCAL TIME ZONE
[ https://issues.apache.org/jira/browse/FLINK-21620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-21620: --- Summary: Support abbreviation TIMESTAMP_LTZ FOR TIMESTAMP WITH LOCAL TIME ZONE (was: [table-planner/table-parser] Support abbreviation TIMESTAMP_LTZ FOR TIMESTAMP WITH LOCAL TIME ZONE) > Support abbreviation TIMESTAMP_LTZ FOR TIMESTAMP WITH LOCAL TIME ZONE > - > > Key: FLINK-21620 > URL: https://issues.apache.org/jira/browse/FLINK-21620 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Table SQL / Planner >Reporter: Leonard Xu >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21622) Introduce function TO_TIMESTAMP_LTZ(numeric [, precision])
Leonard Xu created FLINK-21622: -- Summary: Introduce function TO_TIMESTAMP_LTZ(numeric [, precision]) Key: FLINK-21622 URL: https://issues.apache.org/jira/browse/FLINK-21622 Project: Flink Issue Type: Sub-task Components: Table SQL / API, Table SQL / Planner Reporter: Leonard Xu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21621) Support TIMESTAMP_LTZ arithmetic
Leonard Xu created FLINK-21621: -- Summary: Support TIMESTAMP_LTZ arithmetic Key: FLINK-21621 URL: https://issues.apache.org/jira/browse/FLINK-21621 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Reporter: Leonard Xu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21620) [table-planner/table-parser] Support abbreviation TIMESTAMP_LTZ FOR TIMESTAMP WITH LOCAL TIME ZONE
Leonard Xu created FLINK-21620: -- Summary: [table-planner/table-parser] Support abbreviation TIMESTAMP_LTZ FOR TIMESTAMP WITH LOCAL TIME ZONE Key: FLINK-21620 URL: https://issues.apache.org/jira/browse/FLINK-21620 Project: Flink Issue Type: Sub-task Components: Table SQL / API, Table SQL / Planner Reporter: Leonard Xu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21617) FLIP-162: Consistent Flink SQL time function behavior
Leonard Xu created FLINK-21617: -- Summary: FLIP-162: Consistent Flink SQL time function behavior Key: FLINK-21617 URL: https://issues.apache.org/jira/browse/FLINK-21617 Project: Flink Issue Type: New Feature Components: Table SQL / API Reporter: Leonard Xu https://cwiki.apache.org/confluence/display/FLINK/FLIP-162%3A+Consistent+Flink+SQL+time+function+behavior -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21523) ArrayIndexOutOfBoundsException occurs while run a hive streaming job with partitioned table source
[
https://issues.apache.org/jira/browse/FLINK-21523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17295085#comment-17295085
]
Leonard Xu commented on FLINK-21523:
Make sense to me
> ArrayIndexOutOfBoundsException occurs while run a hive streaming job with
> partitioned table source
> ---
>
> Key: FLINK-21523
> URL: https://issues.apache.org/jira/browse/FLINK-21523
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.12.1
>Reporter: zouyunhe
>Priority: Major
> Labels: pull-request-available
>
> we have two hive table, the ddl as below
> {code:java}
> //test_tbl5
> create table test.test_5
> (dpi int,
> uid bigint)
> partitioned by( day string, hour string) stored as parquet;
> //test_tbl3
> create table test.test_3(
> dpi int,
> uid bigint,
> itime timestamp) stored as parquet;{code}
> then add a partiton to test_tbl5,
> {code:java}
> alter table test_tbl5 add partition(day='2021-02-27',hour='12');
> {code}
> we start a flink streaming job to read hive table test_tbl5 , and write the
> data into test_tbl3, the job's sql as
> {code:java}
> set test_tbl5.streaming-source.enable = true;
> insert into hive.test.test_tbl3 select dpi, uid,
> cast(to_timestamp('2020-08-09 00:00:00') as timestamp(9)) from
> hive.test.test_tbl5 where `day` = '2021-02-27';
> {code}
> and we seen the exception throws
> {code:java}
> 2021-02-28 22:33:16,553 ERROR
> org.apache.flink.runtime.source.coordinator.SourceCoordinatorContext -
> Exception while handling result from async call in SourceCoordinator-Source:
> HiveSource-test.test_tbl5. Triggering job
> failover.org.apache.flink.connectors.hive.FlinkHiveException: Failed to
> enumerate filesat
> org.apache.flink.connectors.hive.ContinuousHiveSplitEnumerator.handleNewSplits(ContinuousHiveSplitEnumerator.java:152)
> ~[flink-connector-hive_2.12-1.12.1.jar:1.12.1]at
> org.apache.flink.runtime.source.coordinator.ExecutorNotifier.lambda$null$4(ExecutorNotifier.java:136)
> ~[flink-dist_2.12-1.12.1.jar:1.12.1]at
> org.apache.flink.util.ThrowableCatchingRunnable.run(ThrowableCatchingRunnable.java:40)
> [flink-dist_2.12-1.12.1.jar:1.12.1]at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> [?:1.8.0_60]at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> [?:1.8.0_60]at java.lang.Thread.run(Thread.java:745) [?:1.8.0_60]Caused
> by: java.lang.ArrayIndexOutOfBoundsException: -1at
> org.apache.flink.connectors.hive.util.HivePartitionUtils.toHiveTablePartition(HivePartitionUtils.java:184)
> ~[flink-connector-hive_2.12-1.12.1.jar:1.12.1]at
> org.apache.flink.connectors.hive.HiveTableSource$HiveContinuousPartitionFetcherContext.toHiveTablePartition(HiveTableSource.java:417)
> ~[flink-connector-hive_2.12-1.12.1.jar:1.12.1]at
> org.apache.flink.connectors.hive.ContinuousHiveSplitEnumerator$PartitionMonitor.call(ContinuousHiveSplitEnumerator.java:237)
> ~[flink-connector-hive_2.12-1.12.1.jar:1.12.1]at
> org.apache.flink.connectors.hive.ContinuousHiveSplitEnumerator$PartitionMonitor.call(ContinuousHiveSplitEnumerator.java:177)
> ~[flink-connector-hive_2.12-1.12.1.jar:1.12.1]at
> org.apache.flink.runtime.source.coordinator.ExecutorNotifier.lambda$notifyReadyAsync$5(ExecutorNotifier.java:133)
> ~[flink-dist_2.12-1.12.1.jar:1.12.1]at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> ~[?:1.8.0_60]at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> ~[?:1.8.0_60]at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> ~[?:1.8.0_60]at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> ~[?:1.8.0_60]... 3 more{code}
> it seems the partitoned field is not found in the source table field list.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21575) Replace InputFormat with BulkFormat in HivePartition Reader
Leonard Xu created FLINK-21575: -- Summary: Replace InputFormat with BulkFormat in HivePartition Reader Key: FLINK-21575 URL: https://issues.apache.org/jira/browse/FLINK-21575 Project: Flink Issue Type: Task Components: Connectors / Hive Reporter: Leonard Xu Currently HivePartition Reader(`HiveInputFormatPartitionReader`) still use legacy interface `InputFormat`, we can migrate it to new interface `BulkFormat` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21431) UpsertKafkaTableITCase.testTemporalJoin hang
[
https://issues.apache.org/jira/browse/FLINK-21431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17288925#comment-17288925
]
Leonard Xu commented on FLINK-21431:
I'll take a look [~dwysakowicz]
> UpsertKafkaTableITCase.testTemporalJoin hang
>
>
> Key: FLINK-21431
> URL: https://issues.apache.org/jira/browse/FLINK-21431
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka
>Affects Versions: 1.13.0
>Reporter: Guowei Ma
>Priority: Major
> Labels: test-stability
>
> This case hangs almost 3 hours:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=13543=logs=ce8f3cc3-c1ea-5281-f5eb-df9ebd24947f=f266c805-9429-58ed-2f9e-482e7b82f58b
> {code:java}
> Test testTemporalJoin[format =
> csv](org.apache.flink.streaming.connectors.kafka.table.UpsertKafkaTableITCase)
> is running.
>
> 23:08:43,259 [ main] INFO
> org.apache.flink.streaming.connectors.kafka.KafkaTestEnvironmentImpl [] -
> Creating topic users_csv 23:08:45,303 [ main] WARN
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer [] - Property
> [transaction.timeout.ms] not specified. Setting it to 360 ms 23:08:45,430
> [ChangelogNormalize(key=[user_id]) -> Calc(select=[user_id, user_name,
> region, CAST(modification_time) AS timestamp]) -> Sink:
> Sink(table=[default_catalog.default_database.users_csv], fields=[user_id,
> user_name, region, timestamp]) (1/1)#0] WARN
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer [] - Using
> AT_LEAST_ONCE semantic, but checkpointing is not enabled. Switching to NONE
> semantic. 23:08:45,438 [ChangelogNormalize(key=[user_id]) ->
> Calc(select=[user_id, user_name, region, CAST(modification_time) AS
> timestamp]) -> Sink: Sink(table=[default_catalog.default_database.users_csv],
> fields=[user_id, user_name, region, timestamp]) (1/1)#0] INFO
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer [] - Starting
> FlinkKafkaInternalProducer (1/1) to produce into default topic users_csv
> 23:08:45,791 [Source: TableSourceScan(table=[[default_catalog,
> default_database, users_csv, watermark=[CAST($3):TIMESTAMP(3)]]],
> fields=[user_id, user_name, region, timestamp]) (1/1)#0] INFO
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase [] -
> Consumer subtask 0 has no restore state. 23:08:45,810 [Source:
> TableSourceScan(table=[[default_catalog, default_database, users_csv,
> watermark=[CAST($3):TIMESTAMP(3)]]], fields=[user_id, user_name, region,
> timestamp]) (1/1)#0] INFO
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase [] -
> Consumer subtask 0 will start reading the following 2 partitions from the
> earliest offsets: [KafkaTopicPartition{topic='users_csv', partition=1},
> KafkaTopicPartition{topic='users_csv', partition=0}] 23:08:45,825 [Legacy
> Source Thread - Source: TableSourceScan(table=[[default_catalog,
> default_database, users_csv, watermark=[CAST($3):TIMESTAMP(3)]]],
> fields=[user_id, user_name, region, timestamp]) (1/1)#0] INFO
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase [] -
> Consumer subtask 0 creating fetcher with offsets
> {KafkaTopicPartition{topic='users_csv', partition=1}=-915623761775,
> KafkaTopicPartition{topic='users_csv', partition=0}=-915623761775}.
> ##[error]The operation was canceled.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21302) Fix NPE when use row_number() in over agg
[
https://issues.apache.org/jira/browse/FLINK-21302?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated FLINK-21302:
---
Issue Type: Bug (was: New Feature)
> Fix NPE when use row_number() in over agg
> -
>
> Key: FLINK-21302
> URL: https://issues.apache.org/jira/browse/FLINK-21302
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: Andy
>Priority: Major
>
> `NullPointException` would be thrown out if SQL contains row_number() in Over
> Aggregate.
> {code:scala}
> @Test
> def testRowNumberOnOver(): Unit = {
> val t = failingDataSource(TestData.tupleData5)
> .toTable(tEnv, 'a, 'b, 'c, 'd, 'e, 'proctime.proctime)
> tEnv.registerTable("MyTable", t)
> val sqlQuery = "SELECT a, ROW_NUMBER() OVER (PARTITION BY a ORDER BY
> proctime()) FROM MyTable"
> val sink = new TestingAppendSink
> tEnv.sqlQuery(sqlQuery).toAppendStream[Row].addSink(sink)
> env.execute()
> }{code}
>
> The following exception would be thrown out.
> {code:java}
> java.lang.NullPointerExceptionjava.lang.NullPointerException at
> scala.collection.mutable.ArrayOps$ofInt$.length$extension(ArrayOps.scala:240)
> at scala.collection.mutable.ArrayOps$ofInt.length(ArrayOps.scala:240) at
> scala.collection.SeqLike$class.size(SeqLike.scala:106) at
> scala.collection.mutable.ArrayOps$ofInt.size(ArrayOps.scala:234) at
> scala.collection.mutable.Builder$class.sizeHint(Builder.scala:69) at
> scala.collection.mutable.ArrayBuilder.sizeHint(ArrayBuilder.scala:22) at
> scala.collection.TraversableLike$class.builder$1(TraversableLike.scala:230)
> at scala.collection.TraversableLike$class.map(TraversableLike.scala:233) at
> scala.collection.mutable.ArrayOps$ofInt.map(ArrayOps.scala:234) at
> org.apache.flink.table.planner.codegen.agg.DeclarativeAggCodeGen.(DeclarativeAggCodeGen.scala:82)
> at
> org.apache.flink.table.planner.codegen.agg.AggsHandlerCodeGenerator$$anonfun$3.apply(AggsHandlerCodeGenerator.scala:222)
> at
> org.apache.flink.table.planner.codegen.agg.AggsHandlerCodeGenerator$$anonfun$3.apply(AggsHandlerCodeGenerator.scala:214)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.ArrayOps$ofRef.foreach(ArrayOps.scala:186) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.mutable.ArrayOps$ofRef.map(ArrayOps.scala:186) at
> org.apache.flink.table.planner.codegen.agg.AggsHandlerCodeGenerator.initialAggregateInformation(AggsHandlerCodeGenerator.scala:214)
> at
> org.apache.flink.table.planner.codegen.agg.AggsHandlerCodeGenerator.generateAggsHandler(AggsHandlerCodeGenerator.scala:325)
> at
> org.apache.flink.table.planner.plan.nodes.exec.stream.StreamExecOverAggregate.createUnboundedOverProcessFunction(StreamExecOverAggregate.java:262)
> at
> org.apache.flink.table.planner.plan.nodes.exec.stream.StreamExecOverAggregate.translateToPlanInternal(StreamExecOverAggregate.java:154)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:128)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecEdge.translateToPlan(ExecEdge.java:247)
> at
> org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecCalc.translateToPlanInternal(CommonExecCalc.java:65)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:128)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecEdge.translateToPlan(ExecEdge.java:247)
> at
> org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecLegacySink.translateToTransformation(CommonExecLegacySink.java:167)
> at
> org.apache.flink.table.planner.plan.nodes.exec.common.CommonExecLegacySink.translateToPlanInternal(CommonExecLegacySink.java:136)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNodeBase.translateToPlan(ExecNodeBase.java:128)
> at
> org.apache.flink.table.planner.delegation.StreamPlanner$$anonfun$translateToPlan$1.apply(StreamPlanner.scala:67)
> at
> org.apache.flink.table.planner.delegation.StreamPlanner$$anonfun$translateToPlan$1.apply(StreamPlanner.scala:66)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891) at
> scala.collection.AbstractIterator.foreach(Iterator.scala:1334) at
> scala.collection.IterableLike$class.foreach(IterableLike.scala:72) at
> scala.collection.AbstractIterable.foreach(Iterable.scala:54) at
>
[jira] [Commented] (FLINK-20495) Elasticsearch6DynamicSinkITCase Hang
[
https://issues.apache.org/jira/browse/FLINK-20495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17279278#comment-17279278
]
Leonard Xu commented on FLINK-20495:
another instance:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12906=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20
> Elasticsearch6DynamicSinkITCase Hang
>
>
> Key: FLINK-20495
> URL: https://issues.apache.org/jira/browse/FLINK-20495
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines, Connectors /
> ElasticSearch, Tests
>Affects Versions: 1.13.0
>Reporter: Huang Xingbo
>Priority: Major
> Labels: test-stability
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10535=logs=d44f43ce-542c-597d-bf94-b0718c71e5e8=03dca39c-73e8-5aaf-601d-328ae5c35f20]
>
> {code:java}
> 2020-12-04T22:39:33.9748225Z [INFO] Running
> org.apache.flink.streaming.connectors.elasticsearch.table.Elasticsearch6DynamicSinkITCase
> 2020-12-04T22:54:51.9486410Z
> ==
> 2020-12-04T22:54:51.9488766Z Process produced no output for 900 seconds.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21203) Don’t collect -U&+U Row When they are equals In the LastRowFunction
[
https://issues.apache.org/jira/browse/FLINK-21203?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17276043#comment-17276043
]
Leonard Xu commented on FLINK-21203:
+1 for the ticket, [~wangpeibin]
About the config option, could you explain more?
> Don’t collect -U&+U Row When they are equals In the LastRowFunction
> -
>
> Key: FLINK-21203
> URL: https://issues.apache.org/jira/browse/FLINK-21203
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Runtime
>Reporter: wangpeibin
>Assignee: wangpeibin
>Priority: Major
>
> In the LastRowFunction , the -U&+U Row will be collected even if they are
> the same, which will increase calculation pressure of the next Operator.
>
> To avoid this, we can optimize the logic of DeduplicateFunctionHelper. Also,
> a config to enable the optimization will be added.
> With the sql followed:
> {quote}select * from
> (select
> *,
> row_number() over (partition by k order by proctime() desc ) as row_num
> from a
> ) t
> where row_num = 1
> {quote}
> Then input 2 row such as :
> {quote}Event("B","1","b"),
> Event("B","1","b")
> {quote}
> Now the output is:
> {quote}(true,+I[B, 1, b, 1])
> (false,-U[B, 1, b, 1])
> (true,+U[B, 1, b, 1])
> {quote}
> After the optimization, the output will be:
> {quote}(true,+I[B, 1, b, 1])
> {quote}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-21183) Support watermark for Hive streaming source
Leonard Xu created FLINK-21183: -- Summary: Support watermark for Hive streaming source Key: FLINK-21183 URL: https://issues.apache.org/jira/browse/FLINK-21183 Project: Flink Issue Type: Improvement Components: Connectors / Hive Reporter: Leonard Xu I've supported streaming hive source in FLINK-17433, but it does not support watermark for streaming hive source yet. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-18903) Flink集成Hive内置时间函数使用报错
[
https://issues.apache.org/jira/browse/FLINK-18903?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu closed FLINK-18903.
--
Resolution: Invalid
Thanks for the report, but please use English in JIRA
> Flink集成Hive内置时间函数使用报错
> -
>
> Key: FLINK-18903
> URL: https://issues.apache.org/jira/browse/FLINK-18903
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.11.1
> Environment: HDP版本：2.4.2.0-258
> Hive版本：1.2.1
> Flink版本：1.11.1
> Flink-sql-client配置见附件：[^sql-client-defaults.yaml]
> Flink jobmanager 启动模式：yarn-session模式
> Flink lib目录已经按官方文档添加如下jar包：
>
> {code:java}
> -rw-r--r-- 1 root root130802 Aug 10 20:09 aircompressor-0.8.jar
> -rw-r--r-- 1 root root 90782 Jul 15 17:24 flink-csv-1.11.1.jar
> -rw-r--r-- 1 root root 108350618 Jul 15 17:30 flink-dist_2.11-1.11.1.jar
> -rw-r--r-- 1 root root 80837 Aug 10 19:41
> flink-hadoop-compatibility_2.11-1.11.1.jar
> -rw-r--r-- 1 root root 94865 Jul 15 17:24 flink-json-1.11.1.jar
> -rw-r--r-- 1 root root 36309656 Aug 11 20:11
> flink-shaded-hadoop-2-uber-2.6.5-10.0.jar
> -rw-r--r-- 1 root root 7712156 Jun 18 10:42
> flink-shaded-zookeeper-3.4.14.jar
> -rw-r--r-- 1 root root 31714606 Aug 10 15:13
> flink-sql-connector-hive-1.2.2_2.11-1.11.0.jar
> -rw-r--r-- 1 root root 3541642 Aug 10 15:12
> flink-sql-connector-kafka_2.11-1.11.0.jar
> -rw-r--r-- 1 root root 2100826 Aug 10 17:13 flink-sql-orc_2.11-1.11.0.jar
> -rw-r--r-- 1 root root 33327194 Jul 15 17:28 flink-table_2.11-1.11.1.jar
> -rw-r--r-- 1 root root 37331759 Jul 15 17:28
> flink-table-blink_2.11-1.11.1.jar
> -rw-r--r-- 1 root root 20599029 Aug 11 22:17 hive-exec-1.2.1.jar
> lrwxrwxrwx 1 root root69 Aug 10 17:15
> hive-metastore-1.2.1000.2.4.2.0-258.jar ->
> /usr/hdp/2.4.2.0-258/hive/lib/hive-metastore-1.2.1000.2.4.2.0-258.jar
> -rw-r--r-- 1 root root918379 Aug 11 22:32
> hive-serde-1.2.1000.2.4.2.0-258.jar
> lrwxrwxrwx 1 root root51 Aug 10 00:19 jersey-core-1.9.jar ->
> /usr/hdp/2.4.2.0-258/hadoop/lib/jersey-core-1.9.jar
> lrwxrwxrwx 1 root root48 Aug 10 17:17 libfb303-0.9.2.jar ->
> /usr/hdp/2.4.2.0-258/hive/lib/libfb303-0.9.2.jar
> -rw-r--r-- 1 root root 67114 Apr 20 20:47 log4j-1.2-api-2.12.1.jar
> -rw-r--r-- 1 root root276771 Apr 20 20:47 log4j-api-2.12.1.jar
> -rw-r--r-- 1 root root 1674433 Apr 20 20:47 log4j-core-2.12.1.jar
> -rw-r--r-- 1 root root 23518 Apr 20 20:47 log4j-slf4j-impl-2.12.1.jar
> -rw-r--r-- 1 root root733071 Aug 10 20:09 orc-core-1.4.3.jar
> {code}
>
>Reporter: 龚旭伦
>Priority: Major
> Labels: flink, hive, sql, 内置函数
> Attachments: flink-sql-client.log, image-2020-08-12-21-59-58-394.png,
> image-2020-08-12-22-17-37-002.png, sql-client-defaults.yaml
>
>
> 【问题描述】
> 我在HDP集群上部署了一套Flink on YARN，并且根据官方文档在Flink
> SQL上集成Hive，集成后发现Hive内置的部分时间函数使用报错（其他函数使用正常），详细如下：
> 报错函数：current_date,current_timestamp
> !image-2020-08-12-21-59-58-394.png|width=585,height=223!
> 详细报错堆栈见附件：[^flink-sql-client.log]
> 【问题分析】
> 1.对current_date函数报错堆栈log的分析，发现产生NullPointerException的代码行为GenericUDFCurrentDate.java:52
> 2.查阅Hive源码：
> !image-2020-08-12-22-17-37-002.png|width=632,height=262!
> 怀疑是SessionState对象在Flink-sql-client中没有初始化而产生的NullPointerException。由于本人知识水平有限，没办法再深入分析其报错原因，麻烦社区帮忙确认并给出修复建议哈。
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-16799) add hive partition limit when read from hive
[ https://issues.apache.org/jira/browse/FLINK-16799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273403#comment-17273403 ] Leonard Xu edited comment on FLINK-16799 at 1/28/21, 8:23 AM: -- [~zhangjun] Could you rebase you code to the latest master? was (Author: leonard xu): [~zhangjun] Any update of this PR? could you rebase you code. > add hive partition limit when read from hive > > > Key: FLINK-16799 > URL: https://issues.apache.org/jira/browse/FLINK-16799 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive >Affects Versions: 1.10.0 >Reporter: Jun Zhang >Assignee: Jun Zhang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Time Spent: 10m > Remaining Estimate: 0h > > add a partition limit when read from hive , a query will not be executed if > it attempts to fetch more partitions per table than the limit configured. > > To avoid full table scans -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-20771) Hive partition is not added when there is a lot of data
[ https://issues.apache.org/jira/browse/FLINK-20771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu closed FLINK-20771. -- Resolution: Fixed > Hive partition is not added when there is a lot of data > --- > > Key: FLINK-20771 > URL: https://issues.apache.org/jira/browse/FLINK-20771 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Reporter: hehuiyuan >Priority: Major > Attachments: image-2020-12-25-18-09-42-707.png, > image-2020-12-25-18-15-07-519.png, image-2020-12-25-18-19-53-746.png > > > Hive partition is not added when the data is huge . > !image-2020-12-25-18-09-42-707.png|width=437,height=115! > Before partition commit, *inProgressPart* will be reinitialize . > the current bucket is active , so the notifyBucketInactive is not executed. > !image-2020-12-25-18-15-07-519.png|width=574,height=192! > > !image-2020-12-25-18-19-53-746.png|width=697,height=56! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-16799) add hive partition limit when read from hive
[ https://issues.apache.org/jira/browse/FLINK-16799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17273403#comment-17273403 ] Leonard Xu commented on FLINK-16799: [~zhangjun] Any update of this PR? could you rebase you code. > add hive partition limit when read from hive > > > Key: FLINK-16799 > URL: https://issues.apache.org/jira/browse/FLINK-16799 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive >Affects Versions: 1.10.0 >Reporter: Jun Zhang >Assignee: Jun Zhang >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > Time Spent: 10m > Remaining Estimate: 0h > > add a partition limit when read from hive , a query will not be executed if > it attempts to fetch more partitions per table than the limit configured. > > To avoid full table scans -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20254) HiveTableSourceITCase.testStreamPartitionReadByCreateTime times out
[
https://issues.apache.org/jira/browse/FLINK-20254?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17272210#comment-17272210
]
Leonard Xu commented on FLINK-20254:
I dig up this and found it's a bug hidden in `ContinuousHiveSplitEnumerator`
and hard to happen:
Only these partitions with same create time are called in different call will
lead to this failure.
But I reproduce it in unit test, I raise up a PR to fix this.
[~lzljs3620320] Appreciate if you can help review.
> HiveTableSourceITCase.testStreamPartitionReadByCreateTime times out
> ---
>
> Key: FLINK-20254
> URL: https://issues.apache.org/jira/browse/FLINK-20254
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Assignee: Leonard Xu
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.13.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=9808=logs=fc5181b0-e452-5c8f-68de-1097947f6483=62110053-334f-5295-a0ab-80dd7e2babbf
> {code}
> 2020-11-19T10:34:23.5591765Z [ERROR] Tests run: 18, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 192.243 s <<< FAILURE! - in
> org.apache.flink.connectors.hive.HiveTableSourceITCase
> 2020-11-19T10:34:23.5593193Z [ERROR]
> testStreamPartitionReadByCreateTime(org.apache.flink.connectors.hive.HiveTableSourceITCase)
> Time elapsed: 120.075 s <<< ERROR!
> 2020-11-19T10:34:23.5593929Z org.junit.runners.model.TestTimedOutException:
> test timed out after 12 milliseconds
> 2020-11-19T10:34:23.5594321Z at java.lang.Thread.sleep(Native Method)
> 2020-11-19T10:34:23.5594777Z at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sleepBeforeRetry(CollectResultFetcher.java:231)
> 2020-11-19T10:34:23.5595378Z at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:119)
> 2020-11-19T10:34:23.5596001Z at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:103)
> 2020-11-19T10:34:23.5596610Z at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:77)
> 2020-11-19T10:34:23.5597218Z at
> org.apache.flink.table.planner.sinks.SelectTableSinkBase$RowIteratorWrapper.hasNext(SelectTableSinkBase.java:115)
> 2020-11-19T10:34:23.5597811Z at
> org.apache.flink.table.api.internal.TableResultImpl$CloseableRowIteratorWrapper.hasNext(TableResultImpl.java:355)
> 2020-11-19T10:34:23.5598555Z at
> org.apache.flink.connectors.hive.HiveTableSourceITCase.fetchRows(HiveTableSourceITCase.java:653)
> 2020-11-19T10:34:23.5599407Z at
> org.apache.flink.connectors.hive.HiveTableSourceITCase.testStreamPartitionReadByCreateTime(HiveTableSourceITCase.java:594)
> 2020-11-19T10:34:23.5599982Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-11-19T10:34:23.5600393Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-11-19T10:34:23.5600865Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-11-19T10:34:23.5601300Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-11-19T10:34:23.5601713Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-11-19T10:34:23.5602211Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-11-19T10:34:23.5602688Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-11-19T10:34:23.5603181Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-11-19T10:34:23.5603753Z at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> 2020-11-19T10:34:23.5604308Z at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> 2020-11-19T10:34:23.5604780Z at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2020-11-19T10:34:23.5605114Z at java.lang.Thread.run(Thread.java:748)
> 2020-11-19T10:34:23.5605299Z
> 2020-11-19T10:34:24.4180149Z [INFO] Running
> org.apache.flink.connectors.hive.TableEnvHiveConnectorITCase
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21013) Blink planner does not ingest timestamp into StreamRecord
[
https://issues.apache.org/jira/browse/FLINK-21013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17271978#comment-17271978
]
Leonard Xu commented on FLINK-21013:
I'd like to fix and back port to 1.11/1.12, please assign this ticket to me.
> Blink planner does not ingest timestamp into StreamRecord
> -
>
> Key: FLINK-21013
> URL: https://issues.apache.org/jira/browse/FLINK-21013
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner, Table SQL / Runtime
>Affects Versions: 1.12.0
>Reporter: Timo Walther
>Priority: Blocker
> Fix For: 1.12.2
>
>
> Currently, the rowtime attribute is not put into the StreamRecord when
> leaving the Table API to DataStream API. The legacy planner supports this,
> but the timestamp is null when using the Blink planner.
> {code}
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setParallelism(1);
> EnvironmentSettings settings =
>
> EnvironmentSettings.newInstance().inStreamingMode().useBlinkPlanner().build();
> StreamTableEnvironment tEnv = StreamTableEnvironment.create(env,
> settings);
> DataStream orderA =
> env.fromCollection(
> Arrays.asList(
> new Order(1L, "beer", 3),
> new Order(1L, "diaper", 4),
> new Order(3L, "rubber", 2)));
> DataStream orderB =
> orderA.assignTimestampsAndWatermarks(
> new AssignerWithPunctuatedWatermarks() {
> @Nullable
> @Override
> public Watermark checkAndGetNextWatermark(
> Order lastElement, long
> extractedTimestamp) {
> return new Watermark(extractedTimestamp);
> }
> @Override
> public long extractTimestamp(Order element, long
> recordTimestamp) {
> return element.user;
> }
> });
> Table tableA = tEnv.fromDataStream(orderB, $("user").rowtime(),
> $("product"), $("amount"));
> // union the two tables
> Table result = tEnv.sqlQuery("SELECT * FROM " + tableA);
> tEnv.toAppendStream(result, Row.class)
> .process(
> new ProcessFunction() {
> @Override
> public void processElement(Row value, Context
> ctx, Collector out)
> throws Exception {
> System.out.println("TIMESTAMP" +
> ctx.timestamp());
> }
> })
> .print();
> env.execute();
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-19436) TPC-DS end-to-end test (Blink planner) failed during shutdown
[
https://issues.apache.org/jira/browse/FLINK-19436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated FLINK-19436:
---
Fix Version/s: 1.11.4
> TPC-DS end-to-end test (Blink planner) failed during shutdown
> -
>
> Key: FLINK-19436
> URL: https://issues.apache.org/jira/browse/FLINK-19436
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner, Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Dian Fu
>Assignee: Leonard Xu
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0, 1.11.4
>
> Attachments: image-2020-11-10-11-08-53-199.png,
> image-2020-11-10-11-09-20-534.png
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=7009=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=2b7514ee-e706-5046-657b-3430666e7bd9
> {code}
> 2020-09-27T22:37:53.2236467Z Stopping taskexecutor daemon (pid: 2992) on host
> fv-az655.
> 2020-09-27T22:37:53.4450715Z Stopping standalonesession daemon (pid: 2699) on
> host fv-az655.
> 2020-09-27T22:37:53.8014537Z Skipping taskexecutor daemon (pid: 11173),
> because it is not running anymore on fv-az655.
> 2020-09-27T22:37:53.8019740Z Skipping taskexecutor daemon (pid: 11561),
> because it is not running anymore on fv-az655.
> 2020-09-27T22:37:53.8022857Z Skipping taskexecutor daemon (pid: 11849),
> because it is not running anymore on fv-az655.
> 2020-09-27T22:37:53.8023616Z Skipping taskexecutor daemon (pid: 12180),
> because it is not running anymore on fv-az655.
> 2020-09-27T22:37:53.8024327Z Skipping taskexecutor daemon (pid: 12950),
> because it is not running anymore on fv-az655.
> 2020-09-27T22:37:53.8025027Z Skipping taskexecutor daemon (pid: 13472),
> because it is not running anymore on fv-az655.
> 2020-09-27T22:37:53.8025727Z Skipping taskexecutor daemon (pid: 16577),
> because it is not running anymore on fv-az655.
> 2020-09-27T22:37:53.8026417Z Skipping taskexecutor daemon (pid: 16959),
> because it is not running anymore on fv-az655.
> 2020-09-27T22:37:53.8027086Z Skipping taskexecutor daemon (pid: 17250),
> because it is not running anymore on fv-az655.
> 2020-09-27T22:37:53.8027770Z Skipping taskexecutor daemon (pid: 17601),
> because it is not running anymore on fv-az655.
> 2020-09-27T22:37:53.8028400Z Stopping taskexecutor daemon (pid: 18438) on
> host fv-az655.
> 2020-09-27T22:37:53.8029314Z
> /home/vsts/work/1/s/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT/bin/taskmanager.sh:
> line 99: 18438 Terminated "${FLINK_BIN_DIR}"/flink-daemon.sh
> $STARTSTOP $ENTRYPOINT "${ARGS[@]}"
> 2020-09-27T22:37:53.8029895Z [FAIL] Test script contains errors.
> 2020-09-27T22:37:53.8032092Z Checking for errors...
> 2020-09-27T22:37:55.3713368Z No errors in log files.
> 2020-09-27T22:37:55.3713935Z Checking for exceptions...
> 2020-09-27T22:37:56.9046391Z No exceptions in log files.
> 2020-09-27T22:37:56.9047333Z Checking for non-empty .out files...
> 2020-09-27T22:37:56.9064402Z No non-empty .out files.
> 2020-09-27T22:37:56.9064859Z
> 2020-09-27T22:37:56.9065588Z [FAIL] 'TPC-DS end-to-end test (Blink planner)'
> failed after 16 minutes and 54 seconds! Test exited with exit code 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
