[jira] [Comment Edited] (FLINK-33470) Implement restore tests for Join node
[ https://issues.apache.org/jira/browse/FLINK-33470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792650#comment-17792650 ] Sergey Nuyanzin edited comment on FLINK-33470 at 12/4/23 7:58 AM: -- I'm sorry however had to revert (also confirmed by [~jhughes]) 18b67b104e025b142a8321e5163edf7fbd439580 and 026bd4be9bafce86ced42d2a07e8b8820f7e6d9d because of ci failures # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=ce3801ad-3bd5-5f06-d165-34d37e757d90=5e4d9387-1dcc-5885-a901-90469b7e6d2f=12081 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=32715a4c-21b8-59a3-4171-744e5ab107eb=ff64056b-5320-5afe-c22c-6fa339e59586=12064 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=f2c100be-250b-5e85-7bbe-176f68fcddc5=05efd11e-5400-54a4-0d27-a4663be008a9=12099 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55160=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12189 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55160=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94=11822 also more details at FLINK-33727 was (Author: sergey nuyanzin): I'm sorry however had to revert (also confirmed by [~jhughes]) 18b67b104e025b142a8321e5163edf7fbd439580 and 026bd4be9bafce86ced42d2a07e8b8820f7e6d9d because of ci failures # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786 # # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=ce3801ad-3bd5-5f06-d165-34d37e757d90=5e4d9387-1dcc-5885-a901-90469b7e6d2f=12081 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=32715a4c-21b8-59a3-4171-744e5ab107eb=ff64056b-5320-5afe-c22c-6fa339e59586=12064 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=f2c100be-250b-5e85-7bbe-176f68fcddc5=05efd11e-5400-54a4-0d27-a4663be008a9=12099 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55160=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12189 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55160=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94=11822 also more details at FLINK-33727 > Implement restore tests for Join node > - > > Key: FLINK-33470 > URL: https://issues.apache.org/jira/browse/FLINK-33470 > Project: Flink > Issue Type: Sub-task >Reporter: Jim Hughes >Assignee: Jim Hughes >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33470) Implement restore tests for Join node
[ https://issues.apache.org/jira/browse/FLINK-33470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792650#comment-17792650 ] Sergey Nuyanzin edited comment on FLINK-33470 at 12/4/23 7:58 AM: -- I'm sorry however had to revert (also confirmed by [~jhughes]) 18b67b104e025b142a8321e5163edf7fbd439580 and 026bd4be9bafce86ced42d2a07e8b8820f7e6d9d because of ci failures # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786 # # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=ce3801ad-3bd5-5f06-d165-34d37e757d90=5e4d9387-1dcc-5885-a901-90469b7e6d2f=12081 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=32715a4c-21b8-59a3-4171-744e5ab107eb=ff64056b-5320-5afe-c22c-6fa339e59586=12064 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=f2c100be-250b-5e85-7bbe-176f68fcddc5=05efd11e-5400-54a4-0d27-a4663be008a9=12099 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55160=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12189 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55160=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94=11822 also more details at FLINK-33727 was (Author: sergey nuyanzin): I'm sorry however had to revert (also confirmed by [~jhughes]) 18b67b104e025b142a8321e5163edf7fbd439580 and 026bd4be9bafce86ced42d2a07e8b8820f7e6d9d because of ci failures # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55160=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12189 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55160=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94=11822 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55160=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94=11884 also more details at FLINK-33727 > Implement restore tests for Join node > - > > Key: FLINK-33470 > URL: https://issues.apache.org/jira/browse/FLINK-33470 > Project: Flink > Issue Type: Sub-task >Reporter: Jim Hughes >Assignee: Jim Hughes >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33470) Implement restore tests for Join node
[ https://issues.apache.org/jira/browse/FLINK-33470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792650#comment-17792650 ] Sergey Nuyanzin edited comment on FLINK-33470 at 12/4/23 7:54 AM: -- I'm sorry however had to revert (also confirmed by [~jhughes]) 18b67b104e025b142a8321e5163edf7fbd439580 and 026bd4be9bafce86ced42d2a07e8b8820f7e6d9d because of ci failures # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55160=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12189 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55160=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94=11822 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55160=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94=11884 also more details at FLINK-33727 was (Author: sergey nuyanzin): I'm sorry however had to revert (also confirmed by [~jhughes]) 18b67b104e025b142a8321e5163edf7fbd439580 and 026bd4be9bafce86ced42d2a07e8b8820f7e6d9d because of ci failures # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55160=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12189 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55160=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94=11884 also more details at FLINK-33727 > Implement restore tests for Join node > - > > Key: FLINK-33470 > URL: https://issues.apache.org/jira/browse/FLINK-33470 > Project: Flink > Issue Type: Sub-task >Reporter: Jim Hughes >Assignee: Jim Hughes >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33731) failover.flip1 package can be rename to failover
[
https://issues.apache.org/jira/browse/FLINK-33731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792696#comment-17792696
]
Zhu Zhu edited comment on FLINK-33731 at 12/4/23 7:46 AM:
--
LGTM. It is a legacy issue. When the new failover logics were introduced, there
were legacy classes directly in package
{{org.apache.flink.runtime.executiongraph.failover}}, so the sub-package was
introduced to avoid mixing them up.
Now that the legacy failover classes are removed already and it's time to do
the renaming.
was (Author: zhuzh):
LGTM. It is a legacy issue. When the new failover logics is introduced, were
legacy classes directly in package
{{org.apache.flink.runtime.executiongraph.failover}}, so the sub-package was
introduced to avoid mixing them up.
Now that the legacy failover classes are removed already and it's time to do
the renaming.
> failover.flip1 package can be rename to failover
>
>
> Key: FLINK-33731
> URL: https://issues.apache.org/jira/browse/FLINK-33731
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Affects Versions: 1.18.0, 1.17.2
>Reporter: Rui Fan
>Assignee: Rui Fan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0
>
>
> Currently, there is a org.apache.flink.runtime.executiongraph.failover.flip1
> package.
> I propose rename the failover.flip1 to failover, in the other word: removing
> the flip1. I have 2 reasons:
> * The naming of the package should be based on business semantics, not FLIP
> number, and the code under the failover.flip1 package has also many changes
> after FLIP-1.
> * All code under the failover.flip1 package are Internal code instead
> @Public code, so they can be renamed directly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33731) failover.flip1 package can be rename to failover
[
https://issues.apache.org/jira/browse/FLINK-33731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792696#comment-17792696
]
Zhu Zhu commented on FLINK-33731:
-
LGTM. It is a legacy issue. When the new failover logics is introduced, were
legacy classes directly in package
{{org.apache.flink.runtime.executiongraph.failover}}, so the sub-package was
introduced to avoid mixing them up.
Now that the legacy failover classes are removed already and it's time to do
the renaming.
> failover.flip1 package can be rename to failover
>
>
> Key: FLINK-33731
> URL: https://issues.apache.org/jira/browse/FLINK-33731
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Affects Versions: 1.18.0, 1.17.2
>Reporter: Rui Fan
>Assignee: Rui Fan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0
>
>
> Currently, there is a org.apache.flink.runtime.executiongraph.failover.flip1
> package.
> I propose rename the failover.flip1 to failover, in the other word: removing
> the flip1. I have 2 reasons:
> * The naming of the package should be based on business semantics, not FLIP
> number, and the code under the failover.flip1 package has also many changes
> after FLIP-1.
> * All code under the failover.flip1 package are Internal code instead
> @Public code, so they can be renamed directly.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLIP-321] Update the docs to add migration periods for deprecated APIs. [flink]
JingGe commented on code in PR #23865:
URL: https://github.com/apache/flink/pull/23865#discussion_r1413472466
##
docs/content/docs/ops/upgrading.md:
##
@@ -60,6 +60,32 @@ Code written against a `PublicEvolving` API in 1.15.2 will
continue to run in 1.
That same code would have to be recompiled when upgrading to 1.16.0 though.
{{< /hint >}}
+### Deprecated API Migration Period
+When an API is deprecated, it is marked with the `@Deprecated` annotation and
a deprecation message is added to the Javadoc.
+According to
[FLIP-321](https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process),
+starting from release 1.18, each deprecated API will have a guaranteed
migration period depending on the API stability level:
+
+|Annotation| Guaranteed Migration Period |
+|::|:--:|
+| `Public` |2 minor releases|
+| `PublicEvolving` |1 minor release |
+| `Experimental` | 1 patch release for the affected minor release |
+
+The source code of a deprecated API will be kept for at least the guaranteed
migration period,
+and may be removed at any point after the migration period has passed.
+
+{{< hint info >}}
+{{< label Example >}}
+Assuming a release sequence of 1.18, 1.19, 1.20, 2.0, 2.1, ..., 3.0,
+- if a `Public` API is deprecated in 1.18, it will not be removed until 2.0.
+- if a `Public` API is deprecated in 1.20, the source code will be kept in 2.0
because the migration period is 2 minor releases. This means the source code
will be removed in 3.0 at the earliest.
Review Comment:
Although the source compatibility described in `API compatibility
guarantees` implicitly said that a `Public` API can only be removed after one
major release, it would be easier for us to follow the deprecation rule
described in this section(i.e. don't make readers think), if we could
explicitly point out that, in this case, for the `Public` API deprecated in
1.20 and carried over to 2.0, even after 2 minor releases, i.e. 2.0 and 2.1, it
still can not be removed in 2.2 and can only be removed in 3.0 at the earliest.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLIP-321] Update the docs to add migration periods for deprecated APIs. [flink]
JingGe commented on code in PR #23865:
URL: https://github.com/apache/flink/pull/23865#discussion_r1413472466
##
docs/content/docs/ops/upgrading.md:
##
@@ -60,6 +60,32 @@ Code written against a `PublicEvolving` API in 1.15.2 will
continue to run in 1.
That same code would have to be recompiled when upgrading to 1.16.0 though.
{{< /hint >}}
+### Deprecated API Migration Period
+When an API is deprecated, it is marked with the `@Deprecated` annotation and
a deprecation message is added to the Javadoc.
+According to
[FLIP-321](https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process),
+starting from release 1.18, each deprecated API will have a guaranteed
migration period depending on the API stability level:
+
+|Annotation| Guaranteed Migration Period |
+|::|:--:|
+| `Public` |2 minor releases|
+| `PublicEvolving` |1 minor release |
+| `Experimental` | 1 patch release for the affected minor release |
+
+The source code of a deprecated API will be kept for at least the guaranteed
migration period,
+and may be removed at any point after the migration period has passed.
+
+{{< hint info >}}
+{{< label Example >}}
+Assuming a release sequence of 1.18, 1.19, 1.20, 2.0, 2.1, ..., 3.0,
+- if a `Public` API is deprecated in 1.18, it will not be removed until 2.0.
+- if a `Public` API is deprecated in 1.20, the source code will be kept in 2.0
because the migration period is 2 minor releases. This means the source code
will be removed in 3.0 at the earliest.
Review Comment:
Although the source compatibility described in `API compatibility
guarantees` implicitly said that a `Public` API can only be removed after one
major release, it would be easier for us to follow the deprecation rule
described in this section, if we could explicitly point out that, in this case,
for the `Public` API deprecated in 1.20 and carried over to 2.0, even after 2
minor releases, i.e. 2.0 and 2.1, it still can not be removed in 2.2 and can
only be removed in 3.0 at the earliest. WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-33734) Merge unaligned checkpoint
Feifan Wang created FLINK-33734: --- Summary: Merge unaligned checkpoint Key: FLINK-33734 URL: https://issues.apache.org/jira/browse/FLINK-33734 Project: Flink Issue Type: Improvement Reporter: Feifan Wang -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [hotfix][core][test] fix ArchUnit typo in pom [flink]
JingGe merged PR #23826: URL: https://github.com/apache/flink/pull/23826 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33722) MATCH_RECOGNIZE in batch mode ignores events order
[
https://issues.apache.org/jira/browse/FLINK-33722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792680#comment-17792680
]
Grzegorz Kołakowski commented on FLINK-33722:
-
Please assign me to the task.
> MATCH_RECOGNIZE in batch mode ignores events order
> --
>
> Key: FLINK-33722
> URL: https://issues.apache.org/jira/browse/FLINK-33722
> Project: Flink
> Issue Type: Bug
> Components: Library / CEP
>Affects Versions: 1.17.1
>Reporter: Grzegorz Kołakowski
>Priority: Major
>
> MATCH_RECOGNIZE in batch mode seems to ignore ORDER BY clause. Let's consider
> the following example:
> {code:sql}
> FROM events
> MATCH_RECOGNIZE (
> PARTITION BY user_id
> ORDER BY ts ASC
> MEASURES
> FIRST(A.ts) as _start,
> LAST(A.ts) as _middle,
> LAST(B.ts) as _finish
> ONE ROW PER MATCH
> AFTER MATCH SKIP PAST LAST ROW
> PATTERN (A{2} B) WITHIN INTERVAL '2' HOURS
> DEFINE
> A AS active is false,
> B AS active is true
> ) AS T {code}
> where _events_ is a Postgresql table containing ~1 records.
> {code:java}
> CREATE TABLE events (
> id INT,
> user_id INT,
> ts TIMESTAMP(3),
> active BOOLEAN,
> WATERMARK FOR ts AS ts - INTERVAL '5' SECOND,
> PRIMARY KEY (id) NOT ENFORCED
> ) WITH (
> 'connector' = 'jdbc',
> 'url' = 'jdbc:postgresql://postgres:5432/test',
> 'username' = 'test',
> 'password' = 'test',
> 'table-name' = 'events'
> ); {code}
> It can happen that _finish is smaller than _start or _middle, which is wrong.
> {noformat}
>user_id _start _middle
> _finish
> 1 2023-11-23 14:34:42.346 2023-11-23 14:34:48.370 2023-11-23
> 14:34:44.264{noformat}
>
> Repository where I reproduced the problem:
> [https://github.com/grzegorz8/flink-match-recognize-in-batch-debugging]
>
>
> According to [~dwysakowicz]: In BATCH the CepOperator is always created to
> process records in processing time:
> [https://github.com/apache/flink/blob/7f7bee70e3ac0d9fb27d7e09b41d6396b748dada/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/batch/BatchExecMatch.java#L54]
> A comparator is passed along to the operator covering the sorting on ts
> field:
> [https://github.com/apache/flink/blob/fea9ffedecf81a97de5c31519ade3bab8228e743/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecMatch.java#L173]
> but this is only secondary sorting. It is applied only within records of the
> same timestamp.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33502) HybridShuffleITCase caused a fatal error
[
https://issues.apache.org/jira/browse/FLINK-33502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792679#comment-17792679
]
Junrui Li commented on FLINK-33502:
---
[~Wencong Liu] Thank you for the clarification. I look forward to seeing your
fix.
> HybridShuffleITCase caused a fatal error
>
>
> Key: FLINK-33502
> URL: https://issues.apache.org/jira/browse/FLINK-33502
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.19.0
>Reporter: Matthias Pohl
>Assignee: Wencong Liu
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0
>
> Attachments: image-2023-11-20-14-37-37-321.png
>
>
> [https://github.com/XComp/flink/actions/runs/6789774296/job/18458197040#step:12:9177]
> {code:java}
> Error: 21:21:35 21:21:35.379 [ERROR] Error occurred in starting fork, check
> output in log
> 9168Error: 21:21:35 21:21:35.379 [ERROR] Process Exit Code: 239
> 9169Error: 21:21:35 21:21:35.379 [ERROR] Crashed tests:
> 9170Error: 21:21:35 21:21:35.379 [ERROR]
> org.apache.flink.test.runtime.HybridShuffleITCase
> 9171Error: 21:21:35 21:21:35.379 [ERROR]
> org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
> 9172Error: 21:21:35 21:21:35.379 [ERROR] Command was /bin/sh -c cd
> /root/flink/flink-tests && /usr/lib/jvm/jdk-11.0.19+7/bin/java -XX:+UseG1GC
> -Xms256m -XX:+IgnoreUnrecognizedVMOptions
> --add-opens=java.base/java.util=ALL-UNNAMED
> --add-opens=java.base/java.io=ALL-UNNAMED -Xmx1536m -jar
> /root/flink/flink-tests/target/surefire/surefirebooter10811559899200556131.jar
> /root/flink/flink-tests/target/surefire 2023-11-07T20-32-50_466-jvmRun4
> surefire6242806641230738408tmp surefire_1603959900047297795160tmp
> 9173Error: 21:21:35 21:21:35.379 [ERROR] Error occurred in starting fork,
> check output in log
> 9174Error: 21:21:35 21:21:35.379 [ERROR] Process Exit Code: 239
> 9175Error: 21:21:35 21:21:35.379 [ERROR] Crashed tests:
> 9176Error: 21:21:35 21:21:35.379 [ERROR]
> org.apache.flink.test.runtime.HybridShuffleITCase
> 9177Error: 21:21:35 21:21:35.379 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:532)
> 9178Error: 21:21:35 21:21:35.379 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:479)
> 9179Error: 21:21:35 21:21:35.379 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:322)
> 9180Error: 21:21:35 21:21:35.379 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:266)
> [...] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [hotfix][core][test] fix ArchUnit typo in pom [flink]
JingGe commented on PR #23826: URL: https://github.com/apache/flink/pull/23826#issuecomment-1837976261 Thanks @TanYuxin-tyx for the review! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33502][network] Make DiskTierProducerAgent#getSegmentIdByIndexOfFirstBufferInSegment return null when the task is released [flink]
WencongLiu commented on PR #23863: URL: https://github.com/apache/flink/pull/23863#issuecomment-1837973875 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-33470) Implement restore tests for Join node
[
https://issues.apache.org/jira/browse/FLINK-33470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792652#comment-17792652
]
Sergey Nuyanzin edited comment on FLINK-33470 at 12/4/23 7:14 AM:
--
a note: how to reproduce it locally
{{ExpandRestoreTest}} should be executed before {{JoinRestoreTest}}
e.g. it could be done like here
{quote}
I was able to find a way to reproduce it locally with 100%.
Just open IntellijIDEA and run all tests for RestoreTestBase
{quote}
was (Author: sergey nuyanzin):
a note: how to reproduce it locally
{quote}
I was able to find a way to reproduce it locally with 100%.
Just open IntellijIDEA and run all tests for RestoreTestBase
{quote}
> Implement restore tests for Join node
> -
>
> Key: FLINK-33470
> URL: https://issues.apache.org/jira/browse/FLINK-33470
> Project: Flink
> Issue Type: Sub-task
>Reporter: Jim Hughes
>Assignee: Jim Hughes
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33500][Runtime] Run storing the JobGraph an asynchronous operation [flink]
flinkbot commented on PR #23866: URL: https://github.com/apache/flink/pull/23866#issuecomment-1837954195 ## CI report: * 8c0d3fdc4aa77efe4a2736044d3715c96e3dacb8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [hotfix][table] refactoring to template method to separate concerns [flink]
fsk119 commented on code in PR #23857:
URL: https://github.com/apache/flink/pull/23857#discussion_r1413440533
##
flink-table/flink-table-common/src/main/java/org/apache/flink/table/utils/print/TableauStyle.java:
##
@@ -115,14 +115,26 @@ public final class TableauStyle implements PrintStyle {
return columnWidths;
}
-@Override
Review Comment:
Why remove Override?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-33500) Run storing the JobGraph an asynchronous operation
[
https://issues.apache.org/jira/browse/FLINK-33500?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-33500:
---
Labels: pull-request-available (was: )
> Run storing the JobGraph an asynchronous operation
> --
>
> Key: FLINK-33500
> URL: https://issues.apache.org/jira/browse/FLINK-33500
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.18.0, 1.17.1, 1.19.0
>Reporter: Matthias Pohl
>Assignee: zhengzhili
>Priority: Major
> Labels: pull-request-available
>
> Currently, submitting a job starts with storing the JobGraph (in HA setups)
> in the {{{}JobGraphStore{}}}. This includes writing the file to S3 (or some
> other remote file system). The job submission is done in the
> {{{}Dispatcher{}}}'s main thread. If writing the {{JobGraph}} is slow, it
> would block any other operation on the {{{}Dispatcher{}}}. See
> [Dispatcher#persistAndRunJob|https://github.com/apache/flink/blob/52cbeb90f32ca36c59590df1daa6748995c9b7f8/flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java#L645]
> as code reference.
> This Jira issue is about moving the job submission into the {{ioExecutor}} as
> an asynchronous call.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [FLINK-33500][Runtime] Run storing the JobGraph an asynchronous operation [flink]
zhengzhili333 opened a new pull request, #23866: URL: https://github.com/apache/flink/pull/23866 ## What is the purpose of the change *Currently, submitting a job starts with storing the JobGraph (in HA setups) in the JobGraphStore. This includes writing the file to S3 (or some other remote file system). The job submission is done in the Dispatcher's main thread. If writing the JobGraph is slow, it would block any other operation on the Dispatcher.* ## Brief change log - *The ZooKeeperStateHandleStore create path in ZooKeeper and locks it then write asynchronously state in Executor* - *The KubernetesStateHandleStore stores key in ConfigMap and write asynchronously state in Executor* - *The dispatcher put JobGraph asynchronously in ioExecutor * - *The dispatcher write To ExecutionGraphInfoStore asynchronously in ioExecutor * ## Verifying this change This change added tests and can be verified as follows: - *Added the Dispatcher JobSubmission test, use ZooKeeperStateHandleStore as JobGraphStore* - *Added the Dispatcher JobSubmission test, use KubernetesStateHandleStore as JobGraphStore* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33386][runtime] Support tasks balancing at slot level for Default Scheduler [flink]
1996fanrui commented on code in PR #23635:
URL: https://github.com/apache/flink/pull/23635#discussion_r1413436882
##
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/GlobalViewDeclarativeSlotPoolBridge.java:
##
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor;
+import org.apache.flink.runtime.jobmaster.JobMasterId;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.taskexecutor.slot.DefaultTimerService;
+import org.apache.flink.runtime.taskexecutor.slot.TimeoutListener;
+import org.apache.flink.runtime.taskexecutor.slot.TimerService;
+import org.apache.flink.runtime.util.ResourceCounter;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.clock.Clock;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+
+/**
+ * {@link SlotPool} implementation which could use the {@link
GlobalViewDeclarativeSlotPoolBridge}
+ * to allocate slots in a global view. Note: It's only used for streaming mode
now.
+ */
+public class GlobalViewDeclarativeSlotPoolBridge extends
DeclarativeSlotPoolBridge
+implements TimeoutListener {
+
+public static final Logger LOG =
+LoggerFactory.getLogger(GlobalViewDeclarativeSlotPoolBridge.class);
+private final Map increasedResourceRequirements;
+
+private final TimerService
timerService;
+
+private final @Nonnull Set receivedNewSlots;
+
+private final @Nonnull Map
preFulfilledFromAvailableSlots;
+private final Time slotRequestMaxInterval;
+
+public GlobalViewDeclarativeSlotPoolBridge(
+JobID jobId,
+DeclarativeSlotPoolFactory declarativeSlotPoolFactory,
+Clock clock,
+Time rpcTimeout,
+Time idleSlotTimeout,
+Time batchSlotTimeout,
+Time slotRequestMaxInterval,
+RequestSlotMatchingStrategy requestSlotMatchingStrategy) {
+super(
+jobId,
+declarativeSlotPoolFactory,
+clock,
+rpcTimeout,
+idleSlotTimeout,
+batchSlotTimeout,
+requestSlotMatchingStrategy);
+this.slotRequestMaxInterval =
Preconditions.checkNotNull(slotRequestMaxInterval);
+this.receivedNewSlots = new HashSet<>();
+this.preFulfilledFromAvailableSlots = new HashMap<>();
+this.increasedResourceRequirements = new HashMap<>();
+this.timerService =
+new DefaultTimerService<>(
+new ScheduledThreadPoolExecutor(1),
+slotRequestMaxInterval.toMilliseconds());
+}
+
+@Override
+protected void internalRequestNewAllocatedSlot(PendingRequest
pendingRequest) {
+pendingRequests.put(pendingRequest.getSlotRequestId(), pendingRequest);
+increasedResourceRequirements.put(pendingRequest.getSlotRequestId(),
false);
+
+timerService.registerTimeout(
+this, slotRequestMaxInterval.getSize(),
slotRequestMaxInterval.getUnit());
+}
+
+@Override
+void newSlotsAreAvailable(Collection newSlots) {
+receivedNewSlots.addAll(newSlots);
+if (newSlots.isEmpty() && receivedNewSlots.isEmpty()) {
+// TODO: Do the matching logic only for available slots.
+}
Re: [PR] [FLINK-33386][runtime] Support tasks balancing at slot level for Default Scheduler [flink]
1996fanrui commented on code in PR #23635:
URL: https://github.com/apache/flink/pull/23635#discussion_r1413436882
##
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/GlobalViewDeclarativeSlotPoolBridge.java:
##
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor;
+import org.apache.flink.runtime.jobmaster.JobMasterId;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.taskexecutor.slot.DefaultTimerService;
+import org.apache.flink.runtime.taskexecutor.slot.TimeoutListener;
+import org.apache.flink.runtime.taskexecutor.slot.TimerService;
+import org.apache.flink.runtime.util.ResourceCounter;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.clock.Clock;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+
+/**
+ * {@link SlotPool} implementation which could use the {@link
GlobalViewDeclarativeSlotPoolBridge}
+ * to allocate slots in a global view. Note: It's only used for streaming mode
now.
+ */
+public class GlobalViewDeclarativeSlotPoolBridge extends
DeclarativeSlotPoolBridge
+implements TimeoutListener {
+
+public static final Logger LOG =
+LoggerFactory.getLogger(GlobalViewDeclarativeSlotPoolBridge.class);
+private final Map increasedResourceRequirements;
+
+private final TimerService
timerService;
+
+private final @Nonnull Set receivedNewSlots;
+
+private final @Nonnull Map
preFulfilledFromAvailableSlots;
+private final Time slotRequestMaxInterval;
+
+public GlobalViewDeclarativeSlotPoolBridge(
+JobID jobId,
+DeclarativeSlotPoolFactory declarativeSlotPoolFactory,
+Clock clock,
+Time rpcTimeout,
+Time idleSlotTimeout,
+Time batchSlotTimeout,
+Time slotRequestMaxInterval,
+RequestSlotMatchingStrategy requestSlotMatchingStrategy) {
+super(
+jobId,
+declarativeSlotPoolFactory,
+clock,
+rpcTimeout,
+idleSlotTimeout,
+batchSlotTimeout,
+requestSlotMatchingStrategy);
+this.slotRequestMaxInterval =
Preconditions.checkNotNull(slotRequestMaxInterval);
+this.receivedNewSlots = new HashSet<>();
+this.preFulfilledFromAvailableSlots = new HashMap<>();
+this.increasedResourceRequirements = new HashMap<>();
+this.timerService =
+new DefaultTimerService<>(
+new ScheduledThreadPoolExecutor(1),
+slotRequestMaxInterval.toMilliseconds());
+}
+
+@Override
+protected void internalRequestNewAllocatedSlot(PendingRequest
pendingRequest) {
+pendingRequests.put(pendingRequest.getSlotRequestId(), pendingRequest);
+increasedResourceRequirements.put(pendingRequest.getSlotRequestId(),
false);
+
+timerService.registerTimeout(
+this, slotRequestMaxInterval.getSize(),
slotRequestMaxInterval.getUnit());
+}
+
+@Override
+void newSlotsAreAvailable(Collection newSlots) {
+receivedNewSlots.addAll(newSlots);
+if (newSlots.isEmpty() && receivedNewSlots.isEmpty()) {
+// TODO: Do the matching logic only for available slots.
+}
Re: [PR] [FLINK-33386][runtime] Support tasks balancing at slot level for Default Scheduler [flink]
1996fanrui commented on code in PR #23635:
URL: https://github.com/apache/flink/pull/23635#discussion_r1413430330
##
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/GlobalViewPhysicalSlotProviderImpl.java:
##
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+/**
+ * Implementation with global view for {@link PhysicalSlotProvider}. Note:
It's only used for
+ * streaming mode now.
+ */
+public class GlobalViewPhysicalSlotProviderImpl extends
PhysicalSlotProviderImpl {
+
+public GlobalViewPhysicalSlotProviderImpl(
+SlotSelectionStrategy slotSelectionStrategy, SlotPool slotPool) {
+super(slotSelectionStrategy, slotPool);
+}
+
+@Override
+protected Map>
tryAllocateFromAvailable(
+Collection slotRequests) {
+Map> availablePhysicalSlots =
new HashMap<>();
+for (PhysicalSlotRequest request : slotRequests) {
+availablePhysicalSlots.put(
Review Comment:
I don't understand why we need this new `tryAllocateFromAvailable`? The old
`tryAllocateFromAvailable` calls
`slotSelectionStrategy.selectBestSlotForProfile` related logic, don't we need
it?
##
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/GlobalViewDeclarativeSlotPoolBridge.java:
##
@@ -0,0 +1,261 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.JobID;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor;
+import org.apache.flink.runtime.jobmaster.JobMasterId;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.taskexecutor.slot.DefaultTimerService;
+import org.apache.flink.runtime.taskexecutor.slot.TimeoutListener;
+import org.apache.flink.runtime.taskexecutor.slot.TimerService;
+import org.apache.flink.runtime.util.ResourceCounter;
+import org.apache.flink.util.FlinkException;
+import org.apache.flink.util.Preconditions;
+import org.apache.flink.util.clock.Clock;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nonnull;
+import javax.annotation.Nullable;
+
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Optional;
+import java.util.Set;
+import java.util.UUID;
+import java.util.concurrent.ScheduledThreadPoolExecutor;
+
+/**
+ * {@link SlotPool} implementation which could use the {@link
GlobalViewDeclarativeSlotPoolBridge}
+ * to allocate slots in a global view. Note: It's only used for streaming mode
now.
+ */
+public class GlobalViewDeclarativeSlotPoolBridge extends
DeclarativeSlotPoolBridge
+implements TimeoutListener {
+
+public static final Logger LOG =
+
Re: [PR] [FLINK-33502][network] Make DiskTierProducerAgent#getSegmentIdByIndexOfFirstBufferInSegment return null when the task is released [flink]
WencongLiu commented on PR #23863: URL: https://github.com/apache/flink/pull/23863#issuecomment-1837925107 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33727][table] Use different sink names for restore tests [flink]
snuyanzin commented on PR #23858: URL: https://github.com/apache/flink/pull/23858#issuecomment-1837922338 closing since finally the change introduced such ci failures was at https://github.com/apache/flink/pull/23861 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33727][table] Use different sink names for restore tests [flink]
snuyanzin closed pull request #23858: [FLINK-33727][table] Use different sink names for restore tests URL: https://github.com/apache/flink/pull/23858 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-33470) Implement restore tests for Join node
[ https://issues.apache.org/jira/browse/FLINK-33470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792650#comment-17792650 ] Sergey Nuyanzin edited comment on FLINK-33470 at 12/4/23 6:27 AM: -- I'm sorry however had to revert (also confirmed by [~jhughes]) 18b67b104e025b142a8321e5163edf7fbd439580 and 026bd4be9bafce86ced42d2a07e8b8820f7e6d9d because of ci failures # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55160=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12189 # https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55160=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94=11884 also more details at FLINK-33727 was (Author: sergey nuyanzin): I'm sorry however had to revert (also confirmed by [~jhughes]) 18b67b104e025b142a8321e5163edf7fbd439580 and 026bd4be9bafce86ced42d2a07e8b8820f7e6d9d because of ci failures https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099 https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786 https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099 https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779 also more details at FLINK-33727 > Implement restore tests for Join node > - > > Key: FLINK-33470 > URL: https://issues.apache.org/jira/browse/FLINK-33470 > Project: Flink > Issue Type: Sub-task >Reporter: Jim Hughes >Assignee: Jim Hughes >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLIP-321] Update the docs to add migration periods for deprecated APIs. [flink]
flinkbot commented on PR #23865: URL: https://github.com/apache/flink/pull/23865#issuecomment-1837911822 ## CI report: * aa4d7a5834e436ea8374efd610bb1e257a9ed5af UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33470) Implement restore tests for Join node
[
https://issues.apache.org/jira/browse/FLINK-33470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792652#comment-17792652
]
Sergey Nuyanzin commented on FLINK-33470:
-
a note: how to reproduce it locally
{quote}
I was able to find a way to reproduce it locally with 100%.
Just open IntellijIDEA and run all tests for RestoreTestBase
{quote}
> Implement restore tests for Join node
> -
>
> Key: FLINK-33470
> URL: https://issues.apache.org/jira/browse/FLINK-33470
> Project: Flink
> Issue Type: Sub-task
>Reporter: Jim Hughes
>Assignee: Jim Hughes
>Priority: Major
> Labels: pull-request-available
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-33727) JoinRestoreTest is failing on AZP
[
https://issues.apache.org/jira/browse/FLINK-33727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin reassigned FLINK-33727:
---
Assignee: Sergey Nuyanzin
> JoinRestoreTest is failing on AZP
> -
>
> Key: FLINK-33727
> URL: https://issues.apache.org/jira/browse/FLINK-33727
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Assignee: Sergey Nuyanzin
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0
>
>
> Since {{JoinRestoreTest}} was introduced in FLINK-33470 it seems to be a
> reason
> {noformat}
> Dec 02 04:42:26 04:42:26.408 [ERROR] Failures:
> Dec 02 04:42:26 04:42:26.408 [ERROR]
> JoinRestoreTest>RestoreTestBase.testRestore:283
> Dec 02 04:42:26 Expecting actual:
> Dec 02 04:42:26 ["+I[9, carol, apple, 9000]",
> Dec 02 04:42:26 "+I[8, bill, banana, 8000]",
> Dec 02 04:42:26 "+I[6, jerry, pen, 6000]"]
> Dec 02 04:42:26 to contain exactly in any order:
> Dec 02 04:42:26 ["+I[Adam, null]",
> Dec 02 04:42:26 "+I[Baker, Research]",
> Dec 02 04:42:26 "+I[Charlie, Human Resources]",
> Dec 02 04:42:26 "+I[Charlie, HR]",
> Dec 02 04:42:26 "+I[Don, Sales]",
> Dec 02 04:42:26 "+I[Victor, null]",
> Dec 02 04:42:26 "+I[Helena, Engineering]",
> Dec 02 04:42:26 "+I[Juliet, Engineering]",
> Dec 02 04:42:26 "+I[Ivana, Research]",
> Dec 02 04:42:26 "+I[Charlie, People Operations]"]
> {noformat}
> examples of failures
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLIP-321] Update the docs to add migration periods for deprecated APIs. [flink]
becketqin commented on PR #23865: URL: https://github.com/apache/flink/pull/23865#issuecomment-1837910118 @mbalassi Do you have time to take a look at the doc update? Thanks! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33727) JoinRestoreTest is failing on AZP
[
https://issues.apache.org/jira/browse/FLINK-33727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792651#comment-17792651
]
Sergey Nuyanzin commented on FLINK-33727:
-
reverted at 18b67b104e025b142a8321e5163edf7fbd439580 and
026bd4be9bafce86ced42d2a07e8b8820f7e6d9d
> JoinRestoreTest is failing on AZP
> -
>
> Key: FLINK-33727
> URL: https://issues.apache.org/jira/browse/FLINK-33727
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0
>
>
> Since {{JoinRestoreTest}} was introduced in FLINK-33470 it seems to be a
> reason
> {noformat}
> Dec 02 04:42:26 04:42:26.408 [ERROR] Failures:
> Dec 02 04:42:26 04:42:26.408 [ERROR]
> JoinRestoreTest>RestoreTestBase.testRestore:283
> Dec 02 04:42:26 Expecting actual:
> Dec 02 04:42:26 ["+I[9, carol, apple, 9000]",
> Dec 02 04:42:26 "+I[8, bill, banana, 8000]",
> Dec 02 04:42:26 "+I[6, jerry, pen, 6000]"]
> Dec 02 04:42:26 to contain exactly in any order:
> Dec 02 04:42:26 ["+I[Adam, null]",
> Dec 02 04:42:26 "+I[Baker, Research]",
> Dec 02 04:42:26 "+I[Charlie, Human Resources]",
> Dec 02 04:42:26 "+I[Charlie, HR]",
> Dec 02 04:42:26 "+I[Don, Sales]",
> Dec 02 04:42:26 "+I[Victor, null]",
> Dec 02 04:42:26 "+I[Helena, Engineering]",
> Dec 02 04:42:26 "+I[Juliet, Engineering]",
> Dec 02 04:42:26 "+I[Ivana, Research]",
> Dec 02 04:42:26 "+I[Charlie, People Operations]"]
> {noformat}
> examples of failures
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-33727) JoinRestoreTest is failing on AZP
[
https://issues.apache.org/jira/browse/FLINK-33727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin resolved FLINK-33727.
-
Fix Version/s: 1.19.0
Resolution: Fixed
> JoinRestoreTest is failing on AZP
> -
>
> Key: FLINK-33727
> URL: https://issues.apache.org/jira/browse/FLINK-33727
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0
>
>
> Since {{JoinRestoreTest}} was introduced in FLINK-33470 it seems to be a
> reason
> {noformat}
> Dec 02 04:42:26 04:42:26.408 [ERROR] Failures:
> Dec 02 04:42:26 04:42:26.408 [ERROR]
> JoinRestoreTest>RestoreTestBase.testRestore:283
> Dec 02 04:42:26 Expecting actual:
> Dec 02 04:42:26 ["+I[9, carol, apple, 9000]",
> Dec 02 04:42:26 "+I[8, bill, banana, 8000]",
> Dec 02 04:42:26 "+I[6, jerry, pen, 6000]"]
> Dec 02 04:42:26 to contain exactly in any order:
> Dec 02 04:42:26 ["+I[Adam, null]",
> Dec 02 04:42:26 "+I[Baker, Research]",
> Dec 02 04:42:26 "+I[Charlie, Human Resources]",
> Dec 02 04:42:26 "+I[Charlie, HR]",
> Dec 02 04:42:26 "+I[Don, Sales]",
> Dec 02 04:42:26 "+I[Victor, null]",
> Dec 02 04:42:26 "+I[Helena, Engineering]",
> Dec 02 04:42:26 "+I[Juliet, Engineering]",
> Dec 02 04:42:26 "+I[Ivana, Research]",
> Dec 02 04:42:26 "+I[Charlie, People Operations]"]
> {noformat}
> examples of failures
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-33727) JoinRestoreTest is failing on AZP
[
https://issues.apache.org/jira/browse/FLINK-33727?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Nuyanzin closed FLINK-33727.
---
> JoinRestoreTest is failing on AZP
> -
>
> Key: FLINK-33727
> URL: https://issues.apache.org/jira/browse/FLINK-33727
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0
>
>
> Since {{JoinRestoreTest}} was introduced in FLINK-33470 it seems to be a
> reason
> {noformat}
> Dec 02 04:42:26 04:42:26.408 [ERROR] Failures:
> Dec 02 04:42:26 04:42:26.408 [ERROR]
> JoinRestoreTest>RestoreTestBase.testRestore:283
> Dec 02 04:42:26 Expecting actual:
> Dec 02 04:42:26 ["+I[9, carol, apple, 9000]",
> Dec 02 04:42:26 "+I[8, bill, banana, 8000]",
> Dec 02 04:42:26 "+I[6, jerry, pen, 6000]"]
> Dec 02 04:42:26 to contain exactly in any order:
> Dec 02 04:42:26 ["+I[Adam, null]",
> Dec 02 04:42:26 "+I[Baker, Research]",
> Dec 02 04:42:26 "+I[Charlie, Human Resources]",
> Dec 02 04:42:26 "+I[Charlie, HR]",
> Dec 02 04:42:26 "+I[Don, Sales]",
> Dec 02 04:42:26 "+I[Victor, null]",
> Dec 02 04:42:26 "+I[Helena, Engineering]",
> Dec 02 04:42:26 "+I[Juliet, Engineering]",
> Dec 02 04:42:26 "+I[Ivana, Research]",
> Dec 02 04:42:26 "+I[Charlie, People Operations]"]
> {noformat}
> examples of failures
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[PR] [FLIP-321] Update the docs to add migration periods for deprecated APIs. [flink]
becketqin opened a new pull request, #23865: URL: https://github.com/apache/flink/pull/23865 ## What is the purpose of the change This PR update the ops doc to add the migration periods for deprecated APIs, which are specified in [FLIP-321](https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process). ## Brief change log See the title. ## Verifying this change Build the docs and check the website locally at the following address: http://localhost:1313/docs/ops/upgrading/#deprecated-api-migration-period ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Reopened] (FLINK-33470) Implement restore tests for Join node
[ https://issues.apache.org/jira/browse/FLINK-33470?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sergey Nuyanzin reopened FLINK-33470: - I'm sorry however had to revert (also confirmed by [~jhughes]) 18b67b104e025b142a8321e5163edf7fbd439580 and 026bd4be9bafce86ced42d2a07e8b8820f7e6d9d because of ci failures https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099 https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786 https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099 https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779 also more details at FLINK-33727 > Implement restore tests for Join node > - > > Key: FLINK-33470 > URL: https://issues.apache.org/jira/browse/FLINK-33470 > Project: Flink > Issue Type: Sub-task >Reporter: Jim Hughes >Assignee: Jim Hughes >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] Revert [FLINK-33470] Implement restore tests for Join node [flink]
snuyanzin merged PR #23861: URL: https://github.com/apache/flink/pull/23861 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-33733) [FLIP-321] Update the stability docs to add the migration periods.
[ https://issues.apache.org/jira/browse/FLINK-33733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jiangjie Qin reassigned FLINK-33733: Assignee: Jiangjie Qin > [FLIP-321] Update the stability docs to add the migration periods. > -- > > Key: FLINK-33733 > URL: https://issues.apache.org/jira/browse/FLINK-33733 > Project: Flink > Issue Type: New Feature > Components: Documentation >Affects Versions: 1.19.0 >Reporter: Jiangjie Qin >Assignee: Jiangjie Qin >Priority: Major > > This ticket updates the docs to add the migration period for deprecated APIs, > which are specified in > [FLIP-321.|https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33724) Application mode doesn't support multiple lines argument
[
https://issues.apache.org/jira/browse/FLINK-33724?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jiaxing Chen updated FLINK-33724:
-
Description:
When running a job in application mode, flink will save program arguments in
flink-conf.yaml. These arguments will be loaded when deploying an application
on YARN. However multiple lines argument cannot be loaded properly.
In my case, I submit a job with sql which is a multiple lines argument:
{code:java}
/bin/flink run-application
-t yarn-application
...
myjob.jar
"INSERT INTO tableA
SELECT
a,
b,
c
FROM tableB;"{code}
In flink-conf.yaml，it saved as:
{code:java}
$internal.application.program-args： "INSERT INTO tableA
SELECT
a,
b,
c
FROM tableB;"{code}
And produced some warning logs when loaded from flink-conf.yaml :
{code:java}
INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: $internal.application.program-args, "INSERT INTO tableA
SELECT a,
WARN org.apache.flink.configuration.GlobalConfiguration [] - Error
while trying to split key and value in configuration file /{working
dir}/flink-conf.yaml:{line num} "SELECT "
...{code}
I dig into the source code and find out that the reader and writer methods of
YAML file cannot properly parse YAML format. Maybe it would be better to use
some third-party tools, such as snakeyaml, to do this job.
was:
When running a job in application mode, program arguments are stored in the
flink-conf.yaml file. These arguments will be loaded when deploying an
application on YARN. However multiple lines argument cannot be loaded properly.
In my case, I submit a job with sql which is a multiple lines argument:
{code:java}
/bin/flink run-application
-t yarn-application
...
myjob.jar
"INSERT INTO tableA
SELECT
a,
b,
c
FROM tableB;"{code}
In flink-conf.yaml，it saved as:
{code:java}
$internal.application.program-args： "INSERT INTO tableA
SELECT
a,
b,
c
FROM tableB;"{code}
And produced some warning logs when loaded from flink-conf.yaml :
{code:java}
INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading
configuration property: $internal.application.program-args, "INSERT INTO tableA
SELECT a,
WARN org.apache.flink.configuration.GlobalConfiguration [] - Error
while trying to split key and value in configuration file /{working
dir}/flink-conf.yaml:{line num} "SELECT "
...{code}
I dig into the source code and find out that the reader and writer methods of
YAML file cannot properly parse YAML format. Maybe it would be better to use
some third-party tools, such as snakeyaml, to do this job.
> Application mode doesn't support multiple lines argument
>
>
> Key: FLINK-33724
> URL: https://issues.apache.org/jira/browse/FLINK-33724
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.18.0
>Reporter: Jiaxing Chen
>Priority: Major
>
> When running a job in application mode, flink will save program arguments in
> flink-conf.yaml. These arguments will be loaded when deploying an application
> on YARN. However multiple lines argument cannot be loaded properly.
> In my case, I submit a job with sql which is a multiple lines argument:
>
> {code:java}
> /bin/flink run-application
> -t yarn-application
> ...
> myjob.jar
> "INSERT INTO tableA
> SELECT
> a,
> b,
> c
> FROM tableB;"{code}
> In flink-conf.yaml，it saved as:
> {code:java}
> $internal.application.program-args： "INSERT INTO tableA
> SELECT
> a,
> b,
> c
> FROM tableB;"{code}
> And produced some warning logs when loaded from flink-conf.yaml :
> {code:java}
> INFO org.apache.flink.configuration.GlobalConfiguration [] -
> Loading configuration property: $internal.application.program-args, "INSERT
> INTO tableA SELECT a,
> WARN org.apache.flink.configuration.GlobalConfiguration [] - Error
> while trying to split key and value in configuration file /{working
> dir}/flink-conf.yaml:{line num} "SELECT "
> ...{code}
> I dig into the source code and find out that the reader and writer methods of
> YAML file cannot properly parse YAML format. Maybe it would be better to use
> some third-party tools, such as snakeyaml, to do this job.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-33733) [FLIP-321] Update the stability docs to add the migration periods.
Jiangjie Qin created FLINK-33733: Summary: [FLIP-321] Update the stability docs to add the migration periods. Key: FLINK-33733 URL: https://issues.apache.org/jira/browse/FLINK-33733 Project: Flink Issue Type: New Feature Components: Documentation Affects Versions: 1.19.0 Reporter: Jiangjie Qin This ticket updates the docs to add the migration period for deprecated APIs, which are specified in [FLIP-321.|https://cwiki.apache.org/confluence/display/FLINK/FLIP-321%3A+Introduce+an+API+deprecation+process] -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33581][core] Deprecate configuration getters/setters that return/set complex Java objects in python flink. [flink]
flinkbot commented on PR #23864: URL: https://github.com/apache/flink/pull/23864#issuecomment-1837896315 ## CI report: * 3fcaa9a433c4c20a27fd4775b2fe0e78a83b2d81 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-33732) Hive Lookup Join ProjectPushDown will encouter ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/FLINK-33732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lekelei closed FLINK-33732.
---
Resolution: Fixed
> Hive Lookup Join ProjectPushDown will encouter ArrayIndexOutOfBoundsException
> -
>
> Key: FLINK-33732
> URL: https://issues.apache.org/jira/browse/FLINK-33732
> Project: Flink
> Issue Type: Bug
>Reporter: lekelei
>Priority: Major
>
> Here is my sql:
> ```
> SET 'execution.runtime-mode' = 'streaming';
> SET 'table.dynamic-table-options.enabled' = 'true';
> SET 'table.exec.source.cdc-events-duplicate' = 'false';
> SET 'pipeline.operator-chaining' = 'false';
> CREATE CATALOG catalog_hive WITH (
> 'type' = 'hive',
> ...
> );
>
> create table kafka_source(
> item1 STRING,
> item2 INT,
> item3 string,
> PRIMARY KEY (item1, item2) NOT ENFORCED,
> process_time as proctime()
> – WATERMARK FOR `ts` AS ts - INTERVAL '10' SECOND
> ) WITH (
> 'connector' = 'kafka',
> 'format' = 'json',
> ...
> );
>
> CREATE TABLE blackhole_sink (
> comp STRING,
> order STRING,
> order_line INT,
> order_sequence INT,
> material_code STRING,
> warehouse_code STRING,
> quantity DOUBLE
> )WITH (
> 'connector' = 'blackhole'
> );
>
> insert into
> blackhole_sink
> select item1,comp from kafka_source a left join
> catalog_hive.db.hive_lookup_tb
> /*+ OPTIONS('streaming-source.partition.include'='latest',
> 'streaming-source.monitor-interval'='60
> min','streaming-source.enable'='true') */
> FOR SYSTEM_TIME AS OF kafka_source.process_time as b on a.item1 = b.comp;
> ```
> The error stack is as follows：
> java.io.IOException: java.io.IOException:
> java.lang.ArrayIndexOutOfBoundsException: 10
> at
> com.netease.sloth.flink.connector.filesystem.table.meta.TableMetaStore.lambda$runSecured$1(TableMetaStore.java:700)
> ~[classes/:?]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_332]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_332]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
> ~[hadoop-common-2.7.5.jar:?]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33732) Hive Lookup Join ProjectPushDown will encouter ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/FLINK-33732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792646#comment-17792646
]
lekelei commented on FLINK-33732:
-
Duplicates the issue [FLINK-30679] Can not load the data of hive dim table when
project-push-down is introduced - ASF JIRA (apache.org), so closed.
> Hive Lookup Join ProjectPushDown will encouter ArrayIndexOutOfBoundsException
> -
>
> Key: FLINK-33732
> URL: https://issues.apache.org/jira/browse/FLINK-33732
> Project: Flink
> Issue Type: Bug
>Reporter: lekelei
>Priority: Major
>
> Here is my sql:
> ```
> SET 'execution.runtime-mode' = 'streaming';
> SET 'table.dynamic-table-options.enabled' = 'true';
> SET 'table.exec.source.cdc-events-duplicate' = 'false';
> SET 'pipeline.operator-chaining' = 'false';
> CREATE CATALOG catalog_hive WITH (
> 'type' = 'hive',
> ...
> );
>
> create table kafka_source(
> item1 STRING,
> item2 INT,
> item3 string,
> PRIMARY KEY (item1, item2) NOT ENFORCED,
> process_time as proctime()
> – WATERMARK FOR `ts` AS ts - INTERVAL '10' SECOND
> ) WITH (
> 'connector' = 'kafka',
> 'format' = 'json',
> ...
> );
>
> CREATE TABLE blackhole_sink (
> comp STRING,
> order STRING,
> order_line INT,
> order_sequence INT,
> material_code STRING,
> warehouse_code STRING,
> quantity DOUBLE
> )WITH (
> 'connector' = 'blackhole'
> );
>
> insert into
> blackhole_sink
> select item1,comp from kafka_source a left join
> catalog_hive.db.hive_lookup_tb
> /*+ OPTIONS('streaming-source.partition.include'='latest',
> 'streaming-source.monitor-interval'='60
> min','streaming-source.enable'='true') */
> FOR SYSTEM_TIME AS OF kafka_source.process_time as b on a.item1 = b.comp;
> ```
> The error stack is as follows：
> java.io.IOException: java.io.IOException:
> java.lang.ArrayIndexOutOfBoundsException: 10
> at
> com.netease.sloth.flink.connector.filesystem.table.meta.TableMetaStore.lambda$runSecured$1(TableMetaStore.java:700)
> ~[classes/:?]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_332]
> at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_332]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
> ~[hadoop-common-2.7.5.jar:?]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33581][core] Deprecate configuration getters/setters that return/set complex Java objects in python flink. [flink]
JunRuiLee commented on PR #23864: URL: https://github.com/apache/flink/pull/23864#issuecomment-1837890764 Hi, @HuangXingBo , Would you be able to take a moment to review this PR? Thank you in advance! -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-33581][core] Deprecate configuration getters/setters that return/set complex Java objects in python flink. [flink]
JunRuiLee opened a new pull request, #23864: URL: https://github.com/apache/flink/pull/23864 ## What is the purpose of the change Deprecate configuration getters/setters that return/set complex Java objects in python flink. ## Brief change log - *Deprecate getter/setter methods related to state backend in python flink.* - *Deprecate getter/setter methods related to CheckpointStorage in python flink.* - *Deprecate getter/setter methods related in the RestartStrategy in python flink.* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (**yes** / no) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33647] Implement restore tests for LookupJoin node [flink]
bvarghese1 commented on code in PR #23814:
URL: https://github.com/apache/flink/pull/23814#discussion_r1413375613
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/LookupJoinTestPrograms.java:
##
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.nodes.exec.stream;
+
+import org.apache.flink.table.test.program.SinkTestStep;
+import org.apache.flink.table.test.program.SourceTestStep;
+import org.apache.flink.table.test.program.TableTestProgram;
+import org.apache.flink.types.Row;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/** {@link TableTestProgram} definitions for testing {@link
StreamExecGroupWindowAggregate}. */
+public class LookupJoinTestPrograms {
+
+static final SourceTestStep CUSTOMERS =
+SourceTestStep.newBuilder("customers_t") // static table
+.addOption("disable-lookup", "false")
+.addOption("filterable-fields", "age")
+.addSchema(
+"id INT PRIMARY KEY NOT ENFORCED",
+"name STRING",
+"age INT",
+"city STRING",
+"state STRING",
+"zipcode INT")
+.producedBeforeRestore(
+Row.of(1, "Bob", 28, "Mountain View",
"California", 94043),
+Row.of(2, "Alice", 32, "San Francisco",
"California", 95016),
+Row.of(3, "Claire", 37, "Austin", "Texas", 73301),
+Row.of(4, "Shannon", 29, "Boise", "Idaho", 83701),
+Row.of(5, "Jake", 42, "New York City", "New York",
10001))
+// Note: Before data state is not persisted for static
tables during savepoint
+.producedAfterRestore(
+Row.of(1, "Bob", 28, "San Jose", "California",
94089),
+Row.of(6, "Joana", 54, "Atlanta", "Georgia",
30033))
+.build();
+
+static final SourceTestStep ORDERS =
+SourceTestStep.newBuilder("orders_t")
+.addOption("filterable-fields", "customer_id")
+.addSchema(
+"order_id INT",
+"customer_id INT",
+"total DOUBLE",
+"order_time STRING",
+"proc_time AS PROCTIME()")
+.producedBeforeRestore(
+Row.of(1, 3, 44.44, "2020-10-10 00:00:01"),
+Row.of(2, 5, 100.02, "2020-10-10 00:00:02"),
+Row.of(4, 2, 92.61, "2020-10-10 00:00:04"),
+Row.of(3, 1, 23.89, "2020-10-10 00:00:03"),
+Row.of(6, 4, 7.65, "2020-10-10 00:00:06"),
+Row.of(5, 2, 12.78, "2020-10-10 00:00:05"))
+.producedAfterRestore(
+Row.of(7, 6, 17.58, "2020-10-10 00:00:07"), // new
customer
+Row.of(9, 1, 143.21, "2020-10-10 00:00:08") //
updated zip code
+)
+.build();
+
+static final List SINK_SCHEMA =
+Arrays.asList(
+"order_id INT",
+"total DOUBLE",
+"id INT",
+"name STRING",
+"age INT",
+"city STRING",
+"state STRING",
+"zipcode INT");
+
+static final TableTestProgram LOOKUP_JOIN_PROJECT_PUSHDOWN =
+TableTestProgram.of(
+"lookup-join-project-pushdown",
+"validates lookup join with project pushdown")
+.setupTableSource(CUSTOMERS)
+.setupTableSource(ORDERS)
+.setupTableSink(
+SinkTestStep.newBuilder("sink_t")
+
Re: [PR] [FLINK-33647] Implement restore tests for LookupJoin node [flink]
bvarghese1 commented on code in PR #23814:
URL: https://github.com/apache/flink/pull/23814#discussion_r1413375404
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/LookupJoinTestPrograms.java:
##
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.nodes.exec.stream;
+
+import org.apache.flink.table.test.program.SinkTestStep;
+import org.apache.flink.table.test.program.SourceTestStep;
+import org.apache.flink.table.test.program.TableTestProgram;
+import org.apache.flink.types.Row;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/** {@link TableTestProgram} definitions for testing {@link
StreamExecGroupWindowAggregate}. */
+public class LookupJoinTestPrograms {
+
+static final SourceTestStep CUSTOMERS =
+SourceTestStep.newBuilder("customers_t") // static table
+.addOption("disable-lookup", "false")
+.addOption("filterable-fields", "age")
+.addSchema(
+"id INT PRIMARY KEY NOT ENFORCED",
+"name STRING",
+"age INT",
+"city STRING",
+"state STRING",
+"zipcode INT")
+.producedBeforeRestore(
+Row.of(1, "Bob", 28, "Mountain View",
"California", 94043),
+Row.of(2, "Alice", 32, "San Francisco",
"California", 95016),
+Row.of(3, "Claire", 37, "Austin", "Texas", 73301),
+Row.of(4, "Shannon", 29, "Boise", "Idaho", 83701),
+Row.of(5, "Jake", 42, "New York City", "New York",
10001))
+// Note: Before data state is not persisted for static
tables during savepoint
+.producedAfterRestore(
+Row.of(1, "Bob", 28, "San Jose", "California",
94089),
+Row.of(6, "Joana", 54, "Atlanta", "Georgia",
30033))
+.build();
+
+static final SourceTestStep ORDERS =
+SourceTestStep.newBuilder("orders_t")
+.addOption("filterable-fields", "customer_id")
+.addSchema(
+"order_id INT",
+"customer_id INT",
+"total DOUBLE",
+"order_time STRING",
+"proc_time AS PROCTIME()")
+.producedBeforeRestore(
+Row.of(1, 3, 44.44, "2020-10-10 00:00:01"),
+Row.of(2, 5, 100.02, "2020-10-10 00:00:02"),
+Row.of(4, 2, 92.61, "2020-10-10 00:00:04"),
+Row.of(3, 1, 23.89, "2020-10-10 00:00:03"),
+Row.of(6, 4, 7.65, "2020-10-10 00:00:06"),
+Row.of(5, 2, 12.78, "2020-10-10 00:00:05"))
+.producedAfterRestore(
+Row.of(7, 6, 17.58, "2020-10-10 00:00:07"), // new
customer
+Row.of(9, 1, 143.21, "2020-10-10 00:00:08") //
updated zip code
+)
+.build();
+
+static final List SINK_SCHEMA =
+Arrays.asList(
+"order_id INT",
+"total DOUBLE",
+"id INT",
+"name STRING",
+"age INT",
+"city STRING",
+"state STRING",
+"zipcode INT");
+
+static final TableTestProgram LOOKUP_JOIN_PROJECT_PUSHDOWN =
+TableTestProgram.of(
+"lookup-join-project-pushdown",
+"validates lookup join with project pushdown")
+.setupTableSource(CUSTOMERS)
+.setupTableSource(ORDERS)
+.setupTableSink(
+SinkTestStep.newBuilder("sink_t")
+
Re: [PR] [FLINK-33647] Implement restore tests for LookupJoin node [flink]
bvarghese1 commented on code in PR #23814:
URL: https://github.com/apache/flink/pull/23814#discussion_r1413375281
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/LookupJoinTestPrograms.java:
##
@@ -0,0 +1,287 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.plan.nodes.exec.stream;
+
+import org.apache.flink.table.test.program.SinkTestStep;
+import org.apache.flink.table.test.program.SourceTestStep;
+import org.apache.flink.table.test.program.TableTestProgram;
+import org.apache.flink.types.Row;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.stream.Collectors;
+
+/** {@link TableTestProgram} definitions for testing {@link
StreamExecGroupWindowAggregate}. */
+public class LookupJoinTestPrograms {
+
+static final SourceTestStep CUSTOMERS =
+SourceTestStep.newBuilder("customers_t") // static table
+.addOption("disable-lookup", "false")
+.addOption("filterable-fields", "age")
+.addSchema(
+"id INT PRIMARY KEY NOT ENFORCED",
+"name STRING",
+"age INT",
+"city STRING",
+"state STRING",
+"zipcode INT")
+.producedBeforeRestore(
+Row.of(1, "Bob", 28, "Mountain View",
"California", 94043),
+Row.of(2, "Alice", 32, "San Francisco",
"California", 95016),
+Row.of(3, "Claire", 37, "Austin", "Texas", 73301),
+Row.of(4, "Shannon", 29, "Boise", "Idaho", 83701),
+Row.of(5, "Jake", 42, "New York City", "New York",
10001))
+// Note: Before data state is not persisted for static
tables during savepoint
+.producedAfterRestore(
+Row.of(1, "Bob", 28, "San Jose", "California",
94089),
+Row.of(6, "Joana", 54, "Atlanta", "Georgia",
30033))
Review Comment:
I think its fine. I have updated the comment to clarify this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33669][doc] Update the documentation for RestartStrategy, Checkpoint Storage, and State Backend. [flink]
1996fanrui commented on code in PR #23847:
URL: https://github.com/apache/flink/pull/23847#discussion_r1413373818
##
docs/content.zh/docs/ops/state/task_failure_recovery.md:
##
@@ -49,20 +49,18 @@ Flink 作业如果没有定义重启策略，则会遵循集群启动时加载
{{< generated/restart_strategy_configuration >}}
除了定义默认的重启策略以外，还可以为每个 Flink 作业单独定义重启策略。
-这个重启策略通过在程序中的 `StreamExecutionEnvironment` 对象上调用 `setRestartStrategy` 方法来设置。
-当然，对于 `StreamExecutionEnvironment` 也同样适用。
下例展示了如何给我们的作业设置固定延时重启策略。
如果发生故障，系统会重启作业 3 次，每两次连续的重启尝试之间等待 10 秒钟。
{{< tabs "2b011473-9a34-4e7b-943b-be4a9071fe3c" >}}
{{< tab "Java" >}}
Review Comment:
Thanks for your feedback, sounds make sense to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33647] Implement restore tests for LookupJoin node [flink]
bvarghese1 commented on code in PR #23814:
URL: https://github.com/apache/flink/pull/23814#discussion_r1413356579
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/stream/LookupJoinJsonPlanTest.java:
##
@@ -1,273 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one
- * or more contributor license agreements. See the NOTICE file
- * distributed with this work for additional information
- * regarding copyright ownership. The ASF licenses this file
- * to you under the Apache License, Version 2.0 (the
- * "License"); you may not use this file except in compliance
- * with the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.flink.table.planner.plan.nodes.exec.stream;
-
-import org.apache.flink.api.common.typeinfo.Types;
-import org.apache.flink.table.api.TableConfig;
-import org.apache.flink.table.api.TableEnvironment;
-import org.apache.flink.table.api.TableSchema;
-import org.apache.flink.table.api.ValidationException;
-import org.apache.flink.table.api.config.OptimizerConfigOptions;
-import
org.apache.flink.table.planner.runtime.utils.InMemoryLookupableTableSource;
-import org.apache.flink.table.planner.utils.StreamTableTestUtil;
-import org.apache.flink.table.planner.utils.TableTestBase;
-import org.apache.flink.types.Row;
-
-import org.junit.jupiter.api.BeforeEach;
-import org.junit.jupiter.api.Test;
-
-import java.util.ArrayList;
-
-import scala.collection.JavaConverters;
-
-import static org.apache.flink.core.testutils.FlinkAssertions.anyCauseMatches;
-import static org.assertj.core.api.Assertions.assertThatThrownBy;
-
-/** Test json serialization/deserialization for LookupJoin. */
-class LookupJoinJsonPlanTest extends TableTestBase {
-
-private StreamTableTestUtil util;
-private TableEnvironment tEnv;
-
-@BeforeEach
-void setup() {
-util = streamTestUtil(TableConfig.getDefault());
-tEnv = util.getTableEnv();
-
-String srcTableA =
-"CREATE TABLE MyTable (\n"
-+ " a int,\n"
-+ " b varchar,\n"
-+ " c bigint,\n"
-+ " proctime as PROCTIME(),\n"
-+ " rowtime as TO_TIMESTAMP(FROM_UNIXTIME(c)),\n"
-+ " watermark for rowtime as rowtime - INTERVAL '1'
second \n"
-+ ") with (\n"
-+ " 'connector' = 'values',\n"
-+ " 'bounded' = 'false')";
-String srcTableB =
-"CREATE TABLE LookupTable (\n"
-+ " id int,\n"
-+ " name varchar,\n"
-+ " age int \n"
-+ ") with (\n"
-+ " 'connector' = 'values',\n"
-+ " 'bounded' = 'false')";
-String sinkTable1 =
-"CREATE TABLE Sink1 (\n"
-+ " a int,\n"
-+ " name varchar,"
-+ " age int"
-+ ") with (\n"
-+ " 'connector' = 'values',\n"
-+ " 'sink-insert-only' = 'false')";
-String sinkTable2 =
-"CREATE TABLE MySink1 (\n"
-+ " a int,\n"
-+ " b varchar,"
-+ " c bigint,"
-+ " proctime timestamp(3),"
-+ " rowtime timestamp(3),"
-+ " id int,"
-+ " name varchar,"
-+ " age int"
-+ ") with (\n"
-+ " 'connector' = 'values',\n"
-+ " 'table-sink-class' = 'DEFAULT')";
-tEnv.executeSql(srcTableA);
-tEnv.executeSql(srcTableB);
-tEnv.executeSql(sinkTable1);
-tEnv.executeSql(sinkTable2);
-}
-
-@Test
-void testJoinTemporalTable() {
-String sinkTableDdl =
-"CREATE TABLE MySink (\n"
-+ " a int,\n"
-+ " b varchar,"
-+ " c bigint,"
-+ " proctime timestamp(3),"
-+ " rowtime timestamp(3),"
-+ " id int,"
-+ " name varchar,"
-+ " age int"
-+ ") with (\n"
-+ " 'connector' = 'values',\n"
-
Re: [PR] [FLINK-33669][doc] Update the documentation for RestartStrategy, Checkpoint Storage, and State Backend. [flink]
JunRuiLee commented on code in PR #23847:
URL: https://github.com/apache/flink/pull/23847#discussion_r1413345247
##
docs/content/docs/ops/state/task_failure_recovery.md:
##
@@ -117,11 +116,11 @@ The fixed delay restart strategy can also be set
programmatically:
{{< tabs "73f5d009-b9af-4bfe-be22-d1c4659fd1ec" >}}
{{< tab "Java" >}}
```java
-StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
-env.setRestartStrategy(RestartStrategies.fixedDelayRestart(
- 3, // number of restart attempts
- Time.of(10, TimeUnit.SECONDS) // delay
-));
+Configuration config = new Configuration();
+config.set(RestartStrategyOptions.RESTART_STRATEGY, "fixed-delay");
+config.set(RestartStrategyOptions.RESTART_STRATEGY_FIXED_DELAY_ATTEMPTS, 3);
// number of restart attempts
+config.set(RestartStrategyOptions.RESTART_STRATEGY_FIXED_DELAY_DELAY,
Duration.ofSeconds(10)); // delay
+StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment(config);
```
Review Comment:
Thank you for bringing this up. I do see the value in providing a
programmatic set example. While it's true that the YAML demo illustrates the
default configuration through the conf file, offering a code snippet showing
how to set the restart strategy programmatically can be beneficial. This would
highlight that the restart strategy can be tailored at the job level, giving
users the flexibility to make adjustments within their code, which can be
particularly useful in scenarios where different jobs may require different
strategies.
Informing users about both methods empowers them with a choice to select the
most appropriate configuration approach based on their specific use case.
Looking forward to your thoughts on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33669][doc] Update the documentation for RestartStrategy, Checkpoint Storage, and State Backend. [flink]
JunRuiLee commented on code in PR #23847:
URL: https://github.com/apache/flink/pull/23847#discussion_r1413345096
##
docs/content.zh/docs/ops/state/task_failure_recovery.md:
##
@@ -49,20 +49,18 @@ Flink 作业如果没有定义重启策略，则会遵循集群启动时加载
{{< generated/restart_strategy_configuration >}}
除了定义默认的重启策略以外，还可以为每个 Flink 作业单独定义重启策略。
-这个重启策略通过在程序中的 `StreamExecutionEnvironment` 对象上调用 `setRestartStrategy` 方法来设置。
-当然，对于 `StreamExecutionEnvironment` 也同样适用。
下例展示了如何给我们的作业设置固定延时重启策略。
如果发生故障，系统会重启作业 3 次，每两次连续的重启尝试之间等待 10 秒钟。
{{< tabs "2b011473-9a34-4e7b-943b-be4a9071fe3c" >}}
{{< tab "Java" >}}
Review Comment:
I appreciate your suggestion to streamline the documentation by focusing on
the options method across all languages. However, I believe it is still
essential to retain code guides for each language to ensure users have clear,
language-specific examples for configuring their Flink jobs.
Regarding the Scala API, it has indeed been deprecated as per FLIP-265 and
is scheduled to be removed in Flink 2.0. I prefer that the Scala-specific
documentation should be removed along with the actual code deprecation in the
upcoming major release.
In the case of the Python API, it was an oversight on my part not to have
deprecated it earlier. I will address this by opening a PR to deprecate the
Python API soon. Once that PR is merged, I will update the documentation
accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33669][doc] Update the documentation for RestartStrategy, Checkpoint Storage, and State Backend. [flink]
JunRuiLee commented on code in PR #23847:
URL: https://github.com/apache/flink/pull/23847#discussion_r1413344964
##
docs/content.zh/docs/deployment/filesystems/gcs.md:
##
@@ -44,7 +44,10 @@ env.readTextFile("gs:///");
stream.writeAsText("gs:///");
// Use GCS as checkpoint storage
Review Comment:
You are absolutely right, and I appreciate your attention to detail. I
overlooked the fact that the comments in other filesystem-related Chinese
documentation were translated. To maintain consistency throughout the
documentation, I will update this PR accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-32611] Redirect to Apache Paimon's link instead of legacy flink table store [flink-web]
JingsongLi merged PR #665: URL: https://github.com/apache/flink-web/pull/665 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33502][network] Make DiskTierProducerAgent#getSegmentIdByIndexOfFirstBufferInSegment return null when the task is released [flink]
flinkbot commented on PR #23863: URL: https://github.com/apache/flink/pull/23863#issuecomment-1837775718 ## CI report: * 5f075724d2329b6a239d681b384ed4ac0d289144 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33731][Scheduler] Remove the redundant flip1 package [flink]
flinkbot commented on PR #23862: URL: https://github.com/apache/flink/pull/23862#issuecomment-1837775224 ## CI report: * f3b29fe0bffcd2511bfaf4f3c3284d0751120259 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-33502][network] Make DiskTierProducerAgent#getSegmentIdByIndexOfFirstBufferInSegment return null when the task is released [flink]
WencongLiu opened a new pull request, #23863: URL: https://github.com/apache/flink/pull/23863 ## What is the purpose of the change *Make DiskTierProducerAgent#getSegmentIdByIndexOfFirstBufferInSegment return null when the task is released.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33541][table-planner] function RAND and RAND_INTEGER should return type nullable if the arguments are nullable [flink]
schnappi17 commented on PR #23779: URL: https://github.com/apache/flink/pull/23779#issuecomment-1837771000 Hi @xuyangzhong, any progress on this? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33731) failover.flip1 package can be rename to failover
[ https://issues.apache.org/jira/browse/FLINK-33731?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33731: --- Labels: pull-request-available (was: ) > failover.flip1 package can be rename to failover > > > Key: FLINK-33731 > URL: https://issues.apache.org/jira/browse/FLINK-33731 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Coordination >Affects Versions: 1.18.0, 1.17.2 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > Currently, there is a org.apache.flink.runtime.executiongraph.failover.flip1 > package. > I propose rename the failover.flip1 to failover, in the other word: removing > the flip1. I have 2 reasons: > * The naming of the package should be based on business semantics, not FLIP > number, and the code under the failover.flip1 package has also many changes > after FLIP-1. > * All code under the failover.flip1 package are Internal code instead > @Public code, so they can be renamed directly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-33731][Scheduler] Remove the redundant flip1 package [flink]
1996fanrui opened a new pull request, #23862: URL: https://github.com/apache/flink/pull/23862 ## What is the purpose of the change Currently, there is a org.apache.flink.runtime.executiongraph.failover.flip1 package. I propose rename the failover.flip1 to failover, in the other word: removing the flip1. I have 2 reasons: - The naming of the package should be based on business semantics, not FLIP number, and the code under the failover.flip1 package has also many changes after FLIP-1. - All code under the failover.flip1 package are Internal code instead of `@Public` code, so they can be renamed directly. ## Brief change log [FLINK-33731][Scheduler] Remove the redundant flip1 package ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector:no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33732) Hive Lookup Join ProjectPushDown will encouter ArrayIndexOutOfBoundsException
[
https://issues.apache.org/jira/browse/FLINK-33732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
lekelei updated FLINK-33732:
Description:
Here is my sql:
```
SET 'execution.runtime-mode' = 'streaming';
SET 'table.dynamic-table-options.enabled' = 'true';
SET 'table.exec.source.cdc-events-duplicate' = 'false';
SET 'pipeline.operator-chaining' = 'false';
CREATE CATALOG catalog_hive WITH (
'type' = 'hive',
...
);
create table kafka_source(
item1 STRING,
item2 INT,
item3 string,
PRIMARY KEY (item1, item2) NOT ENFORCED,
process_time as proctime()
– WATERMARK FOR `ts` AS ts - INTERVAL '10' SECOND
) WITH (
'connector' = 'kafka',
'format' = 'json',
...
);
CREATE TABLE blackhole_sink (
comp STRING,
order STRING,
order_line INT,
order_sequence INT,
material_code STRING,
warehouse_code STRING,
quantity DOUBLE
)WITH (
'connector' = 'blackhole'
);
insert into
blackhole_sink
select item1,comp from kafka_source a left join
catalog_hive.db.hive_lookup_tb
/*+ OPTIONS('streaming-source.partition.include'='latest',
'streaming-source.monitor-interval'='60 min','streaming-source.enable'='true')
*/
FOR SYSTEM_TIME AS OF kafka_source.process_time as b on a.item1 = b.comp;
```
The error stack is as follows：
java.io.IOException: java.io.IOException:
java.lang.ArrayIndexOutOfBoundsException: 10
at
com.netease.sloth.flink.connector.filesystem.table.meta.TableMetaStore.lambda$runSecured$1(TableMetaStore.java:700)
~[classes/:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_332]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_332]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
~[hadoop-common-2.7.5.jar:?]
was:
Here is my sql:
```
SET 'execution.runtime-mode' = 'streaming';
SET 'table.dynamic-table-options.enabled' = 'true';
SET 'table.exec.source.cdc-events-duplicate' = 'false';
SET 'pipeline.operator-chaining' = 'false';
CREATE CATALOG catalog_hive WITH (
'type' = 'hive',
...
);
create table kafka_source(
item1 STRING,
item2 INT,
item3 string,
PRIMARY KEY (item1, item2) NOT ENFORCED,
process_time as proctime()
-- WATERMARK FOR `ts` AS ts - INTERVAL '10' SECOND
) WITH (
'connector' = 'kafka',
'format' = 'json',
...
);
CREATE TABLE blackhole_sink (
comp STRING,
order STRING,
order_line INT,
order_sequence INT,
material_code STRING,
warehouse_code STRING,
quantity DOUBLE
)WITH (
'connector' = 'blackhole'
);
insert into
dwd_pd_purchase_received_detail_arctic_rt_180502_test
select item1,comp from kafka_source a left join
catalog_hive.db.hive_lookup_tb
/*+ OPTIONS('streaming-source.partition.include'='latest',
'streaming-source.monitor-interval'='60 min','streaming-source.enable'='true')
*/
FOR SYSTEM_TIME AS OF kafka_source.process_time as b on a.item1 = b.comp;
```
The error stack is as follows：
java.io.IOException: java.io.IOException:
java.lang.ArrayIndexOutOfBoundsException: 10
at
com.netease.sloth.flink.connector.filesystem.table.meta.TableMetaStore.lambda$runSecured$1(TableMetaStore.java:700)
~[classes/:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_332]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_332]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
~[hadoop-common-2.7.5.jar:?]
> Hive Lookup Join ProjectPushDown will encouter ArrayIndexOutOfBoundsException
> -
>
> Key: FLINK-33732
> URL: https://issues.apache.org/jira/browse/FLINK-33732
> Project: Flink
> Issue Type: Bug
>Reporter: lekelei
>Priority: Major
>
> Here is my sql:
> ```
> SET 'execution.runtime-mode' = 'streaming';
> SET 'table.dynamic-table-options.enabled' = 'true';
> SET 'table.exec.source.cdc-events-duplicate' = 'false';
> SET 'pipeline.operator-chaining' = 'false';
> CREATE CATALOG catalog_hive WITH (
> 'type' = 'hive',
> ...
> );
>
> create table kafka_source(
> item1 STRING,
> item2 INT,
> item3 string,
> PRIMARY KEY (item1, item2) NOT ENFORCED,
> process_time as proctime()
> – WATERMARK FOR `ts` AS ts - INTERVAL '10' SECOND
> ) WITH (
> 'connector' = 'kafka',
> 'format' = 'json',
> ...
> );
>
> CREATE TABLE blackhole_sink (
> comp STRING,
> order STRING,
> order_line INT,
> order_sequence INT,
> material_code STRING,
> warehouse_code STRING,
> quantity DOUBLE
> )WITH (
> 'connector' = 'blackhole'
> );
>
> insert into
> blackhole_sink
> select item1,comp from kafka_source a left join
> catalog_hive.db.hive_lookup_tb
> /*+ OPTIONS('streaming-source.partition.include'='latest',
> 'streaming-source.monitor-interval'='60
> min','streaming-source.enable'='true') */
> FOR SYSTEM_TIME AS OF kafka_source.process_time as b on a.item1 = b.comp;
> ```
> The error stack is as follows：
>
[jira] [Created] (FLINK-33732) Hive Lookup Join ProjectPushDown will encouter ArrayIndexOutOfBoundsException
lekelei created FLINK-33732:
---
Summary: Hive Lookup Join ProjectPushDown will encouter
ArrayIndexOutOfBoundsException
Key: FLINK-33732
URL: https://issues.apache.org/jira/browse/FLINK-33732
Project: Flink
Issue Type: Bug
Reporter: lekelei
Here is my sql:
```
SET 'execution.runtime-mode' = 'streaming';
SET 'table.dynamic-table-options.enabled' = 'true';
SET 'table.exec.source.cdc-events-duplicate' = 'false';
SET 'pipeline.operator-chaining' = 'false';
CREATE CATALOG catalog_hive WITH (
'type' = 'hive',
...
);
create table kafka_source(
item1 STRING,
item2 INT,
item3 string,
PRIMARY KEY (item1, item2) NOT ENFORCED,
process_time as proctime()
-- WATERMARK FOR `ts` AS ts - INTERVAL '10' SECOND
) WITH (
'connector' = 'kafka',
'format' = 'json',
...
);
CREATE TABLE blackhole_sink (
comp STRING,
order STRING,
order_line INT,
order_sequence INT,
material_code STRING,
warehouse_code STRING,
quantity DOUBLE
)WITH (
'connector' = 'blackhole'
);
insert into
dwd_pd_purchase_received_detail_arctic_rt_180502_test
select item1,comp from kafka_source a left join
catalog_hive.db.hive_lookup_tb
/*+ OPTIONS('streaming-source.partition.include'='latest',
'streaming-source.monitor-interval'='60 min','streaming-source.enable'='true')
*/
FOR SYSTEM_TIME AS OF kafka_source.process_time as b on a.item1 = b.comp;
```
The error stack is as follows：
java.io.IOException: java.io.IOException:
java.lang.ArrayIndexOutOfBoundsException: 10
at
com.netease.sloth.flink.connector.filesystem.table.meta.TableMetaStore.lambda$runSecured$1(TableMetaStore.java:700)
~[classes/:?]
at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_332]
at javax.security.auth.Subject.doAs(Subject.java:422) ~[?:1.8.0_332]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1754)
~[hadoop-common-2.7.5.jar:?]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33386][runtime] Support tasks balancing at slot level for Default Scheduler [flink]
RocMarshal commented on PR #23635: URL: https://github.com/apache/flink/pull/23635#issuecomment-1837758141 The waiting mechanism is ready for the review. Would you @KarmaGYZ @1996fanrui help take a look if you were in free time? Thank you very much~ And the verification part about the test would be refactored after external junit5 migrated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33731) failover.flip1 package can be rename to failover
[ https://issues.apache.org/jira/browse/FLINK-33731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792611#comment-17792611 ] Rui Fan commented on FLINK-33731: - Hi [~zhuzh] , what do you think? > failover.flip1 package can be rename to failover > > > Key: FLINK-33731 > URL: https://issues.apache.org/jira/browse/FLINK-33731 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / Coordination >Affects Versions: 1.18.0, 1.17.2 >Reporter: Rui Fan >Assignee: Rui Fan >Priority: Major > Fix For: 1.19.0 > > > Currently, there is a org.apache.flink.runtime.executiongraph.failover.flip1 > package. > I propose rename the failover.flip1 to failover, in the other word: removing > the flip1. I have 2 reasons: > * The naming of the package should be based on business semantics, not FLIP > number, and the code under the failover.flip1 package has also many changes > after FLIP-1. > * All code under the failover.flip1 package are Internal code instead > @Public code, so they can be renamed directly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33731) failover.flip1 package can be rename to failover
Rui Fan created FLINK-33731: --- Summary: failover.flip1 package can be rename to failover Key: FLINK-33731 URL: https://issues.apache.org/jira/browse/FLINK-33731 Project: Flink Issue Type: Technical Debt Components: Runtime / Coordination Affects Versions: 1.17.2, 1.18.0 Reporter: Rui Fan Assignee: Rui Fan Fix For: 1.19.0 Currently, there is a org.apache.flink.runtime.executiongraph.failover.flip1 package. I propose rename the failover.flip1 to failover, in the other word: removing the flip1. I have 2 reasons: * The naming of the package should be based on business semantics, not FLIP number, and the code under the failover.flip1 package has also many changes after FLIP-1. * All code under the failover.flip1 package are Internal code instead @Public code, so they can be renamed directly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33502) HybridShuffleITCase caused a fatal error
[
https://issues.apache.org/jira/browse/FLINK-33502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792610#comment-17792610
]
Wencong Liu commented on FLINK-33502:
-
Thanks [~JunRuiLi] . I have investigated it and found that the root cause is
different with this issue. But the exception caught in the outermost layer is
same. I'll reopen this issue and fix it as soon as possible.
> HybridShuffleITCase caused a fatal error
>
>
> Key: FLINK-33502
> URL: https://issues.apache.org/jira/browse/FLINK-33502
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.19.0
>Reporter: Matthias Pohl
>Assignee: Wencong Liu
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0
>
> Attachments: image-2023-11-20-14-37-37-321.png
>
>
> [https://github.com/XComp/flink/actions/runs/6789774296/job/18458197040#step:12:9177]
> {code:java}
> Error: 21:21:35 21:21:35.379 [ERROR] Error occurred in starting fork, check
> output in log
> 9168Error: 21:21:35 21:21:35.379 [ERROR] Process Exit Code: 239
> 9169Error: 21:21:35 21:21:35.379 [ERROR] Crashed tests:
> 9170Error: 21:21:35 21:21:35.379 [ERROR]
> org.apache.flink.test.runtime.HybridShuffleITCase
> 9171Error: 21:21:35 21:21:35.379 [ERROR]
> org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
> 9172Error: 21:21:35 21:21:35.379 [ERROR] Command was /bin/sh -c cd
> /root/flink/flink-tests && /usr/lib/jvm/jdk-11.0.19+7/bin/java -XX:+UseG1GC
> -Xms256m -XX:+IgnoreUnrecognizedVMOptions
> --add-opens=java.base/java.util=ALL-UNNAMED
> --add-opens=java.base/java.io=ALL-UNNAMED -Xmx1536m -jar
> /root/flink/flink-tests/target/surefire/surefirebooter10811559899200556131.jar
> /root/flink/flink-tests/target/surefire 2023-11-07T20-32-50_466-jvmRun4
> surefire6242806641230738408tmp surefire_1603959900047297795160tmp
> 9173Error: 21:21:35 21:21:35.379 [ERROR] Error occurred in starting fork,
> check output in log
> 9174Error: 21:21:35 21:21:35.379 [ERROR] Process Exit Code: 239
> 9175Error: 21:21:35 21:21:35.379 [ERROR] Crashed tests:
> 9176Error: 21:21:35 21:21:35.379 [ERROR]
> org.apache.flink.test.runtime.HybridShuffleITCase
> 9177Error: 21:21:35 21:21:35.379 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:532)
> 9178Error: 21:21:35 21:21:35.379 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:479)
> 9179Error: 21:21:35 21:21:35.379 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:322)
> 9180Error: 21:21:35 21:21:35.379 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:266)
> [...] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33727) JoinRestoreTest is failing on AZP
[
https://issues.apache.org/jira/browse/FLINK-33727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792599#comment-17792599
]
Jim Hughes commented on FLINK-33727:
That works for me; I approved that PR.
> JoinRestoreTest is failing on AZP
> -
>
> Key: FLINK-33727
> URL: https://issues.apache.org/jira/browse/FLINK-33727
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> Since {{JoinRestoreTest}} was introduced in FLINK-33470 it seems to be a
> reason
> {noformat}
> Dec 02 04:42:26 04:42:26.408 [ERROR] Failures:
> Dec 02 04:42:26 04:42:26.408 [ERROR]
> JoinRestoreTest>RestoreTestBase.testRestore:283
> Dec 02 04:42:26 Expecting actual:
> Dec 02 04:42:26 ["+I[9, carol, apple, 9000]",
> Dec 02 04:42:26 "+I[8, bill, banana, 8000]",
> Dec 02 04:42:26 "+I[6, jerry, pen, 6000]"]
> Dec 02 04:42:26 to contain exactly in any order:
> Dec 02 04:42:26 ["+I[Adam, null]",
> Dec 02 04:42:26 "+I[Baker, Research]",
> Dec 02 04:42:26 "+I[Charlie, Human Resources]",
> Dec 02 04:42:26 "+I[Charlie, HR]",
> Dec 02 04:42:26 "+I[Don, Sales]",
> Dec 02 04:42:26 "+I[Victor, null]",
> Dec 02 04:42:26 "+I[Helena, Engineering]",
> Dec 02 04:42:26 "+I[Juliet, Engineering]",
> Dec 02 04:42:26 "+I[Ivana, Research]",
> Dec 02 04:42:26 "+I[Charlie, People Operations]"]
> {noformat}
> examples of failures
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33727) JoinRestoreTest is failing on AZP
[
https://issues.apache.org/jira/browse/FLINK-33727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792595#comment-17792595
]
Sergey Nuyanzin commented on FLINK-33727:
-
{quote}
I apologize for taking out CI! The order merges resulted in PRs not testing
all of the programs together in one branch until they were merged. (Which is
why I'm willing to suggest disabling all RestoreTests temporarily or reverting
the commits which caused the issue.)
{quote}
I noticed that within FLINK-33470 there was a deletion of {{JoinJsonPlanTest}}
and {{JoinJsonPlanITCase}} and i would guess the idea was to replace it with
new tests... In this case just disabling of new tests is not an option since in
that case we are in a situation when old tests are removed and new are still
not ready.
If revert is more preferable option for you then current PR, then ok I created
a revert PR https://github.com/apache/flink/pull/23861 and if it is ok we can
close the PR for this issue and commit revert PR and reopen FLINK-33470 where
further work could be continued
> JoinRestoreTest is failing on AZP
> -
>
> Key: FLINK-33727
> URL: https://issues.apache.org/jira/browse/FLINK-33727
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> Since {{JoinRestoreTest}} was introduced in FLINK-33470 it seems to be a
> reason
> {noformat}
> Dec 02 04:42:26 04:42:26.408 [ERROR] Failures:
> Dec 02 04:42:26 04:42:26.408 [ERROR]
> JoinRestoreTest>RestoreTestBase.testRestore:283
> Dec 02 04:42:26 Expecting actual:
> Dec 02 04:42:26 ["+I[9, carol, apple, 9000]",
> Dec 02 04:42:26 "+I[8, bill, banana, 8000]",
> Dec 02 04:42:26 "+I[6, jerry, pen, 6000]"]
> Dec 02 04:42:26 to contain exactly in any order:
> Dec 02 04:42:26 ["+I[Adam, null]",
> Dec 02 04:42:26 "+I[Baker, Research]",
> Dec 02 04:42:26 "+I[Charlie, Human Resources]",
> Dec 02 04:42:26 "+I[Charlie, HR]",
> Dec 02 04:42:26 "+I[Don, Sales]",
> Dec 02 04:42:26 "+I[Victor, null]",
> Dec 02 04:42:26 "+I[Helena, Engineering]",
> Dec 02 04:42:26 "+I[Juliet, Engineering]",
> Dec 02 04:42:26 "+I[Ivana, Research]",
> Dec 02 04:42:26 "+I[Charlie, People Operations]"]
> {noformat}
> examples of failures
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] Revert [FLINK-33470] Implement restore tests for Join node [flink]
flinkbot commented on PR #23861: URL: https://github.com/apache/flink/pull/23861#issuecomment-1837630725 ## CI report: * 7be03755f669a2c83d4d67cad67cf2aaadfe2b32 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] Revert [FLINK-33470] Implement restore tests for Join node [flink]
snuyanzin opened a new pull request, #23861: URL: https://github.com/apache/flink/pull/23861 Since current implemenentation leads to ci failures like https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099 https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786 https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099 https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779 it was suggested to revert these changes e.g. at https://issues.apache.org/jira/browse/FLINK-33727#comment-17792535 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-32986][test] Fix createTemporaryFunction type inference error [flink]
jeyhunkarimov commented on PR #23586: URL: https://github.com/apache/flink/pull/23586#issuecomment-1837590317 > @jeyhunkarimov thanks for fixing this! Only one comment. Thanks a lot for the review @lincoln-lil . I addressed your comment -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-28048][connectors] Introduce Source API alternative to FiniteTestSource [flink]
tweise commented on PR #23777: URL: https://github.com/apache/flink/pull/23777#issuecomment-1837588962 LGTM from my side, based on a quick review after @mxm had already covered the details. Regarding complexity: FLIP-27 sources are without doubt more complex for the source implementation/author. That was the price to push past the limitations of the (eventually too simplistic) old SourceFunction and accommodate the rather diverse and complex requirements of feature rich source connectors. Part of FLIP-27 was to reduce implementation complexity by providing building blocks like abstractions for readers and fetchers. Changes proposed here serve a limited/specific scenario for testing. The goal is not to also modify the consuming tests, which would be risky from a regression angle and perhaps not feasible. While it takes a bit of effort to understand the implementation, I also see no obvious winner from an abstraction perspective, at least not one that would be helpful for general connector needs. Please also consider that this effort was delayed till now, lacking criticality beyond finally sunsetting SourceFunction. Given that @afedulov has already put significant effort into this, I would personally focus further efforts elsewhere. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33305] Allow to customize arg line for connector's surefire plugin [flink-connector-shared-utils]
snuyanzin commented on code in PR #25:
URL:
https://github.com/apache/flink-connector-shared-utils/pull/25#discussion_r1413196788
##
pom.xml:
##
@@ -558,6 +563,8 @@ under the License.
US
en
${project.basedir}
+
+
${surefire.module.config}
Review Comment:
That's actually a good question.
Initially I thought it might be useful in case there will be something
similar to `org.apache.flink.runtime.testutils.TestJvmProcess` on connetors
level. However after looking at connector's repos it seems not the case at
least for the moment... So I will remove it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-28048][connectors] Introduce Source API alternative to FiniteTestSource [flink]
tweise commented on code in PR #23777:
URL: https://github.com/apache/flink/pull/23777#discussion_r1413195035
##
flink-connectors/flink-connector-datagen/src/main/java/org/apache/flink/connector/datagen/source/DataGenerators.java:
##
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.datagen.source;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.api.connector.source.SourceReaderFactory;
+import
org.apache.flink.api.connector.source.lib.NumberSequenceSource.NumberSequenceSplit;
+import
org.apache.flink.connector.datagen.functions.IndexLookupGeneratorFunction;
+
+import java.util.Collection;
+import java.util.function.BooleanSupplier;
+
+/** A collection of factory methods for creating data generator-based sources.
*/
+@Experimental
+public class DataGenerators {
+
+/**
+ * TODO: consider moving into test scope TestDataGenerators in datagen
test package after
+ * https://github.com/apache/flink/pull/23553 gets merged.
Review Comment:
The referenced PR was merged. Anything that needs to be updated here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [hotfix][sql-client] remove duplicated code in CliUtils [flink]
flinkbot commented on PR #23860: URL: https://github.com/apache/flink/pull/23860#issuecomment-1837578360 ## CI report: * 2bffbc227737e5d8ce0a130279e092d2950f4ae0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-27529] Fix Intger Comparison For Source Index in Hybrid Source [flink]
varun1729DD commented on PR #23703: URL: https://github.com/apache/flink/pull/23703#issuecomment-1837577966 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [hotfix][sql-client] remove duplicated code in CliUtils [flink]
JingGe opened a new pull request, #23860: URL: https://github.com/apache/flink/pull/23860 ## What is the purpose of the change remove duplicated code in the CliUtils ## Verifying this change This change is a trivial code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-28048][connectors] Introduce Source API alternative to FiniteTestSource [flink]
tweise commented on code in PR #23777:
URL: https://github.com/apache/flink/pull/23777#discussion_r1413186514
##
flink-connectors/flink-connector-datagen/src/main/java/org/apache/flink/connector/datagen/source/DoubleEmittingSourceReaderWithCheckpointsInBetween.java:
##
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.connector.datagen.source;
+
+import org.apache.flink.annotation.Experimental;
+import org.apache.flink.api.connector.source.ReaderOutput;
+import org.apache.flink.api.connector.source.SourceReader;
+import org.apache.flink.api.connector.source.SourceReaderContext;
+import org.apache.flink.api.connector.source.lib.util.IteratorSourceReaderBase;
+import org.apache.flink.api.connector.source.lib.util.IteratorSourceSplit;
+import org.apache.flink.core.io.InputStatus;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import javax.annotation.Nullable;
+
+import java.util.Iterator;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.BooleanSupplier;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * A {@link SourceReader} that synchronizes emission of N elements on the
arrival of the checkpoint
+ * barriers It 1) emits a list of elements without checkpoints in-between, 2)
then waits for two
+ * checkpoints to complete, 3) then re-emits the same elements before 4)
waiting for another two
+ * checkpoints and 5) exiting.
+ *
+ * This lockstep execution is possible because {@code pollNext} and {@code
snapshotState} are
+ * executed in the same thread and the fact that {@code pollNext} can emit N
elements at once. This
+ * reader is meant to be used solely for testing purposes as the substitution
for the {@code
+ * FiniteTestSource} which implements the deprecated {@code SourceFunction}
API.
+ */
+@Experimental
+public class DoubleEmittingSourceReaderWithCheckpointsInBetween<
+E, O, IterT extends Iterator, SplitT extends
IteratorSourceSplit>
+extends IteratorSourceReaderBase {
+
+private final GeneratorFunction generatorFunction;
+
+private BooleanSupplier allowedToExit;
+private int snapshotsCompleted;
+private int snapshotsToWaitFor;
+private boolean done;
+
+public DoubleEmittingSourceReaderWithCheckpointsInBetween(
+SourceReaderContext context,
+GeneratorFunction generatorFunction,
+@Nullable BooleanSupplier allowedToExit) {
+super(context);
+this.generatorFunction = checkNotNull(generatorFunction);
+this.allowedToExit = allowedToExit;
+}
+
+public DoubleEmittingSourceReaderWithCheckpointsInBetween(
+SourceReaderContext context, GeneratorFunction
generatorFunction) {
+super(context);
+this.generatorFunction = checkNotNull(generatorFunction);
+}
+
+//
+
+@Override
+public void start(SourceReaderContext context) {
+try {
+generatorFunction.open(context);
+} catch (Exception e) {
+throw new FlinkRuntimeException("Failed to open the
GeneratorFunction", e);
+}
+}
+
+@Override
+public InputStatus pollNext(ReaderOutput output) {
+// This is the termination path after the test data has been emitted
twice
+if (done) {
+if (allowedToExit != null) { // Termination is controlled
externally
+return allowedToExit.getAsBoolean()
+? InputStatus.END_OF_INPUT
+: InputStatus.NOTHING_AVAILABLE;
+} else {
+return InputStatus.END_OF_INPUT;
+}
+}
+// This is the initial path
+if (currentSplit == null) {
+InputStatus inputStatus = tryMoveToNextSplit();
+switch (inputStatus) {
+case MORE_AVAILABLE:
+emitElements(output);
+break;
+case END_OF_INPUT:
+// This can happen if the source parallelism is larger
than the number of
+// available splits
+
Re: [PR] [FLINK-30064] Move existing Hive connector code from Flink repo to dedicated Hive repo [flink-connector-hive]
snuyanzin commented on PR #5: URL: https://github.com/apache/flink-connector-hive/pull/5#issuecomment-1837569300 rebased since there were new commits in main flink master to hive-connector area also added archunit violations because of reent commit within FLINK-33637 @RyanSkraba moved test packages to `flink.connectors.hive` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33730][doc] update the Flink upgrade savepoint compatibility table doc [flink]
flinkbot commented on PR #23859: URL: https://github.com/apache/flink/pull/23859#issuecomment-1837554278 ## CI report: * ba1b3fd979c8011bf5989c68c5e78ebf40ce2048 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33730) Update the compatibility table to only include last three versions
[ https://issues.apache.org/jira/browse/FLINK-33730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33730: --- Labels: pull-request-available (was: ) > Update the compatibility table to only include last three versions > -- > > Key: FLINK-33730 > URL: https://issues.apache.org/jira/browse/FLINK-33730 > Project: Flink > Issue Type: Improvement > Components: Documentation >Reporter: Jing Ge >Assignee: Jing Ge >Priority: Minor > Labels: pull-request-available > > Update the compatibility table > ([apache-flink:./docs/content/docs/ops/upgrading.md|https://github.com/apache/flink/blob/master/docs/content/docs/ops/upgrading.md#compatibility-table] > and > [apache-flink:./docs/content.zh/docs/ops/upgrading.md|https://github.com/apache/flink/blob/master/docs/content.zh/docs/ops/upgrading.md#compatibility-table]) > according to the discussion[1]. > > [1] https://lists.apache.org/thread/7yx396x5lmtws0s4t0sf9f2psgny11d6 > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-33730][doc] update the Flink upgrade savepoint compatibility table doc [flink]
JingGe opened a new pull request, #23859: URL: https://github.com/apache/flink/pull/23859 ## What is the purpose of the change update the Flink upgrade savepoint compatibility table doc to only include the current and previous releases according to the [Flink Update Policy for old releases ](https://flink.apache.org/downloads/#update-policy-for-old-releases) ## Brief change log - remove all `created with` versions older than 1.7 - remove all `Resumed with` versions older than 1.17 ## Verifying this change This change is a trivial doc cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-33730) Update the compatibility table to only include last three versions
Jing Ge created FLINK-33730: --- Summary: Update the compatibility table to only include last three versions Key: FLINK-33730 URL: https://issues.apache.org/jira/browse/FLINK-33730 Project: Flink Issue Type: Improvement Components: Documentation Reporter: Jing Ge Assignee: Jing Ge Update the compatibility table ([apache-flink:./docs/content/docs/ops/upgrading.md|https://github.com/apache/flink/blob/master/docs/content/docs/ops/upgrading.md#compatibility-table] and [apache-flink:./docs/content.zh/docs/ops/upgrading.md|https://github.com/apache/flink/blob/master/docs/content.zh/docs/ops/upgrading.md#compatibility-table]) according to the discussion[1]. [1] https://lists.apache.org/thread/7yx396x5lmtws0s4t0sf9f2psgny11d6 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32986) The new createTemporaryFunction has some regression of type inference compare to the deprecated registerFunction
[
https://issues.apache.org/jira/browse/FLINK-32986?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792544#comment-17792544
]
lincoln lee commented on FLINK-32986:
-
[~jeyhun] assigned to you.
> The new createTemporaryFunction has some regression of type inference compare
> to the deprecated registerFunction
>
>
> Key: FLINK-32986
> URL: https://issues.apache.org/jira/browse/FLINK-32986
> Project: Flink
> Issue Type: Technical Debt
> Components: Table SQL / API
>Affects Versions: 1.18.0, 1.17.1
>Reporter: lincoln lee
>Assignee: Jeyhun Karimov
>Priority: Major
> Labels: pull-request-available
>
> Current `LookupJoinITCase#testJoinTemporalTableWithUdfFilter` uses a legacy
> form function registration:
> {code}
> tEnv.registerFunction("add", new TestAddWithOpen)
> {code}
> it works fine with the SQL call `add(T.id, 2) > 3` but fails when swith to
> the new api:
> {code}
> tEnv.createTemporaryFunction("add", classOf[TestAddWithOpen])
> // or this
> tEnv.createTemporaryFunction("add", new TestAddWithOpen)
> {code}
> exception:
> {code}
> Caused by: org.apache.flink.table.api.ValidationException: Invalid function
> call:
> default_catalog.default_database.add(BIGINT, INT NOT NULL)
> at
> org.apache.flink.table.types.inference.TypeInferenceUtil.createInvalidCallException(TypeInferenceUtil.java:193)
> at
> org.apache.flink.table.planner.functions.inference.TypeInferenceOperandChecker.checkOperandTypes(TypeInferenceOperandChecker.java:89)
> at
> org.apache.calcite.sql.SqlOperator.checkOperandTypes(SqlOperator.java:753)
> at
> org.apache.calcite.sql.SqlOperator.validateOperands(SqlOperator.java:499)
> at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:335)
> at org.apache.calcite.sql.SqlFunction.deriveType(SqlFunction.java:231)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6302)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl$DeriveTypeVisitor.visit(SqlValidatorImpl.java:6287)
> at org.apache.calcite.sql.SqlCall.accept(SqlCall.java:161)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveTypeImpl(SqlValidatorImpl.java:1869)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.deriveType(SqlValidatorImpl.java:1860)
> at
> org.apache.calcite.sql.type.SqlTypeUtil.deriveType(SqlTypeUtil.java:200)
> at
> org.apache.calcite.sql.type.InferTypes.lambda$static$0(InferTypes.java:47)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.inferUnknownTypes(SqlValidatorImpl.java:2050)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.inferUnknownTypes(SqlValidatorImpl.java:2055)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateWhereOrOn(SqlValidatorImpl.java:4338)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateJoin(SqlValidatorImpl.java:3410)
> at
> org.apache.flink.table.planner.calcite.FlinkCalciteSqlValidator.validateJoin(FlinkCalciteSqlValidator.java:154)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateFrom(SqlValidatorImpl.java:3282)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateSelect(SqlValidatorImpl.java:3603)
> at
> org.apache.calcite.sql.validate.SelectNamespace.validateImpl(SelectNamespace.java:64)
> at
> org.apache.calcite.sql.validate.AbstractNamespace.validate(AbstractNamespace.java:89)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateNamespace(SqlValidatorImpl.java:1050)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateQuery(SqlValidatorImpl.java:1025)
> at org.apache.calcite.sql.SqlSelect.validate(SqlSelect.java:248)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validateScopedExpression(SqlValidatorImpl.java:1000)
> at
> org.apache.calcite.sql.validate.SqlValidatorImpl.validate(SqlValidatorImpl.java:749)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$validate(FlinkPlannerImpl.scala:196)
> ... 49 more
> Caused by: org.apache.flink.table.api.ValidationException: Invalid input
> arguments. Expected signatures are:
> default_catalog.default_database.add(a BIGINT NOT NULL, b INT NOT NULL)
> default_catalog.default_database.add(a BIGINT NOT NULL, b BIGINT NOT NULL)
> at
> org.apache.flink.table.types.inference.TypeInferenceUtil.createInvalidInputException(TypeInferenceUtil.java:180)
> at
> org.apache.flink.table.planner.functions.inference.TypeInferenceOperandChecker.checkOperandTypesOrError(TypeInferenceOperandChecker.java:124)
>
Re: [PR] [FLINK-32986][test] Fix createTemporaryFunction type inference error [flink]
lincoln-lil commented on code in PR #23586:
URL: https://github.com/apache/flink/pull/23586#discussion_r1413081174
##
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/utils/UserDefinedFunctionTestUtils.scala:
##
@@ -382,6 +382,13 @@ object UserDefinedFunctionTestUtils {
eval(a, b.asInstanceOf[Long])
}
+def eval(a: java.lang.Long, b: java.lang.Long): java.lang.Long = {
Review Comment:
Just use `JLong` instead of `java.lang.Long`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-26088][Connectors/ElasticSearch] Add Elasticsearch 8.0 support [flink-connector-elasticsearch]
mtfelisb commented on PR #53: URL: https://github.com/apache/flink-connector-elasticsearch/pull/53#issuecomment-1837517174 > @mtfelisb Are you still active on this PR? Hi, @MartijnVisser. Yes, I'm waiting for review. Also, I'll not implement the `DynamicTableSink` because @rinkako started [here](https://github.com/rinkako/flink-connector-elasticsearch_es8/tree/FLINK-26088/flink-connector-elasticsearch8/src/main/java/org/apache/flink/streaming/connectors/elasticsearch/table) based on my pull request. Thanks by the way @rinkako! ☺️ -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33694][gs-fs-hadoop] Support overriding GCS root URL [flink]
patricklucas commented on PR #23836: URL: https://github.com/apache/flink/pull/23836#issuecomment-1837516482 I considered a slightly different approach, delegating the lookup of the config option to the GCS Hadoop connector itself: https://github.com/apache/flink/compare/master...patricklucas:flink:FLINK-33694_gs_fs_host_v2 If I get some feedback about which of these options is preferred, I can update the PR accordingly. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33727) JoinRestoreTest is failing on AZP
[
https://issues.apache.org/jira/browse/FLINK-33727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792538#comment-17792538
]
Jim Hughes commented on FLINK-33727:
{quote}if so I'm curious whether it would be more helpful for others to have at
least a comment about that in sources
{quote}
Absolutely! The RestoreTest framework is new, and this discussion shows that
there are a number of non-obvious assumptions. [~bvarghese] and I are new to
Flink, and as we've worked with it, we have extended it to have the features
necessary to test various capabilities.
I apologize for taking out CI! The order merges resulted in PRs not testing
all of the programs together in one branch until they were merged. (Which is
why I'm willing to suggest disabling all RestoreTests temporarily or reverting
the commits which caused the issue.)
As an additional improvement to the RestoreTestBase, we could save the SQL text
in a file and fail if it is changed. Of course, then folks could still update
the test files which ought to be "immutable" in some sense.
> JoinRestoreTest is failing on AZP
> -
>
> Key: FLINK-33727
> URL: https://issues.apache.org/jira/browse/FLINK-33727
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> Since {{JoinRestoreTest}} was introduced in FLINK-33470 it seems to be a
> reason
> {noformat}
> Dec 02 04:42:26 04:42:26.408 [ERROR] Failures:
> Dec 02 04:42:26 04:42:26.408 [ERROR]
> JoinRestoreTest>RestoreTestBase.testRestore:283
> Dec 02 04:42:26 Expecting actual:
> Dec 02 04:42:26 ["+I[9, carol, apple, 9000]",
> Dec 02 04:42:26 "+I[8, bill, banana, 8000]",
> Dec 02 04:42:26 "+I[6, jerry, pen, 6000]"]
> Dec 02 04:42:26 to contain exactly in any order:
> Dec 02 04:42:26 ["+I[Adam, null]",
> Dec 02 04:42:26 "+I[Baker, Research]",
> Dec 02 04:42:26 "+I[Charlie, Human Resources]",
> Dec 02 04:42:26 "+I[Charlie, HR]",
> Dec 02 04:42:26 "+I[Don, Sales]",
> Dec 02 04:42:26 "+I[Victor, null]",
> Dec 02 04:42:26 "+I[Helena, Engineering]",
> Dec 02 04:42:26 "+I[Juliet, Engineering]",
> Dec 02 04:42:26 "+I[Ivana, Research]",
> Dec 02 04:42:26 "+I[Charlie, People Operations]"]
> {noformat}
> examples of failures
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33727) JoinRestoreTest is failing on AZP
[
https://issues.apache.org/jira/browse/FLINK-33727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792536#comment-17792536
]
Sergey Nuyanzin edited comment on FLINK-33727 at 12/3/23 2:58 PM:
--
{quote}
Presently, the method `generateTestSetupFiles` is disabled and only run by test
developers before a PR is submitted. This method takes the SQL, gets and saves
a compiled plan, and runs through the beforeRestore data making some
comparisons, and finally stopping the job and taking a savepoint.
{quote}
if so I'm curious whether it would be more helpful for others to have at least
a comment about that in sources?
IMHO it could help others who is editing the code around however not aware of
it and since currently there is no any such info in code and the code is not a
part of "turned on" tests, then it could be simply be broken and neither
contributor nor reviewer could notice that
was (Author: sergey nuyanzin):
{quote}
Presently, the method `generateTestSetupFiles` is disabled and only run by test
developers before a PR is submitted. This method takes the SQL, gets and saves
a compiled plan, and runs through the beforeRestore data making some
comparisons, and finally stopping the job and taking a savepoint.
{quote}
if so I'm curious whether it would be more helpful for others to have at least
a comment about that in sources?
IMHO it could help others who is editing the code around however not aware of
it and since currently there is no any such info in code and the code is not a
part of "turned on" tests, then it could be simply be broken without and
neither contributor nor reviewer could notice that
> JoinRestoreTest is failing on AZP
> -
>
> Key: FLINK-33727
> URL: https://issues.apache.org/jira/browse/FLINK-33727
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> Since {{JoinRestoreTest}} was introduced in FLINK-33470 it seems to be a
> reason
> {noformat}
> Dec 02 04:42:26 04:42:26.408 [ERROR] Failures:
> Dec 02 04:42:26 04:42:26.408 [ERROR]
> JoinRestoreTest>RestoreTestBase.testRestore:283
> Dec 02 04:42:26 Expecting actual:
> Dec 02 04:42:26 ["+I[9, carol, apple, 9000]",
> Dec 02 04:42:26 "+I[8, bill, banana, 8000]",
> Dec 02 04:42:26 "+I[6, jerry, pen, 6000]"]
> Dec 02 04:42:26 to contain exactly in any order:
> Dec 02 04:42:26 ["+I[Adam, null]",
> Dec 02 04:42:26 "+I[Baker, Research]",
> Dec 02 04:42:26 "+I[Charlie, Human Resources]",
> Dec 02 04:42:26 "+I[Charlie, HR]",
> Dec 02 04:42:26 "+I[Don, Sales]",
> Dec 02 04:42:26 "+I[Victor, null]",
> Dec 02 04:42:26 "+I[Helena, Engineering]",
> Dec 02 04:42:26 "+I[Juliet, Engineering]",
> Dec 02 04:42:26 "+I[Ivana, Research]",
> Dec 02 04:42:26 "+I[Charlie, People Operations]"]
> {noformat}
> examples of failures
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33727) JoinRestoreTest is failing on AZP
[
https://issues.apache.org/jira/browse/FLINK-33727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792536#comment-17792536
]
Sergey Nuyanzin commented on FLINK-33727:
-
{quote}
Presently, the method `generateTestSetupFiles` is disabled and only run by test
developers before a PR is submitted. This method takes the SQL, gets and saves
a compiled plan, and runs through the beforeRestore data making some
comparisons, and finally stopping the job and taking a savepoint.
{quote}
if so I'm curious whether it would be more helpful for others to have at least
a comment about that in sources?
IMHO it could help others who is editing the code around however not aware of
it and since currently there is no any such info in code and the code is not a
part of "turned on" tests, then it could be simply be broken without and
neither contributor nor reviewer could notice that
> JoinRestoreTest is failing on AZP
> -
>
> Key: FLINK-33727
> URL: https://issues.apache.org/jira/browse/FLINK-33727
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> Since {{JoinRestoreTest}} was introduced in FLINK-33470 it seems to be a
> reason
> {noformat}
> Dec 02 04:42:26 04:42:26.408 [ERROR] Failures:
> Dec 02 04:42:26 04:42:26.408 [ERROR]
> JoinRestoreTest>RestoreTestBase.testRestore:283
> Dec 02 04:42:26 Expecting actual:
> Dec 02 04:42:26 ["+I[9, carol, apple, 9000]",
> Dec 02 04:42:26 "+I[8, bill, banana, 8000]",
> Dec 02 04:42:26 "+I[6, jerry, pen, 6000]"]
> Dec 02 04:42:26 to contain exactly in any order:
> Dec 02 04:42:26 ["+I[Adam, null]",
> Dec 02 04:42:26 "+I[Baker, Research]",
> Dec 02 04:42:26 "+I[Charlie, Human Resources]",
> Dec 02 04:42:26 "+I[Charlie, HR]",
> Dec 02 04:42:26 "+I[Don, Sales]",
> Dec 02 04:42:26 "+I[Victor, null]",
> Dec 02 04:42:26 "+I[Helena, Engineering]",
> Dec 02 04:42:26 "+I[Juliet, Engineering]",
> Dec 02 04:42:26 "+I[Ivana, Research]",
> Dec 02 04:42:26 "+I[Charlie, People Operations]"]
> {noformat}
> examples of failures
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33727) JoinRestoreTest is failing on AZP
[
https://issues.apache.org/jira/browse/FLINK-33727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792535#comment-17792535
]
Jim Hughes commented on FLINK-33727:
As an alternative, to the existing PR and disabling the RestoreTests, I'm
totally fine with you reverting the commits from my PR:
[https://github.com/apache/flink/pull/23680]
[https://github.com/apache/flink/commit/e886dfdda6cd927548c8af0a88e78171e7ba34a8]
[https://github.com/apache/flink/commit/5edc7d7b18e88cc86e84d197202d8cbb40621864]
> JoinRestoreTest is failing on AZP
> -
>
> Key: FLINK-33727
> URL: https://issues.apache.org/jira/browse/FLINK-33727
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> Since {{JoinRestoreTest}} was introduced in FLINK-33470 it seems to be a
> reason
> {noformat}
> Dec 02 04:42:26 04:42:26.408 [ERROR] Failures:
> Dec 02 04:42:26 04:42:26.408 [ERROR]
> JoinRestoreTest>RestoreTestBase.testRestore:283
> Dec 02 04:42:26 Expecting actual:
> Dec 02 04:42:26 ["+I[9, carol, apple, 9000]",
> Dec 02 04:42:26 "+I[8, bill, banana, 8000]",
> Dec 02 04:42:26 "+I[6, jerry, pen, 6000]"]
> Dec 02 04:42:26 to contain exactly in any order:
> Dec 02 04:42:26 ["+I[Adam, null]",
> Dec 02 04:42:26 "+I[Baker, Research]",
> Dec 02 04:42:26 "+I[Charlie, Human Resources]",
> Dec 02 04:42:26 "+I[Charlie, HR]",
> Dec 02 04:42:26 "+I[Don, Sales]",
> Dec 02 04:42:26 "+I[Victor, null]",
> Dec 02 04:42:26 "+I[Helena, Engineering]",
> Dec 02 04:42:26 "+I[Juliet, Engineering]",
> Dec 02 04:42:26 "+I[Ivana, Research]",
> Dec 02 04:42:26 "+I[Charlie, People Operations]"]
> {noformat}
> examples of failures
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33728) do not rewatch when KubernetesResourceManagerDriver watch fail
[ https://issues.apache.org/jira/browse/FLINK-33728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaogang zhou updated FLINK-33728: -- Description: is it necessary to getResourceEventHandler().onError(throwable) in PodCallbackHandlerImpl# handleError method. We can just neglect the disconnection of watching process. and try to rewatch once new requestResource called > do not rewatch when KubernetesResourceManagerDriver watch fail > -- > > Key: FLINK-33728 > URL: https://issues.apache.org/jira/browse/FLINK-33728 > Project: Flink > Issue Type: New Feature > Components: Deployment / Kubernetes >Reporter: xiaogang zhou >Priority: Major > > is it necessary to > getResourceEventHandler().onError(throwable) > in PodCallbackHandlerImpl# handleError method. > > We can just neglect the disconnection of watching process. and try to rewatch > once new requestResource called -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33729) Events are getting lost when an exception occurs within a processing function
Rafał Trójczak created FLINK-33729: -- Summary: Events are getting lost when an exception occurs within a processing function Key: FLINK-33729 URL: https://issues.apache.org/jira/browse/FLINK-33729 Project: Flink Issue Type: Bug Components: Connectors / Pulsar Affects Versions: 1.15.3 Reporter: Rafał Trójczak We have a Flink job using a Pulsar source that reads from an input topic, and a Pulsar sink that is writing to an output topic. Both Flink and Pulsar connector are of version 1.15.3. The Pulsar version that I use is 2.10.3. Here is a simple project that is intended to reproduce this problem: [https://github.com/trojczak/flink-pulsar-connector-problem/] All of my tests were done on my local Kubernetes cluster using the Flink Kubernetes Operator and Pulsar is running on my local Docker. But the same problem occurred on a "normal" cluster. Expected behavior: When an exception is thrown within the code (or a TaskManager pod is restarted for any other reason, e.g. OOM exception), the processing should be picked up from the last event sent to the output topic. Actual behavior: The events before the failure are sent correctly to the output topic, next some of the events from the input topic are missing, then from some point the events are being processed normally until the next exception is thrown, and so on. Finally, from 100 events that should be sent from the input topic to the output topic, only 40 are sent. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33727][table] Use different sink names for restore tests [flink]
jnh5y commented on PR #23858: URL: https://github.com/apache/flink/pull/23858#issuecomment-1837506076 > Are there any savepoint where currently changed names were participating? Yes. > Then how to find all? Or is it just a question in general? The savepoints are living under directories in https://github.com/apache/flink/tree/master/flink-table/flink-table-planner/src/test/resources/restore-tests, e.g. https://github.com/apache/flink/tree/master/flink-table/flink-table-planner/src/test/resources/restore-tests/stream-exec-join_1/anti-join/savepoint -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33727) JoinRestoreTest is failing on AZP
[
https://issues.apache.org/jira/browse/FLINK-33727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792534#comment-17792534
]
Jim Hughes commented on FLINK-33727:
{quote}..., however tests are continuing passing. Is it expected?
{quote}
"Yes", but the reason is a little confusing.
The RestoreTest framework has two methods: `generateTestSetupFiles` and
`testRestore`.
Presently, the method `generateTestSetupFiles` is disabled and only run by test
developers before a PR is submitted. This method takes the SQL, gets and saves
a compiled plan, and runs through the beforeRestore data making some
comparisons, and finally stopping the job and taking a savepoint.
The second method uses the compiled plan and the savepoint.
Since you are only running the second method, changing the SQL is irrelevant
and not tested (unless you manually run `generateTestSetupFiles`).
{quote}what is wrong with current PR for this JIRA?
{quote}
CI failing is showing that the RestoreTestBase has some limitations/assumptions
around state which we need to address. The current PR fixes CI, but does not
address those, rather it works around them. I'd prefer that we fix the
limitations rather than work around them. That's why I'm suggesting to disable
the RestoreTests as a whole until Monday when Dawid and Timo can weigh in.
> JoinRestoreTest is failing on AZP
> -
>
> Key: FLINK-33727
> URL: https://issues.apache.org/jira/browse/FLINK-33727
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> Since {{JoinRestoreTest}} was introduced in FLINK-33470 it seems to be a
> reason
> {noformat}
> Dec 02 04:42:26 04:42:26.408 [ERROR] Failures:
> Dec 02 04:42:26 04:42:26.408 [ERROR]
> JoinRestoreTest>RestoreTestBase.testRestore:283
> Dec 02 04:42:26 Expecting actual:
> Dec 02 04:42:26 ["+I[9, carol, apple, 9000]",
> Dec 02 04:42:26 "+I[8, bill, banana, 8000]",
> Dec 02 04:42:26 "+I[6, jerry, pen, 6000]"]
> Dec 02 04:42:26 to contain exactly in any order:
> Dec 02 04:42:26 ["+I[Adam, null]",
> Dec 02 04:42:26 "+I[Baker, Research]",
> Dec 02 04:42:26 "+I[Charlie, Human Resources]",
> Dec 02 04:42:26 "+I[Charlie, HR]",
> Dec 02 04:42:26 "+I[Don, Sales]",
> Dec 02 04:42:26 "+I[Victor, null]",
> Dec 02 04:42:26 "+I[Helena, Engineering]",
> Dec 02 04:42:26 "+I[Juliet, Engineering]",
> Dec 02 04:42:26 "+I[Ivana, Research]",
> Dec 02 04:42:26 "+I[Charlie, People Operations]"]
> {noformat}
> examples of failures
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33727][table] Use different sink names for restore tests [flink]
snuyanzin commented on PR #23858: URL: https://github.com/apache/flink/pull/23858#issuecomment-1837503632 Are there any savepoint where currently changed names were participating? Then how to find all? Or is it just a question in general? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-33728) do not rewatch when KubernetesResourceManagerDriver watch fail
xiaogang zhou created FLINK-33728: - Summary: do not rewatch when KubernetesResourceManagerDriver watch fail Key: FLINK-33728 URL: https://issues.apache.org/jira/browse/FLINK-33728 Project: Flink Issue Type: New Feature Components: Deployment / Kubernetes Reporter: xiaogang zhou -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33727][table] Use different sink names for restore tests [flink]
snuyanzin commented on code in PR #23858:
URL: https://github.com/apache/flink/pull/23858#discussion_r1413116026
##
flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/testutils/JoinTestPrograms.java:
##
@@ -142,7 +142,7 @@ public class JoinTestPrograms {
.setupTableSource(SOURCE_T1)
.setupTableSource(SOURCE_T2)
.setupTableSink(
-SinkTestStep.newBuilder("MySink")
+
SinkTestStep.newBuilder("NON_WINDOW_INNER_JOIN_WITH_NULL_Sink")
Review Comment:
makes sense, thanks
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-33727) JoinRestoreTest is failing on AZP
[
https://issues.apache.org/jira/browse/FLINK-33727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792533#comment-17792533
]
Sergey Nuyanzin edited comment on FLINK-33727 at 12/3/23 2:32 PM:
--
{quote}
I'm pretty sure that if you removed the run SQL, that'd be like removing the
section in a JUnit test function which does anything and then asserts that it
works. (That'd explain why the tests pass without it.)
{quote}
I don't understand it
as another experiment I just changed SQL to the wrong SQL with syntax error.
e.g. just add a couple of commas in wrong places
{code:java}
.runSql(
"insert, into, deduplicate_sink "
+ "select order_id, user, product, order_time \n"
+ "FROM ("
+ " SELECT *,"
+ "ROW_NUMBER() OVER (PARTITION BY product ORDER BY event_time ASC) AS
row_num\n"
+ " FROM MyTable)"
+ "WHERE row_num = 1")
{code}
, however tests are continuing passing. Is it expected?
Then it's is not clear what is the reason to have it and how to check that it
tests what it is expected to test?
{quote}
If you are looking to sort things immediately, I'd suggest adding `Disabled` to
`testRestore` here:
https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/testutils/RestoreTestBase.java#L229
{quote}
what is wrong with current PR for this JIRA?
was (Author: sergey nuyanzin):
{quote}
I'm pretty sure that if you removed the run SQL, that'd be like removing the
section in a JUnit test function which does anything and then asserts that it
works. (That'd explain why the tests pass without it.)
{quote}
I don't understand it
as another experiment I just changed SQL to the wrong SQL with syntax error,
however tests are continuing passing. Is it expected?
Then it's is not clear what is the reason to have it and how to check that it
tests what it is expected to test?
{quote}
If you are looking to sort things immediately, I'd suggest adding `Disabled` to
`testRestore` here:
https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/testutils/RestoreTestBase.java#L229
{quote}
what is wrong with current PR for this JIRA?
> JoinRestoreTest is failing on AZP
> -
>
> Key: FLINK-33727
> URL: https://issues.apache.org/jira/browse/FLINK-33727
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> Since {{JoinRestoreTest}} was introduced in FLINK-33470 it seems to be a
> reason
> {noformat}
> Dec 02 04:42:26 04:42:26.408 [ERROR] Failures:
> Dec 02 04:42:26 04:42:26.408 [ERROR]
> JoinRestoreTest>RestoreTestBase.testRestore:283
> Dec 02 04:42:26 Expecting actual:
> Dec 02 04:42:26 ["+I[9, carol, apple, 9000]",
> Dec 02 04:42:26 "+I[8, bill, banana, 8000]",
> Dec 02 04:42:26 "+I[6, jerry, pen, 6000]"]
> Dec 02 04:42:26 to contain exactly in any order:
> Dec 02 04:42:26 ["+I[Adam, null]",
> Dec 02 04:42:26 "+I[Baker, Research]",
> Dec 02 04:42:26 "+I[Charlie, Human Resources]",
> Dec 02 04:42:26 "+I[Charlie, HR]",
> Dec 02 04:42:26 "+I[Don, Sales]",
> Dec 02 04:42:26 "+I[Victor, null]",
> Dec 02 04:42:26 "+I[Helena, Engineering]",
> Dec 02 04:42:26 "+I[Juliet, Engineering]",
> Dec 02 04:42:26 "+I[Ivana, Research]",
> Dec 02 04:42:26 "+I[Charlie, People Operations]"]
> {noformat}
> examples of failures
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33727) JoinRestoreTest is failing on AZP
[
https://issues.apache.org/jira/browse/FLINK-33727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792533#comment-17792533
]
Sergey Nuyanzin commented on FLINK-33727:
-
{quote}
I'm pretty sure that if you removed the run SQL, that'd be like removing the
section in a JUnit test function which does anything and then asserts that it
works. (That'd explain why the tests pass without it.)
{quote}
I don't understand it
as another experiment I just changed SQL to the wrong SQL with syntax error,
however tests are continuing passing. Is it expected?
Then it's is not clear what is the reason to have it and how to check that it
tests what it is expected to test?
{quote}
If you are looking to sort things immediately, I'd suggest adding `Disabled` to
`testRestore` here:
https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/testutils/RestoreTestBase.java#L229
{quote}
what is wrong with current PR for this JIRA?
> JoinRestoreTest is failing on AZP
> -
>
> Key: FLINK-33727
> URL: https://issues.apache.org/jira/browse/FLINK-33727
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> Since {{JoinRestoreTest}} was introduced in FLINK-33470 it seems to be a
> reason
> {noformat}
> Dec 02 04:42:26 04:42:26.408 [ERROR] Failures:
> Dec 02 04:42:26 04:42:26.408 [ERROR]
> JoinRestoreTest>RestoreTestBase.testRestore:283
> Dec 02 04:42:26 Expecting actual:
> Dec 02 04:42:26 ["+I[9, carol, apple, 9000]",
> Dec 02 04:42:26 "+I[8, bill, banana, 8000]",
> Dec 02 04:42:26 "+I[6, jerry, pen, 6000]"]
> Dec 02 04:42:26 to contain exactly in any order:
> Dec 02 04:42:26 ["+I[Adam, null]",
> Dec 02 04:42:26 "+I[Baker, Research]",
> Dec 02 04:42:26 "+I[Charlie, Human Resources]",
> Dec 02 04:42:26 "+I[Charlie, HR]",
> Dec 02 04:42:26 "+I[Don, Sales]",
> Dec 02 04:42:26 "+I[Victor, null]",
> Dec 02 04:42:26 "+I[Helena, Engineering]",
> Dec 02 04:42:26 "+I[Juliet, Engineering]",
> Dec 02 04:42:26 "+I[Ivana, Research]",
> Dec 02 04:42:26 "+I[Charlie, People Operations]"]
> {noformat}
> examples of failures
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33727) JoinRestoreTest is failing on AZP
[
https://issues.apache.org/jira/browse/FLINK-33727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792532#comment-17792532
]
Jim Hughes commented on FLINK-33727:
{quote}...
It seems it relies on some internal state...
{quote}
The tests have internal state. Your testing shows that it not being reset
between test classes!
Thanks for digging into that; that will help us identify what we need to sort
out with the TestRestoreBase.
If you are looking to sort things immediately, I'd suggest adding `Disabled` to
`testRestore` here:
[https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/testutils/RestoreTestBase.java#L229]
That'd turn off all of these tests until [~twalthr] [~dwysakowicz] and I have a
solution.[|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=dwysakowicz]
> JoinRestoreTest is failing on AZP
> -
>
> Key: FLINK-33727
> URL: https://issues.apache.org/jira/browse/FLINK-33727
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> Since {{JoinRestoreTest}} was introduced in FLINK-33470 it seems to be a
> reason
> {noformat}
> Dec 02 04:42:26 04:42:26.408 [ERROR] Failures:
> Dec 02 04:42:26 04:42:26.408 [ERROR]
> JoinRestoreTest>RestoreTestBase.testRestore:283
> Dec 02 04:42:26 Expecting actual:
> Dec 02 04:42:26 ["+I[9, carol, apple, 9000]",
> Dec 02 04:42:26 "+I[8, bill, banana, 8000]",
> Dec 02 04:42:26 "+I[6, jerry, pen, 6000]"]
> Dec 02 04:42:26 to contain exactly in any order:
> Dec 02 04:42:26 ["+I[Adam, null]",
> Dec 02 04:42:26 "+I[Baker, Research]",
> Dec 02 04:42:26 "+I[Charlie, Human Resources]",
> Dec 02 04:42:26 "+I[Charlie, HR]",
> Dec 02 04:42:26 "+I[Don, Sales]",
> Dec 02 04:42:26 "+I[Victor, null]",
> Dec 02 04:42:26 "+I[Helena, Engineering]",
> Dec 02 04:42:26 "+I[Juliet, Engineering]",
> Dec 02 04:42:26 "+I[Ivana, Research]",
> Dec 02 04:42:26 "+I[Charlie, People Operations]"]
> {noformat}
> examples of failures
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33727) JoinRestoreTest is failing on AZP
[
https://issues.apache.org/jira/browse/FLINK-33727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17792532#comment-17792532
]
Jim Hughes edited comment on FLINK-33727 at 12/3/23 2:26 PM:
-
{quote}...
It seems it relies on some internal state...
{quote}
The tests have internal state. Your testing shows that it not being reset
between test classes!
Thanks for digging into that; that will help us identify what we need to sort
out with the TestRestoreBase.
If you are looking to sort things immediately, I'd suggest adding `Disabled` to
`testRestore` here:
[https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/testutils/RestoreTestBase.java#L229]
That'd turn off all of these tests until [~twalthr] [~dwysakowicz] [~bvarghese]
and I have a
solution.[https://issues.apache.org/jira/secure/ViewProfile.jspa?name=dwysakowicz]
was (Author: JIRAUSER284726):
{quote}...
It seems it relies on some internal state...
{quote}
The tests have internal state. Your testing shows that it not being reset
between test classes!
Thanks for digging into that; that will help us identify what we need to sort
out with the TestRestoreBase.
If you are looking to sort things immediately, I'd suggest adding `Disabled` to
`testRestore` here:
[https://github.com/apache/flink/blob/master/flink-table/flink-table-planner/src/test/java/org/apache/flink/table/planner/plan/nodes/exec/testutils/RestoreTestBase.java#L229]
That'd turn off all of these tests until [~twalthr] [~dwysakowicz] and I have a
solution.[|https://issues.apache.org/jira/secure/ViewProfile.jspa?name=dwysakowicz]
> JoinRestoreTest is failing on AZP
> -
>
> Key: FLINK-33727
> URL: https://issues.apache.org/jira/browse/FLINK-33727
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> Since {{JoinRestoreTest}} was introduced in FLINK-33470 it seems to be a
> reason
> {noformat}
> Dec 02 04:42:26 04:42:26.408 [ERROR] Failures:
> Dec 02 04:42:26 04:42:26.408 [ERROR]
> JoinRestoreTest>RestoreTestBase.testRestore:283
> Dec 02 04:42:26 Expecting actual:
> Dec 02 04:42:26 ["+I[9, carol, apple, 9000]",
> Dec 02 04:42:26 "+I[8, bill, banana, 8000]",
> Dec 02 04:42:26 "+I[6, jerry, pen, 6000]"]
> Dec 02 04:42:26 to contain exactly in any order:
> Dec 02 04:42:26 ["+I[Adam, null]",
> Dec 02 04:42:26 "+I[Baker, Research]",
> Dec 02 04:42:26 "+I[Charlie, Human Resources]",
> Dec 02 04:42:26 "+I[Charlie, HR]",
> Dec 02 04:42:26 "+I[Don, Sales]",
> Dec 02 04:42:26 "+I[Victor, null]",
> Dec 02 04:42:26 "+I[Helena, Engineering]",
> Dec 02 04:42:26 "+I[Juliet, Engineering]",
> Dec 02 04:42:26 "+I[Ivana, Research]",
> Dec 02 04:42:26 "+I[Charlie, People Operations]"]
> {noformat}
> examples of failures
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55120=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55129=logs=a9db68b9-a7e0-54b6-0f98-010e0aff39e2=cdd32e0b-6047-565b-c58f-14054472f1be=11786
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55136=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=12099
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=55137=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4=11779
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
