[jira] [Commented] (FLINK-24701) "late events" link in "Timely Stream Processing" document is not working
[ https://issues.apache.org/jira/browse/FLINK-24701?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17527130#comment-17527130 ] Wencong Liu commented on FLINK-24701: - This problem seems to have been solved in Flink 1.16. [~TsReaper] > "late events" link in "Timely Stream Processing" document is not working > > > Key: FLINK-24701 > URL: https://issues.apache.org/jira/browse/FLINK-24701 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.15.0 >Reporter: Caizhi Weng >Priority: Minor > Labels: Starter, starter > > The last but one paragraph of [this > document|https://nightlies.apache.org/flink/flink-docs-master/docs/concepts/time/#notions-of-time-event-time-and-processing-time] > contains a link with the text "late events". However clicking on that link > leads to nothing. > I guess it should point to the > [lateness|https://nightlies.apache.org/flink/flink-docs-master/docs/concepts/time/#lateness] > section in the same page. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FLINK-25970) SerializedThrowable should record type of the original throwable.
[
https://issues.apache.org/jira/browse/FLINK-25970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17527426#comment-17527426
]
Wencong Liu commented on FLINK-25970:
-
Hello [~xtsong] , I have two questions about this issue:
1. Are the examples here real? Under what circumstances will
{code:java}
new SerializedThrowable(xxx).printStackTrace() {code}
print these call stacks?
2. In the source code of 1.14.3, I saw that the call stack of
SerializedThrowable() will be replaced by the construction parameter:
{code:java}
private SerializedThrowable(Throwable exception, Set alreadySeen) {
// ...
// mimic the original exception's stack trace
setStackTrace(exception.getStackTrace());
// ...
} {code}
so the first line of the call stack printed to System.err should be
ConcurrentModificationException?
> SerializedThrowable should record type of the original throwable.
> -
>
> Key: FLINK-25970
> URL: https://issues.apache.org/jira/browse/FLINK-25970
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.14.3
>Reporter: Xintong Song
>Priority: Minor
> Labels: starter
>
> Currently, only the message and stack of the original throwable is preserved
> in {{{}SerializedThrowable{}}}, while the type of the original throwable is
> discarded.
> Sometimes, it would be helpful if message of {{SerializedThrowable}} can also
> include the full class name of the original throwable.
> E.g., in the following stack.
> {code:java}
> Caused by: org.apache.flink.util.SerializedThrowable
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> ~[?:1.8.0_102]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1471) ~[?:1.8.0_102]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1469) ~[?:1.8.0_102]
> ...
> {code}
> It's not that easy to understand what is wrong from this stack. JDK does not
> provide a message for the original exception. We have to look into the JDK
> source codes to find out what's going on. Sometimes it's even more annoying
> having to look for the JDK source codes of the exactly same version in order
> to match the line numbers.
> Turns out the original exception was a {{ConcurrentModificationException}}. I
> think it would be much more straightforward if we can have a stack like the
> following.
> {code}
> Caused by: org.apache.flink.util.SerializedThrowable:
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> ~[?:1.8.0_102]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1471) ~[?:1.8.0_102]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1469) ~[?:1.8.0_102]
> ...
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (FLINK-25970) SerializedThrowable should record type of the original throwable.
[
https://issues.apache.org/jira/browse/FLINK-25970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17528882#comment-17528882
]
Wencong Liu commented on FLINK-25970:
-
Hello [~xtsong], I understand this issue in this way :
SerializedThrowable will return itself as a replacement if an exception occurs
when calling deserializeError(ClassLoader classloader). When this happens, the
call stack printed by logger will not contain the class name of original
exception.
The pr has been associated with this issue. You could take a look at it when
you are free. Thank you very much.
> SerializedThrowable should record type of the original throwable.
> -
>
> Key: FLINK-25970
> URL: https://issues.apache.org/jira/browse/FLINK-25970
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.14.3
>Reporter: Xintong Song
>Priority: Minor
> Labels: pull-request-available, starter
>
> Currently, only the message and stack of the original throwable is preserved
> in {{{}SerializedThrowable{}}}, while the type of the original throwable is
> discarded.
> Sometimes, it would be helpful if message of {{SerializedThrowable}} can also
> include the full class name of the original throwable.
> E.g., in the following stack.
> {code:java}
> Caused by: org.apache.flink.util.SerializedThrowable
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> ~[?:1.8.0_102]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1471) ~[?:1.8.0_102]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1469) ~[?:1.8.0_102]
> ...
> {code}
> It's not that easy to understand what is wrong from this stack. JDK does not
> provide a message for the original exception. We have to look into the JDK
> source codes to find out what's going on. Sometimes it's even more annoying
> having to look for the JDK source codes of the exactly same version in order
> to match the line numbers.
> Turns out the original exception was a {{ConcurrentModificationException}}. I
> think it would be much more straightforward if we can have a stack like the
> following.
> {code}
> Caused by: org.apache.flink.util.SerializedThrowable:
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> ~[?:1.8.0_102]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1471) ~[?:1.8.0_102]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1469) ~[?:1.8.0_102]
> ...
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (FLINK-25970) SerializedThrowable should record type of the original throwable.
[
https://issues.apache.org/jira/browse/FLINK-25970?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17530592#comment-17530592
]
Wencong Liu commented on FLINK-25970:
-
Hello [~xtsong], the problem you mentioned has been fixed and CI test has
passed. Please take a look at PR#19615 when you are free.
> SerializedThrowable should record type of the original throwable.
> -
>
> Key: FLINK-25970
> URL: https://issues.apache.org/jira/browse/FLINK-25970
> Project: Flink
> Issue Type: Improvement
> Components: API / Core
>Affects Versions: 1.14.3
>Reporter: Xintong Song
>Priority: Minor
> Labels: pull-request-available, starter
>
> Currently, only the message and stack of the original throwable is preserved
> in {{{}SerializedThrowable{}}}, while the type of the original throwable is
> discarded.
> Sometimes, it would be helpful if message of {{SerializedThrowable}} can also
> include the full class name of the original throwable.
> E.g., in the following stack.
> {code:java}
> Caused by: org.apache.flink.util.SerializedThrowable
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> ~[?:1.8.0_102]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1471) ~[?:1.8.0_102]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1469) ~[?:1.8.0_102]
> ...
> {code}
> It's not that easy to understand what is wrong from this stack. JDK does not
> provide a message for the original exception. We have to look into the JDK
> source codes to find out what's going on. Sometimes it's even more annoying
> having to look for the JDK source codes of the exactly same version in order
> to match the line numbers.
> Turns out the original exception was a {{ConcurrentModificationException}}. I
> think it would be much more straightforward if we can have a stack like the
> following.
> {code}
> Caused by: org.apache.flink.util.SerializedThrowable:
> java.util.ConcurrentModificationException
> at java.util.HashMap$HashIterator.nextNode(HashMap.java:1437)
> ~[?:1.8.0_102]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1471) ~[?:1.8.0_102]
> at java.util.HashMap$EntryIterator.next(HashMap.java:1469) ~[?:1.8.0_102]
> ...
> {code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (FLINK-27730) Kafka connector document code sink has an error
[ https://issues.apache.org/jira/browse/FLINK-27730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17540576#comment-17540576 ] Wencong Liu commented on FLINK-27730: - Hello [~i-liu] , please show the relevant documentation link, and what is the specific error? > Kafka connector document code sink has an error > --- > > Key: FLINK-27730 > URL: https://issues.apache.org/jira/browse/FLINK-27730 > Project: Flink > Issue Type: Bug > Components: Documentation >Affects Versions: 1.14.4 > Environment: Flink 1.14.4 >Reporter: liuwei >Priority: Major > Fix For: 1.14.4 > > Attachments: kafka-sink.png > > > Kafka Sink document sample code API call error. > -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FLINK-27724) The SubtaskCheckpointCoordinatorImpl#close() isn't called in anywhere
[ https://issues.apache.org/jira/browse/FLINK-27724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17540578#comment-17540578 ] Wencong Liu commented on FLINK-27724: - Hello [~fanrui] , are you sure this problem exists? On the latest master branch, SubtaskCheckpointCoordinatorImpl#close() is called by many modules. > The SubtaskCheckpointCoordinatorImpl#close() isn't called in anywhere > - > > Key: FLINK-27724 > URL: https://issues.apache.org/jira/browse/FLINK-27724 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.14.0, 1.15.0 >Reporter: fanrui >Priority: Major > Fix For: 1.16.0 > > > The SubtaskCheckpointCoordinatorImpl#close() isn't called in anywhere -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FLINK-27703) FileChannelManagerImplTest.testDirectoriesCleanupOnKillWithoutCallerHook failed with The marker file was not found within 10000 msecs
[
https://issues.apache.org/jira/browse/FLINK-27703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17540583#comment-17540583
]
Wencong Liu commented on FLINK-27703:
-
Hello [~hxbks2ks] , I understand from code this test creates a file when start
a new process, but it is not completed within 10s. Please confirm whether the
test can be repeated locally? cc [~gaoyunhaii], [~martijnvisser]
> FileChannelManagerImplTest.testDirectoriesCleanupOnKillWithoutCallerHook
> failed with The marker file was not found within 1 msecs
> -
>
> Key: FLINK-27703
> URL: https://issues.apache.org/jira/browse/FLINK-27703
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Priority: Major
> Labels: test-stability
>
> {code:java}
> 2022-05-19T09:08:49.8088232Z May 19 09:08:49 [ERROR] Failures:
> 2022-05-19T09:08:49.8090850Z May 19 09:08:49 [ERROR]
> FileChannelManagerImplTest.testDirectoriesCleanupOnKillWithoutCallerHook:97->testDirectoriesCleanupOnKill:127
> The marker file was not found within 1 msecs
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=35834&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1&t=24c3384f-1bcb-57b3-224f-51bf973bbee8&l=9744
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] (FLINK-27724) The SubtaskCheckpointCoordinatorImpl#close() isn't called in anywhere
[ https://issues.apache.org/jira/browse/FLINK-27724 ] Wencong Liu deleted comment on FLINK-27724: - was (Author: JIRAUSER281639): Hello [~fanrui] , are you sure this problem exists? On the latest master branch, SubtaskCheckpointCoordinatorImpl#close() is called by many modules. > The SubtaskCheckpointCoordinatorImpl#close() isn't called in anywhere > - > > Key: FLINK-27724 > URL: https://issues.apache.org/jira/browse/FLINK-27724 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.14.0, 1.15.0 >Reporter: fanrui >Priority: Major > Fix For: 1.16.0 > > > The SubtaskCheckpointCoordinatorImpl#close() isn't called in anywhere -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FLINK-27724) The SubtaskCheckpointCoordinatorImpl#close() isn't called in anywhere
[ https://issues.apache.org/jira/browse/FLINK-27724?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17540602#comment-17540602 ] Wencong Liu commented on FLINK-27724: - Sorry, I came to the wrong conclusion just now. In StreamTask#cancel(), Subtaskcheckbpointcoordinatorimpl#close() should probably be called. > The SubtaskCheckpointCoordinatorImpl#close() isn't called in anywhere > - > > Key: FLINK-27724 > URL: https://issues.apache.org/jira/browse/FLINK-27724 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.14.0, 1.15.0 >Reporter: fanrui >Priority: Major > Fix For: 1.16.0 > > > The SubtaskCheckpointCoordinatorImpl#close() isn't called in anywhere -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Commented] (FLINK-30257) SqlClientITCase#testMatchRecognize failed
[
https://issues.apache.org/jira/browse/FLINK-30257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17689133#comment-17689133
]
Wencong Liu commented on FLINK-30257:
-
cc [~martijnvisser]
> SqlClientITCase#testMatchRecognize failed
> -
>
> Key: FLINK-30257
> URL: https://issues.apache.org/jira/browse/FLINK-30257
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.17.0
>Reporter: Martijn Visser
>Assignee: Wencong Liu
>Priority: Major
> Labels: pull-request-available, test-stability
> Attachments: image-2022-12-29-21-47-31-606.png
>
>
> {code:java}
> Nov 30 21:54:41 [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 224.683 s <<< FAILURE! - in SqlClientITCase
> Nov 30 21:54:41 [ERROR] SqlClientITCase.testMatchRecognize Time elapsed:
> 50.164 s <<< FAILURE!
> Nov 30 21:54:41 org.opentest4j.AssertionFailedError:
> Nov 30 21:54:41
> Nov 30 21:54:41 expected: 1
> Nov 30 21:54:41 but was: 0
> Nov 30 21:54:41 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> Nov 30 21:54:41 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> Nov 30 21:54:41 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> Nov 30 21:54:41 at
> SqlClientITCase.verifyNumberOfResultRecords(SqlClientITCase.java:297)
> Nov 30 21:54:41 at
> SqlClientITCase.testMatchRecognize(SqlClientITCase.java:255)
> Nov 30 21:54:41 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 30 21:54:41 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 30 21:54:41 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 30 21:54:41 at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 30 21:54:41 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMetho
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43635&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=160c9ae5-96fd-516e-1c91-deb81f59292a&l=14817
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31020) Read-only mode for Rest API
[ https://issues.apache.org/jira/browse/FLINK-31020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17689172#comment-17689172 ] Wencong Liu commented on FLINK-31020: - Thanks [~omkardeshpande8] for the proposal! I think it is a tricky behavior to only allow GET operations. We cannot guarantee that REST APIs other than submit/cancel/modify do not use POST/PUT operations on the web UI. If you think it's unsafe, you can disable the rest server. > Read-only mode for Rest API > --- > > Key: FLINK-31020 > URL: https://issues.apache.org/jira/browse/FLINK-31020 > Project: Flink > Issue Type: New Feature > Components: Runtime / REST >Affects Versions: 1.16.1 >Reporter: Omkar Deshpande >Priority: Major > > We run Flink jobs on application cluster on Kubernetes. We don't > submit/cancel or modify jobs from rest API or web UI. If there was an option > to enable only GET operations on the rest service, it would greatly solve the > problem of configuring access control and reduce the attack surface. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31020) Read-only mode for Rest API
[ https://issues.apache.org/jira/browse/FLINK-31020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17689173#comment-17689173 ] Wencong Liu commented on FLINK-31020: - cc [~xtsong] > Read-only mode for Rest API > --- > > Key: FLINK-31020 > URL: https://issues.apache.org/jira/browse/FLINK-31020 > Project: Flink > Issue Type: New Feature > Components: Runtime / REST >Affects Versions: 1.16.1 >Reporter: Omkar Deshpande >Priority: Major > > We run Flink jobs on application cluster on Kubernetes. We don't > submit/cancel or modify jobs from rest API or web UI. If there was an option > to enable only GET operations on the rest service, it would greatly solve the > problem of configuring access control and reduce the attack surface. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31092) Hive ITCases fail with OutOfMemoryError
[
https://issues.apache.org/jira/browse/FLINK-31092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17689193#comment-17689193
]
Wencong Liu commented on FLINK-31092:
-
Hi [~mapohl], is the heap dump generated before OOM error occurs?
> Hive ITCases fail with OutOfMemoryError
> ---
>
> Key: FLINK-31092
> URL: https://issues.apache.org/jira/browse/FLINK-31092
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: test-stability
> Attachments: VisualVM-FLINK-31092.png
>
>
> We're experiencing a OutOfMemoryError where the heap space reaches the upper
> limit:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=46161&view=logs&j=fc5181b0-e452-5c8f-68de-1097947f6483&t=995c650b-6573-581c-9ce6-7ad4cc038461&l=23142
> {code}
> Feb 15 05:05:14 [INFO] Running
> org.apache.flink.table.catalog.hive.HiveCatalogITCase
> Feb 15 05:05:17 [INFO] java.lang.OutOfMemoryError: Java heap space
> Feb 15 05:05:17 [INFO] Dumping heap to java_pid9669.hprof ...
> Feb 15 05:05:28 [INFO] Heap dump file created [1957090051 bytes in 11.718
> secs]
> java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.maven.surefire.booter.ForkedBooter.cancelPingScheduler(ForkedBooter.java:209)
> at
> org.apache.maven.surefire.booter.ForkedBooter.acknowledgedExit(ForkedBooter.java:419)
> at
> org.apache.maven.surefire.booter.ForkedBooter.execute(ForkedBooter.java:186)
> at
> org.apache.maven.surefire.booter.ForkedBooter.run(ForkedBooter.java:562)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:548)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31020) Read-only mode for Rest API
[ https://issues.apache.org/jira/browse/FLINK-31020?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17689458#comment-17689458 ] Wencong Liu commented on FLINK-31020: - Sorry [~chesnay] , my statement may not be very accurate. My opinion is consistent with yours. Directly disabling mutating API may affect the normal operation of the web UI. > Read-only mode for Rest API > --- > > Key: FLINK-31020 > URL: https://issues.apache.org/jira/browse/FLINK-31020 > Project: Flink > Issue Type: New Feature > Components: Runtime / REST >Affects Versions: 1.16.1 >Reporter: Omkar Deshpande >Priority: Major > > We run Flink jobs on application cluster on Kubernetes. We don't > submit/cancel or modify jobs from rest API or web UI. If there was an option > to enable only GET operations on the rest service, it would greatly solve the > problem of configuring access control and reduce the attack surface. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] (FLINK-31020) Read-only mode for Rest API
[ https://issues.apache.org/jira/browse/FLINK-31020 ] Wencong Liu deleted comment on FLINK-31020: - was (Author: JIRAUSER281639): cc [~xtsong] > Read-only mode for Rest API > --- > > Key: FLINK-31020 > URL: https://issues.apache.org/jira/browse/FLINK-31020 > Project: Flink > Issue Type: New Feature > Components: Runtime / REST >Affects Versions: 1.16.1 >Reporter: Omkar Deshpande >Priority: Major > > We run Flink jobs on application cluster on Kubernetes. We don't > submit/cancel or modify jobs from rest API or web UI. If there was an option > to enable only GET operations on the rest service, it would greatly solve the > problem of configuring access control and reduce the attack surface. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31176) correct the description of sql gateway configuration
[ https://issues.apache.org/jira/browse/FLINK-31176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691919#comment-17691919 ] Wencong Liu commented on FLINK-31176: - Thanks [~wangkang] ! I'll take a look. > correct the description of sql gateway configuration > > > Key: FLINK-31176 > URL: https://issues.apache.org/jira/browse/FLINK-31176 > Project: Flink > Issue Type: Improvement > Components: Documentation, Table SQL / Gateway >Affects Versions: 1.16.0 >Reporter: wangkang >Priority: Major > Labels: pull-request-available > Attachments: image-2023-02-22-11-17-08-611.png > > > correct the description of sql gateway configuration: > 1.sql-gateway.session.idle-timeout 、sql-gateway.session.check-interval > description in SqlGatewayServiceConfigOptions > 2.GetSessionConfigHeaders and TriggerSessionHeartbeatHeaders class description > !image-2023-02-22-11-17-08-611.png|width=717,height=289! > when setting sql-gateway.session.idle-timeout to negative value,SqlGateway > will throw NumberFormatException,beacause the TimeUtils.pasDuration method > doesn't support the ne negative value,so we should remove the 'or negative > value' description -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31208) KafkaSourceReader overrides meaninglessly a method(pauseOrResumeSplits)
[
https://issues.apache.org/jira/browse/FLINK-31208?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17694026#comment-17694026
]
Wencong Liu commented on FLINK-31208:
-
It looks like some redundant code. cc [~renqs] WDYT?
> KafkaSourceReader overrides meaninglessly a method(pauseOrResumeSplits)
> ---
>
> Key: FLINK-31208
> URL: https://issues.apache.org/jira/browse/FLINK-31208
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
>Reporter: Hongshun Wang
>Priority: Not a Priority
>
> KafkaSourceReader overrides meaninglessly a method(pauseOrResumeSplits)
> ，which is no difference with its Parent class (SourceReaderBase). why not
> remove this override method?
>
> Relative code is here, which we can see is no difference?
> {code:java}
> //org.apache.flink.connector.kafka.source.reader.KafkaSourceReader#pauseOrResumeSplits
> @Override
> public void pauseOrResumeSplits(
> Collection splitsToPause, Collection splitsToResume) {
> splitFetcherManager.pauseOrResumeSplits(splitsToPause, splitsToResume);
> }
> //org.apache.flink.connector.base.source.reader.SourceReaderBase#pauseOrResumeSplits
> @Override
> public void pauseOrResumeSplits(
> Collection splitsToPause, Collection splitsToResume) {
> splitFetcherManager.pauseOrResumeSplits(splitsToPause, splitsToResume);
> }{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30829) Make the backpressure tab could be sort by the backpressure level
[ https://issues.apache.org/jira/browse/FLINK-30829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17694039#comment-17694039 ] Wencong Liu commented on FLINK-30829: - I think if the busy/idle/backpressure is displayed in three columns and each of them can be sorted separately, it will be more clear to users. WDYT? cc [~yunta] [~xtsong] > Make the backpressure tab could be sort by the backpressure level > - > > Key: FLINK-30829 > URL: https://issues.apache.org/jira/browse/FLINK-30829 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Affects Versions: 1.17.0 >Reporter: Zhanghao Chen >Priority: Major > > [FLINK-29998|https://issues.apache.org/jira/browse/FLINK-29998] enables user > to sort the backpressure tab to see which task is busiest. Another common > scenario for backpressure analysis is to find which tasks are backpressured. > We should add support to sort the backpressure tab by backpressure level as > well. > > h4. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31246) Remove PodTemplate description from the SpecChange message
[ https://issues.apache.org/jira/browse/FLINK-31246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17694419#comment-17694419 ] Wencong Liu commented on FLINK-31246: - Hello [~pvary], I'm quite interested in this issue could you please provide the code position? > Remove PodTemplate description from the SpecChange message > -- > > Key: FLINK-31246 > URL: https://issues.apache.org/jira/browse/FLINK-31246 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Reporter: Peter Vary >Priority: Major > > Currently the Spec Change message contains the full PodTemplate twice. > This makes the message seriously big and also contains very little useful > information. > We should abbreviate the message -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30829) Make the backpressure tab could be sort by the backpressure level
[ https://issues.apache.org/jira/browse/FLINK-30829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17695072#comment-17695072 ] Wencong Liu commented on FLINK-30829: - For jobs with high parallelism, it will be very convenient to analyze job status according to sorted busy/idle/backpressure columns. Would you like to continue this? If you don't have time, I can take over. More discussion is also necessary. cc [~Zhanghao Chen] > Make the backpressure tab could be sort by the backpressure level > - > > Key: FLINK-30829 > URL: https://issues.apache.org/jira/browse/FLINK-30829 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Affects Versions: 1.17.0 >Reporter: Zhanghao Chen >Priority: Major > > [FLINK-29998|https://issues.apache.org/jira/browse/FLINK-29998] enables user > to sort the backpressure tab to see which task is busiest. Another common > scenario for backpressure analysis is to find which tasks are backpressured. > We should add support to sort the backpressure tab by backpressure level as > well. > > h4. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30829) Make the backpressure tab could be sort by the backpressure level
[ https://issues.apache.org/jira/browse/FLINK-30829?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17695548#comment-17695548 ] Wencong Liu commented on FLINK-30829: - I think it's necessary to sort the three columns separately. The calculation strategy is different between busy/idle/backpressure, and it may be updated in future. Therefore, it will be inaccurate to infer the top value of backpressure or idle percentage by sorted busy columns. WDYT? [~yunta] > Make the backpressure tab could be sort by the backpressure level > - > > Key: FLINK-30829 > URL: https://issues.apache.org/jira/browse/FLINK-30829 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Affects Versions: 1.17.0 >Reporter: Zhanghao Chen >Priority: Major > > [FLINK-29998|https://issues.apache.org/jira/browse/FLINK-29998] enables user > to sort the backpressure tab to see which task is busiest. Another common > scenario for backpressure analysis is to find which tasks are backpressured. > We should add support to sort the backpressure tab by backpressure level as > well. > > h4. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31298) ConnectionUtilsTest.testFindConnectingAddressWhenGetLocalHostThrows swallows IllegalArgumentException
[
https://issues.apache.org/jira/browse/FLINK-31298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17695974#comment-17695974
]
Wencong Liu commented on FLINK-31298:
-
Hello [~mapohl] , I'd like to take this ticket. SocketOptions.SO_TIMEOUT should
be set to 0.
> ConnectionUtilsTest.testFindConnectingAddressWhenGetLocalHostThrows swallows
> IllegalArgumentException
> -
>
> Key: FLINK-31298
> URL: https://issues.apache.org/jira/browse/FLINK-31298
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.17.0, 1.15.3, 1.16.1
>Reporter: Matthias Pohl
>Priority: Major
> Labels: starter, test-stability
>
> FLINK-24156 introduced {{NetUtils.acceptWithoutTimeout}} which caused the
> test to print a the stacktrace of an {{IllegalArgumentException}}:
> {code}
> Exception in thread "Thread-0" java.lang.IllegalArgumentException:
> serverSocket SO_TIMEOUT option must be 0
> at
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:138)
> at
> org.apache.flink.util.NetUtils.acceptWithoutTimeout(NetUtils.java:139)
> at
> org.apache.flink.runtime.net.ConnectionUtilsTest$1.run(ConnectionUtilsTest.java:83)
> at java.lang.Thread.run(Thread.java:750)
> {code}
> This is also shown in the Maven output of CI runs and might cause confusion.
> The test should be fixed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-27051) CompletedCheckpoint.DiscardObject.discard is not idempotent
[ https://issues.apache.org/jira/browse/FLINK-27051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17696419#comment-17696419 ] Wencong Liu commented on FLINK-27051: - Hello [~mapohl] , I'm quite interested in the issues under this umbrella. For this issue, do you mean the CompletedCheckpoint.DiscardObject.discard should only discard related data at the first time when it's invoked in multiple times？ > CompletedCheckpoint.DiscardObject.discard is not idempotent > --- > > Key: FLINK-27051 > URL: https://issues.apache.org/jira/browse/FLINK-27051 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Affects Versions: 1.15.0 >Reporter: Matthias Pohl >Priority: Major > > `CompletedCheckpoint.DiscardObject.discard` is not implemented in an > idempotent fashion because we're losing the operatorState even in the case of > a failure (see > [CompletedCheckpoint:328||https://github.com/apache/flink/blob/dc419b5639f68bcb0b773763f24179dd3536d713/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CompletedCheckpoint.java#L328]. > This prevents us from retrying the deletion. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-27051) CompletedCheckpoint.DiscardObject.discard is not idempotent
[
https://issues.apache.org/jira/browse/FLINK-27051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17696970#comment-17696970
]
Wencong Liu commented on FLINK-27051:
-
Thanks for the explanation [~mapohl] . I think I can try this issue, could you
please assign to me?
> CompletedCheckpoint.DiscardObject.discard is not idempotent
> ---
>
> Key: FLINK-27051
> URL: https://issues.apache.org/jira/browse/FLINK-27051
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Matthias Pohl
>Priority: Major
>
> {{CompletedCheckpoint.DiscardObject.discard}} is not implemented in an
> idempotent fashion because we're losing the operatorState even in the case of
> a failure (see
> [CompletedCheckpoint:328|https://github.com/apache/flink/blob/dc419b5639f68bcb0b773763f24179dd3536d713/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CompletedCheckpoint.java#L328]).
> This prevents us from retrying the deletion.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31298) ConnectionUtilsTest.testFindConnectingAddressWhenGetLocalHostThrows swallows IllegalArgumentException
[
https://issues.apache.org/jira/browse/FLINK-31298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17697698#comment-17697698
]
Wencong Liu commented on FLINK-31298:
-
cc [~mapohl]
> ConnectionUtilsTest.testFindConnectingAddressWhenGetLocalHostThrows swallows
> IllegalArgumentException
> -
>
> Key: FLINK-31298
> URL: https://issues.apache.org/jira/browse/FLINK-31298
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.17.0, 1.15.3, 1.16.1
>Reporter: Matthias Pohl
>Assignee: Wencong Liu
>Priority: Major
> Labels: pull-request-available, starter, test-stability
>
> FLINK-24156 introduced {{NetUtils.acceptWithoutTimeout}} which caused the
> test to print a the stacktrace of an {{IllegalArgumentException}}:
> {code}
> Exception in thread "Thread-0" java.lang.IllegalArgumentException:
> serverSocket SO_TIMEOUT option must be 0
> at
> org.apache.flink.util.Preconditions.checkArgument(Preconditions.java:138)
> at
> org.apache.flink.util.NetUtils.acceptWithoutTimeout(NetUtils.java:139)
> at
> org.apache.flink.runtime.net.ConnectionUtilsTest$1.run(ConnectionUtilsTest.java:83)
> at java.lang.Thread.run(Thread.java:750)
> {code}
> This is also shown in the Maven output of CI runs and might cause confusion.
> The test should be fixed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-27051) CompletedCheckpoint.DiscardObject.discard is not idempotent
[
https://issues.apache.org/jira/browse/FLINK-27051?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17697772#comment-17697772
]
Wencong Liu commented on FLINK-27051:
-
cc [~mapohl]
> CompletedCheckpoint.DiscardObject.discard is not idempotent
> ---
>
> Key: FLINK-27051
> URL: https://issues.apache.org/jira/browse/FLINK-27051
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Matthias Pohl
>Assignee: Wencong Liu
>Priority: Major
> Labels: pull-request-available
>
> {{CompletedCheckpoint.DiscardObject.discard}} is not implemented in an
> idempotent fashion because we're losing the operatorState even in the case of
> a failure (see
> [CompletedCheckpoint:328|https://github.com/apache/flink/blob/dc419b5639f68bcb0b773763f24179dd3536d713/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/CompletedCheckpoint.java#L328]).
> This prevents us from retrying the deletion.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-28046) Annotate SourceFunction as deprecated
[ https://issues.apache.org/jira/browse/FLINK-28046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17698237#comment-17698237 ] Wencong Liu commented on FLINK-28046: - cc [~Weijie Guo] > Annotate SourceFunction as deprecated > - > > Key: FLINK-28046 > URL: https://issues.apache.org/jira/browse/FLINK-28046 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream >Affects Versions: 1.15.3 >Reporter: Alexander Fedulov >Assignee: Alexander Fedulov >Priority: Major > Labels: pull-request-available, stale-assigned > Fix For: 1.17.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17698338#comment-17698338
]
Wencong Liu commented on FLINK-31378:
-
Hello [~loserwang1024] , the links in your proposal may be not correct? Could
you please give the correct link?
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Priority: Not a Priority
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17698703#comment-17698703
]
Wencong Liu commented on FLINK-31378:
-
The flink-streaming-java module does not have a dependency on flink-clients
module.
[FLINK-15090] Reverse the dependency from flink-streaming-java to flink-client
- ASF JIRA (apache.org)
I think the docs should be modified, WDYT? cc [~Weijie Guo]
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Priority: Not a Priority
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31378) Documentation fails to build due to lack of package
[
https://issues.apache.org/jira/browse/FLINK-31378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17698726#comment-17698726
]
Wencong Liu commented on FLINK-31378:
-
[~loserwang1024] | have a fix for this document. If you like, you can
participate in the review together.
> Documentation fails to build due to lack of package
> ---
>
> Key: FLINK-31378
> URL: https://issues.apache.org/jira/browse/FLINK-31378
> Project: Flink
> Issue Type: Improvement
> Components: Documentation
>Reporter: Hongshun Wang
>Priority: Not a Priority
> Attachments: image-2023-03-10-11-19-35-773.png
>
>
> In [Project Configuration
> Section|[https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/dev/configuration/overview/#running-and-packaging],]
> it shows that "If you want to run your job by simply executing the main
> class, you will need {{flink-runtime}} in your classpath".
> However, when I just add flink-runtime in my classPath, an error is thrown
> like this:"
> No ExecutorFactory found to execute the application".
> It seems that flink-clients is also needed to supply an excutor through Java
> Service Load.
> Could you please add this in official article for beginners like me?
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31358) Display consumed split/partition/queue info on Web UI
[ https://issues.apache.org/jira/browse/FLINK-31358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699559#comment-17699559 ] Wencong Liu commented on FLINK-31358: - I seems a specific design for source subtasks. It's necessary to consider the unified implementation for different operators. > Display consumed split/partition/queue info on Web UI > - > > Key: FLINK-31358 > URL: https://issues.apache.org/jira/browse/FLINK-31358 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Reporter: Zhanghao Chen >Priority: Minor > > Many data sources have the concept of "split", which is a partition of the > whole data (e.g. partition in Kafka, queue in RocketMQ), and each Flink > source task is allocated with a subset of splits to consume. When a job is > lagging on only a few splits, it would be useful for determining whether it > is a data source issue or a Flink issue if users can view which source task > consumes which split on Web UI. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-28538) Eagerly emit a watermark the first time we exceed Long.MIN_VALUE
[
https://issues.apache.org/jira/browse/FLINK-28538?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17699613#comment-17699613
]
Wencong Liu commented on FLINK-28538:
-
Hello [~chesnay] , I think the BoundedOutOfOrderWatermarkGenerator will produce
the watermark for each record.
{code:java}
public class BoundedOutOfOrderWatermarkGenerator extends WatermarkGenerator {
private static final long serialVersionUID = 1L;
private final long delay;
private final int rowtimeIndex;
/**
* @param rowtimeIndex the field index of rowtime attribute, the value of
rowtime should never
* be null.
* @param delay The delay by which watermarks are behind the observed
timestamp.
*/
public BoundedOutOfOrderWatermarkGenerator(int rowtimeIndex, long delay) {
this.delay = delay;
this.rowtimeIndex = rowtimeIndex;
}
@Nullable
@Override
public Long currentWatermark(RowData row) {
return row.getLong(rowtimeIndex) - delay;
}
} {code}
> Eagerly emit a watermark the first time we exceed Long.MIN_VALUE
>
>
> Key: FLINK-28538
> URL: https://issues.apache.org/jira/browse/FLINK-28538
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Chesnay Schepler
>Priority: Major
> Fix For: 1.17.0
>
>
> The built-in watermark generators are set up to start with a Long.MIN_VALUE
> watermark, that is updated on each event and periodically emitted later on.
> As a result, before the first periodic emission is triggered (shortly after
> job (re)start) no element is ever considered late.
> We should be able to remedy that by emitting a watermark on the first event.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30259) Use flink Preconditions Util instead of uncertain Assert keyword to do checking
[
https://issues.apache.org/jira/browse/FLINK-30259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700027#comment-17700027
]
Wencong Liu commented on FLINK-30259:
-
cc [~chesnay]
> Use flink Preconditions Util instead of uncertain Assert keyword to do
> checking
> ---
>
> Key: FLINK-30259
> URL: https://issues.apache.org/jira/browse/FLINK-30259
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes, Formats (JSON, Avro, Parquet,
> ORC, SequenceFile), Table SQL / Planner
>Affects Versions: 1.16.0
>Reporter: Ran Tao
>Priority: Major
>
> The code of some modules of the current Flink project uses the 'assert'
> keyword of java to do checking, which actually depends on the enablement of
> the -enableassertions (-ea) option (default is false, which means some assert
> code can not work), otherwise it may lead to unexpected behavior. In fact,
> flink already has a mature Preconditions tool, we can use it to replace
> 'assert' keyword. it is more clean and consistent with flink.
> The following is an example of some snippets (by using idea, we can find
> other places ).
> RowDataPrintFunction
> {code:java}
> @Override
> public void invoke(RowData value, Context context) {
> Object data = converter.toExternal(value);
> assert data != null;
> writer.write(data.toString());
> }
> {code}
> e.g. if assert not enable,data.toString() will cause NPE.
> KubernetesUtils
> {code:java}
> public static KubernetesConfigMap checkConfigMaps(
> List configMaps, String
> expectedConfigMapName) {
> assert (configMaps.size() == 1);
> assert (configMaps.get(0).getName().equals(expectedConfigMapName));
> return configMaps.get(0);
> }
> {code}
> e.g. if assert not enable,configMaps.get(0)will cause NPE.
> RocksDBOperationUtils
> {code:java}
> if (memoryConfig.isUsingFixedMemoryPerSlot()) {
> assert memoryConfig.getFixedMemoryPerSlot() != null;
> logger.info("Getting fixed-size shared cache for RocksDB.");
> return memoryManager.getExternalSharedMemoryResource(
> FIXED_SLOT_MEMORY_RESOURCE_ID,
> allocator,
> // if assert not enable, here will cause NPE.
> memoryConfig.getFixedMemoryPerSlot().getBytes());
> } else {
> logger.info("Getting managed memory shared cache for
> RocksDB.");
> return memoryManager.getSharedMemoryResourceForManagedMemory(
> MANAGED_MEMORY_RESOURCE_ID, allocator,
> memoryFraction);
> }
> {code}
> e.g. if assert not enable,
> RocksDBOperationUtils#memoryConfig.getFixedMemoryPerSlot().getBytes()) will
> cause NPE.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30077) k8s jobmanager pod repeated restart
[
https://issues.apache.org/jira/browse/FLINK-30077?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700031#comment-17700031
]
Wencong Liu commented on FLINK-30077:
-
Hello [~baibaiwuchang] . Thanks for your proposal! Actually I can't understand
your statement completely. Could you please illustrate this sentence in detail ?
{code:java}
Flink kuberne moudle watch taskmanager pod. We always watch jobmanager pod
status and flink cancel deployment In my particular situation. {code}
> k8s jobmanager pod repeated restart
> ---
>
> Key: FLINK-30077
> URL: https://issues.apache.org/jira/browse/FLINK-30077
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes
>Reporter: hanjie
>Priority: Major
>
> We use Flink K8S. When task exist bug, jobmanager pod repeated start.
> For example：
> -88b95598d-rlzxg 0/1 CrashLoopBackOff 215
> 19h
>
> then, i learned that k8s deployment could only set "restartPolicy:Always".
> ([https://github.com/kubernetes/kubernetes/issues/24725)]
> When jobmanager always restart, I don't think that's understandable。
> Flink kuberne moudle watch taskmanager pod. We always watch jobmanager pod
> status and flink cancel deployment In my particular situation.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29705) Document the least access with RBAC setting for native K8s integration
[ https://issues.apache.org/jira/browse/FLINK-29705?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700032#comment-17700032 ] Wencong Liu commented on FLINK-29705: - Hello [~ouyangwuli] . Are you still working on this? > Document the least access with RBAC setting for native K8s integration > -- > > Key: FLINK-29705 > URL: https://issues.apache.org/jira/browse/FLINK-29705 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes, Documentation >Reporter: Yang Wang >Assignee: ouyangwulin >Priority: Major > > We should document the least access with RBAC settings[1]. And the operator > docs could be taken as a reference[2]. > > [1]. > [https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/resource-providers/native_kubernetes/#rbac] > [2]. > [https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/operations/rbac/] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-28023) Translate "Native Kubernetes" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-28023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700049#comment-17700049 ] Wencong Liu commented on FLINK-28023: - cc [~linqichen178] > Translate "Native Kubernetes" page into Chinese > --- > > Key: FLINK-28023 > URL: https://issues.apache.org/jira/browse/FLINK-28023 > Project: Flink > Issue Type: Improvement > Components: Deployment / Kubernetes >Affects Versions: 1.14.4 >Reporter: linqichen >Priority: Major > > Hi, I think it is very urgent to translate 'native kubernetes' page into > chinese. > Can you assign this to me ?? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-27550) Remove checking yarn queues before submitting job to Yarn
[
https://issues.apache.org/jira/browse/FLINK-27550?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700057#comment-17700057
]
Wencong Liu commented on FLINK-27550:
-
cc [~yangwang166]
> Remove checking yarn queues before submitting job to Yarn
> -
>
> Key: FLINK-27550
> URL: https://issues.apache.org/jira/browse/FLINK-27550
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / YARN
>Reporter: Junfan Zhang
>Priority: Major
> Labels: pull-request-available
>
> When invoking the method of {{checkYarnQueues}} in YarnClusterDescriptor,
> it will check the specified yarnQueue whether exists in the queues gotten by
> the YarnClient.QueueInfo.
> However when using the capacity-scheduler, the yarn queues path should be
> retrieved by the api of {{QueueInfo.getQueuePath}}
> instead of {{getQueueName}}.
> Due to this, it will always print out the yarn all queues log, but it also
> can be submitted to Yarn successfully.
> The api of getQueuePath is introduced in the latest hadoop
> version(https://issues.apache.org/jira/browse/YARN-10658), so it's hard to
> solve this problem in the older hadoop cluster.
> According to the above description, the process of checking is unnecessary.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-26425) Yarn aggregate log files in a rolling fashion for flink
[ https://issues.apache.org/jira/browse/FLINK-26425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700068#comment-17700068 ] Wencong Liu commented on FLINK-26425: - [~davedwwang] , thanks for your proposal! Could you give some examples of the rolling fashion under the settings of spark.yarn.rolledLog.excludePattern and spark.yarn.rolledLog.includePattern? > Yarn aggregate log files in a rolling fashion for flink > --- > > Key: FLINK-26425 > URL: https://issues.apache.org/jira/browse/FLINK-26425 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.14.3 >Reporter: wang da wei >Priority: Major > > For the spark on yarn, you can set the > spark.yarn.rolledLog.includePattern、spark.yarn.rolledLog.excludePattern to > filter the log files and those log files will be aggregated in a rolling > fashion,Do you have similar Settings in Flink？ -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29334) StateHandleStore#releaseAndTryRemoveAll is not used and can be removed
[
https://issues.apache.org/jira/browse/FLINK-29334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700510#comment-17700510
]
Wencong Liu commented on FLINK-29334:
-
Hello [~mapohl], I'd like to take this ticket. Could you please assign to me？
> StateHandleStore#releaseAndTryRemoveAll is not used and can be removed
> --
>
> Key: FLINK-29334
> URL: https://issues.apache.org/jira/browse/FLINK-29334
> Project: Flink
> Issue Type: Technical Debt
> Components: Runtime / Coordination
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Priority: Minor
> Labels: starer
>
> {{StateHandleStore#releaseAndTryRemoveAll}} isn't used in production code.
> There is also not a real reason to do a final cleanup. We should clean up
> component at the right location that than doing a wipe-out at the end.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-27997) How to unregister custom metrics at runtime?
[ https://issues.apache.org/jira/browse/FLINK-27997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700636#comment-17700636 ] Wencong Liu commented on FLINK-27997: - Hello [~henvealf] . Thanks for your proposal. What metrics do you think will cause memory overflow? > How to unregister custom metrics at runtime? > > > Key: FLINK-27997 > URL: https://issues.apache.org/jira/browse/FLINK-27997 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics >Reporter: henvealf >Priority: Major > > How to custom unregister metrics? > Worry about memory overflow due to too many metrics. > Any suggestions for adding a feature to unregister metrics? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-27954) JobVertexFlameGraphHandler does not work on standby Dispatcher
[
https://issues.apache.org/jira/browse/FLINK-27954?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700638#comment-17700638
]
Wencong Liu commented on FLINK-27954:
-
cc [~Weijie Guo]
> JobVertexFlameGraphHandler does not work on standby Dispatcher
> --
>
> Key: FLINK-27954
> URL: https://issues.apache.org/jira/browse/FLINK-27954
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination, Runtime / REST
>Affects Versions: 1.15.0
>Reporter: Chesnay Schepler
>Assignee: Weijie Guo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.17.0, 1.16.2, 1.15.5
>
>
> The {{JobVertexFlameGraphHandler}} relies internally on the
> {{JobVertexThreadInfoTracker}} which calls
> {{ResourceManagerGateway#requestTaskExecutorThreadInfoGateway}} to get a
> gateway for requesting the thread info from the task executors. Since this
> gateway is not serializable it would categorically fail if called from a
> standby dispatcher.
> Instead this should follow the logic of the {{MetricFetcherImpl}}, which
> requests addresses instead and manually connects to the task executors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-27918) SavepointITCase.testStopWithSavepointFailingAfterSnapshotCreation failed with Expected RuntimeException after snapshot creation
[
https://issues.apache.org/jira/browse/FLINK-27918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700643#comment-17700643
]
Wencong Liu commented on FLINK-27918:
-
cc @[~dmvk]
> SavepointITCase.testStopWithSavepointFailingAfterSnapshotCreation failed with
> Expected RuntimeException after snapshot creation
> ---
>
> Key: FLINK-27918
> URL: https://issues.apache.org/jira/browse/FLINK-27918
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Priority: Major
> Labels: auto-deprioritized-critical, test-stability
>
> {code:java}
> 2022-06-06T03:13:54.0165829Z Jun 06 03:13:54 [ERROR]
> org.apache.flink.test.checkpointing.SavepointITCase.testStopWithSavepointFailingAfterSnapshotCreation
> Time elapsed: 0.242 s <<< ERROR!
> 2022-06-06T03:13:54.0167256Z Jun 06 03:13:54
> java.util.concurrent.ExecutionException:
> org.apache.flink.util.FlinkException: Stop with savepoint operation could not
> be completed.
> 2022-06-06T03:13:54.0173825Z Jun 06 03:13:54 at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2022-06-06T03:13:54.0174662Z Jun 06 03:13:54 at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> 2022-06-06T03:13:54.0180645Z Jun 06 03:13:54 at
> org.apache.flink.test.checkpointing.SavepointITCase.testStopWithFailingSourceInOnePipeline(SavepointITCase.java:1175)
> 2022-06-06T03:13:54.0181702Z Jun 06 03:13:54 at
> org.apache.flink.test.checkpointing.SavepointITCase.testStopWithSavepointFailingAfterSnapshotCreation(SavepointITCase.java:1020)
> 2022-06-06T03:13:54.0182472Z Jun 06 03:13:54 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-06-06T03:13:54.0184012Z Jun 06 03:13:54 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-06-06T03:13:54.0185109Z Jun 06 03:13:54 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-06-06T03:13:54.0185907Z Jun 06 03:13:54 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-06-06T03:13:54.0187049Z Jun 06 03:13:54 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2022-06-06T03:13:54.0188081Z Jun 06 03:13:54 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2022-06-06T03:13:54.0189241Z Jun 06 03:13:54 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2022-06-06T03:13:54.0190002Z Jun 06 03:13:54 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2022-06-06T03:13:54.0190704Z Jun 06 03:13:54 at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2022-06-06T03:13:54.0191400Z Jun 06 03:13:54 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2022-06-06T03:13:54.0192051Z Jun 06 03:13:54 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2022-06-06T03:13:54.0192883Z Jun 06 03:13:54 at
> org.apache.flink.util.TestNameProvider$1.evaluate(TestNameProvider.java:45)
> 2022-06-06T03:13:54.0194154Z Jun 06 03:13:54 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:61)
> 2022-06-06T03:13:54.0195096Z Jun 06 03:13:54 at
> org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> 2022-06-06T03:13:54.0196226Z Jun 06 03:13:54 at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> 2022-06-06T03:13:54.0197088Z Jun 06 03:13:54 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> 2022-06-06T03:13:54.0198037Z Jun 06 03:13:54 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> 2022-06-06T03:13:54.0199186Z Jun 06 03:13:54 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> 2022-06-06T03:13:54.0200147Z Jun 06 03:13:54 at
> org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> 2022-06-06T03:13:54.0200956Z Jun 06 03:13:54 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> 2022-06-06T03:13:54.0201666Z Jun 06 03:13:54 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> 2022-06-06T03:13:54.0202304Z Jun 06 03:13:54 at
> org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> 2022-06-06T03:13:54.0202932Z Jun 06 03:13:54 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> 2022-06-06T03:13:54.0203971Z Jun 06 03:13:54 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:54)
> 2022-06-06T03:13:54.0204588Z Jun 06 03:13:54 at
> o
[jira] [Commented] (FLINK-27900) Decouple the advertisedAddress and rest.bind-address
[
https://issues.apache.org/jira/browse/FLINK-27900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700660#comment-17700660
]
Wencong Liu commented on FLINK-27900:
-
Hello [~LucentWong] , I think if advertisedAddress is decoupled from the
bind-address, it means the advertisedAddress can be set by user itself. It's
conflicted with it's meaning of "advertised".
> Decouple the advertisedAddress and rest.bind-address
>
>
> Key: FLINK-27900
> URL: https://issues.apache.org/jira/browse/FLINK-27900
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / REST
>Affects Versions: 1.10.3, 1.12.0, 1.11.6, 1.13.6, 1.14.4
> Environment: Flink 1.13, 1.12, 1.11, 1.10 with ssl
> Deploy Flink in Kubernetes pod with a nginx sidecar for auth
>Reporter: Yu Wang
>Priority: Minor
>
> Currently the Flink Rest api does not have authentication, according to the
> doc
> [https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/deployment/security/security-ssl/#external--rest-connectivity]
> # We set up the Flink cluster in k8s
> # We set up a nginx sidecar to enable auth for Flink Rest api.
> # We set *rest.bind-address* to localhost to hide the original Flink address
> and port
> # We enabled the ssl for the Flink Rest api
> It works fine wen the client tried to call the Flink Rest api with *https*
> scheme.
> But if the client using *http* scheme, the *RedirectingSslHandler* will try
> to redirect the address to the advertised url. According to
> {*}RestServerEndpoint{*}, Flink will use the value of *rest.bind-address* as
> the {*}advertisedAddress{*}. So the client will be redirected to *127.0.0.1*
> and failed to connect the url.
> So we hope the advertisedAddress can be decoupled with rest.bind-addres, to
> provide more flexibility to the Flink deployment.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-27204) FileSystemJobResultStore should operator on the ioExecutor
[
https://issues.apache.org/jira/browse/FLINK-27204?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701046#comment-17701046
]
Wencong Liu commented on FLINK-27204:
-
Hello [~mapohl], I think changing the JobResultStore interface is a great idea.
I’m interested in this task and I'm wondering if you would be willing to
assign it to me? I would be happy to contribute.
> FileSystemJobResultStore should operator on the ioExecutor
> --
>
> Key: FLINK-27204
> URL: https://issues.apache.org/jira/browse/FLINK-27204
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.16.0
>Reporter: Matthias Pohl
>Priority: Major
>
> The {{JobResultStore}} interface is synchronous currently. For the
> {{FileSystemJobResultStore}} this means that (possibly) IO-heavy operations
> have to be explicitly called moved to the ioExecutor within the Dispatcher.
> Instead, we could change the {{JobResultStore}} interface in a way that it
> returns {{CompletableFuture}} instances instead. That would enable us to run
> the {{FileSystemJobResultStore}} operations in the ioExecutor which would be
> set when initializing the {{{}FileSystemJobResultStore{}}}. This would move
> the responsibility of where to run the operation from the {{Dispatcher}} into
> the {{JobResultStore.}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34237) MongoDB connector compile failed with Flink 1.19-SNAPSHOT
[
https://issues.apache.org/jira/browse/FLINK-34237?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17811101#comment-17811101
]
Wencong Liu commented on FLINK-34237:
-
Thanks for the reminder. I'll fix it as soon as possible.
> MongoDB connector compile failed with Flink 1.19-SNAPSHOT
> -
>
> Key: FLINK-34237
> URL: https://issues.apache.org/jira/browse/FLINK-34237
> Project: Flink
> Issue Type: Bug
> Components: API / Core, Connectors / MongoDB
>Reporter: Leonard Xu
>Assignee: Wencong Liu
>Priority: Blocker
> Fix For: 1.19.0
>
>
> {code:java}
> Error: Failed to execute goal
> org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile
> (default-compile) on project flink-connector-mongodb: Compilation failure
> 134Error:
> /home/runner/work/flink-connector-mongodb/flink-connector-mongodb/flink-connector-mongodb/src/main/java/org/apache/flink/connector/mongodb/source/reader/MongoSourceReaderContext.java:[35,8]
> org.apache.flink.connector.mongodb.source.reader.MongoSourceReaderContext is
> not abstract and does not override abstract method getTaskInfo() in
> org.apache.flink.api.connector.source.SourceReaderContext
> 135{code}
> [https://github.com/apache/flink-connector-mongodb/actions/runs/7657281844/job/20867604084]
> This is related to FLINK-33905
> One point: As
> [FLIP-382|https://cwiki.apache.org/confluence/display/FLINK/FLIP-382%3A+Unify+the+Provision+of+Diverse+Metadata+for+Context-like+APIs]
> is accepted, all connectors who implement SourceReaderContext (i.e
> MongoSourceReaderContext) should implement new introduced methods `
> getTaskInfo()` if they want to compile/work with Flink 1.19.
> Another point: The FLIP-382 didn't mentioned the connector backward
> compatibility well, maybe we need to rethink the section. As I just have a
> rough look at the FLIP, maybe [~xtsong] and [~Wencong Liu] could comment
> under this issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-34246) Allow only archive failed job to history server
[ https://issues.apache.org/jira/browse/FLINK-34246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17811468#comment-17811468 ] Wencong Liu commented on FLINK-34246: - Thanks [~qingwei91], for suggesting this. Are you suggesting that we should offer an option that allows the HistoryServer to archive only the failed batch jobs? This requirement seems quite specific. For instance, we would also need to consider archiving the logs of failed streaming jobs. > Allow only archive failed job to history server > --- > > Key: FLINK-34246 > URL: https://issues.apache.org/jira/browse/FLINK-34246 > Project: Flink > Issue Type: Improvement > Components: Client / Job Submission >Reporter: Lim Qing Wei >Priority: Minor > > Hi, I wonder if we can support only archiving Failed job to History Server. > History server is a great tool to allow us to check on previous job, we are > using FLink batch which can run many times throughout the week, we only need > to check job on History Server when it has failed. > It would be more efficient if we can choose to only store a subset of the > data. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (FLINK-32978) Deprecate RichFunction#open(Configuration parameters)
[ https://issues.apache.org/jira/browse/FLINK-32978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wencong Liu reopened FLINK-32978: - > Deprecate RichFunction#open(Configuration parameters) > - > > Key: FLINK-32978 > URL: https://issues.apache.org/jira/browse/FLINK-32978 > Project: Flink > Issue Type: Technical Debt > Components: API / Core >Affects Versions: 1.19.0 >Reporter: Wencong Liu >Assignee: Wencong Liu >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > The > [FLIP-344|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425231] > has decided that the parameter in RichFunction#open will be removed in the > next major version. We should deprecate it now and remove it in Flink 2.0. > The removal will be tracked in > [FLINK-6912|https://issues.apache.org/jira/browse/FLINK-6912]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-32978) Deprecate RichFunction#open(Configuration parameters)
[ https://issues.apache.org/jira/browse/FLINK-32978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wencong Liu closed FLINK-32978. --- Release Note: The RichFunction#open(Configuration parameters) method has been deprecated and will be removed in future versions. Users are encouraged to migrate to the new RichFunction#open(OpenContext openContext) method, which provides a more comprehensive context for initialization. Here are the key changes and recommendations for migration: The open(Configuration parameters) method is now marked as deprecated. A new method open(OpenContext openContext) has been added as a default method to the RichFunction interface. Users should implement the new open(OpenContext openContext) method for function initialization tasks. The new method will be called automatically before the execution of any processing methods (map, join, etc.). If the new open(OpenContext openContext) method is not implemented, Flink will fall back to invoking the deprecated open(Configuration parameters) method. Resolution: Fixed > Deprecate RichFunction#open(Configuration parameters) > - > > Key: FLINK-32978 > URL: https://issues.apache.org/jira/browse/FLINK-32978 > Project: Flink > Issue Type: Technical Debt > Components: API / Core >Affects Versions: 1.19.0 >Reporter: Wencong Liu >Assignee: Wencong Liu >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > The > [FLIP-344|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425231] > has decided that the parameter in RichFunction#open will be removed in the > next major version. We should deprecate it now and remove it in Flink 2.0. > The removal will be tracked in > [FLINK-6912|https://issues.apache.org/jira/browse/FLINK-6912]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32978) Deprecate RichFunction#open(Configuration parameters)
[ https://issues.apache.org/jira/browse/FLINK-32978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17812213#comment-17812213 ] Wencong Liu commented on FLINK-32978: - [~martijnvisser] Thanks for the reminding. I've added the release notes information. > Deprecate RichFunction#open(Configuration parameters) > - > > Key: FLINK-32978 > URL: https://issues.apache.org/jira/browse/FLINK-32978 > Project: Flink > Issue Type: Technical Debt > Components: API / Core >Affects Versions: 1.19.0 >Reporter: Wencong Liu >Assignee: Wencong Liu >Priority: Major > Labels: pull-request-available > Fix For: 1.19.0 > > > The > [FLIP-344|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=263425231] > has decided that the parameter in RichFunction#open will be removed in the > next major version. We should deprecate it now and remove it in Flink 2.0. > The removal will be tracked in > [FLINK-6912|https://issues.apache.org/jira/browse/FLINK-6912]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-34132) Batch WordCount job fails when run with AdaptiveBatch scheduler
[
https://issues.apache.org/jira/browse/FLINK-34132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17813316#comment-17813316
]
Wencong Liu commented on FLINK-34132:
-
Thanks for the reminding. [~zhuzh] I will address these issues when I have some
free time. 😄
> Batch WordCount job fails when run with AdaptiveBatch scheduler
> ---
>
> Key: FLINK-34132
> URL: https://issues.apache.org/jira/browse/FLINK-34132
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.17.1, 1.18.1
>Reporter: Prabhu Joseph
>Assignee: Junrui Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.19.0
>
>
> Batch WordCount job fails when run with AdaptiveBatch scheduler.
> *Repro Steps*
> {code:java}
> flink-yarn-session -Djobmanager.scheduler=adaptive -d
> flink run -d /usr/lib/flink/examples/batch/WordCount.jar --input
> s3://prabhuflinks3/INPUT --output s3://prabhuflinks3/OUT
> {code}
> *Error logs*
> {code:java}
> The program finished with the following exception:
> org.apache.flink.client.program.ProgramInvocationException: The main method
> caused an error: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException:
> org.apache.flink.runtime.client.JobInitializationException: Could not start
> the JobMaster.
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:372)
> at
> org.apache.flink.client.program.PackagedProgram.invokeInteractiveModeForExecution(PackagedProgram.java:222)
> at
> org.apache.flink.client.ClientUtils.executeProgram(ClientUtils.java:105)
> at
> org.apache.flink.client.cli.CliFrontend.executeProgram(CliFrontend.java:851)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:245)
> at
> org.apache.flink.client.cli.CliFrontend.parseAndRun(CliFrontend.java:1095)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$mainInternal$9(CliFrontend.java:1189)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at
> org.apache.flink.runtime.security.contexts.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
> at
> org.apache.flink.client.cli.CliFrontend.mainInternal(CliFrontend.java:1189)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1157)
> Caused by: java.lang.RuntimeException:
> java.util.concurrent.ExecutionException: java.lang.RuntimeException:
> org.apache.flink.runtime.client.JobInitializationException: Could not start
> the JobMaster.
> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
> at
> org.apache.flink.api.java.ExecutionEnvironment.executeAsync(ExecutionEnvironment.java:1067)
> at
> org.apache.flink.client.program.ContextEnvironment.executeAsync(ContextEnvironment.java:144)
> at
> org.apache.flink.client.program.ContextEnvironment.execute(ContextEnvironment.java:73)
> at
> org.apache.flink.examples.java.wordcount.WordCount.main(WordCount.java:106)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.apache.flink.client.program.PackagedProgram.callMainMethod(PackagedProgram.java:355)
> ... 12 more
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException:
> org.apache.flink.runtime.client.JobInitializationException: Could not start
> the JobMaster.
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.api.java.ExecutionEnvironment.executeAsync(ExecutionEnvironment.java:1062)
> ... 20 more
> Caused by: java.lang.RuntimeException:
> org.apache.flink.runtime.client.JobInitializationException: Could not start
> the JobMaster.
> at org.apache.flink.util.ExceptionUtils.rethrow(ExceptionUtils.java:321)
> at
> org.apache.flink.util.function.FunctionUtils.lambda$uncheckedFunction$2(FunctionUtils.java:75)
> at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> at
> java.util.concurrent.CompletableFuture$Completion.exec(CompletableFuture.java:457)
> at java.util.concurren
[jira] [Commented] (FLINK-34448) ChangelogLocalRecoveryITCase#testRestartTM failed fatally with 127 exit code
[
https://issues.apache.org/jira/browse/FLINK-34448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17818186#comment-17818186
]
Wencong Liu commented on FLINK-34448:
-
Maybe [~Yanfei Lei] could take a look 😄.
> ChangelogLocalRecoveryITCase#testRestartTM failed fatally with 127 exit code
>
>
> Key: FLINK-34448
> URL: https://issues.apache.org/jira/browse/FLINK-34448
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.20.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: test-stability
> Attachments: FLINK-34448.head.log.gz
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=57550&view=logs&j=2c3cbe13-dee0-5837-cf47-3053da9a8a78&t=b78d9d30-509a-5cea-1fef-db7abaa325ae&l=8897
> \
> {code}
> Feb 16 02:43:47 02:43:47.142 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:3.2.2:test (integration-tests)
> on project flink-tests:
> Feb 16 02:43:47 02:43:47.142 [ERROR]
> Feb 16 02:43:47 02:43:47.142 [ERROR] Please refer to
> /__w/1/s/flink-tests/target/surefire-reports for the individual test results.
> Feb 16 02:43:47 02:43:47.142 [ERROR] Please refer to dump files (if any
> exist) [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
> Feb 16 02:43:47 02:43:47.142 [ERROR] ExecutionException The forked VM
> terminated without properly saying goodbye. VM crash or System.exit called?
> Feb 16 02:43:47 02:43:47.142 [ERROR] Command was /bin/sh -c cd
> '/__w/1/s/flink-tests' && '/usr/lib/jvm/jdk-11.0.19+7/bin/java'
> '-XX:+UseG1GC' '-Xms256m' '-XX:+IgnoreUnrecognizedVMOptions'
> '--add-opens=java.base/java.util=ALL-UNNAMED'
> '--add-opens=java.base/java.io=ALL-UNNAMED' '-Xmx1536m' '-jar'
> '/__w/1/s/flink-tests/target/surefire/surefirebooter-20240216015747138_560.jar'
> '/__w/1/s/flink-tests/target/surefire' '2024-02-16T01-57-43_286-jvmRun4'
> 'surefire-20240216015747138_558tmp' 'surefire_185-20240216015747138_559tmp'
> Feb 16 02:43:47 02:43:47.142 [ERROR] Error occurred in starting fork, check
> output in log
> Feb 16 02:43:47 02:43:47.142 [ERROR] Process Exit Code: 127
> Feb 16 02:43:47 02:43:47.142 [ERROR] Crashed tests:
> Feb 16 02:43:47 02:43:47.142 [ERROR]
> org.apache.flink.test.checkpointing.ChangelogLocalRecoveryITCase
> Feb 16 02:43:47 02:43:47.142 [ERROR]
> org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
> Feb 16 02:43:47 02:43:47.142 [ERROR] Command was /bin/sh -c cd
> '/__w/1/s/flink-tests' && '/usr/lib/jvm/jdk-11.0.19+7/bin/java'
> '-XX:+UseG1GC' '-Xms256m' '-XX:+IgnoreUnrecognizedVMOptions'
> '--add-opens=java.base/java.util=ALL-UNNAMED'
> '--add-opens=java.base/java.io=ALL-UNNAMED' '-Xmx1536m' '-jar'
> '/__w/1/s/flink-tests/target/surefire/surefirebooter-20240216015747138_560.jar'
> '/__w/1/s/flink-tests/target/surefire' '2024-02-16T01-57-43_286-jvmRun4'
> 'surefire-20240216015747138_558tmp' 'surefire_185-20240216015747138_559tmp'
> Feb 16 02:43:47 02:43:47.142 [ERROR] Error occurred in starting fork, check
> output in log
> Feb 16 02:43:47 02:43:47.142 [ERROR] Process Exit Code: 127
> Feb 16 02:43:47 02:43:47.142 [ERROR] Crashed tests:
> Feb 16 02:43:47 02:43:47.142 [ERROR]
> org.apache.flink.test.checkpointing.ChangelogLocalRecoveryITCase
> Feb 16 02:43:47 02:43:47.142 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:456)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-34543) Support Full Partition Processing On Non-keyed DataStream
Wencong Liu created FLINK-34543: --- Summary: Support Full Partition Processing On Non-keyed DataStream Key: FLINK-34543 URL: https://issues.apache.org/jira/browse/FLINK-34543 Project: Flink Issue Type: Improvement Components: API / DataStream Affects Versions: 1.20.0 Reporter: Wencong Liu Fix For: 1.20.0 1. Introduce MapParititon, SortPartition, Aggregate, Reduce API in DataStream. 2. Introduce SortPartition API in KeyedStream. The related FLIP can be found in [FLIP-380|https://cwiki.apache.org/confluence/display/FLINK/FLIP-380%3A+Support+Full+Partition+Processing+On+Non-keyed+DataStream]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-34543) Support Full Partition Processing On Non-keyed DataStream
[ https://issues.apache.org/jira/browse/FLINK-34543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wencong Liu updated FLINK-34543: Description: 1. Introduce MapParititon, SortPartition, Aggregate, Reduce API in DataStream. 2. Introduce SortPartition API in KeyedStream. The related motivation and design can be found in [FLIP-380|https://cwiki.apache.org/confluence/display/FLINK/FLIP-380%3A+Support+Full+Partition+Processing+On+Non-keyed+DataStream]. was: 1. Introduce MapParititon, SortPartition, Aggregate, Reduce API in DataStream. 2. Introduce SortPartition API in KeyedStream. The related FLIP can be found in [FLIP-380|https://cwiki.apache.org/confluence/display/FLINK/FLIP-380%3A+Support+Full+Partition+Processing+On+Non-keyed+DataStream]. > Support Full Partition Processing On Non-keyed DataStream > - > > Key: FLINK-34543 > URL: https://issues.apache.org/jira/browse/FLINK-34543 > Project: Flink > Issue Type: Improvement > Components: API / DataStream >Affects Versions: 1.20.0 >Reporter: Wencong Liu >Priority: Major > Labels: pull-request-available > Fix For: 1.20.0 > > > 1. Introduce MapParititon, SortPartition, Aggregate, Reduce API in DataStream. > 2. Introduce SortPartition API in KeyedStream. > The related motivation and design can be found in > [FLIP-380|https://cwiki.apache.org/confluence/display/FLINK/FLIP-380%3A+Support+Full+Partition+Processing+On+Non-keyed+DataStream]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30219) Fetch results api in sql gateway return error result.
[ https://issues.apache.org/jira/browse/FLINK-30219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17645913#comment-17645913 ] Wencong Liu commented on FLINK-30219: - Hi, [~Aiden Gong] , could you please illustrate this error in detail? Is the returned format not as expected? > Fetch results api in sql gateway return error result. > - > > Key: FLINK-30219 > URL: https://issues.apache.org/jira/browse/FLINK-30219 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Gateway >Affects Versions: 1.16.0 >Reporter: Aiden Gong >Assignee: Aiden Gong >Priority: Critical > Labels: pull-request-available > Fix For: 1.16.1 > > Attachments: image-2022-11-26-10-38-02-270.png > > > !image-2022-11-26-10-38-02-270.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32502) Remove AbstractLeaderElectionService
[
https://issues.apache.org/jira/browse/FLINK-32502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17740367#comment-17740367
]
Wencong Liu commented on FLINK-32502:
-
Hello [~mapohl] Are you suggesting merging the methods of
AbstractLeaderElectionService to the LeaderElectionService interface? I would
like to address this issue. 😄
> Remove AbstractLeaderElectionService
>
>
> Key: FLINK-32502
> URL: https://issues.apache.org/jira/browse/FLINK-32502
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.18.0
>Reporter: Matthias Pohl
>Priority: Major
>
> {{AbstractLeaderElectionService}} doesn't bring much value anymore and can be
> removed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32523) NotifyCheckpointAbortedITCase.testNotifyCheckpointAborted fails with timeout on AZP
[
https://issues.apache.org/jira/browse/FLINK-32523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17741813#comment-17741813
]
Wencong Liu commented on FLINK-32523:
-
I think we should remove @Test(timeout = TEST_TIMEOUT) in this test and let CI
judge whether it's timed out. WDYT?
> NotifyCheckpointAbortedITCase.testNotifyCheckpointAborted fails with timeout
> on AZP
> ---
>
> Key: FLINK-32523
> URL: https://issues.apache.org/jira/browse/FLINK-32523
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.18.0
>Reporter: Sergey Nuyanzin
>Priority: Critical
> Labels: test-stability
>
> This build
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=50795&view=logs&j=39d5b1d5-3b41-54dc-6458-1e2ddd1cdcf3&t=0c010d0c-3dec-5bf1-d408-7b18988b1b2b&l=8638
> fails with timeout
> {noformat}
> Jul 03 01:26:35 org.junit.runners.model.TestTimedOutException: test timed out
> after 10 milliseconds
> Jul 03 01:26:35 at java.lang.Object.wait(Native Method)
> Jul 03 01:26:35 at java.lang.Object.wait(Object.java:502)
> Jul 03 01:26:35 at
> org.apache.flink.core.testutils.OneShotLatch.await(OneShotLatch.java:61)
> Jul 03 01:26:35 at
> org.apache.flink.test.checkpointing.NotifyCheckpointAbortedITCase.verifyAllOperatorsNotifyAborted(NotifyCheckpointAbortedITCase.java:198)
> Jul 03 01:26:35 at
> org.apache.flink.test.checkpointing.NotifyCheckpointAbortedITCase.testNotifyCheckpointAborted(NotifyCheckpointAbortedITCase.java:189)
> Jul 03 01:26:35 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Jul 03 01:26:35 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Jul 03 01:26:35 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Jul 03 01:26:35 at java.lang.reflect.Method.invoke(Method.java:498)
> Jul 03 01:26:35 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> Jul 03 01:26:35 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> Jul 03 01:26:35 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> Jul 03 01:26:35 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> Jul 03 01:26:35 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> Jul 03 01:26:35 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> Jul 03 01:26:35 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> Jul 03 01:26:35 at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32624) TieredStorageConsumerClientTest.testGetNextBufferFromRemoteTier failed on CI
[
https://issues.apache.org/jira/browse/FLINK-32624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744403#comment-17744403
]
Wencong Liu commented on FLINK-32624:
-
Sorry for the late reply, I'll take a look. [~lincoln.86xy]
> TieredStorageConsumerClientTest.testGetNextBufferFromRemoteTier failed on CI
>
>
> Key: FLINK-32624
> URL: https://issues.apache.org/jira/browse/FLINK-32624
> Project: Flink
> Issue Type: Bug
> Components: API / Core
>Affects Versions: 1.18.0
>Reporter: lincoln lee
>Priority: Major
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=51376&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1&t=24c3384f-1bcb-57b3-224f-51bf973bbee8]
> errors:
> {code}
> Jul 18 11:18:35 11:18:35.412 [ERROR]
> org.apache.flink.runtime.io.network.partition.hybrid.tiered.netty.TieredStorageConsumerClientTest.testGetNextBufferFromRemoteTier
> Time elapsed: 0.014 s <<< FAILURE!
> Jul 18 11:18:35 java.lang.AssertionError:
> Jul 18 11:18:35
> Jul 18 11:18:35 Expecting Optional to contain a value but it was empty.
> Jul 18 11:18:35 at
> org.apache.flink.runtime.io.network.partition.hybrid.tiered.netty.TieredStorageConsumerClientTest.testGetNextBufferFromRemoteTier(TieredStorageConsumerClientTest.java:127)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32624) TieredStorageConsumerClientTest.testGetNextBufferFromRemoteTier failed on CI
[
https://issues.apache.org/jira/browse/FLINK-32624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744409#comment-17744409
]
Wencong Liu commented on FLINK-32624:
-
Thanks [~lincoln.86xy] . I've opened a hot fix. [[hotfix] Fix the
TieredStorageConsumerClientTest by WencongLiu · Pull Request #23017 ·
apache/flink (github.com)|https://github.com/apache/flink/pull/23017]
> TieredStorageConsumerClientTest.testGetNextBufferFromRemoteTier failed on CI
>
>
> Key: FLINK-32624
> URL: https://issues.apache.org/jira/browse/FLINK-32624
> Project: Flink
> Issue Type: Bug
> Components: API / Core
>Affects Versions: 1.18.0
>Reporter: lincoln lee
>Priority: Major
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=51376&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1&t=24c3384f-1bcb-57b3-224f-51bf973bbee8]
> errors:
> {code}
> Jul 18 11:18:35 11:18:35.412 [ERROR]
> org.apache.flink.runtime.io.network.partition.hybrid.tiered.netty.TieredStorageConsumerClientTest.testGetNextBufferFromRemoteTier
> Time elapsed: 0.014 s <<< FAILURE!
> Jul 18 11:18:35 java.lang.AssertionError:
> Jul 18 11:18:35
> Jul 18 11:18:35 Expecting Optional to contain a value but it was empty.
> Jul 18 11:18:35 at
> org.apache.flink.runtime.io.network.partition.hybrid.tiered.netty.TieredStorageConsumerClientTest.testGetNextBufferFromRemoteTier(TieredStorageConsumerClientTest.java:127)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-5336) Make Path immutable
[
https://issues.apache.org/jira/browse/FLINK-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744452#comment-17744452
]
Wencong Liu commented on FLINK-5336:
Hi all, I have checked all the classes that utilize the *Path* class. I found
that there're still some classes are de/serialize the *Path* through
*IOReadableWritable* interface.
# {*}FileSourceSplitSerializer{*}: It de/serializes the *Path* during the
process of de/serializing FileSourceSplit.
# {*}TestManagedSinkCommittableSerializer{*}: It de/serializes the Path during
the process of de/serializing TestManagedCommittable.
# {*}TestManagedFileSourceSplitSerializer{*}: It de/serializes the Path during
the process of de/serializing TestManagedIterableSourceSplit.
For 1, the Path needs to be serialized to save checkpoint data in source.
[~sewen]
For 2/3, the IT case in flink-table-common depends on the Path
de/serialization. [~qingyue]
In summary, I think the Path class should still need to implement the
*IOReadableWritable* interface to support de/serialization. WDYT?
> Make Path immutable
> ---
>
> Key: FLINK-5336
> URL: https://issues.apache.org/jira/browse/FLINK-5336
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataSet
>Reporter: Stephan Ewen
>Priority: Major
> Fix For: 2.0.0
>
>
> The {{Path}} class is currently mutable to support the {{IOReadableWritable}}
> serialization. Since that serialization is not used any more, I suggest to
> drop that interface from Path and make the Path's URI final.
> Being immutable, we can store configures paths properly without the chance of
> them being mutated as side effects.
> Many parts of the code make the assumption that the Path is immutable, being
> susceptible to subtle errors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-5336) Make Path immutable
[
https://issues.apache.org/jira/browse/FLINK-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744452#comment-17744452
]
Wencong Liu edited comment on FLINK-5336 at 7/20/23 2:32 AM:
-
Hi all, I have checked all the classes that utilize the *Path* class. I found
that there're still some classes are de/serializing the *Path* through
*IOReadableWritable* interface.
# {*}FileSourceSplitSerializer{*}: It de/serializes the *Path* during the
process of de/serializing FileSourceSplit.
# {*}TestManagedSinkCommittableSerializer{*}: It de/serializes the Path during
the process of de/serializing TestManagedCommittable.
# {*}TestManagedFileSourceSplitSerializer{*}: It de/serializes the Path during
the process of de/serializing TestManagedIterableSourceSplit.
For 1, the Path needs to be serialized to save checkpoint data in source.
For 2/3, the IT case in flink-table-common depends on the Path
de/serialization. [~qingyue]
In summary, I think the Path class should still need to implement the
*IOReadableWritable* interface to support de/serialization. WDYT?
was (Author: JIRAUSER281639):
Hi all, I have checked all the classes that utilize the *Path* class. I found
that there're still some classes are de/serialize the *Path* through
*IOReadableWritable* interface.
# {*}FileSourceSplitSerializer{*}: It de/serializes the *Path* during the
process of de/serializing FileSourceSplit.
# {*}TestManagedSinkCommittableSerializer{*}: It de/serializes the Path during
the process of de/serializing TestManagedCommittable.
# {*}TestManagedFileSourceSplitSerializer{*}: It de/serializes the Path during
the process of de/serializing TestManagedIterableSourceSplit.
For 1, the Path needs to be serialized to save checkpoint data in source.
[~sewen]
For 2/3, the IT case in flink-table-common depends on the Path
de/serialization. [~qingyue]
In summary, I think the Path class should still need to implement the
*IOReadableWritable* interface to support de/serialization. WDYT?
> Make Path immutable
> ---
>
> Key: FLINK-5336
> URL: https://issues.apache.org/jira/browse/FLINK-5336
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataSet
>Reporter: Stephan Ewen
>Priority: Major
> Fix For: 2.0.0
>
>
> The {{Path}} class is currently mutable to support the {{IOReadableWritable}}
> serialization. Since that serialization is not used any more, I suggest to
> drop that interface from Path and make the Path's URI final.
> Being immutable, we can store configures paths properly without the chance of
> them being mutated as side effects.
> Many parts of the code make the assumption that the Path is immutable, being
> susceptible to subtle errors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-5336) Make Path immutable
[
https://issues.apache.org/jira/browse/FLINK-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744452#comment-17744452
]
Wencong Liu edited comment on FLINK-5336 at 7/20/23 4:13 AM:
-
Hi all, I have checked all the classes that utilize the *Path* class. I found
that there're still some classes are de/serializing the *Path* through
*IOReadableWritable* interface.
# {*}FileSourceSplitSerializer{*}: It de/serializes the *Path* during the
process of de/serializing FileSourceSplit.
# {*}TestManagedSinkCommittableSerializer{*}: It de/serializes the Path during
the process of de/serializing TestManagedCommittable.
# {*}TestManagedFileSourceSplitSerializer{*}: It de/serializes the Path during
the process of de/serializing TestManagedIterableSourceSplit.
For 1, the Path needs to be serialized to save checkpoint data in source.
For 2/3, the IT case in flink-table-common depends on the Path
de/serialization. [~qingyue]
was (Author: JIRAUSER281639):
Hi all, I have checked all the classes that utilize the *Path* class. I found
that there're still some classes are de/serializing the *Path* through
*IOReadableWritable* interface.
# {*}FileSourceSplitSerializer{*}: It de/serializes the *Path* during the
process of de/serializing FileSourceSplit.
# {*}TestManagedSinkCommittableSerializer{*}: It de/serializes the Path during
the process of de/serializing TestManagedCommittable.
# {*}TestManagedFileSourceSplitSerializer{*}: It de/serializes the Path during
the process of de/serializing TestManagedIterableSourceSplit.
For 1, the Path needs to be serialized to save checkpoint data in source.
For 2/3, the IT case in flink-table-common depends on the Path
de/serialization. [~qingyue]
In summary, I think the Path class should still need to implement the
*IOReadableWritable* interface to support de/serialization. WDYT?
> Make Path immutable
> ---
>
> Key: FLINK-5336
> URL: https://issues.apache.org/jira/browse/FLINK-5336
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataSet
>Reporter: Stephan Ewen
>Priority: Major
> Fix For: 2.0.0
>
>
> The {{Path}} class is currently mutable to support the {{IOReadableWritable}}
> serialization. Since that serialization is not used any more, I suggest to
> drop that interface from Path and make the Path's URI final.
> Being immutable, we can store configures paths properly without the chance of
> them being mutated as side effects.
> Many parts of the code make the assumption that the Path is immutable, being
> susceptible to subtle errors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-5336) Make Path immutable
[
https://issues.apache.org/jira/browse/FLINK-5336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17744938#comment-17744938
]
Wencong Liu commented on FLINK-5336:
Thanks [~qingyue] , I think the *FileSourceSplitSerializer* could be modified
to no longer implement the *IOReadableWritable* interface to support
serialization/deserialization on DataInputView/DataOutputView. I'll propose a
FLIP about the specific actions at a later time.
> Make Path immutable
> ---
>
> Key: FLINK-5336
> URL: https://issues.apache.org/jira/browse/FLINK-5336
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataSet
>Reporter: Stephan Ewen
>Priority: Major
> Fix For: 2.0.0
>
>
> The {{Path}} class is currently mutable to support the {{IOReadableWritable}}
> serialization. Since that serialization is not used any more, I suggest to
> drop that interface from Path and make the Path's URI final.
> Being immutable, we can store configures paths properly without the chance of
> them being mutated as side effects.
> Many parts of the code make the assumption that the Path is immutable, being
> susceptible to subtle errors.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-32686) Performance regression on startScheduling.BATCH and startScheduling.STREAMING since 2023-07-24
[ https://issues.apache.org/jira/browse/FLINK-32686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17747733#comment-17747733 ] Wencong Liu commented on FLINK-32686: - Thanks for reporting this. The second regression since 07.24 is caused by commit id: c9e1833642650e0b1ea162371dd7c6d35f2e21b7. The commit disables the repair in [FLINK-32094] startScheduling.BATCH performance regression since May 11th - ASF JIRA (apache.org) unexpectedly and causes the regression in 07.24. I'll open a pull request to fix this. The first regression in 07.09 is not related with FLINK-32094. > Performance regression on startScheduling.BATCH and startScheduling.STREAMING > since 2023-07-24 > --- > > Key: FLINK-32686 > URL: https://issues.apache.org/jira/browse/FLINK-32686 > Project: Flink > Issue Type: Bug >Affects Versions: 1.18.0 >Reporter: Martijn Visser >Priority: Blocker > > http://codespeed.dak8s.net:8000/timeline/#/?exe=5&ben=startScheduling.STREAMING&extr=on&quarts=on&equid=off&env=2&revs=200 > http://codespeed.dak8s.net:8000/timeline/#/?exe=5&ben=startScheduling.BATCH&extr=on&quarts=on&equid=off&env=2&revs=200 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-32686) Performance regression on startScheduling.BATCH and startScheduling.STREAMING since 2023-07-24
[ https://issues.apache.org/jira/browse/FLINK-32686?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17747733#comment-17747733 ] Wencong Liu edited comment on FLINK-32686 at 7/27/23 3:42 AM: -- Thanks for reporting this. The second regression since 07.24 is caused by commit id: c9e1833642650e0b1ea162371dd7c6d35f2e21b7. The commit disables the repair in FLINK-32094 unexpectedly and causes the regression in 07.24. I'll open a pull request to fix this. The first regression in 07.09 is not related with FLINK-32094. was (Author: JIRAUSER281639): Thanks for reporting this. The second regression since 07.24 is caused by commit id: c9e1833642650e0b1ea162371dd7c6d35f2e21b7. The commit disables the repair in [FLINK-32094] startScheduling.BATCH performance regression since May 11th - ASF JIRA (apache.org) unexpectedly and causes the regression in 07.24. I'll open a pull request to fix this. The first regression in 07.09 is not related with FLINK-32094. > Performance regression on startScheduling.BATCH and startScheduling.STREAMING > since 2023-07-24 > --- > > Key: FLINK-32686 > URL: https://issues.apache.org/jira/browse/FLINK-32686 > Project: Flink > Issue Type: Bug >Affects Versions: 1.18.0 >Reporter: Martijn Visser >Priority: Blocker > Labels: pull-request-available > > http://codespeed.dak8s.net:8000/timeline/#/?exe=5&ben=startScheduling.STREAMING&extr=on&quarts=on&equid=off&env=2&revs=200 > http://codespeed.dak8s.net:8000/timeline/#/?exe=5&ben=startScheduling.BATCH&extr=on&quarts=on&equid=off&env=2&revs=200 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-32708) Fix the write logic in remote tier of hybrid shuffle
Wencong Liu created FLINK-32708: --- Summary: Fix the write logic in remote tier of hybrid shuffle Key: FLINK-32708 URL: https://issues.apache.org/jira/browse/FLINK-32708 Project: Flink Issue Type: Bug Components: Runtime / Network Affects Versions: 1.18.0 Reporter: Wencong Liu Fix For: 1.18.0 Currently, on the writer side in the remote tier, the flag file indicating the latest segment id is updated first, followed by the creation of the data file. This results in an incorrect order of file creation and we should fix it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32708) Fix the write logic in remote tier of Hybrid Shuffle
[ https://issues.apache.org/jira/browse/FLINK-32708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wencong Liu updated FLINK-32708: Summary: Fix the write logic in remote tier of Hybrid Shuffle (was: Fix the write logic in remote tier of hybrid shuffle) > Fix the write logic in remote tier of Hybrid Shuffle > > > Key: FLINK-32708 > URL: https://issues.apache.org/jira/browse/FLINK-32708 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.18.0 >Reporter: Wencong Liu >Priority: Major > Labels: pull-request-available > Fix For: 1.18.0 > > > Currently, on the writer side in the remote tier, the flag file indicating > the latest segment id is updated first, followed by the creation of the data > file. This results in an incorrect order of file creation and we should fix > it. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30343) Migrate KubernetesLeaderElectionAndRetrievalITCase
[ https://issues.apache.org/jira/browse/FLINK-30343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17652461#comment-17652461 ] Wencong Liu commented on FLINK-30343: - Hi [~mapohl] , could you please illustrate this issue in detail ? > Migrate KubernetesLeaderElectionAndRetrievalITCase > -- > > Key: FLINK-30343 > URL: https://issues.apache.org/jira/browse/FLINK-30343 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Affects Versions: 1.16.0, 1.17.0 >Reporter: Matthias Pohl >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30530) Flink configuration from user-provided ConfigMap
[
https://issues.apache.org/jira/browse/FLINK-30530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17652743#comment-17652743
]
Wencong Liu commented on FLINK-30530:
-
Hi [~tashoyan] , I'm quite interested in this issue. In which Java file should
I find the problem you describe?
> Flink configuration from user-provided ConfigMap
>
>
> Key: FLINK-30530
> URL: https://issues.apache.org/jira/browse/FLINK-30530
> Project: Flink
> Issue Type: Improvement
> Components: Kubernetes Operator
>Affects Versions: 1.15.2
> Environment: Flink 1.15.2
> Flink Kubernetes operator 1.2.0
>Reporter: Arseniy Tashoyan
>Priority: Major
>
> Currently the Flink configuration can be specified in the YAML descriptor of
> FlinkDeployment via the _flinkConfiguration_ setting:
> {code:yaml}
> flinkConfiguration:
> taskmanager.numberOfTaskSlots: "2"
> ...
> {code}
> Same for the logging configuration:
> {code:yaml}
> logConfiguration:
> "log4j-console.properties": |
> rootLogger.level = DEBUG
> ...{code}
> This makes the YAML descriptor overloaded and huge. In addition, Flink and
> logging configuration may differ for different applications, while the
> Kubernetes settings maybe same for all applications. Therefore it makes sense
> to extract Flink and logging configurations from the YAML descriptor.
> This can be done via a user-provided ConfigMap:
> {code:yaml}
> flinkConfigMap: basic-example-flink-config
> {code}
> In this example we have a Flink application {_}basic-example{_}. The
> _basic-example-flink-config_ ConfigMap contains all config files used by
> Flink: flink-conf.yaml, log4j-console.properties, possibly other files. The
> content of this ConfigMap gets mounted as a volume to {_}/opt/flink/conf{_}.
> Therefore we can have different Flink settings for different applications and
> the same YAML descriptor for all of them (only the value for _flinkConfigMap_
> differs).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30507) EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMinMaxParallelism could not be initialized
[ https://issues.apache.org/jira/browse/FLINK-30507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wencong Liu updated FLINK-30507: Attachment: image-2022-12-29-15-48-55-564.png > EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMinMaxParallelism > could not be initialized > -- > > Key: FLINK-30507 > URL: https://issues.apache.org/jira/browse/FLINK-30507 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.17.0 >Reporter: Qingsheng Ren >Priority: Major > Attachments: image-2022-12-29-15-48-55-564.png > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44245&view=logs&j=4d4a0d10-fca2-5507-8eed-c07f0bdf4887&t=7b25afdf-cc6c-566f-5459-359dc2585798&l=12367 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30507) EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMinMaxParallelism could not be initialized
[ https://issues.apache.org/jira/browse/FLINK-30507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wencong Liu updated FLINK-30507: Attachment: image-2022-12-29-15-51-12-230.png > EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMinMaxParallelism > could not be initialized > -- > > Key: FLINK-30507 > URL: https://issues.apache.org/jira/browse/FLINK-30507 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.17.0 >Reporter: Qingsheng Ren >Priority: Major > Attachments: image-2022-12-29-15-48-55-564.png, > image-2022-12-29-15-51-12-230.png > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44245&view=logs&j=4d4a0d10-fca2-5507-8eed-c07f0bdf4887&t=7b25afdf-cc6c-566f-5459-359dc2585798&l=12367 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30507) EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMinMaxParallelism could not be initialized
[ https://issues.apache.org/jira/browse/FLINK-30507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wencong Liu updated FLINK-30507: Attachment: image-2022-12-29-15-52-17-508.png > EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMinMaxParallelism > could not be initialized > -- > > Key: FLINK-30507 > URL: https://issues.apache.org/jira/browse/FLINK-30507 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.17.0 >Reporter: Qingsheng Ren >Priority: Major > Attachments: image-2022-12-29-15-48-55-564.png, > image-2022-12-29-15-51-12-230.png, image-2022-12-29-15-52-17-508.png > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44245&view=logs&j=4d4a0d10-fca2-5507-8eed-c07f0bdf4887&t=7b25afdf-cc6c-566f-5459-359dc2585798&l=12367 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30507) EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMinMaxParallelism could not be initialized
[ https://issues.apache.org/jira/browse/FLINK-30507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17652754#comment-17652754 ] Wencong Liu commented on FLINK-30507: - Hello [~renqs]. For this failed IT case, I found that the main reason is ZookeeperServer startup timed out. ZookeeperServer uses the minSessionTimeout value as the actual timeout parameter by default, and the default value of minSessionTimeout is 6s. Currently, only maxSessionTimeout = 60s is configured in the startup configuration. Since it's larger than the default value of 6s, this configuration does not take effect. !image-2022-12-29-15-52-17-508.png! !image-2022-12-29-15-48-55-564.png! !image-2022-12-29-15-51-12-230.png! Therefore, minSessionTimeout can be set to avoid the slow startup of ZookeeperServer when resources are tight. I tend to set minSessionTimeout to 12s. Do you think this change is reasonable? > EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMinMaxParallelism > could not be initialized > -- > > Key: FLINK-30507 > URL: https://issues.apache.org/jira/browse/FLINK-30507 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.17.0 >Reporter: Qingsheng Ren >Priority: Major > Attachments: image-2022-12-29-15-48-55-564.png, > image-2022-12-29-15-51-12-230.png, image-2022-12-29-15-52-17-508.png > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44245&view=logs&j=4d4a0d10-fca2-5507-8eed-c07f0bdf4887&t=7b25afdf-cc6c-566f-5459-359dc2585798&l=12367 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30502) Checkpoint state Continued to increase
[ https://issues.apache.org/jira/browse/FLINK-30502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17652755#comment-17652755 ] Wencong Liu commented on FLINK-30502: - Hello [~douheyu] , thanks for your feedback. Could you please take a look? [~Yanfei Lei] > Checkpoint state Continued to increase > -- > > Key: FLINK-30502 > URL: https://issues.apache.org/jira/browse/FLINK-30502 > Project: Flink > Issue Type: Bug > Environment: flink-1.13 >Reporter: heyu dou >Priority: Major > > I am using flink incremental checkpoint and configured state ttl. > But I found a lot of files outside the ttl in the hdfs directory > (checkpoint/shard). > what happened and how can fix it? -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30389) Add retry to read hints
[
https://issues.apache.org/jira/browse/FLINK-30389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17652781#comment-17652781
]
Wencong Liu commented on FLINK-30389:
-
Hello [~lzljs3620320] , I'm quite interested in this issue. Can you provide the
exact location where the exception occurs? Thanks.
> Add retry to read hints
> ---
>
> Key: FLINK-30389
> URL: https://issues.apache.org/jira/browse/FLINK-30389
> Project: Flink
> Issue Type: Improvement
> Components: Table Store
>Reporter: Jingsong Lee
>Priority: Major
> Fix For: table-store-0.3.0
>
>
> For the oss (object store) filesystem. When writing hint file, delete it
> first and then add it. Reading hint file may fail frequently. We don't need
> to return directly in case of failure. We can add a retry.
> {code:java}
> Failed to read hint file LATEST. Falling back to listing files.
> java.io.FileNotFoundException: oss://lake_v4/snapshot/LATEST: No such file or
> directory!
> at
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30257) SqlClientITCase#testMatchRecognize failed
[
https://issues.apache.org/jira/browse/FLINK-30257?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Wencong Liu updated FLINK-30257:
Attachment: image-2022-12-29-21-47-31-606.png
> SqlClientITCase#testMatchRecognize failed
> -
>
> Key: FLINK-30257
> URL: https://issues.apache.org/jira/browse/FLINK-30257
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.17.0
>Reporter: Martijn Visser
>Priority: Major
> Labels: test-stability
> Attachments: image-2022-12-29-21-47-31-606.png
>
>
> {code:java}
> Nov 30 21:54:41 [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 224.683 s <<< FAILURE! - in SqlClientITCase
> Nov 30 21:54:41 [ERROR] SqlClientITCase.testMatchRecognize Time elapsed:
> 50.164 s <<< FAILURE!
> Nov 30 21:54:41 org.opentest4j.AssertionFailedError:
> Nov 30 21:54:41
> Nov 30 21:54:41 expected: 1
> Nov 30 21:54:41 but was: 0
> Nov 30 21:54:41 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> Nov 30 21:54:41 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> Nov 30 21:54:41 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> Nov 30 21:54:41 at
> SqlClientITCase.verifyNumberOfResultRecords(SqlClientITCase.java:297)
> Nov 30 21:54:41 at
> SqlClientITCase.testMatchRecognize(SqlClientITCase.java:255)
> Nov 30 21:54:41 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 30 21:54:41 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 30 21:54:41 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 30 21:54:41 at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 30 21:54:41 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMetho
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43635&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=160c9ae5-96fd-516e-1c91-deb81f59292a&l=14817
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30257) SqlClientITCase#testMatchRecognize failed
[
https://issues.apache.org/jira/browse/FLINK-30257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17652852#comment-17652852
]
Wencong Liu commented on FLINK-30257:
-
Hello [~martijnvisser]. In the situation of resource shortage, this sleep delay
may not be sufficient, and the data in the file may not be flushed. May be
adding timeout could solve the problem?
!image-2022-12-29-21-47-31-606.png!
> SqlClientITCase#testMatchRecognize failed
> -
>
> Key: FLINK-30257
> URL: https://issues.apache.org/jira/browse/FLINK-30257
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.17.0
>Reporter: Martijn Visser
>Priority: Major
> Labels: test-stability
> Attachments: image-2022-12-29-21-47-31-606.png
>
>
> {code:java}
> Nov 30 21:54:41 [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 224.683 s <<< FAILURE! - in SqlClientITCase
> Nov 30 21:54:41 [ERROR] SqlClientITCase.testMatchRecognize Time elapsed:
> 50.164 s <<< FAILURE!
> Nov 30 21:54:41 org.opentest4j.AssertionFailedError:
> Nov 30 21:54:41
> Nov 30 21:54:41 expected: 1
> Nov 30 21:54:41 but was: 0
> Nov 30 21:54:41 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> Nov 30 21:54:41 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> Nov 30 21:54:41 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> Nov 30 21:54:41 at
> SqlClientITCase.verifyNumberOfResultRecords(SqlClientITCase.java:297)
> Nov 30 21:54:41 at
> SqlClientITCase.testMatchRecognize(SqlClientITCase.java:255)
> Nov 30 21:54:41 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 30 21:54:41 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 30 21:54:41 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 30 21:54:41 at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 30 21:54:41 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMetho
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43635&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=160c9ae5-96fd-516e-1c91-deb81f59292a&l=14817
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30234) SourceReaderBase should provide an option to disable numRecordsIn metric registration
[ https://issues.apache.org/jira/browse/FLINK-30234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17652854#comment-17652854 ] Wencong Liu commented on FLINK-30234: - Hello [~renqs] , I think this is a reasonable improvement. I'd like to take this ticket, could you assign this to me? > SourceReaderBase should provide an option to disable numRecordsIn metric > registration > - > > Key: FLINK-30234 > URL: https://issues.apache.org/jira/browse/FLINK-30234 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Affects Versions: 1.17.0 >Reporter: Qingsheng Ren >Priority: Major > > Currently the numRecordsIn metric is pre-registered for all sources in > SourceReaderBase. Considering different implementation of source reader, the > definition of "record" might differ from the one we use in SourceReaderBase, > hence numRecordsIn might be inaccurate. > We could introduce an option in SourceReader to disable the registration of > numRecordsIn in SourceReaderBase and let the actual implementation to report > the metric instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30117) Evaluate operator error log whitelist entries added in FLINK-29475
[ https://issues.apache.org/jira/browse/FLINK-30117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17653069#comment-17653069 ] Wencong Liu commented on FLINK-30117: - Hello [~gaborgsomogyi] , I'm quite interested in this issue, but I can't understand it completely. Could you please illustrate it in detail ? > Evaluate operator error log whitelist entries added in FLINK-29475 > -- > > Key: FLINK-30117 > URL: https://issues.apache.org/jira/browse/FLINK-30117 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Affects Versions: 1.17.0 >Reporter: Gabor Somogyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30087) sql-gateway-api is bundled twice in the distribution
[ https://issues.apache.org/jira/browse/FLINK-30087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17653074#comment-17653074 ] Wencong Liu commented on FLINK-30087: - Hello [~chesnay] , it the module flink-table-uber named "flink-table-api-java-uber" ? > sql-gateway-api is bundled twice in the distribution > > > Key: FLINK-30087 > URL: https://issues.apache.org/jira/browse/FLINK-30087 > Project: Flink > Issue Type: Technical Debt > Components: Build System, Table SQL / Gateway >Affects Versions: 1.17.0 >Reporter: Chesnay Schepler >Priority: Major > Fix For: 1.17.0 > > > The sql-gateway-api is bundled both in flink-sql-gateway (put into opt/) and > flink-table-uber (put into lib/). > It probably(?) shouldn't be part of flink-table-uber. > In any case there's no reason to bundle it twice. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30117) Evaluate operator error log whitelist entries added in FLINK-29475
[ https://issues.apache.org/jira/browse/FLINK-30117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17653268#comment-17653268 ] Wencong Liu commented on FLINK-30117: - Thanks [~gaborgsomogyi] . According to the code part, the error message of opening subtasks in this issue have been eliminated by "grep -v". So what's the next step? > Evaluate operator error log whitelist entries added in FLINK-29475 > -- > > Key: FLINK-30117 > URL: https://issues.apache.org/jira/browse/FLINK-30117 > Project: Flink > Issue Type: Improvement > Components: Kubernetes Operator >Affects Versions: 1.17.0 >Reporter: Gabor Somogyi >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30343) Migrate KubernetesLeaderElectionAndRetrievalITCase
[ https://issues.apache.org/jira/browse/FLINK-30343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17653630#comment-17653630 ] Wencong Liu commented on FLINK-30343: - [~mapohl] Thanks for your reply. I think that even though FLINK-26522 is currently in progress, the current changes to FLINK-30338 are still meaningful. After FLINK-30338 is completed, only a few minor changes may be needed. Old (legacy) test cases should not be needed, right? If you think it's OK, could you please assign the remaining subtasks to me? > Migrate KubernetesLeaderElectionAndRetrievalITCase > -- > > Key: FLINK-30343 > URL: https://issues.apache.org/jira/browse/FLINK-30343 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Affects Versions: 1.16.0, 1.17.0 >Reporter: Matthias Pohl >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30389) Add retry to read hints
[
https://issues.apache.org/jira/browse/FLINK-30389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17653812#comment-17653812
]
Wencong Liu commented on FLINK-30389:
-
Thanks [~lzljs3620320] . A fixed number of retries and detection intervals can
be added to readHints. I'd like to take this ticket. Could you please assign to
me?
> Add retry to read hints
> ---
>
> Key: FLINK-30389
> URL: https://issues.apache.org/jira/browse/FLINK-30389
> Project: Flink
> Issue Type: Improvement
> Components: Table Store
>Reporter: Jingsong Lee
>Priority: Major
> Fix For: table-store-0.3.0
>
>
> For the oss (object store) filesystem. When writing hint file, delete it
> first and then add it. Reading hint file may fail frequently. We don't need
> to return directly in case of failure. We can add a retry.
> {code:java}
> Failed to read hint file LATEST. Falling back to listing files.
> java.io.FileNotFoundException: oss://lake_v4/snapshot/LATEST: No such file or
> directory!
> at
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30234) SourceReaderBase should provide an option to disable numRecordsIn metric registration
[ https://issues.apache.org/jira/browse/FLINK-30234?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17653814#comment-17653814 ] Wencong Liu commented on FLINK-30234: - The discussion has been created. cc [~renqs] [[DISCUSS] Allow source readers extending SourceReaderBase to override numRecordsIn report logic-Apache Mail Archives|https://lists.apache.org/thread/pq7vrqc1vjbzj6of4wm5x8x7pkpdchox] > SourceReaderBase should provide an option to disable numRecordsIn metric > registration > - > > Key: FLINK-30234 > URL: https://issues.apache.org/jira/browse/FLINK-30234 > Project: Flink > Issue Type: Improvement > Components: Connectors / Common >Affects Versions: 1.17.0 >Reporter: Qingsheng Ren >Assignee: Wencong Liu >Priority: Major > > Currently the numRecordsIn metric is pre-registered for all sources in > SourceReaderBase. Considering different implementation of source reader, the > definition of "record" might differ from the one we use in SourceReaderBase, > hence numRecordsIn might be inaccurate. > We could introduce an option in SourceReader to disable the registration of > numRecordsIn in SourceReaderBase and let the actual implementation to report > the metric instead. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30343) Migrate KubernetesLeaderElectionAndRetrievalITCase
[ https://issues.apache.org/jira/browse/FLINK-30343?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17653825#comment-17653825 ] Wencong Liu commented on FLINK-30343: - It makes sense. Let's follow the progress of FLINK-26522. > Migrate KubernetesLeaderElectionAndRetrievalITCase > -- > > Key: FLINK-30343 > URL: https://issues.apache.org/jira/browse/FLINK-30343 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Affects Versions: 1.16.0, 1.17.0 >Reporter: Matthias Pohl >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30556) Improve the logic for enumerating splits for Hive source to avoid potential OOM
[ https://issues.apache.org/jira/browse/FLINK-30556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17654349#comment-17654349 ] Wencong Liu commented on FLINK-30556: - Hello [~luoyuxia] , I'm quite interested in this issue. Could you provide the code location where HiveSource enumerate all splits? > Improve the logic for enumerating splits for Hive source to avoid potential > OOM > --- > > Key: FLINK-30556 > URL: https://issues.apache.org/jira/browse/FLINK-30556 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Affects Versions: 1.16.0 >Reporter: luoyuxia >Priority: Major > > Currently, when read hive source in batch mode, it'll first enumerate all > split for the hive table. But when the table is large, the split will be too > many which may well cause OOM. Some commuity users has also reported this > problem. > We need to optimize the logic for enumerating splits for hive table source to > avoid potential OOM. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30556) Improve the logic for enumerating splits for Hive source to avoid potential OOM
[ https://issues.apache.org/jira/browse/FLINK-30556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17654468#comment-17654468 ] Wencong Liu commented on FLINK-30556: - In my opinion, the method of enumerateSplits in interface FileEnumerator could return the type Iterator rather than the type Collection. This allows the FileSourceSplit to be read only when it is used, WDYT? [~luoyuxia] > Improve the logic for enumerating splits for Hive source to avoid potential > OOM > --- > > Key: FLINK-30556 > URL: https://issues.apache.org/jira/browse/FLINK-30556 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Affects Versions: 1.16.0 >Reporter: luoyuxia >Priority: Major > > Currently, when read hive source in batch mode, it'll first enumerate all > split for the hive table. But when the table is large, the split will be too > many which may well cause OOM. Some commuity users has also reported this > problem. > We need to optimize the logic for enumerating splits for hive table source to > avoid potential OOM. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30507) EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMinMaxParallelism could not be initialized
[ https://issues.apache.org/jira/browse/FLINK-30507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17654969#comment-17654969 ] Wencong Liu commented on FLINK-30507: - Hello [~mapohl]. I should haven't seen this root cause. This issue can be closed. Thanks for your information. > EventTimeWindowCheckpointingITCase.testTumblingTimeWindowWithKVStateMinMaxParallelism > could not be initialized > -- > > Key: FLINK-30507 > URL: https://issues.apache.org/jira/browse/FLINK-30507 > Project: Flink > Issue Type: Bug > Components: Tests >Affects Versions: 1.17.0 >Reporter: Qingsheng Ren >Priority: Major > Labels: test-stability > Attachments: image-2022-12-29-15-48-55-564.png, > image-2022-12-29-15-51-12-230.png, image-2022-12-29-15-52-17-508.png > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=44245&view=logs&j=4d4a0d10-fca2-5507-8eed-c07f0bdf4887&t=7b25afdf-cc6c-566f-5459-359dc2585798&l=12367 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30556) Improve the logic for enumerating splits for Hive source to avoid potential OOM
[ https://issues.apache.org/jira/browse/FLINK-30556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17655791#comment-17655791 ] Wencong Liu commented on FLINK-30556: - Thanks [~luoyuxia]. As the ContinuousHiveSplitEnumerator can solve this problem, I think it is enough. But I will try the implementation of iterator. When the POC is completed, I can associate the PR with this issue. More discussion can be proposed later. WDYT? > Improve the logic for enumerating splits for Hive source to avoid potential > OOM > --- > > Key: FLINK-30556 > URL: https://issues.apache.org/jira/browse/FLINK-30556 > Project: Flink > Issue Type: Sub-task > Components: Connectors / Hive >Affects Versions: 1.16.0 >Reporter: luoyuxia >Priority: Major > > Currently, when read hive source in batch mode, it'll first enumerate all > split for the hive table. But when the table is large, the split will be too > many which may well cause OOM. Some commuity users has also reported this > problem. > We need to optimize the logic for enumerating splits for hive table source to > avoid potential OOM. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29125) Placeholder in Apache Flink Web Frontend to display some "tags" to distinguish between frontends of different clusters
[
https://issues.apache.org/jira/browse/FLINK-29125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676486#comment-17676486
]
Wencong Liu commented on FLINK-29125:
-
I think there are several options to represent the cluster identification tag:
*Option1: Use IP & Port of rest server to identify the cluster.*
We can show the tag like this:
{code:java}
Apache Flink Web Dashboard(47.100.234.179:8081){code}
*Option2: Use a ID to identify the cluster:*
Currently, the cluster can be identified by the DispatcherID owned by
Dispatcher. But I think it's not suitable to use DispatcherID directly, there
are two reasons:
# The DispatcherID is generated by leaderSessionID, and leaderSessionID is
randomly generated during leader selection. If the cluster is standalone,
DispatcherID will use a default value and cause misunderstanding. In this case
all clusters will have a same DispatcherID.
# The length of DispatcherID.toString() is too long, like this
"----". It's not pretty to show it on browser
location bar.
Using a randomly generated fixed-length string to identify the cluster maybe a
better choice. We can show the tag like this:
{code:java}
Apache Flink Web Dashboard(ID: yy787u){code}
*Option3: Enable the user to set a custom name to cluster in flink-conf.yaml*
Maybe user can add a configuration item like this:
{code:java}
flink.cluster-id: "cluster 1"{code}
Then we can show the tag like this:
{code:java}
Apache Flink Web Dashboard(ID: cluster 1){code}
If the user doesn't specify this item, the cluster id will not be shown, or we
can fallback to either *Option1* and {*}Option2{*}.
WDYT？ [~dkrovi] [~junhan] [~xtsong]
> Placeholder in Apache Flink Web Frontend to display some "tags" to
> distinguish between frontends of different clusters
> --
>
> Key: FLINK-29125
> URL: https://issues.apache.org/jira/browse/FLINK-29125
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Web Frontend
>Reporter: Durga Krovi
>Priority: Major
> Labels: pull-request-available
>
> When there are several Apache Flink clusters running and the corresponding
> Web Frontend is opened in browser tabs, it would be great if these UIs can be
> distinguished in a visible way. Port number in the browser location bar might
> be useful.
> In our use case, we switch among multiple clusters, connect to only one
> cluster at a time and use the same port for forwarding. In such a case, there
> is no visible cue to identify the cluster of the UI being accessed on the
> browser.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-29125) Placeholder in Apache Flink Web Frontend to display some "tags" to distinguish between frontends of different clusters
[
https://issues.apache.org/jira/browse/FLINK-29125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676486#comment-17676486
]
Wencong Liu edited comment on FLINK-29125 at 1/13/23 6:36 AM:
--
I think there are several options to represent the cluster identification tag:
*Option1: Use IP & Port of rest server to identify the cluster.*
We can show the tag like this:
{code:java}
Apache Flink Web Dashboard(47.100.234.179:8081){code}
*Option2: Use a ID to identify the cluster:*
Currently, the cluster can be identified by the DispatcherID owned by
Dispatcher. But I think it's not suitable to use DispatcherID directly, there
are two reasons:
# The DispatcherID is generated by leaderSessionID, and leaderSessionID is
randomly generated during leader selection. If the cluster is standalone,
DispatcherID will use a default value and cause misunderstanding. In this case
all clusters will have a same DispatcherID.
# The length of DispatcherID.toString() is too long, like this
"----". It's not pretty to show it on browser
location bar.
Using a randomly generated fixed-length string to identify the cluster maybe a
better choice. We can show the tag like this:
{code:java}
Apache Flink Web Dashboard(ID: yy787u){code}
*Option3: Enable the user to set a custom name to cluster in flink-conf.yaml*
Maybe user can add a configuration item like this:
{code:java}
flink.cluster-id: "cluster 1"{code}
Then we can show the tag like this:
{code:java}
Apache Flink Web Dashboard(ID: cluster 1){code}
If the user doesn't specify this item, the cluster id will not be shown, or we
can fallback to either *Option1* and {*}Option2{*}.
WDYT？ [~dkrovi] [~junhan] [~xtsong] [~ziyang]
was (Author: JIRAUSER281639):
I think there are several options to represent the cluster identification tag:
*Option1: Use IP & Port of rest server to identify the cluster.*
We can show the tag like this:
{code:java}
Apache Flink Web Dashboard(47.100.234.179:8081){code}
*Option2: Use a ID to identify the cluster:*
Currently, the cluster can be identified by the DispatcherID owned by
Dispatcher. But I think it's not suitable to use DispatcherID directly, there
are two reasons:
# The DispatcherID is generated by leaderSessionID, and leaderSessionID is
randomly generated during leader selection. If the cluster is standalone,
DispatcherID will use a default value and cause misunderstanding. In this case
all clusters will have a same DispatcherID.
# The length of DispatcherID.toString() is too long, like this
"----". It's not pretty to show it on browser
location bar.
Using a randomly generated fixed-length string to identify the cluster maybe a
better choice. We can show the tag like this:
{code:java}
Apache Flink Web Dashboard(ID: yy787u){code}
*Option3: Enable the user to set a custom name to cluster in flink-conf.yaml*
Maybe user can add a configuration item like this:
{code:java}
flink.cluster-id: "cluster 1"{code}
Then we can show the tag like this:
{code:java}
Apache Flink Web Dashboard(ID: cluster 1){code}
If the user doesn't specify this item, the cluster id will not be shown, or we
can fallback to either *Option1* and {*}Option2{*}.
WDYT？ [~dkrovi] [~junhan] [~xtsong]
> Placeholder in Apache Flink Web Frontend to display some "tags" to
> distinguish between frontends of different clusters
> --
>
> Key: FLINK-29125
> URL: https://issues.apache.org/jira/browse/FLINK-29125
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Web Frontend
>Reporter: Durga Krovi
>Priority: Major
> Labels: pull-request-available
>
> When there are several Apache Flink clusters running and the corresponding
> Web Frontend is opened in browser tabs, it would be great if these UIs can be
> distinguished in a visible way. Port number in the browser location bar might
> be useful.
> In our use case, we switch among multiple clusters, connect to only one
> cluster at a time and use the same port for forwarding. In such a case, there
> is no visible cue to identify the cluster of the UI being accessed on the
> browser.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30257) SqlClientITCase#testMatchRecognize failed
[
https://issues.apache.org/jira/browse/FLINK-30257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17676563#comment-17676563
]
Wencong Liu commented on FLINK-30257:
-
Thanks [~martijnvisser]. I think we can refactor this test by depending on the
global timeout in Azure. WDYT?
> SqlClientITCase#testMatchRecognize failed
> -
>
> Key: FLINK-30257
> URL: https://issues.apache.org/jira/browse/FLINK-30257
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.17.0
>Reporter: Martijn Visser
>Priority: Major
> Labels: test-stability
> Attachments: image-2022-12-29-21-47-31-606.png
>
>
> {code:java}
> Nov 30 21:54:41 [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 224.683 s <<< FAILURE! - in SqlClientITCase
> Nov 30 21:54:41 [ERROR] SqlClientITCase.testMatchRecognize Time elapsed:
> 50.164 s <<< FAILURE!
> Nov 30 21:54:41 org.opentest4j.AssertionFailedError:
> Nov 30 21:54:41
> Nov 30 21:54:41 expected: 1
> Nov 30 21:54:41 but was: 0
> Nov 30 21:54:41 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> Nov 30 21:54:41 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> Nov 30 21:54:41 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> Nov 30 21:54:41 at
> SqlClientITCase.verifyNumberOfResultRecords(SqlClientITCase.java:297)
> Nov 30 21:54:41 at
> SqlClientITCase.testMatchRecognize(SqlClientITCase.java:255)
> Nov 30 21:54:41 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 30 21:54:41 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 30 21:54:41 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 30 21:54:41 at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 30 21:54:41 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMetho
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43635&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=160c9ae5-96fd-516e-1c91-deb81f59292a&l=14817
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30257) SqlClientITCase#testMatchRecognize failed
[
https://issues.apache.org/jira/browse/FLINK-30257?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17677587#comment-17677587
]
Wencong Liu commented on FLINK-30257:
-
Thanks [~martijnvisser], I've opened a pull request.
> SqlClientITCase#testMatchRecognize failed
> -
>
> Key: FLINK-30257
> URL: https://issues.apache.org/jira/browse/FLINK-30257
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.17.0
>Reporter: Martijn Visser
>Assignee: Wencong Liu
>Priority: Major
> Labels: pull-request-available, test-stability
> Attachments: image-2022-12-29-21-47-31-606.png
>
>
> {code:java}
> Nov 30 21:54:41 [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0,
> Time elapsed: 224.683 s <<< FAILURE! - in SqlClientITCase
> Nov 30 21:54:41 [ERROR] SqlClientITCase.testMatchRecognize Time elapsed:
> 50.164 s <<< FAILURE!
> Nov 30 21:54:41 org.opentest4j.AssertionFailedError:
> Nov 30 21:54:41
> Nov 30 21:54:41 expected: 1
> Nov 30 21:54:41 but was: 0
> Nov 30 21:54:41 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> Nov 30 21:54:41 at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> Nov 30 21:54:41 at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> Nov 30 21:54:41 at
> SqlClientITCase.verifyNumberOfResultRecords(SqlClientITCase.java:297)
> Nov 30 21:54:41 at
> SqlClientITCase.testMatchRecognize(SqlClientITCase.java:255)
> Nov 30 21:54:41 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> Nov 30 21:54:41 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> Nov 30 21:54:41 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> Nov 30 21:54:41 at java.lang.reflect.Method.invoke(Method.java:498)
> Nov 30 21:54:41 at
> org.junit.platform.commons.util.ReflectionUtils.invokeMetho
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43635&view=logs&j=af184cdd-c6d8-5084-0b69-7e9c67b35f7a&t=160c9ae5-96fd-516e-1c91-deb81f59292a&l=14817
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29125) Placeholder in Apache Flink Web Frontend to display some "tags" to distinguish between frontends of different clusters
[ https://issues.apache.org/jira/browse/FLINK-29125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17678074#comment-17678074 ] Wencong Liu commented on FLINK-29125: - Thanks for the reply [~dkrovi] [~martijnvisser] . I also think it needs a discuss and I'll put these options on the proposal. WDYT? cc [~xtsong] [~junhan] > Placeholder in Apache Flink Web Frontend to display some "tags" to > distinguish between frontends of different clusters > -- > > Key: FLINK-29125 > URL: https://issues.apache.org/jira/browse/FLINK-29125 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Reporter: Durga Krovi >Priority: Major > Labels: pull-request-available > > When there are several Apache Flink clusters running and the corresponding > Web Frontend is opened in browser tabs, it would be great if these UIs can be > distinguished in a visible way. Port number in the browser location bar might > be useful. > In our use case, we switch among multiple clusters, connect to only one > cluster at a time and use the same port for forwarding. In such a case, there > is no visible cue to identify the cluster of the UI being accessed on the > browser. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30699) Improve the efficiency of the getRandomString method in the StringUtils class
[ https://issues.apache.org/jira/browse/FLINK-30699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17678078#comment-17678078 ] Wencong Liu commented on FLINK-30699: - Thanks for your proposal! [~TaoZex] . I'm a little confused about the aim of this issue. Do you mean your change will have a better efficiency? It will be more convenient that you design some tests and put the results on this issue. > Improve the efficiency of the getRandomString method in the StringUtils class > - > > Key: FLINK-30699 > URL: https://issues.apache.org/jira/browse/FLINK-30699 > Project: Flink > Issue Type: Improvement > Components: API / Core >Reporter: Bingye Chen >Priority: Minor > Labels: pull-request-available > Attachments: image-2023-01-16-18-13-56-912.png, > image-2023-01-16-18-14-12-939.png > > > This is a util class method that uses data.length to affect efficiency. > !image-2023-01-16-18-13-56-912.png|width=398,height=148! > !image-2023-01-16-18-14-12-939.png|width=398,height=114! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-30699) Improve the efficiency of the getRandomString method in the StringUtils class
[ https://issues.apache.org/jira/browse/FLINK-30699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17678078#comment-17678078 ] Wencong Liu edited comment on FLINK-30699 at 1/18/23 6:14 AM: -- Thanks for your proposal! [~TaoZex] . I'm a little confused about the aim of this issue. Do you mean your change will have a better efficiency? It will be more convincing that you design some tests and put the results on this issue. was (Author: JIRAUSER281639): Thanks for your proposal! [~TaoZex] . I'm a little confused about the aim of this issue. Do you mean your change will have a better efficiency? It will be more convenient that you design some tests and put the results on this issue. > Improve the efficiency of the getRandomString method in the StringUtils class > - > > Key: FLINK-30699 > URL: https://issues.apache.org/jira/browse/FLINK-30699 > Project: Flink > Issue Type: Improvement > Components: API / Core >Reporter: Bingye Chen >Priority: Minor > Labels: pull-request-available > Attachments: image-2023-01-16-18-13-56-912.png, > image-2023-01-16-18-14-12-939.png > > > This is a util class method that uses data.length to affect efficiency. > !image-2023-01-16-18-13-56-912.png|width=398,height=148! > !image-2023-01-16-18-14-12-939.png|width=398,height=114! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30774) flink-utils module
[
https://issues.apache.org/jira/browse/FLINK-30774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17680085#comment-17680085
]
Wencong Liu commented on FLINK-30774:
-
Hello [~mapohl]. I think your proposal is reasonable! But I have a question.
Currently many utility classes exists in the path
"flink-core/src/main/java/org/apache/flink/util". Some of them are only used in
the module flink-core, others are used in the modules depending on flink-core.
How do we sift the classes moved to the flink-utils module? Or we just simply
move all of them to the module flink-util.
> flink-utils module
> --
>
> Key: FLINK-30774
> URL: https://issues.apache.org/jira/browse/FLINK-30774
> Project: Flink
> Issue Type: Improvement
> Components: Build System
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Priority: Major
> Labels: starter
>
> Currently, utility methods generic utility classes like {{Preconditions}} or
> {{AbstractAutoCloseableRegistry}} are collected in {{flink-core}}. The flaw
> of this approach is that we cannot use those classes in modules like
> {{fink-migration-test-utils}}, {{flink-test-utils-junit}},
> {{flink-metrics-core}} or {{flink-annotations}}.
> We might want to have a generic {{flink-utils}} analogously to
> {{flink-test-utils}} that collects Flink-independent utility functionality
> that can be access by any module {{flink-core}} is depending on to make this
> utility functionality available in any Flink-related module.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30739) SqlGatewayRestEndpointStatementITCase failed with NullPointer
[
https://issues.apache.org/jira/browse/FLINK-30739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17680088#comment-17680088
]
Wencong Liu commented on FLINK-30739:
-
I seems like a existed similar issue. cc [~fsk119]
> SqlGatewayRestEndpointStatementITCase failed with NullPointer
> -
>
> Key: FLINK-30739
> URL: https://issues.apache.org/jira/browse/FLINK-30739
> Project: Flink
> Issue Type: Bug
> Components: Tests
>Reporter: Yun Tang
>Priority: Major
>
> Instance
> https://myasuka.visualstudio.com/flink/_build/results?buildId=437&view=logs&j=43a593e7-535d-554b-08cc-244368da36b4&t=82d122c0-8bbf-56f3-4c0d-8e3d69630d0f
> {code:java}
> Jan 18 10:54:20 [ERROR]
> org.apache.flink.table.gateway.service.SqlGatewayServiceStatementITCase.testFlinkSqlStatements
> Time elapsed: 1.37 s <<< FAILURE!
> Jan 18 10:54:20 org.opentest4j.AssertionFailedError:
> Jan 18 10:54:20
> Jan 18 10:54:20 expected:
> Jan 18 10:54:20 "# table.q - CREATE/DROP/SHOW/ALTER/DESCRIBE TABLE
> Jan 18 10:54:20 #
> Jan 18 10:54:20 # Licensed to the Apache Software Foundation (ASF) under
> one or more
> Jan 18 10:54:20 # contributor license agreements. See the NOTICE file
> distributed with
> Jan 18 10:54:20 # this work for additional information regarding copyright
> ownership.
> Jan 18 10:54:20 # The ASF licenses this file to you under the Apache
> License, Version 2.0
> Jan 18 10:54:20 # (the "License"); you may not use this file except in
> compliance with
> Jan 18 10:54:20 # the License. You may obtain a copy of the License at
> Jan 18 10:54:20 #
> Jan 18 10:54:20 # http://www.apache.org/licenses/LICENSE-2.0
> Jan 18 10:54:20 #
> Jan 18 10:54:20 # Unless required by applicable law or agreed to in
> writing, software
> Jan 18 10:54:20 # distributed under the License is distributed on an "AS
> IS" BASIS,
> Jan 18 10:54:20 # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
> express or implied.
> Jan 18 10:54:20 # See the License for the specific language governing
> permissions and
> Jan 18 10:54:20 # limitations under the License.
> Jan 18 10:54:20
> Jan 18 10:54:20 #
> ==
> Jan 18 10:54:20 # validation test
> Jan 18 10:54:20 #
> ==
> Jan 18 10:54:20
> Jan 18 10:54:20 create table tbl(a int, b as invalid_function());
> Jan 18 10:54:20 !output
> Jan 18 10:54:20 org.apache.calcite.sql.validate.SqlValidatorException: No
> match found for function signature invalid_function()
> Jan 18 10:54:20 !error
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33502) HybridShuffleITCase caused a fatal error
[
https://issues.apache.org/jira/browse/FLINK-33502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17798413#comment-17798413
]
Wencong Liu commented on FLINK-33502:
-
Sorry for the late reply. I've just identified the issue and proposed a fix; it
should be stable now. [~mapohl]
> HybridShuffleITCase caused a fatal error
>
>
> Key: FLINK-33502
> URL: https://issues.apache.org/jira/browse/FLINK-33502
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.19.0
>Reporter: Matthias Pohl
>Assignee: Wencong Liu
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.19.0
>
> Attachments: image-2023-11-20-14-37-37-321.png
>
>
> [https://github.com/XComp/flink/actions/runs/6789774296/job/18458197040#step:12:9177]
> {code:java}
> Error: 21:21:35 21:21:35.379 [ERROR] Error occurred in starting fork, check
> output in log
> 9168Error: 21:21:35 21:21:35.379 [ERROR] Process Exit Code: 239
> 9169Error: 21:21:35 21:21:35.379 [ERROR] Crashed tests:
> 9170Error: 21:21:35 21:21:35.379 [ERROR]
> org.apache.flink.test.runtime.HybridShuffleITCase
> 9171Error: 21:21:35 21:21:35.379 [ERROR]
> org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
> 9172Error: 21:21:35 21:21:35.379 [ERROR] Command was /bin/sh -c cd
> /root/flink/flink-tests && /usr/lib/jvm/jdk-11.0.19+7/bin/java -XX:+UseG1GC
> -Xms256m -XX:+IgnoreUnrecognizedVMOptions
> --add-opens=java.base/java.util=ALL-UNNAMED
> --add-opens=java.base/java.io=ALL-UNNAMED -Xmx1536m -jar
> /root/flink/flink-tests/target/surefire/surefirebooter10811559899200556131.jar
> /root/flink/flink-tests/target/surefire 2023-11-07T20-32-50_466-jvmRun4
> surefire6242806641230738408tmp surefire_1603959900047297795160tmp
> 9173Error: 21:21:35 21:21:35.379 [ERROR] Error occurred in starting fork,
> check output in log
> 9174Error: 21:21:35 21:21:35.379 [ERROR] Process Exit Code: 239
> 9175Error: 21:21:35 21:21:35.379 [ERROR] Crashed tests:
> 9176Error: 21:21:35 21:21:35.379 [ERROR]
> org.apache.flink.test.runtime.HybridShuffleITCase
> 9177Error: 21:21:35 21:21:35.379 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:532)
> 9178Error: 21:21:35 21:21:35.379 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:479)
> 9179Error: 21:21:35 21:21:35.379 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:322)
> 9180Error: 21:21:35 21:21:35.379 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:266)
> [...] {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-33905) FLIP-382: Unify the Provision of Diverse Metadata for Context-like APIs
Wencong Liu created FLINK-33905: --- Summary: FLIP-382: Unify the Provision of Diverse Metadata for Context-like APIs Key: FLINK-33905 URL: https://issues.apache.org/jira/browse/FLINK-33905 Project: Flink Issue Type: Improvement Components: API / Core Affects Versions: 1.19.0 Reporter: Wencong Liu This ticket is proposed for [FLIP-382|https://cwiki.apache.org/confluence/display/FLINK/FLIP-382%3A+Unify+the+Provision+of+Diverse+Metadata+for+Context-like+APIs]. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33939) Make husky in runtime-web no longer affect git global hooks
[ https://issues.apache.org/jira/browse/FLINK-33939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17800411#comment-17800411 ] Wencong Liu commented on FLINK-33939: - Thanks for raising this issue! 😄 I completely agree with your proposal to make front-end code detection an optional command execution in our use of husky with runtime-web. By doing this, we can preserve the functionality of any globally configured git hooks. > Make husky in runtime-web no longer affect git global hooks > --- > > Key: FLINK-33939 > URL: https://issues.apache.org/jira/browse/FLINK-33939 > Project: Flink > Issue Type: Improvement > Components: Runtime / Web Frontend >Reporter: Jason TANG >Priority: Minor > > Since runtime-web relies on husky to ensure that front-end code changes are > detected before `git commit`, husky modifies the global git hooks > (core.hooksPath) so that core.hooksPath won't take effect if it's configured > globally, I thought it would be a good idea to make the front-end code > detection a optional command execution, which ensures that the globally > configured hooks are executed correctly. -- This message was sent by Atlassian Jira (v8.20.10#820010)
