[GitHub] [flink] flinkbot commented on pull request #22208: [FLINK-31499] [table-planner] Move SqlCreateTable conversion logic to SqlCreateTableConverter
flinkbot commented on PR #22208: URL: https://github.com/apache/flink/pull/22208#issuecomment-1474732693 ## CI report: * 5d7e324b18d000d6f46e3bfde2dbd654c3dda08c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on pull request #22208: [FLINK-31499] [table-planner] Move SqlCreateTable conversion logic to SqlCreateTableConverter
xuzhiwen1255 commented on PR #22208: URL: https://github.com/apache/flink/pull/22208#issuecomment-1474732352 @wuchong wu @luoyuxia luo Please take a look, thank you very much. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31499) Move SqlCreateTable conversion logic to SqlCreateTableConverter
[
https://issues.apache.org/jira/browse/FLINK-31499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-31499:
---
Labels: pull-request-available (was: )
> Move SqlCreateTable conversion logic to SqlCreateTableConverter
> ---
>
> Key: FLINK-31499
> URL: https://issues.apache.org/jira/browse/FLINK-31499
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Assignee: xuzhiwen
>Priority: Major
> Labels: pull-request-available
>
> Introduce {{SqlCreateTableConverter}} and move the conversion logic of
> SqlCreateTable -> CreateTableOperation to it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] xuzhiwen1255 opened a new pull request, #22208: [FLINK-31499] [table-planner] Move SqlCreateTable conversion logic to SqlCreateTableConverter
xuzhiwen1255 opened a new pull request, #22208: URL: https://github.com/apache/flink/pull/22208 ## What is the purpose of the change Similar to [FLINK-31368](https://issues.apache.org/jira/browse/FLINK-31368), the SqlToOperationConverter is a bit bloated. This PR moves the conversion logic into CreateTableConverter ## Brief change log - Add SqlNodeConverter implementation of SqlCreateTableAs, SqlCreateTable and SqlCreateTableLike - SqlNodeConverter.ConvertContext adds the method of obtaining FlinkPlannerImpl ## Verifying this change This change is already covered by existing tests, such as (please describe tests). ## Does this pull request potentially affect one of the following parts: Dependencies (does it add or upgrade a dependency): no The public API, i.e., is any changed class annotated with @Public(Evolving): no The serializers: no The runtime per-record code paths (performance sensitive): no Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: no The S3 file system connector: no ## Documentation Does this pull request introduce a new feature? no If yes, how is the feature documented? not applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fredia commented on pull request #21822: [FLINK-30863][state] Register local recovery files of changelog before notifyCheckpointComplete()
fredia commented on PR #21822: URL: https://github.com/apache/flink/pull/21822#issuecomment-1474697817 @zoltar9264 Thanks for the review and suggestion, I updated the PR based on the comment. Currently, the `LocalChangelogRegistryImpl#prune()` is removed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fredia commented on a diff in pull request #21822: [FLINK-30863][state] Register local recovery files of changelog before notifyCheckpointComplete()
fredia commented on code in PR #21822:
URL: https://github.com/apache/flink/pull/21822#discussion_r1140930335
##
flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/LocalChangelogRegistryImpl.java:
##
@@ -115,4 +116,32 @@ private void scheduleAsyncDelete(StreamStateHandle
streamStateHandle) {
}
}
}
+
+@Override
+public void close() throws IOException {
+asyncDisposalExecutor.shutdown();
+handleToLastUsedCheckpointID.clear();
+}
+
+/** A changelog handle and corresponding checkpointId in the local
registry. */
+private static final class LocalStateEntry {
+private StreamStateHandle handle;
+private long createdCheckpointId;
+private long lastUsedCheckpointId;
+
+LocalStateEntry(StreamStateHandle handle, long checkpointId) {
+this.handle = handle;
+this.createdCheckpointId = checkpointId;
+this.lastUsedCheckpointId = checkpointId;
+}
+
+public void advance(long checkpointId) {
Review Comment:
Since we don't need `LocalChangelogRegistry#prune(long)`, I deleted the
inner class.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] fredia commented on a diff in pull request #21822: [FLINK-30863][state] Register local recovery files of changelog before notifyCheckpointComplete()
fredia commented on code in PR #21822:
URL: https://github.com/apache/flink/pull/21822#discussion_r1140928595
##
flink-dstl/flink-dstl-dfs/src/main/java/org/apache/flink/changelog/fs/DuplicatingStateChangeFsUploader.java:
##
@@ -51,12 +52,12 @@
* Store the meta of files into {@link ChangelogTaskLocalStateStore}
by
* AsyncCheckpointRunnable#reportCompletedSnapshotStates().
* Pass control of the file to {@link
LocalChangelogRegistry#register} when
- * ChangelogKeyedStateBackend#notifyCheckpointComplete() , files of
the previous
- * checkpoint will be deleted by {@link
LocalChangelogRegistry#discardUpToCheckpoint} at
- * the same time.
+ * ChangelogKeyedStateBackend#buildSnapshotResult , files of the
previous checkpoint will
Review Comment:
Fixed.
##
flink-dstl/flink-dstl-dfs/src/main/java/org/apache/flink/changelog/fs/FsStateChangelogWriter.java:
##
@@ -369,7 +390,6 @@ public void confirm(SequenceNumber from, SequenceNumber to,
long checkpointId) {
changelogRegistry.stopTracking(localHandle);
localChangelogRegistry.register(localHandle,
checkpointId);
});
-localChangelogRegistry.discardUpToCheckpoint(checkpointId);
Review Comment:
Fixed, retain the call of `discardUpToCheckpoint` here and delete the call
in `subsume()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] fredia commented on a diff in pull request #21822: [FLINK-30863][state] Register local recovery files of changelog before notifyCheckpointComplete()
fredia commented on code in PR #21822:
URL: https://github.com/apache/flink/pull/21822#discussion_r1140928577
##
flink-dstl/flink-dstl-dfs/src/main/java/org/apache/flink/changelog/fs/FsStateChangelogStorage.java:
##
@@ -167,6 +167,7 @@ public FsStateChangelogWriter createWriter(
@Override
public void close() throws Exception {
uploader.close();
+localChangelogRegistry.close();
Review Comment:
Fixed,replace with `IOUtils.closeQuietly` .
##
flink-dstl/flink-dstl-dfs/src/main/java/org/apache/flink/changelog/fs/FsStateChangelogWriter.java:
##
@@ -199,15 +200,21 @@ public SequenceNumber nextSequenceNumber() {
return activeSequenceNumber;
}
+@VisibleForTesting
+CompletableFuture>
persist(SequenceNumber from)
Review Comment:
The tests related to the Local file have been modified accordingly, for
other tests I left them as they are.
##
flink-dstl/flink-dstl-dfs/src/main/java/org/apache/flink/changelog/fs/FsStateChangelogWriter.java:
##
@@ -369,7 +390,6 @@ public void confirm(SequenceNumber from, SequenceNumber to,
long checkpointId) {
changelogRegistry.stopTracking(localHandle);
localChangelogRegistry.register(localHandle,
checkpointId);
Review Comment:
Fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] fredia commented on a diff in pull request #21822: [FLINK-30863][state] Register local recovery files of changelog before notifyCheckpointComplete()
fredia commented on code in PR #21822:
URL: https://github.com/apache/flink/pull/21822#discussion_r1140928508
##
flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/LocalChangelogRegistryImpl.java:
##
@@ -87,8 +88,8 @@ public void discardUpToCheckpoint(long upTo) {
public void prune(long checkpointID) {
Set handles =
handleToLastUsedCheckpointID.values().stream()
-.filter(tuple -> tuple.f1 == checkpointID)
-.map(tuple -> tuple.f0)
+.filter(entry -> entry.createdCheckpointId ==
checkpointID)
Review Comment:
Agreed, the pre-emptive upload files would not be re-upload, so those
aborted files may be referenced by subsequent checkpoints. But in case of many
subsequent aborted checkpoints, local state accumulates, all aborted local
state will not be deleted until the next completed materialization.
Considering that the priority of correctness is higher than that of space
amplification, I first deleted this method.
##
flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/LocalChangelogRegistry.java:
##
@@ -51,9 +58,9 @@ public void prune(long checkpointID) {}
* are unregistered when the checkpoint completes, because only one
checkpoint is kept for local
* recovery.
*
- * @param upTo lowest CheckpointID which is still valid.
+ * @param latestSubsumedId latest subsumed checkpointId.
*/
-void discardUpToCheckpoint(long upTo);
+void discardUpToCheckpoint(long latestSubsumedId);
Review Comment:
Previously, `discardUpToCheckpoint` was called in both `confirm()` and
`subsume()`, and in fact this is a duplicate, now it is only called in
`subsume()`, and the logic in `discardUpToCheckpoint` also changed (`<` ->
`<=`).
If `state.checkpoints.num-retained >1`, this will retain more local files
that are not needed, because local recovery only needs one previous chk.
I fixed this by remove `subsume` and only call `discardUpToCheckpoint` when
`confirm()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-29298) LocalBufferPool request buffer from NetworkBufferPool hanging
[ https://issues.apache.org/jira/browse/FLINK-29298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17702051#comment-17702051 ] Weijie Guo edited comment on FLINK-29298 at 3/18/23 3:27 AM: - Hi [~lichen1109]. Unfortunately, this bug is only possible to reproduce in the case of strong buffer competition. However, I wrote a unit test for this PR, which can reproduce the problem with a high probability. In addition, there is another bug (FLINK-31293) that can also cause a similar phenomenon. Whether your job is a batch job or a stream job, there should be no similar problems with batch jobs in the latest release-1.17 and master branches. was (Author: weijie guo): Hi [~lichen1109]. Unfortunately, this bug is only possible to reproduce in the case of strong buffer competition. However, I wrote a unit test for this PR, which can reproduce the problem with a high probability. In addition, there is another Bug (FLINK-31293) that can also cause a similar phenomenon. Whether your job is a batch job or a stream job, there should be no similar problems with batch jobs in the latest release-1.17 and master branches. > LocalBufferPool request buffer from NetworkBufferPool hanging > - > > Key: FLINK-29298 > URL: https://issues.apache.org/jira/browse/FLINK-29298 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.16.0 >Reporter: Weijie Guo >Assignee: Weijie Guo >Priority: Critical > Labels: pull-request-available > Fix For: 1.17.0, 1.16.1 > > Attachments: image-2022-09-14-10-52-15-259.png, > image-2022-09-14-10-58-45-987.png, image-2022-09-14-11-00-47-309.png > > > In the scenario where the buffer contention is fierce, sometimes the task > hang can be observed. Through the thread dump information, we can found that > the task thread is blocked by requestMemorySegmentBlocking forever. After > investigating the dumped heap information, I found that the NetworkBufferPool > actually has many buffers, but the LocalBufferPool is still unavailable and > no buffer has been obtained. > By looking at the code, I am sure that this is a bug in thread race: when the > task thread polled out the last buffer in LocalBufferPool and triggered the > onGlobalPoolAvailable callback itself, it will skip this notification (as > currently the LocalBufferPool is available), which will cause the BufferPool > to eventually become unavailable and will never register a callback to the > NetworkBufferPool. > The conditions for triggering the problem are relatively strict, but I have > found a stable way to reproduce it, I will try to fix and verify this problem. > !image-2022-09-14-10-52-15-259.png|width=1021,height=219! > !image-2022-09-14-10-58-45-987.png|width=997,height=315! > !image-2022-09-14-11-00-47-309.png|width=453,height=121! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29298) LocalBufferPool request buffer from NetworkBufferPool hanging
[ https://issues.apache.org/jira/browse/FLINK-29298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17702051#comment-17702051 ] Weijie Guo commented on FLINK-29298: Hi [~lichen1109]. Unfortunately, this bug is only possible to reproduce in the case of strong buffer competition. However, I wrote a unit test for this PR, which can reproduce the problem with a high probability. In addition, there is another Bug (FLINK-31293) that can also cause a similar phenomenon. Whether your job is a batch job or a stream job, there should be no similar problems with batch jobs in the latest release-1.17 and master branches. > LocalBufferPool request buffer from NetworkBufferPool hanging > - > > Key: FLINK-29298 > URL: https://issues.apache.org/jira/browse/FLINK-29298 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.16.0 >Reporter: Weijie Guo >Assignee: Weijie Guo >Priority: Critical > Labels: pull-request-available > Fix For: 1.17.0, 1.16.1 > > Attachments: image-2022-09-14-10-52-15-259.png, > image-2022-09-14-10-58-45-987.png, image-2022-09-14-11-00-47-309.png > > > In the scenario where the buffer contention is fierce, sometimes the task > hang can be observed. Through the thread dump information, we can found that > the task thread is blocked by requestMemorySegmentBlocking forever. After > investigating the dumped heap information, I found that the NetworkBufferPool > actually has many buffers, but the LocalBufferPool is still unavailable and > no buffer has been obtained. > By looking at the code, I am sure that this is a bug in thread race: when the > task thread polled out the last buffer in LocalBufferPool and triggered the > onGlobalPoolAvailable callback itself, it will skip this notification (as > currently the LocalBufferPool is available), which will cause the BufferPool > to eventually become unavailable and will never register a callback to the > NetworkBufferPool. > The conditions for triggering the problem are relatively strict, but I have > found a stable way to reproduce it, I will try to fix and verify this problem. > !image-2022-09-14-10-52-15-259.png|width=1021,height=219! > !image-2022-09-14-10-58-45-987.png|width=997,height=315! > !image-2022-09-14-11-00-47-309.png|width=453,height=121! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] fredia commented on a diff in pull request #21822: [FLINK-30863][state] Register local recovery files of changelog before notifyCheckpointComplete()
fredia commented on code in PR #21822:
URL: https://github.com/apache/flink/pull/21822#discussion_r1140919763
##
flink-runtime/src/main/java/org/apache/flink/runtime/state/changelog/LocalChangelogRegistryImpl.java:
##
@@ -58,23 +59,23 @@ public void register(StreamStateHandle handle, long
checkpointID) {
handle.getStreamStateHandleID(),
(k, v) -> {
if (v == null) {
-return Tuple2.of(handle, checkpointID);
+return new LocalStateEntry(handle, checkpointID);
} else {
-Preconditions.checkState(handle.equals(v.f0));
-return Tuple2.of(handle, Math.max(v.f1, checkpointID));
+Preconditions.checkState(handle.equals(v.getHandle()));
Review Comment:
No, normally there is no such case: new handle doesn't equal v.getHandle.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] chucheng92 commented on pull request #22179: [FLINK-31380][table] FLIP-297: Support enhanced show catalogs syntax
chucheng92 commented on PR #22179: URL: https://github.com/apache/flink/pull/22179#issuecomment-1474659993 because execution of operation has changed in [FLINK-31507](https://issues.apache.org/jira/browse/FLINK-31507) . I will rebase it. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] (FLINK-31380) Support enhanced show catalogs syntax
[ https://issues.apache.org/jira/browse/FLINK-31380 ] Ran Tao deleted comment on FLINK-31380: - was (Author: lemonjing): Hi, [~jark] . Currently there are three ways to support these auxiliary sql statements. 1.change every operation to support like/ilike or from/in, but future syntax we need to continue change them. (one-by-one way). 2.add a common abstract class to do this, but future change we need to change abstract class either.(abstract class way) 3.decorator model, add concrete decorator to support a certain syntax. Its benefits are very scalable and loosely coupled. currently the pr was implemented in this way. (decorators) I'm wondering if I've overcomplicated the problem(because these classes are not @Public). WDYT? Sorry, this is not discussed in FLIP. FLIP was the first way before, but I found that most of the work was repeated in the implementation. > Support enhanced show catalogs syntax > - > > Key: FLINK-31380 > URL: https://issues.apache.org/jira/browse/FLINK-31380 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Table SQL / Planner >Reporter: Ran Tao >Assignee: Ran Tao >Priority: Major > Labels: pull-request-available > > As FLIP discussed. We will support new syntax for some show operations. > To avoid bloat, this ticket supports ShowCatalogs. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #22207: FLINK-31510. Use getMemorySize instead of getMemory.
flinkbot commented on PR #22207: URL: https://github.com/apache/flink/pull/22207#issuecomment-1474563109 ## CI report: * e6835649ed601e22536f0550e52c089aa4e06da9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31510) Use getMemorySize instead of getMemory
[ https://issues.apache.org/jira/browse/FLINK-31510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-31510: --- Labels: pull-request-available (was: ) > Use getMemorySize instead of getMemory > -- > > Key: FLINK-31510 > URL: https://issues.apache.org/jira/browse/FLINK-31510 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Affects Versions: 1.18.0 >Reporter: Shilun Fan >Priority: Major > Labels: pull-request-available > > In YARN-4844, use getMemorySize instead of getMemory, because using int to > represent memory may exceed the bounds in some cases and produce negative > numbers. > This change was merged in HADOOP-2.8.0, we should use getMemorySize instead > of getMemory. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] slfan1989 opened a new pull request, #22207: FLINK-31510. Use getMemorySize instead of getMemory.
slfan1989 opened a new pull request, #22207: URL: https://github.com/apache/flink/pull/22207 ## What is the purpose of the change JIRA: FLINK-31510. Use getMemorySize instead of getMemory In [YARN-4844](https://issues.apache.org/jira/browse/YARN-4844), use getMemorySize instead of getMemory, because using int to represent memory may exceed the bounds in some cases and produce negative numbers. This change was merged in HADOOP-2.8.0, we should use getMemorySize instead of getMemory. ## Brief change log Use getMemorySize instead of getMemory. ## Verifying this change Manually verified. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-31510) Use getMemorySize instead of getMemory
Shilun Fan created FLINK-31510: -- Summary: Use getMemorySize instead of getMemory Key: FLINK-31510 URL: https://issues.apache.org/jira/browse/FLINK-31510 Project: Flink Issue Type: Improvement Components: Deployment / YARN Affects Versions: 1.18.0 Reporter: Shilun Fan In YARN-4844, use getMemorySize instead of getMemory, because using int to represent memory may exceed the bounds in some cases and produce negative numbers. This change was merged in HADOOP-2.8.0, we should use getMemorySize instead of getMemory. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29298) LocalBufferPool request buffer from NetworkBufferPool hanging
[ https://issues.apache.org/jira/browse/FLINK-29298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17702031#comment-17702031 ] Lee chen commented on FLINK-29298: -- We alse faced to this problem, how to reproduce this problem with a stable way, Thank you sir > LocalBufferPool request buffer from NetworkBufferPool hanging > - > > Key: FLINK-29298 > URL: https://issues.apache.org/jira/browse/FLINK-29298 > Project: Flink > Issue Type: Bug > Components: Runtime / Network >Affects Versions: 1.16.0 >Reporter: Weijie Guo >Assignee: Weijie Guo >Priority: Critical > Labels: pull-request-available > Fix For: 1.17.0, 1.16.1 > > Attachments: image-2022-09-14-10-52-15-259.png, > image-2022-09-14-10-58-45-987.png, image-2022-09-14-11-00-47-309.png > > > In the scenario where the buffer contention is fierce, sometimes the task > hang can be observed. Through the thread dump information, we can found that > the task thread is blocked by requestMemorySegmentBlocking forever. After > investigating the dumped heap information, I found that the NetworkBufferPool > actually has many buffers, but the LocalBufferPool is still unavailable and > no buffer has been obtained. > By looking at the code, I am sure that this is a bug in thread race: when the > task thread polled out the last buffer in LocalBufferPool and triggered the > onGlobalPoolAvailable callback itself, it will skip this notification (as > currently the LocalBufferPool is available), which will cause the BufferPool > to eventually become unavailable and will never register a callback to the > NetworkBufferPool. > The conditions for triggering the problem are relatively strict, but I have > found a stable way to reproduce it, I will try to fix and verify this problem. > !image-2022-09-14-10-52-15-259.png|width=1021,height=219! > !image-2022-09-14-10-58-45-987.png|width=997,height=315! > !image-2022-09-14-11-00-47-309.png|width=453,height=121! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-23159) Correlated sql subquery on the source created via fromValues() failed to compile
[
https://issues.apache.org/jira/browse/FLINK-23159?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17702027#comment-17702027
]
Deepyaman Datta commented on FLINK-23159:
-
Hello! I believe I'm affected by this issue, as shared on the Apache Flink
Slack ([https://apache-flink.slack.com/archives/C03G7LJTS2G/p1679089054243999).]
If I'm understanding correctly, would
[https://github.com/apache/flink/blob/release-1.17.0-rc3/flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/rules/logical/SubQueryDecorrelator.java#L893-L901]
need to be filled out (to not return `null`)? Is this something that would be
reasonable for a new contributor to tackle? I'm unfortunately not at all
familiar with the Flink codebase, but I can give it a shot, if reasonable.
Alternatively, are there any suggested workarounds to this issue?
> Correlated sql subquery on the source created via fromValues() failed to
> compile
>
>
> Key: FLINK-23159
> URL: https://issues.apache.org/jira/browse/FLINK-23159
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.13.0
>Reporter: Yun Gao
>Priority: Major
>
> Correlated subquery like
> {code:java}
> import org.apache.flink.table.api.DataTypes;
> import org.apache.flink.table.api.EnvironmentSettings;
> import org.apache.flink.table.api.Table;
> import org.apache.flink.table.api.TableEnvironment;
> import org.apache.flink.table.types.DataType;
> import org.apache.flink.types.Row;
> import java.util.ArrayList;
> import java.util.List;
> public class SQLQueryTest {
> public static void main(String[] args) {
> EnvironmentSettings settings =
> EnvironmentSettings.newInstance().useBlinkPlanner().inBatchMode()
> .build();
> TableEnvironment tableEnvironment = TableEnvironment.create(settings);
> DataType row = DataTypes.ROW(
> DataTypes.FIELD("flag", DataTypes.STRING()),
> DataTypes.FIELD("id", DataTypes.INT()),
> DataTypes.FIELD("name", DataTypes.STRING())
> );
> Table table = tableEnvironment.fromValues(row, new
> MyListSource("table1").builder());
> tableEnvironment.createTemporaryView("table1", table);
> table = tableEnvironment.fromValues(row, new
> MyListSource("table2").builder());
> tableEnvironment.createTemporaryView("table2", table);
> String sql = "select t1.flag from table1 t1 where t1.name in (select
> t2.name from table2 t2 where t2.id = t1.id)";
> tableEnvironment.explainSql(sql);
> }
> public static class MyListSource {
> private String flag;
> public MyListSource(String flag) {
> this.flag = flag;
> }
> public List builder() {
> List rows = new ArrayList<>();
> for (int i = 2; i < 3; i++) {
> Row row = new Row(3);
> row.setField(0, flag);
> row.setField(1, i);
> row.setField(2, "me");
> rows.add(row);
> }
> return rows;
> }
> }
> }
> {code}
> would throws
> {code:java}
> Exception in thread "main" org.apache.flink.table.api.TableException:
> unexpected correlate variable $cor0 in the plan
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkDecorrelateProgram.checkCorrelVariableExists(FlinkDecorrelateProgram.scala:57)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkDecorrelateProgram.optimize(FlinkDecorrelateProgram.scala:42)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram$$anonfun$optimize$1$$anonfun$apply$1.apply(FlinkGroupProgram.scala:63)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram$$anonfun$optimize$1$$anonfun$apply$1.apply(FlinkGroupProgram.scala:60)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
> at scala.collection.Iterator$class.foreach(Iterator.scala:891)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
> at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
> at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
> at
> scala.collection.TraversableOnce$class.foldLeft(TraversableOnce.scala:157)
> at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram$$anonfun$optimize$1.apply(FlinkGroupProgram.scala:60)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkGroupProgram$$anonfun$optimize$1.apply(FlinkGroupProgram.scala:55)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft$1.apply(TraversableOnce.scala:157)
> at
> scala.collection.TraversableOnce$$anonfun$foldLeft
[GitHub] [flink-kubernetes-operator] robsonpeixoto commented on pull request #93: [FLINK-26706] Introduce Ingress URL templating
robsonpeixoto commented on PR #93:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/93#issuecomment-1474533148
This solution is amazing! Thanks a lot =D
But add this feature in annotation os very useful, for example:
```
ingress:
template: "/{{namespace}}/{{name}}(/|$)(.*)"
className: "nginx"
annotations:
nginx.ingress.kubernetes.io/rewrite-target: "/$2"
nginx.ingress.kubernetes.io/enable-rewrite-log: "true"
nginx.ingress.kubernetes.io/configuration-snippet: |
if ($uri = "/{{namespace}}/{{name}}") {rewrite .*
$1/{{namespace}}/{{name}}/ permanent;}
```
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-31509) REST Service missing sessionAffinity causes job run failure with HA cluster
Emmanuel Leroy created FLINK-31509: -- Summary: REST Service missing sessionAffinity causes job run failure with HA cluster Key: FLINK-31509 URL: https://issues.apache.org/jira/browse/FLINK-31509 Project: Flink Issue Type: Bug Components: Kubernetes Operator Environment: Flink 1.15 on Flink Operator 1.4.0 on Kubernetes 1.25.4, (optionally with Beam 2.46.0) but the issue was observed on Flink 1.14, 1.15 and 1.16 and on Flink Operator 1.2, 1.3, 1.3.1, 1.4.0 Reporter: Emmanuel Leroy When using a Session Cluster with multiple Job Managers, the -rest service load balances the API requests to all job managers, not just the master. When submitting a FlinkSessionJob, I often see errors like: `jar .jar was not found`, because the submission is done in 2 steps: * upload the jar with `v1/jars/upload` which returns the `jar_id` * run the job with `v1/jars//run` Unfortunately, with the Service load balacing between nodes, it is often the case that the jar is uploaded on a JM, and the run request happens on another, where the jar doesn't exist. A simple fix is to append the `sessionAffinity: ClientIP` on the -rest service, where the API calls from a given originating IP will always be routed to the same node. This issue is especially problematic with Beam, where the Beam job submission does not retry to run the job with the jar_id, and will fail, causing it to re-upload a new jar and retrying, until it is lucky enough to get the 2 calls in a row routed to the same node. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-kubernetes-operator] gyfora commented on a diff in pull request #549: [FLINK-31502] Limit the number of scale operations to reduce cluster churn
gyfora commented on code in PR #549: URL: https://github.com/apache/flink-kubernetes-operator/pull/549#discussion_r1140649362 ## docs/layouts/shortcodes/generated/dynamic_section.html: ## @@ -86,6 +86,12 @@ Duration Interval at which periodic savepoints will be triggered. The triggering schedule is not guaranteed, savepoints will be triggered as part of the regular reconcile loop. + +kubernetes.operator.rescaling.cluster-cooldown +1 min +Duration Review Comment: The default value should be 0 I think to preserve the current behavior as this does not affect all kinds of Kubernetes clusters . many are actually fixed sized -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-31508) FLIP-304: Pluggable failure handling for Apache Flink
Panagiotis Garefalakis created FLINK-31508: -- Summary: FLIP-304: Pluggable failure handling for Apache Flink Key: FLINK-31508 URL: https://issues.apache.org/jira/browse/FLINK-31508 Project: Flink Issue Type: New Feature Components: Runtime / REST Reporter: Panagiotis Garefalakis Fix For: 1.18.0 This is an umbrella ticket for [FLIP-304|https://cwiki.apache.org/confluence/display/FLINK/FLIP-304%3A+Pluggable+failure+handling+for+Apache+Flink] -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-31502) Limit the number of concurrent scale operations to reduce cluster churn
[ https://issues.apache.org/jira/browse/FLINK-31502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-31502: --- Labels: pull-request-available (was: ) > Limit the number of concurrent scale operations to reduce cluster churn > --- > > Key: FLINK-31502 > URL: https://issues.apache.org/jira/browse/FLINK-31502 > Project: Flink > Issue Type: Improvement > Components: Autoscaler, Kubernetes Operator >Reporter: Maximilian Michels >Assignee: Maximilian Michels >Priority: Major > Labels: pull-request-available > Fix For: kubernetes-operator-1.5.0 > > > Until we move to using the upcoming Rescale API which recycles pods, we need > to be mindful with how many deployments we scale at the same time because > each of them is going to give up all its pods and require the new number of > required pods. > This can cause churn in the cluster and temporary lead to "unallocatable" > pods which triggers the k8s cluster autoscaler to add more cluster nodes. > That is often not desirable because the actual required resources after the > scaling have been settled, are lower. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-kubernetes-operator] mxm opened a new pull request, #549: [FLINK-31502] Limit the number of scale operations to reduce cluster churn
mxm opened a new pull request, #549: URL: https://github.com/apache/flink-kubernetes-operator/pull/549 Until we move to using the upcoming Rescale API which recycles pods, we need to be mindful with how many deployments we scale at the same time because each of them is going to give up all its pods and require the new number of required pods. This can cause churn in the cluster and temporary lead to "unallocatable" pods which triggers the k8s cluster autoscaler to add more cluster nodes. That is often not desirable because the actual required resources after the scalings have been settled, are lower. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-31490) ZooKeeperLeaderElectionTest.testLeaderShouldBeCorrectedWhenOverwritten times out
[
https://issues.apache.org/jira/browse/FLINK-31490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701917#comment-17701917
]
Chesnay Schepler commented on FLINK-31490:
--
But FLINK-27848 was fixed in 1.17.0; if that change did cause this issue then
it could be 1.17 specific.
> ZooKeeperLeaderElectionTest.testLeaderShouldBeCorrectedWhenOverwritten times
> out
>
>
> Key: FLINK-31490
> URL: https://issues.apache.org/jira/browse/FLINK-31490
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: test-stability
> Attachments: FLINK-31490.mvn.log, FLINK-31490.zookeeper-client.log,
> FLINK-31490.zookeeper-server.log
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=47221&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1&t=24c3384f-1bcb-57b3-224f-51bf973bbee8&l=9448
> {code}
> Mar 16 02:00:54 "main" #1 prio=5 os_prio=0 tid=0x7f488800b800 nid=0x5a15
> waiting on condition [0x7f488fe14000]
> Mar 16 02:00:54java.lang.Thread.State: WAITING (parking)
> Mar 16 02:00:54 at sun.misc.Unsafe.park(Native Method)
> Mar 16 02:00:54 - parking to wait for <0xe4065228> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> Mar 16 02:00:54 at
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> Mar 16 02:00:54 at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> Mar 16 02:00:54 at
> java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:442)
> Mar 16 02:00:54 at
> org.apache.flink.runtime.leaderelection.TestingRetrievalBase.lambda$waitForNewLeader$0(TestingRetrievalBase.java:50)
> Mar 16 02:00:54 at
> org.apache.flink.runtime.leaderelection.TestingRetrievalBase$$Lambda$1377/797057570.get(Unknown
> Source)
> Mar 16 02:00:54 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:150)
> Mar 16 02:00:54 at
> org.apache.flink.runtime.testutils.CommonTestUtils.waitUntilCondition(CommonTestUtils.java:144)
> Mar 16 02:00:54 at
> org.apache.flink.runtime.leaderelection.TestingRetrievalBase.waitForNewLeader(TestingRetrievalBase.java:48)
> Mar 16 02:00:54 at
> org.apache.flink.runtime.leaderelection.ZooKeeperLeaderElectionTest.testLeaderShouldBeCorrectedWhenOverwritten(ZooKeeperLeaderElectionTest.java:479)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] zentol commented on pull request #22052: [FLINK-31278] Disables fork reuse and parallel execution for flink-core package
zentol commented on PR #22052: URL: https://github.com/apache/flink/pull/22052#issuecomment-1474175255 I'd suggest to instead somewhat profile the runtime tests instead. Add a special test that runs last (see https://stackoverflow.com/questions/57624495/junit-test-class-order) that triggers a heap dump (and whatever else we'd like (e.g., memory consumption), run a sub-section of tests, analyze dump. Maybe even as a resource that runs after every single test. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-31380) Support enhanced show catalogs syntax
[ https://issues.apache.org/jira/browse/FLINK-31380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701912#comment-17701912 ] Ran Tao edited comment on FLINK-31380 at 3/17/23 5:26 PM: -- Hi, [~jark] . Currently there are three ways to support these auxiliary sql statements. 1.change every operation to support like/ilike or from/in, but future syntax we need to continue change them. (one-by-one way). 2.add a common abstract class to do this, but future change we need to change abstract class either.(abstract class way) 3.decorator model, add concrete decorator to support a certain syntax. Its benefits are very scalable and loosely coupled. currently the pr was implemented in this way. (decorators) I'm wondering if I've overcomplicated the problem(because these classes are not @Public). WDYT? Sorry, this is not discussed in FLIP. FLIP was the first way before, but I found that most of the work was repeated in the implementation. was (Author: lemonjing): Hi, [~jark] . Currently there are three ways to support these auxiliary sql statements. 1.change every operation to support like/ilike or from/in, but future syntax we need to continue change them. (one-by-one way). 2.add a common abstract class to do this, but future change we need to change abstract class either.(abstract class way) 3.decorator model, add concrete decorator to support a certain syntax. Its benefits are very scalable and loosely coupled. currently the pr was implemented in this way. (decorators) I'm wondering if I've overcomplicated the problem(because these classes are not @Public). WDYT? > Support enhanced show catalogs syntax > - > > Key: FLINK-31380 > URL: https://issues.apache.org/jira/browse/FLINK-31380 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Table SQL / Planner >Reporter: Ran Tao >Assignee: Ran Tao >Priority: Major > Labels: pull-request-available > > As FLIP discussed. We will support new syntax for some show operations. > To avoid bloat, this ticket supports ShowCatalogs. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zentol commented on pull request #22052: [FLINK-31278] Disables fork reuse and parallel execution for flink-core package
zentol commented on PR #22052: URL: https://github.com/apache/flink/pull/22052#issuecomment-1474170781 Still I don't see what this gets you. The test running into the OOM isn't necessarily who's causing it. If some test allocates 99% of memory but finishes, then some subsequent test may end up failing. You'd kinda end up in the same situation you are right now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-31380) Support enhanced show catalogs syntax
[ https://issues.apache.org/jira/browse/FLINK-31380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701912#comment-17701912 ] Ran Tao edited comment on FLINK-31380 at 3/17/23 5:24 PM: -- Hi, [~jark] . Currently there are three ways to support these auxiliary sql statements. 1.change every operation to support like/ilike or from/in, but future syntax we need to continue change them. (one-by-one way). 2.add a common abstract class to do this, but future change we need to change abstract class either.(abstract class way) 3.decorator model, add concrete decorator to support a certain syntax. Its benefits are very scalable and loosely coupled. currently the pr was implemented in this way. (decorators) I'm wondering if I've overcomplicated the problem(because these classes are not @Public). WDYT? was (Author: lemonjing): Hi, [~jark] . Currently there are three ways to support these auxiliary sql statements. 1.change every operation to support like/ilike or from/in, but future syntax we need to continue change them. (one-by-one way). 2.add a common abstract class to do this, but future change we need to change abstract class either.(abstract class way) 3.decorator model, add concrete decorator to support a certain syntax. Its benefits are very scalable and loosely coupled. currently the pr was implemented in this way. I'm wondering if I've overcomplicated the problem(because these classes are not @Public). WDYT? > Support enhanced show catalogs syntax > - > > Key: FLINK-31380 > URL: https://issues.apache.org/jira/browse/FLINK-31380 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Table SQL / Planner >Reporter: Ran Tao >Assignee: Ran Tao >Priority: Major > Labels: pull-request-available > > As FLIP discussed. We will support new syntax for some show operations. > To avoid bloat, this ticket supports ShowCatalogs. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31380) Support enhanced show catalogs syntax
[ https://issues.apache.org/jira/browse/FLINK-31380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701912#comment-17701912 ] Ran Tao commented on FLINK-31380: - Hi, [~jark] . Currently there are three ways to support these auxiliary sql statements. 1.change every operation to support like/ilike or from/in, but future syntax we need to continue change them. (one-by-one way). 2.add a common abstract class to do this, but future change we need to change abstract class either.(abstract class way) 3.decorator model, add concrete decorator to support a certain syntax. Its benefits are very scalable and loosely coupled. currently the pr was implemented in this way. I'm wondering if I've overcomplicated the problem(because these classes are not @Public). WDYT? > Support enhanced show catalogs syntax > - > > Key: FLINK-31380 > URL: https://issues.apache.org/jira/browse/FLINK-31380 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Table SQL / Planner >Reporter: Ran Tao >Assignee: Ran Tao >Priority: Major > Labels: pull-request-available > > As FLIP discussed. We will support new syntax for some show operations. > To avoid bloat, this ticket supports ShowCatalogs. > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] zentol commented on pull request #22052: [FLINK-31278] Disables fork reuse and parallel execution for flink-core package
zentol commented on PR #22052: URL: https://github.com/apache/flink/pull/22052#issuecomment-1474164411 wait a minute. Why is the basic nightly run using `'ubuntu-20.04'` as the pool`? A because of FLINK-18370... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #22052: [FLINK-31278] Disables fork reuse and parallel execution for flink-core package
zentol commented on PR #22052: URL: https://github.com/apache/flink/pull/22052#issuecomment-1474157633 What makes you think they are on azure-owned hosts? They are both scheduled builds in flink-ci, which do use the alibaba machines. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-31504) Move execution logic of CreateOperation out from TableEnvironmentImpl
[
https://issues.apache.org/jira/browse/FLINK-31504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-31504:
---
Assignee: Jark Wu
> Move execution logic of CreateOperation out from TableEnvironmentImpl
> -
>
> Key: FLINK-31504
> URL: https://issues.apache.org/jira/browse/FLINK-31504
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Assignee: Jark Wu
>Priority: Major
>
> This should implement {{ExecutableOperation}} for all the
> {{CreateOperation}}s to move the execution logic out from
> {{TableEnvironmentImpl#executeInternal()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31507) Move execution logic of ShowOperation out from TableEnvironmentImpl
Jark Wu created FLINK-31507:
---
Summary: Move execution logic of ShowOperation out from
TableEnvironmentImpl
Key: FLINK-31507
URL: https://issues.apache.org/jira/browse/FLINK-31507
Project: Flink
Issue Type: Sub-task
Components: Table SQL / Planner
Reporter: Jark Wu
This should implement {{ExecutableOperation}} for all the {{ShowOperation}}s to
move the execution logic out from {{TableEnvironmentImpl#executeInternal()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-connector-cassandra] echauchot commented on pull request #3: [FLINK-26822] Add Cassandra Source
echauchot commented on PR #3: URL: https://github.com/apache/flink-connector-cassandra/pull/3#issuecomment-1474052060 @zentol I pushed for security reasons, I have only the addSplitsBack case to deal with in lazy splits generation scenario. You can start to take a look at the other things if you have time or just wait for this part to be finished. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-31505) Move execution logic of DropOperation out from TableEnvironmentImpl
Jark Wu created FLINK-31505:
---
Summary: Move execution logic of DropOperation out from
TableEnvironmentImpl
Key: FLINK-31505
URL: https://issues.apache.org/jira/browse/FLINK-31505
Project: Flink
Issue Type: Sub-task
Components: Table SQL / Planner
Reporter: Jark Wu
This should implement {{ExecutableOperation}} for all the {{DropOperation}}s to
move the execution logic out from {{TableEnvironmentImpl#executeInternal()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31506) Move execution logic of AlterOperation out from TableEnvironmentImpl
Jark Wu created FLINK-31506:
---
Summary: Move execution logic of AlterOperation out from
TableEnvironmentImpl
Key: FLINK-31506
URL: https://issues.apache.org/jira/browse/FLINK-31506
Project: Flink
Issue Type: Sub-task
Components: Table SQL / Planner
Reporter: Jark Wu
This should implement {{ExecutableOperation}} for all the {{AlterOperation}}s
to move the execution logic out from {{TableEnvironmentImpl#executeInternal()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31504) Move execution logic of CreateOperation out from TableEnvironmentImpl
Jark Wu created FLINK-31504:
---
Summary: Move execution logic of CreateOperation out from
TableEnvironmentImpl
Key: FLINK-31504
URL: https://issues.apache.org/jira/browse/FLINK-31504
Project: Flink
Issue Type: Sub-task
Components: Table SQL / Planner
Reporter: Jark Wu
This should implement {{ExecutableOperation}} for all the {{CreateOperation}}s
to move the execution logic out from {{TableEnvironmentImpl#executeInternal()}}.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-31494) Introduce SqlNodeConverter for SqlToOperationConverter
[
https://issues.apache.org/jira/browse/FLINK-31494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu closed FLINK-31494.
---
Fix Version/s: 1.18.0
Resolution: Fixed
Fixed in master: ba03d4db7ca01c7a2436253b9f9429a2a3fb3219
> Introduce SqlNodeConverter for SqlToOperationConverter
> --
>
> Key: FLINK-31494
> URL: https://issues.apache.org/jira/browse/FLINK-31494
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: xuzhiwen
>Assignee: Jark Wu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> Introduce {{SqlNodeConverter}} for {{SqlToOperationConverter}}, following
> Timo's idea in FLINK-31368
> class like:
> {code:java}
> public interface SqlNodeConverter {
> Operation convertSqlNode(S node, ConvertContext context);
> }
> /** Registry of SqlNode converters. */
> public class SqlNodeConverters {
> private static final Map, SqlNodeConverter> CONVERTERS = new
> HashMap<>();
> static {
> // register all the converters here
> register(new SqlCreateCatalogConverter());
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31494) Introduce SqlNodeConverter for SqlToOperationConverter
[
https://issues.apache.org/jira/browse/FLINK-31494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701863#comment-17701863
]
Jark Wu edited comment on FLINK-31494 at 3/17/23 3:42 PM:
--
Fixed in master: ba03d4db7ca01c7a2436253b9f9429a2a3fb3219 and
fa0dd3559e9697e21795a2634d8d99a0b7efdcf3
was (Author: jark):
Fixed in master: ba03d4db7ca01c7a2436253b9f9429a2a3fb3219
> Introduce SqlNodeConverter for SqlToOperationConverter
> --
>
> Key: FLINK-31494
> URL: https://issues.apache.org/jira/browse/FLINK-31494
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: xuzhiwen
>Assignee: Jark Wu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.18.0
>
>
> Introduce {{SqlNodeConverter}} for {{SqlToOperationConverter}}, following
> Timo's idea in FLINK-31368
> class like:
> {code:java}
> public interface SqlNodeConverter {
> Operation convertSqlNode(S node, ConvertContext context);
> }
> /** Registry of SqlNode converters. */
> public class SqlNodeConverters {
> private static final Map, SqlNodeConverter> CONVERTERS = new
> HashMap<>();
> static {
> // register all the converters here
> register(new SqlCreateCatalogConverter());
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] wuchong merged pull request #22205: [FLINK-31494][table-planner] Introduce SqlNodeConverter for SqlToOperationConverter
wuchong merged PR #22205: URL: https://github.com/apache/flink/pull/22205 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong commented on pull request #22205: [FLINK-31494][table-planner] Introduce SqlNodeConverter for SqlToOperationConverter
wuchong commented on PR #22205: URL: https://github.com/apache/flink/pull/22205#issuecomment-1474031293 Thank you for the review @xuzhiwen1255 . Merging... -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong commented on a diff in pull request #22205: [FLINK-31494][table-planner] Introduce SqlNodeConverter for SqlToOperationConverter
wuchong commented on code in PR #22205:
URL: https://github.com/apache/flink/pull/22205#discussion_r1140393596
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/SqlNodeToOperationConversion.java:
##
@@ -285,11 +287,18 @@ public static Optional convert(
private static Optional convertValidatedSqlNode(
FlinkPlannerImpl flinkPlanner, CatalogManager catalogManager,
SqlNode validated) {
beforeConversion();
-SqlToOperationConverter converter =
-new SqlToOperationConverter(flinkPlanner, catalogManager);
-if (validated instanceof SqlCreateCatalog) {
-return
Optional.of(converter.convertCreateCatalog((SqlCreateCatalog) validated));
-} else if (validated instanceof SqlDropCatalog) {
+
+// delegate conversion to the registered converters first
+SqlNodeConvertContext context = new
SqlNodeConvertContext(flinkPlanner, catalogManager);
+Optional operation =
SqlNodeConverters.convertSqlNode(validated, context);
+if (operation.isPresent()) {
+return operation;
+}
+
+// TODO: all the below conversion logic should be migrated to
SqlNodeConverters
Review Comment:
Yes, all the SqlNodes involved in the conversion should be migrated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-31503) "org.apache.beam.sdk.options.PipelineOptionsRegistrar: Provider org.apache.beam.sdk.options.DefaultPipelineOptionsRegistrar not a subtype" is thrown when executing Pyt
[
https://issues.apache.org/jira/browse/FLINK-31503?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701861#comment-17701861
]
Dian Fu commented on FLINK-31503:
-
If your Flink version doesn't contain this fix, you could simply work around
this issue by adding the following configuration:
SET 'classloader.parent-first-patterns.additional' = 'org.apache.beam.';
> "org.apache.beam.sdk.options.PipelineOptionsRegistrar: Provider
> org.apache.beam.sdk.options.DefaultPipelineOptionsRegistrar not a subtype" is
> thrown when executing Python UDFs in SQL Client
> --
>
> Key: FLINK-31503
> URL: https://issues.apache.org/jira/browse/FLINK-31503
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Reporter: Dian Fu
>Assignee: Dian Fu
>Priority: Major
> Labels: pull-request-available
>
> The following exception will be thrown when executing SQL statements
> containing Python UDFs in SQL Client:
> {code}
> Caused by: java.util.ServiceConfigurationError:
> org.apache.beam.sdk.options.PipelineOptionsRegistrar: Provider
> org.apache.beam.sdk.options.DefaultPipelineOptionsRegistrar not a subtype
> at java.util.ServiceLoader.fail(ServiceLoader.java:239)
> at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:376)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:415)
> at
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet$Builder.addAll(ImmutableSet.java:507)
> at
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSortedSet$Builder.addAll(ImmutableSortedSet.java:528)
> at
> org.apache.beam.sdk.util.common.ReflectHelpers.loadServicesOrdered(ReflectHelpers.java:199)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory$Cache.initializeRegistry(PipelineOptionsFactory.java:2089)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory$Cache.(PipelineOptionsFactory.java:2083)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory$Cache.(PipelineOptionsFactory.java:2047)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.resetCache(PipelineOptionsFactory.java:581)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.(PipelineOptionsFactory.java:547)
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.open(BeamPythonFunctionRunner.java:241)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-31499) Move SqlCreateTable conversion logic to SqlCreateTableConverter
[
https://issues.apache.org/jira/browse/FLINK-31499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-31499:
---
Assignee: xuzhiwen
> Move SqlCreateTable conversion logic to SqlCreateTableConverter
> ---
>
> Key: FLINK-31499
> URL: https://issues.apache.org/jira/browse/FLINK-31499
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Assignee: xuzhiwen
>Priority: Major
>
> Introduce {{SqlCreateTableConverter}} and move the conversion logic of
> SqlCreateTable -> CreateTableOperation to it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-31478) TypeError: a bytes-like object is required, not 'JavaList' is thrown when ds.execute_and_collect() is called on a KeyedStream
[
https://issues.apache.org/jira/browse/FLINK-31478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu closed FLINK-31478.
---
Fix Version/s: 1.16.2
1.18.0
1.17.1
Resolution: Fixed
Fixed in:
- master via 5e059efee864e17939a33f29272a848d00598531
- release-1.17 via ec5a09b3ce56426d1bdc8eeac4bf52cac9be015b
- release-1.16 via cadf4b35fb6f20c8cba310fa54626d0b9bae1361
> TypeError: a bytes-like object is required, not 'JavaList' is thrown when
> ds.execute_and_collect() is called on a KeyedStream
> -
>
> Key: FLINK-31478
> URL: https://issues.apache.org/jira/browse/FLINK-31478
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Reporter: Dian Fu
>Assignee: Dian Fu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.16.2, 1.18.0, 1.17.1
>
>
> {code}
>
> # Licensed to the Apache Software Foundation (ASF) under one
> # or more contributor license agreements. See the NOTICE file
> # distributed with this work for additional information
> # regarding copyright ownership. The ASF licenses this file
> # to you under the Apache License, Version 2.0 (the
> # "License"); you may not use this file except in compliance
> # with the License. You may obtain a copy of the License at
> #
> # http://www.apache.org/licenses/LICENSE-2.0
> #
> # Unless required by applicable law or agreed to in writing, software
> # distributed under the License is distributed on an "AS IS" BASIS,
> # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
> # See the License for the specific language governing permissions and
> # limitations under the License.
>
> import argparse
> import logging
> import sys
> from pyflink.common import WatermarkStrategy, Encoder, Types
> from pyflink.datastream import StreamExecutionEnvironment,
> RuntimeExecutionMode
> from pyflink.datastream.connectors.file_system import (FileSource,
> StreamFormat, FileSink,
>OutputFileConfig,
> RollingPolicy)
> word_count_data = ["To be, or not to be,--that is the question:--",
>"Whether 'tis nobler in the mind to suffer",
>"The slings and arrows of outrageous fortune",
>"Or to take arms against a sea of troubles,",
>"And by opposing end them?--To die,--to sleep,--",
>"No more; and by a sleep to say we end",
>"The heartache, and the thousand natural shocks",
>"That flesh is heir to,--'tis a consummation",
>"Devoutly to be wish'd. To die,--to sleep;--",
>"To sleep! perchance to dream:--ay, there's the rub;",
>"For in that sleep of death what dreams may come,",
>"When we have shuffled off this mortal coil,",
>"Must give us pause: there's the respect",
>"That makes calamity of so long life;",
>"For who would bear the whips and scorns of time,",
>"The oppressor's wrong, the proud man's contumely,",
>"The pangs of despis'd love, the law's delay,",
>"The insolence of office, and the spurns",
>"That patient merit of the unworthy takes,",
>"When he himself might his quietus make",
>"With a bare bodkin? who would these fardels bear,",
>"To grunt and sweat under a weary life,",
>"But that the dread of something after death,--",
>"The undiscover'd country, from whose bourn",
>"No traveller returns,--puzzles the will,",
>"And makes us rather bear those ills we have",
>"Than fly to others that we know not of?",
>"Thus conscience does make cowards of us all;",
>"And thus the native hue of resolution",
>"Is sicklied o'er with the pale cast of thought;",
>"And enterprises of great pith and moment,",
>"With this regard, their currents turn awry,",
>"And lose the name of action.--Soft you now!",
>"The fair Ophelia!--Nymph, in thy orisons",
>"Be all my sins remember'd."]
> def word_count(input_path, output_path):
> env = StreamExecutionEnvironment.get_execution_environment()
> env.set_runtime_mo
[GitHub] [flink] dianfu closed pull request #22190: [FLINK-31478][python] Fix ds.execute_and_collect to support nested tuple
dianfu closed pull request #22190: [FLINK-31478][python] Fix ds.execute_and_collect to support nested tuple URL: https://github.com/apache/flink/pull/22190 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #22206: [FLINK-31503][python] Makes sure it loads Beam classes using the same class loader as the PyFlink classes
flinkbot commented on PR #22206: URL: https://github.com/apache/flink/pull/22206#issuecomment-1473992293 ## CI report: * a0428979c1e0b2870fe6e2710e2584e0ae531420 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31503) "org.apache.beam.sdk.options.PipelineOptionsRegistrar: Provider org.apache.beam.sdk.options.DefaultPipelineOptionsRegistrar not a subtype" is thrown when executing Pytho
[
https://issues.apache.org/jira/browse/FLINK-31503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-31503:
---
Labels: pull-request-available (was: )
> "org.apache.beam.sdk.options.PipelineOptionsRegistrar: Provider
> org.apache.beam.sdk.options.DefaultPipelineOptionsRegistrar not a subtype" is
> thrown when executing Python UDFs in SQL Client
> --
>
> Key: FLINK-31503
> URL: https://issues.apache.org/jira/browse/FLINK-31503
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Reporter: Dian Fu
>Assignee: Dian Fu
>Priority: Major
> Labels: pull-request-available
>
> The following exception will be thrown when executing SQL statements
> containing Python UDFs in SQL Client:
> {code}
> Caused by: java.util.ServiceConfigurationError:
> org.apache.beam.sdk.options.PipelineOptionsRegistrar: Provider
> org.apache.beam.sdk.options.DefaultPipelineOptionsRegistrar not a subtype
> at java.util.ServiceLoader.fail(ServiceLoader.java:239)
> at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
> at
> java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:376)
> at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
> at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
> at
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:415)
> at
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet$Builder.addAll(ImmutableSet.java:507)
> at
> org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSortedSet$Builder.addAll(ImmutableSortedSet.java:528)
> at
> org.apache.beam.sdk.util.common.ReflectHelpers.loadServicesOrdered(ReflectHelpers.java:199)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory$Cache.initializeRegistry(PipelineOptionsFactory.java:2089)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory$Cache.(PipelineOptionsFactory.java:2083)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory$Cache.(PipelineOptionsFactory.java:2047)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.resetCache(PipelineOptionsFactory.java:581)
> at
> org.apache.beam.sdk.options.PipelineOptionsFactory.(PipelineOptionsFactory.java:547)
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.open(BeamPythonFunctionRunner.java:241)
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] dianfu opened a new pull request, #22206: [FLINK-31503][python] Makes sure it loads Beam classes using the same class loader as the PyFlink classes
dianfu opened a new pull request, #22206: URL: https://github.com/apache/flink/pull/22206 … ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluster with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] reswqa commented on pull request #22161: [FLINK-31405] Refactor tests to git rid of timeout of CompletableFuture assertions.
reswqa commented on PR #22161: URL: https://github.com/apache/flink/pull/22161#issuecomment-1473941258 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-31503) "org.apache.beam.sdk.options.PipelineOptionsRegistrar: Provider org.apache.beam.sdk.options.DefaultPipelineOptionsRegistrar not a subtype" is thrown when executing Pytho
Dian Fu created FLINK-31503:
---
Summary: "org.apache.beam.sdk.options.PipelineOptionsRegistrar:
Provider org.apache.beam.sdk.options.DefaultPipelineOptionsRegistrar not a
subtype" is thrown when executing Python UDFs in SQL Client
Key: FLINK-31503
URL: https://issues.apache.org/jira/browse/FLINK-31503
Project: Flink
Issue Type: Bug
Components: API / Python
Reporter: Dian Fu
Assignee: Dian Fu
The following exception will be thrown when executing SQL statements containing
Python UDFs in SQL Client:
{code}
Caused by: java.util.ServiceConfigurationError:
org.apache.beam.sdk.options.PipelineOptionsRegistrar: Provider
org.apache.beam.sdk.options.DefaultPipelineOptionsRegistrar not a subtype
at java.util.ServiceLoader.fail(ServiceLoader.java:239)
at java.util.ServiceLoader.access$300(ServiceLoader.java:185)
at
java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:376)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableCollection$Builder.addAll(ImmutableCollection.java:415)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSet$Builder.addAll(ImmutableSet.java:507)
at
org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.ImmutableSortedSet$Builder.addAll(ImmutableSortedSet.java:528)
at
org.apache.beam.sdk.util.common.ReflectHelpers.loadServicesOrdered(ReflectHelpers.java:199)
at
org.apache.beam.sdk.options.PipelineOptionsFactory$Cache.initializeRegistry(PipelineOptionsFactory.java:2089)
at
org.apache.beam.sdk.options.PipelineOptionsFactory$Cache.(PipelineOptionsFactory.java:2083)
at
org.apache.beam.sdk.options.PipelineOptionsFactory$Cache.(PipelineOptionsFactory.java:2047)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.resetCache(PipelineOptionsFactory.java:581)
at
org.apache.beam.sdk.options.PipelineOptionsFactory.(PipelineOptionsFactory.java:547)
at
org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.open(BeamPythonFunctionRunner.java:241)
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] xuzhiwen1255 commented on pull request #22205: [FLINK-31494][table-planner] Introduce SqlNodeConverter for SqlToOperationConverter
xuzhiwen1255 commented on PR #22205: URL: https://github.com/apache/flink/pull/22205#issuecomment-1473881727 LGTM. Thank you @wuchong for contribution. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xuzhiwen1255 commented on a diff in pull request #22205: [FLINK-31494][table-planner] Introduce SqlNodeConverter for SqlToOperationConverter
xuzhiwen1255 commented on code in PR #22205:
URL: https://github.com/apache/flink/pull/22205#discussion_r1140254867
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/SqlNodeToOperationConversion.java:
##
@@ -285,11 +287,18 @@ public static Optional convert(
private static Optional convertValidatedSqlNode(
FlinkPlannerImpl flinkPlanner, CatalogManager catalogManager,
SqlNode validated) {
beforeConversion();
-SqlToOperationConverter converter =
-new SqlToOperationConverter(flinkPlanner, catalogManager);
-if (validated instanceof SqlCreateCatalog) {
-return
Optional.of(converter.convertCreateCatalog((SqlCreateCatalog) validated));
-} else if (validated instanceof SqlDropCatalog) {
+
+// delegate conversion to the registered converters first
+SqlNodeConvertContext context = new
SqlNodeConvertContext(flinkPlanner, catalogManager);
+Optional operation =
SqlNodeConverters.convertSqlNode(validated, context);
+if (operation.isPresent()) {
+return operation;
+}
+
+// TODO: all the below conversion logic should be migrated to
SqlNodeConverters
Review Comment:
Do all sqlnodes need to implement?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dmvk commented on a diff in pull request #22169: [FLINK-31399] AdaptiveScheduler is able to handle changes in job resource requirements.
dmvk commented on code in PR #22169:
URL: https://github.com/apache/flink/pull/22169#discussion_r1140236168
##
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/AdaptiveScheduler.java:
##
@@ -1050,6 +1055,14 @@ public CreatingExecutionGraph.AssignmentResult
tryToAssignSlots(
.orElseGet(CreatingExecutionGraph.AssignmentResult::notPossible);
}
+@Override
+public void freeExcessiveReservedSlots() {
+for (SlotInfo slotInfo :
declarativeSlotPool.getFreeSlotsInformation()) {
+declarativeSlotPool.freeReservedSlot(
+slotInfo.getAllocationId(), null,
System.currentTimeMillis());
+}
Review Comment:
(tests are still passing after removing this)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dmvk commented on a diff in pull request #22169: [FLINK-31399] AdaptiveScheduler is able to handle changes in job resource requirements.
dmvk commented on code in PR #22169:
URL: https://github.com/apache/flink/pull/22169#discussion_r1140235833
##
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/AdaptiveScheduler.java:
##
@@ -1050,6 +1055,14 @@ public CreatingExecutionGraph.AssignmentResult
tryToAssignSlots(
.orElseGet(CreatingExecutionGraph.AssignmentResult::notPossible);
}
+@Override
+public void freeExcessiveReservedSlots() {
+for (SlotInfo slotInfo :
declarativeSlotPool.getFreeSlotsInformation()) {
+declarativeSlotPool.freeReservedSlot(
+slotInfo.getAllocationId(), null,
System.currentTimeMillis());
+}
Review Comment:
I went through the chain of differently typed callbacks; it seems that the
reserved slot is already freed when the Execution is canceled (~ when the
ExecutionGraph is canceled), which should make this a no-op. We're guaranteed
the EG is canceled before we create a new one.
Good catch, thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] dmvk commented on a diff in pull request #22169: [FLINK-31399] AdaptiveScheduler is able to handle changes in job resource requirements.
dmvk commented on code in PR #22169:
URL: https://github.com/apache/flink/pull/22169#discussion_r1140223431
##
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/AdaptiveScheduler.java:
##
@@ -1213,4 +1268,17 @@ T transitionToState(StateFactory
targetState) {
State getState() {
return state;
}
+
+/**
+ * Check for slots that are idle for more than {@link
JobManagerOptions#SLOT_IDLE_TIMEOUT} and
+ * release them back to the ResourceManager.
+ */
+private void checkIdleSlotTimeout() {
+declarativeSlotPool.releaseIdleSlots(System.currentTimeMillis());
+getMainThreadExecutor()
+.schedule(
+this::checkIdleSlotTimeout,
Review Comment:
> Could you explain why this method is necessary -don't we release the
excessive slots in CreatingExecutionGraph? What is the scenario?
We mark the slots as idle with the current timestamp and still need to keep
them around until `current timestamp + slot idle timeout`; it's up to the slot
pool to decide when slots should be physically released
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-31499) Move SqlCreateTable conversion logic to SqlCreateTableConverter
[
https://issues.apache.org/jira/browse/FLINK-31499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701719#comment-17701719
]
xuzhiwen commented on FLINK-31499:
--
[~jark] Can you assign it to me.
> Move SqlCreateTable conversion logic to SqlCreateTableConverter
> ---
>
> Key: FLINK-31499
> URL: https://issues.apache.org/jira/browse/FLINK-31499
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: Jark Wu
>Priority: Major
>
> Introduce {{SqlCreateTableConverter}} and move the conversion logic of
> SqlCreateTable -> CreateTableOperation to it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31502) Limit the number of concurrent scale operations to reduce cluster churn
Maximilian Michels created FLINK-31502: -- Summary: Limit the number of concurrent scale operations to reduce cluster churn Key: FLINK-31502 URL: https://issues.apache.org/jira/browse/FLINK-31502 Project: Flink Issue Type: Improvement Components: Autoscaler, Kubernetes Operator Reporter: Maximilian Michels Assignee: Maximilian Michels Fix For: kubernetes-operator-1.5.0 Until we move to using the upcoming Rescale API which recycles pods, we need to be mindful with how many deployments we scale at the same time because each of them is going to give up all its pods and require the new number of required pods. This can cause churn in the cluster and temporary lead to "unallocatable" pods which triggers the k8s cluster autoscaler to add more cluster nodes. That is often not desirable because the actual required resources after the scaling have been settled, are lower. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-31494) Introduce SqlNodeConverter for SqlToOperationConverter
[
https://issues.apache.org/jira/browse/FLINK-31494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701683#comment-17701683
]
Jark Wu commented on FLINK-31494:
-
[~xzw0223], the key idea is not putting all the conversion logic in a single
class, otherwise it's huge and hard to maintain.
> Introduce SqlNodeConverter for SqlToOperationConverter
> --
>
> Key: FLINK-31494
> URL: https://issues.apache.org/jira/browse/FLINK-31494
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: xuzhiwen
>Assignee: Jark Wu
>Priority: Major
> Labels: pull-request-available
>
> Introduce {{SqlNodeConverter}} for {{SqlToOperationConverter}}, following
> Timo's idea in FLINK-31368
> class like:
> {code:java}
> public interface SqlNodeConverter {
> Operation convertSqlNode(S node, ConvertContext context);
> }
> /** Registry of SqlNode converters. */
> public class SqlNodeConverters {
> private static final Map, SqlNodeConverter> CONVERTERS = new
> HashMap<>();
> static {
> // register all the converters here
> register(new SqlCreateCatalogConverter());
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31135) ConfigMap DataSize went > 1 MB and cluster stopped working
[
https://issues.apache.org/jira/browse/FLINK-31135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701681#comment-17701681
]
Maximilian Michels commented on FLINK-31135:
Oh, just realized this is unrelated to FLINK-31345 but a separate config map
issue. Reopening :)
> ConfigMap DataSize went > 1 MB and cluster stopped working
> --
>
> Key: FLINK-31135
> URL: https://issues.apache.org/jira/browse/FLINK-31135
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Affects Versions: kubernetes-operator-1.2.0
>Reporter: Sriram Ganesh
>Priority: Major
>
> I am Flink Operator to manage clusters. Flink version: 1.15.2. Flink jobs
> failed with the below error. It seems the config map size went beyond 1 MB
> (default size).
> Since it is managed by the operator and config maps are not updated with any
> manual intervention, I suspect it could be an operator issue.
>
> {code:java}
> Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure
> executing: PUT at:
> https:///api/v1/namespaces//configmaps/-config-map. Message:
> ConfigMap "-config-map" is invalid: []: Too long: must have at most
> 1048576 bytes. Received status: Status(apiVersion=v1, code=422,
> details=StatusDetails(causes=[StatusCause(field=[], message=Too long: must
> have at most 1048576 bytes, reason=FieldValueTooLong,
> additionalProperties={})], group=null, kind=ConfigMap, name=-config-map,
> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
> message=ConfigMap "-config-map" is invalid: []: Too long: must have at
> most 1048576 bytes, metadata=ListMeta(_continue=null,
> remainingItemCount=null, resourceVersion=null, selfLink=null,
> additionalProperties={}), reason=Invalid, status=Failure,
> additionalProperties={}).
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:673)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:612)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:560)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:521)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleUpdate(OperationSupport.java:347)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleUpdate(OperationSupport.java:327)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleUpdate(BaseOperation.java:781)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:183)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:188)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:130)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:41)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.lambda$attemptCheckAndUpdateConfigMap$11(Fabric8FlinkKubeClient.java:325)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
> ~[?:?]
> ... 3 more {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (FLINK-31135) ConfigMap DataSize went > 1 MB and cluster stopped working
[
https://issues.apache.org/jira/browse/FLINK-31135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maximilian Michels reopened FLINK-31135:
> ConfigMap DataSize went > 1 MB and cluster stopped working
> --
>
> Key: FLINK-31135
> URL: https://issues.apache.org/jira/browse/FLINK-31135
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Affects Versions: kubernetes-operator-1.2.0
>Reporter: Sriram Ganesh
>Priority: Major
>
> I am Flink Operator to manage clusters. Flink version: 1.15.2. Flink jobs
> failed with the below error. It seems the config map size went beyond 1 MB
> (default size).
> Since it is managed by the operator and config maps are not updated with any
> manual intervention, I suspect it could be an operator issue.
>
> {code:java}
> Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure
> executing: PUT at:
> https:///api/v1/namespaces//configmaps/-config-map. Message:
> ConfigMap "-config-map" is invalid: []: Too long: must have at most
> 1048576 bytes. Received status: Status(apiVersion=v1, code=422,
> details=StatusDetails(causes=[StatusCause(field=[], message=Too long: must
> have at most 1048576 bytes, reason=FieldValueTooLong,
> additionalProperties={})], group=null, kind=ConfigMap, name=-config-map,
> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
> message=ConfigMap "-config-map" is invalid: []: Too long: must have at
> most 1048576 bytes, metadata=ListMeta(_continue=null,
> remainingItemCount=null, resourceVersion=null, selfLink=null,
> additionalProperties={}), reason=Invalid, status=Failure,
> additionalProperties={}).
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:673)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:612)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:560)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:521)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleUpdate(OperationSupport.java:347)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleUpdate(OperationSupport.java:327)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleUpdate(BaseOperation.java:781)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:183)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:188)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:130)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:41)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.lambda$attemptCheckAndUpdateConfigMap$11(Fabric8FlinkKubeClient.java:325)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
> ~[?:?]
> ... 3 more {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-31135) ConfigMap DataSize went > 1 MB and cluster stopped working
[
https://issues.apache.org/jira/browse/FLINK-31135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maximilian Michels resolved FLINK-31135.
Resolution: Duplicate
Closing because this has been resolved already on the main branch.
> ConfigMap DataSize went > 1 MB and cluster stopped working
> --
>
> Key: FLINK-31135
> URL: https://issues.apache.org/jira/browse/FLINK-31135
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Affects Versions: kubernetes-operator-1.2.0
>Reporter: Sriram Ganesh
>Priority: Major
>
> I am Flink Operator to manage clusters. Flink version: 1.15.2. Flink jobs
> failed with the below error. It seems the config map size went beyond 1 MB
> (default size).
> Since it is managed by the operator and config maps are not updated with any
> manual intervention, I suspect it could be an operator issue.
>
> {code:java}
> Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure
> executing: PUT at:
> https:///api/v1/namespaces//configmaps/-config-map. Message:
> ConfigMap "-config-map" is invalid: []: Too long: must have at most
> 1048576 bytes. Received status: Status(apiVersion=v1, code=422,
> details=StatusDetails(causes=[StatusCause(field=[], message=Too long: must
> have at most 1048576 bytes, reason=FieldValueTooLong,
> additionalProperties={})], group=null, kind=ConfigMap, name=-config-map,
> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
> message=ConfigMap "-config-map" is invalid: []: Too long: must have at
> most 1048576 bytes, metadata=ListMeta(_continue=null,
> remainingItemCount=null, resourceVersion=null, selfLink=null,
> additionalProperties={}), reason=Invalid, status=Failure,
> additionalProperties={}).
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:673)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:612)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:560)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:521)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleUpdate(OperationSupport.java:347)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleUpdate(OperationSupport.java:327)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleUpdate(BaseOperation.java:781)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:183)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:188)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:130)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:41)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.lambda$attemptCheckAndUpdateConfigMap$11(Fabric8FlinkKubeClient.java:325)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
> ~[?:?]
> ... 3 more {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31135) ConfigMap DataSize went > 1 MB and cluster stopped working
[
https://issues.apache.org/jira/browse/FLINK-31135?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701679#comment-17701679
]
Maximilian Michels commented on FLINK-31135:
This has been addressed in FLINK-31345.
> ConfigMap DataSize went > 1 MB and cluster stopped working
> --
>
> Key: FLINK-31135
> URL: https://issues.apache.org/jira/browse/FLINK-31135
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Affects Versions: kubernetes-operator-1.2.0
>Reporter: Sriram Ganesh
>Priority: Major
>
> I am Flink Operator to manage clusters. Flink version: 1.15.2. Flink jobs
> failed with the below error. It seems the config map size went beyond 1 MB
> (default size).
> Since it is managed by the operator and config maps are not updated with any
> manual intervention, I suspect it could be an operator issue.
>
> {code:java}
> Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure
> executing: PUT at:
> https:///api/v1/namespaces//configmaps/-config-map. Message:
> ConfigMap "-config-map" is invalid: []: Too long: must have at most
> 1048576 bytes. Received status: Status(apiVersion=v1, code=422,
> details=StatusDetails(causes=[StatusCause(field=[], message=Too long: must
> have at most 1048576 bytes, reason=FieldValueTooLong,
> additionalProperties={})], group=null, kind=ConfigMap, name=-config-map,
> retryAfterSeconds=null, uid=null, additionalProperties={}), kind=Status,
> message=ConfigMap "-config-map" is invalid: []: Too long: must have at
> most 1048576 bytes, metadata=ListMeta(_continue=null,
> remainingItemCount=null, resourceVersion=null, selfLink=null,
> additionalProperties={}), reason=Invalid, status=Failure,
> additionalProperties={}).
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.requestFailure(OperationSupport.java:673)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.assertResponseCode(OperationSupport.java:612)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:560)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleResponse(OperationSupport.java:521)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleUpdate(OperationSupport.java:347)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.OperationSupport.handleUpdate(OperationSupport.java:327)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.BaseOperation.handleUpdate(BaseOperation.java:781)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.lambda$replace$1(HasMetadataOperation.java:183)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:188)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:130)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> io.fabric8.kubernetes.client.dsl.base.HasMetadataOperation.replace(HasMetadataOperation.java:41)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient.lambda$attemptCheckAndUpdateConfigMap$11(Fabric8FlinkKubeClient.java:325)
> ~[flink-dist-1.15.2.jar:1.15.2]
> at
> java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
> ~[?:?]
> ... 3 more {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] luoyuxia commented on a diff in pull request #22202: [FLINK-29177][hive] Shade org.apache.commons.lang3 in flink-sql-connector-hive to avoid potential class conflict
luoyuxia commented on code in PR #22202: URL: https://github.com/apache/flink/pull/22202#discussion_r1140150472 ## flink-connectors/flink-sql-connector-hive-2.3.9/pom.xml: ## @@ -115,6 +115,10 @@ under the License. com.google org.apache.flink.hive.shaded.com.google + + org.apache.commons.lang3 + org.apache.flink.hive.shaded.org.apache.commons.lang3 Review Comment: I'm not intended to do so. For the case that the hive depends some class like `org.apache.common.xxx` but it hasn't shiped the class `org.apache.common.xxx`. It's provided by hadoop-lib. Without shadding, when we put hadoop-lib, hive-connector together, it works. But after shade, it won't works as the class `org.apache.common.xxx` has been shaded to `org.flink.shade.`, and hadoop-lib can't provide it now. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
leonardBang commented on code in PR #618: URL: https://github.com/apache/flink-web/pull/618#discussion_r1140149272 ## docs/content/posts/2023-03-09-release-1.17.0.md: ## @@ -0,0 +1,485 @@ +--- +authors: +- LeonardXu: + name: "Leonard Xu" + twitter: Leonardxbj +date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images +subtitle: "" +title: Announcing the Release of Apache Flink 1.17 +aliases: +- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date +--- + +The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache +Flink is the leading stream processing standard, and the concept of unified +stream and batch data processing is being successfully adopted in more and more +companies. Thanks to our excellent community and contributors, Apache Flink +continues to grow as a technology and remains one of the most active projects in +the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically +participating and saw the completion of 7 FLIPs and 600+ issues, bringing many +exciting new features and improvements to the community. + + +# Towards Streaming Warehouses + +In order to achieve greater efficiency in the realm of [streaming +warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821), +Flink 1.17 contains substantial improvements to both the performance of batch +processing and the semantics of streaming processing. These improvements +represent a significant stride towards the creation of a more efficient and +streamlined data warehouse, capable of processing large quantities of data in +real-time. + +For batch processing, this release includes several new features and +improvements: + +* **Streaming Warehouse API:** + [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061) + introduces the new Delete and Update API in Flink SQL which works in(only) batch + mode. External storage systems like Flink Table Store can implement row-level + updates via this new API. The ALTER TABLE syntax is enhanced by including the + ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it + easier for users to maintain their table schema. +* **Batch Execution Improvements:** Execution of batch workloads has been + significantly improved in Flink 1.17 in terms of performance, stability and + usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved + with strategy and operator optimizations, such as new join reordering and adaptive + local hash aggregation, Hive aggregate functions improvements, and the hybrid + shuffle mode enhancements. Stability wise, speculative execution now supports + all operators, and the Adaptive Batch Scheduler is more robust against data + skew. Usability wise, the tuning effort required for batch workloads has been + reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode. + The hybrid shuffle is compatible with speculative execution and the Adaptive + Batch Scheduler, next to various configuration simplifications. +* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for + SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced + functionality. Users can use SQL statements to manage job lifecycles, + including displaying job information and stopping running jobs. This provides + a powerful tool for managing Flink jobs. + +For stream processing, the following features and improvements are realized: + +* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect + results or exceptions which is a challenging topic in streaming SQL. Incorrect + optimization plans and functional issues have been fixed, and the experimental + feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails) + is introduced to inform of potential correctness risks and optimization + suggestions to SQL users. +* **Checkpoint Improvements:** The generic incremental checkpoint improvements + enhance the speed and stability of the checkpoint procedure, and the unaligned + checkpoint has improved stability under backpressure and is production-ready + in Flink 1.17. Users can manually trigger checkpoints with self-defined + checkpoint types while a job is running with the newly introduced REST + interface for triggering checkpoints. +* **Watermark Alignment Enhancement:** Efficient watermark processing directly + affects the execution efficiency of event time applications. In Flink 1.17, + [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits) + introduces an improvement to watermark alignment by aligning data emission + across splits within a source op
[GitHub] [flink] luoyuxia commented on a diff in pull request #22202: [FLINK-29177][hive] Shade org.apache.commons.lang3 in flink-sql-connector-hive to avoid potential class conflict
luoyuxia commented on code in PR #22202: URL: https://github.com/apache/flink/pull/22202#discussion_r1140146575 ## flink-connectors/flink-sql-connector-hive-2.3.9/pom.xml: ## @@ -115,6 +115,10 @@ under the License. com.google org.apache.flink.hive.shaded.com.google + Review Comment: Sure. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] luoyuxia commented on a diff in pull request #22197: [FLINK-31487][table-planner] Add targetColumns to DynamicTableSink#Context
luoyuxia commented on code in PR #22197:
URL: https://github.com/apache/flink/pull/22197#discussion_r1139990233
##
flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/sink/DynamicTableSink.java:
##
@@ -167,6 +168,26 @@ interface Context {
* @see LogicalType#supportsOutputConversion(Class)
*/
DataStructureConverter createDataStructureConverter(DataType
consumedDataType);
+
+/**
+ * Returns an {@link Optional} array of column index paths related to
user specified target
+ * column list or {@link Optional#empty()} when not specified. The
array indices are 0-based
+ * and support composite columns within (possibly nested) structures.
+ *
+ * This information comes from the column list of the DML clause,
e.g., for a sink table
+ * t1 which schema is: {@code a STRING, b ROW < b1 INT, b2 STRING>, c
BIGINT}
+ *
+ *
+ * insert: 'insert into t1(a, b.b2) ...', the column list will
be 'a, b.b2', and will
+ * return {@code [[0], [1, 1]]}. The statement 'insert into
target select ...' without
+ * specifying a column list will return {@link Optional#empty()}.
+ * update: 'update target set a=1, b.b1=2 where ...', the column
list will be 'a,
Review Comment:
nit
```suggestion
* update: 'update t1 set a=1, b.b1=2 where ...', the column
list will be 'a,
```
##
flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/sink/DynamicTableSink.java:
##
@@ -167,6 +168,26 @@ interface Context {
* @see LogicalType#supportsOutputConversion(Class)
*/
DataStructureConverter createDataStructureConverter(DataType
consumedDataType);
+
+/**
+ * Returns an {@link Optional} array of column index paths related to
user specified target
+ * column list or {@link Optional#empty()} when not specified. The
array indices are 0-based
+ * and support composite columns within (possibly nested) structures.
+ *
+ * This information comes from the column list of the DML clause,
e.g., for a sink table
+ * t1 which schema is: {@code a STRING, b ROW < b1 INT, b2 STRING>, c
BIGINT}
+ *
+ *
+ * insert: 'insert into t1(a, b.b2) ...', the column list will
be 'a, b.b2', and will
+ * return {@code [[0], [1, 1]]}. The statement 'insert into
target select ...' without
+ * specifying a column list will return {@link Optional#empty()}.
+ * update: 'update target set a=1, b.b1=2 where ...', the column
list will be 'a,
+ * b.b1', will return {@code [[0], [1, 0]]}.
Review Comment:
```suggestion
* b.b1', and will return {@code [[0], [1, 0]]}.
```
##
flink-table/flink-table-common/src/main/java/org/apache/flink/table/connector/sink/DynamicTableSink.java:
##
@@ -167,6 +168,26 @@ interface Context {
* @see LogicalType#supportsOutputConversion(Class)
*/
DataStructureConverter createDataStructureConverter(DataType
consumedDataType);
+
+/**
+ * Returns an {@link Optional} array of column index paths related to
user specified target
+ * column list or {@link Optional#empty()} when not specified. The
array indices are 0-based
+ * and support composite columns within (possibly nested) structures.
+ *
+ * This information comes from the column list of the DML clause,
e.g., for a sink table
+ * t1 which schema is: {@code a STRING, b ROW < b1 INT, b2 STRING>, c
BIGINT}
+ *
+ *
+ * insert: 'insert into t1(a, b.b2) ...', the column list will
be 'a, b.b2', and will
+ * return {@code [[0], [1, 1]]}. The statement 'insert into
target select ...' without
Review Comment:
```suggestion
* return {@code [[0], [1, 1]]}. The statement 'insert into t1
select ...' without
```
?
##
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/SinkModifyOperation.java:
##
@@ -121,6 +135,11 @@ public QueryOperation getChild() {
return child;
}
+/** return an empty array when no column list specified. */
Review Comment:
From the comments, seems like it should be
```
if (targetColumns == null) {
return new int[0][0];;
} else {
return targetColumns
}
```
?
##
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/operations/SqlToOperationConverter.java:
##
@@ -1558,8 +1572,16 @@ private Operation convertUpdate(SqlUpdate sqlUpdate) {
catalogManager.qualifyIdentifier(unresolvedTableIdentifier));
// get query
PlannerQueryOperation queryOperation = new
PlannerQueryOperation(tableModify);
+//
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
leonardBang commented on code in PR #618: URL: https://github.com/apache/flink-web/pull/618#discussion_r1140145036 ## docs/content/posts/2023-03-09-release-1.17.0.md: ## @@ -0,0 +1,485 @@ +--- +authors: +- LeonardXu: + name: "Leonard Xu" + twitter: Leonardxbj +date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images +subtitle: "" +title: Announcing the Release of Apache Flink 1.17 +aliases: +- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date +--- + +The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache +Flink is the leading stream processing standard, and the concept of unified +stream and batch data processing is being successfully adopted in more and more +companies. Thanks to our excellent community and contributors, Apache Flink +continues to grow as a technology and remains one of the most active projects in +the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically +participating and saw the completion of 7 FLIPs and 600+ issues, bringing many +exciting new features and improvements to the community. + + +# Towards Streaming Warehouses + +In order to achieve greater efficiency in the realm of [streaming +warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821), +Flink 1.17 contains substantial improvements to both the performance of batch +processing and the semantics of streaming processing. These improvements +represent a significant stride towards the creation of a more efficient and +streamlined data warehouse, capable of processing large quantities of data in +real-time. + +For batch processing, this release includes several new features and +improvements: + +* **Streaming Warehouse API:** + [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061) + introduces the new Delete and Update API in Flink SQL which works in(only) batch + mode. External storage systems like Flink Table Store can implement row-level + updates via this new API. The ALTER TABLE syntax is enhanced by including the + ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it + easier for users to maintain their table schema. +* **Batch Execution Improvements:** Execution of batch workloads has been + significantly improved in Flink 1.17 in terms of performance, stability and + usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved + with strategy and operator optimizations, such as new join reordering and adaptive + local hash aggregation, Hive aggregate functions improvements, and the hybrid + shuffle mode enhancements. Stability wise, speculative execution now supports + all operators, and the Adaptive Batch Scheduler is more robust against data + skew. Usability wise, the tuning effort required for batch workloads has been + reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode. + The hybrid shuffle is compatible with speculative execution and the Adaptive + Batch Scheduler, next to various configuration simplifications. +* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for + SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced + functionality. Users can use SQL statements to manage job lifecycles, + including displaying job information and stopping running jobs. This provides + a powerful tool for managing Flink jobs. + +For stream processing, the following features and improvements are realized: + +* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect + results or exceptions which is a challenging topic in streaming SQL. Incorrect + optimization plans and functional issues have been fixed, and the experimental + feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails) + is introduced to inform of potential correctness risks and optimization + suggestions to SQL users. +* **Checkpoint Improvements:** The generic incremental checkpoint improvements + enhance the speed and stability of the checkpoint procedure, and the unaligned + checkpoint has improved stability under backpressure and is production-ready + in Flink 1.17. Users can manually trigger checkpoints with self-defined + checkpoint types while a job is running with the newly introduced REST + interface for triggering checkpoints. +* **Watermark Alignment Enhancement:** Efficient watermark processing directly + affects the execution efficiency of event time applications. In Flink 1.17, + [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits) + introduces an improvement to watermark alignment by aligning data emission + across splits within a source
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
leonardBang commented on code in PR #618: URL: https://github.com/apache/flink-web/pull/618#discussion_r1140138561 ## docs/content/posts/2023-03-09-release-1.17.0.md: ## @@ -0,0 +1,485 @@ +--- +authors: +- LeonardXu: + name: "Leonard Xu" + twitter: Leonardxbj +date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images +subtitle: "" +title: Announcing the Release of Apache Flink 1.17 +aliases: +- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date +--- + +The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache +Flink is the leading stream processing standard, and the concept of unified +stream and batch data processing is being successfully adopted in more and more +companies. Thanks to our excellent community and contributors, Apache Flink +continues to grow as a technology and remains one of the most active projects in +the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically +participating and saw the completion of 7 FLIPs and 600+ issues, bringing many +exciting new features and improvements to the community. + + +# Towards Streaming Warehouses + +In order to achieve greater efficiency in the realm of [streaming +warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821), +Flink 1.17 contains substantial improvements to both the performance of batch +processing and the semantics of streaming processing. These improvements +represent a significant stride towards the creation of a more efficient and +streamlined data warehouse, capable of processing large quantities of data in +real-time. + +For batch processing, this release includes several new features and +improvements: + +* **Streaming Warehouse API:** + [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061) + introduces the new Delete and Update API in Flink SQL which works in(only) batch + mode. External storage systems like Flink Table Store can implement row-level + updates via this new API. The ALTER TABLE syntax is enhanced by including the + ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it + easier for users to maintain their table schema. +* **Batch Execution Improvements:** Execution of batch workloads has been + significantly improved in Flink 1.17 in terms of performance, stability and + usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved + with strategy and operator optimizations, such as new join reordering and adaptive + local hash aggregation, Hive aggregate functions improvements, and the hybrid + shuffle mode enhancements. Stability wise, speculative execution now supports + all operators, and the Adaptive Batch Scheduler is more robust against data + skew. Usability wise, the tuning effort required for batch workloads has been + reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode. + The hybrid shuffle is compatible with speculative execution and the Adaptive + Batch Scheduler, next to various configuration simplifications. +* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for + SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced + functionality. Users can use SQL statements to manage job lifecycles, + including displaying job information and stopping running jobs. This provides + a powerful tool for managing Flink jobs. + +For stream processing, the following features and improvements are realized: + +* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect + results or exceptions which is a challenging topic in streaming SQL. Incorrect + optimization plans and functional issues have been fixed, and the experimental + feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails) + is introduced to inform of potential correctness risks and optimization + suggestions to SQL users. +* **Checkpoint Improvements:** The generic incremental checkpoint improvements + enhance the speed and stability of the checkpoint procedure, and the unaligned + checkpoint has improved stability under backpressure and is production-ready + in Flink 1.17. Users can manually trigger checkpoints with self-defined + checkpoint types while a job is running with the newly introduced REST + interface for triggering checkpoints. +* **Watermark Alignment Enhancement:** Efficient watermark processing directly + affects the execution efficiency of event time applications. In Flink 1.17, + [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits) + introduces an improvement to watermark alignment by aligning data emission + across splits within a source
[GitHub] [flink-web] leonardBang commented on a diff in pull request #618: Announcement blogpost for the 1.17 release
leonardBang commented on code in PR #618: URL: https://github.com/apache/flink-web/pull/618#discussion_r1140137456 ## docs/content/posts/2023-03-09-release-1.17.0.md: ## @@ -0,0 +1,485 @@ +--- +authors: +- LeonardXu: + name: "Leonard Xu" + twitter: Leonardxbj +date: "2023-03-09T08:00:00Z" #FIXME: Change to the actual release date, also the date in the filename, and the directory name of linked images +subtitle: "" +title: Announcing the Release of Apache Flink 1.17 +aliases: +- /news/2023/03/09/release-1.17.0.html #FIXME: Change to the actual release date +--- + +The Apache Flink PMC is pleased to announce Apache Flink release 1.17.0. Apache +Flink is the leading stream processing standard, and the concept of unified +stream and batch data processing is being successfully adopted in more and more +companies. Thanks to our excellent community and contributors, Apache Flink +continues to grow as a technology and remains one of the most active projects in +the Apache Software Foundation. Flink 1.17 had 173 contributors enthusiastically +participating and saw the completion of 7 FLIPs and 600+ issues, bringing many +exciting new features and improvements to the community. + + +# Towards Streaming Warehouses + +In order to achieve greater efficiency in the realm of [streaming +warehouse](https://www.alibabacloud.com/blog/more-than-computing-a-new-era-led-by-the-warehouse-architecture-of-apache-flink_598821), +Flink 1.17 contains substantial improvements to both the performance of batch +processing and the semantics of streaming processing. These improvements +represent a significant stride towards the creation of a more efficient and +streamlined data warehouse, capable of processing large quantities of data in +real-time. + +For batch processing, this release includes several new features and +improvements: + +* **Streaming Warehouse API:** + [FLIP-282](https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=235838061) + introduces the new Delete and Update API in Flink SQL which works in(only) batch + mode. External storage systems like Flink Table Store can implement row-level + updates via this new API. The ALTER TABLE syntax is enhanced by including the + ability to ADD/MODIFY/DROP columns, primary keys, and watermarks, making it + easier for users to maintain their table schema. +* **Batch Execution Improvements:** Execution of batch workloads has been + significantly improved in Flink 1.17 in terms of performance, stability and + usability. Performance wise, a 26% TPC-DS improvement on 10T dataset is achieved + with strategy and operator optimizations, such as new join reordering and adaptive + local hash aggregation, Hive aggregate functions improvements, and the hybrid + shuffle mode enhancements. Stability wise, speculative execution now supports + all operators, and the Adaptive Batch Scheduler is more robust against data + skew. Usability wise, the tuning effort required for batch workloads has been + reduced. The Adaptive Batch Scheduler is now the default scheduler in batch mode. + The hybrid shuffle is compatible with speculative execution and the Adaptive + Batch Scheduler, next to various configuration simplifications. +* **SQL Client/Gateway:** Apache Flink 1.17 introduces the "gateway mode" for + SQL Client, allowing users to submit SQL queries to a SQL Gateway for enhanced + functionality. Users can use SQL statements to manage job lifecycles, + including displaying job information and stopping running jobs. This provides + a powerful tool for managing Flink jobs. + +For stream processing, the following features and improvements are realized: + +* **Streaming SQL Semantics:** Non-deterministic operations may bring incorrect + results or exceptions which is a challenging topic in streaming SQL. Incorrect + optimization plans and functional issues have been fixed, and the experimental + feature of [PLAN_ADVICE](https://nightlies.apache.org/flink/flink-docs-release-1.17/docs/dev/table/sql/explain/#explaindetails) + is introduced to inform of potential correctness risks and optimization + suggestions to SQL users. +* **Checkpoint Improvements:** The generic incremental checkpoint improvements + enhance the speed and stability of the checkpoint procedure, and the unaligned + checkpoint has improved stability under backpressure and is production-ready + in Flink 1.17. Users can manually trigger checkpoints with self-defined + checkpoint types while a job is running with the newly introduced REST + interface for triggering checkpoints. +* **Watermark Alignment Enhancement:** Efficient watermark processing directly + affects the execution efficiency of event time applications. In Flink 1.17, + [FLIP-217](https://cwiki.apache.org/confluence/display/FLINK/FLIP-217%3A+Support+watermark+alignment+of+source+splits) + introduces an improvement to watermark alignment by aligning data emission + across splits within a source
[jira] [Created] (FLINK-31501) Move SqlCreateView conversion logic to SqlCreateViewConverter
Jark Wu created FLINK-31501:
---
Summary: Move SqlCreateView conversion logic to
SqlCreateViewConverter
Key: FLINK-31501
URL: https://issues.apache.org/jira/browse/FLINK-31501
Project: Flink
Issue Type: Sub-task
Components: Table SQL / Planner
Reporter: Jark Wu
Introduce {{SqlCreateViewConverter}} and move the conversion logic of
SqlCreateView -> CreateViewOperation to it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31500) Move SqlAlterTableSchema conversion logic to AlterTableSchemaConverter
Jark Wu created FLINK-31500:
---
Summary: Move SqlAlterTableSchema conversion logic to
AlterTableSchemaConverter
Key: FLINK-31500
URL: https://issues.apache.org/jira/browse/FLINK-31500
Project: Flink
Issue Type: Sub-task
Components: Table SQL / Planner
Reporter: Jark Wu
Introduce {{AlterTableSchemaConverter}} and move the conversion logic of
SqlAlterTableSchema -> AlterTableChangeOperation to it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-31499) Move SqlCreateTable conversion logic to SqlCreateTableConverter
Jark Wu created FLINK-31499:
---
Summary: Move SqlCreateTable conversion logic to
SqlCreateTableConverter
Key: FLINK-31499
URL: https://issues.apache.org/jira/browse/FLINK-31499
Project: Flink
Issue Type: Sub-task
Components: Table SQL / Planner
Reporter: Jark Wu
Introduce {{SqlCreateTableConverter}} and move the conversion logic of
SqlCreateTable -> CreateTableOperation to it.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #22205: [FLINK-31494][table-planner] Introduce SqlNodeConverter for SqlToOperationConverter
flinkbot commented on PR #22205: URL: https://github.com/apache/flink/pull/22205#issuecomment-1473709150 ## CI report: * c91d41a200babd67477dd8288f41cf77936ee318 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong commented on pull request #22205: [FLINK-31494][table-planner] Introduce SqlNodeConverter for SqlToOperationConverter
wuchong commented on PR #22205: URL: https://github.com/apache/flink/pull/22205#issuecomment-1473704387 @luoyuxia @xuzhiwen1255 , would be appreciated if you could help to review. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31494) Introduce SqlNodeConverter for SqlToOperationConverter
[
https://issues.apache.org/jira/browse/FLINK-31494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-31494:
---
Labels: pull-request-available (was: )
> Introduce SqlNodeConverter for SqlToOperationConverter
> --
>
> Key: FLINK-31494
> URL: https://issues.apache.org/jira/browse/FLINK-31494
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: xuzhiwen
>Assignee: Jark Wu
>Priority: Major
> Labels: pull-request-available
>
> Introduce {{SqlNodeConverter}} for {{SqlToOperationConverter}}, following
> Timo's idea in FLINK-31368
> class like:
> {code:java}
> public interface SqlNodeConverter {
> Operation convertSqlNode(S node, ConvertContext context);
> }
> /** Registry of SqlNode converters. */
> public class SqlNodeConverters {
> private static final Map, SqlNodeConverter> CONVERTERS = new
> HashMap<>();
> static {
> // register all the converters here
> register(new SqlCreateCatalogConverter());
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] wuchong opened a new pull request, #22205: [FLINK-31494][table-planner] Introduce SqlNodeConverter for SqlToOperationConverter
wuchong opened a new pull request, #22205: URL: https://github.com/apache/flink/pull/22205 ## What is the purpose of the change Similar to [FLINK-31368](https://issues.apache.org/jira/browse/FLINK-31368), the `SqlToOperationConverter` is a bit bloated. This PR is aiming to introduce `SqlNodeConverter` and move conversion logic from `SqlToOperationConverter` in the following PRs. ## Brief change log - Rename `SqlToOperationConverter` into `SqlNodeToOperationConversion`, because the Converter name may confuse developers regarding the relationship with `SqlNodeConverter` - Introduce `SqlNodeConverter` interface and make `SqlToOperationConverter` supports it. - Migrate the conversion logic of `SqlCreateCatalog` to verify the happy path. ## Verifying this change This change is already covered by existing tests, such as *(please describe tests)*. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-31489) OperationManagerTest.testCloseOperation failed because it couldn't find a submitted operation
[
https://issues.apache.org/jira/browse/FLINK-31489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated FLINK-31489:
---
Fix Version/s: 1.17.1
> OperationManagerTest.testCloseOperation failed because it couldn't find a
> submitted operation
> -
>
> Key: FLINK-31489
> URL: https://issues.apache.org/jira/browse/FLINK-31489
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Gateway
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.18.0, 1.17.1
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=47218&view=logs&j=ce3801ad-3bd5-5f06-d165-34d37e757d90&t=5e4d9387-1dcc-5885-a901-90469b7e6d2f&l=13386
> {code}
> Mar 16 02:49:52 [ERROR] Tests run: 7, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 30.433 s <<< FAILURE! - in
> org.apache.flink.table.gateway.service.operation.OperationManagerTest
> Mar 16 02:49:52 [ERROR]
> org.apache.flink.table.gateway.service.operation.OperationManagerTest.testCloseOperation
> Time elapsed: 0.042 s <<< ERROR!
> Mar 16 02:49:52 org.apache.flink.table.gateway.api.utils.SqlGatewayException:
> Can not find the submitted operation in the OperationManager with the
> 1734d6cf-cf52-40c5-804f-809e48a9818a.
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManager.lambda$getOperation$3(OperationManager.java:487)
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManager.readLock(OperationManager.java:518)
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManager.getOperation(OperationManager.java:482)
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManager.awaitOperationTermination(OperationManager.java:149)
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManagerTest.testCloseOperation(OperationManagerTest.java:199)
> Mar 16 02:49:52 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (FLINK-31489) OperationManagerTest.testCloseOperation failed because it couldn't find a submitted operation
[
https://issues.apache.org/jira/browse/FLINK-31489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu resolved FLINK-31489.
Fix Version/s: 1.18.0
(was: 1.17.0)
Resolution: Fixed
> OperationManagerTest.testCloseOperation failed because it couldn't find a
> submitted operation
> -
>
> Key: FLINK-31489
> URL: https://issues.apache.org/jira/browse/FLINK-31489
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Gateway
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.18.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=47218&view=logs&j=ce3801ad-3bd5-5f06-d165-34d37e757d90&t=5e4d9387-1dcc-5885-a901-90469b7e6d2f&l=13386
> {code}
> Mar 16 02:49:52 [ERROR] Tests run: 7, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 30.433 s <<< FAILURE! - in
> org.apache.flink.table.gateway.service.operation.OperationManagerTest
> Mar 16 02:49:52 [ERROR]
> org.apache.flink.table.gateway.service.operation.OperationManagerTest.testCloseOperation
> Time elapsed: 0.042 s <<< ERROR!
> Mar 16 02:49:52 org.apache.flink.table.gateway.api.utils.SqlGatewayException:
> Can not find the submitted operation in the OperationManager with the
> 1734d6cf-cf52-40c5-804f-809e48a9818a.
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManager.lambda$getOperation$3(OperationManager.java:487)
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManager.readLock(OperationManager.java:518)
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManager.getOperation(OperationManager.java:482)
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManager.awaitOperationTermination(OperationManager.java:149)
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManagerTest.testCloseOperation(OperationManagerTest.java:199)
> Mar 16 02:49:52 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-31489) OperationManagerTest.testCloseOperation failed because it couldn't find a submitted operation
[
https://issues.apache.org/jira/browse/FLINK-31489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701652#comment-17701652
]
Leonard Xu edited comment on FLINK-31489 at 3/17/23 11:35 AM:
--
Fixed in :
master: ff08c7b1d7fb9334d624aa74bc5788d4d7ac1edc
release-1.17: f0a7dcc4a1c4b3c4b2e4e7029e8f60fb7e7720d2
was (Author: leonard xu):
Fixed in :
master: ff08c7b1d7fb9334d624aa74bc5788d4d7ac1edc
release-1.17: TODO
> OperationManagerTest.testCloseOperation failed because it couldn't find a
> submitted operation
> -
>
> Key: FLINK-31489
> URL: https://issues.apache.org/jira/browse/FLINK-31489
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Gateway
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.17.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=47218&view=logs&j=ce3801ad-3bd5-5f06-d165-34d37e757d90&t=5e4d9387-1dcc-5885-a901-90469b7e6d2f&l=13386
> {code}
> Mar 16 02:49:52 [ERROR] Tests run: 7, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 30.433 s <<< FAILURE! - in
> org.apache.flink.table.gateway.service.operation.OperationManagerTest
> Mar 16 02:49:52 [ERROR]
> org.apache.flink.table.gateway.service.operation.OperationManagerTest.testCloseOperation
> Time elapsed: 0.042 s <<< ERROR!
> Mar 16 02:49:52 org.apache.flink.table.gateway.api.utils.SqlGatewayException:
> Can not find the submitted operation in the OperationManager with the
> 1734d6cf-cf52-40c5-804f-809e48a9818a.
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManager.lambda$getOperation$3(OperationManager.java:487)
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManager.readLock(OperationManager.java:518)
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManager.getOperation(OperationManager.java:482)
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManager.awaitOperationTermination(OperationManager.java:149)
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManagerTest.testCloseOperation(OperationManagerTest.java:199)
> Mar 16 02:49:52 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] leonardBang merged pull request #22200: [FLINK-31489][sql-gateway] Fix instable OperationManagerTest.testCloseOperation
leonardBang merged PR #22200: URL: https://github.com/apache/flink/pull/22200 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zoltar9264 commented on a diff in pull request #21822: [FLINK-30863][state] Register local recovery files of changelog before notifyCheckpointComplete()
zoltar9264 commented on code in PR #21822:
URL: https://github.com/apache/flink/pull/21822#discussion_r1139825377
##
flink-dstl/flink-dstl-dfs/src/main/java/org/apache/flink/changelog/fs/DuplicatingStateChangeFsUploader.java:
##
@@ -51,12 +52,12 @@
* Store the meta of files into {@link ChangelogTaskLocalStateStore}
by
* AsyncCheckpointRunnable#reportCompletedSnapshotStates().
* Pass control of the file to {@link
LocalChangelogRegistry#register} when
- * ChangelogKeyedStateBackend#notifyCheckpointComplete() , files of
the previous
- * checkpoint will be deleted by {@link
LocalChangelogRegistry#discardUpToCheckpoint} at
- * the same time.
+ * ChangelogKeyedStateBackend#buildSnapshotResult , files of the
previous checkpoint will
Review Comment:
Control transfer is no longer when
`ChangelogKeyedStateBackend#buildSnapshotResult`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] zoltar9264 commented on a diff in pull request #21822: [FLINK-30863][state] Register local recovery files of changelog before notifyCheckpointComplete()
zoltar9264 commented on code in PR #21822:
URL: https://github.com/apache/flink/pull/21822#discussion_r1139807707
##
flink-dstl/flink-dstl-dfs/src/main/java/org/apache/flink/changelog/fs/FsStateChangelogWriter.java:
##
@@ -199,15 +200,21 @@ public SequenceNumber nextSequenceNumber() {
return activeSequenceNumber;
}
+@VisibleForTesting
+CompletableFuture>
persist(SequenceNumber from)
+throws IOException {
+return persist(from, DUMMY_PERSIST_CHECKPOINT);
+}
+
@Override
public CompletableFuture>
persist(
-SequenceNumber from) throws IOException {
+SequenceNumber from, long checkpointId) throws IOException {
LOG.debug(
"persist {} starting from sqn {} (incl.), active sqn: {}",
Review Comment:
we can add checkpointId to debug log
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-31489) OperationManagerTest.testCloseOperation failed because it couldn't find a submitted operation
[
https://issues.apache.org/jira/browse/FLINK-31489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701652#comment-17701652
]
Leonard Xu commented on FLINK-31489:
Fixed in :
master: ff08c7b1d7fb9334d624aa74bc5788d4d7ac1edc
release-1.17: TODO
> OperationManagerTest.testCloseOperation failed because it couldn't find a
> submitted operation
> -
>
> Key: FLINK-31489
> URL: https://issues.apache.org/jira/browse/FLINK-31489
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Gateway
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Assignee: lincoln lee
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.17.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=47218&view=logs&j=ce3801ad-3bd5-5f06-d165-34d37e757d90&t=5e4d9387-1dcc-5885-a901-90469b7e6d2f&l=13386
> {code}
> Mar 16 02:49:52 [ERROR] Tests run: 7, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 30.433 s <<< FAILURE! - in
> org.apache.flink.table.gateway.service.operation.OperationManagerTest
> Mar 16 02:49:52 [ERROR]
> org.apache.flink.table.gateway.service.operation.OperationManagerTest.testCloseOperation
> Time elapsed: 0.042 s <<< ERROR!
> Mar 16 02:49:52 org.apache.flink.table.gateway.api.utils.SqlGatewayException:
> Can not find the submitted operation in the OperationManager with the
> 1734d6cf-cf52-40c5-804f-809e48a9818a.
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManager.lambda$getOperation$3(OperationManager.java:487)
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManager.readLock(OperationManager.java:518)
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManager.getOperation(OperationManager.java:482)
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManager.awaitOperationTermination(OperationManager.java:149)
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManagerTest.testCloseOperation(OperationManagerTest.java:199)
> Mar 16 02:49:52 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-31489) OperationManagerTest.testCloseOperation failed because it couldn't find a submitted operation
[
https://issues.apache.org/jira/browse/FLINK-31489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated FLINK-31489:
---
Priority: Major (was: Critical)
> OperationManagerTest.testCloseOperation failed because it couldn't find a
> submitted operation
> -
>
> Key: FLINK-31489
> URL: https://issues.apache.org/jira/browse/FLINK-31489
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Gateway
>Affects Versions: 1.17.0
>Reporter: Matthias Pohl
>Assignee: lincoln lee
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.17.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=47218&view=logs&j=ce3801ad-3bd5-5f06-d165-34d37e757d90&t=5e4d9387-1dcc-5885-a901-90469b7e6d2f&l=13386
> {code}
> Mar 16 02:49:52 [ERROR] Tests run: 7, Failures: 0, Errors: 1, Skipped: 0,
> Time elapsed: 30.433 s <<< FAILURE! - in
> org.apache.flink.table.gateway.service.operation.OperationManagerTest
> Mar 16 02:49:52 [ERROR]
> org.apache.flink.table.gateway.service.operation.OperationManagerTest.testCloseOperation
> Time elapsed: 0.042 s <<< ERROR!
> Mar 16 02:49:52 org.apache.flink.table.gateway.api.utils.SqlGatewayException:
> Can not find the submitted operation in the OperationManager with the
> 1734d6cf-cf52-40c5-804f-809e48a9818a.
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManager.lambda$getOperation$3(OperationManager.java:487)
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManager.readLock(OperationManager.java:518)
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManager.getOperation(OperationManager.java:482)
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManager.awaitOperationTermination(OperationManager.java:149)
> Mar 16 02:49:52 at
> org.apache.flink.table.gateway.service.operation.OperationManagerTest.testCloseOperation(OperationManagerTest.java:199)
> Mar 16 02:49:52 at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> [...]
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] leonardBang merged pull request #22199: [FLINK-31489][sql-gateway] Fix instable OperationManagerTest.testCloseOperation
leonardBang merged PR #22199: URL: https://github.com/apache/flink/pull/22199 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Aitozi commented on a diff in pull request #22179: [FLINK-31380][table] FLIP-297: Support enhanced show catalogs syntax
Aitozi commented on code in PR #22179:
URL: https://github.com/apache/flink/pull/22179#discussion_r1140063123
##
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/SupportsShowLikeOperation.java:
##
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.operations;
+
+import org.apache.flink.table.operations.utils.OperationLikeType;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/** Abstract show operation supports filter with like. */
+public abstract class SupportsShowLikeOperation implements ShowOperation {
+
+// different like type such as like, ilike
+private final OperationLikeType likeType;
+private final boolean notLike;
+private final String likePattern;
+
+/** Use when there is no sub-clause. */
+protected SupportsShowLikeOperation() {
+this.likeType = null;
Review Comment:
Sorry for misleading, No problem with this comments now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #22204: [Flink 31170] [docs]The spelling error of the document word causes sq…
flinkbot commented on PR #22204: URL: https://github.com/apache/flink/pull/22204#issuecomment-1473630712 ## CI report: * c2ef221f4397634df80b1dd2372bc9c423c2627b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] BoYiZhang opened a new pull request, #22204: [Flink 31170] [docs]The spelling error of the document word causes sq…
BoYiZhang opened a new pull request, #22204: URL: https://github.com/apache/flink/pull/22204 ## What is the purpose of the change - *Fix spelling errors in the create table statement* ## Brief change log - *The create table statement provided by the document is incorrect, and an error is reported during execution* ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (no ) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? ( no) - If yes, how is the feature documented? (not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-connector-aws] dannycranmer commented on a diff in pull request #58: [FLINK-16003][Connectors/Kinesis] Log non-retriable Kinesis exceptions from getRecords
dannycranmer commented on code in PR #58:
URL:
https://github.com/apache/flink-connector-aws/pull/58#discussion_r1140050195
##
flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxy.java:
##
@@ -313,6 +313,9 @@ public GetRecordsResult getRecords(String shardIterator,
int maxRecordsToGet)
} else {
throw ex;
}
+} catch (RuntimeException ex) {
+LOG.error(ex.getClass().getName() + ": " + ex.getMessage());
Review Comment:
Why not `LOG.error(ex)`, or something with a meaningful message
`LOG.error("Encountered non-recoverable error while invoking getRecords.", ex);`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-31494) Introduce SqlNodeConverter for SqlToOperationConverter
[
https://issues.apache.org/jira/browse/FLINK-31494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701631#comment-17701631
]
xuzhiwen edited comment on FLINK-31494 at 3/17/23 10:14 AM:
Hi [~jark] , Will SqlNodeConverters register all SQLnodes? My initial idea is
the same, but I feel that although this reduces the amount of
SqlToOperationConverter code, it will swell in SqlNodeConverters.
was (Author: xzw0223):
hi [~jark] , Will SqlNodeConverters register all SQLnodes? My initial idea is
the same, but I feel that although this reduces the amount of
SqlToOperationConverter code, it will swell in SqlNodeConverters
> Introduce SqlNodeConverter for SqlToOperationConverter
> --
>
> Key: FLINK-31494
> URL: https://issues.apache.org/jira/browse/FLINK-31494
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: xuzhiwen
>Assignee: Jark Wu
>Priority: Major
>
> Introduce {{SqlNodeConverter}} for {{SqlToOperationConverter}}, following
> Timo's idea in FLINK-31368
> class like:
> {code:java}
> public interface SqlNodeConverter {
> Operation convertSqlNode(S node, ConvertContext context);
> }
> /** Registry of SqlNode converters. */
> public class SqlNodeConverters {
> private static final Map, SqlNodeConverter> CONVERTERS = new
> HashMap<>();
> static {
> // register all the converters here
> register(new SqlCreateCatalogConverter());
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-31494) Introduce SqlNodeConverter for SqlToOperationConverter
[
https://issues.apache.org/jira/browse/FLINK-31494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701631#comment-17701631
]
xuzhiwen commented on FLINK-31494:
--
hi [~jark] , Will SqlNodeConverters register all SQLnodes? My initial idea is
the same, but I feel that although this reduces the amount of
SqlToOperationConverter code, it will swell in SqlNodeConverters
> Introduce SqlNodeConverter for SqlToOperationConverter
> --
>
> Key: FLINK-31494
> URL: https://issues.apache.org/jira/browse/FLINK-31494
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: xuzhiwen
>Assignee: Jark Wu
>Priority: Major
>
> Introduce {{SqlNodeConverter}} for {{SqlToOperationConverter}}, following
> Timo's idea in FLINK-31368
> class like:
> {code:java}
> public interface SqlNodeConverter {
> Operation convertSqlNode(S node, ConvertContext context);
> }
> /** Registry of SqlNode converters. */
> public class SqlNodeConverters {
> private static final Map, SqlNodeConverter> CONVERTERS = new
> HashMap<>();
> static {
> // register all the converters here
> register(new SqlCreateCatalogConverter());
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] leonardBang commented on pull request #22201: [FLINK-30959][table][doc-zh] Improve the documentation of UNIX_TIMESTAMP function for different argument formats
leonardBang commented on PR #22201: URL: https://github.com/apache/flink/pull/22201#issuecomment-1473599179 @ruanhang1993 Could we can create a new JIRA issue to track the translation for all sql function? I found some content of these function is outdated. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-31498) DeclartiveSlotManager always request redundant task manager when resource is not enough
Weihua Hu created FLINK-31498: - Summary: DeclartiveSlotManager always request redundant task manager when resource is not enough Key: FLINK-31498 URL: https://issues.apache.org/jira/browse/FLINK-31498 Project: Flink Issue Type: Bug Components: Runtime / Coordination Reporter: Weihua Hu Attachments: image-2023-03-17-18-05-43-088.png Currently redundant task manager check in DeclarativeSlotManager only compare free slots with required redundant slots. when there are no enough resources in YARN/Kubernetes, this mechanism will always try to request new task manager. there are two way to address this. 1. maintain the state of redundant workers to avoid request twice 2. only try to request redundant workers when there is no pending worker The first way will make the logic of redundant worker too complicated, I would like to choose the second way Looking forward to any suggestion. !image-2023-03-17-18-05-43-088.png! -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-30959) Improve the documentation of UNIX_TIMESTAMP for different argument format
[
https://issues.apache.org/jira/browse/FLINK-30959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17700745#comment-17700745
]
Leonard Xu edited comment on FLINK-30959 at 3/17/23 10:09 AM:
--
Resolved in:
* master: f3ec5733f4fbc85d7a6bd27484ac10185a36b1ed
* release-1.17: 7e9cf7af50f87d997dd660bc6300eb791bc0a9cf
* release-1.16: 3d84e591fa172f7520d6c9fc7d2335a7bf67ecd2
was (Author: leonard xu):
Resolved in:
* master: f3ec5733f4fbc85d7a6bd27484ac10185a36b1ed
* release-1.17: 7e9cf7af50f87d997dd660bc6300eb791bc0a9cf
* release-1.16: TODO
> Improve the documentation of UNIX_TIMESTAMP for different argument format
> -
>
> Key: FLINK-30959
> URL: https://issues.apache.org/jira/browse/FLINK-30959
> Project: Flink
> Issue Type: Improvement
> Components: Documentation, Table SQL / API
>Affects Versions: 1.16.1
>Reporter: Yunfeng Zhou
>Assignee: Hang Ruan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.17.0, 1.16.2
>
>
> When running the following pyflink program
>
> {code:python}
> import pandas as pd
> from pyflink.datastream import StreamExecutionEnvironment, HashMapStateBackend
> from pyflink.table import StreamTableEnvironment
> if __name__ == "__main__":
> input_data = pd.DataFrame(
> [
> ["Alex", 100.0, "2022-01-01 08:00:00.001 +0800"],
> ["Emma", 400.0, "2022-01-01 00:00:00.003 +"],
> ["Alex", 200.0, "2022-01-01 08:00:00.005 +0800"],
> ["Emma", 300.0, "2022-01-01 00:00:00.007 +"],
> ["Jack", 500.0, "2022-01-01 08:00:00.009 +0800"],
> ["Alex", 450.0, "2022-01-01 00:00:00.011 +"],
> ],
> columns=["name", "avg_cost", "time"],
> )
> env = StreamExecutionEnvironment.get_execution_environment()
> env.set_state_backend(HashMapStateBackend())
> t_env = StreamTableEnvironment.create(env)
> input_table = t_env.from_pandas(input_data)
> t_env.create_temporary_view("input_table", input_table)
> time_format = "-MM-dd HH:mm:ss.SSS X"
> output_table = t_env.sql_query(
> f"SELECT *, UNIX_TIMESTAMP(`time`, '{time_format}') AS unix_time FROM
> input_table"
> )
> output_table.execute().print()
> {code}
> The actual output is
> {code}
> +++++--+
> | op | name | avg_cost |
> time |unix_time |
> +++++--+
> | +I | Alex | 100.0 |
> 2022-01-01 08:00:00.001 +0800 | 1640995200 |
> | +I | Emma | 400.0 |
> 2022-01-01 00:00:00.003 + | 1640995200 |
> | +I | Alex | 200.0 |
> 2022-01-01 08:00:00.005 +0800 | 1640995200 |
> | +I | Emma | 300.0 |
> 2022-01-01 00:00:00.007 + | 1640995200 |
> | +I | Jack | 500.0 |
> 2022-01-01 08:00:00.009 +0800 | 1640995200 |
> | +I | Alex | 450.0 |
> 2022-01-01 00:00:00.011 + | 1640995200 |
> +++++--+
> {code}
> While the expected result is
> {code:java}
> +++++--+
> | op | name | avg_cost |
> time |unix_time |
> +++++--+
> | +I | Alex | 100.0 |
> 2022-01-01 08:00:00.001 +0800 | 1640995200 |
> | +I | Emma | 400.0 |
> 2022-01-01 00:00:00.003 + | 1640966400 |
> | +I | Alex | 200.0 |
> 2022-01-01 08:00:00.005 +0800 | 1640995200 |
> | +I | Emma | 300.0 |
> 2022-01-01 00:00:00.007 + | 1640966400 |
> | +I | Jack | 500.0 |
> 2022-01-01 08:00:00.009 +0800 | 1640995200
[jira] [Updated] (FLINK-31494) Introduce SqlNodeConverter for SqlToOperationConverter
[
https://issues.apache.org/jira/browse/FLINK-31494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-31494:
Description:
Introduce {{SqlNodeConverter}} for {{SqlToOperationConverter}}, following
Timo's idea in FLINK-31368
class like:
{code:java}
public interface SqlNodeConverter {
Operation convertSqlNode(S node, ConvertContext context);
}
/** Registry of SqlNode converters. */
public class SqlNodeConverters {
private static final Map, SqlNodeConverter> CONVERTERS = new
HashMap<>();
static {
// register all the converters here
register(new SqlCreateCatalogConverter());
}
}
{code}
was:
Introduce OperationExecutor for SqlToOperationConverter, following Timo's idea
in FLINK-31368
class like:
{code:java}
public interface OperationExecutor{
// The Operation supported by the current OperationExecutor
Class supportedOperation();
// The SqlNode type supported by the current OperationExecutor
Class supportedSqlNode();
// OperationExecutor is used to convert the validated sqlNode into Operation
Operation execute(SqlNode validated);
} {code}
Add an OperationExecutors to manage all OperationExecutors and be responsible
for performing conversion sqlNode.
{code:java}
public class OperationExecutors{
private static Map,OperationExecutor > executors = new
HashMap<>();
static{
addExecutor(SqlCreateCatalog.class,CreateCatalogExecutor.class);
// .
}
private static void addExecutor(sqlnodeClass,operationExecutorsClass){
executors.put(sqlnodeClass,operationExecutorsClass.newInstance);
}
public OperationExecutor getExecutor(sqlnodeClass){
return executors.get(validated.getClass());
}
public Operation execute(SqlNode validated){
return executors.get(validated.getClass()).executor(validated);
}
} {code}
This can be used in SqlToOperationConverter.java
{code:java}
private static Optional convertValidatedSqlNode(
FlinkPlannerImpl flinkPlanner, CatalogManager catalogManager, SqlNode
validated) {
OperationExecutor executor = OperationExecutors.getExecutor(validated);
if (executor == null){
return Optional.empty();
}
Operation operation = executor.execute(validated);
if (operation!=null){
return Optional.of(operation);
}
}
{code}
> Introduce SqlNodeConverter for SqlToOperationConverter
> --
>
> Key: FLINK-31494
> URL: https://issues.apache.org/jira/browse/FLINK-31494
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: xuzhiwen
>Assignee: Jark Wu
>Priority: Major
>
> Introduce {{SqlNodeConverter}} for {{SqlToOperationConverter}}, following
> Timo's idea in FLINK-31368
> class like:
> {code:java}
> public interface SqlNodeConverter {
> Operation convertSqlNode(S node, ConvertContext context);
> }
> /** Registry of SqlNode converters. */
> public class SqlNodeConverters {
> private static final Map, SqlNodeConverter> CONVERTERS = new
> HashMap<>();
> static {
> // register all the converters here
> register(new SqlCreateCatalogConverter());
> }
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] leonardBang merged pull request #22186: [FLINK-30959][table][docs] Improve the documentation of UNIX_TIMESTAMP function for different argument formats
leonardBang merged PR #22186: URL: https://github.com/apache/flink/pull/22186 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
