[jira] [Created] (FLINK-25413) Use append yarn and hadoop config to replace overwrite
qiunan created FLINK-25413: -- Summary: Use append yarn and hadoop config to replace overwrite Key: FLINK-25413 URL: https://issues.apache.org/jira/browse/FLINK-25413 Project: Flink Issue Type: Improvement Reporter: qiunan In [FLINK-16005][flink-yarn] Support yarn and hadoop config override. However, I think we should append methods to better meet our production needs. For example, the HDFS cluster in production has its own configuration. We only need to add hdfs that cannot be written to. At the same time, there are many clusters in production. We cannot cover all the configurations. In flink-conf.yaml flink.hadoop.dfs.nameservices: nameservice1 The problem now is, if flink.hadoop.dfs.nameservices: nameservice1 we can only write to the cluster of nameservice1, all other clusters cannot be written, we need to add the configuration of each cluster to write. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Closed] (FLINK-25365) Remove remaining references to planner from Python
[
https://issues.apache.org/jira/browse/FLINK-25365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Timo Walther closed FLINK-25365.
Fix Version/s: 1.15.0
Resolution: Fixed
Fixed in master: 74ed032009aa599c7b69e7205e7a0db3d0b1fd72
> Remove remaining references to planner from Python
> --
>
> Key: FLINK-25365
> URL: https://issues.apache.org/jira/browse/FLINK-25365
> Project: Flink
> Issue Type: Sub-task
> Components: API / Python, Table SQL / Planner
>Reporter: Timo Walther
>Assignee: Francesco Guardiani
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.15.0
>
>
> This issue should conclude the effort of accessing planner classes from the
> Python module.
> E.g. the following classes should be simplified:
> {code}
> org.apache.flink.table.planner.utils.python.PythonTableUtils
> org.apache.flink.table.planner.utils.python.PythonInputFormatTableSource
> org.apache.flink.table.planner.typeutils.DataViewUtils
> {code}
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] twalthr closed pull request #18156: [FLINK-25365][python] Remove remaining references to planner from Python
twalthr closed pull request #18156: URL: https://github.com/apache/flink/pull/18156 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17808) Rename checkpoint meta file to "_metadata" until it has completed writing
[
https://issues.apache.org/jira/browse/FLINK-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463624#comment-17463624
]
Junfan Zhang commented on FLINK-17808:
--
I overlooked this, and thanks [~yunta] for pointing it.
I think If filesystem dont support, we could fall back to original
implementation in {{{}FsCheckpointMetadataOutputStream{}}}. Maybe we should
underline this point in doc.
> Rename checkpoint meta file to "_metadata" until it has completed writing
> -
>
> Key: FLINK-17808
> URL: https://issues.apache.org/jira/browse/FLINK-17808
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.10.0
>Reporter: Yun Tang
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
> Fix For: 1.15.0
>
>
> In practice, some developers or customers would use some strategy to find the
> recent _metadata as the checkpoint to recover (e.g as many proposals in
> FLINK-9043 suggest). However, there existed a "_meatadata" file does not mean
> the checkpoint have been completed as the writing to create the "_meatadata"
> file could break as some force quit (e.g. yarn application -kill).
> We could create the checkpoint meta stream to write data to file named as
> "_metadata.inprogress" and renamed it to "_metadata" once completed writing.
> By doing so, we could ensure the "_metadata" is not broken.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] TanYuxin-tyx commented on a change in pull request #17936: [FLINK-24954][network] Reset read buffer request timeout on buffer recycling for sort-shuffle
TanYuxin-tyx commented on a change in pull request #17936:
URL: https://github.com/apache/flink/pull/17936#discussion_r773664257
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartitionReadSchedulerTest.java
##
@@ -192,6 +194,53 @@ public void testOnReadBufferRequestError() throws
Exception {
assertAllResourcesReleased();
}
+@Test
+public void testIncreaseDeadlineWhenRecycleBuffer() throws Exception {
+testRequestDeadlineInternal(true);
+assertTrue(readScheduler.getBufferRequestDeadline().hasTimeLeft());
+}
+
+@Test
+public void testDeadlineTimeoutWhenNoRecycleBuffer() throws Exception {
+testRequestDeadlineInternal(false);
+assertFalse(readScheduler.getBufferRequestDeadline().hasTimeLeft());
+}
+
+private void testRequestDeadlineInternal(boolean needTriggerRecycleBuffer)
throws Exception {
+
readScheduler.setRequestBufferDeadline(Deadline.fromNow(Duration.ofMillis(500)));
+assertTrue(readScheduler.getBufferRequestDeadline().hasTimeLeft());
+
+readScheduler.run();
+// Waiting for the deadline timeout
+Thread.sleep(800);
+
+if (needTriggerRecycleBuffer) {
+// If at least one buffer is recycled, the deadline will be
increased
+triggerRecycleBuffer();
+}
+}
+
+private void triggerRecycleBuffer() throws Exception {
Review comment:
Thanks again for these comments.
I have updated the code. Could you please help check it again? @wsry
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * 0268004b11d420c07814834ebe4798dfdf7ea5d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27611) * cfd2774e395e402ff9838a0f7aeb0a0f2897f50f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28463) * b976feccf7b2c67f9c5911a01100a86c64b6b953 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28465) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * 0268004b11d420c07814834ebe4798dfdf7ea5d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27611) * cfd2774e395e402ff9838a0f7aeb0a0f2897f50f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28463) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] TanYuxin-tyx commented on a change in pull request #17936: [FLINK-24954][network] Reset read buffer request timeout on buffer recycling for sort-shuffle
TanYuxin-tyx commented on a change in pull request #17936:
URL: https://github.com/apache/flink/pull/17936#discussion_r773663321
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartitionReadSchedulerTest.java
##
@@ -192,6 +194,53 @@ public void testOnReadBufferRequestError() throws
Exception {
assertAllResourcesReleased();
}
+@Test
+public void testIncreaseDeadlineWhenRecycleBuffer() throws Exception {
+testRequestDeadlineInternal(true);
+assertTrue(readScheduler.getBufferRequestDeadline().hasTimeLeft());
Review comment:
Ok, I have updated the tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] TanYuxin-tyx commented on a change in pull request #17936: [FLINK-24954][network] Reset read buffer request timeout on buffer recycling for sort-shuffle
TanYuxin-tyx commented on a change in pull request #17936:
URL: https://github.com/apache/flink/pull/17936#discussion_r773662905
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartitionReadSchedulerTest.java
##
@@ -192,6 +194,53 @@ public void testOnReadBufferRequestError() throws
Exception {
assertAllResourcesReleased();
}
+@Test
+public void testIncreaseDeadlineWhenRecycleBuffer() throws Exception {
Review comment:
Ok, I have added the above tests. For the third case, I added 2 tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-25360) Add State Desc to CheckpointMetadata
[ https://issues.apache.org/jira/browse/FLINK-25360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463622#comment-17463622 ] Yun Tang commented on FLINK-25360: -- [~liufangqi], I think your concerns have two different problems: # AggregatingStateDescriptor does not store the aggFunction on snapshot. This indeed occurs for a long time, and I think this deserve a disscussion in community mailling list. I prefer to store this in state descriptior. # Checkpoint meta does not contains the state descriptors. This is only true for savepoints, and checkpoints actually store the state descrtiptors, see [MetadataV2V3SerializerBase |https://github.com/apache/flink/blob/2b1a9dea74a334adb1fe890f024f4153ad11a985/flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java#L318]. BTW, even in current status, state-processor-API could still work well to read and modify the savepoints. It seems you have some additional requests, could you share what feature you're implementing? Or why you must need the meta stored in savepoint meta? > Add State Desc to CheckpointMetadata > > > Key: FLINK-25360 > URL: https://issues.apache.org/jira/browse/FLINK-25360 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing, Runtime / State Backends >Reporter: 刘方奇 >Priority: Major > Attachments: image-2021-12-17-20-01-42-423.png > > > Now we can't get the State Descriptor info in the checkpoint meta. Like the > case if we use state-processor-api to load state then rewrite state, we can't > flexible use the state. > Maybe there are other cases we need the State Descriptor, so can we add this > info? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * 0268004b11d420c07814834ebe4798dfdf7ea5d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27611) * cfd2774e395e402ff9838a0f7aeb0a0f2897f50f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28463) * b976feccf7b2c67f9c5911a01100a86c64b6b953 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] TanYuxin-tyx commented on a change in pull request #17936: [FLINK-24954][network] Reset read buffer request timeout on buffer recycling for sort-shuffle
TanYuxin-tyx commented on a change in pull request #17936:
URL: https://github.com/apache/flink/pull/17936#discussion_r773661934
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartitionReadScheduler.java
##
@@ -178,6 +185,23 @@ public synchronized void run() {
return new ArrayDeque<>();
}
+@VisibleForTesting
+void setRequestBufferDeadline(Deadline deadline) {
+this.requestBufferDeadline = deadline;
+}
+
+@VisibleForTesting
+Deadline getBufferRequestDeadline() {
Review comment:
Fixed it. And this method is renamed as `getBufferRequestTimeoutTime`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] TanYuxin-tyx commented on a change in pull request #17936: [FLINK-24954][network] Reset read buffer request timeout on buffer recycling for sort-shuffle
TanYuxin-tyx commented on a change in pull request #17936:
URL: https://github.com/apache/flink/pull/17936#discussion_r773661321
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartitionReadScheduler.java
##
@@ -154,8 +160,9 @@ public synchronized void run() {
}
try {
-Deadline deadline = Deadline.fromNow(BUFFER_REQUEST_TIMEOUT);
-while (deadline.hasTimeLeft()) {
+requestBufferDeadline = Deadline.fromNow(BUFFER_REQUEST_TIMEOUT);
+lastNumRecycledBuffers = bufferPool.getNumRecycledBuffers();
+while (getBufferRequestDeadline().hasTimeLeft()) {
Review comment:
Thanks a lot. I think you are right. I have modified the logics of renew
timeout. When checking the timeout time, I use the
(`lastBufferOperationTimestamp` + 5min) as the deadline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17936: [FLINK-24954][network] Reset read buffer request timeout on buffer recycling for sort-shuffle
flinkbot edited a comment on pull request #17936: URL: https://github.com/apache/flink/pull/17936#issuecomment-980533423 ## CI report: * 819684b71438fea2f38144bc70c23cc6525dee37 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27165) * ec7627125864b807502b26946f61a59ec027c625 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28464) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] TanYuxin-tyx commented on a change in pull request #17936: [FLINK-24954][network] Reset read buffer request timeout on buffer recycling for sort-shuffle
TanYuxin-tyx commented on a change in pull request #17936:
URL: https://github.com/apache/flink/pull/17936#discussion_r773659490
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartitionReadScheduler.java
##
@@ -119,6 +119,12 @@
@GuardedBy("lock")
private volatile boolean isReleased;
+/** Number of buffers recycled in the last loop. */
+private long lastNumRecycledBuffers;
Review comment:
This field has been changed to `lastBufferOperationTimestamp`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] TanYuxin-tyx commented on a change in pull request #17936: [FLINK-24954][network] Reset read buffer request timeout on buffer recycling for sort-shuffle
TanYuxin-tyx commented on a change in pull request #17936:
URL: https://github.com/apache/flink/pull/17936#discussion_r773658955
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartitionReadScheduler.java
##
@@ -178,6 +185,23 @@ public synchronized void run() {
return new ArrayDeque<>();
}
+@VisibleForTesting
+void setRequestBufferDeadline(Deadline deadline) {
Review comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] TanYuxin-tyx commented on a change in pull request #17936: [FLINK-24954][network] Reset read buffer request timeout on buffer recycling for sort-shuffle
TanYuxin-tyx commented on a change in pull request #17936:
URL: https://github.com/apache/flink/pull/17936#discussion_r773658627
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartitionReadScheduler.java
##
@@ -119,6 +119,12 @@
@GuardedBy("lock")
private volatile boolean isReleased;
+/** Number of buffers recycled in the last loop. */
+private long lastNumRecycledBuffers;
+
+/** Deadline of requesting buffers. */
+private Deadline requestBufferDeadline;
Review comment:
Fixed. I have move it to the constructor.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * 0268004b11d420c07814834ebe4798dfdf7ea5d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27611) * cfd2774e395e402ff9838a0f7aeb0a0f2897f50f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28463) * b976feccf7b2c67f9c5911a01100a86c64b6b953 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17936: [FLINK-24954][network] Reset read buffer request timeout on buffer recycling for sort-shuffle
flinkbot edited a comment on pull request #17936: URL: https://github.com/apache/flink/pull/17936#issuecomment-980533423 ## CI report: * 819684b71438fea2f38144bc70c23cc6525dee37 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27165) * ec7627125864b807502b26946f61a59ec027c625 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] TanYuxin-tyx commented on a change in pull request #17936: [FLINK-24954][network] Reset read buffer request timeout on buffer recycling for sort-shuffle
TanYuxin-tyx commented on a change in pull request #17936: URL: https://github.com/apache/flink/pull/17936#discussion_r773657411 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/io/network/partition/SortMergeResultPartition.java ## @@ -456,7 +456,7 @@ public ResultSubpartitionView createSubpartitionView( checkState(!isReleased(), "Partition released."); checkState(isFinished(), "Trying to read unfinished blocking partition."); -return readScheduler.crateSubpartitionReader( Review comment: Ok, fixed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * 0268004b11d420c07814834ebe4798dfdf7ea5d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27611) * cfd2774e395e402ff9838a0f7aeb0a0f2897f50f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28463) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * 0268004b11d420c07814834ebe4798dfdf7ea5d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27611) * cfd2774e395e402ff9838a0f7aeb0a0f2897f50f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28463) * b976feccf7b2c67f9c5911a01100a86c64b6b953 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24885) ProcessElement Interface parameter Collector : java.lang.NullPointerException
[ https://issues.apache.org/jira/browse/FLINK-24885?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463612#comment-17463612 ] Caizhi Weng commented on FLINK-24885: - Hi [~wangbaohua] is this issue really fixed? Have you tested that? Which Flink version are you testing? > ProcessElement Interface parameter Collector : java.lang.NullPointerException > -- > > Key: FLINK-24885 > URL: https://issues.apache.org/jira/browse/FLINK-24885 > Project: Flink > Issue Type: Bug > Components: Table SQL / Runtime >Affects Versions: 1.13.1 >Reporter: wangbaohua >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0, 1.13.6, 1.14.3 > > Attachments: error.jpg > > > 2021-11-15 11:11:55,032 INFO com.asap.demo.function.dealMapFunction > [] - size:160 > 2021-11-15 11:11:55,230 WARN org.apache.flink.runtime.taskmanager.Task > [] - Co-Process-Broadcast-Keyed -> Map -> > DataSteamToTable(stream=default_catalog.default_database.Unregistered_DataStream_Source_8, > type=*com.asap.demo.model.BeanField<`account` STRING, `accountId` STRING, > `accountIn` STRING, `accountInName` STRING, `accountInOrgId` STRING, > `accountInOrgName` STRING, `accountInType` STRING, `accountName` STRING, > `accountOrgId` STRING, `accountOrgName` STRING, `accountOut` STRING, > `accountOutName` STRING, `accountOutOrgId` STRING, `accountOutOrgName` > STRING, `accountOutType` STRING, `accountStatus` STRING, `accountType` > STRING, `action` STRING, `actionDesc` STRING, `alarmcontext` STRING, > `alarmgrade` STRING, `alarmtype` STRING, `alertId` STRING, `alertInfo` > STRING, `alertLevel` STRING, `alertSignatureIdL` STRING, `appId` STRING, > `appName` STRING, `appProtocol` STRING, `appType` STRING, `areaId` STRING, > `areaName` STRING, `areaType` STRING, `assetFrom` STRING, `assetId` STRING, > `assetInfo` STRING, `assetIp` STRING, `assetLevel` STRING, `assetName` > STRING, `assetPid` STRING, `assetType` STRING, `assetUse` STRING, > `assetVendor` STRING, `attackStage` STRING, `attackStageCode` STRING, > `attackType` STRING, `attackTypeName` STRING, `authSerNum` STRING, `authTime` > STRING, `authType` STRING, `bankSeqNum` STRING, `batchNo` STRING, > `blackDomain` STRING, `blackDomainDesc` STRING, `blackDomainTag` STRING, > `blackDstIp` STRING, `blackFile` STRING, `blackFileDesc` STRING, > `blackFileTag` STRING, `blackId` STRING, `blackIpTag` STRING, `blackSrcIp` > STRING, `blackTag` STRING, `blackTagMatchCount` STRING, `blackTagMatchDesc` > STRING, `blackUrl` STRING, `blackUrlDesc` STRING, `blackUrlTag` STRING, > `blackVulnCve` STRING, `blackVulnDesc` STRING, `blackVulnName` STRING, > `blackVulnTag` STRING, `branchId` STRING, `branchName` STRING, > `businessSystemName` STRING, `businessType` STRING, `cardId` STRING, > `cascadeSourceIp` STRING, `cascadeSourceName` STRING, `cebUid` STRING, > `certNum` STRING, `certType` STRING, `chainId` STRING, `channel` STRING, > `channelId` STRING, `character` STRING, `charge` STRING, `cifSeqNum` STRING, > `clientInfo` STRING, `clientIp` STRING, `clientMac` STRING, `clientName` > STRING, `clientPort` STRING, `collectTime` TIMESTAMP_LTZ(9), `collectTimeL` > TIMESTAMP_LTZ(9), `command` STRING, `commandLine` STRING, `commandResult` > STRING, `company` STRING, `companyCustomId` STRING, `companyId` STRING, > `completenessTag` STRING, `confidence` STRING, `confidenceLevel` STRING, > `consignedUser` STRING, `contractNo` STRING, `count` STRING, `couponAmount` > STRING, `couponId` STRING, `createTime` TIMESTAMP_LTZ(3), `createTimeL` > BIGINT, `createdBy` STRING, `curType` STRING, `currency` STRING, `currentBal` > STRING, `customLabel1` STRING, `customLabel10` STRING, `customLabel2` STRING, > `customLabel3` STRING, `customLabel4` STRING, `customLabel5` STRING, > `customLabel6` STRING, `customLabel7` STRING, `customLabel8` STRING, > `customLabel9` STRING, `customValue1` STRING, `customValue10` STRING, > `customValue2` STRING, `customValue3` STRING, `customValue4` STRING, > `customValue5` STRING, `customValue6` STRING, `customValue7` STRING, > `customValue8` STRING, `customValue9` STRING, `dataQualityTag` STRING, > `dataType` STRING, `dataTypeName` STRING, `dbInstance` STRING, `dbName` > STRING, `dbTable` STRING, `dbVersion` STRING, `dealSuggest` STRING, > `defVManagerId` STRING, `department` STRING, `deviceCategory` STRING, > `deviceId` STRING, `deviceIp` STRING, `deviceMac` STRING, `deviceName` > STRING, `deviceParentType` STRING, `deviceType` STRING, `deviceVersion` > STRING, `direction` STRING, `directionDesc` STRING, `directionOfAttackTag` > STRING, `domain` STRING, `dstAdminAccount` STRING, `dstAdminEmail` STRING, > `dstAdminFOrgId` STRING, `dstAdmi
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * 0268004b11d420c07814834ebe4798dfdf7ea5d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27611) * cfd2774e395e402ff9838a0f7aeb0a0f2897f50f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28463) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25411) JsonRowSerializationSchema unable to parse TIMESTAMP_LTZ fields
[ https://issues.apache.org/jira/browse/FLINK-25411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Lalwani updated FLINK-25411: - Fix Version/s: 1.13.6 > JsonRowSerializationSchema unable to parse TIMESTAMP_LTZ fields > --- > > Key: FLINK-25411 > URL: https://issues.apache.org/jira/browse/FLINK-25411 > Project: Flink > Issue Type: Bug > Components: API / DataStream, API / Type Serialization System >Affects Versions: 1.13.3, 1.15.0, 1.14.2 >Reporter: Surendra Lalwani >Priority: Critical > Fix For: 1.13.6 > > > While I try to fire a simple query Select current_timestamp from table_name , > it gives error that Could not serialize row and asks me to add shaded flink > dependency for jsr-310. Seems like in the Serializer , the JavaTimeModule is > not added -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * 0268004b11d420c07814834ebe4798dfdf7ea5d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27611) * cfd2774e395e402ff9838a0f7aeb0a0f2897f50f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28463) * b976feccf7b2c67f9c5911a01100a86c64b6b953 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18068: [FLINK-25105][checkpoint] Enables final checkpoint by default
flinkbot edited a comment on pull request #18068: URL: https://github.com/apache/flink/pull/18068#issuecomment-989975508 ## CI report: * 6832524d7d78de814cbeadb44fa8037da5c10ca9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28200) * f0b1ef0bd1e2babc2093a57b0b6b619c99cb551e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28457) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * 0268004b11d420c07814834ebe4798dfdf7ea5d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27611) * cfd2774e395e402ff9838a0f7aeb0a0f2897f50f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28463) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18068: [FLINK-25105][checkpoint] Enables final checkpoint by default
flinkbot edited a comment on pull request #18068: URL: https://github.com/apache/flink/pull/18068#issuecomment-989975508 ## CI report: * 6832524d7d78de814cbeadb44fa8037da5c10ca9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28200) * f0b1ef0bd1e2babc2093a57b0b6b619c99cb551e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28457) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * 0268004b11d420c07814834ebe4798dfdf7ea5d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27611) * cfd2774e395e402ff9838a0f7aeb0a0f2897f50f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28463) * b976feccf7b2c67f9c5911a01100a86c64b6b953 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on pull request #18068: [FLINK-25105][checkpoint] Enables final checkpoint by default
gaoyunhaii commented on pull request #18068: URL: https://github.com/apache/flink/pull/18068#issuecomment-999337047 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * 0268004b11d420c07814834ebe4798dfdf7ea5d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27611) * cfd2774e395e402ff9838a0f7aeb0a0f2897f50f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28463) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * 0268004b11d420c07814834ebe4798dfdf7ea5d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27611) * cfd2774e395e402ff9838a0f7aeb0a0f2897f50f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28463) * b976feccf7b2c67f9c5911a01100a86c64b6b953 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25328) Improvement of reuse segments for join/agg/sort operators in TaskManager for flink olap queries
[ https://issues.apache.org/jira/browse/FLINK-25328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463605#comment-17463605 ] Xintong Song commented on FLINK-25328: -- What I had in mind is something similar to the idle slot / taskmanager timeout, to cache the released segments for a certain configurable time before actually deallocating it. My second point is basically about speeding-up deallocating the cached segments when necessary. If there're pending rocksdb/python requests, likely blocked on deallocating of the segments, we should deallocate asap even the configured caching time is not reached. That should also allow as to configure relative long caching time without worrying about blocking rocksdb/python for too long. > Improvement of reuse segments for join/agg/sort operators in TaskManager for > flink olap queries > --- > > Key: FLINK-25328 > URL: https://issues.apache.org/jira/browse/FLINK-25328 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Affects Versions: 1.14.0, 1.12.5, 1.13.3 >Reporter: Shammon >Priority: Major > > We submit batch jobs to flink session cluster as olap queries, and these > jobs' subtasks in TaskManager are frequently created and destroyed because > they finish their work quickly. Each slot in taskmanager manages > `MemoryManager` for multiple tasks in one job, and the `MemoryManager` is > closed when all the subtasks are finished. Join/Aggregate/Sort and etc. > operators in the subtasks allocate `MemorySegment` via `MemoryManager` and > these `MemorySegment` will be free when they are finished. > > It causes too much memory allocation and free of `MemorySegment` in > taskmanager. For example, a TaskManager contains 50 slots, one job has 3 > join/agg operatos run in the slot, each operator will allocate 2000 segments > and initialize them. If the subtasks of a job take 100ms to execute, then the > taskmanager will execute 10 jobs' subtasks one second and it will allocate > and free 2000 * 3 * 50 * 10 = 300w segments for them. Allocate and free too > many segments from memory will cause two issues: > 1) Increases the CPU usage of taskmanager > 2) Increase the cost of subtasks in taskmanager, which will increase the > latency of job and decrease the qps. > To improve the usage of memory segment between jobs in the same slot, > we propose not drop memory manager when all the subtasks in the slot are > finished. The slot will hold the `MemoryManager` and not free the allocated > `MemorySegment` in it immediately. When some subtasks of another job are > assigned to the slot, they don't need to allocate segments from memory and > can reuse the `MemoryManager` and `MemorySegment` in it. WDYT? [~xtsong] THX -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * 0268004b11d420c07814834ebe4798dfdf7ea5d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27611) * cfd2774e395e402ff9838a0f7aeb0a0f2897f50f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28463) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17988: [FLINK-25010][Connectors/Hive] Speed up hive's createMRSplits by multi thread
flinkbot edited a comment on pull request #17988: URL: https://github.com/apache/flink/pull/17988#issuecomment-984363654 ## CI report: * e7be85162e0b431d518ab1ffe8b59283338d00b7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28411) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong commented on pull request #17988: [FLINK-25010][Connectors/Hive] Speed up hive's createMRSplits by multi thread
wuchong commented on pull request #17988: URL: https://github.com/apache/flink/pull/17988#issuecomment-999330362 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * 0268004b11d420c07814834ebe4798dfdf7ea5d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27611) * cfd2774e395e402ff9838a0f7aeb0a0f2897f50f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28463) * b976feccf7b2c67f9c5911a01100a86c64b6b953 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25411) JsonRowSerializationSchema unable to parse TIMESTAMP_LTZ fields
[ https://issues.apache.org/jira/browse/FLINK-25411?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Lalwani updated FLINK-25411: - Affects Version/s: 1.14.2 1.15.0 > JsonRowSerializationSchema unable to parse TIMESTAMP_LTZ fields > --- > > Key: FLINK-25411 > URL: https://issues.apache.org/jira/browse/FLINK-25411 > Project: Flink > Issue Type: Bug > Components: API / DataStream, API / Type Serialization System >Affects Versions: 1.13.3, 1.15.0, 1.14.2 >Reporter: Surendra Lalwani >Priority: Critical > > While I try to fire a simple query Select current_timestamp from table_name , > it gives error that Could not serialize row and asks me to add shaded flink > dependency for jsr-310. Seems like in the Serializer , the JavaTimeModule is > not added -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * 0268004b11d420c07814834ebe4798dfdf7ea5d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27611) * cfd2774e395e402ff9838a0f7aeb0a0f2897f50f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28463) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17808) Rename checkpoint meta file to "_metadata" until it has completed writing
[ https://issues.apache.org/jira/browse/FLINK-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463604#comment-17463604 ] Yun Tang commented on FLINK-17808: -- [~zuston] It seems FileSystem#createRecoverableWriter is only optional on file systems and various file system implementations may not support this method, did you consider file systems which not support this feature? > Rename checkpoint meta file to "_metadata" until it has completed writing > - > > Key: FLINK-17808 > URL: https://issues.apache.org/jira/browse/FLINK-17808 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing >Affects Versions: 1.10.0 >Reporter: Yun Tang >Priority: Minor > Labels: auto-deprioritized-major, pull-request-available > Fix For: 1.15.0 > > > In practice, some developers or customers would use some strategy to find the > recent _metadata as the checkpoint to recover (e.g as many proposals in > FLINK-9043 suggest). However, there existed a "_meatadata" file does not mean > the checkpoint have been completed as the writing to create the "_meatadata" > file could break as some force quit (e.g. yarn application -kill). > We could create the checkpoint meta stream to write data to file named as > "_metadata.inprogress" and renamed it to "_metadata" once completed writing. > By doing so, we could ensure the "_metadata" is not broken. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * 0268004b11d420c07814834ebe4798dfdf7ea5d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27611) * cfd2774e395e402ff9838a0f7aeb0a0f2897f50f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28463) * b976feccf7b2c67f9c5911a01100a86c64b6b953 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25188) Cannot install PyFlink on MacOS with M1 chip
[
https://issues.apache.org/jira/browse/FLINK-25188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463600#comment-17463600
]
Ada Wong commented on FLINK-25188:
--
I have updated numpy version in Apache Beam. we could bump to Beam 2.36.0.
https://issues.apache.org/jira/browse/BEAM-13397?filter=-2
> Cannot install PyFlink on MacOS with M1 chip
>
>
> Key: FLINK-25188
> URL: https://issues.apache.org/jira/browse/FLINK-25188
> Project: Flink
> Issue Type: Improvement
> Components: API / Python
>Affects Versions: 1.14.0
>Reporter: Ada Wong
>Priority: Major
> Fix For: 1.15.0
>
>
> Need to update dependencies: numpy>=
> 1.20.3、pyarrow>=5.0.0、pandas>=1.3.0、apache-beam==2.36.0
> This following is some dependencies adapt M1 chip informations
> Numpy version:
> [https://stackoverflow.com/questions/65336789/numpy-build-fail-in-m1-big-sur-11-1]
> [https://github.com/numpy/numpy/releases/tag/v1.21.4]
> pyarrow version:
> [https://stackoverflow.com/questions/68385728/installing-pyarrow-cant-copy-build-lib-macosx-11-arm64-3-9-pyarrow-include-ar]
> pandas version:
> [https://github.com/pandas-dev/pandas/issues/40611#issuecomment-901569655]
> Apache beam:
> https://issues.apache.org/jira/browse/BEAM-12957
> https://issues.apache.org/jira/browse/BEAM-11703
> This following is dependency tree after installed successfully
> Although Beam need numpy<1.21.0 and M1 need numpy >=1.21.4, when I using
> numpy 1.20.3 I install successfully on M1 chip.
> {code:java}
> apache-flink==1.14.dev0
> - apache-beam [required: ==2.34.0, installed: 2.34.0]
> - avro-python3 [required: >=1.8.1,<1.10.0,!=1.9.2, installed: 1.9.2.1]
> - crcmod [required: >=1.7,<2.0, installed: 1.7]
> - dill [required: >=0.3.1.1,<0.3.2, installed: 0.3.1.1]
> - fastavro [required: >=0.21.4,<2, installed: 0.23.6]
> - pytz [required: Any, installed: 2021.3]
> - future [required: >=0.18.2,<1.0.0, installed: 0.18.2]
> - grpcio [required: >=1.29.0,<2, installed: 1.42.0]
> - six [required: >=1.5.2, installed: 1.16.0]
> - hdfs [required: >=2.1.0,<3.0.0, installed: 2.6.0]
> - docopt [required: Any, installed: 0.6.2]
> - requests [required: >=2.7.0, installed: 2.26.0]
> - certifi [required: >=2017.4.17, installed: 2021.10.8]
> - charset-normalizer [required: ~=2.0.0, installed: 2.0.9]
> - idna [required: >=2.5,<4, installed: 3.3]
> - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.7]
> - six [required: >=1.9.0, installed: 1.16.0]
> - httplib2 [required: >=0.8,<0.20.0, installed: 0.19.1]
> - pyparsing [required: >=2.4.2,<3, installed: 2.4.7]
> - numpy [required: >=1.14.3,<1.21.0, installed: 1.20.3]
> - oauth2client [required: >=2.0.1,<5, installed: 4.1.3]
> - httplib2 [required: >=0.9.1, installed: 0.19.1]
> - pyparsing [required: >=2.4.2,<3, installed: 2.4.7]
> - pyasn1 [required: >=0.1.7, installed: 0.4.8]
> - pyasn1-modules [required: >=0.0.5, installed: 0.2.8]
> - pyasn1 [required: >=0.4.6,<0.5.0, installed: 0.4.8]
> - rsa [required: >=3.1.4, installed: 4.8]
> - pyasn1 [required: >=0.1.3, installed: 0.4.8]
> - six [required: >=1.6.1, installed: 1.16.0]
> - orjson [required: <4.0, installed: 3.6.5]
> - protobuf [required: >=3.12.2,<4, installed: 3.17.3]
> - six [required: >=1.9, installed: 1.16.0]
> - pyarrow [required: >=0.15.1,<6.0.0, installed: 5.0.0]
> - numpy [required: >=1.16.6, installed: 1.20.3]
> - pydot [required: >=1.2.0,<2, installed: 1.4.2]
> - pyparsing [required: >=2.1.4, installed: 2.4.7]
> - pymongo [required: >=3.8.0,<4.0.0, installed: 3.12.2]
> - python-dateutil [required: >=2.8.0,<3, installed: 2.8.0]
> - six [required: >=1.5, installed: 1.16.0]
> - pytz [required: >=2018.3, installed: 2021.3]
> - requests [required: >=2.24.0,<3.0.0, installed: 2.26.0]
> - certifi [required: >=2017.4.17, installed: 2021.10.8]
> - charset-normalizer [required: ~=2.0.0, installed: 2.0.9]
> - idna [required: >=2.5,<4, installed: 3.3]
> - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.7]
> - typing-extensions [required: >=3.7.0,<4, installed: 3.10.0.2]
> - apache-flink-libraries [required: ==1.14.dev0, installed: 1.14.dev0]
> - avro-python3 [required: >=1.8.1,<1.10.0,!=1.9.2, installed: 1.9.2.1]
> - cloudpickle [required: ==1.2.2, installed: 1.2.2]
> - fastavro [required: >=0.21.4,<0.24, installed: 0.23.6]
> - pytz [required: Any, installed: 2021.3]
> - numpy [required: >=1.20.3, installed: 1.20.3]
> - pandas [required: >=1.3.0, installed: 1.3.0]
> - numpy [required: >=1.17.3, installed: 1.20.3]
> - python-dateutil [required: >=2.7.3, installed: 2.8.0]
> - six [re
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * 0268004b11d420c07814834ebe4798dfdf7ea5d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27611) * cfd2774e395e402ff9838a0f7aeb0a0f2897f50f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28463) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-25261) Changelog not truncated on materialization
[ https://issues.apache.org/jira/browse/FLINK-25261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463599#comment-17463599 ] Yuan Mei edited comment on FLINK-25261 at 12/22/21, 6:40 AM: - Shouldn't truncation happen when a checkpoint is subsumed? I do not think it is a safe/right place to do truncation when materialization completes. was (Author: ym): Shouldn't truncation happen when a checkpoint is subsumed? I do not think it is safe and the right place to do when materialization completes. > Changelog not truncated on materialization > -- > > Key: FLINK-25261 > URL: https://issues.apache.org/jira/browse/FLINK-25261 > Project: Flink > Issue Type: Bug > Components: Runtime / State Backends >Affects Versions: 1.15.0 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > [https://github.com/apache/flink/blob/dcc4d43e413b20f70036e73c61d52e2e1c5afee7/flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java#L640] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18172: [hotfix][doc]update doc variable name reference error
flinkbot edited a comment on pull request #18172: URL: https://github.com/apache/flink/pull/18172#issuecomment-999323746 ## CI report: * 20f4eea5c51af444b31a4535e7751af021a9041c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28462) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
flinkbot edited a comment on pull request #18023: URL: https://github.com/apache/flink/pull/18023#issuecomment-986704278 ## CI report: * 0268004b11d420c07814834ebe4798dfdf7ea5d7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=27611) * cfd2774e395e402ff9838a0f7aeb0a0f2897f50f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-25261) Changelog not truncated on materialization
[ https://issues.apache.org/jira/browse/FLINK-25261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463599#comment-17463599 ] Yuan Mei edited comment on FLINK-25261 at 12/22/21, 6:38 AM: - Shouldn't truncation happen when a checkpoint is subsumed? I do not think it is safe and the right place to do when materialization completes. was (Author: ym): Shouldn't truncation happen when a checkpoint is subsumed? > Changelog not truncated on materialization > -- > > Key: FLINK-25261 > URL: https://issues.apache.org/jira/browse/FLINK-25261 > Project: Flink > Issue Type: Bug > Components: Runtime / State Backends >Affects Versions: 1.15.0 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > [https://github.com/apache/flink/blob/dcc4d43e413b20f70036e73c61d52e2e1c5afee7/flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java#L640] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] wanglijie95 commented on a change in pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
wanglijie95 commented on a change in pull request #18023: URL: https://github.com/apache/flink/pull/18023#discussion_r773634889 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/VertexParallelismInformation.java ## @@ -37,6 +37,14 @@ */ int getMaxParallelism(); +/** + * Set a given vertex's parallelism property. The parallelism can be changed only if the vertex + * parallelism was not decided yet (i.e. was -1). Review comment: I will add checks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wanglijie95 commented on a change in pull request #18023: [FLINK-25032] Allow to create execution vertices and execution edges lazily
wanglijie95 commented on a change in pull request #18023:
URL: https://github.com/apache/flink/pull/18023#discussion_r773634628
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/executiongraph/ExecutionJobVertexTest.java
##
@@ -93,6 +92,63 @@ public void testLazyInitialization() throws Exception {
assertThat(ejv.getTaskVertices().length, is(3));
assertThat(ejv.getInputs().size(), is(0));
assertThat(ejv.getProducedDataSets().length, is(1));
+assertThat(ejv.getOperatorCoordinators().size(), is(0));
+}
+
+@Test(expected = IllegalStateException.class)
+public void testErrorIfInitializationWithoutParallelismDecided() throws
Exception {
+final ExecutionJobVertex ejv = createDynamicExecutionJobVertex();
+
+initializeVertex(ejv);
+}
+
+@Test
+public void testSetParallelismLazily() throws Exception {
+final int parallelism = 3;
+final int defaultMaxParallelism = 13;
+final ExecutionJobVertex ejv =
+createDynamicExecutionJobVertex(-1, -1, defaultMaxParallelism);
+
+assertThat(ejv.isParallelismDecided(), is(false));
+
+ejv.setParallelism(parallelism);
+
+assertThat(ejv.isParallelismDecided(), is(true));
+assertThat(ejv.getParallelism(), is(parallelism));
+
+initializeVertex(ejv);
+
+assertThat(ejv.getTaskVertices().length, is(parallelism));
+}
+
+@Test
+public void testConfiguredMaxParallelismIsRespected() throws Exception {
+final int configuredMaxParallelism = 12;
+final int defaultMaxParallelism = 13;
+final ExecutionJobVertex ejv =
+createDynamicExecutionJobVertex(
+-1, configuredMaxParallelism, defaultMaxParallelism);
+
+assertThat(ejv.getMaxParallelism(), is(configuredMaxParallelism));
+}
+
+@Test
+public void testComputingMaxParallelismFromConfiguredParallelism() throws
Exception {
+final int parallelism = 300;
+final int defaultMaxParallelism = 13;
+final ExecutionJobVertex ejv =
+createDynamicExecutionJobVertex(parallelism, -1,
defaultMaxParallelism);
+
+assertThat(ejv.getMaxParallelism(), is(512));
+}
+
+@Test
+public void testFallingBackToDefaultParallelism() throws Exception {
Review comment:
fixed
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SchedulerBase.java
##
@@ -262,22 +264,55 @@ public static int getDefaultMaxParallelism(JobVertex
vertex) {
normalizeParallelism(vertex.getParallelism()));
}
+// TODO: move to Adaptive Batch Scheduler.
+/**
+ * Compute the {@link VertexParallelismStore} for all given vertices in a
dynamic graph, which
+ * will set defaults and ensure that the returned store contains valid
parallelisms, with the
+ * configured default max parallelism.
+ *
+ * @param vertices the vertices to compute parallelism for
+ * @param defaultMaxParallelism a function for computing a default max
parallelism if none is
Review comment:
fixed
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/ExecutionJobVertex.java
##
@@ -307,13 +323,21 @@ public JobVertexID getJobVertexId() {
@Override
public ExecutionVertex[] getTaskVertices() {
+if (taskVertices == null) {
+LOG.warn(
+"Trying to get execution vertices of an uninitialized job
vertex "
++ getJobVertexId());
+return new ExecutionVertex[0];
+}
return taskVertices;
}
public IntermediateResult[] getProducedDataSets() {
+checkState(isInitialized());
return producedDataSets;
}
+@Nullable
public InputSplitAssigner getSplitAssigner() {
return splitAssigner;
Review comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-25261) Changelog not truncated on materialization
[ https://issues.apache.org/jira/browse/FLINK-25261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463599#comment-17463599 ] Yuan Mei edited comment on FLINK-25261 at 12/22/21, 6:36 AM: - Shouldn't truncation happen when a checkpoint is subsumed? was (Author: ym): Shouldn't truncation happen during checkpoint subsumption? > Changelog not truncated on materialization > -- > > Key: FLINK-25261 > URL: https://issues.apache.org/jira/browse/FLINK-25261 > Project: Flink > Issue Type: Bug > Components: Runtime / State Backends >Affects Versions: 1.15.0 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > [https://github.com/apache/flink/blob/dcc4d43e413b20f70036e73c61d52e2e1c5afee7/flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java#L640] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25261) Changelog not truncated on materialization
[ https://issues.apache.org/jira/browse/FLINK-25261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463599#comment-17463599 ] Yuan Mei commented on FLINK-25261: -- Shouldn't truncation happen during checkpoint subsumption? > Changelog not truncated on materialization > -- > > Key: FLINK-25261 > URL: https://issues.apache.org/jira/browse/FLINK-25261 > Project: Flink > Issue Type: Bug > Components: Runtime / State Backends >Affects Versions: 1.15.0 >Reporter: Roman Khachatryan >Assignee: Roman Khachatryan >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > [https://github.com/apache/flink/blob/dcc4d43e413b20f70036e73c61d52e2e1c5afee7/flink-state-backends/flink-statebackend-changelog/src/main/java/org/apache/flink/state/changelog/ChangelogKeyedStateBackend.java#L640] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot commented on pull request #18172: [hotfix][doc]update doc variable name reference error
flinkbot commented on pull request #18172: URL: https://github.com/apache/flink/pull/18172#issuecomment-999323746 ## CI report: * 20f4eea5c51af444b31a4535e7751af021a9041c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18172: [hotfix][doc]update doc variable name reference error
flinkbot commented on pull request #18172: URL: https://github.com/apache/flink/pull/18172#issuecomment-999323402 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 20f4eea5c51af444b31a4535e7751af021a9041c (Wed Dec 22 06:35:18 UTC 2021) **Warnings:** * **Invalid pull request title: No valid Jira ID provided** Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25411) JsonRowSerializationSchema unable to parse TIMESTAMP_LTZ fields
[ https://issues.apache.org/jira/browse/FLINK-25411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463598#comment-17463598 ] Surendra Lalwani commented on FLINK-25411: -- [~MartijnVisser82] can you please check this > JsonRowSerializationSchema unable to parse TIMESTAMP_LTZ fields > --- > > Key: FLINK-25411 > URL: https://issues.apache.org/jira/browse/FLINK-25411 > Project: Flink > Issue Type: Bug > Components: API / DataStream, API / Type Serialization System >Affects Versions: 1.13.3 >Reporter: Surendra Lalwani >Priority: Critical > > While I try to fire a simple query Select current_timestamp from table_name , > it gives error that Could not serialize row and asks me to add shaded flink > dependency for jsr-310. Seems like in the Serializer , the JavaTimeModule is > not added -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] h1365930778 opened a new pull request #18172: update doc variable name reference error
h1365930778 opened a new pull request #18172: URL: https://github.com/apache/flink/pull/18172 ## What is the purpose of the change This PR fixes the problem with the Python DataStream API documentation page variable "t_env" being written as "table". On this page: https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/dev/python/datastream/intro_to_datastream_api/  ## Verifying this change I built and verified at the rendered docs site locally ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-25412) Upgrade of flink to 1.14.2 is showing internal server errors in the UI
[
https://issues.apache.org/jira/browse/FLINK-25412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
srujankumar updated FLINK-25412:
Description:
Hi
We have recently upgraded the flink from {{1.13.3}} to {{1.14.2}} and then
after upgrade we are facing issues in the UI with internal server error(but the
job is running)
and checkpoints shows no data and after quite sometime it shows checkpoint stats
Attaching the screenshot for your reference
!image-2021-12-22-11-53-41-478.png!
was:
Hi
We have recently upgraded the flink from {{1.13.3}} to {{1.14.2}} and then
after upgrade we are facing issues in the UI with internal server error(but the
job is running)
Attaching the screenshot for your reference
!image-2021-12-22-11-53-41-478.png!
> Upgrade of flink to 1.14.2 is showing internal server errors in the UI
> --
>
> Key: FLINK-25412
> URL: https://issues.apache.org/jira/browse/FLINK-25412
> Project: Flink
> Issue Type: Bug
>Reporter: srujankumar
>Priority: Major
> Attachments: image-2021-12-22-11-53-41-478.png
>
>
> Hi
> We have recently upgraded the flink from {{1.13.3}} to {{1.14.2}} and then
> after upgrade we are facing issues in the UI with internal server error(but
> the job is running)
> and checkpoints shows no data and after quite sometime it shows checkpoint
> stats
> Attaching the screenshot for your reference
> !image-2021-12-22-11-53-41-478.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (FLINK-25412) Upgrade of flink to 1.14.2 is showing internal server errors in the UI
srujankumar created FLINK-25412:
---
Summary: Upgrade of flink to 1.14.2 is showing internal server
errors in the UI
Key: FLINK-25412
URL: https://issues.apache.org/jira/browse/FLINK-25412
Project: Flink
Issue Type: Bug
Reporter: srujankumar
Attachments: image-2021-12-22-11-53-41-478.png
Hi
We have recently upgraded the flink from {{1.13.3}} to {{1.14.2}} and then
after upgrade we are facing issues in the UI with internal server error
Attaching the screenshot for your reference
!image-2021-12-22-11-53-41-478.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Updated] (FLINK-25412) Upgrade of flink to 1.14.2 is showing internal server errors in the UI
[
https://issues.apache.org/jira/browse/FLINK-25412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
srujankumar updated FLINK-25412:
Description:
Hi
We have recently upgraded the flink from {{1.13.3}} to {{1.14.2}} and then
after upgrade we are facing issues in the UI with internal server error(but the
job is running)
Attaching the screenshot for your reference
!image-2021-12-22-11-53-41-478.png!
was:
Hi
We have recently upgraded the flink from {{1.13.3}} to {{1.14.2}} and then
after upgrade we are facing issues in the UI with internal server error
Attaching the screenshot for your reference
!image-2021-12-22-11-53-41-478.png!
> Upgrade of flink to 1.14.2 is showing internal server errors in the UI
> --
>
> Key: FLINK-25412
> URL: https://issues.apache.org/jira/browse/FLINK-25412
> Project: Flink
> Issue Type: Bug
>Reporter: srujankumar
>Priority: Major
> Attachments: image-2021-12-22-11-53-41-478.png
>
>
> Hi
> We have recently upgraded the flink from {{1.13.3}} to {{1.14.2}} and then
> after upgrade we are facing issues in the UI with internal server error(but
> the job is running)
> Attaching the screenshot for your reference
> !image-2021-12-22-11-53-41-478.png!
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] yangjunhan commented on a change in pull request #18148: [FLINK-25372] Add thread dump feature for jobmanager
yangjunhan commented on a change in pull request #18148:
URL: https://github.com/apache/flink/pull/18148#discussion_r773623624
##
File path:
flink-runtime-web/web-dashboard/src/app/pages/job-manager/thread-dump/job-manager-thread-dump.component.ts
##
@@ -0,0 +1,72 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import { ChangeDetectorRef, Component, OnInit, ChangeDetectionStrategy,
OnDestroy } from '@angular/core';
+import { Subject } from 'rxjs';
+import { takeUntil } from 'rxjs/operators';
+
+import { EditorOptions } from 'ng-zorro-antd/code-editor/typings';
+import { flinkEditorOptions } from 'share/common/editor/editor-config';
+
+import { JobManagerService } from 'services';
+
+@Component({
+ selector: 'flink-job-manager-thread-dump',
+ templateUrl: './job-manager-thread-dump.component.html',
+ styleUrls: ['./job-manager-thread-dump.component.less'],
+ changeDetection: ChangeDetectionStrategy.OnPush
+})
+export class JobManagerThreadDumpComponent implements OnInit, OnDestroy {
+ public readonly editorOptions: EditorOptions = flinkEditorOptions;
+
+ public dump = '';
+ public loading = true;
+
+ private readonly destroy$ = new Subject();
+
+ constructor(private readonly jobManagerService: JobManagerService, private
readonly cdr: ChangeDetectorRef) {}
+
+ public ngOnInit(): void {
+this.reload();
+ }
+
+ public ngOnDestroy(): void {
+this.destroy$.next();
+this.destroy$.complete();
+ }
+
+ public reload(): void {
+this.loading = true;
+this.cdr.markForCheck();
+if (this.jobManagerService) {
+ this.jobManagerService
+.loadThreadDump()
+.pipe(takeUntil(this.destroy$))
+.subscribe(
+ data => {
+this.loading = false;
+this.dump = data;
+this.cdr.markForCheck();
+ },
+ () => {
+this.cdr.markForCheck();
Review comment:
you need to set loading false in error as well. Another way is using
`finalize` rxjs operator in the pipe so that you do not need to set loading
false in two places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-24952) Rowtime attributes must not be in the input rows of a regular join. As a workaround you can cast the time attributes of input tables to TIMESTAMP before
[ https://issues.apache.org/jira/browse/FLINK-24952?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangbaohua closed FLINK-24952. -- Resolution: Fixed > Rowtime attributes must not be in the input rows of a regular join. As a > workaround you can cast the time attributes of input tables to TIMESTAMP > before > > > Key: FLINK-24952 > URL: https://issues.apache.org/jira/browse/FLINK-24952 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Affects Versions: 1.13.1 >Reporter: wangbaohua >Priority: Major > > Rowtime attributes must not be in the input rows of a regular join. As a > workaround you can cast the time attributes of input tables to TIMESTAMP > before. > Please check the documentation for the set of currently supported SQL > features. > at > org.apache.flink.table.planner.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:82) > at > org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.$anonfun$optimize$1(FlinkChainedProgram.scala:62) > at > scala.collection.TraversableOnce.$anonfun$foldLeft$1(TraversableOnce.scala:156) > at > scala.collection.TraversableOnce.$anonfun$foldLeft$1$adapted(TraversableOnce.scala:156) > at scala.collection.Iterator.foreach(Iterator.scala:937) > at scala.collection.Iterator.foreach$(Iterator.scala:937) > at scala.collection.AbstractIterator.foreach(Iterator.scala:1425) > at scala.collection.IterableLike.foreach(IterableLike.scala:70) > at scala.collection.IterableLike.foreach$(IterableLike.scala:69) > at scala.collection.AbstractIterable.foreach(Iterable.scala:54) > at scala.collection.TraversableOnce.foldLeft(TraversableOnce.scala:156) > at scala.collection.TraversableOnce.foldLeft$(TraversableOnce.scala:154) > at scala.collection.AbstractTraversable.foldLeft(Traversable.scala:104) > at > org.apache.flink.table.planner.plan.optimize.program.FlinkChainedProgram.optimize(FlinkChainedProgram.scala:58) > at > org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.optimizeTree(StreamCommonSubGraphBasedOptimizer.scala:163) > at > org.apache.flink.table.planner.plan.optimize.StreamCommonSubGraphBasedOptimizer.doOptimize(StreamCommonSubGraphBasedOptimizer.scala:83) > at > org.apache.flink.table.planner.plan.optimize.CommonSubGraphBasedOptimizer.optimize(CommonSubGraphBasedOptimizer.scala:77) > at > org.apache.flink.table.planner.delegation.PlannerBase.optimize(PlannerBase.scala:279) > at > org.apache.flink.table.planner.delegation.PlannerBase.translate(PlannerBase.scala:163) > at > org.apache.flink.table.api.bridge.java.internal.StreamTableEnvironmentImpl.toStreamInternal(StreamTableEnvironmentImpl.java:439) > at > org.apache.flink.table.api.bridge.java.internal.StreamTableEnvironmentImpl.toRetractStream(StreamTableEnvironmentImpl.java:528) > at > org.apache.flink.table.api.bridge.java.internal.StreamTableEnvironmentImpl.toRetractStream(StreamTableEnvironmentImpl.java:517) > at rete.ReteDemo4.test(ReteDemo4.java:478) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306) > at > org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63) > at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329) > at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293) > at org.
[jira] [Closed] (FLINK-25103) KeyedBroadcastProcessFunction run set 6, parallelism ValueState variables A
[
https://issues.apache.org/jira/browse/FLINK-25103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
wangbaohua closed FLINK-25103.
--
Resolution: Fixed
> KeyedBroadcastProcessFunction run set 6, parallelism ValueState variables A
> ---
>
> Key: FLINK-25103
> URL: https://issues.apache.org/jira/browse/FLINK-25103
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
>Affects Versions: 1.14.0
>Reporter: wangbaohua
>Priority: Major
>
> KeyedBroadcastProcessFunction run set 6, parallelism ValueState variables A,
> excuse me how A stored in the six tasks. When I was running, I observed that
> some tasks fetched variable A was null, while others had values .The
> following code :
>
> setParallelism(9);
> ..
> public class dealStreamProcessFunction extends
> KeyedBroadcastProcessFunction,
> StandardEvent> {
> private static final Logger logger =
> LoggerFactory.getLogger(dealStreamProcessFunction.class);
> private transient ValueState> listState;
> private transient ValueState runingFlagState;
> private transient ValueState engineState;
> MapStateDescriptor> ruleStateDescriptor = new
> MapStateDescriptor<>(ContextInfo.RULE_SBROAD_CAST_STATE
> , BasicTypeInfo.STRING_TYPE_INFO
> , new ListTypeInfo<>(String.class));
> InferenceEngine engine;
> /**
> * open方法只会执行一次
> * 可以在这实现初始化的功能
> *
> * @param parameters
> * @throws Exception
> */
> @Override
> public void open(Configuration parameters) throws Exception {
> super.open(parameters);
> ValueStateDescriptor> recentOperatorsDescriptor =
> new ValueStateDescriptor>(
> "recent-operator",
> TypeInformation.of(new TypeHint>() {
> }));
> ValueStateDescriptor runingFlagDescriptor = new
> ValueStateDescriptor(
> "runingFlag",
> Boolean.class);
> ValueStateDescriptor engineDescriptor = new
> ValueStateDescriptor(
> "runingFlag1",
> InferenceEngine.class);
> engineState = getRuntimeContext().getState(engineDescriptor);
> listState = getRuntimeContext().getState(recentOperatorsDescriptor);
> runingFlagState = getRuntimeContext().getState(runingFlagDescriptor);
> logger.info("KeyedBroadcastProcessFunction open");
> }
> @Override
> public void processElement(StandardEvent standardEvent, ReadOnlyContext
> readOnlyContext, Collector collector) throws Exception {
> if(standardEvent == null){
> return;
> }
> List list = null;
> list =
> readOnlyContext.getBroadcastState(ruleStateDescriptor).get(ContextInfo.RULE_SBROAD_CAST_STATE);
> if (list == null) {
> logger.info("RulesBroadcastState is null..");
> List lst = listState.value();
> if (lst == null) {
> lst = new ArrayList<>();
> }
> lst.add(standardEvent);
> listState.update(lst);
> return;
> }
> //第一次进来
> if (runingFlagState.value() == null) {
> logger.info("runingFlagState.value() == null");
> runingFlagState.update(true);
> }
> if (((runingFlagState.value() && list.get(0).equals("1")) ||
> list.get(0).equals("0"))) {
> logger.info("action update.:" + list.size() + ":" +
> runingFlagState.value() + ":" + list.get(0));
> String flag = list.get(0);
> list.remove(0);
> InferenceEngine engine1 =
> InferenceEngine.compile(RuleReader.parseRules(list));
> engineState.update(engine1);
> if (runingFlagState.value() && flag.equals("1")) {
> runingFlagState.update(false);
> }
> }
> if (engineState.value() != null) {
> List listTmp = listState.value();
> if (listTmp != null) {
> for (StandardEvent standardEventTmp : listTmp) {
> logger.info("listState.:" + standardEventTmp);
> match(standardEventTmp, collector);
> }
> listState.clear();
> }
> match(standardEvent, collector);
> } else {
> logger.info("processElement engine is null.:");
> }
> }
> private void match(StandardEvent standardEvent, Collector
> collector) throws IOException {
> PatternMatcher matcher = engineState.value().matcher(standardEvent);
> if (matcher.find()) {
> List actions = matcher.getActions();
> for (Action action : actions) {
> if (standardEvent != null) {
>
[jira] [Closed] (FLINK-24885) ProcessElement Interface parameter Collector : java.lang.NullPointerException
[ https://issues.apache.org/jira/browse/FLINK-24885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangbaohua closed FLINK-24885. -- Resolution: Fixed > ProcessElement Interface parameter Collector : java.lang.NullPointerException > -- > > Key: FLINK-24885 > URL: https://issues.apache.org/jira/browse/FLINK-24885 > Project: Flink > Issue Type: Bug > Components: Table SQL / Runtime >Affects Versions: 1.13.1 >Reporter: wangbaohua >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0, 1.13.6, 1.14.3 > > Attachments: error.jpg > > > 2021-11-15 11:11:55,032 INFO com.asap.demo.function.dealMapFunction > [] - size:160 > 2021-11-15 11:11:55,230 WARN org.apache.flink.runtime.taskmanager.Task > [] - Co-Process-Broadcast-Keyed -> Map -> > DataSteamToTable(stream=default_catalog.default_database.Unregistered_DataStream_Source_8, > type=*com.asap.demo.model.BeanField<`account` STRING, `accountId` STRING, > `accountIn` STRING, `accountInName` STRING, `accountInOrgId` STRING, > `accountInOrgName` STRING, `accountInType` STRING, `accountName` STRING, > `accountOrgId` STRING, `accountOrgName` STRING, `accountOut` STRING, > `accountOutName` STRING, `accountOutOrgId` STRING, `accountOutOrgName` > STRING, `accountOutType` STRING, `accountStatus` STRING, `accountType` > STRING, `action` STRING, `actionDesc` STRING, `alarmcontext` STRING, > `alarmgrade` STRING, `alarmtype` STRING, `alertId` STRING, `alertInfo` > STRING, `alertLevel` STRING, `alertSignatureIdL` STRING, `appId` STRING, > `appName` STRING, `appProtocol` STRING, `appType` STRING, `areaId` STRING, > `areaName` STRING, `areaType` STRING, `assetFrom` STRING, `assetId` STRING, > `assetInfo` STRING, `assetIp` STRING, `assetLevel` STRING, `assetName` > STRING, `assetPid` STRING, `assetType` STRING, `assetUse` STRING, > `assetVendor` STRING, `attackStage` STRING, `attackStageCode` STRING, > `attackType` STRING, `attackTypeName` STRING, `authSerNum` STRING, `authTime` > STRING, `authType` STRING, `bankSeqNum` STRING, `batchNo` STRING, > `blackDomain` STRING, `blackDomainDesc` STRING, `blackDomainTag` STRING, > `blackDstIp` STRING, `blackFile` STRING, `blackFileDesc` STRING, > `blackFileTag` STRING, `blackId` STRING, `blackIpTag` STRING, `blackSrcIp` > STRING, `blackTag` STRING, `blackTagMatchCount` STRING, `blackTagMatchDesc` > STRING, `blackUrl` STRING, `blackUrlDesc` STRING, `blackUrlTag` STRING, > `blackVulnCve` STRING, `blackVulnDesc` STRING, `blackVulnName` STRING, > `blackVulnTag` STRING, `branchId` STRING, `branchName` STRING, > `businessSystemName` STRING, `businessType` STRING, `cardId` STRING, > `cascadeSourceIp` STRING, `cascadeSourceName` STRING, `cebUid` STRING, > `certNum` STRING, `certType` STRING, `chainId` STRING, `channel` STRING, > `channelId` STRING, `character` STRING, `charge` STRING, `cifSeqNum` STRING, > `clientInfo` STRING, `clientIp` STRING, `clientMac` STRING, `clientName` > STRING, `clientPort` STRING, `collectTime` TIMESTAMP_LTZ(9), `collectTimeL` > TIMESTAMP_LTZ(9), `command` STRING, `commandLine` STRING, `commandResult` > STRING, `company` STRING, `companyCustomId` STRING, `companyId` STRING, > `completenessTag` STRING, `confidence` STRING, `confidenceLevel` STRING, > `consignedUser` STRING, `contractNo` STRING, `count` STRING, `couponAmount` > STRING, `couponId` STRING, `createTime` TIMESTAMP_LTZ(3), `createTimeL` > BIGINT, `createdBy` STRING, `curType` STRING, `currency` STRING, `currentBal` > STRING, `customLabel1` STRING, `customLabel10` STRING, `customLabel2` STRING, > `customLabel3` STRING, `customLabel4` STRING, `customLabel5` STRING, > `customLabel6` STRING, `customLabel7` STRING, `customLabel8` STRING, > `customLabel9` STRING, `customValue1` STRING, `customValue10` STRING, > `customValue2` STRING, `customValue3` STRING, `customValue4` STRING, > `customValue5` STRING, `customValue6` STRING, `customValue7` STRING, > `customValue8` STRING, `customValue9` STRING, `dataQualityTag` STRING, > `dataType` STRING, `dataTypeName` STRING, `dbInstance` STRING, `dbName` > STRING, `dbTable` STRING, `dbVersion` STRING, `dealSuggest` STRING, > `defVManagerId` STRING, `department` STRING, `deviceCategory` STRING, > `deviceId` STRING, `deviceIp` STRING, `deviceMac` STRING, `deviceName` > STRING, `deviceParentType` STRING, `deviceType` STRING, `deviceVersion` > STRING, `direction` STRING, `directionDesc` STRING, `directionOfAttackTag` > STRING, `domain` STRING, `dstAdminAccount` STRING, `dstAdminEmail` STRING, > `dstAdminFOrgId` STRING, `dstAdminId` STRING, `dstAdminMobile` STRING, > `dstAdminName` STRING, `dstAdminOrgId` STRING, `dstAdminOrgName` STRING, > `dstAdminType` STRING, `dstA
[GitHub] [flink] jelly-1203 commented on pull request #18017: [FLINK-25171] Validation of duplicate fields in ddl sql
jelly-1203 commented on pull request #18017: URL: https://github.com/apache/flink/pull/18017#issuecomment-999311507 Hi, @godfreyhe Please have time to do the final check. If there is any deficiency, I will continue to improve -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] becketqin commented on pull request #17991: [FLINK-25132][connector/kafka] Move record deserializing from SplitFetcher to RecordEmitter to support object-reusing deserializer
becketqin commented on pull request #17991: URL: https://github.com/apache/flink/pull/17991#issuecomment-999305905 Merged to master: 2b1a9dea74a334adb1fe890f024f4153ad11a985 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] becketqin merged pull request #17991: [FLINK-25132][connector/kafka] Move record deserializing from SplitFetcher to RecordEmitter to support object-reusing deserializer
becketqin merged pull request #17991: URL: https://github.com/apache/flink/pull/17991 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] becketqin commented on pull request #17991: [FLINK-25132][connector/kafka] Move record deserializing from SplitFetcher to RecordEmitter to support object-reusing deserializer
becketqin commented on pull request #17991: URL: https://github.com/apache/flink/pull/17991#issuecomment-999305583 Thanks for the patch. LGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-25411) JsonRowSerializationSchema unable to parse TIMESTAMP_LTZ fields
Surendra Lalwani created FLINK-25411: Summary: JsonRowSerializationSchema unable to parse TIMESTAMP_LTZ fields Key: FLINK-25411 URL: https://issues.apache.org/jira/browse/FLINK-25411 Project: Flink Issue Type: Bug Components: API / DataStream, API / Type Serialization System Affects Versions: 1.13.3 Reporter: Surendra Lalwani While I try to fire a simple query Select current_timestamp from table_name , it gives error that Could not serialize row and asks me to add shaded flink dependency for jsr-310. Seems like in the Serializer , the JavaTimeModule is not added -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (FLINK-25410) Flink CLI should exit when app is accepted with detach mode on Yarn
[ https://issues.apache.org/jira/browse/FLINK-25410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463588#comment-17463588 ] Junfan Zhang edited comment on FLINK-25410 at 12/22/21, 5:47 AM: - Could you help check this feature? [~guoyangze] [~trohrmann] [~xtsong] [~yunta] [~wangyang0918] If OK, please assign to me. PR will be attached sooner Thanks ~ was (Author: zuston): Could you help check this feature? [~guoyangze] [~trohrmann] [~xtsong] [~yunta] If OK, please assign to me. PR will be attached sooner Thanks ~ > Flink CLI should exit when app is accepted with detach mode on Yarn > --- > > Key: FLINK-25410 > URL: https://issues.apache.org/jira/browse/FLINK-25410 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Reporter: Junfan Zhang >Priority: Major > > h2. Why > In our internal streaming platform, we will use flink-cli tool to submit > Flink streaming application on Yarn. > However when encountering Hadoop cluster down and then lots of flink apps > need to be resubmitted, the submitter of worker in our platform will hang at > this time. > Because the Yarn cluster resources are tight and the scheduling efficiency > becomes low when lots of apps needs to be started. > And flink-cli will not exit until the app status changes to running. > In addition, I also think there is no need to wait when app status is > accepted with detach mode on Yarn. > h2. How > When app in accpeted status, flink-cli should exit directly to release > submitter worker process resource. The PR could refer to : > https://github.com/apache/flink/blob/f191becdb42d6df823a103dc4f787c4737baa8e7/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L1224 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (FLINK-25410) Flink CLI should exit when app is accepted with detach mode on Yarn
[ https://issues.apache.org/jira/browse/FLINK-25410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463588#comment-17463588 ] Junfan Zhang edited comment on FLINK-25410 at 12/22/21, 5:43 AM: - Could you help check this feature? [~guoyangze] [~trohrmann] [~xtsong] [~yunta] If OK, please assign to me. PR will be attached sooner Thanks ~ was (Author: zuston): Could you help check this feature? [~guoyangze] [~trohrmann] [~xtsong] [~yunta] Thanks ~ > Flink CLI should exit when app is accepted with detach mode on Yarn > --- > > Key: FLINK-25410 > URL: https://issues.apache.org/jira/browse/FLINK-25410 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Reporter: Junfan Zhang >Priority: Major > > h2. Why > In our internal streaming platform, we will use flink-cli tool to submit > Flink streaming application on Yarn. > However when encountering Hadoop cluster down and then lots of flink apps > need to be resubmitted, the submitter of worker in our platform will hang at > this time. > Because the Yarn cluster resources are tight and the scheduling efficiency > becomes low when lots of apps needs to be started. > And flink-cli will not exit until the app status changes to running. > In addition, I also think there is no need to wait when app status is > accepted with detach mode on Yarn. > h2. How > When app in accpeted status, flink-cli should exit directly to release > submitter worker process resource. The PR could refer to : > https://github.com/apache/flink/blob/f191becdb42d6df823a103dc4f787c4737baa8e7/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L1224 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Comment Edited] (FLINK-25328) Improvement of reuse segments for join/agg/sort operators in TaskManager for flink olap queries
[ https://issues.apache.org/jira/browse/FLINK-25328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463590#comment-17463590 ] Shammon edited comment on FLINK-25328 at 12/22/21, 5:42 AM: Thanks [~xtsong], I think it's interesting about pending requests. I agree that when join/agg operators and RocksDB / Python request more segment and they cant allocate segments from memory any more(the total usage of segments has reached to the maximum), we can pending these requests until new segments are free. I think it's a good improvement about `MemoryManager`. I wonder if I understand the second point correctly, conversely, when some tasks are finished, their free segments should not be deallocated immediately even when there're no pending requests. These segments should be reused by the later tasks, and we can deallocate them in a periodic time to decrease the usage of memory. What do you think? :) was (Author: zjureel): Thanks [~xtsong], I think it's interesting about pending requests. I agree that when join/agg operators and RocksDB / Python request more segment and they cant allocate segments from memory any more(the total usage of segments has reached to the maximum), we can pending these requests until new segments are free. I think it's a good improvement about `MemoryManager`. I wonder if I understand the second point correctly, conversely, when some tasks are finished, their free segments should not be deallocated immediately even when there're no pending requests. These segments should be reused by the later tasks, what do you think? :) > Improvement of reuse segments for join/agg/sort operators in TaskManager for > flink olap queries > --- > > Key: FLINK-25328 > URL: https://issues.apache.org/jira/browse/FLINK-25328 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Affects Versions: 1.14.0, 1.12.5, 1.13.3 >Reporter: Shammon >Priority: Major > > We submit batch jobs to flink session cluster as olap queries, and these > jobs' subtasks in TaskManager are frequently created and destroyed because > they finish their work quickly. Each slot in taskmanager manages > `MemoryManager` for multiple tasks in one job, and the `MemoryManager` is > closed when all the subtasks are finished. Join/Aggregate/Sort and etc. > operators in the subtasks allocate `MemorySegment` via `MemoryManager` and > these `MemorySegment` will be free when they are finished. > > It causes too much memory allocation and free of `MemorySegment` in > taskmanager. For example, a TaskManager contains 50 slots, one job has 3 > join/agg operatos run in the slot, each operator will allocate 2000 segments > and initialize them. If the subtasks of a job take 100ms to execute, then the > taskmanager will execute 10 jobs' subtasks one second and it will allocate > and free 2000 * 3 * 50 * 10 = 300w segments for them. Allocate and free too > many segments from memory will cause two issues: > 1) Increases the CPU usage of taskmanager > 2) Increase the cost of subtasks in taskmanager, which will increase the > latency of job and decrease the qps. > To improve the usage of memory segment between jobs in the same slot, > we propose not drop memory manager when all the subtasks in the slot are > finished. The slot will hold the `MemoryManager` and not free the allocated > `MemorySegment` in it immediately. When some subtasks of another job are > assigned to the slot, they don't need to allocate segments from memory and > can reuse the `MemoryManager` and `MemorySegment` in it. WDYT? [~xtsong] THX -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25328) Improvement of reuse segments for join/agg/sort operators in TaskManager for flink olap queries
[ https://issues.apache.org/jira/browse/FLINK-25328?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463590#comment-17463590 ] Shammon commented on FLINK-25328: - Thanks [~xtsong], I think it's interesting about pending requests. I agree that when join/agg operators and RocksDB / Python request more segment and they cant allocate segments from memory any more(the total usage of segments has reached to the maximum), we can pending these requests until new segments are free. I think it's a good improvement about `MemoryManager`. I wonder if I understand the second point correctly, conversely, when some tasks are finished, their free segments should not be deallocated immediately even when there're no pending requests. These segments should be reused by the later tasks, what do you think? :) > Improvement of reuse segments for join/agg/sort operators in TaskManager for > flink olap queries > --- > > Key: FLINK-25328 > URL: https://issues.apache.org/jira/browse/FLINK-25328 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Affects Versions: 1.14.0, 1.12.5, 1.13.3 >Reporter: Shammon >Priority: Major > > We submit batch jobs to flink session cluster as olap queries, and these > jobs' subtasks in TaskManager are frequently created and destroyed because > they finish their work quickly. Each slot in taskmanager manages > `MemoryManager` for multiple tasks in one job, and the `MemoryManager` is > closed when all the subtasks are finished. Join/Aggregate/Sort and etc. > operators in the subtasks allocate `MemorySegment` via `MemoryManager` and > these `MemorySegment` will be free when they are finished. > > It causes too much memory allocation and free of `MemorySegment` in > taskmanager. For example, a TaskManager contains 50 slots, one job has 3 > join/agg operatos run in the slot, each operator will allocate 2000 segments > and initialize them. If the subtasks of a job take 100ms to execute, then the > taskmanager will execute 10 jobs' subtasks one second and it will allocate > and free 2000 * 3 * 50 * 10 = 300w segments for them. Allocate and free too > many segments from memory will cause two issues: > 1) Increases the CPU usage of taskmanager > 2) Increase the cost of subtasks in taskmanager, which will increase the > latency of job and decrease the qps. > To improve the usage of memory segment between jobs in the same slot, > we propose not drop memory manager when all the subtasks in the slot are > finished. The slot will hold the `MemoryManager` and not free the allocated > `MemorySegment` in it immediately. When some subtasks of another job are > assigned to the slot, they don't need to allocate segments from memory and > can reuse the `MemoryManager` and `MemorySegment` in it. WDYT? [~xtsong] THX -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25410) Flink CLI should exit when app is accepted with detach mode on Yarn
[ https://issues.apache.org/jira/browse/FLINK-25410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463588#comment-17463588 ] Junfan Zhang commented on FLINK-25410: -- Could you help check this feature? [~guoyangze] [~trohrmann] [~xtsong] [~yunta] Thanks ~ > Flink CLI should exit when app is accepted with detach mode on Yarn > --- > > Key: FLINK-25410 > URL: https://issues.apache.org/jira/browse/FLINK-25410 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Reporter: Junfan Zhang >Priority: Major > > h2. Why > In our internal streaming platform, we will use flink-cli tool to submit > Flink streaming application on Yarn. > However when encountering Hadoop cluster down and then lots of flink apps > need to be resubmitted, the submitter of worker in our platform will hang at > this time. > Because the Yarn cluster resources are tight and the scheduling efficiency > becomes low when lots of apps needs to be started. > And flink-cli will not exit until the app status changes to running. > In addition, I also think there is no need to wait when app status is > accepted with detach mode on Yarn. > h2. How > When app in accpeted status, flink-cli should exit directly to release > submitter worker process resource. The PR could refer to : > https://github.com/apache/flink/blob/f191becdb42d6df823a103dc4f787c4737baa8e7/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L1224 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25410) Flink CLI should exit when app is accepted with detach mode on Yarn
[ https://issues.apache.org/jira/browse/FLINK-25410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junfan Zhang updated FLINK-25410: - Description: h2. Why In our internal streaming platform, we will use flink-cli tool to submit Flink streaming application on Yarn. However when encountering Hadoop cluster down and then lots of flink apps need to be resubmitted, the submitter of worker in our platform will hang at this time. Because the Yarn cluster resources are tight and the scheduling efficiency becomes low when lots of apps needs to be started. And flink-cli will not exit until the app status changes to running. In addition, I also think there is no need to wait when app status is accepted with detach mode on Yarn. h2. How When app in accpeted status, flink-cli should exit directly to release submitter worker process resource. The PR could refer to : https://github.com/apache/flink/blob/f191becdb42d6df823a103dc4f787c4737baa8e7/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L1224 was: h2. Why In our internal streaming platform, we will use flink-cli tool to submit Flink streaming application on Yarn. However when encountering Hadoop cluster down and then lots of flink apps need to be resubmitted, the submitter of worker in our platform will hang. Because the Yarn cluster resources are tight, flink-cli will exit until the app's status change to running > Flink CLI should exit when app is accepted with detach mode on Yarn > --- > > Key: FLINK-25410 > URL: https://issues.apache.org/jira/browse/FLINK-25410 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Reporter: Junfan Zhang >Priority: Major > > h2. Why > In our internal streaming platform, we will use flink-cli tool to submit > Flink streaming application on Yarn. > However when encountering Hadoop cluster down and then lots of flink apps > need to be resubmitted, the submitter of worker in our platform will hang at > this time. > Because the Yarn cluster resources are tight and the scheduling efficiency > becomes low when lots of apps needs to be started. > And flink-cli will not exit until the app status changes to running. > In addition, I also think there is no need to wait when app status is > accepted with detach mode on Yarn. > h2. How > When app in accpeted status, flink-cli should exit directly to release > submitter worker process resource. The PR could refer to : > https://github.com/apache/flink/blob/f191becdb42d6df823a103dc4f787c4737baa8e7/flink-yarn/src/main/java/org/apache/flink/yarn/YarnClusterDescriptor.java#L1224 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-25410) Flink CLI should exit when app is accepted with detach mode on Yarn
[ https://issues.apache.org/jira/browse/FLINK-25410?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Junfan Zhang updated FLINK-25410: - Description: h2. Why In our internal streaming platform, we will use flink-cli tool to submit Flink streaming application on Yarn. However when encountering Hadoop cluster down and then lots of flink apps need to be resubmitted, the submitter of worker in our platform will hang. Because the Yarn cluster resources are tight, flink-cli will exit until the app's status change to running > Flink CLI should exit when app is accepted with detach mode on Yarn > --- > > Key: FLINK-25410 > URL: https://issues.apache.org/jira/browse/FLINK-25410 > Project: Flink > Issue Type: Improvement > Components: Deployment / YARN >Reporter: Junfan Zhang >Priority: Major > > h2. Why > In our internal streaming platform, we will use flink-cli tool to submit > Flink streaming application on Yarn. > However when encountering Hadoop cluster down and then lots of flink apps > need to be resubmitted, the submitter of worker in our platform will hang. > Because the Yarn cluster resources are tight, flink-cli will exit until the > app's status change to running -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-25410) Flink CLI should exit when app is accepted with detach mode on Yarn
Junfan Zhang created FLINK-25410: Summary: Flink CLI should exit when app is accepted with detach mode on Yarn Key: FLINK-25410 URL: https://issues.apache.org/jira/browse/FLINK-25410 Project: Flink Issue Type: Improvement Components: Deployment / YARN Reporter: Junfan Zhang -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] xintongsong commented on a change in pull request #18148: [FLINK-25372] Add thread dump feature for jobmanager
xintongsong commented on a change in pull request #18148:
URL: https://github.com/apache/flink/pull/18148#discussion_r773568386
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/dispatcher/Dispatcher.java
##
@@ -693,6 +696,21 @@ JobManagerRunner createJobManagerRunner(JobGraph jobGraph,
long initializationTi
timeout));
}
+@Override
+public CompletableFuture requestThreadDump(Time timeout) {
+final Collection threadDump = JvmUtils.createThreadDump();
+
+final Collection threadInfos =
+threadDump.stream()
+.map(
+threadInfo ->
+ThreadDumpInfo.ThreadInfo.create(
+threadInfo.getThreadName(),
threadInfo.toString()))
+.collect(Collectors.toList());
+
+return
CompletableFuture.completedFuture(ThreadDumpInfo.create(threadInfos));
+}
Review comment:
This is identical to `TaskExecutor#requestThreadDump`. We can
deduplicate this by introducing a static method in `ThreadDumpInfo`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-17808) Rename checkpoint meta file to "_metadata" until it has completed writing
[
https://issues.apache.org/jira/browse/FLINK-17808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463584#comment-17463584
]
Junfan Zhang commented on FLINK-17808:
--
[~yunta]
As Stephan mentioned above, i also think the first one is the better option.
So we could use the {{RecoverableFsDataOutputStream}} close and commit to
ensure the writing file atomicity instead of using the {{FSDataOutputStream}}
in {{FsCheckpointMetadataOutputStream}}. Right?
Please let me know what u think.
> Rename checkpoint meta file to "_metadata" until it has completed writing
> -
>
> Key: FLINK-17808
> URL: https://issues.apache.org/jira/browse/FLINK-17808
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.10.0
>Reporter: Yun Tang
>Priority: Minor
> Labels: auto-deprioritized-major, pull-request-available
> Fix For: 1.15.0
>
>
> In practice, some developers or customers would use some strategy to find the
> recent _metadata as the checkpoint to recover (e.g as many proposals in
> FLINK-9043 suggest). However, there existed a "_meatadata" file does not mean
> the checkpoint have been completed as the writing to create the "_meatadata"
> file could break as some force quit (e.g. yarn application -kill).
> We could create the checkpoint meta stream to write data to file named as
> "_metadata.inprogress" and renamed it to "_metadata" once completed writing.
> By doing so, we could ensure the "_metadata" is not broken.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
flinkbot edited a comment on pull request #17937: URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170 ## CI report: * 463b2c0160ba8892fa654a7689f9a58d516583e5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398) * c1728e5765b33b6ba1140a2f313687eb3bbbaf5f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28460) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17937: [FLINK-25044][testing][Pulsar Connector] Add More Unit Test For Pulsar Source
flinkbot edited a comment on pull request #17937: URL: https://github.com/apache/flink/pull/17937#issuecomment-981287170 ## CI report: * 463b2c0160ba8892fa654a7689f9a58d516583e5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28398) * c1728e5765b33b6ba1140a2f313687eb3bbbaf5f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25398) Show complete stacktrace when requesting thread dump
[
https://issues.apache.org/jira/browse/FLINK-25398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463582#comment-17463582
]
Junfan Zhang commented on FLINK-25398:
--
[~Thesharing] Thanks for your advice. Agree with u, i will introduce the extra
config to control the max depth.
Could you help tell me how to introduce new unit test to cover it? I have no
ideas on it. :D. If just testing the rpc response and request, i think the
previous UT is enough.
> Show complete stacktrace when requesting thread dump
>
>
> Key: FLINK-25398
> URL: https://issues.apache.org/jira/browse/FLINK-25398
> Project: Flink
> Issue Type: Improvement
>Reporter: Junfan Zhang
>Priority: Major
> Labels: pull-request-available
> Attachments: stacktrace.png
>
>
> h2. Why
> Now the stacktrace is not complete when clicking the task executor's
> threaddump in runtime webui. Hence it's hard to the initial calling
> according to the stacktrace.
> Now the thread stacktrace is limited to 8, refer to openjdk:
> [https://github.com/openjdk/jdk/blob/master/src/java.management/share/classes/java/lang/management/ThreadInfo.java#L597]
>
> h2. Solution
> Using the custom {{stringify}} method to return stacktrace instead of using
> {{ThreadInfo.toString}} directly
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18114: [FLINK-25173][table][hive] Introduce CatalogLock and implement HiveCatalogLock
flinkbot edited a comment on pull request #18114: URL: https://github.com/apache/flink/pull/18114#issuecomment-994556266 ## CI report: * ad59747459a1ff9ff00e772282f8f02a955fafdd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28363) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18114: [FLINK-25173][table][hive] Introduce CatalogLock and implement HiveCatalogLock
flinkbot edited a comment on pull request #18114: URL: https://github.com/apache/flink/pull/18114#issuecomment-994556266 ## CI report: * ad59747459a1ff9ff00e772282f8f02a955fafdd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28363) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16571: [hotfix][connector/kafka] fix missing nullable annotion
flinkbot edited a comment on pull request #16571: URL: https://github.com/apache/flink/pull/16571#issuecomment-884882802 ## CI report: * 18b95daab0297d86041683e92112c75b17c17bc8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20843) * 83e4441ce01d748577c9527ebad316ec9ec11fd7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28459) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on pull request #18114: [FLINK-25173][table][hive] Introduce CatalogLock and implement HiveCatalogLock
JingsongLi commented on pull request #18114: URL: https://github.com/apache/flink/pull/18114#issuecomment-999271045 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18014: [FLINK-24857][test][Kafka] Upgrade SourceReaderTestBase t…
flinkbot edited a comment on pull request #18014: URL: https://github.com/apache/flink/pull/18014#issuecomment-986441468 ## CI report: * 163e826eea7eb58d13e9712562a231bdabae9052 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28369) * e632511ff5874b350c838d7bef49e515f7981f47 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28458) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #16571: [hotfix][connector/kafka] fix missing nullable annotion
flinkbot edited a comment on pull request #16571: URL: https://github.com/apache/flink/pull/16571#issuecomment-884882802 ## CI report: * 18b95daab0297d86041683e92112c75b17c17bc8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=20843) * 83e4441ce01d748577c9527ebad316ec9ec11fd7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25286) Improve connector testing framework to support more scenarios
[ https://issues.apache.org/jira/browse/FLINK-25286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463573#comment-17463573 ] Qingsheng Ren commented on FLINK-25286: --- [~monster#12] Thanks for supporting our work on connector testing framework! Currently we are working on a refactor on the interfaces of connector testing framework (FLINK-25287). Feel free to help us to review the PR and have a try on the new interfaces on your connectors~ It'll be appreciated to have any suggestions from connector developers. > Improve connector testing framework to support more scenarios > - > > Key: FLINK-25286 > URL: https://issues.apache.org/jira/browse/FLINK-25286 > Project: Flink > Issue Type: Improvement > Components: Test Infrastructure >Reporter: Qingsheng Ren >Priority: Major > Fix For: 1.15.0 > > > Currently connector testing framework only support tests for DataStream > sources, and available scenarios are quite limited by current interface > design. > This ticket proposes to made improvements to connector testing framework for > supporting more test scenarios, and add test suites for sink and Table/SQL > API. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18014: [FLINK-24857][test][Kafka] Upgrade SourceReaderTestBase t…
flinkbot edited a comment on pull request #18014: URL: https://github.com/apache/flink/pull/18014#issuecomment-986441468 ## CI report: * 163e826eea7eb58d13e9712562a231bdabae9052 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28369) * e632511ff5874b350c838d7bef49e515f7981f47 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25293) Option to let fail if KafkaSource keeps failing to commit offset
[
https://issues.apache.org/jira/browse/FLINK-25293?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463569#comment-17463569
]
Qingsheng Ren commented on FLINK-25293:
---
Thanks for the ticket [~rero] !
Basically Flink doesn't rely on committed offsets for fault tolerance, so we
don't treat it as a severe failure if commit fails since it doesn't break the
correctness of Flink job.
I'm a little bit wondering about your usage of KafkaSource. Is there any cases
that the offset commit failure can only be recovered by restarting the Flink
job? If the purpose of failing the Flink job is just to alert user about the
offset commit failure, I think a better approach is to setup a metric system
and create an alarm to monitor metric "commitsFailed".
> Option to let fail if KafkaSource keeps failing to commit offset
>
>
> Key: FLINK-25293
> URL: https://issues.apache.org/jira/browse/FLINK-25293
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka
>Affects Versions: 1.14.0
> Environment: Flink 1.14.0
>Reporter: rerorero
>Priority: Major
>
> Is it possible to let KafkaSource fail if it keeps failing to commit offset?
>
> I faced an issue where KafkaSource keeps failing and never recover, while
> it's logging like these logs:
> {code:java}
> 2021-12-08 22:18:34,155 INFO
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator [] -
> [Consumer clientId=dbz-cg-1, groupId=dbz-cg] Group coordinator
> b4-pkc-x.asia-northeast1.gcp.confluent.cloud:9092 (id: 2147483643 rack:
> null) is unavailable or invalid due to cause: null.isDisconnected: true.
> Rediscovery will be attempted.
> 2021-12-08 22:18:34,157 WARN
> org.apache.flink.connector.kafka.source.reader.KafkaSourceReader [] - Failed
> to commit consumer offsets for checkpoint 13 {code}
> This is happening not just once, but a couple of times a week (it happens
> when the Kafka broker performs rolling restart). It can be recovered by
> restarting the Flink Job.
> I found other people reporting the similar thing:
> [https://lists.apache.org/thread/8l4f2yb4qwysdn1cj1wjk99tfb79kgs2]. This
> could possibly be a problem with the Kafka client, and of course, the problem
> should be fixed on Kafka side if so.
> However, Flink Kafka connector doesn't provide an automatic way to save this
> situation. KafkaSource keeps retrying forever when a retriable error occurs,
> even if it is not retriable actually:
> [https://github.com/apache/flink/blob/afb29d92c4e76ec6a453459c3d8a08304efec549/flink-connectors/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/source/reader/KafkaSourceReader.java#L144-L148]
> Since it sends metrics of the number of times a commit fails, it could be
> automated by monitoring it and restarting the job, but that would mean we
> need to have a new process to be managed.
> Does it make sense to have KafkaSource have the option like, let the source
> task fail if it keeps failing to commit an offset more than X times?
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[GitHub] [flink] JingsongLi commented on a change in pull request #18135: [FLINK-24809][table-common][table-planner] Fix precision for aggs on DECIMAL types
JingsongLi commented on a change in pull request #18135:
URL: https://github.com/apache/flink/pull/18135#discussion_r773580701
##
File path:
flink-table/flink-table-planner/src/test/scala/org/apache/flink/table/planner/runtime/stream/sql/AggregateITCase.scala
##
@@ -831,34 +921,53 @@ class AggregateITCase(
@Test
def testDifferentTypesSumWithRetract(): Unit = {
-val data = List(
- (1.toByte, 1.toShort, 1, 1L, 1.0F, 1.0, "a"),
- (2.toByte, 2.toShort, 2, 2L, 2.0F, 2.0, "a"),
- (3.toByte, 3.toShort, 3, 3L, 3.0F, 3.0, "a"),
- (3.toByte, 3.toShort, 3, 3L, 3.0F, 3.0, "a"),
- (1.toByte, 1.toShort, 1, 1L, 1.0F, 1.0, "b"),
- (2.toByte, 2.toShort, 2, 2L, 2.0F, 2.0, "b"),
- (3.toByte, 3.toShort, 3, 3L, 3.0F, 3.0, "c"),
- (3.toByte, 3.toShort, 3, 3L, 3.0F, 3.0, "c")
-)
-
-val t = failingDataSource(data).toTable(tEnv, 'a, 'b, 'c, 'd, 'e, 'f, 'g)
-tEnv.registerTable("T", t)
-
-// We use sub-query + limit here to ensure retraction
-val sql =
- """
-|SELECT sum(a), sum(b), sum(c), sum(d), sum(e), sum(f), sum(h) FROM (
-| SELECT *, CAST(c AS DECIMAL(3, 2)) AS h FROM T LIMIT 8
-|) GROUP BY g
- """.stripMargin
+val upsertSourceCurrencyData = List(
+ changelogRow("+I", Byte.box(1), Short.box(1), Int.box(1), Long.box(1),
+Float.box(1.0F), Double.box(1.0), "a"),
+ changelogRow("+I", Byte.box(2), Short.box(2), Int.box(2), Long.box(2),
+Float.box(2.0F), Double.box(2.0), "a"),
+ changelogRow("-D", Byte.box(1), Short.box(1), Int.box(1), Long.box(1),
+Float.box(1.0F), Double.box(1.0), "a"),
+ changelogRow("+I", Byte.box(3), Short.box(3), Int.box(3), Long.box(3),
+Float.box(3.0F), Double.box(3.0), "a"),
+ changelogRow("-D", Byte.box(2), Short.box(2), Int.box(2), Long.box(2),
+Float.box(2.0F), Double.box(2.0), "a"),
+ changelogRow("+I", Byte.box(1), Short.box(1), Int.box(1), Long.box(1),
+Float.box(1.0F), Double.box(1.0), "a"),
+ changelogRow("-D", Byte.box(3), Short.box(3), Int.box(3), Long.box(3),
+Float.box(3.0F), Double.box(3.0), "a"),
+ changelogRow("+I", Byte.box(2), Short.box(2), Int.box(2), Long.box(2),
+Float.box(2.0F), Double.box(2.0), "a"),
+ changelogRow("+I", Byte.box(3), Short.box(3), Int.box(3), Long.box(3),
+Float.box(3.0F), Double.box(3.0), "a"))
+
+val upsertSourceDataId = registerData(upsertSourceCurrencyData);
Review comment:
minor: remove ";"
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/aggfunctions/SumWithRetractAggFunction.java
##
@@ -194,21 +198,31 @@ protected Expression zeroLiteral() {
/** Built-in Decimal Sum with retract aggregate function. */
public static class DecimalSumWithRetractAggFunction extends
SumWithRetractAggFunction {
-private DecimalType decimalType;
+private final DataType resultType;
public DecimalSumWithRetractAggFunction(DecimalType decimalType) {
-this.decimalType = decimalType;
+DecimalType sumType = (DecimalType)
LogicalTypeMerging.findSumAggType(decimalType);
+this.resultType = DataTypes.DECIMAL(sumType.getPrecision(),
sumType.getScale());
}
@Override
public DataType getResultType() {
-DecimalType sumType = (DecimalType)
LogicalTypeMerging.findSumAggType(decimalType);
-return DataTypes.DECIMAL(sumType.getPrecision(),
sumType.getScale());
+return resultType;
}
@Override
protected Expression zeroLiteral() {
return literal(0);
}
+
+protected UnresolvedCallExpression adjustedPlus(
+UnresolvedReferenceExpression arg1,
UnresolvedReferenceExpression arg2) {
+return aggDecimalPlus(arg1, arg2);
+}
+
+protected UnresolvedCallExpression adjustedMinus(
Review comment:
ditto
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/aggfunctions/SumWithRetractAggFunction.java
##
@@ -194,21 +198,31 @@ protected Expression zeroLiteral() {
/** Built-in Decimal Sum with retract aggregate function. */
public static class DecimalSumWithRetractAggFunction extends
SumWithRetractAggFunction {
-private DecimalType decimalType;
+private final DataType resultType;
public DecimalSumWithRetractAggFunction(DecimalType decimalType) {
-this.decimalType = decimalType;
+DecimalType sumType = (DecimalType)
LogicalTypeMerging.findSumAggType(decimalType);
+this.resultType = DataTypes.DECIMAL(sumType.getPrecision(),
sumType.getScale());
}
@Override
public DataType getResultType() {
-DecimalType sumType = (DecimalType)
LogicalTypeMerging.findSumAggType(decimalType)
[GitHub] [flink] flinkbot edited a comment on pull request #17000: [FLINK-23946][clients] Application mode should shutdown gracefully on…
flinkbot edited a comment on pull request #17000: URL: https://github.com/apache/flink/pull/17000#issuecomment-906394364 ## CI report: * 11c78d1d81896eaea685b7a2e097bdf1124a8e45 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28447) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #17819: [FLINK-15816][k8s] Limit kubernetes.cluster-id to a maximum of 40 characters
wangyang0918 commented on pull request #17819: URL: https://github.com/apache/flink/pull/17819#issuecomment-999262358 It is a good idea to introduce such a `enum`. And I do not think we will have the prefix and suffix for the K8s resource names or labels. +1 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18068: [FLINK-25105][checkpoint] Enables final checkpoint by default
flinkbot edited a comment on pull request #18068: URL: https://github.com/apache/flink/pull/18068#issuecomment-989975508 ## CI report: * 6832524d7d78de814cbeadb44fa8037da5c10ca9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28200) * f0b1ef0bd1e2babc2093a57b0b6b619c99cb551e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28457) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18068: [FLINK-25105][checkpoint] Enables final checkpoint by default
flinkbot edited a comment on pull request #18068: URL: https://github.com/apache/flink/pull/18068#issuecomment-989975508 ## CI report: * 6832524d7d78de814cbeadb44fa8037da5c10ca9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28200) * f0b1ef0bd1e2babc2093a57b0b6b619c99cb551e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25360) Add State Desc to CheckpointMetadata
[ https://issues.apache.org/jira/browse/FLINK-25360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463562#comment-17463562 ] 刘方奇 commented on FLINK-25360: - Exactly, it can improve state-processor-api. I think it can also help the checkpoint meta more useful. > Add State Desc to CheckpointMetadata > > > Key: FLINK-25360 > URL: https://issues.apache.org/jira/browse/FLINK-25360 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing, Runtime / State Backends >Reporter: 刘方奇 >Priority: Major > Attachments: image-2021-12-17-20-01-42-423.png > > > Now we can't get the State Descriptor info in the checkpoint meta. Like the > case if we use state-processor-api to load state then rewrite state, we can't > flexible use the state. > Maybe there are other cases we need the State Descriptor, so can we add this > info? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25360) Add State Desc to CheckpointMetadata
[ https://issues.apache.org/jira/browse/FLINK-25360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463561#comment-17463561 ] 刘方奇 commented on FLINK-25360: - [~yunta] THX for your reply. We use state-processor-api to transform state now, but we find that we can not even build the state desc in any way. It seems like not friendly. # we can only get the typeSerializers or something that can not help we build a state desc (AggregatingStateDescriptor need the aggFunction). # even we just get the above, we can not get it easily, cause most of these are stored in the state file. I think now the checkpoint meta do not have the power of self-described. > Add State Desc to CheckpointMetadata > > > Key: FLINK-25360 > URL: https://issues.apache.org/jira/browse/FLINK-25360 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing, Runtime / State Backends >Reporter: 刘方奇 >Priority: Major > Attachments: image-2021-12-17-20-01-42-423.png > > > Now we can't get the State Descriptor info in the checkpoint meta. Like the > case if we use state-processor-api to load state then rewrite state, we can't > flexible use the state. > Maybe there are other cases we need the State Descriptor, so can we add this > info? -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] PatrickRen commented on a change in pull request #18014: [FLINK-24857][test][Kafka] Upgrade SourceReaderTestBase t…
PatrickRen commented on a change in pull request #18014: URL: https://github.com/apache/flink/pull/18014#discussion_r773571166 ## File path: flink-connectors/flink-connector-kafka/src/test/java/org/apache/flink/connector/kafka/source/reader/KafkaSourceReaderTest.java ## @@ -540,6 +540,7 @@ private long getCommittedOffsetMetric(TopicPartition tp, MetricListener listener String.valueOf(tp.partition()), COMMITTED_OFFSET_METRIC_GAUGE); assertThat(committedOffsetGauge.isPresent()).isTrue(); Review comment: I think you forget to remove this line 😄 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-25409) Add cache metric to LookupFunction
[ https://issues.apache.org/jira/browse/FLINK-25409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463553#comment-17463553 ] Yuan Zhu edited comment on FLINK-25409 at 12/22/21, 3:06 AM: - In 1.14.2, neglecting some private LookupFunctions outside of connectors: ||Cache Type||Legacy||Not Legacy|| |Guava cache|JdbcLookupFunction|JdbcRowDataLookupFunction| |Java map|CsvLookupFunction| | |Java map| |FileSystemLookupFunction| |Guava cache | |HBaseRowDataLookupFunction| They all have cache inside. We can extract a common abstract class to init cache metric if enabled and some useful metrics. The implementation logic can be left by child class. If some LookupSource without cache in connectors external repo won't to use metric indeed, they can just implement TableFunction like before or implement the newly abstract LookupFunction without init metric implementation. [~jingzhang] WDYT? was (Author: straw): In 1.14.2, neglecting some private LookupFunctions outside of connectors: ||Cache Type||Legacy||Not Legacy|| |Guava cache|JdbcLookupFunction|JdbcRowDataLookupFunction| |Java map|CsvLookupFunction| | |Java map| |FileSystemLookupFunction| |Guava cache | |HBaseRowDataLookupFunction| They all have cache inside. We can extract a common abstract class to init cache metric if enabled and some useful metrics. The implementation logic can be left by child class. If some LookupSource without cache in connectors external repo won't to use metric indeed, they can just implement TableFunction like before or implement the newly abstract LookupFunction without init metric implementation. > Add cache metric to LookupFunction > -- > > Key: FLINK-25409 > URL: https://issues.apache.org/jira/browse/FLINK-25409 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Ecosystem >Reporter: Yuan Zhu >Priority: Major > > Since we encounter performance problem when lookup join in production env > frequently, adding metrics to monitor Lookup function cache is very helpful > to troubleshoot. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (FLINK-25360) Add State Desc to CheckpointMetadata
[ https://issues.apache.org/jira/browse/FLINK-25360?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17463557#comment-17463557 ] Yun Tang commented on FLINK-25360: -- [~liufangqi] Any changes to checkpoint persistent content requires attention to consider backward compatibility. I think your request sounds more like related to state-processor-API, could you give details on what current state-processor-API cannot do to meet your request? Moreover, do you think we can just change on state-processor-API to meet your request? > Add State Desc to CheckpointMetadata > > > Key: FLINK-25360 > URL: https://issues.apache.org/jira/browse/FLINK-25360 > Project: Flink > Issue Type: Improvement > Components: Runtime / Checkpointing, Runtime / State Backends >Reporter: 刘方奇 >Priority: Major > Attachments: image-2021-12-17-20-01-42-423.png > > > Now we can't get the State Descriptor info in the checkpoint meta. Like the > case if we use state-processor-api to load state then rewrite state, we can't > flexible use the state. > Maybe there are other cases we need the State Descriptor, so can we add this > info? -- This message was sent by Atlassian Jira (v8.20.1#820001)
