[GitHub] [flink] JingGe commented on a diff in pull request #21322: [FLINK-30025][table] the max column width should be resizable while printing the query result to the client console.
JingGe commented on code in PR #21322:
URL: https://github.com/apache/flink/pull/21322#discussion_r1027671723
##
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/TableEnvironmentImpl.java:
##
@@ -926,7 +926,7 @@ private TableResultInternal
executeQueryOperation(QueryOperation operation) {
DataTypeUtils.expandCompositeTypeToSchema(
sinkOperation.getConsumedDataType()),
resultProvider.getRowDataStringConverter(),
-PrintStyle.DEFAULT_MAX_COLUMN_WIDTH,
Review Comment:
Do you mean removing `PrintStyle.DEFAULT_MAX_COLUMN_WIDTH` completely? How
to make it configurable? Could you elaborate on it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] JingGe commented on a diff in pull request #21322: [FLINK-30025][table] the max column width should be resizable while printing the query result to the client console.
JingGe commented on code in PR #21322:
URL: https://github.com/apache/flink/pull/21322#discussion_r102766
##
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/TableConfigOptions.java:
##
@@ -102,6 +102,16 @@ private TableConfigOptions() {}
+ "the session time zone is used during
conversion. The input of option is either a full name "
+ "such as \"America/Los_Angeles\", or a
custom timezone id such as \"GMT-08:00\".");
+@Documentation.TableOption(execMode = Documentation.ExecMode.STREAMING)
+public static final ConfigOption PRINT_MAX_COLUMN_WIDTH =
Review Comment:
"sql-client.display.max-column-width" seems only work in cli. This PR is
focusing on the Table API. I have updated the jira task with more background.
Please check: https://issues.apache.org/jira/browse/FLINK-30025
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] JingGe commented on a diff in pull request #21322: [FLINK-30025][table] the max column width should be resizable while printing the query result to the client console.
JingGe commented on code in PR #21322:
URL: https://github.com/apache/flink/pull/21322#discussion_r102766
##
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/TableConfigOptions.java:
##
@@ -102,6 +102,16 @@ private TableConfigOptions() {}
+ "the session time zone is used during
conversion. The input of option is either a full name "
+ "such as \"America/Los_Angeles\", or a
custom timezone id such as \"GMT-08:00\".");
+@Documentation.TableOption(execMode = Documentation.ExecMode.STREAMING)
+public static final ConfigOption PRINT_MAX_COLUMN_WIDTH =
Review Comment:
"sql-client.display.max-column-width" seems only work in cli. I have updated
the jira task with more background. Please check:
https://issues.apache.org/jira/browse/FLINK-30025
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-30025) table.execute().print() can only use the default max column width
[
https://issues.apache.org/jira/browse/FLINK-30025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Ge updated FLINK-30025:
Description:
when running table API program "table.execute().print();", the columns with
long string value are truncated to 30 chars. E.g.,:
!https://static.dingtalk.com/media/lALPF6XTM7ZO1FXNASrNBEI_1090_298.png_620x1q90.jpg?auth_bizType=%27IM%27=im|width=457,height=125!
I tried set the max width with:
tEnv.getConfig.getConfiguration.setInteger("sql-client.display.max-column-width",
100); It has no effect. How can I set the max-width?
Here is the example code:
val env = StreamExecutionEnvironment.getExecutionEnvironment
val tEnv = StreamTableEnvironment.create(env)
tEnv.getConfig.getConfiguration.setInteger("sql-client.display.max-column-width",
100)
val orderA = env
.fromCollection(Seq(Order(1L, "beer", 3), Order(1L,
"diaper---.diaper---.diaper---.diaper---.", 4), Order(3L, "rubber", 2)))
.toTable(tEnv)
orderA.execute().print()
"sql-client.display.max-column-width" seems only work in cli: SET
'sql-client.display.max-column-width' = '40';
While using Table API, by default, the DEFAULT_MAX_COLUMN_WIDTH in PrintStyle
is used now. It should be configurable.
was:
"sql-client.display.max-column-width" seems only work in cli: SET
'sql-client.display.max-column-width' = '40';
While using Table API, by default, the DEFAULT_MAX_COLUMN_WIDTH in PrintStyle
is used now. It should be configurable.
> table.execute().print() can only use the default max column width
> --
>
> Key: FLINK-30025
> URL: https://issues.apache.org/jira/browse/FLINK-30025
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.16.0, 1.15.2
>Reporter: Jing Ge
>Assignee: Jing Ge
>Priority: Minor
> Labels: pull-request-available
>
> when running table API program "table.execute().print();", the columns with
> long string value are truncated to 30 chars. E.g.,:
> !https://static.dingtalk.com/media/lALPF6XTM7ZO1FXNASrNBEI_1090_298.png_620x1q90.jpg?auth_bizType=%27IM%27=im|width=457,height=125!
> I tried set the max width with:
> tEnv.getConfig.getConfiguration.setInteger("sql-client.display.max-column-width",
> 100); It has no effect. How can I set the max-width?
> Here is the example code:
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> val tEnv = StreamTableEnvironment.create(env)
> tEnv.getConfig.getConfiguration.setInteger("sql-client.display.max-column-width",
> 100)
> val orderA = env
> .fromCollection(Seq(Order(1L, "beer", 3), Order(1L,
> "diaper---.diaper---.diaper---.diaper---.", 4), Order(3L, "rubber", 2)))
> .toTable(tEnv)
> orderA.execute().print()
>
> "sql-client.display.max-column-width" seems only work in cli: SET
> 'sql-client.display.max-column-width' = '40';
> While using Table API, by default, the DEFAULT_MAX_COLUMN_WIDTH in PrintStyle
> is used now. It should be configurable.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30025) table.execute().print() can only use the default max column width
[ https://issues.apache.org/jira/browse/FLINK-30025?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Ge updated FLINK-30025: Description: "sql-client.display.max-column-width" seems only work in cli: SET 'sql-client.display.max-column-width' = '40'; While using Table API, by default, the DEFAULT_MAX_COLUMN_WIDTH in PrintStyle is used now. It should be configurable. was:By default, the DEFAULT_MAX_COLUMN_WIDTH in PrintStyle is used now. It should be configurable. > table.execute().print() can only use the default max column width > -- > > Key: FLINK-30025 > URL: https://issues.apache.org/jira/browse/FLINK-30025 > Project: Flink > Issue Type: Improvement > Components: Table SQL / API >Affects Versions: 1.16.0, 1.15.2 >Reporter: Jing Ge >Assignee: Jing Ge >Priority: Minor > Labels: pull-request-available > > "sql-client.display.max-column-width" seems only work in cli: SET > 'sql-client.display.max-column-width' = '40'; > While using Table API, by default, the DEFAULT_MAX_COLUMN_WIDTH in PrintStyle > is used now. It should be configurable. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] JingsongLi merged pull request #392: [SECURITY] Fix Temporary File Information Disclosure Vulnerability
JingsongLi merged PR #392: URL: https://github.com/apache/flink-table-store/pull/392 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-30091) mvn package verify fails with "Error creating shaded jar: duplicate entry"
[
https://issues.apache.org/jira/browse/FLINK-30091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-30091.
Fix Version/s: table-store-0.3.0
Assignee: Alex Sorokoumov
Resolution: Fixed
master: 0a31766290c36a0c9033da3850146c684a90468d
> mvn package verify fails with "Error creating shaded jar: duplicate entry"
> --
>
> Key: FLINK-30091
> URL: https://issues.apache.org/jira/browse/FLINK-30091
> Project: Flink
> Issue Type: Bug
> Components: Table Store
>Reporter: Alex Sorokoumov
>Assignee: Alex Sorokoumov
>Priority: Major
> Labels: pull-request-available
> Fix For: table-store-0.3.0
>
>
> {{mvn package verify}} fails with
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-shade-plugin:3.1.1:shade (shade-flink) on
> project flink-table-store-dist: Error creating shaded jar: duplicate entry:
> META-INF/services/org.apache.flink.table.store.shaded.org.apache.kafka.common.config.provider.ConfigProvider
> -> [Help 1]
> {noformat}
> The fix is to update {{maven-shade-plugin}} version to 3.4.1.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-table-store] JingsongLi merged pull request #391: [FLINK-30091] Update maven-shade-plugin version to 3.4.1
JingsongLi merged PR #391: URL: https://github.com/apache/flink-table-store/pull/391 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-29985) TaskManager might not close SlotTable on SIGTERM
[ https://issues.apache.org/jira/browse/FLINK-29985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Pohl updated FLINK-29985: -- Component/s: Runtime / Coordination > TaskManager might not close SlotTable on SIGTERM > > > Key: FLINK-29985 > URL: https://issues.apache.org/jira/browse/FLINK-29985 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination, Runtime / Task >Affects Versions: 1.16.0, 1.17.0, 1.15.3 >Reporter: Roman Khachatryan >Priority: Minor > > When TM is stopped by RM, its slot table is closed, causing all its slots to > be released. > However, when TM is stopped by SIGTERM (i.e. external resource manager), its > slot table is NOT closed. > > When a slot is released, the associated resources are released as well, in > particular, MemoryManager. > MemoryManager might hold not only memory, but also arbitrary shared resources > (currently, PythonSharedResources and RocksDBSharedResources). > As of now, RocksDBSharedResources contains only ephemeral resources. Not sure > about PythonSharedResources, but likely it is associated with a separate > process. > That means that in standalone clusters, some resources might not be released. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29985) TaskManager might not close SlotTable on SIGTERM
[ https://issues.apache.org/jira/browse/FLINK-29985?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias Pohl updated FLINK-29985: -- Affects Version/s: 1.17.0 > TaskManager might not close SlotTable on SIGTERM > > > Key: FLINK-29985 > URL: https://issues.apache.org/jira/browse/FLINK-29985 > Project: Flink > Issue Type: Bug > Components: Runtime / Task >Affects Versions: 1.16.0, 1.17.0, 1.15.3 >Reporter: Roman Khachatryan >Priority: Minor > > When TM is stopped by RM, its slot table is closed, causing all its slots to > be released. > However, when TM is stopped by SIGTERM (i.e. external resource manager), its > slot table is NOT closed. > > When a slot is released, the associated resources are released as well, in > particular, MemoryManager. > MemoryManager might hold not only memory, but also arbitrary shared resources > (currently, PythonSharedResources and RocksDBSharedResources). > As of now, RocksDBSharedResources contains only ephemeral resources. Not sure > about PythonSharedResources, but likely it is associated with a separate > process. > That means that in standalone clusters, some resources might not be released. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #388: [FLINK-29232] File store continuous reading support from_timestamp scan mode
JingsongLi commented on code in PR #388:
URL: https://github.com/apache/flink-table-store/pull/388#discussion_r1027654165
##
flink-table-store-connector/src/main/java/org/apache/flink/table/store/connector/source/FileStoreSource.java:
##
@@ -130,4 +119,42 @@ public SplitEnumerator restoreEnu
return new StaticFileStoreSplitEnumerator(context, snapshot,
splits);
}
}
+
+private DataFilePlan batchScanSplits(DataTableScan scan) {
+return scan.plan();
+}
+
+private DataFilePlan continuousFirstScanSplits(
+DataTableScan scan, SnapshotManager snapshotManager) {
+switch (startupMode) {
+case FULL:
+DataFilePlan plan;
+if (table.options().changelogProducer()
+== CoreOptions.ChangelogProducer.FULL_COMPACTION) {
+// Read the results of the last full compaction.
+// Only full compaction results will appear on the max
level.
+plan = scan.withLevel(table.options().numLevels() -
1).plan();
+} else {
+plan = scan.plan();
+}
+return plan;
+case LATEST:
+return new DataFilePlan(
+snapshotManager.latestSnapshotId(),
Collections.emptyList());
+case FROM_TIMESTAMP:
Review Comment:
I created https://issues.apache.org/jira/browse/FLINK-30110 for this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-30110) Enable from-timestamp log scan when timestamp-millis is configured
Jingsong Lee created FLINK-30110: Summary: Enable from-timestamp log scan when timestamp-millis is configured Key: FLINK-30110 URL: https://issues.apache.org/jira/browse/FLINK-30110 Project: Flink Issue Type: Improvement Components: Table Store Reporter: Jingsong Lee Fix For: table-store-0.3.0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-ml] lindong28 merged pull request #175: [FLINK-29602] Add Transformer for SQLTransformer
lindong28 merged PR #175: URL: https://github.com/apache/flink-ml/pull/175 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #21356: [BP-1.15][FLINK-16582][runtime][test] Fix the buffer leaks in NettyBufferPoolTest
flinkbot commented on PR #21356: URL: https://github.com/apache/flink/pull/21356#issuecomment-1321559701 ## CI report: * 2d4b39d29e32619f29d6832121e96cf9fac494ff UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-29602) Add Transformer for SQLTransformer
[ https://issues.apache.org/jira/browse/FLINK-29602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin resolved FLINK-29602. -- Resolution: Fixed > Add Transformer for SQLTransformer > -- > > Key: FLINK-29602 > URL: https://issues.apache.org/jira/browse/FLINK-29602 > Project: Flink > Issue Type: New Feature > Components: Library / Machine Learning >Reporter: Yunfeng Zhou >Assignee: Yunfeng Zhou >Priority: Major > Labels: pull-request-available > Fix For: ml-2.2.0 > > > Add Transformer for SQLTransformer. > Its function would be at least equivalent to Spark's > org.apache.spark.ml.feature.SQLTransformer. The relevant PR should contain > the following components: > * Java implementation/test (Must include) > * Python implementation/test (Optional) > * Markdown document (Optional) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] flinkbot commented on pull request #21355: [BP-1.16][FLINK-16582][runtime][test] Fix the buffer leaks in NettyBufferPoolTest
flinkbot commented on PR #21355: URL: https://github.com/apache/flink/pull/21355#issuecomment-1321559574 ## CI report: * fca350ae0fa2d8da21ec5782063a975fe9c453fc UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-29602) Add Transformer for SQLTransformer
[ https://issues.apache.org/jira/browse/FLINK-29602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin reassigned FLINK-29602: Assignee: Yunfeng Zhou > Add Transformer for SQLTransformer > -- > > Key: FLINK-29602 > URL: https://issues.apache.org/jira/browse/FLINK-29602 > Project: Flink > Issue Type: New Feature > Components: Library / Machine Learning >Affects Versions: ml-2.2.0 >Reporter: Yunfeng Zhou >Assignee: Yunfeng Zhou >Priority: Major > Labels: pull-request-available > Fix For: ml-2.2.0 > > > Add Transformer for SQLTransformer. > Its function would be at least equivalent to Spark's > org.apache.spark.ml.feature.SQLTransformer. The relevant PR should contain > the following components: > * Java implementation/test (Must include) > * Python implementation/test (Optional) > * Markdown document (Optional) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29602) Add Transformer for SQLTransformer
[ https://issues.apache.org/jira/browse/FLINK-29602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dong Lin updated FLINK-29602: - Affects Version/s: (was: ml-2.2.0) > Add Transformer for SQLTransformer > -- > > Key: FLINK-29602 > URL: https://issues.apache.org/jira/browse/FLINK-29602 > Project: Flink > Issue Type: New Feature > Components: Library / Machine Learning >Reporter: Yunfeng Zhou >Assignee: Yunfeng Zhou >Priority: Major > Labels: pull-request-available > Fix For: ml-2.2.0 > > > Add Transformer for SQLTransformer. > Its function would be at least equivalent to Spark's > org.apache.spark.ml.feature.SQLTransformer. The relevant PR should contain > the following components: > * Java implementation/test (Must include) > * Python implementation/test (Optional) > * Markdown document (Optional) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-ml] lindong28 commented on pull request #175: [FLINK-29602] Add Transformer for SQLTransformer
lindong28 commented on PR #175: URL: https://github.com/apache/flink-ml/pull/175#issuecomment-1321558329 Thanks for the update! LGTM. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on pull request #14319: [FLINK-16582][runtime][test] Fix the buffer leaks in NettyBufferPoolTest
gaoyunhaii commented on PR #14319: URL: https://github.com/apache/flink/pull/14319#issuecomment-1321558015 Ah, very thanks for that! I'll finalize the issue today. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] XComp commented on pull request #14319: [FLINK-16582][runtime][test] Fix the buffer leaks in NettyBufferPoolTest
XComp commented on PR #14319: URL: https://github.com/apache/flink/pull/14319#issuecomment-1321557610 The backports are created. Feel free to have a look, merge all PRs after the CI is green and finalize the Jira issue. Thanks :-) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] XComp opened a new pull request, #21356: [FLINK-16582][runtime][test] Fix the buffer leaks in NettyBufferPoolTest
XComp opened a new pull request, #21356: URL: https://github.com/apache/flink/pull/21356 1.15 backport for PR https://github.com/apache/flink/pull/14319 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] gaoyunhaii commented on pull request #14319: [FLINK-16582][runtime][test] Fix the buffer leaks in NettyBufferPoolTest
gaoyunhaii commented on PR #14319: URL: https://github.com/apache/flink/pull/14319#issuecomment-1321555445 Hi @XComp very thanks for reviewing and very sorry for missing the previous message, I'll create the backports and merge it after the ci passed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #388: [FLINK-29232] File store continuous reading support from_timestamp scan mode
JingsongLi commented on code in PR #388:
URL: https://github.com/apache/flink-table-store/pull/388#discussion_r1027645758
##
flink-table-store-connector/src/main/java/org/apache/flink/table/store/connector/source/FileStoreSource.java:
##
@@ -130,4 +119,42 @@ public SplitEnumerator restoreEnu
return new StaticFileStoreSplitEnumerator(context, snapshot,
splits);
}
}
+
+private DataFilePlan batchScanSplits(DataTableScan scan) {
+return scan.plan();
+}
+
+private DataFilePlan continuousFirstScanSplits(
+DataTableScan scan, SnapshotManager snapshotManager) {
+switch (startupMode) {
+case FULL:
+DataFilePlan plan;
+if (table.options().changelogProducer()
+== CoreOptions.ChangelogProducer.FULL_COMPACTION) {
+// Read the results of the last full compaction.
+// Only full compaction results will appear on the max
level.
+plan = scan.withLevel(table.options().numLevels() -
1).plan();
+} else {
+plan = scan.plan();
+}
+return plan;
+case LATEST:
+return new DataFilePlan(
+snapshotManager.latestSnapshotId(),
Collections.emptyList());
+case FROM_TIMESTAMP:
Review Comment:
Maybe we can let the default value of `log.scan` is none, and explain in
documentation?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] XComp opened a new pull request, #21355: [FLINK-16582][runtime][test] Fix the buffer leaks in NettyBufferPoolTest
XComp opened a new pull request, #21355: URL: https://github.com/apache/flink/pull/21355 1.16 backport for PR #14319 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #388: [FLINK-29232] File store continuous reading support from_timestamp scan mode
JingsongLi commented on code in PR #388:
URL: https://github.com/apache/flink-table-store/pull/388#discussion_r1027645758
##
flink-table-store-connector/src/main/java/org/apache/flink/table/store/connector/source/FileStoreSource.java:
##
@@ -130,4 +119,42 @@ public SplitEnumerator restoreEnu
return new StaticFileStoreSplitEnumerator(context, snapshot,
splits);
}
}
+
+private DataFilePlan batchScanSplits(DataTableScan scan) {
+return scan.plan();
+}
+
+private DataFilePlan continuousFirstScanSplits(
+DataTableScan scan, SnapshotManager snapshotManager) {
+switch (startupMode) {
+case FULL:
+DataFilePlan plan;
+if (table.options().changelogProducer()
+== CoreOptions.ChangelogProducer.FULL_COMPACTION) {
+// Read the results of the last full compaction.
+// Only full compaction results will appear on the max
level.
+plan = scan.withLevel(table.options().numLevels() -
1).plan();
+} else {
+plan = scan.plan();
+}
+return plan;
+case LATEST:
+return new DataFilePlan(
+snapshotManager.latestSnapshotId(),
Collections.emptyList());
+case FROM_TIMESTAMP:
Review Comment:
Maybe we can let the default value of `log.scan` is none, and explain in
comments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #388: [FLINK-29232] File store continuous reading support from_timestamp scan mode
JingsongLi commented on code in PR #388:
URL: https://github.com/apache/flink-table-store/pull/388#discussion_r1027644999
##
flink-table-store-connector/src/test/java/org/apache/flink/table/store/connector/ContinuousFileStoreITCase.java:
##
@@ -114,6 +121,55 @@ public void testContinuousLatest() throws TimeoutException
{
.containsExactlyInAnyOrder(Row.of("7", "8", "9"), Row.of("10",
"11", "12"));
}
+@Test
+public void testContinuousFromTimestamp() throws Exception {
+String sql =
+"SELECT * FROM T1 /*+ OPTIONS('log.scan'='from-timestamp',
'log.scan.timestamp-millis'='%s') */";
+
+// empty table
+BlockingIterator iterator =
BlockingIterator.of(streamSqlIter(sql, 0));
+batchSql("INSERT INTO T1 VALUES ('1', '2', '3'), ('4', '5', '6')");
+batchSql("INSERT INTO T1 VALUES ('7', '8', '9'), ('10', '11', '12')");
+assertThat(iterator.collect(2))
+.containsExactlyInAnyOrder(Row.of("1", "2", "3"), Row.of("4",
"5", "6"));
+iterator.close();
+
+SnapshotManager snapshotManager =
+new SnapshotManager(
+new Path(path,
"default_catalog.catalog/default_database.db/T1"));
+List snapshots =
+new
ArrayList<>(ImmutableList.copyOf(snapshotManager.snapshots()));
+snapshots.sort(Comparator.comparingLong(Snapshot::timeMillis));
+Snapshot first = snapshots.get(0);
+Snapshot second = snapshots.get(1);
+
+// before second snapshot
+iterator = BlockingIterator.of(streamSqlIter(sql, second.timeMillis()
- 1));
+batchSql("INSERT INTO T1 VALUES ('13', '14', '15')");
+assertThat(iterator.collect(3))
+.containsExactlyInAnyOrder(
+Row.of("7", "8", "9"), Row.of("10", "11", "12"),
Row.of("13", "14", "15"));
+iterator.close();
+
+// from second snapshot
+iterator = BlockingIterator.of(streamSqlIter(sql,
second.timeMillis()));
+assertThat(iterator.collect(3))
+.containsExactlyInAnyOrder(
+Row.of("7", "8", "9"), Row.of("10", "11", "12"),
Row.of("13", "14", "15"));
+iterator.close();
+
+// from start
+iterator = BlockingIterator.of(streamSqlIter(sql, first.timeMillis() -
1));
Review Comment:
Good~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #388: [FLINK-29232] File store continuous reading support from_timestamp scan mode
JingsongLi commented on code in PR #388:
URL: https://github.com/apache/flink-table-store/pull/388#discussion_r1027644463
##
flink-table-store-connector/src/main/java/org/apache/flink/table/store/connector/source/FileStoreSource.java:
##
@@ -130,4 +119,42 @@ public SplitEnumerator restoreEnu
return new StaticFileStoreSplitEnumerator(context, snapshot,
splits);
}
}
+
+private DataFilePlan batchScanSplits(DataTableScan scan) {
+return scan.plan();
+}
+
+private DataFilePlan continuousFirstScanSplits(
+DataTableScan scan, SnapshotManager snapshotManager) {
+switch (startupMode) {
+case FULL:
+DataFilePlan plan;
+if (table.options().changelogProducer()
+== CoreOptions.ChangelogProducer.FULL_COMPACTION) {
+// Read the results of the last full compaction.
+// Only full compaction results will appear on the max
level.
+plan = scan.withLevel(table.options().numLevels() -
1).plan();
+} else {
+plan = scan.plan();
+}
+return plan;
+case LATEST:
+return new DataFilePlan(
+snapshotManager.latestSnapshotId(),
Collections.emptyList());
+case FROM_TIMESTAMP:
Review Comment:
Good point, I think we can improve this later, because now we can not unset
option in Flink SQL, we can wait for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] XComp commented on pull request #14319: [FLINK-16582][runtime][test] Fix the buffer leaks in NettyBufferPoolTest
XComp commented on PR #14319: URL: https://github.com/apache/flink/pull/14319#issuecomment-1321551142 I force-pushed the rebase myself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #388: [FLINK-29232] File store continuous reading support from_timestamp scan mode
JingsongLi commented on code in PR #388:
URL: https://github.com/apache/flink-table-store/pull/388#discussion_r1027639409
##
flink-table-store-connector/src/main/java/org/apache/flink/table/store/connector/source/FileStoreSource.java:
##
@@ -130,4 +119,42 @@ public SplitEnumerator restoreEnu
return new StaticFileStoreSplitEnumerator(context, snapshot,
splits);
}
}
+
+private DataFilePlan batchScanSplits(DataTableScan scan) {
+return scan.plan();
+}
+
+private DataFilePlan continuousFirstScanSplits(
+DataTableScan scan, SnapshotManager snapshotManager) {
+switch (startupMode) {
+case FULL:
+DataFilePlan plan;
+if (table.options().changelogProducer()
+== CoreOptions.ChangelogProducer.FULL_COMPACTION) {
+// Read the results of the last full compaction.
+// Only full compaction results will appear on the max
level.
+plan = scan.withLevel(table.options().numLevels() -
1).plan();
+} else {
+plan = scan.plan();
+}
+return plan;
+case LATEST:
Review Comment:
This is from kafka connector.
I think `INCREMENTAL` contains `LATEST` `EARLIEST` `FROM-TIMESTAMP`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #388: [FLINK-29232] File store continuous reading support from_timestamp scan mode
JingsongLi commented on code in PR #388:
URL: https://github.com/apache/flink-table-store/pull/388#discussion_r1027638918
##
flink-table-store-connector/src/main/java/org/apache/flink/table/store/connector/source/FileStoreSource.java:
##
@@ -130,4 +119,42 @@ public SplitEnumerator restoreEnu
return new StaticFileStoreSplitEnumerator(context, snapshot,
splits);
}
}
+
+private DataFilePlan batchScanSplits(DataTableScan scan) {
+return scan.plan();
+}
+
+private DataFilePlan continuousFirstScanSplits(
+DataTableScan scan, SnapshotManager snapshotManager) {
+switch (startupMode) {
Review Comment:
I think we can extract this logical to `SnapshotEnumerator.startup`.
We don't need to introduce `DataFilePlanBuilder`, it is just a simple call.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-29801) OperatorCoordinator need open the way to operate metricGroup interface
[ https://issues.apache.org/jira/browse/FLINK-29801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636464#comment-17636464 ] yuemeng commented on FLINK-29801: - Hi [~ruanhang1993], [~zhuzh] A further consideration is good, very happy to hear that and looking forward to this Flip, I will focus on any progress of this Flip and It would be even better if I could help it finish, If any part needs me to join in， just let me know > OperatorCoordinator need open the way to operate metricGroup interface > -- > > Key: FLINK-29801 > URL: https://issues.apache.org/jira/browse/FLINK-29801 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics >Reporter: yuemeng >Assignee: yuemeng >Priority: Major > Labels: pull-request-available > > Currently, We have no way to get metric group instances in OperatorCoordinator > In some cases, we may report some metric in OperatorCoordinator such as Flink > hudi integrate scene， some meta will send to operator coordinator to commit > to hdfs or hms > but we also need to report some metrics in operator coordinator for monitor > purpose -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] XComp commented on a diff in pull request #21300: [FLINK-29423][rest] Remove custom JobDetails serializer
XComp commented on code in PR #21300:
URL: https://github.com/apache/flink/pull/21300#discussion_r1027615498
##
flink-runtime/src/main/java/org/apache/flink/runtime/messages/webmonitor/JobDetails.java:
##
@@ -93,6 +92,29 @@ public class JobDetails implements Serializable {
*/
private final Map>
currentExecutionAttempts;
+@JsonCreator
+public JobDetails(
+@JsonProperty(FIELD_NAME_JOB_ID) @JsonDeserialize(using =
JobIDDeserializer.class)
+JobID jobId,
+@JsonProperty(FIELD_NAME_JOB_NAME) String jobName,
+@JsonProperty(FIELD_NAME_START_TIME) long startTime,
+@JsonProperty(FIELD_NAME_END_TIME) long endTime,
+@JsonProperty(FIELD_NAME_DURATION) long duration,
+@JsonProperty(FIELD_NAME_STATUS) JobStatus status,
+@JsonProperty(FIELD_NAME_LAST_MODIFICATION) long lastUpdateTime,
+@JsonProperty(FIELD_NAME_TASKS) Map taskInfo) {
+this(
+jobId,
+jobName,
+startTime,
+endTime,
+duration,
+status,
+lastUpdateTime,
+extractNumTasksPerState(taskInfo),
+taskInfo.get(FIELD_NAME_TOTAL_NUMBER_TASKS));
+}
+
@VisibleForTesting
Review Comment:
```suggestion
```
This is not true anymore since you're using this constructor in the
constructor annotated with `@JsonCreator`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #21354: [FLINK-30084][Runtime] Remove unused method notifyAllocationFailure.
flinkbot commented on PR #21354: URL: https://github.com/apache/flink/pull/21354#issuecomment-1321511902 ## CI report: * 157c1984af79af7af52769b9037a13e824cda211 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-30084) Remove unused ResourceActions.notifyAllocationFailure
[ https://issues.apache.org/jira/browse/FLINK-30084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-30084: --- Labels: pull-request-available (was: ) > Remove unused ResourceActions.notifyAllocationFailure > - > > Key: FLINK-30084 > URL: https://issues.apache.org/jira/browse/FLINK-30084 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Weihua Hu >Priority: Major > Labels: pull-request-available > > as commented in https://github.com/apache/flink/pull/21233 > ResourceActions.notifyAllocationFailure is not used, these related API should > be removed. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] huwh opened a new pull request, #21354: [FLINK-30084][Runtime] Remove unused method notifyAllocationFailure.
huwh opened a new pull request, #21354: URL: https://github.com/apache/flink/pull/21354 ## What is the purpose of the change The methods ResourceActions#notifyAllocationFailure and JobMasterGateway#notifyAllocationFailure are not used and should be removed. ## Brief change log Remove ResourceActions#notifyAllocationFailure and JobMasterGateway#notifyAllocationFailure and the related unit tests. ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on a diff in pull request #438: [FLINK-29974] Not allowing the cancelling the which are already in the completed state.
gyfora commented on code in PR #438:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/438#discussion_r1027609040
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java:
##
@@ -362,11 +362,22 @@ public void cancelSessionJob(
FlinkSessionJob sessionJob, UpgradeMode upgradeMode, Configuration
conf)
throws Exception {
-var jobStatus = sessionJob.getStatus().getJobStatus();
+var sessionJobStatus = sessionJob.getStatus();
+var jobStatus = sessionJobStatus.getJobStatus();
var jobIdString = jobStatus.getJobId();
Preconditions.checkNotNull(jobIdString, "The job to be suspend should
not be null");
var jobId = JobID.fromHexString(jobIdString);
Optional savepointOpt = Optional.empty();
+
+if (ReconciliationUtils.isJobInTerminalState(sessionJobStatus)) {
+LOG.info("Job is already in terminal state. JobID {}",
jobId.toHexString());
+return;
+} else if (!ReconciliationUtils.isJobRunning(sessionJobStatus)) {
Review Comment:
I think we don’t need this branch here, this would prevent us from
suspending failing jobs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] gyfora commented on a diff in pull request #438: [FLINK-29974] Not allowing the cancelling the which are already in the completed state.
gyfora commented on code in PR #438:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/438#discussion_r1027607734
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java:
##
@@ -362,11 +362,21 @@ public void cancelSessionJob(
FlinkSessionJob sessionJob, UpgradeMode upgradeMode, Configuration
conf)
throws Exception {
-var jobStatus = sessionJob.getStatus().getJobStatus();
+var sessionJobStatus = sessionJob.getStatus();
+var jobStatus = sessionJobStatus.getJobStatus();
var jobIdString = jobStatus.getJobId();
Preconditions.checkNotNull(jobIdString, "The job to be suspend should
not be null");
var jobId = JobID.fromHexString(jobIdString);
Optional savepointOpt = Optional.empty();
+
+if (ReconciliationUtils.isJobInTerminalState(sessionJobStatus)) {
+LOG.info("Job is already in terminal state.");
Review Comment:
Ah ok, then I misunderstood what you wrote earlier a little :)
yea that sounds good!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] XComp commented on pull request #21252: [FLINK-29830][Connector/Pulsar] Create the topic with schema before consuming messages in PulsarSinkITCase
XComp commented on PR #21252: URL: https://github.com/apache/flink/pull/21252#issuecomment-1321488646 Thanks for taking over @tisonkun. I was offline for a week anyway. So, good that you unblocked PR #21249 by merging this PR. > Style concern, if it's not enforced by the spotless plugin, varies from person to person. I believe who writes the most part of the code has the right to write code in his/her style. I agree with your view on that. Most of my comments were of cosmetic nature and not really blocking the PR. I created a follow-up issue FLINK-30109 to discuss the sneaky utility methods, though. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #21252: [FLINK-29830][Connector/Pulsar] Create the topic with schema before consuming messages in PulsarSinkITCase
XComp commented on code in PR #21252:
URL: https://github.com/apache/flink/pull/21252#discussion_r1027592579
##
flink-connectors/flink-connector-pulsar/src/test/java/org/apache/flink/connector/pulsar/testutils/runtime/PulsarRuntimeOperator.java:
##
@@ -180,6 +180,10 @@ public void createTopic(String topic, int
numberOfPartitions) {
}
}
+public void createSchema(String topic, Schema schema) {
+sneakyAdmin(() -> admin().schemas().createSchema(topic,
schema.getSchemaInfo()));
Review Comment:
Thanks for the brief clarification, @syhily . I created FLINK-30109 to cover
the issue. We either should add more documentation clarifying why we do this or
removing it as it's a workaround that might raise questions to readers. Let's
move the discussion into FLINK-30109. I'm curious about your view on that and
why the Pulsar API is forcing us to do this exception transformation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-kubernetes-operator] rgsriram commented on a diff in pull request #438: [FLINK-29974] Not allowing the cancelling the which are already in the completed state.
rgsriram commented on code in PR #438:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/438#discussion_r1027591433
##
flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java:
##
@@ -362,11 +362,21 @@ public void cancelSessionJob(
FlinkSessionJob sessionJob, UpgradeMode upgradeMode, Configuration
conf)
throws Exception {
-var jobStatus = sessionJob.getStatus().getJobStatus();
+var sessionJobStatus = sessionJob.getStatus();
+var jobStatus = sessionJobStatus.getJobStatus();
var jobIdString = jobStatus.getJobId();
Preconditions.checkNotNull(jobIdString, "The job to be suspend should
not be null");
var jobId = JobID.fromHexString(jobIdString);
Optional savepointOpt = Optional.empty();
+
+if (ReconciliationUtils.isJobInTerminalState(sessionJobStatus)) {
+LOG.info("Job is already in terminal state.");
Review Comment:
Yes, You're right.
Let me put it clearly, there are the following scenarios.
1. Cancel the same job via cli more than 1 time-> Throws an error which is
out of the scope of the operator. (IMO)
2. Cancelled the job via cli and suspended the same job via operator -> No
error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-30109) Checked exceptions are sneakingly transformed into unchecked exceptions in the Pulsar
Matthias Pohl created FLINK-30109:
-
Summary: Checked exceptions are sneakingly transformed into
unchecked exceptions in the Pulsar
Key: FLINK-30109
URL: https://issues.apache.org/jira/browse/FLINK-30109
Project: Flink
Issue Type: Technical Debt
Components: Connectors / Pulsar, Documentation
Affects Versions: 1.15.2, 1.16.0, 1.17.0
Reporter: Matthias Pohl
[PulsarExceptionUtils|https://github.com/apache/flink/blob/c675f786c51038801161e861826d1c54654f0dde/flink-connectors/flink-connector-pulsar/src/main/java/org/apache/flink/connector/pulsar/common/utils/PulsarExceptionUtils.java#L33]
provides {{sneaky*}} utility methods for hiding checked exceptions. This is
rather unusual coding. Based on what's provided in the code I would have
concerns as a reader that we're not handling errors properly in calling code.
Either, we remove these methods and add proper exception handling or we add
proper documentation on why this workaround is necessary.
[~syhily] already hinted in his [FLINK-29830 PR
comment|https://github.com/apache/flink/pull/21252#discussion_r1019822514] that
this is related to flaws of the Pulsar API.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29461) ProcessDataStreamStreamingTests.test_process_function unstable
[
https://issues.apache.org/jira/browse/FLINK-29461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636431#comment-17636431
]
Leonard Xu commented on FLINK-29461:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43332=logs=e92ecf6d-e207-5a42-7ff7-528ff0c5b259=40fc352e-9b4c-5fd8-363f-628f24b01ec2
> ProcessDataStreamStreamingTests.test_process_function unstable
> --
>
> Key: FLINK-29461
> URL: https://issues.apache.org/jira/browse/FLINK-29461
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 2022-09-29T02:10:45.3571648Z Sep 29 02:10:45 self =
> testMethod=test_process_function>
> 2022-09-29T02:10:45.3572279Z Sep 29 02:10:45
> 2022-09-29T02:10:45.3572810Z Sep 29 02:10:45 def
> test_process_function(self):
> 2022-09-29T02:10:45.3573495Z Sep 29 02:10:45
> self.env.set_parallelism(1)
> 2022-09-29T02:10:45.3574148Z Sep 29 02:10:45
> self.env.get_config().set_auto_watermark_interval(2000)
> 2022-09-29T02:10:45.3580634Z Sep 29 02:10:45
> self.env.set_stream_time_characteristic(TimeCharacteristic.EventTime)
> 2022-09-29T02:10:45.3583194Z Sep 29 02:10:45 data_stream =
> self.env.from_collection([(1, '1603708211000'),
> 2022-09-29T02:10:45.3584515Z Sep 29 02:10:45
>(2, '1603708224000'),
> 2022-09-29T02:10:45.3585957Z Sep 29 02:10:45
>(3, '1603708226000'),
> 2022-09-29T02:10:45.3587132Z Sep 29 02:10:45
>(4, '1603708289000')],
> 2022-09-29T02:10:45.3588094Z Sep 29 02:10:45
> type_info=Types.ROW([Types.INT(), Types.STRING()]))
> 2022-09-29T02:10:45.3589090Z Sep 29 02:10:45
> 2022-09-29T02:10:45.3589949Z Sep 29 02:10:45 class
> MyProcessFunction(ProcessFunction):
> 2022-09-29T02:10:45.3590710Z Sep 29 02:10:45
> 2022-09-29T02:10:45.3591856Z Sep 29 02:10:45 def
> process_element(self, value, ctx):
> 2022-09-29T02:10:45.3592873Z Sep 29 02:10:45
> current_timestamp = ctx.timestamp()
> 2022-09-29T02:10:45.3593862Z Sep 29 02:10:45
> current_watermark = ctx.timer_service().current_watermark()
> 2022-09-29T02:10:45.3594915Z Sep 29 02:10:45 yield "current
> timestamp: {}, current watermark: {}, current_value: {}"\
> 2022-09-29T02:10:45.3596201Z Sep 29 02:10:45
> .format(str(current_timestamp), str(current_watermark), str(value))
> 2022-09-29T02:10:45.3597089Z Sep 29 02:10:45
> 2022-09-29T02:10:45.3597942Z Sep 29 02:10:45 watermark_strategy =
> WatermarkStrategy.for_monotonous_timestamps()\
> 2022-09-29T02:10:45.3599260Z Sep 29 02:10:45
> .with_timestamp_assigner(SecondColumnTimestampAssigner())
> 2022-09-29T02:10:45.3600611Z Sep 29 02:10:45
> data_stream.assign_timestamps_and_watermarks(watermark_strategy)\
> 2022-09-29T02:10:45.3601877Z Sep 29 02:10:45
> .process(MyProcessFunction(),
> output_type=Types.STRING()).add_sink(self.test_sink)
> 2022-09-29T02:10:45.3603527Z Sep 29 02:10:45 self.env.execute('test
> process function')
> 2022-09-29T02:10:45.3604445Z Sep 29 02:10:45 results =
> self.test_sink.get_results()
> 2022-09-29T02:10:45.3605684Z Sep 29 02:10:45 expected = ["current
> timestamp: 1603708211000, current watermark: "
> 2022-09-29T02:10:45.3607157Z Sep 29 02:10:45
> "-9223372036854775808, current_value: Row(f0=1, f1='1603708211000')",
> 2022-09-29T02:10:45.3608256Z Sep 29 02:10:45 "current
> timestamp: 1603708224000, current watermark: "
> 2022-09-29T02:10:45.3609650Z Sep 29 02:10:45
> "-9223372036854775808, current_value: Row(f0=2, f1='1603708224000')",
> 2022-09-29T02:10:45.3610854Z Sep 29 02:10:45 "current
> timestamp: 1603708226000, current watermark: "
> 2022-09-29T02:10:45.3612279Z Sep 29 02:10:45
> "-9223372036854775808, current_value: Row(f0=3, f1='1603708226000')",
> 2022-09-29T02:10:45.3613382Z Sep 29 02:10:45 "current
> timestamp: 1603708289000, current watermark: "
> 2022-09-29T02:10:45.3615683Z Sep 29 02:10:45
> "-9223372036854775808, current_value: Row(f0=4, f1='1603708289000')"]
> 2022-09-29T02:10:45.3617687Z Sep 29 02:10:45 >
> self.assert_equals_sorted(expected, results)
> 2022-09-29T02:10:45.3618620Z Sep 29 02:10:45
> 2022-09-29T02:10:45.3619425Z Sep 29 02:10:45
> pyflink/datastream/tests/test_data_stream.py:986:
[jira] [Commented] (FLINK-26037) TaskManagerRunnerTest JVM crash with exit code 239
[
https://issues.apache.org/jira/browse/FLINK-26037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636430#comment-17636430
]
Leonard Xu commented on FLINK-26037:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43332=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=125e07e7-8de0-5c6c-a541-a567415af3ef
> TaskManagerRunnerTest JVM crash with exit code 239
> --
>
> Key: FLINK-26037
> URL: https://issues.apache.org/jira/browse/FLINK-26037
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0, 1.17.0
>Reporter: Yun Gao
>Assignee: Chesnay Schepler
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> {code:java}
> Feb 09 02:22:52 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test)
> on project flink-runtime: There are test failures.
> Feb 09 02:22:52 [ERROR]
> Feb 09 02:22:52 [ERROR] Please refer to
> /__w/2/s/flink-runtime/target/surefire-reports for the individual test
> results.
> Feb 09 02:22:52 [ERROR] Please refer to dump files (if any exist)
> [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
> Feb 09 02:22:52 [ERROR] ExecutionException The forked VM terminated without
> properly saying goodbye. VM crash or System.exit called?
> Feb 09 02:22:52 [ERROR] Command was /bin/sh -c cd /__w/2/s/flink-runtime &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
> -Dmvn.forkNumber=2 -XX:+UseG1GC -Duser.country=US -Duser.language=en -jar
> /__w/2/s/flink-runtime/target/surefire/surefirebooter2302747870022684931.jar
> /__w/2/s/flink-runtime/target/surefire 2022-02-09T01-58-20_619-jvmRun2
> surefire8313405181432833392tmp surefire_2318122906826894495431tmp
> Feb 09 02:22:52 [ERROR] Error occurred in starting fork, check output in log
> Feb 09 02:22:52 [ERROR] Process Exit Code: 239
> Feb 09 02:22:52 [ERROR] Crashed tests:
> Feb 09 02:22:52 [ERROR]
> org.apache.flink.runtime.taskexecutor.TaskManagerRunnerTest
> Feb 09 02:22:52 [ERROR]
> org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
> Feb 09 02:22:52 [ERROR] Command was /bin/sh -c cd /__w/2/s/flink-runtime &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
> -Dmvn.forkNumber=2 -XX:+UseG1GC -Duser.country=US -Duser.language=en -jar
> /__w/2/s/flink-runtime/target/surefire/surefirebooter2302747870022684931.jar
> /__w/2/s/flink-runtime/target/surefire 2022-02-09T01-58-20_619-jvmRun2
> surefire8313405181432833392tmp surefire_2318122906826894495431tmp
> Feb 09 02:22:52 [ERROR] Error occurred in starting fork, check output in log
> Feb 09 02:22:52 [ERROR] Process Exit Code: 239
> Feb 09 02:22:52 [ERROR] Crashed tests:
> Feb 09 02:22:52 [ERROR]
> org.apache.flink.runtime.taskexecutor.TaskManagerRunnerTest
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:532)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:479)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:322)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:266)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1314)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1159)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:932)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> Feb 09 02:22:52 [ERROR] at
>
[jira] [Created] (FLINK-30108) flink-core module tests exited with code 143
Leonard Xu created FLINK-30108:
--
Summary: flink-core module tests exited with code 143
Key: FLINK-30108
URL: https://issues.apache.org/jira/browse/FLINK-30108
Project: Flink
Issue Type: Bug
Components: API / Core, Tests
Affects Versions: 1.17.0
Reporter: Leonard Xu
{noformat}
Nov 18 01:02:58 [INFO] Tests run: 16, Failures: 0, Errors: 0, Skipped: 0, Time
elapsed: 109.22 s - in
org.apache.flink.runtime.operators.hash.InPlaceMutableHashTableTest
Nov 18 01:18:09
==
Nov 18 01:18:09 Process produced no output for 900 seconds.
Nov 18 01:18:09
==
Nov 18 01:18:09
==
Nov 18 01:18:09 The following Java processes are running (JPS)
Nov 18 01:18:09
==
Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError
Nov 18 01:18:09 924 Launcher
Nov 18 01:18:09 23421 surefirebooter1178962604207099497.jar
Nov 18 01:18:09 11885 Jps
Nov 18 01:18:09
==
Nov 18 01:18:09 Printing stack trace of Java process 924
Nov 18 01:18:09
==
Picked up JAVA_TOOL_OPTIONS: -XX:+HeapDumpOnOutOfMemoryError
Nov 18 01:18:09 2022-11-18 01:18:09
Nov 18 01:18:09 Full thread dump OpenJDK 64-Bit Server VM (25.292-b10 mixed
mode):
...
...
...
Nov 18 01:18:09
==
Nov 18 01:18:09 Printing stack trace of Java process 11885
Nov 18 01:18:09
==
11885: No such process
Nov 18 01:18:09 Killing process with pid=923 and all descendants
/__w/2/s/tools/ci/watchdog.sh: line 113: 923 Terminated $cmd
Nov 18 01:18:10 Process exited with EXIT CODE: 143.
Nov 18 01:18:10 Trying to KILL watchdog (919).
Nov 18 01:18:10 Searching for .dump, .dumpstream and related files in '/__w/2/s'
Nov 18 01:18:16 Moving
'/__w/2/s/flink-runtime/target/surefire-reports/2022-11-18T00-55-55_041-jvmRun3.dumpstream'
to target directory ('/__w/_temp/debug_files')
Nov 18 01:18:16 Moving
'/__w/2/s/flink-runtime/target/surefire-reports/2022-11-18T00-55-55_041-jvmRun3.dump'
to target directory ('/__w/_temp/debug_files')
The STDIO streams did not close within 10 seconds of the exit event from
process '/bin/bash'. This may indicate a child process inherited the STDIO
streams and has not yet exited.
##[error]Bash exited with code '143'.
Finishing: Test - core
{noformat}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43277=logs=0e7be18f-84f2-53f0-a32d-4a5e4a174679=7c1d86e3-35bd-5fd5-3b7c-30c126a78702
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-26037) TaskManagerRunnerTest JVM crash with exit code 239
[
https://issues.apache.org/jira/browse/FLINK-26037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636429#comment-17636429
]
Leonard Xu commented on FLINK-26037:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43277=logs=77a9d8e1-d610-59b3-fc2a-4766541e0e33=125e07e7-8de0-5c6c-a541-a567415af3ef
> TaskManagerRunnerTest JVM crash with exit code 239
> --
>
> Key: FLINK-26037
> URL: https://issues.apache.org/jira/browse/FLINK-26037
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0, 1.17.0
>Reporter: Yun Gao
>Assignee: Chesnay Schepler
>Priority: Critical
> Labels: pull-request-available, test-stability
>
> {code:java}
> Feb 09 02:22:52 [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:3.0.0-M5:test (default-test)
> on project flink-runtime: There are test failures.
> Feb 09 02:22:52 [ERROR]
> Feb 09 02:22:52 [ERROR] Please refer to
> /__w/2/s/flink-runtime/target/surefire-reports for the individual test
> results.
> Feb 09 02:22:52 [ERROR] Please refer to dump files (if any exist)
> [date].dump, [date]-jvmRun[N].dump and [date].dumpstream.
> Feb 09 02:22:52 [ERROR] ExecutionException The forked VM terminated without
> properly saying goodbye. VM crash or System.exit called?
> Feb 09 02:22:52 [ERROR] Command was /bin/sh -c cd /__w/2/s/flink-runtime &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
> -Dmvn.forkNumber=2 -XX:+UseG1GC -Duser.country=US -Duser.language=en -jar
> /__w/2/s/flink-runtime/target/surefire/surefirebooter2302747870022684931.jar
> /__w/2/s/flink-runtime/target/surefire 2022-02-09T01-58-20_619-jvmRun2
> surefire8313405181432833392tmp surefire_2318122906826894495431tmp
> Feb 09 02:22:52 [ERROR] Error occurred in starting fork, check output in log
> Feb 09 02:22:52 [ERROR] Process Exit Code: 239
> Feb 09 02:22:52 [ERROR] Crashed tests:
> Feb 09 02:22:52 [ERROR]
> org.apache.flink.runtime.taskexecutor.TaskManagerRunnerTest
> Feb 09 02:22:52 [ERROR]
> org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
> Feb 09 02:22:52 [ERROR] Command was /bin/sh -c cd /__w/2/s/flink-runtime &&
> /usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -Xms256m -Xmx2048m
> -Dmvn.forkNumber=2 -XX:+UseG1GC -Duser.country=US -Duser.language=en -jar
> /__w/2/s/flink-runtime/target/surefire/surefirebooter2302747870022684931.jar
> /__w/2/s/flink-runtime/target/surefire 2022-02-09T01-58-20_619-jvmRun2
> surefire8313405181432833392tmp surefire_2318122906826894495431tmp
> Feb 09 02:22:52 [ERROR] Error occurred in starting fork, check output in log
> Feb 09 02:22:52 [ERROR] Process Exit Code: 239
> Feb 09 02:22:52 [ERROR] Crashed tests:
> Feb 09 02:22:52 [ERROR]
> org.apache.flink.runtime.taskexecutor.TaskManagerRunnerTest
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:532)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkPerTestSet(ForkStarter.java:479)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:322)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:266)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1314)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1159)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:932)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
> Feb 09 02:22:52 [ERROR] at
> org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
> Feb 09 02:22:52 [ERROR] at
>
[jira] [Commented] (FLINK-29836) PulsarUnorderedSourceITCase failed due to an expected job restart not happening
[
https://issues.apache.org/jira/browse/FLINK-29836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636428#comment-17636428
]
Leonard Xu commented on FLINK-29836:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43278=logs=a5ef94ef-68c2-57fd-3794-dc108ed1c495=2c68b137-b01d-55c9-e603-3ff3f320364b
> PulsarUnorderedSourceITCase failed due to an expected job restart not
> happening
> ---
>
> Key: FLINK-29836
> URL: https://issues.apache.org/jira/browse/FLINK-29836
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.15.3
>Reporter: Matthias Pohl
>Priority: Major
> Labels: test-stability
>
> [This build
> failed|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42681=logs=a5ef94ef-68c2-57fd-3794-dc108ed1c495=2c68b137-b01d-55c9-e603-3ff3f320364b=27377]
> due to a missing job restart:
> {code:java}
> Nov 01 02:04:20 Expecting
> Nov 01 02:04:20Nov 01 02:04:20 java.lang.RuntimeException: Job restarted
> Nov 01 02:04:20 at
> org.apache.flink.streaming.api.operators.collect.UncheckpointedCollectResultBuffer.sinkRestarted(UncheckpointedCollectResultBuffer.java:42)
> Nov 01 02:04:20 at
> org.apache.flink.streaming.api.operators.collect.AbstractCollectResultBuffer.dealWithResponse(AbstractCollectResultBuffer.java:87)
> Nov 01 02:04:20 at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:147)
> Nov 01 02:04:20 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
> Nov 01 02:04:20 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> Nov 01 02:04:20 at
> org.apache.flink.connector.testframe.utils.UnorderedCollectIteratorAssert.compareWithExactlyOnceSemantic(UnorderedCollectIteratorAssert.java:108)
> Nov 01 02:04:20 at
> org.apache.flink.connector.testframe.utils.UnorderedCollectIteratorAssert.matchesRecordsFromSource(UnorderedCollectIteratorAssert.java:79)
> Nov 01 02:04:20 at
> org.apache.flink.connector.pulsar.testutils.source.UnorderedSourceTestSuiteBase.lambda$checkResultWithSemantic$0(UnorderedSourceTestSuiteBase.java:53)
> Nov 01 02:04:20 at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
> Nov 01 02:04:20 at
> java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1632)
> Nov 01 02:04:20 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Nov 01 02:04:20 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Nov 01 02:04:20 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Nov 01 02:04:20 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> Nov 01 02:04:20 ]>
> Nov 01 02:04:20 to be completed within 2M.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30107) Unstable test ChangelogRecoveryITCase#testMaterialization failed on azure
[
https://issues.apache.org/jira/browse/FLINK-30107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated FLINK-30107:
---
Summary: Unstable test ChangelogRecoveryITCase#testMaterialization failed
on azure (was: ChangelogRecoveryITCase#testMaterialization)
> Unstable test ChangelogRecoveryITCase#testMaterialization failed on azure
> -
>
> Key: FLINK-30107
> URL: https://issues.apache.org/jira/browse/FLINK-30107
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.16.0
>Reporter: Leonard Xu
>Priority: Major
>
> {noformat}
> Nov 18 06:18:39 [ERROR] Errors:
> Nov 18 06:18:39 [ERROR] ChangelogRecoveryITCase.testMaterialization
> Nov 18 06:18:39 [INFO] Run 1: PASS
> Nov 18 06:18:39 [ERROR] Run 2: org.apache.flink.runtime.JobException:
> Recovery is suppressed by
> FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=2,
> backoffTimeMS=0)
> Nov 18 06:18:39 [INFO] Run 3: PASS
> Nov 18 06:18:39 [INFO]
> Nov 18 06:18:39 [INFO]
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43279=logs=4d4a0d10-fca2-5507-8eed-c07f0bdf4887=7b25afdf-cc6c-566f-5459-359dc2585798
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30107) ChangelogRecoveryITCase#testMaterialization
Leonard Xu created FLINK-30107:
--
Summary: ChangelogRecoveryITCase#testMaterialization
Key: FLINK-30107
URL: https://issues.apache.org/jira/browse/FLINK-30107
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing
Affects Versions: 1.16.0
Reporter: Leonard Xu
{noformat}
Nov 18 06:18:39 [ERROR] Errors:
Nov 18 06:18:39 [ERROR] ChangelogRecoveryITCase.testMaterialization
Nov 18 06:18:39 [INFO] Run 1: PASS
Nov 18 06:18:39 [ERROR] Run 2: org.apache.flink.runtime.JobException:
Recovery is suppressed by
FixedDelayRestartBackoffTimeStrategy(maxNumberRestartAttempts=2,
backoffTimeMS=0)
Nov 18 06:18:39 [INFO] Run 3: PASS
Nov 18 06:18:39 [INFO]
Nov 18 06:18:39 [INFO]
{noformat}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43279=logs=4d4a0d10-fca2-5507-8eed-c07f0bdf4887=7b25afdf-cc6c-566f-5459-359dc2585798
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30106) Build python wheels on macs failed due to install for crcmod error
Leonard Xu created FLINK-30106:
--
Summary: Build python wheels on macs failed due to install for
crcmod error
Key: FLINK-30106
URL: https://issues.apache.org/jira/browse/FLINK-30106
Project: Flink
Issue Type: Bug
Components: API / Python, Build System / CI
Affects Versions: 1.16.0
Reporter: Leonard Xu
{noformat}
note: This error originates from a subprocess, and is likely not a
problem with pip.
ERROR: Failed building wheel for crcmod
Running setup.py clean for crcmod
error: subprocess-exited-with-error
× python setup.py clean did not run successfully.
│ exit code: 1
╰─> [14 lines of output]
error: Multiple top-level packages discovered in a flat-layout:
['python3', 'python2'].
To avoid accidental inclusion of unwanted files or directories,
setuptools will not proceed with this build.
If you are trying to create a single distribution with multiple
packages
on purpose, you should not rely on automatic discovery.
Instead, consider the following options:
1. set up custom discovery (`find` directive with `include` or
`exclude`)
2. use a `src-layout`
3. explicitly set `py_modules` or `packages` with a list of names
To find more information, look for "package discovery" on
setuptools docs.
[end of output]
note: This error originates from a subprocess, and is likely not a
problem with pip.
ERROR: Failed cleaning build dir for crcmod
Building wheel for dill (setup.py): started
Building wheel for dill (setup.py): finished with status 'done'
Created wheel for dill: filename=dill-0.3.1.1-py3-none-any.whl
size=78544
sha256=9ce160b3c3e2e1dcd24e136c59fb84ef9bd072b4715c6b34536d3ff9c39a5962
{noformat}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43279=logs=f73b5736-8355-5390-ec71-4dfdec0ce6c5=90f7230e-bf5a-531b-8566-ad48d3e03bbb
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-30103) Test InputFormatCacheLoaderTest.checkCounter failed due to unexpected value on azure
[
https://issues.apache.org/jira/browse/FLINK-30103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636425#comment-17636425
]
Leonard Xu commented on FLINK-30103:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43279=logs=0c940707-2659-5648-cbe6-a1ad63045f0a=075c2716-8010-5565-fe08-3c4bb45824a4
> Test InputFormatCacheLoaderTest.checkCounter failed due to unexpected value
> on azure
>
>
> Key: FLINK-30103
> URL: https://issues.apache.org/jira/browse/FLINK-30103
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.16.0
>Reporter: Leonard Xu
>Priority: Major
>
> {noformat}
> Nov 20 02:43:43 [ERROR] Failures:
> Nov 20 02:43:43 [ERROR] InputFormatCacheLoaderTest.checkCounter:74
> Nov 20 02:43:43 Expecting AtomicInteger(0) to have value:
> Nov 20 02:43:43 0
> Nov 20 02:43:43 but did not
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43319=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29755) PulsarSourceUnorderedE2ECase.testSavepoint failed because of missing TaskManagers
[
https://issues.apache.org/jira/browse/FLINK-29755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636424#comment-17636424
]
Leonard Xu commented on FLINK-29755:
[~syhily] I also found some shade error log and pulsar internal error log,
these error logs really make the troubleshoot harder, could you take a look ?
{noformat}
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: 2022-11-19 03:05:19,977 ERROR
org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.rejectedExecution
[] - Failed to submit a listener notification task. Event loop shut down?
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT: java.lang.NoClassDefFoundError:
org/apache/pulsar/shade/io/netty/util/concurrent/GlobalEventExecutor$2
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT:at
org.apache.pulsar.shade.io.netty.util.concurrent.GlobalEventExecutor.startThread(GlobalEventExecutor.java:223)
~[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT:at
org.apache.pulsar.shade.io.netty.util.concurrent.GlobalEventExecutor.execute0(GlobalEventExecutor.java:211)
~[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT:at
org.apache.pulsar.shade.io.netty.util.concurrent.GlobalEventExecutor.execute(GlobalEventExecutor.java:205)
~[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT:at
org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:841)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT:at
org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:499)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT:at
org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.setValue0(DefaultPromise.java:616)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT:at
org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.setSuccess0(DefaultPromise.java:605)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT:at
org.apache.pulsar.shade.io.netty.util.concurrent.DefaultPromise.setSuccess(DefaultPromise.java:96)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT:at
org.apache.pulsar.shade.io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:1057)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT:at
org.apache.pulsar.shade.io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
[blob_p-fb94d82f266979b2959919c77d8d46821bf01b74-6789386b595a9ff48b74a062fd69a96e:2.10.2]
03:05:19,978 [docker-java-stream-848840694] INFO
org.apache.flink.connector.testframe.container.FlinkContainerTestEnvironment []
- [JobManager] STDOUT:at
org.apache.pulsar.shade.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
[jira] [Comment Edited] (FLINK-28326) ResultPartitionTest.testIdleAndBackPressuredTime failed with AssertError
[
https://issues.apache.org/jira/browse/FLINK-28326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636423#comment-17636423
]
Qingsheng Ren edited comment on FLINK-28326 at 11/21/22 4:16 AM:
-
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43136=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=24c3384f-1bcb-57b3-224f-51bf973bbee8=8114]
Looks like the issue happens again. [~Weijie Guo] any thought?
was (Author: renqs):
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43136=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=24c3384f-1bcb-57b3-224f-51bf973bbee8=8114
> ResultPartitionTest.testIdleAndBackPressuredTime failed with AssertError
>
>
> Key: FLINK-28326
> URL: https://issues.apache.org/jira/browse/FLINK-28326
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Assignee: Weijie Guo
>Priority: Major
> Labels: pull-request-available, stale-assigned, test-stability
> Fix For: 1.17.0
>
>
> {code:java}
> 2022-06-30T09:23:24.0469768Z Jun 30 09:23:24 [INFO]
> 2022-06-30T09:23:24.0470382Z Jun 30 09:23:24 [ERROR] Failures:
> 2022-06-30T09:23:24.0471581Z Jun 30 09:23:24 [ERROR]
> ResultPartitionTest.testIdleAndBackPressuredTime:414
> 2022-06-30T09:23:24.0472898Z Jun 30 09:23:24 Expected: a value greater than
> <0L>
> 2022-06-30T09:23:24.0474090Z Jun 30 09:23:24 but: <0L> was equal to <0L>
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37406=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=24c3384f-1bcb-57b3-224f-51bf973bbee8
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (FLINK-28326) ResultPartitionTest.testIdleAndBackPressuredTime failed with AssertError
[
https://issues.apache.org/jira/browse/FLINK-28326?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Qingsheng Ren reopened FLINK-28326:
---
> ResultPartitionTest.testIdleAndBackPressuredTime failed with AssertError
>
>
> Key: FLINK-28326
> URL: https://issues.apache.org/jira/browse/FLINK-28326
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Assignee: Weijie Guo
>Priority: Major
> Labels: pull-request-available, stale-assigned, test-stability
> Fix For: 1.17.0
>
>
> {code:java}
> 2022-06-30T09:23:24.0469768Z Jun 30 09:23:24 [INFO]
> 2022-06-30T09:23:24.0470382Z Jun 30 09:23:24 [ERROR] Failures:
> 2022-06-30T09:23:24.0471581Z Jun 30 09:23:24 [ERROR]
> ResultPartitionTest.testIdleAndBackPressuredTime:414
> 2022-06-30T09:23:24.0472898Z Jun 30 09:23:24 Expected: a value greater than
> <0L>
> 2022-06-30T09:23:24.0474090Z Jun 30 09:23:24 but: <0L> was equal to <0L>
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37406=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=24c3384f-1bcb-57b3-224f-51bf973bbee8
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-28326) ResultPartitionTest.testIdleAndBackPressuredTime failed with AssertError
[
https://issues.apache.org/jira/browse/FLINK-28326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636423#comment-17636423
]
Qingsheng Ren commented on FLINK-28326:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43136=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=24c3384f-1bcb-57b3-224f-51bf973bbee8=8114
> ResultPartitionTest.testIdleAndBackPressuredTime failed with AssertError
>
>
> Key: FLINK-28326
> URL: https://issues.apache.org/jira/browse/FLINK-28326
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.16.0
>Reporter: Huang Xingbo
>Assignee: Weijie Guo
>Priority: Major
> Labels: pull-request-available, stale-assigned, test-stability
> Fix For: 1.17.0
>
>
> {code:java}
> 2022-06-30T09:23:24.0469768Z Jun 30 09:23:24 [INFO]
> 2022-06-30T09:23:24.0470382Z Jun 30 09:23:24 [ERROR] Failures:
> 2022-06-30T09:23:24.0471581Z Jun 30 09:23:24 [ERROR]
> ResultPartitionTest.testIdleAndBackPressuredTime:414
> 2022-06-30T09:23:24.0472898Z Jun 30 09:23:24 Expected: a value greater than
> <0L>
> 2022-06-30T09:23:24.0474090Z Jun 30 09:23:24 but: <0L> was equal to <0L>
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=37406=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=24c3384f-1bcb-57b3-224f-51bf973bbee8
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29755) PulsarSourceUnorderedE2ECase.testSavepoint failed because of missing TaskManagers
[
https://issues.apache.org/jira/browse/FLINK-29755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636422#comment-17636422
]
Leonard Xu commented on FLINK-29755:
PulsarSourceUnorderedE2ECase.testSourceSingleSplit failed on azure may due to
same issue
https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_apis/build/builds/43307/logs/847
> PulsarSourceUnorderedE2ECase.testSavepoint failed because of missing
> TaskManagers
> -
>
> Key: FLINK-29755
> URL: https://issues.apache.org/jira/browse/FLINK-29755
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Matthias Pohl
>Priority: Critical
> Labels: test-stability
> Attachments: PulsarSourceUnorderedE2ECase.testSavepoint.log
>
>
> [This
> build|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42325=logs=af184cdd-c6d8-5084-0b69-7e9c67b35f7a=160c9ae5-96fd-516e-1c91-deb81f59292a=13932]
> failed (not exclusively) due to a problem with
> {{PulsarSourceUnorderedE2ECase.testSavepoint}}. It seems like there were no
> TaskManagers spun up which resulted in the test job failing with a
> {{NoResourceAvailableException}}.
> {code}
> org.apache.flink.runtime.jobmaster.slotpool.DeclarativeSlotPoolBridge [] -
> Could not acquire the minimum required resources, failing slot requests.
> Acquired: []. Current slot pool status: Registered TMs: 0, registered slots:
> 0 free slots: 0
> {code}
> I didn't raise this one to critical because it looks like a missing
> TaskManager test environment issue. I attached the e2e test-specific logs to
> the Jira issue.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink] fsk119 commented on a diff in pull request #21322: [FLINK-30025][table] the max column width should be resizable while printing the query result to the client console.
fsk119 commented on code in PR #21322:
URL: https://github.com/apache/flink/pull/21322#discussion_r1027549706
##
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/TableConfigOptions.java:
##
@@ -102,6 +102,16 @@ private TableConfigOptions() {}
+ "the session time zone is used during
conversion. The input of option is either a full name "
+ "such as \"America/Los_Angeles\", or a
custom timezone id such as \"GMT-08:00\".");
+@Documentation.TableOption(execMode = Documentation.ExecMode.STREAMING)
+public static final ConfigOption PRINT_MAX_COLUMN_WIDTH =
Review Comment:
Currently SQL Client also has the option
`sql-client.display.max-column-width`. When users set the option
`sql-client.display.max-column-width` or `sql-client.display.max-column-width`
in the client, can we adjust the column width?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] fsk119 commented on a diff in pull request #21322: [FLINK-30025][table] the max column width should be resizable while printing the query result to the client console.
fsk119 commented on code in PR #21322:
URL: https://github.com/apache/flink/pull/21322#discussion_r1027555804
##
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/TableEnvironmentImpl.java:
##
@@ -926,7 +926,7 @@ private TableResultInternal
executeQueryOperation(QueryOperation operation) {
DataTypeUtils.expandCompositeTypeToSchema(
sinkOperation.getConsumedDataType()),
resultProvider.getRowDataStringConverter(),
-PrintStyle.DEFAULT_MAX_COLUMN_WIDTH,
Review Comment:
I think we should remove the PrintStyle.DEFAULT_MAX_COLUMN_WIDTH now.
##
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/config/TableConfigOptions.java:
##
@@ -102,6 +102,16 @@ private TableConfigOptions() {}
+ "the session time zone is used during
conversion. The input of option is either a full name "
+ "such as \"America/Los_Angeles\", or a
custom timezone id such as \"GMT-08:00\".");
+@Documentation.TableOption(execMode = Documentation.ExecMode.STREAMING)
+public static final ConfigOption PRINT_MAX_COLUMN_WIDTH =
Review Comment:
Currently SQL Client also has the option
`sql-client.display.max-column-width`. When users set the option
`sql-client.display.max-column-width` in the client, can we adjust the column
width?
##
docs/layouts/shortcodes/generated/table_config_configuration.html:
##
@@ -62,6 +62,12 @@
Enum
Strategy how to restore catalog objects such as tables,
functions, or data types using a given plan and performing catalog lookups if
necessary. It influences the need for catalog metadata to bepresent and enables
partial enrichment of plan information.Possible
values:"ALL": Reads all metadata about catalog tables, functions, or
data types that has been persisted in the plan. The strategy performs a catalog
lookup by identifier to fill in missing information or enrich mutable options.
If the original object is not available in the catalog anymore, pipelines can
still be restored if all information necessary is contained in the
plan."ALL_ENFORCED": Requires that all metadata about catalog tables,
functions, or data types has been persisted in the plan. The strategy will
neither perform a catalog lookup by identifier nor enrich mutable options with
catalog information. A restore will fail if not all information necessary is
contained in the
plan."IDENTIFIER": Uses only the identifier of catalog tables,
functions, or data types and always performs a catalog lookup. A restore will
fail if the original object is not available in the catalog anymore. Additional
metadata that might be contained in the plan will be ignored.
+
+table.print.max-column-width Streaming
Review Comment:
I think the name should be discussed. I lean to
`table.display.max-column-width`. cc @wuchong @godfreyhe
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] ruanhang1993 commented on pull request #21197: [FLINK-29801] OperatorCoordinator need open the way to operate metric…
ruanhang1993 commented on PR #21197: URL: https://github.com/apache/flink/pull/21197#issuecomment-1321431135 > @ruanhang1993 would you work with @JerryYue-M on the FLIP? > > cc @PatrickRen > FLIP for the change and a discuss it in the community. I would love to help. @JerryYue-M , I will create a FLIP about this issue. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-29801) OperatorCoordinator need open the way to operate metricGroup interface
[ https://issues.apache.org/jira/browse/FLINK-29801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636421#comment-17636421 ] Hang Ruan edited comment on FLINK-29801 at 11/21/22 3:52 AM: - Hi, [~MengYue] , For the question 1, what I mean is better to pass the metric group in the `OperatorCoordinator.Context` instead of using it by invoking the method `registeMetric` provided in the PR. For the question 2, it seems that the implementations of `OperatorCoordinator` need to distinguish the metric group name by self, which may need to be done by Flink. IMO, I think we need a new metric group for the `OperatorCoordinator`. And we need the FLIP for the change and a discuss it in the community. This needs to spend some time in the discussion it in the community. was (Author: ruanhang1993): Hi, [~MengYue] , For the question 1, what I means is better to pass the metric group in the `OperatorCoordinator.Context` instead of using it by invoking the method `registeMetric` provided in the PR. For the question 2, it seems that the implementations of `OperatorCoordinator` need to distinguish the metric group name by self, which may need to be done by Flink. IMO, I think we need a new metric group for the `OperatorCoordinator`. And we need the FLIP for the change and a discuss it in the community. This needs to spend some time in the discussion it in the community. > OperatorCoordinator need open the way to operate metricGroup interface > -- > > Key: FLINK-29801 > URL: https://issues.apache.org/jira/browse/FLINK-29801 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics >Reporter: yuemeng >Assignee: yuemeng >Priority: Major > Labels: pull-request-available > > Currently, We have no way to get metric group instances in OperatorCoordinator > In some cases, we may report some metric in OperatorCoordinator such as Flink > hudi integrate scene， some meta will send to operator coordinator to commit > to hdfs or hms > but we also need to report some metrics in operator coordinator for monitor > purpose -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29801) OperatorCoordinator need open the way to operate metricGroup interface
[ https://issues.apache.org/jira/browse/FLINK-29801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636421#comment-17636421 ] Hang Ruan commented on FLINK-29801: --- Hi, [~MengYue] , For the question 1, what I means is better to pass the metric group in the `OperatorCoordinator.Context` instead of using it by invoking the method `registeMetric` provided in the PR. For the question 2, it seems that the implementations of `OperatorCoordinator` need to distinguish the metric group name by self, which may need to be done by Flink. IMO, I think we need a new metric group for the `OperatorCoordinator`. And we need the FLIP for the change and a discuss it in the community. This needs to spend some time in the discussion it in the community. > OperatorCoordinator need open the way to operate metricGroup interface > -- > > Key: FLINK-29801 > URL: https://issues.apache.org/jira/browse/FLINK-29801 > Project: Flink > Issue Type: Improvement > Components: Runtime / Metrics >Reporter: yuemeng >Assignee: yuemeng >Priority: Major > Labels: pull-request-available > > Currently, We have no way to get metric group instances in OperatorCoordinator > In some cases, we may report some metric in OperatorCoordinator such as Flink > hudi integrate scene， some meta will send to operator coordinator to commit > to hdfs or hms > but we also need to report some metrics in operator coordinator for monitor > purpose -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30105) SubtaskExecutionAttemptAccumulatorsInfoTest failed with JVM exit code 239
Qingsheng Ren created FLINK-30105:
-
Summary: SubtaskExecutionAttemptAccumulatorsInfoTest failed with
JVM exit code 239
Key: FLINK-30105
URL: https://issues.apache.org/jira/browse/FLINK-30105
Project: Flink
Issue Type: Bug
Components: Runtime / REST
Affects Versions: 1.17.0
Reporter: Qingsheng Ren
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43124=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=24c3384f-1bcb-57b3-224f-51bf973bbee8=8164]
{code:java}
Nov 14 08:08:38 [ERROR]
org.apache.flink.runtime.rest.messages.job.SubtaskExecutionAttemptAccumulatorsInfoTest
Nov 14 08:08:38 [ERROR]
org.apache.maven.surefire.booter.SurefireBooterForkException:
ExecutionException The forked VM terminated without properly saying goodbye. VM
crash or System.exit called?
Nov 14 08:08:38 [ERROR] Command was /bin/sh -c cd /__w/1/s/flink-runtime &&
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java -XX:+UseG1GC -Xms256m -Xmx768m
-jar
/__w/1/s/flink-runtime/target/surefire/surefirebooter285048696362473297.jar
/__w/1/s/flink-runtime/target/surefire 2022-11-14T08-03-00_124-jvmRun3
surefire3531696916359342131tmp surefire_26906836589585616499tmp
Nov 14 08:08:38 [ERROR] Error occurred in starting fork, check output in log
Nov 14 08:08:38 [ERROR] Process Exit Code: 239
Nov 14 08:08:38 [ERROR] Crashed tests:
Nov 14 08:08:38 [ERROR]
org.apache.flink.runtime.rest.messages.job.SubtaskExecutionAttemptAccumulatorsInfoTest
Nov 14 08:08:38 [ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.awaitResultsDone(ForkStarter.java:532)
Nov 14 08:08:38 [ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.runSuitesForkOnceMultiple(ForkStarter.java:405)
Nov 14 08:08:38 [ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:321)
Nov 14 08:08:38 [ERROR] at
org.apache.maven.plugin.surefire.booterclient.ForkStarter.run(ForkStarter.java:266)
Nov 14 08:08:38 [ERROR] at
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeProvider(AbstractSurefireMojo.java:1314)
Nov 14 08:08:38 [ERROR] at
org.apache.maven.plugin.surefire.AbstractSurefireMojo.executeAfterPreconditionsChecked(AbstractSurefireMojo.java:1159)
Nov 14 08:08:38 [ERROR] at
org.apache.maven.plugin.surefire.AbstractSurefireMojo.execute(AbstractSurefireMojo.java:932)
Nov 14 08:08:38 [ERROR] at
org.apache.maven.plugin.DefaultBuildPluginManager.executeMojo(DefaultBuildPluginManager.java:132)
Nov 14 08:08:38 [ERROR] at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:208)
Nov 14 08:08:38 [ERROR] at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:153)
Nov 14 08:08:38 [ERROR] at
org.apache.maven.lifecycle.internal.MojoExecutor.execute(MojoExecutor.java:145)
Nov 14 08:08:38 [ERROR] at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:116)
Nov 14 08:08:38 [ERROR] at
org.apache.maven.lifecycle.internal.LifecycleModuleBuilder.buildProject(LifecycleModuleBuilder.java:80)
Nov 14 08:08:38 [ERROR] at
org.apache.maven.lifecycle.internal.builder.singlethreaded.SingleThreadedBuilder.build(SingleThreadedBuilder.java:51)
Nov 14 08:08:38 [ERROR] at
org.apache.maven.lifecycle.internal.LifecycleStarter.execute(LifecycleStarter.java:120)
Nov 14 08:08:38 [ERROR] at
org.apache.maven.DefaultMaven.doExecute(DefaultMaven.java:355)
Nov 14 08:08:38 [ERROR] at
org.apache.maven.DefaultMaven.execute(DefaultMaven.java:155)
Nov 14 08:08:38 [ERROR] at
org.apache.maven.cli.MavenCli.execute(MavenCli.java:584)
Nov 14 08:08:38 [ERROR] at
org.apache.maven.cli.MavenCli.doMain(MavenCli.java:216)
Nov 14 08:08:38 [ERROR] at org.apache.maven.cli.MavenCli.main(MavenCli.java:160)
Nov 14 08:08:38 [ERROR] at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
Method)
Nov 14 08:08:38 [ERROR] at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
Nov 14 08:08:38 [ERROR] at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
Nov 14 08:08:38 [ERROR] at java.lang.reflect.Method.invoke(Method.java:498)
Nov 14 08:08:38 [ERROR] at
org.codehaus.plexus.classworlds.launcher.Launcher.launchEnhanced(Launcher.java:289)
Nov 14 08:08:38 [ERROR] at
org.codehaus.plexus.classworlds.launcher.Launcher.launch(Launcher.java:229)
Nov 14 08:08:38 [ERROR] at
org.codehaus.plexus.classworlds.launcher.Launcher.mainWithExitCode(Launcher.java:415)
Nov 14 08:08:38 [ERROR] at
org.codehaus.plexus.classworlds.launcher.Launcher.main(Launcher.java:356)
Nov 14 08:08:38 [ERROR] Caused by:
org.apache.maven.surefire.booter.SurefireBooterForkException: The forked VM
terminated without properly saying goodbye. VM crash or System.exit called?
Nov 14 08:08:38 [ERROR] Command was /bin/sh -c cd /__w/1/s/flink-runtime &&
/usr/lib/jvm/java-8-openjdk-amd64/jre/bin/java
[jira] [Commented] (FLINK-26974) Python EmbeddedThreadDependencyTests.test_add_python_file failed on azure
[
https://issues.apache.org/jira/browse/FLINK-26974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636418#comment-17636418
]
Qingsheng Ren commented on FLINK-26974:
---
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43111=logs=3e4dd1a2-fe2f-5e5d-a581-48087e718d53=b4612f28-e3b5-5853-8a8b-610ae894217a=26123
> Python EmbeddedThreadDependencyTests.test_add_python_file failed on azure
> -
>
> Key: FLINK-26974
> URL: https://issues.apache.org/jira/browse/FLINK-26974
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.15.0, 1.16.0, 1.17.0
>Reporter: Yun Gao
>Assignee: Huang Xingbo
>Priority: Minor
> Labels: auto-deprioritized-major, test-stability
>
> {code:java}
> Mar 31 10:49:17 === FAILURES
> ===
> Mar 31 10:49:17 __
> EmbeddedThreadDependencyTests.test_add_python_file __
> Mar 31 10:49:17
> Mar 31 10:49:17 self =
> testMethod=test_add_python_file>
> Mar 31 10:49:17
> Mar 31 10:49:17 def test_add_python_file(self):
> Mar 31 10:49:17 python_file_dir = os.path.join(self.tempdir,
> "python_file_dir_" + str(uuid.uuid4()))
> Mar 31 10:49:17 os.mkdir(python_file_dir)
> Mar 31 10:49:17 python_file_path = os.path.join(python_file_dir,
> "test_dependency_manage_lib.py")
> Mar 31 10:49:17 with open(python_file_path, 'w') as f:
> Mar 31 10:49:17 f.write("def add_two(a):\nraise
> Exception('This function should not be called!')")
> Mar 31 10:49:17 self.t_env.add_python_file(python_file_path)
> Mar 31 10:49:17
> Mar 31 10:49:17 python_file_dir_with_higher_priority = os.path.join(
> Mar 31 10:49:17 self.tempdir, "python_file_dir_" +
> str(uuid.uuid4()))
> Mar 31 10:49:17 os.mkdir(python_file_dir_with_higher_priority)
> Mar 31 10:49:17 python_file_path_higher_priority =
> os.path.join(python_file_dir_with_higher_priority,
> Mar 31 10:49:17
> "test_dependency_manage_lib.py")
> Mar 31 10:49:17 with open(python_file_path_higher_priority, 'w') as f:
> Mar 31 10:49:17 f.write("def add_two(a):\nreturn a + 2")
> Mar 31 10:49:17
> self.t_env.add_python_file(python_file_path_higher_priority)
> Mar 31 10:49:17
> Mar 31 10:49:17 def plus_two(i):
> Mar 31 10:49:17 from test_dependency_manage_lib import add_two
> Mar 31 10:49:17 return add_two(i)
> Mar 31 10:49:17
> Mar 31 10:49:17 self.t_env.create_temporary_system_function(
> Mar 31 10:49:17 "add_two", udf(plus_two, DataTypes.BIGINT(),
> DataTypes.BIGINT()))
> Mar 31 10:49:17 table_sink = source_sink_utils.TestAppendSink(
> Mar 31 10:49:17 ['a', 'b'], [DataTypes.BIGINT(),
> DataTypes.BIGINT()])
> Mar 31 10:49:17 self.t_env.register_table_sink("Results", table_sink)
> Mar 31 10:49:17 t = self.t_env.from_elements([(1, 2), (2, 5), (3,
> 1)], ['a', 'b'])
> Mar 31 10:49:17 > t.select(expr.call("add_two", t.a),
> t.a).execute_insert("Results").wait()
> Mar 31 10:49:17
> Mar 31 10:49:17 pyflink/table/tests/test_dependency.py:63:
> Mar 31 10:49:17 _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
> _ _ _ _ _ _ _ _ _
> Mar 31 10:49:17 pyflink/table/table_result.py:76: in wait
> Mar 31 10:49:17 get_method(self._j_table_result, "await")()
> Mar 31 10:49:17
> .tox/py38-cython/lib/python3.8/site-packages/py4j/java_gateway.py:1321: in
> __call__
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=34001=logs=821b528f-1eed-5598-a3b4-7f748b13f261=6bb545dd-772d-5d8c-f258-f5085fba3295=27239
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-30097) CachedDataStream java example in the document is not correct
[
https://issues.apache.org/jira/browse/FLINK-30097?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xingbo Huang reassigned FLINK-30097:
Assignee: Xuannan Su
> CachedDataStream java example in the document is not correct
>
>
> Key: FLINK-30097
> URL: https://issues.apache.org/jira/browse/FLINK-30097
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.16.0
>Reporter: Prabhu Joseph
>Assignee: Xuannan Su
>Priority: Minor
>
> CachedDataStream java example in the document is not correct -
> [https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/overview/#datastream-rarr-cacheddatastream]
>
> {code:java}
> DataStream dataStream = //...
> CachedDataStream cachedDataStream = dataStream.cache();{code}
> The example shows to invoke cache() on a DataStream instance but DataStream
> class does not have cache() method. The right usage is to call cache() on an
> instance of DataStreamSource/SideOutputDataStream/SingleOutputStreamOperator.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-29462) LookupJoinITCase failed on azure due to classloader leaking
[
https://issues.apache.org/jira/browse/FLINK-29462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636417#comment-17636417
]
Qingsheng Ren commented on FLINK-29462:
---
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43111=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94=18169]
> LookupJoinITCase failed on azure due to classloader leaking
> ---
>
> Key: FLINK-29462
> URL: https://issues.apache.org/jira/browse/FLINK-29462
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.0
>Reporter: Yun Gao
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 09:53:45,656 [ForkJoinPool.commonPool-worker-8] WARN
> org.apache.flink.table.runtime.generated.GeneratedClass [] - Failed to
> compile split code, falling back to original code
> org.apache.flink.util.FlinkRuntimeException:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.
> at
> org.apache.flink.table.runtime.generated.CompileUtils.compile(CompileUtils.java:94)
> ~[flink-table-runtime-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.compile(GeneratedClass.java:97)
> ~[flink-table-runtime-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.runtime.generated.GeneratedClass.newInstance(GeneratedClass.java:68)
> ~[flink-table-runtime-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.runtime.keyselector.GenericRowDataKeySelector.open(GenericRowDataKeySelector.java:50)
> ~[flink-table-runtime-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.runtime.functions.table.lookup.fullcache.inputformat.InputSplitCacheLoadTask.(InputSplitCacheLoadTask.java:60)
> ~[flink-table-runtime-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.runtime.functions.table.lookup.fullcache.inputformat.InputFormatCacheLoader.createCacheLoadTask(InputFormatCacheLoader.java:135)
> ~[flink-table-runtime-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.runtime.functions.table.lookup.fullcache.inputformat.InputFormatCacheLoader.lambda$reloadCache$0(InputFormatCacheLoader.java:84)
> ~[flink-table-runtime-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> [?:1.8.0_292]
> at
> java.util.Spliterators$ArraySpliterator.forEachRemaining(Spliterators.java:948)
> [?:1.8.0_292]
> at
> java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482)
> [?:1.8.0_292]
> at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> [?:1.8.0_292]
> at
> java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:708)
> [?:1.8.0_292]
> at
> java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
> [?:1.8.0_292]
> at
> java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:566)
> [?:1.8.0_292]
> at
> org.apache.flink.table.runtime.functions.table.lookup.fullcache.inputformat.InputFormatCacheLoader.reloadCache(InputFormatCacheLoader.java:85)
> [flink-table-runtime-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> org.apache.flink.table.runtime.functions.table.lookup.fullcache.CacheLoader.run(CacheLoader.java:105)
> [flink-table-runtime-1.17-SNAPSHOT.jar:1.17-SNAPSHOT]
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
> [?:1.8.0_292]
> at
> java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1632)
> [?:1.8.0_292]
> at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> [?:1.8.0_292]
> at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> [?:1.8.0_292]
> at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> [?:1.8.0_292]
> at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> [?:1.8.0_292]
> Caused by:
> org.apache.flink.shaded.guava30.com.google.common.util.concurrent.UncheckedExecutionException:
> org.apache.flink.api.common.InvalidProgramException: Table program cannot be
> compiled. This is a bug. Please file an issue.
> at
> org.apache.flink.shaded.guava30.com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2051)
> ~[flink-shaded-guava-30.1.1-jre-15.0.jar:30.1.1-jre-15.0]
> at
> org.apache.flink.shaded.guava30.com.google.common.cache.LocalCache.get(LocalCache.java:3962)
> ~[flink-shaded-guava-30.1.1-jre-15.0.jar:30.1.1-jre-15.0]
> at
>
[jira] [Commented] (FLINK-29836) PulsarUnorderedSourceITCase failed due to an expected job restart not happening
[
https://issues.apache.org/jira/browse/FLINK-29836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636415#comment-17636415
]
Leonard Xu commented on FLINK-29836:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43308=logs=a5ef94ef-68c2-57fd-3794-dc108ed1c495=2c68b137-b01d-55c9-e603-3ff3f320364b
> PulsarUnorderedSourceITCase failed due to an expected job restart not
> happening
> ---
>
> Key: FLINK-29836
> URL: https://issues.apache.org/jira/browse/FLINK-29836
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Pulsar
>Affects Versions: 1.15.3
>Reporter: Matthias Pohl
>Priority: Major
> Labels: test-stability
>
> [This build
> failed|https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=42681=logs=a5ef94ef-68c2-57fd-3794-dc108ed1c495=2c68b137-b01d-55c9-e603-3ff3f320364b=27377]
> due to a missing job restart:
> {code:java}
> Nov 01 02:04:20 Expecting
> Nov 01 02:04:20Nov 01 02:04:20 java.lang.RuntimeException: Job restarted
> Nov 01 02:04:20 at
> org.apache.flink.streaming.api.operators.collect.UncheckpointedCollectResultBuffer.sinkRestarted(UncheckpointedCollectResultBuffer.java:42)
> Nov 01 02:04:20 at
> org.apache.flink.streaming.api.operators.collect.AbstractCollectResultBuffer.dealWithResponse(AbstractCollectResultBuffer.java:87)
> Nov 01 02:04:20 at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:147)
> Nov 01 02:04:20 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:106)
> Nov 01 02:04:20 at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:80)
> Nov 01 02:04:20 at
> org.apache.flink.connector.testframe.utils.UnorderedCollectIteratorAssert.compareWithExactlyOnceSemantic(UnorderedCollectIteratorAssert.java:108)
> Nov 01 02:04:20 at
> org.apache.flink.connector.testframe.utils.UnorderedCollectIteratorAssert.matchesRecordsFromSource(UnorderedCollectIteratorAssert.java:79)
> Nov 01 02:04:20 at
> org.apache.flink.connector.pulsar.testutils.source.UnorderedSourceTestSuiteBase.lambda$checkResultWithSemantic$0(UnorderedSourceTestSuiteBase.java:53)
> Nov 01 02:04:20 at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1640)
> Nov 01 02:04:20 at
> java.util.concurrent.CompletableFuture$AsyncRun.exec(CompletableFuture.java:1632)
> Nov 01 02:04:20 at
> java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
> Nov 01 02:04:20 at
> java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
> Nov 01 02:04:20 at
> java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
> Nov 01 02:04:20 at
> java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
> Nov 01 02:04:20 ]>
> Nov 01 02:04:20 to be completed within 2M.
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-26624) Running HA (hashmap, async) end-to-end test failed on azure due to unable to find master logs
[
https://issues.apache.org/jira/browse/FLINK-26624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636414#comment-17636414
]
Leonard Xu commented on FLINK-26624:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43308=logs=e9d3d34f-3d15-59f4-0e3e-35067d100dfe=f8a6d3eb-38cf-5cca-9a99-d0badeb5fe62
> Running HA (hashmap, async) end-to-end test failed on azure due to unable to
> find master logs
> -
>
> Key: FLINK-26624
> URL: https://issues.apache.org/jira/browse/FLINK-26624
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.15.0
>Reporter: Yun Gao
>Priority: Minor
> Labels: auto-deprioritized-major, test-stability
>
> {code:java}
> Mar 12 04:31:15 Waiting for text Completed checkpoint [1-9]* for job
> 699ebf9bdcb51a9fe76db5463027d34c to appear 2 of times in logs...
> grep:
> /home/vsts/work/_temp/debug_files/flink-logs/*standalonesession-1*.log*: No
> such file or directory
> Mar 12 04:31:16 Starting standalonesession daemon on host fv-az302-918.
> grep:
> /home/vsts/work/_temp/debug_files/flink-logs/*standalonesession-1*.log*: No
> such file or directory
> Mar 12 04:41:23 A timeout occurred waiting for Completed checkpoint [1-9]*
> for job 699ebf9bdcb51a9fe76db5463027d34c to appear 2 of times in logs.
> Mar 12 04:41:23 Stopping job timeout watchdog (with pid=272045)
> Mar 12 04:41:23 Killing JM watchdog @ 273681
> Mar 12 04:41:23 Killing TM watchdog @ 274268
> Mar 12 04:41:23 [FAIL] Test script contains errors.
> Mar 12 04:41:23 Checking of logs skipped.
> Mar 12 04:41:23
> Mar 12 04:41:23 [FAIL] 'Running HA (hashmap, async) end-to-end test' failed
> after 10 minutes and 31 seconds! Test exited with exit code 1
> Mar 12 04:41:23
> 04:41:23 ##[group]Environment Information
> Mar 12 04:41:24 Searching for .dump, .dumpstream and related files in
> '/home/vsts/work/1/s'
> dmesg: read kernel buffer failed: Operation not permitted
> Mar 12 04:41:28 Stopping taskexecutor daemon (pid: 272837) on host
> fv-az302-918.
> Mar 12 04:41:29 Stopping standalonesession daemon (pid: 274590) on host
> fv-az302-918.
> Mar 12 04:41:35 Stopping zookeeper...
> Mar 12 04:41:36 Stopping zookeeper daemon (pid: 272248) on host fv-az302-918.
> The STDIO streams did not close within 10 seconds of the exit event from
> process '/usr/bin/bash'. This may indicate a child process inherited the
> STDIO streams and has not yet exited.
> ##[error]Bash exited with code '1'.
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=32945=logs=bea52777-eaf8-5663-8482-18fbc3630e81=b2642e3a-5b86-574d-4c8a-f7e2842bfb14
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30104) Snapshot deployment failed due to could not transfer artifact org.apache.flink:flink-runtime-web:jar:1.15-20221119.010836-325 from/to apache.snapshots
[
https://issues.apache.org/jira/browse/FLINK-30104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu updated FLINK-30104:
---
Summary: Snapshot deployment failed due to could not transfer artifact
org.apache.flink:flink-runtime-web:jar:1.15-20221119.010836-325 from/to
apache.snapshots (was: Snapshot deployment failed due to Could not transfer
artifact org.apache.flink:flink-runtime-web:jar:1.15-20221119.010836-325
from/to apache.snapshots)
> Snapshot deployment failed due to could not transfer artifact
> org.apache.flink:flink-runtime-web:jar:1.15-20221119.010836-325 from/to
> apache.snapshots
> --
>
> Key: FLINK-30104
> URL: https://issues.apache.org/jira/browse/FLINK-30104
> Project: Flink
> Issue Type: Bug
> Components: Build System / CI
>Affects Versions: 1.15.2
>Reporter: Leonard Xu
>Priority: Major
>
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-deploy-plugin:2.8.2:deploy (default-deploy) on
> project flink-runtime-web: Failed to deploy artifacts: Could not transfer
> artifact org.apache.flink:flink-runtime-web:jar:1.15-20221119.010836-325
> from/to apache.snapshots.https
> (https://repository.apache.org/content/repositories/snapshots): Failed to
> transfer file:
> https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-runtime-web/1.15-SNAPSHOT/flink-runtime-web-1.15-20221119.010836-325.jar.
> Return code is: 502, ReasonPhrase: Proxy Error. -> [Help 1]
> {noformat}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43308=logs=eca6b3a6-1600-56cc-916a-c549b3cde3ff=e9844b5e-5aa3-546b-6c3e-5395c7c0cac7
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (FLINK-30104) Snapshot deployment failed due to Could not transfer artifact org.apache.flink:flink-runtime-web:jar:1.15-20221119.010836-325 from/to apache.snapshots
Leonard Xu created FLINK-30104:
--
Summary: Snapshot deployment failed due to Could not transfer
artifact org.apache.flink:flink-runtime-web:jar:1.15-20221119.010836-325
from/to apache.snapshots
Key: FLINK-30104
URL: https://issues.apache.org/jira/browse/FLINK-30104
Project: Flink
Issue Type: Bug
Components: Build System / CI
Affects Versions: 1.15.2
Reporter: Leonard Xu
{noformat}
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-deploy-plugin:2.8.2:deploy (default-deploy) on
project flink-runtime-web: Failed to deploy artifacts: Could not transfer
artifact org.apache.flink:flink-runtime-web:jar:1.15-20221119.010836-325
from/to apache.snapshots.https
(https://repository.apache.org/content/repositories/snapshots): Failed to
transfer file:
https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-runtime-web/1.15-SNAPSHOT/flink-runtime-web-1.15-20221119.010836-325.jar.
Return code is: 502, ReasonPhrase: Proxy Error. -> [Help 1]
{noformat}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43308=logs=eca6b3a6-1600-56cc-916a-c549b3cde3ff=e9844b5e-5aa3-546b-6c3e-5395c7c0cac7
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-30101) Always use StandaloneClientHAServices to create RestClusterClient when retriving a Flink on YARN cluster client
[ https://issues.apache.org/jira/browse/FLINK-30101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhanghao Chen updated FLINK-30101: -- Description: *Problem* Currently, the procedure of retrieving a Flink on YARN cluster client is as follows (in YarnClusterDescriptor#retrieve method): # Get application report from YARN # Set rest.address & rest.port using the info from application report # Create a new RestClusterClient using the updated configuration, will use client HA serivce to fetch the rest.address & rest.port if HA is enabled Here, we can see that the usage of client HA in step 3 is redundant, as we've already got the rest.address & rest.port from YARN application report. When ZK HA is enabled, this would take ~1.5 s to initialize client HA services and fetch the rest IP & port. 1.5 s can mean a lot for latency-sensitive client operations. In my company, we use Flink client to submit short-running session jobs and e2e latency is critical. The job submission time is around 10 s on average, and 1.5s would mean a 15% time saving. *Proposal* When retrieving a Flink on YARN cluster client, use StandaloneClientHAServices to create RestClusterClient instead as we have pre-fetched rest.address & rest.port from YARN application report. This is also what we did in KubernetesClusterDescriptor. was: *Problem* Currently, the procedure of retrieving a Flink on YARN cluster client is as follows (in YarnClusterDescriptor#retrieve method): # Get application report from YARN # Set rest.address & rest.port using the info from application report # Create a new RestClusterClient using the updated configuration, will use client HA serivce to fetch the rest.address & rest.port if HA is enabled Here, we can see that the usage of client HA in step 3 is redundant, as we've already got the rest.address & rest.port from YARN application report. When ZK HA is enabled, this would take ~1.5 s to initialize client HA services and fetch the rest IP & port. 1.5 s can mean a lot for latency-sensitive client operations. In my company, we use Flink client to submit short-running session jobs and e2e latency is critical. The job submission time is around 10 s on average, and 1.5s would mean 15% of time saving. *Proposal* When retrieving a Flink on YARN cluster client, use StandaloneClientHAServices to create RestClusterClient instead as we have pre-fetched rest.address & rest.port from YARN application report. This is also what we did in KubernetesClusterDescriptor. > Always use StandaloneClientHAServices to create RestClusterClient when > retriving a Flink on YARN cluster client > > > Key: FLINK-30101 > URL: https://issues.apache.org/jira/browse/FLINK-30101 > Project: Flink > Issue Type: Improvement > Components: Client / Job Submission >Affects Versions: 1.16.0 >Reporter: Zhanghao Chen >Priority: Major > Fix For: 1.17.0 > > > *Problem* > Currently, the procedure of retrieving a Flink on YARN cluster client is as > follows (in YarnClusterDescriptor#retrieve method): > # Get application report from YARN > # Set rest.address & rest.port using the info from application report > # Create a new RestClusterClient using the updated configuration, will use > client HA serivce to fetch the rest.address & rest.port if HA is enabled > Here, we can see that the usage of client HA in step 3 is redundant, as we've > already got the rest.address & rest.port from YARN application report. When > ZK HA is enabled, this would take ~1.5 s to initialize client HA services and > fetch the rest IP & port. > 1.5 s can mean a lot for latency-sensitive client operations. In my company, > we use Flink client to submit short-running session jobs and e2e latency is > critical. The job submission time is around 10 s on average, and 1.5s would > mean a 15% time saving. > *Proposal* > When retrieving a Flink on YARN cluster client, use > StandaloneClientHAServices to > create RestClusterClient instead as we have pre-fetched rest.address & > rest.port from YARN application report. This is also what we did in > KubernetesClusterDescriptor. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29427) LookupJoinITCase failed with classloader problem
[
https://issues.apache.org/jira/browse/FLINK-29427?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636412#comment-17636412
]
Leonard Xu commented on FLINK-29427:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43309=logs=f2c100be-250b-5e85-7bbe-176f68fcddc5=05efd11e-5400-54a4-0d27-a4663be008a9
> LookupJoinITCase failed with classloader problem
>
>
> Key: FLINK-29427
> URL: https://issues.apache.org/jira/browse/FLINK-29427
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.16.0, 1.17.0
>Reporter: Huang Xingbo
>Assignee: Alexander Smirnov
>Priority: Critical
> Labels: test-stability
>
> {code:java}
> 2022-09-27T02:49:20.9501313Z Sep 27 02:49:20 Caused by:
> org.codehaus.janino.InternalCompilerException: Compiling
> "KeyProjection$108341": Trying to access closed classloader. Please check if
> you store classloaders directly or indirectly in static fields. If the
> stacktrace suggests that the leak occurs in a third party library and cannot
> be fixed immediately, you can disable this check with the configuration
> 'classloader.check-leaked-classloader'.
> 2022-09-27T02:49:20.9502654Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.compileUnit(UnitCompiler.java:382)
> 2022-09-27T02:49:20.9503366Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:237)
> 2022-09-27T02:49:20.9504044Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.compileToClassLoader(SimpleCompiler.java:465)
> 2022-09-27T02:49:20.9504704Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:216)
> 2022-09-27T02:49:20.9505341Z Sep 27 02:49:20 at
> org.codehaus.janino.SimpleCompiler.cook(SimpleCompiler.java:207)
> 2022-09-27T02:49:20.9505965Z Sep 27 02:49:20 at
> org.codehaus.commons.compiler.Cookable.cook(Cookable.java:80)
> 2022-09-27T02:49:20.9506584Z Sep 27 02:49:20 at
> org.codehaus.commons.compiler.Cookable.cook(Cookable.java:75)
> 2022-09-27T02:49:20.9507261Z Sep 27 02:49:20 at
> org.apache.flink.table.runtime.generated.CompileUtils.doCompile(CompileUtils.java:104)
> 2022-09-27T02:49:20.9507883Z Sep 27 02:49:20 ... 30 more
> 2022-09-27T02:49:20.9509266Z Sep 27 02:49:20 Caused by:
> java.lang.IllegalStateException: Trying to access closed classloader. Please
> check if you store classloaders directly or indirectly in static fields. If
> the stacktrace suggests that the leak occurs in a third party library and
> cannot be fixed immediately, you can disable this check with the
> configuration 'classloader.check-leaked-classloader'.
> 2022-09-27T02:49:20.9510835Z Sep 27 02:49:20 at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.ensureInner(FlinkUserCodeClassLoaders.java:184)
> 2022-09-27T02:49:20.9511760Z Sep 27 02:49:20 at
> org.apache.flink.util.FlinkUserCodeClassLoaders$SafetyNetWrapperClassLoader.loadClass(FlinkUserCodeClassLoaders.java:192)
> 2022-09-27T02:49:20.9512456Z Sep 27 02:49:20 at
> java.lang.Class.forName0(Native Method)
> 2022-09-27T02:49:20.9513014Z Sep 27 02:49:20 at
> java.lang.Class.forName(Class.java:348)
> 2022-09-27T02:49:20.9513649Z Sep 27 02:49:20 at
> org.codehaus.janino.ClassLoaderIClassLoader.findIClass(ClassLoaderIClassLoader.java:89)
> 2022-09-27T02:49:20.9514339Z Sep 27 02:49:20 at
> org.codehaus.janino.IClassLoader.loadIClass(IClassLoader.java:312)
> 2022-09-27T02:49:20.9514990Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.findTypeByName(UnitCompiler.java:8556)
> 2022-09-27T02:49:20.9515659Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6749)
> 2022-09-27T02:49:20.9516337Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.getReferenceType(UnitCompiler.java:6594)
> 2022-09-27T02:49:20.9516989Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.getType2(UnitCompiler.java:6573)
> 2022-09-27T02:49:20.9517632Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler.access$13900(UnitCompiler.java:215)
> 2022-09-27T02:49:20.9518319Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22$1.visitReferenceType(UnitCompiler.java:6481)
> 2022-09-27T02:49:20.9519018Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22$1.visitReferenceType(UnitCompiler.java:6476)
> 2022-09-27T02:49:20.9519680Z Sep 27 02:49:20 at
> org.codehaus.janino.Java$ReferenceType.accept(Java.java:3928)
> 2022-09-27T02:49:20.9520386Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22.visitType(UnitCompiler.java:6476)
> 2022-09-27T02:49:20.9521042Z Sep 27 02:49:20 at
> org.codehaus.janino.UnitCompiler$22.visitType(UnitCompiler.java:6469)
> 2022-09-27T02:49:20.9521677Z Sep 27 02:49:20 at
>
[jira] [Comment Edited] (FLINK-29072) Migrate DDL-related schema handling to the new Schema framework
[ https://issues.apache.org/jira/browse/FLINK-29072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636404#comment-17636404 ] Yubin Li edited comment on FLINK-29072 at 11/21/22 3:21 AM: Hi, [~qingyue] [~luoyuxia], I have implemented the feature in https://issues.apache.org/jira/browse/FLINK-29679, migrate created table to new schema framework, so we can make all flink tables including hive connector to have new version schema, would bring a lot of benefits like get column comments, now ci has passed, would you please give a review? was (Author: liyubin117): Hi, [~qingyue] [~luoyuxia], I have implemented the feature in https://issues.apache.org/jira/browse/FLINK-29679, migrate created table to new schema framework, so we can make all flink tables including hive connector to have new version schema, would bring lots benefits like get column comments, now ci has passed, would you please give a review? > Migrate DDL-related schema handling to the new Schema framework > --- > > Key: FLINK-29072 > URL: https://issues.apache.org/jira/browse/FLINK-29072 > Project: Flink > Issue Type: Technical Debt > Components: Table SQL / API >Affects Versions: 1.16.0 >Reporter: Jane Chan >Priority: Major > > [FLIP-164|https://cwiki.apache.org/confluence/display/FLINK/FLIP-164%3A+Improve+Schema+Handling+in+Catalogs] > introduces the new Schema framework. Yet all DDL-related functionalities > (such as CREATE TABLE, CREATE TABLE LIKE, ALTER TABLE SET/RESET, ALTER TABLE > ADD/DROP CONSTRAINT, SHOW CREATE TABLE, etc.) are based on the deprecated > CatalogTableImpl and TableSchema. > The FLIP said, "For backward compatibility, we leave Catalog#createTable and > Catalog#alterTable untouched." Therefore, issues like FLINK-18958, > FLINK-28690, etc., cannot be resolved. And new functionalities like ALTER > TABLE ADD/MODIFY face a dilemma of implementing against the new framework but > lose backward consistency. > Fully migrating to the new framework takes a lot of effort and maybe a > long-term plan, but at least it's the time to put it on the agenda to have a > track. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-29072) Migrate DDL-related schema handling to the new Schema framework
[ https://issues.apache.org/jira/browse/FLINK-29072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636404#comment-17636404 ] Yubin Li edited comment on FLINK-29072 at 11/21/22 3:20 AM: Hi, [~qingyue] [~luoyuxia], I have implemented the feature in https://issues.apache.org/jira/browse/FLINK-29679, migrate created table to new schema framework, so we can make all flink tables including hive connector to have new version schema, would bring lots benefits like get column comments, now ci has passed, would you please give a review? was (Author: liyubin117): Hi, [~qingyue] [~luoyuxia], I have implemented the feature in https://issues.apache.org/jira/browse/FLINK-29679, it support all flink tables including hive connector, now ci has passed and column comment displayed, would you please give a review? > Migrate DDL-related schema handling to the new Schema framework > --- > > Key: FLINK-29072 > URL: https://issues.apache.org/jira/browse/FLINK-29072 > Project: Flink > Issue Type: Technical Debt > Components: Table SQL / API >Affects Versions: 1.16.0 >Reporter: Jane Chan >Priority: Major > > [FLIP-164|https://cwiki.apache.org/confluence/display/FLINK/FLIP-164%3A+Improve+Schema+Handling+in+Catalogs] > introduces the new Schema framework. Yet all DDL-related functionalities > (such as CREATE TABLE, CREATE TABLE LIKE, ALTER TABLE SET/RESET, ALTER TABLE > ADD/DROP CONSTRAINT, SHOW CREATE TABLE, etc.) are based on the deprecated > CatalogTableImpl and TableSchema. > The FLIP said, "For backward compatibility, we leave Catalog#createTable and > Catalog#alterTable untouched." Therefore, issues like FLINK-18958, > FLINK-28690, etc., cannot be resolved. And new functionalities like ALTER > TABLE ADD/MODIFY face a dilemma of implementing against the new framework but > lose backward consistency. > Fully migrating to the new framework takes a lot of effort and maybe a > long-term plan, but at least it's the time to put it on the agenda to have a > track. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30103) Test InputFormatCacheLoaderTest.checkCounter failed due to unexpected value on azure
Leonard Xu created FLINK-30103:
--
Summary: Test InputFormatCacheLoaderTest.checkCounter failed due
to unexpected value on azure
Key: FLINK-30103
URL: https://issues.apache.org/jira/browse/FLINK-30103
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Affects Versions: 1.16.0
Reporter: Leonard Xu
{noformat}
Nov 20 02:43:43 [ERROR] Failures:
Nov 20 02:43:43 [ERROR] InputFormatCacheLoaderTest.checkCounter:74
Nov 20 02:43:43 Expecting AtomicInteger(0) to have value:
Nov 20 02:43:43 0
Nov 20 02:43:43 but did not
{noformat}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43319=logs=de826397-1924-5900-0034-51895f69d4b7=f311e913-93a2-5a37-acab-4a63e1328f94
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[GitHub] [flink-table-store] JingsongLi closed pull request #390: [WIP] Extract BucketComputer
JingsongLi closed pull request #390: [WIP] Extract BucketComputer URL: https://github.com/apache/flink-table-store/pull/390 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-30102) Unstable test EventTimeWindowCheckpointingITCase.testPreAggregatedSlidingTimeWindow failed runs on azure
Leonard Xu created FLINK-30102:
--
Summary: Unstable test
EventTimeWindowCheckpointingITCase.testPreAggregatedSlidingTimeWindow failed
runs on azure
Key: FLINK-30102
URL: https://issues.apache.org/jira/browse/FLINK-30102
Project: Flink
Issue Type: Bug
Components: Runtime / Checkpointing, Tests
Affects Versions: 1.16.0
Reporter: Leonard Xu
{noformat}
Nov 20 06:26:32 [ERROR] Failures:
Nov 20 06:26:32 [ERROR]
EventTimeWindowCheckpointingITCase.testPreAggregatedSlidingTimeWindow
Nov 20 06:26:32 [INFO] Run 1: PASS
Nov 20 06:26:32 [ERROR] Run 2: Job execution failed.
Nov 20 06:26:32 [INFO] Run 3: PASS
Nov 20 06:26:32 [INFO] Run 4: PASS
Nov 20 06:26:32 [INFO] Run 5: PASS
Nov 20 06:26:32 [INFO] Run 6: PASS
Nov 20 06:26:32 [INFO] Run 7: PASS
Nov 20 06:26:32 [INFO] Run 8: PASS
Nov 20 06:26:32 [INFO] Run 9: PASS
Nov 20 06:26:32 [INFO] Run 10: PASS
{noformat}
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=43319=logs=4d4a0d10-fca2-5507-8eed-c07f0bdf4887=7b25afdf-cc6c-566f-5459-359dc2585798=10358
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-29614) Introduce Spark writer for table store
[ https://issues.apache.org/jira/browse/FLINK-29614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee updated FLINK-29614: - Fix Version/s: table-store-0.3.0 > Introduce Spark writer for table store > -- > > Key: FLINK-29614 > URL: https://issues.apache.org/jira/browse/FLINK-29614 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: table-store-0.3.0 > > > The main difficulty is that the Spark SourceV2 interface currently does not > support custom distribution, and the Table Store must have consistent > distribution. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29614) Introduce Spark writer for table store
[ https://issues.apache.org/jira/browse/FLINK-29614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-29614: --- Labels: pull-request-available (was: ) > Introduce Spark writer for table store > -- > > Key: FLINK-29614 > URL: https://issues.apache.org/jira/browse/FLINK-29614 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > > The main difficulty is that the Spark SourceV2 interface currently does not > support custom distribution, and the Table Store must have consistent > distribution. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink-table-store] JingsongLi opened a new pull request, #394: [FLINK-29614] Introduce Spark writer for table store
JingsongLi opened a new pull request, #394: URL: https://github.com/apache/flink-table-store/pull/394 - Supports spark batch write by v1 write. - Supports spark delete. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-table-store] JingsongLi commented on a diff in pull request #388: [FLINK-29232] File store continuous reading support from_timestamp scan mode
JingsongLi commented on code in PR #388:
URL: https://github.com/apache/flink-table-store/pull/388#discussion_r1027507379
##
flink-table-store-core/src/main/java/org/apache/flink/table/store/file/utils/SnapshotManager.java:
##
@@ -90,6 +90,22 @@ public boolean snapshotExists(long snapshotId) {
}
}
+public @Nullable Long earlierThanTimeMills(long timestampMills) {
+Long earliest = earliestSnapshotId();
+Long latest = latestSnapshotId();
+if (earliest == null || latest == null) {
+return null;
+}
+
+for (long i = latest; i >= earliest; i--) {
+long commitTime = snapshot(i).timeMillis();
+if (commitTime < timestampMills) {
+return i;
+}
+}
+return earliest - 1;
Review Comment:
The earliest snapshots here refer to those that have not expired.
earliest should be changed after expiration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-30101) Always use StandaloneClientHAServices to create RestClusterClient when retriving a Flink on YARN cluster client
[ https://issues.apache.org/jira/browse/FLINK-30101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636408#comment-17636408 ] Zhanghao Chen commented on FLINK-30101: --- [~xtsong] Looking forwarding to your opinions on the proposal~ Much thanks > Always use StandaloneClientHAServices to create RestClusterClient when > retriving a Flink on YARN cluster client > > > Key: FLINK-30101 > URL: https://issues.apache.org/jira/browse/FLINK-30101 > Project: Flink > Issue Type: Improvement > Components: Client / Job Submission >Affects Versions: 1.16.0 >Reporter: Zhanghao Chen >Priority: Major > Fix For: 1.17.0 > > > *Problem* > Currently, the procedure of retrieving a Flink on YARN cluster client is as > follows (in YarnClusterDescriptor#retrieve method): > # Get application report from YARN > # Set rest.address & rest.port using the info from application report > # Create a new RestClusterClient using the updated configuration, will use > client HA serivce to fetch the rest.address & rest.port if HA is enabled > Here, we can see that the usage of client HA in step 3 is redundant, as we've > already got the rest.address & rest.port from YARN application report. When > ZK HA is enabled, this would take ~1.5 s to initialize client HA services and > fetch the rest IP & port. > 1.5 s can mean a lot for latency-sensitive client operations. In my company, > we use Flink client to submit short-running session jobs and e2e latency is > critical. The job submission time is around 10 s on average, and 1.5s would > mean 15% of time saving. > *Proposal* > When retrieving a Flink on YARN cluster client, use > StandaloneClientHAServices to > create RestClusterClient instead as we have pre-fetched rest.address & > rest.port from YARN application report. This is also what we did in > KubernetesClusterDescriptor. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-30101) Always use StandaloneClientHAServices to create RestClusterClient when retriving a Flink on YARN cluster client
[ https://issues.apache.org/jira/browse/FLINK-30101?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhanghao Chen updated FLINK-30101: -- Summary: Always use StandaloneClientHAServices to create RestClusterClient when retriving a Flink on YARN cluster client (was: YARN client should ) > Always use StandaloneClientHAServices to create RestClusterClient when > retriving a Flink on YARN cluster client > > > Key: FLINK-30101 > URL: https://issues.apache.org/jira/browse/FLINK-30101 > Project: Flink > Issue Type: Improvement > Components: Client / Job Submission >Affects Versions: 1.16.0 >Reporter: Zhanghao Chen >Priority: Major > Fix For: 1.17.0 > > > *Problem* > Currently, the procedure of retrieving a Flink on YARN cluster client is as > follows (in YarnClusterDescriptor#retrieve method): > # Get application report from YARN > # Set rest.address & rest.port using the info from application report > # Create a new RestClusterClient using the updated configuration, will use > client HA serivce to fetch the rest.address & rest.port if HA is enabled > Here, we can see that the usage of client HA in step 3 is redundant, as we've > already got the rest.address & rest.port from YARN application report. When > ZK HA is enabled, this would take ~1.5 s to initialize client HA services and > fetch the rest IP & port. > 1.5 s can mean a lot for latency-sensitive client operations. In my company, > we use Flink client to submit short-running session jobs and e2e latency is > critical. The job submission time is around 10 s on average, and 1.5s would > mean 15% of time saving. > *Proposal* > When retrieving a Flink on YARN cluster client, use > StandaloneClientHAServices to > create RestClusterClient instead as we have pre-fetched rest.address & > rest.port from YARN application report. This is also what we did in > KubernetesClusterDescriptor. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-30101) YARN client should
Zhanghao Chen created FLINK-30101: - Summary: YARN client should Key: FLINK-30101 URL: https://issues.apache.org/jira/browse/FLINK-30101 Project: Flink Issue Type: Improvement Components: Client / Job Submission Affects Versions: 1.16.0 Reporter: Zhanghao Chen Fix For: 1.17.0 *Problem* Currently, the procedure of retrieving a Flink on YARN cluster client is as follows (in YarnClusterDescriptor#retrieve method): # Get application report from YARN # Set rest.address & rest.port using the info from application report # Create a new RestClusterClient using the updated configuration, will use client HA serivce to fetch the rest.address & rest.port if HA is enabled Here, we can see that the usage of client HA in step 3 is redundant, as we've already got the rest.address & rest.port from YARN application report. When ZK HA is enabled, this would take ~1.5 s to initialize client HA services and fetch the rest IP & port. 1.5 s can mean a lot for latency-sensitive client operations. In my company, we use Flink client to submit short-running session jobs and e2e latency is critical. The job submission time is around 10 s on average, and 1.5s would mean 15% of time saving. *Proposal* When retrieving a Flink on YARN cluster client, use StandaloneClientHAServices to create RestClusterClient instead as we have pre-fetched rest.address & rest.port from YARN application report. This is also what we did in KubernetesClusterDescriptor. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] XComp commented on a diff in pull request #20755: [FLINK-29092][tests] Use base path bucket assigner
XComp commented on code in PR #20755:
URL: https://github.com/apache/flink/pull/20755#discussion_r1027503224
##
flink-formats/flink-hadoop-bulk/src/test/java/org/apache/flink/formats/hadoop/bulk/HadoopPathBasedPartFileWriterTest.java:
##
@@ -82,13 +84,23 @@ public void testWriteFile() throws Exception {
DataStream stream =
env.addSource(new FiniteTestSource<>(data),
TypeInformation.of(String.class));
Configuration configuration = new Configuration();
-
+// Elements from source assign to one bucket , and produce two part
after checkpoint.
Review Comment:
Thanks for the clarification. In that case, we might want to move the second
part of the comment further up (see my [related comment
above](https://github.com/apache/flink/pull/20755#discussion_r1027502971)).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] XComp commented on a diff in pull request #20755: [FLINK-29092][tests] Use base path bucket assigner
XComp commented on code in PR #20755:
URL: https://github.com/apache/flink/pull/20755#discussion_r1027502971
##
flink-formats/flink-hadoop-bulk/src/test/java/org/apache/flink/formats/hadoop/bulk/HadoopPathBasedPartFileWriterTest.java:
##
@@ -82,13 +82,14 @@ public void testWriteFile() throws Exception {
DataStream stream =
env.addSource(new FiniteTestSource<>(data),
TypeInformation.of(String.class));
Review Comment:
```suggestion
// FiniteTestSource causes two output files to be generated
downstream
env.addSource(new FiniteTestSource<>(data),
TypeInformation.of(String.class));
```
we want to move the comment about generating two output files to the code
line that's actually causing this behavior.
##
flink-formats/flink-hadoop-bulk/src/test/java/org/apache/flink/formats/hadoop/bulk/HadoopPathBasedPartFileWriterTest.java:
##
@@ -82,13 +84,23 @@ public void testWriteFile() throws Exception {
DataStream stream =
env.addSource(new FiniteTestSource<>(data),
TypeInformation.of(String.class));
Configuration configuration = new Configuration();
-
+// Elements from source assign to one bucket , and produce two part
after checkpoint.
Review Comment:
Thanks for the clarification. In that case, we might want to move the second
part of the comment further up (see my related comment).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-29092) HadoopPathBasedPartFileWriterTest.testWriteFile failed with AssertionError
[
https://issues.apache.org/jira/browse/FLINK-29092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl updated FLINK-29092:
--
Affects Version/s: 1.15.2
> HadoopPathBasedPartFileWriterTest.testWriteFile failed with AssertionError
> --
>
> Key: FLINK-29092
> URL: https://issues.apache.org/jira/browse/FLINK-29092
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.16.0, 1.15.2
>Reporter: Xingbo Huang
>Assignee: KurtDing
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.17.0
>
> Attachments: image-2022-09-05-17-31-44-813.png
>
>
> {code:java}
> 2022-08-24T02:00:01.1670618Z Aug 24 02:00:01 [ERROR]
> org.apache.flink.formats.hadoop.bulk.HadoopPathBasedPartFileWriterTest.testWriteFile
> Time elapsed: 2.311 s <<< FAILURE!
> 2022-08-24T02:00:01.1671250Z Aug 24 02:00:01 java.lang.AssertionError:
> 2022-08-24T02:00:01.1671626Z Aug 24 02:00:01
> 2022-08-24T02:00:01.1672001Z Aug 24 02:00:01 Expected size: 1 but was: 2 in:
> 2022-08-24T02:00:01.1673656Z Aug 24 02:00:01
> [DeprecatedRawLocalFileStatus{path=file:/tmp/junit3893779198554813459/junit1595046776902782406/2022-08-24--02;
> isDirectory=true; modification_time=166130640;
> access_time=1661306400396; owner=; group=; permission=rwxrwxrwx;
> isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false},
> 2022-08-24T02:00:01.1676131Z Aug 24 02:00:01
> DeprecatedRawLocalFileStatus{path=file:/tmp/junit3893779198554813459/junit1595046776902782406/2022-08-24--01;
> isDirectory=true; modification_time=166130640;
> access_time=1661306400326; owner=; group=; permission=rwxrwxrwx;
> isSymlink=false; hasAcl=false; isEncrypted=false; isErasureCoded=false}]
> 2022-08-24T02:00:01.1677339Z Aug 24 02:00:01 at
> org.apache.flink.formats.hadoop.bulk.HadoopPathBasedPartFileWriterTest.validateResult(HadoopPathBasedPartFileWriterTest.java:107)
> 2022-08-24T02:00:01.1678274Z Aug 24 02:00:01 at
> org.apache.flink.formats.hadoop.bulk.HadoopPathBasedPartFileWriterTest.testWriteFile(HadoopPathBasedPartFileWriterTest.java:97)
> 2022-08-24T02:00:01.1679017Z Aug 24 02:00:01 at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2022-08-24T02:00:01.1679666Z Aug 24 02:00:01 at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2022-08-24T02:00:01.1680369Z Aug 24 02:00:01 at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2022-08-24T02:00:01.1681019Z Aug 24 02:00:01 at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2022-08-24T02:00:01.1681666Z Aug 24 02:00:01 at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:59)
> 2022-08-24T02:00:01.1682385Z Aug 24 02:00:01 at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2022-08-24T02:00:01.1683094Z Aug 24 02:00:01 at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:56)
> 2022-08-24T02:00:01.1683965Z Aug 24 02:00:01 at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2022-08-24T02:00:01.1684713Z Aug 24 02:00:01 at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> 2022-08-24T02:00:01.1685956Z Aug 24 02:00:01 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:299)
> 2022-08-24T02:00:01.1687185Z Aug 24 02:00:01 at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:293)
> 2022-08-24T02:00:01.1688347Z Aug 24 02:00:01 at
> java.util.concurrent.FutureTask.run(FutureTask.java:266)
> 2022-08-24T02:00:01.1689324Z Aug 24 02:00:01 at
> java.lang.Thread.run(Thread.java:748)
> 2022-08-24T02:00:01.1690062Z Aug 24 02:00:01
> 2022-08-24T02:00:03.4727706Z Aug 24 02:00:03 Formatting using clusterid:
> testClusterID
> 2022-08-24T02:00:07.9860626Z Aug 24 02:00:07 [INFO] Tests run: 12, Failures:
> 0, Errors: 0, Skipped: 0, Time elapsed: 13.712 s - in
> org.apache.flink.formats.hadoop.bulk.committer.HadoopRenameCommitterHDFSTest
> 2022-08-24T02:00:08.4139747Z Aug 24 02:00:08 [INFO]
> 2022-08-24T02:00:08.4140678Z Aug 24 02:00:08 [INFO] Results:
> 2022-08-24T02:00:08.4141326Z Aug 24 02:00:08 [INFO]
> 2022-08-24T02:00:08.4142008Z Aug 24 02:00:08 [ERROR] Failures:
> 2022-08-24T02:00:08.4144242Z Aug 24 02:00:08 [ERROR]
> HadoopPathBasedPartFileWriterTest.testWriteFile:97->validateResult:107
> 2022-08-24T02:00:08.4145317Z Aug 24 02:00:08 Expected size: 1 but was: 2 in:
> 2022-08-24T02:00:08.4147711Z Aug 24 02:00:08
>
[jira] [Updated] (FLINK-29679) Migrate to new schema framework & show column comment
[ https://issues.apache.org/jira/browse/FLINK-29679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yubin Li updated FLINK-29679: - Parent: FLINK-29072 Issue Type: Sub-task (was: New Feature) > Migrate to new schema framework & show column comment > - > > Key: FLINK-29679 > URL: https://issues.apache.org/jira/browse/FLINK-29679 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Affects Versions: 1.17.0 >Reporter: Yubin Li >Assignee: Yubin Li >Priority: Major > Labels: pull-request-available > > comment is very helpful to make table schema user-friendly, many data > analysers rely on such message to write sql adaptive to corresponding > business logics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-29072) Migrate DDL-related schema handling to the new Schema framework
[ https://issues.apache.org/jira/browse/FLINK-29072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636404#comment-17636404 ] Yubin Li commented on FLINK-29072: -- Hi, [~qingyue] [~luoyuxia], I have implemented the feature in https://issues.apache.org/jira/browse/FLINK-29679, it support all flink tables including hive connector, now ci has passed and column comment displayed, would you please give a review? > Migrate DDL-related schema handling to the new Schema framework > --- > > Key: FLINK-29072 > URL: https://issues.apache.org/jira/browse/FLINK-29072 > Project: Flink > Issue Type: Technical Debt > Components: Table SQL / API >Affects Versions: 1.16.0 >Reporter: Jane Chan >Priority: Major > > [FLIP-164|https://cwiki.apache.org/confluence/display/FLINK/FLIP-164%3A+Improve+Schema+Handling+in+Catalogs] > introduces the new Schema framework. Yet all DDL-related functionalities > (such as CREATE TABLE, CREATE TABLE LIKE, ALTER TABLE SET/RESET, ALTER TABLE > ADD/DROP CONSTRAINT, SHOW CREATE TABLE, etc.) are based on the deprecated > CatalogTableImpl and TableSchema. > The FLIP said, "For backward compatibility, we leave Catalog#createTable and > Catalog#alterTable untouched." Therefore, issues like FLINK-18958, > FLINK-28690, etc., cannot be resolved. And new functionalities like ALTER > TABLE ADD/MODIFY face a dilemma of implementing against the new framework but > lose backward consistency. > Fully migrating to the new framework takes a lot of effort and maybe a > long-term plan, but at least it's the time to put it on the agenda to have a > track. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-29679) Migrate to new schema framework & show column comment
[ https://issues.apache.org/jira/browse/FLINK-29679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yubin Li updated FLINK-29679: - Summary: Migrate to new schema framework & show column comment (was: DESCRIBE statement shows column comment) > Migrate to new schema framework & show column comment > - > > Key: FLINK-29679 > URL: https://issues.apache.org/jira/browse/FLINK-29679 > Project: Flink > Issue Type: New Feature > Components: Table SQL / API >Affects Versions: 1.17.0 >Reporter: Yubin Li >Assignee: Yubin Li >Priority: Major > Labels: pull-request-available > > comment is very helpful to make table schema user-friendly, many data > analysers rely on such message to write sql adaptive to corresponding > business logics. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-30093) [Flink SQL][Protobuf] CompileException when querying Kafka topic using google.protobuf.Timestamp
[
https://issues.apache.org/jira/browse/FLINK-30093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636400#comment-17636400
]
Suhan Mao commented on FLINK-30093:
---
[~rmetzger] Thanks for your findings!
I did not consider that user will use *google.protobuf.Timestamp* in the proto
file. So importing definition is not supported in current flink version, but it
definitely should be supported in the future.
`
> [Flink SQL][Protobuf] CompileException when querying Kafka topic using
> google.protobuf.Timestamp
> -
>
> Key: FLINK-30093
> URL: https://issues.apache.org/jira/browse/FLINK-30093
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Ecosystem
>Affects Versions: 1.16.0
> Environment: Mac OS Ventura
>Reporter: James Mcguire
>Priority: Major
> Attachments: taskmanager_172.22.0 (1).4_46291-40eec2_log
>
>
> I am encountering an issue when trying to use Flink SQL to query a Kafka
> topic that uses {{{}google.protobuf.Timestamp{}}}.
>
> When attempting to use Flink SQL to query a protobuf serialized Kafka topic
> that uses {{{}google.protobuf.Timestamp{}}}, a
> {{org.codehaus.commons.compiler.CompileException: Line 23, Column 5: Cannot
> determine simple type name "com" }}error occurs when trying to query the
> table.
>
> *Replication steps:*
> 1. Use a protobuf definition that contains a
> {{{}google.protobuf.Timestamp{}}}:
> {noformat}
> syntax = "proto3";
> package example.message;
> import "google/protobuf/timestamp.proto";
> option java_package = "com.example.message";
> option java_multiple_files = true;
> message Test {
> int64 id = 1;
> google.protobuf.Timestamp created_at = 5;
> }{noformat}
> 2. Use protobuf definition to produce message to topic
> 3. Confirm message is deserializable by protoc:
> {code:java}
> kcat -C -t development.example.message -b localhost:9092 -o -1 -e -q -D "" |
> protoc --decode=example.message.Test
> --proto_path=/Users/jamesmcguire/repos/flink-proto-example/schemas/
> example/message/test.proto
> id: 123
> created_at {
> seconds: 456
> nanos: 789
> }{code}
> 4. Create table in Flink SQL using kafka connector and protobuf format
> {code:java}
> CREATE TABLE tests (
> id BIGINT,
> created_at row
> )
> COMMENT ''
> WITH (
> 'connector' = 'kafka',
> 'format' = 'protobuf',
> 'protobuf.message-class-name' = 'com.example.message.Test',
> 'properties.auto.offset.reset' = 'earliest',
> 'properties.bootstrap.servers' = 'host.docker.internal:9092',
> 'properties.group.id' = 'test-1',
> 'topic' = 'development.example.message'
> );{code}
> 5. Run query in Flink SQL and encounter error:
> {code:java}
> Flink SQL> select * from tests;
> [ERROR] Could not execute SQL statement. Reason:
> org.codehaus.commons.compiler.CompileException: Line 23, Column 5: Cannot
> determine simple type name "com" {code}
> {*}NOTE{*}: If you repeat steps 4-5 without {{created_at row nanos INT>}} in the table, step 5 will complete successfully.
> 6. Observe in attached log file, Flink appears to be using the incorrect
> namespace (should be {{google.protobuf.Timestamp):}}
> {code:java}
> com.example.message.Timestamp message3 = message0.getCreatedAt(); {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-23035) Add explicit method to StateChangelogWriter to write metadata
[ https://issues.apache.org/jira/browse/FLINK-23035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636399#comment-17636399 ] Yanfei Lei commented on FLINK-23035: [~binh] How is it going? we intend to finish it in 1.17, the feature freezing date is set on *January 17th, 2023*. > Add explicit method to StateChangelogWriter to write metadata > - > > Key: FLINK-23035 > URL: https://issues.apache.org/jira/browse/FLINK-23035 > Project: Flink > Issue Type: Technical Debt > Components: Runtime / State Backends >Reporter: Roman Khachatryan >Assignee: Xinbin Huang >Priority: Minor > Fix For: 1.17.0 > > > Currently, metadata is written to the state changelog using the same > StateChangelogWriter.append() method as data. > However, it doesn't belong to a specific group, and should be read first on > recovery. Because of that, -1 is used. > An explicit append() without keygroup would be less fragile (probably still > using -1 under the hood). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[GitHub] [flink] wanglijie95 commented on pull request #20097: [FLINK-28284][Connectors/Jdbc] Add JdbcSink with new format
wanglijie95 commented on PR #20097: URL: https://github.com/apache/flink/pull/20097#issuecomment-1321372510 Thanks @eskabetxe for creating this PR. I have 3 questions about the changes: 1. Do we need to introduce `JdbcProducer/NonXaJdbcProducer`? I think the existing `JdbcOutputFormat` can cover all functionality of non-xa sink, can we reuse it? 2. Does the non-xa sink need to implement `StatefulSink` and `TwoPhaseCommittingSink`? I think the non-xa sink neither has state nor supports 2PC. 3. I don't see support for xa sink, is it not in the current plan? Some my thoughts: I think we can introduce 2 implementations: `JdbcXaSink` and `JdbcNonXaSink`. `JdbcXaSink` implements `StatefulSink` and `TwoPhaseCommittingSink`, the `JdbcNonXaSink` implements the based `org.apache.flink.api.connector.sink2.Sink`. These two implementations can then be exposed to the user via `.sink` and `.exactlyOnceSink`, just like the legacy sink in `org.apache.flink.connector.jdbc.JdbcSink`. WDYT? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-ml] yunfengzhou-hub commented on a diff in pull request #175: [FLINK-29602] Add Transformer for SQLTransformer
yunfengzhou-hub commented on code in PR #175:
URL: https://github.com/apache/flink-ml/pull/175#discussion_r1027464467
##
flink-ml-lib/src/main/java/org/apache/flink/ml/feature/sqltransformer/SQLTransformer.java:
##
@@ -0,0 +1,194 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.ml.feature.sqltransformer;
+
+import org.apache.flink.api.common.functions.AggregateFunction;
+import org.apache.flink.api.common.functions.FlatMapFunction;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.ml.api.Transformer;
+import org.apache.flink.ml.common.datastream.EndOfStreamWindows;
+import org.apache.flink.ml.param.Param;
+import org.apache.flink.ml.util.ParamUtils;
+import org.apache.flink.ml.util.ReadWriteUtils;
+import org.apache.flink.streaming.api.datastream.DataStream;
+import org.apache.flink.table.api.Schema;
+import org.apache.flink.table.api.Table;
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.table.api.bridge.java.StreamTableEnvironment;
+import org.apache.flink.table.api.internal.TableImpl;
+import org.apache.flink.types.Row;
+import org.apache.flink.types.RowKind;
+import org.apache.flink.util.Collector;
+import org.apache.flink.util.Preconditions;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * SQLTransformer implements the transformations that are defined by SQL
statement.
+ *
+ * Currently we only support SQL syntax like `SELECT ... FROM __THIS__ ...`
where `__THIS__`
+ * represents the input table and cannot be modified.
+ *
+ * The select clause specifies the fields, constants, and expressions to
display in the output.
+ * Except the cases described in the note section below, it can be any select
clause that Flink SQL
+ * supports. Users can also use Flink SQL built-in function and UDFs to
operate on these selected
+ * columns.
+ *
+ * For example, SQLTransformer supports statements like:
+ *
+ *
+ * `SELECT a, a + b AS a_b FROM __THIS__`
+ * `SELECT a, SQRT(b) AS b_sqrt FROM __THIS__ where a > 5`
+ * `SELECT a, b, SUM(c) AS c_sum FROM __THIS__ GROUP BY a, b`
+ *
+ *
+ * Note: This operator only generates append-only/insert-only table as its
output. If the output
+ * table could possibly contain retract messages(e.g. perform `SELECT ... FROM
__THIS__ GROUP BY
+ * ...` operation on a table in streaming mode), this operator would aggregate
all changelogs and
+ * only output the final state. The records in the final state would be output
in the order they
+ * were last modified.
+ */
+public class SQLTransformer
+implements Transformer,
SQLTransformerParams {
+static final String TABLE_IDENTIFIER = "__THIS__";
+
+private static final String INSERT_ONLY_EXCEPTION_PATTERN =
+"^.* doesn't support consuming .* changes which is produced by
node .*$";
+
+private final Map, Object> paramMap = new HashMap<>();
+
+public SQLTransformer() {
+ParamUtils.initializeMapWithDefaultValues(paramMap, this);
+}
+
+@Override
+public Table[] transform(Table... inputs) {
+Preconditions.checkArgument(inputs.length == 1);
+StreamTableEnvironment tEnv =
+(StreamTableEnvironment) ((TableImpl)
inputs[0]).getTableEnvironment();
+String statement = getStatement().replace(TABLE_IDENTIFIER,
inputs[0].toString());
+
+Table outputTable = tEnv.sqlQuery(statement);
+
+if (!isInsertOnlyTable(tEnv, outputTable)) {
+Schema schema =
+
Schema.newBuilder().fromResolvedSchema(outputTable.getResolvedSchema()).build();
+DataStream outputStream = tEnv.toChangelogStream(outputTable,
schema);
+
+outputStream =
+outputStream
+.windowAll(EndOfStreamWindows.get())
+.aggregate(
+new ChangeLogStreamToDataStreamFunction(),
+Types.LIST(outputStream.getType()),
+Types.LIST(outputStream.getType()))
+
[jira] [Closed] (FLINK-27874) KeyedStateCheckpointingITCase Timeout
[
https://issues.apache.org/jira/browse/FLINK-27874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yanfei Lei closed FLINK-27874.
--
Resolution: Cannot Reproduce
> KeyedStateCheckpointingITCase Timeout
> -
>
> Key: FLINK-27874
> URL: https://issues.apache.org/jira/browse/FLINK-27874
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.16.0
>Reporter: Shengkai Fang
>Priority: Major
>
> The test fails in the CI.
>
> {code:java}
> "main" #1 prio=5 os_prio=0 tid=0x7f216400b800 nid=0x21fb waiting on
> condition [0x7f216c07f000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x82d016d8> (a
> java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1989)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1951)
> at
> org.apache.flink.test.checkpointing.KeyedStateCheckpointingITCase.testProgramWithBackend(KeyedStateCheckpointingITCase.java:175)
> at
> org.apache.flink.test.checkpointing.KeyedStateCheckpointingITCase.testWithMemoryBackendSync(KeyedStateCheckpointingITCase.java:104)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=36200=logs=38d6b56a-d502-56fb-7b73-c09f8fe7becd=6e6509fa-8a5d-5a6c-e17e-64f5ecc17842=13690
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-27874) KeyedStateCheckpointingITCase Timeout
[
https://issues.apache.org/jira/browse/FLINK-27874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636392#comment-17636392
]
Yanfei Lei commented on FLINK-27874:
Since this hasn't been reproduced in 5 months, I'm going to close this ticket,
please reopen it if the bug reproduces again.
> KeyedStateCheckpointingITCase Timeout
> -
>
> Key: FLINK-27874
> URL: https://issues.apache.org/jira/browse/FLINK-27874
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.16.0
>Reporter: Shengkai Fang
>Priority: Major
>
> The test fails in the CI.
>
> {code:java}
> "main" #1 prio=5 os_prio=0 tid=0x7f216400b800 nid=0x21fb waiting on
> condition [0x7f216c07f000]
>java.lang.Thread.State: WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0x82d016d8> (a
> java.util.concurrent.CompletableFuture$Signaller)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> at
> java.util.concurrent.CompletableFuture$Signaller.block(CompletableFuture.java:1707)
> at
> java.util.concurrent.ForkJoinPool.managedBlock(ForkJoinPool.java:3323)
> at
> java.util.concurrent.CompletableFuture.waitingGet(CompletableFuture.java:1742)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1989)
> at
> org.apache.flink.streaming.api.environment.StreamExecutionEnvironment.execute(StreamExecutionEnvironment.java:1951)
> at
> org.apache.flink.test.checkpointing.KeyedStateCheckpointingITCase.testProgramWithBackend(KeyedStateCheckpointingITCase.java:175)
> at
> org.apache.flink.test.checkpointing.KeyedStateCheckpointingITCase.testWithMemoryBackendSync(KeyedStateCheckpointingITCase.java:104)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=36200=logs=38d6b56a-d502-56fb-7b73-c09f8fe7becd=6e6509fa-8a5d-5a6c-e17e-64f5ecc17842=13690
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Closed] (FLINK-28010) Use deleteRange to optimize the clear operation of RocksDBMapState.
[
https://issues.apache.org/jira/browse/FLINK-28010?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yanfei Lei closed FLINK-28010.
--
Resolution: Won't Fix
> Use deleteRange to optimize the clear operation of RocksDBMapState.
> ---
>
> Key: FLINK-28010
> URL: https://issues.apache.org/jira/browse/FLINK-28010
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: ming li
>Assignee: ming li
>Priority: Major
> Labels: pull-request-available
>
> [FLINK-21321|https://issues.apache.org/jira/browse/FLINK-21321] has
> introduced {{deleteRange}} for fast clipping of Incremental checkpoint, so
> can the {{clear}} method in {{RocksDBMapState}} be replaced with
> {{{}deleteRange{}}}?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-28010) Use deleteRange to optimize the clear operation of RocksDBMapState.
[
https://issues.apache.org/jira/browse/FLINK-28010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636391#comment-17636391
]
Yanfei Lei commented on FLINK-28010:
Because the experimental results of
[iterator|https://github.com/apache/flink/pull/20405#discussion_r951119238] are
not as expected, as [~Ming Li] said, "maybe this is not a good optimization, we
moved the cost of delete with Iterator to read, and this cost will persist
unless a compaction occurs". I am going to close this ticket, please reopen
this if there are some other optimizations.
> Use deleteRange to optimize the clear operation of RocksDBMapState.
> ---
>
> Key: FLINK-28010
> URL: https://issues.apache.org/jira/browse/FLINK-28010
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: ming li
>Assignee: ming li
>Priority: Major
> Labels: pull-request-available
>
> [FLINK-21321|https://issues.apache.org/jira/browse/FLINK-21321] has
> introduced {{deleteRange}} for fast clipping of Incremental checkpoint, so
> can the {{clear}} method in {{RocksDBMapState}} be replaced with
> {{{}deleteRange{}}}?
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-28695) Fail to send partition request to restarted taskmanager
[
https://issues.apache.org/jira/browse/FLINK-28695?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17636390#comment-17636390
]
nobleyd commented on FLINK-28695:
-
[~pnowojski] Hi, we have the same problem, and adjusting
`taskmanager.network.max-num-tcp-connections` words fine.
> Fail to send partition request to restarted taskmanager
> ---
>
> Key: FLINK-28695
> URL: https://issues.apache.org/jira/browse/FLINK-28695
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, Runtime / Network
>Affects Versions: 1.15.0, 1.15.1
>Reporter: Simonas
>Assignee: Rui Fan
>Priority: Critical
> Labels: pull-request-available
> Attachments: deployment.txt, image-2022-11-20-16-16-45-705.png,
> job_log.txt, jobmanager_config.txt, jobmanager_logs.txt, pod_restart.txt,
> taskmanager_config.txt
>
>
> After upgrade to *1.15.1* we started getting error while running JOB
>
> {code:java}
> org.apache.flink.runtime.io.network.netty.exception.LocalTransportException:
> Sending the partition request to '/XXX.XXX.XX.32:6121 (#0)' failed.at
> org.apache.flink.runtime.io.network.netty.NettyPartitionRequestClient$1.operationComplete(NettyPartitionRequestClient.java:145)
> {code}
> {code:java}
> Caused by:
> org.apache.flink.shaded.netty4.io.netty.channel.StacklessClosedChannelException
>
> atrg.apache.flink.shaded.netty4.io.netty.channel.AbstractChannel$AbstractUnsafe.write(Object,
> ChannelPromise)(Unknown Source){code}
> After investigation we managed narrow it down to the exact behavior then this
> issue happens:
> # Deploying JOB on fresh kubernetes session cluster with multiple
> TaskManagers: TM1 and TM2 is successful. Job has multiple partitions running
> on both TM1 and TM2.
> # One TaskManager TM2 (XXX.XXX.XX.32) fails for unrelated issue. For example
> OOM exception.
> # Kubernetes POD with mentioned TaskManager TM2 is restarted. POD retains
> same IP address as before.
> # JobManager is able to pickup the restarted TM2 (XXX.XXX.XX.32)
> # JOB is restarted because it was running on the failed TaskManager TM2
> # TM1 data channel to TM2 is closed and we get LocalTransportException:
> Sending the partition request to '/XXX.XXX.XX.32:6121 (#0)' failed during JOB
> running stage.
> # When we explicitly delete pod with TM2 it creates new POD with different
> IP address and JOB is able to start again.
> Important to note that we didn't encountered this issue with previous
> *1.14.4* version and TaskManager restarts didn't cause such error.
> Please note attached kubernetes deployments and reduced logs from JobManager.
> TaskManager logs did show errors before error, but doesn't show anything
> significant after restart.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (FLINK-29614) Introduce Spark writer for table store
[ https://issues.apache.org/jira/browse/FLINK-29614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee reassigned FLINK-29614: Assignee: Jingsong Lee > Introduce Spark writer for table store > -- > > Key: FLINK-29614 > URL: https://issues.apache.org/jira/browse/FLINK-29614 > Project: Flink > Issue Type: New Feature > Components: Table Store >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > > The main difficulty is that the Spark SourceV2 interface currently does not > support custom distribution, and the Table Store must have consistent > distribution. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Closed] (FLINK-30094) Flink Table Sotre's Lookup Join document may needs left table DDL example.
[ https://issues.apache.org/jira/browse/FLINK-30094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-30094. Fix Version/s: table-store-0.3.0 Assignee: Hang HOU Resolution: Fixed master: fe780cf70aea008fead49a14ac0b71cd403db97b > Flink Table Sotre's Lookup Join document may needs left table DDL example. > -- > > Key: FLINK-30094 > URL: https://issues.apache.org/jira/browse/FLINK-30094 > Project: Flink > Issue Type: Improvement > Components: Table Store >Affects Versions: 1.16.0 >Reporter: Hang HOU >Assignee: Hang HOU >Priority: Minor > Labels: pull-request-available > Fix For: table-store-0.3.0 > > > In this “Lookup Join” character, official document don't give a DDL example > about the left table. Maybe some first time users who using lookup join with > right table which in table store will get some confusions about "Processing > Time Temporal Join". -- This message was sent by Atlassian Jira (v8.20.10#820010)
