Re: [PR] [FLINK-33792] Generate the same code for the same logic [flink]
flinkbot commented on PR #23984: URL: https://github.com/apache/flink/pull/23984#issuecomment-1867349736 ## CI report: * d996c020cea9582b5df989f4eb277efb395931fb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33217) Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array
[
https://issues.apache.org/jira/browse/FLINK-33217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799692#comment-17799692
]
Jane Chan commented on FLINK-33217:
---
Sorry for being late.
Hi, [~xuyangzhong], thanks for the exploration. While you mentioned that
{quote}Because the LEFT join, the type row is forced nullable, and the row type
is changes to `ROW(VARCHAR not null)` by
`FlinkTypeFactory#createTypeWithNullability` that overrides its super class.
And the diff about nullable cases this failure.
{quote}
Actually, the desired nullability change for the type`ROW(VARCHAR not null) not
null` after a LEFT OUTER JOIN should be `ROW(VARCHAR)`, rather than
`ROW(VARCHAR not null)`.
> Flink SQL: UNNEST fails with on LEFT JOIN with NOT NULL type in array
> -
>
> Key: FLINK-33217
> URL: https://issues.apache.org/jira/browse/FLINK-33217
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.15.3, 1.18.0, 1.19.0
>Reporter: Robert Metzger
>Priority: Major
> Attachments: UnnestNullErrorTest.scala
>
>
> Steps to reproduce:
> Take a column of type
> {code:java}
> business_data ARRAY
> {code}
> Take this query
> {code:java}
> select bd_name from reproduce_unnest LEFT JOIN
> UNNEST(reproduce_unnest.business_data) AS exploded_bd(bd_name) ON true
> {code}
> And get this error
> {code:java}
> Caused by: java.lang.AssertionError: Type mismatch:
> rowtype of rel before registration: RecordType(VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" NOT NULL ARRAY business_data, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" bd_name) NOT NULL
> rowtype of rel after registration: RecordType(VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" NOT NULL ARRAY business_data, VARCHAR(2147483647) CHARACTER
> SET "UTF-16LE" NOT NULL f0) NOT NULL
> Difference:
> bd_name: VARCHAR(2147483647) CHARACTER SET "UTF-16LE" -> VARCHAR(2147483647)
> CHARACTER SET "UTF-16LE" NOT NULL
> at org.apache.calcite.util.Litmus$1.fail(Litmus.java:32)
> at org.apache.calcite.plan.RelOptUtil.equal(RelOptUtil.java:2206)
> at
> org.apache.calcite.rel.AbstractRelNode.onRegister(AbstractRelNode.java:275)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.registerImpl(VolcanoPlanner.java:1270)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.register(VolcanoPlanner.java:598)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.ensureRegistered(VolcanoPlanner.java:613)
> at
> org.apache.calcite.plan.volcano.VolcanoPlanner.changeTraits(VolcanoPlanner.java:498)
> at
> org.apache.calcite.tools.Programs$RuleSetProgram.run(Programs.java:315)
> at
> org.apache.flink.table.planner.plan.optimize.program.FlinkVolcanoProgram.optimize(FlinkVolcanoProgram.scala:62)
> {code}
> I have implemented a small test case, which fails against Flink 1.15, 1.8 and
> the latest master branch.
> Workarounds:
> 1. Drop "NOT NULL" in array type
> 2. Drop "LEFT" from "LEFT JOIN".
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33927][table] Cleanup usage of deprecated ExecutionConfigOptions#(TABLE_EXEC_SHUFFLE_MODE, TABLE_EXEC_LEGACY_TRANSFORMATION_UIDS) [flink]
flinkbot commented on PR #23983: URL: https://github.com/apache/flink/pull/23983#issuecomment-1867344725 ## CI report: * d8230c9858c38019428e42bc8a7ee411fdb2d345 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33792) Generate the same code for the same logic
[ https://issues.apache.org/jira/browse/FLINK-33792?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33792: --- Labels: pull-request-available (was: ) > Generate the same code for the same logic > - > > Key: FLINK-33792 > URL: https://issues.apache.org/jira/browse/FLINK-33792 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Dan Zou >Assignee: Dan Zou >Priority: Major > Labels: pull-request-available > > Generate the same code for the same logic, so that we may reuse the generated > code between different jobs. This is the precondition for FLINK-28691. The > current issue is we use a self-incrementing counter in CodeGenUtils#newName, > it means we could not get the same generated class between two queries even > when they are exactly the same. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] [FLINK-33792] Generate the same code for the same logic [flink]
zoudan opened a new pull request, #23984: URL: https://github.com/apache/flink/pull/23984 ## What is the purpose of the change This pull request ensure that we generate the same code for the same logic, it is a precondition for sharing generated classes between different jobs. ## Brief change log - add a name counter in each `CodeGeneratorContext` and use it when we generate names for variables. ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing This change added tests and can be verified as follows: - Add a new test: CodeGenUtilsTest#testNewName - Add tests in existing class: HashCodeGeneratorTest#testHashWithIndependentNameCounter and ProjectionCodeGeneratorTest#testHashWithIndependentNameCounter ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (docs) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-33927][table] Cleanup usage of deprecated ExecutionConfigOptions#(TABLE_EXEC_SHUFFLE_MODE, TABLE_EXEC_LEGACY_TRANSFORMATION_UIDS) [flink]
liyubin117 opened a new pull request, #23983: URL: https://github.com/apache/flink/pull/23983 ## What is the purpose of the change Cleanup the usage of deprecated org.apache.flink.table.api.config.ExecutionConfigOptions#TABLE_EXEC_SHUFFLE_MODE, org.apache.flink.table.api.config.ExecutionConfigOptions#TABLE_EXEC_LEGACY_TRANSFORMATION_UIDS. ## Brief change log cleanup deprecated usage. ## Verifying this change no need additional verify besides CI ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33927) Cleanup usage of deprecated ExecutionConfigOptions#(TABLE_EXEC_SHUFFLE_MODE, TABLE_EXEC_LEGACY_TRANSFORMATION_UIDS)
[ https://issues.apache.org/jira/browse/FLINK-33927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33927: --- Labels: pull-request-available (was: ) > Cleanup usage of deprecated ExecutionConfigOptions#(TABLE_EXEC_SHUFFLE_MODE, > TABLE_EXEC_LEGACY_TRANSFORMATION_UIDS) > --- > > Key: FLINK-33927 > URL: https://issues.apache.org/jira/browse/FLINK-33927 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.19.0 >Reporter: Yubin Li >Priority: Major > Labels: pull-request-available > > |org.apache.flink.table.api.config.ExecutionConfigOptions#TABLE_EXEC_SHUFFLE_MODE| > |org.apache.flink.table.api.config.ExecutionConfigOptions#TABLE_EXEC_LEGACY_TRANSFORMATION_UIDS| -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33490) Validate the name conflicts when creating view
[
https://issues.apache.org/jira/browse/FLINK-33490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799688#comment-17799688
]
Benchao Li commented on FLINK-33490:
[~xuyangzhong] No, I mean
{code:SQL}
CREATE VIEW MyView(col1, col2) AS -- Note that, I've specified the column names
to the view, hence it should not throw exception even if the query's name
conflicts.
SELECT a, a
FROM T
{code}
> Validate the name conflicts when creating view
> --
>
> Key: FLINK-33490
> URL: https://issues.apache.org/jira/browse/FLINK-33490
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Shengkai Fang
>Assignee: xuyang
>Priority: Major
> Labels: pull-request-available
>
> We should forbid
> ```
> CREATE VIEW id_view AS
> SELECT id, uid AS id FROM id_table
> ```
> As the SQL standards states,
> If is specified, then:
> i) If any two columns in the table specified by the have
> equivalent s, or if any column of that table has an
> implementation-dependent name, then a shall be specified.
> ii) Equivalent s shall not be specified more than once in the
> .
> Many databases also throw exception when view name conflicts, e.g. mysql,
> postgres.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-33917) IllegalArgumentException: hostname can't be null
[
https://issues.apache.org/jira/browse/FLINK-33917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799686#comment-17799686
]
Tom commented on FLINK-33917:
-
I've also raised this as a JDK bug in the mean time
https://bugs.java.com/bugdatabase/view_bug?bug_id=JDK-8322667
> IllegalArgumentException: hostname can't be null
>
>
> Key: FLINK-33917
> URL: https://issues.apache.org/jira/browse/FLINK-33917
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Tom
>Priority: Major
> Labels: pull-request-available
>
> In certain scenarios, if the hostname contains certain characters it will
> throw an exception when it tries to initialize the `InetSocketAddress`
>
> {code:java}
> java.lang.IllegalArgumentException: hostname can't be null at
> java.base/java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
> at
> java.base/java.net.InetSocketAddress.(InetSocketAddress.java:216) {code}
>
> [https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L236]
>
> {code:java}
> @Override
> public boolean isJobManagerPortReady(Configuration config) {
> final URI uri;
> try (var clusterClient = getClusterClient(config)) {
> uri = URI.create(clusterClient.getWebInterfaceURL());
> } catch (Exception ex) {
> throw new FlinkRuntimeException(ex);
> }
> SocketAddress socketAddress = new InetSocketAddress(uri.getHost(),
> uri.getPort());
> Socket socket = new Socket();
> try {
> socket.connect(socketAddress, 1000);
> socket.close();
> return true;
> } catch (IOException e) {
> return false;
> }
> }
> {code}
>
> Here's a simple test to reproduce
>
> URL
> {code:java}
> @ParameterizedTest
> @ValueSource(
> strings = {"http://127.0.0.1:8081;, "http://123-dev:8081;,
> "http://dev-test.abc:8081;, "http://dev-test.1a:8081;,
> "http://dev-test.abc01:8081"})
> void testURLAddresses(String inputAddress) {
> assertDoesNotThrow(
> () -> {
> final URL url = new URL(inputAddress);
> new InetSocketAddress(url.getHost(), url.getPort());
> });
> } {code}
>
> URI
>
> {code:java}
> @ParameterizedTest
> @ValueSource(
> strings = {"http://127.0.0.1:8081;, "http://123-dev:8081;,
> "http://dev-test.abc:8081;, "http://dev-test.1a:8081;,
> "http://dev-test.abc01:8081"})
> void testURIAddresses(String inputAddress) {
> assertDoesNotThrow(
> () -> {
> final URI uri = new URI(inputAddress);
> new InetSocketAddress(uri.getHost(), uri.getPort());
> });
> } {code}
>
> All test cases past except for "http://dev-test.1a:8081; which is a valid
> flink host url, but not a valid URI
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
flashJd commented on code in PR #728:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/728#discussion_r1434780474
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##
@@ -264,4 +262,21 @@ private static Map
getVertexParallelismOverrides(
});
return overrides;
}
+
+private boolean blockScalingExecution(
+Context context,
+Map scalingSummaries,
+Configuration conf,
+Instant now) {
+var scaleEnabled = conf.get(SCALING_ENABLED);
+var isExcluded = CalendarUtils.inExcludedPeriods(conf, now);
+autoScalerEventHandler.handleScalingEvent(
+context,
+scalingSummaries,
+scaleEnabled,
+isExcluded,
+conf.get(SCALING_EVENT_INTERVAL));
+
+ return !scaleEnabled || isExcluded;
Review Comment:
Thanks for the review. A small question, what do you think of the frequent
log printing, e.g. `Stabilizing until ...` and `Metric window not full until
...`, is it worth to put this frequent log to autoscale event handler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
flashJd commented on code in PR #728:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/728#discussion_r1434780474
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##
@@ -264,4 +262,21 @@ private static Map
getVertexParallelismOverrides(
});
return overrides;
}
+
+private boolean blockScalingExecution(
+Context context,
+Map scalingSummaries,
+Configuration conf,
+Instant now) {
+var scaleEnabled = conf.get(SCALING_ENABLED);
+var isExcluded = CalendarUtils.inExcludedPeriods(conf, now);
+autoScalerEventHandler.handleScalingEvent(
+context,
+scalingSummaries,
+scaleEnabled,
+isExcluded,
+conf.get(SCALING_EVENT_INTERVAL));
+
+ return !scaleEnabled || isExcluded;
Review Comment:
Thanks for the review. A small question, what do you think of the frequent
log printing, `Stabilizing until ...` and `Metric window not full until ...`,
is it worth to put this frequent log to autoscale event handler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
flashJd commented on code in PR #728:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/728#discussion_r1434780474
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##
@@ -264,4 +262,21 @@ private static Map
getVertexParallelismOverrides(
});
return overrides;
}
+
+private boolean blockScalingExecution(
+Context context,
+Map scalingSummaries,
+Configuration conf,
+Instant now) {
+var scaleEnabled = conf.get(SCALING_ENABLED);
+var isExcluded = CalendarUtils.inExcludedPeriods(conf, now);
+autoScalerEventHandler.handleScalingEvent(
+context,
+scalingSummaries,
+scaleEnabled,
+isExcluded,
+conf.get(SCALING_EVENT_INTERVAL));
+
+ return !scaleEnabled || isExcluded;
Review Comment:
Thanks for the review. A small question, what do you think of the frequent
log printing, `Stabilizing until ...` and `Metric window not full until ...`,
is it worth to put some frequent log to autoscale event handler.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-33927) Cleanup usage of deprecated ExecutionConfigOptions#(TABLE_EXEC_SHUFFLE_MODE, TABLE_EXEC_LEGACY_TRANSFORMATION_UIDS)
[ https://issues.apache.org/jira/browse/FLINK-33927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yubin Li updated FLINK-33927: - Description: |org.apache.flink.table.api.config.ExecutionConfigOptions#TABLE_EXEC_SHUFFLE_MODE| |org.apache.flink.table.api.config.ExecutionConfigOptions#TABLE_EXEC_LEGACY_TRANSFORMATION_UIDS| was: |org.apache.flink.table.api.config.ExecutionConfigOptions#TABLE_EXEC_SHUFFLE_MODE|Use ExecutionOptions#BATCH_SHUFFLE_MODE instead| |org.apache.flink.table.api.config.ExecutionConfigOptions#TABLE_EXEC_LEGACY_TRANSFORMATION_UIDS|Use #TABLE_EXEC_UID_GENERATION instead| > Cleanup usage of deprecated ExecutionConfigOptions#(TABLE_EXEC_SHUFFLE_MODE, > TABLE_EXEC_LEGACY_TRANSFORMATION_UIDS) > --- > > Key: FLINK-33927 > URL: https://issues.apache.org/jira/browse/FLINK-33927 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Affects Versions: 1.19.0 >Reporter: Yubin Li >Priority: Major > > |org.apache.flink.table.api.config.ExecutionConfigOptions#TABLE_EXEC_SHUFFLE_MODE| > |org.apache.flink.table.api.config.ExecutionConfigOptions#TABLE_EXEC_LEGACY_TRANSFORMATION_UIDS| -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (FLINK-33927) Cleanup usage of deprecated ExecutionConfigOptions#(TABLE_EXEC_SHUFFLE_MODE, TABLE_EXEC_LEGACY_TRANSFORMATION_UIDS)
Yubin Li created FLINK-33927: Summary: Cleanup usage of deprecated ExecutionConfigOptions#(TABLE_EXEC_SHUFFLE_MODE, TABLE_EXEC_LEGACY_TRANSFORMATION_UIDS) Key: FLINK-33927 URL: https://issues.apache.org/jira/browse/FLINK-33927 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Affects Versions: 1.19.0 Reporter: Yubin Li |org.apache.flink.table.api.config.ExecutionConfigOptions#TABLE_EXEC_SHUFFLE_MODE|Use ExecutionOptions#BATCH_SHUFFLE_MODE instead| |org.apache.flink.table.api.config.ExecutionConfigOptions#TABLE_EXEC_LEGACY_TRANSFORMATION_UIDS|Use #TABLE_EXEC_UID_GENERATION instead| -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
flashJd commented on code in PR #728:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/728#discussion_r1434766546
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/event/AutoScalerEventHandler.java:
##
@@ -83,12 +92,27 @@ default void handleScalingEvent(
}
static String scalingReport(
-Map scalingSummaries, boolean
scalingEnabled) {
-StringBuilder sb =
-new StringBuilder(
-scalingEnabled
-? SCALING_SUMMARY_HEADER_SCALING_ENABLED
-: SCALING_SUMMARY_HEADER_SCALING_DISABLED);
+Map scalingSummaries,
+boolean scalingEnabled,
+boolean isExcluded,
+Configuration config) {
+StringBuilder sb = new StringBuilder();
+if (!scalingEnabled) {
+sb.append(
+String.format(
+SCALING_SUMMARY_HEADER_SCALING_EXECUTION_DISABLED,
+SCALING_ENABLED.key(),
+false));
+} else if (isExcluded) {
+sb.append(
+String.format(
+SCALING_SUMMARY_HEADER_SCALING_EXECUTION_DISABLED,
+EXCLUDED_PERIODS.key(),
+config.get(EXCLUDED_PERIODS)));
+} else {
+sb.append(SCALING_SUMMARY_HEADER_SCALING_EXECUTION_ENABLED);
+}
Review Comment:
Modify the parameter of method `handleScalingEvent` to replace the
scalingEnabled and scalingBlocked flags via a generic String
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-33490) Validate the name conflicts when creating view
[
https://issues.apache.org/jira/browse/FLINK-33490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799679#comment-17799679
]
xuyang commented on FLINK-33490:
Hi, [~libenchao] .IIUC, do you mean we need to support the following SQL?
{code:java}
create view MyView as select t1.a, t2.a from SrcTable t1 join SrcTable t2 on
t1.b = t2.b {code}
If I understand it wrong, can you give an example?
> Validate the name conflicts when creating view
> --
>
> Key: FLINK-33490
> URL: https://issues.apache.org/jira/browse/FLINK-33490
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Shengkai Fang
>Assignee: xuyang
>Priority: Major
> Labels: pull-request-available
>
> We should forbid
> ```
> CREATE VIEW id_view AS
> SELECT id, uid AS id FROM id_table
> ```
> As the SQL standards states,
> If is specified, then:
> i) If any two columns in the table specified by the have
> equivalent s, or if any column of that table has an
> implementation-dependent name, then a shall be specified.
> ii) Equivalent s shall not be specified more than once in the
> .
> Many databases also throw exception when view name conflicts, e.g. mysql,
> postgres.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
flashJd commented on code in PR #728: URL: https://github.com/apache/flink-kubernetes-operator/pull/728#discussion_r1434764455 ## flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java: ## @@ -264,4 +262,21 @@ private static Map getVertexParallelismOverrides( }); return overrides; } + +private boolean blockScalingExecution( Review Comment: Renamed to `checkIfBlockedAndTriggerScalingEvent` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33917) IllegalArgumentException: hostname can't be null
[
https://issues.apache.org/jira/browse/FLINK-33917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom updated FLINK-33917:
Description:
In certain scenarios, if the hostname contains certain characters it will throw
an exception when it tries to initialize the `InetSocketAddress`
{code:java}
java.lang.IllegalArgumentException: hostname can't be null at
java.base/java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
at java.base/java.net.InetSocketAddress.(InetSocketAddress.java:216)
{code}
[https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L236]
{code:java}
@Override
public boolean isJobManagerPortReady(Configuration config) {
final URI uri;
try (var clusterClient = getClusterClient(config)) {
uri = URI.create(clusterClient.getWebInterfaceURL());
} catch (Exception ex) {
throw new FlinkRuntimeException(ex);
}
SocketAddress socketAddress = new InetSocketAddress(uri.getHost(),
uri.getPort());
Socket socket = new Socket();
try {
socket.connect(socketAddress, 1000);
socket.close();
return true;
} catch (IOException e) {
return false;
}
}
{code}
Here's a simple test to reproduce
URL
{code:java}
@ParameterizedTest
@ValueSource(
strings = {"http://127.0.0.1:8081;, "http://123-dev:8081;,
"http://dev-test.abc:8081;, "http://dev-test.1a:8081;,
"http://dev-test.abc01:8081"})
void testURLAddresses(String inputAddress) {
assertDoesNotThrow(
() -> {
final URL url = new URL(inputAddress);
new InetSocketAddress(url.getHost(), url.getPort());
});
} {code}
URI
{code:java}
@ParameterizedTest
@ValueSource(
strings = {"http://127.0.0.1:8081;, "http://123-dev:8081;,
"http://dev-test.abc:8081;, "http://dev-test.1a:8081;,
"http://dev-test.abc01:8081"})
void testURIAddresses(String inputAddress) {
assertDoesNotThrow(
() -> {
final URI uri = new URI(inputAddress);
new InetSocketAddress(uri.getHost(), uri.getPort());
});
} {code}
All test cases past except for "http://dev-test.1a:8081; which is a valid
flink host url, but not a valid URI
was:
In certain scenarios, if the hostname contains certain characters it will throw
an exception when it tries to initialize the `InetSocketAddress`
{code:java}
java.lang.IllegalArgumentException: hostname can't be null at
java.base/java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
at java.base/java.net.InetSocketAddress.(InetSocketAddress.java:216)
{code}
[https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L236]
{code:java}
@Override
public boolean isJobManagerPortReady(Configuration config) {
final URI uri;
try (var clusterClient = getClusterClient(config)) {
uri = URI.create(clusterClient.getWebInterfaceURL());
} catch (Exception ex) {
throw new FlinkRuntimeException(ex);
}
SocketAddress socketAddress = new InetSocketAddress(uri.getHost(),
uri.getPort());
Socket socket = new Socket();
try {
socket.connect(socketAddress, 1000);
socket.close();
return true;
} catch (IOException e) {
return false;
}
}
{code}
Here's a simple test to reproduce
URL
{code:java}
@ParameterizedTest
@ValueSource(
strings = {"http://127.0.0.1:8081;, "http://123-dev:8081;,
"http://dev-test.abc:8081;, "http://dev-test.1a:8081;,
"http://dev-test.abc01:8081"})
void testURLAddresses(String inputAddress) {
assertDoesNotThrow(
() -> {
final URL url = new URL(inputAddress);
new InetSocketAddress(url.getHost(), url.getPort());
});
} {code}
URI
{code:java}
@ParameterizedTest
@ValueSource(
strings = {"http://127.0.0.1:8081;, "http://123-dev:8081;,
"http://dev-test.abc:8081;, "http://dev-test.1a:8081;,
"http://dev-test.abc01:8081"})
void testURIAddresses(String inputAddress) {
assertDoesNotThrow(
() -> {
final URI uri = new URI(inputAddress);
new InetSocketAddress(uri.getHost(), uri.getPort());
[jira] [Comment Edited] (FLINK-33917) IllegalArgumentException: hostname can't be null
[
https://issues.apache.org/jira/browse/FLINK-33917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799677#comment-17799677
]
Tom edited comment on FLINK-33917 at 12/22/23 6:25 AM:
---
Actually uri("123-test") does work. The issue here is the '.' followed by a
number
uri("123-test.abc") works, but uri("dev-test.1") does not.
Here's a test case you can run
{code:java}
@ParameterizedTest
@ValueSource(
strings = {"http://127.0.0.1:8081;, "http://123-dev:8081;,
"http://dev-test.abc:8081;, "http://dev-test.1a:8081;,
"http://dev-test.abc01:8081"})
void testURIAddresses(String inputAddress) {
assertDoesNotThrow(
() -> {
final URI uri = new URI(inputAddress);
new InetSocketAddress(uri.getHost(), uri.getPort());
});
} {code}
was (Author: JIRAUSER303121):
Actually uri("123-test") does work. The issue here is the '.' followed by a
number
uri("123-test.abc") works, but uri("dev-test.01") does not.
Here's a test case you can run
{code:java}
@ParameterizedTest
@ValueSource(
strings = {"http://127.0.0.1:8081;, "http://123-dev:8081;,
"http://dev-test.abc:8081;, "http://dev-test.1a:8081;,
"http://dev-test.abc01:8081"})
void testURIAddresses(String inputAddress) {
assertDoesNotThrow(
() -> {
final URI uri = new URI(inputAddress);
new InetSocketAddress(uri.getHost(), uri.getPort());
});
} {code}
> IllegalArgumentException: hostname can't be null
>
>
> Key: FLINK-33917
> URL: https://issues.apache.org/jira/browse/FLINK-33917
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Tom
>Priority: Major
> Labels: pull-request-available
>
> In certain scenarios, if the hostname contains certain characters it will
> throw an exception when it tries to initialize the `InetSocketAddress`
>
> {code:java}
> java.lang.IllegalArgumentException: hostname can't be null at
> java.base/java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
> at
> java.base/java.net.InetSocketAddress.(InetSocketAddress.java:216) {code}
>
> [https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L236]
>
> {code:java}
> @Override
> public boolean isJobManagerPortReady(Configuration config) {
> final URI uri;
> try (var clusterClient = getClusterClient(config)) {
> uri = URI.create(clusterClient.getWebInterfaceURL());
> } catch (Exception ex) {
> throw new FlinkRuntimeException(ex);
> }
> SocketAddress socketAddress = new InetSocketAddress(uri.getHost(),
> uri.getPort());
> Socket socket = new Socket();
> try {
> socket.connect(socketAddress, 1000);
> socket.close();
> return true;
> } catch (IOException e) {
> return false;
> }
> }
> {code}
>
> Here's a simple test to reproduce
>
> URL
> {code:java}
> @ParameterizedTest
> @ValueSource(
> strings = {"http://127.0.0.1:8081;, "http://123-dev:8081;,
> "http://dev-test.abc:8081;, "http://dev-test.1a:8081;,
> "http://dev-test.abc01:8081"})
> void testURLAddresses(String inputAddress) {
> assertDoesNotThrow(
> () -> {
> final URL url = new URL(inputAddress);
> new InetSocketAddress(url.getHost(), url.getPort());
> });
> } {code}
>
> URI
>
> {code:java}
> @ParameterizedTest
> @ValueSource(
> strings = {"http://127.0.0.1:8081;, "http://123-dev:8081;,
> "http://dev-test.abc:8081;, "http://dev-test.1a:8081;,
> "http://dev-test.abc01:8081"})
> void testURIAddresses(String inputAddress) {
> assertDoesNotThrow(
> () -> {
> final URI uri = new URI(inputAddress);
> new InetSocketAddress(uri.getHost(), uri.getPort());
> });
> } {code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33917) IllegalArgumentException: hostname can't be null
[
https://issues.apache.org/jira/browse/FLINK-33917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799677#comment-17799677
]
Tom edited comment on FLINK-33917 at 12/22/23 6:25 AM:
---
Actually uri("123-test") does work. The issue here is the '.' followed by a
number
uri("123-test.abc") works, but uri("dev-test.01") does not.
Here's a test case you can run
{code:java}
@ParameterizedTest
@ValueSource(
strings = {"http://127.0.0.1:8081;, "http://123-dev:8081;,
"http://dev-test.abc:8081;, "http://dev-test.1a:8081;,
"http://dev-test.abc01:8081"})
void testURIAddresses(String inputAddress) {
assertDoesNotThrow(
() -> {
final URI uri = new URI(inputAddress);
new InetSocketAddress(uri.getHost(), uri.getPort());
});
} {code}
was (Author: JIRAUSER303121):
Actually uri("123-test") does work. The issue here is the '.' followed by a
number
uri("123-test.abc") works, but uri("dev-test.01") does not.
Here's a test case you can run
@ParameterizedTest
@ValueSource(
strings = \{"http://127.0.0.1:8081;, "http://123-dev:8081;,
"http://dev-test.abc:8081;, "http://dev-test.1a:8081;,
"http://dev-test.abc01:8081"})
void testURIAddresses(String inputAddress) {
assertDoesNotThrow(
() -> {
final URI url = new URI(inputAddress);
new InetSocketAddress(url.getHost(), url.getPort());
});
}
> IllegalArgumentException: hostname can't be null
>
>
> Key: FLINK-33917
> URL: https://issues.apache.org/jira/browse/FLINK-33917
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Tom
>Priority: Major
> Labels: pull-request-available
>
> In certain scenarios, if the hostname contains certain characters it will
> throw an exception when it tries to initialize the `InetSocketAddress`
>
> {code:java}
> java.lang.IllegalArgumentException: hostname can't be null at
> java.base/java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
> at
> java.base/java.net.InetSocketAddress.(InetSocketAddress.java:216) {code}
>
> [https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L236]
>
> {code:java}
> @Override
> public boolean isJobManagerPortReady(Configuration config) {
> final URI uri;
> try (var clusterClient = getClusterClient(config)) {
> uri = URI.create(clusterClient.getWebInterfaceURL());
> } catch (Exception ex) {
> throw new FlinkRuntimeException(ex);
> }
> SocketAddress socketAddress = new InetSocketAddress(uri.getHost(),
> uri.getPort());
> Socket socket = new Socket();
> try {
> socket.connect(socketAddress, 1000);
> socket.close();
> return true;
> } catch (IOException e) {
> return false;
> }
> }
> {code}
>
> Here's a simple test to reproduce
>
> URL
> {code:java}
> @ParameterizedTest
> @ValueSource(
> strings = {"http://127.0.0.1:8081;, "http://123-dev:8081;,
> "http://dev-test.abc:8081;, "http://dev-test.1a:8081;,
> "http://dev-test.abc01:8081"})
> void testURLAddresses(String inputAddress) {
> assertDoesNotThrow(
> () -> {
> final URL url = new URL(inputAddress);
> new InetSocketAddress(url.getHost(), url.getPort());
> });
> } {code}
>
> URI
>
> {code:java}
> @ParameterizedTest
> @ValueSource(
> strings = {"http://127.0.0.1:8081;, "http://123-dev:8081;,
> "http://dev-test.abc:8081;, "http://dev-test.1a:8081;,
> "http://dev-test.abc01:8081"})
> void testURIAddresses(String inputAddress) {
> assertDoesNotThrow(
> () -> {
> final URI uri = new URI(inputAddress);
> new InetSocketAddress(uri.getHost(), uri.getPort());
> });
> } {code}
>
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33917) IllegalArgumentException: hostname can't be null
[
https://issues.apache.org/jira/browse/FLINK-33917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tom updated FLINK-33917:
Description:
In certain scenarios, if the hostname contains certain characters it will throw
an exception when it tries to initialize the `InetSocketAddress`
{code:java}
java.lang.IllegalArgumentException: hostname can't be null at
java.base/java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
at java.base/java.net.InetSocketAddress.(InetSocketAddress.java:216)
{code}
[https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L236]
{code:java}
@Override
public boolean isJobManagerPortReady(Configuration config) {
final URI uri;
try (var clusterClient = getClusterClient(config)) {
uri = URI.create(clusterClient.getWebInterfaceURL());
} catch (Exception ex) {
throw new FlinkRuntimeException(ex);
}
SocketAddress socketAddress = new InetSocketAddress(uri.getHost(),
uri.getPort());
Socket socket = new Socket();
try {
socket.connect(socketAddress, 1000);
socket.close();
return true;
} catch (IOException e) {
return false;
}
}
{code}
Here's a simple test to reproduce
URL
{code:java}
@ParameterizedTest
@ValueSource(
strings = {"http://127.0.0.1:8081;, "http://123-dev:8081;,
"http://dev-test.abc:8081;, "http://dev-test.1a:8081;,
"http://dev-test.abc01:8081"})
void testURLAddresses(String inputAddress) {
assertDoesNotThrow(
() -> {
final URL url = new URL(inputAddress);
new InetSocketAddress(url.getHost(), url.getPort());
});
} {code}
URI
{code:java}
@ParameterizedTest
@ValueSource(
strings = {"http://127.0.0.1:8081;, "http://123-dev:8081;,
"http://dev-test.abc:8081;, "http://dev-test.1a:8081;,
"http://dev-test.abc01:8081"})
void testURIAddresses(String inputAddress) {
assertDoesNotThrow(
() -> {
final URI uri = new URI(inputAddress);
new InetSocketAddress(uri.getHost(), uri.getPort());
});
} {code}
was:
In certain scenarios, if the hostname contains certain characters it will throw
an exception when it tries to initialize the `InetSocketAddress`
{code:java}
java.lang.IllegalArgumentException: hostname can't be null at
java.base/java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
at java.base/java.net.InetSocketAddress.(InetSocketAddress.java:216)
{code}
[https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L236]
{code:java}
@Override
public boolean isJobManagerPortReady(Configuration config) {
final URI uri;
try (var clusterClient = getClusterClient(config)) {
uri = URI.create(clusterClient.getWebInterfaceURL());
} catch (Exception ex) {
throw new FlinkRuntimeException(ex);
}
SocketAddress socketAddress = new InetSocketAddress(uri.getHost(),
uri.getPort());
Socket socket = new Socket();
try {
socket.connect(socketAddress, 1000);
socket.close();
return true;
} catch (IOException e) {
return false;
}
}
{code}
> IllegalArgumentException: hostname can't be null
>
>
> Key: FLINK-33917
> URL: https://issues.apache.org/jira/browse/FLINK-33917
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Tom
>Priority: Major
> Labels: pull-request-available
>
> In certain scenarios, if the hostname contains certain characters it will
> throw an exception when it tries to initialize the `InetSocketAddress`
>
> {code:java}
> java.lang.IllegalArgumentException: hostname can't be null at
> java.base/java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
> at
> java.base/java.net.InetSocketAddress.(InetSocketAddress.java:216) {code}
>
>
[jira] [Commented] (FLINK-33917) IllegalArgumentException: hostname can't be null
[
https://issues.apache.org/jira/browse/FLINK-33917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799677#comment-17799677
]
Tom commented on FLINK-33917:
-
Actually uri("123-test") does work. The issue here is the '.' followed by a
number
uri("123-test.abc") works, but uri("dev-test.01") does not.
Here's a test case you can run
@ParameterizedTest
@ValueSource(
strings = \{"http://127.0.0.1:8081;, "http://123-dev:8081;,
"http://dev-test.abc:8081;, "http://dev-test.1a:8081;,
"http://dev-test.abc01:8081"})
void testURIAddresses(String inputAddress) {
assertDoesNotThrow(
() -> {
final URI url = new URI(inputAddress);
new InetSocketAddress(url.getHost(), url.getPort());
});
}
> IllegalArgumentException: hostname can't be null
>
>
> Key: FLINK-33917
> URL: https://issues.apache.org/jira/browse/FLINK-33917
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Reporter: Tom
>Priority: Major
> Labels: pull-request-available
>
> In certain scenarios, if the hostname contains certain characters it will
> throw an exception when it tries to initialize the `InetSocketAddress`
>
> {code:java}
> java.lang.IllegalArgumentException: hostname can't be null at
> java.base/java.net.InetSocketAddress.checkHost(InetSocketAddress.java:149)
> at
> java.base/java.net.InetSocketAddress.(InetSocketAddress.java:216) {code}
>
> [https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java|https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/AbstractFlinkService.java#L236]
>
> {code:java}
> @Override
> public boolean isJobManagerPortReady(Configuration config) {
> final URI uri;
> try (var clusterClient = getClusterClient(config)) {
> uri = URI.create(clusterClient.getWebInterfaceURL());
> } catch (Exception ex) {
> throw new FlinkRuntimeException(ex);
> }
> SocketAddress socketAddress = new InetSocketAddress(uri.getHost(),
> uri.getPort());
> Socket socket = new Socket();
> try {
> socket.connect(socketAddress, 1000);
> socket.close();
> return true;
> } catch (IOException e) {
> return false;
> }
> }
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33583][table] support state ttl hint for join [flink]
LadyForest commented on PR #23752: URL: https://github.com/apache/flink/pull/23752#issuecomment-1867289962 Hi @xuyangzhong, sorry for being late. Could you mind rebasing the master and resolving the conflicts? I'll take a look within this week. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-32850) [JUnit5 Migration] The io package of flink-runtime module
[ https://issues.apache.org/jira/browse/FLINK-32850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu resolved FLINK-32850. Resolution: Implemented All PRs merged, close this ticket > [JUnit5 Migration] The io package of flink-runtime module > - > > Key: FLINK-32850 > URL: https://issues.apache.org/jira/browse/FLINK-32850 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Rui Fan >Assignee: Jiabao Sun >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32850) [JUnit5 Migration] The io package of flink-runtime module
[ https://issues.apache.org/jira/browse/FLINK-32850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-32850: --- Affects Version/s: 1.18.0 > [JUnit5 Migration] The io package of flink-runtime module > - > > Key: FLINK-32850 > URL: https://issues.apache.org/jira/browse/FLINK-32850 > Project: Flink > Issue Type: Sub-task > Components: Tests >Affects Versions: 1.18.0 >Reporter: Rui Fan >Assignee: Jiabao Sun >Priority: Minor > Labels: pull-request-available > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-32850) [JUnit5 Migration] The io package of flink-runtime module
[ https://issues.apache.org/jira/browse/FLINK-32850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Leonard Xu updated FLINK-32850: --- Fix Version/s: 1.19.0 > [JUnit5 Migration] The io package of flink-runtime module > - > > Key: FLINK-32850 > URL: https://issues.apache.org/jira/browse/FLINK-32850 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Rui Fan >Assignee: Jiabao Sun >Priority: Minor > Labels: pull-request-available > Fix For: 1.19.0 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32850) [JUnit5 Migration] The io package of flink-runtime module
[ https://issues.apache.org/jira/browse/FLINK-32850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799659#comment-17799659 ] Jiabao Sun commented on FLINK-32850: Thanks [~Weijie Guo], this task is finished. Could you help close this issue as well? > [JUnit5 Migration] The io package of flink-runtime module > - > > Key: FLINK-32850 > URL: https://issues.apache.org/jira/browse/FLINK-32850 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Rui Fan >Assignee: Jiabao Sun >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33653) Introduce a benchmark for balanced tasks scheduling
[ https://issues.apache.org/jira/browse/FLINK-33653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan reassigned FLINK-33653: --- Assignee: RocMarshal > Introduce a benchmark for balanced tasks scheduling > --- > > Key: FLINK-33653 > URL: https://issues.apache.org/jira/browse/FLINK-33653 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33874) Support resource request wait mechanism at DefaultDeclarativeSlotPool side for Default Scheduler
[ https://issues.apache.org/jira/browse/FLINK-33874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan reassigned FLINK-33874: --- Assignee: RocMarshal > Support resource request wait mechanism at DefaultDeclarativeSlotPool side > for Default Scheduler > > > Key: FLINK-33874 > URL: https://issues.apache.org/jira/browse/FLINK-33874 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Assigned] (FLINK-33875) Support slots wait mechanism at DeclarativeSlotPoolBridge side for Default Scheduler
[ https://issues.apache.org/jira/browse/FLINK-33875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rui Fan reassigned FLINK-33875: --- Assignee: RocMarshal > Support slots wait mechanism at DeclarativeSlotPoolBridge side for Default > Scheduler > > > Key: FLINK-33875 > URL: https://issues.apache.org/jira/browse/FLINK-33875 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Task >Reporter: RocMarshal >Assignee: RocMarshal >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-32850) [JUnit5 Migration] The io package of flink-runtime module
[ https://issues.apache.org/jira/browse/FLINK-32850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799652#comment-17799652 ] Weijie Guo commented on FLINK-32850: master(1.19) via 462708228df253de09e0a002ee0f2bee94e701c5. > [JUnit5 Migration] The io package of flink-runtime module > - > > Key: FLINK-32850 > URL: https://issues.apache.org/jira/browse/FLINK-32850 > Project: Flink > Issue Type: Sub-task > Components: Tests >Reporter: Rui Fan >Assignee: Jiabao Sun >Priority: Minor > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-32850][flink-runtime][JUnit5 Migration] The io.network.partition package of flink-runtime module [flink]
reswqa closed pull request #23974: [FLINK-32850][flink-runtime][JUnit5 Migration] The io.network.partition package of flink-runtime module URL: https://github.com/apache/flink/pull/23974 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-32513) Job in BATCH mode with a significant number of transformations freezes on method StreamGraphGenerator.existsUnboundedSource()
[
https://issues.apache.org/jira/browse/FLINK-32513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799647#comment-17799647
]
Zhu Zhu edited comment on FLINK-32513 at 12/22/23 2:46 AM:
---
The problem happens because the {{getTransitivePredecessors()}} of
{{TwoInputTransformation}} is not properly implemented. If the two inputs share
the same upstream node, that node will be visited twice. It results in a 2^N(N
= number of TwoInputTransformation) time cost to iterate and space cost to
store the predecessors.
A possible solution can be adding a cache of predecessors for each
Transformation and using LinkedHashSet to deduplicate the predecessors.
Note that a few other transformations can lead to the same problem too, e.g.
UnionTransformation, AbstractMultipleInputTransformation.
was (Author: zhuzh):
The problem happens because the {{getTransitivePredecessors()}} of
{{TwoInputTransformation}} is not properly implemented, which results in a
2^N(N = number of TwoInputTransformation) time cost to iterate and space cost
to store the predecessors.
A possible solution can be adding a cache of predecessors for each
Transformation and using LinkedHashSet to deduplicate the predecessors.
Note that a few other transformations can lead to the same problem too, e.g.
UnionTransformation, AbstractMultipleInputTransformation.
> Job in BATCH mode with a significant number of transformations freezes on
> method StreamGraphGenerator.existsUnboundedSource()
> -
>
> Key: FLINK-32513
> URL: https://issues.apache.org/jira/browse/FLINK-32513
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.15.3, 1.16.1, 1.17.1
> Environment: All modes (local, k8s session, k8s application, ...)
> Flink 1.15.3
> Flink 1.16.1
> Flink 1.17.1
>Reporter: Vladislav Keda
>Priority: Critical
> Attachments: image-2023-07-10-17-26-46-544.png
>
>
> Flink job executed in BATCH mode with a significant number of transformations
> (more than 30 in my case) takes very long time to start due to the method
> StreamGraphGenerator.existsUnboundedSource(). Also, during the execution of
> the method, a lot of memory is consumed, which causes the GC to fire
> frequently.
> Thread Dump:
> {code:java}
> "main@1" prio=5 tid=0x1 nid=NA runnable
> java.lang.Thread.State: RUNNABLE
> at java.util.ArrayList.addAll(ArrayList.java:702)
> at
> org.apache.flink.streaming.api.transformations.TwoInputTransformation.getTransitivePredecessors(TwoInputTransformation.java:224)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.PartitionTransformation.getTransitivePredecessors(PartitionTransformation.java:95)
> at
> org.apache.flink.streaming.api.transformations.TwoInputTransformation.getTransitivePredecessors(TwoInputTransformation.java:223)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
>
Re: [PR] [FLINK-32850][flink-runtime][JUnit5 Migration] The io.network.partition package of flink-runtime module [flink]
Jiabao-Sun commented on PR #23974: URL: https://github.com/apache/flink/pull/23974#issuecomment-1867163701 > After this PR, whether all tests in `io.network` have been migrated? Yes. Thanks a lot. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-32513) Job in BATCH mode with a significant number of transformations freezes on method StreamGraphGenerator.existsUnboundedSource()
[
https://issues.apache.org/jira/browse/FLINK-32513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799647#comment-17799647
]
Zhu Zhu commented on FLINK-32513:
-
The problem happens because the {{getTransitivePredecessors()}} of
{{TwoInputTransformation}} is not properly implemented, which results in a
2^N(N = number of TwoInputTransformation) time cost to iterate and space cost
to store the predecessors.
A possible solution can be adding a cache of predecessors for each
Transformation and using LinkedHashSet to deduplicate the predecessors.
Note that a few other transformations can lead to the same problem too, e.g.
UnionTransformation, AbstractMultipleInputTransformation.
> Job in BATCH mode with a significant number of transformations freezes on
> method StreamGraphGenerator.existsUnboundedSource()
> -
>
> Key: FLINK-32513
> URL: https://issues.apache.org/jira/browse/FLINK-32513
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.15.3, 1.16.1, 1.17.1
> Environment: All modes (local, k8s session, k8s application, ...)
> Flink 1.15.3
> Flink 1.16.1
> Flink 1.17.1
>Reporter: Vladislav Keda
>Priority: Critical
> Attachments: image-2023-07-10-17-26-46-544.png
>
>
> Flink job executed in BATCH mode with a significant number of transformations
> (more than 30 in my case) takes very long time to start due to the method
> StreamGraphGenerator.existsUnboundedSource(). Also, during the execution of
> the method, a lot of memory is consumed, which causes the GC to fire
> frequently.
> Thread Dump:
> {code:java}
> "main@1" prio=5 tid=0x1 nid=NA runnable
> java.lang.Thread.State: RUNNABLE
> at java.util.ArrayList.addAll(ArrayList.java:702)
> at
> org.apache.flink.streaming.api.transformations.TwoInputTransformation.getTransitivePredecessors(TwoInputTransformation.java:224)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.PartitionTransformation.getTransitivePredecessors(PartitionTransformation.java:95)
> at
> org.apache.flink.streaming.api.transformations.TwoInputTransformation.getTransitivePredecessors(TwoInputTransformation.java:223)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.PartitionTransformation.getTransitivePredecessors(PartitionTransformation.java:95)
> at
> org.apache.flink.streaming.api.transformations.TwoInputTransformation.getTransitivePredecessors(TwoInputTransformation.java:223)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
> org.apache.flink.streaming.api.transformations.OneInputTransformation.getTransitivePredecessors(OneInputTransformation.java:174)
> at
>
Re: [PR] [FLINK-32850][flink-runtime][JUnit5 Migration] The io.network.partition package of flink-runtime module [flink]
reswqa commented on PR #23974: URL: https://github.com/apache/flink/pull/23974#issuecomment-1867145877 After this PR, whether all tests in `io.network` have been migrated? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-25476) CharType lost in the creation of MaxAggFunction & MinAggFunction
[
https://issues.apache.org/jira/browse/FLINK-25476?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
dalongliu resolved FLINK-25476.
---
Fix Version/s: 1.19.0
Resolution: Fixed
> CharType lost in the creation of MaxAggFunction & MinAggFunction
>
>
> Key: FLINK-25476
> URL: https://issues.apache.org/jira/browse/FLINK-25476
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.15.0
>Reporter: zoucao
>Assignee: Xu Yang
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> pull-request-available
> Fix For: 1.19.0
>
>
> CharType lost in AggFunctionFactory#createMinAggFunction(Retract) and
> AggFunctionFactory#createMaxAggFunction(Retract).
> If execute
> {code:java}
> SELECT Max('a') ...
> {code}
> , The following exception will throw.
>
> Caused by: org.apache.flink.table.api.TableException: Max aggregate function
> does not support type: ''CHAR''.
> Please re-check the data type.
> at
> org.apache.flink.table.planner.plan.utils.AggFunctionFactory.createMaxAggFunction(AggFunctionFactory.scala:395)
> at
> org.apache.flink.table.planner.plan.utils.AggFunctionFactory.createAggFunction(AggFunctionFactory.scala:76)
> at
> org.apache.flink.table.planner.plan.utils.AggregateUtil$.$anonfun$transformToAggregateInfoList$1(AggregateUtil.scala:444)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (FLINK-25476) CharType lost in the creation of MaxAggFunction & MinAggFunction
[
https://issues.apache.org/jira/browse/FLINK-25476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799640#comment-17799640
]
dalongliu commented on FLINK-25476:
---
Merged to master: 085859ec19f59d2f8e73632e96fae59f7821c7d0
> CharType lost in the creation of MaxAggFunction & MinAggFunction
>
>
> Key: FLINK-25476
> URL: https://issues.apache.org/jira/browse/FLINK-25476
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.15.0
>Reporter: zoucao
>Assignee: Xu Yang
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> pull-request-available
>
> CharType lost in AggFunctionFactory#createMinAggFunction(Retract) and
> AggFunctionFactory#createMaxAggFunction(Retract).
> If execute
> {code:java}
> SELECT Max('a') ...
> {code}
> , The following exception will throw.
>
> Caused by: org.apache.flink.table.api.TableException: Max aggregate function
> does not support type: ''CHAR''.
> Please re-check the data type.
> at
> org.apache.flink.table.planner.plan.utils.AggFunctionFactory.createMaxAggFunction(AggFunctionFactory.scala:395)
> at
> org.apache.flink.table.planner.plan.utils.AggFunctionFactory.createAggFunction(AggFunctionFactory.scala:76)
> at
> org.apache.flink.table.planner.plan.utils.AggregateUtil$.$anonfun$transformToAggregateInfoList$1(AggregateUtil.scala:444)
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-25476][table-planner] support CHAR type in function MAX and MIN [flink]
lsyldliu closed pull request #18375: [FLINK-25476][table-planner] support CHAR type in function MAX and MIN URL: https://github.com/apache/flink/pull/18375 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [BP-1.18][FLINK-27082][ci] Adds a github-actions profile that disables certain tests that do not run in GHA [flink]
flinkbot commented on PR #23982: URL: https://github.com/apache/flink/pull/23982#issuecomment-1867064898 ## CI report: * 6c905021e5b09a3537a68d208749bdc8ee1061f5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-27082][ci] Disables tests that relies on disabling file permissions [flink]
XComp merged PR #23962: URL: https://github.com/apache/flink/pull/23962 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [BP-1.18][FLINK-27082][ci] Adds a github-actions profile that disables certain tests that do not run in GHA [flink]
XComp commented on PR #23982: URL: https://github.com/apache/flink/pull/23982#issuecomment-1867063585 @snuyanzin can you do another quick pass over this one considering that there were some conflicts which I had to resolve? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33440] Update Flink version matrix, add 1.18.0 and 1.19-SNAPSHOT to GH workflows [flink-connector-hbase]
snuyanzin commented on PR #35: URL: https://github.com/apache/flink-connector-hbase/pull/35#issuecomment-1867027544 There was snappy update, probably need to add explicit dependency to cope with https://github.com/apache/flink-connector-hbase/actions/runs/7085132836/job/19876358681?pr=35#step:14:78 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-27082][ci] Disables tests that relies on disabling file permissions [flink]
snuyanzin commented on code in PR #23962:
URL: https://github.com/apache/flink/pull/23962#discussion_r1434572736
##
flink-core/src/test/java/org/apache/flink/core/fs/local/LocalFileSystemTest.java:
##
@@ -236,6 +237,7 @@ public void testRenameNonExistingFile() throws IOException {
}
@Test
+@Ignore("FLINK-27082: Fails in Docker with root user")
Review Comment:
Awesome, thanks for making it real :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-27082][ci] Disables tests that relies on disabling file permissions [flink]
XComp commented on code in PR #23962:
URL: https://github.com/apache/flink/pull/23962#discussion_r1434558666
##
flink-core/src/test/java/org/apache/flink/core/fs/local/LocalFileSystemTest.java:
##
@@ -236,6 +237,7 @@ public void testRenameNonExistingFile() throws IOException {
}
@Test
+@Ignore("FLINK-27082: Fails in Docker with root user")
Review Comment:
I updated the PR according to your idea. :+1:
The following two GHA workflow runs include this PR:
* [build #68](https://github.com/XComp/flink/actions/runs/7292799860) does
not have the profile activated
* [build #70](https://github.com/XComp/flink/actions/runs/7292859288) does
have the profile activated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-33817) Allow ReadDefaultValues = False for non primitive types on Proto3
[ https://issues.apache.org/jira/browse/FLINK-33817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799584#comment-17799584 ] Sai Sharath Dandi commented on FLINK-33817: --- [~libenchao] , Can you please take a look at this ticket and assign it to me? > Allow ReadDefaultValues = False for non primitive types on Proto3 > - > > Key: FLINK-33817 > URL: https://issues.apache.org/jira/browse/FLINK-33817 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.18.0 >Reporter: Sai Sharath Dandi >Priority: Major > > *Background* > > The current Protobuf format > [implementation|https://github.com/apache/flink/blob/c3e2d163a637dca5f49522721109161bd7ebb723/flink-formats/flink-protobuf/src/main/java/org/apache/flink/formats/protobuf/deserialize/ProtoToRowConverter.java] > always sets ReadDefaultValues=False when using Proto3 version. This can > cause severe performance degradation for large Protobuf schemas with OneOf > fields as the entire generated code needs to be executed during > deserialization even when certain fields are not present in the data to be > deserialized and all the subsequent nested Fields can be skipped. Proto3 > supports hasXXX() methods for checking field presence for non primitive types > since Proto version > [3.15|https://github.com/protocolbuffers/protobuf/releases/tag/v3.15.0]. In > the internal performance benchmarks in our company, we've seen almost 10x > difference in performance for one of our real production usecase when > allowing to set ReadDefaultValues=False with proto3 version. The exact > difference in performance depends on the schema complexity and data payload > but we should allow readDefaultValue=False in general. > > *Solution* > > Support using ReadDefaultValues=False when using Proto3 version. We need to > be careful to check for field presence only on non-primitive types if > ReadDefaultValues is false and version used is Proto3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (FLINK-33817) Allow ReadDefaultValues = False for non primitive types on Proto3
[ https://issues.apache.org/jira/browse/FLINK-33817?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sai Sharath Dandi updated FLINK-33817: -- Description: *Background* The current Protobuf format [implementation|https://github.com/apache/flink/blob/c3e2d163a637dca5f49522721109161bd7ebb723/flink-formats/flink-protobuf/src/main/java/org/apache/flink/formats/protobuf/deserialize/ProtoToRowConverter.java] always sets ReadDefaultValues=False when using Proto3 version. This can cause severe performance degradation for large Protobuf schemas with OneOf fields as the entire generated code needs to be executed during deserialization even when certain fields are not present in the data to be deserialized and all the subsequent nested Fields can be skipped. Proto3 supports hasXXX() methods for checking field presence for non primitive types since Proto version [3.15|https://github.com/protocolbuffers/protobuf/releases/tag/v3.15.0]. In the internal performance benchmarks in our company, we've seen almost 10x difference in performance for one of our real production usecase when allowing to set ReadDefaultValues=False with proto3 version. The exact difference in performance depends on the schema complexity and data payload but we should allow user to set readDefaultValue=False in general. *Solution* Support using ReadDefaultValues=False when using Proto3 version. We need to be careful to check for field presence only on non-primitive types if ReadDefaultValues is false and version used is Proto3 was: *Background* The current Protobuf format [implementation|https://github.com/apache/flink/blob/c3e2d163a637dca5f49522721109161bd7ebb723/flink-formats/flink-protobuf/src/main/java/org/apache/flink/formats/protobuf/deserialize/ProtoToRowConverter.java] always sets ReadDefaultValues=False when using Proto3 version. This can cause severe performance degradation for large Protobuf schemas with OneOf fields as the entire generated code needs to be executed during deserialization even when certain fields are not present in the data to be deserialized and all the subsequent nested Fields can be skipped. Proto3 supports hasXXX() methods for checking field presence for non primitive types since Proto version [3.15|https://github.com/protocolbuffers/protobuf/releases/tag/v3.15.0]. In the internal performance benchmarks in our company, we've seen almost 10x difference in performance for one of our real production usecase when allowing to set ReadDefaultValues=False with proto3 version. The exact difference in performance depends on the schema complexity and data payload but we should allow readDefaultValue=False in general. *Solution* Support using ReadDefaultValues=False when using Proto3 version. We need to be careful to check for field presence only on non-primitive types if ReadDefaultValues is false and version used is Proto3 > Allow ReadDefaultValues = False for non primitive types on Proto3 > - > > Key: FLINK-33817 > URL: https://issues.apache.org/jira/browse/FLINK-33817 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.18.0 >Reporter: Sai Sharath Dandi >Priority: Major > > *Background* > > The current Protobuf format > [implementation|https://github.com/apache/flink/blob/c3e2d163a637dca5f49522721109161bd7ebb723/flink-formats/flink-protobuf/src/main/java/org/apache/flink/formats/protobuf/deserialize/ProtoToRowConverter.java] > always sets ReadDefaultValues=False when using Proto3 version. This can > cause severe performance degradation for large Protobuf schemas with OneOf > fields as the entire generated code needs to be executed during > deserialization even when certain fields are not present in the data to be > deserialized and all the subsequent nested Fields can be skipped. Proto3 > supports hasXXX() methods for checking field presence for non primitive types > since Proto version > [3.15|https://github.com/protocolbuffers/protobuf/releases/tag/v3.15.0]. In > the internal performance benchmarks in our company, we've seen almost 10x > difference in performance for one of our real production usecase when > allowing to set ReadDefaultValues=False with proto3 version. The exact > difference in performance depends on the schema complexity and data payload > but we should allow user to set readDefaultValue=False in general. > > *Solution* > > Support using ReadDefaultValues=False when using Proto3 version. We need to > be careful to check for field presence only on non-primitive types if > ReadDefaultValues is false and version used is Proto3 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33611) Support Large Protobuf Schemas
[ https://issues.apache.org/jira/browse/FLINK-33611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799560#comment-17799560 ] Sai Sharath Dandi edited comment on FLINK-33611 at 12/21/23 9:23 PM: - [~libenchao] , The constant pool size [limit|https://docs.oracle.com/javase/specs/jvms/se12/html/jvms-4.html#jvms-4.4] is 65536 entries in Java. The constant pool size includes a lot of things but if we count only the identifier names and assume there are 2 identifiers(one for protobuf and one for rowdata) the generated code used for each field in the schema for rough estimation. There cannot be more than 65536/2 = 32768 fields in the Protobuf schema. Of course, the actual number is lower than that because we did not include split method names, class names etc.. was (Author: JIRAUSER298466): [~libenchao] , The constant pool size [limit|https://docs.oracle.com/javase/specs/jvms/se12/html/jvms-4.html#jvms-4.4] is 65536 entries in java. The constant pool size includes a lot of things but if we count only the identifier names and assume there are 2 identifiers(one for protobuf and one for rowdata) the generated code used for each field in the schema for rough estimation. There cannot be more than 65536/2 = 32768 fields in the Protobuf schema. Of course, the actual number is lower than that because we did not include split method names, class names etc.. > Support Large Protobuf Schemas > -- > > Key: FLINK-33611 > URL: https://issues.apache.org/jira/browse/FLINK-33611 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.18.0 >Reporter: Sai Sharath Dandi >Assignee: Sai Sharath Dandi >Priority: Major > Labels: pull-request-available > > h3. Background > Flink serializes and deserializes protobuf format data by calling the decode > or encode method in GeneratedProtoToRow_XXX.java generated by codegen to > parse byte[] data into Protobuf Java objects. FLINK-32650 has introduced the > ability to split the generated code to improve the performance for large > Protobuf schemas. However, this is still not sufficient to support some > larger protobuf schemas as the generated code exceeds the java constant pool > size [limit|https://en.wikipedia.org/wiki/Java_class_file#The_constant_pool] > and we can see errors like "Too many constants" when trying to compile the > generated code. > *Solution* > Since we already have the split code functionality already introduced, the > main proposal here is to now reuse the variable names across different split > method scopes. This will greatly reduce the constant pool size. One more > optimization is to only split the last code segment also only when the size > exceeds split threshold limit. Currently, the last segment of the generated > code is always being split which can lead to too many split methods and thus > exceed the constant pool size limit -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33611) Support Large Protobuf Schemas
[ https://issues.apache.org/jira/browse/FLINK-33611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799560#comment-17799560 ] Sai Sharath Dandi edited comment on FLINK-33611 at 12/21/23 9:22 PM: - [~libenchao] , The constant pool size [limit|https://docs.oracle.com/javase/specs/jvms/se12/html/jvms-4.html#jvms-4.4] is 65536 entries in java. The constant pool size includes a lot of things but if we count only the identifier names and assume there are 2 identifiers(one for protobuf and one for rowdata) the generated code used for each field in the schema for rough estimation. There cannot be more than 65536/2 = 32768 fields in the Protobuf schema. Of course, the actual number is lower than that because we did not include split method names, class names etc.. was (Author: JIRAUSER298466): [~libenchao] , The constant pool size [limit|https://docs.oracle.com/javase/specs/jvms/se12/html/jvms-4.html#jvms-4.4] is 65536 entries in java. The constant pool size includes a lot of things but if we count only the identifier names and assume there are 2 identifiers in the generated code used for each field in the schema for rough estimation. There cannot be more than 65536/2 = 32768 fields in the Protobuf schema. Of course, the actual number is lower than that because we did not include split method names, class names etc.. > Support Large Protobuf Schemas > -- > > Key: FLINK-33611 > URL: https://issues.apache.org/jira/browse/FLINK-33611 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.18.0 >Reporter: Sai Sharath Dandi >Assignee: Sai Sharath Dandi >Priority: Major > Labels: pull-request-available > > h3. Background > Flink serializes and deserializes protobuf format data by calling the decode > or encode method in GeneratedProtoToRow_XXX.java generated by codegen to > parse byte[] data into Protobuf Java objects. FLINK-32650 has introduced the > ability to split the generated code to improve the performance for large > Protobuf schemas. However, this is still not sufficient to support some > larger protobuf schemas as the generated code exceeds the java constant pool > size [limit|https://en.wikipedia.org/wiki/Java_class_file#The_constant_pool] > and we can see errors like "Too many constants" when trying to compile the > generated code. > *Solution* > Since we already have the split code functionality already introduced, the > main proposal here is to now reuse the variable names across different split > method scopes. This will greatly reduce the constant pool size. One more > optimization is to only split the last code segment also only when the size > exceeds split threshold limit. Currently, the last segment of the generated > code is always being split which can lead to too many split methods and thus > exceed the constant pool size limit -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33611) Support Large Protobuf Schemas
[ https://issues.apache.org/jira/browse/FLINK-33611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799583#comment-17799583 ] Sai Sharath Dandi commented on FLINK-33611: --- Fwiw, I've run into this issue for one of our real production use case schema at my company which has about 44,368 fields in the Protobuf schema > Support Large Protobuf Schemas > -- > > Key: FLINK-33611 > URL: https://issues.apache.org/jira/browse/FLINK-33611 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.18.0 >Reporter: Sai Sharath Dandi >Assignee: Sai Sharath Dandi >Priority: Major > Labels: pull-request-available > > h3. Background > Flink serializes and deserializes protobuf format data by calling the decode > or encode method in GeneratedProtoToRow_XXX.java generated by codegen to > parse byte[] data into Protobuf Java objects. FLINK-32650 has introduced the > ability to split the generated code to improve the performance for large > Protobuf schemas. However, this is still not sufficient to support some > larger protobuf schemas as the generated code exceeds the java constant pool > size [limit|https://en.wikipedia.org/wiki/Java_class_file#The_constant_pool] > and we can see errors like "Too many constants" when trying to compile the > generated code. > *Solution* > Since we already have the split code functionality already introduced, the > main proposal here is to now reuse the variable names across different split > method scopes. This will greatly reduce the constant pool size. One more > optimization is to only split the last code segment also only when the size > exceeds split threshold limit. Currently, the last segment of the generated > code is always being split which can lead to too many split methods and thus > exceed the constant pool size limit -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Comment Edited] (FLINK-33611) Support Large Protobuf Schemas
[ https://issues.apache.org/jira/browse/FLINK-33611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799560#comment-17799560 ] Sai Sharath Dandi edited comment on FLINK-33611 at 12/21/23 8:18 PM: - [~libenchao] , The constant pool size [limit|https://docs.oracle.com/javase/specs/jvms/se12/html/jvms-4.html#jvms-4.4] is 65536 entries in java. The constant pool size includes a lot of things but if we count only the identifier names and assume there are 2 identifiers in the generated code used for each field in the schema for rough estimation. There cannot be more than 65536/2 = 32768 fields in the Protobuf schema. Of course, the actual number is lower than that because we did not include split method names, class names etc.. was (Author: JIRAUSER298466): [~libenchao] , The constant pool size [limit|https://docs.oracle.com/javase/specs/jvms/se12/html/jvms-4.html#jvms-4.4] is 65536 entries in java. The constant pool size includes a lot of things but if we count only the identifier names and assume there are 2 identifiers used for each field in the schema for rough estimation. There cannot be more than 65536/2 = 32768 fields in the Protobuf schema. Of course, the actual number is lower than that because we did not include split method names, class names etc.. > Support Large Protobuf Schemas > -- > > Key: FLINK-33611 > URL: https://issues.apache.org/jira/browse/FLINK-33611 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.18.0 >Reporter: Sai Sharath Dandi >Assignee: Sai Sharath Dandi >Priority: Major > Labels: pull-request-available > > h3. Background > Flink serializes and deserializes protobuf format data by calling the decode > or encode method in GeneratedProtoToRow_XXX.java generated by codegen to > parse byte[] data into Protobuf Java objects. FLINK-32650 has introduced the > ability to split the generated code to improve the performance for large > Protobuf schemas. However, this is still not sufficient to support some > larger protobuf schemas as the generated code exceeds the java constant pool > size [limit|https://en.wikipedia.org/wiki/Java_class_file#The_constant_pool] > and we can see errors like "Too many constants" when trying to compile the > generated code. > *Solution* > Since we already have the split code functionality already introduced, the > main proposal here is to now reuse the variable names across different split > method scopes. This will greatly reduce the constant pool size. One more > optimization is to only split the last code segment also only when the size > exceeds split threshold limit. Currently, the last segment of the generated > code is always being split which can lead to too many split methods and thus > exceed the constant pool size limit -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (FLINK-33611) Support Large Protobuf Schemas
[ https://issues.apache.org/jira/browse/FLINK-33611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799560#comment-17799560 ] Sai Sharath Dandi commented on FLINK-33611: --- [~libenchao] , The constant pool size [limit|https://docs.oracle.com/javase/specs/jvms/se12/html/jvms-4.html#jvms-4.4] is 65536 entries in java. The constant pool size includes a lot of things but if we count only the identifier names and assume there are 2 identifiers used for each field in the schema for rough estimation. There cannot be more than 65536/2 = 32768 fields in the Protobuf schema. Of course, the actual number is lower than that because we did not include split method names, class names etc.. > Support Large Protobuf Schemas > -- > > Key: FLINK-33611 > URL: https://issues.apache.org/jira/browse/FLINK-33611 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.18.0 >Reporter: Sai Sharath Dandi >Assignee: Sai Sharath Dandi >Priority: Major > Labels: pull-request-available > > h3. Background > Flink serializes and deserializes protobuf format data by calling the decode > or encode method in GeneratedProtoToRow_XXX.java generated by codegen to > parse byte[] data into Protobuf Java objects. FLINK-32650 has introduced the > ability to split the generated code to improve the performance for large > Protobuf schemas. However, this is still not sufficient to support some > larger protobuf schemas as the generated code exceeds the java constant pool > size [limit|https://en.wikipedia.org/wiki/Java_class_file#The_constant_pool] > and we can see errors like "Too many constants" when trying to compile the > generated code. > *Solution* > Since we already have the split code functionality already introduced, the > main proposal here is to now reuse the variable names across different split > method scopes. This will greatly reduce the constant pool size. One more > optimization is to only split the last code segment also only when the size > exceeds split threshold limit. Currently, the last segment of the generated > code is always being split which can lead to too many split methods and thus > exceed the constant pool size limit -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33611] [flink-protobuf] Support Large Protobuf Schemas [flink]
sharath1709 commented on PR #23937: URL: https://github.com/apache/flink/pull/23937#issuecomment-1866817077 @libenchao Thanks for the quick response. Sure, let's try to reach an agreement on the JIRA first -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33926) Can't start a job with a jar in the system classpath in native k8s mode
[
https://issues.apache.org/jira/browse/FLINK-33926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trystan updated FLINK-33926:
Description:
It appears that the combination of the running operator-controlled jobs in
native k8s + application mode + using a job jar in the classpath is invalid.
Avoiding dynamic classloading (as specified in the
[docs|https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code])
is beneficial for some jobs. This affects at least Flink 1.16.1 and operator
1.6.0.
FLINK-29288 seems to have addressed this for standalone mode. If I am
misunderstanding how to correctly build jars for this native k8s scenario,
apologies for the noise and any pointers would be appreciated!
Perhaps related, the [spec
documentation|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#jobspec]
declares it optional, but isn't clear about under what conditions that applies.
* Putting the jar in the system classpath and pointing *jarURI* to that jar
leads to linkage errors.
* Not including *jarURI* leads to NullPointerExceptions in the operator:
{code:java}
{"type":"org.apache.flink.kubernetes.operator.exception.ReconciliationException","message":"java.lang.NullPointerException","stackTrace":"org.apache.flink.kubernetes.operator.exception.ReconciliationException:
java.lang.NullPointerException\n\tat
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:148)\n\tat
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:56)\n\tat
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:138)\n\tat
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:96)\n\tat
org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)\n\tat
io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:95)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62)\n\tat
io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:414)\n\tat
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)\n\tat
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\nCaused by:
java.lang.NullPointerException\n\tat
org.apache.flink.kubernetes.utils.KubernetesUtils.checkJarFileForApplicationMode(KubernetesUtils.java:407)\n\tat
org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployApplicationCluster(KubernetesClusterDescriptor.java:207)\n\tat
org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer.run(ApplicationClusterDeployer.java:67)\n\tat
org.apache.flink.kubernetes.operator.service.NativeFlinkService.deployApplicationCluster(Native","additionalMetadata":{},"throwableList":[{"type":"java.lang.NullPointerException","additionalMetadata":{}}]}
{code}
was:
It appears that the combination of the running operator-controlled jobs in
native k8s + application mode + using a job jar in the classpath is invalid.
Avoiding dynamic classloading (as specified in the
[docs|https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code])
is beneficial for some jobs. This affects at least Flink 1.16.1.
FLINK-29288 seems to have addressed this for standalone mode. If I am
misunderstanding how to correctly build jars for this native k8s scenario,
apologies for the noise and any pointers would be appreciated!
Perhaps related, the [spec
documentation|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#jobspec]
declares it optional, but isn't clear about under what conditions that applies.
* Putting the jar in the system classpath and pointing *jarURI* to that jar
leads to linkage errors.
* Not including *jarURI* leads to NullPointerExceptions in the operator:
{code:java}
{"type":"org.apache.flink.kubernetes.operator.exception.ReconciliationException","message":"java.lang.NullPointerException","stackTrace":"org.apache.flink.kubernetes.operator.exception.ReconciliationException:
java.lang.NullPointerException\n\tat
[jira] [Updated] (FLINK-33926) Can't start a job with a jar in the system classpath in native k8s mode
[
https://issues.apache.org/jira/browse/FLINK-33926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trystan updated FLINK-33926:
Component/s: Kubernetes Operator
> Can't start a job with a jar in the system classpath in native k8s mode
> ---
>
> Key: FLINK-33926
> URL: https://issues.apache.org/jira/browse/FLINK-33926
> Project: Flink
> Issue Type: Bug
> Components: Kubernetes Operator
>Affects Versions: kubernetes-operator-1.6.0
>Reporter: Trystan
>Priority: Major
>
> It appears that the combination of the running operator-controlled jobs in
> native k8s + application mode + using a job jar in the classpath is invalid.
> Avoiding dynamic classloading (as specified in the
> [docs|https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code])
> is beneficial for some jobs. This affects at least Flink 1.16.1.
>
> FLINK-29288 seems to have addressed this for standalone mode. If I am
> misunderstanding how to correctly build jars for this native k8s scenario,
> apologies for the noise and any pointers would be appreciated!
>
> Perhaps related, the [spec
> documentation|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#jobspec]
> declares it optional, but isn't clear about under what conditions that
> applies.
> * Putting the jar in the system classpath and pointing *jarURI* to that jar
> leads to linkage errors.
> * Not including *jarURI* leads to NullPointerExceptions in the operator:
> {code:java}
> {"type":"org.apache.flink.kubernetes.operator.exception.ReconciliationException","message":"java.lang.NullPointerException","stackTrace":"org.apache.flink.kubernetes.operator.exception.ReconciliationException:
> java.lang.NullPointerException\n\tat
> org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:148)\n\tat
>
> org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:56)\n\tat
>
> io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:138)\n\tat
>
> io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:96)\n\tat
>
> org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)\n\tat
>
> io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:95)\n\tat
>
> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139)\n\tat
>
> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119)\n\tat
>
> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89)\n\tat
>
> io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62)\n\tat
>
> io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:414)\n\tat
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)\n\tat
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\nCaused by:
> java.lang.NullPointerException\n\tat
> org.apache.flink.kubernetes.utils.KubernetesUtils.checkJarFileForApplicationMode(KubernetesUtils.java:407)\n\tat
>
> org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployApplicationCluster(KubernetesClusterDescriptor.java:207)\n\tat
>
> org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer.run(ApplicationClusterDeployer.java:67)\n\tat
>
> org.apache.flink.kubernetes.operator.service.NativeFlinkService.deployApplicationCluster(Native","additionalMetadata":{},"throwableList":[{"type":"java.lang.NullPointerException","additionalMetadata":{}}]}
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33926) Can't start a job with a jar in the system classpath in native k8s mode
[
https://issues.apache.org/jira/browse/FLINK-33926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trystan updated FLINK-33926:
Description:
It appears that the combination of the running operator-controlled jobs in
native k8s + application mode + using a job jar in the classpath is invalid.
Avoiding dynamic classloading (as specified in the
[docs|https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code])
is beneficial for some jobs. This affects at least Flink 1.16.1 and Kubernetes
Operator 1.6.0.
FLINK-29288 seems to have addressed this for standalone mode. If I am
misunderstanding how to correctly build jars for this native k8s scenario,
apologies for the noise and any pointers would be appreciated!
Perhaps related, the [spec
documentation|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#jobspec]
declares it optional, but isn't clear about under what conditions that applies.
* Putting the jar in the system classpath and pointing *jarURI* to that jar
leads to linkage errors.
* Not including *jarURI* leads to NullPointerExceptions in the operator:
{code:java}
{"type":"org.apache.flink.kubernetes.operator.exception.ReconciliationException","message":"java.lang.NullPointerException","stackTrace":"org.apache.flink.kubernetes.operator.exception.ReconciliationException:
java.lang.NullPointerException\n\tat
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:148)\n\tat
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:56)\n\tat
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:138)\n\tat
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:96)\n\tat
org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)\n\tat
io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:95)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62)\n\tat
io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:414)\n\tat
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)\n\tat
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\nCaused by:
java.lang.NullPointerException\n\tat
org.apache.flink.kubernetes.utils.KubernetesUtils.checkJarFileForApplicationMode(KubernetesUtils.java:407)\n\tat
org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployApplicationCluster(KubernetesClusterDescriptor.java:207)\n\tat
org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer.run(ApplicationClusterDeployer.java:67)\n\tat
org.apache.flink.kubernetes.operator.service.NativeFlinkService.deployApplicationCluster(Native","additionalMetadata":{},"throwableList":[{"type":"java.lang.NullPointerException","additionalMetadata":{}}]}
{code}
was:
It appears that the combination of the running operator-controlled jobs in
native k8s + application mode + using a job jar in the classpath is invalid.
Avoiding dynamic classloading (as specified in the
[docs|https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code])
is beneficial for some jobs. This affects at least Flink 1.16.1 and operator
1.6.0.
FLINK-29288 seems to have addressed this for standalone mode. If I am
misunderstanding how to correctly build jars for this native k8s scenario,
apologies for the noise and any pointers would be appreciated!
Perhaps related, the [spec
documentation|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#jobspec]
declares it optional, but isn't clear about under what conditions that applies.
* Putting the jar in the system classpath and pointing *jarURI* to that jar
leads to linkage errors.
* Not including *jarURI* leads to NullPointerExceptions in the operator:
{code:java}
{"type":"org.apache.flink.kubernetes.operator.exception.ReconciliationException","message":"java.lang.NullPointerException","stackTrace":"org.apache.flink.kubernetes.operator.exception.ReconciliationException:
[jira] [Updated] (FLINK-33926) Can't start a job with a jar in the system classpath in native k8s mode
[
https://issues.apache.org/jira/browse/FLINK-33926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trystan updated FLINK-33926:
Description:
It appears that the combination of the running operator-controlled jobs in
native k8s + application mode + using a job jar in the classpath is invalid.
Avoiding dynamic classloading (as specified in the
[docs|https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code])
is beneficial for some jobs. This affects at least Flink 1.16.1.
FLINK-29288 seems to have addressed this for standalone mode. If I am
misunderstanding how to correctly build jars for this native k8s scenario,
apologies for the noise and any pointers would be appreciated!
Perhaps related, the [spec
documentation|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#jobspec]
declares it optional, but isn't clear about under what conditions that applies.
* Putting the jar in the system classpath and pointing *jarURI* to that jar
leads to linkage errors.
* Not including *jarURI* leads to NullPointerExceptions in the operator:
{code:java}
{"type":"org.apache.flink.kubernetes.operator.exception.ReconciliationException","message":"java.lang.NullPointerException","stackTrace":"org.apache.flink.kubernetes.operator.exception.ReconciliationException:
java.lang.NullPointerException\n\tat
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:148)\n\tat
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:56)\n\tat
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:138)\n\tat
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:96)\n\tat
org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)\n\tat
io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:95)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62)\n\tat
io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:414)\n\tat
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)\n\tat
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\nCaused by:
java.lang.NullPointerException\n\tat
org.apache.flink.kubernetes.utils.KubernetesUtils.checkJarFileForApplicationMode(KubernetesUtils.java:407)\n\tat
org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployApplicationCluster(KubernetesClusterDescriptor.java:207)\n\tat
org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer.run(ApplicationClusterDeployer.java:67)\n\tat
org.apache.flink.kubernetes.operator.service.NativeFlinkService.deployApplicationCluster(Native","additionalMetadata":{},"throwableList":[{"type":"java.lang.NullPointerException","additionalMetadata":{}}]}
{code}
was:
It appears that the combination of the running operator-controlled jobs in
native k8s + application mode + using a job jar in the classpath is invalid.
Avoiding dynamic classloading (as specified in the
[docs|https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code])
is beneficial for some jobs. This affects at least Flink 1.16.1.
FLINK-29288 seems to have addressed this for standalone mode. If I am
misunderstanding how to correctly build jars for this scenario, apologies for
the noise and any pointers would be appreciated!
Perhaps related, the [spec
documentation|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#jobspec]
declares it optional, but isn't clear about under what conditions that applies.
* Putting the jar in the system classpath and pointing *jarURI* to that jar
leads to linkage errors.
* Not including *jarURI* leads to NullPointerExceptions in the operator:
{code:java}
{"type":"org.apache.flink.kubernetes.operator.exception.ReconciliationException","message":"java.lang.NullPointerException","stackTrace":"org.apache.flink.kubernetes.operator.exception.ReconciliationException:
java.lang.NullPointerException\n\tat
[jira] [Updated] (FLINK-33926) Can't start a job with a jar in the system classpath in native k8s mode
[
https://issues.apache.org/jira/browse/FLINK-33926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trystan updated FLINK-33926:
Description:
It appears that the combination of the running operator-controlled jobs in
native k8s + application mode + using a job jar in the classpath is invalid.
Avoiding dynamic classloading (as specified in the
[docs|https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code])
is beneficial for some jobs. This affects at least Flink 1.16.1.
FLINK-29288 seems to have addressed this for standalone mode. If I am
misunderstanding how to correctly build jars for this scenario, apologies for
the noise and any pointers would be appreciated!
Perhaps related, the [spec
documentation|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#jobspec]
declares it optional, but isn't clear about under what conditions that applies.
* Putting the jar in the system classpath and pointing *jarURI* to that jar
leads to linkage errors.
* Not including *jarURI* leads to NullPointerExceptions in the operator:
{code:java}
{"type":"org.apache.flink.kubernetes.operator.exception.ReconciliationException","message":"java.lang.NullPointerException","stackTrace":"org.apache.flink.kubernetes.operator.exception.ReconciliationException:
java.lang.NullPointerException\n\tat
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:148)\n\tat
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:56)\n\tat
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:138)\n\tat
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:96)\n\tat
org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)\n\tat
io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:95)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62)\n\tat
io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:414)\n\tat
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)\n\tat
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\nCaused by:
java.lang.NullPointerException\n\tat
org.apache.flink.kubernetes.utils.KubernetesUtils.checkJarFileForApplicationMode(KubernetesUtils.java:407)\n\tat
org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployApplicationCluster(KubernetesClusterDescriptor.java:207)\n\tat
org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer.run(ApplicationClusterDeployer.java:67)\n\tat
org.apache.flink.kubernetes.operator.service.NativeFlinkService.deployApplicationCluster(Native","additionalMetadata":{},"throwableList":[{"type":"java.lang.NullPointerException","additionalMetadata":{}}]}
{code}
was:
It appears that the combination of the running operator-controlled jobs in
native k8s application mode + using a job jar in the classpath is invalid.
Avoiding dynamic classloading (as specified in the
[docs|https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code])
is beneficial for some jobs. This affects at least Flink 1.16.1.
FLINK-29288 addressed this for standalone mode.
Perhaps related, the [spec
documentation|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#jobspec]
declares it optional, but isn't clear about under what conditions that applies.
* Putting the jar in the system classpath and pointing *jarURI* to that jar
leads to linkage errors.
* Not including *jarURI* leads to NullPointerExceptions in the operator:
{code:java}
{"type":"org.apache.flink.kubernetes.operator.exception.ReconciliationException","message":"java.lang.NullPointerException","stackTrace":"org.apache.flink.kubernetes.operator.exception.ReconciliationException:
java.lang.NullPointerException\n\tat
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:148)\n\tat
[jira] [Created] (FLINK-33926) Can't start a job with a jar in the system classpath in native k8s mode
Trystan created FLINK-33926:
---
Summary: Can't start a job with a jar in the system classpath in
native k8s mode
Key: FLINK-33926
URL: https://issues.apache.org/jira/browse/FLINK-33926
Project: Flink
Issue Type: Bug
Affects Versions: kubernetes-operator-1.6.0
Reporter: Trystan
It appears that the combination of the running operator-controlled jobs in
native k8s mode + using a job jar in the classpath is invalid. Avoiding dynamic
classloading (as specified in the
[docs|https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code])
is beneficial for some jobs. This affects at least Flink 1.16.1.
[FLINK-29288|https://issues.apache.org/jira/browse/FLINK-29288] addressed this
for standalone mode.
Perhaps related, the [spec
documentation|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#jobspec]
declares it optional, but isn't clear about under what conditions that applies.
* Putting the jar in the system classpath and pointing *jarURI* to that jar
leads to linkage errors.
* Not including *jarURI* leads to NullPointerExceptions in the operator:
{code:java}
{"type":"org.apache.flink.kubernetes.operator.exception.ReconciliationException","message":"java.lang.NullPointerException","stackTrace":"org.apache.flink.kubernetes.operator.exception.ReconciliationException:
java.lang.NullPointerException\n\tat
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:148)\n\tat
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:56)\n\tat
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:138)\n\tat
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:96)\n\tat
org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)\n\tat
io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:95)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62)\n\tat
io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:414)\n\tat
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)\n\tat
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\nCaused by:
java.lang.NullPointerException\n\tat
org.apache.flink.kubernetes.utils.KubernetesUtils.checkJarFileForApplicationMode(KubernetesUtils.java:407)\n\tat
org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployApplicationCluster(KubernetesClusterDescriptor.java:207)\n\tat
org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer.run(ApplicationClusterDeployer.java:67)\n\tat
org.apache.flink.kubernetes.operator.service.NativeFlinkService.deployApplicationCluster(Native","additionalMetadata":{},"throwableList":[{"type":"java.lang.NullPointerException","additionalMetadata":{}}]}
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (FLINK-33926) Can't start a job with a jar in the system classpath in native k8s mode
[
https://issues.apache.org/jira/browse/FLINK-33926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Trystan updated FLINK-33926:
Description:
It appears that the combination of the running operator-controlled jobs in
native k8s application mode + using a job jar in the classpath is invalid.
Avoiding dynamic classloading (as specified in the
[docs|https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code])
is beneficial for some jobs. This affects at least Flink 1.16.1.
FLINK-29288 addressed this for standalone mode.
Perhaps related, the [spec
documentation|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#jobspec]
declares it optional, but isn't clear about under what conditions that applies.
* Putting the jar in the system classpath and pointing *jarURI* to that jar
leads to linkage errors.
* Not including *jarURI* leads to NullPointerExceptions in the operator:
{code:java}
{"type":"org.apache.flink.kubernetes.operator.exception.ReconciliationException","message":"java.lang.NullPointerException","stackTrace":"org.apache.flink.kubernetes.operator.exception.ReconciliationException:
java.lang.NullPointerException\n\tat
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:148)\n\tat
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:56)\n\tat
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:138)\n\tat
io.javaoperatorsdk.operator.processing.Controller$1.execute(Controller.java:96)\n\tat
org.apache.flink.kubernetes.operator.metrics.OperatorJosdkMetrics.timeControllerExecution(OperatorJosdkMetrics.java:80)\n\tat
io.javaoperatorsdk.operator.processing.Controller.reconcile(Controller.java:95)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.reconcileExecution(ReconciliationDispatcher.java:139)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleReconcile(ReconciliationDispatcher.java:119)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleDispatch(ReconciliationDispatcher.java:89)\n\tat
io.javaoperatorsdk.operator.processing.event.ReconciliationDispatcher.handleExecution(ReconciliationDispatcher.java:62)\n\tat
io.javaoperatorsdk.operator.processing.event.EventProcessor$ReconcilerExecutor.run(EventProcessor.java:414)\n\tat
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)\n\tat
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)\n\tat java.base/java.lang.Thread.run(Unknown Source)\nCaused by:
java.lang.NullPointerException\n\tat
org.apache.flink.kubernetes.utils.KubernetesUtils.checkJarFileForApplicationMode(KubernetesUtils.java:407)\n\tat
org.apache.flink.kubernetes.KubernetesClusterDescriptor.deployApplicationCluster(KubernetesClusterDescriptor.java:207)\n\tat
org.apache.flink.client.deployment.application.cli.ApplicationClusterDeployer.run(ApplicationClusterDeployer.java:67)\n\tat
org.apache.flink.kubernetes.operator.service.NativeFlinkService.deployApplicationCluster(Native","additionalMetadata":{},"throwableList":[{"type":"java.lang.NullPointerException","additionalMetadata":{}}]}
{code}
was:
It appears that the combination of the running operator-controlled jobs in
native k8s mode + using a job jar in the classpath is invalid. Avoiding dynamic
classloading (as specified in the
[docs|https://nightlies.apache.org/flink/flink-docs-release-1.16/docs/ops/debugging/debugging_classloading/#avoiding-dynamic-classloading-for-user-code])
is beneficial for some jobs. This affects at least Flink 1.16.1.
[FLINK-29288|https://issues.apache.org/jira/browse/FLINK-29288] addressed this
for standalone mode.
Perhaps related, the [spec
documentation|https://nightlies.apache.org/flink/flink-kubernetes-operator-docs-main/docs/custom-resource/reference/#jobspec]
declares it optional, but isn't clear about under what conditions that applies.
* Putting the jar in the system classpath and pointing *jarURI* to that jar
leads to linkage errors.
* Not including *jarURI* leads to NullPointerExceptions in the operator:
{code:java}
{"type":"org.apache.flink.kubernetes.operator.exception.ReconciliationException","message":"java.lang.NullPointerException","stackTrace":"org.apache.flink.kubernetes.operator.exception.ReconciliationException:
java.lang.NullPointerException\n\tat
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:148)\n\tat
org.apache.flink.kubernetes.operator.controller.FlinkDeploymentController.reconcile(FlinkDeploymentController.java:56)\n\tat
[PR] Mircea - module 1 [flink-training]
ness-MirceaLutic opened a new pull request, #68: URL: https://github.com/apache/flink-training/pull/68 (no comment) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
gyfora commented on code in PR #728: URL: https://github.com/apache/flink-kubernetes-operator/pull/728#discussion_r1434220037 ## flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java: ## @@ -264,4 +262,21 @@ private static Map getVertexParallelismOverrides( }); return overrides; } + +private boolean blockScalingExecution( Review Comment: We should rename this to `triggerScalingEvent` or `checkIfBlockedAndTriggerScalingEvent` or add javadocs or something that suggests that the scaling event is triggered inside -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
gyfora commented on code in PR #728:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/728#discussion_r1434216202
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/event/AutoScalerEventHandler.java:
##
@@ -83,12 +92,27 @@ default void handleScalingEvent(
}
static String scalingReport(
-Map scalingSummaries, boolean
scalingEnabled) {
-StringBuilder sb =
-new StringBuilder(
-scalingEnabled
-? SCALING_SUMMARY_HEADER_SCALING_ENABLED
-: SCALING_SUMMARY_HEADER_SCALING_DISABLED);
+Map scalingSummaries,
+boolean scalingEnabled,
+boolean isExcluded,
+Configuration config) {
+StringBuilder sb = new StringBuilder();
+if (!scalingEnabled) {
+sb.append(
+String.format(
+SCALING_SUMMARY_HEADER_SCALING_EXECUTION_DISABLED,
+SCALING_ENABLED.key(),
+false));
+} else if (isExcluded) {
+sb.append(
+String.format(
+SCALING_SUMMARY_HEADER_SCALING_EXECUTION_DISABLED,
+EXCLUDED_PERIODS.key(),
+config.get(EXCLUDED_PERIODS)));
+} else {
+sb.append(SCALING_SUMMARY_HEADER_SCALING_EXECUTION_ENABLED);
+}
Review Comment:
I think it would make the code much cleaner to simply submit a string to the
event handler instead of the 2 booleans.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
gyfora commented on code in PR #728:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/728#discussion_r1434212997
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##
@@ -264,4 +262,21 @@ private static Map
getVertexParallelismOverrides(
});
return overrides;
}
+
+private boolean blockScalingExecution(
+Context context,
+Map scalingSummaries,
+Configuration conf,
+Instant now) {
+var scaleEnabled = conf.get(SCALING_ENABLED);
+var isExcluded = CalendarUtils.inExcludedPeriods(conf, now);
+autoScalerEventHandler.handleScalingEvent(
+context,
+scalingSummaries,
+scaleEnabled,
+isExcluded,
+conf.get(SCALING_EVENT_INTERVAL));
+
+ return !scaleEnabled || isExcluded;
Review Comment:
Sorry, I misunderstood the code, I am fine with this :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
gyfora commented on code in PR #728:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/728#discussion_r1434211420
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##
@@ -264,4 +262,21 @@ private static Map
getVertexParallelismOverrides(
});
return overrides;
}
+
+private boolean blockScalingExecution(
+Context context,
+Map scalingSummaries,
+Configuration conf,
+Instant now) {
+var scaleEnabled = conf.get(SCALING_ENABLED);
+var isExcluded = CalendarUtils.inExcludedPeriods(conf, now);
+autoScalerEventHandler.handleScalingEvent(
+context,
+scalingSummaries,
+scaleEnabled,
+isExcluded,
+conf.get(SCALING_EVENT_INTERVAL));
+
+ return !scaleEnabled || isExcluded;
Review Comment:
Ah sorry I did not realise that you also changed the event message for the
basic recommendations... my bad
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
gyfora commented on code in PR #728:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/728#discussion_r1434209212
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##
@@ -264,4 +262,21 @@ private static Map
getVertexParallelismOverrides(
});
return overrides;
}
+
+private boolean blockScalingExecution(
+Context context,
+Map scalingSummaries,
+Configuration conf,
+Instant now) {
+var scaleEnabled = conf.get(SCALING_ENABLED);
+var isExcluded = CalendarUtils.inExcludedPeriods(conf, now);
+autoScalerEventHandler.handleScalingEvent(
+context,
+scalingSummaries,
+scaleEnabled,
+isExcluded,
+conf.get(SCALING_EVENT_INTERVAL));
+
+ return !scaleEnabled || isExcluded;
Review Comment:
In the current logic we send 2 types of events on a parallelism change:
1. Scaling Recommendation (when disabled) -> `Recommended parallelism
change: {parallelisms}`
2. Scaling Event (when enabled) -> `Scaling Vertices: {parallelisms}`
The PR basically adds a 3rd type of event which looks like a recommendation
but it doesn't say recommendation it prints:
`Scaling execution disabled by config: configkey:false {parallelism:map}`
I don't think that this is fully consistent with the current logic. We
intended to keep the recommendations, but we changed the code so that it's not
a recommendation anymore but looks different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-20772) RocksDBValueState with TTL occurs NullPointerException when calling update(null) method
[
https://issues.apache.org/jira/browse/FLINK-20772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799448#comment-17799448
]
Seongbae Chang commented on FLINK-20772:
[~masteryhx], [~Zakelly] Sorry for the late response. Nowadays, I had not read
my private e-mail.
I'm sad to miss the opportunity to contribute Flink, but I'm absolutely happy
it for Flink to be enhanced. Thank you for your interest and contribution.
> RocksDBValueState with TTL occurs NullPointerException when calling
> update(null) method
>
>
> Key: FLINK-20772
> URL: https://issues.apache.org/jira/browse/FLINK-20772
> Project: Flink
> Issue Type: Bug
> Components: Runtime / State Backends
>Affects Versions: 1.11.2
> Environment: Flink version: 1.11.2
> Flink Cluster: Standalone cluster with 3 Job managers and Task managers on
> CentOS 7
>Reporter: Seongbae Chang
>Assignee: Zakelly Lan
>Priority: Not a Priority
> Labels: auto-deprioritized-major, auto-deprioritized-minor,
> beginner, pull-request-available
> Fix For: 1.19.0
>
>
> h2. Problem
> * I use ValueState for my custom trigger and set TTL for these ValueState in
> RocksDB backend environment.
> * I found an error when I used this code. I know that
> ValueState.update(null) works equally to ValueState.clear() in general.
> Unfortunately, this error occurs after using TTL
> {code:java}
> // My Code
> ctx.getPartitionedState(batchTotalSizeStateDesc).update(null);
> {code}
> * I tested this in Flink 1.11.2, but I think it would be a problem in upper
> versions.
> * Plus, I'm a beginner. So, if there is any problem in this discussion
> issue, please give me advice about that. And I'll fix it!
> {code:java}
> // Error Stacktrace
> Caused by: TimerException{org.apache.flink.util.FlinkRuntimeException: Error
> while adding data to RocksDB}
> ... 12 more
> Caused by: org.apache.flink.util.FlinkRuntimeException: Error while adding
> data to RocksDB
> at
> org.apache.flink.contrib.streaming.state.RocksDBValueState.update(RocksDBValueState.java:108)
> at
> org.apache.flink.runtime.state.ttl.TtlValueState.update(TtlValueState.java:50)
> at .onProcessingTime(ActionBatchTimeTrigger.java:102)
> at .onProcessingTime(ActionBatchTimeTrigger.java:29)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$Context.onProcessingTime(WindowOperator.java:902)
> at
> org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.onProcessingTime(WindowOperator.java:498)
> at
> org.apache.flink.streaming.api.operators.InternalTimerServiceImpl.onProcessingTime(InternalTimerServiceImpl.java:260)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1220)
> ... 11 more
> Caused by: java.lang.NullPointerException
> at
> org.apache.flink.api.common.typeutils.base.IntSerializer.serialize(IntSerializer.java:69)
> at
> org.apache.flink.api.common.typeutils.base.IntSerializer.serialize(IntSerializer.java:32)
> at
> org.apache.flink.api.common.typeutils.CompositeSerializer.serialize(CompositeSerializer.java:142)
> at
> org.apache.flink.contrib.streaming.state.AbstractRocksDBState.serializeValueInternal(AbstractRocksDBState.java:158)
> at
> org.apache.flink.contrib.streaming.state.AbstractRocksDBState.serializeValue(AbstractRocksDBState.java:178)
> at
> org.apache.flink.contrib.streaming.state.AbstractRocksDBState.serializeValue(AbstractRocksDBState.java:167)
> at
> org.apache.flink.contrib.streaming.state.RocksDBValueState.update(RocksDBValueState.java:106)
> ... 18 more
> {code}
>
> h2. Reason
> * It relates to RocksDBValueState with TTLValueState
> * In RocksDBValueState(as well as other types of ValueState),
> *.update(null)* has to be caught in if-clauses(null checking). However, it
> skips the null checking and then tries to serialize the null value.
> {code:java}
> //
> https://github.com/apache/flink/blob/release-1.11/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBValueState.java#L96-L110
> @Override
> public void update(V value) {
> if (value == null) {
> clear();
> return;
> }
>
> try {
> backend.db.put(columnFamily, writeOptions,
> serializeCurrentKeyWithGroupAndNamespace(), serializeValue(value));
> } catch (Exception e) {
> throw new FlinkRuntimeException("Error while adding data to RocksDB",
> e);
> }
> }{code}
> * It is because that TtlValueState wraps the value(null) with the
> LastAccessTime and makes the new TtlValue Object with the null value.
>
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
mxm commented on code in PR #728:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/728#discussion_r1434199585
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##
@@ -264,4 +262,21 @@ private static Map
getVertexParallelismOverrides(
});
return overrides;
}
+
+private boolean blockScalingExecution(
+Context context,
+Map scalingSummaries,
+Configuration conf,
+Instant now) {
+var scaleEnabled = conf.get(SCALING_ENABLED);
+var isExcluded = CalendarUtils.inExcludedPeriods(conf, now);
+autoScalerEventHandler.handleScalingEvent(
+context,
+scalingSummaries,
+scaleEnabled,
+isExcluded,
+conf.get(SCALING_EVENT_INTERVAL));
+
+ return !scaleEnabled || isExcluded;
Review Comment:
I don't feel strongly about including the cause. We can also just send out
scaling events only for actual scaling operations. What @flashJd did is
actually more in line with the current code which does include meta information
like "scalingEnabled" in the event message. If we decide to skip those, for
consistency, I would also remove the "scalingEnabled" flag and only ever emit
scaling events when scaling is enabled and not blocked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33365] include filters with Lookup joins [flink-connector-jdbc]
snuyanzin commented on code in PR #79:
URL:
https://github.com/apache/flink-connector-jdbc/pull/79#discussion_r1434166658
##
flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/table/JdbcDynamicTableSource.java:
##
@@ -96,28 +97,115 @@ public JdbcDynamicTableSource(
public LookupRuntimeProvider getLookupRuntimeProvider(LookupContext
context) {
// JDBC only support non-nested look up keys
String[] keyNames = new String[context.getKeys().length];
+
for (int i = 0; i < keyNames.length; i++) {
int[] innerKeyArr = context.getKeys()[i];
Preconditions.checkArgument(
innerKeyArr.length == 1, "JDBC only support non-nested
look up keys");
keyNames[i] =
DataType.getFieldNames(physicalRowDataType).get(innerKeyArr[0]);
}
+
final RowType rowType = (RowType) physicalRowDataType.getLogicalType();
+
+String[] conditions = null;
+
+if (this.resolvedPredicates != null) {
+conditions = new String[this.resolvedPredicates.size()];
+int processedPushdownParamsIndex = 0;
+for (int i = 0; i < this.resolvedPredicates.size(); i++) {
+String resolvedPredicate = this.resolvedPredicates.get(i);
+
+/*
+ * This replace seems like it should be using a Flink class to
resolve the parameter. It does not
+ * effect the dialects as the placeholder comes from
JdbcFilterPushdownPreparedStatementVisitor.
+ *
+ * Here is what has been considered as alternatives.
+ *
+ * We cannot use the way this is done in
getScanRuntimeProvider, as the index we have is the index
+ * into the filters, but it needs the index into the fields.
For example one lookup key and one filter
+ * would both have an index of 0, which the subsequent code
would incorrectly resolve to the first
+ * field.
+ * We cannot use the PreparedStatement as we have not got
access to the statement here.
+ * We cannot use ParameterizedPredicate as it takes the filter
expression as input (e.g EQUALS(...)
+ * not the form we have here an example would be ('field1'= ?).
+ *
+ * An entry in the resolvedPredicates list may have more than
one associated pushdown parameter, for example
+ * a query like this : ... on e.type = 2 and (e.age = 50 OR
height > 90) and a.ip = e.ip;
+ * will have 2 resolvedPredicates and 3 pushdownParams. The
2nd and 3rd pushdownParams will be for the second
+ * resolvedPredicate.
+ *
+ */
+ArrayList paramsForThisPredicate = new ArrayList();
+char placeholderChar =
+
JdbcFilterPushdownPreparedStatementVisitor.PUSHDOWN_PREDICATE_PLACEHOLDER
+.charAt(0);
+
+int count =
+(int) resolvedPredicate.chars().filter(ch -> ch ==
placeholderChar).count();
+
+for (int j = processedPushdownParamsIndex;
+j < processedPushdownParamsIndex + count;
+j++) {
+
paramsForThisPredicate.add(this.pushdownParams[j].toString());
+}
+processedPushdownParamsIndex = processedPushdownParamsIndex +
count;
Review Comment:
I'm not sure this is a reliable approach
for instance if we take the same test
`org.apache.flink.connector.jdbc.table.JdbcTablePlanTest#testLookupJoinWithORFilter`
and rename `age` column with``?age` like
```java
util.tableEnv()
.executeSql(
"CREATE TABLE d ( "
+ "ip varchar(20), type int, ```?age` int"
+ ") WITH ("
+ " 'connector'='jdbc',"
+ " 'url'='jdbc:derby:memory:test1',"
+ " 'table-name'='d'"
+ ")");
```
then apply this rename to the query in test like
```java
@Test
public void testLookupJoinWithORFilter() {
util.verifyExecPlan(
"SELECT * FROM a left join d FOR SYSTEM_TIME AS OF
a.proctime on (d.```?age` = 50 OR d.type = 1) and a.ip = d.ip");
}
```
and now if we run this test it starts failing with
`java.lang.ArrayIndexOutOfBoundsException: 2`
like
```
Caused by: java.lang.ArrayIndexOutOfBoundsException: 2
at
org.apache.flink.connector.jdbc.table.JdbcDynamicTableSource.getLookupRuntimeProvider(JdbcDynamicTableSource.java:149)
at
Re: [PR] [FLINK-27082][ci] Disables tests that relies on disabling file permissions [flink]
XComp commented on code in PR #23962:
URL: https://github.com/apache/flink/pull/23962#discussion_r1434112468
##
flink-core/src/test/java/org/apache/flink/core/fs/local/LocalFileSystemTest.java:
##
@@ -236,6 +237,7 @@ public void testRenameNonExistingFile() throws IOException {
}
@Test
+@Ignore("FLINK-27082: Fails in Docker with root user")
Review Comment:
That's actually a good idea. I haven't thought of it. But that way, we could
disable the tests in GHA but keep them in Azure CI. I'm gonna look into it :+1:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33132] Flink Connector Redshift TableSink Implementation [flink-connector-aws]
Samrat002 commented on PR #114: URL: https://github.com/apache/flink-connector-aws/pull/114#issuecomment-1866305970 @hlteoh37, @vahmed-hamdy please review in free time -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [hotfix] [docs] Update README.md to remove Maven [flink]
snuyanzin commented on PR #23981: URL: https://github.com/apache/flink/pull/23981#issuecomment-1866302758 having maven wrapper does not protect from running with installed maven -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-33902) Switch to OpenSSL legacy algorithms
[
https://issues.apache.org/jira/browse/FLINK-33902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias Pohl resolved FLINK-33902.
---
Fix Version/s: 1.19.0
1.18.1
Resolution: Fixed
master:
[87738d3f0e5f6022b6decdf47045503de67d0a72|https://github.com/apache/flink/commit/87738d3f0e5f6022b6decdf47045503de67d0a72]
1.18:
[2ee3beeaa2302d96e1c420b03ecc0e1e22812110|https://github.com/apache/flink/commit/2ee3beeaa2302d96e1c420b03ecc0e1e22812110]
> Switch to OpenSSL legacy algorithms
> ---
>
> Key: FLINK-33902
> URL: https://issues.apache.org/jira/browse/FLINK-33902
> Project: Flink
> Issue Type: Sub-task
> Components: Build System
>Affects Versions: 1.18.0, 1.19.0
>Reporter: Matthias Pohl
>Assignee: Matthias Pohl
>Priority: Major
> Labels: github-actions, pull-request-available
> Fix For: 1.19.0, 1.18.1
>
>
> In FLINK-33550 we discovered that the GHA runners provided by GitHub have a
> newer version of OpenSSL installed which caused errors in the SSL tests:
> {code:java}
> Certificate was added to keystore
> Certificate was added to keystore
> Certificate reply was installed in keystore
> Error outputting keys and certificates
> 40F767F1D97F:error:0308010C:digital envelope
> routines:inner_evp_generic_fetch:unsupported:../crypto/evp/evp_fetch.c:349:Global
> default library context, Algorithm (RC2-40-CBC : 0), Properties ()
> Nov 14 15:39:21 [FAIL] Test script contains errors. {code}
> The workaround is to enable legacy algorithms using the {{-legacy}} parameter
> in 3.0.0+. We might need to check whether that works for older OpenSSL
> version (in Azure CI).
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [BP-1.18][FLINK-33902][ci] Adds -legacy to openssl command [flink]
XComp merged PR #23979: URL: https://github.com/apache/flink/pull/23979 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [BP-1.18][FLINK-33902][ci] Adds -legacy to openssl command [flink]
XComp commented on PR #23979: URL: https://github.com/apache/flink/pull/23979#issuecomment-1866296953 I'm merging w/o extra review because there were no conflicts in the backport and the parent PR was approved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33902][ci] Adds -legacy to openssl command [flink]
XComp merged PR #23961: URL: https://github.com/apache/flink/pull/23961 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [hotfix] [docs] Update README.md to remove Maven [flink]
flinkbot commented on PR #23981: URL: https://github.com/apache/flink/pull/23981#issuecomment-1866287852 ## CI report: * 8bac98b51ccb4a13d5a498e377f17bb86b953b1d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [hotfix] [docs] Update README.md to remove Maven [flink]
asafm opened a new pull request, #23981: URL: https://github.com/apache/flink/pull/23981 ## What is the purpose of the change Small fix to README ## Brief change log No need for pre-installed maven, since we have Maven Wrapper ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing This change is a trivial rework / code cleanup without any test coverage. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33365] include filters with Lookup joins [flink-connector-jdbc]
snuyanzin commented on code in PR #79:
URL:
https://github.com/apache/flink-connector-jdbc/pull/79#discussion_r1434090851
##
pom.xml:
##
@@ -63,6 +63,9 @@ under the License.
flink-connector-jdbc-parent
1
+
+-XX:+UseG1GC -Xms256m
-XX:+IgnoreUnrecognizedVMOptions
${surefire.module.config}
+
Review Comment:
Why is this a part of the PR?
It is already in main
I guess you need to rebase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33365] include filters with Lookup joins [flink-connector-jdbc]
snuyanzin commented on code in PR #79: URL: https://github.com/apache/flink-connector-jdbc/pull/79#discussion_r1434090595 ## flink-connector-jdbc/pom.xml: ## @@ -42,6 +42,10 @@ under the License. 21.8.0.0 418 1.12.10 + --add-opens=java.base/java.util=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED + Review Comment: Why is this a part of the PR? It is already in main I guess you need to rebase -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33365] include filters with Lookup joins [flink-connector-jdbc]
snuyanzin commented on code in PR #79:
URL:
https://github.com/apache/flink-connector-jdbc/pull/79#discussion_r1434090293
##
.github/workflows/push_pr.yml:
##
@@ -25,7 +25,14 @@ jobs:
compile_and_test:
strategy:
matrix:
-flink: [1.16-SNAPSHOT, 1.17-SNAPSHOT, 1.18-SNAPSHOT]
+flink: [ 1.16-SNAPSHOT, 1.17-SNAPSHOT ]
+jdk: [ '8, 11' ]
+include:
+ - flink: 1.18-SNAPSHOT
+jdk: '8, 11, 17'
+ - flink: 1.19-SNAPSHOT
+jdk: '8, 11, 17, 21'
Review Comment:
Why is this a part of the PR?
It is already in main
I guess you need to rebase
##
.github/workflows/weekly.yml:
##
@@ -34,6 +34,11 @@ jobs:
branch: main
}, {
flink: 1.18-SNAPSHOT,
+ jdk: '8, 11, 17',
+ branch: main
+}, {
+ flink: 1.19-SNAPSHOT,
+ jdk: '8, 11, 17, 21',
Review Comment:
Why is this a part of the PR?
It is already in main
I guess you need to rebase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33365] include filters with Lookup joins [flink-connector-jdbc]
snuyanzin commented on code in PR #79:
URL:
https://github.com/apache/flink-connector-jdbc/pull/79#discussion_r1434090293
##
.github/workflows/push_pr.yml:
##
@@ -25,7 +25,14 @@ jobs:
compile_and_test:
strategy:
matrix:
-flink: [1.16-SNAPSHOT, 1.17-SNAPSHOT, 1.18-SNAPSHOT]
+flink: [ 1.16-SNAPSHOT, 1.17-SNAPSHOT ]
+jdk: [ '8, 11' ]
+include:
+ - flink: 1.18-SNAPSHOT
+jdk: '8, 11, 17'
+ - flink: 1.19-SNAPSHOT
+jdk: '8, 11, 17, 21'
Review Comment:
Why is is this a part of the PR?
It is already in main
I guess you need to rebase
##
.github/workflows/weekly.yml:
##
@@ -34,6 +34,11 @@ jobs:
branch: main
}, {
flink: 1.18-SNAPSHOT,
+ jdk: '8, 11, 17',
+ branch: main
+}, {
+ flink: 1.19-SNAPSHOT,
+ jdk: '8, 11, 17, 21',
Review Comment:
Why is is this a part of the PR?
It is already in main
I guess you need to rebase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33365] include filters with Lookup joins [flink-connector-jdbc]
snuyanzin commented on code in PR #79:
URL:
https://github.com/apache/flink-connector-jdbc/pull/79#discussion_r1434090595
##
flink-connector-jdbc/pom.xml:
##
@@ -42,6 +42,10 @@ under the License.
21.8.0.0
418
1.12.10
+
--add-opens=java.base/java.util=ALL-UNNAMED
--add-opens=java.base/java.lang=ALL-UNNAMED
+
Review Comment:
Why is is this a part of the PR?
It is already in main
I guess you need to rebase
##
pom.xml:
##
@@ -63,6 +63,9 @@ under the License.
flink-connector-jdbc-parent
1
+
+-XX:+UseG1GC -Xms256m
-XX:+IgnoreUnrecognizedVMOptions
${surefire.module.config}
+
Review Comment:
Why is is this a part of the PR?
It is already in main
I guess you need to rebase
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
gyfora commented on code in PR #728:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/728#discussion_r1434066148
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##
@@ -264,4 +262,21 @@ private static Map
getVertexParallelismOverrides(
});
return overrides;
}
+
+private boolean blockScalingExecution(
+Context context,
+Map scalingSummaries,
+Configuration conf,
+Instant now) {
+var scaleEnabled = conf.get(SCALING_ENABLED);
+var isExcluded = CalendarUtils.inExcludedPeriods(conf, now);
+autoScalerEventHandler.handleScalingEvent(
+context,
+scalingSummaries,
+scaleEnabled,
+isExcluded,
+conf.get(SCALING_EVENT_INTERVAL));
+
+ return !scaleEnabled || isExcluded;
Review Comment:
If scaling is blocked or disabled the event that we send out is a
recommendation. Wouldn't it make sense to have uniform events for
recommendation regardless of the cause? The user always knows why it's only a
recommendation, either because the autoscaler is disabled completely or if in a
blocking period
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Cleanup usage of deprecated IdleStateRetentionTime related method in org.apache.flink.table.api.TableConfig [flink]
liyubin117 commented on PR #23980: URL: https://github.com/apache/flink/pull/23980#issuecomment-1866172964 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-33611) Support Large Protobuf Schemas
[ https://issues.apache.org/jira/browse/FLINK-33611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799400#comment-17799400 ] Benchao Li commented on FLINK-33611: I'm wondering how large will it be that can reach this limit? > Support Large Protobuf Schemas > -- > > Key: FLINK-33611 > URL: https://issues.apache.org/jira/browse/FLINK-33611 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.18.0 >Reporter: Sai Sharath Dandi >Assignee: Sai Sharath Dandi >Priority: Major > Labels: pull-request-available > > h3. Background > Flink serializes and deserializes protobuf format data by calling the decode > or encode method in GeneratedProtoToRow_XXX.java generated by codegen to > parse byte[] data into Protobuf Java objects. FLINK-32650 has introduced the > ability to split the generated code to improve the performance for large > Protobuf schemas. However, this is still not sufficient to support some > larger protobuf schemas as the generated code exceeds the java constant pool > size [limit|https://en.wikipedia.org/wiki/Java_class_file#The_constant_pool] > and we can see errors like "Too many constants" when trying to compile the > generated code. > *Solution* > Since we already have the split code functionality already introduced, the > main proposal here is to now reuse the variable names across different split > method scopes. This will greatly reduce the constant pool size. One more > optimization is to only split the last code segment also only when the size > exceeds split threshold limit. Currently, the last segment of the generated > code is always being split which can lead to too many split methods and thus > exceed the constant pool size limit -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33611] [flink-protobuf] Support Large Protobuf Schemas [flink]
libenchao commented on PR #23937: URL: https://github.com/apache/flink/pull/23937#issuecomment-1866145686 @sharath1709 Thanks for your contribution. Per the community's convention, we need to reach an agreement on the issue before we go to the PR review. I'll comment on the corresponding Jira, and let's get back here after we reach the agreement. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
mxm commented on code in PR #728:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/728#discussion_r1433991220
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/event/AutoScalerEventHandler.java:
##
@@ -83,12 +92,27 @@ default void handleScalingEvent(
}
static String scalingReport(
-Map scalingSummaries, boolean
scalingEnabled) {
-StringBuilder sb =
-new StringBuilder(
-scalingEnabled
-? SCALING_SUMMARY_HEADER_SCALING_ENABLED
-: SCALING_SUMMARY_HEADER_SCALING_DISABLED);
+Map scalingSummaries,
+boolean scalingEnabled,
+boolean isExcluded,
+Configuration config) {
+StringBuilder sb = new StringBuilder();
+if (!scalingEnabled) {
+sb.append(
+String.format(
+SCALING_SUMMARY_HEADER_SCALING_EXECUTION_DISABLED,
+SCALING_ENABLED.key(),
+false));
+} else if (isExcluded) {
+sb.append(
+String.format(
+SCALING_SUMMARY_HEADER_SCALING_EXECUTION_DISABLED,
+EXCLUDED_PERIODS.key(),
+config.get(EXCLUDED_PERIODS)));
+} else {
+sb.append(SCALING_SUMMARY_HEADER_SCALING_EXECUTION_ENABLED);
+}
Review Comment:
I also didn't like this change at first, but I think the additional code
here is justified to provide better error reporting to the user.
As an alternative, we could replace the `scalingEnabled` and
`scalingBlocked` flags via a generic String which allows to pass in any
additional explanation by the caller.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
mxm commented on code in PR #728:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/728#discussion_r1433989601
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##
@@ -264,4 +262,21 @@ private static Map
getVertexParallelismOverrides(
});
return overrides;
}
+
+private boolean blockScalingExecution(
+Context context,
+Map scalingSummaries,
+Configuration conf,
+Instant now) {
+var scaleEnabled = conf.get(SCALING_ENABLED);
+var isExcluded = CalendarUtils.inExcludedPeriods(conf, now);
+autoScalerEventHandler.handleScalingEvent(
+context,
+scalingSummaries,
+scaleEnabled,
+isExcluded,
+conf.get(SCALING_EVENT_INTERVAL));
+
+ return !scaleEnabled || isExcluded;
Review Comment:
I was contemplating to comment on this as well, but after looking at the
code, there is some benefit to hinting whether scaling is disabled or blocked.
By treating blocked scaling as "disabled", we yield less information to the
user. I think it will be beneficial to learn WHY the scaling is disabled, i.e.
scaling disabled or scaling blocked.
>The event handler really has nothing to do with this feature (and many
other things that happen in the scaling executor)
Not sure about that. Event handlers are supposed to process events and in
the context of the autoscaler, a blocked scaling decision is a type of event we
may present to the user. We are already leaking whether scaling is enabled or
not to the event handler, why not also annotate why it is disabled?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-33490) Validate the name conflicts when creating view
[
https://issues.apache.org/jira/browse/FLINK-33490?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17799398#comment-17799398
]
Benchao Li commented on FLINK-33490:
The standard states that:
{quote}i) If any two columns in the table specified by the
have equivalent s, or if any column of that table has an
implementation-dependent name, then a shall be specified.
{quote}
Hence if we specify columns list to the view, it shall not throw exception when
columns names from the query conflict. However, the current fix does not allow
it, can we improve it? [~xuyangzhong] [~fsk119]
> Validate the name conflicts when creating view
> --
>
> Key: FLINK-33490
> URL: https://issues.apache.org/jira/browse/FLINK-33490
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Affects Versions: 1.19.0
>Reporter: Shengkai Fang
>Assignee: xuyang
>Priority: Major
> Labels: pull-request-available
>
> We should forbid
> ```
> CREATE VIEW id_view AS
> SELECT id, uid AS id FROM id_table
> ```
> As the SQL standards states,
> If is specified, then:
> i) If any two columns in the table specified by the have
> equivalent s, or if any column of that table has an
> implementation-dependent name, then a shall be specified.
> ii) Equivalent s shall not be specified more than once in the
> .
> Many databases also throw exception when view name conflicts, e.g. mysql,
> postgres.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
gyfora commented on code in PR #728:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/728#discussion_r1433973975
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##
@@ -264,4 +262,21 @@ private static Map
getVertexParallelismOverrides(
});
return overrides;
}
+
+private boolean blockScalingExecution(
+Context context,
+Map scalingSummaries,
+Configuration conf,
+Instant now) {
+var scaleEnabled = conf.get(SCALING_ENABLED);
+var isExcluded = CalendarUtils.inExcludedPeriods(conf, now);
+autoScalerEventHandler.handleScalingEvent(
+context,
+scalingSummaries,
+scaleEnabled,
+isExcluded,
+conf.get(SCALING_EVENT_INTERVAL));
+
+ return !scaleEnabled || isExcluded;
Review Comment:
The event handler really has nothing to do with this feature (and many other
things that happen in the scaling executor)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] [FLINK-33795] Add a new config to forbid autoscale execution in the configured excluded periods [flink-kubernetes-operator]
gyfora commented on code in PR #728:
URL:
https://github.com/apache/flink-kubernetes-operator/pull/728#discussion_r1433973067
##
flink-autoscaler/src/main/java/org/apache/flink/autoscaler/ScalingExecutor.java:
##
@@ -264,4 +262,21 @@ private static Map
getVertexParallelismOverrides(
});
return overrides;
}
+
+private boolean blockScalingExecution(
+Context context,
+Map scalingSummaries,
+Configuration conf,
+Instant now) {
+var scaleEnabled = conf.get(SCALING_ENABLED);
+var isExcluded = CalendarUtils.inExcludedPeriods(conf, now);
+autoScalerEventHandler.handleScalingEvent(
+context,
+scalingSummaries,
+scaleEnabled,
+isExcluded,
+conf.get(SCALING_EVENT_INTERVAL));
+
+ return !scaleEnabled || isExcluded;
Review Comment:
I am not a big fan of scattering features across many interfaces without a
very good reason. It makes the event handler and the whole logic more complex
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Re: [PR] Cleanup usage of deprecated IdleStateRetentionTime related method in org.apache.flink.table.api.TableConfig [flink]
flinkbot commented on PR #23980: URL: https://github.com/apache/flink/pull/23980#issuecomment-1866115231 ## CI report: * 620ea94f2829ebc8c55ab0657261c65f6881054b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-33925) Extended failure handling for bulk requests (elasticsearch back port)
[ https://issues.apache.org/jira/browse/FLINK-33925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-33925: --- Labels: pull-request-available (was: ) > Extended failure handling for bulk requests (elasticsearch back port) > - > > Key: FLINK-33925 > URL: https://issues.apache.org/jira/browse/FLINK-33925 > Project: Flink > Issue Type: Improvement > Components: Connectors / Opensearch >Affects Versions: opensearch-1.0.1 >Reporter: Peter Schulz >Priority: Major > Labels: pull-request-available > > This is a back port of the implementation for the elasticsearch connector, > see FLINK-32028, to achieve consistent APIs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [PR] [FLINK-33925][connectors/opensearch] Allow customising bulk failure handling [flink-connector-opensearch]
boring-cyborg[bot] commented on PR #39: URL: https://github.com/apache/flink-connector-opensearch/pull/39#issuecomment-1866114172 Thanks for opening this pull request! Please check out our contributing guidelines. (https://flink.apache.org/contributing/how-to-contribute.html) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[PR] [FLINK-33925][connectors/opensearch] Allow customising bulk failure handling [flink-connector-opensearch]
schulzp opened a new pull request, #39: URL: https://github.com/apache/flink-connector-opensearch/pull/39 This is a back port of the implementation for the elasticsearch connector, see [FLINK-32028](https://issues.apache.org/jira/browse/FLINK-32028), to achieve consistent APIs. Extracted `BulkResponseInspector` interface to allow custom handling of (partially) failed bulk requests. If not overridden, default behaviour remains unchanged and partial failures are escalated. * fixes [FLINK-33925](https://issues.apache.org/jira/browse/FLINK-33925) * allows custom metrics to be exposed -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-33925) Extended failure handling for bulk requests (elasticsearch back port)
Peter Schulz created FLINK-33925: Summary: Extended failure handling for bulk requests (elasticsearch back port) Key: FLINK-33925 URL: https://issues.apache.org/jira/browse/FLINK-33925 Project: Flink Issue Type: Improvement Components: Connectors / Opensearch Affects Versions: opensearch-1.0.1 Reporter: Peter Schulz This is a back port of the implementation for the elasticsearch connector, see FLINK-32028, to achieve consistent APIs. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[PR] Cleanup usage of deprecated IdleStateRetentionTime related method in org.apache.flink.table.api.TableConfig [flink]
liyubin117 opened a new pull request, #23980: URL: https://github.com/apache/flink/pull/23980 ## What is the purpose of the change Cleanup the usage of deprecated org.apache.flink.table.api.TableConfig#getMinIdleStateRetentionTime, org.apache.flink.table.api.TableConfig#getMaxIdleStateRetentionTime. ## Brief change log use new api. ## Verifying this change no need additional verify besides CI ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? no -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [BP-1.18][FLINK-33902][ci] Adds -legacy to openssl command [flink]
flinkbot commented on PR #23979: URL: https://github.com/apache/flink/pull/23979#issuecomment-1866078496 ## CI report: * 7d32f61389784a833100d065c17ffd2fca753eb4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
Re: [PR] [FLINK-33902][ci] Adds -legacy to openssl command [flink]
XComp commented on PR #23961: URL: https://github.com/apache/flink/pull/23961#issuecomment-1866076157 Thanks for approving the PR. I created the corresponding 1.18 backport. :+1: -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
