[GitHub] [flink] flinkbot edited a comment on pull request #17562: [FLINK-16206][table-planner] Support JSON_ARRAYAGG
flinkbot edited a comment on pull request #17562: URL: https://github.com/apache/flink/pull/17562#issuecomment-951200622 ## CI report: * 7a90ad8e577b79b3f68fbed12a5824f6b822c129 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25434) * dc21e87ad1ed66a9ce41d2a93e52bd5a7108bc82 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25498) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Airblader commented on a change in pull request #17562: [FLINK-16206][table-planner] Support JSON_ARRAYAGG
Airblader commented on a change in pull request #17562:
URL: https://github.com/apache/flink/pull/17562#discussion_r737159905
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/functions/aggfunctions/JsonArrayAggFunction.java
##
@@ -0,0 +1,182 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.planner.functions.aggfunctions;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.table.api.DataTypes;
+import org.apache.flink.table.api.TableException;
+import org.apache.flink.table.api.dataview.ListView;
+import org.apache.flink.table.data.StringData;
+import org.apache.flink.table.functions.BuiltInFunctionDefinitions;
+import
org.apache.flink.table.planner.plan.rules.logical.WrapJsonAggFunctionArgumentsRule;
+import
org.apache.flink.table.runtime.functions.aggregate.BuiltInAggregateFunction;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.LogicalType;
+import org.apache.flink.table.types.utils.DataTypeUtils;
+
+import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.JsonNode;
+import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.node.ArrayNode;
+import
org.apache.flink.shaded.jackson2.com.fasterxml.jackson.databind.util.RawValue;
+
+import org.apache.calcite.sql.SqlJsonConstructorNullClause;
+
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.stream.Collectors;
+
+import static
org.apache.flink.table.runtime.functions.SqlJsonUtils.createArrayNode;
+import static
org.apache.flink.table.runtime.functions.SqlJsonUtils.getNodeFactory;
+import static
org.apache.flink.table.runtime.functions.SqlJsonUtils.serializeJson;
+
+/**
+ * Implementation for {@link
BuiltInFunctionDefinitions#JSON_ARRAYAGG_ABSENT_ON_NULL} / {@link
+ * BuiltInFunctionDefinitions#JSON_ARRAYAGG_NULL_ON_NULL}.
+ *
+ * Note that this function only ever receives strings to accumulate because
{@link
+ * WrapJsonAggFunctionArgumentsRule} wraps arguments into {@link
+ * BuiltInFunctionDefinitions#JSON_STRING}.
+ */
+@Internal
+public class JsonArrayAggFunction
+extends BuiltInAggregateFunction {
+
+private static final long serialVersionUID = 1L;
+
+/**
+ * Marker that represents a {@code null} since {@link ListView} does not
allow {@code null}s.
+ *
+ * Note that due to {@link WrapJsonAggFunctionArgumentsRule} and the
fact that this function
+ * already only receives JSON strings, this value cannot be created by the
user and is thus safe
+ * to use.
+ */
+private static final StringData NULL_STR = StringData.fromString("null");
+
+private final transient List argumentTypes;
+private final SqlJsonConstructorNullClause onNull;
+
+public JsonArrayAggFunction(LogicalType[] argumentTypes,
SqlJsonConstructorNullClause onNull) {
+this.argumentTypes =
+Arrays.stream(argumentTypes)
+.map(DataTypeUtils::toInternalDataType)
+.collect(Collectors.toList());
+
+this.onNull = onNull;
+}
+
+@Override
+public List getArgumentDataTypes() {
+return argumentTypes;
+}
+
+@Override
+public DataType getOutputDataType() {
+return DataTypes.STRING();
+}
+
+@Override
+public DataType getAccumulatorDataType() {
+return DataTypes.STRUCTURED(
+Accumulator.class,
+DataTypes.FIELD("list",
ListView.newListViewDataType(DataTypes.STRING(;
+}
+
+@Override
+public Accumulator createAccumulator() {
+return new Accumulator();
+}
+
+public void resetAccumulator(Accumulator acc) {
+acc.list.clear();
+}
+
+public void accumulate(Accumulator acc, StringData itemData) throws
Exception {
+if (itemData == null) {
+switch (onNull) {
+case NULL_ON_NULL:
+acc.list.add(NULL_STR);
+break;
+case ABSENT_ON_NULL:
+break;
+default:
+throw new TableException(
+
[jira] [Created] (FLINK-24664) Support merge for JSON_ARRAYAGG
Ingo Bürk created FLINK-24664: - Summary: Support merge for JSON_ARRAYAGG Key: FLINK-24664 URL: https://issues.apache.org/jira/browse/FLINK-24664 Project: Flink Issue Type: Sub-task Components: Table SQL / API Reporter: Ingo Bürk For JSON_ARRAYAGG we currently do not support merge (required e.g. for hop windows) because arrays are order-sensitive, and we cannot ensure the correct order. There are a few ways we could solve this: * Making use of WITHIN GROUP syntax, which is supported by Calcite, but not yet for JSON_ARRAYAGG in the parser. * Allowing incorrect behavior, but clearly documenting this case. In this case we should at least try to find a way to have a deterministic order of the accumulators to be merged as to not make the result non-deterministic. * Try to actually solve this somehow such that behavior is correct. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17562: [FLINK-16206][table-planner] Support JSON_ARRAYAGG
flinkbot edited a comment on pull request #17562: URL: https://github.com/apache/flink/pull/17562#issuecomment-951200622 ## CI report: * 7a90ad8e577b79b3f68fbed12a5824f6b822c129 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25434) * dc21e87ad1ed66a9ce41d2a93e52bd5a7108bc82 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24663) PyFlink failed to get the site packege path because of SyntaxError in shell command
[
https://issues.apache.org/jira/browse/FLINK-24663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jackwangcs updated FLINK-24663:
---
Summary: PyFlink failed to get the site packege path because of SyntaxError
in shell command (was: PyFlink failed to get the site packege path because of
no quote in shell command)
> PyFlink failed to get the site packege path because of SyntaxError in shell
> command
> ---
>
> Key: FLINK-24663
> URL: https://issues.apache.org/jira/browse/FLINK-24663
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.12.3
>Reporter: jackwangcs
>Priority: Major
>
> Flink throws an exception when it tries to install 3rd party dependencies:
> {code:java}
> Caused by: java.io.IOException: Failed to execute the command: python -c
> import sys;from distutils.dist import Distribution;install_obj =
> Distribution().get_command_obj('install', create=True);install_obj.prefix =
> sys.argv[1];install_obj.finalize_options();installed_dir =
> [install_obj.install_purelib];install_obj.install_purelib !=
> install_obj.install_platlib and
> installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> > 1 and print(installed_dir[1])
> /mnt/yarn/usercache/hadoop/appcache/application_1629776785656_0100/python-dist-fb549cea-0857-4b11-9eb7-7818eaa3f561/python-requirements
> output: File "", line 1
> import sys;from distutils.dist import Distribution;install_obj =
> Distribution().get_command_obj('install', create=True);install_obj.prefix =
> sys.argv[1];install_obj.finalize_options();installed_dir =
> [install_obj.install_purelib];install_obj.install_purelib !=
> install_obj.install_platlib and
> installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> > 1 and print(installed_dir[1])
>
>
>
>
> ^SyntaxError: invalid syntax
> at
> org.apache.flink.python.util.PythonEnvironmentManagerUtils.execute(PythonEnvironmentManagerUtils.java:211)
> at
> org.apache.flink.python.util.PythonEnvironmentManagerUtils.getSitePackagesPath(PythonEnvironmentManagerUtils.java:171)
> at
> org.apache.flink.python.util.PythonEnvironmentManagerUtils.pipInstallRequirements(PythonEnvironmentManagerUtils.java:99)
> at
> org.apache.flink.python.env.beam.ProcessPythonEnvironmentManager.createEnvironment(ProcessPythonEnvironmentManager.java:169)
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.createPythonExecutionEnvironment(BeamPythonFunctionRunner.java:339)
> {code}
> This can be reproduced by running the python script in a bash shell:
> {code:java}
> python3 -c import sys;from distutils.dist import Distribution;install_obj =
> Distribution().get_command_obj('install',
> create=True);print(sys.argv[1]);install_obj.prefix =
> sys.argv[1];install_obj.finalize_options();installed_dir =
> [install_obj.install_purelib];install_obj.install_purelib !=
> install_obj.install_platlib and
> installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> > 1 and print(installed_dir[1]) /tmp/requirements
> -bash: syntax error near unexpected token `(' {code}
> The solution is to quote all argements to execute:
> {code:java}
> python3 "-c" "import sys;from distutils.dist import Distribution;install_obj
> = Distribution().get_command_obj('install',
> create=True);print(sys.argv[1]);install_obj.prefix =
> sys.argv[1];install_obj.finalize_options();installed_dir =
> [install_obj.install_purelib];install_obj.install_purelib !=
> install_obj.install_platlib and
> installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> > 1 and print(installed_dir[1])" "/tmp/requirements"
> /tmp/requirements
> /tmp/requirements/lib/python3.6/site-packages{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24663) PyFlink failed to get the site packege path because of no quote in shell command
[
https://issues.apache.org/jira/browse/FLINK-24663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jackwangcs updated FLINK-24663:
---
Description:
Flink throws an exception when it tries to install 3rd party dependencies:
{code:java}
Caused by: java.io.IOException: Failed to execute the command: python -c import
sys;from distutils.dist import Distribution;install_obj =
Distribution().get_command_obj('install', create=True);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1])
/mnt/yarn/usercache/hadoop/appcache/application_1629776785656_0100/python-dist-fb549cea-0857-4b11-9eb7-7818eaa3f561/python-requirements
output: File "", line 1
import sys;from distutils.dist import Distribution;install_obj =
Distribution().get_command_obj('install', create=True);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1])
^SyntaxError: invalid syntax
at
org.apache.flink.python.util.PythonEnvironmentManagerUtils.execute(PythonEnvironmentManagerUtils.java:211)
at
org.apache.flink.python.util.PythonEnvironmentManagerUtils.getSitePackagesPath(PythonEnvironmentManagerUtils.java:171)
at
org.apache.flink.python.util.PythonEnvironmentManagerUtils.pipInstallRequirements(PythonEnvironmentManagerUtils.java:99)
at
org.apache.flink.python.env.beam.ProcessPythonEnvironmentManager.createEnvironment(ProcessPythonEnvironmentManager.java:169)
at
org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.createPythonExecutionEnvironment(BeamPythonFunctionRunner.java:339)
{code}
This can be reproduced by running the python script in a bash shell:
{code:java}
python3 -c import sys;from distutils.dist import Distribution;install_obj =
Distribution().get_command_obj('install',
create=True);print(sys.argv[1]);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1]) /tmp/requirements
-bash: syntax error near unexpected token `(' {code}
The solution is to quote all argements to execute:
{code:java}
python3 "-c" "import sys;from distutils.dist import Distribution;install_obj =
Distribution().get_command_obj('install',
create=True);print(sys.argv[1]);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1])" "/tmp/requirements"
/tmp/requirements
/tmp/requirements/lib/python3.6/site-packages{code}
was:
Flink throws an exception when it tries to install 3rd party dependencies:
{code:java}
Caused by: java.io.IOException: Failed to execute the command: python -c import
sys;from distutils.dist import Distribution;install_obj =
Distribution().get_command_obj('install', create=True);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1])
/mnt/yarn/usercache/hadoop/appcache/application_1629776785656_0100/python-dist-fb549cea-0857-4b11-9eb7-7818eaa3f561/python-requirementsCaused
by: java.io.IOException: Failed to execute the command: python -c import
sys;from distutils.dist import Distribution;install_obj =
Distribution().get_command_obj('install', create=True);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1])
/mnt/yarn/usercache/hadoop/appcache/application_1629776785656_0100/python-di
[GitHub] [flink] flinkbot edited a comment on pull request #17549: [FLINK-16205][table-planner] Support JSON_OBJECTAGG
flinkbot edited a comment on pull request #17549: URL: https://github.com/apache/flink/pull/17549#issuecomment-949678124 ## CI report: * 5ea492bfda82c87d79bf7ab6fb999f65e8c872ca Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25435) * 284794c4156ff18a56b3bf823882abf775d9fd36 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25497) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21068) Add new timeout options for Elasticsearch connector
[
https://issues.apache.org/jira/browse/FLINK-21068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434690#comment-17434690
]
Fabian Paul commented on FLINK-21068:
-
[~henriquemota] I think we can support the feature with Flink 1.15.
> Add new timeout options for Elasticsearch connector
> ---
>
> Key: FLINK-21068
> URL: https://issues.apache.org/jira/browse/FLINK-21068
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / ElasticSearch
>Affects Versions: 1.12.1
>Reporter: jinfeng
>Priority: Minor
> Labels: auto-deprioritized-major
>
> Currently, the connection.max-retry-timeout seems not work with new
> elasticsearch connector. The elasticsearch community has Remove
> setMaxRetryTimeoutMillis from RestClientBuilder. We can set timeout options
> when create RestHighLevelClient in
> Elasticsearch7ApiCallBridge , like
> {code:java}
> //代码占位符
> @Override
> public RestHighLevelClient createClient(Map clientConfig)
> throws IOException {
>RestClientBuilder builder = RestClient.builder(httpHosts.toArray(new
> HttpHost[httpHosts.size()]));
>builder.setRequestConfigCallback(new
> RestClientBuilder.RequestConfigCallback() {
> @Override
> public RequestConfig.Builder
> customizeRequestConfig(RequestConfig.Builder builder) {
> if (clientConfig.containsKey(CONFIG_KEY_CONNECTION_TIMEOUT)) {
>
> builder.setConnectTimeout(Integer.valueOf(clientConfig.get(CONFIG_KEY_CONNECTION_TIMEOUT)));
> }
> if (clientConfig.containsKey(CONFIG_KEY_CONNECTION_SOCKET_TIMEOUT)) {
>
> builder.setSocketTimeout(Integer.valueOf(clientConfig.get(CONFIG_KEY_CONNECTION_SOCKET_TIMEOUT)));
> }
> if (clientConfig.containsKey(CONFIG_KEY_CONNECTION_REQUEST_TIMEOUT))
> {
>
> builder.setConnectionRequestTimeout(Integer.valueOf(clientConfig.get(CONFIG_KEY_CONNECTION_REQUEST_TIMEOUT)));
> }
> return builder;
> }
>});
> {code}
>
> So, we can add three table config to config eleasticsearch timeout.
> connection.timeout
> connection.socket-timeout
> connection.request-timeout
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-24663) PyFlink failed to get the site packege path because of no quote in shell command
[
https://issues.apache.org/jira/browse/FLINK-24663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jackwangcs updated FLINK-24663:
---
Description:
Flink throws an exception when it tries to install 3rd party dependencies:
{code:java}
Caused by: java.io.IOException: Failed to execute the command: python -c import
sys;from distutils.dist import Distribution;install_obj =
Distribution().get_command_obj('install', create=True);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1])
/mnt/yarn/usercache/hadoop/appcache/application_1629776785656_0100/python-dist-fb549cea-0857-4b11-9eb7-7818eaa3f561/python-requirementsCaused
by: java.io.IOException: Failed to execute the command: python -c import
sys;from distutils.dist import Distribution;install_obj =
Distribution().get_command_obj('install', create=True);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1])
/mnt/yarn/usercache/hadoop/appcache/application_1629776785656_0100/python-dist-fb549cea-0857-4b11-9eb7-7818eaa3f561/python-requirementsoutput:
File "", line 1 import sys;from distutils.dist import
Distribution;install_obj = Distribution().get_command_obj('install',
create=True);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1])
^SyntaxError: invalid syntax
at
org.apache.flink.python.util.PythonEnvironmentManagerUtils.execute(PythonEnvironmentManagerUtils.java:211)
at
org.apache.flink.python.util.PythonEnvironmentManagerUtils.getSitePackagesPath(PythonEnvironmentManagerUtils.java:171)
at
org.apache.flink.python.util.PythonEnvironmentManagerUtils.pipInstallRequirements(PythonEnvironmentManagerUtils.java:99)
at
org.apache.flink.python.env.beam.ProcessPythonEnvironmentManager.createEnvironment(ProcessPythonEnvironmentManager.java:169)
at
org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.createPythonExecutionEnvironment(BeamPythonFunctionRunner.java:339)
{code}
This can be reproduced by running the python script in a bash shell:
{code:java}
python3 -c import sys;from distutils.dist import Distribution;install_obj =
Distribution().get_command_obj('install',
create=True);print(sys.argv[1]);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1]) /tmp/requirements
-bash: syntax error near unexpected token `(' {code}
The solution is to quota all argements to execute:
{code:java}
python3 "-c" "import sys;from distutils.dist import Distribution;install_obj =
Distribution().get_command_obj('install',
create=True);print(sys.argv[1]);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1])" "/tmp/requirements"
/tmp/requirements
/tmp/requirements/lib/python3.6/site-packages{code}
was:
Flink throws an exception when it tries to install 3rd party dependencies:
{code:java}
Caused by: java.io.IOException: Failed to execute the command: python -c import
sys;from distutils.dist import Distribution;install_obj =
Distribution().get_command_obj('install', create=True);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1])
/mnt/yarn/usercache/hadoop/appcache/application_1629776785656_0100/python-di
[GitHub] [flink] Airblader commented on pull request #17549: [FLINK-16205][table-planner] Support JSON_OBJECTAGG
Airblader commented on pull request #17549:
URL: https://github.com/apache/flink/pull/17549#issuecomment-952588166
Should work now:
```
++
| count |
++
|{"Hello":1} |
| {"Hello2":2} |
| {"Ciao":1} |
++
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17549: [FLINK-16205][table-planner] Support JSON_OBJECTAGG
flinkbot edited a comment on pull request #17549: URL: https://github.com/apache/flink/pull/17549#issuecomment-949678124 ## CI report: * 5ea492bfda82c87d79bf7ab6fb999f65e8c872ca Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25435) * 284794c4156ff18a56b3bf823882abf775d9fd36 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-24663) PyFlink failed to get the site packege path because of no quote in shell command
jackwangcs created FLINK-24663:
--
Summary: PyFlink failed to get the site packege path because of no
quote in shell command
Key: FLINK-24663
URL: https://issues.apache.org/jira/browse/FLINK-24663
Project: Flink
Issue Type: Bug
Components: API / Python
Affects Versions: 1.12.3
Reporter: jackwangcs
Flink throws an exception when it tries to install 3rd party dependencies:
{code:java}
Caused by: java.io.IOException: Failed to execute the command: python -c import
sys;from distutils.dist import Distribution;install_obj =
Distribution().get_command_obj('install', create=True);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1])
/mnt/yarn/usercache/hadoop/appcache/application_1629776785656_0100/python-dist-fb549cea-0857-4b11-9eb7-7818eaa3f561/python-requirementsCaused
by: java.io.IOException: Failed to execute the command: python -c import
sys;from distutils.dist import Distribution;install_obj =
Distribution().get_command_obj('install', create=True);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1])
/mnt/yarn/usercache/hadoop/appcache/application_1629776785656_0100/python-dist-fb549cea-0857-4b11-9eb7-7818eaa3f561/python-requirementsoutput:
File "", line 1 import sys;from distutils.dist import
Distribution;install_obj = Distribution().get_command_obj('install',
create=True);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1])
^SyntaxError: invalid syntax
at
org.apache.flink.python.util.PythonEnvironmentManagerUtils.execute(PythonEnvironmentManagerUtils.java:211)
at
org.apache.flink.python.util.PythonEnvironmentManagerUtils.getSitePackagesPath(PythonEnvironmentManagerUtils.java:171)
at
org.apache.flink.python.util.PythonEnvironmentManagerUtils.pipInstallRequirements(PythonEnvironmentManagerUtils.java:99)
at
org.apache.flink.python.env.beam.ProcessPythonEnvironmentManager.createEnvironment(ProcessPythonEnvironmentManager.java:169)
at
org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.createPythonExecutionEnvironment(BeamPythonFunctionRunner.java:339)
{code}
This can be reproduced by running the python script in a bash shell:
{code:java}
python3 -c import sys;from distutils.dist import Distribution;install_obj =
Distribution().get_command_obj('install',
create=True);print(sys.argv[1]);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1]) /tmp/requirements
-bash: syntax error near unexpected token `(' {code}
The solution is to quota all argements to execute:
{code:java}
python3 -c "import sys;from distutils.dist import Distribution;install_obj =
Distribution().get_command_obj('install',
create=True);print(sys.argv[1]);install_obj.prefix =
sys.argv[1];install_obj.finalize_options();installed_dir =
[install_obj.install_purelib];install_obj.install_purelib !=
install_obj.install_platlib and
installed_dir.append(install_obj.install_platlib);print(installed_dir[0]);len(installed_dir)
> 1 and print(installed_dir[1])" "/tmp/requirements"
/tmp/requirements
/tmp/requirements/lib/python3.6/site-packages {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17556: [FLINK-24627][tests] add some Junit5 extensions to replace the existed Junit4 rules
flinkbot edited a comment on pull request #17556: URL: https://github.com/apache/flink/pull/17556#issuecomment-950605426 ## CI report: * 3cd20469f38e3f4e11e7e51790c7eaa2c4d74c78 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25486) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17520: [FLINK-24565][avro] Port avro file format factory to BulkReaderFormatFactory
flinkbot edited a comment on pull request #17520: URL: https://github.com/apache/flink/pull/17520#issuecomment-946592172 ## CI report: * de579d6e83aa94b92dc583c79c34ac085fb4b912 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25485) * afbf2856adea77d035d6a225662d1893c7a7f13e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25496) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-24661) ConfigOption add isSecret method to judge sensitive options
[ https://issues.apache.org/jira/browse/FLINK-24661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ada Wong updated FLINK-24661: - Description: Related ticket https://issues.apache.org/jira/browse/FLINK-24381 [~chesnay] Should this ticket modify connectors for example ES and JDBC password option, or just only modify core module not connector modules? was: Related ticket https://issues.apache.org/jira/browse/FLINK-24381 [~chesnay] This ticket modify connectors for example ES and JDBC password option, or only modify core module not connector modules? > ConfigOption add isSecret method to judge sensitive options > --- > > Key: FLINK-24661 > URL: https://issues.apache.org/jira/browse/FLINK-24661 > Project: Flink > Issue Type: Improvement > Components: API / Core >Affects Versions: 1.13.3 >Reporter: Ada Wong >Priority: Major > > Related ticket https://issues.apache.org/jira/browse/FLINK-24381 > [~chesnay] > Should this ticket modify connectors for example ES and JDBC password option, > or just only modify core module not connector modules? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] Airblader commented on a change in pull request #17549: [FLINK-16205][table-planner] Support JSON_OBJECTAGG
Airblader commented on a change in pull request #17549:
URL: https://github.com/apache/flink/pull/17549#discussion_r737143328
##
File path: docs/data/sql_functions.yml
##
@@ -766,6 +766,27 @@ json:
)
)
```
+ - sql: JSON_OBJECTAGG([KEY] key VALUE value [ { NULL | ABSENT } ON NULL ])
+table: jsonObjectAgg(JsonOnNull, keyExpression, valueExpression)
+description: |
+ Builds a JSON object string by aggregating over key-value expressions.
Review comment:
I'll rephrase the headline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17520: [FLINK-24565][avro] Port avro file format factory to BulkReaderFormatFactory
flinkbot edited a comment on pull request #17520: URL: https://github.com/apache/flink/pull/17520#issuecomment-946592172 ## CI report: * de579d6e83aa94b92dc583c79c34ac085fb4b912 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25485) * afbf2856adea77d035d6a225662d1893c7a7f13e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17569: [FLINK-24648] CEIL and FLOOR should support DECADE, CENTURY, MILLENNIUM
flinkbot edited a comment on pull request #17569: URL: https://github.com/apache/flink/pull/17569#issuecomment-951868685 ## CI report: * 2730f733c2cd70dea5c1c9862e1a016ecd9705e2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25467) * 8d34e32482fd19e3e0bb1e64af793ce5d3138b98 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25495) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #17537: [FLINK-19722][table-planner]Pushdown Watermark to SourceProvider (new Source)
JingsongLi commented on a change in pull request #17537:
URL: https://github.com/apache/flink/pull/17537#discussion_r737139591
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecTableSourceScan.java
##
@@ -97,9 +104,40 @@ public DynamicTableSourceSpec getTableSourceSpec() {
return createInputFormatTransformation(env, inputFormat,
outputTypeInfo, operatorName);
} else if (provider instanceof SourceProvider) {
Source source = ((SourceProvider)
provider).createSource();
-// TODO: Push down watermark strategy to source scan
-return env.fromSource(
-source, WatermarkStrategy.noWatermarks(),
operatorName, outputTypeInfo)
+// don't use rowTypes from CatalogTable
+// because the rowType number may be reduced by ProjectionPushDown
+RowType sourceRowType = outputTypeInfo.toRowType();
+SourceAbilityContext sourceAbilityContext =
+new SourceAbilityContext(planner.getFlinkContext(),
sourceRowType);
+
+WatermarkStrategy watermarkStrategy = null;
+
+if (tableSource instanceof SupportsWatermarkPushDown) {
Review comment:
If DDL don't declare watermark, we can just use `fromSource(Source,
noWatermarks)` to create `DataStreamSource`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-24661) ConfigOption add isSecret method to judge sensitive options
[ https://issues.apache.org/jira/browse/FLINK-24661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ada Wong updated FLINK-24661: - Description: Related ticket https://issues.apache.org/jira/browse/FLINK-24381 [~chesnay] This ticket modify connectors for example ES and JDBC password option, or only modify core module not connector modules? was: Related ticket https://issues.apache.org/jira/browse/FLINK-24381 [~chesnay] > ConfigOption add isSecret method to judge sensitive options > --- > > Key: FLINK-24661 > URL: https://issues.apache.org/jira/browse/FLINK-24661 > Project: Flink > Issue Type: Improvement > Components: API / Core >Affects Versions: 1.13.3 >Reporter: Ada Wong >Priority: Major > > Related ticket https://issues.apache.org/jira/browse/FLINK-24381 > [~chesnay] > This ticket modify connectors for example ES and JDBC password option, or > only modify core module not connector modules? > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17569: [FLINK-24648] CEIL and FLOOR should support DECADE, CENTURY, MILLENNIUM
flinkbot edited a comment on pull request #17569: URL: https://github.com/apache/flink/pull/17569#issuecomment-951868685 ## CI report: * 2730f733c2cd70dea5c1c9862e1a016ecd9705e2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25467) * 8d34e32482fd19e3e0bb1e64af793ce5d3138b98 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-23914) Make connector testing framework more verbose on test failure
[ https://issues.apache.org/jira/browse/FLINK-23914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434677#comment-17434677 ] Arvid Heise commented on FLINK-23914: - Merged into master as 27ec51c56ad23217b162c458d2878fe2af4e963f..db1df307f8240ff7cd495978556c0efdf4247e94. > Make connector testing framework more verbose on test failure > - > > Key: FLINK-23914 > URL: https://issues.apache.org/jira/browse/FLINK-23914 > Project: Flink > Issue Type: Improvement > Components: Tests >Affects Versions: 1.14.0 >Reporter: Qingsheng Ren >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > Currently testing framework doesn't provide enough debugging if test case > fails. We need to add more logs on test failure to reveal more information > for debugging. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] AHeise commented on pull request #16940: [FLINK-23914][connector/testing-framework] Improve logging and error message in testing framework
AHeise commented on pull request #16940: URL: https://github.com/apache/flink/pull/16940#issuecomment-952569674 Thank you. Merged. Can you create a backport PR? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] AHeise merged pull request #16940: [FLINK-23914][connector/testing-framework] Improve logging and error message in testing framework
AHeise merged pull request #16940: URL: https://github.com/apache/flink/pull/16940 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14891: [FLINK-21289][deployment] FIX missing load pipeline.classpaths in app…
flinkbot edited a comment on pull request #14891: URL: https://github.com/apache/flink/pull/14891#issuecomment-774628149 ## CI report: * 150a5be15d8605d4fe2214e2e02a97d9bd73bdf7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13637) * d520fe1077ed24965f8aa005113d2b81d5de02f1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25493) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] PChou commented on pull request #14891: [FLINK-21289][deployment] FIX missing load pipeline.classpaths in app…
PChou commented on pull request #14891: URL: https://github.com/apache/flink/pull/14891#issuecomment-952562549 `ClassPathPackagedProgramRetriever` is replaced by `DefaultPackagedProgramRetriever` on master branch, and `DefaultPackagedProgramRetriever` has been support `Configuration` as parameter. So the last commit just change `DefaultPackagedProgramRetriever` to fix the problem. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14891: [FLINK-21289][deployment] FIX missing load pipeline.classpaths in app…
flinkbot edited a comment on pull request #14891: URL: https://github.com/apache/flink/pull/14891#issuecomment-774628149 ## CI report: * 150a5be15d8605d4fe2214e2e02a97d9bd73bdf7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=13637) * d520fe1077ed24965f8aa005113d2b81d5de02f1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17463: [Hotfix] Fix typos.
flinkbot edited a comment on pull request #17463: URL: https://github.com/apache/flink/pull/17463#issuecomment-942081615 ## CI report: * 3dd575059ec3e136b282f1bfb5c7fd538fe40b87 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25483) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xuyangzhong commented on a change in pull request #17537: [FLINK-19722][table-planner]Pushdown Watermark to SourceProvider (new Source)
xuyangzhong commented on a change in pull request #17537:
URL: https://github.com/apache/flink/pull/17537#discussion_r737116940
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecTableSourceScan.java
##
@@ -97,9 +104,40 @@ public DynamicTableSourceSpec getTableSourceSpec() {
return createInputFormatTransformation(env, inputFormat,
outputTypeInfo, operatorName);
} else if (provider instanceof SourceProvider) {
Source source = ((SourceProvider)
provider).createSource();
-// TODO: Push down watermark strategy to source scan
-return env.fromSource(
-source, WatermarkStrategy.noWatermarks(),
operatorName, outputTypeInfo)
+// don't use rowTypes from CatalogTable
+// because the rowType number may be reduced by ProjectionPushDown
+RowType sourceRowType = outputTypeInfo.toRowType();
+SourceAbilityContext sourceAbilityContext =
+new SourceAbilityContext(planner.getFlinkContext(),
sourceRowType);
+
+WatermarkStrategy watermarkStrategy = null;
+
+if (tableSource instanceof SupportsWatermarkPushDown) {
Review comment:
One case: if we don't declare a watermark, the source should pushdown
noWatermarks?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17555: [FLINK-24381][table] Add support for setting Options sensitive
flinkbot edited a comment on pull request #17555: URL: https://github.com/apache/flink/pull/17555#issuecomment-950569581 ## CI report: * 907d080f302fcd5851afb178e981a122de09bbb5 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25484) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #17537: [FLINK-19722][table-planner]Pushdown Watermark to SourceProvider (new Source)
JingsongLi commented on a change in pull request #17537:
URL: https://github.com/apache/flink/pull/17537#discussion_r737114021
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecTableSourceScan.java
##
@@ -97,9 +104,40 @@ public DynamicTableSourceSpec getTableSourceSpec() {
return createInputFormatTransformation(env, inputFormat,
outputTypeInfo, operatorName);
} else if (provider instanceof SourceProvider) {
Source source = ((SourceProvider)
provider).createSource();
-// TODO: Push down watermark strategy to source scan
-return env.fromSource(
-source, WatermarkStrategy.noWatermarks(),
operatorName, outputTypeInfo)
+// don't use rowTypes from CatalogTable
+// because the rowType number may be reduced by ProjectionPushDown
+RowType sourceRowType = outputTypeInfo.toRowType();
+SourceAbilityContext sourceAbilityContext =
+new SourceAbilityContext(planner.getFlinkContext(),
sourceRowType);
+
+WatermarkStrategy watermarkStrategy = null;
+
+if (tableSource instanceof SupportsWatermarkPushDown) {
Review comment:
Can we always push down watermark to new source? Is there a semantic or
implementation problem?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] xuyangzhong commented on a change in pull request #17537: [FLINK-19722][table-planner]Pushdown Watermark to SourceProvider (new Source)
xuyangzhong commented on a change in pull request #17537:
URL: https://github.com/apache/flink/pull/17537#discussion_r737111607
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecTableSourceScan.java
##
@@ -97,9 +104,40 @@ public DynamicTableSourceSpec getTableSourceSpec() {
return createInputFormatTransformation(env, inputFormat,
outputTypeInfo, operatorName);
} else if (provider instanceof SourceProvider) {
Source source = ((SourceProvider)
provider).createSource();
-// TODO: Push down watermark strategy to source scan
-return env.fromSource(
-source, WatermarkStrategy.noWatermarks(),
operatorName, outputTypeInfo)
+// don't use rowTypes from CatalogTable
+// because the rowType number may be reduced by ProjectionPushDown
+RowType sourceRowType = outputTypeInfo.toRowType();
+SourceAbilityContext sourceAbilityContext =
+new SourceAbilityContext(planner.getFlinkContext(),
sourceRowType);
+
+WatermarkStrategy watermarkStrategy = null;
+
+if (tableSource instanceof SupportsWatermarkPushDown) {
Review comment:
The one way to get the watermark strategy is to generate strategy by
watermarkExpr in watermarkPushdownSpec, in which the "applyWatermark" is
useless actually because we always regenerate strategy. And the code now is
with this style.
The another way I think better is that since the watermark strategy is in
the dynamicTableSource now and we can add a function named
"getWatermarkStrategy" to get it directly. But before I didn't do this plan
because I think the "get" function should be added to the implement
SupportsWatermarkPushDown, and it will change the implement's current style and
cause more widely influence.
Maybe I'd better do the second way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] xuyangzhong commented on a change in pull request #17537: [FLINK-19722][table-planner]Pushdown Watermark to SourceProvider (new Source)
xuyangzhong commented on a change in pull request #17537:
URL: https://github.com/apache/flink/pull/17537#discussion_r737105753
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecTableSourceScan.java
##
@@ -97,9 +104,40 @@ public DynamicTableSourceSpec getTableSourceSpec() {
return createInputFormatTransformation(env, inputFormat,
outputTypeInfo, operatorName);
} else if (provider instanceof SourceProvider) {
Source source = ((SourceProvider)
provider).createSource();
-// TODO: Push down watermark strategy to source scan
-return env.fromSource(
-source, WatermarkStrategy.noWatermarks(),
operatorName, outputTypeInfo)
+// don't use rowTypes from CatalogTable
+// because the rowType number may be reduced by ProjectionPushDown
+RowType sourceRowType = outputTypeInfo.toRowType();
+SourceAbilityContext sourceAbilityContext =
+new SourceAbilityContext(planner.getFlinkContext(),
sourceRowType);
+
+WatermarkStrategy watermarkStrategy = null;
+
+if (tableSource instanceof SupportsWatermarkPushDown) {
Review comment:
In my vision, if the dynamic table source doesn't support
watermarkPushdown, or can't apply the PushWatermarkIntoTableSourceScanRule,
maybe we can't push the watermark down to the source.
Now if the actual watermark can't be pushed into source like above
situations, the WatermarkStrategy.noWatermarks() will be pushed into the source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] xuyangzhong commented on a change in pull request #17537: [FLINK-19722][table-planner]Pushdown Watermark to SourceProvider (new Source)
xuyangzhong commented on a change in pull request #17537:
URL: https://github.com/apache/flink/pull/17537#discussion_r737105590
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecTableSourceScan.java
##
@@ -97,9 +104,40 @@ public DynamicTableSourceSpec getTableSourceSpec() {
return createInputFormatTransformation(env, inputFormat,
outputTypeInfo, operatorName);
} else if (provider instanceof SourceProvider) {
Source source = ((SourceProvider)
provider).createSource();
-// TODO: Push down watermark strategy to source scan
-return env.fromSource(
-source, WatermarkStrategy.noWatermarks(),
operatorName, outputTypeInfo)
+// don't use rowTypes from CatalogTable
+// because the rowType number may be reduced by ProjectionPushDown
+RowType sourceRowType = outputTypeInfo.toRowType();
+SourceAbilityContext sourceAbilityContext =
+new SourceAbilityContext(planner.getFlinkContext(),
sourceRowType);
+
+WatermarkStrategy watermarkStrategy = null;
+
+if (tableSource instanceof SupportsWatermarkPushDown) {
Review comment:
In my vision, if the dynamic table source doesn't support
watermarkPushdown, or can't apply the PushWatermarkIntoTableSourceScanRule,
maybe we can't push the watermark down to the source.
Now if the actual watermark can't be pushed into source like above
situations, the WatermarkStrategy.noWatermarks() will be pushed into the source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] tsreaper commented on a change in pull request #17570: [FLINK-24654][table] Fix NPE on RetractableTopNFunction when some records were cleared by state ttl
tsreaper commented on a change in pull request #17570:
URL: https://github.com/apache/flink/pull/17570#discussion_r737100142
##
File path:
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/rank/RetractableTopNFunction.java
##
@@ -455,9 +464,18 @@ private boolean retractRecordWithoutRowNumber(
// sends the record if there is a record recently upgrades
to Top-N
int index = Long.valueOf(rankEnd - nextRank).intValue();
List inputs = dataState.get(key);
-RowData toAdd = inputs.get(index);
-collectInsert(out, toAdd);
-break;
+if (inputs == null) {
+// Skip the data if it's state is cleared because of
state ttl.
+if (lenient) {
+LOG.warn(STATE_CLEARED_WARN_MSG);
Review comment:
Shall we update the value of `sortedMap`? If not `treeMap` and
`dataState` will be inconsistent. Or shall we put `treeMap` and `dataState` in
an atomic state object to avoid this problem once and for all.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] tsreaper commented on a change in pull request #17570: [FLINK-24654][table] Fix NPE on RetractableTopNFunction when some records were cleared by state ttl
tsreaper commented on a change in pull request #17570:
URL: https://github.com/apache/flink/pull/17570#discussion_r737099501
##
File path:
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/rank/RetractableTopNFunction.java
##
@@ -455,9 +464,18 @@ private boolean retractRecordWithoutRowNumber(
// sends the record if there is a record recently upgrades
to Top-N
int index = Long.valueOf(rankEnd - nextRank).intValue();
List inputs = dataState.get(key);
-RowData toAdd = inputs.get(index);
-collectInsert(out, toAdd);
-break;
+if (inputs == null) {
+// Skip the data if it's state is cleared because of
state ttl.
+if (lenient) {
+LOG.warn(STATE_CLEARED_WARN_MSG);
+} else {
+throw new RuntimeException(STATE_CLEARED_WARN_MSG);
+}
+} else {
+RowData toAdd = inputs.get(index);
+collectInsert(out, toAdd);
+break;
Review comment:
OK I understand. When we discover that records of this sort key do not
exist in states, we should just skip this sort key as if nothing happens and
continue processing. So we should only break out of the loop here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-ml] guoweiM commented on a change in pull request #16: [FLINK-9][iteration] Support per-round iteration
guoweiM commented on a change in pull request #16:
URL: https://github.com/apache/flink-ml/pull/16#discussion_r736240831
##
File path:
flink-ml-iteration/src/main/java/org/apache/flink/iteration/PerRoundSubGraphBuilder.java
##
@@ -0,0 +1,67 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.iteration;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.iteration.compile.DraftExecutionEnvironment;
+import org.apache.flink.iteration.operator.OperatorWrapper;
+import org.apache.flink.iteration.operator.perround.PerRoundOperatorWrapper;
+import org.apache.flink.streaming.api.datastream.DataStream;
+
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.flink.util.Preconditions.checkArgument;
+
+/** Allows to add per-round subgraph inside the iteration body. */
+@Internal
+public class PerRoundSubGraphBuilder {
Review comment:
This class is not used.
##
File path:
flink-ml-iteration/src/main/java/org/apache/flink/iteration/datacache/nonkeyed/DataCacheWriter.java
##
@@ -0,0 +1,94 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.iteration.datacache.nonkeyed;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.core.fs.FSDataOutputStream;
+import org.apache.flink.core.fs.FileSystem;
+import org.apache.flink.core.fs.Path;
+import org.apache.flink.core.memory.DataOutputView;
+import org.apache.flink.core.memory.DataOutputViewStreamWrapper;
+
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.List;
+import java.util.function.Supplier;
+
+/** Records the data received and replayed them on required. */
+public class DataCacheWriter {
+
+private final TypeSerializer serializer;
+
+private final FileSystem fileSystem;
+
+private final Supplier pathGenerator;
+
+private final List finishSegments;
+
+private Path currentPath;
+
+private FSDataOutputStream outputStream;
+
+private DataOutputView outputView;
+
+private int currentSegmentCount;
+
+public DataCacheWriter(
+TypeSerializer serializer, FileSystem fileSystem,
Supplier pathGenerator)
+throws IOException {
+this.serializer = serializer;
+this.fileSystem = fileSystem;
+this.pathGenerator = pathGenerator;
+
+this.finishSegments = new ArrayList<>();
+
+startNewSegment();
+}
+
+public void addRecord(T record) throws IOException {
+serializer.serialize(record, outputView);
+currentSegmentCount += 1;
+}
+
+public List finishAddingRecords() throws IOException {
+finishCurrentSegment();
+return finishSegments;
+}
+
+public List getFinishSegments() {
+return finishSegments;
+}
+
+@VisibleForTesting
+void startNewSegment() throws IOException {
+this.currentPath = pathGenerator.get();
+this.outputStream = fileSystem.create(currentPath,
FileSystem.WriteMode.NO_OVERWRITE);
+this.outputView = new DataOutputViewStreamWrapper(outputStream);
Review comment:
I am a little curious why there is not have a `SegmentWriter` just as
`SegmentReader`.
We might could remove some the mutable member such as `currentPath` &
`outputStream` etc.
##
File path:
flink-ml-iteration/src/main/java/
[GitHub] [flink] flinkbot edited a comment on pull request #17516: [FLINK-23391][connector/kafka] Fix flaky Kafka source metric test by retrying notifyCheckpointComplete until success or timeout
flinkbot edited a comment on pull request #17516: URL: https://github.com/apache/flink/pull/17516#issuecomment-946494939 ## CI report: * 1c05c2461a12b061e88930ba92fb5d5c9d4e57d9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25205) * 2e0b1630a93c95d0db9a0262e7031f766bd472bd UNKNOWN * a313fe10583fb092610b1855bdd0780c7a662d82 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25492) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17516: [FLINK-23391][connector/kafka] Fix flaky Kafka source metric test by retrying notifyCheckpointComplete until success or timeout
flinkbot edited a comment on pull request #17516: URL: https://github.com/apache/flink/pull/17516#issuecomment-946494939 ## CI report: * 1c05c2461a12b061e88930ba92fb5d5c9d4e57d9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25205) * 2e0b1630a93c95d0db9a0262e7031f766bd472bd UNKNOWN * a313fe10583fb092610b1855bdd0780c7a662d82 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17516: [FLINK-23391][connector/kafka] Fix flaky Kafka source metric test by retrying notifyCheckpointComplete until success or timeout
flinkbot edited a comment on pull request #17516: URL: https://github.com/apache/flink/pull/17516#issuecomment-946494939 ## CI report: * 1c05c2461a12b061e88930ba92fb5d5c9d4e57d9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25205) * 2e0b1630a93c95d0db9a0262e7031f766bd472bd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #17537: [FLINK-19722][table-planner]Pushdown Watermark to SourceProvider (new Source)
JingsongLi commented on a change in pull request #17537:
URL: https://github.com/apache/flink/pull/17537#discussion_r737083797
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecTableSourceScan.java
##
@@ -97,9 +104,40 @@ public DynamicTableSourceSpec getTableSourceSpec() {
return createInputFormatTransformation(env, inputFormat,
outputTypeInfo, operatorName);
} else if (provider instanceof SourceProvider) {
Source source = ((SourceProvider)
provider).createSource();
-// TODO: Push down watermark strategy to source scan
-return env.fromSource(
-source, WatermarkStrategy.noWatermarks(),
operatorName, outputTypeInfo)
+// don't use rowTypes from CatalogTable
+// because the rowType number may be reduced by ProjectionPushDown
+RowType sourceRowType = outputTypeInfo.toRowType();
+SourceAbilityContext sourceAbilityContext =
+new SourceAbilityContext(planner.getFlinkContext(),
sourceRowType);
+
+WatermarkStrategy watermarkStrategy = null;
+
+if (tableSource instanceof SupportsWatermarkPushDown) {
Review comment:
If the source uses `SourceProvider` and implements
`SupportsWatermarkPushDown`.
The `applyWatermark` is useless, is this correct?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #17537: [FLINK-19722][table-planner]Pushdown Watermark to SourceProvider (new Source)
JingsongLi commented on a change in pull request #17537:
URL: https://github.com/apache/flink/pull/17537#discussion_r737082587
##
File path:
flink-table/flink-table-planner/src/main/java/org/apache/flink/table/planner/plan/nodes/exec/common/CommonExecTableSourceScan.java
##
@@ -97,9 +104,40 @@ public DynamicTableSourceSpec getTableSourceSpec() {
return createInputFormatTransformation(env, inputFormat,
outputTypeInfo, operatorName);
} else if (provider instanceof SourceProvider) {
Source source = ((SourceProvider)
provider).createSource();
-// TODO: Push down watermark strategy to source scan
-return env.fromSource(
-source, WatermarkStrategy.noWatermarks(),
operatorName, outputTypeInfo)
+// don't use rowTypes from CatalogTable
+// because the rowType number may be reduced by ProjectionPushDown
+RowType sourceRowType = outputTypeInfo.toRowType();
+SourceAbilityContext sourceAbilityContext =
+new SourceAbilityContext(planner.getFlinkContext(),
sourceRowType);
+
+WatermarkStrategy watermarkStrategy = null;
+
+if (tableSource instanceof SupportsWatermarkPushDown) {
Review comment:
If a source uses a `SourceProvider`, under what circumstances does he
not want watermark pushed down to the source?
We can think about: always push down watermark to new source.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17516: [FLINK-23391][connector/kafka] Fix flaky Kafka source metric test by retrying notifyCheckpointComplete until success or timeout
flinkbot edited a comment on pull request #17516: URL: https://github.com/apache/flink/pull/17516#issuecomment-946494939 ## CI report: * 1c05c2461a12b061e88930ba92fb5d5c9d4e57d9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25205) * 2e0b1630a93c95d0db9a0262e7031f766bd472bd UNKNOWN * a313fe10583fb092610b1855bdd0780c7a662d82 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17516: [FLINK-23391][connector/kafka] Fix flaky Kafka source metric test by retrying notifyCheckpointComplete until success or timeout
flinkbot edited a comment on pull request #17516: URL: https://github.com/apache/flink/pull/17516#issuecomment-946494939 ## CI report: * 1c05c2461a12b061e88930ba92fb5d5c9d4e57d9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25205) * 2e0b1630a93c95d0db9a0262e7031f766bd472bd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] PatrickRen commented on pull request #16940: [FLINK-23914][connector/testing-framework] Improve logging and error message in testing framework
PatrickRen commented on pull request #16940: URL: https://github.com/apache/flink/pull/16940#issuecomment-952509350 Thanks @AHeise ! AZP run has passed. Could you help to merge the PR when you are available? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tsreaper commented on a change in pull request #17570: [FLINK-24654][table] Fix NPE on RetractableTopNFunction when some records were cleared by state ttl
tsreaper commented on a change in pull request #17570:
URL: https://github.com/apache/flink/pull/17570#discussion_r737077180
##
File path:
flink-table/flink-table-runtime/src/main/java/org/apache/flink/table/runtime/operators/rank/RetractableTopNFunction.java
##
@@ -455,9 +464,18 @@ private boolean retractRecordWithoutRowNumber(
// sends the record if there is a record recently upgrades
to Top-N
int index = Long.valueOf(rankEnd - nextRank).intValue();
List inputs = dataState.get(key);
-RowData toAdd = inputs.get(index);
-collectInsert(out, toAdd);
-break;
+if (inputs == null) {
+// Skip the data if it's state is cleared because of
state ttl.
+if (lenient) {
+LOG.warn(STATE_CLEARED_WARN_MSG);
+} else {
+throw new RuntimeException(STATE_CLEARED_WARN_MSG);
+}
+} else {
+RowData toAdd = inputs.get(index);
+collectInsert(out, toAdd);
+break;
Review comment:
`break` should be in the outer `else` block starting from line 463?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17516: [FLINK-23391][connector/kafka] Fix flaky Kafka source metric test by retrying notifyCheckpointComplete until success or timeout
flinkbot edited a comment on pull request #17516: URL: https://github.com/apache/flink/pull/17516#issuecomment-946494939 ## CI report: * 1c05c2461a12b061e88930ba92fb5d5c9d4e57d9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25205) * 2e0b1630a93c95d0db9a0262e7031f766bd472bd UNKNOWN * a313fe10583fb092610b1855bdd0780c7a662d82 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17516: [FLINK-23391][connector/kafka] Fix flaky Kafka source metric test by retrying notifyCheckpointComplete until success or timeout
flinkbot edited a comment on pull request #17516: URL: https://github.com/apache/flink/pull/17516#issuecomment-946494939 ## CI report: * 1c05c2461a12b061e88930ba92fb5d5c9d4e57d9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25205) * 2e0b1630a93c95d0db9a0262e7031f766bd472bd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17516: [FLINK-23391][connector/kafka] Fix flaky Kafka source metric test by retrying notifyCheckpointComplete until success or timeout
flinkbot edited a comment on pull request #17516: URL: https://github.com/apache/flink/pull/17516#issuecomment-946494939 ## CI report: * 1c05c2461a12b061e88930ba92fb5d5c9d4e57d9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25205) * 2e0b1630a93c95d0db9a0262e7031f766bd472bd UNKNOWN * a313fe10583fb092610b1855bdd0780c7a662d82 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17516: [FLINK-23391][connector/kafka] Fix flaky Kafka source metric test by retrying notifyCheckpointComplete until success or timeout
flinkbot edited a comment on pull request #17516: URL: https://github.com/apache/flink/pull/17516#issuecomment-946494939 ## CI report: * 1c05c2461a12b061e88930ba92fb5d5c9d4e57d9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25205) * 2e0b1630a93c95d0db9a0262e7031f766bd472bd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] PatrickRen commented on pull request #17516: [FLINK-23391][connector/kafka] Fix flaky Kafka source metric test by retrying notifyCheckpointComplete until success or timeout
PatrickRen commented on pull request #17516: URL: https://github.com/apache/flink/pull/17516#issuecomment-952504483 Sorry for my late response @AHeise . I updated the commit message and rebased the latest master -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17516: [FLINK-23391][connector/kafka] Fix flaky Kafka source metric test by retrying notifyCheckpointComplete until success or timeout
flinkbot edited a comment on pull request #17516: URL: https://github.com/apache/flink/pull/17516#issuecomment-946494939 ## CI report: * 1c05c2461a12b061e88930ba92fb5d5c9d4e57d9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25205) * 2e0b1630a93c95d0db9a0262e7031f766bd472bd UNKNOWN * a313fe10583fb092610b1855bdd0780c7a662d82 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-24662) PyFlink sphinx check failed with "node class 'meta' is already registered, its visitors will be overridden"
[
https://issues.apache.org/jira/browse/FLINK-24662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu closed FLINK-24662.
---
Fix Version/s: 1.13.4
1.14.1
1.15.0
1.12.6
Resolution: Fixed
Fixed in
- master via aebd31054366581dde0445b2aa73f63cd6687d24
- release-1.14 via 808b32b758e010b1d60ce62ae3bb43ac8845b6c1
- release-1.13 via c62f7634ca8836d207a024de0032b04fc8125d4c
- release-1.12 via 9233b9b6513d6e912a620b8f18a93fa73909622c
> PyFlink sphinx check failed with "node class 'meta' is already registered,
> its visitors will be overridden"
> ---
>
> Key: FLINK-24662
> URL: https://issues.apache.org/jira/browse/FLINK-24662
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Tests
>Affects Versions: 1.13.0, 1.14.0, 1.15.0
>Reporter: Dian Fu
>Assignee: Dian Fu
>Priority: Major
> Fix For: 1.12.6, 1.15.0, 1.14.1, 1.13.4
>
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=25481&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602&t=8d78fe4f-d658-5c70-12f8-4921589024c3]
> {code}
> ==mypy checks... [SUCCESS]===
> Oct 26 22:08:34 rm -rf _build/*
> Oct 26 22:08:34 /__w/1/s/flink-python/dev/.conda/bin/sphinx-build -b html -d
> _build/doctrees -a -W . _build/html
> Oct 26 22:08:34 Running Sphinx v2.4.4
> Oct 26 22:08:34
> Oct 26 22:08:34 Warning, treated as error:
> Oct 26 22:08:34 node class 'meta' is already registered, its visitors will be
> overridden
> Oct 26 22:08:34 Makefile:76: recipe for target 'html' failed
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-24662) PyFlink sphinx check failed with "node class 'meta' is already registered, its visitors will be overridden"
[
https://issues.apache.org/jira/browse/FLINK-24662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434649#comment-17434649
]
Dian Fu commented on FLINK-24662:
-
It should be caused by the latest release docutils
[https://pypi.org/project/docutils/0.18/#history]
To fix it, we could simply set the version to an old version.
> PyFlink sphinx check failed with "node class 'meta' is already registered,
> its visitors will be overridden"
> ---
>
> Key: FLINK-24662
> URL: https://issues.apache.org/jira/browse/FLINK-24662
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Tests
>Affects Versions: 1.13.0, 1.14.0, 1.15.0
>Reporter: Dian Fu
>Assignee: Dian Fu
>Priority: Major
>
> [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=25481&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602&t=8d78fe4f-d658-5c70-12f8-4921589024c3]
> {code}
> ==mypy checks... [SUCCESS]===
> Oct 26 22:08:34 rm -rf _build/*
> Oct 26 22:08:34 /__w/1/s/flink-python/dev/.conda/bin/sphinx-build -b html -d
> _build/doctrees -a -W . _build/html
> Oct 26 22:08:34 Running Sphinx v2.4.4
> Oct 26 22:08:34
> Oct 26 22:08:34 Warning, treated as error:
> Oct 26 22:08:34 node class 'meta' is already registered, its visitors will be
> overridden
> Oct 26 22:08:34 Makefile:76: recipe for target 'html' failed
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-21289) Application mode ignores the pipeline.classpaths configuration
[
https://issues.apache.org/jira/browse/FLINK-21289?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434648#comment-17434648
]
Zhou Parker commented on FLINK-21289:
-
[~wangyang0918] Since it’s been a long time in the past, I am not sure if this
BUG has been fixed. I need to review this PR conflict
> Application mode ignores the pipeline.classpaths configuration
> --
>
> Key: FLINK-21289
> URL: https://issues.apache.org/jira/browse/FLINK-21289
> Project: Flink
> Issue Type: Bug
> Components: Client / Job Submission, Deployment / Kubernetes,
> Deployment / YARN
>Affects Versions: 1.11.2, 1.12.1
> Environment: flink: 1.11
> kubernetes: 1.15
>
>Reporter: Zhou Parker
>Assignee: Zhou Parker
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.6, 1.15.0, 1.14.1, 1.13.4
>
> Attachments: 0001-IMP.patch
>
>
> 我尝试将flink作业以application
> mode方式提交到kubernetes上运行。但程序的依赖包并不完全存在于local:///opt/flink/usrlib/.jar中。导致找不到类。
> 在yarn上可以工作，是因为我们用 {color:#ff}-C [http://|http:///]
> {color}的方式，让依赖可以被URLClassloader加载。
> 但我发现，当实验提交到kubernetes时，-C只会在 configmap/flink-conf.yaml
> 中生成一个pipeline.classpaths 配置条目。我们的main函数可以执行，但是在加载外部依赖类的时候提示找不到类。
> 通过阅读源码，*我发现运行用户代码的类加载器实际并没有把 pipeline.classpaths
> 中的条目加入候选URL*，这导致了无法加载类的情况。从源码中，我也发现，通过将依赖包放在usrlib目录下（默认的userClassPath）可以解决问题。但我们的依赖可能是动态的，不合适一次性打到镜像里面。
> 我提议可以改进这个过程，将pipeline.classpaths也加入到对应的类加载器。这个改动很小，我自己经过测试，可以完美解决问题。
>
>
> English translation:
> I'm trying to submit flink job to kubernetes cluster with application mode,
> but throw ClassNotFoundException when some dependency class is not shipped in
> kind of local:///opt/flink/usrlib/.jar.
> This works on yarn, since we use {color:#ff}-C
> [http://|http:///]{color} command line style that let dependency
> class can be load by URLClassloader.
> But i figure out that not works on kubernetes. When submit to kubernetes
> cluster, -C is only shipped as item "pipeline.classpaths" in
> configmap/flink-conf.yaml。
> After read the source code, *i find out that the Classloader launching the
> "main" entry of user code miss consider adding pipeline.classpaths into
> candidates URLs*. from source code, i also learn that we can ship the
> dependency jar in the usrlib dir to solve the problem. But that failed for
> us, we are not _preferred_ to ship dependencies in image at compile time,
> since dependencies are known dynamically in runtime
> I proposed to improve the process, let the Classloader consider usrlib as
> well as pipeline.classpaths, this is a quite little change. I test the
> solution and it works quite well
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-24541) ConfigurationUtils#assembleDynamicConfigsStr should consider special characters
[ https://issues.apache.org/jira/browse/FLINK-24541?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song closed FLINK-24541. Fix Version/s: (was: 1.14.1) Resolution: Fixed Fixed via: - master (1.15): 44e75ecdf9ca15f8a028f77efd6cc7da5a7e4549 > ConfigurationUtils#assembleDynamicConfigsStr should consider special > characters > --- > > Key: FLINK-24541 > URL: https://issues.apache.org/jira/browse/FLINK-24541 > Project: Flink > Issue Type: Bug > Components: Runtime / Configuration >Affects Versions: 1.14.0 >Reporter: Yangze Guo >Assignee: Yangze Guo >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > Without quoting, some special characters will be misunderstood by shell, e.g. > ';' used in list type options. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] xintongsong closed pull request #17533: [FLINK-24541][runtime] Quoting the external resource list in generati…
xintongsong closed pull request #17533: URL: https://github.com/apache/flink/pull/17533 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-24662) PyFlink sphinx check failed with "node class 'meta' is already registered, its visitors will be overridden"
Dian Fu created FLINK-24662:
---
Summary: PyFlink sphinx check failed with "node class 'meta' is
already registered, its visitors will be overridden"
Key: FLINK-24662
URL: https://issues.apache.org/jira/browse/FLINK-24662
Project: Flink
Issue Type: Bug
Components: API / Python, Tests
Affects Versions: 1.14.0, 1.13.0, 1.15.0
Reporter: Dian Fu
Assignee: Dian Fu
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=25481&view=logs&j=9cada3cb-c1d3-5621-16da-0f718fb86602&t=8d78fe4f-d658-5c70-12f8-4921589024c3]
{code}
==mypy checks... [SUCCESS]===
Oct 26 22:08:34 rm -rf _build/*
Oct 26 22:08:34 /__w/1/s/flink-python/dev/.conda/bin/sphinx-build -b html -d
_build/doctrees -a -W . _build/html
Oct 26 22:08:34 Running Sphinx v2.4.4
Oct 26 22:08:34
Oct 26 22:08:34 Warning, treated as error:
Oct 26 22:08:34 node class 'meta' is already registered, its visitors will be
overridden
Oct 26 22:08:34 Makefile:76: recipe for target 'html' failed
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17520: [FLINK-24565][avro] Port avro file format factory to BulkReaderFormatFactory
flinkbot edited a comment on pull request #17520: URL: https://github.com/apache/flink/pull/17520#issuecomment-946592172 ## CI report: * de579d6e83aa94b92dc583c79c34ac085fb4b912 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25485) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17556: [FLINK-24627][tests] add some Junit5 extensions to replace the existed Junit4 rules
flinkbot edited a comment on pull request #17556: URL: https://github.com/apache/flink/pull/17556#issuecomment-950605426 ## CI report: * 756027340bfe8a48fd5947c6eddd8db22beca44a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25452) * 3cd20469f38e3f4e11e7e51790c7eaa2c4d74c78 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-24661) ConfigOption add isSecret method to judge sensitive options
Ada Wong created FLINK-24661: Summary: ConfigOption add isSecret method to judge sensitive options Key: FLINK-24661 URL: https://issues.apache.org/jira/browse/FLINK-24661 Project: Flink Issue Type: Improvement Components: API / Core Affects Versions: 1.13.3 Reporter: Ada Wong Related ticket https://issues.apache.org/jira/browse/FLINK-24381 [~chesnay] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17556: [FLINK-24627][tests] add some Junit5 extensions to replace the existed Junit4 rules
flinkbot edited a comment on pull request #17556: URL: https://github.com/apache/flink/pull/17556#issuecomment-950605426 ## CI report: * 756027340bfe8a48fd5947c6eddd8db22beca44a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25452) * 3cd20469f38e3f4e11e7e51790c7eaa2c4d74c78 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25486) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17520: [FLINK-24565][avro] Port avro file format factory to BulkReaderFormatFactory
flinkbot edited a comment on pull request #17520: URL: https://github.com/apache/flink/pull/17520#issuecomment-946592172 ## CI report: * 368a7c41cde52ad3c0fde6c30e95ddf5308f8c16 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25458) * de579d6e83aa94b92dc583c79c34ac085fb4b912 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25485) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-24610) Flink projection common expression elimination
[ https://issues.apache.org/jira/browse/FLINK-24610?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434275#comment-17434275 ] Lyn Zhang edited comment on FLINK-24610 at 10/27/21, 2:33 AM: -- [~jark] Please help to review this issue when you are free was (Author: zicat): [~jark] Please help to review this issue if you are free > Flink projection common expression elimination > -- > > Key: FLINK-24610 > URL: https://issues.apache.org/jira/browse/FLINK-24610 > Project: Flink > Issue Type: Improvement > Components: Table SQL / Planner >Reporter: Lyn Zhang >Priority: Major > Labels: pull-request-available > > Flink sql projection not support reusing common expression in code generator. > For example the sql: SELECT split(a, '__'_)[1], split(a, '__'_)[2] FROM > table.The function split(a, '_') will be invoked 2 times per row data. > We can use CalcProgram.getExprList to generator each expression and > CalcProgram.getProjectList to eliminate common expression. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #17520: [FLINK-24565][avro] Port avro file format factory to BulkReaderFormatFactory
flinkbot edited a comment on pull request #17520: URL: https://github.com/apache/flink/pull/17520#issuecomment-946592172 ## CI report: * 368a7c41cde52ad3c0fde6c30e95ddf5308f8c16 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25458) * de579d6e83aa94b92dc583c79c34ac085fb4b912 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] ruanhang1993 commented on a change in pull request #17556: [FLINK-24627][tests] add some Junit5 extensions to replace the existed Junit4 rules
ruanhang1993 commented on a change in pull request #17556:
URL: https://github.com/apache/flink/pull/17556#discussion_r737055199
##
File path:
flink-test-utils-parent/flink-test-utils-junit/src/main/java/org/apache/flink/core/testutils/AllCallbackWrapper.java
##
@@ -0,0 +1,25 @@
+package org.apache.flink.core.testutils;
+
+import org.junit.jupiter.api.extension.AfterAllCallback;
+import org.junit.jupiter.api.extension.BeforeAllCallback;
+import org.junit.jupiter.api.extension.ExtensionContext;
+
+/** An extension wrap logic for {@link BeforeAllCallback} and {@link
AfterAllCallback}. */
+public class AllCallbackWrapper
Review comment:
Yes, I think you are right. I will remove the generic type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] tsreaper commented on a change in pull request #17520: [FLINK-24565][avro] Port avro file format factory to BulkReaderFormatFactory
tsreaper commented on a change in pull request #17520:
URL: https://github.com/apache/flink/pull/17520#discussion_r737052999
##
File path: flink-formats/flink-avro/pom.xml
##
@@ -47,13 +47,11 @@ under the License.
-
org.apache.flink
- flink-table-common
+
flink-table-runtime_${scala.binary.version}
Review comment:
I would say `flink-table-api-java`. This `FileSystemConnectorOptions` is
a `@PublicEvolving` API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] ruanhang1993 commented on a change in pull request #17556: [FLINK-24627][tests] add some Junit5 extensions to replace the existed Junit4 rules
ruanhang1993 commented on a change in pull request #17556:
URL: https://github.com/apache/flink/pull/17556#discussion_r737048082
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/zookeeper/ZooKeeperExtension.java
##
@@ -0,0 +1,64 @@
+package org.apache.flink.runtime.zookeeper;
+
+import org.apache.flink.core.testutils.CustomExtension;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.curator.test.TestingServer;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/**
+ * {@link org.junit.jupiter.api.extension.Extension} which starts a {@link
+ * org.apache.zookeeper.server.ZooKeeperServer}.
+ */
+public class ZooKeeperExtension implements CustomExtension {
+private static final Logger LOG =
LoggerFactory.getLogger(ZooKeeperExtension.class);
+
+@Nullable private TestingServer zooKeeperServer;
+
+public String getConnectString() {
+verifyIsRunning();
+return zooKeeperServer.getConnectString();
+}
+
+private void verifyIsRunning() {
+Preconditions.checkState(zooKeeperServer != null);
+}
+
+private void terminateZooKeeperServer() throws IOException {
+if (zooKeeperServer != null) {
+zooKeeperServer.stop();
+zooKeeperServer = null;
+}
+}
+
+@Override
+public void after(ExtensionContext context) throws Exception {
+try {
+terminateZooKeeperServer();
+} catch (IOException e) {
+LOG.warn("Could not properly terminate the {}.",
getClass().getSimpleName(), e);
+}
+}
+
+@Override
+public void before(ExtensionContext context) throws Exception {
+terminateZooKeeperServer();
Review comment:
I keep the same logic in `ZooKeeperExtension`, which invokes this method
in its `before()` method. I am not sure what will happen if we remove this code.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17463: [Hotfix] Fix typos.
flinkbot edited a comment on pull request #17463: URL: https://github.com/apache/flink/pull/17463#issuecomment-942081615 ## CI report: * 8e67703b26eb31fc91419ba343e9ee30b46bdb53 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25477) * 3dd575059ec3e136b282f1bfb5c7fd538fe40b87 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25483) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17463: [Hotfix] Fix typos.
flinkbot edited a comment on pull request #17463: URL: https://github.com/apache/flink/pull/17463#issuecomment-942081615 ## CI report: * 8e67703b26eb31fc91419ba343e9ee30b46bdb53 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25477) * 3dd575059ec3e136b282f1bfb5c7fd538fe40b87 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] meetjunsu commented on pull request #17542: [FLINK-17782] Add array,map,row types support for parquet row writer
meetjunsu commented on pull request #17542: URL: https://github.com/apache/flink/pull/17542#issuecomment-952456458 @snuyanzin updated in latest commit -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20286) Support streaming source for filesystem SQL connector
[ https://issues.apache.org/jira/browse/FLINK-20286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434628#comment-17434628 ] Hadi commented on FLINK-20286: -- [~slinkydeveloper] can this support in Flink SQL ? I found no place to set the config. > Support streaming source for filesystem SQL connector > - > > Key: FLINK-20286 > URL: https://issues.apache.org/jira/browse/FLINK-20286 > Project: Flink > Issue Type: New Feature > Components: Connectors / FileSystem, Table SQL / Ecosystem >Reporter: Jark Wu >Priority: Minor > Labels: auto-deprioritized-major > > Currenlty, the filesystem SQL connector only supports bounded source. It > would be great to support streaming read just like Hive connector. It should > monitor the new added files and read the content of new files. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-24641) Flink SQL FileSystem read hdfs Path, when Path's file name change, the sql will not find the file.
[
https://issues.apache.org/jira/browse/FLINK-24641?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Hadi closed FLINK-24641.
Resolution: Duplicate
> Flink SQL FileSystem read hdfs Path, when Path's file name change, the sql
> will not find the file.
> --
>
> Key: FLINK-24641
> URL: https://issues.apache.org/jira/browse/FLINK-24641
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.13.2
>Reporter: Hadi
>Priority: Minor
> Attachments: screenshot-1.png
>
>
> Hello, big bro.
> when I use the Flink SQL to read HDFS Files like this:
> {code:java}
> CREATE TABLE `cfg_city`(
> `provincecode` int, `city_id` int, `city_name` string)
> WITH (
> 'connector'='filesystem',
> 'path'='viewfs://path1/path2/path3/cfg_city',
> 'format' = 'csv',
> 'csv.field-delimiter' = ',',
> 'csv.ignore-parse-errors' = 'true'
> )
> ;
> {code}
> if the task failed and retry, this will throw a exception:
> !screenshot-1.png!
> I found , because the path 'viewfs://path1/path2/path3/cfg_city' file name
> changed. but it not get new FileList.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-24641) Flink SQL FileSystem read hdfs Path, when Path's file name change, the sql will not find the file.
[
https://issues.apache.org/jira/browse/FLINK-24641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434625#comment-17434625
]
Hadi commented on FLINK-24641:
--
[~MartijnVisser] thank u for ur replay. a duplicate issues i see.
thank u again.
> Flink SQL FileSystem read hdfs Path, when Path's file name change, the sql
> will not find the file.
> --
>
> Key: FLINK-24641
> URL: https://issues.apache.org/jira/browse/FLINK-24641
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.13.2
>Reporter: Hadi
>Priority: Minor
> Attachments: screenshot-1.png
>
>
> Hello, big bro.
> when I use the Flink SQL to read HDFS Files like this:
> {code:java}
> CREATE TABLE `cfg_city`(
> `provincecode` int, `city_id` int, `city_name` string)
> WITH (
> 'connector'='filesystem',
> 'path'='viewfs://path1/path2/path3/cfg_city',
> 'format' = 'csv',
> 'csv.field-delimiter' = ',',
> 'csv.ignore-parse-errors' = 'true'
> )
> ;
> {code}
> if the task failed and retry, this will throw a exception:
> !screenshot-1.png!
> I found , because the path 'viewfs://path1/path2/path3/cfg_city' file name
> changed. but it not get new FileList.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-24660) Allow setting KafkaSubscriber in KafkaSourceBuilder
Mason Chen created FLINK-24660: -- Summary: Allow setting KafkaSubscriber in KafkaSourceBuilder Key: FLINK-24660 URL: https://issues.apache.org/jira/browse/FLINK-24660 Project: Flink Issue Type: Improvement Components: Connectors / Kafka Affects Versions: 1.13.3, 1.14.0 Reporter: Mason Chen Some users may have a different mechanism for subscribing the set of topics/partitions. The builder can allow user custom implementations of KafkaSubscriber -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18983) Job doesn't changed to failed if close function has blocked
[
https://issues.apache.org/jira/browse/FLINK-18983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-18983:
---
Labels: stale-major stale-minor (was: stale-major)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Job doesn't changed to failed if close function has blocked
> ---
>
> Key: FLINK-18983
> URL: https://issues.apache.org/jira/browse/FLINK-18983
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Task
>Affects Versions: 1.11.0, 1.12.0
>Reporter: YufeiLiu
>Priority: Minor
> Labels: stale-major, stale-minor
>
> If a operator throw a exception, it will break process loop and dispose all
> operator. But state will never switch to FAILED if block in Function.close,
> and JobMaster can't know the final state and do restart.
> Task have {{TaskCancelerWatchDog}} to kill process if cancellation timeout,
> but it doesn't work for FAILED task.TAskThread will allways hang at:
> org.apache.flink.streaming.runtime.tasks.StreamTask#cleanUpInvoke
> Test case:
> {code:java}
> Configuration configuration = new Configuration();
> configuration.set(TaskManagerOptions.TASK_CANCELLATION_TIMEOUT, 1L);
> StreamExecutionEnvironment env =
> StreamExecutionEnvironment.createLocalEnvironment(2, configuration);
> env.addSource(...)
> .process(new ProcessFunction() {
> @Override
> public void processElement(String value, Context ctx,
> Collector out) throws Exception {
> if (getRuntimeContext().getIndexOfThisSubtask() == 0) {
> throw new RuntimeException();
> }
> }
> @Override
> public void close() throws Exception {
> if (getRuntimeContext().getIndexOfThisSubtask() == 0) {
> Thread.sleep(1000);
> }
> }
> }).setParallelism(2)
> .print();
> {code}
> In this case, job will block at close action and never change to FAILED.
> If change thread which subtaskIndex == 1 to sleep, TM will exit after
> TASK_CANCELLATION_TIMEOUT.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-23525) Docker command fails on Azure: Exit code 137 returned from process: file name '/usr/bin/docker'
[
https://issues.apache.org/jira/browse/FLINK-23525?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-23525:
---
Labels: auto-deprioritized-blocker stale-critical test-stability (was:
auto-deprioritized-blocker test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Critical but is unassigned and neither itself nor its Sub-Tasks have been
updated for 14 days. I have gone ahead and marked it "stale-critical". If this
ticket is critical, please either assign yourself or give an update.
Afterwards, please remove the label or in 7 days the issue will be
deprioritized.
> Docker command fails on Azure: Exit code 137 returned from process: file name
> '/usr/bin/docker'
> ---
>
> Key: FLINK-23525
> URL: https://issues.apache.org/jira/browse/FLINK-23525
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Affects Versions: 1.14.0, 1.13.1
>Reporter: Dawid Wysakowicz
>Priority: Critical
> Labels: auto-deprioritized-blocker, stale-critical,
> test-stability
> Attachments: screenshot-1.png
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=21053&view=logs&j=4d4a0d10-fca2-5507-8eed-c07f0bdf4887&t=7b25afdf-cc6c-566f-5459-359dc2585798&l=10034
> {code}
> ##[error]Exit code 137 returned from process: file name '/usr/bin/docker',
> arguments 'exec -i -u 1001 -w /home/vsts_azpcontainer
> 9dca235e075b70486fac576ee17cee722940edf575e5478e0a52def5b46c28b5
> /__a/externals/node/bin/node /__w/_temp/containerHandlerInvoker.js'.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18574) DownloadPipelineArtifact failes occasionally
[
https://issues.apache.org/jira/browse/FLINK-18574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-18574:
---
Labels: auto-deprioritized-critical stale-major test-stability (was:
auto-deprioritized-critical test-stability)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Major but is unassigned and neither itself nor its Sub-Tasks have been updated
for 60 days. I have gone ahead and added a "stale-major" to the issue". If this
ticket is a Major, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> DownloadPipelineArtifact failes occasionally
>
>
> Key: FLINK-18574
> URL: https://issues.apache.org/jira/browse/FLINK-18574
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Reporter: Dian Fu
>Priority: Major
> Labels: auto-deprioritized-critical, stale-major, test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4414&view=logs&j=bdd9ea51-4de2-506a-d4d9-f3930e4d2355&t=c1dcc74d-b153-580b-95a4-73d5cd91b503
> {code}
> 2020-07-11T20:38:15.9784577Z ##[section]Starting: DownloadPipelineArtifact
> 2020-07-11T20:38:15.9799212Z
> ==
> 2020-07-11T20:38:15.9799811Z Task : Download Pipeline Artifacts
> 2020-07-11T20:38:15.9800305Z Description : Download build and pipeline
> artifacts
> 2020-07-11T20:38:15.9800753Z Version : 2.3.1
> 2020-07-11T20:38:15.9801328Z Author : Microsoft Corporation
> 2020-07-11T20:38:15.9801942Z Help :
> https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/download-pipeline-artifact
> 2020-07-11T20:38:15.9802614Z
> ==
> 2020-07-11T20:38:16.5226960Z Download from the specified build: #4414
> 2020-07-11T20:38:16.5246147Z Download artifact to:
> /home/agent07/myagent/_work/2/flink_artifact
> 2020-07-11T20:38:20.6397951Z Information, ApplicationInsightsTelemetrySender
> will correlate events with X-TFS-Session 38415228-535b-4652-8afc-41e3fcfa5c67
> 2020-07-11T20:38:20.6546003Z Information, DedupManifestArtifactClient will
> correlate http requests with X-TFS-Session
> 38415228-535b-4652-8afc-41e3fcfa5c67
> 2020-07-11T20:38:20.6563787Z Information, Minimatch patterns: [**]
> 2020-07-11T20:38:20.7577072Z Information,
> ArtifactHttpRetryMessageHandler.SendAsync:
> https://vsblobprodsu6weu.vsblob.visualstudio.com/A2d3c0ac8-fecf-45be-8407-6d87302181a9/_apis/dedup/nodes/4C7343B40F7AB12049F6929A6A4ECD738EFADAC85D010AF5877ACD0682CC6C5802
> attempt 1/6 failed with StatusCode RedirectMethod, IsRetryableResponse False
> 2020-07-11T20:38:35.8527079Z Information, Filtered 130283 files from the
> Minimatch filters supplied.
> 2020-07-11T20:38:35.8865820Z Information, Downloaded 0.0 MB out of 1,170.3 MB
> (0%).
> 2020-07-11T20:38:40.8869716Z Information, Downloaded 4.6 MB out of 1,170.3 MB
> (0%).
> 2020-07-11T20:38:45.9379430Z Information, Downloaded 24.5 MB out of 1,170.3
> MB (2%).
> 2020-07-11T20:38:50.9539945Z Information, Downloaded 41.8 MB out of 1,170.3
> MB (4%).
> 2020-07-11T20:38:53.5482129Z Warning,
> [https://nsevsblobprodsu6weus42.blob.core.windows.net/db2d3c0ac8fecf45be84076d87302181a9/454487F48A36900D2D7FE8928376FDB49F9F3BF90B3A5474B7EF8D9ABE8BDC7201?sv=2019-02-02&sr=b&sig=HGudFZh7Cd3TOBUMGO7rIq0DJxPQORzHkSVs3iYeDPc%3D&spr=https&se=2020-07-12T21%3A16%3A18Z&sp=r&rscl=x-e2eid-e61f9696-89054b42-93868ec3-c0739ba7-session-38415228-535b4652-8afc41e3-fcfa5c67]
> Try 1/5, retryable exception caught. Retrying in 00:00:01.
> System.Net.Http.HttpRequestException: An error occurred while sending the
> request.
> 2020-07-11T20:38:53.5484031Z ---> System.Net.Http.CurlException: Couldn't
> connect to server
> 2020-07-11T20:38:53.5484420Zat
> System.Net.Http.CurlHandler.ThrowIfCURLEError(CURLcode error)
> 2020-07-11T20:38:53.5484929Zat
> System.Net.Http.CurlHandler.MultiAgent.FinishRequest(StrongToWeakReference`1
> easyWrapper, CURLcode messageResult)
> 2020-07-11T20:38:53.5485733Z--- End of inner exception stack trace ---
> 2020-07-11T20:38:53.5486394Zat
> System.Net.Http.HttpClient.FinishSendAsyncBuffered(Task`1 sendTask,
> HttpRequestMessage request, CancellationTokenSource cts, Boolean disposeCts)
> 2020-07-11T20:38:53.5487612Zat
> Microsoft.VisualStudio.Services.Common.TaskCancellationExtensions.EnforceCancellation[TResult](Task`1
> task, CancellationToken cancellationToken, Func`1 makeMessage, String file,
> String member, Int32 line)
> 2020-07-11T20:38:53.5488656Zat
> Microsof
[jira] [Updated] (FLINK-12912) Incorrect handling of task.checkpoint.alignment.max-size when one checkpoint subsumes another one
[
https://issues.apache.org/jira/browse/FLINK-12912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Flink Jira Bot updated FLINK-12912:
---
Labels: stale-major stale-minor (was: stale-major)
I am the [Flink Jira Bot|https://github.com/apache/flink-jira-bot/] and I help
the community manage its development. I see this issues has been marked as
Minor but is unassigned and neither itself nor its Sub-Tasks have been updated
for 180 days. I have gone ahead and marked it "stale-minor". If this ticket is
still Minor, please either assign yourself or give an update. Afterwards,
please remove the label or in 7 days the issue will be deprioritized.
> Incorrect handling of task.checkpoint.alignment.max-size when one checkpoint
> subsumes another one
> -
>
> Key: FLINK-12912
> URL: https://issues.apache.org/jira/browse/FLINK-12912
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.6.4, 1.7.2, 1.8.0
>Reporter: Piotr Nowojski
>Priority: Minor
> Labels: stale-major, stale-minor
>
> {{BarrierBuffer#numQueuedBytes}} which is used to evaluate {{
> task.checkpoint.alignment.max-size}} limit, is not correctly handled if one
> checkpoint subsumes another one.
> The max size limit is checked against a sum of {{numQueuedBytes}} and
> {{bufferBlocker.getBytesBlocked()}}. The {{getBytesBlocked}} keeps tracks of
> the alignment size of the only most latest checkpoint. The bug is
> {{BarrierBuffer#releaseBlocksAndResetBarriers()}} method, where while
> handling first subsumed checkpoint in the branch:
> {code:java}
> if (currentBuffered == null) {
> // common case: no more buffered data
> currentBuffered =
> bufferBlocker.rollOverReusingResources();
> if (currentBuffered != null) {
> currentBuffered.open();
> }
> }
> {code}
> we clear the {{bufferBlocker.getBytesBlocked()}} counter, while we do not
> update {{numQueuedBytes}} counter.
> For example when first checkpoint approached to 99.9% of max alignment size
> when it was subsumed, due to this bug calculated alignment size drops to 0
> bytes. For subsequent subsumed checkpoints {{numQueuedBytes}} is correctly
> updated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-24580) Kinesis connect time out error is not handled as recoverable
[
https://issues.apache.org/jira/browse/FLINK-24580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17434604#comment-17434604
]
John Karp commented on FLINK-24580:
---
I've tested out this patch, it seems to work in my manual testing:
https://github.com/apache/flink/compare/master...john-karp:retry-connect?expand=1
(The resulting log message is "Got recoverable SdkClientException. Backing off
for 261 millis (com.amazonaws.SdkClientException: Unable to execute HTTP
request: Connect to kinesis.us-east-1.amazonaws.com:443
[kinesis.us-east-1.amazonaws.com/3.227.250.226] failed: connect timed out)")
If the patch makes sense, I can add any appropriate unit tests.
> Kinesis connect time out error is not handled as recoverable
>
>
> Key: FLINK-24580
> URL: https://issues.apache.org/jira/browse/FLINK-24580
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kinesis
>Affects Versions: 1.13.2
>Reporter: John Karp
>Priority: Major
>
> Several times a day, transient Kinesis errors cause our Flink job to fail:
> {noformat}
> org.apache.flink.kinesis.shaded.com.amazonaws.SdkClientException: Unable to
> execute HTTP request: Connect to kinesis.us-east-1.amazonaws.com:443
> [kinesis.us-east-1.amazonaws.com/3.91.171.253] failed: connect timed out
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleRetryableException(AmazonHttpClient.java:1207)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1153)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.doExecute(AmazonHttpClient.java:802)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeWithTimer(AmazonHttpClient.java:770)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.execute(AmazonHttpClient.java:744)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutor.access$500(AmazonHttpClient.java:704)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient$RequestExecutionBuilderImpl.execute(AmazonHttpClient.java:686)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:550)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:530)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.doInvoke(AmazonKinesisClient.java:2893)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2860)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.invoke(AmazonKinesisClient.java:2849)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.executeGetRecords(AmazonKinesisClient.java:1319)
> at
> org.apache.flink.kinesis.shaded.com.amazonaws.services.kinesis.AmazonKinesisClient.getRecords(AmazonKinesisClient.java:1288)
> at
> org.apache.flink.streaming.connectors.kinesis.proxy.KinesisProxy.getRecords(KinesisProxy.java:292)
> at
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.polling.PollingRecordPublisher.getRecords(PollingRecordPublisher.java:168)
> at
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.polling.PollingRecordPublisher.run(PollingRecordPublisher.java:113)
> at
> org.apache.flink.streaming.connectors.kinesis.internals.publisher.polling.PollingRecordPublisher.run(PollingRecordPublisher.java:102)
> at
> org.apache.flink.streaming.connectors.kinesis.internals.ShardConsumer.run(ShardConsumer.java:114)
> at
> java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
> at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
> at
> java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by:
> org.apache.flink.kinesis.shaded.org.apache.http.conn.ConnectTimeoutException:
> Connect to kinesis.us-east-1.amazonaws.com:443
> [kinesis.us-east-1.amazonaws.com/3.91.171.253] failed: connect timed out
> at
> org.apache.flink.kinesis.shaded.org.apache.http.impl.conn.DefaultHttpClientConnectionOperator.connect(DefaultHttpClientConnectionOperator.java:151)
> at
> org.apache.flink.kinesis.shaded.org.apache.http.impl.conn.PoolingHttpClientCo
[GitHub] [flink] rkhachatryan commented on a change in pull request #15204: [FLINK-21783] Allow extension of HeapBackend SnapshotStrategy
rkhachatryan commented on a change in pull request #15204: URL: https://github.com/apache/flink/pull/15204#discussion_r736856052 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/state/CheckpointableKeyedStateBackend.java ## @@ -45,5 +46,6 @@ * write out a savepoint in the common/unified format. */ @Nonnull -SavepointResources savepoint() throws Exception; +SavepointResources savepoint(long checkpointId, CheckpointOptions checkpointOptions) Review comment: `checkpointId` is supposed to be used to decide whether to make an incremental or a full snapshot. `checkpointOptions` can be removed, you are right. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17572: [FLINK-23647][checkpointing] Wait for CheckpointsCleaner to complete before closing cluster
flinkbot edited a comment on pull request #17572: URL: https://github.com/apache/flink/pull/17572#issuecomment-952040222 ## CI report: * 2e33c87239fdd6a082ed3ae41a4e2c49c5e92031 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25478) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17463: [Hotfix] Fix typos.
flinkbot edited a comment on pull request #17463: URL: https://github.com/apache/flink/pull/17463#issuecomment-942081615 ## CI report: * 8e67703b26eb31fc91419ba343e9ee30b46bdb53 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25477) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17573: [FLINK-24190][runtime] Forbid to split the first record in the buffer if it physically fit it.
flinkbot edited a comment on pull request #17573: URL: https://github.com/apache/flink/pull/17573#issuecomment-952043255 ## CI report: * 0c03f9744353ae2b4140d334bc00649d311ee142 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25476) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17572: [FLINK-23647][checkpointing] Wait for CheckpointsCleaner to complete before closing cluster
flinkbot edited a comment on pull request #17572: URL: https://github.com/apache/flink/pull/17572#issuecomment-952040222 ## CI report: * 5c20827f354a6f033d0ccfd71d353f9e5f6669c7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25475) * 2e33c87239fdd6a082ed3ae41a4e2c49c5e92031 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25478) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17472: [FLINK-24486][rest] Make async result store duration configurable
flinkbot edited a comment on pull request #17472: URL: https://github.com/apache/flink/pull/17472#issuecomment-943076656 ## CI report: * d1470a5514b0990a18976d1dbb7ab5767dd00e69 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25466) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17472: [FLINK-24486][rest] Make async result store duration configurable
flinkbot edited a comment on pull request #17472: URL: https://github.com/apache/flink/pull/17472#issuecomment-943076656 ## CI report: * d1470a5514b0990a18976d1dbb7ab5767dd00e69 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25466) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17472: [FLINK-24486][rest] Make async result store duration configurable
flinkbot edited a comment on pull request #17472: URL: https://github.com/apache/flink/pull/17472#issuecomment-943076656 ## CI report: * d1470a5514b0990a18976d1dbb7ab5767dd00e69 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25466) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on pull request #17472: [FLINK-24486][rest] Make async result store duration configurable
zentol commented on pull request #17472: URL: https://github.com/apache/flink/pull/17472#issuecomment-952185469 @flinkbot run azure -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #17556: [FLINK-24627][tests] add some Junit5 extensions to replace the existed Junit4 rules
zentol commented on a change in pull request #17556:
URL: https://github.com/apache/flink/pull/17556#discussion_r735366139
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/zookeeper/ZooKeeperExtension.java
##
@@ -0,0 +1,64 @@
+package org.apache.flink.runtime.zookeeper;
+
+import org.apache.flink.core.testutils.CustomExtension;
+import org.apache.flink.util.Preconditions;
+
+import org.apache.curator.test.TestingServer;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import javax.annotation.Nullable;
+
+import java.io.IOException;
+
+/**
+ * {@link org.junit.jupiter.api.extension.Extension} which starts a {@link
+ * org.apache.zookeeper.server.ZooKeeperServer}.
+ */
+public class ZooKeeperExtension implements CustomExtension {
+private static final Logger LOG =
LoggerFactory.getLogger(ZooKeeperExtension.class);
+
+@Nullable private TestingServer zooKeeperServer;
+
+public String getConnectString() {
+verifyIsRunning();
+return zooKeeperServer.getConnectString();
+}
+
+private void verifyIsRunning() {
+Preconditions.checkState(zooKeeperServer != null);
+}
+
+private void terminateZooKeeperServer() throws IOException {
+if (zooKeeperServer != null) {
+zooKeeperServer.stop();
+zooKeeperServer = null;
+}
+}
+
+@Override
+public void after(ExtensionContext context) throws Exception {
+try {
+terminateZooKeeperServer();
+} catch (IOException e) {
+LOG.warn("Could not properly terminate the {}.",
getClass().getSimpleName(), e);
+}
+}
+
+@Override
+public void before(ExtensionContext context) throws Exception {
+terminateZooKeeperServer();
Review comment:
Why is this necessary?
##
File path:
flink-test-utils-parent/flink-test-utils-junit/src/main/java/org/apache/flink/core/testutils/AllCallbackWrapper.java
##
@@ -0,0 +1,25 @@
+package org.apache.flink.core.testutils;
+
+import org.junit.jupiter.api.extension.AfterAllCallback;
+import org.junit.jupiter.api.extension.BeforeAllCallback;
+import org.junit.jupiter.api.extension.ExtensionContext;
+
+/** An extension wrap logic for {@link BeforeAllCallback} and {@link
AfterAllCallback}. */
+public class AllCallbackWrapper
Review comment:
The generic type doesn't seem to get us anything, because T is not
returned anywhere.
##
File path:
flink-test-utils-parent/flink-test-utils-junit/src/main/java/org/apache/flink/testutils/junit/RetryTestExecutionExtension.java
##
@@ -0,0 +1,103 @@
+package org.apache.flink.testutils.junit;
+
+import org.junit.jupiter.api.extension.AfterEachCallback;
+import org.junit.jupiter.api.extension.ConditionEvaluationResult;
+import org.junit.jupiter.api.extension.ExecutionCondition;
+import org.junit.jupiter.api.extension.ExtensionContext;
+import org.junit.jupiter.api.extension.TestExecutionExceptionHandler;
+import org.opentest4j.TestAbortedException;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.Map;
+
+import static org.apache.flink.testutils.junit.RetryExtension.RETRY_KEY;
+import static org.apache.flink.testutils.junit.RetryExtension.RETRY_NAMESPACE;
+import static org.apache.flink.testutils.junit.RetryExtension.getTestMethodKey;
+
+/** Extension to decide whether a retry test should run. */
+public class RetryTestExecutionExtension
+implements ExecutionCondition, TestExecutionExceptionHandler,
AfterEachCallback {
+public static final Logger LOG =
LoggerFactory.getLogger(RetryTestExecutionExtension.class);
+private int retryIndex;
+private int totalTimes;
Review comment:
```suggestion
private final int retryIndex;
private final int totalTimes;
```
##
File path:
flink-test-utils-parent/flink-test-utils-junit/src/main/java/org/apache/flink/core/testutils/AllCallbackWrapper.java
##
@@ -0,0 +1,25 @@
+package org.apache.flink.core.testutils;
+
+import org.junit.jupiter.api.extension.AfterAllCallback;
+import org.junit.jupiter.api.extension.BeforeAllCallback;
+import org.junit.jupiter.api.extension.ExtensionContext;
+
+/** An extension wrap logic for {@link BeforeAllCallback} and {@link
AfterAllCallback}. */
+public class AllCallbackWrapper
+implements BeforeAllCallback, AfterAllCallback {
+private T customExtension;
Review comment:
```suggestion
private final T customExtension;
```
##
File path:
flink-test-utils-parent/flink-test-utils-junit/src/main/java/org/apache/flink/core/testutils/EachCallbackWrapper.java
##
@@ -0,0 +1,25 @@
+package org.apache.flink.core.testutils;
+
+import org.junit.jupiter.api.extension.AfterEachCallback;
+import org.junit.jupiter.api.extension.BeforeEachCallback;
+import org.junit.jupiter.api.extension.ExtensionContext;
+
+/** An extension
[GitHub] [flink] flinkbot edited a comment on pull request #17535: [FLINK-24609][akka][build] Use same Scala properties as root pom
flinkbot edited a comment on pull request #17535: URL: https://github.com/apache/flink/pull/17535#issuecomment-948487840 ## CI report: * 906e27be8cc278d75fd10d5216ca34275168df96 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25471) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17572: [FLINK-23647][checkpointing] Wait for CheckpointsCleaner to complete before closing cluster
flinkbot edited a comment on pull request #17572: URL: https://github.com/apache/flink/pull/17572#issuecomment-952040222 ## CI report: * 5c20827f354a6f033d0ccfd71d353f9e5f6669c7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25475) * 2e33c87239fdd6a082ed3ae41a4e2c49c5e92031 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25478) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17572: [FLINK-23647][checkpointing] Wait for CheckpointsCleaner to complete before closing cluster
flinkbot edited a comment on pull request #17572: URL: https://github.com/apache/flink/pull/17572#issuecomment-952040222 ## CI report: * 5c20827f354a6f033d0ccfd71d353f9e5f6669c7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25475) * 2e33c87239fdd6a082ed3ae41a4e2c49c5e92031 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17572: [FLINK-23647][checkpointing] Wait for CheckpointsCleaner to complete before closing cluster
flinkbot edited a comment on pull request #17572: URL: https://github.com/apache/flink/pull/17572#issuecomment-952040222 ## CI report: * 5c20827f354a6f033d0ccfd71d353f9e5f6669c7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25475) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17572: [FLINK-23647][checkpointing] Wait for CheckpointsCleaner to complete before closing cluster
flinkbot edited a comment on pull request #17572: URL: https://github.com/apache/flink/pull/17572#issuecomment-952040222 ## CI report: * 5c20827f354a6f033d0ccfd71d353f9e5f6669c7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25475) * 2e33c87239fdd6a082ed3ae41a4e2c49c5e92031 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17572: [FLINK-23647][checkpointing] Wait for CheckpointsCleaner to complete before closing cluster
flinkbot edited a comment on pull request #17572: URL: https://github.com/apache/flink/pull/17572#issuecomment-952040222 ## CI report: * 5c20827f354a6f033d0ccfd71d353f9e5f6669c7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25475) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17571: [FLINK-23015][table-runtime-blink] Implement streaming window Deduplicate operator
flinkbot edited a comment on pull request #17571: URL: https://github.com/apache/flink/pull/17571#issuecomment-951900326 ## CI report: * 62c680fc5948737a1cfce1730886f54501deff84 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25470) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17572: [FLINK-23647][checkpointing] Wait for CheckpointsCleaner to complete before closing cluster
flinkbot edited a comment on pull request #17572: URL: https://github.com/apache/flink/pull/17572#issuecomment-952040222 ## CI report: * 5c20827f354a6f033d0ccfd71d353f9e5f6669c7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25475) * 2e33c87239fdd6a082ed3ae41a4e2c49c5e92031 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17572: [FLINK-23647][checkpointing] Wait for CheckpointsCleaner to complete before closing cluster
flinkbot edited a comment on pull request #17572: URL: https://github.com/apache/flink/pull/17572#issuecomment-952040222 ## CI report: * 5c20827f354a6f033d0ccfd71d353f9e5f6669c7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25475) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #17572: [FLINK-23647][checkpointing] Wait for CheckpointsCleaner to complete before closing cluster
flinkbot edited a comment on pull request #17572: URL: https://github.com/apache/flink/pull/17572#issuecomment-952040222 ## CI report: * 5c20827f354a6f033d0ccfd71d353f9e5f6669c7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=25475) * 2e33c87239fdd6a082ed3ae41a4e2c49c5e92031 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
