[GitHub] [flink] dannycranmer commented on a change in pull request #18603: [FLINK-25610][connector/firehose] Adding table api base for kinesis firehose
dannycranmer commented on a change in pull request #18603: URL: https://github.com/apache/flink/pull/18603#discussion_r803703950 ## File path: flink-connectors/flink-connector-aws-kinesis-firehose/src/main/java/org/apache/flink/connector/firehose/table/KinesisFirehoseConnectorOptions.java ## @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.connector.firehose.table; + +import org.apache.flink.annotation.PublicEvolving; +import org.apache.flink.configuration.ConfigOption; +import org.apache.flink.configuration.ConfigOptions; +import org.apache.flink.connector.base.table.AsyncSinkConnectorOptions; + +/** Options for the Kinesis firehose connector. */ +@PublicEvolving +public class KinesisFirehoseConnectorOptions extends AsyncSinkConnectorOptions { + +public static final ConfigOption DELIVERY_STREAM = +ConfigOptions.key("delivery-stream") +.stringType() +.noDefaultValue() +.withDescription( +"Name of the Kinesis firehose delivery stream backing this table."); Review comment: Capital F for "Firehose" ## File path: flink-connectors/flink-connector-aws-kinesis-firehose/src/main/java/org/apache/flink/connector/firehose/table/KinesisFirehoseConnectorOptions.java ## @@ -0,0 +1,43 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.connector.firehose.table; + +import org.apache.flink.annotation.PublicEvolving; +import org.apache.flink.configuration.ConfigOption; +import org.apache.flink.configuration.ConfigOptions; +import org.apache.flink.connector.base.table.AsyncSinkConnectorOptions; + +/** Options for the Kinesis firehose connector. */ +@PublicEvolving +public class KinesisFirehoseConnectorOptions extends AsyncSinkConnectorOptions { + +public static final ConfigOption DELIVERY_STREAM = +ConfigOptions.key("delivery-stream") +.stringType() +.noDefaultValue() +.withDescription( +"Name of the Kinesis firehose delivery stream backing this table."); + +public static final ConfigOption SINK_FAIL_ON_ERROR = +ConfigOptions.key("sink.fail-on-error") +.booleanType() +.defaultValue(false) +.withDescription( +"Optional fail on error value for kinesis sink, default is false"); Review comment: typo "kinesis" > "Kinesis Firehose" ## File path: flink-connectors/flink-connector-aws-kinesis-firehose/src/main/java/org/apache/flink/connector/firehose/table/KinesisFirehoseDynamicSink.java ## @@ -0,0 +1,185 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDI
[GitHub] [flink] zentol commented on a change in pull request #18705: [FLINK-26073][Connectors][Twitter] Remove Twitter connector
zentol commented on a change in pull request #18705: URL: https://github.com/apache/flink/pull/18705#discussion_r803742070 ## File path: flink-examples/flink-examples-streaming/pom.xml ## @@ -168,20 +162,6 @@ under the License. unpack - Review comment: hmm, doesn't seem to do anything. The twitter jar doesn't contain anything in the included directory after all. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] CrynetLogistics commented on a change in pull request #18553: [FLINK-25846][FLINK-25848] Async Sink does not gracefully shutdown on Cancel, KDS Sink does not fast fail when invalid co
CrynetLogistics commented on a change in pull request #18553: URL: https://github.com/apache/flink/pull/18553#discussion_r803742615 ## File path: flink-connectors/flink-connector-aws-kinesis-firehose/src/test/java/org/apache/flink/connector/firehose/sink/KinesisFirehoseSinkITCase.java ## @@ -78,42 +76,36 @@ @Before public void setup() throws Exception { System.setProperty(SdkSystemSetting.CBOR_ENABLED.property(), "false"); -httpClient = AWSServicesTestUtils.createHttpClient(mockFirehoseContainer.getEndpoint()); -s3AsyncClient = createS3Client(mockFirehoseContainer.getEndpoint(), httpClient); -firehoseAsyncClient = getFirehoseClient(mockFirehoseContainer.getEndpoint(), httpClient); -iamAsyncClient = createIamClient(mockFirehoseContainer.getEndpoint(), httpClient); +s3HttpClient = createHttpClient(mockFirehoseContainer.getEndpoint()); +firehoseHttpClient = createHttpClient(mockFirehoseContainer.getEndpoint()); +iamHttpClient = createHttpClient(mockFirehoseContainer.getEndpoint()); +s3AsyncClient = createS3Client(mockFirehoseContainer.getEndpoint(), s3HttpClient); +firehoseAsyncClient = +getFirehoseClient(mockFirehoseContainer.getEndpoint(), firehoseHttpClient); +iamAsyncClient = createIamClient(mockFirehoseContainer.getEndpoint(), iamHttpClient); +env = StreamExecutionEnvironment.getExecutionEnvironment(); Review comment: Any ideas on what might be the root cause here? This is something I spent Wednesday, Thursday and Friday looking at last week, with unfortunately no breakthrough. However, I did find that separate http clients resolved the issue. Would you be able to assist? I believe the root cause is not in this codebase but in one of the sdk clients. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] metaswirl commented on a change in pull request #18689: [FLINK-21439][runtime] Exception history adaptive scheduler
metaswirl commented on a change in pull request #18689: URL: https://github.com/apache/flink/pull/18689#discussion_r803742746 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/StateWithExecutionGraph.java ## @@ -148,6 +162,33 @@ public Logger getLogger() { return logger; } +protected Throwable extractError(TaskExecutionStateTransition taskExecutionStateTransition) { Review comment: It's a small method that's easy to understand. Why would you want to remove it? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18153: [FLINK-25568][connectors/elasticsearch] Add Elasticsearch 7 Source
flinkbot edited a comment on pull request #18153: URL: https://github.com/apache/flink/pull/18153#issuecomment-997756404 ## CI report: * f34977eb02ad9dceb02c108cdbf344ebcf75e737 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31113) * 7cecfb69874b3e340b01232bc780eab2e3a43765 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31127) * cb936e98d7be5f4898c3c087dd28d6c2a907304b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31142) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18624: [FLINK-25388][table-planner] Add consumedOptions to ExecNodeMetadata
flinkbot edited a comment on pull request #18624: URL: https://github.com/apache/flink/pull/18624#issuecomment-1029176049 ## CI report: * 1d0ac8779a7961062b728ad0e0dcc8c227dacbe4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31105) * 135b18a48b23afa0a28cde1ff570304bd0326101 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31148) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18669: [FLINK-25943][connector/common] Add buffered requests to snapshot state in AsyncSyncWriter.
flinkbot edited a comment on pull request #18669: URL: https://github.com/apache/flink/pull/18669#issuecomment-1033040800 ## CI report: * 5a104bf0fe75823e4946852d66fb5ad117d585b6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31104) * b35192da26215222b167cdffbb9f45091fb4c559 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31125) * 0821dc818041b4be18d1327b9f5d5ca18deff594 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31139) * f7b2480909019bf786dcc9d58680be202458e2c4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18693: [FLINK-25987][state/changelog] Make lastAppendedSqn optional
flinkbot edited a comment on pull request #18693: URL: https://github.com/apache/flink/pull/18693#issuecomment-1034140883 ## CI report: * 71fd9a5a7917a6bc423a45ca3deec8b84007c9e3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31089) * 3e33bf10866f986c2778b5ed8f9fb183aadeb11d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31137) * b34ba68d832332d79137cfe1ff45cd1efae38e41 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment
flinkbot edited a comment on pull request #18702: URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544 ## CI report: * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) * 2cc0ad9e6652642762f48c030a2ad5339881123b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18706: [FLINK-26072][Connectors][NiFi] Mark NiFi Source and Sink as deprecated
flinkbot edited a comment on pull request #18706: URL: https://github.com/apache/flink/pull/18706#issuecomment-1034953890 ## CI report: * bf78620ba038d62e133c9ef66eb39e0e2313561f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31143) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18707: [FLINK-25995][table-planner] Make implicit assumption of SQL local hash explicit
flinkbot edited a comment on pull request #18707: URL: https://github.com/apache/flink/pull/18707#issuecomment-1034994010 ## CI report: * 5e2128cc02aa0cfdb9b074edd5b0cab02c3daed6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31149) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] metaswirl commented on a change in pull request #18689: [FLINK-21439][runtime] Exception history adaptive scheduler
metaswirl commented on a change in pull request #18689: URL: https://github.com/apache/flink/pull/18689#discussion_r803748090 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/executiongraph/DefaultExecutionGraph.java ## @@ -1576,4 +1576,15 @@ public ExecutionDeploymentListener getExecutionDeploymentListener() { public boolean isDynamic() { return isDynamic; } + +@Override +public Optional getExecutionVertexId(ExecutionAttemptID id) { +Execution execution = this.getRegisteredExecutions().get(id); +return Optional.ofNullable(execution).map(Execution::getVertex).map(ExecutionVertex::getID); +} + +@Override +public Optional getExecutionVertex(final ExecutionVertexID executionVertexId) { Review comment: The question is how we want to react, when there is no ExecutionVertex? Is this an error case? (It appears at least in the tests.) If it's not an error case, do we want to throw an exception here and then catch it and make it an Optional at a later point? Sounds rather complicated. I rather have an Optional to start with. Also, the method `getExecutionVertexOrThrow` is part of `InternalExecutionGraphAccessor`. It is not accessible from `ExecutionGraph`. (We can of course create a copy of it in the ExecutionGraph.) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] metaswirl commented on a change in pull request #18689: [FLINK-21439][runtime] Exception history adaptive scheduler
metaswirl commented on a change in pull request #18689: URL: https://github.com/apache/flink/pull/18689#discussion_r803748913 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/Executing.java ## @@ -80,49 +89,53 @@ public void cancel() { getExecutionGraph(), getExecutionGraphHandler(), getOperatorCoordinatorHandler()); } +private void handleFailure(Failure failure) { +failureCollection.add(failure); +FailureResult failureResult = context.howToHandleFailure(failure); +transitionOnFailure(failureResult); +} + @Override public void handleGlobalFailure(Throwable cause) { -handleAnyFailure(cause); +handleFailure(Failure.createGlobal(cause)); } -private void handleAnyFailure(Throwable cause) { -final FailureResult failureResult = context.howToHandleFailure(cause); +@Override +boolean updateTaskExecutionState(TaskExecutionStateTransition taskExecutionStateTransition) { +final boolean successfulUpdate = +getExecutionGraph().updateState(taskExecutionStateTransition); + +if (successfulUpdate +&& taskExecutionStateTransition.getExecutionState() == ExecutionState.FAILED) { +handleFailure( +Failure.createLocal( +extractError(taskExecutionStateTransition), + extractExecutionVertexID(taskExecutionStateTransition))); +} + +return successfulUpdate; Review comment: Sounds good, will check. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] XComp commented on pull request #18637: [FLINK-25433][runtime] Adds retry mechanism to DefaultResourceCleaner
XComp commented on pull request #18637: URL: https://github.com/apache/flink/pull/18637#issuecomment-1035005706 Rebased the branch after FLINK-26065 is resolved. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18153: [FLINK-25568][connectors/elasticsearch] Add Elasticsearch 7 Source
flinkbot edited a comment on pull request #18153: URL: https://github.com/apache/flink/pull/18153#issuecomment-997756404 ## CI report: * f34977eb02ad9dceb02c108cdbf344ebcf75e737 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31113) * 7cecfb69874b3e340b01232bc780eab2e3a43765 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31127) * cb936e98d7be5f4898c3c087dd28d6c2a907304b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31142) * aab3087762ad1405ff910b1573f7307317a3ccd4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18637: [FLINK-25433][runtime] Adds retry mechanism to DefaultResourceCleaner
flinkbot edited a comment on pull request #18637: URL: https://github.com/apache/flink/pull/18637#issuecomment-1030235273 ## CI report: * 429fe69c2ea086a2e13a7b99d922085618299396 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31062) * 7a1f59c89cc282433be0c496e30275f004be7a89 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18669: [FLINK-25943][connector/common] Add buffered requests to snapshot state in AsyncSyncWriter.
flinkbot edited a comment on pull request #18669: URL: https://github.com/apache/flink/pull/18669#issuecomment-1033040800 ## CI report: * 5a104bf0fe75823e4946852d66fb5ad117d585b6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31104) * b35192da26215222b167cdffbb9f45091fb4c559 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31125) * 0821dc818041b4be18d1327b9f5d5ca18deff594 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31139) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18693: [FLINK-25987][state/changelog] Make lastAppendedSqn optional
flinkbot edited a comment on pull request #18693: URL: https://github.com/apache/flink/pull/18693#issuecomment-1034140883 ## CI report: * 71fd9a5a7917a6bc423a45ca3deec8b84007c9e3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31089) * 3e33bf10866f986c2778b5ed8f9fb183aadeb11d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31137) * b34ba68d832332d79137cfe1ff45cd1efae38e41 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31150) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18692: [FLINK-26015] Fixes object store bug
flinkbot edited a comment on pull request #18692: URL: https://github.com/apache/flink/pull/18692#issuecomment-1034131249 ## CI report: * 7645e2a83b9789816399536fbf369dc65eafcbf7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31092) * b20d5b2c72912bd1d97cd61d3c38da95c52700a8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment
flinkbot edited a comment on pull request #18702: URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544 ## CI report: * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) * 6b2d3af9ac7e956e358c6058809f49368d6c9c0a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18707: [FLINK-25995][table-planner] Make implicit assumption of SQL local hash explicit
flinkbot edited a comment on pull request #18707: URL: https://github.com/apache/flink/pull/18707#issuecomment-1034994010 ## CI report: * 5e2128cc02aa0cfdb9b074edd5b0cab02c3daed6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31149) * 7e5e0f5f2aa79f47e79357721719cdcb971e32b3 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18706: [FLINK-26072][Connectors][NiFi] Mark NiFi Source and Sink as deprecated
flinkbot edited a comment on pull request #18706: URL: https://github.com/apache/flink/pull/18706#issuecomment-1034953890 ## CI report: * bf78620ba038d62e133c9ef66eb39e0e2313561f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31143) * f104642af11fced6f110afd39424871efa89ae9f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24275) Allow idempotent job cancellation
[ https://issues.apache.org/jira/browse/FLINK-24275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17490310#comment-17490310 ] Joe Moser commented on FLINK-24275: --- [~chesnay] i was referring to {_}409{_}, so not a successful operation. > Allow idempotent job cancellation > - > > Key: FLINK-24275 > URL: https://issues.apache.org/jira/browse/FLINK-24275 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Reporter: Austin Cawley-Edwards >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > As a user of Flink, I want to be able to cancel a job from an external system > in a fault-tolerant way without guessing if the job has already been > cancelled. > > Currently, the cancel endpoint (PATCH /jobs/:jobid?mode=cancel) will return a > 404 if the job is already cancelled. This makes it hard to detect if the job > truly doesn't exist, or if it is already in the desired state. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18153: [FLINK-25568][connectors/elasticsearch] Add Elasticsearch 7 Source
flinkbot edited a comment on pull request #18153: URL: https://github.com/apache/flink/pull/18153#issuecomment-997756404 ## CI report: * f34977eb02ad9dceb02c108cdbf344ebcf75e737 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31113) * 7cecfb69874b3e340b01232bc780eab2e3a43765 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31127) * cb936e98d7be5f4898c3c087dd28d6c2a907304b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31142) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18637: [FLINK-25433][runtime] Adds retry mechanism to DefaultResourceCleaner
flinkbot edited a comment on pull request #18637: URL: https://github.com/apache/flink/pull/18637#issuecomment-1030235273 ## CI report: * 429fe69c2ea086a2e13a7b99d922085618299396 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31062) * 7a1f59c89cc282433be0c496e30275f004be7a89 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31151) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] RocMarshal commented on a change in pull request #18653: [FLINK-25825][connector-jdbc] MySqlCatalogITCase fails on azure
RocMarshal commented on a change in pull request #18653: URL: https://github.com/apache/flink/pull/18653#discussion_r803760543 ## File path: flink-connectors/flink-connector-jdbc/src/test/java/org/apache/flink/connector/jdbc/table/UnsignedTypeConversionITCase.java ## @@ -94,70 +103,76 @@ .withDatabaseName(DEFAULT_DB_NAME) .withLogConsumer(new Slf4jLogConsumer(LOGGER)); -private Connection connection; private StreamTableEnvironment tEnv; @Test public void testUnsignedType() throws Exception { prepare(); // write data to db -tEnv.executeSql(String.format("insert into jdbc_sink select %s from data", COLUMNS)) +tEnv.executeSql( +String.format( +"insert into jdbc_sink select %s from data", +String.join(",", COLUMNS))) .await(); // read data from db using jdbc connection and compare -PreparedStatement query = -connection.prepareStatement( -String.format("select %s from %s", COLUMNS, TABLE_NAME)); -ResultSet resultSet = query.executeQuery(); -while (resultSet.next()) { - Assertions.assertThat(resultSet.getObject("tiny_c")).isEqualTo(127); - Assertions.assertThat(resultSet.getObject("tiny_un_c")).isEqualTo(255); - Assertions.assertThat(resultSet.getObject("small_c")).isEqualTo(32767); - Assertions.assertThat(resultSet.getObject("small_un_c")).isEqualTo(65535); - Assertions.assertThat(resultSet.getObject("int_c")).isEqualTo(2147483647); - Assertions.assertThat(resultSet.getObject("int_un_c")).isEqualTo(4294967295L); - Assertions.assertThat(resultSet.getObject("big_c")).isEqualTo(9223372036854775807L); -Assertions.assertThat(resultSet.getObject("big_un_c")) -.isEqualTo(new BigInteger("18446744073709551615")); +try (Connection con = + DriverManager.getConnection(MYSQL_CONTAINER.getJdbcUrl(), USER, PASSWORD); +PreparedStatement ps = +con.prepareStatement( +String.format( +"select %s from %s", +String.join(",", COLUMNS), TABLE_NAME))) { +ResultSet resultSet = ps.executeQuery(); +while (resultSet.next()) { +assertThat(resultSet.getObject("tiny_c")).isEqualTo(127); +assertThat(resultSet.getObject("tiny_un_c")).isEqualTo(255); +assertThat(resultSet.getObject("small_c")).isEqualTo(32767); +assertThat(resultSet.getObject("small_un_c")).isEqualTo(65535); +assertThat(resultSet.getObject("int_c")).isEqualTo(2147483647); + assertThat(resultSet.getObject("int_un_c")).isEqualTo(4294967295L); + assertThat(resultSet.getObject("big_c")).isEqualTo(9223372036854775807L); +assertThat(resultSet.getObject("big_un_c")) +.isEqualTo(new BigInteger("18446744073709551615")); +} } // read data from db using flink and compare Iterator collected = -tEnv.executeSql(String.format("select %s from jdbc_source", COLUMNS)).collect(); +tEnv.executeSql( +String.format( +"select %s from jdbc_source", String.join(",", COLUMNS))) +.collect(); List result = CollectionUtil.iteratorToList(collected); -List expected = Collections.singletonList(Row.ofKind(RowKind.INSERT, ROW)); -Assertions.assertThat(result).isEqualTo(expected); - -connection.close(); +assertThat(result).containsOnly(Row.ofKind(RowKind.INSERT, ROW)); } private void prepare() throws Exception { -MYSQL_CONTAINER.start(); - -connection = DriverManager.getConnection(MYSQL_CONTAINER.getJdbcUrl(), USER, PASSWORD); tEnv = StreamTableEnvironment.create(StreamExecutionEnvironment.getExecutionEnvironment()); Review comment: @slinkydeveloper Done. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18669: [FLINK-25943][connector/common] Add buffered requests to snapshot state in AsyncSyncWriter.
flinkbot edited a comment on pull request #18669: URL: https://github.com/apache/flink/pull/18669#issuecomment-1033040800 ## CI report: * 5a104bf0fe75823e4946852d66fb5ad117d585b6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31104) * b35192da26215222b167cdffbb9f45091fb4c559 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31125) * 0821dc818041b4be18d1327b9f5d5ca18deff594 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31139) * f7b2480909019bf786dcc9d58680be202458e2c4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18653: [FLINK-25825][connector-jdbc] MySqlCatalogITCase fails on azure
flinkbot edited a comment on pull request #18653: URL: https://github.com/apache/flink/pull/18653#issuecomment-1032212433 ## CI report: * 764c5fdf856b0930f7fdf8d470256e84d05a646a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31117) * 1c96c6553cd12da3d699383ec26353315864871a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dmvk commented on a change in pull request #18689: [FLINK-21439][runtime] Exception history adaptive scheduler
dmvk commented on a change in pull request #18689: URL: https://github.com/apache/flink/pull/18689#discussion_r803761345 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/StateWithExecutionGraph.java ## @@ -148,6 +162,33 @@ public Logger getLogger() { return logger; } +protected Throwable extractError(TaskExecutionStateTransition taskExecutionStateTransition) { +Throwable cause = taskExecutionStateTransition.getError(userCodeClassLoader); +if (cause == null) { +cause = new FlinkException("Unknown failure cause. Probably related to FLINK-21376."); +} +return cause; +} + +protected Optional extractExecutionVertexID( +TaskExecutionStateTransition taskExecutionStateTransition) { +return executionGraph.getExecutionVertexId(taskExecutionStateTransition.getID()); Review comment: If you want to go this way, I'd rather do something along the lines of `Optional ExecutionGraph#getExecution(...)`, because that's something we'd be able to reuse on other places immediately; But feel free to ignore this, there's nothing wrong with this approach; I mostly feel uneasy about adding new methods to already overloaded interfaces, but in this case it doesn't add complexity -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment
flinkbot edited a comment on pull request #18702: URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544 ## CI report: * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) * 425f2f16c73e8302ec0be19b6c9994cfccfee40d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18701: [FLINK-26071][table-api-java][table-planner] Now Planner#compilePlan fails if the plan cannot be serialized
flinkbot edited a comment on pull request #18701: URL: https://github.com/apache/flink/pull/18701#issuecomment-1034766277 ## CI report: * a3fdc9763094051769e040d5211db40196fe493c Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31116) * e001ada7d0947721903b47ce861fe0ac846ea2db Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31135) * c2e56d970d3d0e19d4534ca60f98d21b9d0764d1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31147) * c2b74e16dc3528608d017847c2f9856cbabf227e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18692: [FLINK-26015] Fixes object store bug
flinkbot edited a comment on pull request #18692: URL: https://github.com/apache/flink/pull/18692#issuecomment-1034131249 ## CI report: * 7645e2a83b9789816399536fbf369dc65eafcbf7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31092) * b20d5b2c72912bd1d97cd61d3c38da95c52700a8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31152) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18706: [FLINK-26072][Connectors][NiFi] Mark NiFi Source and Sink as deprecated
flinkbot edited a comment on pull request #18706: URL: https://github.com/apache/flink/pull/18706#issuecomment-1034953890 ## CI report: * bf78620ba038d62e133c9ef66eb39e0e2313561f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31143) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18705: [FLINK-26073][Connectors][Twitter] Remove Twitter connector
flinkbot edited a comment on pull request #18705: URL: https://github.com/apache/flink/pull/18705#issuecomment-1034936625 ## CI report: * 0559b7a557c7e34f4564c4280ae0b9d2977b42da Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31140) * b303ba40b73e934b6f113fd8875c0725f1256913 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18707: [FLINK-25995][table-planner] Make implicit assumption of SQL local hash explicit
flinkbot edited a comment on pull request #18707: URL: https://github.com/apache/flink/pull/18707#issuecomment-1034994010 ## CI report: * 5e2128cc02aa0cfdb9b074edd5b0cab02c3daed6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31149) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] MartijnVisser commented on pull request #18706: [FLINK-26072][Connectors][NiFi] Mark NiFi Source and Sink as deprecated
MartijnVisser commented on pull request #18706: URL: https://github.com/apache/flink/pull/18706#issuecomment-1035022729 > They should also be marked as deprecate in the documentation. Thanks, I've added a deprecation warning to the docs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18391: [FLINK-25478][chaneglog] Correct the state register logic of ChangelogStateBackendHandle
Myasuka commented on a change in pull request #18391: URL: https://github.com/apache/flink/pull/18391#discussion_r803767353 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java ## @@ -415,7 +440,7 @@ KeyedStateHandle deserializeKeyedStateHandle( int baseSize = dis.readInt(); List base = new ArrayList<>(baseSize); for (int i = 0; i < baseSize; i++) { -base.add(deserializeKeyedStateHandle(dis, context)); + base.add(Preconditions.checkNotNull(deserializeKeyedStateHandle(dis, context))); Review comment: Since we already write the size of collection out, it's really strange that the materialized collection having `null` as its elements. I think a check here could act as a safety guard here. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18153: [FLINK-25568][connectors/elasticsearch] Add Elasticsearch 7 Source
flinkbot edited a comment on pull request #18153: URL: https://github.com/apache/flink/pull/18153#issuecomment-997756404 ## CI report: * f34977eb02ad9dceb02c108cdbf344ebcf75e737 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31113) * 7cecfb69874b3e340b01232bc780eab2e3a43765 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31127) * cb936e98d7be5f4898c3c087dd28d6c2a907304b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31142) * aab3087762ad1405ff910b1573f7307317a3ccd4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dmvk commented on a change in pull request #18689: [FLINK-21439][runtime] Exception history adaptive scheduler
dmvk commented on a change in pull request #18689: URL: https://github.com/apache/flink/pull/18689#discussion_r803770073 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/StopWithSavepoint.java ## @@ -134,30 +143,53 @@ public JobStatus getJobStatus() { return JobStatus.RUNNING; } +private void handleFailure(Failure failure) { +failureCollection.add(failure); +FailureResult failureResult = context.howToHandleFailure(failure); +transitionOnFailure(failureResult); +} + @Override public void handleGlobalFailure(Throwable cause) { -handleAnyFailure(cause); +handleFailure(Failure.createGlobal(cause)); } @Override boolean updateTaskExecutionState(TaskExecutionStateTransition taskExecutionStateTransition) { final boolean successfulUpdate = getExecutionGraph().updateState(taskExecutionStateTransition); -if (successfulUpdate) { -if (taskExecutionStateTransition.getExecutionState() == ExecutionState.FAILED) { -Throwable cause = taskExecutionStateTransition.getError(userCodeClassLoader); -handleAnyFailure( -cause == null -? new FlinkException( -"Unknown failure cause. Probably related to FLINK-21376.") -: cause); -} +if (successfulUpdate +&& taskExecutionStateTransition.getExecutionState() == ExecutionState.FAILED) { +handleFailure( +Failure.createLocal( +extractError(taskExecutionStateTransition), + extractExecutionVertexID(taskExecutionStateTransition))); } return successfulUpdate; } +private void transitionOnFailure(FailureResult failureResult) { Review comment: nvm, I've missed that `StateWithExecutionGraph` can't always transition to `restarting` and `failing` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18649: [FLINK-25844][table-api-java] Introduce StatementSet#compilePlan
flinkbot edited a comment on pull request #18649: URL: https://github.com/apache/flink/pull/18649#issuecomment-1031570072 ## CI report: * 114478aae924d51669e7809bc26c6c82c4d5955a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31119) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18653: [FLINK-25825][connector-jdbc] MySqlCatalogITCase fails on azure
flinkbot edited a comment on pull request #18653: URL: https://github.com/apache/flink/pull/18653#issuecomment-1032212433 ## CI report: * 764c5fdf856b0930f7fdf8d470256e84d05a646a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31117) * 1c96c6553cd12da3d699383ec26353315864871a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31153) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18664: [FLINK-25907][runtime][security] Add pluggable delegation token manager
flinkbot edited a comment on pull request #18664: URL: https://github.com/apache/flink/pull/18664#issuecomment-1032585099 ## CI report: * 357ed24c86356875da2a328ea57546ff86b39d5f Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31120) * 01046c38b403a797e9228a434d194144e6eed82b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31133) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18669: [FLINK-25943][connector/common] Add buffered requests to snapshot state in AsyncSyncWriter.
flinkbot edited a comment on pull request #18669: URL: https://github.com/apache/flink/pull/18669#issuecomment-1033040800 ## CI report: * 5a104bf0fe75823e4946852d66fb5ad117d585b6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31104) * b35192da26215222b167cdffbb9f45091fb4c559 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31125) * 0821dc818041b4be18d1327b9f5d5ca18deff594 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31139) * f7b2480909019bf786dcc9d58680be202458e2c4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31154) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment
flinkbot edited a comment on pull request #18702: URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544 ## CI report: * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18705: [FLINK-26073][Connectors][Twitter] Remove Twitter connector
flinkbot edited a comment on pull request #18705: URL: https://github.com/apache/flink/pull/18705#issuecomment-1034936625 ## CI report: * 0559b7a557c7e34f4564c4280ae0b9d2977b42da Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31140) * b303ba40b73e934b6f113fd8875c0725f1256913 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31155) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18701: [FLINK-26071][table-api-java][table-planner] Now Planner#compilePlan fails if the plan cannot be serialized
flinkbot edited a comment on pull request #18701: URL: https://github.com/apache/flink/pull/18701#issuecomment-1034766277 ## CI report: * a3fdc9763094051769e040d5211db40196fe493c Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31116) * e001ada7d0947721903b47ce861fe0ac846ea2db Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31135) * c2e56d970d3d0e19d4534ca60f98d21b9d0764d1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31147) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18706: [FLINK-26072][Connectors][NiFi] Mark NiFi Source and Sink as deprecated
flinkbot edited a comment on pull request #18706: URL: https://github.com/apache/flink/pull/18706#issuecomment-1034953890 ## CI report: * bf78620ba038d62e133c9ef66eb39e0e2313561f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31143) * f104642af11fced6f110afd39424871efa89ae9f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18707: [FLINK-25995][table-planner] Make implicit assumption of SQL local hash explicit
flinkbot edited a comment on pull request #18707: URL: https://github.com/apache/flink/pull/18707#issuecomment-1034994010 ## CI report: * 5e2128cc02aa0cfdb9b074edd5b0cab02c3daed6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31149) * 7e5e0f5f2aa79f47e79357721719cdcb971e32b3 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] slinkydeveloper commented on a change in pull request #18653: [FLINK-25825][connector-jdbc] MySqlCatalogITCase fails on azure
slinkydeveloper commented on a change in pull request #18653: URL: https://github.com/apache/flink/pull/18653#discussion_r803774457 ## File path: flink-connectors/flink-connector-jdbc/src/test/java/org/apache/flink/connector/jdbc/table/UnsignedTypeConversionITCase.java ## @@ -21,161 +21,124 @@ import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment; import org.apache.flink.table.api.DataTypes; import org.apache.flink.table.api.Table; +import org.apache.flink.table.api.TableEnvironment; import org.apache.flink.table.api.bridge.java.StreamTableEnvironment; import org.apache.flink.test.util.AbstractTestBase; import org.apache.flink.types.Row; +import org.apache.flink.types.RowKind; +import org.apache.flink.util.CloseableIterator; import org.apache.flink.util.CollectionUtil; -import ch.vorburger.exec.ManagedProcessException; -import ch.vorburger.mariadb4j.DB; -import ch.vorburger.mariadb4j.DBConfigurationBuilder; -import org.junit.After; -import org.junit.Before; -import org.junit.BeforeClass; +import org.junit.ClassRule; import org.junit.Test; import org.slf4j.Logger; import org.slf4j.LoggerFactory; +import org.testcontainers.containers.MySQLContainer; +import org.testcontainers.containers.output.Slf4jLogConsumer; +import org.testcontainers.utility.DockerImageName; import java.math.BigDecimal; -import java.math.BigInteger; import java.sql.Connection; import java.sql.DriverManager; import java.sql.PreparedStatement; import java.sql.ResultSet; import java.sql.SQLException; -import java.sql.Statement; -import java.util.Collections; -import java.util.Iterator; +import java.util.Arrays; +import java.util.HashMap; import java.util.List; -import java.util.stream.Collectors; +import java.util.Map; -import static org.apache.flink.table.api.Expressions.row; -import static org.junit.Assert.assertEquals; +import static java.lang.String.format; +import static java.lang.String.join; +import static org.assertj.core.api.Assertions.assertThat; /** - * Test unsigned type conversion between Flink and JDBC driver mysql, the test underlying use - * MariaDB to mock a DB which use mysql driver too. + * Test unsigned type conversion between Flink and JDBC driver mysql, the test underlying use MySQL + * to mock a DB. */ public class UnsignedTypeConversionITCase extends AbstractTestBase { -private static final Logger logger = +private static final Logger LOGGER = LoggerFactory.getLogger(UnsignedTypeConversionITCase.class); + +private static final DockerImageName MYSQL_57_IMAGE = DockerImageName.parse("mysql:5.7.34"); private static final String DEFAULT_DB_NAME = "test"; private static final String TABLE_NAME = "unsigned_test"; -private static final int INITIALIZE_DB_MAX_RETRY = 3; -private static DB db; -private static String dbUrl; -private static Connection connection; - -private StreamTableEnvironment tEnv; - -@BeforeClass -public static void prepareMariaDB() throws IllegalStateException { -boolean initDbSuccess = false; -int i = 0; -// The initialization of maria db instance is a little unstable according to past CI tests. -// Add retry logic here to avoid initialization failure. -while (i < INITIALIZE_DB_MAX_RETRY) { -try { -db = DB.newEmbeddedDB(DBConfigurationBuilder.newBuilder().build()); -db.start(); -dbUrl = db.getConfiguration().getURL(DEFAULT_DB_NAME); -connection = DriverManager.getConnection(dbUrl); -try (Statement statement = connection.createStatement()) { -statement.execute("CREATE DATABASE IF NOT EXISTS `" + DEFAULT_DB_NAME + "`;"); -ResultSet resultSet = -statement.executeQuery( -"SELECT SCHEMA_NAME FROM " -+ "INFORMATION_SCHEMA.SCHEMATA WHERE SCHEMA_NAME = '" -+ DEFAULT_DB_NAME -+ "';"); -if (resultSet.next()) { -String dbName = resultSet.getString(1); -initDbSuccess = DEFAULT_DB_NAME.equalsIgnoreCase(dbName); -} +private static final String USER = "root"; +private static final String PASSWORD = ""; +private static final List COLUMNS = +Arrays.asList( +"tiny_c", +"tiny_un_c", +"small_c", +"small_un_c", +"int_c", +"int_un_c", +"big_c", +"big_un_c"); + +private static final Map DEFAULT_CONTAINER_ENV_MAP = +new HashMap() { +{ +put("MYSQL_ROOT_HOST", "%"); } -} catch (E
[GitHub] [flink] vahmed-hamdy opened a new pull request #18708: Flink 25610 firehose connector backup
vahmed-hamdy opened a new pull request #18708: URL: https://github.com/apache/flink/pull/18708 ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change Please make sure both new and modified tests in this PR follows the conventions defined in our code quality guide: https://flink.apache.org/contributing/code-style-and-quality-common.html#testing *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] pnowojski commented on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment
pnowojski commented on pull request #18702: URL: https://github.com/apache/flink/pull/18702#issuecomment-1035032817 The initial version was causing a small performance regression. After a couple of optimisations there is no visible regression when the watermark alignment is not used. ``` "Benchmark","Mode","Threads","Samples","Score","Score Error (99.9%)","Unit","Param: sourceType" "org.apache.flink.benchmark.InputBenchmark.mapRebalanceMapSink","thrpt",1,30,9936.451429,255.258997,"ops/ms",LEGACY "org.apache.flink.benchmark.InputBenchmark.mapRebalanceMapSink","thrpt",1,30,14651.620943,383.589770,"ops/ms",F27_UNBOUNDED "org.apache.flink.benchmark.InputBenchmark.mapSink","thrpt",1,30,15669.405419,199.914977,"ops/ms",LEGACY "org.apache.flink.benchmark.InputBenchmark.mapSink","thrpt",1,30,30679.668113,265.650771,"ops/ms",F27_UNBOUNDED ``` However with watermark alignment enabled, most likely due to the `WatermarkTrackingOutput` wrapper, there is a small (under 3%) extra overhead visible in the `mapSink.F27_UNBOUNDED` benchmark. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18153: [FLINK-25568][connectors/elasticsearch] Add Elasticsearch 7 Source
flinkbot edited a comment on pull request #18153: URL: https://github.com/apache/flink/pull/18153#issuecomment-997756404 ## CI report: * f34977eb02ad9dceb02c108cdbf344ebcf75e737 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31113) * 7cecfb69874b3e340b01232bc780eab2e3a43765 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31127) * cb936e98d7be5f4898c3c087dd28d6c2a907304b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31142) * aab3087762ad1405ff910b1573f7307317a3ccd4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31156) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18708: Flink 25610 firehose connector backup
flinkbot commented on pull request #18708: URL: https://github.com/apache/flink/pull/18708#issuecomment-1035036984 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 6e752a3332dd1b2f7b256f54d6aa662f8bacc4ba (Thu Feb 10 15:13:18 UTC 2022) **Warnings:** * **5 pom.xml files were touched**: Check for build and licensing issues. * No documentation files were touched! Remember to keep the Flink docs up to date! * **Invalid pull request title: No valid Jira ID provided** Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-26076) Fix ArchUnit violations in SourceMetricsITCase
Dawid Wysakowicz created FLINK-26076: Summary: Fix ArchUnit violations in SourceMetricsITCase Key: FLINK-26076 URL: https://issues.apache.org/jira/browse/FLINK-26076 Project: Flink Issue Type: Improvement Components: Tests Reporter: Dawid Wysakowicz Assignee: Dawid Wysakowicz Fix For: 1.15.0 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] rkhachatryan commented on a change in pull request #18391: [FLINK-25478][chaneglog] Correct the state register logic of ChangelogStateBackendHandle
rkhachatryan commented on a change in pull request #18391: URL: https://github.com/apache/flink/pull/18391#discussion_r803781575 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java ## @@ -415,7 +440,7 @@ KeyedStateHandle deserializeKeyedStateHandle( int baseSize = dis.readInt(); List base = new ArrayList<>(baseSize); for (int i = 0; i < baseSize; i++) { -base.add(deserializeKeyedStateHandle(dis, context)); + base.add(Preconditions.checkNotNull(deserializeKeyedStateHandle(dis, context))); Review comment: I agree, it's quite strange; but nevertheless possible. Currently, this is a normal case; with a null-check, this will be an exceptional case, recovery will fail in what was a normal case before. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26076) Fix ArchUnit violations in Source(Sink)MetricsITCase
[ https://issues.apache.org/jira/browse/FLINK-26076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dawid Wysakowicz updated FLINK-26076: - Summary: Fix ArchUnit violations in Source(Sink)MetricsITCase (was: Fix ArchUnit violations in SourceMetricsITCase) > Fix ArchUnit violations in Source(Sink)MetricsITCase > > > Key: FLINK-26076 > URL: https://issues.apache.org/jira/browse/FLINK-26076 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz >Priority: Major > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18696: [hotfix][docs] project config pages
flinkbot edited a comment on pull request #18696: URL: https://github.com/apache/flink/pull/18696#issuecomment-1034481691 ## CI report: * 2921ed917d22b64cb99f05dfdd0f7e119dd86880 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31078) * ccb88999c20036553d971851cdc0c38670fb102a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18701: [FLINK-26071][table-api-java][table-planner] Now Planner#compilePlan fails if the plan cannot be serialized
flinkbot edited a comment on pull request #18701: URL: https://github.com/apache/flink/pull/18701#issuecomment-1034766277 ## CI report: * a3fdc9763094051769e040d5211db40196fe493c Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31116) * e001ada7d0947721903b47ce861fe0ac846ea2db Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31135) * c2e56d970d3d0e19d4534ca60f98d21b9d0764d1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31147) * c2b74e16dc3528608d017847c2f9856cbabf227e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment
flinkbot edited a comment on pull request #18702: URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544 ## CI report: * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) * f730132ffa59826d88df89d4bd6ac1e5c93639d3 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18706: [FLINK-26072][Connectors][NiFi] Mark NiFi Source and Sink as deprecated
flinkbot edited a comment on pull request #18706: URL: https://github.com/apache/flink/pull/18706#issuecomment-1034953890 ## CI report: * bf78620ba038d62e133c9ef66eb39e0e2313561f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31143) * f104642af11fced6f110afd39424871efa89ae9f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31157) * 12773467f6c3eebf4dcd2befcb0bb323c706b6c7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18707: [FLINK-25995][table-planner] Make implicit assumption of SQL local hash explicit
flinkbot edited a comment on pull request #18707: URL: https://github.com/apache/flink/pull/18707#issuecomment-1034994010 ## CI report: * 5e2128cc02aa0cfdb9b074edd5b0cab02c3daed6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31149) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18708: Flink 25610 firehose connector backup
flinkbot commented on pull request #18708: URL: https://github.com/apache/flink/pull/18708#issuecomment-1035039830 ## CI report: * 6e752a3332dd1b2f7b256f54d6aa662f8bacc4ba UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #18709: [FLINK-26076] Fix ArchUnit violations in Source(Sink)MetricsITCase
dawidwys commented on pull request #18709: URL: https://github.com/apache/flink/pull/18709#issuecomment-1035041800 Do you mind taking a look @zentol @JingGe ? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-24275) Allow idempotent job cancellation
[ https://issues.apache.org/jira/browse/FLINK-24275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17490322#comment-17490322 ] Chesnay Schepler commented on FLINK-24275: -- IMO we should only document things (in particular in the open api spec) if we can enforce&guarantee it in some way, which currently isn't possible due to technical limitations. This improvement is tracked in https://issues.apache.org/jira/browse/FLINK-26045. Instead we noted it in the release notes. > Allow idempotent job cancellation > - > > Key: FLINK-24275 > URL: https://issues.apache.org/jira/browse/FLINK-24275 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Reporter: Austin Cawley-Edwards >Assignee: Chesnay Schepler >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > > As a user of Flink, I want to be able to cancel a job from an external system > in a fault-tolerant way without guessing if the job has already been > cancelled. > > Currently, the cancel endpoint (PATCH /jobs/:jobid?mode=cancel) will return a > 404 if the job is already cancelled. This makes it hard to detect if the job > truly doesn't exist, or if it is already in the desired state. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] dawidwys opened a new pull request #18709: [FLINK-26076] Fix ArchUnit violations in Source(Sink)MetricsITCase
dawidwys opened a new pull request #18709: URL: https://github.com/apache/flink/pull/18709 ## What is the purpose of the change Fix ArchUnit violations regarding InMemoryReporter ## Brief change log * parameterize search methods with JobID instead of recreating the reporter (and cluster) for every test ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26076) Fix ArchUnit violations in Source(Sink)MetricsITCase
[ https://issues.apache.org/jira/browse/FLINK-26076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-26076: --- Labels: pull-request-available (was: ) > Fix ArchUnit violations in Source(Sink)MetricsITCase > > > Key: FLINK-26076 > URL: https://issues.apache.org/jira/browse/FLINK-26076 > Project: Flink > Issue Type: Improvement > Components: Tests >Reporter: Dawid Wysakowicz >Assignee: Dawid Wysakowicz >Priority: Major > Labels: pull-request-available > Fix For: 1.15.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] dmvk commented on a change in pull request #18689: [FLINK-21439][runtime] Exception history adaptive scheduler
dmvk commented on a change in pull request #18689: URL: https://github.com/apache/flink/pull/18689#discussion_r803786630 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/StateWithExecutionGraph.java ## @@ -148,6 +162,33 @@ public Logger getLogger() { return logger; } +protected Throwable extractError(TaskExecutionStateTransition taskExecutionStateTransition) { Review comment: After the change it would be used only within this class. I'd be OK with just making it private, so it's not exposed to the `AdaptiveScheduler`. But in general: ``` final boolean successfulUpdate = getExecutionGraph().updateState(taskExecutionStateTransition); if (successfulUpdate && taskExecutionStateTransition.getExecutionState() == ExecutionState.FAILED) { final Throwable error = taskExecutionStateTransition.getError(classLoader); if (error == null) { error = new FlinkException("Unknown failure cause. Probably related to FLINK-21376."); } handleFailure( Failure.createLocal( error, extractExecutionVertexID(taskExecutionStateTransition))); } return successfulUpdate; ``` is no less readable than ``` final boolean successfulUpdate = getExecutionGraph().updateState(taskExecutionStateTransition); if (successfulUpdate && taskExecutionStateTransition.getExecutionState() == ExecutionState.FAILED) { handleFailure( Failure.createLocal( extractError(taskExecutionStateTransition), extractExecutionVertexID(taskExecutionStateTransition))); } return successfulUpdate; ``` and it avoids additional noise / extra method (eg, when you read the code, you need to go to `extractError` to see what it really does). This will become especially visible when we get rid of the else clause. But again, I have no strong feelings if we just change it to private. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dawidwys merged pull request #18643: [hotfix][docs] replace deprecated class and fix typos
dawidwys merged pull request #18643: URL: https://github.com/apache/flink/pull/18643 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] slinkydeveloper opened a new pull request #18710: [FLINK-26055][table][annotations] Modified CompiledPlan#getFlinkVersion return type to FlinkVersion
slinkydeveloper opened a new pull request #18710: URL: https://github.com/apache/flink/pull/18710 ## What is the purpose of the change This PR exposes and persists the flink version in `CompiledPlan` from the `FlinkVersion` enum, rather than from the resource file in `flink-runtime` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18709: [FLINK-26076] Fix ArchUnit violations in Source(Sink)MetricsITCase
flinkbot commented on pull request #18709: URL: https://github.com/apache/flink/pull/18709#issuecomment-1035047288 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 17f2ffbe7f9a8dd9315e7ee8c1cc3acad30728bc (Thu Feb 10 15:22:32 UTC 2022) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26055) Use new FlinkVersion enum for serde plans
[ https://issues.apache.org/jira/browse/FLINK-26055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-26055: --- Labels: pull-request-available (was: ) > Use new FlinkVersion enum for serde plans > - > > Key: FLINK-26055 > URL: https://issues.apache.org/jira/browse/FLINK-26055 > Project: Flink > Issue Type: Sub-task >Reporter: Timo Walther >Assignee: Francesco Guardiani >Priority: Major > Labels: pull-request-available > > The new FlinkVersion enum should be used to encode a Flink version from and > to JSON. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] dmvk commented on a change in pull request #18689: [FLINK-21439][runtime] Exception history adaptive scheduler
dmvk commented on a change in pull request #18689: URL: https://github.com/apache/flink/pull/18689#discussion_r803790449 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/StateWithExecutionGraph.java ## @@ -148,6 +162,33 @@ public Logger getLogger() { return logger; } +protected Throwable extractError(TaskExecutionStateTransition taskExecutionStateTransition) { +Throwable cause = taskExecutionStateTransition.getError(userCodeClassLoader); +if (cause == null) { +cause = new FlinkException("Unknown failure cause. Probably related to FLINK-21376."); +} +return cause; +} + +protected Optional extractExecutionVertexID( +TaskExecutionStateTransition taskExecutionStateTransition) { +return executionGraph.getExecutionVertexId(taskExecutionStateTransition.getID()); +} + +protected static Optional convertFailures( +Function> lookup, +List failureCollection) { +if (failureCollection.isEmpty()) { +return Optional.empty(); +} +Failure first = failureCollection.remove(0); +Set entries = new HashSet<>(); +for (Failure failure : failureCollection) { +entries.add(failure.toExceptionHistoryEntry(lookup)); Review comment: I guess it could happen if TM doesn't get the ack response from the JM (some kind of transient network failure). But I'm not really sure whether TM retries than. If it adds too much complexity I think it's OK not to bother with that. Maybe we should just use `List` instead of `Set` for collecting the entries to make the fact that we're not trying to be smart explicit. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18667: [FLINK-25990][table] Expose uid generator for DataStream/Transformation providers
flinkbot edited a comment on pull request #18667: URL: https://github.com/apache/flink/pull/18667#issuecomment-1032823071 ## CI report: * 4af7f825a1bcbed0a6fda02c9445d9288a175e30 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30988) * ca7a596b31ce7e89969dcf7bf58b9a7cdea38310 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26060) Make Python specific exec nodes unsupported
[ https://issues.apache.org/jira/browse/FLINK-26060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francesco Guardiani updated FLINK-26060: Component/s: Table SQL / Runtime > Make Python specific exec nodes unsupported > > > Key: FLINK-26060 > URL: https://issues.apache.org/jira/browse/FLINK-26060 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Runtime >Reporter: Francesco Guardiani >Assignee: Francesco Guardiani >Priority: Major > Labels: pull-request-available > > These nodes are using the old type system, which is going to be removed soon. > We should avoid supporting them in the persisted plan, as we cannot commit to > support them. Once migrated to the new type system, PyFlink won't need these > nodes anymore and will just rely on Table new function stack. For more > details, also check https://issues.apache.org/jira/browse/FLINK-25231 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18696: [hotfix][docs] project config pages
flinkbot edited a comment on pull request #18696: URL: https://github.com/apache/flink/pull/18696#issuecomment-1034481691 ## CI report: * 2921ed917d22b64cb99f05dfdd0f7e119dd86880 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31078) * ccb88999c20036553d971851cdc0c38670fb102a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31158) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18701: [FLINK-26071][table-api-java][table-planner] Now Planner#compilePlan fails if the plan cannot be serialized
flinkbot edited a comment on pull request #18701: URL: https://github.com/apache/flink/pull/18701#issuecomment-1034766277 ## CI report: * a3fdc9763094051769e040d5211db40196fe493c Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31116) * e001ada7d0947721903b47ce861fe0ac846ea2db Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31135) * c2e56d970d3d0e19d4534ca60f98d21b9d0764d1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31147) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment
flinkbot edited a comment on pull request #18702: URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544 ## CI report: * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-26055) Use new FlinkVersion enum for serde plans
[ https://issues.apache.org/jira/browse/FLINK-26055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francesco Guardiani updated FLINK-26055: Component/s: Table SQL / API > Use new FlinkVersion enum for serde plans > - > > Key: FLINK-26055 > URL: https://issues.apache.org/jira/browse/FLINK-26055 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API >Reporter: Timo Walther >Assignee: Francesco Guardiani >Priority: Major > Labels: pull-request-available > > The new FlinkVersion enum should be used to encode a Flink version from and > to JSON. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18706: [FLINK-26072][Connectors][NiFi] Mark NiFi Source and Sink as deprecated
flinkbot edited a comment on pull request #18706: URL: https://github.com/apache/flink/pull/18706#issuecomment-1034953890 ## CI report: * bf78620ba038d62e133c9ef66eb39e0e2313561f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31143) * f104642af11fced6f110afd39424871efa89ae9f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31157) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18707: [FLINK-25995][table-planner] Make implicit assumption of SQL local hash explicit
flinkbot edited a comment on pull request #18707: URL: https://github.com/apache/flink/pull/18707#issuecomment-1034994010 ## CI report: * 5e2128cc02aa0cfdb9b074edd5b0cab02c3daed6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31149) * 7e5e0f5f2aa79f47e79357721719cdcb971e32b3 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18708: Flink 25610 firehose connector backup
flinkbot edited a comment on pull request #18708: URL: https://github.com/apache/flink/pull/18708#issuecomment-1035039830 ## CI report: * 6e752a3332dd1b2f7b256f54d6aa662f8bacc4ba Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31159) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18709: [FLINK-26076] Fix ArchUnit violations in Source(Sink)MetricsITCase
flinkbot commented on pull request #18709: URL: https://github.com/apache/flink/pull/18709#issuecomment-1035050631 ## CI report: * 17f2ffbe7f9a8dd9315e7ee8c1cc3acad30728bc UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dmvk commented on a change in pull request #18689: [FLINK-21439][runtime] Exception history adaptive scheduler
dmvk commented on a change in pull request #18689: URL: https://github.com/apache/flink/pull/18689#discussion_r803794145 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/adaptive/StateWithExecutionGraph.java ## @@ -148,6 +162,33 @@ public Logger getLogger() { return logger; } +protected Throwable extractError(TaskExecutionStateTransition taskExecutionStateTransition) { +Throwable cause = taskExecutionStateTransition.getError(userCodeClassLoader); +if (cause == null) { +cause = new FlinkException("Unknown failure cause. Probably related to FLINK-21376."); +} +return cause; +} + +protected Optional extractExecutionVertexID( +TaskExecutionStateTransition taskExecutionStateTransition) { +return executionGraph.getExecutionVertexId(taskExecutionStateTransition.getID()); +} + +protected static Optional convertFailures( +Function> lookup, +List failureCollection) { +if (failureCollection.isEmpty()) { +return Optional.empty(); +} +Failure first = failureCollection.remove(0); +Set entries = new HashSet<>(); +for (Failure failure : failureCollection) { +entries.add(failure.toExceptionHistoryEntry(lookup)); Review comment: @zentol Do you know if `updateTaskExecutionState` could be retried by the TM? I guess we need to have something along these lines to make sure that the EG can transition to the terminal state. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18710: [FLINK-26055][table][annotations] Modified CompiledPlan#getFlinkVersion return type to FlinkVersion
flinkbot commented on pull request #18710: URL: https://github.com/apache/flink/pull/18710#issuecomment-1035051190 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit b7dc4ee5acd86161c61df1a11adb4df336d39900 (Thu Feb 10 15:25:41 UTC 2022) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] Myasuka commented on a change in pull request #18391: [FLINK-25478][chaneglog] Correct the state register logic of ChangelogStateBackendHandle
Myasuka commented on a change in pull request #18391: URL: https://github.com/apache/flink/pull/18391#discussion_r803795941 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/checkpoint/metadata/MetadataV2V3SerializerBase.java ## @@ -415,7 +440,7 @@ KeyedStateHandle deserializeKeyedStateHandle( int baseSize = dis.readInt(); List base = new ArrayList<>(baseSize); for (int i = 0; i < baseSize; i++) { -base.add(deserializeKeyedStateHandle(dis, context)); + base.add(Preconditions.checkNotNull(deserializeKeyedStateHandle(dis, context))); Review comment: I prefer to throw exception here to act as a safety guard. Otherwise, if we take the slient exception here, we should at least add warnings. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-25600) Support new statement set syntax in sql client and update docs
[ https://issues.apache.org/jira/browse/FLINK-25600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17490329#comment-17490329 ] Francesco Guardiani commented on FLINK-25600: - Hi [~wenlong.lwl] any updates on this one? > Support new statement set syntax in sql client and update docs > -- > > Key: FLINK-25600 > URL: https://issues.apache.org/jira/browse/FLINK-25600 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / API, Table SQL / Client >Reporter: Wenlong Lyu >Assignee: Wenlong Lyu >Priority: Major > > this is a follow up of FLINK-25392, to finish adding the new statement set: > 1. the new statement set need multi line parsing support in sql client, which > is not supported currently: > execute statement set begin > insert xxx; > insert xxx; > end; > 2. we need to update the doc to introduce the new syntax -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (FLINK-26077) Support operators send request to Coordinator and return a response
Jark Wu created FLINK-26077: --- Summary: Support operators send request to Coordinator and return a response Key: FLINK-26077 URL: https://issues.apache.org/jira/browse/FLINK-26077 Project: Flink Issue Type: New Feature Reporter: Jark Wu Currently, the communitcation between Operator and Coordiator is sigle-way. That means, after Operator sending a message to Coordiator, it can't wait to get the response message. In some senarios, the Operator may need to retrieve some information stored in the Coordinator. Thus, it would be great if we can have a two-way communication. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Updated] (FLINK-26077) Support operators send request to Coordinator and return a response
[ https://issues.apache.org/jira/browse/FLINK-26077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-26077: Component/s: Runtime / Coordination > Support operators send request to Coordinator and return a response > --- > > Key: FLINK-26077 > URL: https://issues.apache.org/jira/browse/FLINK-26077 > Project: Flink > Issue Type: New Feature > Components: Runtime / Coordination >Reporter: Jark Wu >Priority: Major > > Currently, the communitcation between Operator and Coordiator is sigle-way. > That means, after Operator sending a message to Coordiator, it can't wait to > get the response message. In some senarios, the Operator may need to retrieve > some information stored in the Coordinator. Thus, it would be great if we can > have a two-way communication. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18058: [FLINK-24571][connectors/elasticsearch] Supports a system time function(now() and current_timestamp) in index pattern
flinkbot edited a comment on pull request #18058: URL: https://github.com/apache/flink/pull/18058#issuecomment-988949831 ## CI report: * 93c33001cf55690369281de939bd79bb3727ad9a UNKNOWN * 107f4c437fb8f4af9a8dd143a2c3e68e72b66ceb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28435) * 5c5c6cc4aa76981978d7015e494fe74f7a8c283b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18667: [FLINK-25990][table] Expose uid generator for DataStream/Transformation providers
flinkbot edited a comment on pull request #18667: URL: https://github.com/apache/flink/pull/18667#issuecomment-1032823071 ## CI report: * 4af7f825a1bcbed0a6fda02c9445d9288a175e30 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=30988) * ca7a596b31ce7e89969dcf7bf58b9a7cdea38310 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31160) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18670: [FLINK-25999] Deprecate Per-Job Mode
flinkbot edited a comment on pull request #18670: URL: https://github.com/apache/flink/pull/18670#issuecomment-1033075650 ## CI report: * 8f553fdda7a749ef4d6398f3239ab2103e629ecf Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31121) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18701: [FLINK-26071][table-api-java][table-planner] Now Planner#compilePlan fails if the plan cannot be serialized
flinkbot edited a comment on pull request #18701: URL: https://github.com/apache/flink/pull/18701#issuecomment-1034766277 ## CI report: * a3fdc9763094051769e040d5211db40196fe493c Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31116) * e001ada7d0947721903b47ce861fe0ac846ea2db Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31135) * c2e56d970d3d0e19d4534ca60f98d21b9d0764d1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31147) * c2b74e16dc3528608d017847c2f9856cbabf227e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18702: [FLINK-24441][source] Block SourceOperator when watermarks are out of alignment
flinkbot edited a comment on pull request #18702: URL: https://github.com/apache/flink/pull/18702#issuecomment-1034842544 ## CI report: * 1c798c2c84e8285f7a7784038583a194bc42b3ea Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31118) * f730132ffa59826d88df89d4bd6ac1e5c93639d3 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18707: [FLINK-25995][table-planner] Make implicit assumption of SQL local hash explicit
flinkbot edited a comment on pull request #18707: URL: https://github.com/apache/flink/pull/18707#issuecomment-1034994010 ## CI report: * 5e2128cc02aa0cfdb9b074edd5b0cab02c3daed6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31149) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18706: [FLINK-26072][Connectors][NiFi] Mark NiFi Source and Sink as deprecated
flinkbot edited a comment on pull request #18706: URL: https://github.com/apache/flink/pull/18706#issuecomment-1034953890 ## CI report: * bf78620ba038d62e133c9ef66eb39e0e2313561f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31143) * f104642af11fced6f110afd39424871efa89ae9f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31157) * 12773467f6c3eebf4dcd2befcb0bb323c706b6c7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18709: [FLINK-26076] Fix ArchUnit violations in Source(Sink)MetricsITCase
flinkbot edited a comment on pull request #18709: URL: https://github.com/apache/flink/pull/18709#issuecomment-1035050631 ## CI report: * 17f2ffbe7f9a8dd9315e7ee8c1cc3acad30728bc Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31162) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #18710: [FLINK-26055][table][annotations] Modified CompiledPlan#getFlinkVersion return type to FlinkVersion
flinkbot commented on pull request #18710: URL: https://github.com/apache/flink/pull/18710#issuecomment-1035062536 ## CI report: * b7dc4ee5acd86161c61df1a11adb4df336d39900 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-26075) Persist per-ExecNode configuration
[ https://issues.apache.org/jira/browse/FLINK-26075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Marios Trivyzas reassigned FLINK-26075: --- Assignee: Marios Trivyzas > Persist per-ExecNode configuration > -- > > Key: FLINK-26075 > URL: https://issues.apache.org/jira/browse/FLINK-26075 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Timo Walther >Assignee: Marios Trivyzas >Priority: Major > > Even though a compiled plan is static, some configuration options still > change the topology of an ExecNode. In general, we will request users to keep > the configuration constant between Flink versions. However, setting > configuration more fine-grained per-ExecNode is a frequently requested > feature. It can also allow us to set the parallelism more fine-grained in the > future. > We need the following infrastructure for the mentioned use cases: > - Every ExecNode can have a configuration > - By default the configuration per node are the values from the global > configuration using the keys from the ExecNodeMetadata annotation. > - We persist the ExecNode configuration in the JSON plan. > - If the persisted plan contains a configuration, the persisted configuration > is merged with the global configuration and has priority. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[GitHub] [flink] flinkbot edited a comment on pull request #18058: [FLINK-24571][connectors/elasticsearch] Supports a system time function(now() and current_timestamp) in index pattern
flinkbot edited a comment on pull request #18058: URL: https://github.com/apache/flink/pull/18058#issuecomment-988949831 ## CI report: * 93c33001cf55690369281de939bd79bb3727ad9a UNKNOWN * 107f4c437fb8f4af9a8dd143a2c3e68e72b66ceb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=28435) * 5c5c6cc4aa76981978d7015e494fe74f7a8c283b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31163) * 94ea8311ad71362f05e29699c109d35b5a06f6d6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #18688: [FLINK-26060][table-planner] Remove persisted plan feature support for Python UDFs
flinkbot edited a comment on pull request #18688: URL: https://github.com/apache/flink/pull/18688#issuecomment-1033917965 ## CI report: * a1e93a73c296c113f974f93e44262d156f4ff102 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=31122) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run azure` re-run the last Azure build -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: issues-unsubscr...@flink.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org