[jira] [Created] (FLINK-20768) Support routing field for Elasticsearch connector
wangsan created FLINK-20768: --- Summary: Support routing field for Elasticsearch connector Key: FLINK-20768 URL: https://issues.apache.org/jira/browse/FLINK-20768 Project: Flink Issue Type: Improvement Components: Connectors / ElasticSearch Reporter: wangsan Routing in Elasticsearch can help with search efficency for large scale dataset, we should support this feature as an optional config in Elasticsearch connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] xintongsong commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
xintongsong commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548814271
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/execution/ThresholdExceedException.java
##
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.execution;
+
+/**
+ * Exception thrown when a threshold exceeds.
+ */
+public class ThresholdExceedException extends RuntimeException {
Review comment:
Why is this class in the path `runtime/execution`?
I think we should make it an inner class of `ThresholdMeter`, given that the
two classes are closely related.
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/execution/ThresholdExceedException.java
##
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.execution;
+
+/**
+ * Exception thrown when a threshold exceeds.
+ */
+public class ThresholdExceedException extends RuntimeException {
+
+ private static final long serialVersionUID = 241871312848349L;
Review comment:
`serialVersionUID` should start from `1L`.
https://flink.apache.org/contributing/code-style-and-quality-java.html#java-serialization
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final Supplier currentTimeMillisSupplier;
+ private final double maximumFailureRate;
+ private final Time failureInterval;
+ private final Queue failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(double maximumFailureRate, Time
failureInterval) {
+ this(maximumFailureRate, failureInterval,
System::currentTimeMillis);
+ }
+
+ @VisibleForTesting
+ public TimestampBasedFailureRater(double maximumFailureRate, Time
failureInterval, Supplier customSupplier) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failur
[jira] [Created] (FLINK-20769) Support minibatch to optimize Python UDAF
Huang Xingbo created FLINK-20769: Summary: Support minibatch to optimize Python UDAF Key: FLINK-20769 URL: https://issues.apache.org/jira/browse/FLINK-20769 Project: Flink Issue Type: Improvement Components: API / Python Reporter: Huang Xingbo Fix For: 1.13.0 Support minibatch to optimize Python UDAF -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] lirui-apache commented on pull request #14452: [FLINK-20710][Hive] HiveTableInputFormat and HiveSourceFileEnumerator will throw IllegalArgumentException when creating splits for table
lirui-apache commented on pull request #14452: URL: https://github.com/apache/flink/pull/14452#issuecomment-751213880 > @JingsongLi , do we have any plan to support write(INSERT INTO and INSERT OVERWRITE) hive partitions on different nameservices? It's very common that some old partitions of a Hive table on different nameservices and the latest partitions and table location in hivemetastore on an another nameservice. I don't think we support writing to hive partitions on different FS at the moment. But it should be possible to support such features. If you're interested, PR is always welcome ~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong closed pull request #14447: [FLINK-20664][k8s] Support setting service account for TaskManager pod.
xintongsong closed pull request #14447: URL: https://github.com/apache/flink/pull/14447 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JackWangCS commented on pull request #14452: [FLINK-20710][Hive] HiveTableInputFormat and HiveSourceFileEnumerator will throw IllegalArgumentException when creating splits for tables
JackWangCS commented on pull request #14452: URL: https://github.com/apache/flink/pull/14452#issuecomment-751214826 Thank you for your reply. I will create a ticket to implement the feature. And there are some design decisions that need to be discussed. I will ping you on the new JIRA ticket about the details. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JackWangCS edited a comment on pull request #14452: [FLINK-20710][Hive] HiveTableInputFormat and HiveSourceFileEnumerator will throw IllegalArgumentException when creating splits for
JackWangCS edited a comment on pull request #14452: URL: https://github.com/apache/flink/pull/14452#issuecomment-751214826 Hi @JingsongLi , thank you for your reply. I will create a ticket to implement the feature. And there are some design decisions that need to be discussed. I will ping you on the new JIRA ticket about the details. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20664) Support setting service account for TaskManager pod
[
https://issues.apache.org/jira/browse/FLINK-20664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xintong Song closed FLINK-20664.
Resolution: Fixed
Fixed via
* master (1.13): 4e5448cc332090d7bd2664f60278534d945e1211
* release-1.12: d3ec48b3cdc7aaf6cef8978bf7c0545be7ce43e6
> Support setting service account for TaskManager pod
> ---
>
> Key: FLINK-20664
> URL: https://issues.apache.org/jira/browse/FLINK-20664
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes
>Affects Versions: 1.12.0
>Reporter: Yang Wang
>Assignee: Boris Lublinsky
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.13.0, 1.12.1
>
>
> Currently, we only set the service account for JobManager. The TaskManager is
> using the default service account. Before the KubernetesHAService is
> introduced, it works because the TaskManager does not need to access the K8s
> resource(e.g. ConfigMap) directly. But now the TaskManager needs to watch
> ConfigMap and retrieve leader address. So if the default service account does
> not have enough permission, users could not specify a valid service account
> for TaskManager.
>
> We should introduce a new config option for TaskManager service account.
> {{kubernetes.taskmanager.service-account}}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14473: [FLINK-20702][python] Support Map Operation Chained Together in Python Table API
flinkbot edited a comment on pull request #14473: URL: https://github.com/apache/flink/pull/14473#issuecomment-750013641 ## CI report: * 8700f3fff87a3b634fda6a74ac7e1430a4c60ae0 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11270) * 99809001066eca34ab608150d270e0b7f720b0f9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20505) Yarn provided lib does not work with http paths.
[
https://issues.apache.org/jira/browse/FLINK-20505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254788#comment-17254788
]
zoucao commented on FLINK-20505:
[~xintongsong]，
Maybe close this ticket is a better choice, beacuse new comers can't get any
useful information about `yarn provided libs` from our discussing.:(
> Yarn provided lib does not work with http paths.
>
>
> Key: FLINK-20505
> URL: https://issues.apache.org/jira/browse/FLINK-20505
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.12.0, 1.11.2
>Reporter: Xintong Song
>Assignee: zoucao
>Priority: Major
> Labels: pull-request-available
>
> If an http path is used for provided lib, the following exception will be
> thrown on the resource manager side:
> {code:java}
> 2020-12-04 17:01:28.955 ERROR org.apache.flink.yarn.YarnResourceManager -
> Could not start TaskManager in container containerXX.
> org.apache.flink.util.FlinkException: Error to parse
> YarnLocalResourceDescriptor from YarnLocalResourceDescriptor{key=X.jar,
> path=https://XXX.jar, size=-1, modificationTime=0, visibility=APPLICATION}
> at
> org.apache.flink.yarn.YarnLocalResourceDescriptor.fromString(YarnLocalResourceDescriptor.java:99)
> at
> org.apache.flink.yarn.Utils.decodeYarnLocalResourceDescriptorListFromString(Utils.java:721)
> at org.apache.flink.yarn.Utils.createTaskExecutorContext(Utils.java:626)
> at
> org.apache.flink.yarn.YarnResourceManager.getOrCreateContainerLaunchContext(YarnResourceManager.java:746)
> at
> org.apache.flink.yarn.YarnResourceManager.createTaskExecutorLaunchContext(YarnResourceManager.java:726)
> at
> org.apache.flink.yarn.YarnResourceManager.startTaskExecutorInContainer(YarnResourceManager.java:500)
> at
> org.apache.flink.yarn.YarnResourceManager.onContainersOfResourceAllocated(YarnResourceManager.java:455)
> at
> org.apache.flink.yarn.YarnResourceManager.lambda$onContainersAllocated$1(YarnResourceManager.java:415)
> {code}
> The problem is that, `HttpFileSystem#getFilsStatus` returns file status with
> length `-1`, while `YarnLocalResourceDescriptor` does not recognize the
> negative file length.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] jrthe42 opened a new pull request #14493: [FLINK-20768][Connectors/ElasticSearch] Support routing field for Elasticsearch connector
jrthe42 opened a new pull request #14493: URL: https://github.com/apache/flink/pull/14493 ## What is the purpose of the change This pull request add support for routing field in ElasticSearch connector. ## Brief change log - Add `routing.filed-name` option to ES DynamicTableSink - If `routing.filed-name` is set, will generate routing field for each `IndexRequest`, `UpdateRequest` and `DeleteRequest`. ## Verifying this change This change added tests in `Elasticsearch7DynamicSinkITCase` and `Elasticsearch6DynamicSinkITCase`. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (docs) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14473: [FLINK-20702][python] Support Map Operation Chained Together in Python Table API
flinkbot edited a comment on pull request #14473: URL: https://github.com/apache/flink/pull/14473#issuecomment-750013641 ## CI report: * 8700f3fff87a3b634fda6a74ac7e1430a4c60ae0 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11270) * 99809001066eca34ab608150d270e0b7f720b0f9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11325) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20768) Support routing field for Elasticsearch connector
[ https://issues.apache.org/jira/browse/FLINK-20768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-20768: --- Labels: pull-request-available (was: ) > Support routing field for Elasticsearch connector > - > > Key: FLINK-20768 > URL: https://issues.apache.org/jira/browse/FLINK-20768 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Reporter: wangsan >Priority: Major > Labels: pull-request-available > > Routing in Elasticsearch can help with search efficency for large scale > dataset, we should support this feature as an optional config in > Elasticsearch connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14493: [FLINK-20768][Connectors/ElasticSearch] Support routing field for Elasticsearch connector
flinkbot commented on pull request #14493: URL: https://github.com/apache/flink/pull/14493#issuecomment-751218776 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 179ea826af4470c6c1d9a94da5f4270f16e81f40 (Fri Dec 25 09:25:36 UTC 2020) **Warnings:** * **2 pom.xml files were touched**: Check for build and licensing issues. * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-20768).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sv3ndk opened a new pull request #14494: uses lambda in Learn Flink example instead of FilterFunction
sv3ndk opened a new pull request #14494: URL: https://github.com/apache/flink/pull/14494 ## What is the purpose of the change *Illustrate java 8's lambda style usage in `.filter()` instead of `FilterFunction` in the first example of the "Learn Flink" section* Java 8 is now probably widespread enough so we can safely assume everyone can now rely on this syntax. (I hope this can be considered as trivial enough no to require a Jira ticket?) ## Brief change log * updated both English and Mandarin documentation, now using a lambda ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency):no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented?not applicable This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14494: uses lambda in Learn Flink example instead of FilterFunction
flinkbot commented on pull request #14494: URL: https://github.com/apache/flink/pull/14494#issuecomment-751221444 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 87391a27aaaf4a4ca2bc15d6043c831348279de4 (Fri Dec 25 09:40:26 UTC 2020) **Warnings:** * **Invalid pull request title: No valid Jira ID provided** Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] WeiZhong94 commented on a change in pull request #14492: [FLINK-20756][python] Add PythonCalcSplitConditionRexFieldRule
WeiZhong94 commented on a change in pull request #14492:
URL: https://github.com/apache/flink/pull/14492#discussion_r548843229
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/rules/logical/PythonCalcSplitRule.scala
##
@@ -168,12 +168,34 @@ abstract class
PythonCalcSplitProjectionRuleBase(description: String)
}
}
+abstract class PythonCalcSplitRexFieldRuleBase(description: String)
+ extends PythonCalcSplitRuleBase(description) {
+
+ override def needConvert(program: RexProgram, node: RexNode): Boolean = {
+node match {
+ case x: RexFieldAccess => x.getReferenceExpr match {
+case y: RexLocalRef if isPythonCall(program.expandLocalRef(y)) => true
+case _ => false
+ }
+ case _ => false
+}
+ }
+
+ protected def containsFieldAccessAfterPythonCall(node: RexNode): Boolean = {
+node match {
+ case call: RexCall =>
call.getOperands.exists(containsFieldAccessAfterPythonCall)
+ case x: RexFieldAccess => isPythonCall(x.getReferenceExpr)
Review comment:
We should call the `containsFieldAccessAfterPythonCall` here if the
expression is not a python call to handle the situation like `where
javaRowFunc(pyRowFunc5(a)).f0 is NULL`. And maybe we can combine this logic
into `FunctionFinder` as the field access, java call, python call may appear
alternately. That should be able to better to cover various situations.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] zhisheng17 opened a new pull request #14495: [hotfix][runtime]fix typo in HistoryServerUtils class
zhisheng17 opened a new pull request #14495: URL: https://github.com/apache/flink/pull/14495 ## What is the purpose of the change fix typo in HistoryServerUtils class This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #14489: [FLINK-20764] In BatchGroupedReduceOperator, fix timer logic
xintongsong commented on pull request #14489: URL: https://github.com/apache/flink/pull/14489#issuecomment-751222183 Thanks @aljoscha for opening the PR and @gaoyunhaii for the review. LGTM. Merging this now. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong closed pull request #14489: [FLINK-20764] In BatchGroupedReduceOperator, fix timer logic
xintongsong closed pull request #14489: URL: https://github.com/apache/flink/pull/14489 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14495: [hotfix][runtime]fix typo in HistoryServerUtils class
flinkbot commented on pull request #14495: URL: https://github.com/apache/flink/pull/14495#issuecomment-751222385 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 06d95b81212492278e8bc45f78a86e720b739f0d (Fri Dec 25 09:46:49 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20764) BatchGroupedReduceOperator does not emit results for singleton inputs
[ https://issues.apache.org/jira/browse/FLINK-20764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-20764: - Fix Version/s: 1.13.0 > BatchGroupedReduceOperator does not emit results for singleton inputs > - > > Key: FLINK-20764 > URL: https://issues.apache.org/jira/browse/FLINK-20764 > Project: Flink > Issue Type: Bug > Components: API / DataStream >Reporter: Aljoscha Krettek >Assignee: Aljoscha Krettek >Priority: Blocker > Labels: pull-request-available > Fix For: 1.13.0, 1.12.1 > > > This was reported on the ML: > https://lists.apache.org/thread.html/rf24ff56408d3f07a02a3726336229a5919e62c4179c9eb8361a5b76b%40%3Cuser.flink.apache.org%3E. > The problem is that the logic for setting up the "final" timer does not kick > in when there is only on input. > I'll cut a PR for that and also add tests for the operator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20764) BatchGroupedReduceOperator does not emit results for singleton inputs
[ https://issues.apache.org/jira/browse/FLINK-20764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song updated FLINK-20764: - Affects Version/s: 1.12.0 > BatchGroupedReduceOperator does not emit results for singleton inputs > - > > Key: FLINK-20764 > URL: https://issues.apache.org/jira/browse/FLINK-20764 > Project: Flink > Issue Type: Bug > Components: API / DataStream >Affects Versions: 1.12.0 >Reporter: Aljoscha Krettek >Assignee: Aljoscha Krettek >Priority: Blocker > Labels: pull-request-available > Fix For: 1.13.0, 1.12.1 > > > This was reported on the ML: > https://lists.apache.org/thread.html/rf24ff56408d3f07a02a3726336229a5919e62c4179c9eb8361a5b76b%40%3Cuser.flink.apache.org%3E. > The problem is that the logic for setting up the "final" timer does not kick > in when there is only on input. > I'll cut a PR for that and also add tests for the operator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14493: [FLINK-20768][Connectors/ElasticSearch] Support routing field for Elasticsearch connector
flinkbot commented on pull request #14493: URL: https://github.com/apache/flink/pull/14493#issuecomment-751222472 ## CI report: * 179ea826af4470c6c1d9a94da5f4270f16e81f40 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14494: uses lambda in Learn Flink example instead of FilterFunction
flinkbot commented on pull request #14494: URL: https://github.com/apache/flink/pull/14494#issuecomment-751222501 ## CI report: * 87391a27aaaf4a4ca2bc15d6043c831348279de4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20764) BatchGroupedReduceOperator does not emit results for singleton inputs
[ https://issues.apache.org/jira/browse/FLINK-20764?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song closed FLINK-20764. Resolution: Fixed Fixed via: * master (1.13): b474d2872f4a82d838ac23252254253df4ca2309 * release-1.12: bdbbb3907865342a7da5be899313422f23d386ee > BatchGroupedReduceOperator does not emit results for singleton inputs > - > > Key: FLINK-20764 > URL: https://issues.apache.org/jira/browse/FLINK-20764 > Project: Flink > Issue Type: Bug > Components: API / DataStream >Affects Versions: 1.12.0 >Reporter: Aljoscha Krettek >Assignee: Aljoscha Krettek >Priority: Blocker > Labels: pull-request-available > Fix For: 1.13.0, 1.12.1 > > > This was reported on the ML: > https://lists.apache.org/thread.html/rf24ff56408d3f07a02a3726336229a5919e62c4179c9eb8361a5b76b%40%3Cuser.flink.apache.org%3E. > The problem is that the logic for setting up the "final" timer does not kick > in when there is only on input. > I'll cut a PR for that and also add tests for the operator. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20770) Incorrect description for config option kubernetes.rest-service.exposed.type
Yang Wang created FLINK-20770:
-
Summary: Incorrect description for config option
kubernetes.rest-service.exposed.type
Key: FLINK-20770
URL: https://issues.apache.org/jira/browse/FLINK-20770
Project: Flink
Issue Type: Improvement
Components: Deployment / Kubernetes
Affects Versions: 1.11.3, 1.12.0
Reporter: Yang Wang
{code:java}
public static final ConfigOption REST_SERVICE_EXPOSED_TYPE =
key("kubernetes.rest-service.exposed.type")
.enumType(ServiceExposedType.class)
.defaultValue(ServiceExposedType.LoadBalancer)
.withDescription("The type of the rest service (ClusterIP or NodePort or
LoadBalancer). " +
"When set to ClusterIP, the rest service will not be created.");
{code}
The description of the config option is not correct. We will always create the
rest service after refactoring the Kubernetes decorators in FLINK-16194.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhisheng17 commented on pull request #14495: [hotfix][runtime]fix typo in HistoryServerUtils class
zhisheng17 commented on pull request #14495: URL: https://github.com/apache/flink/pull/14495#issuecomment-751224053 the code will error if remove the `;` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhisheng17 commented on pull request #14495: [hotfix][runtime]fix typo in HistoryServerUtils class
zhisheng17 commented on pull request #14495: URL: https://github.com/apache/flink/pull/14495#issuecomment-751224512 what about use class instead of enum? @tillrohrmann This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] shmyer commented on pull request #14084: [FLINK-15867][table-planner-blink] Support time-related types for FIRST_VALUE and LAST_VALUE aggregate functions
shmyer commented on pull request #14084: URL: https://github.com/apache/flink/pull/14084#issuecomment-751224702 Hi @leonardBang I would have time next week if that's not too late This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14472: [Flink-20736] Separate implementation of BatchExecLimit,StreamExecLimit,StreamExecRank
flinkbot edited a comment on pull request #14472: URL: https://github.com/apache/flink/pull/14472#issuecomment-750005686 ## CI report: * 9135cbb7f107ee96b15de4b0a3453d60ee5b9de6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11316) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11298) * b04180b32ac8950c9dabaa7f8142ed6bff033304 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14493: [FLINK-20768][Connectors/ElasticSearch] Support routing field for Elasticsearch connector
flinkbot edited a comment on pull request #14493: URL: https://github.com/apache/flink/pull/14493#issuecomment-751222472 ## CI report: * 179ea826af4470c6c1d9a94da5f4270f16e81f40 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11326) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14494: uses lambda in Learn Flink example instead of FilterFunction
flinkbot edited a comment on pull request #14494: URL: https://github.com/apache/flink/pull/14494#issuecomment-751222501 ## CI report: * 87391a27aaaf4a4ca2bc15d6043c831348279de4 UNKNOWN * 8c529ec8d1edb62ba45784709f3d313b7ef37c4c UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14495: [hotfix][runtime]fix typo in HistoryServerUtils class
flinkbot commented on pull request #14495: URL: https://github.com/apache/flink/pull/14495#issuecomment-751225470 ## CI report: * 06d95b81212492278e8bc45f78a86e720b739f0d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20743) Print ContainerId For RemoteTransportException
[ https://issues.apache.org/jira/browse/FLINK-20743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254795#comment-17254795 ] Zhijiang commented on FLINK-20743: -- Thanks for creating this issue [~清月]. I remembered I also encountered the same concern while debugging some failover issues before. I guess we might rely on the port info to trace the other required infos such as container ID from job manager log. But indeed it is not very convinenet and efficient. So in general I supoort your proposal for improving the debugging process, but I am a bit worried that it might not be easy to pass container ID into the network stack which might touch many components. Anyway, you can try out your way and I can assign this ticket to you if you desire to contribute it. :) > Print ContainerId For RemoteTransportException > -- > > Key: FLINK-20743 > URL: https://issues.apache.org/jira/browse/FLINK-20743 > Project: Flink > Issue Type: Improvement > Components: Runtime / Network >Affects Versions: 1.10.0, 1.11.1, 1.12.1 >Reporter: yang gang >Priority: Major > Attachments: image-2020-12-23-15-13-21-226.png > > > !image-2020-12-23-15-13-21-226.png|width=970,height=291! > RemoteTransportException, this exception reminds the user which service has > a problem by means of Ip/Port. > When we troubleshoot the problem, the information is not accurate enough. > Usually at this time we need to look at the running log of the container that > has the problem, but when we see this log, it also shows that the container > has died, so pass Ip/ The port method can no longer quickly locate a specific > container. > So I hope that when such an exception occurs, I hope to print the > containerId。 > E.g: > Connection unexpectedly closed by remote task manager > 'hostName/ip:port/containerId'. This might indicate that the remote task > manager was lost. > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-20771) Hive partition is not added when there is a lot of data
hehuiyuan created FLINK-20771: - Summary: Hive partition is not added when there is a lot of data Key: FLINK-20771 URL: https://issues.apache.org/jira/browse/FLINK-20771 Project: Flink Issue Type: Bug Components: Connectors / Hive Reporter: hehuiyuan Attachments: image-2020-12-25-18-09-42-707.png, image-2020-12-25-18-15-07-519.png Hive partition is not added when the data is huge . !image-2020-12-25-18-09-42-707.png|width=437,height=115! Before partition commit, *inProgressPart* will be reinitialize . But bucket is active , the partition is !image-2020-12-25-18-15-07-519.png|width=574,height=192! bucket is active , so the notifyBucketInactive is not executed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20771) Hive partition is not added when there is a lot of data
[ https://issues.apache.org/jira/browse/FLINK-20771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hehuiyuan updated FLINK-20771: -- Attachment: image-2020-12-25-18-19-53-746.png > Hive partition is not added when there is a lot of data > --- > > Key: FLINK-20771 > URL: https://issues.apache.org/jira/browse/FLINK-20771 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Reporter: hehuiyuan >Priority: Major > Attachments: image-2020-12-25-18-09-42-707.png, > image-2020-12-25-18-15-07-519.png, image-2020-12-25-18-19-53-746.png > > > Hive partition is not added when the data is huge . > !image-2020-12-25-18-09-42-707.png|width=437,height=115! > Before partition commit, *inProgressPart* will be reinitialize . > But bucket is active , the partition is > !image-2020-12-25-18-15-07-519.png|width=574,height=192! > bucket is active , so the notifyBucketInactive is not executed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20771) Hive partition is not added when there is a lot of data
[ https://issues.apache.org/jira/browse/FLINK-20771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hehuiyuan updated FLINK-20771: -- Description: Hive partition is not added when the data is huge . !image-2020-12-25-18-09-42-707.png|width=437,height=115! Before partition commit, *inProgressPart* will be reinitialize . But bucket is active , the partition is !image-2020-12-25-18-15-07-519.png|width=574,height=192! bucket is active , so the notifyBucketInactive is not executed. !image-2020-12-25-18-19-53-746.png|width=697,height=56! was: Hive partition is not added when the data is huge . !image-2020-12-25-18-09-42-707.png|width=437,height=115! Before partition commit, *inProgressPart* will be reinitialize . But bucket is active , the partition is !image-2020-12-25-18-15-07-519.png|width=574,height=192! bucket is active , so the notifyBucketInactive is not executed. > Hive partition is not added when there is a lot of data > --- > > Key: FLINK-20771 > URL: https://issues.apache.org/jira/browse/FLINK-20771 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Reporter: hehuiyuan >Priority: Major > Attachments: image-2020-12-25-18-09-42-707.png, > image-2020-12-25-18-15-07-519.png, image-2020-12-25-18-19-53-746.png > > > Hive partition is not added when the data is huge . > !image-2020-12-25-18-09-42-707.png|width=437,height=115! > Before partition commit, *inProgressPart* will be reinitialize . > But bucket is active , the partition is > !image-2020-12-25-18-15-07-519.png|width=574,height=192! > bucket is active , so the notifyBucketInactive is not executed. > !image-2020-12-25-18-19-53-746.png|width=697,height=56! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20771) Hive partition is not added when there is a lot of data
[ https://issues.apache.org/jira/browse/FLINK-20771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hehuiyuan updated FLINK-20771: -- Description: Hive partition is not added when the data is huge . !image-2020-12-25-18-09-42-707.png|width=437,height=115! Before partition commit, *inProgressPart* will be reinitialize . the current bucket is active , so the notifyBucketInactive is not executed. !image-2020-12-25-18-15-07-519.png|width=574,height=192! !image-2020-12-25-18-19-53-746.png|width=697,height=56! was: Hive partition is not added when the data is huge . !image-2020-12-25-18-09-42-707.png|width=437,height=115! Before partition commit, *inProgressPart* will be reinitialize . But bucket is active , the partition is !image-2020-12-25-18-15-07-519.png|width=574,height=192! bucket is active , so the notifyBucketInactive is not executed. !image-2020-12-25-18-19-53-746.png|width=697,height=56! > Hive partition is not added when there is a lot of data > --- > > Key: FLINK-20771 > URL: https://issues.apache.org/jira/browse/FLINK-20771 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Reporter: hehuiyuan >Priority: Major > Attachments: image-2020-12-25-18-09-42-707.png, > image-2020-12-25-18-15-07-519.png, image-2020-12-25-18-19-53-746.png > > > Hive partition is not added when the data is huge . > !image-2020-12-25-18-09-42-707.png|width=437,height=115! > Before partition commit, *inProgressPart* will be reinitialize . > the current bucket is active , so the notifyBucketInactive is not executed. > !image-2020-12-25-18-15-07-519.png|width=574,height=192! > > !image-2020-12-25-18-19-53-746.png|width=697,height=56! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] xiaoHoly commented on pull request #14495: [hotfix][runtime]fix typo in HistoryServerUtils class
xiaoHoly commented on pull request #14495: URL: https://github.com/apache/flink/pull/14495#issuecomment-751227923 hi,@zhisheng17 .I have learned some information, and I think this idea makes sense. Share with you now。https://blog.csdn.net/weixin_39525007/article/details/110726349 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14472: [Flink-20736] Separate implementation of BatchExecLimit,StreamExecLimit,StreamExecRank
flinkbot edited a comment on pull request #14472: URL: https://github.com/apache/flink/pull/14472#issuecomment-750005686 ## CI report: * 9135cbb7f107ee96b15de4b0a3453d60ee5b9de6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11316) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11298) * b04180b32ac8950c9dabaa7f8142ed6bff033304 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11330) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14494: uses lambda in Learn Flink example instead of FilterFunction
flinkbot edited a comment on pull request #14494: URL: https://github.com/apache/flink/pull/14494#issuecomment-751222501 ## CI report: * 87391a27aaaf4a4ca2bc15d6043c831348279de4 UNKNOWN * 8c529ec8d1edb62ba45784709f3d313b7ef37c4c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11327) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14495: [hotfix][runtime]fix typo in HistoryServerUtils class
flinkbot edited a comment on pull request #14495: URL: https://github.com/apache/flink/pull/14495#issuecomment-751225470 ## CI report: * 06d95b81212492278e8bc45f78a86e720b739f0d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11331) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20416) Need a cached catalog for batch SQL job
[
https://issues.apache.org/jira/browse/FLINK-20416?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254799#comment-17254799
]
Rui Li commented on FLINK-20416:
[~shared_ptr] Thanks for the design doc. Instead of adding a wrapper of a
delegate catalog, perhaps another option is to encapsulate some cache utility
for a catalog to use, so that each catalog implementation has more flexibility
about the caching behavior. Maybe we should start a discussion in the dev
mailing list so that more people can be involved.
> Need a cached catalog for batch SQL job
> ---
>
> Key: FLINK-20416
> URL: https://issues.apache.org/jira/browse/FLINK-20416
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common, Connectors / Hive, Table SQL / API,
> Table SQL / Planner
>Reporter: Sebastian Liu
>Priority: Major
> Labels: pull-request-available
>
> For OLAP scenarios, There are usually some analytical queries which running
> time is relatively short. These queries are also sensitive to latency. In the
> current Blink sql processing, parse/validate/optimize stages are all need
> meta data from catalog API. But each request to the catalog requires re-run
> of the underlying meta query.
>
> We may need a cached catalog which can cache the table schema and statistic
> info to avoid unnecessary repeated meta requests.
> I have submitted a related PR for adding a genetic cached catalog, which can
> delegate other implementations of {{AbstractCatalog. }}
> {{[https://github.com/apache/flink/pull/14260]}}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20772) [DISCUSS] RocksDBValueState with TTL occurs NullPointerException when calling update(null) method
Seongbae Chang created FLINK-20772:
--
Summary: [DISCUSS] RocksDBValueState with TTL occurs
NullPointerException when calling update(null) method
Key: FLINK-20772
URL: https://issues.apache.org/jira/browse/FLINK-20772
Project: Flink
Issue Type: Bug
Components: Runtime / State Backends
Affects Versions: 1.11.2
Environment: Flink version: 1.11.2
Flink Cluster: Standalone cluster with 3 Job managers and Task managers on
CentOS 7
Reporter: Seongbae Chang
h2. Problem
* I use ValueState for my custom trigger and set TTL for these ValueState in
RocksDB backend environment.
* I found an error when I used this code. I know that ValueState.update(null)
works equally to ValueState.clear() in general. Unfortunately, this error
occurs after using TTL
{code:java}
// My Code
ctx.getPartitionedState(batchTotalSizeStateDesc).update(null);
{code}
* I tested this in Flink 1.11.2, but I think it would be a problem in upper
versions.
* Plus, I'm a beginner. So, if there is any problem in this discussion issue,
please give me advice about that. And I'll fix it!
{code:java}
// Error Stacktrace
Caused by: TimerException{org.apache.flink.util.FlinkRuntimeException: Error
while adding data to RocksDB}
... 12 more
Caused by: org.apache.flink.util.FlinkRuntimeException: Error while adding data
to RocksDB
at
org.apache.flink.contrib.streaming.state.RocksDBValueState.update(RocksDBValueState.java:108)
at
org.apache.flink.runtime.state.ttl.TtlValueState.update(TtlValueState.java:50)
at .onProcessingTime(ActionBatchTimeTrigger.java:102)
at .onProcessingTime(ActionBatchTimeTrigger.java:29)
at
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator$Context.onProcessingTime(WindowOperator.java:902)
at
org.apache.flink.streaming.runtime.operators.windowing.WindowOperator.onProcessingTime(WindowOperator.java:498)
at
org.apache.flink.streaming.api.operators.InternalTimerServiceImpl.onProcessingTime(InternalTimerServiceImpl.java:260)
at
org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1220)
... 11 more
Caused by: java.lang.NullPointerException
at
org.apache.flink.api.common.typeutils.base.IntSerializer.serialize(IntSerializer.java:69)
at
org.apache.flink.api.common.typeutils.base.IntSerializer.serialize(IntSerializer.java:32)
at
org.apache.flink.api.common.typeutils.CompositeSerializer.serialize(CompositeSerializer.java:142)
at
org.apache.flink.contrib.streaming.state.AbstractRocksDBState.serializeValueInternal(AbstractRocksDBState.java:158)
at
org.apache.flink.contrib.streaming.state.AbstractRocksDBState.serializeValue(AbstractRocksDBState.java:178)
at
org.apache.flink.contrib.streaming.state.AbstractRocksDBState.serializeValue(AbstractRocksDBState.java:167)
at
org.apache.flink.contrib.streaming.state.RocksDBValueState.update(RocksDBValueState.java:106)
... 18 more
{code}
h2. Reason
* It relates to RocksDBValueState with TTLValueState
* In RocksDBValueState(as well as other types of ValueState), *.update(null)*
has to be caught in if-clauses(null checking). However, it skips the null
checking and then tries to serialize the null value.
{code:java}
//
https://github.com/apache/flink/blob/release-1.11/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBValueState.java#L96-L110
@Override
public void update(V value) {
if (value == null) {
clear();
return;
}
try {
backend.db.put(columnFamily, writeOptions,
serializeCurrentKeyWithGroupAndNamespace(), serializeValue(value));
} catch (Exception e) {
throw new FlinkRuntimeException("Error while adding data to RocksDB",
e);
}
}{code}
* It is because that TtlValueState wraps the value(null) with the
LastAccessTime and makes the new TtlValue Object with the null value.
{code:java}
//
https://github.com/apache/flink/blob/release-1.11/flink-runtime/src/main/java/org/apache/flink/runtime/state/ttl/TtlValueState.java#L47-L51
@Override
public void update(T value) throws IOException {
accessCallback.run();
original.update(wrapWithTs(value));
}
{code}
{code:java}
//
https://github.com/apache/flink/blob/release-1.11/flink-runtime/src/main/java/org/apache/flink/runtime/state/ttl/TtlUtils.java#L46-L48
static TtlValue wrapWithTs(V value, long ts) {
return new TtlValue<>(value, ts);
}{code}
{code:java}
//
https://github.com/apache/flink/blob/release-1.11/flink-runtime/src/main/java/org/apache/flink/runtime/state/ttl/TtlValue.java
public class TtlValue implements Serializable {
private static final long serialVersionUID = 5221129704201125020L;
@Nullable
private final T use
[GitHub] [flink] flinkbot edited a comment on pull request #14492: [FLINK-20756][python] Add PythonCalcSplitConditionRexFieldRule
flinkbot edited a comment on pull request #14492: URL: https://github.com/apache/flink/pull/14492#issuecomment-751194123 ## CI report: * 8d7ba4cf9f64e73076055013e3ccb18b7bd9ec3c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11322) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14494: uses lambda in Learn Flink example instead of FilterFunction
flinkbot edited a comment on pull request #14494: URL: https://github.com/apache/flink/pull/14494#issuecomment-751222501 ## CI report: * 87391a27aaaf4a4ca2bc15d6043c831348279de4 UNKNOWN * 8c529ec8d1edb62ba45784709f3d313b7ef37c4c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11327) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20768) Support routing field for Elasticsearch connector
[ https://issues.apache.org/jira/browse/FLINK-20768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-20768: Component/s: Table SQL / Ecosystem > Support routing field for Elasticsearch connector > - > > Key: FLINK-20768 > URL: https://issues.apache.org/jira/browse/FLINK-20768 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch, Table SQL / Ecosystem >Reporter: wangsan >Priority: Major > Labels: pull-request-available > > Routing in Elasticsearch can help with search efficency for large scale > dataset, we should support this feature as an optional config in > Elasticsearch connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20768) Support routing field for Elasticsearch connector
[ https://issues.apache.org/jira/browse/FLINK-20768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254807#comment-17254807 ] Jark Wu commented on FLINK-20768: - Hi [~wangsan], before submitting a pull request, we should discuss and reach a consensus on the proposal first. Could you explain a bit more what is routing in ES and what's the Public API you want to introduce? > Support routing field for Elasticsearch connector > - > > Key: FLINK-20768 > URL: https://issues.apache.org/jira/browse/FLINK-20768 > Project: Flink > Issue Type: Improvement > Components: Connectors / ElasticSearch >Reporter: wangsan >Priority: Major > Labels: pull-request-available > > Routing in Elasticsearch can help with search efficency for large scale > dataset, we should support this feature as an optional config in > Elasticsearch connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] HuangXingBo opened a new pull request #14496: [FLINK-20620][python] Port BatchExecPythonCalc and StreamExecPythonCalc to Java
HuangXingBo opened a new pull request #14496: URL: https://github.com/apache/flink/pull/14496 ## What is the purpose of the change This pull request will Port BatchExecPythonCalc and StreamExecPythonCalc to Java ## Brief change log - *Introduce `CommonPythonUtil`* ## Verifying this change This change added tests and can be verified as follows: - *Original Tests* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20620) Port BatchExecPythonCalc and StreamExecPythonCalc to Java
[
https://issues.apache.org/jira/browse/FLINK-20620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-20620:
---
Labels: pull-request-available (was: )
> Port BatchExecPythonCalc and StreamExecPythonCalc to Java
> -
>
> Key: FLINK-20620
> URL: https://issues.apache.org/jira/browse/FLINK-20620
> Project: Flink
> Issue Type: Sub-task
> Components: API / Python, Table SQL / Planner
>Reporter: godfrey he
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> https://issues.apache.org/jira/browse/FLINK-20610 will separate the
> implementation of BatchExecCalc and StreamExecCalc, and port BatchExecCalc
> and StreamExecCalc to Java.
> {{StreamExecPythonCalc}} extends from {{CommonExecPythonCalc}} and
> {{CommonExecPythonCalc}} extends from {{CommonPythonBase}}, they are all
> Scala classes, and involves a lot of code. Java class can't extend Scala
> interface with default implementation. So I create an issue separately to
> port them to Java.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] HuangXingBo commented on pull request #14496: [FLINK-20620][python] Port BatchExecPythonCalc and StreamExecPythonCalc to Java
HuangXingBo commented on pull request #14496: URL: https://github.com/apache/flink/pull/14496#issuecomment-751240558 @godfreyhe @dianfu Could you help review it? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14496: [FLINK-20620][python] Port BatchExecPythonCalc and StreamExecPythonCalc to Java
flinkbot commented on pull request #14496: URL: https://github.com/apache/flink/pull/14496#issuecomment-751240803 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit b320965a86dfb89ae406cf19c77dd2e5f294d95f (Fri Dec 25 12:11:47 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-20620).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13800: [FLINK-19650][connectors jdbc]Support the limit push down for the Jdb…
flinkbot edited a comment on pull request #13800: URL: https://github.com/apache/flink/pull/13800#issuecomment-717132623 ## CI report: * f5e7891cc45eba2d0d8f28dabdf596834faf0f0a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11318) * bfbae787e414d2cd1c9dec08f5bbfb3a9b91e6ce UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14496: [FLINK-20620][python] Port BatchExecPythonCalc and StreamExecPythonCalc to Java
flinkbot commented on pull request #14496: URL: https://github.com/apache/flink/pull/14496#issuecomment-751242619 ## CI report: * b320965a86dfb89ae406cf19c77dd2e5f294d95f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-20773) Support allow-unescaped-control-chars option for JSON format
xiaozilong created FLINK-20773: -- Summary: Support allow-unescaped-control-chars option for JSON format Key: FLINK-20773 URL: https://issues.apache.org/jira/browse/FLINK-20773 Project: Flink Issue Type: Improvement Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) Affects Versions: 1.12.0 Reporter: xiaozilong Attachments: image-2020-12-25-20-21-50-637.png Can we add a option `allow-unquoted-ctrl-char` for json format because of it will throw exception when exist illegal unquoted characters in data. !image-2020-12-25-20-21-50-637.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20773) Support allow-unescaped-control-chars option for JSON format
[ https://issues.apache.org/jira/browse/FLINK-20773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaozilong updated FLINK-20773: --- Description: Can we add an option `allow-unquoted-ctrl-char` for json format because of it will throw exception when exist illegal unquoted characters in data. !image-2020-12-25-20-21-50-637.png! was: Can we add a option `allow-unquoted-ctrl-char` for json format because of it will throw exception when exist illegal unquoted characters in data. !image-2020-12-25-20-21-50-637.png! > Support allow-unescaped-control-chars option for JSON format > > > Key: FLINK-20773 > URL: https://issues.apache.org/jira/browse/FLINK-20773 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.12.0 >Reporter: xiaozilong >Priority: Major > Attachments: image-2020-12-25-20-21-50-637.png > > > Can we add an option `allow-unquoted-ctrl-char` for json format because of it > will throw exception when exist illegal unquoted characters in data. > !image-2020-12-25-20-21-50-637.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20773) Support allow-unescaped-control-chars option for JSON format
[ https://issues.apache.org/jira/browse/FLINK-20773?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254818#comment-17254818 ] xiaozilong commented on FLINK-20773: cc [~jark] > Support allow-unescaped-control-chars option for JSON format > > > Key: FLINK-20773 > URL: https://issues.apache.org/jira/browse/FLINK-20773 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.12.0 >Reporter: xiaozilong >Priority: Major > Attachments: image-2020-12-25-20-21-50-637.png > > > Can we add a option `allow-unquoted-ctrl-char` for json format because of it > will throw exception when exist illegal unquoted characters in data. > !image-2020-12-25-20-21-50-637.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20773) Support allow-unescaped-control-chars option for JSON format
[ https://issues.apache.org/jira/browse/FLINK-20773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaozilong updated FLINK-20773: --- Description: Can we add an option `allow-unquoted-ctrl-char` for json format because of it will throw exception when exist illegal unquoted characters in the data. !image-2020-12-25-20-21-50-637.png! was: Can we add an option `allow-unquoted-ctrl-char` for json format because of it will throw exception when exist illegal unquoted characters in data. !image-2020-12-25-20-21-50-637.png! > Support allow-unescaped-control-chars option for JSON format > > > Key: FLINK-20773 > URL: https://issues.apache.org/jira/browse/FLINK-20773 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.12.0 >Reporter: xiaozilong >Priority: Major > Attachments: image-2020-12-25-20-21-50-637.png > > > Can we add an option `allow-unquoted-ctrl-char` for json format because of it > will throw exception when exist illegal unquoted characters in the data. > !image-2020-12-25-20-21-50-637.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] dianfu closed pull request #14473: [FLINK-20702][python] Support Map Operation Chained Together in Python Table API
dianfu closed pull request #14473: URL: https://github.com/apache/flink/pull/14473 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20702) Support Map Operation Chained Together in Python Table API
[
https://issues.apache.org/jira/browse/FLINK-20702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu closed FLINK-20702.
---
Resolution: Fixed
Merged to master via febce3597bc927a839bcd7dae679c531401b6ad7
> Support Map Operation Chained Together in Python Table API
> --
>
> Key: FLINK-20702
> URL: https://issues.apache.org/jira/browse/FLINK-20702
> Project: Flink
> Issue Type: Sub-task
> Components: API / Python
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> {code:java}
> tab.map(func1).map(func2){code}
> func1 and func2 can be merged into one python function executed in the same
> Python Operation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20702) Support map operation chained together in Python Table API
[
https://issues.apache.org/jira/browse/FLINK-20702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-20702:
Summary: Support map operation chained together in Python Table API (was:
Support Map Operation Chained Together in Python Table API)
> Support map operation chained together in Python Table API
> --
>
> Key: FLINK-20702
> URL: https://issues.apache.org/jira/browse/FLINK-20702
> Project: Flink
> Issue Type: Sub-task
> Components: API / Python
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> {code:java}
> tab.map(func1).map(func2){code}
> func1 and func2 can be merged into one python function executed in the same
> Python Operation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20507) Support aggregate operation in Python Table API
[
https://issues.apache.org/jira/browse/FLINK-20507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-20507:
Summary: Support aggregate operation in Python Table API (was: Support
Aggregate Operation in Python Table API)
> Support aggregate operation in Python Table API
> ---
>
> Key: FLINK-20507
> URL: https://issues.apache.org/jira/browse/FLINK-20507
> Project: Flink
> Issue Type: Sub-task
> Components: API / Python
>Reporter: Huang Xingbo
>Assignee: Huang Xingbo
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> Support Python UDAF for Aggregate Operation in Python Table API
> The usage:
> {code:java}
> t = ... # type: Table, table schema: [a: String, b: Int, c: Int]
> # aggregate General Python UDAF
> t_env.create_temporary_function("agg", GeneralPythonAggregateFunction())
> t.group_by(t.c).select("agg(a)")
> # aggregate Pandas UDAF
> mean_max_udaf = udaf(lambda a: Row(a.mean(), a.max()),
> result_type=DataTypes.ROW(
> [DataTypes.FIELD("a", DataTypes.FLOAT()),
> DataTypes.FIELD("b", DataTypes.INT()),
> func_type="pandas")
> t.group_by(t.a).aggregate(mean_max_udaf(t.b).alias("d", "f")).select("a, d,
> f"){code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-20738) Separate the implementation of batch group aggregate nodes
[ https://issues.apache.org/jira/browse/FLINK-20738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] godfrey he reassigned FLINK-20738: -- Assignee: godfrey he > Separate the implementation of batch group aggregate nodes > -- > > Key: FLINK-20738 > URL: https://issues.apache.org/jira/browse/FLINK-20738 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: godfrey he >Assignee: godfrey he >Priority: Major > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-20766) Separate the implementation of stream sort nodes
[ https://issues.apache.org/jira/browse/FLINK-20766?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] godfrey he reassigned FLINK-20766: -- Assignee: Wenlong Lyu > Separate the implementation of stream sort nodes > > > Key: FLINK-20766 > URL: https://issues.apache.org/jira/browse/FLINK-20766 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Wenlong Lyu >Assignee: Wenlong Lyu >Priority: Major > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13800: [FLINK-19650][connectors jdbc]Support the limit push down for the Jdb…
flinkbot edited a comment on pull request #13800: URL: https://github.com/apache/flink/pull/13800#issuecomment-717132623 ## CI report: * f5e7891cc45eba2d0d8f28dabdf596834faf0f0a Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11318) * bfbae787e414d2cd1c9dec08f5bbfb3a9b91e6ce Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11332) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14496: [FLINK-20620][python] Port BatchExecPythonCalc and StreamExecPythonCalc to Java
flinkbot edited a comment on pull request #14496: URL: https://github.com/apache/flink/pull/14496#issuecomment-751242619 ## CI report: * b320965a86dfb89ae406cf19c77dd2e5f294d95f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11333) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] godfreyhe closed pull request #14455: [FLINK-20706][table-planner-blink] Separate the implementation of BatchExecUnion and StreamExecUnion
godfreyhe closed pull request #14455: URL: https://github.com/apache/flink/pull/14455 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20706) Separate the implementation of BatchExecUnion and StreamExecUnion
[ https://issues.apache.org/jira/browse/FLINK-20706?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] godfrey he closed FLINK-20706. -- Resolution: Fixed Fixed in 1.13.0: 4f323bac..f3d03eb4 > Separate the implementation of BatchExecUnion and StreamExecUnion > - > > Key: FLINK-20706 > URL: https://issues.apache.org/jira/browse/FLINK-20706 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: godfrey he >Assignee: godfrey he >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] kezhuw commented on a change in pull request #14466: [FLINK-20657][connectors/jdbc] Migrate jdbc InputFormat/LookupFunction to SimpleJdbcConnectionProvider for connection establishmen
kezhuw commented on a change in pull request #14466:
URL: https://github.com/apache/flink/pull/14466#discussion_r548872516
##
File path:
flink-connectors/flink-connector-jdbc/src/main/java/org/apache/flink/connector/jdbc/internal/connection/JdbcConnectionProvider.java
##
@@ -20,13 +20,40 @@
import org.apache.flink.annotation.Internal;
import java.sql.Connection;
+import java.sql.SQLException;
/**
* JDBC connection provider.
*/
@Internal
public interface JdbcConnectionProvider {
- Connection getConnection() throws Exception;
+ /**
+* Get existing connection.
+*
+* @return existing connection
+*/
+ Connection getExistingConnection();
Review comment:
`@Nullable` added.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] kezhuw commented on pull request #14466: [FLINK-20657][connectors/jdbc] Migrate jdbc InputFormat/LookupFunction to SimpleJdbcConnectionProvider for connection establishment
kezhuw commented on pull request #14466: URL: https://github.com/apache/flink/pull/14466#issuecomment-751249646 @wuchong I have squashed and rebased all commits on latest master. Please take another look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] godfreyhe commented on pull request #14472: [Flink-20736] Separate implementation of BatchExecLimit,StreamExecLimit,StreamExecRank
godfreyhe commented on pull request #14472: URL: https://github.com/apache/flink/pull/14472#issuecomment-751252395 Thanks for the update, LGTM except some minor adjustments, I will do this in my local and then merge the pr once azure is green This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14466: [FLINK-20657][connectors/jdbc] Migrate jdbc InputFormat/LookupFunction to SimpleJdbcConnectionProvider for connection establishment
flinkbot edited a comment on pull request #14466: URL: https://github.com/apache/flink/pull/14466#issuecomment-749592580 ## CI report: * 265349b7e273ffcc388f37fd31248eb37e271550 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11301) * cfb29832f3281af5b385f0ef87f17875224f474e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14466: [FLINK-20657][connectors/jdbc] Migrate jdbc InputFormat/LookupFunction to SimpleJdbcConnectionProvider for connection establishment
flinkbot edited a comment on pull request #14466: URL: https://github.com/apache/flink/pull/14466#issuecomment-749592580 ## CI report: * 265349b7e273ffcc388f37fd31248eb37e271550 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11301) * cfb29832f3281af5b385f0ef87f17875224f474e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11336) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14493: [FLINK-20768][Connectors/ElasticSearch] Support routing field for Elasticsearch connector
flinkbot edited a comment on pull request #14493: URL: https://github.com/apache/flink/pull/14493#issuecomment-751222472 ## CI report: * 179ea826af4470c6c1d9a94da5f4270f16e81f40 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11326) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14472: [Flink-20736] Separate implementation of BatchExecLimit,StreamExecLimit,StreamExecRank
flinkbot edited a comment on pull request #14472: URL: https://github.com/apache/flink/pull/14472#issuecomment-750005686 ## CI report: * b04180b32ac8950c9dabaa7f8142ed6bff033304 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11330) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] blublinsky commented on pull request #14281: FLINK-20359 Added Owner Reference to Job Manager in native kubernetes setup
blublinsky commented on pull request #14281: URL: https://github.com/apache/flink/pull/14281#issuecomment-751265666 We could, which creates a lot of code, which not really add clarity or simplicity. Right now it is just 9 lines of code. What is the point of creation additional class, that is doing exactly the same? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13800: [FLINK-19650][connectors jdbc]Support the limit push down for the Jdb…
flinkbot edited a comment on pull request #13800: URL: https://github.com/apache/flink/pull/13800#issuecomment-717132623 ## CI report: * bfbae787e414d2cd1c9dec08f5bbfb3a9b91e6ce Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11332) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14496: [FLINK-20620][python] Port BatchExecPythonCalc and StreamExecPythonCalc to Java
flinkbot edited a comment on pull request #14496: URL: https://github.com/apache/flink/pull/14496#issuecomment-751242619 ## CI report: * b320965a86dfb89ae406cf19c77dd2e5f294d95f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11333) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20771) Hive partition is not added when there is a lot of data
[ https://issues.apache.org/jira/browse/FLINK-20771?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254876#comment-17254876 ] zhuxiaoshang commented on FLINK-20771: -- Hi [~hehuiyuan],this is dumplicated to https://issues.apache.org/jira/browse/FLINK-20213 > Hive partition is not added when there is a lot of data > --- > > Key: FLINK-20771 > URL: https://issues.apache.org/jira/browse/FLINK-20771 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Reporter: hehuiyuan >Priority: Major > Attachments: image-2020-12-25-18-09-42-707.png, > image-2020-12-25-18-15-07-519.png, image-2020-12-25-18-19-53-746.png > > > Hive partition is not added when the data is huge . > !image-2020-12-25-18-09-42-707.png|width=437,height=115! > Before partition commit, *inProgressPart* will be reinitialize . > the current bucket is active , so the notifyBucketInactive is not executed. > !image-2020-12-25-18-15-07-519.png|width=574,height=192! > > !image-2020-12-25-18-19-53-746.png|width=697,height=56! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14466: [FLINK-20657][connectors/jdbc] Migrate jdbc InputFormat/LookupFunction to SimpleJdbcConnectionProvider for connection establishment
flinkbot edited a comment on pull request #14466: URL: https://github.com/apache/flink/pull/14466#issuecomment-749592580 ## CI report: * cfb29832f3281af5b385f0ef87f17875224f474e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11336) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] xintongsong opened a new pull request #405: Add Apache Flink release 1.12.1
xintongsong opened a new pull request #405: URL: https://github.com/apache/flink-web/pull/405 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] godfreyhe closed pull request #14472: [Flink-20736] Separate implementation of BatchExecLimit,StreamExecLimit,StreamExecRank
godfreyhe closed pull request #14472: URL: https://github.com/apache/flink/pull/14472 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20736) Separate the implementation of BatchExecLimit and StreamExecLimit
[ https://issues.apache.org/jira/browse/FLINK-20736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] godfrey he closed FLINK-20736. -- Resolution: Fixed Fixed in 1.13.0: 677d392a..a25b9b84 > Separate the implementation of BatchExecLimit and StreamExecLimit > - > > Key: FLINK-20736 > URL: https://issues.apache.org/jira/browse/FLINK-20736 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Wenlong Lyu >Assignee: Wenlong Lyu >Priority: Major > Fix For: 1.13.0 > > > This is a subtask to separate physical limit and exec limit implementation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-20773) Support allow-unescaped-control-chars option for JSON format
[ https://issues.apache.org/jira/browse/FLINK-20773?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] xiaozilong updated FLINK-20773: --- Description: Can we add an option `allow-unescaped-control-chars` for json format because of it will throw exception when exist illegal unquoted characters in the data. !image-2020-12-25-20-21-50-637.png! was: Can we add an option `allow-unquoted-ctrl-char` for json format because of it will throw exception when exist illegal unquoted characters in the data. !image-2020-12-25-20-21-50-637.png! > Support allow-unescaped-control-chars option for JSON format > > > Key: FLINK-20773 > URL: https://issues.apache.org/jira/browse/FLINK-20773 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile) >Affects Versions: 1.12.0 >Reporter: xiaozilong >Priority: Major > Attachments: image-2020-12-25-20-21-50-637.png > > > Can we add an option `allow-unescaped-control-chars` for json format because > of it will throw exception when exist illegal unquoted characters in the data. > !image-2020-12-25-20-21-50-637.png! > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20762) Remove unused checkpointStorage path after job finished
[
https://issues.apache.org/jira/browse/FLINK-20762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254936#comment-17254936
]
Liu commented on FLINK-20762:
-
[~yunta] Thank you for your information. I have read the FLINK-15012. It is the
same problem with this ticket. I will keep watch FLINK-15012.
> Remove unused checkpointStorage path after job finished
> ---
>
> Key: FLINK-20762
> URL: https://issues.apache.org/jira/browse/FLINK-20762
> Project: Flink
> Issue Type: Improvement
>Reporter: Liu
>Priority: Minor
>
> Current checkpoint structure is as following:
> {code:java}
> /user-defined-checkpoint-dir
> |
> + --shared/
> + --taskowned/
> + --chk-1/
> + --chk-2/...
> {code}
> After cancelling job, the checkpointStorage will not be removed. For
> some cases, the dictionary can not be removed. For example, external
> checkpoint is retained or taskowned is used.
> If all the sub-dictionaries are empty, maybe the checkpointStorage can be
> removed. Otherwise, there will be lots of unused dictionaries with time going
> on. I wonder whether this is a problem. If so, I would like to fix it. Thank
> you.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-20762) Remove unused checkpointStorage path after job finished
[
https://issues.apache.org/jira/browse/FLINK-20762?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liu closed FLINK-20762.
---
Resolution: Duplicate
> Remove unused checkpointStorage path after job finished
> ---
>
> Key: FLINK-20762
> URL: https://issues.apache.org/jira/browse/FLINK-20762
> Project: Flink
> Issue Type: Improvement
>Reporter: Liu
>Priority: Minor
>
> Current checkpoint structure is as following:
> {code:java}
> /user-defined-checkpoint-dir
> |
> + --shared/
> + --taskowned/
> + --chk-1/
> + --chk-2/...
> {code}
> After cancelling job, the checkpointStorage will not be removed. For
> some cases, the dictionary can not be removed. For example, external
> checkpoint is retained or taskowned is used.
> If all the sub-dictionaries are empty, maybe the checkpointStorage can be
> removed. Otherwise, there will be lots of unused dictionaries with time going
> on. I wonder whether this is a problem. If so, I would like to fix it. Thank
> you.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17482) KafkaITCase.testMultipleSourcesOnePartition unstable
[
https://issues.apache.org/jira/browse/FLINK-17482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17254938#comment-17254938
]
Huang Xingbo commented on FLINK-17482:
--
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=11339&view=logs&j=c5f0071e-1851-543e-9a45-9ac140befc32&t=1fb1a56f-e8b5-5a82-00a0-a2db7757b4f5]
> KafkaITCase.testMultipleSourcesOnePartition unstable
>
>
> Key: FLINK-17482
> URL: https://issues.apache.org/jira/browse/FLINK-17482
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Robert Metzger
>Priority: Major
> Labels: test-stability
> Fix For: 1.13.0
>
>
> CI run:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=454&view=logs&j=c5f0071e-1851-543e-9a45-9ac140befc32&t=684b1416-4c17-504e-d5ab-97ee44e08a20
> {code}
> 07:29:40,472 [main] INFO
> org.apache.flink.streaming.connectors.kafka.KafkaTestBase[] -
> -
> [ERROR] Tests run: 22, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 152.018 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kafka.KafkaITCase
> [ERROR]
> testMultipleSourcesOnePartition(org.apache.flink.streaming.connectors.kafka.KafkaITCase)
> Time elapsed: 4.257 s <<< FAILURE!
> java.lang.AssertionError: Test failed: Job execution failed.
> at org.junit.Assert.fail(Assert.java:88)
> at org.apache.flink.test.util.TestUtils.tryExecute(TestUtils.java:45)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaConsumerTestBase.runMultipleSourcesOnePartitionExactlyOnceTest(KafkaConsumerTestBase.java:963)
> at
> org.apache.flink.streaming.connectors.kafka.KafkaITCase.testMultipleSourcesOnePartition(KafkaITCase.java:107)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:298)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:292)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
flinkbot edited a comment on pull request #8952: URL: https://github.com/apache/flink/pull/8952#issuecomment-513724324 ## CI report: * d083b630115604e34b0a74498890aedbff61b2a7 UNKNOWN * 67e9e1e4c013b78eaee9fc9d215ef400bba5cc59 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=11309) * c6cd36a8dfd854ecf63b2d248a4a28f889c873ec UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548938069
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/execution/ThresholdExceedException.java
##
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.execution;
+
+/**
+ * Exception thrown when a threshold exceeds.
+ */
+public class ThresholdExceedException extends RuntimeException {
Review comment:
Agree. Moved into TimestampBasedThreshold as suggested in later comment.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548938104
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/execution/ThresholdExceedException.java
##
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.execution;
+
+/**
+ * Exception thrown when a threshold exceeds.
+ */
+public class ThresholdExceedException extends RuntimeException {
+
+ private static final long serialVersionUID = 241871312848349L;
Review comment:
Fixed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548938183
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/FailureRaterUtil.java
##
@@ -0,0 +1,37 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.configuration.ResourceManagerOptions;
+import org.apache.flink.util.Preconditions;
+
+/**
+ * Failure rate util.
+ */
+public class FailureRaterUtil {
Review comment:
Agree. It should be moved to TimestampBasedFailureRater after
NoFailreRater is removed.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548938212
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/ThresholdMeter.java
##
@@ -0,0 +1,36 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.metrics.Meter;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+/**
+ * Metric for measuring rate with Threshold.
+ */
+public interface ThresholdMeter extends Meter {
+
+ /**
+* Check whether the current rate of events exceed threshold.
+* If yes, throw ThresholdExceedException.
+*
+* @throws ThresholdExceedException
+*/
+ void checkAgainstThreshold() throws ThresholdExceedException;
Review comment:
Updated accordingly.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
HuangZhenQiu commented on a change in pull request #8952: URL: https://github.com/apache/flink/pull/8952#discussion_r548938231 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java ## @@ -0,0 +1,93 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.flink.runtime.failurerate; Review comment: Updated Accordingly. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548939285
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
Review comment:
There actually a three-layer concept here. ThresholdMeter is a Meter
that has a checkAgainstThreshold interface. The TimestampBasedThresholdMeter is
a concrete implementation. The failure rate is actually a customized
TimestampBasedThresholdMeter, as we put the fromConfiguration and exception
message in the class. I am good with split into three layers. But as it is the
only use case, probably it will be better to keep it as it is?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548939286
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final Supplier currentTimeMillisSupplier;
+ private final double maximumFailureRate;
+ private final Time failureInterval;
+ private final Queue failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(double maximumFailureRate, Time
failureInterval) {
+ this(maximumFailureRate, failureInterval,
System::currentTimeMillis);
+ }
+
+ @VisibleForTesting
+ public TimestampBasedFailureRater(double maximumFailureRate, Time
failureInterval, Supplier customSupplier) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failureInterval = failureInterval;
+ this.failureTimestamps = new ArrayDeque<>();
+ this.currentTimeMillisSupplier = customSupplier;
+ }
+
+ @Override
+ public void markEvent() {
+ failureTimestamps.add(System.currentTimeMillis());
+ failureCounter++;
+ }
+
+ @Override
+ public void markEvent(long n) {
+ for (int i = 0; i < n; i++) {
+ failureTimestamps.add(System.currentTimeMillis());
+ }
+ failureCounter = failureCounter + n;
+ }
+
+ @Override
+ public double getRate() {
+ return getCurrentFailureRate() /
(failureInterval.toMilliseconds() / MILLISECONDS_PER_SECOND);
+ }
+
+ @Override
+ public long getCount() {
+ return failureCounter;
+ }
+
+ @Override
+ public void checkAgainstThreshold() throws ThresholdExceedException {
+ if (getCurrentFailureRate() >= maximumFailureRate) {
+ throw new
ThresholdExceedException(String.format("Maximum number of failed workers %f"
+ + " is detected in Resource Manager",
getCurrentFailureRate()));
+ }
+ }
+
+ private double getCurrentFailureRate() {
+ Long currentTimeStamp = System.currentTimeMillis();
Review comment:
Done.
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548939312
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/failurerate/TimestampBasedFailureRater.java
##
@@ -0,0 +1,93 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.failurerate;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.execution.ThresholdExceedException;
+
+import java.util.ArrayDeque;
+import java.util.Queue;
+import java.util.function.Supplier;
+
+/**
+ * A timestamp queue based failure rater implementation.
+ */
+public class TimestampBasedFailureRater implements ThresholdMeter {
+ private static final Double MILLISECONDS_PER_SECOND = 1000.0;
+ private final Supplier currentTimeMillisSupplier;
+ private final double maximumFailureRate;
+ private final Time failureInterval;
+ private final Queue failureTimestamps;
+ private long failureCounter = 0;
+
+ public TimestampBasedFailureRater(double maximumFailureRate, Time
failureInterval) {
+ this(maximumFailureRate, failureInterval,
System::currentTimeMillis);
+ }
+
+ @VisibleForTesting
+ public TimestampBasedFailureRater(double maximumFailureRate, Time
failureInterval, Supplier customSupplier) {
+ this.maximumFailureRate = maximumFailureRate;
+ this.failureInterval = failureInterval;
+ this.failureTimestamps = new ArrayDeque<>();
+ this.currentTimeMillisSupplier = customSupplier;
+ }
+
+ @Override
+ public void markEvent() {
+ failureTimestamps.add(System.currentTimeMillis());
+ failureCounter++;
+ }
+
+ @Override
+ public void markEvent(long n) {
+ for (int i = 0; i < n; i++) {
+ failureTimestamps.add(System.currentTimeMillis());
+ }
+ failureCounter = failureCounter + n;
+ }
+
+ @Override
+ public double getRate() {
+ return getCurrentFailureRate() /
(failureInterval.toMilliseconds() / MILLISECONDS_PER_SECOND);
+ }
+
+ @Override
+ public long getCount() {
+ return failureCounter;
+ }
+
+ @Override
+ public void checkAgainstThreshold() throws ThresholdExceedException {
+ if (getCurrentFailureRate() >= maximumFailureRate) {
Review comment:
Updated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548939317
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/metrics/MetricNames.java
##
@@ -67,6 +67,7 @@ private MetricNames() {
public static final String CHECKPOINT_ALIGNMENT_TIME =
"checkpointAlignmentTime";
public static final String CHECKPOINT_START_DELAY_TIME =
"checkpointStartDelayNanos";
+ public static final String WORKER_FAILURE_RATE = "workFailureRate";
Review comment:
Done.
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/ResourceManager.java
##
@@ -108,6 +111,8 @@
public static final String RESOURCE_MANAGER_NAME = "resourcemanager";
+ protected final Logger log = LoggerFactory.getLogger(getClass());
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on a change in pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
HuangZhenQiu commented on a change in pull request #8952:
URL: https://github.com/apache/flink/pull/8952#discussion_r548939375
##
File path:
flink-core/src/main/java/org/apache/flink/configuration/ResourceManagerOptions.java
##
@@ -67,6 +69,33 @@
"for streaming workloads, which may fail if there are
not enough slots. Note that this configuration option does not take " +
"effect for standalone clusters, where how many slots
are allocated is not controlled by Flink.");
+ /**
+* Defines the maximum number of worker (YARN / Mesos / Kubernetes)
failures per minute before rejecting subsequent worker
+* requests until the failure rate falls below the maximum. It is to
quickly catch external dependency caused
+* workers failure and wait for retry interval before sending new
request. By default, the value is set to 10/min.
+*/
+ public static final ConfigOption MAXIMUM_WORKERS_FAILURE_RATE =
ConfigOptions
+ .key("resourcemanager.start-worker.max-failure-rate")
+ .doubleType()
+ .defaultValue(10.0)
+ .withDescription("Defines the maximum number of worker (YARN /
Mesos) failures per minute before rejecting" +
+ " subsequent worker requests until the failure rate
falls below the maximum. It is to quickly catch" +
+ " external dependency caused workers failure and
terminate job accordingly." +
+ " By default, the value is set to 10/min.");
+
+ /**
+* Defines the worker creation interval in milliseconds. In case of
worker creation failures, we should wait for an interval before
+* trying to create new workers when the failure rate exceeds.
Otherwise, ActiveResourceManager will always re-requesting
+* the worker, which keeps the main thread busy.
+*/
+ public static final ConfigOption
WORKER_CREATION_RETRY_INTERVAL = ConfigOptions
+ .key("resourcemanager.start-worker.retry-interval")
+ .durationType()
+ .defaultValue(Duration.ofMillis(30))
Review comment:
Updated to 3 seconds as the default value of original Kubernetes
interval.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] HuangZhenQiu commented on pull request #8952: [FLINK-10868][flink-runtime] Add failure rater for resource manager
HuangZhenQiu commented on pull request #8952: URL: https://github.com/apache/flink/pull/8952#issuecomment-751315528 @xintongsong Thanks for the comments. I do like the simplified version after grouping classes and utils together. For the TimestampBasedFailureRater, I would prefer to leave it as it is. Otherwise, we need to split the 100 lines of code into ThresholdMeter, TimestampBasedThresholdMeter, and FailureRater. I think we can keep it simple and refactor it once ThresholdMeter, TimestampBasedThresholdMeter can be usable in other components. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
