[GitHub] [flink] wangyang0918 commented on issue #9965: [FLINK-10935][kubernetes]Implement KubeClient with Faric8 Kubernetes clients
wangyang0918 commented on issue #9965: [FLINK-10935][kubernetes]Implement KubeClient with Faric8 Kubernetes clients URL: https://github.com/apache/flink/pull/9965#issuecomment-561046399 @zhuzhurk I have addressed all your comments. Please take a look. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wangyang0918 commented on a change in pull request #9965: [FLINK-10935][kubernetes]Implement KubeClient with Faric8 Kubernetes clients
wangyang0918 commented on a change in pull request #9965:
[FLINK-10935][kubernetes]Implement KubeClient with Faric8 Kubernetes clients
URL: https://github.com/apache/flink/pull/9965#discussion_r353028432
##
File path:
flink-kubernetes/src/test/java/org/apache/flink/kubernetes/KubernetesTestBase.java
##
@@ -0,0 +1,193 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes;
+
+import org.apache.flink.configuration.ConfigConstants;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.kubernetes.configuration.KubernetesConfigOptions;
+import
org.apache.flink.kubernetes.configuration.KubernetesConfigOptionsInternal;
+import org.apache.flink.kubernetes.kubeclient.Fabric8FlinkKubeClient;
+import org.apache.flink.kubernetes.kubeclient.FlinkKubeClient;
+import org.apache.flink.kubernetes.kubeclient.decorators.Decorator;
+import org.apache.flink.kubernetes.kubeclient.decorators.InitializerDecorator;
+import
org.apache.flink.kubernetes.kubeclient.decorators.OwnerReferenceDecorator;
+import org.apache.flink.kubernetes.kubeclient.decorators.ServiceDecorator;
+import org.apache.flink.kubernetes.kubeclient.resources.KubernetesService;
+import org.apache.flink.kubernetes.utils.Constants;
+import org.apache.flink.runtime.clusterframework.BootstrapTools;
+import org.apache.flink.test.util.TestBaseUtils;
+import org.apache.flink.util.TestLogger;
+
+import io.fabric8.kubernetes.api.model.LoadBalancerIngress;
+import io.fabric8.kubernetes.api.model.LoadBalancerStatus;
+import io.fabric8.kubernetes.api.model.Service;
+import io.fabric8.kubernetes.api.model.ServiceStatusBuilder;
+import io.fabric8.kubernetes.api.model.WatchEvent;
+import io.fabric8.kubernetes.client.KubernetesClient;
+import org.junit.Before;
+import org.junit.Rule;
+import org.junit.rules.TemporaryFolder;
+
+import java.io.File;
+import java.io.IOException;
+import java.util.ArrayList;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+
+/**
+ * Base test class for Kubernetes.
+ */
+public class KubernetesTestBase extends TestLogger {
+ @Rule
+ public MixedKubernetesServer server = new MixedKubernetesServer(true,
true);
+
+ @Rule
+ public TemporaryFolder temporaryFolder = new TemporaryFolder();
+
+ private File flinkConfDir;
+
+ protected final String nameSpace = "test";
Review comment:
I will make these variables `protected static final`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] godfreyhe commented on a change in pull request #10315: [FLINK-14552][table] Enable partition statistics in blink planner
godfreyhe commented on a change in pull request #10315: [FLINK-14552][table] Enable partition statistics in blink planner URL: https://github.com/apache/flink/pull/10315#discussion_r353028551 ## File path: flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/plan/schema/TableSourceTable.scala ## @@ -50,7 +51,8 @@ class TableSourceTable[T]( statistic: FlinkStatistic, val tableSource: TableSource[T], val isStreamingMode: Boolean, -val catalogTable: CatalogTable) +val catalogTable: CatalogTable, +val tableIdentifier: Option[ObjectIdentifier]) Review comment: It's better to make the comment of `tableIdentifier` field more clear This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wangyang0918 commented on a change in pull request #9985: [FLINK-10968][kubernetes] Implement TaskManager Entrypoint for Kubernetes.
wangyang0918 commented on a change in pull request #9985:
[FLINK-10968][kubernetes] Implement TaskManager Entrypoint for Kubernetes.
URL: https://github.com/apache/flink/pull/9985#discussion_r353029210
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskManagerRunner.java
##
@@ -331,6 +311,25 @@ public static void runTaskManager(Configuration
configuration, ResourceID resour
taskManagerRunner.start();
}
+ public static void runTaskManagerSecurely(String[] args, ResourceID
resourceID) {
+ try {
+ final Configuration configuration =
loadConfiguration(args);
+
+ FileSystem.initialize(configuration,
PluginUtils.createPluginManagerFromRootFolder(configuration));
+
+ SecurityUtils.install(new
SecurityConfiguration(configuration));
+
+
SecurityUtils.getInstalledContext().runSecured((Callable) () -> {
Review comment:
Right. I will remove the `(Callable)`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] pnowojski merged pull request #10347: [FLINK-14516][network] Remove non credit-based flow control code
pnowojski merged pull request #10347: [FLINK-14516][network] Remove non credit-based flow control code URL: https://github.com/apache/flink/pull/10347 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] pnowojski commented on issue #10347: [FLINK-14516][network] Remove non credit-based flow control code
pnowojski commented on issue #10347: [FLINK-14516][network] Remove non credit-based flow control code URL: https://github.com/apache/flink/pull/10347#issuecomment-561048333 Thanks for the review @zhijiangW. Merging This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-14516) Remove non credit based network code
[
https://issues.apache.org/jira/browse/FLINK-14516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski closed FLINK-14516.
--
Fix Version/s: 1.10.0
Release Note: Non credit-based network flow control code was removed
alongside of the configuration option {{ taskmanager.network.credit-model }}.
It's default value {{true}} is now the only option in Flink.
Resolution: Fixed
Merged to master as 2dc0e3a3663f6c09ad726dfacad53760c47e15a0^ and
2dc0e3a3663f6c09ad726dfacad53760c47e15a0
> Remove non credit based network code
>
>
> Key: FLINK-14516
> URL: https://issues.apache.org/jira/browse/FLINK-14516
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Network
>Affects Versions: 1.9.0
>Reporter: Piotr Nowojski
>Assignee: Piotr Nowojski
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> After [a survey on the dev mailing
> list|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/SURVEY-Dropping-non-Credit-based-Flow-Control-td33714.html]
> the feedback was that old code path is not used and no longer needed. Based
> on that we should be safe to drop it and make credit based flow control the
> only option (currently it's the default).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] TisonKun commented on issue #10346: [FLINK-14972] Make Remote(Stream)Environment use Executors.
TisonKun commented on issue #10346: [FLINK-14972] Make Remote(Stream)Environment use Executors. URL: https://github.com/apache/flink/pull/10346#issuecomment-561050084 Hi @kl0u @aljoscha FLINK-14992 is waiting for this pull request. I don't see a blocker prevents this pr from merging. Could you take a look? I'm anyway start the work of FLINK-14992 tomorrow, which would conflict with the pull request. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wangyang0918 commented on a change in pull request #9985: [FLINK-10968][kubernetes] Implement TaskManager Entrypoint for Kubernetes.
wangyang0918 commented on a change in pull request #9985:
[FLINK-10968][kubernetes] Implement TaskManager Entrypoint for Kubernetes.
URL: https://github.com/apache/flink/pull/9985#discussion_r353031884
##
File path:
flink-yarn/src/main/java/org/apache/flink/yarn/YarnTaskExecutorRunner.java
##
@@ -97,22 +91,7 @@ private static void run(String[] args) {
final String currDir = ENV.get(Environment.PWD.key());
LOG.info("Current working Directory: {}", currDir);
- // Some dynamic properties for task manager are added to args, such as
managed memory size.
- final CommandLineParser
commandLineParser =
- new CommandLineParser<>(new
ClusterConfigurationParserFactory());
-
- ClusterConfiguration clusterConfiguration = null;
- try {
- clusterConfiguration =
commandLineParser.parse(args);
- } catch (FlinkParseException e) {
- LOG.error("Could not parse command line
arguments {}.", args, e);
-
commandLineParser.printHelp(YarnTaskExecutorRunner.class.getSimpleName());
- System.exit(1);
Review comment:
I think we should not `System.exit(1)` here. We should use
`INIT_ERROR_EXIT_CODE` instead.
We just can not use `TaskManagerRunner#runTaskManagerSecurely` for
`YarnTaskExecutorRunner`. Because the configuration needs to be updated and
then use the updated configuration to start task executor. It is the unique
logics of Yarn.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] TisonKun edited a comment on issue #10346: [FLINK-14972] Make Remote(Stream)Environment use Executors.
TisonKun edited a comment on issue #10346: [FLINK-14972] Make Remote(Stream)Environment use Executors. URL: https://github.com/apache/flink/pull/10346#issuecomment-561050084 Hi @kl0u @aljoscha FLINK-14992 is waiting for this pull request. I don't see a blocker prevents this pr from merging. Could you take a look? I'll anyway start the work of FLINK-14992 tomorrow, which would conflict with the pull request. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] TisonKun commented on issue #10370: [FLINK-14947] Introduce LocalExecutor and make Local(Stream)Environment use it
TisonKun commented on issue #10370: [FLINK-14947] Introduce LocalExecutor and make Local(Stream)Environment use it URL: https://github.com/apache/flink/pull/10370#issuecomment-561049953 Hi @kl0u @aljoscha FLINK-14992 is waiting for this pull request. I don't see a blocker prevents this pr from merging. Could you take a look? I'm anyway start the work of FLINK-14992 tomorrow, which would conflict with the pull request. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] TisonKun edited a comment on issue #10370: [FLINK-14947] Introduce LocalExecutor and make Local(Stream)Environment use it
TisonKun edited a comment on issue #10370: [FLINK-14947] Introduce LocalExecutor and make Local(Stream)Environment use it URL: https://github.com/apache/flink/pull/10370#issuecomment-561049953 Hi @kl0u @aljoscha FLINK-14992 is waiting for this pull request. I don't see a blocker prevents this pr from merging. Could you take a look? I'll anyway start the work of FLINK-14992 tomorrow, which would conflict with the pull request. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wangyang0918 commented on a change in pull request #9985: [FLINK-10968][kubernetes] Implement TaskManager Entrypoint for Kubernetes.
wangyang0918 commented on a change in pull request #9985:
[FLINK-10968][kubernetes] Implement TaskManager Entrypoint for Kubernetes.
URL: https://github.com/apache/flink/pull/9985#discussion_r353032583
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/utils/Constants.java
##
@@ -0,0 +1,27 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.kubernetes.utils;
+
+/**
+ * Constants for kubernetes.
+ */
+public class Constants {
Review comment:
I will make this PR depends on #FLINK-10935. The class `Constants` will be
created in that PR. We just add a new variable `ENV_FLINK_POD_NAME`. Does it
make sense?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Resolved] (FLINK-15028) YARNSessionFIFOSecuredITCase test failed with java11 on travis
[
https://issues.apache.org/jira/browse/FLINK-15028?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li resolved FLINK-15028.
---
Resolution: Duplicate
> YARNSessionFIFOSecuredITCase test failed with java11 on travis
> --
>
> Key: FLINK-15028
> URL: https://issues.apache.org/jira/browse/FLINK-15028
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.10.0
>Reporter: Yu Li
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.10.0
>
>
> YARNSessionFIFOSecuredITCase failed at the {{setup}} phase with below
> exceptions with java11:
> {code}
> Caused by: java.lang.ClassNotFoundException: javax.activation.DataSource
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> {code}
> https://api.travis-ci.org/v3/job/619217945/log.txt
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10325: [FLINK-14512][table] Introduce listPartitionsByFilter to Catalog
flinkbot edited a comment on issue #10325: [FLINK-14512][table] Introduce listPartitionsByFilter to Catalog URL: https://github.com/apache/flink/pull/10325#issuecomment-558654173 ## CI report: * e41fee5e6b021d3d92abee1fcfbcc7344650bac3 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/138243833) * 224d719bcb6d52c9417031ef357d24f055e3de01 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/139093320) * 0e76e8658c794c4bfc20c2dcaffe1c55394adb50 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/139095677) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-15023) Remove on-heap managed memory
[
https://issues.apache.org/jira/browse/FLINK-15023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986693#comment-16986693
]
Zhu Zhu edited comment on FLINK-15023 at 12/3/19 8:19 AM:
--
Should we also rename the {{offHeapManagedMemory}} to {{managedMemory}}?
was (Author: zhuzh):
Shall we also rename the {{offHeapManagedMemory}} to {{managedMemory}}?
> Remove on-heap managed memory
> -
>
> Key: FLINK-15023
> URL: https://issues.apache.org/jira/browse/FLINK-15023
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Configuration
>Reporter: Xintong Song
>Priority: Major
>
> As mentioned in [this discussion
> thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Make-Managed-Memory-always-off-heap-Adjustment-to-FLIP-49-td35365.html],
> we want to make managed memory always off-heap.
> This task include changes:
> * Remove `MEMORY_OFF_HEAP`, `MANAGED_MEMORY_OFFHEAP_FRACTION` and
> `MANAGED_MEMORY_OFFHEAP_SIZE` from `TaskManagerOptions`.
> * Remove `onHeapManagedMemory` from `ResourceProfile`, `ResourceSpec` and
> `TaskExecutorResourceSpec`.
> * Remove on heap managed memory from `MemoryManager`
> * Remove on heap managed memory fraction from `StreamConfig`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15023) Remove on-heap managed memory
[
https://issues.apache.org/jira/browse/FLINK-15023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986693#comment-16986693
]
Zhu Zhu commented on FLINK-15023:
-
Shall we also rename the {{offHeapManagedMemory}} to {{managedMemory}}?
> Remove on-heap managed memory
> -
>
> Key: FLINK-15023
> URL: https://issues.apache.org/jira/browse/FLINK-15023
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Configuration
>Reporter: Xintong Song
>Priority: Major
>
> As mentioned in [this discussion
> thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Make-Managed-Memory-always-off-heap-Adjustment-to-FLIP-49-td35365.html],
> we want to make managed memory always off-heap.
> This task include changes:
> * Remove `MEMORY_OFF_HEAP`, `MANAGED_MEMORY_OFFHEAP_FRACTION` and
> `MANAGED_MEMORY_OFFHEAP_SIZE` from `TaskManagerOptions`.
> * Remove `onHeapManagedMemory` from `ResourceProfile`, `ResourceSpec` and
> `TaskExecutorResourceSpec`.
> * Remove on heap managed memory from `MemoryManager`
> * Remove on heap managed memory fraction from `StreamConfig`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15004) Choose two-phase Aggregate if the statistics is unknown
[
https://issues.apache.org/jira/browse/FLINK-15004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
godfrey he updated FLINK-15004:
---
Summary: Choose two-phase Aggregate if the statistics is unknown (was:
Choose SortMergeJoin instead of HashJoin if the statistics is unknown)
> Choose two-phase Aggregate if the statistics is unknown
> ---
>
> Key: FLINK-15004
> URL: https://issues.apache.org/jira/browse/FLINK-15004
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.9.1
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, blink planner will use default rowCount value (defined in
> {{FlinkPreparingTableBase#DEFAULT_ROWCOUNT}} ) when the statistics is
> unknown, and maybe choose {{HashJoin}} instead of {{SortMergeJoin}}. The job
> will hang if the build side of {{HashJoin}} has huge input size. So It's
> better to use {{SortMergeJoin}} for execution stability if the statistics is
> unknown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zhuzhurk commented on issue #9965: [FLINK-10935][kubernetes]Implement KubeClient with Faric8 Kubernetes clients
zhuzhurk commented on issue #9965: [FLINK-10935][kubernetes]Implement KubeClient with Faric8 Kubernetes clients URL: https://github.com/apache/flink/pull/9965#issuecomment-561053083 LGTM. +1 to merge it. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wangyang0918 commented on a change in pull request #9985: [FLINK-10968][kubernetes] Implement TaskManager Entrypoint for Kubernetes.
wangyang0918 commented on a change in pull request #9985:
[FLINK-10968][kubernetes] Implement TaskManager Entrypoint for Kubernetes.
URL: https://github.com/apache/flink/pull/9985#discussion_r353031884
##
File path:
flink-yarn/src/main/java/org/apache/flink/yarn/YarnTaskExecutorRunner.java
##
@@ -97,22 +91,7 @@ private static void run(String[] args) {
final String currDir = ENV.get(Environment.PWD.key());
LOG.info("Current working Directory: {}", currDir);
- // Some dynamic properties for task manager are added to args, such as
managed memory size.
- final CommandLineParser
commandLineParser =
- new CommandLineParser<>(new
ClusterConfigurationParserFactory());
-
- ClusterConfiguration clusterConfiguration = null;
- try {
- clusterConfiguration =
commandLineParser.parse(args);
- } catch (FlinkParseException e) {
- LOG.error("Could not parse command line
arguments {}.", args, e);
-
commandLineParser.printHelp(YarnTaskExecutorRunner.class.getSimpleName());
- System.exit(1);
Review comment:
I think we should not `System.exit(1)` here. We should use
`INIT_ERROR_EXIT_CODE` instead. The exception of
`TaskManagerRunner.loadConfiguration` will be caught.
We just can not use `TaskManagerRunner#runTaskManagerSecurely` for
`YarnTaskExecutorRunner`. Because the configuration needs to be updated and
then use the updated configuration to start task executor. It is the unique
logics of Yarn.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] wangyang0918 commented on issue #9965: [FLINK-10935][kubernetes]Implement KubeClient with Faric8 Kubernetes clients
wangyang0918 commented on issue #9965: [FLINK-10935][kubernetes]Implement KubeClient with Faric8 Kubernetes clients URL: https://github.com/apache/flink/pull/9965#issuecomment-561054031 @TisonKun Please have a final check and help to merge. Really thanks for your kind help. -:) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-15004) Choose two-phase Aggregate if the statistics is unknown
[
https://issues.apache.org/jira/browse/FLINK-15004?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
godfrey he updated FLINK-15004:
---
Description: Currently, blink planner will use default rowCount value
(defined in {{FlinkPreparingTableBase#DEFAULT_ROWCOUNT}} ) when the statistics
is unknown, and maybe choose one-phase Aggregate. The job will hang if the data
is skewed. So It's better to use two-phase Aggregate for execution stability if
the statistics is unknown. (was: Currently, blink planner will use default
rowCount value (defined in {{FlinkPreparingTableBase#DEFAULT_ROWCOUNT}} ) when
the statistics is unknown, and maybe choose {{HashJoin}} instead of
{{SortMergeJoin}}. The job will hang if the build side of {{HashJoin}} has huge
input size. So It's better to use {{SortMergeJoin}} for execution stability if
the statistics is unknown.)
> Choose two-phase Aggregate if the statistics is unknown
> ---
>
> Key: FLINK-15004
> URL: https://issues.apache.org/jira/browse/FLINK-15004
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.9.1
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, blink planner will use default rowCount value (defined in
> {{FlinkPreparingTableBase#DEFAULT_ROWCOUNT}} ) when the statistics is
> unknown, and maybe choose one-phase Aggregate. The job will hang if the data
> is skewed. So It's better to use two-phase Aggregate for execution stability
> if the statistics is unknown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15023) Remove on-heap managed memory
[ https://issues.apache.org/jira/browse/FLINK-15023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986698#comment-16986698 ] Xintong Song commented on FLINK-15023: -- True, we should also do that. > Remove on-heap managed memory > - > > Key: FLINK-15023 > URL: https://issues.apache.org/jira/browse/FLINK-15023 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Configuration >Reporter: Xintong Song >Priority: Major > > As mentioned in [this discussion > thread|http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Make-Managed-Memory-always-off-heap-Adjustment-to-FLIP-49-td35365.html], > we want to make managed memory always off-heap. > This task include changes: > * Remove `MEMORY_OFF_HEAP`, `MANAGED_MEMORY_OFFHEAP_FRACTION` and > `MANAGED_MEMORY_OFFHEAP_SIZE` from `TaskManagerOptions`. > * Remove `onHeapManagedMemory` from `ResourceProfile`, `ResourceSpec` and > `TaskExecutorResourceSpec`. > * Remove on heap managed memory from `MemoryManager` > * Remove on heap managed memory fraction from `StreamConfig`. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9965: [FLINK-10935][kubernetes]Implement KubeClient with Faric8 Kubernetes clients
flinkbot edited a comment on issue #9965: [FLINK-10935][kubernetes]Implement KubeClient with Faric8 Kubernetes clients URL: https://github.com/apache/flink/pull/9965#issuecomment-544813931 ## CI report: * 6f90b457e56a0a8cb45d63c1b05b47d2e38030a1 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132938440) * 86aa5ce8f77faf233c51a7231b3f71e518fd6c92 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132962300) * 7851d845a43f799627b2c788ace8eb7e6caccb03 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/133133283) * d49bb0a622e1667baffd29f19fdcc60d0022fe82 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133366914) * 0e88e0e5be77f450c82cbc460ea4f02a1effc920 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133998649) * 9afcc7ba840186c68f36b30d6b28b8c1cbf09b61 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/137145224) * 06e6b2bee4c1788b150f2b83c43eb4723709864b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138017584) * 816a11afe713e736cdfd2eb566762ee2addf7071 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138708643) * e80691082c9fc8ac704b0bccaa5180b7de0718d5 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138770607) * e85e76c018b7606381c5869e0c1054e02c4a2321 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138957695) * d1c41ae28960b3eea21fc662e771fe44c1c53b12 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] StephanEwen commented on issue #10382: [FLINK-15014][state-processor-api] Refactor KeyedStateInputFormat to support multiple types of user functions
StephanEwen commented on issue #10382: [FLINK-15014][state-processor-api] Refactor KeyedStateInputFormat to support multiple types of user functions URL: https://github.com/apache/flink/pull/10382#issuecomment-561055507 @flinkbot approve all This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] StephanEwen commented on issue #10382: [FLINK-15014][state-processor-api] Refactor KeyedStateInputFormat to support multiple types of user functions
StephanEwen commented on issue #10382: [FLINK-15014][state-processor-api] Refactor KeyedStateInputFormat to support multiple types of user functions URL: https://github.com/apache/flink/pull/10382#issuecomment-561055534 Looks fine to me as well. +1 Will merge this... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #10296: [FLINK-14691][table]Add use/create/drop/alter database operation and support it in flink/blink planner
KurtYoung commented on a change in pull request #10296: [FLINK-14691][table]Add
use/create/drop/alter database operation and support it in flink/blink planner
URL: https://github.com/apache/flink/pull/10296#discussion_r353028884
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/TableEnvironmentImpl.java
##
@@ -469,21 +479,84 @@ public void sqlUpdate(String stmt) {
createTableOperation.getCatalogTable(),
createTableOperation.getTableIdentifier(),
createTableOperation.isIgnoreIfExists());
+ } else if (operation instanceof CreateDatabaseOperation) {
+ CreateDatabaseOperation createDatabaseOperation =
(CreateDatabaseOperation) operation;
+ Catalog catalog =
getCatalogOrThrowException(createDatabaseOperation.getCatalogName());
+ String exMsg = getDDLOpExecuteErrorMsg("CREATE
DATABASE", createDatabaseOperation.getCatalogName());
+ try {
+ catalog.createDatabase(
+
createDatabaseOperation.getDatabaseName(),
+
createDatabaseOperation.getCatalogDatabase(),
+
createDatabaseOperation.isIgnoreIfExists());
+ } catch (DatabaseAlreadyExistException e) {
+ throw new ValidationException(exMsg, e);
+ } catch (Exception e) {
+ throw new TableException(exMsg, e);
+ }
} else if (operation instanceof DropTableOperation) {
DropTableOperation dropTableOperation =
(DropTableOperation) operation;
catalogManager.dropTable(
dropTableOperation.getTableIdentifier(),
dropTableOperation.isIfExists());
- } else if (operation instanceof UseCatalogOperation) {
+ } else if (operation instanceof DropDatabaseOperation) {
+ DropDatabaseOperation dropDatabaseOperation =
(DropDatabaseOperation) operation;
+ Catalog catalog =
getCatalogOrThrowException(dropDatabaseOperation.getCatalogName());
+ String exMsg = getDDLOpExecuteErrorMsg("DROP DATABASE",
dropDatabaseOperation.getCatalogName());
Review comment:
ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #10296: [FLINK-14691][table]Add use/create/drop/alter database operation and support it in flink/blink planner
KurtYoung commented on a change in pull request #10296: [FLINK-14691][table]Add
use/create/drop/alter database operation and support it in flink/blink planner
URL: https://github.com/apache/flink/pull/10296#discussion_r353028839
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/TableEnvironmentImpl.java
##
@@ -469,21 +479,84 @@ public void sqlUpdate(String stmt) {
createTableOperation.getCatalogTable(),
createTableOperation.getTableIdentifier(),
createTableOperation.isIgnoreIfExists());
+ } else if (operation instanceof CreateDatabaseOperation) {
+ CreateDatabaseOperation createDatabaseOperation =
(CreateDatabaseOperation) operation;
+ Catalog catalog =
getCatalogOrThrowException(createDatabaseOperation.getCatalogName());
+ String exMsg = getDDLOpExecuteErrorMsg("CREATE
DATABASE", createDatabaseOperation.getCatalogName());
Review comment:
replace `CREATE DATABASE` with `createDatabaseOperation.asSummaryString`?
And you don't need provide catalog name anymore
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #10296: [FLINK-14691][table]Add use/create/drop/alter database operation and support it in flink/blink planner
KurtYoung commented on a change in pull request #10296: [FLINK-14691][table]Add
use/create/drop/alter database operation and support it in flink/blink planner
URL: https://github.com/apache/flink/pull/10296#discussion_r353029423
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/TableEnvironmentImpl.java
##
@@ -469,21 +479,84 @@ public void sqlUpdate(String stmt) {
createTableOperation.getCatalogTable(),
createTableOperation.getTableIdentifier(),
createTableOperation.isIgnoreIfExists());
+ } else if (operation instanceof CreateDatabaseOperation) {
+ CreateDatabaseOperation createDatabaseOperation =
(CreateDatabaseOperation) operation;
+ Catalog catalog =
getCatalogOrThrowException(createDatabaseOperation.getCatalogName());
+ String exMsg = getDDLOpExecuteErrorMsg("CREATE
DATABASE", createDatabaseOperation.getCatalogName());
+ try {
+ catalog.createDatabase(
+
createDatabaseOperation.getDatabaseName(),
+
createDatabaseOperation.getCatalogDatabase(),
+
createDatabaseOperation.isIgnoreIfExists());
+ } catch (DatabaseAlreadyExistException e) {
+ throw new ValidationException(exMsg, e);
+ } catch (Exception e) {
+ throw new TableException(exMsg, e);
+ }
} else if (operation instanceof DropTableOperation) {
DropTableOperation dropTableOperation =
(DropTableOperation) operation;
catalogManager.dropTable(
dropTableOperation.getTableIdentifier(),
dropTableOperation.isIfExists());
- } else if (operation instanceof UseCatalogOperation) {
+ } else if (operation instanceof DropDatabaseOperation) {
+ DropDatabaseOperation dropDatabaseOperation =
(DropDatabaseOperation) operation;
+ Catalog catalog =
getCatalogOrThrowException(dropDatabaseOperation.getCatalogName());
+ String exMsg = getDDLOpExecuteErrorMsg("DROP DATABASE",
dropDatabaseOperation.getCatalogName());
+ try {
+ catalog.dropDatabase(
+
dropDatabaseOperation.getDatabaseName(),
+
dropDatabaseOperation.isIfExists(),
+
dropDatabaseOperation.isCascade());
+ } catch (DatabaseNotExistException |
DatabaseNotEmptyException e) {
+ throw new ValidationException(exMsg, e);
+ } catch (Exception e) {
+ throw new TableException(exMsg, e);
+ }
+ } else if (operation instanceof AlterDatabaseOperation) {
+ AlterDatabaseOperation alterDatabaseOperation =
(AlterDatabaseOperation) operation;
+ Catalog catalog =
getCatalogOrThrowException(alterDatabaseOperation.getCatalogName());
+ String exMsg = getDDLOpExecuteErrorMsg("ALTER
DATABASE", alterDatabaseOperation.getCatalogName());
+ try {
+ catalog.alterDatabase(
+
alterDatabaseOperation.getDatabaseName(),
+
alterDatabaseOperation.getCatalogDatabase(),
+ false);
+ } catch (DatabaseNotExistException e) {
+ throw new ValidationException(exMsg, e);
+ } catch (Exception e) {
+ throw new TableException(exMsg, e);
+ }
+ } else if (operation instanceof UseOperation) {
Review comment:
just copy the content of `applyUseOperation` to here
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #10296: [FLINK-14691][table]Add use/create/drop/alter database operation and support it in flink/blink planner
KurtYoung commented on a change in pull request #10296: [FLINK-14691][table]Add
use/create/drop/alter database operation and support it in flink/blink planner
URL: https://github.com/apache/flink/pull/10296#discussion_r353029579
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/TableEnvironmentImpl.java
##
@@ -469,21 +479,84 @@ public void sqlUpdate(String stmt) {
createTableOperation.getCatalogTable(),
createTableOperation.getTableIdentifier(),
createTableOperation.isIgnoreIfExists());
+ } else if (operation instanceof CreateDatabaseOperation) {
+ CreateDatabaseOperation createDatabaseOperation =
(CreateDatabaseOperation) operation;
+ Catalog catalog =
getCatalogOrThrowException(createDatabaseOperation.getCatalogName());
+ String exMsg = getDDLOpExecuteErrorMsg("CREATE
DATABASE", createDatabaseOperation.getCatalogName());
+ try {
+ catalog.createDatabase(
+
createDatabaseOperation.getDatabaseName(),
+
createDatabaseOperation.getCatalogDatabase(),
+
createDatabaseOperation.isIgnoreIfExists());
+ } catch (DatabaseAlreadyExistException e) {
+ throw new ValidationException(exMsg, e);
+ } catch (Exception e) {
+ throw new TableException(exMsg, e);
+ }
} else if (operation instanceof DropTableOperation) {
DropTableOperation dropTableOperation =
(DropTableOperation) operation;
catalogManager.dropTable(
dropTableOperation.getTableIdentifier(),
dropTableOperation.isIfExists());
- } else if (operation instanceof UseCatalogOperation) {
+ } else if (operation instanceof DropDatabaseOperation) {
+ DropDatabaseOperation dropDatabaseOperation =
(DropDatabaseOperation) operation;
+ Catalog catalog =
getCatalogOrThrowException(dropDatabaseOperation.getCatalogName());
+ String exMsg = getDDLOpExecuteErrorMsg("DROP DATABASE",
dropDatabaseOperation.getCatalogName());
+ try {
+ catalog.dropDatabase(
+
dropDatabaseOperation.getDatabaseName(),
+
dropDatabaseOperation.isIfExists(),
+
dropDatabaseOperation.isCascade());
+ } catch (DatabaseNotExistException |
DatabaseNotEmptyException e) {
+ throw new ValidationException(exMsg, e);
+ } catch (Exception e) {
+ throw new TableException(exMsg, e);
+ }
+ } else if (operation instanceof AlterDatabaseOperation) {
+ AlterDatabaseOperation alterDatabaseOperation =
(AlterDatabaseOperation) operation;
+ Catalog catalog =
getCatalogOrThrowException(alterDatabaseOperation.getCatalogName());
+ String exMsg = getDDLOpExecuteErrorMsg("ALTER
DATABASE", alterDatabaseOperation.getCatalogName());
+ try {
+ catalog.alterDatabase(
+
alterDatabaseOperation.getDatabaseName(),
+
alterDatabaseOperation.getCatalogDatabase(),
+ false);
+ } catch (DatabaseNotExistException e) {
+ throw new ValidationException(exMsg, e);
+ } catch (Exception e) {
+ throw new TableException(exMsg, e);
+ }
+ } else if (operation instanceof UseOperation) {
+ applyUseOperation((UseOperation) operation);
+ } else {
+ throw new
TableException(UNSUPPORTED_QUERY_IN_SQL_UPDATE_MSG);
+ }
+ }
+
+ /**Apply use operation to current table environment. */
+ private void applyUseOperation(UseOperation operation) {
+ if (operation instanceof UseCatalogOperation) {
UseCatalogOperation useCatalogOperation =
(UseCatalogOperation) operation;
catalogManager.setCurrentCatalog(useCatalogOperation.getCatalogName());
+ } else if (operation instanceof UseDatabaseOperation) {
+
[GitHub] [flink] KurtYoung commented on a change in pull request #10296: [FLINK-14691][table]Add use/create/drop/alter database operation and support it in flink/blink planner
KurtYoung commented on a change in pull request #10296: [FLINK-14691][table]Add
use/create/drop/alter database operation and support it in flink/blink planner
URL: https://github.com/apache/flink/pull/10296#discussion_r353028953
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/TableEnvironmentImpl.java
##
@@ -469,21 +479,84 @@ public void sqlUpdate(String stmt) {
createTableOperation.getCatalogTable(),
createTableOperation.getTableIdentifier(),
createTableOperation.isIgnoreIfExists());
+ } else if (operation instanceof CreateDatabaseOperation) {
+ CreateDatabaseOperation createDatabaseOperation =
(CreateDatabaseOperation) operation;
+ Catalog catalog =
getCatalogOrThrowException(createDatabaseOperation.getCatalogName());
+ String exMsg = getDDLOpExecuteErrorMsg("CREATE
DATABASE", createDatabaseOperation.getCatalogName());
+ try {
+ catalog.createDatabase(
+
createDatabaseOperation.getDatabaseName(),
+
createDatabaseOperation.getCatalogDatabase(),
+
createDatabaseOperation.isIgnoreIfExists());
+ } catch (DatabaseAlreadyExistException e) {
+ throw new ValidationException(exMsg, e);
+ } catch (Exception e) {
+ throw new TableException(exMsg, e);
+ }
} else if (operation instanceof DropTableOperation) {
DropTableOperation dropTableOperation =
(DropTableOperation) operation;
catalogManager.dropTable(
dropTableOperation.getTableIdentifier(),
dropTableOperation.isIfExists());
- } else if (operation instanceof UseCatalogOperation) {
+ } else if (operation instanceof DropDatabaseOperation) {
+ DropDatabaseOperation dropDatabaseOperation =
(DropDatabaseOperation) operation;
+ Catalog catalog =
getCatalogOrThrowException(dropDatabaseOperation.getCatalogName());
+ String exMsg = getDDLOpExecuteErrorMsg("DROP DATABASE",
dropDatabaseOperation.getCatalogName());
+ try {
+ catalog.dropDatabase(
+
dropDatabaseOperation.getDatabaseName(),
+
dropDatabaseOperation.isIfExists(),
+
dropDatabaseOperation.isCascade());
+ } catch (DatabaseNotExistException |
DatabaseNotEmptyException e) {
+ throw new ValidationException(exMsg, e);
+ } catch (Exception e) {
+ throw new TableException(exMsg, e);
+ }
+ } else if (operation instanceof AlterDatabaseOperation) {
+ AlterDatabaseOperation alterDatabaseOperation =
(AlterDatabaseOperation) operation;
+ Catalog catalog =
getCatalogOrThrowException(alterDatabaseOperation.getCatalogName());
+ String exMsg = getDDLOpExecuteErrorMsg("ALTER
DATABASE", alterDatabaseOperation.getCatalogName());
Review comment:
ditto
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #10296: [FLINK-14691][table]Add use/create/drop/alter database operation and support it in flink/blink planner
KurtYoung commented on a change in pull request #10296: [FLINK-14691][table]Add use/create/drop/alter database operation and support it in flink/blink planner URL: https://github.com/apache/flink/pull/10296#discussion_r353023956 ## File path: flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/api/internal/TableEnvironmentImpl.java ## @@ -94,6 +102,10 @@ protected final FunctionCatalog functionCatalog; protected final Planner planner; protected final Parser parser; + private static final String UNSUPPORTED_QUERY_IN_SQL_UPDATE_MSG = + "Unsupported SQL query! sqlUpdate() only accepts a single SQL statement of type " + + "INSERT, CREATE TABLE, DROP TABLE, USE CATALOG, USE [catalog.]database, " + Review comment: nit: USE [CATALOG.]DATABASE This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] KurtYoung commented on a change in pull request #10296: [FLINK-14691][table]Add use/create/drop/alter database operation and support it in flink/blink planner
KurtYoung commented on a change in pull request #10296: [FLINK-14691][table]Add
use/create/drop/alter database operation and support it in flink/blink planner
URL: https://github.com/apache/flink/pull/10296#discussion_r353030027
##
File path:
flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/catalog/GenericInMemoryCatalog.java
##
@@ -116,6 +116,29 @@ public void dropDatabase(String databaseName, boolean
ignoreIfNotExists)
// Make sure the database is empty
if (isDatabaseEmpty(databaseName)) {
databases.remove(databaseName);
+ } else if (cascade) {
+ // delete all tables in this database and then
delete the database.
+ List deleteTablePaths =
tables.keySet().stream()
+
.filter(op ->
op.getDatabaseName().equals(databaseName))
Review comment:
nit: format doesn't seem right
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] azagrebin commented on issue #10161: [FLINK-13986][runtime] Clean up legacy code for FLIP-49.
azagrebin commented on issue #10161: [FLINK-13986][runtime] Clean up legacy
code for FLIP-49.
URL: https://github.com/apache/flink/pull/10161#issuecomment-561057717
Actually, I did not notice that we do not snapshot the env vars before the
test and restore them afterwards, like we do e.g. in `PluginConfigTest`:
```
private static Map oldEnvVariables;
@Before
public static void setup() {
oldEnvVariables = System.getenv();
}
@After
public void teardown() {
if (oldEnvVariables != null) {
CommonTestUtils.setEnv(oldEnvVariables, true);
}
}
```
This is a cleaner approach.
Could you add this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-15008) Tests in flink-yarn-tests fail with ClassNotFoundException (JDK11)
[
https://issues.apache.org/jira/browse/FLINK-15008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986713#comment-16986713
]
Yun Tang commented on FLINK-15008:
--
This is because the [removal of
{{java.activation}}|https://docs.oracle.com/en/java/javase/11/migrate/index.html#JSMIG-GUID-F640FA9D-FB66-4D85-AD2B-D931174C09A3]
in JDK11. We could manually add dependency below to fix this:
{code:java}
javax.activation
activation
1.1.1
{code}
> Tests in flink-yarn-tests fail with ClassNotFoundException (JDK11)
> --
>
> Key: FLINK-15008
> URL: https://issues.apache.org/jira/browse/FLINK-15008
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Tests
>Affects Versions: 1.10.0
>Reporter: Gary Yao
>Priority: Blocker
> Labels: jdk11
>
> {noformat}
> 1) Error injecting constructor, java.lang.NoClassDefFoundError:
> javax/activation/DataSource
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.JAXBContextResolver.(JAXBContextResolver.java:41)
> at
> org.apache.hadoop.yarn.server.resourcemanager.webapp.RMWebApp.setup(RMWebApp.java:51)
> while locating
> org.apache.hadoop.yarn.server.resourcemanager.webapp.JAXBContextResolver
> 1 error
> at com.google.inject.internal.InjectorImpl$4.get(InjectorImpl.java:987)
> at
> com.google.inject.internal.InjectorImpl.getInstance(InjectorImpl.java:1013)
> at
> com.sun.jersey.guice.spi.container.GuiceComponentProviderFactory$GuiceInstantiatedComponentProvider.getInstance(GuiceComponentProviderFactory.java:332)
> at
> com.sun.jersey.core.spi.component.ioc.IoCProviderFactory$ManagedSingleton.(IoCProviderFactory.java:179)
> at
> com.sun.jersey.core.spi.component.ioc.IoCProviderFactory.wrap(IoCProviderFactory.java:100)
> at
> com.sun.jersey.core.spi.component.ioc.IoCProviderFactory._getComponentProvider(IoCProviderFactory.java:93)
> at
> com.sun.jersey.core.spi.component.ProviderFactory.getComponentProvider(ProviderFactory.java:153)
> at
> com.sun.jersey.core.spi.component.ProviderServices.getComponent(ProviderServices.java:251)
> at
> com.sun.jersey.core.spi.component.ProviderServices.getProviders(ProviderServices.java:148)
> at
> com.sun.jersey.core.spi.factory.ContextResolverFactory.init(ContextResolverFactory.java:83)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl._initiate(WebApplicationImpl.java:1271)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.access$700(WebApplicationImpl.java:169)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:775)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl$13.f(WebApplicationImpl.java:771)
> at com.sun.jersey.spi.inject.Errors.processWithErrors(Errors.java:193)
> at
> com.sun.jersey.server.impl.application.WebApplicationImpl.initiate(WebApplicationImpl.java:771)
> at
> com.sun.jersey.guice.spi.container.servlet.GuiceContainer.initiate(GuiceContainer.java:121)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer$InternalWebComponent.initiate(ServletContainer.java:318)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.load(WebComponent.java:609)
> at
> com.sun.jersey.spi.container.servlet.WebComponent.init(WebComponent.java:210)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:373)
> at
> com.sun.jersey.spi.container.servlet.ServletContainer.init(ServletContainer.java:710)
> at
> com.google.inject.servlet.FilterDefinition.init(FilterDefinition.java:114)
> at
> com.google.inject.servlet.ManagedFilterPipeline.initPipeline(ManagedFilterPipeline.java:98)
> at com.google.inject.servlet.GuiceFilter.init(GuiceFilter.java:172)
> at org.mortbay.jetty.servlet.FilterHolder.doStart(FilterHolder.java:97)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at
> org.mortbay.jetty.servlet.ServletHandler.initialize(ServletHandler.java:713)
> at org.mortbay.jetty.servlet.Context.startContext(Context.java:140)
> at
> org.mortbay.jetty.webapp.WebAppContext.startContext(WebAppContext.java:1282)
> at
> org.mortbay.jetty.handler.ContextHandler.doStart(ContextHandler.java:518)
> at
> org.mortbay.jetty.webapp.WebAppContext.doStart(WebAppContext.java:499)
> at
> org.mortbay.component.AbstractLifeCycle.start(AbstractLifeCycle.java:50)
> at
> org.mortbay.jetty.handler.HandlerCollection.doStart(HandlerCollection.java:152)
> at
> org.mortbay.jetty.handler.ContextHandlerCollection.doStart(ContextHandlerCollection.java:156)
> at
> org.mortbay.component.AbstractLifeCycl
[GitHub] [flink] azagrebin edited a comment on issue #10161: [FLINK-13986][runtime] Clean up legacy code for FLIP-49.
azagrebin edited a comment on issue #10161: [FLINK-13986][runtime] Clean up
legacy code for FLIP-49.
URL: https://github.com/apache/flink/pull/10161#issuecomment-561057717
Actually, I did not notice that we do not snapshot the env vars before the
test and restore them afterwards, like we do e.g. in `PluginConfigTest`:
```
private static Map oldEnvVariables;
@Before
public static void setup() {
oldEnvVariables = System.getenv();
}
@After
public void teardown() {
if (oldEnvVariables != null) {
CommonTestUtils.setEnv(oldEnvVariables, true);
}
}
```
This is a cleaner approach, then we do not need to unset all vars at the end
of the test.
Could you add this?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (FLINK-14936) Introduce MemoryManager#computeMemorySize to calculate managed memory from a fraction
[ https://issues.apache.org/jira/browse/FLINK-14936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986720#comment-16986720 ] Andrey Zagrebin commented on FLINK-14936: - Ok, as we already have MemoryManager#computeNumberOfPages, I think we can quickly add MemoryManager#computeMemorySize for now and create a ticket to refactor both methods into LimittingMemoryManager later. > Introduce MemoryManager#computeMemorySize to calculate managed memory from a > fraction > - > > Key: FLINK-14936 > URL: https://issues.apache.org/jira/browse/FLINK-14936 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.10.0 >Reporter: Zhu Zhu >Priority: Major > > A MemoryManager#computeMemorySize(double fraction) is needed to calculate > managed memory bytes from a fraction. > It can be helpful for operators to get the memory size it can reserve and for > further #reserveMemory. (Similar to #computeNumberOfPages). > Here are two cases that may need this method in near future: > 1. [Python operator memory > management|https://lists.apache.org/thread.html/dd4dedeb9354c2ee559cd2f15629c719853915b5efb31a0eafee9361@%3Cdev.flink.apache.org%3E] > 2. [Statebackend memory > management|https://issues.apache.org/jira/browse/FLINK-14883] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15004) Choose two-phase Aggregate if the statistics is unknown
[
https://issues.apache.org/jira/browse/FLINK-15004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986719#comment-16986719
]
Kurt Young commented on FLINK-15004:
Is row count sufficient for us to decide whether we want to have one or two
phase aggregation? I think the key's ndv will be much more important here.
> Choose two-phase Aggregate if the statistics is unknown
> ---
>
> Key: FLINK-15004
> URL: https://issues.apache.org/jira/browse/FLINK-15004
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.9.1
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, blink planner will use default rowCount value (defined in
> {{FlinkPreparingTableBase#DEFAULT_ROWCOUNT}} ) when the statistics is
> unknown, and maybe choose one-phase Aggregate. The job will hang if the data
> is skewed. So It's better to use two-phase Aggregate for execution stability
> if the statistics is unknown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] xintongsong commented on issue #10161: [FLINK-13986][runtime] Clean up legacy code for FLIP-49.
xintongsong commented on issue #10161: [FLINK-13986][runtime] Clean up legacy code for FLIP-49. URL: https://github.com/apache/flink/pull/10161#issuecomment-561059932 @azagrebin, updated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zhuzhurk commented on issue #9986: [FLINK-10933][kubernetes] Implement KubernetesSessionClusterEntrypoint.
zhuzhurk commented on issue #9986: [FLINK-10933][kubernetes] Implement KubernetesSessionClusterEntrypoint. URL: https://github.com/apache/flink/pull/9986#issuecomment-561060222 The change needs to be based on #9984 to compile I think. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] godfreyhe commented on issue #9284: [FLINK-13502] move CatalogTableStatisticsConverter & TreeNode to correct package
godfreyhe commented on issue #9284: [FLINK-13502] move CatalogTableStatisticsConverter & TreeNode to correct package URL: https://github.com/apache/flink/pull/9284#issuecomment-561060689 @flinkbot run travis This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15032) Remove the eagerly serialization from `RemoteRpcInvocation`
[
https://issues.apache.org/jira/browse/FLINK-15032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986721#comment-16986721
]
Biao Liu commented on FLINK-15032:
--
Hi [~maguowei], thanks for the impressive proposal.
I have not fully understood the scenario. Here are some questions of mine.
{quote}Consider a job that has 1k parallelism and has a 1m union list state.
When deploying the 1k tasks, the eager serialization would use 1G memory
instantly(Some time the serialization amplifies the memory usage). However, the
serialized object is only used when the Akka sends the message. So we could
reduce the memory pressure if we only serialize the object when the message
would be sent by the Akka.{quote}
Do you mean we could postpone the serialization till
{{RemoteRpcInvocation#writeObject}}?
{quote}Furthermore, Akka would serialize the message at last and all the
XXXGateway related class could be visible by the RPC level. Because of that, I
think the serialization in the constructor of `RemoteRpcInvocation` could be
avoided.{quote}
Could you explain it a bit more? Do you mean there is no visible issue of
postponing the serialization? Or we don't need serialization anymore?
{quote}In summary, this Jira proposes to remove the serialization at the
constructor of `RemoteRpcInvocation`.{quote}
Do we still need serialization in {{wirteObject}} after removing serialization
in constructor?
> Remove the eagerly serialization from `RemoteRpcInvocation`
>
>
> Key: FLINK-15032
> URL: https://issues.apache.org/jira/browse/FLINK-15032
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Guowei Ma
>Priority: Major
>
> Currently, the constructor of `RemoteRpcInvocation` serializes the
> `parameterTypes` and `arg` of an RPC call. This could lead to two problems:
> # Consider a job that has 1k parallelism and has a 1m union list state. When
> deploying the 1k tasks, the eager serialization would use 1G memory
> instantly(Some time the serialization amplifies the memory usage). However,
> the serialized object is only used when the Akka sends the message. So we
> could reduce the memory pressure if we only serialize the object when the
> message would be sent by the Akka.
> # Furthermore, Akka would serialize the message at last and all the
> XXXGateway related class could be visible by the RPC level. Because of that,
> I think the serialization in the constructor of `RemoteRpcInvocation` could
> be avoided. I also do a simple test and find this could reduce the time cost
> of the RPC call. The 1k number of RPC calls with 1m `String` message: The
> current version costs around 2700ms; the Nonserialization version cost about
> 37ms.
>
> In summary, this Jira proposes to remove the serialization at the constructor
> of `RemoteRpcInvocation`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9284: [FLINK-13502] move CatalogTableStatisticsConverter & TreeNode to correct package
flinkbot edited a comment on issue #9284: [FLINK-13502] move CatalogTableStatisticsConverter & TreeNode to correct package URL: https://github.com/apache/flink/pull/9284#issuecomment-516662768 ## CI report: * fff36fd941c13764ad55fc85e4604ab64689aa21 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/121345562) * a7345a9c435a4afbac09b639764c316ec66e1d4e : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] kl0u commented on issue #10370: [FLINK-14947] Introduce LocalExecutor and make Local(Stream)Environment use it
kl0u commented on issue #10370: [FLINK-14947] Introduce LocalExecutor and make Local(Stream)Environment use it URL: https://github.com/apache/flink/pull/10370#issuecomment-561064160 Merged. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] kl0u closed pull request #10370: [FLINK-14947] Introduce LocalExecutor and make Local(Stream)Environment use it
kl0u closed pull request #10370: [FLINK-14947] Introduce LocalExecutor and make Local(Stream)Environment use it URL: https://github.com/apache/flink/pull/10370 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Closed] (FLINK-14947) Implement LocalExecutor as new Executor interface
[
https://issues.apache.org/jira/browse/FLINK-14947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kostas Kloudas closed FLINK-14947.
--
Resolution: Fixed
Merged with 41176ba0189ef2782449a1b7d4d074adc8a4af6f
> Implement LocalExecutor as new Executor interface
> -
>
> Key: FLINK-14947
> URL: https://issues.apache.org/jira/browse/FLINK-14947
> Project: Flink
> Issue Type: Sub-task
> Components: Client / Job Submission
>Reporter: Zili Chen
>Assignee: Kostas Kloudas
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We can replace {{PlanExecutor}} things with new Executor interface. One of
> this series is implement a {{LocalExecutor}} that execute pipeline within a
> {{MiniCluster}}. For proper lifecycle management I would wait for FLINK-14762
> & FLINK-14948 being merged.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10340: [FLINK-14978][table-api] Introduce constraint class hierarchy required for primary keys
flinkbot edited a comment on issue #10340: [FLINK-14978][table-api] Introduce constraint class hierarchy required for primary keys URL: https://github.com/apache/flink/pull/10340#issuecomment-559115958 ## CI report: * 40e3dce0c5ee3aa81dc7f773e3d6804e2de6f836 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138425542) * d29b368850d98b8586947633cc944a87f56a0686 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/139022683) * 7741e08586fd404dbcfbced7400310e9682398b8 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-15032) Remove the eagerly serialization from `RemoteRpcInvocation`
[
https://issues.apache.org/jira/browse/FLINK-15032?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986721#comment-16986721
]
Biao Liu edited comment on FLINK-15032 at 12/3/19 8:52 AM:
---
Hi [~maguowei], thanks for the impressive proposal.
Though I have not fully understood the scenario. Here are some questions of
mine.
{quote}Consider a job that has 1k parallelism and has a 1m union list state.
When deploying the 1k tasks, the eager serialization would use 1G memory
instantly(Some time the serialization amplifies the memory usage). However, the
serialized object is only used when the Akka sends the message. So we could
reduce the memory pressure if we only serialize the object when the message
would be sent by the Akka.
{quote}
Do you mean we could postpone the serialization till
{{RemoteRpcInvocation#writeObject}}?
{quote}Furthermore, Akka would serialize the message at last and all the
XXXGateway related class could be visible by the RPC level. Because of that, I
think the serialization in the constructor of `RemoteRpcInvocation` could be
avoided.
{quote}
Could you explain it a bit more? Do you mean there is no visible issue of
postponing the serialization? Or we don't need serialization anymore?
{quote}In summary, this Jira proposes to remove the serialization at the
constructor of `RemoteRpcInvocation`.
{quote}
Do we still need serialization in {{wirteObject}} after removing serialization
in constructor?
was (Author: sleepy):
Hi [~maguowei], thanks for the impressive proposal.
I have not fully understood the scenario. Here are some questions of mine.
{quote}Consider a job that has 1k parallelism and has a 1m union list state.
When deploying the 1k tasks, the eager serialization would use 1G memory
instantly(Some time the serialization amplifies the memory usage). However, the
serialized object is only used when the Akka sends the message. So we could
reduce the memory pressure if we only serialize the object when the message
would be sent by the Akka.{quote}
Do you mean we could postpone the serialization till
{{RemoteRpcInvocation#writeObject}}?
{quote}Furthermore, Akka would serialize the message at last and all the
XXXGateway related class could be visible by the RPC level. Because of that, I
think the serialization in the constructor of `RemoteRpcInvocation` could be
avoided.{quote}
Could you explain it a bit more? Do you mean there is no visible issue of
postponing the serialization? Or we don't need serialization anymore?
{quote}In summary, this Jira proposes to remove the serialization at the
constructor of `RemoteRpcInvocation`.{quote}
Do we still need serialization in {{wirteObject}} after removing serialization
in constructor?
> Remove the eagerly serialization from `RemoteRpcInvocation`
>
>
> Key: FLINK-15032
> URL: https://issues.apache.org/jira/browse/FLINK-15032
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Reporter: Guowei Ma
>Priority: Major
>
> Currently, the constructor of `RemoteRpcInvocation` serializes the
> `parameterTypes` and `arg` of an RPC call. This could lead to two problems:
> # Consider a job that has 1k parallelism and has a 1m union list state. When
> deploying the 1k tasks, the eager serialization would use 1G memory
> instantly(Some time the serialization amplifies the memory usage). However,
> the serialized object is only used when the Akka sends the message. So we
> could reduce the memory pressure if we only serialize the object when the
> message would be sent by the Akka.
> # Furthermore, Akka would serialize the message at last and all the
> XXXGateway related class could be visible by the RPC level. Because of that,
> I think the serialization in the constructor of `RemoteRpcInvocation` could
> be avoided. I also do a simple test and find this could reduce the time cost
> of the RPC call. The 1k number of RPC calls with 1m `String` message: The
> current version costs around 2700ms; the Nonserialization version cost about
> 37ms.
>
> In summary, this Jira proposes to remove the serialization at the constructor
> of `RemoteRpcInvocation`.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] TisonKun opened a new pull request #10392: [FLINK-14854][client] Add executeAsync() method to execution environments
TisonKun opened a new pull request #10392: [FLINK-14854][client] Add executeAsync() method to execution environments URL: https://github.com/apache/flink/pull/10392 ## What is the purpose of the change Add a new `executeAsync()` method which returns a future of `JobClient`. This exposes the new executor/job client work on the user API. I added such methods in `ExecutionEnvironment` & `StreamExecutionEnvironment` and export as Scala API. However, I don't patch code for make it works in every environment that hijacks `execution` method. The reason is that 1) future work will port (Local|Remote)Environment under new Executor abstraction so then they will work. 2) CollectionEnvironment doesn't support something like `JobClient`. Actually I think such environment can be replaced with LocalEnvironment. Currently, I do only check not null for `ExecutorFactory` in `executeAsync` and throw an exception with description when we cannot find one, i.e., Local, Remote, Collection Environment. It will hints users that these envs don't support such feature(now). ## Verifying this change `executeAsync()` is a lightweight interface for default parameter. And with this change all codepath previously goes into `execute(jobName)` now also goes into `executeAsync(jobName)`. Thus I think the change is already covered by existing tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes, we introduces more user APIs) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (yes) - If yes, how is the feature documented? (JavaDocs) cc @aljoscha @kl0u This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14854) Add executeAsync() method to execution environments
[
https://issues.apache.org/jira/browse/FLINK-14854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-14854:
---
Labels: pull-request-available (was: )
> Add executeAsync() method to execution environments
> ---
>
> Key: FLINK-14854
> URL: https://issues.apache.org/jira/browse/FLINK-14854
> Project: Flink
> Issue Type: Sub-task
> Components: API / DataSet, API / DataStream
>Reporter: Aljoscha Krettek
>Assignee: Zili Chen
>Priority: Major
> Labels: pull-request-available
>
> The new {{executeAsync()}} method should return a {{JobClient}}. This exposes
> the new executor/job client work on the user API.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] TisonKun commented on a change in pull request #10392: [FLINK-14854][client] Add executeAsync() method to execution environments
TisonKun commented on a change in pull request #10392: [FLINK-14854][client]
Add executeAsync() method to execution environments
URL: https://github.com/apache/flink/pull/10392#discussion_r353050400
##
File path:
flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/CollectionTestEnvironment.java
##
@@ -42,11 +42,6 @@ public JobExecutionResult getLastJobExecutionResult() {
}
}
- @Override
- public JobExecutionResult execute() throws Exception {
Review comment:
IMO `execute()` should be a thin interface for adding default parameter, so
we should mark it as `final` and this override makes non-sense.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10392: [FLINK-14854][client] Add executeAsync() method to execution environments
flinkbot commented on issue #10392: [FLINK-14854][client] Add executeAsync() method to execution environments URL: https://github.com/apache/flink/pull/10392#issuecomment-561065761 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 233f83b16fc6cdc8ceae87982d565d68480a25ca (Tue Dec 03 08:57:00 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10390: [FLINK-15004] [table-planner-blink] Choose two-phase Aggregate if the row count is unknown
flinkbot edited a comment on issue #10390: [FLINK-15004] [table-planner-blink] Choose two-phase Aggregate if the row count is unknown URL: https://github.com/apache/flink/pull/10390#issuecomment-560995750 ## CI report: * b954b01901d861cdb57d970ee7c27e659210e8df : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/139085348) * 68493d2af7629016c98ebad49a536f85de8247d9 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9965: [FLINK-10935][kubernetes]Implement KubeClient with Faric8 Kubernetes clients
flinkbot edited a comment on issue #9965: [FLINK-10935][kubernetes]Implement KubeClient with Faric8 Kubernetes clients URL: https://github.com/apache/flink/pull/9965#issuecomment-544813931 ## CI report: * 6f90b457e56a0a8cb45d63c1b05b47d2e38030a1 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132938440) * 86aa5ce8f77faf233c51a7231b3f71e518fd6c92 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132962300) * 7851d845a43f799627b2c788ace8eb7e6caccb03 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/133133283) * d49bb0a622e1667baffd29f19fdcc60d0022fe82 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133366914) * 0e88e0e5be77f450c82cbc460ea4f02a1effc920 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133998649) * 9afcc7ba840186c68f36b30d6b28b8c1cbf09b61 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/137145224) * 06e6b2bee4c1788b150f2b83c43eb4723709864b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138017584) * 816a11afe713e736cdfd2eb566762ee2addf7071 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138708643) * e80691082c9fc8ac704b0bccaa5180b7de0718d5 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138770607) * e85e76c018b7606381c5869e0c1054e02c4a2321 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138957695) * d1c41ae28960b3eea21fc662e771fe44c1c53b12 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/139101718) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9985: [FLINK-10968][kubernetes] Implement TaskManager Entrypoint for Kubernetes.
flinkbot edited a comment on issue #9985: [FLINK-10968][kubernetes] Implement TaskManager Entrypoint for Kubernetes. URL: https://github.com/apache/flink/pull/9985#issuecomment-545891561 ## CI report: * e74d80aaf05f631c7757483023657bee6f533e45 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133371125) * 31756d78d9b4e26b4c4a2f605216a4ce39ec847e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/133998569) * bb8213a916c182834d8f558dbb2e74aea6461779 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134010358) * 26c2e1fc03f752032ec67ea7ee008718c7875db7 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/137145291) * 1d9e7da8c6317253b27719c627e0a672156f4eea : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138017419) * ed20fe035c70880e9ab82e5465056c4542e69002 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138704970) * 0834ad81085a8851ef865111d9adbc0e840cc806 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/138984373) * c799f988facdff719c77b4c2e758e5751cd12c50 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/139079045) * 2c401c0e2190ba33a8578e1afa4923496e73c69b : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10017: [FLINK-14019][python] add support for managing environment and dependencies of Python UDF in Flink Python API
flinkbot edited a comment on issue #10017: [FLINK-14019][python] add support for managing environment and dependencies of Python UDF in Flink Python API URL: https://github.com/apache/flink/pull/10017#issuecomment-546947394 ## CI report: * c851f9713cef338dd135c3e982f138d68bfbe33d : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/133821281) * 85c34fa3fe7ec4b016438faf40358b66f043e37e : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/134189114) * 880095643c38982cd28d8558fcd0426c38e3cf67 : UNKNOWN * 362c69a059433aeed3c05426ba5e8aae96522263 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/134530748) * d59863bf8e1d95df1db00bfdd73aa1cb19caa8fa : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/135220068) * b79460aa27c8e1a10ccfa7de914d9362ac6ad7c8 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/136068705) * 0b1c1af9b44a42e3eb2c30969fc801970051f88e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/137379346) * 1ae7d82c2ef097d71a304e5a57be546c7129813b : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/137557615) * 6074499cf1c05b89af3b1e2525f21361414a330a : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/137561657) * e027f28287fd4d3e5b79f4dfe0782adf3f93f12c : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138165917) * 195f77371db5e159a68f9a6d376d3d8932b81988 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138189743) * 2c0d810025c8f59c441fbd5d6888f33dba3f2fbc : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138547277) * 3dfa8b0fce6780b30181289785c711057d4bb278 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138566115) * 45b6e92e1d0e054b462d763118bf2010955b2b3a : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138907643) * 37d0de7d71471aa327270df92d49c04f8c2f : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/138947327) * 7d00e0001b382e13afd32812dcb643b4c7029266 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138957717) * 5ef0379819c9c04eba6ecc478e4f0f2f040b4f7b : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/139079057) * 212806453d8276daa35fb8ac0fa7a1407c390af6 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/139095687) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Created] (FLINK-15033) Remove unused RemoteEnvirnment.executeRemotely() (FLINK-11048)
Kostas Kloudas created FLINK-15033: -- Summary: Remove unused RemoteEnvirnment.executeRemotely() (FLINK-11048) Key: FLINK-15033 URL: https://issues.apache.org/jira/browse/FLINK-15033 Project: Flink Issue Type: Sub-task Components: Client / Job Submission Affects Versions: 1.10.0 Reporter: Kostas Kloudas Assignee: Kostas Kloudas -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10161: [FLINK-13986][runtime] Clean up legacy code for FLIP-49.
flinkbot edited a comment on issue #10161: [FLINK-13986][runtime] Clean up legacy code for FLIP-49. URL: https://github.com/apache/flink/pull/10161#issuecomment-552882313 ## CI report: * 2c0501f41bea1da031777069dd46eb17c5ae8038 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/136122509) * a93fe47a7f1a91c8a33e7cac2bfc095e3f17012b : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/136270502) * 649d050fe4173a390df026156f6e9bae4f346360 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/136299400) * 08dafb5d2c6f38599bf86c06516465aeaa324941 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/136328000) * 8700267a462544e3d51aa40baa60ea07482305c5 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/136460282) * b90d7ea0e63b16b064f7e54e886202fff63a7516 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/136474353) * 0366a60deac3f1da4902e76a7879bd75996dc15b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/137744753) * 2d1421dc8595923aa04200a90dcc3a1eb2f9229e : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/137877709) * f778ef42223274946279a328552684b7c03e1d1b : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/138184361) * 298d6c299a83f2e2424fce5b54b25dc11b2fdcbc : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138768101) * d126350c6b1276105208247a472d54e7dc8b8dc3 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138907671) * 35bc04059fb8225e2cadf7f9fcd870e9bd2cd2ee : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/139003018) * feb744562f2b4796afac8518e2c3d65f61c705d3 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/139055297) * 3f52a34adc23caf2bd65c59e75684f3af8ea5604 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] TisonKun commented on a change in pull request #10392: [FLINK-14854][client] Add executeAsync() method to execution environments
TisonKun commented on a change in pull request #10392: [FLINK-14854][client]
Add executeAsync() method to execution environments
URL: https://github.com/apache/flink/pull/10392#discussion_r353055224
##
File path:
flink-test-utils-parent/flink-test-utils/src/main/java/org/apache/flink/test/util/CollectionTestEnvironment.java
##
@@ -42,11 +42,6 @@ public JobExecutionResult getLastJobExecutionResult() {
}
}
- @Override
- public JobExecutionResult execute() throws Exception {
Review comment:
hmmm mark method as `final` breaks b/w comp. but I think it can be removed
on the fly for these meaningless override.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[GitHub] [flink] dianfu commented on issue #10017: [FLINK-14019][python] add support for managing environment and dependencies of Python UDF in Flink Python API
dianfu commented on issue #10017: [FLINK-14019][python] add support for managing environment and dependencies of Python UDF in Flink Python API URL: https://github.com/apache/flink/pull/10017#issuecomment-561070492 @WeiZhong94 Thanks for the update. LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] carp84 commented on issue #8340: [FLINK-12395][table] Add more detailed javadoc for getDescription() and getDetailedDescription() in catalog object interfaces
carp84 commented on issue #8340: [FLINK-12395][table] Add more detailed javadoc for getDescription() and getDetailedDescription() in catalog object interfaces URL: https://github.com/apache/flink/pull/8340#issuecomment-561071168 @bowenli86 Since we are approaching feature freeze and branch cut for 1.10 release, mind give a quick update here about where we are now? The changes here seem to be all on javadoc thus easy to get merged if they correctly reflect the latest semantic? Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-12395) Add more detailed javadoc for getDescription() and getDetailedDescription() in catalog object interfaces
[
https://issues.apache.org/jira/browse/FLINK-12395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986731#comment-16986731
]
Yu Li commented on FLINK-12395:
---
Drop a comment in PR and simply a ping here to confirm whether we still target
at getting this in for 1.10.0 release [~phoenixjiangnan]? Thanks.
> Add more detailed javadoc for getDescription() and getDetailedDescription()
> in catalog object interfaces
>
>
> Key: FLINK-12395
> URL: https://issues.apache.org/jira/browse/FLINK-12395
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Bowen Li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> We need to add more javadoc for {{getDescritpion()}} and
> {{getDetailedDescription()}} APIs in CatalogBaseTable, including in which
> cases they will be called, and how we expect developers to implement them
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-14936) Introduce MemoryManager#computeMemorySize to calculate managed memory from a fraction
[ https://issues.apache.org/jira/browse/FLINK-14936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrey Zagrebin reassigned FLINK-14936: --- Assignee: Zhu Zhu > Introduce MemoryManager#computeMemorySize to calculate managed memory from a > fraction > - > > Key: FLINK-14936 > URL: https://issues.apache.org/jira/browse/FLINK-14936 > Project: Flink > Issue Type: Improvement > Components: Runtime / Coordination >Affects Versions: 1.10.0 >Reporter: Zhu Zhu >Assignee: Zhu Zhu >Priority: Major > > A MemoryManager#computeMemorySize(double fraction) is needed to calculate > managed memory bytes from a fraction. > It can be helpful for operators to get the memory size it can reserve and for > further #reserveMemory. (Similar to #computeNumberOfPages). > Here are two cases that may need this method in near future: > 1. [Python operator memory > management|https://lists.apache.org/thread.html/dd4dedeb9354c2ee559cd2f15629c719853915b5efb31a0eafee9361@%3Cdev.flink.apache.org%3E] > 2. [Statebackend memory > management|https://issues.apache.org/jira/browse/FLINK-14883] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] spoon-lz opened a new pull request #10393: [hotfix] Update string "chk-" to constant
spoon-lz opened a new pull request #10393: [hotfix] Update string "chk-" to constant URL: https://github.com/apache/flink/pull/10393 ## What is the purpose of the change Update string "chk-" to constant ## Brief change log N/A ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: N/A ## Documentation N/A This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10393: [hotfix] Update string "chk-" to constant
flinkbot commented on issue #10393: [hotfix] Update string "chk-" to constant URL: https://github.com/apache/flink/pull/10393#issuecomment-561073821 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit bc31a8ef2dcf05e175a4d47ab900a274756b461b (Tue Dec 03 09:17:53 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-12412) Allow ListTypeInfo used for java.util.List and MapTypeInfo used for java.util.Map
[
https://issues.apache.org/jira/browse/FLINK-12412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li updated FLINK-12412:
--
Fix Version/s: (was: 1.10.0)
1.11.0
[~yangfei] [~twalthr] Hi guys, I could see the linked PR already closed and no
real action making progress, so I will change the version to 1.11.0 and please
feel free to close it if you think nothing more to do here. Thanks.
> Allow ListTypeInfo used for java.util.List and MapTypeInfo used for
> java.util.Map
> -
>
> Key: FLINK-12412
> URL: https://issues.apache.org/jira/browse/FLINK-12412
> Project: Flink
> Issue Type: New Feature
> Components: API / Type Serialization System
>Affects Versions: 1.9.0
>Reporter: YangFei
>Assignee: YangFei
>Priority: Major
> Labels: pull-request-available, starer
> Fix For: 1.11.0
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
>
> {code:java}
> // code placeholder
> public static class UserBehavior {
> public long userId;
> public long itemId;
> public int categoryId;
> public long timestamp;
> public List comments;
> }
> public static void main(String[] args) throws Exception {
> PojoTypeInfo pojoType = (PojoTypeInfo)
> TypeExtractor.createTypeInfo(UserBehavior.class);
> }
> {code}
>
> The filed comments in UserBehavior will be extracted by TypeExtractor to
> GenericType .
> I think it can be extracted to ListTypeInfo .
> This would be a big improvement as in many cases classes including List or Map
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9284: [FLINK-13502] move CatalogTableStatisticsConverter & TreeNode to correct package
flinkbot edited a comment on issue #9284: [FLINK-13502] move CatalogTableStatisticsConverter & TreeNode to correct package URL: https://github.com/apache/flink/pull/9284#issuecomment-516662768 ## CI report: * fff36fd941c13764ad55fc85e4604ab64689aa21 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/121345562) * a7345a9c435a4afbac09b639764c316ec66e1d4e : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/139105430) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10340: [FLINK-14978][table-api] Introduce constraint class hierarchy required for primary keys
flinkbot edited a comment on issue #10340: [FLINK-14978][table-api] Introduce constraint class hierarchy required for primary keys URL: https://github.com/apache/flink/pull/10340#issuecomment-559115958 ## CI report: * 40e3dce0c5ee3aa81dc7f773e3d6804e2de6f836 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138425542) * d29b368850d98b8586947633cc944a87f56a0686 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/139022683) * 7741e08586fd404dbcfbced7400310e9682398b8 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/139105444) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10346: [FLINK-14972] Make Remote(Stream)Environment use Executors.
flinkbot edited a comment on issue #10346: [FLINK-14972] Make Remote(Stream)Environment use Executors. URL: https://github.com/apache/flink/pull/10346#issuecomment-559445838 ## CI report: * 9d133e6ba91f6d688401f493cc39729b4b34feb6 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/138552538) * 8ef4796fe5eefa1ba4a8c0411687f1594b84d637 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138557811) * 73d9858a69d860ca089ea776ddb9126d810c9792 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138672498) * de55ac4f29e099597a253c6312fbd089204a039d : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/138717503) * 269e359359c1253756b884ff3d18cbd0ffa46362 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138722990) * ae729f2d987d25e52c67e60d1f176f0fb9d821c9 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] wangyang0918 commented on issue #9986: [FLINK-10933][kubernetes] Implement KubernetesSessionClusterEntrypoint.
wangyang0918 commented on issue #9986: [FLINK-10933][kubernetes] Implement KubernetesSessionClusterEntrypoint. URL: https://github.com/apache/flink/pull/9986#issuecomment-561078620 @zhuzhurk Yes, your are right. This is PR depends on #9984. I will rebase and trigger the travis again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10392: [FLINK-14854][client] Add executeAsync() method to execution environments
flinkbot commented on issue #10392: [FLINK-14854][client] Add executeAsync() method to execution environments URL: https://github.com/apache/flink/pull/10392#issuecomment-561079535 ## CI report: * fd0d9488583ccee29304a015efb3a026000a : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #10393: [hotfix] Update string "chk-" to constant
flinkbot commented on issue #10393: [hotfix] Update string "chk-" to constant URL: https://github.com/apache/flink/pull/10393#issuecomment-561079588 ## CI report: * bc31a8ef2dcf05e175a4d47ab900a274756b461b : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10390: [FLINK-15004] [table-planner-blink] Choose two-phase Aggregate if the row count is unknown
flinkbot edited a comment on issue #10390: [FLINK-15004] [table-planner-blink] Choose two-phase Aggregate if the row count is unknown URL: https://github.com/apache/flink/pull/10390#issuecomment-560995750 ## CI report: * b954b01901d861cdb57d970ee7c27e659210e8df : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/139085348) * 68493d2af7629016c98ebad49a536f85de8247d9 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/139105480) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot commented on issue #8340: [FLINK-12395][table] Add more detailed javadoc for getDescription() and getDetailedDescription() in catalog object interfaces
flinkbot commented on issue #8340: [FLINK-12395][table] Add more detailed javadoc for getDescription() and getDetailedDescription() in catalog object interfaces URL: https://github.com/apache/flink/pull/8340#issuecomment-561079242 ## CI report: * 8a0e8d22ee908d22c6ec9474fde076c5fd7f206d : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15004) Choose two-phase Aggregate if the statistics is unknown
[
https://issues.apache.org/jira/browse/FLINK-15004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986742#comment-16986742
]
godfrey he commented on FLINK-15004:
[~ykt836], yes, `ndv` had been considered before: if `ndv` is unknown, the
planner will choose two-phase aggregate.
`DistinctRowCount` metadata handler could return null which means unknown,
while `RowCount` metadata handler always returns primitive type and planner
does not know whether the inputs have real row count or just use default value.
the issue mainly solves the following scenarios: `ndv` is known, while row
count is unknown.
> Choose two-phase Aggregate if the statistics is unknown
> ---
>
> Key: FLINK-15004
> URL: https://issues.apache.org/jira/browse/FLINK-15004
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.9.1
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, blink planner will use default rowCount value (defined in
> {{FlinkPreparingTableBase#DEFAULT_ROWCOUNT}} ) when the statistics is
> unknown, and maybe choose one-phase Aggregate. The job will hang if the data
> is skewed. So It's better to use two-phase Aggregate for execution stability
> if the statistics is unknown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-14378) Cleanup rocksDB lib folder if fail to load library
[
https://issues.apache.org/jira/browse/FLINK-14378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986741#comment-16986741
]
Nico Kruber commented on FLINK-14378:
-
I believe a proper cleanup should cover both scenarios and a fix for this one
probably also fixes the other issue. I'm closing FLINK-15011 as a duplicate.
Just to clarify here: we should also cleanup the {{rocksdb-lib-}}
directory upon graceful shutdown.
> Cleanup rocksDB lib folder if fail to load library
> --
>
> Key: FLINK-14378
> URL: https://issues.apache.org/jira/browse/FLINK-14378
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Reporter: Yun Tang
>Assignee: Yun Tang
>Priority: Major
>
> This improvement is inspired due to some of our machines need some time to
> load the rocksDB library. When some other unrecoverable exceptions continue
> to happen and the process to load library would be interrupted which cause
> the {{rocksdb-lib}} folder created but not cleaned up. As the job continues
> to failover, the {{rocksdb-lib}} folder would be created more and more. We
> even come across that machine was running out of inodes!
> Details could refer to current
> [implementation|https://github.com/apache/flink/blob/80b27a150026b7b5cb707bd9fa3e17f565bb8112/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBStateBackend.java#L860]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-15004) Choose two-phase Aggregate if the statistics is unknown
[
https://issues.apache.org/jira/browse/FLINK-15004?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986742#comment-16986742
]
godfrey he edited comment on FLINK-15004 at 12/3/19 9:34 AM:
-
[~ykt836], yes, `ndv` had been considered before: if `ndv` is unknown, the
planner will choose two-phase aggregate.
`DistinctRowCount` metadata handler could return null which means unknown,
while `RowCount` metadata handler always returns primitive type and planner
does not know whether the inputs have real row count or just use default value.
this issue mainly solves the following scenarios: `ndv` is known, while row
count is unknown.
was (Author: godfreyhe):
[~ykt836], yes, `ndv` had been considered before: if `ndv` is unknown, the
planner will choose two-phase aggregate.
`DistinctRowCount` metadata handler could return null which means unknown,
while `RowCount` metadata handler always returns primitive type and planner
does not know whether the inputs have real row count or just use default value.
the issue mainly solves the following scenarios: `ndv` is known, while row
count is unknown.
> Choose two-phase Aggregate if the statistics is unknown
> ---
>
> Key: FLINK-15004
> URL: https://issues.apache.org/jira/browse/FLINK-15004
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Affects Versions: 1.9.0, 1.9.1
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.10.0
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> Currently, blink planner will use default rowCount value (defined in
> {{FlinkPreparingTableBase#DEFAULT_ROWCOUNT}} ) when the statistics is
> unknown, and maybe choose one-phase Aggregate. The job will hang if the data
> is skewed. So It's better to use two-phase Aggregate for execution stability
> if the statistics is unknown.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-15011) RocksDB temp directory not cleaned up
[
https://issues.apache.org/jira/browse/FLINK-15011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nico Kruber closed FLINK-15011.
---
Resolution: Duplicate
> RocksDB temp directory not cleaned up
> -
>
> Key: FLINK-15011
> URL: https://issues.apache.org/jira/browse/FLINK-15011
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / State Backends
>Affects Versions: 1.9.1
>Reporter: Nico Kruber
>Priority: Major
>
> When starting a Flink cluster with 2 TMs, then starting a job with RocksDB
> with
> {code:java}
> state.backend: rocksdb {code}
> it will create temp directories {{rocksdb-lib-}} where it extracts the
> native libraries to. After shutting down the Flink cluster, these directories
> remain (but their contents are cleaned up at least).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #9965: [FLINK-10935][kubernetes]Implement KubeClient with Faric8 Kubernetes clients
flinkbot edited a comment on issue #9965: [FLINK-10935][kubernetes]Implement KubeClient with Faric8 Kubernetes clients URL: https://github.com/apache/flink/pull/9965#issuecomment-544813931 ## CI report: * 6f90b457e56a0a8cb45d63c1b05b47d2e38030a1 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132938440) * 86aa5ce8f77faf233c51a7231b3f71e518fd6c92 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/132962300) * 7851d845a43f799627b2c788ace8eb7e6caccb03 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/133133283) * d49bb0a622e1667baffd29f19fdcc60d0022fe82 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133366914) * 0e88e0e5be77f450c82cbc460ea4f02a1effc920 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133998649) * 9afcc7ba840186c68f36b30d6b28b8c1cbf09b61 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/137145224) * 06e6b2bee4c1788b150f2b83c43eb4723709864b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138017584) * 816a11afe713e736cdfd2eb566762ee2addf7071 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138708643) * e80691082c9fc8ac704b0bccaa5180b7de0718d5 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138770607) * e85e76c018b7606381c5869e0c1054e02c4a2321 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138957695) * d1c41ae28960b3eea21fc662e771fe44c1c53b12 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/139101718) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #9985: [FLINK-10968][kubernetes] Implement TaskManager Entrypoint for Kubernetes.
flinkbot edited a comment on issue #9985: [FLINK-10968][kubernetes] Implement TaskManager Entrypoint for Kubernetes. URL: https://github.com/apache/flink/pull/9985#issuecomment-545891561 ## CI report: * e74d80aaf05f631c7757483023657bee6f533e45 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/133371125) * 31756d78d9b4e26b4c4a2f605216a4ce39ec847e : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/133998569) * bb8213a916c182834d8f558dbb2e74aea6461779 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/134010358) * 26c2e1fc03f752032ec67ea7ee008718c7875db7 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/137145291) * 1d9e7da8c6317253b27719c627e0a672156f4eea : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138017419) * ed20fe035c70880e9ab82e5465056c4542e69002 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138704970) * 0834ad81085a8851ef865111d9adbc0e840cc806 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/138984373) * c799f988facdff719c77b4c2e758e5751cd12c50 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/139079045) * 2c401c0e2190ba33a8578e1afa4923496e73c69b : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/139105485) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-13502) CatalogTableStatisticsConverter & TreeNode should be in correct package
[
https://issues.apache.org/jira/browse/FLINK-13502?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
godfrey he updated FLINK-13502:
---
Fix Version/s: 1.10.0
> CatalogTableStatisticsConverter & TreeNode should be in correct package
> ---
>
> Key: FLINK-13502
> URL: https://issues.apache.org/jira/browse/FLINK-13502
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Critical
> Fix For: 1.10.0
>
>
> currently, {{CatalogTableStatisticsConverter}} is in
> {{org.apache.flink.table.util}}, {{TreeNode}} is in
> {{org.apache.flink.table.planner.plan}}. {{CatalogTableStatisticsConverter}}
> should be in {{org.apache.flink.table.planner.utils}}, {{TreeNode}} should be
> in {{org.apache.flink.table.planner.expressions}}.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-11937) Resolve small file problem in RocksDB incremental checkpoint
[ https://issues.apache.org/jira/browse/FLINK-11937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Congxian Qiu(klion26) updated FLINK-11937: -- Fix Version/s: (was: 1.10.0) 1.11.0 > Resolve small file problem in RocksDB incremental checkpoint > > > Key: FLINK-11937 > URL: https://issues.apache.org/jira/browse/FLINK-11937 > Project: Flink > Issue Type: New Feature > Components: Runtime / Checkpointing >Reporter: Congxian Qiu(klion26) >Assignee: Congxian Qiu(klion26) >Priority: Major > Labels: pull-request-available > Fix For: 1.11.0 > > Time Spent: 0.5h > Remaining Estimate: 0h > > Currently when incremental checkpoint is enabled in RocksDBStateBackend a > separate file will be generated on DFS for each sst file. This may cause > “file flood” when running intensive workload (many jobs with high > parallelism) in big cluster. According to our observation in Alibaba > production, such file flood introduces at lease two drawbacks when using HDFS > as the checkpoint storage FileSystem: 1) huge number of RPC request issued to > NN which may burst its response queue; 2) huge number of files causes big > pressure on NN’s on-heap memory. > In Flink we ever noticed similar small file flood problem and tried to > resolved it by introducing ByteStreamStateHandle(FLINK-2808), but this > solution has its limitation that if we configure the threshold too low there > will still be too many small files, while if too high the JM will finally > OOM, thus could hardly resolve the issue in case of using RocksDBStateBackend > with incremental snapshot strategy. > We propose a new OutputStream called > FileSegmentCheckpointStateOutputStream(FSCSOS) to fix the problem. FSCSOS > will reuse the same underlying distributed file until its size exceeds a > preset threshold. We > plan to complete the work in 3 steps: firstly introduce FSCSOS, secondly > resolve the specific storage amplification issue on FSCSOS, and lastly add an > option to reuse FSCSOS across multiple checkpoints to further reduce the DFS > file number. > More details please refer to the attached design doc. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-12427) Translate the "Flink DataStream API Programming Guide" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-12427?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li updated FLINK-12427: -- Fix Version/s: (was: 1.10.0) 1.11.0 Labels: (was: pull-request-available) Changing fix version to 1.11.0 since no progress here and assignee seems to be inactive. > Translate the "Flink DataStream API Programming Guide" page into Chinese > > > Key: FLINK-12427 > URL: https://issues.apache.org/jira/browse/FLINK-12427 > Project: Flink > Issue Type: Sub-task > Components: chinese-translation, Documentation >Affects Versions: 1.9.0 >Reporter: YangFei >Assignee: YangFei >Priority: Major > Fix For: 1.11.0 > > > [https://ci.apache.org/projects/flink/flink-docs-master/dev/datastream_api.html] > files locate /flink/docs/dev/datastream_api.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on issue #10161: [FLINK-13986][runtime] Clean up legacy code for FLIP-49.
flinkbot edited a comment on issue #10161: [FLINK-13986][runtime] Clean up legacy code for FLIP-49. URL: https://github.com/apache/flink/pull/10161#issuecomment-552882313 ## CI report: * 2c0501f41bea1da031777069dd46eb17c5ae8038 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/136122509) * a93fe47a7f1a91c8a33e7cac2bfc095e3f17012b : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/136270502) * 649d050fe4173a390df026156f6e9bae4f346360 : CANCELED [Build](https://travis-ci.com/flink-ci/flink/builds/136299400) * 08dafb5d2c6f38599bf86c06516465aeaa324941 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/136328000) * 8700267a462544e3d51aa40baa60ea07482305c5 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/136460282) * b90d7ea0e63b16b064f7e54e886202fff63a7516 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/136474353) * 0366a60deac3f1da4902e76a7879bd75996dc15b : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/137744753) * 2d1421dc8595923aa04200a90dcc3a1eb2f9229e : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/137877709) * f778ef42223274946279a328552684b7c03e1d1b : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/138184361) * 298d6c299a83f2e2424fce5b54b25dc11b2fdcbc : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138768101) * d126350c6b1276105208247a472d54e7dc8b8dc3 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138907671) * 35bc04059fb8225e2cadf7f9fcd870e9bd2cd2ee : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/139003018) * feb744562f2b4796afac8518e2c3d65f61c705d3 : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/139055297) * 3f52a34adc23caf2bd65c59e75684f3af8ea5604 : PENDING [Build](https://travis-ci.com/flink-ci/flink/builds/139105509) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Updated] (FLINK-14663) Distinguish unknown column stats and zero
[
https://issues.apache.org/jira/browse/FLINK-14663?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-14663:
---
Labels: pull-request-available (was: )
> Distinguish unknown column stats and zero
> -
>
> Key: FLINK-14663
> URL: https://issues.apache.org/jira/browse/FLINK-14663
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive, Table SQL / API
>Reporter: Kurt Young
>Priority: Major
> Labels: pull-request-available
>
> When converting from hive stats to flink's column stats, we didn't check
> whether some columns stats is really set or just an initial value. For
> example:
> {code:java}
> // code placeholder
> LongColumnStatsData longColStats = stats.getLongStats();
> return new CatalogColumnStatisticsDataLong(
> longColStats.getLowValue(),
> longColStats.getHighValue(),
> longColStats.getNumDVs(),
> longColStats.getNumNulls());
> {code}
> Hive `LongColumnStatsData` actually has information whether some stats is
> set through APIs like `isSetNumDVs()`. And the initial values are all 0, it
> will confuse us is it really 0 or just an initial value.
>
> We can use -1 to represent UNKNOWN value for column stats.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] zjuwangg opened a new pull request #10394: [FLINK-14663]Distinguish catalogColumnStats' unknown value and real values
zjuwangg opened a new pull request #10394: [FLINK-14663]Distinguish catalogColumnStats' unknown value and real values URL: https://github.com/apache/flink/pull/10394 ## What is the purpose of the change *When converting from hive stats to Flink's column stats, we didn't check whether some columns stats is really set or just an initial value. This PR we aim to change the CatalogColumnStatisticsDataBase and its subclass's stats from primitive types to corresponding boxed-type.* ## Brief change log - *[68e7752](https://github.com/apache/flink/commit/68e77528300eadcbeaa62372d676b01e268c39a8) Distinguish catalogColumnStats' unknown value and real values* ## Verifying this change This change is already covered by existing tests, such as *HiveCatalogHiveMetadataTest#testAlterTableColumnStatistics* and so on. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-15012) Checkpoint directory not cleaned up
[
https://issues.apache.org/jira/browse/FLINK-15012?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986760#comment-16986760
]
Nico Kruber commented on FLINK-15012:
-
Well, we do have a lot of temp directories that will be deleted with
{{stop-cluster.sh}}, e.g. blobStorage or flink-io.
However, the checkpoint directory may be special because it is shared between
the JobManager and the TaskManager processes. Even if the JobManager cleans
this up, some TaskManager could still be writing to it in case a checkpoint was
concurrently being created. I did not try, but I am a bit concerned whether
this may happen in a real cluster setup as well, for example in K8s where you
may kill the Flink cluster (along with all running jobs) through K8s. Since we
claim that the checkpoint lifecycle is managed by Flink, it should actually
always do the cleanup*
Looking at the code you linked for ZooKeeperCompletedCheckpointStore as well as
how StandaloneCompletedCheckpointStore implement their {{shutdown() }}method, I
am also wondering why they only clean up completed checkpoints. Shouldn't they
also clean up in-process checkpoints (if possible)?
* There may be some strings attached but then they would need to be documented
so that DevOps may account for that and eventually do a manual cleanup (if the
checkpoint path lets them identify what to delete).
> Checkpoint directory not cleaned up
> ---
>
> Key: FLINK-15012
> URL: https://issues.apache.org/jira/browse/FLINK-15012
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Checkpointing
>Affects Versions: 1.9.1
>Reporter: Nico Kruber
>Priority: Major
>
> I started a Flink cluster with 2 TMs using {{start-cluster.sh}} and the
> following config (in addition to the default {{flink-conf.yaml}})
> {code:java}
> state.checkpoints.dir: file:///path/to/checkpoints/
> state.backend: rocksdb {code}
> After submitting a jobwith checkpoints enabled (every 5s), checkpoints show
> up, e.g.
> {code:java}
> bb969f842bbc0ecc3b41b7fbe23b047b/
> ├── chk-2
> │ ├── 238969e1-6949-4b12-98e7-1411c186527c
> │ ├── 2702b226-9cfc-4327-979d-e5508ab2e3d5
> │ ├── 4c51cb24-6f71-4d20-9d4c-65ed6e826949
> │ ├── e706d574-c5b2-467a-8640-1885ca252e80
> │ └── _metadata
> ├── shared
> └── taskowned {code}
> If I shut down the cluster via {{stop-cluster.sh}}, these files will remain
> on disk and not be cleaned up.
> In contrast, if I cancel the job, at least {{chk-2}} will be deleted, but
> still leaving the (empty) directories.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13861) No new checkpoint will be trigged when canceling an expired checkpoint failed
[
https://issues.apache.org/jira/browse/FLINK-13861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Congxian Qiu(klion26) updated FLINK-13861:
--
Fix Version/s: (was: 1.10.0)
> No new checkpoint will be trigged when canceling an expired checkpoint failed
> -
>
> Key: FLINK-13861
> URL: https://issues.apache.org/jira/browse/FLINK-13861
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.7.2, 1.8.1, 1.9.0
>Reporter: Congxian Qiu(klion26)
>Assignee: Congxian Qiu(klion26)
>Priority: Major
>
> I encountered this problem in our private fork of Flink, after taking a look
> at the current master branch of Apache Flink, I think the problem exists here
> also.
> Problem Detail:
> 1. checkpoint canceled because of expiration, so will call the canceller
> such as below
> {code:java}
> final Runnable canceller = () -> {
>synchronized (lock) {
> // only do the work if the checkpoint is not discarded anyways
> // note that checkpoint completion discards the pending checkpoint
> object
> if (!checkpoint.isDiscarded()) {
> LOG.info("Checkpoint {} of job {} expired before completing.",
> checkpointID, job);
> failPendingCheckpoint(checkpoint,
> CheckpointFailureReason.CHECKPOINT_EXPIRED);
> pendingCheckpoints.remove(checkpointID);
> rememberRecentCheckpointId(checkpointID);
> triggerQueuedRequests();
> }
>}
> };{code}
>
> But failPendingCheckpoint may throw exceptions because it will call
> {{CheckpointCoordinator#failPendingCheckpoint}}
> -> {{PendingCheckpoint#abort}}
> -> {{PendingCheckpoint#reportFailedCheckpoint}}
> -> initialize a FailedCheckpointStates, may throw an exception by
> {{checkArgument}}
> Did not find more about why there ever failed the {{checkArgument
> currently(this problem did not reproduce frequently)}}, will create an issue
> for that if I have more findings.
>
> 2. when trigger checkpoint next, we'll first check if there already are too
> many checkpoints such as below
> {code:java}
> private void checkConcurrentCheckpoints() throws CheckpointException {
>if (pendingCheckpoints.size() >= maxConcurrentCheckpointAttempts) {
> triggerRequestQueued = true;
> if (currentPeriodicTrigger != null) {
> currentPeriodicTrigger.cancel(false);
> currentPeriodicTrigger = null;
> }
> throw new
> CheckpointException(CheckpointFailureReason.TOO_MANY_CONCURRENT_CHECKPOINTS);
>}
> }
> {code}
> the {{pendingCheckpoints.zie() >= maxConcurrentCheckpoitnAttempts}} will
> always true
> 3. no checkpoint will be triggered ever from that on.
> Because of the {{failPendingCheckpoint}} may throw Exception, so we may
> place the remove pending checkpoint logic in a finally block.
> I'd like to file a pr for this if this really needs to fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-13861) No new checkpoint will be trigged when canceling an expired checkpoint failed
[
https://issues.apache.org/jira/browse/FLINK-13861?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986763#comment-16986763
]
Congxian Qiu(klion26) commented on FLINK-13861:
---
[~pnowojski] [~SleePy] thanks for your reply. and sorry for my late reply(I
thought that I have replied after the off-line discuss with [~SleePy])
Yes, I'm interested in the root cause also, but we did not encounter it
anymore. After the off-line discuss with [~SleePy], we reach an agreement that
ignores the exception directly may not be a good solution, ignoring the
exception directly need we take attention to more and more corner cases, maybe
fail the whole job when encountering such a case is more reasonable.
As the 1.10 will code freeze soon and I have some other work on hands now, so I
just remove the fix version temporarily. and will come back soon :)
> No new checkpoint will be trigged when canceling an expired checkpoint failed
> -
>
> Key: FLINK-13861
> URL: https://issues.apache.org/jira/browse/FLINK-13861
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.7.2, 1.8.1, 1.9.0
>Reporter: Congxian Qiu(klion26)
>Assignee: Congxian Qiu(klion26)
>Priority: Major
> Fix For: 1.10.0
>
>
> I encountered this problem in our private fork of Flink, after taking a look
> at the current master branch of Apache Flink, I think the problem exists here
> also.
> Problem Detail:
> 1. checkpoint canceled because of expiration, so will call the canceller
> such as below
> {code:java}
> final Runnable canceller = () -> {
>synchronized (lock) {
> // only do the work if the checkpoint is not discarded anyways
> // note that checkpoint completion discards the pending checkpoint
> object
> if (!checkpoint.isDiscarded()) {
> LOG.info("Checkpoint {} of job {} expired before completing.",
> checkpointID, job);
> failPendingCheckpoint(checkpoint,
> CheckpointFailureReason.CHECKPOINT_EXPIRED);
> pendingCheckpoints.remove(checkpointID);
> rememberRecentCheckpointId(checkpointID);
> triggerQueuedRequests();
> }
>}
> };{code}
>
> But failPendingCheckpoint may throw exceptions because it will call
> {{CheckpointCoordinator#failPendingCheckpoint}}
> -> {{PendingCheckpoint#abort}}
> -> {{PendingCheckpoint#reportFailedCheckpoint}}
> -> initialize a FailedCheckpointStates, may throw an exception by
> {{checkArgument}}
> Did not find more about why there ever failed the {{checkArgument
> currently(this problem did not reproduce frequently)}}, will create an issue
> for that if I have more findings.
>
> 2. when trigger checkpoint next, we'll first check if there already are too
> many checkpoints such as below
> {code:java}
> private void checkConcurrentCheckpoints() throws CheckpointException {
>if (pendingCheckpoints.size() >= maxConcurrentCheckpointAttempts) {
> triggerRequestQueued = true;
> if (currentPeriodicTrigger != null) {
> currentPeriodicTrigger.cancel(false);
> currentPeriodicTrigger = null;
> }
> throw new
> CheckpointException(CheckpointFailureReason.TOO_MANY_CONCURRENT_CHECKPOINTS);
>}
> }
> {code}
> the {{pendingCheckpoints.zie() >= maxConcurrentCheckpoitnAttempts}} will
> always true
> 3. no checkpoint will be triggered ever from that on.
> Because of the {{failPendingCheckpoint}} may throw Exception, so we may
> place the remove pending checkpoint logic in a finally block.
> I'd like to file a pr for this if this really needs to fix.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on issue #10394: [FLINK-14663]Distinguish catalogColumnStats' unknown value and real values
flinkbot commented on issue #10394: [FLINK-14663]Distinguish catalogColumnStats' unknown value and real values URL: https://github.com/apache/flink/pull/10394#issuecomment-561089712 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 68e77528300eadcbeaa62372d676b01e268c39a8 (Tue Dec 03 09:54:53 UTC 2019) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-14663).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] zjuwangg commented on issue #10394: [FLINK-14663]Distinguish catalogColumnStats' unknown value and real values
zjuwangg commented on issue #10394: [FLINK-14663]Distinguish catalogColumnStats' unknown value and real values URL: https://github.com/apache/flink/pull/10394#issuecomment-561090470 cc @godfreyhe @KurtYoung @bowenli86 to have a review. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] jinglining commented on a change in pull request #10359: [FLINK-14813][metrics] Provide `isBackPressured` Task metric
jinglining commented on a change in pull request #10359: [FLINK-14813][metrics] Provide `isBackPressured` Task metric URL: https://github.com/apache/flink/pull/10359#discussion_r353079869 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java ## @@ -351,6 +353,7 @@ public Task( this.taskManagerConfig = Preconditions.checkNotNull(taskManagerConfig); this.metrics = metricGroup; + metricGroup.gauge(MetricNames.IS_BACKPRESSURED, new BackPressureGauge(this)); Review comment: > If you're going for a Gauge, why not just `metricGroup.gauge(MetricNames.IS_BACKPRESSURED, this::isBackPressured);`? > > Also, since this effectively leaks a reference to an object under construction, have you verified that it is safe to call `isBackpressured` without the constructor having finished? Under what circumstances would call `isBackpressured` without the constructor having finished？ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (FLINK-14566) Enable ResourceSpec to get/set whether managed memory is used
[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986766#comment-16986766
]
Andrey Zagrebin commented on FLINK-14566:
-
[~zhuzh]
I think you are right it raises a lot of questions and would need more time to
understand.
Now I tend to agree with the previous suggestion to expose method for
{{Transformation}}/{{StreamNode}} to set weights for managed memory. We should
just carefully handle the case of already set known spec and then trying to set
the weight for managed memory. I think we should not allow mixing weight and
known spec.
If we decide to add WeightedResourceSpec later we can adjust the weight setting
methods accordingly.
> Enable ResourceSpec to get/set whether managed memory is used
> -
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Critical
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> To achieve this, we need an interface to set/get whether an operator uses
> managed memory. It can be part of {{ResourceSpec}} or directly locates in the
> operator description classes ({{Transformation}}/{{StreamNode}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] jinglining commented on a change in pull request #10359: [FLINK-14813][metrics] Provide `isBackPressured` Task metric
jinglining commented on a change in pull request #10359: [FLINK-14813][metrics] Provide `isBackPressured` Task metric URL: https://github.com/apache/flink/pull/10359#discussion_r353079869 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java ## @@ -351,6 +353,7 @@ public Task( this.taskManagerConfig = Preconditions.checkNotNull(taskManagerConfig); this.metrics = metricGroup; + metricGroup.gauge(MetricNames.IS_BACKPRESSURED, new BackPressureGauge(this)); Review comment: > If you're going for a Gauge, why not just `metricGroup.gauge(MetricNames.IS_BACKPRESSURED, this::isBackPressured);`? > > Also, since this effectively leaks a reference to an object under construction, have you verified that it is safe to call `isBackpressured` without the constructor having finished? Under what circumstances would call `isBackpressured` without the constructor having finished？ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] flinkbot edited a comment on issue #10245: [FLINK-10936][kubernetes] Implement Kubernetes command line tools to support session cluster.
flinkbot edited a comment on issue #10245: [FLINK-10936][kubernetes] Implement Kubernetes command line tools to support session cluster. URL: https://github.com/apache/flink/pull/10245#issuecomment-555375223 ## CI report: * 71f8401be1cb214e56f94e486c495ef7c800cfc4 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/137142029) * 9a125a1b09b4c9f0d450d8913402373a7bb2595d : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/138017463) * 34a1a08081ba754f8fc0a5531dfded37c0492986 : UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Comment Edited] (FLINK-14566) Enable ResourceSpec to get/set whether managed memory is used
[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986766#comment-16986766
]
Andrey Zagrebin edited comment on FLINK-14566 at 12/3/19 10:00 AM:
---
[~zhuzh]
I think you are right it raises a lot of questions and would need more time to
understand.
Now I tend to agree with the previous suggestion to expose method for
{{Transformation}}/{{StreamNode}} to set weights for managed memory. We should
just carefully handle the case of already set known spec and then trying to set
the weight for managed memory. I think we should not allow mixing weight and
known spec. Also, I would not touch ResourceSpec now and keep the weight in
{{Transformation}}/{{StreamNode.}}
If we decide to add WeightedResourceSpec later we can adjust internally the
weight setting methods and ResourceSpec accordingly.
was (Author: azagrebin):
[~zhuzh]
I think you are right it raises a lot of questions and would need more time to
understand.
Now I tend to agree with the previous suggestion to expose method for
{{Transformation}}/{{StreamNode}} to set weights for managed memory. We should
just carefully handle the case of already set known spec and then trying to set
the weight for managed memory. I think we should not allow mixing weight and
known spec.
If we decide to add WeightedResourceSpec later we can adjust the weight setting
methods accordingly.
> Enable ResourceSpec to get/set whether managed memory is used
> -
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Priority: Critical
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> To achieve this, we need an interface to set/get whether an operator uses
> managed memory. It can be part of {{ResourceSpec}} or directly locates in the
> operator description classes ({{Transformation}}/{{StreamNode}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-14566) Enable ResourceSpec to get/set whether managed memory is used
[
https://issues.apache.org/jira/browse/FLINK-14566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Zagrebin reassigned FLINK-14566:
---
Assignee: Zhu Zhu
> Enable ResourceSpec to get/set whether managed memory is used
> -
>
> Key: FLINK-14566
> URL: https://issues.apache.org/jira/browse/FLINK-14566
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Coordination
>Affects Versions: 1.10.0
>Reporter: Zhu Zhu
>Assignee: Zhu Zhu
>Priority: Critical
> Fix For: 1.10.0
>
>
> To calculate managed memory fraction for an operator with UNKNOWN resources,
> we need to know whether the operator will use managed memory, according to
> FLINK-14062.
> To achieve this, we need an interface to set/get whether an operator uses
> managed memory. It can be part of {{ResourceSpec}} or directly locates in the
> operator description classes ({{Transformation}}/{{StreamNode}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] jinglining commented on a change in pull request #10359: [FLINK-14813][metrics] Provide `isBackPressured` Task metric
jinglining commented on a change in pull request #10359: [FLINK-14813][metrics] Provide `isBackPressured` Task metric URL: https://github.com/apache/flink/pull/10359#discussion_r353079869 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskmanager/Task.java ## @@ -351,6 +353,7 @@ public Task( this.taskManagerConfig = Preconditions.checkNotNull(taskManagerConfig); this.metrics = metricGroup; + metricGroup.gauge(MetricNames.IS_BACKPRESSURED, new BackPressureGauge(this)); Review comment: > If you're going for a Gauge, why not just `metricGroup.gauge(MetricNames.IS_BACKPRESSURED, this::isBackPressured);`? > > Also, since this effectively leaks a reference to an object under construction, have you verified that it is safe to call `isBackpressured` without the constructor having finished? Under what circumstances would call `isBackpressured` without the constructor having finished？ If has this circumstance, maybe register it in org.apache.flink.runtime.taskexecutor.TaskExecutor#submitTask will be better. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [flink] rkhachatryan commented on a change in pull request #10345: [FLINK-12484][runtime] synchronize all mailbox actions
rkhachatryan commented on a change in pull request #10345:
[FLINK-12484][runtime] synchronize all mailbox actions
URL: https://github.com/apache/flink/pull/10345#discussion_r353081981
##
File path:
flink-streaming-java/src/test/java/org/apache/flink/streaming/runtime/tasks/StreamTaskExecutionDecorationTest.java
##
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.runtime.tasks;
+
+import org.apache.flink.runtime.checkpoint.CheckpointMetaData;
+import org.apache.flink.runtime.checkpoint.CheckpointOptions;
+import org.apache.flink.runtime.checkpoint.CheckpointType;
+import org.apache.flink.runtime.state.CheckpointStorageLocationReference;
+import org.apache.flink.runtime.util.FatalExitExceptionHandler;
+import org.apache.flink.streaming.api.operators.StreamOperator;
+import org.apache.flink.streaming.runtime.tasks.mailbox.Mail;
+import org.apache.flink.streaming.runtime.tasks.mailbox.MailboxDefaultAction;
+import org.apache.flink.util.function.ThrowingRunnable;
+
+import org.junit.After;
+import org.junit.Assert;
+import org.junit.Before;
+import org.junit.Test;
+
+import java.util.concurrent.Callable;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+/**
+ * Verifies that {@link StreamTask} {@link StreamTaskActionExecutor decorates
execution} of actions that potentially needs to be synchronized.
+ */
+public class StreamTaskExecutionDecorationTest {
+ private CountingStreamTaskActionExecutor decorator;
+ private StreamTask> task;
+
+ @Test
+ public void testAbortCheckpointOnBarrierIsDecorated() throws Exception {
+ task.abortCheckpointOnBarrier(1, null);
+ verify();
+ }
+
+ @Test
+ public void testTriggerCheckpointOnBarrierIsDecorated() throws
Exception {
+ task.triggerCheckpointOnBarrier(new CheckpointMetaData(1, 2),
new CheckpointOptions(CheckpointType.CHECKPOINT, new
CheckpointStorageLocationReference(new byte[]{1})), null);
+ verify();
+ }
+
+ @Test
+ public void testTriggerCheckpointAsyncIsDecorated() throws Exception {
+ task.triggerCheckpointAsync(new CheckpointMetaData(1, 2), new
CheckpointOptions(CheckpointType.CHECKPOINT, new
CheckpointStorageLocationReference(new byte[]{1})), false);
+ new Thread(() -> {
+ try {
+ Thread.sleep(50);
+ } catch (InterruptedException ex) {
+ ex.printStackTrace();
+ } finally {
+ task.mailboxProcessor.allActionsCompleted();
+ }
+ }).start();
+ task.mailboxProcessor.runMailboxLoop();
+ verify();
+ }
+
+ @Test
+ public void testMailboxExecutorIsDecorated() throws Exception {
+ CountDownLatch latch = new CountDownLatch(1);
+
task.mailboxProcessor.getMainMailboxExecutor().asExecutor("test").execute(() ->
{
+ try {
+ verify();
+ } finally {
+ latch.countDown();
+ }
+ });
+ new Thread(() -> {
+ try {
+ latch.await();
+ task.mailboxProcessor.allActionsCompleted();
+ } catch (Exception ex) {
+ ex.printStackTrace();
+ }
+ }).start();
+ task.mailboxProcessor.runMailboxLoop();
+ }
+
+ @Before
+ public void before() {
+ decorator = new CountingStreamTaskActionExecutor();
+ task = new StreamTask>(new
StreamTaskTest.DeclineDummyEnvironment(), null,
FatalExitExceptionHandler.INSTANCE, decorator) {
+ @Override
+ protected void init() {
+ }
+
+ @Override
+ protected void
processInput(MailboxDefaultAction.C
[GitHub] [flink] flinkbot edited a comment on issue #9284: [FLINK-13502] move CatalogTableStatisticsConverter & TreeNode to correct package
flinkbot edited a comment on issue #9284: [FLINK-13502] move CatalogTableStatisticsConverter & TreeNode to correct package URL: https://github.com/apache/flink/pull/9284#issuecomment-516662768 ## CI report: * fff36fd941c13764ad55fc85e4604ab64689aa21 : FAILURE [Build](https://travis-ci.com/flink-ci/flink/builds/121345562) * a7345a9c435a4afbac09b639764c316ec66e1d4e : SUCCESS [Build](https://travis-ci.com/flink-ci/flink/builds/139105430) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
