[GitHub] [flink] flinkbot commented on pull request #12350: [FLINK-17824][e2e] Introduce timeout to 'resume savepoint' test
flinkbot commented on pull request #12350: URL: https://github.com/apache/flink/pull/12350#issuecomment-634468809 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 5a37c0125f808a477da831a4a1a5acbaec92133e (Wed May 27 07:00:01 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #12264: [FLINK-17558][netty] Release partitions asynchronously
tillrohrmann commented on a change in pull request #12264: URL: https://github.com/apache/flink/pull/12264#discussion_r430897263 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/taskexecutor/TaskManagerServices.java ## @@ -265,10 +265,15 @@ public static TaskManagerServices fromConfiguration( // start the I/O manager, it will create some temp directories. final IOManager ioManager = new IOManagerAsync(taskManagerServicesConfiguration.getTmpDirPaths()); + final ExecutorService ioExecutor = Executors.newFixedThreadPool( Review comment: I think it is actually a good idea to move it away from the `RpcService's` executor because this is the same pool which is used to run the `RpcEndpoints`. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on a change in pull request #12264: [FLINK-17558][netty] Release partitions asynchronously
tillrohrmann commented on a change in pull request #12264:
URL: https://github.com/apache/flink/pull/12264#discussion_r430897493
##
File path:
flink-test-utils-parent/flink-test-utils-junit/src/main/java/org/apache/flink/testutils/executor/TestExecutorServiceResource.java
##
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.testutils.executor;
+
+import org.junit.rules.ExternalResource;
+
+import java.util.concurrent.ExecutorService;
+import java.util.function.Supplier;
+
+/**
+ * Resource which starts an {@link ExecutorService} for testing purposes.
+ */
+public class TestExecutorServiceResource extends ExternalResource {
Review comment:
Can we unify them and get rid of one of them?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] tillrohrmann closed pull request #12347: [FLINK-17750][tests] Harden YARNHighavailabilityITCase by increasing ZK session timeout to 20s
tillrohrmann closed pull request #12347: URL: https://github.com/apache/flink/pull/12347 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12334: [FLINK-17934][FLINK-17955][fs-connector] StreamingFileWriter should set chainingStrategy & BucketLifeCycleListener should just in Buc
flinkbot edited a comment on pull request #12334: URL: https://github.com/apache/flink/pull/12334#issuecomment-633879796 ## CI report: * e3f49ff243a1ef02b1fe3caeb4001e42d2c38636 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2242) * 96752af4aabbb367eb4150477dc1b835dbf303dd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2244) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann closed pull request #12341: [FLINK-17938] Move YarnTestBase.findFile and YarnTestBase.RootDirFilenameFilter to o.a.f.y.u.TestUtils
tillrohrmann closed pull request #12341: URL: https://github.com/apache/flink/pull/12341 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17404) Running HA per-job cluster (rocks, non-incremental) gets stuck killing a non-existing pid
[
https://issues.apache.org/jira/browse/FLINK-17404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117450#comment-17117450
]
Robert Metzger commented on FLINK-17404:
Note to self: The test logs: "Config uses deprecated configuration key
'high-availability.zookeeper.storageDir' instead of proper key
'high-availability.storageDir'"
> Running HA per-job cluster (rocks, non-incremental) gets stuck killing a
> non-existing pid
> -
>
> Key: FLINK-17404
> URL: https://issues.apache.org/jira/browse/FLINK-17404
> Project: Flink
> Issue Type: Bug
> Components: Test Infrastructure, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Critical
> Labels: test-stability
> Attachments: 255
>
>
> CI log: https://api.travis-ci.org/v3/job/678609505/log.txt
> {code}
> Waiting for text Completed checkpoint [1-9]* for job
> to appear 2 of times in logs...
> grep:
> /home/travis/build/apache/flink/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT/log/*standalonejob-2*.log:
> No such file or directory
> grep:
> /home/travis/build/apache/flink/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT/log/*standalonejob-2*.log:
> No such file or directory
> Starting standalonejob daemon on host
> travis-job-e606668f-b674-49c0-8590-e3508e22b99d.
> grep:
> /home/travis/build/apache/flink/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT/log/*standalonejob-2*.log:
> No such file or directory
> grep:
> /home/travis/build/apache/flink/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT/log/*standalonejob-2*.log:
> No such file or directory
> Killed TM @ 18864
> kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or
> kill -l [sigspec]
> Killed TM @
> No output has been received in the last 10m0s, this potentially indicates a
> stalled build or something wrong with the build itself.
> Check the details on how to adjust your build configuration on:
> https://docs.travis-ci.com/user/common-build-problems/#build-times-out-because-no-output-was-received
> The build has been terminated
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] klion26 commented on a change in pull request #12313: [FLINK-17005][docs] Translate the CREATE TABLE ... LIKE syntax documentation to Chinese
klion26 commented on a change in pull request #12313:
URL: https://github.com/apache/flink/pull/12313#discussion_r430865901
##
File path: docs/dev/table/sql/create.zh.md
##
@@ -208,14 +215,14 @@ CREATE TABLE Orders (
**注意：** 使用 `CREATE TABLE` 语句注册的表均可用作 table source 和 table sink。 在被 DML
语句引用前，我们无法决定其实际用于 source 抑或是 sink。
-**LIKE clause**
+**LIKE**
+
+`LIKE` 子句来源于两种 SQL 特性的变体/组合（Feature T171，“表定义中的LIKE语法” 和 Feature
T173，“表定义中的LIKE语法扩展”）。LIKE 子句可以基于现有表的定义去创建新表，并且可以扩展或排除原始表中的某些部分。与 SQL 标准相反，LIKE
子句必须在 CREATE 语句中定义。这是因为 LIKE 子句可以应用于表定义的多个部分，而不仅仅是 schema 部分。
Review comment:
```suggestion
`LIKE` 子句来源于两种 SQL 特性的变体/组合（Feature T171，“表定义中的 LIKE 语法” 和 Feature
T173，“表定义中的 LIKE 语法扩展”）。LIKE 子句可以基于现有表的定义去创建新表，并且可以扩展或排除原始表中的某些部分。与 SQL
标准相反，LIKE 子句必须在 CREATE 语句中定义。这是因为 LIKE 子句可以应用于表定义的多个部分，而不仅仅是 schema 部分。
```
另外这里的实现细节我不是太清楚，但是从字面上看 “at the top-level of a CREATE statement" 这里翻译成”必须在
CREATE 语句中定义“，我比较好奇这里说的 ”top-level“ 是指什么意思呢？
##
File path: docs/dev/table/sql/create.zh.md
##
@@ -208,14 +215,14 @@ CREATE TABLE Orders (
**注意：** 使用 `CREATE TABLE` 语句注册的表均可用作 table source 和 table sink。 在被 DML
语句引用前，我们无法决定其实际用于 source 抑或是 sink。
-**LIKE clause**
+**LIKE**
Review comment:
这里单纯的用 `LIKE` 会觉的不太完整
##
File path: docs/dev/table/sql/create.zh.md
##
@@ -249,59 +257,60 @@ CREATE TABLE Orders_with_watermark (
);
{% endhighlight %}
-The merging logic of table features can be controlled with `like options`.
+表属性的合并逻辑可以用 `like options` 来控制。
-You can control the merging behavior of:
+可以控制合并的表属性如下：
-* CONSTRAINTS - constraints such as primary and unique keys
-* GENERATED - computed columns
-* OPTIONS - connector options that describe connector and format properties
-* PARTITIONS - partition of the tables
-* WATERMARKS - watermark declarations
+* CONSTRAINTS - 主键和唯一键约束
+* GENERATED - 计算列
+* OPTIONS - 连接器和格式化的配置项
+* PARTITIONS - 表分区信息
+* WATERMARKS - watermark 定义
-with three different merging strategies:
+并且有三种不同的表属性合并策略：
-* INCLUDING - Includes the feature of the source table, fails on duplicate
entries, e.g. if an option with the same key exists in both tables.
-* EXCLUDING - Does not include the given feature of the source table.
-* OVERWRITING - Includes the feature of the source table, overwrites duplicate
entries of the source table with properties of the new table, e.g. if an option
with the same key exists in both tables, the one from the current statement
will be used.
+* INCLUDING - 新表包含源表（source table）所有的表属性，如果和源表的表属性重复则会直接失败，例如新表和源表存在相同 key 的属性。
+* EXCLUDING - 新表不包含源表指定的任何表属性。
+* OVERWRITING - 新表包含源表的表属性，但如果出现重复项，则会用新表的表属性覆盖源表中的重复表属性，例如，两个表中都存在相同 key
的属性，则会使用当前语句中定义的 key 的属性值。
-Additionally, you can use the `INCLUDING/EXCLUDING ALL` option to specify what
should be the strategy if there was no specific strategy defined, i.e. if you
use `EXCLUDING ALL INCLUDING WATERMARKS` only the watermarks will be included
from the source table.
+此外，如果没有特别指定合并策略，也可以使用 `INCLUDING/EXCLUDING ALL` 这种声明方式来指定使用怎样的合并策略，例如使用
`EXCLUDING ALL INCLUDING WATERMARKS`，那么代表只有源表的 WATERMARKS 属性才会被包含进新表。
-Example:
+示例如下：
{% highlight sql %}
--- A source table stored in a filesystem
+
+-- 存储在文件系统的源表
CREATE TABLE Orders_in_file (
user BIGINT,
product STRING,
order_time_string STRING,
order_time AS to_timestamp(order_time)
-
+
)
PARTITIONED BY user
WITH (
'connector' = 'filesystem'
'path' = '...'
);
--- A corresponding table we want to store in kafka
+-- 对应存储在 kafka 的源表
CREATE TABLE Orders_in_kafka (
--- Add watermark definition
+-- 添加 watermark 定义
WATERMARK FOR order_time AS order_time - INTERVAL '5' SECOND
) WITH (
'connector': 'kafka'
...
)
LIKE Orders_in_file (
--- Exclude everything besides the computed columns which we need to
generate the watermark for.
--- We do not want to have the partitions or filesystem options as those do
not apply to kafka.
+-- 排除需要生成 watermark 的计算列之外的所有内容。
+-- 去除不适用于 kafka 的所有分区和文件系统的相关属性。
EXCLUDING ALL
INCLUDING GENERATED
);
{% endhighlight %}
-If you provide no like options, `INCLUDING ALL OVERWRITING OPTIONS` will be
used as a default.
+如果未提供 like 配置项（like options），默认将使用 `INCLUDING ALL OVERWRITING OPTIONS` 的合并策略。
-**NOTE** You cannot control the behavior of merging physical fields. Those
will be merged as if you applied the `INCLUDING` strategy.
+**注意：** 您无法选择物理列的合并策略，当物理列进行合并时就如使用了 `INCLUDING` 策略。
Review comment:
这个有个小疑问，`physical fields` 就是指物理列吗？这个是和 逻辑列 对应的吗？另外在网上搜索发现 “物理列”这个描述好少
##
File path: docs/dev/table/sql/create.zh.md
##
@@ -249,59 +257,60 @@ CREATE TABLE Orders_with_watermark (
);
{% endhighlight %}
-The merging logic of table features can be controlled with `like options`.
+表属性的合并逻辑可以用 `like options` 来控制。
-You can control the merging behavior of:
+可以控制合并的表属性如下：
-* CONSTRAINTS - constraints such as primary and unique keys
-* GENERATED - computed columns
-* OPTION
[GitHub] [flink] flinkbot commented on pull request #12349: [FLINK-17722][python][build system] (followups) Keeps all jars of lib directory in CachedFiles
flinkbot commented on pull request #12349: URL: https://github.com/apache/flink/pull/12349#issuecomment-634476258 ## CI report: * 9a36906676bd0d2307a340c5cdee233c264111d0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11752: [FLINK-14713][rest]Show All Attempts For Vertex SubTask In Rest Api
flinkbot edited a comment on pull request #11752: URL: https://github.com/apache/flink/pull/11752#issuecomment-613965719 ## CI report: * ed1e68d9afea99af66fe8d3b9ad4aa181cab4907 UNKNOWN * b38b0a4ebb41da904231c886d6c52e79cec36b0e UNKNOWN * 2c61ddf0397a81249caab0de78b05e25ecc21fb4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=1256) * 832118b39c5fc3a7f04f785cbe97f274b24527ad Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2246) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12349: [FLINK-17722][python][build system] (followups) Keeps all jars of lib directory in CachedFiles
flinkbot edited a comment on pull request #12349: URL: https://github.com/apache/flink/pull/12349#issuecomment-634476258 ## CI report: * 9a36906676bd0d2307a340c5cdee233c264111d0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2248) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12350: [FLINK-17824][e2e] Introduce timeout to 'resume savepoint' test
flinkbot commented on pull request #12350: URL: https://github.com/apache/flink/pull/12350#issuecomment-634485040 ## CI report: * 5a37c0125f808a477da831a4a1a5acbaec92133e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17470) Flink task executor process permanently hangs on `flink-daemon.sh stop`, deletes PID file
[
https://issues.apache.org/jira/browse/FLINK-17470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117471#comment-17117471
]
Hunter Herman commented on FLINK-17470:
---
We have observed the same problem with 1.9 on the same system setup/java
version, however far less frequently. Flink 1.10 experiences this issue around
1 in 3 or 4 restarts, while flink 1.9 seems more like 1 in 15 or so.
> Flink task executor process permanently hangs on `flink-daemon.sh stop`,
> deletes PID file
> -
>
> Key: FLINK-17470
> URL: https://issues.apache.org/jira/browse/FLINK-17470
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.10.0
> Environment:
> {code:java}
> $ uname -a
> Linux hostname.local 3.10.0-1062.9.1.el7.x86_64 #1 SMP Fri Dec 6 15:49:49 UTC
> 2019 x86_64 x86_64 x86_64 GNU/Linux
> $ lsb_release -a
> LSB Version: :core-4.1-amd64:core-4.1-noarch
> Distributor ID: CentOS
> Description: CentOS Linux release 7.7.1908 (Core)
> Release: 7.7.1908
> Codename: Core
> {code}
> Flink version 1.10
>
>Reporter: Hunter Herman
>Priority: Major
> Attachments: flink_jstack.log, flink_mixed_jstack.log
>
>
> Hi Flink team!
> We've attempted to upgrade our flink 1.9 cluster to 1.10, but are
> experiencing reproducible instability on shutdown. Speciically, it appears
> that the `kill` issued in the `stop` case of flink-daemon.sh is causing the
> task executor process to hang permanently. Specifically, the process seems to
> be hanging in the
> `org.apache.flink.runtime.util.JvmShutdownSafeguard$DelayedTerminator.run` in
> a `Thread.sleep()` call. I think this is a bizarre behavior. Also note that
> every thread in the process is BLOCKED. on a `pthread_cond_wait` call. Is
> this an OS level issue? Banging my head on a wall here. See attached stack
> traces for details.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] kl0u commented on a change in pull request #12339: [FLINK-17744] StreamContextEnvironment#execute cannot be call JobListener#onJobExecuted
kl0u commented on a change in pull request #12339:
URL: https://github.com/apache/flink/pull/12339#discussion_r430918068
##
File path:
flink-clients/src/main/java/org/apache/flink/client/program/StreamContextEnvironment.java
##
@@ -71,31 +74,44 @@ public JobExecutionResult execute(StreamGraph streamGraph)
throws Exception {
JobClient jobClient = executeAsync(streamGraph);
JobExecutionResult jobExecutionResult;
- if (getConfiguration().getBoolean(DeploymentOptions.ATTACHED)) {
- CompletableFuture
jobExecutionResultFuture =
+ List jobListeners = getJobListeners();
Review comment:
I would suggest to put all the "old" `execute` in another method,
something like `getJobExecutionResult()` and then, the actual `execute` can
contain only the `try {} catch {}` logic. This will look sth like:
```
private JobExecutionResult getJobExecutionResult(final JobClient jobClient)
throws Exception {
checkNotNull(jobClient);
if (getConfiguration().getBoolean(DeploymentOptions.ATTACHED)) {
CompletableFuture
jobExecutionResultFuture =
jobClient.getJobExecutionResult(getUserClassloader());
if
(getConfiguration().getBoolean(DeploymentOptions.SHUTDOWN_IF_ATTACHED)) {
Thread shutdownHook =
ShutdownHookUtil.addShutdownHook(
() -> {
// wait a smidgen to
allow the async request to go through before
// the jvm exits
jobClient.cancel().get(1, TimeUnit.SECONDS);
},
StreamContextEnvironment.class.getSimpleName(),
LOG);
jobExecutionResultFuture.whenComplete((ignored,
throwable) ->
ShutdownHookUtil.removeShutdownHook(shutdownHook,
StreamContextEnvironment.class.getSimpleName(), LOG));
}
JobExecutionResult jobExecutionResult =
jobExecutionResultFuture.get();
System.out.println(jobExecutionResult);
return jobExecutionResult;
} else {
return new
DetachedJobExecutionResult(jobClient.getJobID());
}
}
@Override
public JobExecutionResult execute(StreamGraph streamGraph) throws
Exception {
JobClient jobClient = executeAsync(streamGraph);
List jobListeners = getJobListeners();
try {
final JobExecutionResult jobExecutionResult =
getJobExecutionResult(jobClient);
jobListeners.forEach(jobListener ->
jobListener.onJobExecuted(jobExecutionResult, null));
return jobExecutionResult;
} catch (Throwable t) {
jobListeners.forEach(jobListener -> {
jobListener.onJobExecuted(null,
ExceptionUtils.stripExecutionException(t));
});
ExceptionUtils.rethrowException(t);
// never reached, only make javac happy
return null;
}
}
```
Please check that this is correct because I have not verified it.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12350: [FLINK-17824][e2e] Introduce timeout to 'resume savepoint' test
flinkbot edited a comment on pull request #12350: URL: https://github.com/apache/flink/pull/12350#issuecomment-634485040 ## CI report: * 5a37c0125f808a477da831a4a1a5acbaec92133e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2249) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17935) Logs could not show up when deploying Flink on Yarn via "--executor"
[
https://issues.apache.org/jira/browse/FLINK-17935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117485#comment-17117485
]
Kostas Kloudas commented on FLINK-17935:
My only concern with moving the configuration to the {{YarnClusterDescriptor}}
itself is that this will tie it to an implicit contract (having to set an
environment variable) which is hard to maintain. Also for its testability, it
would be nice if we could just pass a complete configuration to the descriptor
itself and not set fields inside there (as much as possible).
> Logs could not show up when deploying Flink on Yarn via "--executor"
>
>
> Key: FLINK-17935
> URL: https://issues.apache.org/jira/browse/FLINK-17935
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Yang Wang
>Assignee: Kostas Kloudas
>Priority: Critical
> Fix For: 1.11.0
>
>
> {code:java}
> ./bin/flink run -d -p 5 -e yarn-per-job
> examples/streaming/WindowJoin.jar{code}
> When we use the {{-e/--executor}} to specify the deploy target to Yarn
> per-job, the logs could not show up. The root cause is we do not set the
> logging files in {{ExecutorCLI}}. We only do it in the
> {{FlinkYarnSessionCli}}.
> If we use {{-m yarn-cluster}}, everything works well.
>
> Maybe we should move the {{setLogConfigFileInConfig}} to
> {{YarnClusterDescriptor}} to avoid this problem. cc [~kkl0u]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] rkhachatryan opened a new pull request #12351: [FLINK-17820][task][checkpointing] Backport to 1.11: Respect fileSizeThreshold in FsCheckpointStateOutputStream.flush()
rkhachatryan opened a new pull request #12351: URL: https://github.com/apache/flink/pull/12351 __This is a backport of #12332 to `1.11`__ ## What is the purpose of the change See FLINK-17820 for full problem description. In short, Flink ignores `state.backend.fs.memory-threshold` and always creates a file per subtask per checkpoint. This PR makes `FsCheckpointStateOutputStream.flush()` to take `fileStateThreshold` into account. ## Verifying this change This change added tests: - `FsCheckpointStreamFactoryTest.testFlushUnderThreshold` - `FsCheckpointStreamFactoryTest.testFlushAboveThreshold` - `ChannelStateCheckpointWriterTest.testSmallFilesNotWritten` ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12351: [FLINK-17820][task][checkpointing] Backport to 1.11: Respect fileSizeThreshold in FsCheckpointStateOutputStream.flush()
flinkbot commented on pull request #12351: URL: https://github.com/apache/flink/pull/12351#issuecomment-634496976 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit a57f600a677c9b0e7431c107c8652a71b83d029f (Wed May 27 08:00:50 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17907) flink-table-api-java: Compilation failure
[ https://issues.apache.org/jira/browse/FLINK-17907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117491#comment-17117491 ] Aihua Li commented on FLINK-17907: -- when i update the jdk version to" Java(TM) SE Runtime Environment (build 1.8.0_131-b11)", this failure is disappeared. [~jark] could you help me to close this bug? thanks > flink-table-api-java: Compilation failure > - > > Key: FLINK-17907 > URL: https://issues.apache.org/jira/browse/FLINK-17907 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.11.0 > Environment: local env >Reporter: Aihua Li >Priority: Blocker > Fix For: 1.11.0 > > > When i execute the command "mvn clean install -B -U -DskipTests > -Dcheckstyle.skip=true -Drat.ignoreErrors -Dmaven.javadoc.skip " in branch > "master" and "release\-1.11" to install flink in my local env, i meet this > failure: > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile > (default-compile) on project flink-table-api-java: Compilation failure > [ERROR] > flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/utils/AggregateOperationFactory.java:[550,53] > unreported exception X; must be caught or declared to be thrown > [ERROR] -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e > switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, please > read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the > command > [ERROR] mvn -rf :flink-table-api-java -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-17463) BlobCacheCleanupTest.testPermanentBlobCleanup:133->verifyJobCleanup:432 » FileAlreadyExists
[
https://issues.apache.org/jira/browse/FLINK-17463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Gary Yao reassigned FLINK-17463:
Assignee: Gary Yao
> BlobCacheCleanupTest.testPermanentBlobCleanup:133->verifyJobCleanup:432 »
> FileAlreadyExists
> ---
>
> Key: FLINK-17463
> URL: https://issues.apache.org/jira/browse/FLINK-17463
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Assignee: Gary Yao
>Priority: Critical
> Labels: test-stability
> Fix For: 1.11.0
>
>
> CI run:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=317&view=logs&j=0da23115-68bb-5dcd-192c-bd4c8adebde1&t=4ed44b66-cdd6-5dcf-5f6a-88b07dda665d
> {code}
> [ERROR] Tests run: 5, Failures: 0, Errors: 1, Skipped: 1, Time elapsed: 2.73
> s <<< FAILURE! - in org.apache.flink.runtime.blob.BlobCacheCleanupTest
> [ERROR]
> testPermanentBlobCleanup(org.apache.flink.runtime.blob.BlobCacheCleanupTest)
> Time elapsed: 2.028 s <<< ERROR!
> java.nio.file.FileAlreadyExistsException:
> /tmp/junit7984674749832216773/junit1629420330972938723/blobStore-296d1a51-8917-4db1-a920-5d4e17e6fa36/job_3bafac5425979b4fe2fa2c7726f8dd5b
> at
> sun.nio.fs.UnixException.translateToIOException(UnixException.java:88)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
> at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
> at
> sun.nio.fs.UnixFileSystemProvider.createDirectory(UnixFileSystemProvider.java:384)
> at java.nio.file.Files.createDirectory(Files.java:674)
> at java.nio.file.Files.createAndCheckIsDirectory(Files.java:781)
> at java.nio.file.Files.createDirectories(Files.java:727)
> at
> org.apache.flink.runtime.blob.BlobUtils.getStorageLocation(BlobUtils.java:196)
> at
> org.apache.flink.runtime.blob.PermanentBlobCache.getStorageLocation(PermanentBlobCache.java:222)
> at
> org.apache.flink.runtime.blob.BlobServerCleanupTest.checkFilesExist(BlobServerCleanupTest.java:213)
> at
> org.apache.flink.runtime.blob.BlobCacheCleanupTest.verifyJobCleanup(BlobCacheCleanupTest.java:432)
> at
> org.apache.flink.runtime.blob.BlobCacheCleanupTest.testPermanentBlobCleanup(BlobCacheCleanupTest.java:133)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-17957) Forbidden syntax "CREATE SYSTEM FUNCTION" for sql parser
Danny Chen created FLINK-17957: -- Summary: Forbidden syntax "CREATE SYSTEM FUNCTION" for sql parser Key: FLINK-17957 URL: https://issues.apache.org/jira/browse/FLINK-17957 Project: Flink Issue Type: Bug Components: Table SQL / API Affects Versions: 1.11.0 Reporter: Danny Chen Fix For: 1.12.0 This syntax is invalid, but the parser still works. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] danny0405 opened a new pull request #12352: [FLINK-17717][sql-parser] Throws for DDL create temporary system func…
danny0405 opened a new pull request #12352: URL: https://github.com/apache/flink/pull/12352 …tion with composite table path Temporary system function with composite identifier is never supported. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-17907) flink-table-api-java: Compilation failure
[ https://issues.apache.org/jira/browse/FLINK-17907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-17907. Fix Version/s: (was: 1.11.0) Resolution: Duplicate > flink-table-api-java: Compilation failure > - > > Key: FLINK-17907 > URL: https://issues.apache.org/jira/browse/FLINK-17907 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.11.0 > Environment: local env >Reporter: Aihua Li >Priority: Blocker > > When i execute the command "mvn clean install -B -U -DskipTests > -Dcheckstyle.skip=true -Drat.ignoreErrors -Dmaven.javadoc.skip " in branch > "master" and "release\-1.11" to install flink in my local env, i meet this > failure: > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile > (default-compile) on project flink-table-api-java: Compilation failure > [ERROR] > flink-table/flink-table-api-java/src/main/java/org/apache/flink/table/operations/utils/AggregateOperationFactory.java:[550,53] > unreported exception X; must be caught or declared to be thrown > [ERROR] -> [Help 1] > [ERROR] > [ERROR] To see the full stack trace of the errors, re-run Maven with the -e > switch. > [ERROR] Re-run Maven using the -X switch to enable full debug logging. > [ERROR] > [ERROR] For more information about the errors and possible solutions, please > read the following articles: > [ERROR] [Help 1] > http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException > [ERROR] > [ERROR] After correcting the problems, you can resume the build with the > command > [ERROR] mvn -rf :flink-table-api-java -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] JingsongLi commented on a change in pull request #12323: [FLINK-17925][fs-connector] Fix Filesystem options to default values and types
JingsongLi commented on a change in pull request #12323:
URL: https://github.com/apache/flink/pull/12323#discussion_r430936699
##
File path:
flink-table/flink-table-runtime-blink/src/main/java/org/apache/flink/table/filesystem/FileSystemOptions.java
##
@@ -132,12 +133,12 @@
public static final ConfigOption SINK_PARTITION_COMMIT_TRIGGER =
key("sink.partition-commit.trigger")
.stringType()
- .defaultValue("partition-time")
+ .defaultValue("process-time")
.withDescription("Trigger type for
partition commit:" +
Review comment:
After some offline discussion, I think `partition-time` is more direct.
Adjust the comments further.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-17717) Throws for DDL create temporary system function with composite table path
[
https://issues.apache.org/jira/browse/FLINK-17717?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-17717:
---
Labels: pull-request-available (was: )
> Throws for DDL create temporary system function with composite table path
> -
>
> Key: FLINK-17717
> URL: https://issues.apache.org/jira/browse/FLINK-17717
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.11.0
>Reporter: Danny Chen
>Assignee: Danny Chen
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> Currently, we support syntax
> {code:sql}
> create temporary system function catalog.db.func_name as function_class
> {code}
> But actually we drop the catalog and db silently, the temporary system
> function never has custom table paths, it belongs always to the system and
> current session, so, we should limit the table path to simple identifier.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #12352: [FLINK-17717][sql-parser] Throws for DDL create temporary system func…
flinkbot commented on pull request #12352: URL: https://github.com/apache/flink/pull/12352#issuecomment-634506407 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit d66997401817c7cd66ce77b6bcfa48401e6c163b (Wed May 27 08:19:24 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-17958) Kubernetes session constantly allocates taskmanagers after cancel a job
Yang Wang created FLINK-17958:
-
Summary: Kubernetes session constantly allocates taskmanagers
after cancel a job
Key: FLINK-17958
URL: https://issues.apache.org/jira/browse/FLINK-17958
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.11.0, 1.12.0
Reporter: Yang Wang
Fix For: 1.11.0
When i am testing the {{kubernetes-session.sh}}, i find that the
{{KubernetesResourceManager}} will constantly allocate taskmanager after cancel
a job. I think it may be caused by a bug of the following code. When the
{{dividend}} is 0 and {{divisor}} is bigger than 1, the return value will be 1.
However, we expect it to be 0.
{code:java}
/**
* Divide and rounding up to integer.
* E.g., divideRoundUp(3, 2) returns 2.
* @param dividend value to be divided by the divisor
* @param divisor value by which the dividend is to be divided
* @return the quotient rounding up to integer
*/
public static int divideRoundUp(int dividend, int divisor) {
return (dividend - 1) / divisor + 1;
}{code}
How to reproduce this issue?
# Start a Kubernetes session
# Submit a Flink job to the existing session
# Cancel the job and wait for the TaskManager released via idle timeout
# More and more TaskManagers will be allocated
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12323: [FLINK-17925][fs-connector] Fix Filesystem options to default values and types
flinkbot edited a comment on pull request #12323: URL: https://github.com/apache/flink/pull/12323#issuecomment-633592742 ## CI report: * 5183374cf406535be6ba4c8a4de100e88a0d658c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2243) * 0619f577f658e1d912e5625d71a672f0b0889196 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12351: [FLINK-17820][task][checkpointing] Backport to 1.11: Respect fileSizeThreshold in FsCheckpointStateOutputStream.flush()
flinkbot commented on pull request #12351: URL: https://github.com/apache/flink/pull/12351#issuecomment-634508548 ## CI report: * a57f600a677c9b0e7431c107c8652a71b83d029f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on pull request #12334: [FLINK-17934][FLINK-17955][fs-connector] StreamingFileWriter should set chainingStrategy & BucketLifeCycleListener should just in Buckets
JingsongLi commented on pull request #12334: URL: https://github.com/apache/flink/pull/12334#issuecomment-634510340 Thanks @sjwiesman @gaoyunhaii for review, merging... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17260) StreamingKafkaITCase failure on Azure
[
https://issues.apache.org/jira/browse/FLINK-17260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117502#comment-17117502
]
Jingsong Lee commented on FLINK-17260:
--
another instance:
[https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_apis/build/builds/2190/logs/133]
> StreamingKafkaITCase failure on Azure
> -
>
> Key: FLINK-17260
> URL: https://issues.apache.org/jira/browse/FLINK-17260
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Roman Khachatryan
>Priority: Blocker
> Labels: test-stability
> Fix For: 1.11.0
>
>
> [https://dev.azure.com/rmetzger/5bd3ef0a-4359-41af-abca-811b04098d2e/_apis/build/builds/7544/logs/165]
>
> {code:java}
> 2020-04-16T00:12:32.2848429Z [INFO] Running
> org.apache.flink.tests.util.kafka.StreamingKafkaITCase
> 2020-04-16T00:14:47.9100927Z [ERROR] Tests run: 3, Failures: 1, Errors: 0,
> Skipped: 0, Time elapsed: 135.621 s <<< FAILURE! - in
> org.apache.flink.tests.util.k afka.StreamingKafkaITCase
> 2020-04-16T00:14:47.9103036Z [ERROR] testKafka[0:
> kafka-version:0.10.2.0](org.apache.flink.tests.util.kafka.StreamingKafkaITCase)
> Time elapsed: 46.222 s <<< FAILURE!
> 2020-04-16T00:14:47.9104033Z java.lang.AssertionError:
> expected:<[elephant,27,64213]> but was:<[]>
> 2020-04-16T00:14:47.9104638Zat org.junit.Assert.fail(Assert.java:88)
> 2020-04-16T00:14:47.9105148Zat
> org.junit.Assert.failNotEquals(Assert.java:834)
> 2020-04-16T00:14:47.9105701Zat
> org.junit.Assert.assertEquals(Assert.java:118)
> 2020-04-16T00:14:47.9106239Zat
> org.junit.Assert.assertEquals(Assert.java:144)
> 2020-04-16T00:14:47.9107177Zat
> org.apache.flink.tests.util.kafka.StreamingKafkaITCase.testKafka(StreamingKafkaITCase.java:162)
> 2020-04-16T00:14:47.9107845Zat
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-04-16T00:14:47.9108434Zat
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-04-16T00:14:47.9109318Zat
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-04-16T00:14:47.9109914Zat
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-04-16T00:14:47.9110434Zat
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-04-16T00:14:47.9110985Zat
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-04-16T00:14:47.9111548Zat
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-04-16T00:14:47.9112083Zat
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-04-16T00:14:47.9112629Zat
> org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2020-04-16T00:14:47.9113145Zat
> org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:48)
> 2020-04-16T00:14:47.9113637Zat
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> 2020-04-16T00:14:47.9114072Zat
> org.junit.rules.RunRules.evaluate(RunRules.java:20)
> 2020-04-16T00:14:47.9114490Zat
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-04-16T00:14:47.9115256Zat
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-04-16T00:14:47.9115791Zat
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-04-16T00:14:47.9116292Zat
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-04-16T00:14:47.9116736Zat
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-04-16T00:14:47.9117779Zat
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-04-16T00:14:47.9118274Zat
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-04-16T00:14:47.9118766Zat
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-04-16T00:14:47.9119204Zat
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-04-16T00:14:47.9119625Zat
> org.junit.runners.Suite.runChild(Suite.java:128)
> 2020-04-16T00:14:47.9120005Zat
> org.junit.runners.Suite.runChild(Suite.java:27)
> 2020-04-16T00:14:47.9120428Zat
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-04-16T00:14:47.9120876Zat
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-04-16T00:14:47.9121350Zat
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-04-16T00:14:47.9121805Zat
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-04-16T00:14:47.9122273Zat
> org.junit.runners.Pa
[GitHub] [flink] JingsongLi commented on pull request #12334: [FLINK-17934][FLINK-17955][fs-connector] StreamingFileWriter should set chainingStrategy & BucketLifeCycleListener should just in Buckets
JingsongLi commented on pull request #12334: URL: https://github.com/apache/flink/pull/12334#issuecomment-634510197 Test case should be: https://issues.apache.org/jira/browse/FLINK-17260 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingsongLi merged pull request #12334: [FLINK-17934][FLINK-17955][fs-connector] StreamingFileWriter should set chainingStrategy & BucketLifeCycleListener should just in Buckets
JingsongLi merged pull request #12334: URL: https://github.com/apache/flink/pull/12334 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] AHeise opened a new pull request #12353: [FLINK-17322][network] Fixes BroadcastRecordWriter overwriting memory segments on first finished BufferConsumer.
AHeise opened a new pull request #12353: URL: https://github.com/apache/flink/pull/12353 ## What is the purpose of the change `BroadcastRecordWriter#randomEmit` initialized buffer consumers for other non-target channels incorrectly leading to separate buffer reference counting and subsequently released buffers too early. ## Brief change log - Fixes `BroadcastRecordWriter#randomEmit`. ## Verifying this change This change added tests and can be verified as follows: - Added unit test in `BroadcastRecordWriterTest` - Added integration tests for latency markers and broadcast ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-17934) StreamingFileWriter should set chainingStrategy
[
https://issues.apache.org/jira/browse/FLINK-17934?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-17934.
Resolution: Fixed
master: cdfb0304c9e982795fa4c839559ca0283db9b424
release-1.11: 83c0eab3675814cbfb46670a4c4411351bfcd183
> StreamingFileWriter should set chainingStrategy
> ---
>
> Key: FLINK-17934
> URL: https://issues.apache.org/jira/browse/FLINK-17934
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Reporter: Jingsong Lee
>Assignee: Jingsong Lee
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> {{StreamingFileWriter}} should be eagerly chained whenever possible.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17322) Enable latency tracker would corrupt the broadcast state
[
https://issues.apache.org/jira/browse/FLINK-17322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-17322:
---
Labels: pull-request-available (was: )
> Enable latency tracker would corrupt the broadcast state
>
>
> Key: FLINK-17322
> URL: https://issues.apache.org/jira/browse/FLINK-17322
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network
>Affects Versions: 1.9.3, 1.10.1
>Reporter: Yun Tang
>Assignee: Arvid Heise
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.0
>
> Attachments:
> Telematics2-feature-flink-1.10-latency-tracking-broken.zip
>
>
> This bug is reported from user mail list:
>
> [http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Latency-tracking-together-with-broadcast-state-can-cause-job-failure-td34013.html]
> Execute {{BroadcastStateIT#broadcastStateWorksWithLatencyTracking}} would
> easily reproduce this problem.
> From current information, the broadcast element would be corrupt once we
> enable {{env.getConfig().setLatencyTrackingInterval(2000)}}.
> The exception stack trace would be: (based on current master branch)
> {code:java}
> Caused by: java.io.IOException: Corrupt stream, found tag: 84
> at
> org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:217)
> ~[classes/:?]
> at
> org.apache.flink.streaming.runtime.streamrecord.StreamElementSerializer.deserialize(StreamElementSerializer.java:46)
> ~[classes/:?]
> at
> org.apache.flink.runtime.plugable.NonReusingDeserializationDelegate.read(NonReusingDeserializationDelegate.java:55)
> ~[classes/:?]
> at
> org.apache.flink.runtime.io.network.api.serialization.SpillingAdaptiveSpanningRecordDeserializer.getNextRecord(SpillingAdaptiveSpanningRecordDeserializer.java:157)
> ~[classes/:?]
> at
> org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:123)
> ~[classes/:?]
> at
> org.apache.flink.streaming.runtime.io.StreamTwoInputProcessor.processInput(StreamTwoInputProcessor.java:181)
> ~[classes/:?]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:332)
> ~[classes/:?]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxStep(MailboxProcessor.java:206)
> ~[classes/:?]
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:196)
> ~[classes/:?]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:505)
> ~[classes/:?]
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:485)
> ~[classes/:?]
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:720)
> ~[classes/:?]
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:544)
> ~[classes/:?]
> at java.lang.Thread.run(Thread.java:748) ~[?:1.8.0_144]
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-17955) BucketLifeCycleListener should just in Buckets
[ https://issues.apache.org/jira/browse/FLINK-17955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-17955. Resolution: Fixed master: 5a9fe5d431cc7722a0af0ab2d6a95460e7e38e77 release-1.11: fe1b533a855b72a9ce3757778a40920e89d7f00c > BucketLifeCycleListener should just in Buckets > -- > > Key: FLINK-17955 > URL: https://issues.apache.org/jira/browse/FLINK-17955 > Project: Flink > Issue Type: Bug > Components: Connectors / FileSystem >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Fix For: 1.11.0 > > > We should keep BucketLifeCycleListener just in runtime. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #12353: [FLINK-17322][network] Fixes BroadcastRecordWriter overwriting memory segments on first finished BufferConsumer.
flinkbot commented on pull request #12353: URL: https://github.com/apache/flink/pull/12353#issuecomment-634512738 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 6141cd7164c5febb45e97f68bc9873dcd6789e21 (Wed May 27 08:31:50 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17923) It will throw MemoryAllocationException if rocksdb statebackend and Python UDF are used in the same slot
[
https://issues.apache.org/jira/browse/FLINK-17923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117512#comment-17117512
]
Zhu Zhu commented on FLINK-17923:
-
We([~yunta]) just find that we can also have option #4 that users set
"state.backend.rocksdb.memory.fixed-per-slot" for python+RocksDB jobs to let
RocksDB instances use off-heap memory instead of managed memory. There would be
no managed memory reservation exception since only python operators will
reserve managed memory.
> It will throw MemoryAllocationException if rocksdb statebackend and Python
> UDF are used in the same slot
> --
>
> Key: FLINK-17923
> URL: https://issues.apache.org/jira/browse/FLINK-17923
> Project: Flink
> Issue Type: Bug
> Components: API / Python, Runtime / State Backends
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Dian Fu
>Priority: Blocker
> Fix For: 1.11.0
>
>
> For the following job:
> {code}
> import logging
> import os
> import shutil
> import sys
> import tempfile
> from pyflink.datastream import StreamExecutionEnvironment
> from pyflink.table import TableConfig, StreamTableEnvironment, DataTypes
> from pyflink.table.udf import udf
> def word_count():
> content = "line Licensed to the Apache Software Foundation ASF under one
> " \
> "line or more contributor license agreements See the NOTICE
> file " \
> "line distributed with this work for additional information " \
> "line regarding copyright ownership The ASF licenses this file
> " \
> "to you under the Apache License Version the " \
> "License you may not use this file except in compliance " \
> "with the License"
> t_config = TableConfig()
> env = StreamExecutionEnvironment.get_execution_environment()
> t_env = StreamTableEnvironment.create(env, t_config)
> # register Results table in table environment
> tmp_dir = tempfile.gettempdir()
> result_path = tmp_dir + '/result'
> if os.path.exists(result_path):
> try:
> if os.path.isfile(result_path):
> os.remove(result_path)
> else:
> shutil.rmtree(result_path)
> except OSError as e:
> logging.error("Error removing directory: %s - %s.", e.filename,
> e.strerror)
> logging.info("Results directory: %s", result_path)
> sink_ddl = """
> create table Results(
> word VARCHAR,
> `count` BIGINT
> ) with (
> 'connector' = 'blackhole'
> )
> """
> t_env.sql_update(sink_ddl)
> @udf(input_types=[DataTypes.BIGINT()], result_type=DataTypes.BIGINT())
> def inc(count):
> return count + 1
> t_env.register_function("inc", inc)
> elements = [(word, 1) for word in content.split(" ")]
> t_env.from_elements(elements, ["word", "count"]) \
> .group_by("word") \
> .select("word, count(1) as count") \
> .select("word, inc(count) as count") \
> .insert_into("Results")
> t_env.execute("word_count")
> if __name__ == '__main__':
> logging.basicConfig(stream=sys.stdout, level=logging.INFO,
> format="%(message)s")
> word_count()
> {code}
> It will throw the following exception if rocksdb state backend is used:
> {code}
> Caused by: org.apache.flink.util.FlinkException: Could not restore keyed
> state backend for KeyedProcessOperator_c27dcf7b54ef6bfd6cff02ca8870b681_(1/1)
> from any of the 1 provided restore options.
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(BackendRestorerProcedure.java:135)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.keyedStatedBackend(StreamTaskStateInitializerImpl.java:317)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.streamOperatorStateContext(StreamTaskStateInitializerImpl.java:144)
> ... 9 more
> Caused by: java.io.IOException: Failed to acquire shared cache resource for
> RocksDB
> at
> org.apache.flink.contrib.streaming.state.RocksDBOperationUtils.allocateSharedCachesIfConfigured(RocksDBOperationUtils.java:212)
> at
> org.apache.flink.contrib.streaming.state.RocksDBStateBackend.createKeyedStateBackend(RocksDBStateBackend.java:516)
> at
> org.apache.flink.streaming.api.operators.StreamTaskStateInitializerImpl.lambda$keyedStatedBackend$1(StreamTaskStateInitializerImpl.java:301)
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.attemptCreateAndRestore(BackendRestorerProcedure.java:142)
> at
> org.apache.flink.streaming.api.operators.BackendRestorerProcedure.createAndRestore(Backe
[jira] [Commented] (FLINK-17935) Logs could not show up when deploying Flink on Yarn via "--executor"
[
https://issues.apache.org/jira/browse/FLINK-17935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117514#comment-17117514
]
Yang Wang commented on FLINK-17935:
---
What i mean is not directly get the {{configDir}} from environment in
{{YarnClusterDescriptor}}. Instead, we add a new argument {{String configDir}}
to constructor of {{YarnClusterDescriptor}}.
I do not insist on this. Moving it to {{YarnClusterClientFactory}} also makes
sense to me.
> Logs could not show up when deploying Flink on Yarn via "--executor"
>
>
> Key: FLINK-17935
> URL: https://issues.apache.org/jira/browse/FLINK-17935
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Yang Wang
>Assignee: Kostas Kloudas
>Priority: Critical
> Fix For: 1.11.0
>
>
> {code:java}
> ./bin/flink run -d -p 5 -e yarn-per-job
> examples/streaming/WindowJoin.jar{code}
> When we use the {{-e/--executor}} to specify the deploy target to Yarn
> per-job, the logs could not show up. The root cause is we do not set the
> logging files in {{ExecutorCLI}}. We only do it in the
> {{FlinkYarnSessionCli}}.
> If we use {{-m yarn-cluster}}, everything works well.
>
> Maybe we should move the {{setLogConfigFileInConfig}} to
> {{YarnClusterDescriptor}} to avoid this problem. cc [~kkl0u]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wuchong commented on a change in pull request #12275: [FLINK-16021][table-common] DescriptorProperties.putTableSchema does …
wuchong commented on a change in pull request #12275:
URL: https://github.com/apache/flink/pull/12275#discussion_r430947071
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/descriptors/DescriptorProperties.java
##
@@ -669,6 +677,14 @@ public DataType getDataType(String key) {
}
}
+ // Extract unique constraints.
+ String pkConstraintNameKey = key + '.' + PRIMARY_KEY_NAME;
+ final Optional pkConstraintNameOpt =
optionalGet(pkConstraintNameKey);
+ if (pkConstraintNameOpt.isPresent()) {
+ final String pkColumnsKey = key + '.' +
PRIMARY_KEY_COLUMNS;
+ final String columns =
optionalGet(pkColumnsKey).orElseThrow(exceptionSupplier(pkColumnsKey));
+ schemaBuilder.primaryKey(pkConstraintNameOpt.get(),
columns.split(","));
+ }
Review comment:
Throw an exception if the the name is not present?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-17958) Kubernetes session constantly allocates taskmanagers after cancel a job
[
https://issues.apache.org/jira/browse/FLINK-17958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117515#comment-17117515
]
Xintong Song commented on FLINK-17958:
--
True. I agree that the bug in {{MathUtils#divideRoundUp}} is the cause of the
problem.
I think we should do the following things.
* Make {{MathUtils#divideRoundUp}} return 0 when {{dividend}} is 0.
* Check that {{dividend >= 0 && divisor > 0}}, and throw an exception otherwise.
* Add more test cases for {{MathUtils#divideRoundUp}}.
* In {{WorkerSpecContainerResourceAdapter#normalize}}, we should return
{{unitValue}} in case {{MathUtils#divideRoundUp}} returns 0.
> Kubernetes session constantly allocates taskmanagers after cancel a job
> ---
>
> Key: FLINK-17958
> URL: https://issues.apache.org/jira/browse/FLINK-17958
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Yang Wang
>Priority: Blocker
> Fix For: 1.11.0
>
>
> When i am testing the {{kubernetes-session.sh}}, i find that the
> {{KubernetesResourceManager}} will constantly allocate taskmanager after
> cancel a job. I think it may be caused by a bug of the following code. When
> the {{dividend}} is 0 and {{divisor}} is bigger than 1, the return value will
> be 1. However, we expect it to be 0.
> {code:java}
> /**
> * Divide and rounding up to integer.
> * E.g., divideRoundUp(3, 2) returns 2.
> * @param dividend value to be divided by the divisor
> * @param divisor value by which the dividend is to be divided
> * @return the quotient rounding up to integer
> */
> public static int divideRoundUp(int dividend, int divisor) {
>return (dividend - 1) / divisor + 1;
> }{code}
>
> How to reproduce this issue?
> # Start a Kubernetes session
> # Submit a Flink job to the existing session
> # Cancel the job and wait for the TaskManager released via idle timeout
> # More and more TaskManagers will be allocated
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12351: [FLINK-17820][task][checkpointing] Backport to 1.11: Respect fileSizeThreshold in FsCheckpointStateOutputStream.flush()
flinkbot edited a comment on pull request #12351: URL: https://github.com/apache/flink/pull/12351#issuecomment-634508548 ## CI report: * a57f600a677c9b0e7431c107c8652a71b83d029f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2251) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12323: [FLINK-17925][fs-connector] Fix Filesystem options to default values and types
flinkbot edited a comment on pull request #12323: URL: https://github.com/apache/flink/pull/12323#issuecomment-633592742 ## CI report: * 5183374cf406535be6ba4c8a4de100e88a0d658c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2243) * 0619f577f658e1d912e5625d71a672f0b0889196 UNKNOWN * 8edb73c8fa0ebc89495e45a21a413e7649244ce6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-17959) Exception: "CANCELLED: call already cancelled" is thrown when run python udf
Hequn Cheng created FLINK-17959:
---
Summary: Exception: "CANCELLED: call already cancelled" is thrown
when run python udf
Key: FLINK-17959
URL: https://issues.apache.org/jira/browse/FLINK-17959
Project: Flink
Issue Type: Bug
Components: API / Python
Affects Versions: 1.10.1, 1.11.0
Reporter: Hequn Cheng
The exception is thrown when running Python UDF:
{code:java}
May 27, 2020 3:20:49 PM
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.SerializingExecutor run
SEVERE: Exception while executing runnable
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed@3960b30e
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.StatusRuntimeException: CANCELLED:
call already cancelled
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Status.asRuntimeException(Status.java:524)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ServerCalls$ServerCallStreamObserverImpl.onCompleted(ServerCalls.java:366)
at
org.apache.beam.runners.fnexecution.state.GrpcStateService$Inbound.onError(GrpcStateService.java:145)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.stub.ServerCalls$StreamingServerCallHandler$StreamingServerCallListener.onCancel(ServerCalls.java:270)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.PartialForwardingServerCallListener.onCancel(PartialForwardingServerCallListener.java:40)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ForwardingServerCallListener.onCancel(ForwardingServerCallListener.java:23)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.ForwardingServerCallListener$SimpleForwardingServerCallListener.onCancel(ForwardingServerCallListener.java:40)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.Contexts$ContextualizedServerCallListener.onCancel(Contexts.java:96)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerCallImpl$ServerStreamListenerImpl.closed(ServerCallImpl.java:337)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ServerImpl$JumpToApplicationThreadServerStreamListener$1Closed.runInContext(ServerImpl.java:793)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
at
org.apache.beam.vendor.grpc.v1p21p0.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{code}
The job can output the right results however it seems something goes wrong
during the shutdown procedure.
You can reproduce the exception with the following code(note: the exception
happens occasionally):
{code}
from pyflink.datastream import StreamExecutionEnvironment
from pyflink.table import StreamTableEnvironment, DataTypes
from pyflink.table.descriptors import Schema, OldCsv, FileSystem
from pyflink.table.udf import udf
env = StreamExecutionEnvironment.get_execution_environment()
env.set_parallelism(1)
t_env = StreamTableEnvironment.create(env)
add = udf(lambda i, j: i + j, [DataTypes.BIGINT(), DataTypes.BIGINT()],
DataTypes.BIGINT())
t_env.register_function("add", add)
t_env.connect(FileSystem().path('/tmp/input')) \
.with_format(OldCsv()
.field('a', DataTypes.BIGINT())
.field('b', DataTypes.BIGINT())) \
.with_schema(Schema()
.field('a', DataTypes.BIGINT())
.field('b', DataTypes.BIGINT())) \
.create_temporary_table('mySource')
t_env.connect(FileSystem().path('/tmp/output')) \
.with_format(OldCsv()
.field('sum', DataTypes.BIGINT())) \
.with_schema(Schema()
.field('sum', DataTypes.BIGINT())) \
.create_temporary_table('mySink')
t_env.from_path('mySource')\
.select("add(a, b)") \
.insert_into('mySink')
t_env.execute("tutorial_job")
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12275: [FLINK-16021][table-common] DescriptorProperties.putTableSchema does …
flinkbot edited a comment on pull request #12275: URL: https://github.com/apache/flink/pull/12275#issuecomment-631915106 ## CI report: * e027ff0822c8621eb8f061888cc15fd7465133ba Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2241) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12353: [FLINK-17322][network] Fixes BroadcastRecordWriter overwriting memory segments on first finished BufferConsumer.
flinkbot commented on pull request #12353: URL: https://github.com/apache/flink/pull/12353#issuecomment-634518102 ## CI report: * 6141cd7164c5febb45e97f68bc9873dcd6789e21 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12352: [FLINK-17717][sql-parser] Throws for DDL create temporary system func…
flinkbot commented on pull request #12352: URL: https://github.com/apache/flink/pull/12352#issuecomment-634517986 ## CI report: * d66997401817c7cd66ce77b6bcfa48401e6c163b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17942) Count distinct could not clean state in WindowOperator
[ https://issues.apache.org/jira/browse/FLINK-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117517#comment-17117517 ] Benchao Li commented on FLINK-17942: [~jark] Yes. I'd like to fix this. Actually, we need to fix for the regular aggregate count distinct too, do you think we need to open another issue or fix it in this issue? > Count distinct could not clean state in WindowOperator > -- > > Key: FLINK-17942 > URL: https://issues.apache.org/jira/browse/FLINK-17942 > Project: Flink > Issue Type: Bug >Affects Versions: 1.9.3, 1.10.1, 1.11.0 >Reporter: Benchao Li >Priority: Major > Fix For: 1.11.0 > > > MapView.clear() is generated in NamespaceAggsHandleFunction.cleanup, however > it's never been called in WindowOperator in blink planner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wuchong commented on pull request #12284: [FLINK-17689][kafka][table] Add integration tests for changelog sourc…
wuchong commented on pull request #12284: URL: https://github.com/apache/flink/pull/12284#issuecomment-634518764 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] HuangXingBo commented on pull request #12349: [FLINK-17722][python][build system] (followups) Keeps all jars of lib directory in CachedFiles
HuangXingBo commented on pull request #12349: URL: https://github.com/apache/flink/pull/12349#issuecomment-634518956 It is the nightly build in my free Azure account. https://dev.azure.com/hxbks2ks/FLINK-TEST/_build/results?buildId=288&view=artifacts&type=publishedArtifacts. And I test the wheel packages is the same as the built from the source code. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-17942) Count distinct could not clean state in WindowOperator
[ https://issues.apache.org/jira/browse/FLINK-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu reassigned FLINK-17942: --- Assignee: Benchao Li > Count distinct could not clean state in WindowOperator > -- > > Key: FLINK-17942 > URL: https://issues.apache.org/jira/browse/FLINK-17942 > Project: Flink > Issue Type: Bug >Affects Versions: 1.9.3, 1.10.1, 1.11.0 >Reporter: Benchao Li >Assignee: Benchao Li >Priority: Major > Fix For: 1.11.0 > > > MapView.clear() is generated in NamespaceAggsHandleFunction.cleanup, however > it's never been called in WindowOperator in blink planner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17960) Improve commands in the "Common Questions" document for PyFlink
Hequn Cheng created FLINK-17960:
---
Summary: Improve commands in the "Common Questions" document for
PyFlink
Key: FLINK-17960
URL: https://issues.apache.org/jira/browse/FLINK-17960
Project: Flink
Issue Type: Improvement
Components: API / Python
Affects Versions: 1.11.0
Reporter: Hequn Cheng
Currently, in the ["Common Questions"
|https://ci.apache.org/projects/flink/flink-docs-master/dev/table/python/common_questions.html#preparing-python-virtual-environment]document,
we have the command `$ setup-pyflink-virtual-env.sh` to run the script.
However, the script is not executable. It would be better to replace the
command with `$ sh setup-pyflink-virtual-env.sh` and add download command.
{code}
$ curl -O
https://ci.apache.org/projects/flink/flink-docs-master/downloads/setup-pyflink-virtual-env.sh
$ sh setup-pyflink-virtual-env.sh
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17942) Count distinct could not clean state in WindowOperator
[ https://issues.apache.org/jira/browse/FLINK-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117531#comment-17117531 ] Jark Wu commented on FLINK-17942: - Assigned to you [~libenchao]. Didn't we call the cleanup in regular aggregate? > Count distinct could not clean state in WindowOperator > -- > > Key: FLINK-17942 > URL: https://issues.apache.org/jira/browse/FLINK-17942 > Project: Flink > Issue Type: Bug >Affects Versions: 1.9.3, 1.10.1, 1.11.0 >Reporter: Benchao Li >Assignee: Benchao Li >Priority: Major > Fix For: 1.11.0 > > > MapView.clear() is generated in NamespaceAggsHandleFunction.cleanup, however > it's never been called in WindowOperator in blink planner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] SteNicholas opened a new pull request #12354: [FLINK-17572][runtime] Remove checkpoint alignment buffered metric from webui
SteNicholas opened a new pull request #12354: URL: https://github.com/apache/flink/pull/12354 ## What is the purpose of the change *After avoid caching buffers for blocked input channels before barrier alignment, runtime never cache buffers while checkpoint barrier alignment, therefore checkpoint alignment buffered metric would always be 0, which should remove it directly in `CheckpointStatistics `, `CheckpointingStatistics`, `TaskCheckpointStatistics`, `TaskCheckpointStatisticsWithSubtaskDetails` and `SubtaskCheckpointStatistics`.* ## Brief change log - *Remove `alignmentBuffered` attribute in `CheckpointStatistics `, `CheckpointingStatistics`, `TaskCheckpointStatistics`, `TaskCheckpointStatisticsWithSubtaskDetails` and `SubtaskCheckpointStatistics`* - *Remove `alignment_buffered` in Checkpoint Detail from `job-checkpoints.component.html`.* - *Remove `alignment_buffered` column in document of `/jobs/:jobid/checkpoints` rest interface.* ## Verifying this change - *Modify test object create by `CheckpointStatistics `, `CheckpointingStatistics`, `TaskCheckpointStatistics`, `TaskCheckpointStatisticsWithSubtaskDetails` and `SubtaskCheckpointStatistics` in `CheckpointingStatisticsTest`, `TaskCheckpointStatisticsTest` and `TaskCheckpointStatisticsWithSubtaskDetailsTest`.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (**not applicable** / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17572) Remove checkpoint alignment buffered metric from webui
[ https://issues.apache.org/jira/browse/FLINK-17572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-17572: --- Labels: pull-request-available (was: ) > Remove checkpoint alignment buffered metric from webui > -- > > Key: FLINK-17572 > URL: https://issues.apache.org/jira/browse/FLINK-17572 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Web Frontend >Affects Versions: 1.11.0 >Reporter: Yingjie Cao >Assignee: Nicholas Jiang >Priority: Minor > Labels: pull-request-available > Fix For: 1.12.0 > > > After FLINK-16404, we never cache buffers while checkpoint barrier alignment, > so the checkpoint alignment buffered metric will be always 0, we should > remove it directly. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-17961) Create an Elasticsearch source
Etienne Chauchot created FLINK-17961: Summary: Create an Elasticsearch source Key: FLINK-17961 URL: https://issues.apache.org/jira/browse/FLINK-17961 Project: Flink Issue Type: New Feature Reporter: Etienne Chauchot There is only an Elasticsearch sink available. There are opensource github repos such as [this one|[https://github.com/mnubo/flink-elasticsearch-source-connector]]. Also the apache bahir project does not provide an Elasticsearch source connector for flink either. IMHO I think the project would benefit from having an stock source connector for ES alongside with the available sink connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17961) Create an Elasticsearch source
[ https://issues.apache.org/jira/browse/FLINK-17961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Etienne Chauchot updated FLINK-17961: - Component/s: Connectors / ElasticSearch > Create an Elasticsearch source > -- > > Key: FLINK-17961 > URL: https://issues.apache.org/jira/browse/FLINK-17961 > Project: Flink > Issue Type: New Feature > Components: Connectors / ElasticSearch >Reporter: Etienne Chauchot >Priority: Minor > > There is only an Elasticsearch sink available. There are opensource github > repos such as [this > one|[https://github.com/mnubo/flink-elasticsearch-source-connector]]. Also > the apache bahir project does not provide an Elasticsearch source connector > for flink either. IMHO I think the project would benefit from having an stock > source connector for ES alongside with the available sink connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #12354: [FLINK-17572][runtime] Remove checkpoint alignment buffered metric from webui
flinkbot commented on pull request #12354: URL: https://github.com/apache/flink/pull/12354#issuecomment-634525524 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 18d7c8a353f4851e8830bf91986fa95e8119de8a (Wed May 27 08:56:40 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17422) Create user document for the external resource framework and the GPU plugin..
[ https://issues.apache.org/jira/browse/FLINK-17422?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117538#comment-17117538 ] Yangze Guo commented on FLINK-17422: Hi, there. I've drafted a user document for the external resource framework. It would be appreciated if you could take a look and any feedback is welcome. BTW, could someone kindly assign this to me? > Create user document for the external resource framework and the GPU plugin.. > - > > Key: FLINK-17422 > URL: https://issues.apache.org/jira/browse/FLINK-17422 > Project: Flink > Issue Type: Sub-task > Components: Documentation >Affects Versions: 1.11.0 >Reporter: Xintong Song >Priority: Major > Fix For: 1.11.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17942) Count distinct could not clean state in WindowOperator
[ https://issues.apache.org/jira/browse/FLINK-17942?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117540#comment-17117540 ] Benchao Li commented on FLINK-17942: [~jark] I've checked the code, it did call the cleanup already. > Count distinct could not clean state in WindowOperator > -- > > Key: FLINK-17942 > URL: https://issues.apache.org/jira/browse/FLINK-17942 > Project: Flink > Issue Type: Bug >Affects Versions: 1.9.3, 1.10.1, 1.11.0 >Reporter: Benchao Li >Assignee: Benchao Li >Priority: Major > Fix For: 1.11.0 > > > MapView.clear() is generated in NamespaceAggsHandleFunction.cleanup, however > it's never been called in WindowOperator in blink planner. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12352: [FLINK-17717][sql-parser] Throws for DDL create temporary system func…
flinkbot edited a comment on pull request #12352: URL: https://github.com/apache/flink/pull/12352#issuecomment-634517986 ## CI report: * d66997401817c7cd66ce77b6bcfa48401e6c163b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2254) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12323: [FLINK-17925][fs-connector] Fix Filesystem options to default values and types
flinkbot edited a comment on pull request #12323: URL: https://github.com/apache/flink/pull/12323#issuecomment-633592742 ## CI report: * 5183374cf406535be6ba4c8a4de100e88a0d658c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2243) * 0619f577f658e1d912e5625d71a672f0b0889196 UNKNOWN * 8edb73c8fa0ebc89495e45a21a413e7649244ce6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2250) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12353: [FLINK-17322][network] Fixes BroadcastRecordWriter overwriting memory segments on first finished BufferConsumer.
flinkbot edited a comment on pull request #12353: URL: https://github.com/apache/flink/pull/12353#issuecomment-634518102 ## CI report: * 6141cd7164c5febb45e97f68bc9873dcd6789e21 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2255) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] godfreyhe opened a new pull request #12355: [FLINK-17893] [sql-client] SQL CLI should print the root cause if the statement is invalid
godfreyhe opened a new pull request #12355: URL: https://github.com/apache/flink/pull/12355 ## What is the purpose of the change *FLINK-17728 supports parsing statements via sql parser, the SqlCommandParser will parse a statement via sql parser first (only ValidationException will be checked). If parse failed, then SqlCommandParser parses a statement via regex matching. So the SqlParseException and other exception are missing. The pr aims to fix this. The solution is changing the parse strategy: try to use regex matching to find a command. if nothing is found, use sql parser to parse the statement and don't catch any exception in SqlCommandParser.* ## Brief change log - *Update the parse strategy* - *Catch and print the SqlExecutionException in CliClient#parseCommand method* ## Verifying this change This change added tests and can be verified as follows: - *Simplify SqlCommandParseTest* - *add CliClientTest#testCreateTableWithInvalidDdl case to validate the case mentioned in the jira* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / **no**) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / **no**) - The serializers: (yes / **no** / don't know) - The runtime per-record code paths (performance sensitive): (yes / **no** / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** / don't know) - The S3 file system connector: (yes / **no** / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / **no**) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / **not documented**) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] godfreyhe commented on pull request #12355: [FLINK-17893] [sql-client] SQL CLI should print the root cause if the statement is invalid
godfreyhe commented on pull request #12355: URL: https://github.com/apache/flink/pull/12355#issuecomment-634528771 cc @danny0405 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17893) SQL-CLI no exception stack
[ https://issues.apache.org/jira/browse/FLINK-17893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-17893: --- Labels: pull-request-available (was: ) > SQL-CLI no exception stack > -- > > Key: FLINK-17893 > URL: https://issues.apache.org/jira/browse/FLINK-17893 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Reporter: Jingsong Lee >Assignee: godfrey he >Priority: Blocker > Labels: pull-request-available > Fix For: 1.11.0 > > > If write a wrong DDL, only "[ERROR] Unknown or invalid SQL statement" message. > No exception stack in client and logs. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-17958) Kubernetes session constantly allocates taskmanagers after cancel a job
[
https://issues.apache.org/jira/browse/FLINK-17958?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrey Zagrebin reassigned FLINK-17958:
---
Assignee: Xintong Song
> Kubernetes session constantly allocates taskmanagers after cancel a job
> ---
>
> Key: FLINK-17958
> URL: https://issues.apache.org/jira/browse/FLINK-17958
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Yang Wang
>Assignee: Xintong Song
>Priority: Blocker
> Fix For: 1.11.0
>
>
> When i am testing the {{kubernetes-session.sh}}, i find that the
> {{KubernetesResourceManager}} will constantly allocate taskmanager after
> cancel a job. I think it may be caused by a bug of the following code. When
> the {{dividend}} is 0 and {{divisor}} is bigger than 1, the return value will
> be 1. However, we expect it to be 0.
> {code:java}
> /**
> * Divide and rounding up to integer.
> * E.g., divideRoundUp(3, 2) returns 2.
> * @param dividend value to be divided by the divisor
> * @param divisor value by which the dividend is to be divided
> * @return the quotient rounding up to integer
> */
> public static int divideRoundUp(int dividend, int divisor) {
>return (dividend - 1) / divisor + 1;
> }{code}
>
> How to reproduce this issue?
> # Start a Kubernetes session
> # Submit a Flink job to the existing session
> # Cancel the job and wait for the TaskManager released via idle timeout
> # More and more TaskManagers will be allocated
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-17750) YARNHighAvailabilityITCase.testKillYarnSessionClusterEntrypoint failed on azure
[
https://issues.apache.org/jira/browse/FLINK-17750?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-17750.
-
Resolution: Fixed
Fixed via
master: 93b81a38efe4dce41c985cd93862d6cbf0c8d0c4
1.11.0: 0e9ddbcbfae556860c0c5323e616cc82f55efe6a
> YARNHighAvailabilityITCase.testKillYarnSessionClusterEntrypoint failed on
> azure
> ---
>
> Key: FLINK-17750
> URL: https://issues.apache.org/jira/browse/FLINK-17750
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.0
>Reporter: Roman Khachatryan
>Assignee: Till Rohrmann
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.11.0
>
> Attachments: jobmanager.log
>
>
> [https://dev.azure.com/khachatryanroman/810e80cc-0656-4d3c-9d8c-186764456a01/_apis/build/builds/6/logs/156]
>
> {code:java}
> 2020-05-15T23:42:29.5307581Z [ERROR]
> testKillYarnSessionClusterEntrypoint(org.apache.flink.yarn.YARNHighAvailabilityITCase)
> Time elapsed: 21.68 s <<< ERROR!
> 2020-05-15T23:42:29.5308406Z java.util.concurrent.ExecutionException:
> 2020-05-15T23:42:29.5308864Z
> org.apache.flink.runtime.rest.util.RestClientException: [Internal server
> error., 2020-05-15T23:42:29.5309678Z java.util.concurrent.TimeoutException:
> Invocation of public abstract java.util.concurrent.CompletableFuture
> org.apache.flink.runt
> ime.dispatcher.DispatcherGateway.requestJob(org.apache.flink.api.common.JobID,org.apache.flink.api.common.time.Time)
> timed out.
> 2020-05-15T23:42:29.5310322Zat com.sun.proxy.$Proxy33.requestJob(Unknown
> Source)
> 2020-05-15T23:42:29.5311018Zat
> org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraphInternal(DefaultExecutionGraphCach
> e.java:103)
> 2020-05-15T23:42:29.5311704Zat
> org.apache.flink.runtime.rest.handler.legacy.DefaultExecutionGraphCache.getExecutionGraph(DefaultExecutionGraphCache.java:7
> 1)
> 2020-05-15T23:42:29.5312355Zat
> org.apache.flink.runtime.rest.handler.job.AbstractExecutionGraphHandler.handleRequest(AbstractExecutionGraphHandler.java:75
> )
> 2020-05-15T23:42:29.5312924Zat
> org.apache.flink.runtime.rest.handler.AbstractRestHandler.respondToRequest(AbstractRestHandler.java:73)
> 2020-05-15T23:42:29.5313423Zat
> org.apache.flink.runtime.rest.handler.AbstractHandler.respondAsLeader(AbstractHandler.java:172)
> 2020-05-15T23:42:29.5314497Zat
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.lambda$channelRead0$0(LeaderRetrievalHandler.java:81)
> 2020-05-15T23:42:29.5315083Zat
> java.util.Optional.ifPresent(Optional.java:159)
> 2020-05-15T23:42:29.5315474Zat
> org.apache.flink.util.OptionalConsumer.ifPresent(OptionalConsumer.java:46)
> 2020-05-15T23:42:29.5315979Zat
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:78)
> 2020-05-15T23:42:29.5316520Zat
> org.apache.flink.runtime.rest.handler.LeaderRetrievalHandler.channelRead0(LeaderRetrievalHandler.java:49)
> 2020-05-15T23:42:29.5317092Zat
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:10
> 5)
> 2020-05-15T23:42:29.5317705Zat
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:374)
> 2020-05-15T23:42:29.5318586Zat
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:360)
> 2020-05-15T23:42:29.5319249Zat
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext
> .java:352)
> 2020-05-15T23:42:29.5319729Zat
> org.apache.flink.runtime.rest.handler.router.RouterHandler.routed(RouterHandler.java:110)
> 2020-05-15T23:42:29.5320136Zat
> org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:89)
> 2020-05-15T23:42:29.5320742Zat
> org.apache.flink.runtime.rest.handler.router.RouterHandler.channelRead0(RouterHandler.java:54)
> 2020-05-15T23:42:29.5321195Zat
> org.apache.flink.shaded.netty4.io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:10
> 5)
> 2020-05-15T23:42:29.5321730Zat
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:374)
> 2020-05-15T23:42:29.5322263Zat
> org.apache.flink.shaded.netty4.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerConte
> xt.java:360)
> 2020-05-15T23:42:29.5322
[jira] [Closed] (FLINK-17938) Cannot run mvn clean verify flink-yarn-tests
[
https://issues.apache.org/jira/browse/FLINK-17938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-17938.
-
Resolution: Fixed
Fixed via
master: e1989bf3b598a565501f4d040ef9627aa34e3ded
1.11.0: 8b8ed695714d878fe69ca3dad8e7540b41c5d4e7
> Cannot run mvn clean verify flink-yarn-tests
>
>
> Key: FLINK-17938
> URL: https://issues.apache.org/jira/browse/FLINK-17938
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Tests
>Affects Versions: 1.11.0
>Reporter: Till Rohrmann
>Assignee: Till Rohrmann
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> As part of FLINK-11086, we introduced the setting of the yarn class path in a
> static initializer of {{YarnTestBase.java:199}}. The yarn class path file
> will be generated by the {{maven-dependency-plugin}} in the {{package}}
> phase. Due to this, the {{yarn.classpath}} file won't be accessible to all
> users of the {{YarnTestBase}} class which are run in a previous phase (e.g.
> {{UtilsTest.testUberjarLocator}}).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #12355: [FLINK-17893] [sql-client] SQL CLI should print the root cause if the statement is invalid
flinkbot commented on pull request #12355: URL: https://github.com/apache/flink/pull/12355#issuecomment-634529264 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 578b299f37b20eab2cd6ce64160cbbfc3ba10479 (Wed May 27 09:03:44 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong merged pull request #12342: [FLINK-17751] [table-planner-blink] Fix proctime defined in ddl can not work with over window in Table api
wuchong merged pull request #12342: URL: https://github.com/apache/flink/pull/12342 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-17751) proctime defined in ddl can't work with over window in Table api
[
https://issues.apache.org/jira/browse/FLINK-17751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu closed FLINK-17751.
---
Resolution: Fixed
- master (1.12.0): 0e67f18846e974881af4487202f48a2e6165e8ac
- 1.11.0: a0800628fcd59ffbbf9889ad1b4fe3a4218b807e
> proctime defined in ddl can't work with over window in Table api
>
>
> Key: FLINK-17751
> URL: https://issues.apache.org/jira/browse/FLINK-17751
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Planner
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> the following test will get {{org.apache.flink.table.api.ValidationException:
> Ordering must be defined on a time attribute.}}
> {code:scala}
> @Test
> def testProcTimeTableSourceOverWindow(): Unit = {
> val ddl =
> s"""
> |CREATE TABLE procTimeT (
> | id int,
> | val bigint,
> | name varchar(32),
> | proctime as PROCTIME()
> |) WITH (
> | 'connector' = 'projectable-values',
> | 'bounded' = 'false'
> |)
>""".stripMargin
> util.tableEnv.executeSql(ddl)
> val t = util.tableEnv.from("procTimeT")
> .window(Over partitionBy 'id orderBy 'proctime preceding 2.hours as 'w)
> .select('id, 'name, 'val.sum over 'w as 'valSum)
> .filter('valSum > 100)
> util.verifyPlan(t)
> }
> {code}
> The reason is: the type of proctime is {{TIMESTAMP(3) NOT null}}, while
> {{LegacyTypeInfoDataTypeConverter}} does not handle the mapping between
> {{Types.LOCAL_DATE_TIME}} and {{DataTypes.TIMESTAMP(3)}} with not null.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13009) YARNSessionCapacitySchedulerITCase#testVCoresAreSetCorrectlyAndJobManagerHostnameAreShownInWebInterfaceAndDynamicPropertiesAndYarnApplicationNameAndTaskManagerSlots thro
[
https://issues.apache.org/jira/browse/FLINK-13009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Wang updated FLINK-13009:
--
Fix Version/s: (was: 1.11.0)
> YARNSessionCapacitySchedulerITCase#testVCoresAreSetCorrectlyAndJobManagerHostnameAreShownInWebInterfaceAndDynamicPropertiesAndYarnApplicationNameAndTaskManagerSlots
> throws NPE on Travis
> -
>
> Key: FLINK-13009
> URL: https://issues.apache.org/jira/browse/FLINK-13009
> Project: Flink
> Issue Type: Test
> Components: Deployment / YARN, Tests
>Affects Versions: 1.8.0
>Reporter: Congxian Qiu(klion26)
>Priority: Major
> Labels: test-stability
>
> The test
> {{YARNSessionCapacitySchedulerITCase#testVCoresAreSetCorrectlyAndJobManagerHostnameAreShownInWebInterfaceAndDynamicPropertiesAndYarnApplicationNameAndTaskManagerSlots}}
> throws NPE on Travis.
> NPE throws from RMAppAttemptMetrics.java#128, and the following is the code
> from hadoop-2.8.3[1]
> {code:java}
> // Only add in the running containers if this is the active attempt.
> 128 RMAppAttempt currentAttempt = rmContext.getRMApps()
> 129 .get(attemptId.getApplicationId()).getCurrentAppAttempt();
> {code}
>
> log [https://api.travis-ci.org/v3/job/550689578/log.txt]
> [1]
> [https://github.com/apache/hadoop/blob/b3fe56402d908019d99af1f1f4fc65cb1d1436a2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptMetrics.java#L128]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-13009) YARNSessionCapacitySchedulerITCase#testVCoresAreSetCorrectlyAndJobManagerHostnameAreShownInWebInterfaceAndDynamicPropertiesAndYarnApplicationNameAndTaskManagerSlots thro
[
https://issues.apache.org/jira/browse/FLINK-13009?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yang Wang updated FLINK-13009:
--
Priority: Major (was: Critical)
> YARNSessionCapacitySchedulerITCase#testVCoresAreSetCorrectlyAndJobManagerHostnameAreShownInWebInterfaceAndDynamicPropertiesAndYarnApplicationNameAndTaskManagerSlots
> throws NPE on Travis
> -
>
> Key: FLINK-13009
> URL: https://issues.apache.org/jira/browse/FLINK-13009
> Project: Flink
> Issue Type: Test
> Components: Deployment / YARN, Tests
>Affects Versions: 1.8.0
>Reporter: Congxian Qiu(klion26)
>Priority: Major
> Labels: test-stability
> Fix For: 1.11.0
>
>
> The test
> {{YARNSessionCapacitySchedulerITCase#testVCoresAreSetCorrectlyAndJobManagerHostnameAreShownInWebInterfaceAndDynamicPropertiesAndYarnApplicationNameAndTaskManagerSlots}}
> throws NPE on Travis.
> NPE throws from RMAppAttemptMetrics.java#128, and the following is the code
> from hadoop-2.8.3[1]
> {code:java}
> // Only add in the running containers if this is the active attempt.
> 128 RMAppAttempt currentAttempt = rmContext.getRMApps()
> 129 .get(attemptId.getApplicationId()).getCurrentAppAttempt();
> {code}
>
> log [https://api.travis-ci.org/v3/job/550689578/log.txt]
> [1]
> [https://github.com/apache/hadoop/blob/b3fe56402d908019d99af1f1f4fc65cb1d1436a2/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptMetrics.java#L128]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17887) Improve interface of ScanFormatFactory and SinkFormatFactory
[
https://issues.apache.org/jira/browse/FLINK-17887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-17887:
Priority: Blocker (was: Major)
> Improve interface of ScanFormatFactory and SinkFormatFactory
>
>
> Key: FLINK-17887
> URL: https://issues.apache.org/jira/browse/FLINK-17887
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Jark Wu
>Assignee: Jark Wu
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> There is some problem with current ScanForamtFactory and SinkFormatFactory
> interfaces:
> 1) {{ScanFormat#createScanFormat}} only accepts {{ScanTableSource.Context}},
> which means it can’t work in lookup source.
> 2) The naming of {{ScanFormat}} also indicates it is only used in scan
> source. But a lookup source should be able to work with format too.
> 3) It’s confusing that {{ScanFormatFactory#createScanFormat}} and
> {{ScanFormat#createScanFormat}} (create itself?)
> The proposed new interface strucutre:
> {code:java}
> interface DeserializationFormatFactory {
> DeserializationSchemaProvider createDeserializationSchemaProvider(
> DynamicTableFactory.Context context, ReadableConfig formatOptions);
> }
> interface SerializationFormatFactory {
> SerializationSchemaProvider createSerializationSchemaProvider(
>DynamicTableFactory.Context context, ReadableConfig formatOptions);
> }
> interface DeserializationSchemaProvider {
> DeserializationSchema createDeserializationSchema(
> DynamicTableSource.Context context, DataType producedDataType)
> }
> interface SerializationSchemaProvider{
> SerializationSchema createSerializationSchema(
> DynamicTableSink.Context context, DataType consumedDataType);
> }
> {code}
> The top-level user-facing interfaces are still
> {{DeserializationFormatFactory}} and {{SerializationFormatFactory}}. This is
> also good for future evolution, e.g. introducing new {{EncoderFormatFactory}}
> and {{DecoderFormatFactory}} if we have a new runtime {{Encoder}} {{Decoder}}
> in the future. This also solves our concerns: no concepts of SourceFormat and
> SinkFormat .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17902) Support the new interfaces about temporary functions in PyFlink
[ https://issues.apache.org/jira/browse/FLINK-17902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117562#comment-17117562 ] Nicholas Jiang commented on FLINK-17902: [~hequn8128][~dian.fu], could you please assign this issue to me for supporting temporary functions interface? I have interest in this. > Support the new interfaces about temporary functions in PyFlink > --- > > Key: FLINK-17902 > URL: https://issues.apache.org/jira/browse/FLINK-17902 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Dian Fu >Priority: Major > Fix For: 1.12.0 > > > The interfaces such as createTemporarySystemFunction, > dropTemporarySystemFunction, createFunction, dropFunction, > createTemporaryFunction, dropTemporaryFunction in the Java TableEnvironment > are currently not available in the PyFlink. The aim of this JIRA is to add > support of them in PyFlink. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17887) Improve interface of ScanFormatFactory and SinkFormatFactory
[
https://issues.apache.org/jira/browse/FLINK-17887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117565#comment-17117565
]
Jark Wu commented on FLINK-17887:
-
I upgrade the priority to BLOCKER, because the current public interface has
problem. We shouldn't release halfway interface.
> Improve interface of ScanFormatFactory and SinkFormatFactory
>
>
> Key: FLINK-17887
> URL: https://issues.apache.org/jira/browse/FLINK-17887
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / API
>Reporter: Jark Wu
>Assignee: Jark Wu
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> There is some problem with current ScanForamtFactory and SinkFormatFactory
> interfaces:
> 1) {{ScanFormat#createScanFormat}} only accepts {{ScanTableSource.Context}},
> which means it can’t work in lookup source.
> 2) The naming of {{ScanFormat}} also indicates it is only used in scan
> source. But a lookup source should be able to work with format too.
> 3) It’s confusing that {{ScanFormatFactory#createScanFormat}} and
> {{ScanFormat#createScanFormat}} (create itself?)
> The proposed new interface strucutre:
> {code:java}
> interface DeserializationFormatFactory {
> DeserializationSchemaProvider createDeserializationSchemaProvider(
> DynamicTableFactory.Context context, ReadableConfig formatOptions);
> }
> interface SerializationFormatFactory {
> SerializationSchemaProvider createSerializationSchemaProvider(
>DynamicTableFactory.Context context, ReadableConfig formatOptions);
> }
> interface DeserializationSchemaProvider {
> DeserializationSchema createDeserializationSchema(
> DynamicTableSource.Context context, DataType producedDataType)
> }
> interface SerializationSchemaProvider{
> SerializationSchema createSerializationSchema(
> DynamicTableSink.Context context, DataType consumedDataType);
> }
> {code}
> The top-level user-facing interfaces are still
> {{DeserializationFormatFactory}} and {{SerializationFormatFactory}}. This is
> also good for future evolution, e.g. introducing new {{EncoderFormatFactory}}
> and {{DecoderFormatFactory}} if we have a new runtime {{Encoder}} {{Decoder}}
> in the future. This also solves our concerns: no concepts of SourceFormat and
> SinkFormat .
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] GJL commented on a change in pull request #12278: [FLINK-17019][runtime] Fulfill slot requests in request order
GJL commented on a change in pull request #12278:
URL: https://github.com/apache/flink/pull/12278#discussion_r430919148
##
File path:
flink-runtime/src/test/java/org/apache/flink/runtime/jobmaster/slotpool/SlotPoolRequestCompletionTest.java
##
@@ -103,7 +123,7 @@ private void runSlotRequestCompletionTest(
// check that the slot requests get completed in
sequential order
for (int i = 0; i < slotRequestIds.size(); i++) {
final CompletableFuture
slotRequestFuture = slotRequests.get(i);
- slotRequestFuture.get();
+ assertThat(slotRequestFuture.getNow(null),
not(is(nullValue(;
Review comment:
`is(not(nullValue))` reads nicer
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SlotPoolImpl.java
##
@@ -698,13 +680,13 @@ boolean offerSlot(
componentMainThreadExecutor.assertRunningInMainThread();
- final PendingRequest pendingRequest =
pendingRequests.removeKeyB(allocationID);
+ final PendingRequest pendingRequest =
pendingRequests.getKeyB(allocationID);
Review comment:
It looks like that this should have been in a separate commit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Comment Edited] (FLINK-17901) Add module interface in PyFlink
[ https://issues.apache.org/jira/browse/FLINK-17901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117567#comment-17117567 ] Nicholas Jiang edited comment on FLINK-17901 at 5/27/20, 9:19 AM: -- Hequn ChengDian Fu, could you please assign this issue to me for supporting load_module interface? I have interest in this. was (Author: nicholasjiang): Hequn ChengDian Fu, could you please assign this issue to me for supporting temporary functions interface? I have interest in this. > Add module interface in PyFlink > --- > > Key: FLINK-17901 > URL: https://issues.apache.org/jira/browse/FLINK-17901 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Dian Fu >Priority: Major > Labels: starter > Fix For: 1.12.0 > > > The "load_module" and "unload_module" interfaces in the Java TableEnvironment > are not available in the PyFlink Table API. We should provide these > interfaces in PyFlink Table API as I think these interfaces are also valuable > for Python users. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-17901) Add module interface in PyFlink
[ https://issues.apache.org/jira/browse/FLINK-17901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117567#comment-17117567 ] Nicholas Jiang edited comment on FLINK-17901 at 5/27/20, 9:19 AM: -- Hequn ChengDian Fu, could you please assign this issue to me for supporting module interface? I have interest in this. was (Author: nicholasjiang): Hequn ChengDian Fu, could you please assign this issue to me for supporting load_module interface? I have interest in this. > Add module interface in PyFlink > --- > > Key: FLINK-17901 > URL: https://issues.apache.org/jira/browse/FLINK-17901 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Dian Fu >Priority: Major > Labels: starter > Fix For: 1.12.0 > > > The "load_module" and "unload_module" interfaces in the Java TableEnvironment > are not available in the PyFlink Table API. We should provide these > interfaces in PyFlink Table API as I think these interfaces are also valuable > for Python users. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17901) Add module interface in PyFlink
[ https://issues.apache.org/jira/browse/FLINK-17901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117567#comment-17117567 ] Nicholas Jiang commented on FLINK-17901: Hequn ChengDian Fu, could you please assign this issue to me for supporting temporary functions interface? I have interest in this. > Add module interface in PyFlink > --- > > Key: FLINK-17901 > URL: https://issues.apache.org/jira/browse/FLINK-17901 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Dian Fu >Priority: Major > Labels: starter > Fix For: 1.12.0 > > > The "load_module" and "unload_module" interfaces in the Java TableEnvironment > are not available in the PyFlink Table API. We should provide these > interfaces in PyFlink Table API as I think these interfaces are also valuable > for Python users. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-17901) Add module interface in PyFlink
[ https://issues.apache.org/jira/browse/FLINK-17901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu reassigned FLINK-17901: --- Assignee: Nicholas Jiang > Add module interface in PyFlink > --- > > Key: FLINK-17901 > URL: https://issues.apache.org/jira/browse/FLINK-17901 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Dian Fu >Assignee: Nicholas Jiang >Priority: Major > Labels: starter > Fix For: 1.12.0 > > > The "load_module" and "unload_module" interfaces in the Java TableEnvironment > are not available in the PyFlink Table API. We should provide these > interfaces in PyFlink Table API as I think these interfaces are also valuable > for Python users. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-17901) Add module interface in PyFlink
[ https://issues.apache.org/jira/browse/FLINK-17901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117567#comment-17117567 ] Nicholas Jiang edited comment on FLINK-17901 at 5/27/20, 9:20 AM: -- [~hequn8128],[~dian.fu], could you please assign this issue to me for supporting module interface? I have interest in this. was (Author: nicholasjiang): Hequn ChengDian Fu, could you please assign this issue to me for supporting module interface? I have interest in this. > Add module interface in PyFlink > --- > > Key: FLINK-17901 > URL: https://issues.apache.org/jira/browse/FLINK-17901 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Dian Fu >Priority: Major > Labels: starter > Fix For: 1.12.0 > > > The "load_module" and "unload_module" interfaces in the Java TableEnvironment > are not available in the PyFlink Table API. We should provide these > interfaces in PyFlink Table API as I think these interfaces are also valuable > for Python users. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink-statefun] abc863377 opened a new pull request #117: [FLINK-17690] Python function wrapper omits docstr
abc863377 opened a new pull request #117: URL: https://github.com/apache/flink-statefun/pull/117 Use decorator to avoid losing docstr. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-17901) Add module interface in PyFlink
[ https://issues.apache.org/jira/browse/FLINK-17901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117567#comment-17117567 ] Nicholas Jiang edited comment on FLINK-17901 at 5/27/20, 9:20 AM: -- [~hequn8128], [~dian.fu], could you please assign this issue to me for supporting module interface? I have interest in this. was (Author: nicholasjiang): [~hequn8128],[~dian.fu], could you please assign this issue to me for supporting module interface? I have interest in this. > Add module interface in PyFlink > --- > > Key: FLINK-17901 > URL: https://issues.apache.org/jira/browse/FLINK-17901 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Dian Fu >Priority: Major > Labels: starter > Fix For: 1.12.0 > > > The "load_module" and "unload_module" interfaces in the Java TableEnvironment > are not available in the PyFlink Table API. We should provide these > interfaces in PyFlink Table API as I think these interfaces are also valuable > for Python users. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] wangyang0918 commented on pull request #12330: [FLINK-17565][k8s] Backport to 1.11
wangyang0918 commented on pull request #12330: URL: https://github.com/apache/flink/pull/12330#issuecomment-634538235 cc @zhijiangW Could you help with the merging? This is a critical bug and i really hope it could be backported to release-1.11 before next release candidate. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17690) Python function wrapper omits docstr
[ https://issues.apache.org/jira/browse/FLINK-17690?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-17690: --- Labels: pull-request-available (was: ) > Python function wrapper omits docstr > > > Key: FLINK-17690 > URL: https://issues.apache.org/jira/browse/FLINK-17690 > Project: Flink > Issue Type: Improvement > Components: Stateful Functions >Reporter: Igal Shilman >Assignee: UnityLung >Priority: Minor > Labels: pull-request-available > > Statefun Python SDK has a connivance bind method, that wraps a functions. > The > [wrapper|https://github.com/apache/flink-statefun/blob/master/statefun-python-sdk/statefun/core.py#L182] > would omit the docstr of the wrapper function. A common practice would be to > use [https://docs.python.org/3/library/functools.html#functools.wraps] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-17902) Support the new interfaces about temporary functions in PyFlink
[ https://issues.apache.org/jira/browse/FLINK-17902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117562#comment-17117562 ] Nicholas Jiang edited comment on FLINK-17902 at 5/27/20, 9:20 AM: -- [~hequn8128], [~dian.fu], could you please assign this issue to me for supporting temporary functions interface? I have interest in this. was (Author: nicholasjiang): [~hequn8128][~dian.fu], could you please assign this issue to me for supporting temporary functions interface? I have interest in this. > Support the new interfaces about temporary functions in PyFlink > --- > > Key: FLINK-17902 > URL: https://issues.apache.org/jira/browse/FLINK-17902 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Dian Fu >Priority: Major > Fix For: 1.12.0 > > > The interfaces such as createTemporarySystemFunction, > dropTemporarySystemFunction, createFunction, dropFunction, > createTemporaryFunction, dropTemporaryFunction in the Java TableEnvironment > are currently not available in the PyFlink. The aim of this JIRA is to add > support of them in PyFlink. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17901) Add module interface in PyFlink
[ https://issues.apache.org/jira/browse/FLINK-17901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117570#comment-17117570 ] Dian Fu commented on FLINK-17901: - Thanks [~nicholasjiang], have assigned it to you :) > Add module interface in PyFlink > --- > > Key: FLINK-17901 > URL: https://issues.apache.org/jira/browse/FLINK-17901 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Dian Fu >Assignee: Nicholas Jiang >Priority: Major > Labels: starter > Fix For: 1.12.0 > > > The "load_module" and "unload_module" interfaces in the Java TableEnvironment > are not available in the PyFlink Table API. We should provide these > interfaces in PyFlink Table API as I think these interfaces are also valuable > for Python users. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17916) Provide API to separate KafkaShuffle's Producer and Consumer to different jobs
[ https://issues.apache.org/jira/browse/FLINK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-17916: - Fix Version/s: (was: 1.11.0) 1.12.0 > Provide API to separate KafkaShuffle's Producer and Consumer to different jobs > -- > > Key: FLINK-17916 > URL: https://issues.apache.org/jira/browse/FLINK-17916 > Project: Flink > Issue Type: Improvement > Components: API / DataStream, Connectors / Kafka >Affects Versions: 1.11.0 >Reporter: Yuan Mei >Assignee: Yuan Mei >Priority: Major > Fix For: 1.12.0 > > > Follow up of FLINK-15670 > *Separate sink (producer) and source (consumer) to different jobs* > * In the same job, a sink and a source are recovered independently according > to regional failover. However, they share the same checkpoint coordinator and > correspondingly, share the same global checkpoint snapshot. > * That says if the consumer fails, the producer can not commit written data > because of two-phase commit set-up (the producer needs a checkpoint-complete > signal to complete the second stage) > * Same applies to the producer > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] azagrebin commented on a change in pull request #11245: [FLINK-15794][Kubernetes] Generate the Kubernetes default image version
azagrebin commented on a change in pull request #11245:
URL: https://github.com/apache/flink/pull/11245#discussion_r430979974
##
File path:
flink-kubernetes/src/main/java/org/apache/flink/kubernetes/configuration/KubernetesConfigOptions.java
##
@@ -121,8 +122,11 @@
public static final ConfigOption CONTAINER_IMAGE =
key("kubernetes.container.image")
.stringType()
- .defaultValue("flink:latest")
- .withDescription("Image to use for Flink containers.");
+ .defaultValue("flink:" + EnvironmentInformation.getVersion() +
"-scala_" + EnvironmentInformation.getScalaVersion())
+ .withDescription("Image to use for Flink containers. " +
+ "The specified image MUST be based upon Apache Flink " +
Review comment:
What is the conclusion here?
##
File path: docs/ops/deployment/docker.md
##
@@ -46,6 +46,11 @@ For example, you can use the following aliases:
* `flink:latest` → `flink:-scala_`
* `flink:1.11` → `flink:1.11.-scala_2.11`
+It is recommended to always use an explicit version tag of the docker image
that specifies both the needed Flink and Scala
Review comment:
```suggestion
Note It is recommended to always use
an explicit version tag of the docker image that specifies both the needed
Flink and Scala
```
This is indeed good addition. I would highlight it with a note label.
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/util/EnvironmentInformation.java
##
@@ -103,6 +103,15 @@ public static String getGitCommitTimeString() {
return getVersionsInstance().gitCommitTimeStr;
}
+ /**
+* Returns the exact name of the Dockerimage for Flink that is needed
to run in.
+*
+* @return The "name:tag" of the Flink docker image.
+*/
+ public static String getFlinkDockerImageNameAndTag() {
Review comment:
My concern is also that `EnvironmentInformation` is responsible for the
real runtime information about the execution environment. The real docker image
may be different.
What are the expected users of this method? If there are indeed more than
just a default value for native kubernetes, maybe it makes sense to introduce
`DockerUtils` class or so.
##
File path: docs/ops/deployment/kubernetes.md
##
@@ -262,7 +261,7 @@ spec:
spec:
containers:
- name: jobmanager
-image: flink:{% if site.is_stable
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest{% endif %}
+image: flink:{% if site.is_stable
%}{{site.version}}-scala{{site.scala_version_suffix}}{% else %}latest # This
contains the latest released Flink built against Scala 2.12{% endif %}
Review comment:
Is it not easier to keep information about `latest->2.12` in one place:
docker hub tag page?
Maybe we could point to tags page here if you think that Scala version can
be confusing for users?
If we switch to Scala 2.11 in `latest` tag, this becomes stale.
##
File path: docs/ops/deployment/kubernetes.md
##
@@ -159,7 +159,6 @@ with the `kubectl` command:
## Appendix
### Common cluster resource definitions
-
Review comment:
This change seems to be unrelated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-17935) Logs could not show up when deploying Flink on Yarn via "--executor"
[
https://issues.apache.org/jira/browse/FLINK-17935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117572#comment-17117572
]
Echo Lee commented on FLINK-17935:
--
[~fly_in_gis] Yes, I agree with this unified solution. But first need to solve
how flinkConfDir should be set.
> Logs could not show up when deploying Flink on Yarn via "--executor"
>
>
> Key: FLINK-17935
> URL: https://issues.apache.org/jira/browse/FLINK-17935
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Yang Wang
>Assignee: Kostas Kloudas
>Priority: Critical
> Fix For: 1.11.0
>
>
> {code:java}
> ./bin/flink run -d -p 5 -e yarn-per-job
> examples/streaming/WindowJoin.jar{code}
> When we use the {{-e/--executor}} to specify the deploy target to Yarn
> per-job, the logs could not show up. The root cause is we do not set the
> logging files in {{ExecutorCLI}}. We only do it in the
> {{FlinkYarnSessionCli}}.
> If we use {{-m yarn-cluster}}, everything works well.
>
> Maybe we should move the {{setLogConfigFileInConfig}} to
> {{YarnClusterDescriptor}} to avoid this problem. cc [~kkl0u]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17961) Create an Elasticsearch source
[ https://issues.apache.org/jira/browse/FLINK-17961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Etienne Chauchot updated FLINK-17961: - Description: There is only an Elasticsearch sink available. There are opensource github repos such as [this one|[https://github.com/mnubo/flink-elasticsearch-source-connector]]. Also the apache bahir project does not provide an Elasticsearch source connector for flink either. IMHO I think the project would benefit from having an bundled source connector for ES alongside with the available sink connector. (was: There is only an Elasticsearch sink available. There are opensource github repos such as [this one|[https://github.com/mnubo/flink-elasticsearch-source-connector]]. Also the apache bahir project does not provide an Elasticsearch source connector for flink either. IMHO I think the project would benefit from having an stock source connector for ES alongside with the available sink connector.) > Create an Elasticsearch source > -- > > Key: FLINK-17961 > URL: https://issues.apache.org/jira/browse/FLINK-17961 > Project: Flink > Issue Type: New Feature > Components: Connectors / ElasticSearch >Reporter: Etienne Chauchot >Priority: Minor > > There is only an Elasticsearch sink available. There are opensource github > repos such as [this > one|[https://github.com/mnubo/flink-elasticsearch-source-connector]]. Also > the apache bahir project does not provide an Elasticsearch source connector > for flink either. IMHO I think the project would benefit from having an > bundled source connector for ES alongside with the available sink connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-15821) Allow configuring Kafka partition / topic discovery in StateFun Kafka ingress
[
https://issues.apache.org/jira/browse/FLINK-15821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117575#comment-17117575
]
UnityLung commented on FLINK-15821:
---
Hi,
I would like to fix this.
Could you please assign it to me?
Thanks a lot.
> Allow configuring Kafka partition / topic discovery in StateFun Kafka ingress
> -
>
> Key: FLINK-15821
> URL: https://issues.apache.org/jira/browse/FLINK-15821
> Project: Flink
> Issue Type: Sub-task
> Components: Stateful Functions
>Affects Versions: statefun-2.0.0
>Reporter: Tzu-Li (Gordon) Tai
>Priority: Major
>
> This is already implemented in the {{FlinkKafkaConsumer}}, so it would only
> be a matter of exposing it through Stateful Function's Kafka ingress.
> Proposed API:
> {code}
> KafkaIngressBuilder#withTopics(java.util.regex.Pattern regexPattern)
> KafkaIngressBuilder#enableDiscovery(java.time.Duration discoveryInterval)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15819) Add more StateFun Kafka ingress features that are already available in FlinkKafkaConsumer
[
https://issues.apache.org/jira/browse/FLINK-15819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117573#comment-17117573
]
UnityLung commented on FLINK-15819:
---
Hi,
I would like to fix this.
Could you please assign it to me?
Thanks a lot.
> Add more StateFun Kafka ingress features that are already available in
> FlinkKafkaConsumer
> -
>
> Key: FLINK-15819
> URL: https://issues.apache.org/jira/browse/FLINK-15819
> Project: Flink
> Issue Type: New Feature
> Components: Stateful Functions
>Affects Versions: statefun-2.0.0
>Reporter: Tzu-Li (Gordon) Tai
>Priority: Major
>
> This is an umbrella JIRA to list all pending {{FlinkKafkaConsumer}} features
> that are reasonable to expose through Stateful Function's Kafka ingress.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-15820) Allow disabling / enabling auto offset committing to Kafka in StateFun Kafka ingress
[
https://issues.apache.org/jira/browse/FLINK-15820?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117574#comment-17117574
]
UnityLung commented on FLINK-15820:
---
Hi,
I would like to fix this.
Could you please assign it to me?
Thanks a lot.
> Allow disabling / enabling auto offset committing to Kafka in StateFun Kafka
> ingress
>
>
> Key: FLINK-15820
> URL: https://issues.apache.org/jira/browse/FLINK-15820
> Project: Flink
> Issue Type: Sub-task
> Components: Stateful Functions
>Affects Versions: statefun-2.0.0
>Reporter: Tzu-Li (Gordon) Tai
>Priority: Major
>
> This is already supported in {{FlinkKafkaConsumer}}, so it would only be a
> matter of exposing it through the Stateful Functions Kafka ingress.
> It would be reasonable to support this with it being disabled by default, so
> that users do not always need to set the consumer group id and have a minimal
> setup to get started with using the ingress.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] azagrebin commented on a change in pull request #12278: [FLINK-17019][runtime] Fulfill slot requests in request order
azagrebin commented on a change in pull request #12278: URL: https://github.com/apache/flink/pull/12278#discussion_r430441131 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SlotPoolImpl.java ## @@ -112,6 +114,9 @@ /** The requests that are waiting for the resource manager to be connected. */ private final LinkedHashMap waitingForResourceManager; + /** Maps a request to its allocation. */ + private final BiMap requestedAllocations; Review comment: Looking into the implementation of `DualKeyLinkedMap` for `pendingRequests`, it seems we can just remove the first matching `SlotRequestId` and then remap the orphaned `SlotRequestId` to its `AllocationID`. The original insertion ordering should not suffer in `DualKeyLinkedMap.aMap`. If so, we could remove `requestedAllocations`. EDIT: `waitingForResourceManager` -> `pendingRequests` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17916) Provide API to separate KafkaShuffle's Producer and Consumer to different jobs
[ https://issues.apache.org/jira/browse/FLINK-17916?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yuan Mei updated FLINK-17916: - Affects Version/s: (was: 1.11.0) 1.12.0 > Provide API to separate KafkaShuffle's Producer and Consumer to different jobs > -- > > Key: FLINK-17916 > URL: https://issues.apache.org/jira/browse/FLINK-17916 > Project: Flink > Issue Type: Improvement > Components: API / DataStream, Connectors / Kafka >Affects Versions: 1.12.0 >Reporter: Yuan Mei >Assignee: Yuan Mei >Priority: Major > Fix For: 1.12.0 > > > Follow up of FLINK-15670 > *Separate sink (producer) and source (consumer) to different jobs* > * In the same job, a sink and a source are recovered independently according > to regional failover. However, they share the same checkpoint coordinator and > correspondingly, share the same global checkpoint snapshot. > * That says if the consumer fails, the producer can not commit written data > because of two-phase commit set-up (the producer needs a checkpoint-complete > signal to complete the second stage) > * Same applies to the producer > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17960) Improve commands in the "Common Questions" document for PyFlink
[
https://issues.apache.org/jira/browse/FLINK-17960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117581#comment-17117581
]
Nicholas Jiang commented on FLINK-17960:
[~hequn8128], I would like to modify script of "Common Questions" document
according to your suggestion. Could you please assign this to me？
> Improve commands in the "Common Questions" document for PyFlink
> ---
>
> Key: FLINK-17960
> URL: https://issues.apache.org/jira/browse/FLINK-17960
> Project: Flink
> Issue Type: Improvement
> Components: API / Python
>Affects Versions: 1.11.0
>Reporter: Hequn Cheng
>Priority: Major
>
> Currently, in the ["Common Questions"
> |https://ci.apache.org/projects/flink/flink-docs-master/dev/table/python/common_questions.html#preparing-python-virtual-environment]document,
> we have the command `$ setup-pyflink-virtual-env.sh` to run the script.
> However, the script is not executable. It would be better to replace the
> command with `$ sh setup-pyflink-virtual-env.sh` and add download command.
> {code}
> $ curl -O
> https://ci.apache.org/projects/flink/flink-docs-master/downloads/setup-pyflink-virtual-env.sh
> $ sh setup-pyflink-virtual-env.sh
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
