[jira] [Commented] (FLINK-18974) Translate the 'User-Defined Functions' page of "Application Development's DataStream API" into Chinese
[ https://issues.apache.org/jira/browse/FLINK-18974?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197403#comment-17197403 ] Robert Metzger commented on FLINK-18974: Thanks a lot for the contribution. Merged to master in https://github.com/apache/flink/commit/8f2385b22cdce0e3e163eb215826d149a21bc45a. > Translate the 'User-Defined Functions' page of "Application Development's > DataStream API" into Chinese > -- > > Key: FLINK-18974 > URL: https://issues.apache.org/jira/browse/FLINK-18974 > Project: Flink > Issue Type: Improvement > Components: chinese-translation, Documentation >Affects Versions: 1.10.0, 1.10.1, 1.11.0, 1.11.1 >Reporter: Roc Marshal >Assignee: CloseRiver >Priority: Major > Labels: Translation, pull-request-available, translation-zh > > The page url is > https://ci.apache.org/projects/flink/flink-docs-release-1.11/zh/dev/user_defined_functions.html > The markdown file is located in flink/docs/dev/user_defined_functions.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-18974) Translate the 'User-Defined Functions' page of "Application Development's DataStream API" into Chinese
[ https://issues.apache.org/jira/browse/FLINK-18974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger closed FLINK-18974. -- Resolution: Fixed > Translate the 'User-Defined Functions' page of "Application Development's > DataStream API" into Chinese > -- > > Key: FLINK-18974 > URL: https://issues.apache.org/jira/browse/FLINK-18974 > Project: Flink > Issue Type: Improvement > Components: chinese-translation, Documentation >Affects Versions: 1.10.0, 1.10.1, 1.11.0, 1.11.1 >Reporter: Roc Marshal >Assignee: CloseRiver >Priority: Major > Labels: Translation, pull-request-available, translation-zh > Fix For: 1.12.0 > > > The page url is > https://ci.apache.org/projects/flink/flink-docs-release-1.11/zh/dev/user_defined_functions.html > The markdown file is located in flink/docs/dev/user_defined_functions.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18974) Translate the 'User-Defined Functions' page of "Application Development's DataStream API" into Chinese
[ https://issues.apache.org/jira/browse/FLINK-18974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger updated FLINK-18974: --- Fix Version/s: 1.12.0 > Translate the 'User-Defined Functions' page of "Application Development's > DataStream API" into Chinese > -- > > Key: FLINK-18974 > URL: https://issues.apache.org/jira/browse/FLINK-18974 > Project: Flink > Issue Type: Improvement > Components: chinese-translation, Documentation >Affects Versions: 1.10.0, 1.10.1, 1.11.0, 1.11.1 >Reporter: Roc Marshal >Assignee: CloseRiver >Priority: Major > Labels: Translation, pull-request-available, translation-zh > Fix For: 1.12.0 > > > The page url is > https://ci.apache.org/projects/flink/flink-docs-release-1.11/zh/dev/user_defined_functions.html > The markdown file is located in flink/docs/dev/user_defined_functions.zh.md -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] rmetzger commented on pull request #13225: [FLINK-18974][docs-zh]Translate the 'User-Defined Functions' page of "Application Development's DataStream API" into Chinese
rmetzger commented on pull request #13225: URL: https://github.com/apache/flink/pull/13225#issuecomment-693925440 Thanks for the review! Merging ... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger merged pull request #13225: [FLINK-18974][docs-zh]Translate the 'User-Defined Functions' page of "Application Development's DataStream API" into Chinese
rmetzger merged pull request #13225: URL: https://github.com/apache/flink/pull/13225 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] klion26 commented on a change in pull request #13225: [FLINK-18974][docs-zh]Translate the 'User-Defined Functions' page of "Application Development's DataStream API" into Chinese
klion26 commented on a change in pull request #13225:
URL: https://github.com/apache/flink/pull/13225#discussion_r489984016
##
File path: docs/dev/user_defined_functions.zh.md
##
@@ -147,95 +153,78 @@ data.map (new RichMapFunction[String, Int] {
-Rich functions provide, in addition to the user-defined function (map,
-reduce, etc), four methods: `open`, `close`, `getRuntimeContext`, and
-`setRuntimeContext`. These are useful for parameterizing the function
-(see [Passing Parameters to Functions]({{ site.baseurl
}}/dev/batch/index.html#passing-parameters-to-functions)),
-creating and finalizing local state, accessing broadcast variables (see
-[Broadcast Variables]({{ site.baseurl
}}/dev/batch/index.html#broadcast-variables)), and for accessing runtime
-information such as accumulators and counters (see
-[Accumulators and Counters](#accumulators--counters)), and information
-on iterations (see [Iterations]({{ site.baseurl }}/dev/batch/iterations.html)).
+除了用户自定义的 function（map，reduce 等），Rich functions
还提供了四个方法：`open`、`close`、`getRuntimeContext` 和
+`setRuntimeContext`。这些方法对于参数化 function
+（参阅 [给 function 传递参数]({% link dev/batch/index.zh.md
%}#passing-parameters-to-functions)），
+创建和最终确定本地状态，访问广播变量（参阅
+[广播变量]({% link dev/batch/index.zh.md %}#broadcast-variables
)），以及访问运行时信息，例如累加器和计数器（参阅
Review comment:
@zhuxiaoshang @RocMarshal 这个地方的链接我怀疑是有问题的，我看了下，这个地方是 FLINK-17009
引入的，但是没有找到更早的历史。
怀疑的原因是，这个文档讲的是 datastream 的 udfs，但是这里的链接是 batch 相关的。
不过这个和整体的翻译无关，能否记录并查找一下这个地方是否真的有问题，正确的链接应该是什么？如果有问题的话，可以再提一个 PR 来进行修改
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13404: [FLINK-16100][docs-zh]Translate "Reading & Writing Hive Tables" page …
flinkbot edited a comment on pull request #13404: URL: https://github.com/apache/flink/pull/13404#issuecomment-693785870 ## CI report: * c9569f50a4707d5fbd27ff15e2745e078d92e39d Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6579) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13397: [FLINK-19178][runtime] Extend managed memory weight/fraction interfaces for various use cases.
flinkbot edited a comment on pull request #13397: URL: https://github.com/apache/flink/pull/13397#issuecomment-693261990 ## CI report: * b8e95c022986da9f0b0d1b18e205f7cbc05f51ab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6578) * 7c2c20f97637a965be7c74243592f1157d1ac9f7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6583) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13358: [FLINK-11779] CLI ignores -m parameter if high-availability is ZOOKEEPER
flinkbot edited a comment on pull request #13358: URL: https://github.com/apache/flink/pull/13358#issuecomment-689274090 ## CI report: * 30192c3868c507f128d525cfe4db23f573c9f22a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6543) * f6e6e94da73c668aa873d7155aae158418f6bc2f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6582) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18222) "Avro Confluent Schema Registry nightly end-to-end test" unstable with "Kafka cluster did not start after 120 seconds"
[
https://issues.apache.org/jira/browse/FLINK-18222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197385#comment-17197385
]
Robert Metzger commented on FLINK-18222:
I backported the fix to "release-1.11" in
https://github.com/apache/flink/commit/5ad541120f18a1dd34736b5fd18b349b456fc96a.
> "Avro Confluent Schema Registry nightly end-to-end test" unstable with "Kafka
> cluster did not start after 120 seconds"
> --
>
> Key: FLINK-18222
> URL: https://issues.apache.org/jira/browse/FLINK-18222
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3045=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> {code}
> 2020-06-09T15:16:48.1427795Z
> ==
> 2020-06-09T15:16:48.1428609Z Running 'Avro Confluent Schema Registry nightly
> end-to-end test'
> 2020-06-09T15:16:48.1429204Z
> ==
> 2020-06-09T15:16:48.1438117Z TEST_DATA_DIR:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-48143298170
> 2020-06-09T15:16:48.2985167Z Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT
> 2020-06-09T15:16:48.3157575Z Downloading Kafka from
> https://archive.apache.org/dist/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz
> 2020-06-09T15:16:48.3214487Z % Total% Received % Xferd Average Speed
> TimeTime Time Current
> 2020-06-09T15:16:48.3215154Z Dload Upload
> Total SpentLeft Speed
> 2020-06-09T15:16:48.3215597Z
> 2020-06-09T15:16:48.3528820Z 0 00 00 0 0 0
> --:--:-- --:--:-- --:--:-- 0
> 2020-06-09T15:16:49.3421526Z 0 00 00 0 0 0
> --:--:-- --:--:-- --:--:-- 0
> 2020-06-09T15:16:50.3415678Z 8 35.8M8 2960k0 0 2896k 0
> 0:00:12 0:00:01 0:00:11 2896k
> 2020-06-09T15:16:51.3406836Z 23 35.8M 23 8544k0 0 4226k 0
> 0:00:08 0:00:02 0:00:06 4225k
> 2020-06-09T15:16:51.6553485Z 70 35.8M 70 25.2M0 0 8550k 0
> 0:00:04 0:00:03 0:00:01 8548k
> 2020-06-09T15:16:51.6555606Z 100 35.8M 100 35.8M0 0 10.7M 0
> 0:00:03 0:00:03 --:--:-- 10.7M
> 2020-06-09T15:16:51.9818041Z Downloading confluent from
> http://packages.confluent.io/archive/3.2/confluent-oss-3.2.0-2.11.tar.gz
> 2020-06-09T15:16:51.9880242Z % Total% Received % Xferd Average Speed
> TimeTime Time Current
> 2020-06-09T15:16:51.9880983Z Dload Upload
> Total SpentLeft Speed
> 2020-06-09T15:16:51.9914252Z
> 2020-06-09T15:16:52.3398614Z 0 00 00 0 0 0
> --:--:-- --:--:-- --:--:-- 0
> 2020-06-09T15:16:53.3399552Z 9 398M9 39.5M0 0 111M 0
> 0:00:03 --:--:-- 0:00:03 111M
> 2020-06-09T15:16:53.9149276Z 47 398M 47 188M0 0 139M 0
> 0:00:02 0:00:01 0:00:01 138M
> 2020-06-09T15:16:53.9150980Z 100 398M 100 398M0 0 206M 0
> 0:00:01 0:00:01 --:--:-- 206M
> 2020-06-09T15:17:04.3565942Z Waiting for broker...
> 2020-06-09T15:17:12.4215170Z Waiting for broker...
> 2020-06-09T15:17:14.3012835Z Waiting for broker...
> 2020-06-09T15:17:16.1965074Z Waiting for broker...
> 2020-06-09T15:17:18.1102274Z Waiting for broker...
> 2020-06-09T15:17:19.9929632Z Waiting for broker...
> 2020-06-09T15:17:21.8607172Z Waiting for broker...
> 2020-06-09T15:17:23.7802949Z Waiting for broker...
> 2020-06-09T15:17:25.6695260Z Waiting for broker...
> 2020-06-09T15:17:27.5536417Z Waiting for broker...
> 2020-06-09T15:17:29.4327778Z Waiting for broker...
> 2020-06-09T15:17:31.3203091Z Waiting for broker...
> 2020-06-09T15:17:33.1987150Z Waiting for broker...
> 2020-06-09T15:17:35.0694860Z Waiting for broker...
> 2020-06-09T15:17:36.9595576Z Waiting for broker...
> 2020-06-09T15:17:38.9243558Z Waiting for broker...
> 2020-06-09T15:17:40.7984064Z Waiting for broker...
> 2020-06-09T15:17:42.6676095Z Waiting for broker...
> 2020-06-09T15:17:44.5628797Z Waiting for broker...
> 2020-06-09T15:17:46.4374532Z Waiting for broker...
> 2020-06-09T15:17:48.3086761Z Waiting for broker...
> 2020-06-09T15:17:50.1574336Z Waiting for broker...
> 2020-06-09T15:17:52.0432952Z Waiting for broker...
>
[jira] [Updated] (FLINK-18222) "Avro Confluent Schema Registry nightly end-to-end test" unstable with "Kafka cluster did not start after 120 seconds"
[
https://issues.apache.org/jira/browse/FLINK-18222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-18222:
---
Fix Version/s: 1.11.3
> "Avro Confluent Schema Registry nightly end-to-end test" unstable with "Kafka
> cluster did not start after 120 seconds"
> --
>
> Key: FLINK-18222
> URL: https://issues.apache.org/jira/browse/FLINK-18222
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0, 1.11.3
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3045=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> {code}
> 2020-06-09T15:16:48.1427795Z
> ==
> 2020-06-09T15:16:48.1428609Z Running 'Avro Confluent Schema Registry nightly
> end-to-end test'
> 2020-06-09T15:16:48.1429204Z
> ==
> 2020-06-09T15:16:48.1438117Z TEST_DATA_DIR:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-48143298170
> 2020-06-09T15:16:48.2985167Z Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT
> 2020-06-09T15:16:48.3157575Z Downloading Kafka from
> https://archive.apache.org/dist/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz
> 2020-06-09T15:16:48.3214487Z % Total% Received % Xferd Average Speed
> TimeTime Time Current
> 2020-06-09T15:16:48.3215154Z Dload Upload
> Total SpentLeft Speed
> 2020-06-09T15:16:48.3215597Z
> 2020-06-09T15:16:48.3528820Z 0 00 00 0 0 0
> --:--:-- --:--:-- --:--:-- 0
> 2020-06-09T15:16:49.3421526Z 0 00 00 0 0 0
> --:--:-- --:--:-- --:--:-- 0
> 2020-06-09T15:16:50.3415678Z 8 35.8M8 2960k0 0 2896k 0
> 0:00:12 0:00:01 0:00:11 2896k
> 2020-06-09T15:16:51.3406836Z 23 35.8M 23 8544k0 0 4226k 0
> 0:00:08 0:00:02 0:00:06 4225k
> 2020-06-09T15:16:51.6553485Z 70 35.8M 70 25.2M0 0 8550k 0
> 0:00:04 0:00:03 0:00:01 8548k
> 2020-06-09T15:16:51.6555606Z 100 35.8M 100 35.8M0 0 10.7M 0
> 0:00:03 0:00:03 --:--:-- 10.7M
> 2020-06-09T15:16:51.9818041Z Downloading confluent from
> http://packages.confluent.io/archive/3.2/confluent-oss-3.2.0-2.11.tar.gz
> 2020-06-09T15:16:51.9880242Z % Total% Received % Xferd Average Speed
> TimeTime Time Current
> 2020-06-09T15:16:51.9880983Z Dload Upload
> Total SpentLeft Speed
> 2020-06-09T15:16:51.9914252Z
> 2020-06-09T15:16:52.3398614Z 0 00 00 0 0 0
> --:--:-- --:--:-- --:--:-- 0
> 2020-06-09T15:16:53.3399552Z 9 398M9 39.5M0 0 111M 0
> 0:00:03 --:--:-- 0:00:03 111M
> 2020-06-09T15:16:53.9149276Z 47 398M 47 188M0 0 139M 0
> 0:00:02 0:00:01 0:00:01 138M
> 2020-06-09T15:16:53.9150980Z 100 398M 100 398M0 0 206M 0
> 0:00:01 0:00:01 --:--:-- 206M
> 2020-06-09T15:17:04.3565942Z Waiting for broker...
> 2020-06-09T15:17:12.4215170Z Waiting for broker...
> 2020-06-09T15:17:14.3012835Z Waiting for broker...
> 2020-06-09T15:17:16.1965074Z Waiting for broker...
> 2020-06-09T15:17:18.1102274Z Waiting for broker...
> 2020-06-09T15:17:19.9929632Z Waiting for broker...
> 2020-06-09T15:17:21.8607172Z Waiting for broker...
> 2020-06-09T15:17:23.7802949Z Waiting for broker...
> 2020-06-09T15:17:25.6695260Z Waiting for broker...
> 2020-06-09T15:17:27.5536417Z Waiting for broker...
> 2020-06-09T15:17:29.4327778Z Waiting for broker...
> 2020-06-09T15:17:31.3203091Z Waiting for broker...
> 2020-06-09T15:17:33.1987150Z Waiting for broker...
> 2020-06-09T15:17:35.0694860Z Waiting for broker...
> 2020-06-09T15:17:36.9595576Z Waiting for broker...
> 2020-06-09T15:17:38.9243558Z Waiting for broker...
> 2020-06-09T15:17:40.7984064Z Waiting for broker...
> 2020-06-09T15:17:42.6676095Z Waiting for broker...
> 2020-06-09T15:17:44.5628797Z Waiting for broker...
> 2020-06-09T15:17:46.4374532Z Waiting for broker...
> 2020-06-09T15:17:48.3086761Z Waiting for broker...
> 2020-06-09T15:17:50.1574336Z Waiting for broker...
> 2020-06-09T15:17:52.0432952Z Waiting for broker...
> 2020-06-09T15:17:53.9406541Z Waiting for broker...
> 2020-06-09T15:17:55.8162052Z Waiting for broker...
> 2020-06-09T15:17:57.7090015Z
[GitHub] [flink-web] zhuzhurk commented on pull request #377: Add Apache Flink release 1.11.2
zhuzhurk commented on pull request #377: URL: https://github.com/apache/flink-web/pull/377#issuecomment-693882177 Thanks for reviewing! @dianfu @zhijiangW The date is updated. Merging. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] zhuzhurk merged pull request #377: Add Apache Flink release 1.11.2
zhuzhurk merged pull request #377: URL: https://github.com/apache/flink-web/pull/377 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13397: [FLINK-19178][runtime] Extend managed memory weight/fraction interfaces for various use cases.
flinkbot edited a comment on pull request #13397: URL: https://github.com/apache/flink/pull/13397#issuecomment-693261990 ## CI report: * b8e95c022986da9f0b0d1b18e205f7cbc05f51ab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6578) * 7c2c20f97637a965be7c74243592f1157d1ac9f7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13358: [FLINK-11779] CLI ignores -m parameter if high-availability is ZOOKEEPER
flinkbot edited a comment on pull request #13358: URL: https://github.com/apache/flink/pull/13358#issuecomment-689274090 ## CI report: * 30192c3868c507f128d525cfe4db23f573c9f22a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6543) * f6e6e94da73c668aa873d7155aae158418f6bc2f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13397: [FLINK-19178][runtime] Extend managed memory weight/fraction interfaces for various use cases.
flinkbot edited a comment on pull request #13397: URL: https://github.com/apache/flink/pull/13397#issuecomment-693261990 ## CI report: * b8e95c022986da9f0b0d1b18e205f7cbc05f51ab Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6578) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13311: [FLINK-18721] Migrate YarnResourceManager to the new YarnResourceManagerDriver
flinkbot edited a comment on pull request #13311: URL: https://github.com/apache/flink/pull/13311#issuecomment-686196434 ## CI report: * b7660009b48d48261b767855d8973c63741ab493 UNKNOWN * f16d78f22d3d66e763f5629a377936bcb20b80c4 UNKNOWN * c9a880c2279b909d953bdef26a5f532a4fd66bfb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6523) * c64348dac8aa33d893d1ff55ac15e5be9d8f8dce Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6581) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13375: [FLINK-18815][filesystems][tests] Join SafetyNetCloseableRegistry's reaper thread before assertion
flinkbot edited a comment on pull request #13375: URL: https://github.com/apache/flink/pull/13375#issuecomment-691410175 ## CI report: * 0e3916e68d38b432ebc33248d3a1ce7f7213b1f8 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6572) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] danny0405 commented on pull request #13291: [FLINK-18988][table] Continuous query with LATERAL and LIMIT produces…
danny0405 commented on pull request #13291: URL: https://github.com/apache/flink/pull/13291#issuecomment-693799225 > I'm trying to base an example on this PR. But the results differ between batch and streaming mode. It seems that the batch mode now outputs the global maximum. The patch is only for streaming purpose, let me check if batch mode is supported. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13402: [FLINK-19262][API/DataStream] Can not setParallelism for FLIP-27 source
flinkbot edited a comment on pull request #13402: URL: https://github.com/apache/flink/pull/13402#issuecomment-693424540 ## CI report: * 4b7f5b86b91caffdb8436913467d165254000745 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6556) * f4a438df036bada30853b6450931e5b116dce9a1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6575) * 83ff7dd5e54dca0b65e99cbf6409dbf6ae0428a4 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6580) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13311: [FLINK-18721] Migrate YarnResourceManager to the new YarnResourceManagerDriver
flinkbot edited a comment on pull request #13311: URL: https://github.com/apache/flink/pull/13311#issuecomment-686196434 ## CI report: * b7660009b48d48261b767855d8973c63741ab493 UNKNOWN * f16d78f22d3d66e763f5629a377936bcb20b80c4 UNKNOWN * c9a880c2279b909d953bdef26a5f532a4fd66bfb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6523) * c64348dac8aa33d893d1ff55ac15e5be9d8f8dce UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19264) Identical jobs cannot be run concurrently
[
https://issues.apache.org/jira/browse/FLINK-19264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197360#comment-17197360
]
Zhu Zhu commented on FLINK-19264:
-
Thanks for reporting and investigating this issue!
I had wrongly assumed that a {{JobVertexID}} is generated randomly. But it is
not. Instead, it is generated from the hash of StreamNode, which is always the
same in the same topology. That's why in [~aljoscha]'s case {{JobVertexID}}
from different jobs can be the same.
Including the {{JobID}} in {{ExecutionAttemptID}} sounds a valid solution to
me, given that a Flink cluster will not accept a job with duplicated {{JobID}}.
I'm also thinking whether we can further {{JobVertexID}} and
{{IntermediateDataSetID}} to consist of {{JobID}} in the future. In this way,
the composite {{ExecutionAttemptID}} will be naturally unique across jobs.
> Identical jobs cannot be run concurrently
> -
>
> Key: FLINK-19264
> URL: https://issues.apache.org/jira/browse/FLINK-19264
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Aljoscha Krettek
>Priority: Blocker
>
> While working on FLINK-19123 I noticed that jobs often fail on
> {{MiniCluster}} when multiple jobs are running at the same time.
> I created a reproducer here:
> https://github.com/aljoscha/flink/tree/flink-19123-fix-test-stream-env-alternative.
> You can run {{MiniClusterConcurrencyITCase}} to see the problem in action.
> Sometimes the test will succeed, sometimes it will fail with
> {code}
> java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.test.example.MiniClusterConcurrencyITCase.submitConcurrently(MiniClusterConcurrencyITCase.java:60)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
> at
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
> at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job
> execution failed.
> at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:147)
> at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$2(MiniClusterJobClient.java:107)
> at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:229)
> at
>
[GitHub] [flink] flinkbot edited a comment on pull request #13404: [FLINK-16100][docs-zh]Translate "Reading & Writing Hive Tables" page …
flinkbot edited a comment on pull request #13404: URL: https://github.com/apache/flink/pull/13404#issuecomment-693785870 ## CI report: * c9569f50a4707d5fbd27ff15e2745e078d92e39d Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6579) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13397: [FLINK-19178][runtime] Extend managed memory weight/fraction interfaces for various use cases.
flinkbot edited a comment on pull request #13397: URL: https://github.com/apache/flink/pull/13397#issuecomment-693261990 ## CI report: * 1e0209f0df58ecdc3460d426262fc89d035b83f6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6550) * b8e95c022986da9f0b0d1b18e205f7cbc05f51ab Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6578) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13371: [FLINK-17879][python] update Python row serializer to support RowKind
flinkbot edited a comment on pull request #13371: URL: https://github.com/apache/flink/pull/13371#issuecomment-690084042 ## CI report: * 6aaa5136a260683572b62d428f54678d9454e0c1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6555) * 1743fdb906cf61fde264a942c0569187edd7b1d6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6577) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-19265) Simplify handling of 'NoMoreSplitsEvent'
[
https://issues.apache.org/jira/browse/FLINK-19265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197346#comment-17197346
]
Jiangjie Qin edited comment on FLINK-19265 at 9/17/20, 3:32 AM:
Thanks for the suggestion, Stephan. I think that makes a lot of sense. We
should add something like the following method in the SplitEnumeratorContext.
{code:java}
void finishSplitsAssignment(int subtask)
{code}
was (Author: becket_qin):
Thanks for the suggestion, Stephan. I think that makes a lot of sense. We
should add something like the following method in the SplitEnumeratorContext.
{code}
{{void finishSplitsAssignment(int subtask)}}
{code}
> Simplify handling of 'NoMoreSplitsEvent'
>
>
> Key: FLINK-19265
> URL: https://issues.apache.org/jira/browse/FLINK-19265
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Major
> Fix For: 1.12.0
>
>
> Split assignment events are treated specially by the source API. Users do not
> create them directly but call methods on the contexts to assign splits.
> The {{NoMoreSplitsEvent}} is in contrast to that a custom user event and
> needs to be handled like a custom event, when sent by enumerators and
> received by the readers.
> That seems a bit confusing and inconsistent, given that NoMoreSplits is
> essential for all bounded stream use cases and is on the same level as the
> {{AddSplitEvent}}.
> I suggest that we treat "no more splits" similarly, by having either a custom
> method or a custom "SplitAssignment" on the context and reader.
> [~jqin] Curious what would be your take on this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19265) Simplify handling of 'NoMoreSplitsEvent'
[
https://issues.apache.org/jira/browse/FLINK-19265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197346#comment-17197346
]
Jiangjie Qin commented on FLINK-19265:
--
Thanks for the suggestion, Stephan. I think that makes a lot of sense. We
should add something like the following method in the SplitEnumeratorContext.
{code}
{{void finishSplitsAssignment(int subtask)}}
{code}
> Simplify handling of 'NoMoreSplitsEvent'
>
>
> Key: FLINK-19265
> URL: https://issues.apache.org/jira/browse/FLINK-19265
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Common
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Major
> Fix For: 1.12.0
>
>
> Split assignment events are treated specially by the source API. Users do not
> create them directly but call methods on the contexts to assign splits.
> The {{NoMoreSplitsEvent}} is in contrast to that a custom user event and
> needs to be handled like a custom event, when sent by enumerators and
> received by the readers.
> That seems a bit confusing and inconsistent, given that NoMoreSplits is
> essential for all bounded stream use cases and is on the same level as the
> {{AddSplitEvent}}.
> I suggest that we treat "no more splits" similarly, by having either a custom
> method or a custom "SplitAssignment" on the context and reader.
> [~jqin] Curious what would be your take on this.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #13404: [FLINK-16100][docs-zh]Translate "Reading & Writing Hive Tables" page …
flinkbot commented on pull request #13404: URL: https://github.com/apache/flink/pull/13404#issuecomment-693785870 ## CI report: * c9569f50a4707d5fbd27ff15e2745e078d92e39d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13398: [FLINK-18604][connectors/HBase] HBase ConnectorDescriptor can not work in Table API
flinkbot edited a comment on pull request #13398: URL: https://github.com/apache/flink/pull/13398#issuecomment-693384597 ## CI report: * 462b148cfa64be886adaa5d3bdd1b8b01ba258b2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6551) * 09f58aeb9ac653be17d8a9b5b6eaaee02e415ad2 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6576) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13402: [FLINK-19262][API/DataStream] Can not setParallelism for FLIP-27 source
flinkbot edited a comment on pull request #13402: URL: https://github.com/apache/flink/pull/13402#issuecomment-693424540 ## CI report: * 4b7f5b86b91caffdb8436913467d165254000745 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6556) * f4a438df036bada30853b6450931e5b116dce9a1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6575) * 83ff7dd5e54dca0b65e99cbf6409dbf6ae0428a4 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13397: [FLINK-19178][runtime] Extend managed memory weight/fraction interfaces for various use cases.
flinkbot edited a comment on pull request #13397: URL: https://github.com/apache/flink/pull/13397#issuecomment-693261990 ## CI report: * 1e0209f0df58ecdc3460d426262fc89d035b83f6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6550) * b8e95c022986da9f0b0d1b18e205f7cbc05f51ab UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13371: [FLINK-17879][python] update Python row serializer to support RowKind
flinkbot edited a comment on pull request #13371: URL: https://github.com/apache/flink/pull/13371#issuecomment-690084042 ## CI report: * 6aaa5136a260683572b62d428f54678d9454e0c1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6555) * 1743fdb906cf61fde264a942c0569187edd7b1d6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-19179) Implement the managed memory fraction calculation logic
[ https://issues.apache.org/jira/browse/FLINK-19179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xintong Song reassigned FLINK-19179: Assignee: Xintong Song > Implement the managed memory fraction calculation logic > --- > > Key: FLINK-19179 > URL: https://issues.apache.org/jira/browse/FLINK-19179 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Xintong Song >Assignee: Xintong Song >Priority: Major > Fix For: 1.12.0 > > > This also means migrating the batch operator use cases. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] meiminda353690383 commented on pull request #11911: [FLINK-16100][docs-zh]Translate "Reading & Writing Hive Tables" page …
meiminda353690383 commented on pull request #11911: URL: https://github.com/apache/flink/pull/11911#issuecomment-693783938 I have resubmitted a pull request https://github.com/apache/flink/pull/13404/commits This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] meiminda353690383 closed pull request #11911: [FLINK-16100][docs-zh]Translate "Reading & Writing Hive Tables" page …
meiminda353690383 closed pull request #11911: URL: https://github.com/apache/flink/pull/11911 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13404: [FLINK-16100][docs-zh]Translate "Reading & Writing Hive Tables" page …
flinkbot commented on pull request #13404: URL: https://github.com/apache/flink/pull/13404#issuecomment-693782320 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit c9569f50a4707d5fbd27ff15e2745e078d92e39d (Thu Sep 17 03:12:52 UTC 2020) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] meiminda353690383 opened a new pull request #13404: [FLINK-16100][docs-zh]Translate "Reading & Writing Hive Tables" page …
meiminda353690383 opened a new pull request #13404: URL: https://github.com/apache/flink/pull/13404 …of "Hive Integration" into Chinese ## What is the purpose of the change *(For example: This pull request makes task deployment go through the blob server, rather than through RPC. That way we avoid re-transferring them on each deployment (during recovery).)* ## Brief change log *(for example:)* - *The TaskInfo is stored in the blob store on job creation time as a persistent artifact* - *Deployments RPC transmits only the blob storage reference* - *TaskManagers retrieve the TaskInfo from the blob cache* ## Verifying this change *(Please pick either of the following options)* This change is a trivial rework / code cleanup without any test coverage. *(or)* This change is already covered by existing tests, such as *(please describe tests)*. *(or)* This change added tests and can be verified as follows: *(example:)* - *Added integration tests for end-to-end deployment with large payloads (100MB)* - *Extended integration test for recovery after master (JobManager) failure* - *Added test that validates that TaskInfo is transferred only once across recoveries* - *Manually verified the change by running a 4 node cluser with 2 JobManagers and 4 TaskManagers, a stateful streaming program, and killing one JobManager and two TaskManagers during the execution, verifying that recovery happens correctly.* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (yes / no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (yes / no) - The serializers: (yes / no / don't know) - The runtime per-record code paths (performance sensitive): (yes / no / don't know) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / no / don't know) - The S3 file system connector: (yes / no / don't know) ## Documentation - Does this pull request introduce a new feature? (yes / no) - If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong commented on pull request #13397: [FLINK-19178][runtime] Extend managed memory weight/fraction interfaces for various use cases.
xintongsong commented on pull request #13397: URL: https://github.com/apache/flink/pull/13397#issuecomment-693780307 Thanks for the review, @KarmaGYZ. Comments addressed. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #13397: [FLINK-19178][runtime] Extend managed memory weight/fraction interfaces for various use cases.
xintongsong commented on a change in pull request #13397:
URL: https://github.com/apache/flink/pull/13397#discussion_r489915944
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java
##
@@ -849,26 +843,23 @@ private static void
setManagedMemoryFractionForSlotSharingGroup(
}
private static void setManagedMemoryFractionForOperator(
- final ResourceSpec operatorResourceSpec,
final ResourceSpec groupResourceSpec,
final int operatorManagedMemoryWeight,
final int groupManagedMemoryWeight,
final StreamConfig operatorConfig) {
- final double managedMemoryFraction;
-
if (groupResourceSpec.equals(ResourceSpec.UNKNOWN)) {
- managedMemoryFraction = groupManagedMemoryWeight > 0
- ?
getFractionRoundedDown(operatorManagedMemoryWeight, groupManagedMemoryWeight)
- : 0.0;
+ operatorConfig.setManagedMemoryFraction(
+ groupManagedMemoryWeight > 0 ?
+
getFractionRoundedDown(operatorManagedMemoryWeight, groupManagedMemoryWeight) :
+ 0.0);
} else {
- final long groupManagedMemoryBytes =
groupResourceSpec.getManagedMemory().getBytes();
- managedMemoryFraction = groupManagedMemoryBytes > 0
- ?
getFractionRoundedDown(operatorResourceSpec.getManagedMemory().getBytes(),
groupManagedMemoryBytes)
- : 0.0;
+ // Supporting for fine grained resource specs is still
under developing.
+ // This branch should not be executed in production.
Not throwing exception for testing purpose.
+ LOG.error("Failed setting managed memory fractions. " +
+ " Operators may not be able to use
managed memory properly." +
+ " Calculating managed memory fractions
with fine grained resource spec is currently not supported.");
Review comment:
FYI, I've created FLINK-19267 to track the fraction calculation issue
for fine grained resource specs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] pyscala commented on a change in pull request #13402: [FLINK-19262][API/DataStream] Can not setParallelism for FLIP-27 source
pyscala commented on a change in pull request #13402:
URL: https://github.com/apache/flink/pull/13402#discussion_r489915990
##
File path:
flink-streaming-java/src/test/java/org/apache/flink/streaming/api/datastream/DataStreamSourceTest.java
##
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.datastream;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.mocks.MockSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+
+/**
+ * Unit test for {@link DataStreamSource}.
+ */
+public class DataStreamSourceTest {
+
+ /**
+* test Constructor for new Sources (FLIP-27).
+*/
+ @Test
+ public void testConstructor(){
Review comment:
Done.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-19267) Calculate managed memory fractions for fine grained resource specs
Xintong Song created FLINK-19267: Summary: Calculate managed memory fractions for fine grained resource specs Key: FLINK-19267 URL: https://issues.apache.org/jira/browse/FLINK-19267 Project: Flink Issue Type: Task Components: Runtime / Coordination Reporter: Xintong Song This is a follow-up issue of FLIP-141, to support managed memory calculation for fine grained resource specs w.r.t. various use cases, per the discussion [here|https://github.com/apache/flink/pull/13397#discussion_r489250595]. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13402: [FLINK-19262][API/DataStream] Can not setParallelism for FLIP-27 source
flinkbot edited a comment on pull request #13402: URL: https://github.com/apache/flink/pull/13402#issuecomment-693424540 ## CI report: * 4b7f5b86b91caffdb8436913467d165254000745 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6556) * f4a438df036bada30853b6450931e5b116dce9a1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6575) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13398: [FLINK-18604][connectors/HBase] HBase ConnectorDescriptor can not work in Table API
flinkbot edited a comment on pull request #13398: URL: https://github.com/apache/flink/pull/13398#issuecomment-693384597 ## CI report: * 462b148cfa64be886adaa5d3bdd1b8b01ba258b2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6551) * 09f58aeb9ac653be17d8a9b5b6eaaee02e415ad2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #13397: [FLINK-19178][runtime] Extend managed memory weight/fraction interfaces for various use cases.
xintongsong commented on a change in pull request #13397:
URL: https://github.com/apache/flink/pull/13397#discussion_r489912221
##
File path:
flink-core/src/main/java/org/apache/flink/configuration/TaskManagerOptions.java
##
@@ -401,6 +403,22 @@
.withDescription("Fraction of Total Flink Memory to be
used as Managed Memory, if Managed Memory size is not"
+ " explicitly specified.");
+ /**
+* Weights of managed memory consumers.
+*/
+ // Do not advertise this option until the feature is completed.
+ @Documentation.ExcludeFromDocumentation
+ public static final ConfigOption>
MANAGED_MEMORY_CONSUMER_WEIGHTS =
+ key("taskmanager.memory.managed.consumer-weights")
+ .mapType()
+ .defaultValue(new HashMap() {{
+ put("DATAPROC", "70");
+ put("PYTHON", "30");
Review comment:
Since map type `ConfigOption` only accepts `Map`, I'll
use a static inner class with static final string constants for the consumer
names, to avoid converting enum values to strings.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11911: [FLINK-16100][docs-zh]Translate "Reading & Writing Hive Tables" page …
flinkbot edited a comment on pull request #11911: URL: https://github.com/apache/flink/pull/11911#issuecomment-619396940 ## CI report: * 52876598cd1fc4df85c3bc8f15063897be98feb6 UNKNOWN * 77d2fcaf3976761d009dd25e657601222521388e UNKNOWN * cb944914a96f89f90317456c5d153340a0e1c9be Travis: [SUCCESS](https://travis-ci.com/github/flink-ci/flink/builds/164165034) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=681) * b065253babc77a1d1d53798b1e2e76baede04afc UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-19258) Fix the wrong example of "csv.line-delimiter" in CSV documentation
[
https://issues.apache.org/jira/browse/FLINK-19258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu reassigned FLINK-19258:
---
Assignee: dalongliu
> Fix the wrong example of "csv.line-delimiter" in CSV documentation
> --
>
> Key: FLINK-19258
> URL: https://issues.apache.org/jira/browse/FLINK-19258
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Formats (JSON, Avro, Parquet, ORC,
> SequenceFile), Table SQL / Ecosystem
>Reporter: Jark Wu
>Assignee: dalongliu
>Priority: Major
>
> In the documentation [1], it says:
> {code}
> 'csv.line-delimiter' = U&'\\000D' specifies the to use carriage return \r as
> line delimiter.
> 'csv.line-delimiter' = U&'\\000A' specifies the to use line feed \n as line
> delimiter.
> {code}
> This is wrong. They should be U&'\000D' and U&'\000A'.
> [1]:
> https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connectors/formats/csv.html#csv-line-delimiter
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19098) Make Rowdata converters public
[ https://issues.apache.org/jira/browse/FLINK-19098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197335#comment-17197335 ] Jark Wu commented on FLINK-19098: - Thanks for the reminder and sorry for the delay [~Brian Zhou]. There are too much things on my list. Will review it today or tomorrow. > Make Rowdata converters public > -- > > Key: FLINK-19098 > URL: https://issues.apache.org/jira/browse/FLINK-19098 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Affects Versions: 1.11.0 >Reporter: Brian Zhou >Assignee: Brian Zhou >Priority: Major > Labels: pull-request-available > > Some of the Rowdata converters(SeDer between Rowdata and format objects like > GenericRecord/JsonNode) are private or package-private (like Json), this is > not easy for other third-party connector projects to utilize to implement its > own format factory in Table API. > For example, Pravega connector is now developing a schema-registry-based > format factory. The [Pravega schema > registry|https://github.com/pravega/schema-registry] is a rest service > similar with confluent registry , but it can help to serialize/deserialize > json/avro/protobuf/custom format data. It will help a lot if these converters > are public. > Noticed in FLINK-16048, we have already moved the avro converters out and > made them public. Similarly, it should be safe to make at least json and csv > format converters public. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13402: [FLINK-19262][API/DataStream] Can not setParallelism for FLIP-27 source
flinkbot edited a comment on pull request #13402: URL: https://github.com/apache/flink/pull/13402#issuecomment-693424540 ## CI report: * 4b7f5b86b91caffdb8436913467d165254000745 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6556) * f4a438df036bada30853b6450931e5b116dce9a1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #13397: [FLINK-19178][runtime] Extend managed memory weight/fraction interfaces for various use cases.
xintongsong commented on a change in pull request #13397:
URL: https://github.com/apache/flink/pull/13397#discussion_r489903233
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamNode.java
##
@@ -49,6 +48,8 @@
@Internal
public class StreamNode implements Serializable {
+ public static final int DEFAULT_MANAGED_MEMORY_WEIGHT = 1;
Review comment:
This does not really matter since the constant is soon removed in the
next commit.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] pyscala removed a comment on pull request #13398: [FLINK-18604][connectors/HBase] HBase ConnectorDescriptor can not work in Table API
pyscala removed a comment on pull request #13398: URL: https://github.com/apache/flink/pull/13398#issuecomment-693351525 cc @JingsongLi , Looking forward to your reply This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] leonardBang commented on pull request #13128: [FLINK-18795][hbase] Support for HBase 2
leonardBang commented on pull request #13128: URL: https://github.com/apache/flink/pull/13128#issuecomment-693772490 > We will run some further cluster tests with this and after that I would like to merge it tomorrow if no other objections. > @leonardBang do you have time to review it? Sorry for the late reply, I was on vacation, and I'll review it today. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #13397: [FLINK-19178][runtime] Extend managed memory weight/fraction interfaces for various use cases.
xintongsong commented on a change in pull request #13397:
URL: https://github.com/apache/flink/pull/13397#discussion_r489895561
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java
##
@@ -177,11 +178,13 @@ private JobGraph createJobGraph() {
setSlotSharingAndCoLocation();
+ // For now, only consider managed memory for batch algorithms.
+ // TODO: extend managed memory fraction calculations w.r.t.
various managed memory use cases.
setManagedMemoryFraction(
Collections.unmodifiableMap(jobVertices),
Collections.unmodifiableMap(vertexConfigs),
Collections.unmodifiableMap(chainedConfigs),
- id ->
streamGraph.getStreamNode(id).getManagedMemoryWeight());
+ id ->
streamGraph.getStreamNode(id).getManagedMemoryUseCaseWeights().getOrDefault(ManagedMemoryUseCase.BATCH_OP,
0));
Review comment:
Not sure about this. I think we don't really want a default weight here.
The purpose for this default is to avoid NPE when performing `mapToInt` and
`sum` in `setManagedMemoryFractionForSlotSharingGroup`.
Maybe we should not set default value here and handles the `null` values in
`setManagedMemoryFractionForSlotSharingGroup`. I think it should be
straightforward to convert `null` to `0` for a `sum` processing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-16947) ArtifactResolutionException: Could not transfer artifact. Entry [...] has not been leased from this pool
[
https://issues.apache.org/jira/browse/FLINK-16947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197332#comment-17197332
]
Dian Fu commented on FLINK-16947:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=6568=logs=245e1f2e-ba5b-5570-d689-25ae21e5302f=02d88c1a-f1b3-5a8c-4b4a-cf43c70f99e1
> ArtifactResolutionException: Could not transfer artifact. Entry [...] has
> not been leased from this pool
> -
>
> Key: FLINK-16947
> URL: https://issues.apache.org/jira/browse/FLINK-16947
> Project: Flink
> Issue Type: Bug
> Components: Build System / Azure Pipelines
>Reporter: Piotr Nowojski
>Assignee: Robert Metzger
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/rmetzger/Flink/_build/results?buildId=6982=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> Build of flink-metrics-availability-test failed with:
> {noformat}
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.22.1:test (end-to-end-tests)
> on project flink-metrics-availability-test: Unable to generate classpath:
> org.apache.maven.artifact.resolver.ArtifactResolutionException: Could not
> transfer artifact org.apache.maven.surefire:surefire-grouper:jar:2.22.1
> from/to google-maven-central
> (https://maven-central-eu.storage-download.googleapis.com/maven2/): Entry
> [id:13][route:{s}->https://maven-central-eu.storage-download.googleapis.com:443][state:null]
> has not been leased from this pool
> [ERROR] org.apache.maven.surefire:surefire-grouper:jar:2.22.1
> [ERROR]
> [ERROR] from the specified remote repositories:
> [ERROR] google-maven-central
> (https://maven-central-eu.storage-download.googleapis.com/maven2/,
> releases=true, snapshots=false),
> [ERROR] apache.snapshots (https://repository.apache.org/snapshots,
> releases=false, snapshots=true)
> [ERROR] Path to dependency:
> [ERROR] 1) dummy:dummy:jar:1.0
> [ERROR] 2) org.apache.maven.surefire:surefire-junit47:jar:2.22.1
> [ERROR] 3) org.apache.maven.surefire:common-junit48:jar:2.22.1
> [ERROR] 4) org.apache.maven.surefire:surefire-grouper:jar:2.22.1
> [ERROR] -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re-run Maven using the -X switch to enable full debug logging.
> [ERROR]
> [ERROR] For more information about the errors and possible solutions, please
> read the following articles:
> [ERROR] [Help 1]
> http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
> [ERROR]
> [ERROR] After correcting the problems, you can resume the build with the
> command
> [ERROR] mvn -rf :flink-metrics-availability-test
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18222) "Avro Confluent Schema Registry nightly end-to-end test" unstable with "Kafka cluster did not start after 120 seconds"
[
https://issues.apache.org/jira/browse/FLINK-18222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197331#comment-17197331
]
Dian Fu commented on FLINK-18222:
-
Instance on 1.11:
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=6568=logs=91bf6583-3fb2-592f-e4d4-d79d79c3230a=03dbd840-5430-533d-d1a7-05d0ebe03873]
[~rmetzger] What about also back-porting the fix to 1.11?
> "Avro Confluent Schema Registry nightly end-to-end test" unstable with "Kafka
> cluster did not start after 120 seconds"
> --
>
> Key: FLINK-18222
> URL: https://issues.apache.org/jira/browse/FLINK-18222
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=3045=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> {code}
> 2020-06-09T15:16:48.1427795Z
> ==
> 2020-06-09T15:16:48.1428609Z Running 'Avro Confluent Schema Registry nightly
> end-to-end test'
> 2020-06-09T15:16:48.1429204Z
> ==
> 2020-06-09T15:16:48.1438117Z TEST_DATA_DIR:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-48143298170
> 2020-06-09T15:16:48.2985167Z Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-1.11-SNAPSHOT-bin/flink-1.11-SNAPSHOT
> 2020-06-09T15:16:48.3157575Z Downloading Kafka from
> https://archive.apache.org/dist/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz
> 2020-06-09T15:16:48.3214487Z % Total% Received % Xferd Average Speed
> TimeTime Time Current
> 2020-06-09T15:16:48.3215154Z Dload Upload
> Total SpentLeft Speed
> 2020-06-09T15:16:48.3215597Z
> 2020-06-09T15:16:48.3528820Z 0 00 00 0 0 0
> --:--:-- --:--:-- --:--:-- 0
> 2020-06-09T15:16:49.3421526Z 0 00 00 0 0 0
> --:--:-- --:--:-- --:--:-- 0
> 2020-06-09T15:16:50.3415678Z 8 35.8M8 2960k0 0 2896k 0
> 0:00:12 0:00:01 0:00:11 2896k
> 2020-06-09T15:16:51.3406836Z 23 35.8M 23 8544k0 0 4226k 0
> 0:00:08 0:00:02 0:00:06 4225k
> 2020-06-09T15:16:51.6553485Z 70 35.8M 70 25.2M0 0 8550k 0
> 0:00:04 0:00:03 0:00:01 8548k
> 2020-06-09T15:16:51.6555606Z 100 35.8M 100 35.8M0 0 10.7M 0
> 0:00:03 0:00:03 --:--:-- 10.7M
> 2020-06-09T15:16:51.9818041Z Downloading confluent from
> http://packages.confluent.io/archive/3.2/confluent-oss-3.2.0-2.11.tar.gz
> 2020-06-09T15:16:51.9880242Z % Total% Received % Xferd Average Speed
> TimeTime Time Current
> 2020-06-09T15:16:51.9880983Z Dload Upload
> Total SpentLeft Speed
> 2020-06-09T15:16:51.9914252Z
> 2020-06-09T15:16:52.3398614Z 0 00 00 0 0 0
> --:--:-- --:--:-- --:--:-- 0
> 2020-06-09T15:16:53.3399552Z 9 398M9 39.5M0 0 111M 0
> 0:00:03 --:--:-- 0:00:03 111M
> 2020-06-09T15:16:53.9149276Z 47 398M 47 188M0 0 139M 0
> 0:00:02 0:00:01 0:00:01 138M
> 2020-06-09T15:16:53.9150980Z 100 398M 100 398M0 0 206M 0
> 0:00:01 0:00:01 --:--:-- 206M
> 2020-06-09T15:17:04.3565942Z Waiting for broker...
> 2020-06-09T15:17:12.4215170Z Waiting for broker...
> 2020-06-09T15:17:14.3012835Z Waiting for broker...
> 2020-06-09T15:17:16.1965074Z Waiting for broker...
> 2020-06-09T15:17:18.1102274Z Waiting for broker...
> 2020-06-09T15:17:19.9929632Z Waiting for broker...
> 2020-06-09T15:17:21.8607172Z Waiting for broker...
> 2020-06-09T15:17:23.7802949Z Waiting for broker...
> 2020-06-09T15:17:25.6695260Z Waiting for broker...
> 2020-06-09T15:17:27.5536417Z Waiting for broker...
> 2020-06-09T15:17:29.4327778Z Waiting for broker...
> 2020-06-09T15:17:31.3203091Z Waiting for broker...
> 2020-06-09T15:17:33.1987150Z Waiting for broker...
> 2020-06-09T15:17:35.0694860Z Waiting for broker...
> 2020-06-09T15:17:36.9595576Z Waiting for broker...
> 2020-06-09T15:17:38.9243558Z Waiting for broker...
> 2020-06-09T15:17:40.7984064Z Waiting for broker...
> 2020-06-09T15:17:42.6676095Z Waiting for broker...
> 2020-06-09T15:17:44.5628797Z Waiting for broker...
> 2020-06-09T15:17:46.4374532Z Waiting for broker...
> 2020-06-09T15:17:48.3086761Z Waiting for broker...
>
[GitHub] [flink] JingsongLi commented on a change in pull request #13402: [FLINK-19262][API/DataStream] Can not setParallelism for FLIP-27 source
JingsongLi commented on a change in pull request #13402:
URL: https://github.com/apache/flink/pull/13402#discussion_r489892921
##
File path:
flink-streaming-java/src/test/java/org/apache/flink/streaming/api/datastream/DataStreamSourceTest.java
##
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.datastream;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.mocks.MockSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+
+/**
+ * Unit test for {@link DataStreamSource}.
+ */
+public class DataStreamSourceTest {
+
+ /**
+* test Constructor for new Sources (FLIP-27).
+*/
+ @Test
+ public void testConstructor(){
Review comment:
@pyscala I think Stephan means: `testConstructor() {` instead of
`testConstructor(){`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] pyscala commented on a change in pull request #13402: [FLINK-19262][API/DataStream] Can not setParallelism for FLIP-27 source
pyscala commented on a change in pull request #13402:
URL: https://github.com/apache/flink/pull/13402#discussion_r489892331
##
File path:
flink-streaming-java/src/test/java/org/apache/flink/streaming/api/datastream/DataStreamSourceTest.java
##
@@ -0,0 +1,53 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.api.datastream;
+
+import org.apache.flink.api.common.eventtime.WatermarkStrategy;
+import org.apache.flink.api.connector.source.Boundedness;
+import org.apache.flink.api.connector.source.mocks.MockSource;
+import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
+
+import org.junit.Test;
+
+import static org.junit.Assert.assertEquals;
+
+/**
+ * Unit test for {@link DataStreamSource}.
+ */
+public class DataStreamSourceTest {
+
+ /**
+* test Constructor for new Sources (FLIP-27).
+*/
+ @Test
+ public void testConstructor(){
Review comment:
Thanks for your reminder.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13313: [FLINK-18722][mesos] Migrate MesosResourceManager to the new MesosResourceManagerDriver
flinkbot edited a comment on pull request #13313: URL: https://github.com/apache/flink/pull/13313#issuecomment-686326763 ## CI report: * 33ea3fa14ddf2ead07a9f0b29ad60a057a474a75 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6541) * c5d95b9560a4288f9a8248c66bef45531c685e56 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6574) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #13397: [FLINK-19178][runtime] Extend managed memory weight/fraction interfaces for various use cases.
xintongsong commented on a change in pull request #13397:
URL: https://github.com/apache/flink/pull/13397#discussion_r489890377
##
File path: flink-core/src/main/java/org/apache/flink/api/dag/Transformation.java
##
@@ -139,12 +142,12 @@ public static int getNewNodeId() {
private ResourceSpec preferredResources = ResourceSpec.DEFAULT;
/**
-* This weight indicates how much this transformation relies on managed
memory, so that
-* transformation highly relies on managed memory would be able to
acquire more managed
-* memory in runtime (linear association). Note that it only works in
cases of UNKNOWN
-* resources.
+* Each entry in this map represents a use case that this
transformation needs managed memory for. The key of the
+* entry indicates the use case, while the value is the
use-case-specific weight for this transformation. Managed
+* memory reserved for an OPERATOR scope use case will be shared by all
the declaring transformations within a slot
+* according to this weight. For SLOT scope use cases, the weights are
ignored.
*/
- private int managedMemoryWeight = DEFAULT_MANAGED_MEMORY_WEIGHT;
+ private final Map
managedMemoryUseCaseWeights = new HashMap<>();
Review comment:
`ROCKSDB` and `PYTHON` are legal. The map accepts both `OPERATOR` and
`SLOT` scope use cases. But only for `OPERATOR` scope use cases do the weights
(values of the map) matter.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #13397: [FLINK-19178][runtime] Extend managed memory weight/fraction interfaces for various use cases.
xintongsong commented on a change in pull request #13397:
URL: https://github.com/apache/flink/pull/13397#discussion_r489890553
##
File path: flink-core/src/main/java/org/apache/flink/api/dag/Transformation.java
##
@@ -139,12 +142,12 @@ public static int getNewNodeId() {
private ResourceSpec preferredResources = ResourceSpec.DEFAULT;
/**
-* This weight indicates how much this transformation relies on managed
memory, so that
-* transformation highly relies on managed memory would be able to
acquire more managed
-* memory in runtime (linear association). Note that it only works in
cases of UNKNOWN
-* resources.
+* Each entry in this map represents a use case that this
transformation needs managed memory for. The key of the
+* entry indicates the use case, while the value is the
use-case-specific weight for this transformation. Managed
+* memory reserved for an OPERATOR scope use case will be shared by all
the declaring transformations within a slot
+* according to this weight. For SLOT scope use cases, the weights are
ignored.
*/
- private int managedMemoryWeight = DEFAULT_MANAGED_MEMORY_WEIGHT;
+ private final Map
managedMemoryUseCaseWeights = new HashMap<>();
Review comment:
I'll try to clarify a bit in the JavaDocs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] lsyldliu commented on pull request #11830: [FLINK-17096] [table] Support state ttl for Mini-Batch Group Agg using StateTtlConfig
lsyldliu commented on pull request #11830: URL: https://github.com/apache/flink/pull/11830#issuecomment-693767869 @wuchong I have updated the code, azure test passed, you can help review it again! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] xintongsong commented on a change in pull request #13397: [FLINK-19178][runtime] Extend managed memory weight/fraction interfaces for various use cases.
xintongsong commented on a change in pull request #13397:
URL: https://github.com/apache/flink/pull/13397#discussion_r489886220
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/graph/StreamingJobGraphGenerator.java
##
@@ -849,26 +843,23 @@ private static void
setManagedMemoryFractionForSlotSharingGroup(
}
private static void setManagedMemoryFractionForOperator(
- final ResourceSpec operatorResourceSpec,
final ResourceSpec groupResourceSpec,
final int operatorManagedMemoryWeight,
final int groupManagedMemoryWeight,
final StreamConfig operatorConfig) {
- final double managedMemoryFraction;
-
if (groupResourceSpec.equals(ResourceSpec.UNKNOWN)) {
- managedMemoryFraction = groupManagedMemoryWeight > 0
- ?
getFractionRoundedDown(operatorManagedMemoryWeight, groupManagedMemoryWeight)
- : 0.0;
+ operatorConfig.setManagedMemoryFraction(
+ groupManagedMemoryWeight > 0 ?
+
getFractionRoundedDown(operatorManagedMemoryWeight, groupManagedMemoryWeight) :
+ 0.0);
} else {
- final long groupManagedMemoryBytes =
groupResourceSpec.getManagedMemory().getBytes();
- managedMemoryFraction = groupManagedMemoryBytes > 0
- ?
getFractionRoundedDown(operatorResourceSpec.getManagedMemory().getBytes(),
groupManagedMemoryBytes)
- : 0.0;
+ // Supporting for fine grained resource specs is still
under developing.
+ // This branch should not be executed in production.
Not throwing exception for testing purpose.
+ LOG.error("Failed setting managed memory fractions. " +
+ " Operators may not be able to use
managed memory properly." +
+ " Calculating managed memory fractions
with fine grained resource spec is currently not supported.");
Review comment:
Adding a `TODO` message and creating a follow-up issue sound good to me.
The problem of throwing an exception is that, existing test cases using fine
grained resource specs will fail on generating job graphs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-19264) Identical jobs cannot be run concurrently
[
https://issues.apache.org/jira/browse/FLINK-19264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197326#comment-17197326
]
Yangze Guo commented on FLINK-19264:
Thanks for the analysis and suggestion, [~chesnay], and [~sewen]. I think
including the JobID will be good enough. Another solution could be adding a
short random suffix to the ExecutionAttemptID, just like what we do in
FLINK-17579.
I'd like to take this ticket when we figure out there is no other bugs that
cause this issue.
> Identical jobs cannot be run concurrently
> -
>
> Key: FLINK-19264
> URL: https://issues.apache.org/jira/browse/FLINK-19264
> Project: Flink
> Issue Type: Bug
>Affects Versions: 1.12.0
>Reporter: Aljoscha Krettek
>Priority: Blocker
>
> While working on FLINK-19123 I noticed that jobs often fail on
> {{MiniCluster}} when multiple jobs are running at the same time.
> I created a reproducer here:
> https://github.com/aljoscha/flink/tree/flink-19123-fix-test-stream-env-alternative.
> You can run {{MiniClusterConcurrencyITCase}} to see the problem in action.
> Sometimes the test will succeed, sometimes it will fail with
> {code}
> java.util.concurrent.ExecutionException:
> org.apache.flink.runtime.client.JobExecutionException: Job execution failed.
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
> at
> org.apache.flink.test.example.MiniClusterConcurrencyITCase.submitConcurrently(MiniClusterConcurrencyITCase.java:60)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at
> com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:69)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
> at
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:220)
> at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:53)
> Caused by: org.apache.flink.runtime.client.JobExecutionException: Job
> execution failed.
> at
> org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:147)
> at
> org.apache.flink.runtime.minicluster.MiniClusterJobClient.lambda$getJobExecutionResult$2(MiniClusterJobClient.java:107)
> at
> java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:616)
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:591)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> at
> org.apache.flink.runtime.rpc.akka.AkkaInvocationHandler.lambda$invokeRpc$0(AkkaInvocationHandler.java:229)
> at
> java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:774)
> at
> java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
> at
> java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:1975)
> at
>
[GitHub] [flink] flinkbot edited a comment on pull request #13313: [FLINK-18722][mesos] Migrate MesosResourceManager to the new MesosResourceManagerDriver
flinkbot edited a comment on pull request #13313: URL: https://github.com/apache/flink/pull/13313#issuecomment-686326763 ## CI report: * 33ea3fa14ddf2ead07a9f0b29ad60a057a474a75 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6541) * c5d95b9560a4288f9a8248c66bef45531c685e56 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17295) Refactor the ExecutionAttemptID to consist of ExecutionVertexID and attemptNumber
[ https://issues.apache.org/jira/browse/FLINK-17295?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197322#comment-17197322 ] Yangze Guo commented on FLINK-17295: Thanks for the pointer, [~sewen]. I think including the JobID will be good enough. Another solution could be adding a short random suffix to the ExecutionAttemptID, just like what we do in FLINK-17579. WDYT? > Refactor the ExecutionAttemptID to consist of ExecutionVertexID and > attemptNumber > - > > Key: FLINK-17295 > URL: https://issues.apache.org/jira/browse/FLINK-17295 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Coordination >Reporter: Yangze Guo >Assignee: Yangze Guo >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > Make the ExecutionAttemptID being composed of (ExecutionVertexID, > attemptNumber). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] xintongsong commented on pull request #13313: [FLINK-18722][mesos] Migrate MesosResourceManager to the new MesosResourceManagerDriver
xintongsong commented on pull request #13313: URL: https://github.com/apache/flink/pull/13313#issuecomment-693760238 Rebase to resolve conflicts. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13403: [Hotfix][Docs] Update rest_v1_dispathcer.html
flinkbot edited a comment on pull request #13403: URL: https://github.com/apache/flink/pull/13403#issuecomment-693610429 ## CI report: * 0f52d8c1b46c31f08e710a157822b93d15027b48 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6571) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6566) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-19184) Add Batch Physical Pandas Group Aggregate Rule and RelNode
[ https://issues.apache.org/jira/browse/FLINK-19184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu reassigned FLINK-19184: --- Assignee: Huang Xingbo > Add Batch Physical Pandas Group Aggregate Rule and RelNode > -- > > Key: FLINK-19184 > URL: https://issues.apache.org/jira/browse/FLINK-19184 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Huang Xingbo >Assignee: Huang Xingbo >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > Add Batch Physical Pandas Group Aggregate Rule and RelNode -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-19186) Add Python building blocks to make sure the basic functionality of Pandas Batch Group Aggregation could work
[ https://issues.apache.org/jira/browse/FLINK-19186?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu reassigned FLINK-19186: --- Assignee: Huang Xingbo > Add Python building blocks to make sure the basic functionality of Pandas > Batch Group Aggregation could work > > > Key: FLINK-19186 > URL: https://issues.apache.org/jira/browse/FLINK-19186 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Huang Xingbo >Assignee: Huang Xingbo >Priority: Major > Fix For: 1.12.0 > > > We need to add a few Python building blocks such as > PandasAggregateFunctionOperation, PythonAggregateFunction, etc for Pandas > Aggregation execution. > This PR makes sure that a basic end to end Pandas Batch Group Aggregation > could be executed. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-19173) Add Pandas Batch Group Aggregation Function Operator
[ https://issues.apache.org/jira/browse/FLINK-19173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu closed FLINK-19173. --- Resolution: Fixed master: 7925647e3ae67a0a8914b4eb1644c318023eeb88 > Add Pandas Batch Group Aggregation Function Operator > > > Key: FLINK-19173 > URL: https://issues.apache.org/jira/browse/FLINK-19173 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Huang Xingbo >Assignee: Huang Xingbo >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > Add Pandas Batch Group Aggregation Function Operator -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] dianfu closed pull request #13369: [FLINK-19173][python] Introduce BatchArrowPythonGroupAggregateFunctionOperator
dianfu closed pull request #13369: URL: https://github.com/apache/flink/pull/13369 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19098) Make Rowdata converters public
[ https://issues.apache.org/jira/browse/FLINK-19098?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197312#comment-17197312 ] Brian Zhou commented on FLINK-19098: I have raised a PR and left open for a week. It has passed the Azure build, can [~jark] and [~danny0405] have a review for that? Thanks > Make Rowdata converters public > -- > > Key: FLINK-19098 > URL: https://issues.apache.org/jira/browse/FLINK-19098 > Project: Flink > Issue Type: Improvement > Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table > SQL / Ecosystem >Affects Versions: 1.11.0 >Reporter: Brian Zhou >Assignee: Brian Zhou >Priority: Major > Labels: pull-request-available > > Some of the Rowdata converters(SeDer between Rowdata and format objects like > GenericRecord/JsonNode) are private or package-private (like Json), this is > not easy for other third-party connector projects to utilize to implement its > own format factory in Table API. > For example, Pravega connector is now developing a schema-registry-based > format factory. The [Pravega schema > registry|https://github.com/pravega/schema-registry] is a rest service > similar with confluent registry , but it can help to serialize/deserialize > json/avro/protobuf/custom format data. It will help a lot if these converters > are public. > Noticed in FLINK-16048, we have already moved the avro converters out and > made them public. Similarly, it should be safe to make at least json and csv > format converters public. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13357: [FLINK-19165] Refactor the UnilateralSortMerger
flinkbot edited a comment on pull request #13357: URL: https://github.com/apache/flink/pull/13357#issuecomment-689034696 ## CI report: * 756d84832df46992d406a579bd154dfc92c83e37 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6542) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13375: [FLINK-18815][filesystems][tests] Join SafetyNetCloseableRegistry's reaper thread before assertion
flinkbot edited a comment on pull request #13375: URL: https://github.com/apache/flink/pull/13375#issuecomment-691410175 ## CI report: * c10af2d6ed4136ff9a8ae30ab0841343635bf0f4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6468) * 0e3916e68d38b432ebc33248d3a1ce7f7213b1f8 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6572) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13403: [Hotfix][Docs] Update rest_v1_dispathcer.html
flinkbot edited a comment on pull request #13403: URL: https://github.com/apache/flink/pull/13403#issuecomment-693610429 ## CI report: * 0f52d8c1b46c31f08e710a157822b93d15027b48 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6571) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6566) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13375: [FLINK-18815][filesystems][tests] Join SafetyNetCloseableRegistry's reaper thread before assertion
flinkbot edited a comment on pull request #13375: URL: https://github.com/apache/flink/pull/13375#issuecomment-691410175 ## CI report: * c10af2d6ed4136ff9a8ae30ab0841343635bf0f4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6468) * 0e3916e68d38b432ebc33248d3a1ce7f7213b1f8 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13403: [Hotfix][Docs] Update rest_v1_dispathcer.html
flinkbot edited a comment on pull request #13403: URL: https://github.com/apache/flink/pull/13403#issuecomment-693610429 ## CI report: * 0f52d8c1b46c31f08e710a157822b93d15027b48 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6566) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6571) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangxlong commented on pull request #13403: [Hotfix][Docs] Update rest_v1_dispathcer.html
wangxlong commented on pull request #13403: URL: https://github.com/apache/flink/pull/13403#issuecomment-693720997 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13357: [FLINK-19165] Refactor the UnilateralSortMerger
flinkbot edited a comment on pull request #13357: URL: https://github.com/apache/flink/pull/13357#issuecomment-689034696 ## CI report: * f48774d7b5e0a14f1e7270591481cc2f83316854 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6569) * 756d84832df46992d406a579bd154dfc92c83e37 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6542) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13357: [FLINK-19165] Refactor the UnilateralSortMerger
flinkbot edited a comment on pull request #13357: URL: https://github.com/apache/flink/pull/13357#issuecomment-689034696 ## CI report: * 87c7f63785dda33d0a9bd210ff4d49962abbe638 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6564) * f48774d7b5e0a14f1e7270591481cc2f83316854 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6569) * 756d84832df46992d406a579bd154dfc92c83e37 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13392: [FLINK-18713][table-planner-blink] Change duration configOption to duration type
flinkbot edited a comment on pull request #13392: URL: https://github.com/apache/flink/pull/13392#issuecomment-692540614 ## CI report: * f3af04d02472f93c5d49e0262ce7957cdd910707 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6565) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13228: [FLINK-19026][network] Improve threading model of CheckpointBarrierUnaligner
flinkbot edited a comment on pull request #13228: URL: https://github.com/apache/flink/pull/13228#issuecomment-679099456 ## CI report: * d141f10ef060063162dc073b1cd66729f5f75a3b UNKNOWN * 89b0550c7b223f304b67a6f64dfc4252509eb76e UNKNOWN * 834c28da93d4b24e562e017979b483e288db02a2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6570) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13357: [FLINK-19165] Refactor the UnilateralSortMerger
flinkbot edited a comment on pull request #13357: URL: https://github.com/apache/flink/pull/13357#issuecomment-689034696 ## CI report: * 87c7f63785dda33d0a9bd210ff4d49962abbe638 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6564) * f48774d7b5e0a14f1e7270591481cc2f83316854 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6569) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13228: [FLINK-19026][network] Improve threading model of CheckpointBarrierUnaligner
flinkbot edited a comment on pull request #13228: URL: https://github.com/apache/flink/pull/13228#issuecomment-679099456 ## CI report: * d141f10ef060063162dc073b1cd66729f5f75a3b UNKNOWN * 89b0550c7b223f304b67a6f64dfc4252509eb76e UNKNOWN * a1fb1c3114305ed5fce898dfcc03abed4d6963d4 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6539) * 834c28da93d4b24e562e017979b483e288db02a2 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13403: [Hotfix][Docs] Update rest_v1_dispathcer.html
flinkbot edited a comment on pull request #13403: URL: https://github.com/apache/flink/pull/13403#issuecomment-693610429 ## CI report: * 0f52d8c1b46c31f08e710a157822b93d15027b48 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6566) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13357: [FLINK-19165] Refactor the UnilateralSortMerger
flinkbot edited a comment on pull request #13357: URL: https://github.com/apache/flink/pull/13357#issuecomment-689034696 ## CI report: * 05cef2abaec734dddea631576862d7c374e320c7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6561) * 87c7f63785dda33d0a9bd210ff4d49962abbe638 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6564) * f48774d7b5e0a14f1e7270591481cc2f83316854 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6569) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13357: [FLINK-19165] Refactor the UnilateralSortMerger
flinkbot edited a comment on pull request #13357: URL: https://github.com/apache/flink/pull/13357#issuecomment-689034696 ## CI report: * 54ee6987c7054a42fa52d04ec61fae74a816792c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6348) * 05cef2abaec734dddea631576862d7c374e320c7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6561) * 87c7f63785dda33d0a9bd210ff4d49962abbe638 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6564) * f48774d7b5e0a14f1e7270591481cc2f83316854 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6569) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11830: [FLINK-17096] [table] Support state ttl for Mini-Batch Group Agg using StateTtlConfig
flinkbot edited a comment on pull request #11830: URL: https://github.com/apache/flink/pull/11830#issuecomment-616574921 ## CI report: * e1eaf8cc2cd85ba3bf7ee684663001783809c719 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6559) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-15467) Should wait for the end of the source thread during the Task cancellation
[
https://issues.apache.org/jira/browse/FLINK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197216#comment-17197216
]
Jun Qin commented on FLINK-15467:
-
Great! Thanks [~AHeise]
> Should wait for the end of the source thread during the Task cancellation
> -
>
> Key: FLINK-15467
> URL: https://issues.apache.org/jira/browse/FLINK-15467
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.9.0, 1.9.1, 1.10.1
>Reporter: ming li
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0, 1.10.3, 1.11.3
>
>
> In the new mailBox model, SourceStreamTask starts a source thread to run
> user methods, and the current execution thread will block on mailbox.takeMail
> (). When a task cancels, the TaskCanceler thread will cancel the task and
> interrupt the execution thread. Therefore, the execution thread of
> SourceStreamTask will throw InterruptedException, then cancel the task again,
> and throw an exception.
> {code:java}
> //代码占位符
> @Override
> protected void performDefaultAction(ActionContext context) throws Exception {
>// Against the usual contract of this method, this implementation is not
> step-wise but blocking instead for
>// compatibility reasons with the current source interface (source
> functions run as a loop, not in steps).
>sourceThread.start();
>// We run an alternative mailbox loop that does not involve default
> actions and synchronizes around actions.
>try {
> runAlternativeMailboxLoop();
>} catch (Exception mailboxEx) {
> // We cancel the source function if some runtime exception escaped the
> mailbox.
> if (!isCanceled()) {
> cancelTask();
> }
> throw mailboxEx;
>}
>sourceThread.join();
>if (!isFinished) {
> sourceThread.checkThrowSourceExecutionException();
>}
>context.allActionsCompleted();
> }
> {code}
> When all tasks of this TaskExecutor are canceled, the blob file will be
> cleaned up. But the real source thread is not finished at this time, which
> will cause a ClassNotFoundException when loading a new class. In this case,
> the source thread may not be able to properly clean up and release resources
> (such as closing child threads, cleaning up local files, etc.). Therefore, I
> think we should mark this task canceled or finished after the execution of
> the source thread is completed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13357: [FLINK-19165] Refactor the UnilateralSortMerger
flinkbot edited a comment on pull request #13357: URL: https://github.com/apache/flink/pull/13357#issuecomment-689034696 ## CI report: * 54ee6987c7054a42fa52d04ec61fae74a816792c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6348) * 05cef2abaec734dddea631576862d7c374e320c7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6561) * 87c7f63785dda33d0a9bd210ff4d49962abbe638 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6564) * f48774d7b5e0a14f1e7270591481cc2f83316854 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13403: [Hotfix][Docs] Update rest_v1_dispathcer.html
flinkbot edited a comment on pull request #13403: URL: https://github.com/apache/flink/pull/13403#issuecomment-693610429 ## CI report: * 0f52d8c1b46c31f08e710a157822b93d15027b48 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6566) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-15467) Should wait for the end of the source thread during the Task cancellation
[
https://issues.apache.org/jira/browse/FLINK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise resolved FLINK-15467.
-
Resolution: Fixed
> Should wait for the end of the source thread during the Task cancellation
> -
>
> Key: FLINK-15467
> URL: https://issues.apache.org/jira/browse/FLINK-15467
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.9.0, 1.9.1, 1.10.1
>Reporter: ming li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0, 1.10.3, 1.11.3
>
>
> In the new mailBox model, SourceStreamTask starts a source thread to run
> user methods, and the current execution thread will block on mailbox.takeMail
> (). When a task cancels, the TaskCanceler thread will cancel the task and
> interrupt the execution thread. Therefore, the execution thread of
> SourceStreamTask will throw InterruptedException, then cancel the task again,
> and throw an exception.
> {code:java}
> //代码占位符
> @Override
> protected void performDefaultAction(ActionContext context) throws Exception {
>// Against the usual contract of this method, this implementation is not
> step-wise but blocking instead for
>// compatibility reasons with the current source interface (source
> functions run as a loop, not in steps).
>sourceThread.start();
>// We run an alternative mailbox loop that does not involve default
> actions and synchronizes around actions.
>try {
> runAlternativeMailboxLoop();
>} catch (Exception mailboxEx) {
> // We cancel the source function if some runtime exception escaped the
> mailbox.
> if (!isCanceled()) {
> cancelTask();
> }
> throw mailboxEx;
>}
>sourceThread.join();
>if (!isFinished) {
> sourceThread.checkThrowSourceExecutionException();
>}
>context.allActionsCompleted();
> }
> {code}
> When all tasks of this TaskExecutor are canceled, the blob file will be
> cleaned up. But the real source thread is not finished at this time, which
> will cause a ClassNotFoundException when loading a new class. In this case,
> the source thread may not be able to properly clean up and release resources
> (such as closing child threads, cleaning up local files, etc.). Therefore, I
> think we should mark this task canceled or finished after the execution of
> the source thread is completed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (FLINK-15467) Should wait for the end of the source thread during the Task cancellation
[
https://issues.apache.org/jira/browse/FLINK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise reassigned FLINK-15467:
---
Assignee: Roman Khachatryan
> Should wait for the end of the source thread during the Task cancellation
> -
>
> Key: FLINK-15467
> URL: https://issues.apache.org/jira/browse/FLINK-15467
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.9.0, 1.9.1, 1.10.1
>Reporter: ming li
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0, 1.10.3, 1.11.3
>
>
> In the new mailBox model, SourceStreamTask starts a source thread to run
> user methods, and the current execution thread will block on mailbox.takeMail
> (). When a task cancels, the TaskCanceler thread will cancel the task and
> interrupt the execution thread. Therefore, the execution thread of
> SourceStreamTask will throw InterruptedException, then cancel the task again,
> and throw an exception.
> {code:java}
> //代码占位符
> @Override
> protected void performDefaultAction(ActionContext context) throws Exception {
>// Against the usual contract of this method, this implementation is not
> step-wise but blocking instead for
>// compatibility reasons with the current source interface (source
> functions run as a loop, not in steps).
>sourceThread.start();
>// We run an alternative mailbox loop that does not involve default
> actions and synchronizes around actions.
>try {
> runAlternativeMailboxLoop();
>} catch (Exception mailboxEx) {
> // We cancel the source function if some runtime exception escaped the
> mailbox.
> if (!isCanceled()) {
> cancelTask();
> }
> throw mailboxEx;
>}
>sourceThread.join();
>if (!isFinished) {
> sourceThread.checkThrowSourceExecutionException();
>}
>context.allActionsCompleted();
> }
> {code}
> When all tasks of this TaskExecutor are canceled, the blob file will be
> cleaned up. But the real source thread is not finished at this time, which
> will cause a ClassNotFoundException when loading a new class. In this case,
> the source thread may not be able to properly clean up and release resources
> (such as closing child threads, cleaning up local files, etc.). Therefore, I
> think we should mark this task canceled or finished after the execution of
> the source thread is completed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #13403: [Hotfix][Docs] Update rest_v1_dispathcer.html
flinkbot commented on pull request #13403: URL: https://github.com/apache/flink/pull/13403#issuecomment-693610429 ## CI report: * 0f52d8c1b46c31f08e710a157822b93d15027b48 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13392: [FLINK-18713][table-planner-blink] Change duration configOption to duration type
flinkbot edited a comment on pull request #13392: URL: https://github.com/apache/flink/pull/13392#issuecomment-692540614 ## CI report: * Unknown: [CANCELED](TBD) * f3af04d02472f93c5d49e0262ce7957cdd910707 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6565) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-15467) Should wait for the end of the source thread during the Task cancellation
[
https://issues.apache.org/jira/browse/FLINK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197200#comment-17197200
]
Arvid Heise commented on FLINK-15467:
-
Merged to release-1.10 a20fb4a796d8df91e4ebbcdc0b35cdd75145c574.
> Should wait for the end of the source thread during the Task cancellation
> -
>
> Key: FLINK-15467
> URL: https://issues.apache.org/jira/browse/FLINK-15467
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.9.0, 1.9.1, 1.10.1
>Reporter: ming li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.3
>
>
> In the new mailBox model, SourceStreamTask starts a source thread to run
> user methods, and the current execution thread will block on mailbox.takeMail
> (). When a task cancels, the TaskCanceler thread will cancel the task and
> interrupt the execution thread. Therefore, the execution thread of
> SourceStreamTask will throw InterruptedException, then cancel the task again,
> and throw an exception.
> {code:java}
> //代码占位符
> @Override
> protected void performDefaultAction(ActionContext context) throws Exception {
>// Against the usual contract of this method, this implementation is not
> step-wise but blocking instead for
>// compatibility reasons with the current source interface (source
> functions run as a loop, not in steps).
>sourceThread.start();
>// We run an alternative mailbox loop that does not involve default
> actions and synchronizes around actions.
>try {
> runAlternativeMailboxLoop();
>} catch (Exception mailboxEx) {
> // We cancel the source function if some runtime exception escaped the
> mailbox.
> if (!isCanceled()) {
> cancelTask();
> }
> throw mailboxEx;
>}
>sourceThread.join();
>if (!isFinished) {
> sourceThread.checkThrowSourceExecutionException();
>}
>context.allActionsCompleted();
> }
> {code}
> When all tasks of this TaskExecutor are canceled, the blob file will be
> cleaned up. But the real source thread is not finished at this time, which
> will cause a ClassNotFoundException when loading a new class. In this case,
> the source thread may not be able to properly clean up and release resources
> (such as closing child threads, cleaning up local files, etc.). Therefore, I
> think we should mark this task canceled or finished after the execution of
> the source thread is completed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-15467) Should wait for the end of the source thread during the Task cancellation
[
https://issues.apache.org/jira/browse/FLINK-15467?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise updated FLINK-15467:
Fix Version/s: 1.10.3
> Should wait for the end of the source thread during the Task cancellation
> -
>
> Key: FLINK-15467
> URL: https://issues.apache.org/jira/browse/FLINK-15467
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.9.0, 1.9.1, 1.10.1
>Reporter: ming li
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0, 1.10.3, 1.11.3
>
>
> In the new mailBox model, SourceStreamTask starts a source thread to run
> user methods, and the current execution thread will block on mailbox.takeMail
> (). When a task cancels, the TaskCanceler thread will cancel the task and
> interrupt the execution thread. Therefore, the execution thread of
> SourceStreamTask will throw InterruptedException, then cancel the task again,
> and throw an exception.
> {code:java}
> //代码占位符
> @Override
> protected void performDefaultAction(ActionContext context) throws Exception {
>// Against the usual contract of this method, this implementation is not
> step-wise but blocking instead for
>// compatibility reasons with the current source interface (source
> functions run as a loop, not in steps).
>sourceThread.start();
>// We run an alternative mailbox loop that does not involve default
> actions and synchronizes around actions.
>try {
> runAlternativeMailboxLoop();
>} catch (Exception mailboxEx) {
> // We cancel the source function if some runtime exception escaped the
> mailbox.
> if (!isCanceled()) {
> cancelTask();
> }
> throw mailboxEx;
>}
>sourceThread.join();
>if (!isFinished) {
> sourceThread.checkThrowSourceExecutionException();
>}
>context.allActionsCompleted();
> }
> {code}
> When all tasks of this TaskExecutor are canceled, the blob file will be
> cleaned up. But the real source thread is not finished at this time, which
> will cause a ClassNotFoundException when loading a new class. In this case,
> the source thread may not be able to properly clean up and release resources
> (such as closing child threads, cleaning up local files, etc.). Therefore, I
> think we should mark this task canceled or finished after the execution of
> the source thread is completed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] AHeise merged pull request #13383: [FLINK-15467][task] Wait for sourceTaskThread to finish before exiting from StreamTask.invoke [1.10]
AHeise merged pull request #13383: URL: https://github.com/apache/flink/pull/13383 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-19109) Split Reader eats chained periodic watermarks
[
https://issues.apache.org/jira/browse/FLINK-19109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arvid Heise resolved FLINK-19109.
-
Resolution: Fixed
> Split Reader eats chained periodic watermarks
> -
>
> Key: FLINK-19109
> URL: https://issues.apache.org/jira/browse/FLINK-19109
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.10.0, 1.10.1, 1.11.0, 1.10.2, 1.11.1
>Reporter: David Anderson
>Assignee: Roman Khachatryan
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.12.0, 1.10.3, 1.11.2
>
>
> Attempting to generate watermarks chained to the Split Reader /
> ContinuousFileReaderOperator, as in
> {code:java}
> SingleOutputStreamOperator results = env
> .readTextFile(...)
> .map(...)
> .assignTimestampsAndWatermarks(bounded)
> .keyBy(...)
> .process(...);{code}
> leads to the Watermarks failing to be produced. Breaking the chain, via
> {{disableOperatorChaining()}} or a {{rebalance}}, works around the bug. Using
> punctuated watermarks also avoids the issue.
> Looking at this in the debugger reveals that timer service is being
> prematurely quiesced.
> In many respects this is FLINK-7666 brought back to life.
> The problem is not present in 1.9.3.
> There's a minimal reproducible example in
> [https://github.com/alpinegizmo/flink-question-001/tree/bug].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #13403: [Hotfix][Docs] Update rest_v1_dispathcer.html
flinkbot commented on pull request #13403: URL: https://github.com/apache/flink/pull/13403#issuecomment-693603286 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 0f52d8c1b46c31f08e710a157822b93d15027b48 (Wed Sep 16 19:08:11 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **Invalid pull request title: No valid Jira ID provided** Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19109) Split Reader eats chained periodic watermarks
[
https://issues.apache.org/jira/browse/FLINK-19109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17197195#comment-17197195
]
Arvid Heise commented on FLINK-19109:
-
Merged in release-1.10: 05ff71c813650f65604d960978f93ba82f09a48d
> Split Reader eats chained periodic watermarks
> -
>
> Key: FLINK-19109
> URL: https://issues.apache.org/jira/browse/FLINK-19109
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.10.0, 1.10.1, 1.11.0, 1.10.2, 1.11.1
>Reporter: David Anderson
>Assignee: Roman Khachatryan
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.2, 1.10.3
>
>
> Attempting to generate watermarks chained to the Split Reader /
> ContinuousFileReaderOperator, as in
> {code:java}
> SingleOutputStreamOperator results = env
> .readTextFile(...)
> .map(...)
> .assignTimestampsAndWatermarks(bounded)
> .keyBy(...)
> .process(...);{code}
> leads to the Watermarks failing to be produced. Breaking the chain, via
> {{disableOperatorChaining()}} or a {{rebalance}}, works around the bug. Using
> punctuated watermarks also avoids the issue.
> Looking at this in the debugger reveals that timer service is being
> prematurely quiesced.
> In many respects this is FLINK-7666 brought back to life.
> The problem is not present in 1.9.3.
> There's a minimal reproducible example in
> [https://github.com/alpinegizmo/flink-question-001/tree/bug].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
