[GitHub] [flink] Shawn-Hx commented on pull request #13308: [hotfix][docs-zh] Fix invalid links in "Concepts & Common API" page of "Table API & SQL"
Shawn-Hx commented on pull request #13308: URL: https://github.com/apache/flink/pull/13308#issuecomment-686955574 Add `` tags for all titles according to @klion26 's [advice](https://github.com/apache/flink/pull/13279#discussion_r483402111). Hi, @klion26. Could you help to review this PR at your convenience ? Thank you~ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19126) Failed to run job in yarn-cluster mode due to No Executor found.
[
https://issues.apache.org/jira/browse/FLINK-19126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17190596#comment-17190596
]
Tang Yan commented on FLINK-19126:
--
[~fly_in_gis] Thanks for your reply. I've got succeeded after modifying the
bin/flink script as below.
# Add HADOOP_CLASSPATH to allow the usage of Hadoop file systems
{color:#FF}HADOOP_CLASSPATH=`hadoop classpath`{color}
exec $JAVA_RUN $JVM_ARGS $FLINK_ENV_JAVA_OPTS "${log_setting[@]}" -classpath
"`manglePathList
"$CC_CLASSPATH:$INTERNAL_HADOOP_CLASSPATHS{color:#FF}:$HADOOP_CLASSPATH{color}"`"
org.apache.flink.client.cli.CliFrontend "$@"
> Failed to run job in yarn-cluster mode due to No Executor found.
>
>
> Key: FLINK-19126
> URL: https://issues.apache.org/jira/browse/FLINK-19126
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.1
>Reporter: Tang Yan
>Priority: Major
>
> I've build the flink package successfully, but when I run the below command,
> it failed to submit the jobs.
> [yanta@flink-1.11]$ bin/flink run -m yarn-cluster -p 2 -c
> org.apache.flink.examples.java.wordcount.WordCount
> examples/batch/WordCount.jar --input hdfs:///user/yanta/aa.txt --output
> hdfs:///user/yanta/result.txt
> Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR or
> HADOOP_CLASSPATH was set.
> The program
> finished with the following exception:
> java.lang.IllegalStateException: No Executor found. Please make sure to
> export the HADOOP_CLASSPATH environment variable or have hadoop in your
> classpath. For more information refer to the "Deployment & Operations"
> section of the official Apache Flink documentation. at
> org.apache.flink.yarn.cli.FallbackYarnSessionCli.isActive(FallbackYarnSessionCli.java:59)
> at
> org.apache.flink.client.cli.CliFrontend.validateAndGetActiveCommandLine(CliFrontend.java:1090)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:218) at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:916)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:992)
> at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:992)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13225: [FLINK-18974][docs-zh]Translate the 'User-Defined Functions' page of "Application Development's DataStream API" into Chinese
flinkbot edited a comment on pull request #13225: URL: https://github.com/apache/flink/pull/13225#issuecomment-678953566 ## CI report: * 9d5f591223a0e46a22e24bbeb084a6b59c48a445 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6140) * f6141fa38f42a92fe10ca6b707813f0253ba5f3f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6191) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13308: [hotfix][docs-zh] Fix invalid links in "Concepts & Common API" page of "Table API & SQL"
flinkbot edited a comment on pull request #13308: URL: https://github.com/apache/flink/pull/13308#issuecomment-685773533 ## CI report: * 7844c7a2675e2691811a91421f65948505620522 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6108) * c7ee03155bcc8fa288704c726635e8cf14bdcfde UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13324: [hotfix][docs] Fix ExecutionEnvironment.scala doc error
flinkbot edited a comment on pull request #13324: URL: https://github.com/apache/flink/pull/13324#issuecomment-686875157 ## CI report: * 04c250f4f16667c122c12e4b793204337d204c59 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6185) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13323: FLINK-19137: Bump Parquet from 1.10.0 to 1.11.1
flinkbot edited a comment on pull request #13323: URL: https://github.com/apache/flink/pull/13323#issuecomment-686869045 ## CI report: * 084ec77b7f7b01dc6314797cb3fa0aa2455432f7 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6184) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] rmetzger commented on a change in pull request #375: [blog] Flink Community Update - August'20
rmetzger commented on a change in pull request #375:
URL: https://github.com/apache/flink-web/pull/375#discussion_r483432310
##
File path: _posts/2020-08-31-community-update.md
##
@@ -0,0 +1,222 @@
+---
+layout: post
+title: "Flink Community Update - August'20"
+date: 2020-08-31T08:00:00.000Z
+categories: news
+authors:
+- morsapaes:
+ name: "Marta Paes"
+ twitter: "morsapaes"
+
+excerpt: Ah, so much for a quiet August month. This time around, we bring you
some new Flink Improvement Proposals (FLIPs), a preview of the upcoming Flink
Stateful Functions 2.2 release and a look into how far Flink has come in
comparison to 2019.
+---
+
+Ah, so much for a quiet August month. This time around, we bring you some new
Flink Improvement Proposals (FLIPs), a preview of the upcoming [Flink Stateful
Functions](https://ci.apache.org/projects/flink/flink-statefun-docs-master/)
2.2 release and a look into how far Flink has come in comparison to 2019.
+
+{% toc %}
+
+# The Past Month in Flink
+
+## Flink Releases
+
+### Getting Ready for Flink Stateful Functions 2.2
+
+The details of the next release of [Stateful
Functions](https://ci.apache.org/projects/flink/flink-statefun-docs-master/)
are under discussion in [this @dev mailing list
thread](http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Next-Stateful-Functions-Release-td44063.html),
and the feature freeze is set for **September 10th** — so, you can expect
Stateful Functions 2.2 to be released soon after! Some of the most relevant
features in the upcoming release are:
+
+* **DataStream API interoperability**, allowing users to embed Stateful
Functions pipelines in regular [DataStream
API](https://ci.apache.org/projects/flink/flink-docs-stable/dev/datastream_api.html)
programs with `DataStream` ingress/egress.
+
+* **Fine-grained control over state** for remote functions, including the
ability to configure different state expiration modes for each individual
function.
+
+As the community around StateFun grows, the release cycle will follow this
pattern of smaller and more frequent releases to incorporate user feedback and
allow for faster iteration. If you’d like to get involved, we’re always looking
for [new contributors](https://github.com/apache/flink-statefun#contributing)!
+
+### Flink 1.10.2
+
+The community has announced the second patch version to cover some outstanding
issues in Flink 1.10. You can find a detailed list with all the improvements
and bugfixes that went into Flink 1.10.2 in the [announcement
blogpost](https://flink.apache.org/news/2020/08/25/release-1.10.2.html).
+
+
Review comment:
You could have mentioned the Flink 1.11.2 plans as well (or did this
discussion start after you wrote the blog?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13325: [FLINK-15974][python] Support to use the Python UDF directly in the Python Table API
flinkbot edited a comment on pull request #13325: URL: https://github.com/apache/flink/pull/13325#issuecomment-686925091 ## CI report: * 26abf1ecf936d0232c6d49c0bdd4fe3beae6252e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6187) * 6f65076badbf59b1ded4e7778a0349d489138e7f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] rmetzger commented on a change in pull request #375: [blog] Flink Community Update - August'20
rmetzger commented on a change in pull request #375:
URL: https://github.com/apache/flink-web/pull/375#discussion_r483434583
##
File path: _posts/2020-08-31-community-update.md
##
@@ -0,0 +1,222 @@
+---
+layout: post
+title: "Flink Community Update - August'20"
+date: 2020-08-31T08:00:00.000Z
+categories: news
+authors:
+- morsapaes:
+ name: "Marta Paes"
+ twitter: "morsapaes"
+
+excerpt: Ah, so much for a quiet August month. This time around, we bring you
some new Flink Improvement Proposals (FLIPs), a preview of the upcoming Flink
Stateful Functions 2.2 release and a look into how far Flink has come in
comparison to 2019.
+---
+
+Ah, so much for a quiet August month. This time around, we bring you some new
Flink Improvement Proposals (FLIPs), a preview of the upcoming [Flink Stateful
Functions](https://ci.apache.org/projects/flink/flink-statefun-docs-master/)
2.2 release and a look into how far Flink has come in comparison to 2019.
+
+{% toc %}
+
+# The Past Month in Flink
+
+## Flink Releases
+
+### Getting Ready for Flink Stateful Functions 2.2
+
+The details of the next release of [Stateful
Functions](https://ci.apache.org/projects/flink/flink-statefun-docs-master/)
are under discussion in [this @dev mailing list
thread](http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Next-Stateful-Functions-Release-td44063.html),
and the feature freeze is set for **September 10th** — so, you can expect
Stateful Functions 2.2 to be released soon after! Some of the most relevant
features in the upcoming release are:
+
+* **DataStream API interoperability**, allowing users to embed Stateful
Functions pipelines in regular [DataStream
API](https://ci.apache.org/projects/flink/flink-docs-stable/dev/datastream_api.html)
programs with `DataStream` ingress/egress.
+
+* **Fine-grained control over state** for remote functions, including the
ability to configure different state expiration modes for each individual
function.
+
+As the community around StateFun grows, the release cycle will follow this
pattern of smaller and more frequent releases to incorporate user feedback and
allow for faster iteration. If you’d like to get involved, we’re always looking
for [new contributors](https://github.com/apache/flink-statefun#contributing)!
+
+### Flink 1.10.2
+
+The community has announced the second patch version to cover some outstanding
issues in Flink 1.10. You can find a detailed list with all the improvements
and bugfixes that went into Flink 1.10.2 in the [announcement
blogpost](https://flink.apache.org/news/2020/08/25/release-1.10.2.html).
+
+
+
+## New Flink Improvement Proposals (FLIPs)
+
+The number of FLIPs being created and discussed in the @dev mailing list is
growing week over week, as the Flink 1.12 release takes form and some
longer-term efforts are kicked-off. Below are some of the new FLIPs to keep an
eye out for!
+
+
+
+
+ #
+
+
+
+
+
+ https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866741";>FLIP-131
+
+Consolidate User-Facing APIs and Deprecate the DataSet
API
+The community proposes to deprecate the DataSet API in favor of the
Table API/SQL and the DataStream API, in the long run. For this to be feasible,
both APIs first need to be https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=158866741#FLIP131:ConsolidatetheuserfacingDataflowSDKs/APIs(anddeprecatetheDataSetAPI)-ProposedChanges">adapted
and expanded to support the additional use cases currently covered by the
DataSet API.
+ The first discussion to branch out of this "umbrella" FLIP is
around support for a batch execution mode in the DataStream API (https://cwiki.apache.org/confluence/display/FLINK/FLIP-134%3A+Semantics+of+Bounded+Applications+on+the+DataStream+API";>FLIP-134).
+
+
+
+
+ https://cwiki.apache.org/confluence/display/FLINK/FLIP-135+Approximate+Task-Local+Recovery";>FLIP-135
+
+Approximate Task-Local Recovery
+To better accommodate recovery scenarios where a certain amount of
data loss is tolerable, but a full pipeline restart is not desirable, the
community plans to introduce a new failover strategy that allows to restart
only the failed task(s). Approximate task-local recovery will allow users to
trade consistency for fast failure recovery, which is handy for use cases like
online training.
+
+
+
+
+ https://cwiki.apache.org/confluence/display/FLINK/FLIP-136%3A++Improve+interoperability+between+DataStream+and+Table+API";>FLIP-136
+
+Improve the interoperability between DataStream and Table
API
+The Table API has seen a great deal of refactoring and new features
in recent releases, but the interfaces to and from the DataStream API haven't
been updated accordingly. The work in this FLIP will cover multiple known gaps
to improve interoperability and expose important functi
[GitHub] [flink] mbalassi commented on pull request #13128: [FLINK-18795][hbase] Support for HBase 2
mbalassi commented on pull request #13128: URL: https://github.com/apache/flink/pull/13128#issuecomment-686965680 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] morsapaes commented on a change in pull request #375: [blog] Flink Community Update - August'20
morsapaes commented on a change in pull request #375:
URL: https://github.com/apache/flink-web/pull/375#discussion_r483435844
##
File path: _posts/2020-08-31-community-update.md
##
@@ -0,0 +1,222 @@
+---
+layout: post
+title: "Flink Community Update - August'20"
+date: 2020-08-31T08:00:00.000Z
+categories: news
+authors:
+- morsapaes:
+ name: "Marta Paes"
+ twitter: "morsapaes"
+
+excerpt: Ah, so much for a quiet August month. This time around, we bring you
some new Flink Improvement Proposals (FLIPs), a preview of the upcoming Flink
Stateful Functions 2.2 release and a look into how far Flink has come in
comparison to 2019.
+---
+
+Ah, so much for a quiet August month. This time around, we bring you some new
Flink Improvement Proposals (FLIPs), a preview of the upcoming [Flink Stateful
Functions](https://ci.apache.org/projects/flink/flink-statefun-docs-master/)
2.2 release and a look into how far Flink has come in comparison to 2019.
+
+{% toc %}
+
+# The Past Month in Flink
+
+## Flink Releases
+
+### Getting Ready for Flink Stateful Functions 2.2
+
+The details of the next release of [Stateful
Functions](https://ci.apache.org/projects/flink/flink-statefun-docs-master/)
are under discussion in [this @dev mailing list
thread](http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Next-Stateful-Functions-Release-td44063.html),
and the feature freeze is set for **September 10th** — so, you can expect
Stateful Functions 2.2 to be released soon after! Some of the most relevant
features in the upcoming release are:
+
+* **DataStream API interoperability**, allowing users to embed Stateful
Functions pipelines in regular [DataStream
API](https://ci.apache.org/projects/flink/flink-docs-stable/dev/datastream_api.html)
programs with `DataStream` ingress/egress.
+
+* **Fine-grained control over state** for remote functions, including the
ability to configure different state expiration modes for each individual
function.
+
+As the community around StateFun grows, the release cycle will follow this
pattern of smaller and more frequent releases to incorporate user feedback and
allow for faster iteration. If you’d like to get involved, we’re always looking
for [new contributors](https://github.com/apache/flink-statefun#contributing)!
+
+### Flink 1.10.2
+
+The community has announced the second patch version to cover some outstanding
issues in Flink 1.10. You can find a detailed list with all the improvements
and bugfixes that went into Flink 1.10.2 in the [announcement
blogpost](https://flink.apache.org/news/2020/08/25/release-1.10.2.html).
+
+
Review comment:
Yeah, it started after I opened the PR. But for minor releases I usually
only include the release itself once it's out.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] shuiqiangchen commented on pull request #13292: [FLINK-19097][python] Support add_jars() for Python StreamExecutionEnvironment.
shuiqiangchen commented on pull request #13292: URL: https://github.com/apache/flink/pull/13292#issuecomment-686966367 Hi @dianfu Thank you for your comments. Currently, users are not able to get/set Configurations for a StreamExecutionEnvironment since the StreamExecutionEnvironment.getConfiguration() is a protected method. So we choose to add these two new interface that the Java StreamExecutionEnvironment does not have, this is actually a bit controversial. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-web] rmetzger commented on a change in pull request #375: [blog] Flink Community Update - August'20
rmetzger commented on a change in pull request #375:
URL: https://github.com/apache/flink-web/pull/375#discussion_r483436620
##
File path: _posts/2020-08-31-community-update.md
##
@@ -0,0 +1,222 @@
+---
+layout: post
+title: "Flink Community Update - August'20"
+date: 2020-08-31T08:00:00.000Z
+categories: news
+authors:
+- morsapaes:
+ name: "Marta Paes"
+ twitter: "morsapaes"
+
+excerpt: Ah, so much for a quiet August month. This time around, we bring you
some new Flink Improvement Proposals (FLIPs), a preview of the upcoming Flink
Stateful Functions 2.2 release and a look into how far Flink has come in
comparison to 2019.
+---
+
+Ah, so much for a quiet August month. This time around, we bring you some new
Flink Improvement Proposals (FLIPs), a preview of the upcoming [Flink Stateful
Functions](https://ci.apache.org/projects/flink/flink-statefun-docs-master/)
2.2 release and a look into how far Flink has come in comparison to 2019.
+
+{% toc %}
+
+# The Past Month in Flink
+
+## Flink Releases
+
+### Getting Ready for Flink Stateful Functions 2.2
+
+The details of the next release of [Stateful
Functions](https://ci.apache.org/projects/flink/flink-statefun-docs-master/)
are under discussion in [this @dev mailing list
thread](http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/Next-Stateful-Functions-Release-td44063.html),
and the feature freeze is set for **September 10th** — so, you can expect
Stateful Functions 2.2 to be released soon after! Some of the most relevant
features in the upcoming release are:
+
+* **DataStream API interoperability**, allowing users to embed Stateful
Functions pipelines in regular [DataStream
API](https://ci.apache.org/projects/flink/flink-docs-stable/dev/datastream_api.html)
programs with `DataStream` ingress/egress.
+
+* **Fine-grained control over state** for remote functions, including the
ability to configure different state expiration modes for each individual
function.
+
+As the community around StateFun grows, the release cycle will follow this
pattern of smaller and more frequent releases to incorporate user feedback and
allow for faster iteration. If you’d like to get involved, we’re always looking
for [new contributors](https://github.com/apache/flink-statefun#contributing)!
+
+### Flink 1.10.2
+
+The community has announced the second patch version to cover some outstanding
issues in Flink 1.10. You can find a detailed list with all the improvements
and bugfixes that went into Flink 1.10.2 in the [announcement
blogpost](https://flink.apache.org/news/2020/08/25/release-1.10.2.html).
+
+
Review comment:
okay, I see.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-19126) Failed to run job in yarn-cluster mode due to No Executor found.
[
https://issues.apache.org/jira/browse/FLINK-19126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17190600#comment-17190600
]
Till Rohrmann commented on FLINK-19126:
---
Why isn't it enough to export {{HADOOP_CLASSPATH}}? Where did Flink fail if you
did it like this [~Tang Yan]?
> Failed to run job in yarn-cluster mode due to No Executor found.
>
>
> Key: FLINK-19126
> URL: https://issues.apache.org/jira/browse/FLINK-19126
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.1
>Reporter: Tang Yan
>Priority: Major
>
> I've build the flink package successfully, but when I run the below command,
> it failed to submit the jobs.
> [yanta@flink-1.11]$ bin/flink run -m yarn-cluster -p 2 -c
> org.apache.flink.examples.java.wordcount.WordCount
> examples/batch/WordCount.jar --input hdfs:///user/yanta/aa.txt --output
> hdfs:///user/yanta/result.txt
> Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR or
> HADOOP_CLASSPATH was set.
> The program
> finished with the following exception:
> java.lang.IllegalStateException: No Executor found. Please make sure to
> export the HADOOP_CLASSPATH environment variable or have hadoop in your
> classpath. For more information refer to the "Deployment & Operations"
> section of the official Apache Flink documentation. at
> org.apache.flink.yarn.cli.FallbackYarnSessionCli.isActive(FallbackYarnSessionCli.java:59)
> at
> org.apache.flink.client.cli.CliFrontend.validateAndGetActiveCommandLine(CliFrontend.java:1090)
> at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:218) at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:916)
> at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:992)
> at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
> at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:992)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] sujun1020 commented on a change in pull request #13306: [FLINK-17779][Connectors/ORC]Orc file format support filter push down
sujun1020 commented on a change in pull request #13306:
URL: https://github.com/apache/flink/pull/13306#discussion_r483439249
##
File path:
flink-formats/flink-orc/src/main/java/org/apache/flink/orc/OrcFileSystemFormatFactory.java
##
@@ -83,14 +96,227 @@ private static Properties getOrcProperties(ReadableConfig
options) {
return orcProperties;
}
+ private boolean isUnaryValid(CallExpression callExpression) {
Review comment:
@JingsongLi thanks for your review, do you mean that all the codes
related to filter pushdown conversion below are put in the class OrcFilters ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13128: [FLINK-18795][hbase] Support for HBase 2
flinkbot edited a comment on pull request #13128: URL: https://github.com/apache/flink/pull/13128#issuecomment-672766836 ## CI report: * 313b80f4474455d7013b0852929b5a8458f391a1 UNKNOWN * ece578a9fdbaee4d815de501187d92a729790c9b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6174) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6196) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sujun1020 commented on a change in pull request #13306: [FLINK-17779][Connectors/ORC]Orc file format support filter push down
sujun1020 commented on a change in pull request #13306:
URL: https://github.com/apache/flink/pull/13306#discussion_r483439597
##
File path:
flink-formats/flink-orc/src/main/java/org/apache/flink/orc/OrcFileSystemFormatFactory.java
##
@@ -83,14 +96,227 @@ private static Properties getOrcProperties(ReadableConfig
options) {
return orcProperties;
}
+ private boolean isUnaryValid(CallExpression callExpression) {
+ return callExpression.getChildren().size() == 1 &&
callExpression.getChildren().get(0) instanceof FieldReferenceExpression;
+ }
+
+ private boolean isBinaryValid(CallExpression callExpression) {
+ return callExpression.getChildren().size() == 2 &&
((callExpression.getChildren().get(0) instanceof FieldReferenceExpression &&
callExpression.getChildren().get(1) instanceof ValueLiteralExpression) ||
+ (callExpression.getChildren().get(0) instanceof
ValueLiteralExpression && callExpression.getChildren().get(1) instanceof
FieldReferenceExpression));
+ }
+
+ public OrcSplitReader.Predicate toOrcPredicate(Expression expression) {
+ if (expression instanceof CallExpression) {
+ CallExpression callExp = (CallExpression) expression;
+ FunctionDefinition funcDef =
callExp.getFunctionDefinition();
+
+ if (funcDef == BuiltInFunctionDefinitions.IS_NULL ||
funcDef == BuiltInFunctionDefinitions.IS_NOT_NULL || funcDef ==
BuiltInFunctionDefinitions.NOT) {
Review comment:
do you means: static final ImmutableMap> FILTERS ?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13325: [FLINK-15974][python] Support to use the Python UDF directly in the Python Table API
flinkbot edited a comment on pull request #13325: URL: https://github.com/apache/flink/pull/13325#issuecomment-686925091 ## CI report: * 26abf1ecf936d0232c6d49c0bdd4fe3beae6252e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6187) * 6f65076badbf59b1ded4e7778a0349d489138e7f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6195) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13308: [hotfix][docs-zh] Fix invalid links in "Concepts & Common API" page of "Table API & SQL"
flinkbot edited a comment on pull request #13308: URL: https://github.com/apache/flink/pull/13308#issuecomment-685773533 ## CI report: * 7844c7a2675e2691811a91421f65948505620522 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6108) * c7ee03155bcc8fa288704c726635e8cf14bdcfde Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6194) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] sujun1020 commented on a change in pull request #13306: [FLINK-17779][Connectors/ORC]Orc file format support filter push down
sujun1020 commented on a change in pull request #13306:
URL: https://github.com/apache/flink/pull/13306#discussion_r483439854
##
File path:
flink-formats/flink-orc/src/main/java/org/apache/flink/orc/OrcFileSystemFormatFactory.java
##
@@ -83,14 +96,227 @@ private static Properties getOrcProperties(ReadableConfig
options) {
return orcProperties;
}
+ private boolean isUnaryValid(CallExpression callExpression) {
+ return callExpression.getChildren().size() == 1 &&
callExpression.getChildren().get(0) instanceof FieldReferenceExpression;
+ }
+
+ private boolean isBinaryValid(CallExpression callExpression) {
+ return callExpression.getChildren().size() == 2 &&
((callExpression.getChildren().get(0) instanceof FieldReferenceExpression &&
callExpression.getChildren().get(1) instanceof ValueLiteralExpression) ||
+ (callExpression.getChildren().get(0) instanceof
ValueLiteralExpression && callExpression.getChildren().get(1) instanceof
FieldReferenceExpression));
+ }
+
+ public OrcSplitReader.Predicate toOrcPredicate(Expression expression) {
+ if (expression instanceof CallExpression) {
+ CallExpression callExp = (CallExpression) expression;
+ FunctionDefinition funcDef =
callExp.getFunctionDefinition();
+
+ if (funcDef == BuiltInFunctionDefinitions.IS_NULL ||
funcDef == BuiltInFunctionDefinitions.IS_NOT_NULL || funcDef ==
BuiltInFunctionDefinitions.NOT) {
+ if (!isUnaryValid(callExp)) {
+ // not a valid predicate
+ LOG.debug("Unsupported predicate [{}]
cannot be pushed into OrcFileSystemFormatFactory.", callExp);
+ return null;
+ }
+
+ PredicateLeaf.Type colType =
toOrcType(((FieldReferenceExpression)
callExp.getChildren().get(0)).getOutputDataType());
+ if (colType == null) {
+ // unsupported type
+ LOG.debug("Unsupported predicate [{}]
cannot be pushed into OrcTableSource.", callExp);
+ return null;
+ }
+
+ String colName = getColumnName(callExp);
+
+ if (funcDef ==
BuiltInFunctionDefinitions.IS_NULL) {
+ return new
OrcSplitReader.IsNull(colName, colType);
+ } else if (funcDef ==
BuiltInFunctionDefinitions.IS_NOT_NULL) {
+ return new OrcSplitReader.Not(
+ new
OrcSplitReader.IsNull(colName, colType));
+ } else {
+ OrcSplitReader.Predicate c =
toOrcPredicate(callExp.getChildren().get(0));
+ if (c == null) {
+ return null;
+ } else {
+ return new
OrcSplitReader.Not(c);
+ }
+ }
+ } else if (funcDef == BuiltInFunctionDefinitions.OR) {
+ if (callExp.getChildren().size() < 2) {
+ return null;
+ }
+ Expression left = callExp.getChildren().get(0);
+ Expression right = callExp.getChildren().get(1);
+
+ OrcSplitReader.Predicate c1 =
toOrcPredicate(left);
+ OrcSplitReader.Predicate c2 =
toOrcPredicate(right);
+ if (c1 == null || c2 == null) {
+ return null;
+ } else {
+ return new OrcSplitReader.Or(c1, c2);
+ }
+ } else {
+ if (!isBinaryValid(callExp)) {
+ // not a valid predicate
+ LOG.debug("Unsupported predicate [{}]
cannot be pushed into OrcFileSystemFormatFactory.", callExp);
+ return null;
+ }
+
+ PredicateLeaf.Type litType =
getLiteralType(callExp);
+ if (litType == null) {
+ // unsupported literal type
+ LOG.debug("Unsupported predicate [{}]
cannot be pushed into OrcFileSystemF
[jira] [Commented] (FLINK-19126) Failed to run job in yarn-cluster mode due to No Executor found.
[ https://issues.apache.org/jira/browse/FLINK-19126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17190603#comment-17190603 ] Tang Yan commented on FLINK-19126: -- [~trohrmann] If I do export HADOOP_CLASSPATH=`hadoop classpath` in linux shell and then do bin/flink run , it can work well. But if I add "export HADOOP_CLASSPATH=`hadoop classpath`" in bin/flink script, it still failed due the same error as Jira description. > Failed to run job in yarn-cluster mode due to No Executor found. > > > Key: FLINK-19126 > URL: https://issues.apache.org/jira/browse/FLINK-19126 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN >Affects Versions: 1.11.1 >Reporter: Tang Yan >Priority: Major > > I've build the flink package successfully, but when I run the below command, > it failed to submit the jobs. > [yanta@flink-1.11]$ bin/flink run -m yarn-cluster -p 2 -c > org.apache.flink.examples.java.wordcount.WordCount > examples/batch/WordCount.jar --input hdfs:///user/yanta/aa.txt --output > hdfs:///user/yanta/result.txt > Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR or > HADOOP_CLASSPATH was set. > The program > finished with the following exception: > java.lang.IllegalStateException: No Executor found. Please make sure to > export the HADOOP_CLASSPATH environment variable or have hadoop in your > classpath. For more information refer to the "Deployment & Operations" > section of the official Apache Flink documentation. at > org.apache.flink.yarn.cli.FallbackYarnSessionCli.isActive(FallbackYarnSessionCli.java:59) > at > org.apache.flink.client.cli.CliFrontend.validateAndGetActiveCommandLine(CliFrontend.java:1090) > at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:218) at > org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:916) > at > org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:992) > at > org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) > at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:992) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] JingsongLi commented on a change in pull request #13306: [FLINK-17779][Connectors/ORC]Orc file format support filter push down
JingsongLi commented on a change in pull request #13306:
URL: https://github.com/apache/flink/pull/13306#discussion_r483441917
##
File path:
flink-formats/flink-orc/src/main/java/org/apache/flink/orc/OrcFileSystemFormatFactory.java
##
@@ -83,14 +96,227 @@ private static Properties getOrcProperties(ReadableConfig
options) {
return orcProperties;
}
+ private boolean isUnaryValid(CallExpression callExpression) {
+ return callExpression.getChildren().size() == 1 &&
callExpression.getChildren().get(0) instanceof FieldReferenceExpression;
+ }
+
+ private boolean isBinaryValid(CallExpression callExpression) {
+ return callExpression.getChildren().size() == 2 &&
((callExpression.getChildren().get(0) instanceof FieldReferenceExpression &&
callExpression.getChildren().get(1) instanceof ValueLiteralExpression) ||
+ (callExpression.getChildren().get(0) instanceof
ValueLiteralExpression && callExpression.getChildren().get(1) instanceof
FieldReferenceExpression));
+ }
+
+ public OrcSplitReader.Predicate toOrcPredicate(Expression expression) {
+ if (expression instanceof CallExpression) {
+ CallExpression callExp = (CallExpression) expression;
+ FunctionDefinition funcDef =
callExp.getFunctionDefinition();
+
+ if (funcDef == BuiltInFunctionDefinitions.IS_NULL ||
funcDef == BuiltInFunctionDefinitions.IS_NOT_NULL || funcDef ==
BuiltInFunctionDefinitions.NOT) {
Review comment:
Yes
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] tillrohrmann closed pull request #13227: [FLINK-18959][Runtime] Try to revert MiniDispatcher for archiveExecutionGraph
tillrohrmann closed pull request #13227: URL: https://github.com/apache/flink/pull/13227 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #13306: [FLINK-17779][Connectors/ORC]Orc file format support filter push down
JingsongLi commented on a change in pull request #13306:
URL: https://github.com/apache/flink/pull/13306#discussion_r483441711
##
File path:
flink-formats/flink-orc/src/main/java/org/apache/flink/orc/OrcFileSystemFormatFactory.java
##
@@ -83,14 +96,227 @@ private static Properties getOrcProperties(ReadableConfig
options) {
return orcProperties;
}
+ private boolean isUnaryValid(CallExpression callExpression) {
+ return callExpression.getChildren().size() == 1 &&
callExpression.getChildren().get(0) instanceof FieldReferenceExpression;
+ }
+
+ private boolean isBinaryValid(CallExpression callExpression) {
+ return callExpression.getChildren().size() == 2 &&
((callExpression.getChildren().get(0) instanceof FieldReferenceExpression &&
callExpression.getChildren().get(1) instanceof ValueLiteralExpression) ||
+ (callExpression.getChildren().get(0) instanceof
ValueLiteralExpression && callExpression.getChildren().get(1) instanceof
FieldReferenceExpression));
+ }
+
+ public OrcSplitReader.Predicate toOrcPredicate(Expression expression) {
+ if (expression instanceof CallExpression) {
+ CallExpression callExp = (CallExpression) expression;
+ FunctionDefinition funcDef =
callExp.getFunctionDefinition();
+
+ if (funcDef == BuiltInFunctionDefinitions.IS_NULL ||
funcDef == BuiltInFunctionDefinitions.IS_NOT_NULL || funcDef ==
BuiltInFunctionDefinitions.NOT) {
+ if (!isUnaryValid(callExp)) {
+ // not a valid predicate
+ LOG.debug("Unsupported predicate [{}]
cannot be pushed into OrcFileSystemFormatFactory.", callExp);
+ return null;
+ }
+
+ PredicateLeaf.Type colType =
toOrcType(((FieldReferenceExpression)
callExp.getChildren().get(0)).getOutputDataType());
+ if (colType == null) {
+ // unsupported type
+ LOG.debug("Unsupported predicate [{}]
cannot be pushed into OrcTableSource.", callExp);
+ return null;
+ }
+
+ String colName = getColumnName(callExp);
+
+ if (funcDef ==
BuiltInFunctionDefinitions.IS_NULL) {
+ return new
OrcSplitReader.IsNull(colName, colType);
+ } else if (funcDef ==
BuiltInFunctionDefinitions.IS_NOT_NULL) {
+ return new OrcSplitReader.Not(
+ new
OrcSplitReader.IsNull(colName, colType));
+ } else {
+ OrcSplitReader.Predicate c =
toOrcPredicate(callExp.getChildren().get(0));
+ if (c == null) {
+ return null;
+ } else {
+ return new
OrcSplitReader.Not(c);
+ }
+ }
+ } else if (funcDef == BuiltInFunctionDefinitions.OR) {
+ if (callExp.getChildren().size() < 2) {
+ return null;
+ }
+ Expression left = callExp.getChildren().get(0);
+ Expression right = callExp.getChildren().get(1);
+
+ OrcSplitReader.Predicate c1 =
toOrcPredicate(left);
+ OrcSplitReader.Predicate c2 =
toOrcPredicate(right);
+ if (c1 == null || c2 == null) {
+ return null;
+ } else {
+ return new OrcSplitReader.Or(c1, c2);
+ }
+ } else {
+ if (!isBinaryValid(callExp)) {
+ // not a valid predicate
+ LOG.debug("Unsupported predicate [{}]
cannot be pushed into OrcFileSystemFormatFactory.", callExp);
+ return null;
+ }
+
+ PredicateLeaf.Type litType =
getLiteralType(callExp);
+ if (litType == null) {
+ // unsupported literal type
+ LOG.debug("Unsupported predicate [{}]
cannot be pushed into OrcFileSystem
[GitHub] [flink] JingsongLi commented on a change in pull request #13306: [FLINK-17779][Connectors/ORC]Orc file format support filter push down
JingsongLi commented on a change in pull request #13306:
URL: https://github.com/apache/flink/pull/13306#discussion_r483442277
##
File path:
flink-formats/flink-orc/src/main/java/org/apache/flink/orc/OrcFileSystemFormatFactory.java
##
@@ -83,14 +96,227 @@ private static Properties getOrcProperties(ReadableConfig
options) {
return orcProperties;
}
+ private boolean isUnaryValid(CallExpression callExpression) {
Review comment:
Yes, we can have a helper class `OrcFilters`, provides static helper
methods.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] tillrohrmann closed pull request #13318: [BP-1.11][FLINK-18959][Runtime] Try to revert MiniDispatcher for archiveExecutionGraph and shutdown cluster upon cancel.
tillrohrmann closed pull request #13318: URL: https://github.com/apache/flink/pull/13318 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on pull request #13318: [BP-1.11][FLINK-18959][Runtime] Try to revert MiniDispatcher for archiveExecutionGraph and shutdown cluster upon cancel.
tillrohrmann commented on pull request #13318: URL: https://github.com/apache/flink/pull/13318#issuecomment-686973295 Manually merged via 7e2294c2b5ca9a17c924ff948a0dcd87d90c8327 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] shuiqiangchen edited a comment on pull request #13292: [FLINK-19097][python] Support add_jars() for Python StreamExecutionEnvironment.
shuiqiangchen edited a comment on pull request #13292: URL: https://github.com/apache/flink/pull/13292#issuecomment-686966367 Hi @dianfu Thank you for your comments. Currently, users are not able to get/set Configurations for a StreamExecutionEnvironment since the StreamExecutionEnvironment.getConfiguration() is a protected method. Here we have two approaches: 1. Making the StreamExecutionEnvironment.getConfiguration() public in Python DataStream API. 2. Add these two new interface that the Java StreamExecutionEnvironment does not have, this is actually a bit controversial. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JingsongLi merged pull request #13301: [FLINK-19121][hive] Avoid accessing HDFS frequently in HiveBulkWriterFactory
JingsongLi merged pull request #13301: URL: https://github.com/apache/flink/pull/13301 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] tillrohrmann commented on pull request #13317: [BP-1.10][FLINK-18959][Runtime] Try to revert MiniDispatcher for archiveExecutionGraph and shutdown cluster upon cancel
tillrohrmann commented on pull request #13317: URL: https://github.com/apache/flink/pull/13317#issuecomment-686973783 Manually merged via 40f81b7794bde837aee8acac8944929ec43b6ef6 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-18959) Fail to archiveExecutionGraph because job is not finished when dispatcher close
[
https://issues.apache.org/jira/browse/FLINK-18959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Till Rohrmann closed FLINK-18959.
-
Resolution: Fixed
Fixed via
master: d9bddfd3c49203034141fafa0b1d522e7ea0b6e7
1.11.2: 7e2294c2b5ca9a17c924ff948a0dcd87d90c8327
1.10.3: 40f81b7794bde837aee8acac8944929ec43b6ef6
> Fail to archiveExecutionGraph because job is not finished when dispatcher
> close
> ---
>
> Key: FLINK-18959
> URL: https://issues.apache.org/jira/browse/FLINK-18959
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.10.0, 1.12.0, 1.11.1
>Reporter: Liu
>Assignee: Liu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.2, 1.10.3
>
> Attachments: flink-debug-log
>
>
> When job is cancelled, we expect to see it in flink's history server. But I
> can not see my job after it is cancelled.
> After digging into the problem, I find that the function
> archiveExecutionGraph is not executed. Below is the brief log:
> {panel:title=log}
> 2020-08-14 15:10:06,406 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph
> [flink-akka.actor.default-dispatcher- 15] - Job EtlAndWindow
> (6f784d4cc5bae88a332d254b21660372) switched from state RUNNING to CANCELLING.
> 2020-08-14 15:10:06,415 DEBUG
> org.apache.flink.runtime.dispatcher.MiniDispatcher
> [flink-akka.actor.default-dispatcher-3] - Shutting down per-job cluster
> because the job was canceled.
> 2020-08-14 15:10:06,629 INFO
> org.apache.flink.runtime.dispatcher.MiniDispatcher
> [flink-akka.actor.default-dispatcher-3] - Stopping dispatcher
> akka.tcp://flink@bjfk-c9865.yz02:38663/user/dispatcher.
> 2020-08-14 15:10:06,629 INFO
> org.apache.flink.runtime.dispatcher.MiniDispatcher
> [flink-akka.actor.default-dispatcher-3] - Stopping all currently running jobs
> of dispatcher akka.tcp://flink@bjfk-c9865.yz02:38663/user/dispatcher.
> 2020-08-14 15:10:06,631 INFO org.apache.flink.runtime.jobmaster.JobMaster
> [flink-akka.actor.default-dispatcher-29] - Stopping the JobMaster for job
> EtlAndWindow(6f784d4cc5bae88a332d254b21660372).
> 2020-08-14 15:10:06,632 DEBUG org.apache.flink.runtime.jobmaster.JobMaster
> [flink-akka.actor.default-dispatcher-29] - Disconnect TaskExecutor
> container_e144_1590060720089_2161_01_06 because: Stopping JobMaster for
> job EtlAndWindow(6f784d4cc5bae88a332d254b21660372).
> 2020-08-14 15:10:06,646 INFO
> org.apache.flink.runtime.executiongraph.ExecutionGraph
> [flink-akka.actor.default-dispatcher-29] - Job EtlAndWindow
> (6f784d4cc5bae88a332d254b21660372) switched from state CANCELLING to CANCELED.
> 2020-08-14 15:10:06,664 DEBUG
> org.apache.flink.runtime.dispatcher.MiniDispatcher
> [flink-akka.actor.default-dispatcher-4] - There is a newer JobManagerRunner
> for the job 6f784d4cc5bae88a332d254b21660372.
> {panel}
> From the log, we can see that job is not finished when dispatcher closes. The
> process is as following:
> * Receive cancel command and send it to all tasks async.
> * In MiniDispatcher, begin to shutting down per-job cluster.
> * Stopping dispatcher and remove job.
> * Job is cancelled and callback is executed in method startJobManagerRunner.
> * Because job is removed before, so currentJobManagerRunner is null which
> not equals to the original jobManagerRunner. In this case,
> archivedExecutionGraph will not be uploaded.
> In normal cases, I find that job is cancelled first and then dispatcher is
> stopped so that archivedExecutionGraph will succeed. But I think that the
> order is not constrained and it is hard to know which comes first.
> Above is what I suspected. If so, then we should fix it.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] tillrohrmann closed pull request #13317: [BP-1.10][FLINK-18959][Runtime] Try to revert MiniDispatcher for archiveExecutionGraph and shutdown cluster upon cancel
tillrohrmann closed pull request #13317: URL: https://github.com/apache/flink/pull/13317 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-19121) Avoid accessing HDFS frequently in HiveBulkWriterFactory
[ https://issues.apache.org/jira/browse/FLINK-19121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee closed FLINK-19121. Resolution: Fixed master: 41c3a19b235ad1351e9376d2d70101bd8090a4a8 release-1.11: f96c562a23dfb1f5a4b562a32099738a3f3db3e6 > Avoid accessing HDFS frequently in HiveBulkWriterFactory > > > Key: FLINK-19121 > URL: https://issues.apache.org/jira/browse/FLINK-19121 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.12.0, 1.11.1 >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Blocker > Labels: pull-request-available > Fix For: 1.11.2 > > > In HadoopPathBasedBulkWriter, getSize will invoke `FileSystem.exists` and > `FileSystem.getFileStatus`, but it is invoked per record. > There will be lots of visits to HDFS, may make HDFS pressure too high. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-19121) Avoid accessing HDFS frequently in HiveBulkWriterFactory
[ https://issues.apache.org/jira/browse/FLINK-19121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhu Zhu updated FLINK-19121: Fix Version/s: 1.12.0 > Avoid accessing HDFS frequently in HiveBulkWriterFactory > > > Key: FLINK-19121 > URL: https://issues.apache.org/jira/browse/FLINK-19121 > Project: Flink > Issue Type: Bug > Components: Connectors / Hive >Affects Versions: 1.12.0, 1.11.1 >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Blocker > Labels: pull-request-available > Fix For: 1.12.0, 1.11.2 > > > In HadoopPathBasedBulkWriter, getSize will invoke `FileSystem.exists` and > `FileSystem.getFileStatus`, but it is invoked per record. > There will be lots of visits to HDFS, may make HDFS pressure too high. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13128: [FLINK-18795][hbase] Support for HBase 2
flinkbot edited a comment on pull request #13128: URL: https://github.com/apache/flink/pull/13128#issuecomment-672766836 ## CI report: * 313b80f4474455d7013b0852929b5a8458f391a1 UNKNOWN * ece578a9fdbaee4d815de501187d92a729790c9b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6196) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6174) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] shuiqiangchen edited a comment on pull request #13292: [FLINK-19097][python] Support add_jars() for Python StreamExecutionEnvironment.
shuiqiangchen edited a comment on pull request #13292: URL: https://github.com/apache/flink/pull/13292#issuecomment-686966367 Hi @dianfu Thank you for your comments. Currently, users are not able to get/set Configurations for a StreamExecutionEnvironment since the StreamExecutionEnvironment.getConfiguration() is a protected method. Here we have two options: 1. Making the StreamExecutionEnvironment.getConfiguration() public in Python DataStream API. 2. Add these two new interface that the Java StreamExecutionEnvironment does not have, this is actually a bit controversial. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13284: [FLINK-17016][runtime] Enable pipelined region scheduling
flinkbot edited a comment on pull request #13284: URL: https://github.com/apache/flink/pull/13284#issuecomment-683683000 ## CI report: * 71281ac4921c174c214f2393e169e7140698af2d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6177) * c89d0141f4a854bc7deb4512a2b2007fccfb13fd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13321: [FLINK-14870] Remove nullable assumption of task slot sharing group
flinkbot edited a comment on pull request #13321: URL: https://github.com/apache/flink/pull/13321#issuecomment-686567896 ## CI report: * d79f152fa91b8bc555af6fb2a00a8d62b184be5a UNKNOWN * 7039064922aaec22752ac84e4e9d41d663e68a14 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6175) * bcd04d78e47e1c3a9b0cb0165d625f424dea4951 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #13181: [FLINK-18957] Implement logical request bulk tracking in SlotSharingExecutionSlotAllocator
azagrebin commented on a change in pull request #13181:
URL: https://github.com/apache/flink/pull/13181#discussion_r483450043
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SchedulerImpl.java
##
@@ -98,14 +101,15 @@ public SchedulerImpl(
"Scheduler is not initialized with proper main thread
executor. " +
"Call to Scheduler.start(...) required.");
- this.bulkSlotProvider = new
BulkSlotProviderImpl(slotSelectionStrategy, slotPool);
+ this.slotRequestBulkChecker =
PhysicalSlotRequestBulkCheckerImpl.fromSlotPool(slotPool,
SystemClock.getInstance());
+ this.bulkSlotProvider = new
BulkSlotProviderImpl(slotSelectionStrategy, slotPool, slotRequestBulkChecker);
Review comment:
Alright, thanks for the idea. It required a bit more changes but it
looks working. The change is the commit:
`[hotfix] Do not mix BulkSlotProvider into SlotProvider/SchedulerImpl`
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/PhysicalSlotProviderImpl.java
##
@@ -87,9 +86,12 @@
SlotRequestId slotRequestId,
ResourceProfile resourceProfile,
boolean willSlotBeOccupiedIndefinitely) {
- return willSlotBeOccupiedIndefinitely ?
- slotPool.requestNewAllocatedSlot(slotRequestId,
resourceProfile, null) :
- slotPool.requestNewAllocatedBatchSlot(slotRequestId,
resourceProfile);
+ if (willSlotBeOccupiedIndefinitely) {
+ return slotPool.requestNewAllocatedSlot(slotRequestId,
resourceProfile, null);
+ } else {
+ slotPool.disableBatchSlotRequestTimeoutCheck();
Review comment:
true
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #13181: [FLINK-18957] Implement logical request bulk tracking in SlotSharingExecutionSlotAllocator
azagrebin commented on a change in pull request #13181:
URL: https://github.com/apache/flink/pull/13181#discussion_r481844415
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SharingPhysicalSlotRequestBulk.java
##
@@ -0,0 +1,86 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequestBulk;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.function.BiConsumer;
+import java.util.stream.Stream;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+class SharingPhysicalSlotRequestBulk implements PhysicalSlotRequestBulk {
+ private final Map>
executions;
+
+ private final Map
pendingRequests;
+
+ private final Map
fulfilledRequests;
+
+ private final BiConsumer
logicalSlotRequestCanceller;
+
+ SharingPhysicalSlotRequestBulk(
+ Map>
executions,
+ Map
pendingRequests,
+ Map
fulfilledRequests,
+ BiConsumer
logicalSlotRequestCanceller) {
+ this.executions = checkNotNull(executions);
+ this.pendingRequests = checkNotNull(pendingRequests);
+ this.fulfilledRequests = checkNotNull(fulfilledRequests);
+ this.logicalSlotRequestCanceller =
checkNotNull(logicalSlotRequestCanceller);
+ }
+
+ @Override
+ public Collection getPendingRequests() {
+ return pendingRequests.values();
+ }
+
+ @Override
+ public Set getAllocationIdsOfFulfilledRequests() {
+ return new HashSet<>(fulfilledRequests.values());
+ }
+
+ @Override
+ public void cancel(Throwable cause) {
+ // pending requests must be canceled first otherwise they might
be fulfilled by
+ // allocated slots released from this bulk
+ Stream
+ .concat(
+ pendingRequests.keySet().stream(),
+ fulfilledRequests.keySet().stream())
+ .flatMap(group -> executions.get(group).stream())
+ .forEach(id -> logicalSlotRequestCanceller.accept(id,
cause));
+ }
+
+ void markFulfilled(ExecutionSlotSharingGroup group, AllocationID
allocationID) {
+ pendingRequests.remove(group);
+ fulfilledRequests.put(group, allocationID);
+ }
+
+ void clear() {
+ pendingRequests.clear();
Review comment:
It is written where it is used atm.
The class specifically exists for the `SlotSharingExecutionSlotAllocator`.
I can rename the method to `clearPendingRequests` and add a doc comment
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Created] (FLINK-19139) flink-conf.yml read values from environment variables
Mohammad Hossein Gerami created FLINK-19139:
---
Summary: flink-conf.yml read values from environment variables
Key: FLINK-19139
URL: https://issues.apache.org/jira/browse/FLINK-19139
Project: Flink
Issue Type: Improvement
Components: Runtime / Configuration
Reporter: Mohammad Hossein Gerami
I would like to sync my Flink configs with environment variables.
For example:
state.backend.rocksdb.metrics.actual-delayed-write-rate: ${ENABLE_METRICS}
state.backend.rocksdb.metrics.background-errors: ${ENABLE_METRICS}
state.backend.rocksdb.metrics.block-cache-capacity: ${ENABLE_METRICS}
...
state.backend.rocksdb.metrics.block-cache-pinned-usage: ${ENABLE_METRICS}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (FLINK-18641) "Failure to finalize checkpoint" error in MasterTriggerRestoreHook
[
https://issues.apache.org/jira/browse/FLINK-18641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17190612#comment-17190612
]
Zhu Zhu edited comment on FLINK-18641 at 9/4/20, 7:59 AM:
--
Hi [~becket_qin], what's the status of the pull request?
was (Author: zhuzh):
Hi [~becket_qin], that's the status of the pull request?
> "Failure to finalize checkpoint" error in MasterTriggerRestoreHook
> --
>
> Key: FLINK-18641
> URL: https://issues.apache.org/jira/browse/FLINK-18641
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.11.0
>Reporter: Brian Zhou
>Assignee: Jiangjie Qin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.2
>
>
> https://github.com/pravega/flink-connectors is a Pravega connector for Flink.
> The ReaderCheckpointHook[1] class uses the Flink `MasterTriggerRestoreHook`
> interface to trigger the Pravega checkpoint during Flink checkpoints to make
> sure the data recovery. The checkpoint recovery tests are running fine in
> Flink 1.10, but it has below issues in Flink 1.11 causing the tests time out.
> Suspect it is related to the checkpoint coordinator thread model changes in
> Flink 1.11
> Error stacktrace:
> {code}
> 2020-07-09 15:39:39,999 30945 [jobmanager-future-thread-5] WARN
> o.a.f.runtime.jobmaster.JobMaster - Error while processing checkpoint
> acknowledgement message
> org.apache.flink.runtime.checkpoint.CheckpointException: Could not finalize
> the pending checkpoint 3. Failure reason: Failure to finalize checkpoint.
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1033)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:948)
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.lambda$acknowledgeCheckpoint$4(SchedulerBase.java:802)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.flink.util.SerializedThrowable: Pending checkpoint has
> not been fully acknowledged yet
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
> at
> org.apache.flink.runtime.checkpoint.PendingCheckpoint.finalizeCheckpoint(PendingCheckpoint.java:298)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1021)
> ... 9 common frames omitted
> {code}
> More detail in this mailing thread:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Pravega-connector-cannot-recover-from-the-checkpoint-due-to-quot-Failure-to-finalize-checkpoint-quot-td36652.html
> Also in https://github.com/pravega/flink-connectors/issues/387
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18641) "Failure to finalize checkpoint" error in MasterTriggerRestoreHook
[
https://issues.apache.org/jira/browse/FLINK-18641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17190612#comment-17190612
]
Zhu Zhu commented on FLINK-18641:
-
Hi [~becket_qin], that's the status of the pull request?
> "Failure to finalize checkpoint" error in MasterTriggerRestoreHook
> --
>
> Key: FLINK-18641
> URL: https://issues.apache.org/jira/browse/FLINK-18641
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing
>Affects Versions: 1.11.0
>Reporter: Brian Zhou
>Assignee: Jiangjie Qin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.2
>
>
> https://github.com/pravega/flink-connectors is a Pravega connector for Flink.
> The ReaderCheckpointHook[1] class uses the Flink `MasterTriggerRestoreHook`
> interface to trigger the Pravega checkpoint during Flink checkpoints to make
> sure the data recovery. The checkpoint recovery tests are running fine in
> Flink 1.10, but it has below issues in Flink 1.11 causing the tests time out.
> Suspect it is related to the checkpoint coordinator thread model changes in
> Flink 1.11
> Error stacktrace:
> {code}
> 2020-07-09 15:39:39,999 30945 [jobmanager-future-thread-5] WARN
> o.a.f.runtime.jobmaster.JobMaster - Error while processing checkpoint
> acknowledgement message
> org.apache.flink.runtime.checkpoint.CheckpointException: Could not finalize
> the pending checkpoint 3. Failure reason: Failure to finalize checkpoint.
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1033)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveAcknowledgeMessage(CheckpointCoordinator.java:948)
> at
> org.apache.flink.runtime.scheduler.SchedulerBase.lambda$acknowledgeCheckpoint$4(SchedulerBase.java:802)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: org.apache.flink.util.SerializedThrowable: Pending checkpoint has
> not been fully acknowledged yet
> at
> org.apache.flink.util.Preconditions.checkState(Preconditions.java:195)
> at
> org.apache.flink.runtime.checkpoint.PendingCheckpoint.finalizeCheckpoint(PendingCheckpoint.java:298)
> at
> org.apache.flink.runtime.checkpoint.CheckpointCoordinator.completePendingCheckpoint(CheckpointCoordinator.java:1021)
> ... 9 common frames omitted
> {code}
> More detail in this mailing thread:
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Pravega-connector-cannot-recover-from-the-checkpoint-due-to-quot-Failure-to-finalize-checkpoint-quot-td36652.html
> Also in https://github.com/pravega/flink-connectors/issues/387
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] tillrohrmann commented on a change in pull request #13316: [FLINK-14422][runtime] Expose network memory usage to TaskManagerDetailsHandler's endpoint
tillrohrmann commented on a change in pull request #13316:
URL: https://github.com/apache/flink/pull/13316#discussion_r483452267
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPool.java
##
@@ -271,12 +271,28 @@ public int getTotalNumberOfMemorySegments() {
return totalNumberOfMemorySegments;
}
+ public long getTotalMemory() {
+ return getTotalNumberOfMemorySegments() * memorySegmentSize;
+ }
+
public int getNumberOfAvailableMemorySegments() {
synchronized (availableMemorySegments) {
return availableMemorySegments.size();
}
}
+ public long getAvailableMemory() {
+ return getNumberOfAvailableMemorySegments() * memorySegmentSize;
+ }
+
+ public int getNumberOfUsedMemorySegments() {
+ return getTotalNumberOfMemorySegments() -
getNumberOfAvailableMemorySegments();
+ }
+
+ public long getUsedMemory() {
+ return getNumberOfUsedMemorySegments() * memorySegmentSize;
+ }
Review comment:
What is the semantic of this method? Shall it return the number of
handed out memory segments to a set of `LocalBufferPools` or shall it return
the size of the actually used memory? If it is the latter, then I am afraid
that memory which has been handed out to the `LocalBufferPools` might also be
idling there. Hence this value won't tell us how much memory is currently been
used.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] shuiqiangchen opened a new pull request #13327: [FLINK-19134][python] Add BasicArrayTypeInfo and coder for PrimitiveArrayTypeInfo.
shuiqiangchen opened a new pull request #13327: URL: https://github.com/apache/flink/pull/13327 ## What is the purpose of the change Add BasicArrayTypeInfo and coder for PrimitiveArrayTypeInfo. ## Brief change log - Added BasicArrayTypeInfo for arrays boxed primitive types. - Added coder for PrimitiveArrayTypes. ## Verifying this change This pull request has test case covered by test_primitive_array_type_info() and test_basic_array_type_info() in test_data_stream.py. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): ( no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? ( not documented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19134) Fix the converter of array coder for Python DataStream API.
[ https://issues.apache.org/jira/browse/FLINK-19134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-19134: --- Labels: pull-request-available (was: ) > Fix the converter of array coder for Python DataStream API. > --- > > Key: FLINK-19134 > URL: https://issues.apache.org/jira/browse/FLINK-19134 > Project: Flink > Issue Type: Bug > Components: API / Python >Reporter: Shuiqiang Chen >Priority: Major > Labels: pull-request-available > > The proto to coder converter for Array type is mistakenly added a list of > element coders. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #13327: [FLINK-19134][python] Add BasicArrayTypeInfo and coder for PrimitiveArrayTypeInfo.
flinkbot commented on pull request #13327: URL: https://github.com/apache/flink/pull/13327#issuecomment-686987024 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 352380e8ab8941541fbc1773bcd29820bcf646c3 (Fri Sep 04 08:03:42 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-19134).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19134) Add BasicArrayTypeInfo and coder for PrimitiveArrayTypeInfo for Python DataStream API.
[ https://issues.apache.org/jira/browse/FLINK-19134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shuiqiang Chen updated FLINK-19134: --- Summary: Add BasicArrayTypeInfo and coder for PrimitiveArrayTypeInfo for Python DataStream API. (was: Fix the converter of array coder for Python DataStream API.) > Add BasicArrayTypeInfo and coder for PrimitiveArrayTypeInfo for Python > DataStream API. > -- > > Key: FLINK-19134 > URL: https://issues.apache.org/jira/browse/FLINK-19134 > Project: Flink > Issue Type: Bug > Components: API / Python >Reporter: Shuiqiang Chen >Priority: Major > Labels: pull-request-available > > The proto to coder converter for Array type is mistakenly added a list of > element coders. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13284: [FLINK-17016][runtime] Enable pipelined region scheduling
flinkbot edited a comment on pull request #13284: URL: https://github.com/apache/flink/pull/13284#issuecomment-683683000 ## CI report: * 71281ac4921c174c214f2393e169e7140698af2d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6177) * c89d0141f4a854bc7deb4512a2b2007fccfb13fd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6200) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13316: [FLINK-14422][runtime] Expose network memory usage to TaskManagerDetailsHandler's endpoint
flinkbot edited a comment on pull request #13316: URL: https://github.com/apache/flink/pull/13316#issuecomment-686398901 ## CI report: * e37cb771c66ed8cab48e0b7abd53132fb15dfca3 UNKNOWN * 5ccdc55dc70a57332c50bd59c0a8e6e59a29bffd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6169) * 194b07dfb332eca1b8d99d0fe1425e117eff9b33 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dawidwys commented on pull request #13326: [FLINK-19133] Open custom partitioners in KafkaSerializationSchemaWrapper
dawidwys commented on pull request #13326: URL: https://github.com/apache/flink/pull/13326#issuecomment-686998060 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13181: [FLINK-18957] Implement logical request bulk tracking in SlotSharingExecutionSlotAllocator
flinkbot edited a comment on pull request #13181: URL: https://github.com/apache/flink/pull/13181#issuecomment-675091412 ## CI report: * e7553689356882de1ffe606400d1255d1d757bc4 UNKNOWN * a96b2db52a0db507e0077266c8e9cb947413e1ba Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6176) * d34648bab5e9a642c97db6e98f80f8059500c145 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13225: [FLINK-18974][docs-zh]Translate the 'User-Defined Functions' page of "Application Development's DataStream API" into Chinese
flinkbot edited a comment on pull request #13225: URL: https://github.com/apache/flink/pull/13225#issuecomment-678953566 ## CI report: * f6141fa38f42a92fe10ca6b707813f0253ba5f3f Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6191) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13321: [FLINK-14870] Remove nullable assumption of task slot sharing group
flinkbot edited a comment on pull request #13321: URL: https://github.com/apache/flink/pull/13321#issuecomment-686567896 ## CI report: * d79f152fa91b8bc555af6fb2a00a8d62b184be5a UNKNOWN * 7039064922aaec22752ac84e4e9d41d663e68a14 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6175) * bcd04d78e47e1c3a9b0cb0165d625f424dea4951 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6201) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13326: [FLINK-19133] Open custom partitioners in KafkaSerializationSchemaWrapper
flinkbot edited a comment on pull request #13326: URL: https://github.com/apache/flink/pull/13326#issuecomment-686943261 ## CI report: * e8a32ce376b09df4766fd6ddd439d8f634210ad7 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6190) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6203) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13327: [FLINK-19134][python] Add BasicArrayTypeInfo and coder for PrimitiveArrayTypeInfo for Python DataStream API.
flinkbot commented on pull request #13327: URL: https://github.com/apache/flink/pull/13327#issuecomment-687006937 ## CI report: * 352380e8ab8941541fbc1773bcd29820bcf646c3 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuxiaoshang opened a new pull request #13328: [host fix]fix typo in the java doc
zhuxiaoshang opened a new pull request #13328: URL: https://github.com/apache/flink/pull/13328 ## What is the purpose of the change fix typo in the java doc ## Brief change log fix typo in the java doc ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (no) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuxiaoshang closed pull request #13328: [host fix]fix typo in the java doc
zhuxiaoshang closed pull request #13328: URL: https://github.com/apache/flink/pull/13328 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13328: [host fix]fix typo in the java doc
flinkbot commented on pull request #13328: URL: https://github.com/apache/flink/pull/13328#issuecomment-687008053 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 8a0aac79ace68245692645d00aaa05e95762a78c (Fri Sep 04 08:32:16 UTC 2020) **Warnings:** * **Invalid pull request title: No valid Jira ID provided** Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuxiaoshang opened a new pull request #13329: [hot fix]fix typo in the java doc
zhuxiaoshang opened a new pull request #13329: URL: https://github.com/apache/flink/pull/13329 ## What is the purpose of the change fix typo in the java doc ## Brief change log fix typo in the java doc ## Verifying this change This change is a trivial rework / code cleanup without any test coverage. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (no) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13329: [hot fix]fix typo in the java doc
flinkbot commented on pull request #13329: URL: https://github.com/apache/flink/pull/13329#issuecomment-687013800 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 54964991f6b3ec7b12f2cb045d29a2a33dad1011 (Fri Sep 04 08:44:23 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **Invalid pull request title: No valid Jira ID provided** Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13181: [FLINK-18957] Implement logical request bulk tracking in SlotSharingExecutionSlotAllocator
flinkbot edited a comment on pull request #13181: URL: https://github.com/apache/flink/pull/13181#issuecomment-675091412 ## CI report: * e7553689356882de1ffe606400d1255d1d757bc4 UNKNOWN * a96b2db52a0db507e0077266c8e9cb947413e1ba Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6176) * d34648bab5e9a642c97db6e98f80f8059500c145 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6202) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13327: [FLINK-19134][python] Add BasicArrayTypeInfo and coder for PrimitiveArrayTypeInfo for Python DataStream API.
flinkbot edited a comment on pull request #13327: URL: https://github.com/apache/flink/pull/13327#issuecomment-687006937 ## CI report: * 352380e8ab8941541fbc1773bcd29820bcf646c3 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6204) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #13181: [FLINK-18957] Implement logical request bulk tracking in SlotSharingExecutionSlotAllocator
zhuzhurk commented on a change in pull request #13181:
URL: https://github.com/apache/flink/pull/13181#discussion_r483462252
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultScheduler.java
##
@@ -95,12 +97,15 @@
private final Set verticesWaitingForRestart;
+ private final Consumer slartUpAction;
Review comment:
`slartUpAction` -> `startUpAction `
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SlotProvider.java
##
@@ -62,11 +51,13 @@ default void start(ComponentMainThreadExecutor
mainThreadExecutor) {
* @param allocationTimeout after which the allocation fails with a
timeout exception
* @return The future of the allocation
*/
- CompletableFuture allocateSlot(
+ default CompletableFuture allocateSlot(
SlotRequestId slotRequestId,
ScheduledUnit scheduledUnit,
SlotProfile slotProfile,
- Time allocationTimeout);
+ Time allocationTimeout) {
Review comment:
Looks to me default body is not needed. I tried removing the default
body and flink-runtime still compiles.
Correct me if I miss anything.
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultSchedulerFactory.java
##
@@ -125,24 +153,50 @@ static SchedulingStrategyFactory
createSchedulingStrategyFactory(final ScheduleM
}
}
- private static ExecutionSlotAllocatorFactory
createExecutionSlotAllocatorFactory(
+ private static DefaultSchedulerComponents
createPipelinedRegionSchedulerComponents(
final ScheduleMode scheduleMode,
- final SlotProvider slotProvider,
- final Time slotRequestTimeout,
- final SchedulingStrategyFactory
schedulingStrategyFactory) {
-
- if (schedulingStrategyFactory instanceof
PipelinedRegionSchedulingStrategy.Factory) {
- return new OneSlotPerExecutionSlotAllocatorFactory(
- slotProvider,
- scheduleMode !=
ScheduleMode.LAZY_FROM_SOURCES_WITH_BATCH_SLOT_REQUEST,
- slotRequestTimeout);
- } else {
- final SlotProviderStrategy slotProviderStrategy =
SlotProviderStrategy.from(
- scheduleMode,
- slotProvider,
- slotRequestTimeout);
-
- return new
DefaultExecutionSlotAllocatorFactory(slotProviderStrategy);
+ final Configuration jobMasterConfiguration,
+ final SlotPool slotPool,
+ final Time slotRequestTimeout) {
+ final SlotSelectionStrategy slotSelectionStrategy =
selectSlotSelectionStrategy(jobMasterConfiguration);
+ final BulkSlotProvider bulkSlotProvider = new
BulkSlotProviderImpl(slotSelectionStrategy, slotPool);
+ final ExecutionSlotAllocatorFactory allocatorFactory = new
OneSlotPerExecutionSlotAllocatorFactory(
+ bulkSlotProvider,
+ scheduleMode !=
ScheduleMode.LAZY_FROM_SOURCES_WITH_BATCH_SLOT_REQUEST,
+ slotRequestTimeout);
+ return new DefaultSchedulerComponents(
+ new PipelinedRegionSchedulingStrategy.Factory(),
+ bulkSlotProvider::start,
+ allocatorFactory);
+ }
+
+ @Nonnull
+ private static SlotSelectionStrategy
selectSlotSelectionStrategy(@Nonnull Configuration configuration) {
+ final boolean evenlySpreadOutSlots =
configuration.getBoolean(ClusterOptions.EVENLY_SPREAD_OUT_SLOTS_STRATEGY);
+
+ final SlotSelectionStrategy
locationPreferenceSlotSelectionStrategy;
+
+ locationPreferenceSlotSelectionStrategy = evenlySpreadOutSlots ?
+
LocationPreferenceSlotSelectionStrategy.createEvenlySpreadOut() :
+ LocationPreferenceSlotSelectionStrategy.createDefault();
+
+ return
configuration.getBoolean(CheckpointingOptions.LOCAL_RECOVERY) ?
+
PreviousAllocationSlotSelectionStrategy.create(locationPreferenceSlotSelectionStrategy)
:
+ locationPreferenceSlotSelectionStrategy;
+ }
+
+ private static class DefaultSchedulerComponents {
+ private final SchedulingStrategyFactory
schedulingStrategyFactory;
+ private final Consumer
slartUpAction;
Review comment:
`startUpAction` -> `startUpAction`
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/DefaultSchedulerFactory.java
##
@@ -125,24 +153,50 @@ static SchedulingStrategyFactory
c
[GitHub] [flink] flinkbot edited a comment on pull request #13274: [FLINK-19061][hive] HiveCatalog fails to get partition column stats i…
flinkbot edited a comment on pull request #13274: URL: https://github.com/apache/flink/pull/13274#issuecomment-682368291 ## CI report: * 7835ad0187076a7dae2ee7e8806337f1b3db01c1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5956) * 6d99f26131d335a4548c7eb7d5c64611d3de8094 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #13329: [hot fix]fix typo in the java doc
flinkbot commented on pull request #13329: URL: https://github.com/apache/flink/pull/13329#issuecomment-687023379 ## CI report: * 54964991f6b3ec7b12f2cb045d29a2a33dad1011 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13308: [hotfix][docs-zh] Fix invalid links in "Concepts & Common API" page of "Table API & SQL"
flinkbot edited a comment on pull request #13308: URL: https://github.com/apache/flink/pull/13308#issuecomment-685773533 ## CI report: * c7ee03155bcc8fa288704c726635e8cf14bdcfde Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6194) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13274: [FLINK-19061][hive] HiveCatalog fails to get partition column stats i…
flinkbot edited a comment on pull request #13274: URL: https://github.com/apache/flink/pull/13274#issuecomment-682368291 ## CI report: * 7835ad0187076a7dae2ee7e8806337f1b3db01c1 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5956) * 6d99f26131d335a4548c7eb7d5c64611d3de8094 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6205) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13316: [FLINK-14422][runtime] Expose network memory usage to TaskManagerDetailsHandler's endpoint
flinkbot edited a comment on pull request #13316: URL: https://github.com/apache/flink/pull/13316#issuecomment-686398901 ## CI report: * e37cb771c66ed8cab48e0b7abd53132fb15dfca3 UNKNOWN * 5ccdc55dc70a57332c50bd59c0a8e6e59a29bffd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6169) * 194b07dfb332eca1b8d99d0fe1425e117eff9b33 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6206) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13329: [hot fix]fix typo in the java doc
flinkbot edited a comment on pull request #13329: URL: https://github.com/apache/flink/pull/13329#issuecomment-687023379 ## CI report: * 54964991f6b3ec7b12f2cb045d29a2a33dad1011 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6207) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-19140) Join with Table Function (UDTF) SQL example is incorrect
Benchao Li created FLINK-19140:
--
Summary: Join with Table Function (UDTF) SQL example is incorrect
Key: FLINK-19140
URL: https://issues.apache.org/jira/browse/FLINK-19140
Project: Flink
Issue Type: Bug
Components: Documentation, Table SQL / API
Reporter: Benchao Li
Section "*Join with Table Function (UDTF)*" of
[https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/queries.html]
currently is:
{code:SQL}
SELECT users, tag
FROM Orders, LATERAL TABLE(unnest_udtf(tags)) t AS tag
-- and
SELECT users, tag
FROM Orders LEFT JOIN LATERAL TABLE(unnest_udtf(tags)) t AS tag ON TRUE
{code}
however, it should be:
{code:SQL}
SELECT users, tag
FROM Orders, LATERAL TABLE(unnest_udtf(tags)) AS t(tag)
-- and
SELECT users, tag
FROM Orders LEFT JOIN LATERAL TABLE(unnest_udtf(tags)) AS t(tag) ON TRUE
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] Myasuka commented on a change in pull request #13179: [FLINK-18978][state-backends] Support full table scan of key and namespace from statebackend
Myasuka commented on a change in pull request #13179:
URL: https://github.com/apache/flink/pull/13179#discussion_r483451813
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/state/KeyedStateBackend.java
##
@@ -75,6 +76,13 @@
*/
Stream getKeys(String state, N namespace);
+ /**
+* @return A stream of all keys for the given state and namespace.
Modifications to the state during iterating
+* over it keys are not supported.
+* @param state State variable for which existing keys will be returned.
Review comment:
I think we'd better give explicit description that we cannot ensure the
order of returned tuples, which means the tuples might be visited one by one
across different namespaces. What do you think?
##
File path:
flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/iterator/RocksStateKeysAndNamespaceIterator.java
##
@@ -0,0 +1,120 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.contrib.streaming.state.iterator;
+
+import org.apache.flink.api.common.typeutils.TypeSerializer;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.contrib.streaming.state.RocksDBKeySerializationUtils;
+import org.apache.flink.contrib.streaming.state.RocksIteratorWrapper;
+import org.apache.flink.core.memory.DataInputDeserializer;
+import org.apache.flink.util.FlinkRuntimeException;
+
+import javax.annotation.Nonnull;
+
+import java.io.IOException;
+import java.util.Iterator;
+import java.util.NoSuchElementException;
+
+/**
+ * Adapter class to bridge between {@link RocksIteratorWrapper} and {@link
Iterator} to iterate over the keys
+ * and namespaces. This class is not thread safe.
+ *
+ * @param the type of the iterated keys in RocksDB.
+ * @param the type of the iterated namespaces in RocksDB.
+ */
+public class RocksStateKeysAndNamespaceIterator implements
Iterator>, AutoCloseable {
Review comment:
I try to implement the abstract class and its extended class like:
~~~java
public abstract class AbstractRocksStateKeysIterator implements
AutoCloseable {
@Nonnull
protected final RocksIteratorWrapper iterator;
@Nonnull
protected final String state;
@Nonnull
protected final TypeSerializer keySerializer;
protected final boolean ambiguousKeyPossible;
protected final int keyGroupPrefixBytes;
protected final DataInputDeserializer byteArrayDataInputView;
public AbstractRocksStateKeysIterator(
@Nonnull RocksIteratorWrapper iterator,
@Nonnull String state,
@Nonnull TypeSerializer keySerializer,
int keyGroupPrefixBytes,
boolean ambiguousKeyPossible) {
this.iterator = iterator;
this.state = state;
this.keySerializer = keySerializer;
this.keyGroupPrefixBytes = keyGroupPrefixBytes;
this.ambiguousKeyPossible = ambiguousKeyPossible;
this.byteArrayDataInputView = new DataInputDeserializer();
}
protected K deserializeKey(byte[] keyBytes, DataInputDeserializer
readView) throws IOException {
readView.setBuffer(keyBytes, keyGroupPrefixBytes,
keyBytes.length - keyGroupPrefixBytes);
return RocksDBKeySerializationUtils.readKey(
keySerializer,
byteArrayDataInputView,
ambiguousKeyPossible);
}
@Override
public void close() {
iterator.close();
}
}
~~~
~~~java
public class RocksStateKeysIterator extends
AbstractRocksStateKeysIterator implements Iterator {
@Nonnull
private final byte[] namespaceBytes;
private K nextKey;
private K previousKey;
public RocksStateKeysIterator(
@Nonnull RocksIteratorWrapper iterator,
@Nonnull String state,
@Nonnull TypeSerializer keySerializer,
int
[GitHub] [flink] flinkbot edited a comment on pull request #13325: [FLINK-15974][python] Support to use the Python UDF directly in the Python Table API
flinkbot edited a comment on pull request #13325: URL: https://github.com/apache/flink/pull/13325#issuecomment-686925091 ## CI report: * 26abf1ecf936d0232c6d49c0bdd4fe3beae6252e Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6187) * 6f65076badbf59b1ded4e7778a0349d489138e7f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6195) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19140) Join with Table Function (UDTF) SQL example is incorrect
[
https://issues.apache.org/jira/browse/FLINK-19140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17190645#comment-17190645
]
Timo Walther commented on FLINK-19140:
--
btw the `AS t(tag)` is optional starting 1.12 due to the new FLIP-65 functions
> Join with Table Function (UDTF) SQL example is incorrect
>
>
> Key: FLINK-19140
> URL: https://issues.apache.org/jira/browse/FLINK-19140
> Project: Flink
> Issue Type: Bug
> Components: Documentation, Table SQL / API
>Reporter: Benchao Li
>Priority: Major
>
> Section "*Join with Table Function (UDTF)*" of
> [https://ci.apache.org/projects/flink/flink-docs-master/dev/table/sql/queries.html]
> currently is:
> {code:SQL}
> SELECT users, tag
> FROM Orders, LATERAL TABLE(unnest_udtf(tags)) t AS tag
> -- and
> SELECT users, tag
> FROM Orders LEFT JOIN LATERAL TABLE(unnest_udtf(tags)) t AS tag ON TRUE
> {code}
>
> however, it should be:
> {code:SQL}
> SELECT users, tag
> FROM Orders, LATERAL TABLE(unnest_udtf(tags)) AS t(tag)
> -- and
> SELECT users, tag
> FROM Orders LEFT JOIN LATERAL TABLE(unnest_udtf(tags)) AS t(tag) ON TRUE
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19126) Failed to run job in yarn-cluster mode due to No Executor found.
[ https://issues.apache.org/jira/browse/FLINK-19126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17190648#comment-17190648 ] Till Rohrmann commented on FLINK-19126: --- Does this mean that we can close this ticket [~Tang Yan]? > Failed to run job in yarn-cluster mode due to No Executor found. > > > Key: FLINK-19126 > URL: https://issues.apache.org/jira/browse/FLINK-19126 > Project: Flink > Issue Type: Bug > Components: Deployment / YARN >Affects Versions: 1.11.1 >Reporter: Tang Yan >Priority: Major > > I've build the flink package successfully, but when I run the below command, > it failed to submit the jobs. > [yanta@flink-1.11]$ bin/flink run -m yarn-cluster -p 2 -c > org.apache.flink.examples.java.wordcount.WordCount > examples/batch/WordCount.jar --input hdfs:///user/yanta/aa.txt --output > hdfs:///user/yanta/result.txt > Setting HADOOP_CONF_DIR=/etc/hadoop/conf because no HADOOP_CONF_DIR or > HADOOP_CLASSPATH was set. > The program > finished with the following exception: > java.lang.IllegalStateException: No Executor found. Please make sure to > export the HADOOP_CLASSPATH environment variable or have hadoop in your > classpath. For more information refer to the "Deployment & Operations" > section of the official Apache Flink documentation. at > org.apache.flink.yarn.cli.FallbackYarnSessionCli.isActive(FallbackYarnSessionCli.java:59) > at > org.apache.flink.client.cli.CliFrontend.validateAndGetActiveCommandLine(CliFrontend.java:1090) > at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:218) at > org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:916) > at > org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:992) > at > org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30) > at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:992) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13102: [FLINK-18515][Kinesis] Adding FanOutRecordPublisher for Kinesis EFO support
flinkbot edited a comment on pull request #13102: URL: https://github.com/apache/flink/pull/13102#issuecomment-671238228 ## CI report: * aeadd967bd4bdfc0352e79b194da6e0c508cfb96 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5664) * 2d0fb75c7d741e191192c6647ffeb5a44b155e6f UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dannycranmer commented on a change in pull request #13102: [FLINK-18515][Kinesis] Adding FanOutRecordPublisher for Kinesis EFO support
dannycranmer commented on a change in pull request #13102:
URL: https://github.com/apache/flink/pull/13102#discussion_r483515110
##
File path:
flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxyV2.java
##
@@ -30,15 +34,28 @@
@Internal
public class KinesisProxyV2 implements KinesisProxyV2Interface {
+ /** An Asynchronous client used to communicate with AWS services. */
private final KinesisAsyncClient kinesisAsyncClient;
/**
-* Create a new KinesisProxyV2 based on the supplied configuration
properties.
+* Create a new KinesisProxyV2 using the provided Async Client.
*
* @param kinesisAsyncClient the kinesis async client used to
communicate with Kinesis
*/
public KinesisProxyV2(final KinesisAsyncClient kinesisAsyncClient) {
this.kinesisAsyncClient =
Preconditions.checkNotNull(kinesisAsyncClient);
}
+ @Override
+ public CompletableFuture subscribeToShard(
+ final SubscribeToShardRequest request,
+ final SubscribeToShardResponseHandler responseHandler) {
+ return kinesisAsyncClient.subscribeToShard(request,
responseHandler);
+ }
+
+ @Override
+ public void close() {
+ kinesisAsyncClient.close();
Review comment:
@tzulitai I later realised that I need to close the Async HTTP Client
used by this client explicitly. This has been done in the next PR
(registration). If you want me to pull that back into this PR let me know. The
result of not closing the HTTP Client is that the thread pool is not shutdown
when the job is cancelled/errors.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13102: [FLINK-18515][Kinesis] Adding FanOutRecordPublisher for Kinesis EFO support
flinkbot edited a comment on pull request #13102: URL: https://github.com/apache/flink/pull/13102#issuecomment-671238228 ## CI report: * aeadd967bd4bdfc0352e79b194da6e0c508cfb96 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5664) * 2d0fb75c7d741e191192c6647ffeb5a44b155e6f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6208) * d82d0c6582050aa4364b4c7dda2c3053f3c2d756 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13189: [FLINK-18661][Kinesis] Stream consumer Registration/Deregistration
flinkbot edited a comment on pull request #13189: URL: https://github.com/apache/flink/pull/13189#issuecomment-675350675 ## CI report: * 7594c2debe3cfeb8d4a216b65b5bda88054f12c3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5933) * 17ab6b978f786d321781a589867fdd331731 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] XComp commented on a change in pull request #13316: [FLINK-14422][runtime] Expose network memory usage to TaskManagerDetailsHandler's endpoint
XComp commented on a change in pull request #13316:
URL: https://github.com/apache/flink/pull/13316#discussion_r483525366
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/io/network/buffer/NetworkBufferPool.java
##
@@ -271,12 +271,28 @@ public int getTotalNumberOfMemorySegments() {
return totalNumberOfMemorySegments;
}
+ public long getTotalMemory() {
+ return getTotalNumberOfMemorySegments() * memorySegmentSize;
+ }
+
public int getNumberOfAvailableMemorySegments() {
synchronized (availableMemorySegments) {
return availableMemorySegments.size();
}
}
+ public long getAvailableMemory() {
+ return getNumberOfAvailableMemorySegments() * memorySegmentSize;
+ }
+
+ public int getNumberOfUsedMemorySegments() {
+ return getTotalNumberOfMemorySegments() -
getNumberOfAvailableMemorySegments();
+ }
+
+ public long getUsedMemory() {
+ return getNumberOfUsedMemorySegments() * memorySegmentSize;
+ }
Review comment:
Good catch. I didn't consider that. I'm gonna look into accessing the
`LocalBufferPools`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] WeiZhong94 commented on pull request #13314: [FLINK-18913][docs][python] Add a "TableEnvironment" document under the "Python API" -> "Table API User's Guide" section
WeiZhong94 commented on pull request #13314: URL: https://github.com/apache/flink/pull/13314#issuecomment-687059355 @dianfu Thanks for your review! I have updated this PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13189: [FLINK-18661][Kinesis] Stream consumer Registration/Deregistration
flinkbot edited a comment on pull request #13189: URL: https://github.com/apache/flink/pull/13189#issuecomment-675350675 ## CI report: * 7594c2debe3cfeb8d4a216b65b5bda88054f12c3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=5933) * 17ab6b978f786d321781a589867fdd331731 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6209) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13314: [FLINK-18913][docs][python] Add a "TableEnvironment" document under the "Python API" -> "Table API User's Guide" section
flinkbot edited a comment on pull request #13314: URL: https://github.com/apache/flink/pull/13314#issuecomment-686373715 ## CI report: * ab431319119d2cb4865ea961c4386da3d90394fe Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6149) * 7566c3c966f7771fc6cb14970c9ccf318c6ba5ce UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13326: [FLINK-19133] Open custom partitioners in KafkaSerializationSchemaWrapper
flinkbot edited a comment on pull request #13326: URL: https://github.com/apache/flink/pull/13326#issuecomment-686943261 ## CI report: * e8a32ce376b09df4766fd6ddd439d8f634210ad7 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6190) Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6203) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] RocMarshal commented on pull request #13235: [FLINK-19036][docs-zh] Translate page 'Application Profiling & Debugging' of 'Debugging & Monitoring' into Chinese
RocMarshal commented on pull request #13235: URL: https://github.com/apache/flink/pull/13235#issuecomment-687066370 ping @wuchong @dianfu @alpinegizmo This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-19124) Some JobClient methods are not ergonomic, require ClassLoader argument
[
https://issues.apache.org/jira/browse/FLINK-19124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17190670#comment-17190670
]
Aljoscha Krettek commented on FLINK-19124:
--
Perfect! I think you can start by removing the {{ClassLoader}} from the methods
and then fix step-by-step the things that break. Let me know if you're blocked
on anything and need any input.
> Some JobClient methods are not ergonomic, require ClassLoader argument
> --
>
> Key: FLINK-19124
> URL: https://issues.apache.org/jira/browse/FLINK-19124
> Project: Flink
> Issue Type: Improvement
> Components: API / DataStream
>Reporter: Aljoscha Krettek
>Assignee: Nicholas Jiang
>Priority: Major
> Labels: starter
>
> Both {{getAccumulators()}} and {{getJobExecutionResult()}} result require the
> user to give a {{ClassLoader}}. In a lot of cases the {{JobClient}} is
> created in a context where we already know the user-code {{ClassLoader}}. We
> should see whether we can remove the class loader from those methods. This
> might require that users give a class loader when constructing a
> {{JobClient}} themselves for an already running job but the trade-off seems
> to be worth it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13314: [FLINK-18913][docs][python] Add a "TableEnvironment" document under the "Python API" -> "Table API User's Guide" section
flinkbot edited a comment on pull request #13314: URL: https://github.com/apache/flink/pull/13314#issuecomment-686373715 ## CI report: * ab431319119d2cb4865ea961c4386da3d90394fe Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6149) * 7566c3c966f7771fc6cb14970c9ccf318c6ba5ce Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6210) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-14406) Add metric for managed memory
[
https://issues.apache.org/jira/browse/FLINK-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matthias reassigned FLINK-14406:
Assignee: Matthias
> Add metric for managed memory
> -
>
> Key: FLINK-14406
> URL: https://issues.apache.org/jira/browse/FLINK-14406
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Metrics, Runtime / Task
>Reporter: lining
>Assignee: Matthias
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If a user wants to get memory used in time, as there's no manage memory's
> metrics, it couldn't get it.
> *Propose*
> * add default memory type in MemoryManager
>
> {code:java}
> public static final MemoryType DEFAULT_MEMORY_TYPE = MemoryType.OFF_HEAP;
> {code}
> * add getManagedMemoryTotal in TaskExecutor:
>
> {code:java}
> public long getManagedMemoryTotal() {
> return this.taskSlotTable.getAllocatedSlots().stream().mapToLong(
> slot ->
> slot.getMemoryManager().getMemorySizeByType(MemoryManager.DEFAULT_MEMORY_TYPE)
> ).sum();
> }
> {code}
>
> * add getManagedMemoryUsed in TaskExecutor:
>
> {code:java}
> public long getManagedMemoryUsed() {
> return this.taskSlotTable.getAllocatedSlots().stream().mapToLong(
> slot ->
> slot.getMemoryManager().getMemorySizeByType(MemoryManager.DEFAULT_MEMORY_TYPE)
> - slot.getMemoryManager().availableMemory(MemoryManager.DEFAULT_MEMORY_TYPE)
> ).sum();
> }
> {code}
>
> * add instantiateMemoryManagerMetrics in MetricUtils
>
> {code:java}
> public static void instantiateMemoryManagerMetrics(MetricGroup
> statusMetricGroup, TaskExecutor taskExecutor) {
> checkNotNull(statusMetricGroup);
> MetricGroup memoryManagerGroup =
> statusMetricGroup.addGroup("Managed").addGroup("Memory");
> memoryManagerGroup.>gauge("TotalCapacity",
> taskExecutor::getManagedMemoryTotal);
> memoryManagerGroup.>gauge("MemoryUsed",
> taskExecutor::getManagedMemoryUsed);
> }
> {code}
> * register it in TaskManagerRunner#startTaskManager
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] sujun1020 commented on pull request #13306: [FLINK-17779][Connectors/ORC]Orc file format support filter push down
sujun1020 commented on pull request #13306: URL: https://github.com/apache/flink/pull/13306#issuecomment-687076363 @JingsongLi Thanks for the review. Updated the PR. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-14406) Add metric for managed memory
[
https://issues.apache.org/jira/browse/FLINK-14406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17190677#comment-17190677
]
Matthias commented on FLINK-14406:
--
I discussed different approaches with [~trohrmann]:
* Using the metric system to expose the used memory
* The metric collection runs in a different thread - we have to take care of
thread-safety here!
* It's only enabled if the UI is requesting it - less load on the
{{TaskExecutor}}
* It's the standard way for exposing metrics of the system which might be
also relevant for other components.
* Collecting the allocated memory in the {{TaskExecutor}} using the
{{TaskExecutor}} as an observer on each {{MemoryManager}} instance
* It's always enabled putting unnecessary load on the {{TaskExecutor}}.
* The connection between {{TaskExecutor}} and {{MemoryManager}} does not
exist, yet. This would create some coupling between these two components that
are not necessary.
We decided to go for the approach also already proposed by [~lining] in the
issue's description. We have to consider thread-safety, though.
> Add metric for managed memory
> -
>
> Key: FLINK-14406
> URL: https://issues.apache.org/jira/browse/FLINK-14406
> Project: Flink
> Issue Type: Sub-task
> Components: Runtime / Metrics, Runtime / Task
>Reporter: lining
>Assignee: Matthias
>Priority: Major
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> If a user wants to get memory used in time, as there's no manage memory's
> metrics, it couldn't get it.
> *Propose*
> * add default memory type in MemoryManager
>
> {code:java}
> public static final MemoryType DEFAULT_MEMORY_TYPE = MemoryType.OFF_HEAP;
> {code}
> * add getManagedMemoryTotal in TaskExecutor:
>
> {code:java}
> public long getManagedMemoryTotal() {
> return this.taskSlotTable.getAllocatedSlots().stream().mapToLong(
> slot ->
> slot.getMemoryManager().getMemorySizeByType(MemoryManager.DEFAULT_MEMORY_TYPE)
> ).sum();
> }
> {code}
>
> * add getManagedMemoryUsed in TaskExecutor:
>
> {code:java}
> public long getManagedMemoryUsed() {
> return this.taskSlotTable.getAllocatedSlots().stream().mapToLong(
> slot ->
> slot.getMemoryManager().getMemorySizeByType(MemoryManager.DEFAULT_MEMORY_TYPE)
> - slot.getMemoryManager().availableMemory(MemoryManager.DEFAULT_MEMORY_TYPE)
> ).sum();
> }
> {code}
>
> * add instantiateMemoryManagerMetrics in MetricUtils
>
> {code:java}
> public static void instantiateMemoryManagerMetrics(MetricGroup
> statusMetricGroup, TaskExecutor taskExecutor) {
> checkNotNull(statusMetricGroup);
> MetricGroup memoryManagerGroup =
> statusMetricGroup.addGroup("Managed").addGroup("Memory");
> memoryManagerGroup.>gauge("TotalCapacity",
> taskExecutor::getManagedMemoryTotal);
> memoryManagerGroup.>gauge("MemoryUsed",
> taskExecutor::getManagedMemoryUsed);
> }
> {code}
> * register it in TaskManagerRunner#startTaskManager
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #13306: [FLINK-17779][Connectors/ORC]Orc file format support filter push down
flinkbot edited a comment on pull request #13306: URL: https://github.com/apache/flink/pull/13306#issuecomment-685717187 ## CI report: * a7f4abab05ad875bf6acef6510fcec6df7d2dc6c UNKNOWN * 8048b8f9ded32203cb0ca0761414ef7186a68e5f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6132) * c7628fc4b97f68c44d7f5e9cf316814c9f73db18 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dianfu merged pull request #13325: [FLINK-15974][python] Support to use the Python UDF directly in the Python Table API
dianfu merged pull request #13325: URL: https://github.com/apache/flink/pull/13325 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13325: [FLINK-15974][python] Support to use the Python UDF directly in the Python Table API
flinkbot edited a comment on pull request #13325: URL: https://github.com/apache/flink/pull/13325#issuecomment-686925091 ## CI report: * 6f65076badbf59b1ded4e7778a0349d489138e7f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=6195) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-19141) Flink Job Submitted on Yarn Does not Work
Yunhui created FLINK-19141:
--
Summary: Flink Job Submitted on Yarn Does not Work
Key: FLINK-19141
URL: https://issues.apache.org/jira/browse/FLINK-19141
Project: Flink
Issue Type: Bug
Components: Client / Job Submission
Affects Versions: 1.11.1
Reporter: Yunhui
I first launch a cluster on yarn.
Then I submit my job with the following command
{code:java}
$flink_path/bin/flink run \
-d -m $host_port \
-c MyMainClass my-jar.jar{code}
It take a long time to schedule. And it ended with the following Exception. But
it works for flink-1.10.1
{code:java}
org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
Could not allocate the required slot within slot request timeout. Please make
sure that the cluster has enough resources.at
org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
~[?:1.8.0_77]at
org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
~[?:1.8.0_77]at
org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
~[?:1.8.0_77]at
org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
~[?:1.8.0_77]at
org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
[flink-dist_2.11-1.11.1.jar:1.11.1]at
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
[flink-dist_2.11-1.11.1.jar:1.11.1]at
scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
[flink-dist_2.11-1.11.1.jar:1.11.1]at
akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
[flink-dist_2.11-1.11.1.jar:1.11.1]at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
[flink-dist_2.11-1.11.1.jar:1.11.1]at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171)
[flink-dist_2.11-1.11.1.
[jira] [Closed] (FLINK-15974) Support to use the Python UDF directly in the Python Table API
[
https://issues.apache.org/jira/browse/FLINK-15974?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu closed FLINK-15974.
---
Resolution: Fixed
master: bffc61ac7dad8541bde77ee6aed2b72385f420d8
> Support to use the Python UDF directly in the Python Table API
> --
>
> Key: FLINK-15974
> URL: https://issues.apache.org/jira/browse/FLINK-15974
> Project: Flink
> Issue Type: Sub-task
> Components: API / Python
>Reporter: Dian Fu
>Assignee: Dian Fu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> Currently, a Python UDF has been registered before using in Python Table API,
> e.g.
> {code}
> t_env.register_function("inc", inc)
> table.select("inc(id)") \
> .insert_into("sink")
> {code}
> It would be great if we could support to use Python UDF directly in the
> Python Table API, e.g.
> {code}
> table.select(inc("id")) \
> .insert_into("sink")
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-web] asfgit closed pull request #375: [blog] Flink Community Update - August'20
asfgit closed pull request #375: URL: https://github.com/apache/flink-web/pull/375 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-19141) Flink Job Submitted on Yarn Does not Work
[
https://issues.apache.org/jira/browse/FLINK-19141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yunhui updated FLINK-19141:
---
Description:
I first launch a cluster on yarn.
{code:java}
$flink_path/bin/yarn-session.sh \
-qu dev \
-d -nm flink_cluster_1.11 \
-jm 8192 \
-tm 12288 \
-s 2 \
-D taskmanager.memory.framework.off-heap.size=2048m \
-D taskmanager.memory.managed.size=0{code}
Then I submit my job with the following command
{code:java}
$flink_path/bin/flink run \
-d -m $host_port \
-c MyMainClass my-jar.jar{code}
It take a long time to schedule. And it ended with the following Exception. But
it works for flink-1.10.1
{code:java}
org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
Could not allocate the required slot within slot request timeout. Please make
sure that the cluster has enough resources.at
org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
~[?:1.8.0_77]at
org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
~[?:1.8.0_77]at
org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
~[?:1.8.0_77]at
org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
~[?:1.8.0_77]at
org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
[flink-dist_2.11-1.11.1.jar:1.11.1]at
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
[flink-dist_2.11-1.11.1.jar:1.11.1]at
scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
[flink-dist_2.11-1.11.1.jar:1.11.1]at
akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
[flink-dist_2.11-1.11.1.jar:1.11.1]at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
[flink-dist_2.11-1.11.1.jar:1.11.1]at
scala.PartialFunction$OrElse.applyOrElse(PartialFunctio
[jira] [Updated] (FLINK-19141) Flink Job Submitted on Yarn Does not Work
[
https://issues.apache.org/jira/browse/FLINK-19141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yunhui updated FLINK-19141:
---
Description:
I first launch a cluster on yarn.
{code:java}
$flink_path/bin/yarn-session.sh \
-qu dev \
-d -nm flink_cluster_1.11 \
-jm 8192 \
-tm 12288 \
-s 2 \
-D taskmanager.memory.framework.off-heap.size=2048m \
-D taskmanager.memory.managed.size=0{code}
Then I submit my job with the following command
{code:java}
$flink_path/bin/flink run \
-d -m $host_port \
-c MyMainClass \
my-jar.jar{code}
It take a long time to schedule. And it ended with the following Exception. But
it works for flink-1.10.1
{code:java}
org.apache.flink.runtime.jobmanager.scheduler.NoResourceAvailableException:
Could not allocate the required slot within slot request timeout. Please make
sure that the cluster has enough resources.at
org.apache.flink.runtime.scheduler.DefaultScheduler.maybeWrapWithNoResourceAvailableException(DefaultScheduler.java:441)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.scheduler.DefaultScheduler.lambda$assignResourceOrHandleError$6(DefaultScheduler.java:422)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
~[?:1.8.0_77]at
org.apache.flink.runtime.jobmaster.slotpool.SchedulerImpl.lambda$internalAllocateSlot$0(SchedulerImpl.java:168)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
~[?:1.8.0_77]at
org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$SingleTaskSlot.release(SlotSharingManager.java:726)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.release(SlotSharingManager.java:537)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.jobmaster.slotpool.SlotSharingManager$MultiTaskSlot.lambda$new$0(SlotSharingManager.java:432)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:822)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:797)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
~[?:1.8.0_77]at
org.apache.flink.runtime.concurrent.FutureUtils.lambda$forwardTo$21(FutureUtils.java:1120)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:760)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:736)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
~[?:1.8.0_77]at
java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
~[?:1.8.0_77]at
org.apache.flink.runtime.concurrent.FutureUtils$Timeout.run(FutureUtils.java:1036)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:402)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:195)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:74)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:152)
~[flink-dist_2.11-1.11.1.jar:1.11.1]at
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26)
[flink-dist_2.11-1.11.1.jar:1.11.1]at
akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21)
[flink-dist_2.11-1.11.1.jar:1.11.1]at
scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123)
[flink-dist_2.11-1.11.1.jar:1.11.1]at
akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21)
[flink-dist_2.11-1.11.1.jar:1.11.1]at
scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170)
[flink-dist_2.11-1.11.1.jar:1.11.1]at
scala.PartialFunction$OrElse.applyOrElse(Partia
[GitHub] [flink] hequn8128 merged pull request #13203: [FLINK-18984][python][docs] Add tutorial documentation for Python DataStream API
hequn8128 merged pull request #13203: URL: https://github.com/apache/flink/pull/13203 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-18984) Add tutorial documentation for Python DataStream API
[ https://issues.apache.org/jira/browse/FLINK-18984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hequn Cheng closed FLINK-18984. --- Resolution: Resolved > Add tutorial documentation for Python DataStream API > > > Key: FLINK-18984 > URL: https://issues.apache.org/jira/browse/FLINK-18984 > Project: Flink > Issue Type: Sub-task > Components: API / Python >Reporter: Hequn Cheng >Assignee: Hequn Cheng >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
