[GitHub] [flink] flinkbot edited a comment on pull request #12283: [FLINK-16975][documentation] Add docs for FileSystem connector
flinkbot edited a comment on pull request #12283: URL: https://github.com/apache/flink/pull/12283#issuecomment-632089857 ## CI report: * b3fd51b309c78d9dd5056eed35dc2fe388665899 UNKNOWN * c29ec64e9bc260015c4b57395ace4a578926fa16 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2383) * 8569d299f41b57033debc94d33ab11af91452140 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2689) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zentol commented on a change in pull request #12447: [FLINK-18069][CI] Test if Java/Scaladocs builds are passing in the compile stage
zentol commented on a change in pull request #12447: URL: https://github.com/apache/flink/pull/12447#discussion_r435040079 ## File path: tools/ci/compile.sh ## @@ -54,10 +54,22 @@ if [ $EXIT_CODE != 0 ]; then exit $EXIT_CODE fi -echo " Checking Scaladocs " +echo " Checking Javadocs and Scaladocs " + +# use the same invocation as on buildbot (https://svn.apache.org/repos/infra/infrastructure/buildbot/aegis/buildmaster/master1/projects/flink.conf) +run_mvn javadoc:aggregate -Paggregate-scaladoc -DadditionalJOption='-Xdoclint:none' \ Review comment: same deal as scala, move the output into a file. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dawidwys closed pull request #12455: [FLINK-17935] Move setting yarn.log-config-file to YarnClusterClientFactory
dawidwys closed pull request #12455: URL: https://github.com/apache/flink/pull/12455 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11541: [FLINK-15416][network] add task manager netty client retry mechenism
flinkbot edited a comment on pull request #11541: URL: https://github.com/apache/flink/pull/11541#issuecomment-604812212 ## CI report: * b5ef19285000d692d6b1b8b0aa7a37ce35c216f5 UNKNOWN * baf2c55f7bb0b9df7bdef94083c947693b75eae2 UNKNOWN * e5419f41a18eaae05814837d679f58480a2cb94c UNKNOWN * a5dc1df6976661f4dc56c5e739b54dd90efc660d Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/169527021) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2679) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12474: [javadocs] Fix typo in TwoPhaseCommitSinkFunction.java
flinkbot edited a comment on pull request #12474: URL: https://github.com/apache/flink/pull/12474#issuecomment-638579512 ## CI report: * 7c4289c4f4b7752e5693f631c4e84c1ebc0d8beb Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2675) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12410: [FLINK-13782][table-api] Implement type strategies for IF ELSE expression
flinkbot edited a comment on pull request #12410: URL: https://github.com/apache/flink/pull/12410#issuecomment-635946149 ## CI report: * ce34ce1e7b16cf8c6bf359ffadbf5d3e96613947 UNKNOWN * 9e2c9729f2c4fb3af0e339c9f7046bf1cef2fb60 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2659) * 5e2c5e2f0d9babcc26b6223449b3ab6f8ac97c3e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] carp84 commented on a change in pull request #12144: [FLINK-17384][flink-dist] support read hbase conf dir from flink.conf and change HBaseConfiguration construction.
carp84 commented on a change in pull request #12144:
URL: https://github.com/apache/flink/pull/12144#discussion_r435046468
##
File path:
flink-connectors/flink-connector-hbase/src/main/java/org/apache/flink/connector/hbase/util/HBaseConfigurationUtil.java
##
@@ -22,20 +22,85 @@
import org.apache.flink.util.Preconditions;
import org.apache.hadoop.conf.Configuration;
+import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.io.Writable;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.DataInputStream;
import java.io.DataOutputStream;
+import java.io.File;
import java.io.IOException;
/**
- * This class helps to do serialization for hadoop Configuration.
+ * This class helps to do serialization for hadoop Configuration and
HBase-related classes.
*/
@Internal
public class HBaseConfigurationUtil {
+ private static final Logger LOG =
LoggerFactory.getLogger(HBaseConfigurationUtil.class);
+
+ @SuppressWarnings("deprecation")
+ public static Configuration getHBaseConfiguration() {
+
+ // Instantiate an HBaseConfiguration to load the
hbase-default.xml and hbase-site.xml from the classpath.
+ Configuration result = new HBaseConfiguration().create();
Review comment:
It seems this line is not updated in the new commit, maybe you missed
the change? @liuyongvs Thanks.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wuchong commented on a change in pull request #12436: [FLINK-17847][table sql / planner] ArrayIndexOutOfBoundsException happens in StreamExecCalc operator
wuchong commented on a change in pull request #12436:
URL: https://github.com/apache/flink/pull/12436#discussion_r435038062
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/codegen/calls/ScalarOperatorGens.scala
##
@@ -1643,19 +1648,25 @@ object ScalarOperatorGens {
val resultTypeTerm = primitiveTypeTermForType(componentInfo)
val defaultTerm = primitiveDefaultValue(componentInfo)
+index.literalValue match {
+ case Some(v: Int) if v < 1 =>
+throw new ValidationException(
+ s"Array element access needs an index starting at 1 but was $v.")
+ case _ => //nothing
+}
val idxStr = s"${index.resultTerm} - 1"
val arrayIsNull = s"${array.resultTerm}.isNullAt($idxStr)"
val arrayGet =
rowFieldReadAccess(ctx, idxStr, array.resultTerm, componentInfo)
val arrayAccessCode =
- s"""
- |${array.code}
- |${index.code}
- |boolean $nullTerm = ${array.nullTerm} || ${index.nullTerm} ||
$arrayIsNull;
- |$resultTypeTerm $resultTerm = $nullTerm ? $defaultTerm : $arrayGet;
- |""".stripMargin
-
+s"""
+|${array.code}
+|${index.code}
+|boolean $nullTerm = ${array.nullTerm} || ${index.nullTerm} ||
$arrayIsNull ||
Review comment:
Move `$arrayIsNull` to the end? Otherwise, it is possible to throw
`ArrayIndexOutOfBoundsException`. You can try to test `f11[3]`. Maybe we should
rename `arrayIsNull` to `arrayElementIsNull`.
##
File path:
flink-table/flink-table-planner-blink/src/main/scala/org/apache/flink/table/planner/codegen/calls/ScalarOperatorGens.scala
##
@@ -1643,19 +1648,25 @@ object ScalarOperatorGens {
val resultTypeTerm = primitiveTypeTermForType(componentInfo)
val defaultTerm = primitiveDefaultValue(componentInfo)
+index.literalValue match {
+ case Some(v: Int) if v < 1 =>
+throw new ValidationException(
+ s"Array element access needs an index starting at 1 but was $v.")
+ case _ => //nothing
+}
val idxStr = s"${index.resultTerm} - 1"
val arrayIsNull = s"${array.resultTerm}.isNullAt($idxStr)"
val arrayGet =
rowFieldReadAccess(ctx, idxStr, array.resultTerm, componentInfo)
val arrayAccessCode =
- s"""
- |${array.code}
- |${index.code}
- |boolean $nullTerm = ${array.nullTerm} || ${index.nullTerm} ||
$arrayIsNull;
- |$resultTypeTerm $resultTerm = $nullTerm ? $defaultTerm : $arrayGet;
- |""".stripMargin
-
+s"""
+|${array.code}
+|${index.code}
+|boolean $nullTerm = ${array.nullTerm} || ${index.nullTerm} ||
$arrayIsNull ||
+| ${idxStr} < 0 || ${idxStr} >= ${array.resultTerm}.size();
Review comment:
`${idxStr}` -> `$idxStr`.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-15066) Cannot run multiple `insert into csvTable values ()`
[
https://issues.apache.org/jira/browse/FLINK-15066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski updated FLINK-15066:
---
Affects Version/s: 1.10.1
> Cannot run multiple `insert into csvTable values ()`
>
>
> Key: FLINK-15066
> URL: https://issues.apache.org/jira/browse/FLINK-15066
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.10.1
>Reporter: Kurt Young
>Assignee: Jingsong Lee
>Priority: Blocker
> Fix For: 1.11.0
>
>
> I created a csv table in sql client, and tried to insert some data into this
> table.
> The first insert into success, but the second one failed with exception:
> {code:java}
> // Caused by: java.io.IOException: File or directory /.../xxx.csv already
> exists. Existing files and directories are not overwritten in NO_OVERWRITE
> mode. Use OVERWRITE mode to overwrite existing files and directories.at
> org.apache.flink.core.fs.FileSystem.initOutPathLocalFS(FileSystem.java:817)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] lsyldliu commented on pull request #12303: [FLINK-17625] [table] Fix ArrayIndexOutOfBoundsException in AppendOnlyTopNFunction
lsyldliu commented on pull request #12303: URL: https://github.com/apache/flink/pull/12303#issuecomment-638737701 cc @beyond1920 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lsyldliu commented on pull request #12303: [FLINK-17625] [table] Fix ArrayIndexOutOfBoundsException in AppendOnlyTopNFunction
lsyldliu commented on pull request #12303: URL: https://github.com/apache/flink/pull/12303#issuecomment-638736205 cc @wuchong This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Comment Edited] (FLINK-18109) Manually test external resource framework with GPUDriver
[
https://issues.apache.org/jira/browse/FLINK-18109?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125740#comment-17125740
]
Yangze Guo edited comment on FLINK-18109 at 6/4/20, 9:40 AM:
-
Test log:
* Standalone mode, Linux, 2 NVIDIA GPUs
** normal mode, works well.
** coordination mode, works well.
* Kubernetes mode
** build a custom image from Flink-1.11.0-RC1
** {{bin/kubernetes-session.sh and submit WordCount job}}
** log of task executor shows everything works well.
* Yarn
** 2.10, works well.
** 3.1, works well.
was (Author: karmagyz):
Test log:
* Standalone mode, Linux, 2 NVIDIA GPUs
** normal mode, works well.
** coordination mode, works well.
* Kubernetes mode
** build a custom image from Flink-1.11.0-RC1
** {{bin/kubernetes-session.sh and submit WordCount job}}
** log of task executor shows everything works well
* Yarn
** Todo
> Manually test external resource framework with GPUDriver
>
>
> Key: FLINK-18109
> URL: https://issues.apache.org/jira/browse/FLINK-18109
> Project: Flink
> Issue Type: Sub-task
>Affects Versions: 1.11.0
>Reporter: Yangze Guo
>Priority: Blocker
> Fix For: 1.11.0
>
>
> We need to:
> - Test it in standalone mode.
> ** The discovery script works well in default mode
> ** The discovery script works well in coordination mode.
> - Test it with Yarn 2.10 and 3.1.
> ** GPU resources are allocated successfully.
> - Test it with Kubernetes
> ** GPU resources are allocated successfully.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] JingsongLi commented on a change in pull request #12319: [FLINK-17406][doc] Add documentation about dynamic table options
JingsongLi commented on a change in pull request #12319:
URL: https://github.com/apache/flink/pull/12319#discussion_r435129153
##
File path: docs/dev/table/sql/hints.md
##
@@ -0,0 +1,73 @@
+---
+title: "SQL Hints"
+nav-parent_id: sql
+nav-pos: 6
+---
+
+
+* This will be replaced by the TOC
+{:toc}
+
Review comment:
Add one line to explain Table hints. This is a doc for hints, not only
for `Dynamic Table Options`
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] JingsongLi commented on a change in pull request #12319: [FLINK-17406][doc] Add documentation about dynamic table options
JingsongLi commented on a change in pull request #12319:
URL: https://github.com/apache/flink/pull/12319#discussion_r435129153
##
File path: docs/dev/table/sql/hints.md
##
@@ -0,0 +1,73 @@
+---
+title: "SQL Hints"
+nav-parent_id: sql
+nav-pos: 6
+---
+
+
+* This will be replaced by the TOC
+{:toc}
+
Review comment:
Add one line to explain Table hints.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] wuchong commented on pull request #12381: [FLINK-17941][sql-client] Switching catalog or database doesn't work …
wuchong commented on pull request #12381: URL: https://github.com/apache/flink/pull/12381#issuecomment-638742371 This has been addressed by #12431. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12144: [FLINK-17384][flink-dist] support read hbase conf dir from flink.conf and change HBaseConfiguration construction.
flinkbot edited a comment on pull request #12144: URL: https://github.com/apache/flink/pull/12144#issuecomment-628475376 ## CI report: * d36b959fb16a91c15babda10dd884cbbdec58420 UNKNOWN * 33b1ef0748285bc503363458a684ab237a64cd62 UNKNOWN * e701eb7677b535e07e4d5642ae9280cab084b854 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2668) * 30ac550d12c2eca22eaa4dab8f9186d7296355cd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2703) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18055) Catalog does not exist in SQL Client
[
https://issues.apache.org/jira/browse/FLINK-18055?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125756#comment-17125756
]
Jark Wu commented on FLINK-18055:
-
[~dwysakowicz], I discussed with [~lirui] and yes, FLINK-17941 is a duplicate.
We both agree to use the pull request of this issue.
> Catalog does not exist in SQL Client
>
>
> Key: FLINK-18055
> URL: https://issues.apache.org/jira/browse/FLINK-18055
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.11.0
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> Flink SQL> show catalogs;
> default_catalog
> hive
> Flink SQL> use catalog hive;
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.catalog.exceptions.CatalogException: A catalog with
> name [`hive`] does not exist.
> The reason is {{SqlCommandParser}} adds {{``}} for catalog name, which is
> unnecessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-18055) Catalog does not exist in SQL Client
[
https://issues.apache.org/jira/browse/FLINK-18055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu closed FLINK-18055.
---
Resolution: Fixed
- master (1.12.0): aa40ffd94a19ed9b57988d42b8c9262ef60e172e
- 1.11.0: f391b0d433ed16f33018837f42c383992953a7bc
> Catalog does not exist in SQL Client
>
>
> Key: FLINK-18055
> URL: https://issues.apache.org/jira/browse/FLINK-18055
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.11.0
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> Flink SQL> show catalogs;
> default_catalog
> hive
> Flink SQL> use catalog hive;
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.catalog.exceptions.CatalogException: A catalog with
> name [`hive`] does not exist.
> The reason is {{SqlCommandParser}} adds {{``}} for catalog name, which is
> unnecessary.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-17941) Switching catalog or database doesn't work from SQL CLI
[
https://issues.apache.org/jira/browse/FLINK-17941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu closed FLINK-17941.
---
Resolution: Duplicate
> Switching catalog or database doesn't work from SQL CLI
> ---
>
> Key: FLINK-17941
> URL: https://issues.apache.org/jira/browse/FLINK-17941
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Reporter: Rui Li
>Assignee: Rui Li
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> To reproduce:
> {noformat}
> Flink SQL> create database db1;
> [INFO] Database has been created.
> Flink SQL> use db1;
> [ERROR] Could not execute SQL statement. Reason:
> org.apache.flink.table.catalog.exceptions.CatalogException: A database with
> name [`myhive`.`db1`] does not exist in the catalog: [myhive].
> Flink SQL> show databases;
> db1
> default
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-docker] KeatsPeeks commented on pull request #9: [FLINK-16260][docker] Add docker images based on Java 11
KeatsPeeks commented on pull request #9: URL: https://github.com/apache/flink-docker/pull/9#issuecomment-638750132 Any update on this one ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-18121) Support creating Docker image from local Flink distribution
Till Rohrmann created FLINK-18121: - Summary: Support creating Docker image from local Flink distribution Key: FLINK-18121 URL: https://issues.apache.org/jira/browse/FLINK-18121 Project: Flink Issue Type: Improvement Components: Dockerfiles Affects Versions: docker-1.11.0.0 Reporter: Till Rohrmann Currently, https://github.com/apache/flink-docker/blob/dev-master/Dockerfile-debian.template only supports to create a Docker image from a Flink distribution which is hosted on a web server. I think it would be helpful if we could also create a Docker image from a Flink distribution which is stored on one's local file system. That way, one would not have to upload the file or start a web server for serving it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zentol commented on pull request #12479: [FLINK-17260] Fix StreamingKafkaITCase instabilities
zentol commented on pull request #12479: URL: https://github.com/apache/flink/pull/12479#issuecomment-638753608 We should do the same for the `org.apache.flink.tests.util.kafka.SQLClientKafkaITCase` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18121) Support creating Docker image from local Flink distribution
[ https://issues.apache.org/jira/browse/FLINK-18121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125783#comment-17125783 ] Robert Metzger commented on FLINK-18121: FYI: I opened a PR documenting the current approach: https://github.com/apache/flink-docker/pull/24 > Support creating Docker image from local Flink distribution > --- > > Key: FLINK-18121 > URL: https://issues.apache.org/jira/browse/FLINK-18121 > Project: Flink > Issue Type: Improvement > Components: Dockerfiles >Affects Versions: docker-1.11.0.0 >Reporter: Till Rohrmann >Priority: Major > > Currently, > https://github.com/apache/flink-docker/blob/dev-master/Dockerfile-debian.template > only supports to create a Docker image from a Flink distribution which is > hosted on a web server. I think it would be helpful if we could also create a > Docker image from a Flink distribution which is stored on one's local file > system. That way, one would not have to upload the file or start a web server > for serving it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12460: [FLINK-18063][checkpointing] Fix the race condition for aborting current checkpoint in CheckpointBarrierUnaligner
flinkbot edited a comment on pull request #12460: URL: https://github.com/apache/flink/pull/12460#issuecomment-638102290 ## CI report: * 1d838c03d2b9f9744cae6fc03a919db72cc0efd9 UNKNOWN * b2c4d6aaf78612e7214cc49d22e2b90fe29b93b6 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2686) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12455: [FLINK-17935] Move setting yarn.log-config-file to YarnClusterClientFactory
flinkbot edited a comment on pull request #12455: URL: https://github.com/apache/flink/pull/12455#issuecomment-638007072 ## CI report: * 7e9143f4531e9a7477a1e2848020d7c69644a3be Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2666) * 278da87279e640156951deef07ee9e4e24cab361 UNKNOWN * 76cf3b92819b6a77bf9c50b6d7674f5df3f4c8f7 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12352: [FLINK-17717][sql-parser] Throws for DDL create temporary system func…
flinkbot edited a comment on pull request #12352: URL: https://github.com/apache/flink/pull/12352#issuecomment-634517986 ## CI report: * 6900843764e0fe637babcb429194ef620be73b50 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2278) * 7dc817125d0684987814d45192eab87a41ebc5b5 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12436: [FLINK-17847][table sql / planner] ArrayIndexOutOfBoundsException happens in StreamExecCalc operator
flinkbot edited a comment on pull request #12436: URL: https://github.com/apache/flink/pull/12436#issuecomment-637399261 ## CI report: * d553feca300e2668e3e5c16c1c524d73acde141c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2627) * 4d1f156fd758a44f4c394a478b6f37bc2de61c86 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2711) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12456: [FLINK-17113][sql-cli] Refactor view support in SQL Client
flinkbot edited a comment on pull request #12456: URL: https://github.com/apache/flink/pull/12456#issuecomment-638032908 ## CI report: * b5b8172d4ccddf197a084d38e6dcef13ee27da8e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2614) * 12e3932ed8d74a59c0d4c28441b336284a2dffeb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2713) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12481: [FLINK-18120] Don't expand documentation sections by default
flinkbot edited a comment on pull request #12481: URL: https://github.com/apache/flink/pull/12481#issuecomment-638746420 ## CI report: * 57276170065599217eaf7a77f451d74a9311a870 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2714) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong closed pull request #12441: [FLINK-17918][table-blink] Fix AppendOnlyTopNFunction shouldn't mutate list value of MapState
wuchong closed pull request #12441: URL: https://github.com/apache/flink/pull/12441 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong commented on pull request #12441: [FLINK-17918][table-blink] Fix AppendOnlyTopNFunction shouldn't mutate list value of MapState
wuchong commented on pull request #12441: URL: https://github.com/apache/flink/pull/12441#issuecomment-638761387 Thanks for the reviewing @AHeise . Will merge this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wuchong commented on pull request #12436: [FLINK-17847][table sql / planner] ArrayIndexOutOfBoundsException happens in StreamExecCalc operator
wuchong commented on pull request #12436: URL: https://github.com/apache/flink/pull/12436#issuecomment-638765700 Do you have other concerns @libenchao ? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12435: [FLINK-18059] [sql-client] Fix create/drop catalog statement can not be executed in sql client
flinkbot edited a comment on pull request #12435: URL: https://github.com/apache/flink/pull/12435#issuecomment-637372360 ## CI report: * b5ef005c90fb95a63c97d06c488adf5e4abe2972 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2626) * 223701abe70c0815eb1f35af7383727385ce2e90 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12436: [FLINK-17847][table sql / planner] ArrayIndexOutOfBoundsException happens in StreamExecCalc operator
flinkbot edited a comment on pull request #12436: URL: https://github.com/apache/flink/pull/12436#issuecomment-637399261 ## CI report: * d553feca300e2668e3e5c16c1c524d73acde141c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2627) * 4d1f156fd758a44f4c394a478b6f37bc2de61c86 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2711) * 63831f0ba3dc2a6ec31acc7fb1cfd1b51288945b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12326: [FLINK-16577] [table-planner-blink] Fix numeric type mismatch error in column interval relmetadata
flinkbot edited a comment on pull request #12326: URL: https://github.com/apache/flink/pull/12326#issuecomment-633786013 ## CI report: * 0b14ce7cf01c6664a3a98c35dd1c494a7ad4fc32 UNKNOWN * 23311a70dcb1522127aaadf906875d536b64c166 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2171) * 8c198bc1bb0b6be3321a01898e09d7492169b47e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18071) CoordinatorEventsExactlyOnceITCase.checkListContainsSequence fails on CI
[
https://issues.apache.org/jira/browse/FLINK-18071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-18071:
---
Affects Version/s: 1.11.0
> CoordinatorEventsExactlyOnceITCase.checkListContainsSequence fails on CI
>
>
> Key: FLINK-18071
> URL: https://issues.apache.org/jira/browse/FLINK-18071
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination, Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> CI:
> https://dev.azure.com/georgeryan1322/Flink/_build/results?buildId=330=logs=6e58d712-c5cc-52fb-0895-6ff7bd56c46b=f30a8e80-b2cf-535c-9952-7f521a4ae374
> {code}
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.795
> s <<< FAILURE! - in
> org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase
> [ERROR]
> test(org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase)
> Time elapsed: 4.647 s <<< FAILURE!
> java.lang.AssertionError: List did not contain expected sequence of 200
> elements, but was: [152, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
> 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,
> 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52,
> 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71,
> 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90,
> 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107,
> 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122,
> 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137,
> 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152,
> 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167,
> 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182,
> 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197,
> 198, 199]
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase.failList(CoordinatorEventsExactlyOnceITCase.java:160)
> at
> org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase.checkListContainsSequence(CoordinatorEventsExactlyOnceITCase.java:148)
> at
> org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase.test(CoordinatorEventsExactlyOnceITCase.java:143)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18122) Kubernetes test fails with "error: timed out waiting for the condition on jobs/flink-job-cluster"
Robert Metzger created FLINK-18122:
--
Summary: Kubernetes test fails with "error: timed out waiting for
the condition on jobs/flink-job-cluster"
Key: FLINK-18122
URL: https://issues.apache.org/jira/browse/FLINK-18122
Project: Flink
Issue Type: Bug
Components: Deployment / Kubernetes, Tests
Affects Versions: 1.11.0
Reporter: Robert Metzger
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2697=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
{code}
2020-06-04T09:25:40.7205843Z service/flink-job-cluster created
2020-06-04T09:25:40.9661515Z job.batch/flink-job-cluster created
2020-06-04T09:25:41.2189123Z deployment.apps/flink-task-manager created
2020-06-04T10:32:32.6402983Z error: timed out waiting for the condition on
jobs/flink-job-cluster
2020-06-04T10:32:33.8057757Z error: unable to upgrade connection: container not
found ("flink-task-manager")
2020-06-04T10:32:33.8111302Z sort: cannot read:
'/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-56335570120/out/kubernetes_wc_out*':
No such file or directory
2020-06-04T10:32:33.8124455Z FAIL WordCount: Output hash mismatch. Got
d41d8cd98f00b204e9800998ecf8427e, expected e682ec6622b5e83f2eb614617d5ab2cf.
2020-06-04T10:32:33.8125379Z head hexdump of actual:
2020-06-04T10:32:33.8136133Z head: cannot open
'/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-56335570120/out/kubernetes_wc_out*'
for reading: No such file or directory
2020-06-04T10:32:33.8344715Z Debugging failed Kubernetes test:
2020-06-04T10:32:33.8345469Z Currently existing Kubernetes resources
2020-06-04T10:32:36.4977853Z I0604 10:32:36.497383 13191 request.go:621]
Throttling request took 1.198606989s, request:
GET:https://10.1.0.4:8443/apis/rbac.authorization.k8s.io/v1?timeout=32s
2020-06-04T10:32:46.6975735Z I0604 10:32:46.697234 13191 request.go:621]
Throttling request took 4.398107353s, request:
GET:https://10.1.0.4:8443/apis/authorization.k8s.io/v1?timeout=32s
2020-06-04T10:32:57.4978637Z I0604 10:32:57.497209 13191 request.go:621]
Throttling request took 1.198449167s, request:
GET:https://10.1.0.4:8443/apis/apps/v1?timeout=32s
2020-06-04T10:33:07.4980104Z I0604 10:33:07.497320 13191 request.go:621]
Throttling request took 4.198274438s, request:
GET:https://10.1.0.4:8443/apis/apiextensions.k8s.io/v1?timeout=32s
2020-06-04T10:33:18.4976060Z I0604 10:33:18.497258 13191 request.go:621]
Throttling request took 1.19871495s, request:
GET:https://10.1.0.4:8443/apis/apps/v1?timeout=32s
2020-06-04T10:33:28.4979129Z I0604 10:33:28.497276 13191 request.go:621]
Throttling request took 4.198369672s, request:
GET:https://10.1.0.4:8443/apis/rbac.authorization.k8s.io/v1?timeout=32s
2020-06-04T10:33:30.9182069Z NAME READY
STATUS RESTARTS AGE
2020-06-04T10:33:30.9184099Z pod/flink-job-cluster-dtb67 0/1
ErrImageNeverPull 0 67m
2020-06-04T10:33:30.9184869Z pod/flink-task-manager-74ccc9bd9-psqwm 0/1
ErrImageNeverPull 0 67m
2020-06-04T10:33:30.9185226Z
2020-06-04T10:33:30.9185926Z NAMETYPECLUSTER-IP
EXTERNAL-IP PORT(S)
AGE
2020-06-04T10:33:30.9186832Z service/flink-job-cluster NodePort
10.111.92.199
6123:32501/TCP,6124:31360/TCP,6125:30025/TCP,8081:30081/TCP 67m
2020-06-04T10:33:30.9187545Z service/kubernetes ClusterIP 10.96.0.1
443/TCP
68m
2020-06-04T10:33:30.9187976Z
2020-06-04T10:33:30.9188472Z NAME READY
UP-TO-DATE AVAILABLE AGE
2020-06-04T10:33:30.9189179Z deployment.apps/flink-task-manager 0/1 1
0 67m
2020-06-04T10:33:30.9189508Z
2020-06-04T10:33:30.9189815Z NAME
DESIRED CURRENT READY AGE
2020-06-04T10:33:30.9190418Z replicaset.apps/flink-task-manager-74ccc9bd9 1
1 0 67m
2020-06-04T10:33:30.9190662Z
2020-06-04T10:33:30.9190891Z NAME COMPLETIONS
DURATION AGE
2020-06-04T10:33:30.9191423Z job.batch/flink-job-cluster 0/1 67m
67m
2020-06-04T10:33:33.7840921Z I0604 10:33:33.783675 13482 request.go:621]
Throttling request took 1.198522435s, request:
GET:https://10.1.0.4:8443/apis/batch/v1beta1?timeout=32s
2020-06-04T10:33:43.7842354Z I0604 10:33:43.783896 13482 request.go:621]
Throttling request took 4.198512626s, request:
GET:https://10.1.0.4:8443/apis/rbac.authorization.k8s.io/v1?timeout=32s
2020-06-04T10:33:54.7840871Z I0604 10:33:54.783716 13482 request.go:621]
Throttling request took 1.198557318s, request:
[jira] [Updated] (FLINK-18071) CoordinatorEventsExactlyOnceITCase.checkListContainsSequence fails on CI
[
https://issues.apache.org/jira/browse/FLINK-18071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-18071:
---
Fix Version/s: 1.11.0
> CoordinatorEventsExactlyOnceITCase.checkListContainsSequence fails on CI
>
>
> Key: FLINK-18071
> URL: https://issues.apache.org/jira/browse/FLINK-18071
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination, Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: test-stability
> Fix For: 1.11.0, 1.12.0
>
>
> CI:
> https://dev.azure.com/georgeryan1322/Flink/_build/results?buildId=330=logs=6e58d712-c5cc-52fb-0895-6ff7bd56c46b=f30a8e80-b2cf-535c-9952-7f521a4ae374
> {code}
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.795
> s <<< FAILURE! - in
> org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase
> [ERROR]
> test(org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase)
> Time elapsed: 4.647 s <<< FAILURE!
> java.lang.AssertionError: List did not contain expected sequence of 200
> elements, but was: [152, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
> 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,
> 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52,
> 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71,
> 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90,
> 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107,
> 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122,
> 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137,
> 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152,
> 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167,
> 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182,
> 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197,
> 198, 199]
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase.failList(CoordinatorEventsExactlyOnceITCase.java:160)
> at
> org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase.checkListContainsSequence(CoordinatorEventsExactlyOnceITCase.java:148)
> at
> org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase.test(CoordinatorEventsExactlyOnceITCase.java:143)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] pnowojski edited a comment on pull request #11541: [FLINK-15416][network] add task manager netty client retry mechenism
pnowojski edited a comment on pull request #11541: URL: https://github.com/apache/flink/pull/11541#issuecomment-638644705 There are a couple of things left to do there. 1. For example test is marked with Ignore and I’m not sure if that’s a mistake, as the commit marking it ignored seemed to be fixing something. But it was 5 years ago so who knows… removing Ignore might introduce test instability, so it would have to be thoroughly checked. 2. There are rebase conflicts. 3. I think in your test you should be joining the spawned threads and checking their’s exceptions. If you prefer we can take over those issues and merge this PR in a couple of weeks - after release 1.11 testing is done. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] StephanEwen commented on pull request #12481: [FLINK-18120] Don't expand documentation sections by default
StephanEwen commented on pull request #12481: URL: https://github.com/apache/flink/pull/12481#issuecomment-638791801 +1 This makes sense. The docs page currently shows too much contents immediately, which can be overwhelming. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] krasinski edited a comment on pull request #12462: [FLINK-18045] Fix Kerberos credentials checking
krasinski edited a comment on pull request #12462: URL: https://github.com/apache/flink/pull/12462#issuecomment-638494158 thank you for the review @walterddr, I've addressed all the comments and added some more unit tests I've also introduced a small change in `hasHDFSDelegationToken` method - taking advantage of the UGI parameter instead of retrieving it one more time seems more natural edit: need to find another way to init kerberos config, tests worked on CI but failed on CI This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18122) Kubernetes test fails with "error: timed out waiting for the condition on jobs/flink-job-cluster"
[
https://issues.apache.org/jira/browse/FLINK-18122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125855#comment-17125855
]
Robert Metzger commented on FLINK-18122:
Ah, good catch.
The test should have a better error handling to fail if the docker image build
failed?
> Kubernetes test fails with "error: timed out waiting for the condition on
> jobs/flink-job-cluster"
> -
>
> Key: FLINK-18122
> URL: https://issues.apache.org/jira/browse/FLINK-18122
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2697=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> {code}
> 2020-06-04T09:25:40.7205843Z service/flink-job-cluster created
> 2020-06-04T09:25:40.9661515Z job.batch/flink-job-cluster created
> 2020-06-04T09:25:41.2189123Z deployment.apps/flink-task-manager created
> 2020-06-04T10:32:32.6402983Z error: timed out waiting for the condition on
> jobs/flink-job-cluster
> 2020-06-04T10:32:33.8057757Z error: unable to upgrade connection: container
> not found ("flink-task-manager")
> 2020-06-04T10:32:33.8111302Z sort: cannot read:
> '/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-56335570120/out/kubernetes_wc_out*':
> No such file or directory
> 2020-06-04T10:32:33.8124455Z FAIL WordCount: Output hash mismatch. Got
> d41d8cd98f00b204e9800998ecf8427e, expected e682ec6622b5e83f2eb614617d5ab2cf.
> 2020-06-04T10:32:33.8125379Z head hexdump of actual:
> 2020-06-04T10:32:33.8136133Z head: cannot open
> '/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-56335570120/out/kubernetes_wc_out*'
> for reading: No such file or directory
> 2020-06-04T10:32:33.8344715Z Debugging failed Kubernetes test:
> 2020-06-04T10:32:33.8345469Z Currently existing Kubernetes resources
> 2020-06-04T10:32:36.4977853Z I0604 10:32:36.497383 13191 request.go:621]
> Throttling request took 1.198606989s, request:
> GET:https://10.1.0.4:8443/apis/rbac.authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:32:46.6975735Z I0604 10:32:46.697234 13191 request.go:621]
> Throttling request took 4.398107353s, request:
> GET:https://10.1.0.4:8443/apis/authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:32:57.4978637Z I0604 10:32:57.497209 13191 request.go:621]
> Throttling request took 1.198449167s, request:
> GET:https://10.1.0.4:8443/apis/apps/v1?timeout=32s
> 2020-06-04T10:33:07.4980104Z I0604 10:33:07.497320 13191 request.go:621]
> Throttling request took 4.198274438s, request:
> GET:https://10.1.0.4:8443/apis/apiextensions.k8s.io/v1?timeout=32s
> 2020-06-04T10:33:18.4976060Z I0604 10:33:18.497258 13191 request.go:621]
> Throttling request took 1.19871495s, request:
> GET:https://10.1.0.4:8443/apis/apps/v1?timeout=32s
> 2020-06-04T10:33:28.4979129Z I0604 10:33:28.497276 13191 request.go:621]
> Throttling request took 4.198369672s, request:
> GET:https://10.1.0.4:8443/apis/rbac.authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:33:30.9182069Z NAME READY
> STATUS RESTARTS AGE
> 2020-06-04T10:33:30.9184099Z pod/flink-job-cluster-dtb67 0/1
> ErrImageNeverPull 0 67m
> 2020-06-04T10:33:30.9184869Z pod/flink-task-manager-74ccc9bd9-psqwm 0/1
> ErrImageNeverPull 0 67m
> 2020-06-04T10:33:30.9185226Z
> 2020-06-04T10:33:30.9185926Z NAMETYPE
> CLUSTER-IP EXTERNAL-IP PORT(S)
> AGE
> 2020-06-04T10:33:30.9186832Z service/flink-job-cluster NodePort
> 10.111.92.199
> 6123:32501/TCP,6124:31360/TCP,6125:30025/TCP,8081:30081/TCP 67m
> 2020-06-04T10:33:30.9187545Z service/kubernetes ClusterIP
> 10.96.0.1 443/TCP
> 68m
> 2020-06-04T10:33:30.9187976Z
> 2020-06-04T10:33:30.9188472Z NAME READY
> UP-TO-DATE AVAILABLE AGE
> 2020-06-04T10:33:30.9189179Z deployment.apps/flink-task-manager 0/1 1
> 0 67m
> 2020-06-04T10:33:30.9189508Z
> 2020-06-04T10:33:30.9189815Z NAME
> DESIRED CURRENT READY AGE
> 2020-06-04T10:33:30.9190418Z replicaset.apps/flink-task-manager-74ccc9bd9 1
> 1 0 67m
> 2020-06-04T10:33:30.9190662Z
> 2020-06-04T10:33:30.9190891Z NAME COMPLETIONS
> DURATION AGE
> 2020-06-04T10:33:30.9191423Z job.batch/flink-job-cluster 0/1
[jira] [Created] (FLINK-18124) Add documentation for new FLIP-27 source interface
Stephan Ewen created FLINK-18124:
Summary: Add documentation for new FLIP-27 source interface
Key: FLINK-18124
URL: https://issues.apache.org/jira/browse/FLINK-18124
Project: Flink
Issue Type: Sub-task
Components: Documentation
Reporter: Stephan Ewen
Assignee: Stephan Ewen
Fix For: 1.11.0
The documentation should be under {{Application Development / DataStream API /
Data Sources }}.
We need to add sections about
- Data Source Concepts
- Data Source API
- Connector Base (Split Reader Library)
- Event Time and Watermarks
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12468: [FLINK-18075] Wrap the SerializationSchema in KafkaSerializationSchema in Kafka connector
flinkbot edited a comment on pull request #12468: URL: https://github.com/apache/flink/pull/12468#issuecomment-638243685 ## CI report: * 4ae1da7c5aee0ff73e4461f0d9235dd60857f1d7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2651) * 7bf026a4e23a89df952f6b361ab8fe435c0a612d UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12326: [FLINK-16577] [table-planner-blink] Fix numeric type mismatch error in column interval relmetadata
flinkbot edited a comment on pull request #12326: URL: https://github.com/apache/flink/pull/12326#issuecomment-633786013 ## CI report: * 0b14ce7cf01c6664a3a98c35dd1c494a7ad4fc32 UNKNOWN * 23311a70dcb1522127aaadf906875d536b64c166 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2171) * 8c198bc1bb0b6be3321a01898e09d7492169b47e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2725) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #11541: [FLINK-15416][network] add task manager netty client retry mechenism
flinkbot edited a comment on pull request #11541: URL: https://github.com/apache/flink/pull/11541#issuecomment-604812212 ## CI report: * b5ef19285000d692d6b1b8b0aa7a37ce35c216f5 UNKNOWN * baf2c55f7bb0b9df7bdef94083c947693b75eae2 UNKNOWN * e5419f41a18eaae05814837d679f58480a2cb94c UNKNOWN * 34a70a74c479f177784b5ba4983b50072c4f7e5f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2702) * f64492a673ed412c925b42152c4ce469d4ccbb0c Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2715) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18125) Skip CI execution on documentation pull requests
[ https://issues.apache.org/jira/browse/FLINK-18125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125858#comment-17125858 ] Robert Metzger commented on FLINK-18125: [~chesnay] I'm not sure if we can skip the entire CI execution, because I don't know if the tests contain something for the docs. If that's the case, we could at least skip e2e execution. > Skip CI execution on documentation pull requests > > > Key: FLINK-18125 > URL: https://issues.apache.org/jira/browse/FLINK-18125 > Project: Flink > Issue Type: Improvement > Components: Build System / Azure Pipelines >Reporter: Robert Metzger >Priority: Major > > In order to save some resources, we can skip the CI execution on > documentation-only changes (whole changeset). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12478: [FLINK-17869][task][checkpointing] Fix race condition when caling ChannelStateWriter.abort
flinkbot edited a comment on pull request #12478: URL: https://github.com/apache/flink/pull/12478#issuecomment-638685986 ## CI report: * bb036ae536e7705d8dfd8b8a5654198bbf4fa5c2 UNKNOWN * 420b7b3f645625b0b43f056440509cdd53bbd4ef Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2694) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18103) Test Pandas DataFrame and Flink Table conversion
[ https://issues.apache.org/jira/browse/FLINK-18103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125746#comment-17125746 ] Dian Fu commented on FLINK-18103: - Verified the following: Convert from a PyFlink Table to a Pandas DataFrame * batch mode of old planner and blink planner * stream mode with bounded data set of old planner and blink planner (only supports append stream at present) * test different data types such as float, array, row, etc. Convert from a Pandas DataFrame to a PyFlink Table * batch mode of blink planner (doesn't support batch mode of old planner) * stream mode of old planner and blink planner * test that it could infer the table schema from the given pandas DataFrame * test DataFrames with NaN data > Test Pandas DataFrame and Flink Table conversion > > > Key: FLINK-18103 > URL: https://issues.apache.org/jira/browse/FLINK-18103 > Project: Flink > Issue Type: Test > Components: API / Python >Reporter: sunjincheng >Assignee: Dian Fu >Priority: Blocker > Labels: release-testing > Fix For: 1.11.0 > > > 1) Test convert a Pandas DataFrame to Flink Table > 2) Test convert a Flink Table to a Pandas DataFrame -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-18103) Test Pandas DataFrame and Flink Table conversion
[ https://issues.apache.org/jira/browse/FLINK-18103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu closed FLINK-18103. --- Resolution: Done > Test Pandas DataFrame and Flink Table conversion > > > Key: FLINK-18103 > URL: https://issues.apache.org/jira/browse/FLINK-18103 > Project: Flink > Issue Type: Test > Components: API / Python >Reporter: sunjincheng >Assignee: Dian Fu >Priority: Blocker > Labels: release-testing > Fix For: 1.11.0 > > > 1) Test convert a Pandas DataFrame to Flink Table > 2) Test convert a Flink Table to a Pandas DataFrame -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] kl0u closed pull request #12463: [FLINK-18087][yarn] Fix uploading user artifact for Yarn job cluster
kl0u closed pull request #12463: URL: https://github.com/apache/flink/pull/12463 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-18087) Uploading user artifact for Yarn job cluster could not work
[
https://issues.apache.org/jira/browse/FLINK-18087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kostas Kloudas closed FLINK-18087.
--
Resolution: Fixed
Fixed on master with d56454a3a0abe9ba34d1fec27864863bbd8a7c5f
and on release-1.11 with f394995281f13c888303025735adb56cc952
> Uploading user artifact for Yarn job cluster could not work
> ---
>
> Key: FLINK-18087
> URL: https://issues.apache.org/jira/browse/FLINK-18087
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Yang Wang
>Assignee: Yang Wang
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.0, 1.12.0
>
>
> In FLINK-17632, we add the support remote user jar. However, uploading user
> artifact for Yarn job cluster is broken exceptionally. In the following code,
> we should only upload local files. Now it has the contrary behavior.
> {code:java}
> // only upload local files
> if (Utils.isRemotePath(entry.getValue().filePath)) {
>Path localPath = new Path(entry.getValue().filePath);
>Tuple2 remoteFileInfo =
> fileUploader.uploadLocalFileToRemote(localPath, entry.getKey());
>jobGraph.setUserArtifactRemotePath(entry.getKey(),
> remoteFileInfo.f0.toString());
> }
> {code}
>
> Another problem is the related tests {{testPerJobModeWithDistributedCache}}
> does not fail because we do not fetch the artifact from remote
> filesystem(i.e. HDFS). We directly get it from local file. It also needs to
> be enhanced.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wuchong closed pull request #12381: [FLINK-17941][sql-client] Switching catalog or database doesn't work …
wuchong closed pull request #12381: URL: https://github.com/apache/flink/pull/12381 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhuzhurk commented on a change in pull request #12375: [FLINK-17017][runtime] Implements bulk allocation for physical slots
zhuzhurk commented on a change in pull request #12375: URL: https://github.com/apache/flink/pull/12375#discussion_r435126559 ## File path: flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/SlotPoolImpl.java ## @@ -447,12 +452,40 @@ public void releaseSlot(@Nonnull SlotRequestId slotRequestId, @Nullable Throwabl componentMainThreadExecutor.assertRunningInMainThread(); + checkState(batchSlotTimeoutCheckEnabled, "batch slot timeout check is disabled unexpectedly."); + final PendingRequest pendingRequest = PendingRequest.createBatchRequest(slotRequestId, resourceProfile); return requestNewAllocatedSlotInternal(pendingRequest) .thenApply(Function.identity()); } + @Override + public CompletableFuture requestNewAllocatedSlotWithoutTimeout( Review comment: Agreed it would be better to explicitly disable the timeout check. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12456: [FLINK-17113][sql-cli] Refactor view support in SQL Client
flinkbot edited a comment on pull request #12456: URL: https://github.com/apache/flink/pull/12456#issuecomment-638032908 ## CI report: * b5b8172d4ccddf197a084d38e6dcef13ee27da8e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2614) * 12e3932ed8d74a59c0d4c28441b336284a2dffeb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12480: [hotfix] Remove redundant backslash in the log of GPUDriver
flinkbot edited a comment on pull request #12480: URL: https://github.com/apache/flink/pull/12480#issuecomment-638732843 ## CI report: * 0704f5331a783e4d3dbd7c404e4b80c27e7b6a60 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2707) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12436: [FLINK-17847][table sql / planner] ArrayIndexOutOfBoundsException happens in StreamExecCalc operator
flinkbot edited a comment on pull request #12436: URL: https://github.com/apache/flink/pull/12436#issuecomment-637399261 ## CI report: * d553feca300e2668e3e5c16c1c524d73acde141c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2627) * 4d1f156fd758a44f4c394a478b6f37bc2de61c86 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12481: [FLINK-18120] Don't expand documentation sections by default
flinkbot commented on pull request #12481: URL: https://github.com/apache/flink/pull/12481#issuecomment-638746420 ## CI report: * 57276170065599217eaf7a77f451d74a9311a870 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12455: [FLINK-17935] Move setting yarn.log-config-file to YarnClusterClientFactory
flinkbot edited a comment on pull request #12455: URL: https://github.com/apache/flink/pull/12455#issuecomment-638007072 ## CI report: * 7e9143f4531e9a7477a1e2848020d7c69644a3be Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2666) * 278da87279e640156951deef07ee9e4e24cab361 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18075) Kafka connector does not call open method of (de)serialization schema
[ https://issues.apache.org/jira/browse/FLINK-18075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aljoscha Krettek updated FLINK-18075: - Issue Type: Bug (was: Improvement) > Kafka connector does not call open method of (de)serialization schema > - > > Key: FLINK-18075 > URL: https://issues.apache.org/jira/browse/FLINK-18075 > Project: Flink > Issue Type: Bug > Components: Connectors / Kafka, Formats (JSON, Avro, Parquet, ORC, > SequenceFile) >Affects Versions: 1.11.0, 1.12.0 >Reporter: Seth Wiesman >Assignee: Dawid Wysakowicz >Priority: Blocker > Labels: pull-request-available > Fix For: 1.11.0 > > > The Kafka consumer and producer do not call the open methods of plain > (De)SerializationSchema interfaces. Only the Keyed and Kafka specific > interfaces. The updated SQL implementations such as > AvroRowDataSeriailzationSchema use these methods and so SQL queries using > avro and kafka will fail in a null pointer exception. > cc [~aljoscha] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18073) AvroRowDataSerializationSchema is not always serializable
[
https://issues.apache.org/jira/browse/FLINK-18073?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek updated FLINK-18073:
-
Issue Type: Bug (was: Improvement)
> AvroRowDataSerializationSchema is not always serializable
> -
>
> Key: FLINK-18073
> URL: https://issues.apache.org/jira/browse/FLINK-18073
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile), Table
> SQL / Ecosystem
>Reporter: Seth Wiesman
>Assignee: Jark Wu
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.0, 1.12.0
>
>
> AvroRowDeserializationSchema member runtimeConverter may hold a reference to
> non serializable avro schema fields. This value should be transient and
> generated inside of open.
> {code:java}
> Caused by: java.io.NotSerializableException:
> org.apache.avro.Schema$RecordSchema
> at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1184)
> ~[?:1.8.0_252]
> at java.io.ObjectOutputStream.writeArray(ObjectOutputStream.java:1378)
> ~[?:1.8.0_252]
> at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
> ~[?:1.8.0_252]
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
> ~[?:1.8.0_252]
> at
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
> ~[?:1.8.0_252]
> at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
> ~[?:1.8.0_252]
> at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
> ~[?:1.8.0_252]
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1548)
> ~[?:1.8.0_252]
> at
> java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1509)
> ~[?:1.8.0_252]
> at
> java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1432)
> ~[?:1.8.0_252]
> at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1178)
> ~[?:1.8.0_252]
> at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:348)
> ~[?:1.8.0_252]
> at
> org.apache.flink.util.InstantiationUtil.serializeObject(InstantiationUtil.java:586)
> ~[flink-dist_2.12-1.11-stream.jar:1.11-stream]
> at
> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:133)
> ~[flink-dist_2.12-1.11-stream.jar:1.11-stream]
> at
> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:126)
> ~[flink-dist_2.12-1.11-stream.jar:1.11-stream]
> at
> org.apache.flink.api.java.ClosureCleaner.clean(ClosureCleaner.java:71)
> ~[flink-dist_2.12-1.11-stream.jar:1.11-stream]
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.(FlinkKafkaProducer.java:632)
> ~[flink-sql-connector-kafka_2.12-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.(FlinkKafkaProducer.java:522)
> ~[flink-sql-connector-kafka_2.12-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
> at
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducer.(FlinkKafkaProducer.java:483)
> ~[flink-sql-connector-kafka_2.12-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
> at
> org.apache.flink.streaming.connectors.kafka.table.KafkaDynamicSink.createKafkaProducer(KafkaDynamicSink.java:60)
> ~[flink-sql-connector-kafka_2.12-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
> at
> org.apache.flink.streaming.connectors.kafka.table.KafkaDynamicSinkBase.getSinkRuntimeProvider(KafkaDynamicSinkBase.java:84)
> ~[flink-sql-connector-kafka_2.12-1.11-SNAPSHOT.jar:1.11-SNAPSHOT]
> at
> org.apache.flink.table.planner.plan.nodes.common.CommonPhysicalSink.createSinkTransformation(CommonPhysicalSink.scala:69)
> ~[flink-table-blink_2.12-1.11-stream.jar:1.11-stream]
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToPlanInternal(StreamExecSink.scala:108)
> ~[flink-table-blink_2.12-1.11-stream.jar:1.11-stream]
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToPlanInternal(StreamExecSink.scala:43)
> ~[flink-table-blink_2.12-1.11-stream.jar:1.11-stream]
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNode.translateToPlan(ExecNode.scala:58)
> ~[flink-table-blink_2.12-1.11-stream.jar:1.11-stream]
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNode.translateToPlan$(ExecNode.scala:56)
> ~[flink-table-blink_2.12-1.11-stream.jar:1.11-stream]
> at
> org.apache.flink.table.planner.plan.nodes.physical.stream.StreamExecSink.translateToPlan(StreamExecSink.scala:43)
> ~[flink-table-blink_2.12-1.11-stream.jar:1.11-stream]
> at
>
[jira] [Updated] (FLINK-17935) Logs could not show up when deploying Flink on Yarn via "--executor"
[
https://issues.apache.org/jira/browse/FLINK-17935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Kostas Kloudas updated FLINK-17935:
---
Priority: Blocker (was: Critical)
> Logs could not show up when deploying Flink on Yarn via "--executor"
>
>
> Key: FLINK-17935
> URL: https://issues.apache.org/jira/browse/FLINK-17935
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Yang Wang
>Assignee: Kostas Kloudas
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> {code:java}
> ./bin/flink run -d -p 5 -e yarn-per-job
> examples/streaming/WindowJoin.jar{code}
> When we use the {{-e/--executor}} to specify the deploy target to Yarn
> per-job, the logs could not show up. The root cause is we do not set the
> logging files in {{ExecutorCLI}}. We only do it in the
> {{FlinkYarnSessionCli}}.
> If we use {{-m yarn-cluster}}, everything works well.
>
> Maybe we should move the {{setLogConfigFileInConfig}} to
> {{YarnClusterDescriptor}} to avoid this problem. cc [~kkl0u]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-17824) "Resuming Savepoint" e2e stalls indefinitely
[
https://issues.apache.org/jira/browse/FLINK-17824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125795#comment-17125795
]
Robert Metzger commented on FLINK-17824:
Another instance on master (which makes the Hadoop 3.1.3 theory less likely)
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2696=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> "Resuming Savepoint" e2e stalls indefinitely
> -
>
> Key: FLINK-17824
> URL: https://issues.apache.org/jira/browse/FLINK-17824
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Tests
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Major
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> CI;
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=1887=logs=91bf6583-3fb2-592f-e4d4-d79d79c3230a=94459a52-42b6-5bfc-5d74-690b5d3c6de8
> {code}
> 2020-05-19T21:05:52.9696236Z
> ==
> 2020-05-19T21:05:52.9696860Z Running 'Resuming Savepoint (file, async, scale
> down) end-to-end test'
> 2020-05-19T21:05:52.9697243Z
> ==
> 2020-05-19T21:05:52.9713094Z TEST_DATA_DIR:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-52970362751
> 2020-05-19T21:05:53.1194478Z Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-1.12-SNAPSHOT-bin/flink-1.12-SNAPSHOT
> 2020-05-19T21:05:53.2180375Z Starting cluster.
> 2020-05-19T21:05:53.9986167Z Starting standalonesession daemon on host
> fv-az558.
> 2020-05-19T21:05:55.5997224Z Starting taskexecutor daemon on host fv-az558.
> 2020-05-19T21:05:55.6223837Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-05-19T21:05:57.0552482Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-05-19T21:05:57.9446865Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-05-19T21:05:59.0098434Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-05-19T21:06:00.0569710Z Dispatcher REST endpoint is up.
> 2020-05-19T21:06:07.7099937Z Job (a92a74de8446a80403798bb4806b73f3) is
> running.
> 2020-05-19T21:06:07.7855906Z Waiting for job to process up to 200 records,
> current progress: 114 records ...
> 2020-05-19T21:06:55.5755111Z
> 2020-05-19T21:06:55.5756550Z
>
> 2020-05-19T21:06:55.5757225Z The program finished with the following
> exception:
> 2020-05-19T21:06:55.5757566Z
> 2020-05-19T21:06:55.5765453Z org.apache.flink.util.FlinkException: Could not
> stop with a savepoint job "a92a74de8446a80403798bb4806b73f3".
> 2020-05-19T21:06:55.5766873Z at
> org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:485)
> 2020-05-19T21:06:55.5767980Z at
> org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:854)
> 2020-05-19T21:06:55.5769014Z at
> org.apache.flink.client.cli.CliFrontend.stop(CliFrontend.java:477)
> 2020-05-19T21:06:55.5770052Z at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:921)
> 2020-05-19T21:06:55.5771107Z at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:982)
> 2020-05-19T21:06:55.5772223Z at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
> 2020-05-19T21:06:55.5773325Z at
> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:982)
> 2020-05-19T21:06:55.5774871Z Caused by:
> java.util.concurrent.ExecutionException:
> java.util.concurrent.CompletionException:
> java.util.concurrent.CompletionException:
> org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint
> Coordinator is suspending.
> 2020-05-19T21:06:55.5777183Z at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2020-05-19T21:06:55.5778884Z at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
> 2020-05-19T21:06:55.5779920Z at
> org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:483)
> 2020-05-19T21:06:55.5781175Z ... 6 more
> 2020-05-19T21:06:55.5782391Z Caused by:
> java.util.concurrent.CompletionException:
> java.util.concurrent.CompletionException:
> org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint
> Coordinator is suspending.
> 2020-05-19T21:06:55.5783885Z at
> org.apache.flink.runtime.scheduler.SchedulerBase.lambda$stopWithSavepoint$9(SchedulerBase.java:890)
> 2020-05-19T21:06:55.5784992Z at
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
> 2020-05-19T21:06:55.5786492Z
[jira] [Updated] (FLINK-17824) "Resuming Savepoint" e2e stalls indefinitely
[
https://issues.apache.org/jira/browse/FLINK-17824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Robert Metzger updated FLINK-17824:
---
Priority: Critical (was: Major)
> "Resuming Savepoint" e2e stalls indefinitely
> -
>
> Key: FLINK-17824
> URL: https://issues.apache.org/jira/browse/FLINK-17824
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Tests
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Assignee: Robert Metzger
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> CI;
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=1887=logs=91bf6583-3fb2-592f-e4d4-d79d79c3230a=94459a52-42b6-5bfc-5d74-690b5d3c6de8
> {code}
> 2020-05-19T21:05:52.9696236Z
> ==
> 2020-05-19T21:05:52.9696860Z Running 'Resuming Savepoint (file, async, scale
> down) end-to-end test'
> 2020-05-19T21:05:52.9697243Z
> ==
> 2020-05-19T21:05:52.9713094Z TEST_DATA_DIR:
> /home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-52970362751
> 2020-05-19T21:05:53.1194478Z Flink dist directory:
> /home/vsts/work/1/s/flink-dist/target/flink-1.12-SNAPSHOT-bin/flink-1.12-SNAPSHOT
> 2020-05-19T21:05:53.2180375Z Starting cluster.
> 2020-05-19T21:05:53.9986167Z Starting standalonesession daemon on host
> fv-az558.
> 2020-05-19T21:05:55.5997224Z Starting taskexecutor daemon on host fv-az558.
> 2020-05-19T21:05:55.6223837Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-05-19T21:05:57.0552482Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-05-19T21:05:57.9446865Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-05-19T21:05:59.0098434Z Waiting for Dispatcher REST endpoint to come
> up...
> 2020-05-19T21:06:00.0569710Z Dispatcher REST endpoint is up.
> 2020-05-19T21:06:07.7099937Z Job (a92a74de8446a80403798bb4806b73f3) is
> running.
> 2020-05-19T21:06:07.7855906Z Waiting for job to process up to 200 records,
> current progress: 114 records ...
> 2020-05-19T21:06:55.5755111Z
> 2020-05-19T21:06:55.5756550Z
>
> 2020-05-19T21:06:55.5757225Z The program finished with the following
> exception:
> 2020-05-19T21:06:55.5757566Z
> 2020-05-19T21:06:55.5765453Z org.apache.flink.util.FlinkException: Could not
> stop with a savepoint job "a92a74de8446a80403798bb4806b73f3".
> 2020-05-19T21:06:55.5766873Z at
> org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:485)
> 2020-05-19T21:06:55.5767980Z at
> org.apache.flink.client.cli.CliFrontend.runClusterAction(CliFrontend.java:854)
> 2020-05-19T21:06:55.5769014Z at
> org.apache.flink.client.cli.CliFrontend.stop(CliFrontend.java:477)
> 2020-05-19T21:06:55.5770052Z at
> org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:921)
> 2020-05-19T21:06:55.5771107Z at
> org.apache.flink.client.cli.CliFrontend.lambda$main$10(CliFrontend.java:982)
> 2020-05-19T21:06:55.5772223Z at
> org.apache.flink.runtime.security.contexts.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
> 2020-05-19T21:06:55.5773325Z at
> org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:982)
> 2020-05-19T21:06:55.5774871Z Caused by:
> java.util.concurrent.ExecutionException:
> java.util.concurrent.CompletionException:
> java.util.concurrent.CompletionException:
> org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint
> Coordinator is suspending.
> 2020-05-19T21:06:55.5777183Z at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
> 2020-05-19T21:06:55.5778884Z at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1928)
> 2020-05-19T21:06:55.5779920Z at
> org.apache.flink.client.cli.CliFrontend.lambda$stop$5(CliFrontend.java:483)
> 2020-05-19T21:06:55.5781175Z ... 6 more
> 2020-05-19T21:06:55.5782391Z Caused by:
> java.util.concurrent.CompletionException:
> java.util.concurrent.CompletionException:
> org.apache.flink.runtime.checkpoint.CheckpointException: Checkpoint
> Coordinator is suspending.
> 2020-05-19T21:06:55.5783885Z at
> org.apache.flink.runtime.scheduler.SchedulerBase.lambda$stopWithSavepoint$9(SchedulerBase.java:890)
> 2020-05-19T21:06:55.5784992Z at
> java.util.concurrent.CompletableFuture.uniHandle(CompletableFuture.java:836)
> 2020-05-19T21:06:55.5786492Z at
> java.util.concurrent.CompletableFuture$UniHandle.tryFire(CompletableFuture.java:811)
> 2020-05-19T21:06:55.5787601Z at
> java.util.concurrent.CompletableFuture$Completion.run(CompletableFuture.java:456)
>
[jira] [Closed] (FLINK-17918) LIMIT queries are failed when adding sleeping time of async checkpoint
[
https://issues.apache.org/jira/browse/FLINK-17918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu closed FLINK-17918.
---
Resolution: Fixed
[hotfix][table-planner-blink] Store last watermark in state to emit on recovery
for EventTimeProcessOperator
- master (1.12.0): 550d4e168bdf068b2c4eadd51878abaa98f0bb3c
- 1.11.0: 7c72335a3dbf2a8de7469d2bef8074d1e33284c9
[FLINK-17918][table-blink] Fix AppendOnlyTopNFunction shouldn't mutate list
value of MapState
- master (1.12.0): 5ab403dafa3bc5ee8654173fb543969dfdd746f8
- 1.11.0: a290c8a3af4083fb32febfc7f955a9bdce08e4b8
> LIMIT queries are failed when adding sleeping time of async checkpoint
> --
>
> Key: FLINK-17918
> URL: https://issues.apache.org/jira/browse/FLINK-17918
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Table SQL / Runtime
>Affects Versions: 1.11.0
>Reporter: Piotr Nowojski
>Assignee: Jark Wu
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> When we change the timing of operations (sleep after emit first record and
> sleep for async operation of checkpoint) with this
> [commit|https://github.com/apache/flink/commit/c05a0d865989c9959047cebcf2cd68b3838cc699],
> the test
> {{org.apache.flink.table.planner.runtime.stream.sql.AggregateITCase#testDifferentTypesSumWithRetract}}
> in flink-table-planner-blink is failed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12410: [FLINK-13782][table-api] Implement type strategies for IF ELSE expression
flinkbot edited a comment on pull request #12410: URL: https://github.com/apache/flink/pull/12410#issuecomment-635946149 ## CI report: * ce34ce1e7b16cf8c6bf359ffadbf5d3e96613947 UNKNOWN * 5e2c5e2f0d9babcc26b6223449b3ab6f8ac97c3e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2692) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12436: [FLINK-17847][table sql / planner] ArrayIndexOutOfBoundsException happens in StreamExecCalc operator
flinkbot edited a comment on pull request #12436: URL: https://github.com/apache/flink/pull/12436#issuecomment-637399261 ## CI report: * d553feca300e2668e3e5c16c1c524d73acde141c Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2627) * 4d1f156fd758a44f4c394a478b6f37bc2de61c86 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2711) * 63831f0ba3dc2a6ec31acc7fb1cfd1b51288945b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2721) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12435: [FLINK-18059] [sql-client] Fix create/drop catalog statement can not be executed in sql client
flinkbot edited a comment on pull request #12435: URL: https://github.com/apache/flink/pull/12435#issuecomment-637372360 ## CI report: * b5ef005c90fb95a63c97d06c488adf5e4abe2972 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2626) * 223701abe70c0815eb1f35af7383727385ce2e90 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2720) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12455: [FLINK-17935] Move setting yarn.log-config-file to YarnClusterClientFactory
flinkbot edited a comment on pull request #12455: URL: https://github.com/apache/flink/pull/12455#issuecomment-638007072 ## CI report: * 278da87279e640156951deef07ee9e4e24cab361 UNKNOWN * 76cf3b92819b6a77bf9c50b6d7674f5df3f4c8f7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2712) * ca75aac93b4b9716e3d1df62f878b8bd7a7c9aab Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2722) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-18123) KubernetesClusterDescriptorTest.testDeployApplicationCluster:135->checkUpdatedConfigAndResourceSetting:194 expected: but was:
Robert Metzger created FLINK-18123:
--
Summary:
KubernetesClusterDescriptorTest.testDeployApplicationCluster:135->checkUpdatedConfigAndResourceSetting:194
expected: but was:
Key: FLINK-18123
URL: https://issues.apache.org/jira/browse/FLINK-18123
Project: Flink
Issue Type: Bug
Components: Deployment / Kubernetes
Affects Versions: 1.12.0
Reporter: Robert Metzger
CI:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2696=logs=fc5181b0-e452-5c8f-68de-1097947f6483=62110053-334f-5295-a0ab-80dd7e2babbf
{code}
2020-06-04T10:17:00.2971211Z [ERROR] Failures:
2020-06-04T10:17:00.2973074Z [ERROR]
KubernetesClusterDescriptorTest.testDeployApplicationCluster:135->checkUpdatedConfigAndResourceSetting:194
expected: but was:
2020-06-04T10:17:00.2973889Z [ERROR] Errors:
2020-06-04T10:17:00.2974940Z [ERROR]
KubernetesClusterDescriptorTest.testDeployHighAvailabilitySessionCluster:86->deploySessionCluster:181
» ClusterDeployment
2020-06-04T10:17:00.2976255Z [ERROR]
KubernetesClusterDescriptorTest.testDeploySessionCluster:77->deploySessionCluster:181
» ClusterDeployment
2020-06-04T10:17:00.2978026Z [ERROR]
KubernetesClusterDescriptorTest.testKillCluster:112->deploySessionCluster:181
» ClusterDeployment
2020-06-04T10:17:00.2980261Z [ERROR]
KubernetesResourceManagerTest.testCreateTaskManagerPodFailedAndRetry:306 »
Execution
2020-06-04T10:17:00.2981587Z [ERROR]
KubernetesResourceManagerTest.testDuplicatedPodAdded:400 » Execution
java.lang...
2020-06-04T10:17:00.2982675Z [ERROR]
KubernetesResourceManagerTest.testGetWorkerNodesFromPreviousAttempts:277 »
Execution
2020-06-04T10:17:00.2983856Z [ERROR]
KubernetesResourceManagerTest.testPodAddedBeforeCreateTaskManagerPodFutureComplete:485
» Execution
2020-06-04T10:17:00.2984993Z [ERROR]
KubernetesResourceManagerTest.testPodTerminatedBeforeAdded:425 » Execution
jav...
2020-06-04T10:17:00.2986083Z [ERROR]
KubernetesResourceManagerTest.testPreviousAttemptPodAdded:368 » Execution
java...
2020-06-04T10:17:00.2987522Z [ERROR]
KubernetesResourceManagerTest.testStartAndRecoverVariousResourceSpec:328 »
NullPointer
2020-06-04T10:17:00.2988812Z [ERROR]
KubernetesResourceManagerTest.testStartAndStopWorker:188 » Execution
java.lang...
2020-06-04T10:17:00.2989909Z [ERROR]
KubernetesResourceManagerTest.testTaskManagerPodTerminated:236 » Execution
jav...
2020-06-04T10:17:00.2990987Z [ERROR]
Fabric8FlinkKubeClientTest>KubernetesTestBase.setup:101->onSetup:102 »
NullPointer
2020-06-04T10:17:00.2992098Z [ERROR]
Fabric8FlinkKubeClientTest>KubernetesTestBase.setup:101->onSetup:102 »
NullPointer
2020-06-04T10:17:00.2993175Z [ERROR]
Fabric8FlinkKubeClientTest>KubernetesTestBase.setup:101->onSetup:102 »
NullPointer
2020-06-04T10:17:00.2994271Z [ERROR]
Fabric8FlinkKubeClientTest>KubernetesTestBase.setup:101->onSetup:102 »
NullPointer
2020-06-04T10:17:00.2995381Z [ERROR]
Fabric8FlinkKubeClientTest>KubernetesTestBase.setup:101->onSetup:102 »
NullPointer
2020-06-04T10:17:00.2996474Z [ERROR]
Fabric8FlinkKubeClientTest>KubernetesTestBase.setup:101->onSetup:102 »
NullPointer
2020-06-04T10:17:00.2997869Z [ERROR]
Fabric8FlinkKubeClientTest>KubernetesTestBase.setup:101->onSetup:102 »
NullPointer
2020-06-04T10:17:00.2999050Z [ERROR]
Fabric8FlinkKubeClientTest>KubernetesTestBase.setup:101->onSetup:102 »
NullPointer
2020-06-04T10:17:00.3000190Z [ERROR]
Fabric8FlinkKubeClientTest>KubernetesTestBase.setup:101->onSetup:102 »
NullPointer
2020-06-04T10:17:00.3001184Z [ERROR]
FlinkConfMountDecoratorTest.testConfigMap:84 » NullPointer
2020-06-04T10:17:00.3002153Z [ERROR]
FlinkConfMountDecoratorTest.testDecoratedFlinkContainer:214 » NullPointer
2020-06-04T10:17:00.3003179Z [ERROR]
FlinkConfMountDecoratorTest.testDecoratedFlinkPodWithLog4j:133 » NullPointer
2020-06-04T10:17:00.3004271Z [ERROR]
FlinkConfMountDecoratorTest.testDecoratedFlinkPodWithLog4jAndLogback:186 »
NullPointer
2020-06-04T10:17:00.3005594Z [ERROR]
FlinkConfMountDecoratorTest.testDecoratedFlinkPodWithLogback:159 » NullPointer
2020-06-04T10:17:00.3006692Z [ERROR]
FlinkConfMountDecoratorTest.testDecoratedFlinkPodWithoutLog4jAndLogback:104 »
NullPointer
2020-06-04T10:17:00.3007991Z [ERROR]
FlinkConfMountDecoratorTest.testWhetherPodOrContainerIsDecorated:74 »
NullPointer
2020-06-04T10:17:00.3009236Z [ERROR]
JavaCmdJobManagerDecoratorTest.testContainerStartCommandTemplate1:200 »
NullPointer
2020-06-04T10:17:00.3010379Z [ERROR]
JavaCmdJobManagerDecoratorTest.testContainerStartCommandTemplate2:228 »
NullPointer
2020-06-04T10:17:00.3011431Z [ERROR]
JavaCmdJobManagerDecoratorTest.testStartCommandWithLog4j:114 » NullPointer
2020-06-04T10:17:00.3012454Z [ERROR]
JavaCmdJobManagerDecoratorTest.testStartCommandWithLog4jAndLogback:143 »
NullPointer
[GitHub] [flink] pnowojski commented on a change in pull request #12460: [FLINK-18063][checkpointing] Fix the race condition for aborting current checkpoint in CheckpointBarrierUnaligner
pnowojski commented on a change in pull request #12460:
URL: https://github.com/apache/flink/pull/12460#discussion_r435188801
##
File path:
flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/io/CheckpointBarrierUnaligner.java
##
@@ -154,46 +154,51 @@ public void processBarrier(CheckpointBarrier
receivedBarrier, int channelIndex)
@Override
public void processCancellationBarrier(CancelCheckpointMarker
cancelBarrier) throws Exception {
long cancelledId = cancelBarrier.getCheckpointId();
- if (LOG.isDebugEnabled()) {
- LOG.debug("{}: Checkpoint {} canceled, aborting
alignment.", taskName, cancelledId);
- }
-
- if (currentConsumedCheckpointId >= cancelledId &&
!isCheckpointPending()) {
- return;
- }
+ // tag whether we should abort checkpoint from task thread view
+ boolean shouldAbort1 = false;
- if (isCheckpointPending()) {
+ if (cancelledId > currentConsumedCheckpointId) {
+ currentConsumedCheckpointId = cancelledId;
+ shouldAbort1 = true;
+ } else if (cancelledId == currentConsumedCheckpointId &&
isCheckpointPending()) {
LOG.warn("{}: Received cancellation barrier for
checkpoint {} before completing current checkpoint {}. " +
"Skipping current checkpoint.",
taskName,
cancelledId,
currentConsumedCheckpointId);
resetConsumedBarriers();
+ shouldAbort1 = true;
+ }
+
+ // tag whether we should abort checkpoint from
threadSafeUnaligner view
+ boolean shouldAbort2 =
threadSafeUnaligner.setCancelledCheckpointId(cancelledId);
Review comment:
> If shouldAbort2 true and shouldAbort1 false, that means the
notifyBarrierReceived triggered by netty thread happen earlier.
But this `notifyBarrierReceived` from netty thread, would enqueue mailbox
action.
If the mailbox action hasn't yet been executed, we could mark this
checkpoint as aborted here, in this method (task thread), and prevent mail from
executing (task thread).
If the mailbox action has already executed (task thread), it could have left
the `currentConsumedCheckpointId` field already up to date, so this method
would already know whether to abort or not from `shouldAbort1`.
Or is it about aborting the checkpoint after `notifyBarrierReceived` from
netty thread, but before enqueued mailbox action was executed?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] rkhachatryan commented on pull request #12457: [FLINK-18050][task][checkpointing] Fix double buffer recycling
rkhachatryan commented on pull request #12457: URL: https://github.com/apache/flink/pull/12457#issuecomment-638795603 CI failure unrelated (`Resuming Savepoint` end-to-end test failed with timeout - FLINK-17824). Succeeded on my private branch: https://dev.azure.com/khachatryanroman/flink/_build/results?buildId=86=results. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-18125) Skip CI execution on documentation pull requests
Robert Metzger created FLINK-18125: -- Summary: Skip CI execution on documentation pull requests Key: FLINK-18125 URL: https://issues.apache.org/jira/browse/FLINK-18125 Project: Flink Issue Type: Improvement Components: Build System / Azure Pipelines Reporter: Robert Metzger In order to save some resources, we can skip the CI execution on documentation-only changes (whole changeset). -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17961) Create an Elasticsearch source
[ https://issues.apache.org/jira/browse/FLINK-17961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125806#comment-17125806 ] Etienne Chauchot commented on FLINK-17961: -- [~chesnay] ES source can definitely mask the overall complexity to the user. As an example in Apache Beam ([available here|https://github.com/apache/beam/blob/e1963c11f9a853564d62f83993dec08ed8a9321f/sdks/java/io/elasticsearch/src/main/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIO.java#L156]) what we do is we use sliced scroll to split the input collection for parallel reading and apply it to the user ES query or to a default _select * from index_ when there is no provided query. Thus, the user API remains simple with _ESIO.read().from(index).withQuery(query)._ My worries here are more related to streaming and failover capabilities raised by Aljoscha. Even though ES is a main source (not an enrichment one IMO) it does not meet some Flink expectancies (cf comments above). So the question is reduced to: is it worth investing some time to make an ES source still? Regarding the thread on an ES table source, I'll read it and comment if I have anything useful to say. > Create an Elasticsearch source > -- > > Key: FLINK-17961 > URL: https://issues.apache.org/jira/browse/FLINK-17961 > Project: Flink > Issue Type: New Feature > Components: Connectors / ElasticSearch >Reporter: Etienne Chauchot >Priority: Minor > > There is only an Elasticsearch sink available. There are opensource github > repos such as [this > one|[https://github.com/mnubo/flink-elasticsearch-source-connector]]. Also > the apache bahir project does not provide an Elasticsearch source connector > for flink either. IMHO I think the project would benefit from having an > bundled source connector for ES alongside with the available sink connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #11541: [FLINK-15416][network] add task manager netty client retry mechenism
flinkbot edited a comment on pull request #11541: URL: https://github.com/apache/flink/pull/11541#issuecomment-604812212 ## CI report: * b5ef19285000d692d6b1b8b0aa7a37ce35c216f5 UNKNOWN * baf2c55f7bb0b9df7bdef94083c947693b75eae2 UNKNOWN * e5419f41a18eaae05814837d679f58480a2cb94c UNKNOWN * a5dc1df6976661f4dc56c5e739b54dd90efc660d Travis: [FAILURE](https://travis-ci.com/github/flink-ci/flink/builds/169527021) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2679) * 34a70a74c479f177784b5ba4983b50072c4f7e5f Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2702) * f64492a673ed412c925b42152c4ce469d4ccbb0c Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2715) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12352: [FLINK-17717][sql-parser] Throws for DDL create temporary system func…
flinkbot edited a comment on pull request #12352: URL: https://github.com/apache/flink/pull/12352#issuecomment-634517986 ## CI report: * 6900843764e0fe637babcb429194ef620be73b50 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2278) * 7dc817125d0684987814d45192eab87a41ebc5b5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2716) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12444: [FLINK-18067]Change default value of yarnMinAllocationMB from zero to DEFAULT_RM_S…
flinkbot edited a comment on pull request #12444: URL: https://github.com/apache/flink/pull/12444#issuecomment-637596755 ## CI report: * 16ec8b694dc07564b09b60040d20b0da9c0e80c4 UNKNOWN * b2e066ffdb1333b0b14aacc5955e4d0841a57096 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2690) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12456: [FLINK-17113][sql-cli] Refactor view support in SQL Client
flinkbot edited a comment on pull request #12456: URL: https://github.com/apache/flink/pull/12456#issuecomment-638032908 ## CI report: * 12e3932ed8d74a59c0d4c28441b336284a2dffeb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2713) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12455: [FLINK-17935] Move setting yarn.log-config-file to YarnClusterClientFactory
flinkbot edited a comment on pull request #12455: URL: https://github.com/apache/flink/pull/12455#issuecomment-638007072 ## CI report: * 278da87279e640156951deef07ee9e4e24cab361 UNKNOWN * 76cf3b92819b6a77bf9c50b6d7674f5df3f4c8f7 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2712) * ca75aac93b4b9716e3d1df62f878b8bd7a7c9aab UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12475: [FLINK-18076][table sql / client] Sql client uses wrong class loader when parsing queries
flinkbot edited a comment on pull request #12475: URL: https://github.com/apache/flink/pull/12475#issuecomment-638593958 ## CI report: * 41240cd524255ffc7cf63d2bc2ee9ff38f3abcbd UNKNOWN * fda75fa6db31dc6cc66f5b4fa7f86db3d4e26356 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2681) * 49c19b93edb74ef702c97a4742f4936145913f35 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=2717) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18121) Support creating Docker image from local Flink distribution
[ https://issues.apache.org/jira/browse/FLINK-18121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125815#comment-17125815 ] Yang Wang commented on FLINK-18121: --- If this ticket is done, then FLINK-17857 could also benefit from this and will not have to use host machine address to fetch the flink.tar.gz. > Support creating Docker image from local Flink distribution > --- > > Key: FLINK-18121 > URL: https://issues.apache.org/jira/browse/FLINK-18121 > Project: Flink > Issue Type: Improvement > Components: Dockerfiles >Affects Versions: docker-1.11.0.0 >Reporter: Till Rohrmann >Priority: Major > > Currently, > https://github.com/apache/flink-docker/blob/dev-master/Dockerfile-debian.template > only supports to create a Docker image from a Flink distribution which is > hosted on a web server. I think it would be helpful if we could also create a > Docker image from a Flink distribution which is stored on one's local file > system. That way, one would not have to upload the file or start a web server > for serving it. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zhijiangW commented on pull request #12460: [FLINK-18063][checkpointing] Fix the race condition for aborting current checkpoint in CheckpointBarrierUnaligner
zhijiangW commented on pull request #12460: URL: https://github.com/apache/flink/pull/12460#issuecomment-638773131 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-17961) Create an Elasticsearch source
[ https://issues.apache.org/jira/browse/FLINK-17961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Etienne Chauchot closed FLINK-17961. Release Note: Closing this because it duplicates: https://issues.apache.org/jira/browse/FLINK-16713 and the above ticket refers a design doc. Resolution: Duplicate > Create an Elasticsearch source > -- > > Key: FLINK-17961 > URL: https://issues.apache.org/jira/browse/FLINK-17961 > Project: Flink > Issue Type: New Feature > Components: Connectors / ElasticSearch >Reporter: Etienne Chauchot >Priority: Minor > > There is only an Elasticsearch sink available. There are opensource github > repos such as [this > one|[https://github.com/mnubo/flink-elasticsearch-source-connector]]. Also > the apache bahir project does not provide an Elasticsearch source connector > for flink either. IMHO I think the project would benefit from having an > bundled source connector for ES alongside with the available sink connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Reopened] (FLINK-17961) Create an Elasticsearch source
[ https://issues.apache.org/jira/browse/FLINK-17961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Etienne Chauchot reopened FLINK-17961: -- Move comment to the proper location (not release notes) > Create an Elasticsearch source > -- > > Key: FLINK-17961 > URL: https://issues.apache.org/jira/browse/FLINK-17961 > Project: Flink > Issue Type: New Feature > Components: Connectors / ElasticSearch >Reporter: Etienne Chauchot >Priority: Minor > > There is only an Elasticsearch sink available. There are opensource github > repos such as [this > one|[https://github.com/mnubo/flink-elasticsearch-source-connector]]. Also > the apache bahir project does not provide an Elasticsearch source connector > for flink either. IMHO I think the project would benefit from having an > bundled source connector for ES alongside with the available sink connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-17961) Create an Elasticsearch source
[ https://issues.apache.org/jira/browse/FLINK-17961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Etienne Chauchot closed FLINK-17961. Resolution: Duplicate Closing this because it duplicates: https://issues.apache.org/jira/browse/FLINK-16713 and the above ticket refers a design doc. > Create an Elasticsearch source > -- > > Key: FLINK-17961 > URL: https://issues.apache.org/jira/browse/FLINK-17961 > Project: Flink > Issue Type: New Feature > Components: Connectors / ElasticSearch >Reporter: Etienne Chauchot >Priority: Minor > > There is only an Elasticsearch sink available. There are opensource github > repos such as [this > one|[https://github.com/mnubo/flink-elasticsearch-source-connector]]. Also > the apache bahir project does not provide an Elasticsearch source connector > for flink either. IMHO I think the project would benefit from having an > bundled source connector for ES alongside with the available sink connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17961) Create an Elasticsearch source
[ https://issues.apache.org/jira/browse/FLINK-17961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Etienne Chauchot updated FLINK-17961: - Release Note: (was: Closing this because it duplicates: https://issues.apache.org/jira/browse/FLINK-16713 and the above ticket refers a design doc.) > Create an Elasticsearch source > -- > > Key: FLINK-17961 > URL: https://issues.apache.org/jira/browse/FLINK-17961 > Project: Flink > Issue Type: New Feature > Components: Connectors / ElasticSearch >Reporter: Etienne Chauchot >Priority: Minor > > There is only an Elasticsearch sink available. There are opensource github > repos such as [this > one|[https://github.com/mnubo/flink-elasticsearch-source-connector]]. Also > the apache bahir project does not provide an Elasticsearch source connector > for flink either. IMHO I think the project would benefit from having an > bundled source connector for ES alongside with the available sink connector. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-18020) SQLClientKafkaITCase.testKafka failed on Travis
[
https://issues.apache.org/jira/browse/FLINK-18020?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aljoscha Krettek reassigned FLINK-18020:
Assignee: Aljoscha Krettek
> SQLClientKafkaITCase.testKafka failed on Travis
> ---
>
> Key: FLINK-18020
> URL: https://issues.apache.org/jira/browse/FLINK-18020
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Table SQL / Client, Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Dawid Wysakowicz
>Assignee: Aljoscha Krettek
>Priority: Blocker
> Fix For: 1.11.0
>
>
> {code}
> [ERROR] Tests run: 3, Failures: 1, Errors: 0, Skipped: 0, Time elapsed:
> 151.386 s <<< FAILURE! - in
> org.apache.flink.tests.util.kafka.SQLClientKafkaITCase
> [ERROR] testKafka[1: kafka-version:0.11
> kafka-sql-version:.*kafka-0.11.jar](org.apache.flink.tests.util.kafka.SQLClientKafkaITCase)
> Time elapsed: 49.811 s <<< FAILURE!
> java.lang.AssertionError:
> Expected: ["2018-03-12 08:00:00.000,Alice,This was a warning.,2,Success
> constant folding.", "2018-03-12 09:00:00.000,Bob,This was another
> warning.,1,Success constant folding.", "2018-03-12 09:00:00.000,Steve,This
> was another info.,2,Success constant folding.", "2018-03-12
> 09:00:00.000,Alice,This was a info.,1,Success constant folding."] in any order
> but: Not matched: "2018-03-12 08:00:00.000,Alice,This was a
> warning.,6,Success constant folding."
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.junit.Assert.assertThat(Assert.java:956)
> at org.junit.Assert.assertThat(Assert.java:923)
> at
> org.apache.flink.tests.util.kafka.SQLClientKafkaITCase.checkCsvResultFile(SQLClientKafkaITCase.java:241)
> at
> org.apache.flink.tests.util.kafka.SQLClientKafkaITCase.testKafka(SQLClientKafkaITCase.java:172)
> {code}
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2323=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18071) CoordinatorEventsExactlyOnceITCase.checkListContainsSequence fails on CI
[
https://issues.apache.org/jira/browse/FLINK-18071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125842#comment-17125842
]
Robert Metzger commented on FLINK-18071:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2697=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=4ed44b66-cdd6-5dcf-5f6a-88b07dda665d
> CoordinatorEventsExactlyOnceITCase.checkListContainsSequence fails on CI
>
>
> Key: FLINK-18071
> URL: https://issues.apache.org/jira/browse/FLINK-18071
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination, Tests
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> CI:
> https://dev.azure.com/georgeryan1322/Flink/_build/results?buildId=330=logs=6e58d712-c5cc-52fb-0895-6ff7bd56c46b=f30a8e80-b2cf-535c-9952-7f521a4ae374
> {code}
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 8.795
> s <<< FAILURE! - in
> org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase
> [ERROR]
> test(org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase)
> Time elapsed: 4.647 s <<< FAILURE!
> java.lang.AssertionError: List did not contain expected sequence of 200
> elements, but was: [152, 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14,
> 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33,
> 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52,
> 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71,
> 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90,
> 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107,
> 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122,
> 123, 124, 125, 126, 127, 128, 129, 130, 131, 132, 133, 134, 135, 136, 137,
> 138, 139, 140, 141, 142, 143, 144, 145, 146, 147, 148, 149, 150, 151, 152,
> 153, 154, 155, 156, 157, 158, 159, 160, 161, 162, 163, 164, 165, 166, 167,
> 168, 169, 170, 171, 172, 173, 174, 175, 176, 177, 178, 179, 180, 181, 182,
> 183, 184, 185, 186, 187, 188, 189, 190, 191, 192, 193, 194, 195, 196, 197,
> 198, 199]
> at org.junit.Assert.fail(Assert.java:88)
> at
> org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase.failList(CoordinatorEventsExactlyOnceITCase.java:160)
> at
> org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase.checkListContainsSequence(CoordinatorEventsExactlyOnceITCase.java:148)
> at
> org.apache.flink.runtime.operators.coordination.CoordinatorEventsExactlyOnceITCase.test(CoordinatorEventsExactlyOnceITCase.java:143)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18122) Kubernetes test fails with "error: timed out waiting for the condition on jobs/flink-job-cluster"
[
https://issues.apache.org/jira/browse/FLINK-18122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125850#comment-17125850
]
Chesnay Schepler commented on FLINK-18122:
--
The docker image build failed, and it likely re-used the image from a previous
run:
{code}
Step 4/14 : RUN set -ex; wget -nv -O /usr/local/bin/gosu
"https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg
--print-architecture)"; wget -nv -O /usr/local/bin/gosu.asc "
---> Running in eba9fe332456
+ dpkg --print-architecture
+ wget -nv -O /usr/local/bin/gosu
https://github.com/tianon/gosu/releases/download/1.11/gosu-amd64
2020-06-04 09:25:20
URL:https://github-production-release-asset-2e65be.s3.amazonaws.com/19708981/82e9dd00-d091-11e8-8734-a1caffcee352?X-Amz-Algorithm=AWS4-HMAC-SHA256=AKIAIWNJYAX4CSVEH53A%2F20200604%2Fus-east-1%2Fs3%2Faws4_request=20200604T092519Z=300=dd2292d73c781cad49cafc4064206ce385dca050762004be10d4ae5e1f5eaa9e=host_id=0_id=19708981=attachment%3B%20filename%3Dgosu-amd64=application%2Foctet-stream
[2294944/2294944] -> "/usr/local/bin/gosu" [1]
+ dpkg --print-architecture
+ wget -nv -O /usr/local/bin/gosu.asc
https://github.com/tianon/gosu/releases/download/1.11/gosu-amd64.asc
GnuTLS: Error in the pull function.
Unable to establish SSL connection.
The command '/bin/sh -c set -ex; wget -nv -O /usr/local/bin/gosu
"https://github.com/tianon/gosu/releases/download/$GOSU_VERSION/gosu-$(dpkg
--print-architecture)"; wget -nv -O /usr/local/bin/gosu.asc "
~/work/1/s
{code}
> Kubernetes test fails with "error: timed out waiting for the condition on
> jobs/flink-job-cluster"
> -
>
> Key: FLINK-18122
> URL: https://issues.apache.org/jira/browse/FLINK-18122
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2697=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> {code}
> 2020-06-04T09:25:40.7205843Z service/flink-job-cluster created
> 2020-06-04T09:25:40.9661515Z job.batch/flink-job-cluster created
> 2020-06-04T09:25:41.2189123Z deployment.apps/flink-task-manager created
> 2020-06-04T10:32:32.6402983Z error: timed out waiting for the condition on
> jobs/flink-job-cluster
> 2020-06-04T10:32:33.8057757Z error: unable to upgrade connection: container
> not found ("flink-task-manager")
> 2020-06-04T10:32:33.8111302Z sort: cannot read:
> '/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-56335570120/out/kubernetes_wc_out*':
> No such file or directory
> 2020-06-04T10:32:33.8124455Z FAIL WordCount: Output hash mismatch. Got
> d41d8cd98f00b204e9800998ecf8427e, expected e682ec6622b5e83f2eb614617d5ab2cf.
> 2020-06-04T10:32:33.8125379Z head hexdump of actual:
> 2020-06-04T10:32:33.8136133Z head: cannot open
> '/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-56335570120/out/kubernetes_wc_out*'
> for reading: No such file or directory
> 2020-06-04T10:32:33.8344715Z Debugging failed Kubernetes test:
> 2020-06-04T10:32:33.8345469Z Currently existing Kubernetes resources
> 2020-06-04T10:32:36.4977853Z I0604 10:32:36.497383 13191 request.go:621]
> Throttling request took 1.198606989s, request:
> GET:https://10.1.0.4:8443/apis/rbac.authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:32:46.6975735Z I0604 10:32:46.697234 13191 request.go:621]
> Throttling request took 4.398107353s, request:
> GET:https://10.1.0.4:8443/apis/authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:32:57.4978637Z I0604 10:32:57.497209 13191 request.go:621]
> Throttling request took 1.198449167s, request:
> GET:https://10.1.0.4:8443/apis/apps/v1?timeout=32s
> 2020-06-04T10:33:07.4980104Z I0604 10:33:07.497320 13191 request.go:621]
> Throttling request took 4.198274438s, request:
> GET:https://10.1.0.4:8443/apis/apiextensions.k8s.io/v1?timeout=32s
> 2020-06-04T10:33:18.4976060Z I0604 10:33:18.497258 13191 request.go:621]
> Throttling request took 1.19871495s, request:
> GET:https://10.1.0.4:8443/apis/apps/v1?timeout=32s
> 2020-06-04T10:33:28.4979129Z I0604 10:33:28.497276 13191 request.go:621]
> Throttling request took 4.198369672s, request:
> GET:https://10.1.0.4:8443/apis/rbac.authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:33:30.9182069Z NAME READY
> STATUS RESTARTS AGE
> 2020-06-04T10:33:30.9184099Z pod/flink-job-cluster-dtb67 0/1
> ErrImageNeverPull 0 67m
> 2020-06-04T10:33:30.9184869Z pod/flink-task-manager-74ccc9bd9-psqwm 0/1
> ErrImageNeverPull 0 67m
[jira] [Commented] (FLINK-18122) Kubernetes test fails with "error: timed out waiting for the condition on jobs/flink-job-cluster"
[
https://issues.apache.org/jira/browse/FLINK-18122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125860#comment-17125860
]
Chesnay Schepler commented on FLINK-18122:
--
Yes; it should be enough to check for errors in
{{flink-end-to-end-tests/test-scripts/common_docker.sh:build_image}}
> Kubernetes test fails with "error: timed out waiting for the condition on
> jobs/flink-job-cluster"
> -
>
> Key: FLINK-18122
> URL: https://issues.apache.org/jira/browse/FLINK-18122
> Project: Flink
> Issue Type: Bug
> Components: Deployment / Kubernetes, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=2697=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=1e2bbe5b-4657-50be-1f07-d84bfce5b1f5
> {code}
> 2020-06-04T09:25:40.7205843Z service/flink-job-cluster created
> 2020-06-04T09:25:40.9661515Z job.batch/flink-job-cluster created
> 2020-06-04T09:25:41.2189123Z deployment.apps/flink-task-manager created
> 2020-06-04T10:32:32.6402983Z error: timed out waiting for the condition on
> jobs/flink-job-cluster
> 2020-06-04T10:32:33.8057757Z error: unable to upgrade connection: container
> not found ("flink-task-manager")
> 2020-06-04T10:32:33.8111302Z sort: cannot read:
> '/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-56335570120/out/kubernetes_wc_out*':
> No such file or directory
> 2020-06-04T10:32:33.8124455Z FAIL WordCount: Output hash mismatch. Got
> d41d8cd98f00b204e9800998ecf8427e, expected e682ec6622b5e83f2eb614617d5ab2cf.
> 2020-06-04T10:32:33.8125379Z head hexdump of actual:
> 2020-06-04T10:32:33.8136133Z head: cannot open
> '/home/vsts/work/1/s/flink-end-to-end-tests/test-scripts/temp-test-directory-56335570120/out/kubernetes_wc_out*'
> for reading: No such file or directory
> 2020-06-04T10:32:33.8344715Z Debugging failed Kubernetes test:
> 2020-06-04T10:32:33.8345469Z Currently existing Kubernetes resources
> 2020-06-04T10:32:36.4977853Z I0604 10:32:36.497383 13191 request.go:621]
> Throttling request took 1.198606989s, request:
> GET:https://10.1.0.4:8443/apis/rbac.authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:32:46.6975735Z I0604 10:32:46.697234 13191 request.go:621]
> Throttling request took 4.398107353s, request:
> GET:https://10.1.0.4:8443/apis/authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:32:57.4978637Z I0604 10:32:57.497209 13191 request.go:621]
> Throttling request took 1.198449167s, request:
> GET:https://10.1.0.4:8443/apis/apps/v1?timeout=32s
> 2020-06-04T10:33:07.4980104Z I0604 10:33:07.497320 13191 request.go:621]
> Throttling request took 4.198274438s, request:
> GET:https://10.1.0.4:8443/apis/apiextensions.k8s.io/v1?timeout=32s
> 2020-06-04T10:33:18.4976060Z I0604 10:33:18.497258 13191 request.go:621]
> Throttling request took 1.19871495s, request:
> GET:https://10.1.0.4:8443/apis/apps/v1?timeout=32s
> 2020-06-04T10:33:28.4979129Z I0604 10:33:28.497276 13191 request.go:621]
> Throttling request took 4.198369672s, request:
> GET:https://10.1.0.4:8443/apis/rbac.authorization.k8s.io/v1?timeout=32s
> 2020-06-04T10:33:30.9182069Z NAME READY
> STATUS RESTARTS AGE
> 2020-06-04T10:33:30.9184099Z pod/flink-job-cluster-dtb67 0/1
> ErrImageNeverPull 0 67m
> 2020-06-04T10:33:30.9184869Z pod/flink-task-manager-74ccc9bd9-psqwm 0/1
> ErrImageNeverPull 0 67m
> 2020-06-04T10:33:30.9185226Z
> 2020-06-04T10:33:30.9185926Z NAMETYPE
> CLUSTER-IP EXTERNAL-IP PORT(S)
> AGE
> 2020-06-04T10:33:30.9186832Z service/flink-job-cluster NodePort
> 10.111.92.199
> 6123:32501/TCP,6124:31360/TCP,6125:30025/TCP,8081:30081/TCP 67m
> 2020-06-04T10:33:30.9187545Z service/kubernetes ClusterIP
> 10.96.0.1 443/TCP
> 68m
> 2020-06-04T10:33:30.9187976Z
> 2020-06-04T10:33:30.9188472Z NAME READY
> UP-TO-DATE AVAILABLE AGE
> 2020-06-04T10:33:30.9189179Z deployment.apps/flink-task-manager 0/1 1
> 0 67m
> 2020-06-04T10:33:30.9189508Z
> 2020-06-04T10:33:30.9189815Z NAME
> DESIRED CURRENT READY AGE
> 2020-06-04T10:33:30.9190418Z replicaset.apps/flink-task-manager-74ccc9bd9 1
> 1 0 67m
> 2020-06-04T10:33:30.9190662Z
> 2020-06-04T10:33:30.9190891Z NAME COMPLETIONS
> DURATION AGE
> 2020-06-04T10:33:30.9191423Z
[GitHub] [flink] zhuzhurk commented on a change in pull request #12375: [FLINK-17017][runtime] Implements bulk allocation for physical slots
zhuzhurk commented on a change in pull request #12375:
URL: https://github.com/apache/flink/pull/12375#discussion_r435194969
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/jobmaster/slotpool/BulkSlotProviderImpl.java
##
@@ -0,0 +1,271 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.jobmaster.slotpool;
+
+import org.apache.flink.annotation.VisibleForTesting;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.ResourceProfile;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.ComponentMainThreadExecutor;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmaster.SlotInfo;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.util.clock.Clock;
+import org.apache.flink.util.clock.SystemClock;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Optional;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.concurrent.TimeoutException;
+import java.util.stream.Collectors;
+import java.util.stream.Stream;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+
+/**
+ * Default implementation of {@link BulkSlotProvider}.
+ */
+class BulkSlotProviderImpl implements BulkSlotProvider {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(BulkSlotProviderImpl.class);
+
+ private ComponentMainThreadExecutor componentMainThreadExecutor;
+
+ private final SlotSelectionStrategy slotSelectionStrategy;
+
+ private final SlotPool slotPool;
+
+ private final Clock clock;
+
+ private final PhysicalSlotRequestBulkTracker slotRequestBulkTracker;
+
+ BulkSlotProviderImpl(final SlotSelectionStrategy slotSelectionStrategy,
final SlotPool slotPool) {
+ this(slotSelectionStrategy, slotPool,
SystemClock.getInstance());
+ }
+
+ @VisibleForTesting
+ BulkSlotProviderImpl(
+ final SlotSelectionStrategy slotSelectionStrategy,
+ final SlotPool slotPool,
+ final Clock clock) {
+
+ this.slotSelectionStrategy =
checkNotNull(slotSelectionStrategy);
+ this.slotPool = checkNotNull(slotPool);
+ this.clock = checkNotNull(clock);
+
+ this.slotRequestBulkTracker = new
PhysicalSlotRequestBulkTracker(clock);
+
+ this.componentMainThreadExecutor = new
ComponentMainThreadExecutor.DummyComponentMainThreadExecutor(
+ "Scheduler is not initialized with proper main thread
executor. " +
+ "Call to BulkSlotProvider.start(...)
required.");
+ }
+
+ @Override
+ public void start(final ComponentMainThreadExecutor mainThreadExecutor)
{
+ this.componentMainThreadExecutor = mainThreadExecutor;
+ }
+
+ @Override
+ public CompletableFuture>
allocatePhysicalSlots(
+ final Collection
physicalSlotRequests,
+ final Time timeout) {
+
+ componentMainThreadExecutor.assertRunningInMainThread();
+
+ LOG.debug("Received {} slot requests.",
physicalSlotRequests.size());
+
+ final PhysicalSlotRequestBulk slotRequestBulk = new
PhysicalSlotRequestBulk(physicalSlotRequests);
+
+ final List>
resultFutures = new ArrayList<>(physicalSlotRequests.size());
+ for (PhysicalSlotRequest request : physicalSlotRequests) {
+ final CompletableFuture
resultFuture =
+ allocatePhysicalSlot(request,
timeout).thenApply(result -> {
+ slotRequestBulk.markRequestFulfilled(
+ result.getSlotRequestId(),
+
[jira] [Commented] (FLINK-17726) Scheduler should take care of tasks directly canceled by TaskManager
[ https://issues.apache.org/jira/browse/FLINK-17726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125632#comment-17125632 ] Nicholas Jiang commented on FLINK-17726: I have already discuss with [~trohrmann] offline. And considering that I have completed other issues I claimed, I would like to make a design about this in a few these days, and discuss with you [~zhuzh] about the design. Thanks for [~zhuzh] and [~trohrmann] explanation for the solution. > Scheduler should take care of tasks directly canceled by TaskManager > > > Key: FLINK-17726 > URL: https://issues.apache.org/jira/browse/FLINK-17726 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination, Runtime / Task >Affects Versions: 1.11.0, 1.12.0 >Reporter: Zhu Zhu >Assignee: Nicholas Jiang >Priority: Critical > Fix For: 1.12.0, 1.11.1 > > > JobManager will not trigger failure handling when receiving CANCELED task > update. > This is because CANCELED tasks are usually caused by another FAILED task. > These CANCELED tasks will be restarted by the failover process triggered > FAILED task. > However, if a task is directly CANCELED by TaskManager due to its own runtime > issue, the task will not be recovered by JM and thus the job would hang. > This is a potential issue and we should avoid it. > A possible solution is to let JobManager treat tasks transitioning to > CANCELED from all states except from CANCELING as failed tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-17918) LIMIT queries are failed when adding sleeping time of async checkpoint
[
https://issues.apache.org/jira/browse/FLINK-17918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125637#comment-17125637
]
Jark Wu commented on FLINK-17918:
-
FYI, I have updated the title and description of this issue and forked out a
new issue FLINK-18118 for the remaining problems.
> LIMIT queries are failed when adding sleeping time of async checkpoint
> --
>
> Key: FLINK-17918
> URL: https://issues.apache.org/jira/browse/FLINK-17918
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Table SQL / Runtime
>Affects Versions: 1.11.0
>Reporter: Piotr Nowojski
>Assignee: Jark Wu
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> When we change the timing of operations (sleep after emit first record and
> sleep for async operation of checkpoint) with this
> [commit|https://github.com/apache/flink/commit/c05a0d865989c9959047cebcf2cd68b3838cc699],
> the test
> {{org.apache.flink.table.planner.runtime.stream.sql.AggregateITCase#testDifferentTypesSumWithRetract}}
> in flink-table-planner-blink is failed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] rkhachatryan opened a new pull request #12478: [FLINK-17869][task][checkpointing] Fix race condition when caling ChannelStateWriter.abort
rkhachatryan opened a new pull request #12478: URL: https://github.com/apache/flink/pull/12478 __This is a draft PR. It depends on #12457 and should not be merged before__ ## What is the purpose of the change Before FLINK-17218 there were two problems: 1. Exceeding `ChannelStateWriterImpl.maxCheckpoints` 1. Race condition with out of order barriers (see be8fbcf506b8dd38e5425cf772a55f033f0962b0 of this PR) FLINK-17218 (24ff415f1b76392f75dea7c3538558d24fcb7058) fixed the first one but introduced a new race condition when netty thread modifies a ChannelStateWriter map used by the task thread. This PR reverts FLINK-17218 and addresses the issues above. ## Brief change log 1. Revert 24ff415f1b76392f75dea7c3538558d24fcb7058 `"[FLINK-17218][checkpointing] Ensuring that ChannelStateWriter aborts previous checkpoints before a new checkpoint` 1. Abort channel state write if checkpoint is subsumed - using future callback and task thread - fix race condition 1. Increase `ChannelStateWriterImpl.DEFAULT_MAX_CHECKPOINTS` - fix the original problem of `FLINK-17218` 1. Ignore out of order checkpoints in `SubtaskCheckpointCoordinator` - fix a more subtle race condition 1. Unignore `UnalignedCheckpointITCase` ## Verifying this change Unignore `UnalignedCheckpointITCase`. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-17726) Scheduler should take care of tasks directly canceled by TaskManager
[ https://issues.apache.org/jira/browse/FLINK-17726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125653#comment-17125653 ] Zhu Zhu commented on FLINK-17726: - Thanks for the updates. [~nicholasjiang] Looking forward to your design. > Scheduler should take care of tasks directly canceled by TaskManager > > > Key: FLINK-17726 > URL: https://issues.apache.org/jira/browse/FLINK-17726 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination, Runtime / Task >Affects Versions: 1.11.0, 1.12.0 >Reporter: Zhu Zhu >Assignee: Nicholas Jiang >Priority: Critical > Fix For: 1.12.0, 1.11.1 > > > JobManager will not trigger failure handling when receiving CANCELED task > update. > This is because CANCELED tasks are usually caused by another FAILED task. > These CANCELED tasks will be restarted by the failover process triggered > FAILED task. > However, if a task is directly CANCELED by TaskManager due to its own runtime > issue, the task will not be recovered by JM and thus the job would hang. > This is a potential issue and we should avoid it. > A possible solution is to let JobManager treat tasks transitioning to > CANCELED from all states except from CANCELING as failed tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] azagrebin commented on a change in pull request #12256: [FLINK-17018][runtime] Allocates slots in bulks for pipelined region scheduling
azagrebin commented on a change in pull request #12256:
URL: https://github.com/apache/flink/pull/12256#discussion_r435061970
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/OneSlotPerExecutionSlotAllocator.java
##
@@ -0,0 +1,277 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.runtime.scheduler;
+
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.runtime.clusterframework.types.AllocationID;
+import org.apache.flink.runtime.clusterframework.types.SlotProfile;
+import org.apache.flink.runtime.concurrent.FutureUtils;
+import org.apache.flink.runtime.jobmanager.scheduler.Locality;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.SlotOwner;
+import org.apache.flink.runtime.jobmaster.SlotRequestId;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.PhysicalSlotRequest;
+import org.apache.flink.runtime.jobmaster.slotpool.SingleLogicalSlot;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.strategy.ExecutionVertexID;
+import org.apache.flink.runtime.taskmanager.TaskManagerLocation;
+import org.apache.flink.util.FlinkException;
+
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.concurrent.CompletableFuture;
+import java.util.function.Function;
+import java.util.stream.Collectors;
+
+import static org.apache.flink.util.Preconditions.checkNotNull;
+import static org.apache.flink.util.Preconditions.checkState;
+
+/**
+ * This slot allocator will request one physical slot for each single
execution vertex.
+ * The slots will be requested in bulks so that the {@link SlotProvider} can
check
+ * whether this bulk of slot requests can be fulfilled at the same time.
+ * It has several limitations:
+ *
+ * 1. Slot sharing will be ignored.
+ *
+ * 2. Co-location constraints are not allowed.
+ *
+ * 3. Intra-bulk input location preferences will be ignored.
+ */
+public class OneSlotPerExecutionSlotAllocator extends
AbstractExecutionSlotAllocator {
+
+ private static final Logger LOG =
LoggerFactory.getLogger(OneSlotPerExecutionSlotAllocator.class);
+
+ private final SlotProvider slotProvider;
+
+ private final SlotOwner slotOwner;
+
+ private final boolean slotWillBeOccupiedIndefinitely;
+
+ private final Time allocationTimeout;
+
+ public OneSlotPerExecutionSlotAllocator(
+ final SlotProvider slotProvider,
Review comment:
Could we already make `OneSlotPerExecutionSlotAllocator` depending on
`BulkSlotProvider` if `Scheduler` extends `BulkSlotProvider`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-18116) E2E performance test
[ https://issues.apache.org/jira/browse/FLINK-18116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125663#comment-17125663 ] Aljoscha Krettek commented on FLINK-18116: -- Please use a meaningful Issue title, such as "Manually test E2E performance on Flink 1.11", if we need this issue at all. > E2E performance test > > > Key: FLINK-18116 > URL: https://issues.apache.org/jira/browse/FLINK-18116 > Project: Flink > Issue Type: Sub-task > Components: API / Core, API / DataStream, API / State Processor, > Build System, Client / Job Submission >Affects Versions: 1.11.0 >Reporter: Aihua Li >Assignee: Aihua Li >Priority: Blocker > Labels: release-testing > Fix For: 1.11.0 > > > it's mainly to verify the performance don't less than 1.10 version by > checking the metrics of end-to-end performance test,such as qps，latency . -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18076) Sql client uses wrong class loader when parsing queries
[
https://issues.apache.org/jira/browse/FLINK-18076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17125668#comment-17125668
]
Leonard Xu commented on FLINK-18076:
[~dwysakowicz] sure, thanks.
> Sql client uses wrong class loader when parsing queries
> ---
>
> Key: FLINK-18076
> URL: https://issues.apache.org/jira/browse/FLINK-18076
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Client
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Dawid Wysakowicz
>Assignee: Leonard Xu
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.11.0
>
>
> Sql-client when parsing queries does not use the user class loader from
> {{ExecutionContext}}. This makes it impossible to query any sources if the
> dependencies are added with {{-j}} flag.
> In order to reproduce it try querying e.g. KafkaDynamicSource with
> {code}
> CREATE TABLE MyUserTable (
>f0 BIGINT
> ) WITH (
> 'connector' = 'kafka',
> 'topic' = 'topic_name', -- required: topic name from which the table is read
> -- required: specify the Kafka server connection string
> 'properties.bootstrap.servers' = 'localhost:9092',
> -- required for Kafka source, optional for Kafka sink, specify consumer
> group
> 'properties.group.id' = 'testGroup',
> -- optional: valid modes are "earliest-offset", "latest-offset",
> "group-offsets", "specific-offsets" or "timestamp"
> 'scan.startup.mode' = 'earliest-offset',
>'format' = 'avro'
> );
> SELECT * FROM MyUserTable;
> {code}
> It give exception:
> {code}
> Exception in thread "main" org.apache.flink.table.client.SqlClientException:
> Unexpected exception. This is a bug. Please consider filing an issue.
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:213)
> Caused by: org.apache.flink.table.client.gateway.SqlExecutionException:
> Invalidate SQL statement.
> at
> org.apache.flink.table.client.cli.SqlCommandParser.parseBySqlParser(SqlCommandParser.java:95)
> at
> org.apache.flink.table.client.cli.SqlCommandParser.parse(SqlCommandParser.java:79)
> at
> org.apache.flink.table.client.cli.CliClient.parseCommand(CliClient.java:256)
> at org.apache.flink.table.client.cli.CliClient.open(CliClient.java:212)
> at org.apache.flink.table.client.SqlClient.openCli(SqlClient.java:142)
> at org.apache.flink.table.client.SqlClient.start(SqlClient.java:114)
> at org.apache.flink.table.client.SqlClient.main(SqlClient.java:201)
> Caused by: org.apache.flink.table.api.ValidationException: Unable to create a
> source for reading table 'default_catalog.default_database.MyUserTable'.
> Table options are:
> 'connector'='kafka'
> 'format'='avro'
> 'properties.bootstrap.servers'='localhost:9092'
> 'properties.group.id'='testGroup'
> 'scan.startup.mode'='earliest-offset'
> 'topic'='topic_name'
> at
> org.apache.flink.table.factories.FactoryUtil.createTableSource(FactoryUtil.java:125)
> at
> org.apache.flink.table.planner.plan.schema.CatalogSourceTable.buildTableScan(CatalogSourceTable.scala:135)
> at
> org.apache.flink.table.planner.plan.schema.CatalogSourceTable.toRel(CatalogSourceTable.scala:78)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.toRel(SqlToRelConverter.java:3492)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertIdentifier(SqlToRelConverter.java:2415)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2102)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertFrom(SqlToRelConverter.java:2051)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelectImpl(SqlToRelConverter.java:661)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertSelect(SqlToRelConverter.java:642)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQueryRecursive(SqlToRelConverter.java:3345)
> at
> org.apache.calcite.sql2rel.SqlToRelConverter.convertQuery(SqlToRelConverter.java:568)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.org$apache$flink$table$planner$calcite$FlinkPlannerImpl$$rel(FlinkPlannerImpl.scala:164)
> at
> org.apache.flink.table.planner.calcite.FlinkPlannerImpl.rel(FlinkPlannerImpl.scala:151)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.toQueryOperation(SqlToOperationConverter.java:773)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convertSqlQuery(SqlToOperationConverter.java:745)
> at
> org.apache.flink.table.planner.operations.SqlToOperationConverter.convert(SqlToOperationConverter.java:238)
> at
> org.apache.flink.table.planner.delegation.ParserImpl.parse(ParserImpl.java:66)
> at
>
[jira] [Assigned] (FLINK-18118) Some SQL Jobs with two input operators are loosing data with unaligned checkpoints
[
https://issues.apache.org/jira/browse/FLINK-18118?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Piotr Nowojski reassigned FLINK-18118:
--
Assignee: Arvid Heise
> Some SQL Jobs with two input operators are loosing data with unaligned
> checkpoints
> --
>
> Key: FLINK-18118
> URL: https://issues.apache.org/jira/browse/FLINK-18118
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Checkpointing, Table SQL / Runtime
>Affects Versions: 1.11.0
>Reporter: Jark Wu
>Assignee: Arvid Heise
>Priority: Blocker
> Fix For: 1.11.0
>
>
> After trying to enable unaligned checkpoints by default, a lot of Blink
> streaming SQL/Table API tests containing joins or set operations are throwing
> errors that are indicating we are loosing some data (full records, without
> deserialisation errors). Example errors:
> {noformat}
> [ERROR] Failures:
> [ERROR] JoinITCase.testFullJoinWithEqualPk:775 expected: 3,3, null,4, null,5)> but was:
> [ERROR] JoinITCase.testStreamJoinWithSameRecord:391 expected: 1,1,1,1, 2,2,2,2, 2,2,2,2, 3,3,3,3, 3,3,3,3, 4,4,4,4, 4,4,4,4, 5,5,5,5,
> 5,5,5,5)> but was:
> [ERROR] SemiAntiJoinStreamITCase.testAntiJoin:352 expected:<0> but was:<1>
> [ERROR] SetOperatorsITCase.testIntersect:55 expected: 2,2,Hello, 3,2,Hello world)> but was:
> [ERROR] JoinITCase.testJoinPushThroughJoin:1272 expected: 2,1,Hello, 2,1,Hello world)> but was:
> {noformat}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] kl0u commented on pull request #12455: [FLINK-17935] Move setting yarn.log-config-file to YarnClusterClientFactory
kl0u commented on pull request #12455: URL: https://github.com/apache/flink/pull/12455#issuecomment-638692082 Yep, this seems to be a mistake. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
