[GitHub] [flink] azagrebin edited a comment on pull request #14199: [FLINK-20118][file connector] Extend FileSourceTextLinesITCase with JM/TM failover tests
azagrebin edited a comment on pull request #14199: URL: https://github.com/apache/flink/pull/14199#issuecomment-735617972 Thanks for the review @StephanEwen I addressed comments, the commits can be also just merged into one before merge. I can merge the PR if you want. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] azagrebin edited a comment on pull request #14199: [FLINK-20118][file connector] Extend FileSourceTextLinesITCase with JM/TM failover tests
azagrebin edited a comment on pull request #14199: URL: https://github.com/apache/flink/pull/14199#issuecomment-735617972 Thanks for the review @StephanEwen I addressed comments, the commits can be also just merged into one before merge. I can merge the PR if you want after at least CI compile gives green. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] azagrebin commented on pull request #14199: [FLINK-20118][file connector] Extend FileSourceTextLinesITCase with JM/TM failover tests
azagrebin commented on pull request #14199: URL: https://github.com/apache/flink/pull/14199#issuecomment-735617972 Thanks for the review @StephanEwen I addressed comments, the commits can be also just merged into one before merge. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #14199: [FLINK-20118][file connector] Extend FileSourceTextLinesITCase with JM/TM failover tests
azagrebin commented on a change in pull request #14199:
URL: https://github.com/apache/flink/pull/14199#discussion_r532397107
##
File path:
flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/src/FileSourceTextLinesITCase.java
##
@@ -61,20 +82,83 @@
@ClassRule
public static final TemporaryFolder TMP_FOLDER = new TemporaryFolder();
- @ClassRule
- public static final MiniClusterWithClientResource MINI_CLUSTER = new
MiniClusterWithClientResource(
- new MiniClusterResourceConfiguration.Builder()
- .setNumberTaskManagers(1)
- .setNumberSlotsPerTaskManager(PARALLELISM)
- .build());
+ private static TestingMiniCluster miniCluster;
- //
+ private static TestingEmbeddedHaServices highAvailabilityServices;
+
+ private static CompletedCheckpointStore checkpointStore;
+
+ @BeforeClass
+ public static void setupClass() throws Exception {
+ highAvailabilityServices = new
HaServices(TestingUtils.defaultExecutor(),
+ () -> checkpointStore,
+ new StandaloneCheckpointIDCounter());
+
+ final Configuration configuration = createConfiguration();
+
+ miniCluster = new TestingMiniCluster(
+ new TestingMiniClusterConfiguration.Builder()
+ .setConfiguration(configuration)
+ .setNumTaskManagers(1)
+ .setNumSlotsPerTaskManager(PARALLELISM)
+
.setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .build(),
+ () -> highAvailabilityServices);
+
+ miniCluster.start();
+ }
+
+ private static Configuration createConfiguration() throws IOException {
+ final Configuration configuration = new Configuration();
+ configuration.set(CheckpointingOptions.STATE_BACKEND,
"filesystem");
Review comment:
True, we can remove `filesystem`. I think it is just a leftover from my
experiments that the checkpoint data stays available between fail-overs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #14199: [FLINK-20118][file connector] Extend FileSourceTextLinesITCase with JM/TM failover tests
azagrebin commented on a change in pull request #14199:
URL: https://github.com/apache/flink/pull/14199#discussion_r532397548
##
File path:
flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/src/FileSourceTextLinesITCase.java
##
@@ -84,18 +168,33 @@ public void testBoundedTextFileSource() throws Exception {
writeHiddenJunkFiles(testDir);
final FileSource source = FileSource
- .forRecordStreamFormat(new TextLineFormat(),
Path.fromLocalFile(testDir))
- .build();
+ .forRecordStreamFormat(new TextLineFormat(),
Path.fromLocalFile(testDir))
+ .build();
- final StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
+ final StreamExecutionEnvironment env = new
TestStreamEnvironment(miniCluster, PARALLELISM);
env.setParallelism(PARALLELISM);
+ env.enableCheckpointing(10L);
Review comment:
makes sense
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14238: [FLINK-20347] Rework YARN documentation page
rmetzger commented on a change in pull request #14238:
URL: https://github.com/apache/flink/pull/14238#discussion_r532397327
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/scheduler/SchedulerBase.java
##
@@ -1020,7 +1020,7 @@ private void startAllOperatorCoordinators() {
catch (Throwable t) {
ExceptionUtils.rethrowIfFatalErrorOrOOM(t);
coordinators.forEach(IOUtils::closeQuietly);
- throw new FlinkRuntimeException("Failed to start the
operator coordinators", t);
+ failJob(new FlinkRuntimeException("Failed to start the
operator coordinators", t));
Review comment:
oh damn. I was trying something out. I should not use `git commit -am`
;)
Thanks a lot for this important catch!
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] azagrebin commented on a change in pull request #14199: [FLINK-20118][file connector] Extend FileSourceTextLinesITCase with JM/TM failover tests
azagrebin commented on a change in pull request #14199:
URL: https://github.com/apache/flink/pull/14199#discussion_r532397107
##
File path:
flink-connectors/flink-connector-files/src/test/java/org/apache/flink/connector/file/src/FileSourceTextLinesITCase.java
##
@@ -61,20 +82,83 @@
@ClassRule
public static final TemporaryFolder TMP_FOLDER = new TemporaryFolder();
- @ClassRule
- public static final MiniClusterWithClientResource MINI_CLUSTER = new
MiniClusterWithClientResource(
- new MiniClusterResourceConfiguration.Builder()
- .setNumberTaskManagers(1)
- .setNumberSlotsPerTaskManager(PARALLELISM)
- .build());
+ private static TestingMiniCluster miniCluster;
- //
+ private static TestingEmbeddedHaServices highAvailabilityServices;
+
+ private static CompletedCheckpointStore checkpointStore;
+
+ @BeforeClass
+ public static void setupClass() throws Exception {
+ highAvailabilityServices = new
HaServices(TestingUtils.defaultExecutor(),
+ () -> checkpointStore,
+ new StandaloneCheckpointIDCounter());
+
+ final Configuration configuration = createConfiguration();
+
+ miniCluster = new TestingMiniCluster(
+ new TestingMiniClusterConfiguration.Builder()
+ .setConfiguration(configuration)
+ .setNumTaskManagers(1)
+ .setNumSlotsPerTaskManager(PARALLELISM)
+
.setRpcServiceSharing(RpcServiceSharing.DEDICATED)
+ .build(),
+ () -> highAvailabilityServices);
+
+ miniCluster.start();
+ }
+
+ private static Configuration createConfiguration() throws IOException {
+ final Configuration configuration = new Configuration();
+ configuration.set(CheckpointingOptions.STATE_BACKEND,
"filesystem");
Review comment:
I think, we can remove `filesystem`. I think it is just a leftover from
my experiments that the checkpoint data stays available between fail-overs.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Closed] (FLINK-19172) [AbstractFileStateBackend]
[
https://issues.apache.org/jira/browse/FLINK-19172?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yu Li closed FLINK-19172.
-
Resolution: Not A Problem
In Flink we use the code convention that method parameters are non-null unless
marked with {{@Nullable}} annotation, which also applies to the
{{AbstractFileStateBackend#validatePath}} method. And we could see the
[caller|https://github.com/apache/flink/blob/master/flink-runtime/src/main/java/org/apache/flink/runtime/state/filesystem/AbstractFileStateBackend.java#L109-L110]
of this method follows this rule.
Please feel free to reopen the issue if you have any other concerns
[~alessiosavi]. Thanks.
> [AbstractFileStateBackend]
> --
>
> Key: FLINK-19172
> URL: https://issues.apache.org/jira/browse/FLINK-19172
> Project: Flink
> Issue Type: Bug
> Components: FileSystems, Runtime / State Backends
>Affects Versions: 1.8.0
>Reporter: Alessio Savi
>Priority: Minor
> Attachments: Flink.PNG
>
>
> The method `validatePath` of class `AbstractFileStateBackend` does not check
> if the pathPart retrived from the input `Path` is blank. Instead, it only
> check if it is null.
> Is this a bug?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14152: [FLINK-19082][doc] Add docs for temporal table and temporal table join
flinkbot edited a comment on pull request #14152: URL: https://github.com/apache/flink/pull/14152#issuecomment-731145130 ## CI report: * dfa09ece231af0ac4c36c21c9b192aa97c6b70f1 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10306) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14238: [FLINK-20347] Rework YARN documentation page
rmetzger commented on a change in pull request #14238:
URL: https://github.com/apache/flink/pull/14238#discussion_r532395243
##
File path: docs/deployment/filesystems/index.zh.md
##
@@ -82,7 +82,7 @@ cp ./opt/flink-s3-fs-hadoop-{{ site.version }}.jar
./plugins/s3-fs-hadoop/
## Hadoop 文件系统 (HDFS) 及其其他实现
所有 Flink 无法找到直接支持的文件系统均将回退为 Hadoop。
-当 `flink-runtime` 和 Hadoop 类包含在 classpath 中时，所有的 Hadoop 文件系统将自动可用。参见 **[Hadoop
集成]({% link deployment/resource-providers/hadoop.zh.md %})**。
+
Review comment:
Thanks ;)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-20415) Parsed QueryOperation does not contain watermark in schema
[
https://issues.apache.org/jira/browse/FLINK-20415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ingo Bürk updated FLINK-20415:
--
Description:
A parsed QueryOperation (e.g. from Parser#parse) does not contain the watermark
information in its schema (QueryOperation#getTableSchema). The information does
appear to be available in the calcite tree.
This can be reproduced by creating a simple table with a watermark, e.g.:
{code:java}
CREATE TABLE x (
id INT,
ts TIMESTAMP(3),
WATERMARK FOR ts AS ts
) WITH (
'connector' = 'datagen'
){code}
… and then parsing a statement like `SELECT * FROM x`.
API-wise, QueryOperation is currently marked internal (and as such this is only
marked as an improvement), but TableSchema is public-evolving, and if
QueryOperation was as well this could be seen as a bug instead.
Note that this might also affect other descendants of Operation which contain
table schema information (I haven't tested all of them), e.g.
CatalogQueryOperation.
was:
A parsed QueryOperation (e.g. from Parser#parse) does not contain the watermark
information in its schema (QueryOperation#getTableSchema). The information does
appear to be available in the calcite tree.
This can be reproduced by creating a simple table with a watermark, e.g.:
{code:java}
CREATE TABLE x (
id INT,
ts TIMESTAMP(3),
WATERMARK FOR ts AS ts
) WITH (
'connector' = 'datagen'
){code}
… and then parsing a statement like `SELECT * FROM x`.
API-wise, QueryOperation is currently marked internal (and as such this is only
marked as an improvement), but TableSchema is public-evolving, and if
QueryOperation was as well this could be seen as a bug instead.
Note that this might also affect other descendats of Operation which contain
table schema information (I haven't tested all of them), e.g.
CatalogQueryOperation.
> Parsed QueryOperation does not contain watermark in schema
> --
>
> Key: FLINK-20415
> URL: https://issues.apache.org/jira/browse/FLINK-20415
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.11.2
>Reporter: Ingo Bürk
>Priority: Minor
>
> A parsed QueryOperation (e.g. from Parser#parse) does not contain the
> watermark information in its schema (QueryOperation#getTableSchema). The
> information does appear to be available in the calcite tree.
> This can be reproduced by creating a simple table with a watermark, e.g.:
> {code:java}
> CREATE TABLE x (
> id INT,
> ts TIMESTAMP(3),
> WATERMARK FOR ts AS ts
> ) WITH (
> 'connector' = 'datagen'
> ){code}
> … and then parsing a statement like `SELECT * FROM x`.
> API-wise, QueryOperation is currently marked internal (and as such this is
> only marked as an improvement), but TableSchema is public-evolving, and if
> QueryOperation was as well this could be seen as a bug instead.
>
> Note that this might also affect other descendants of Operation which contain
> table schema information (I haven't tested all of them), e.g.
> CatalogQueryOperation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-20415) Parsed QueryOperation does not contain watermark in schema
[
https://issues.apache.org/jira/browse/FLINK-20415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ingo Bürk updated FLINK-20415:
--
Description:
A parsed QueryOperation (e.g. from Parser#parse) does not contain the watermark
information in its schema (QueryOperation#getTableSchema). The information does
appear to be available in the calcite tree.
This can be reproduced by creating a simple table with a watermark, e.g.:
{code:java}
CREATE TABLE x (
id INT,
ts TIMESTAMP(3),
WATERMARK FOR ts AS ts
) WITH (
'connector' = 'datagen'
){code}
… and then parsing a statement like `SELECT * FROM x`.
API-wise, QueryOperation is currently marked internal (and as such this is only
marked as an improvement), but TableSchema is public-evolving, and if
QueryOperation was as well this could be seen as a bug instead.
Note that this might also affect other descendats of Operation which contain
table schema information (I haven't tested all of them), e.g.
CatalogQueryOperation.
was:
A parsed QueryOperation (e.g. from Parser#parse) does not contain the watermark
information in its schema (QueryOperation#getTableSchema). The information does
appear to be available in the calcite tree.
This can be reproduced by creating a simple table with a watermark, e.g.:
{code:java}
CREATE TABLE x (
id INT,
ts TIMESTAMP(3),
WATERMARK FOR ts AS ts
) WITH (
'connector' = 'datagen'
){code}
… and then parsing a statement like `SELECT * FROM x`.
API-wise, QueryOperation is currently marked internal (and as such this is only
marked as an improvement), but TableSchema is public-evolving, and if
QueryOperation was as well this could be seen as a bug instead.
> Parsed QueryOperation does not contain watermark in schema
> --
>
> Key: FLINK-20415
> URL: https://issues.apache.org/jira/browse/FLINK-20415
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / API
>Affects Versions: 1.11.2
>Reporter: Ingo Bürk
>Priority: Minor
>
> A parsed QueryOperation (e.g. from Parser#parse) does not contain the
> watermark information in its schema (QueryOperation#getTableSchema). The
> information does appear to be available in the calcite tree.
> This can be reproduced by creating a simple table with a watermark, e.g.:
> {code:java}
> CREATE TABLE x (
> id INT,
> ts TIMESTAMP(3),
> WATERMARK FOR ts AS ts
> ) WITH (
> 'connector' = 'datagen'
> ){code}
> … and then parsing a statement like `SELECT * FROM x`.
> API-wise, QueryOperation is currently marked internal (and as such this is
> only marked as an improvement), but TableSchema is public-evolving, and if
> QueryOperation was as well this could be seen as a bug instead.
>
> Note that this might also affect other descendats of Operation which contain
> table schema information (I haven't tested all of them), e.g.
> CatalogQueryOperation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-19863) SQLClientHBaseITCase.testHBase failed with "java.io.IOException: Process failed due to timeout"
[
https://issues.apache.org/jira/browse/FLINK-19863?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240524#comment-17240524
]
Robert Metzger commented on FLINK-19863:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10273=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> SQLClientHBaseITCase.testHBase failed with "java.io.IOException: Process
> failed due to timeout"
> ---
>
> Key: FLINK-19863
> URL: https://issues.apache.org/jira/browse/FLINK-19863
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Assignee: Leonard Xu
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8541=logs=91bf6583-3fb2-592f-e4d4-d79d79c3230a=3425d8ba-5f03-540a-c64b-51b8481bf7d6
> {code}
> 00:50:02,589 [main] INFO
> org.apache.flink.tests.util.flink.FlinkDistribution [] - Stopping
> Flink cluster.
> 00:50:04,106 [main] INFO
> org.apache.flink.tests.util.flink.FlinkDistribution [] - Stopping
> Flink cluster.
> 00:50:04,741 [main] INFO
> org.apache.flink.tests.util.flink.LocalStandaloneFlinkResource [] - Backed up
> logs to
> /home/vsts/work/1/s/flink-end-to-end-tests/artifacts/flink-b3924665-1ac9-4309-8255-20f0dc94e7b9.
> 00:50:04,788 [main] INFO
> org.apache.flink.tests.util.hbase.LocalStandaloneHBaseResource [] - Stopping
> HBase Cluster
> 00:50:16,243 [main] ERROR
> org.apache.flink.tests.util.hbase.SQLClientHBaseITCase [] -
>
> Test testHBase[0:
> hbase-version:1.4.3](org.apache.flink.tests.util.hbase.SQLClientHBaseITCase)
> failed with:
> java.io.IOException: Process failed due to timeout.
> at
> org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlocking(AutoClosableProcess.java:130)
> at
> org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlocking(AutoClosableProcess.java:108)
> at
> org.apache.flink.tests.util.flink.FlinkDistribution.submitSQLJob(FlinkDistribution.java:221)
> at
> org.apache.flink.tests.util.flink.LocalStandaloneFlinkResource$StandaloneClusterController.submitSQLJob(LocalStandaloneFlinkResource.java:196)
> at
> org.apache.flink.tests.util.hbase.SQLClientHBaseITCase.executeSqlStatements(SQLClientHBaseITCase.java:215)
> at
> org.apache.flink.tests.util.hbase.SQLClientHBaseITCase.testHBase(SQLClientHBaseITCase.java:152)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20415) Parsed QueryOperation does not contain watermark in schema
Ingo Bürk created FLINK-20415:
-
Summary: Parsed QueryOperation does not contain watermark in schema
Key: FLINK-20415
URL: https://issues.apache.org/jira/browse/FLINK-20415
Project: Flink
Issue Type: Improvement
Components: Table SQL / API
Affects Versions: 1.11.2
Reporter: Ingo Bürk
A parsed QueryOperation (e.g. from Parser#parse) does not contain the watermark
information in its schema (QueryOperation#getTableSchema). The information does
appear to be available in the calcite tree.
This can be reproduced by creating a simple table with a watermark, e.g.:
{code:java}
CREATE TABLE x (
id INT,
ts TIMESTAMP(3),
WATERMARK FOR ts AS ts
) WITH (
'connector' = 'datagen'
){code}
… and then parsing a statement like `SELECT * FROM x`.
API-wise, QueryOperation is currently marked internal (and as such this is only
marked as an improvement), but TableSchema is public-evolving, and if
QueryOperation was as well this could be seen as a bug instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #10847: [FLINK-15578][connectors/jdbc] implement exactly once JDBC sink
flinkbot edited a comment on pull request #10847: URL: https://github.com/apache/flink/pull/10847#issuecomment-573933799 ## CI report: * 693e48ac1e9cd413f54e7cbc4c46b9ae190ce848 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10300) * 46042bff26813c15199ebe6fc9a9d3c1dd4e5860 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10308) * 1a5d68a1e74c02bb85534ffdde081fcb03d70917 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on a change in pull request #14254: [FLINK-20357][docs] Split HA documentation up into a general overview and the specific implementations
rmetzger commented on a change in pull request #14254:
URL: https://github.com/apache/flink/pull/14254#discussion_r532384255
##
File path: docs/deployment/ha/zookeeper_ha.md
##
@@ -23,113 +23,104 @@ specific language governing permissions and limitations
under the License.
-->
-## ZooKeeper HA Services
+Flink's ZooKeeper HA services use [ZooKeeper](http://zookeeper.apache.org) for
high availability services.
-One high availability services implementation uses ZooKeeper.
+* Toc
+{:toc}
-### Configuration
+Flink leverages **[ZooKeeper](http://zookeeper.apache.org)** for *distributed
coordination* between all running JobManager instances.
+ZooKeeper is a separate service from Flink, which provides highly reliable
distributed coordination via leader election and light-weight consistent state
storage.
+Check out [ZooKeeper's Getting Started
Guide](http://zookeeper.apache.org/doc/current/zookeeperStarted.html) for more
information about ZooKeeper.
+Flink includes scripts to [bootstrap a simple ZooKeeper](#bootstrap-zookeeper)
installation.
-To enable JobManager High Availability you have to set the **high-availability
mode** to *zookeeper*, configure a **ZooKeeper quorum** and set up a **masters
file** with all JobManagers hosts and their web UI ports.
+## Configuration
-Flink leverages **[ZooKeeper](http://zookeeper.apache.org)** for *distributed
coordination* between all running JobManager instances. ZooKeeper is a separate
service from Flink, which provides highly reliable distributed coordination via
leader election and light-weight consistent state storage. Check out
[ZooKeeper's Getting Started
Guide](http://zookeeper.apache.org/doc/current/zookeeperStarted.html) for more
information about ZooKeeper. Flink includes scripts to [bootstrap a simple
ZooKeeper](#bootstrap-zookeeper) installation.
+In order to start an HA-cluster you have to configure the following
configuration keys:
- Masters File (masters)
-
-In order to start an HA-cluster configure the *masters* file in `conf/masters`:
-
-- **masters file**: The *masters file* contains all hosts, on which
JobManagers are started, and the ports to which the web user interface binds.
-
-
-jobManagerAddress1:webUIPort1
-[...]
-jobManagerAddressX:webUIPortX
-
-
-By default, the job manager will pick a *random port* for inter process
communication. You can change this via the
**`high-availability.jobmanager.port`** key. This key accepts single ports
(e.g. `50010`), ranges (`5-50025`), or a combination of both
(`50010,50011,50020-50025,50050-50075`).
-
- Config File (flink-conf.yaml)
-
-In order to start an HA-cluster add the following configuration keys to
`conf/flink-conf.yaml`:
-
-- **high-availability mode** (required): The *high-availability mode* has to
be set in `conf/flink-conf.yaml` to *zookeeper* in order to enable high
availability mode.
-Alternatively this option can be set to FQN of factory class Flink should use
to create HighAvailabilityServices instance.
+- **high-availability mode** (required):
+The `high-availability` option has to be set to *zookeeper*.
high-availability: zookeeper
-- **ZooKeeper quorum** (required): A *ZooKeeper quorum* is a replicated group
of ZooKeeper servers, which provide the distributed coordination service.
+- **ZooKeeper quorum** (required):
+A *ZooKeeper quorum* is a replicated group of ZooKeeper servers, which provide
the distributed coordination service.
high-availability.zookeeper.quorum:
address1:2181[,...],addressX:2181
Each *addressX:port* refers to a ZooKeeper server, which is reachable by
Flink at the given address and port.
-- **ZooKeeper root** (recommended): The *root ZooKeeper node*, under which all
cluster nodes are placed.
+- **ZooKeeper root** (recommended):
+The *root ZooKeeper node*, under which all cluster nodes are placed.
- high-availability.zookeeper.path.root: /flink
+ high-availability.zookeeper.path.root: /flink
-- **ZooKeeper cluster-id** (recommended): The *cluster-id ZooKeeper node*,
under which all required coordination data for a cluster is placed.
+- **ZooKeeper cluster-id** (recommended):
+The *cluster-id ZooKeeper node*, under which all required coordination data
for a cluster is placed.
high-availability.cluster-id: /default_ns # important: customize per
cluster
- **Important**: You should not set this value manually when running a YARN
- cluster, a per-job YARN session, or on another cluster manager. In those
- cases a cluster-id is automatically being generated based on the application
- id. Manually setting a cluster-id overrides this behaviour in YARN.
- Specifying a cluster-id with the -z CLI option, in turn, overrides manual
- configuration. If you are running multiple Flink HA clusters on bare metal,
- you have to manually configure separate cluster-ids for each cluster.
+ **Important**:
+ You should not set this value manually when running on
[jira] [Commented] (FLINK-20195) Jobs endpoint returns duplicated jobs
[
https://issues.apache.org/jira/browse/FLINK-20195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240522#comment-17240522
]
Ingo Bürk commented on FLINK-20195:
---
[~chesnay] Yes, we only observed this during a transition to CANCELED.
> Jobs endpoint returns duplicated jobs
> -
>
> Key: FLINK-20195
> URL: https://issues.apache.org/jira/browse/FLINK-20195
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination, Runtime / REST
>Affects Versions: 1.11.2
>Reporter: Ingo Bürk
>Priority: Minor
> Fix For: 1.12.0
>
>
> The GET /jobs endpoint can, for a split second, return a duplicated job after
> it has been cancelled. This occurred in Ververica Platform after canceling a
> job (using PATCH /jobs/\{jobId}) and calling GET /jobs.
> I've reproduced this and queried the endpoint in a relatively tight loop (~
> every 0.5s) to log the responses of GET /jobs and got this:
>
>
> {code:java}
> …
> {"jobs":[{"id":"e110531c08dd4e3dbbfcf7afc1629c3d","status":"RUNNING"},{"id":"53fd11db25394308862c997dce9ef990","status":"CANCELLING"}]}
> {"jobs":[{"id":"e110531c08dd4e3dbbfcf7afc1629c3d","status":"RUNNING"},{"id":"53fd11db25394308862c997dce9ef990","status":"CANCELLING"}]}
> {"jobs":[{"id":"e110531c08dd4e3dbbfcf7afc1629c3d","status":"FAILED"},{"id":"53fd11db25394308862c997dce9ef990","status":"CANCELED"},{"id":"53fd11db25394308862c997dce9ef990","status":"CANCELED"}]}
> {"jobs":[{"id":"53fd11db25394308862c997dce9ef990","status":"CANCELED"},{"id":"e110531c08dd4e3dbbfcf7afc1629c3d","status":"FAILED"}]}
> {"jobs":[{"id":"53fd11db25394308862c997dce9ef990","status":"CANCELED"},{"id":"e110531c08dd4e3dbbfcf7afc1629c3d","status":"FAILED"}]}
> …{code}
>
> You can see in in between that for just a moment, the endpoint returned the
> same Job ID twice.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] wangyang0918 commented on a change in pull request #14254: [FLINK-20357][docs] Split HA documentation up into a general overview and the specific implementations
wangyang0918 commented on a change in pull request #14254:
URL: https://github.com/apache/flink/pull/14254#discussion_r532384451
##
File path: docs/deployment/ha/kubernetes_ha.md
##
@@ -2,7 +2,7 @@
title: "Kubernetes HA Services"
nav-title: Kubernetes HA Services
nav-parent_id: ha
-nav-pos: 2
+nav-pos: 6
Review comment:
nit: I am not sure about why we have the `nav-pos` 6 for
`kubernetes_ha.md` and 5 for `zookeeper_ha.md`. Does `2` and `1` also make
sense?
##
File path: docs/deployment/ha/kubernetes_ha.md
##
@@ -23,77 +23,50 @@ specific language governing permissions and limitations
under the License.
-->
-## Kubernetes Cluster High Availability
-Kubernetes high availability service could support both [standalone Flink on
Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %})
and [native Kubernetes integration]({% link
deployment/resource-providers/native_kubernetes.md %}).
+Flink's Kubernetes HA services use [Kubernetes](https://kubernetes.io/) for
high availability services.
-When running Flink JobManager as a Kubernetes deployment, the replica count
should be configured to 1 or greater.
-* The value `1` means that a new JobManager will be launched to take over
leadership if the current one terminates exceptionally.
-* The value `N` (greater than 1) means that multiple JobManagers will be
launched simultaneously while one is active and others are standby. Starting
more than one JobManager will make the recovery faster.
+* Toc
+{:toc}
-### Configuration
-{% highlight yaml %}
-kubernetes.cluster-id:
-high-availability:
org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
-high-availability.storageDir: hdfs:///flink/recovery
-{% endhighlight %}
+Kubernetes high availability services can only be used when deploying to
Kubernetes.
+Consequently, they can be configured when using [standalone Flink on
Kubernetes]({% link deployment/resource-providers/standalone/kubernetes.md %})
or the [native Kubernetes integration]({% link
deployment/resource-providers/native_kubernetes.md %})
- Example: Highly Available Standalone Flink Cluster on Kubernetes
-Both session and job/application clusters support using the Kubernetes high
availability service. Users just need to add the following Flink config options
to [flink-configuration-configmap.yaml]({% link
deployment/resource-providers/standalone/kubernetes.md
%}#common-cluster-resource-definitions). All other yamls do not need to be
updated.
-
-Note The filesystem which corresponds to
the scheme of your configured HA storage directory must be available to the
runtime. Refer to [custom Flink image]({% link
deployment/resource-providers/standalone/docker.md %}#customize-flink-image)
and [enable plugins]({% link deployment/resource-providers/standalone/docker.md
%}#using-plugins) for more information.
-
-{% highlight yaml %}
-apiVersion: v1
-kind: ConfigMap
-metadata:
- name: flink-config
- labels:
-app: flink
-data:
- flink-conf.yaml: |+
- ...
-kubernetes.cluster-id:
-high-availability:
org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
-high-availability.storageDir: hdfs:///flink/recovery
-restart-strategy: fixed-delay
-restart-strategy.fixed-delay.attempts: 10
- ...
-{% endhighlight %}
+## Configuration
- Example: Highly Available Native Kubernetes Cluster
-Using the following command to start a native Flink application cluster on
Kubernetes with high availability configured.
-{% highlight bash %}
-$ ./bin/flink run-application -p 8 -t kubernetes-application \
- -Dkubernetes.cluster-id= \
- -Dtaskmanager.memory.process.size=4096m \
- -Dkubernetes.taskmanager.cpu=2 \
- -Dtaskmanager.numberOfTaskSlots=4 \
- -Dkubernetes.container.image= \
-
-Dhigh-availability=org.apache.flink.kubernetes.highavailability.KubernetesHaServicesFactory
\
- -Dhigh-availability.storageDir=s3://flink/flink-ha \
- -Drestart-strategy=fixed-delay -Drestart-strategy.fixed-delay.attempts=10 \
-
-Dcontainerized.master.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar
\
-
-Dcontainerized.taskmanager.env.ENABLE_BUILT_IN_PLUGINS=flink-s3-fs-hadoop-{{site.version}}.jar
\
- local:///opt/flink/examples/streaming/StateMachineExample.jar
-{% endhighlight %}
+In order to start an HA-cluster you have to configure the following
configuration keys:
-### High Availability Data Clean Up
-Currently, when a Flink job reached the terminal state (`FAILED`, `CANCELED`,
`FINISHED`), all the HA data, including metadata in Kubernetes ConfigMap and HA
state on DFS, will be cleaned up.
+- **high-availability mode** (required):
+The [`high-availability`]({% link deployment/config.md %}#high-availability-1)
option has to be set to `KubernetesHaServicesFactory`.
-So the following command will only shut down the Flink session cluster and
leave all the HA related ConfigMaps, state untouched.
-{%
[GitHub] [flink] flinkbot edited a comment on pull request #13983: [FLINK-19989][python] Add collect operation in Python DataStream API
flinkbot edited a comment on pull request #13983: URL: https://github.com/apache/flink/pull/13983#issuecomment-723574143 ## CI report: * 89743b74e9f51dcb0b202a6ea54e2bd94947170e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10284) * 0e851de8b2911475922422c569a1e28a065e37b6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10309) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #10847: [FLINK-15578][connectors/jdbc] implement exactly once JDBC sink
flinkbot edited a comment on pull request #10847: URL: https://github.com/apache/flink/pull/10847#issuecomment-573933799 ## CI report: * 693e48ac1e9cd413f54e7cbc4c46b9ae190ce848 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10300) * 46042bff26813c15199ebe6fc9a9d3c1dd4e5860 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10308) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14238: [FLINK-20347] Rework YARN documentation page
wangyang0918 commented on a change in pull request #14238:
URL: https://github.com/apache/flink/pull/14238#discussion_r532339513
##
File path: docs/deployment/resource-providers/yarn.md
##
@@ -0,0 +1,238 @@
+---
+title: "Apache Hadoop YARN"
+nav-title: YARN
+nav-parent_id: resource_providers
+nav-pos: 4
+---
+
+
+* This will be replaced by the TOC
+{:toc}
+
+## Getting Started
+
+This *Getting Started* section guides you through setting up a fully
functional Flink Cluster on YARN.
+
+### Introduction
+
+[Apache Hadoop
YARN](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html)
is a resource provider popular with many data processing frameworks.
+Flink services are submitted to YARN's ResourceManager, which spawns
containers on machines managed by YARN NodeManagers. Flink deploys its
JobManager and TaskManager instances into such containers.
+
+Flink can dynamically allocate and de-allocate TaskManager resources depending
on the number of processing slots required by the job(s) running on the
JobManager.
+
+### Preparation
+
+This *Getting Started* section assumes a functional YARN environment, starting
from version 2.4.1. YARN environments are provided most conveniently through
services such as Amazon EMR, Google Cloud DataProc or products like Cloudera.
[Manually setting up a YARN environment
locally](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html)
or [on a
cluster](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html)
is not recommended for following through this *Getting Started* tutorial.
+
+- Make sure your YARN cluster is ready for accepting Flink applications by
running `yarn top`. It should show no error messages.
+- Download a recent Flink distribution from the [download page]({{
site.download_url }}) and unpack it.
+- **Important** Make sure that the `HADOOP_CLASSPATH` environment variable is
set up (it can be checked by running `echo $HADOOP_CLASSPATH`). If not, set it
up using
+
+{% highlight bash %}
+export HADOOP_CLASSPATH=`hadoop classpath`
+{% endhighlight %}
+
+### Starting a Flink Session on YARN
+
+Once you've made sure that the `HADOOP_CLASSPATH` environment variable is set,
you can launch a Flink on YARN session, and submit an example job:
+
+{% highlight bash %}
+
+# we assume to be in the root directory of the unzipped Flink distribution
+
+# (0) export HADOOP_CLASSPATH
+export HADOOP_CLASSPATH=`hadoop classpath`
+
+# (1) Start YARN Session
+./bin/yarn-session.sh --detached
+
+# (2) You can now access the Flink Web Interface through the URL printed in
the last lines of the command output, or through the YARN ResourceManager web
UI.
+
+# (3) Submit example job
+./bin/flink run ./examples/streaming/TopSpeedWindowing.jar
+
+# (4) Stop YARN session (replace the application id based on the output of the
yarn-session.sh command)
+echo "stop" | ./bin/yarn-session.sh -yid application_X_XXX
+{% endhighlight %}
+
+Congratulations! You have successfully run a Flink application by deploying
Flink on YARN.
+
+{% top %}
+
+## Deployment Modes Supported by Flink on YARN
+
+For production use, we recommend deploying Flink Applications in the [Per-job
or Application Mode]({% link deployment/index.md %}#deployment-modes), as these
modes provide a better isolation for the Applications.
+
+### Application Mode
+
+Application Mode will launch a Flink cluster on YARN, where the main() method
of the application jar gets executed on the JobManager in YARN.
+The cluster will shut down as soon as the application has finished. You can
manually stop the cluster using `yarn application -kill ` or by
cancelling the Flink job.
+
+{% highlight bash %}
+./bin/flink run-application -t yarn-application
./examples/streaming/TopSpeedWindowing.jar
+{% endhighlight %}
+
+
+Once an Application Mode cluster is deployed, you can interact with it for
operations like cancelling or taking a savepoint.
+
+{% highlight bash %}
+# List running job on the cluster
+./bin/flink list -t yarn-application -Dyarn.application.id=application__YY
+# Cancel running job
+./bin/flink cancel -t yarn-application
-Dyarn.application.id=application__YY
+{% endhighlight %}
+
+Note that cancelling your job on an Application Cluster will stop the cluster.
+
+To unlock the full potential of the application mode, consider using it with
the `yarn.provided.lib.dirs` configuration option
+and pre-upload your application jar to a location accessible by all nodes in
your cluster. In this case, the
+command could look like:
+
+{% highlight bash %}
+./bin/flink run-application -t yarn-application \
+ -Dyarn.provided.lib.dirs="hdfs://myhdfs/my-remote-flink-dist-dir" \
+ hdfs://myhdfs/jars/my-application.jar
+{% endhighlight %}
+
+The above will allow the job submission to be extra lightweight as the needed
Flink jars and the application jar
+are going to be picked up by
[GitHub] [flink] flinkbot edited a comment on pull request #13983: [FLINK-19989][python] Add collect operation in Python DataStream API
flinkbot edited a comment on pull request #13983: URL: https://github.com/apache/flink/pull/13983#issuecomment-723574143 ## CI report: * 89743b74e9f51dcb0b202a6ea54e2bd94947170e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10284) * 0e851de8b2911475922422c569a1e28a065e37b6 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13488: [FLINK-19381][docs] Fix docs for savepoint relocation
flinkbot edited a comment on pull request #13488: URL: https://github.com/apache/flink/pull/13488#issuecomment-699590439 ## CI report: * c8ba0ba6a2af6587da888a0365176c9e33759bad Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9954) * 3b8d5db8e7a5799df55ba6ce17a197e0fa01fe67 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10307) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #10847: [FLINK-15578][connectors/jdbc] implement exactly once JDBC sink
flinkbot edited a comment on pull request #10847: URL: https://github.com/apache/flink/pull/10847#issuecomment-573933799 ## CI report: * 693e48ac1e9cd413f54e7cbc4c46b9ae190ce848 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10300) * 46042bff26813c15199ebe6fc9a9d3c1dd4e5860 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] SteNicholas removed a comment on pull request #13983: [FLINK-19989][python] Add collect operation in Python DataStream API
SteNicholas removed a comment on pull request #13983: URL: https://github.com/apache/flink/pull/13983#issuecomment-735070443 @dianfu , I have followed your comments above mentioned. Please help to review again. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14165: [FLINK-19687][table] Support to get execution plan from StatementSet
flinkbot edited a comment on pull request #14165: URL: https://github.com/apache/flink/pull/14165#issuecomment-732020040 ## CI report: * c5728be97ab3b29a7dc4834b9fb2bbf63a0143dd Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10303) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14152: [FLINK-19082][doc] Add docs for temporal table and temporal table join
flinkbot edited a comment on pull request #14152: URL: https://github.com/apache/flink/pull/14152#issuecomment-731145130 ## CI report: * 25f4b1496af119c99555cfaf302a8eca568b1b8f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10082) * dfa09ece231af0ac4c36c21c9b192aa97c6b70f1 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10306) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13488: [FLINK-19381][docs] Fix docs for savepoint relocation
flinkbot edited a comment on pull request #13488: URL: https://github.com/apache/flink/pull/13488#issuecomment-699590439 ## CI report: * c8ba0ba6a2af6587da888a0365176c9e33759bad Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=9954) * 3b8d5db8e7a5799df55ba6ce17a197e0fa01fe67 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] rmetzger commented on pull request #14256: [FLINK-20396][checkpointing] Add a 'subtaskReset()' method to the OperatorCoordinator.
rmetzger commented on pull request #14256: URL: https://github.com/apache/flink/pull/14256#issuecomment-735580490 I reported the e2e failure in https://issues.apache.org/jira/browse/FLINK-20410. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20410) SQLClientSchemaRegistryITCase.testWriting failed with "Subject 'user_behavior' not found.; error code: 40401"
[
https://issues.apache.org/jira/browse/FLINK-20410?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240502#comment-17240502
]
Robert Metzger commented on FLINK-20410:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10291=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> SQLClientSchemaRegistryITCase.testWriting failed with "Subject
> 'user_behavior' not found.; error code: 40401"
> -
>
> Key: FLINK-20410
> URL: https://issues.apache.org/jira/browse/FLINK-20410
> Project: Flink
> Issue Type: Bug
> Components: Formats (JSON, Avro, Parquet, ORC, SequenceFile)
>Affects Versions: 1.12.0, 1.13.0
>Reporter: Dian Fu
>Priority: Critical
> Labels: test-stability
> Fix For: 1.12.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10276=logs=91bf6583-3fb2-592f-e4d4-d79d79c3230a=3425d8ba-5f03-540a-c64b-51b8481bf7d6
> {code}
> 2020-11-28T01:14:08.6444305Z Nov 28 01:14:08 [ERROR]
> testWriting(org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase)
> Time elapsed: 74.818 s <<< ERROR!
> 2020-11-28T01:14:08.6445353Z Nov 28 01:14:08
> io.confluent.kafka.schemaregistry.client.rest.exceptions.RestClientException:
> Subject 'user_behavior' not found.; error code: 40401
> 2020-11-28T01:14:08.6446071Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.rest.RestService.sendHttpRequest(RestService.java:292)
> 2020-11-28T01:14:08.6446910Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.rest.RestService.httpRequest(RestService.java:352)
> 2020-11-28T01:14:08.6447522Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.rest.RestService.getAllVersions(RestService.java:769)
> 2020-11-28T01:14:08.6448352Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.rest.RestService.getAllVersions(RestService.java:760)
> 2020-11-28T01:14:08.6449091Z Nov 28 01:14:08 at
> io.confluent.kafka.schemaregistry.client.CachedSchemaRegistryClient.getAllVersions(CachedSchemaRegistryClient.java:364)
> 2020-11-28T01:14:08.6449878Z Nov 28 01:14:08 at
> org.apache.flink.tests.util.kafka.SQLClientSchemaRegistryITCase.testWriting(SQLClientSchemaRegistryITCase.java:195)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14152: [FLINK-19082][doc] Add docs for temporal table and temporal table join
flinkbot edited a comment on pull request #14152: URL: https://github.com/apache/flink/pull/14152#issuecomment-731145130 ## CI report: * 25f4b1496af119c99555cfaf302a8eca568b1b8f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10082) * dfa09ece231af0ac4c36c21c9b192aa97c6b70f1 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13800: [FLINK-19650][connectors jdbc]Support the limit push down for the Jdb…
flinkbot edited a comment on pull request #13800: URL: https://github.com/apache/flink/pull/13800#issuecomment-717132623 ## CI report: * d03e417c4b828ffb3e3b8061d0a9b5dd9a38b018 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8374) * 6e7748ddea8ff84587d15a1c8050b14b51cf1fcb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10304) * e7a4ae2ed2de185200b562bb882af275a49e972e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10305) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] leonardBang commented on pull request #14152: [FLINK-19082][doc] Add docs for temporal table and temporal table join
leonardBang commented on pull request #14152: URL: https://github.com/apache/flink/pull/14152#issuecomment-735560931 @sjwiesman I've rebased again, do you have time to have a look? thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] carp84 commented on pull request #13488: [FLINK-19381][docs] Fix docs for savepoint relocation
carp84 commented on pull request #13488: URL: https://github.com/apache/flink/pull/13488#issuecomment-735552504 @klion26 Please also check @StephanEwen's comments [here](https://github.com/apache/flink/pull/14175#pullrequestreview-540475738). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #13800: [FLINK-19650][connectors jdbc]Support the limit push down for the Jdb…
flinkbot edited a comment on pull request #13800: URL: https://github.com/apache/flink/pull/13800#issuecomment-717132623 ## CI report: * d03e417c4b828ffb3e3b8061d0a9b5dd9a38b018 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8374) * 6e7748ddea8ff84587d15a1c8050b14b51cf1fcb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10304) * e7a4ae2ed2de185200b562bb882af275a49e972e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangyang0918 commented on a change in pull request #14238: [FLINK-20347] Rework YARN documentation page
wangyang0918 commented on a change in pull request #14238:
URL: https://github.com/apache/flink/pull/14238#discussion_r532329248
##
File path: docs/deployment/index.zh.md
##
@@ -25,47 +25,158 @@ specific language governing permissions and limitations
under the License.
-->
-When deciding how and where to run Flink, there's a wide range of options
available.
+Flink is a versatile framework, supporting many different deployment scenarios
in a mix and match fashion.
+
+Below, we briefly explain the building blocks of a Flink cluster, their
purpose and available implementations.
+If you just want to start Flink locally, we recommend setting up a [Standalone
Cluster]({% link deployment/resource-providers/standalone/index.md %}).
* This will be replaced by the TOC
{:toc}
+
+## Overview and Reference Architecture
+
+The figure below shows the building blocks of every Flink cluster. There is
always somewhere a client running. It takes the code of the Flink applications,
transforms it into a JobGraph and submits it to the JobManager.
+
+The JobManager distributes the work onto the TaskManagers, where the actual
operators (such as sources, transformations and sinks) are running.
+
+When deploying Flink, there are often multiple options available for each
building block. We have listed them in the table below the figure.
+
+
+
Review comment:
I really like the architecture figure and table. It helps the users to
dive into the deployment quickly.
##
File path: docs/deployment/filesystems/index.zh.md
##
@@ -82,7 +82,7 @@ cp ./opt/flink-s3-fs-hadoop-{{ site.version }}.jar
./plugins/s3-fs-hadoop/
## Hadoop 文件系统 (HDFS) 及其其他实现
所有 Flink 无法找到直接支持的文件系统均将回退为 Hadoop。
-当 `flink-runtime` 和 Hadoop 类包含在 classpath 中时，所有的 Hadoop 文件系统将自动可用。参见 **[Hadoop
集成]({% link deployment/resource-providers/hadoop.zh.md %})**。
+
Review comment:
Hmm. I think we should leave the following Chinese sentence.
当 `flink-runtime` 和 Hadoop 类包含在 classpath 中时，所有的 Hadoop 文件系统将自动可用。
The English version is as following.
All Hadoop file systems are automatically available when `flink-runtime` and
the Hadoop libraries are on the classpath.
##
File path: docs/deployment/resource-providers/yarn.md
##
@@ -0,0 +1,200 @@
+---
+title: "Apache Hadoop YARN"
+nav-title: YARN
+nav-parent_id: resource_providers
+nav-pos: 4
+---
+
+
+* This will be replaced by the TOC
+{:toc}
+
+## Getting Started
+
+This *Getting Started* section guides you through setting up a fully
functional Flink Cluster on YARN.
+
+### Introduction
+
+[Apache Hadoop
YARN](https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html)
is a resource provider popular with many data processing frameworks.
+Flink services are submitted to YARN's ResourceManager, which spawns
containers on machines managed by YARN NodeManagers. Flink deploys its
JobManager and TaskManager instances into such containers.
+
+Flink can dynamically allocate and de-allocate TaskManager resources depending
on the number of processing slots required by the job(s) running on the
JobManager.
+
+### Preparation
+
+This *Getting Started* section assumes a functional YARN environment, starting
from version 2.4.1. YARN environments are provided most conveniently through
services such as Amazon EMR, Google Cloud DataProc or products like Cloudera.
[Manually setting up a YARN environment
locally](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/SingleCluster.html)
or [on a
cluster](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/ClusterSetup.html)
is not recommended for following through this *Getting Started* tutorial.
+
+
+- Make sure your YARN cluster is ready for accepting Flink applications by
running `yarn top`. It should show no error messages.
+- Download a recent Flink distribution from the [download page]({{
site.download_url }}) and unpack it.
+- **Important** Make sure that the `HADOOP_CLASSPATH` environment variable is
set up (it can be checked by running `echo $HADOOP_CLASSPATH`). If not, set it
up using
+
+{% highlight bash %}
+export HADOOP_CLASSPATH=`hadoop classpath`
+{% endhighlight %}
+
+
+### Starting a Flink Session on YARN
Review comment:
+1 for also setting the `HADOOP_CONF_DIR` here. Especially when I am
submitting a Flink job from my Mac to a remote Yarn cluster. I should set the
two environments(`HADOOP_CONF_DIR` and `HADOOP_CLASSPATH`).
`HADOOP_HOME` is unnecessary when the `HADOOP_CONF_DIR` has been set.
##
File path: docs/deployment/resource-providers/yarn.md
##
@@ -0,0 +1,238 @@
+---
+title: "Apache Hadoop YARN"
+nav-title: YARN
+nav-parent_id: resource_providers
+nav-pos: 4
+---
+
+
+* This will be replaced by the TOC
+{:toc}
+
+## Getting Started
+
+This *Getting Started* section guides you through setting up a fully
functional Flink Cluster on YARN.
+
+### Introduction
+
[GitHub] [flink] flinkbot edited a comment on pull request #13800: [FLINK-19650][connectors jdbc]Support the limit push down for the Jdb…
flinkbot edited a comment on pull request #13800: URL: https://github.com/apache/flink/pull/13800#issuecomment-717132623 ## CI report: * d03e417c4b828ffb3e3b8061d0a9b5dd9a38b018 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=8374) * 6e7748ddea8ff84587d15a1c8050b14b51cf1fcb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Resolved] (FLINK-20288) Correct documentation about savepoint self-contained
[ https://issues.apache.org/jira/browse/FLINK-20288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yu Li resolved FLINK-20288. --- Resolution: Duplicate [~klion26] thanks for the reminder, closing this one as duplication of FLINK-19381 and let's track the work there [~yunta]. > Correct documentation about savepoint self-contained > > > Key: FLINK-20288 > URL: https://issues.apache.org/jira/browse/FLINK-20288 > Project: Flink > Issue Type: Bug > Components: Documentation, Runtime / Checkpointing >Affects Versions: 1.11.0 >Reporter: Yun Tang >Assignee: Yun Tang >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0, 1.11.4 > > > Savepoint self-contained has been supported while the documentation still > remain as not supported, we should fix that description. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] carp84 closed pull request #14175: [FLINK-20288][docs] Correct documentation about savepoint self-contained
carp84 closed pull request #14175: URL: https://github.com/apache/flink/pull/14175 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] carp84 commented on pull request #14175: [FLINK-20288][docs] Correct documentation about savepoint self-contained
carp84 commented on pull request #14175: URL: https://github.com/apache/flink/pull/14175#issuecomment-735533931 @klion26 I think you're right, this PR duplicates #13488 . Please check out Stephan's comments and address them in #13488 as well, thanks. @Myasuka @StephanEwen I'm closing this PR and let's further track it in #13488 , thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] carp84 commented on pull request #131: [FLINK-18968] Translate README.md to Chinese
carp84 commented on pull request #131: URL: https://github.com/apache/flink-statefun/pull/131#issuecomment-735532283 Closing this PR since it's already superseded by #176 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] carp84 closed pull request #131: [FLINK-18968] Translate README.md to Chinese
carp84 closed pull request #131: URL: https://github.com/apache/flink-statefun/pull/131 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20405) The LAG function in over window is not implemented correctly
[
https://issues.apache.org/jira/browse/FLINK-20405?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Leonard Xu closed FLINK-20405.
--
Resolution: Duplicate
> The LAG function in over window is not implemented correctly
>
>
> Key: FLINK-20405
> URL: https://issues.apache.org/jira/browse/FLINK-20405
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.12.0
>Reporter: Leonard Xu
>Priority: Major
>
> For LAG(input, offset, default) function in over window, it always return
> current row's input no matter how the offset is set.
> After see the codegen code of the function, I think the implementation is not
> correct and need to correct.
> {code:java}
> // the offset and default value is never used
> public UnboundedOverAggregateHelper$24(java.lang.Object[] references) throws
> Exception {constant$14 = ((int) 1);
> constant$14isNull = false;
> constant$15 = ((org.apache.flink.table.data.binary.BinaryStringData)
> str$13);constant$15isNull = false;
> typeSerializer$19 =
> (((org.apache.flink.table.runtime.typeutils.StringDataSerializer)
> references[0])); }
> public void accumulate(org.apache.flink.table.data.RowData accInput) throws
> Exception {
> org.apache.flink.table.data.binary.BinaryStringData field$21;
> boolean isNull$21;
> org.apache.flink.table.data.binary.BinaryStringData field$22;
> isNull$21 = accInput.isNullAt(2);field$21 =
> org.apache.flink.table.data.binary.BinaryStringData.EMPTY_UTF8;if
> (!isNull$21) { field$21 =
> ((org.apache.flink.table.data.binary.BinaryStringData)
> accInput.getString(2));}field$22 = field$21;
> if (!isNull$21) { field$22 =
> (org.apache.flink.table.data.binary.BinaryStringData)
> (typeSerializer$19.copy(field$22));}
> if (agg0_leadlag != field$22) { agg0_leadlag =
> ((org.apache.flink.table.data.binary.BinaryStringData)
> typeSerializer$19.copy(field$22));} ;
> agg0_leadlagIsNull = isNull$21; }
> {code}
>
> The question comes from user mail list[1]
> [1]
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/FlinkSQL-kafka-gt-dedup-gt-kafka-td39335.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20405) The LAG function in over window is not implemented correctly
[
https://issues.apache.org/jira/browse/FLINK-20405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240468#comment-17240468
]
Leonard Xu commented on FLINK-20405:
[~libenchao] You're wright, I will close and link to the earlier one.
> The LAG function in over window is not implemented correctly
>
>
> Key: FLINK-20405
> URL: https://issues.apache.org/jira/browse/FLINK-20405
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.12.0
>Reporter: Leonard Xu
>Priority: Major
>
> For LAG(input, offset, default) function in over window, it always return
> current row's input no matter how the offset is set.
> After see the codegen code of the function, I think the implementation is not
> correct and need to correct.
> {code:java}
> // the offset and default value is never used
> public UnboundedOverAggregateHelper$24(java.lang.Object[] references) throws
> Exception {constant$14 = ((int) 1);
> constant$14isNull = false;
> constant$15 = ((org.apache.flink.table.data.binary.BinaryStringData)
> str$13);constant$15isNull = false;
> typeSerializer$19 =
> (((org.apache.flink.table.runtime.typeutils.StringDataSerializer)
> references[0])); }
> public void accumulate(org.apache.flink.table.data.RowData accInput) throws
> Exception {
> org.apache.flink.table.data.binary.BinaryStringData field$21;
> boolean isNull$21;
> org.apache.flink.table.data.binary.BinaryStringData field$22;
> isNull$21 = accInput.isNullAt(2);field$21 =
> org.apache.flink.table.data.binary.BinaryStringData.EMPTY_UTF8;if
> (!isNull$21) { field$21 =
> ((org.apache.flink.table.data.binary.BinaryStringData)
> accInput.getString(2));}field$22 = field$21;
> if (!isNull$21) { field$22 =
> (org.apache.flink.table.data.binary.BinaryStringData)
> (typeSerializer$19.copy(field$22));}
> if (agg0_leadlag != field$22) { agg0_leadlag =
> ((org.apache.flink.table.data.binary.BinaryStringData)
> typeSerializer$19.copy(field$22));} ;
> agg0_leadlagIsNull = isNull$21; }
> {code}
>
> The question comes from user mail list[1]
> [1]
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/FlinkSQL-kafka-gt-dedup-gt-kafka-td39335.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] carp84 commented on a change in pull request #14248: [FLINK-20287][docs] Add documentation of how to switch memory allocator in Flink docker image
carp84 commented on a change in pull request #14248: URL: https://github.com/apache/flink/pull/14248#discussion_r532335274 ## File path: docs/deployment/resource-providers/standalone/docker.zh.md ## @@ -268,6 +268,18 @@ The `ENABLE_BUILT_IN_PLUGINS` should contain a list of plugin jar file names sep There are also more [advanced ways](#advanced-customization) for customizing the Flink image. +### Switch memory allocator + +Flink introduced `jemalloc` as default memory allocator to resolve memory fragmentation problem (please refer to [FLINK-19125](https://issues.apache.org/jira/browse/FLINK-19125)). +User could still switch back to previous `glibc` as memory allocator if wanting to keep behavior as before or meeting some unexpected memory related problem. +Please contact Flink community via user mail list if you found any regression under current `jemalloc` memory allocator. + +You could switch back to `glibc` as memory allocator by passing `disable-jemalloc` parameter: Review comment: ```suggestion You could switch back to use `glibc` as memory allocator to restore the old behavior or if any unexpected memory consumption or problem observed (and please report the issue via JIRA or mailing list if you found any), by passing `disable-jemalloc` parameter: ``` ## File path: docs/deployment/resource-providers/standalone/docker.md ## @@ -268,6 +268,18 @@ The `ENABLE_BUILT_IN_PLUGINS` should contain a list of plugin jar file names sep There are also more [advanced ways](#advanced-customization) for customizing the Flink image. +### Switch memory allocator + +Flink introduced `jemalloc` as default memory allocator to resolve memory fragmentation problem (please refer to [FLINK-19125](https://issues.apache.org/jira/browse/FLINK-19125)). +User could still switch back to previous `glibc` as memory allocator if wanting to keep behavior as before or meeting some unexpected memory related problem. +Please contact Flink community via user mail list if you found any regression under current `jemalloc` memory allocator. + +You could switch back to `glibc` as memory allocator by passing `disable-jemalloc` parameter: Review comment: ```suggestion You could switch back to use `glibc` as memory allocator to restore the old behavior or if any unexpected memory consumption or problem observed (and please report the issue via JIRA or mailing list if you found any), by passing `disable-jemalloc` parameter: ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20414) 【HBase】FileNotFoundException: File /tmp/hbase-deploy/hbase/lib does not exist
[
https://issues.apache.org/jira/browse/FLINK-20414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240466#comment-17240466
]
Jark Wu commented on FLINK-20414:
-
Did you find the reason? [~zhisheng]
> 【HBase】FileNotFoundException: File /tmp/hbase-deploy/hbase/lib does not exist
> -
>
> Key: FLINK-20414
> URL: https://issues.apache.org/jira/browse/FLINK-20414
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase, Table SQL / Ecosystem
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
> Fix For: 1.12.0
>
>
> {code:java}
> CREATE TABLE yarn_log_datagen_test_hbase_sink (
> appid INT,
> message STRING
> ) WITH (
> 'connector' = 'datagen',
> 'rows-per-second'='10',
> 'fields.appid.kind'='random',
> 'fields.appid.min'='1',
> 'fields.appid.max'='1000',
> 'fields.message.length'='100'
> );
> CREATE TABLE hbase_test1 (
> rowkey INT,
> family1 ROW
> ) WITH (
> 'connector' = 'hbase-1.4',
> 'table-name' = 'test_flink',
> 'zookeeper.quorum' = 'xxx:2181',
> 'sink.parallelism' = '2',
> 'sink.buffer-flush.interval' = '1',
> 'sink.buffer-flush.max-rows' = '1',
> 'sink.buffer-flush.max-size' = '1'
> );
> INSERT INTO hbase_test1 SELECT appid, ROW(message) FROM
> yarn_log_datagen_test_hbase_sink;
> {code}
> I run the sql, has exception, and data is not write into hbase, i add the
> flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar in the lib folder
>
> {code:java}
> 2020-11-30 10:49:45,772 DEBUG
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
> [] - Finding class again:
> org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException2020-11-30
> 10:49:45,774 DEBUG
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
> [] - Class org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException
> not found - using dynamical class loader2020-11-30 10:49:45,774 DEBUG
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
> [] - Finding class:
> org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException2020-11-30
> 10:49:45,774 DEBUG
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
> [] - Loading new jar files, if any2020-11-30 10:49:45,776 WARN
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
> [] - Failed to check remote dir status
> /tmp/hbase-deploy/hbase/libjava.io.FileNotFoundException: File
> /tmp/hbase-deploy/hbase/lib does not exist.at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:795)
> ~[hadoop-hdfs-2.7.3.4.jar:?]at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:106)
> ~[hadoop-hdfs-2.7.3.4.jar:?]at
> org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:853)
> ~[hadoop-hdfs-2.7.3.4.jar:?]at
> org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:849)
> ~[hadoop-hdfs-2.7.3.4.jar:?]at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-2.7.3.jar:?]at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:860)
> ~[hadoop-hdfs-2.7.3.4.jar:?]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader.loadNewJars(DynamicClassLoader.java:206)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader.tryRefreshClass(DynamicClassLoader.java:168)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader.loadClass(DynamicClassLoader.java:140)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> java.lang.Class.forName0(Native Method) ~[?:1.8.0_92]at
> java.lang.Class.forName(Class.java:348) [?:1.8.0_92]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.protobuf.ProtobufUtil.toException(ProtobufUtil.java:1753)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.protobuf.ResponseConverter.getResults(ResponseConverter.java:157)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:180)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:53)
>
[jira] [Assigned] (FLINK-20315) Optimize the compaction section for the document of table filesystem sink.
[ https://issues.apache.org/jira/browse/FLINK-20315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingsong Lee reassigned FLINK-20315: Assignee: Jingsong Lee > Optimize the compaction section for the document of table filesystem sink. > -- > > Key: FLINK-20315 > URL: https://issues.apache.org/jira/browse/FLINK-20315 > Project: Flink > Issue Type: Improvement > Components: Connectors / FileSystem, Documentation, Table SQL / > Ecosystem >Affects Versions: 1.12.0 >Reporter: Yun Gao >Assignee: Jingsong Lee >Priority: Critical > Fix For: 1.12.0 > > > Since if the files are big and the compaction use too much time, it will > backpressure the job and also extend the time period of checkpoint, we might > explicitly warns users about the possibility of this case to avoid perplexity. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (FLINK-20334) Introduce function endpoint path templating in module YAML specifications
[
https://issues.apache.org/jira/browse/FLINK-20334?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tzu-Li (Gordon) Tai reassigned FLINK-20334:
---
Assignee: Tzu-Li (Gordon) Tai
> Introduce function endpoint path templating in module YAML specifications
> -
>
> Key: FLINK-20334
> URL: https://issues.apache.org/jira/browse/FLINK-20334
> Project: Flink
> Issue Type: Sub-task
> Components: Stateful Functions
>Reporter: Tzu-Li (Gordon) Tai
>Assignee: Tzu-Li (Gordon) Tai
>Priority: Major
> Fix For: statefun-2.3.0
>
>
> In the current module specifications, function endpoints are defined like so:
> {code}
> functions:
> - function:
> meta:
> kind: http
> type: com.foo/world
> spec:
> endpoint: http://localhost:5959/statefun
> {code}
> A list of functions and their corresponding service endpoints are listed
> statically in the module specification file, which is loaded once on system
> startup. The system may only route messages to functions that have been
> defined. This prevents users from adding new functions to the application,
> without having to restart the system and reload new module specifications.
> We propose that instead of specifying functions, users should specify a
> "family" of function endpoints, like so:
> {code}
> functionEndpoints:
> - functionEndpoint:
> meta:
> kind: http
> spec:
> target:
> typename:
> namespace: com.foo.bar
> function: * # (can be wildcard * or a specific name)
> urlPathTemplate:
> "https://bar.foo.com:8000/{typename.function};
> connectTimeout: 1min
> # ... (other connection related configs that is shared for this
> endpoint family)
> {code}
> Note how users no longer define eager state per individual function. This is
> made possible by FLINK-20265, where state is now defined in the functions
> instead of in the module specifications.
> Function endpoint templating should only be supported in a new module
> specification format version (next version being {{3.0}}), where the previous
> way of defining individual functions is no longer supported.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink-statefun] tzulitai commented on pull request #181: [FLINK-18810][sdk] Golang remote functions SDK
tzulitai commented on pull request #181: URL: https://github.com/apache/flink-statefun/pull/181#issuecomment-735523466 Concerning the licensing: Since we won't be releasing artifacts for the Go SDK, I think ensuring compatibility for ASF 2.0 breaks down to: - Are there any copied source codes? If yes, the source license should be mentioned in a NOTICE file. - Are there any dependencies with a license not compatible with ASF 2.0? - Are there any dependencies that are modified in a way that violates its original license (e.g. shading). I quickly skimmed through the dependencies: - `github.com/golang/protobuf`: BSD 3-clause - `github.com/stretchr/testify`: MIT - `github.com/valyala/bytebufferpool`: MIT - `google.golang.org/protobuf`: BSD 3-clause Transitive dependencies are neither bundled or modified / copied in any way. Therefore, I think we should good on the licensing side ✅ This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-20414) 【HBase】FileNotFoundException: File /tmp/hbase-deploy/hbase/lib does not exist
[
https://issues.apache.org/jira/browse/FLINK-20414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhisheng closed FLINK-20414.
Resolution: Fixed
> 【HBase】FileNotFoundException: File /tmp/hbase-deploy/hbase/lib does not exist
> -
>
> Key: FLINK-20414
> URL: https://issues.apache.org/jira/browse/FLINK-20414
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase, Table SQL / Ecosystem
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
> Fix For: 1.12.0
>
>
> {code:java}
> CREATE TABLE yarn_log_datagen_test_hbase_sink (
> appid INT,
> message STRING
> ) WITH (
> 'connector' = 'datagen',
> 'rows-per-second'='10',
> 'fields.appid.kind'='random',
> 'fields.appid.min'='1',
> 'fields.appid.max'='1000',
> 'fields.message.length'='100'
> );
> CREATE TABLE hbase_test1 (
> rowkey INT,
> family1 ROW
> ) WITH (
> 'connector' = 'hbase-1.4',
> 'table-name' = 'test_flink',
> 'zookeeper.quorum' = 'xxx:2181',
> 'sink.parallelism' = '2',
> 'sink.buffer-flush.interval' = '1',
> 'sink.buffer-flush.max-rows' = '1',
> 'sink.buffer-flush.max-size' = '1'
> );
> INSERT INTO hbase_test1 SELECT appid, ROW(message) FROM
> yarn_log_datagen_test_hbase_sink;
> {code}
> I run the sql, has exception, and data is not write into hbase, i add the
> flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar in the lib folder
>
> {code:java}
> 2020-11-30 10:49:45,772 DEBUG
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
> [] - Finding class again:
> org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException2020-11-30
> 10:49:45,774 DEBUG
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
> [] - Class org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException
> not found - using dynamical class loader2020-11-30 10:49:45,774 DEBUG
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
> [] - Finding class:
> org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException2020-11-30
> 10:49:45,774 DEBUG
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
> [] - Loading new jar files, if any2020-11-30 10:49:45,776 WARN
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
> [] - Failed to check remote dir status
> /tmp/hbase-deploy/hbase/libjava.io.FileNotFoundException: File
> /tmp/hbase-deploy/hbase/lib does not exist.at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:795)
> ~[hadoop-hdfs-2.7.3.4.jar:?]at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:106)
> ~[hadoop-hdfs-2.7.3.4.jar:?]at
> org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:853)
> ~[hadoop-hdfs-2.7.3.4.jar:?]at
> org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:849)
> ~[hadoop-hdfs-2.7.3.4.jar:?]at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-2.7.3.jar:?]at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:860)
> ~[hadoop-hdfs-2.7.3.4.jar:?]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader.loadNewJars(DynamicClassLoader.java:206)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader.tryRefreshClass(DynamicClassLoader.java:168)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader.loadClass(DynamicClassLoader.java:140)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> java.lang.Class.forName0(Native Method) ~[?:1.8.0_92]at
> java.lang.Class.forName(Class.java:348) [?:1.8.0_92]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.protobuf.ProtobufUtil.toException(ProtobufUtil.java:1753)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.protobuf.ResponseConverter.getResults(ResponseConverter.java:157)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:180)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:53)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
>
[jira] [Updated] (FLINK-20414) 【HBase】FileNotFoundException: File /tmp/hbase-deploy/hbase/lib does not exist
[
https://issues.apache.org/jira/browse/FLINK-20414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jark Wu updated FLINK-20414:
Component/s: Table SQL / Ecosystem
> 【HBase】FileNotFoundException: File /tmp/hbase-deploy/hbase/lib does not exist
> -
>
> Key: FLINK-20414
> URL: https://issues.apache.org/jira/browse/FLINK-20414
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase, Table SQL / Ecosystem
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
> Fix For: 1.12.0
>
>
> {code:java}
> CREATE TABLE yarn_log_datagen_test_hbase_sink (
> appid INT,
> message STRING
> ) WITH (
> 'connector' = 'datagen',
> 'rows-per-second'='10',
> 'fields.appid.kind'='random',
> 'fields.appid.min'='1',
> 'fields.appid.max'='1000',
> 'fields.message.length'='100'
> );
> CREATE TABLE hbase_test1 (
> rowkey INT,
> family1 ROW
> ) WITH (
> 'connector' = 'hbase-1.4',
> 'table-name' = 'test_flink',
> 'zookeeper.quorum' = 'xxx:2181',
> 'sink.parallelism' = '2',
> 'sink.buffer-flush.interval' = '1',
> 'sink.buffer-flush.max-rows' = '1',
> 'sink.buffer-flush.max-size' = '1'
> );
> INSERT INTO hbase_test1 SELECT appid, ROW(message) FROM
> yarn_log_datagen_test_hbase_sink;
> {code}
> I run the sql, has exception, and data is not write into hbase, i add the
> flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar in the lib folder
>
> {code:java}
> 2020-11-30 10:49:45,772 DEBUG
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
> [] - Finding class again:
> org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException2020-11-30
> 10:49:45,774 DEBUG
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
> [] - Class org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException
> not found - using dynamical class loader2020-11-30 10:49:45,774 DEBUG
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
> [] - Finding class:
> org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException2020-11-30
> 10:49:45,774 DEBUG
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
> [] - Loading new jar files, if any2020-11-30 10:49:45,776 WARN
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
> [] - Failed to check remote dir status
> /tmp/hbase-deploy/hbase/libjava.io.FileNotFoundException: File
> /tmp/hbase-deploy/hbase/lib does not exist.at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:795)
> ~[hadoop-hdfs-2.7.3.4.jar:?]at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:106)
> ~[hadoop-hdfs-2.7.3.4.jar:?]at
> org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:853)
> ~[hadoop-hdfs-2.7.3.4.jar:?]at
> org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:849)
> ~[hadoop-hdfs-2.7.3.4.jar:?]at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-2.7.3.jar:?]at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:860)
> ~[hadoop-hdfs-2.7.3.4.jar:?]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader.loadNewJars(DynamicClassLoader.java:206)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader.tryRefreshClass(DynamicClassLoader.java:168)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader.loadClass(DynamicClassLoader.java:140)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> java.lang.Class.forName0(Native Method) ~[?:1.8.0_92]at
> java.lang.Class.forName(Class.java:348) [?:1.8.0_92]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.protobuf.ProtobufUtil.toException(ProtobufUtil.java:1753)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.protobuf.ResponseConverter.getResults(ResponseConverter.java:157)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:180)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:53)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
>
[jira] [Commented] (FLINK-20405) The LAG function in over window is not implemented correctly
[
https://issues.apache.org/jira/browse/FLINK-20405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240461#comment-17240461
]
Benchao Li commented on FLINK-20405:
This is duplicated with FLINK-19449 ?
> The LAG function in over window is not implemented correctly
>
>
> Key: FLINK-20405
> URL: https://issues.apache.org/jira/browse/FLINK-20405
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.12.0
>Reporter: Leonard Xu
>Priority: Major
>
> For LAG(input, offset, default) function in over window, it always return
> current row's input no matter how the offset is set.
> After see the codegen code of the function, I think the implementation is not
> correct and need to correct.
> {code:java}
> // the offset and default value is never used
> public UnboundedOverAggregateHelper$24(java.lang.Object[] references) throws
> Exception {constant$14 = ((int) 1);
> constant$14isNull = false;
> constant$15 = ((org.apache.flink.table.data.binary.BinaryStringData)
> str$13);constant$15isNull = false;
> typeSerializer$19 =
> (((org.apache.flink.table.runtime.typeutils.StringDataSerializer)
> references[0])); }
> public void accumulate(org.apache.flink.table.data.RowData accInput) throws
> Exception {
> org.apache.flink.table.data.binary.BinaryStringData field$21;
> boolean isNull$21;
> org.apache.flink.table.data.binary.BinaryStringData field$22;
> isNull$21 = accInput.isNullAt(2);field$21 =
> org.apache.flink.table.data.binary.BinaryStringData.EMPTY_UTF8;if
> (!isNull$21) { field$21 =
> ((org.apache.flink.table.data.binary.BinaryStringData)
> accInput.getString(2));}field$22 = field$21;
> if (!isNull$21) { field$22 =
> (org.apache.flink.table.data.binary.BinaryStringData)
> (typeSerializer$19.copy(field$22));}
> if (agg0_leadlag != field$22) { agg0_leadlag =
> ((org.apache.flink.table.data.binary.BinaryStringData)
> typeSerializer$19.copy(field$22));} ;
> agg0_leadlagIsNull = isNull$21; }
> {code}
>
> The question comes from user mail list[1]
> [1]
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/FlinkSQL-kafka-gt-dedup-gt-kafka-td39335.html
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14259: [FLINK-20290][runtime] Fix split duplication issues in Sources
flinkbot edited a comment on pull request #14259: URL: https://github.com/apache/flink/pull/14259#issuecomment-735471312 ## CI report: * eb435ff41287c9bc7d77e0ed5a711e10f8b53385 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10302) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20378) Watermark generation check TIMESTAMP_WITHOUT_TIME_ZONE
[
https://issues.apache.org/jira/browse/FLINK-20378?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240457#comment-17240457
]
wxmimperio commented on FLINK-20378:
[~jark]
Thanks,I will continue to follow this issue.
> Watermark generation check TIMESTAMP_WITHOUT_TIME_ZONE
> --
>
> Key: FLINK-20378
> URL: https://issues.apache.org/jira/browse/FLINK-20378
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.11.1
>Reporter: wxmimperio
>Priority: Major
>
>
>
> {code:java}
> def generateWatermarkGenerator(
> config: TableConfig,
> inputType: RowType,
> watermarkExpr: RexNode): GeneratedWatermarkGenerator = {
> // validation
> val watermarkOutputType =
> FlinkTypeFactory.toLogicalType(watermarkExpr.getType)
> if (watermarkOutputType.getTypeRoot !=
> LogicalTypeRoot.TIMESTAMP_WITHOUT_TIME_ZONE) {
> throw new CodeGenException(
> "WatermarkGenerator only accepts output data type of TIMESTAMP," +
> " but is " + watermarkOutputType)
> }
> {code}
>
> Why does watermark generation need to be detected as
> TIMESTAMP_WITHOUT_TIME_ZONE?
> If I remove this check, what effect will it have on the watermark?
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] caozhen1937 commented on pull request #14088: [FLINK-20168][docs-zh] Translate page 'Flink Architecture' into Chinese.
caozhen1937 commented on pull request #14088: URL: https://github.com/apache/flink/pull/14088#issuecomment-735519308 hi @wuchong ,please help to review if you have time. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] fsk119 commented on a change in pull request #14224: [FLINK-20274][docs-zh] Translate page 'testing' into Chinese
fsk119 commented on a change in pull request #14224:
URL: https://github.com/apache/flink/pull/14224#discussion_r532314361
##
File path: docs/dev/stream/testing.zh.md
##
@@ -62,7 +62,7 @@ class IncrementMapFunction extends MapFunction[Long, Long] {
-It is very easy to unit test such a function with your favorite testing
framework by passing suitable arguments and verifying the output.
+通过传递合适的参数并验证输出，可以很容易的使用你喜欢的测试框架对这样的函数进行单元测试。
Review comment:
缺了主语
"的" -> "地"
##
File path: docs/dev/stream/testing.zh.md
##
@@ -568,20 +568,20 @@ object CollectSink {
-A few remarks on integration testing with `MiniClusterWithClientResource`:
+关于使用 `MiniClusterWithClientResource` 进行集成测试的几点备注：
-* In order not to copy your whole pipeline code from production to test, make
sources and sinks pluggable in your production code and inject special test
sources and test sinks in your tests.
+* 为了不将整个 pipeline 代码从生产复制到测试，请将 source 和 sink 插入到生产代码中，并在测试中注入特殊的测试 source 和测试
sink。
-* The static variable in `CollectSink` is used here because Flink serializes
all operators before distributing them across a cluster.
-Communicating with operators instantiated by a local Flink mini cluster via
static variables is one way around this issue.
-Alternatively, you could write the data to files in a temporary directory with
your test sink.
+* 这里使用 `CollectSink` 中的静态变量，是因为Flink 在将所有算子分布到整个集群之前先对其进行了序列化。
+解决此问题的一种方法是与本地 Flink 小型集群通过静态变量实例化的算子进行通信。
Review comment:
"静态变量实例化的算子" -> "实例化算子的静态变量"
##
File path: docs/dev/stream/testing.zh.md
##
@@ -142,17 +142,17 @@ class IncrementFlatMapFunctionTest extends FlatSpec with
MockFactory {
-### Unit Testing Stateful or Timely UDFs & Custom Operators
+### 对有状态或及时 UDF 和自定义算子进行单元测试
-Testing the functionality of a user-defined function, which makes use of
managed state or timers is more difficult because it involves testing the
interaction between the user code and Flink's runtime.
-For this Flink comes with a collection of so called test harnesses, which can
be used to test such user-defined functions as well as custom operators:
+测试使用管理状态或定时器的用户自定义函数的功能更加困难，因为它涉及到测试用户代码和 Flink 运行时的交互。
Review comment:
测试使用管理状态或定时器的用户自定义函数的功能更加困难 -> 对使用管理状态或定时器的用户自定义函数的功能测试会更加困难
##
File path: docs/dev/stream/testing.zh.md
##
@@ -177,7 +177,7 @@ To use the test harnesses a set of additional dependencies
(test scoped) is need
{% endhighlight %}
-Now, the test harnesses can be used to push records and watermarks into your
user-defined functions or custom operators, control processing time and finally
assert on the output of the operator (including side outputs).
+现在，可以使用测试工具将记录和 watermark 推送到用户自定义函数或自定义算子中，控制处理时间，最后对算子的输出（包括旁路输出）进行断言。
Review comment:
'assert' -> 校验
##
File path: docs/dev/stream/testing.zh.md
##
@@ -568,20 +568,20 @@ object CollectSink {
-A few remarks on integration testing with `MiniClusterWithClientResource`:
+关于使用 `MiniClusterWithClientResource` 进行集成测试的几点备注：
-* In order not to copy your whole pipeline code from production to test, make
sources and sinks pluggable in your production code and inject special test
sources and test sinks in your tests.
+* 为了不将整个 pipeline 代码从生产复制到测试，请将 source 和 sink 插入到生产代码中，并在测试中注入特殊的测试 source 和测试
sink。
-* The static variable in `CollectSink` is used here because Flink serializes
all operators before distributing them across a cluster.
Review comment:
> The static variable in `CollectSink` is used here
I think it's more straightforward to illustrate which static field is used.
Therefore, I think we'd better to modify to
"The static variable `values` in `CollectSink`"
##
File path: docs/dev/stream/testing.zh.md
##
@@ -364,7 +364,7 @@ class PassThroughProcessFunction extends
ProcessFunction[Integer, Integer] {
-It is very easy to unit test such a function with
`ProcessFunctionTestHarnesses` by passing suitable arguments and verifying the
output.
+通过传递合适的参数并验证输出，对使用 `ProcessFunctionTestHarnesses` 这样的函数进行单元测试是非常容易的。
Review comment:
通过传递合适的参数，使用`ProcessFunctionTestHarnesses` 是很容易进行单元测试并验证输出。
##
File path: docs/dev/stream/testing.zh.md
##
@@ -325,18 +325,18 @@ class StatefulFlatMapTest extends FlatSpec with Matchers
with BeforeAndAfter {
-Many more examples for the usage of these test harnesses can be found in the
Flink code base, e.g.:
+在 Flink 代码库里可以找到更多使用这些测试工具的示例，例如：
-* `org.apache.flink.streaming.runtime.operators.windowing.WindowOperatorTest`
is a good example for testing operators and user-defined functions, which
depend on processing or event time.
-*
`org.apache.flink.streaming.api.functions.sink.filesystem.LocalStreamingFileSinkTest`
shows how to test a custom sink with the `AbstractStreamOperatorTestHarness`.
Specifically, it uses `AbstractStreamOperatorTestHarness.snapshot` and
[jira] [Updated] (FLINK-20414) 【HBase】FileNotFoundException: File /tmp/hbase-deploy/hbase/lib does not exist
[
https://issues.apache.org/jira/browse/FLINK-20414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhisheng updated FLINK-20414:
-
Summary: 【HBase】FileNotFoundException: File /tmp/hbase-deploy/hbase/lib
does not exist (was: 【HBase】)
> 【HBase】FileNotFoundException: File /tmp/hbase-deploy/hbase/lib does not exist
> -
>
> Key: FLINK-20414
> URL: https://issues.apache.org/jira/browse/FLINK-20414
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase
>Affects Versions: 1.12.0
>Reporter: zhisheng
>Priority: Major
> Fix For: 1.12.0
>
>
> {code:java}
> CREATE TABLE yarn_log_datagen_test_hbase_sink (
> appid INT,
> message STRING
> ) WITH (
> 'connector' = 'datagen',
> 'rows-per-second'='10',
> 'fields.appid.kind'='random',
> 'fields.appid.min'='1',
> 'fields.appid.max'='1000',
> 'fields.message.length'='100'
> );
> CREATE TABLE hbase_test1 (
> rowkey INT,
> family1 ROW
> ) WITH (
> 'connector' = 'hbase-1.4',
> 'table-name' = 'test_flink',
> 'zookeeper.quorum' = 'xxx:2181',
> 'sink.parallelism' = '2',
> 'sink.buffer-flush.interval' = '1',
> 'sink.buffer-flush.max-rows' = '1',
> 'sink.buffer-flush.max-size' = '1'
> );
> INSERT INTO hbase_test1 SELECT appid, ROW(message) FROM
> yarn_log_datagen_test_hbase_sink;
> {code}
> I run the sql, has exception, and data is not write into hbase, i add the
> flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar in the lib folder
>
> {code:java}
> 2020-11-30 10:49:45,772 DEBUG
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
> [] - Finding class again:
> org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException2020-11-30
> 10:49:45,774 DEBUG
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
> [] - Class org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException
> not found - using dynamical class loader2020-11-30 10:49:45,774 DEBUG
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
> [] - Finding class:
> org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException2020-11-30
> 10:49:45,774 DEBUG
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
> [] - Loading new jar files, if any2020-11-30 10:49:45,776 WARN
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
> [] - Failed to check remote dir status
> /tmp/hbase-deploy/hbase/libjava.io.FileNotFoundException: File
> /tmp/hbase-deploy/hbase/lib does not exist.at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:795)
> ~[hadoop-hdfs-2.7.3.4.jar:?]at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:106)
> ~[hadoop-hdfs-2.7.3.4.jar:?]at
> org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:853)
> ~[hadoop-hdfs-2.7.3.4.jar:?]at
> org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:849)
> ~[hadoop-hdfs-2.7.3.4.jar:?]at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> ~[hadoop-common-2.7.3.jar:?]at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:860)
> ~[hadoop-hdfs-2.7.3.4.jar:?]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader.loadNewJars(DynamicClassLoader.java:206)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader.tryRefreshClass(DynamicClassLoader.java:168)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader.loadClass(DynamicClassLoader.java:140)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> java.lang.Class.forName0(Native Method) ~[?:1.8.0_92]at
> java.lang.Class.forName(Class.java:348) [?:1.8.0_92]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.protobuf.ProtobufUtil.toException(ProtobufUtil.java:1753)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.protobuf.ResponseConverter.getResults(ResponseConverter.java:157)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:180)
> [flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
> org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:53)
>
[jira] [Created] (FLINK-20414) 【HBase】
zhisheng created FLINK-20414:
Summary: 【HBase】
Key: FLINK-20414
URL: https://issues.apache.org/jira/browse/FLINK-20414
Project: Flink
Issue Type: Bug
Components: Connectors / HBase
Affects Versions: 1.12.0
Reporter: zhisheng
Fix For: 1.12.0
{code:java}
CREATE TABLE yarn_log_datagen_test_hbase_sink (
appid INT,
message STRING
) WITH (
'connector' = 'datagen',
'rows-per-second'='10',
'fields.appid.kind'='random',
'fields.appid.min'='1',
'fields.appid.max'='1000',
'fields.message.length'='100'
);
CREATE TABLE hbase_test1 (
rowkey INT,
family1 ROW
) WITH (
'connector' = 'hbase-1.4',
'table-name' = 'test_flink',
'zookeeper.quorum' = 'xxx:2181',
'sink.parallelism' = '2',
'sink.buffer-flush.interval' = '1',
'sink.buffer-flush.max-rows' = '1',
'sink.buffer-flush.max-size' = '1'
);
INSERT INTO hbase_test1 SELECT appid, ROW(message) FROM
yarn_log_datagen_test_hbase_sink;
{code}
I run the sql, has exception, and data is not write into hbase, i add the
flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar in the lib folder
{code:java}
2020-11-30 10:49:45,772 DEBUG
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
[] - Finding class again:
org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException2020-11-30
10:49:45,774 DEBUG
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
[] - Class org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException not
found - using dynamical class loader2020-11-30 10:49:45,774 DEBUG
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
[] - Finding class:
org.apache.hadoop.hbase.regionserver.NoSuchColumnFamilyException2020-11-30
10:49:45,774 DEBUG
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
[] - Loading new jar files, if any2020-11-30 10:49:45,776 WARN
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader
[] - Failed to check remote dir status
/tmp/hbase-deploy/hbase/libjava.io.FileNotFoundException: File
/tmp/hbase-deploy/hbase/lib does not exist.at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:795)
~[hadoop-hdfs-2.7.3.4.jar:?]at
org.apache.hadoop.hdfs.DistributedFileSystem.access$700(DistributedFileSystem.java:106)
~[hadoop-hdfs-2.7.3.4.jar:?]at
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:853)
~[hadoop-hdfs-2.7.3.4.jar:?]at
org.apache.hadoop.hdfs.DistributedFileSystem$18.doCall(DistributedFileSystem.java:849)
~[hadoop-hdfs-2.7.3.4.jar:?]at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
~[hadoop-common-2.7.3.jar:?]at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:860)
~[hadoop-hdfs-2.7.3.4.jar:?]at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader.loadNewJars(DynamicClassLoader.java:206)
[flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader.tryRefreshClass(DynamicClassLoader.java:168)
[flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.util.DynamicClassLoader.loadClass(DynamicClassLoader.java:140)
[flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
java.lang.Class.forName0(Native Method) ~[?:1.8.0_92]at
java.lang.Class.forName(Class.java:348) [?:1.8.0_92]at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.protobuf.ProtobufUtil.toException(ProtobufUtil.java:1753)
[flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.protobuf.ResponseConverter.getResults(ResponseConverter.java:157)
[flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:180)
[flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.MultiServerCallable.call(MultiServerCallable.java:53)
[flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
[flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
org.apache.flink.hbase.shaded.org.apache.hadoop.hbase.client.AsyncProcess$AsyncRequestFutureImpl$SingleServerRequestRunnable.run(AsyncProcess.java:806)
[flink-sql-connector-hbase-1.4_2.11-1.12-SNAPSHOT.jar:1.12-SNAPSHOT]at
[GitHub] [flink] flinkbot edited a comment on pull request #14165: [FLINK-19687][table] Support to get execution plan from StatementSet
flinkbot edited a comment on pull request #14165: URL: https://github.com/apache/flink/pull/14165#issuecomment-732020040 ## CI report: * 560977065e15581698d2eec2307cd19a319ee918 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10286) * c5728be97ab3b29a7dc4834b9fb2bbf63a0143dd Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10303) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-16258) Update test utilities to FLIP-65
[
https://issues.apache.org/jira/browse/FLINK-16258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240446#comment-17240446
]
CaoZhen commented on FLINK-16258:
-
Hi [~twalthr], I'm so sorry to stay for this task for so long. This task is
difficult for me. I am still familiar with the new api. If there is a suitable
candidate, please assign this to him。
> Update test utilities to FLIP-65
>
>
> Key: FLINK-16258
> URL: https://issues.apache.org/jira/browse/FLINK-16258
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: Timo Walther
>Assignee: CaoZhen
>Priority: Major
>
> Currently, {{TestingTableEnvironment.registerFunction}} and
> {{TestingTableEnvironment.registerFunction}} are still using the old type
> system.
> This means that most tests still work on the old type system and actually
> don't test real API user-facing behavior.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20411) The annotation config do not support complex structure
[
https://issues.apache.org/jira/browse/FLINK-20411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240443#comment-17240443
]
Yang Wang commented on FLINK-20411:
---
I think the ConfigOption in Map type could support complex structure now. All
the thing you need to do is escape the strings. You could find more information
here[1]. In your case, the following config option could work.
{code:java}
-Dkubernetes.jobmanager.annotations="'k1:''{"a1":"v1","a2":"v2"}''','k2:''{"a1":"v1","a2":"v2"}'''"{code}
[1].
https://github.com/apache/flink/blob/5af1d007e26e0fb437028d5882d79fe09baf937a/flink-core/src/main/java/org/apache/flink/configuration/StructuredOptionsSplitter.java#L56
> The annotation config do not support complex structure
> --
>
> Key: FLINK-20411
> URL: https://issues.apache.org/jira/browse/FLINK-20411
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes
>Affects Versions: 1.11.2
>Reporter: Aitozi
>Priority: Minor
>
> Now we support user to set annotations by these config, the format is
> "k1:v1,k2:v2".
> # kubernetes.rest-service.annotations
> # kubernetes.jobmanager.annotations
> # kubernetes.taskmanager.annotations
> But the annotation may be more complex structure like
> "k1:\{"a1":"v1","a2":"v2"},k2:\{"a1":"v1","a2":"v2"}", It can not be parsed.
> Shall we support annotation and labels just like the environment variables
> use a prefix split mechanism, like :
> # kubernetes.rest-service.annotation.k1="\{"a1":"v1","a2":"v2"}"
> By this we can set some default cluster annotations and labels and no need to
> worry about overridden by user config.
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14165: [FLINK-19687][table] Support to get execution plan from StatementSet
flinkbot edited a comment on pull request #14165: URL: https://github.com/apache/flink/pull/14165#issuecomment-732020040 ## CI report: * 560977065e15581698d2eec2307cd19a319ee918 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10286) * c5728be97ab3b29a7dc4834b9fb2bbf63a0143dd UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] V1ncentzzZ edited a comment on pull request #14165: [FLINK-19687][table] Support to get execution plan from StatementSet
V1ncentzzZ edited a comment on pull request #14165: URL: https://github.com/apache/flink/pull/14165#issuecomment-735046296 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] V1ncentzzZ edited a comment on pull request #14165: [FLINK-19687][table] Support to get execution plan from StatementSet
V1ncentzzZ edited a comment on pull request #14165: URL: https://github.com/apache/flink/pull/14165#issuecomment-735046296 @flinkbot run azure. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20388) Supports users setting operators' metrics name
[ https://issues.apache.org/jira/browse/FLINK-20388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240425#comment-17240425 ] Jark Wu commented on FLINK-20388: - [~chesnay], yes, this is original from SQL, because the generated operator name can be very large. But I think DataStream users will also have this problem when the operator name is generated, not defined manually. > Supports users setting operators' metrics name > -- > > Key: FLINK-20388 > URL: https://issues.apache.org/jira/browse/FLINK-20388 > Project: Flink > Issue Type: Improvement > Components: API / DataStream, Runtime / Metrics >Reporter: hailong wang >Priority: Major > > Currently, we only support users setting operators name. > And we use those in the topology to distinguish operators, at the same time, > as the operator metrics name. > If the operator name length is larger than 80, we truncate it simply. > I think we can allow users to set operator metrics name like operators name. > If the user is not set, use the current way. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18634) FlinkKafkaProducerITCase.testRecoverCommittedTransaction failed with "Timeout expired after 60000milliseconds while awaiting InitProducerId"
[
https://issues.apache.org/jira/browse/FLINK-18634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240421#comment-17240421
]
Dian Fu commented on FLINK-18634:
-
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=10297=logs=1fc6e7bf-633c-5081-c32a-9dea24b05730=80a658d1-f7f6-5d93-2758-53ac19fd5b19
> FlinkKafkaProducerITCase.testRecoverCommittedTransaction failed with "Timeout
> expired after 6milliseconds while awaiting InitProducerId"
>
>
> Key: FLINK-18634
> URL: https://issues.apache.org/jira/browse/FLINK-18634
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Kafka, Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Dian Fu
>Priority: Major
> Labels: test-stability
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4590=logs=c5f0071e-1851-543e-9a45-9ac140befc32=684b1416-4c17-504e-d5ab-97ee44e08a20
> {code}
> 2020-07-17T11:43:47.9693015Z [ERROR] Tests run: 12, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 269.399 s <<< FAILURE! - in
> org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase
> 2020-07-17T11:43:47.9693862Z [ERROR]
> testRecoverCommittedTransaction(org.apache.flink.streaming.connectors.kafka.FlinkKafkaProducerITCase)
> Time elapsed: 60.679 s <<< ERROR!
> 2020-07-17T11:43:47.9694737Z org.apache.kafka.common.errors.TimeoutException:
> org.apache.kafka.common.errors.TimeoutException: Timeout expired after
> 6milliseconds while awaiting InitProducerId
> 2020-07-17T11:43:47.9695376Z Caused by:
> org.apache.kafka.common.errors.TimeoutException: Timeout expired after
> 6milliseconds while awaiting InitProducerId
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20014) Resolve CVE-2020-11022 and CVE-2020-11023 in scala-compiler
[ https://issues.apache.org/jira/browse/FLINK-20014?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240411#comment-17240411 ] Chesnay Schepler commented on FLINK-20014: -- Does this affect users running Flink, or only developers building it? > Resolve CVE-2020-11022 and CVE-2020-11023 in scala-compiler > --- > > Key: FLINK-20014 > URL: https://issues.apache.org/jira/browse/FLINK-20014 > Project: Flink > Issue Type: Improvement > Components: Build System >Reporter: Alan Leong >Priority: Major > > Update scala version to resolve CVE-2020-11022 and CVE-2020-11023 in > scala-compiler. This issue was addressed in Scala 2.12.12 > [https://github.com/scala/scala/pull/8963.|https://github.com/scala/bug/issues/11974] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20195) Jobs endpoint returns duplicated jobs
[
https://issues.apache.org/jira/browse/FLINK-20195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240410#comment-17240410
]
Chesnay Schepler commented on FLINK-20195:
--
Am I understanding you correctly that this happens when the job transitioned to
CANCELED? Or can it be reproduced for any job, regardless of state transitions?
If it just happens during a transition, then this likely occurs due to
{{Dispatcher#requestMultipleJobDetails}} not having a fully consistent view
over all jobs. It first queries all job masters for it's respective job, and
then the execution graph store where completed jobs reside in. It is
conceivable that a JM can return a job, the job is then archived to the store,
and then we retrieve the same job from the store.
An easy fix would be to de-duplicate entries based on the Job ID.
> Jobs endpoint returns duplicated jobs
> -
>
> Key: FLINK-20195
> URL: https://issues.apache.org/jira/browse/FLINK-20195
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination, Runtime / REST
>Affects Versions: 1.11.2
>Reporter: Ingo Bürk
>Priority: Minor
> Fix For: 1.12.0
>
>
> The GET /jobs endpoint can, for a split second, return a duplicated job after
> it has been cancelled. This occurred in Ververica Platform after canceling a
> job (using PATCH /jobs/\{jobId}) and calling GET /jobs.
> I've reproduced this and queried the endpoint in a relatively tight loop (~
> every 0.5s) to log the responses of GET /jobs and got this:
>
>
> {code:java}
> …
> {"jobs":[{"id":"e110531c08dd4e3dbbfcf7afc1629c3d","status":"RUNNING"},{"id":"53fd11db25394308862c997dce9ef990","status":"CANCELLING"}]}
> {"jobs":[{"id":"e110531c08dd4e3dbbfcf7afc1629c3d","status":"RUNNING"},{"id":"53fd11db25394308862c997dce9ef990","status":"CANCELLING"}]}
> {"jobs":[{"id":"e110531c08dd4e3dbbfcf7afc1629c3d","status":"FAILED"},{"id":"53fd11db25394308862c997dce9ef990","status":"CANCELED"},{"id":"53fd11db25394308862c997dce9ef990","status":"CANCELED"}]}
> {"jobs":[{"id":"53fd11db25394308862c997dce9ef990","status":"CANCELED"},{"id":"e110531c08dd4e3dbbfcf7afc1629c3d","status":"FAILED"}]}
> {"jobs":[{"id":"53fd11db25394308862c997dce9ef990","status":"CANCELED"},{"id":"e110531c08dd4e3dbbfcf7afc1629c3d","status":"FAILED"}]}
> …{code}
>
> You can see in in between that for just a moment, the endpoint returned the
> same Job ID twice.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20092) [Java 11] Multi-thread Flink compilation not working
[
https://issues.apache.org/jira/browse/FLINK-20092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240407#comment-17240407
]
Chesnay Schepler commented on FLINK-20092:
--
We cannot do that in general, because otherwise users would still see
dependencies that we have shaded.
I don't see a way to solve this that isn't significantly more error-prone
(e.g., manually marking dependencies as optional of they are shaded).
> [Java 11] Multi-thread Flink compilation not working
>
>
> Key: FLINK-20092
> URL: https://issues.apache.org/jira/browse/FLINK-20092
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Reporter: Maciej Bryński
>Priority: Major
>
> I'd like to use maven -T option when compiling flink.
> {code:java}
> mvn -T 2C clean install -D"scala-2.12" -DskipTests{code}
> Unfortunately my build is stuck on:
> {code:java}
> [INFO] --- maven-shade-plugin:3.2.1:shade (shade-flink) @
> flink-fs-hadoop-shaded ---
> [INFO] Including org.apache.hadoop:hadoop-common:jar:3.1.0 in the shaded jar.
> [INFO] Including org.apache.hadoop:hadoop-annotations:jar:3.1.0 in the shaded
> jar.
> [INFO] Including com.google.guava:guava:jar:11.0.2 in the shaded jar.
> [INFO] Including commons-io:commons-io:jar:2.7 in the shaded jar.
> [INFO] Including commons-collections:commons-collections:jar:3.2.2 in the
> shaded jar.
> [INFO] Including commons-logging:commons-logging:jar:1.1.3 in the shaded jar.
> [INFO] Including commons-lang:commons-lang:jar:2.6 in the shaded jar.
> [INFO] Including commons-beanutils:commons-beanutils:jar:1.9.3 in the shaded
> jar.
> [INFO] Including org.apache.commons:commons-configuration2:jar:2.1.1 in the
> shaded jar.
> [INFO] Including org.apache.commons:commons-lang3:jar:3.3.2 in the shaded jar.
> [INFO] Including com.google.re2j:re2j:jar:1.1 in the shaded jar.
> [INFO] Including org.apache.hadoop:hadoop-auth:jar:3.1.0 in the shaded jar.
> [INFO] Including org.apache.htrace:htrace-core4:jar:4.1.0-incubating in the
> shaded jar.
> [INFO] Including com.fasterxml.jackson.core:jackson-databind:jar:2.10.1 in
> the shaded jar.
> [INFO] Including com.fasterxml.jackson.core:jackson-annotations:jar:2.10.1 in
> the shaded jar.
> [INFO] Including com.fasterxml.jackson.core:jackson-core:jar:2.10.1 in the
> shaded jar.
> [INFO] Including org.codehaus.woodstox:stax2-api:jar:3.1.4 in the shaded jar.
> [INFO] Including com.fasterxml.woodstox:woodstox-core:jar:5.0.3 in the shaded
> jar.
> [INFO] Including org.apache.flink:force-shading:jar:1.12-SNAPSHOT in the
> shaded jar.
> [INFO] No artifact matching filter io.netty:netty
> [WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.
> [WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.
> [WARNING] Discovered module-info.class. Shading will break its strong
> encapsulation.
> [INFO] Replacing original artifact with shaded artifact.
> [INFO] Replacing
> /home/maverick/flink/flink-filesystems/flink-fs-hadoop-shaded/target/flink-fs-hadoop-shaded-1.12-SNAPSHOT.jar
> with
> /home/maverick/flink/flink-filesystems/flink-fs-hadoop-shaded/target/flink-fs-hadoop-shaded-1.12-SNAPSHOT-shaded.jar
> [INFO] Dependency-reduced POM written at:
> /home/maverick/flink/flink-filesystems/flink-fs-hadoop-shaded/target/dependency-reduced-pom.xml
> {code}
> Can we make flink compilation working with multiple maven threads ?
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-20388) Supports users setting operators' metrics name
[ https://issues.apache.org/jira/browse/FLINK-20388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240404#comment-17240404 ] Chesnay Schepler commented on FLINK-20388: -- Is this purely about SQL? The operator name used for metrics is derived from the actual operator name, and I see little reason to change this behavior. > Supports users setting operators' metrics name > -- > > Key: FLINK-20388 > URL: https://issues.apache.org/jira/browse/FLINK-20388 > Project: Flink > Issue Type: Improvement > Components: API / DataStream, Runtime / Metrics >Reporter: hailong wang >Priority: Major > > Currently, we only support users setting operators name. > And we use those in the topology to distinguish operators, at the same time, > as the operator metrics name. > If the operator name length is larger than 80, we truncate it simply. > I think we can allow users to set operator metrics name like operators name. > If the user is not set, use the current way. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20404) ZooKeeper quorum fails to start due to missing log4j library
[
https://issues.apache.org/jira/browse/FLINK-20404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240403#comment-17240403
]
Chesnay Schepler commented on FLINK-20404:
--
This is due to ZK 3.4 having a hard dependency on log4j 1, which we cannot fix.
You can try using ZK 3.5 (presumably ZK was decoupled from log4j1 in
ZOOKEEPER-1371), or use Flink with log4j1 as described in the
[documentation|https://ci.apache.org/projects/flink/flink-docs-release-1.11/monitoring/logging.html#configuring-log4j1].
> ZooKeeper quorum fails to start due to missing log4j library
>
>
> Key: FLINK-20404
> URL: https://issues.apache.org/jira/browse/FLINK-20404
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.11.2
>Reporter: Pedro Miguel Rainho Chaves
>Priority: Blocker
> Fix For: 1.12.0, 1.11.3
>
>
> Upon starting a zookeeper quorum using flink's bootstrapped zookeeper, it
> throws the following exception.
>
> {code:java}
> 2020-11-27 13:13:38,371 ERROR
> org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer [] - Error
> running ZooKeeper quorum peer: org/apache/log4j/jmx/HierarchyDynamicMBean
> java.lang.NoClassDefFoundError: org/apache/log4j/jmx/HierarchyDynamicMBean
> at
> org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:51)
> ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-11.0]
> at
> org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:125)
> ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-11.0]
> at
> org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer.runFlinkZkQuorumPeer(FlinkZooKeeperQuorumPeer.java:123)
> ~[flink-dist_2.11-1.11.2.jar:1.11.2]
> at
> org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer.main(FlinkZooKeeperQuorumPeer.java:79)
> [flink-dist_2.11-1.11.2.jar:1.11.2]
> Caused by: java.lang.ClassNotFoundException:
> org.apache.log4j.jmx.HierarchyDynamicMBean
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> ~[?:1.8.0_262]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> ~[?:1.8.0_262]
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> ~[?:1.8.0_262]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ~[?:1.8.0_262]
> ... 4 more
> {code}
> This happens because the new flink version is missing a log4j library. This
> can be solved by adding log4j-1.2.17.jar to the classpath, nonetheless the
> bootstrapped zookeepeer version should be compatible with the log4j2
> libraries that come with flink's default installation.
>
> *Steps to reproduce:*
> # Fresh install of flink version 1.11.2
> # Change the zookeeper config to start as a quorum
> {code:java}
> server.1=:2888:3888
> server.2=:2888:3888{code}
> # Start zookeeper
> # /bin/zookeeper.sh start-foreground 1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-20404) ZooKeeper quorum fails to start due to missing log4j library
[
https://issues.apache.org/jira/browse/FLINK-20404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler closed FLINK-20404.
Fix Version/s: (was: 1.11.3)
(was: 1.12.0)
Resolution: Won't Fix
> ZooKeeper quorum fails to start due to missing log4j library
>
>
> Key: FLINK-20404
> URL: https://issues.apache.org/jira/browse/FLINK-20404
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.11.2
>Reporter: Pedro Miguel Rainho Chaves
>Priority: Blocker
>
> Upon starting a zookeeper quorum using flink's bootstrapped zookeeper, it
> throws the following exception.
>
> {code:java}
> 2020-11-27 13:13:38,371 ERROR
> org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer [] - Error
> running ZooKeeper quorum peer: org/apache/log4j/jmx/HierarchyDynamicMBean
> java.lang.NoClassDefFoundError: org/apache/log4j/jmx/HierarchyDynamicMBean
> at
> org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:51)
> ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-11.0]
> at
> org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:125)
> ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-11.0]
> at
> org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer.runFlinkZkQuorumPeer(FlinkZooKeeperQuorumPeer.java:123)
> ~[flink-dist_2.11-1.11.2.jar:1.11.2]
> at
> org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer.main(FlinkZooKeeperQuorumPeer.java:79)
> [flink-dist_2.11-1.11.2.jar:1.11.2]
> Caused by: java.lang.ClassNotFoundException:
> org.apache.log4j.jmx.HierarchyDynamicMBean
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> ~[?:1.8.0_262]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> ~[?:1.8.0_262]
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> ~[?:1.8.0_262]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ~[?:1.8.0_262]
> ... 4 more
> {code}
> This happens because the new flink version is missing a log4j library. This
> can be solved by adding log4j-1.2.17.jar to the classpath, nonetheless the
> bootstrapped zookeepeer version should be compatible with the log4j2
> libraries that come with flink's default installation.
>
> *Steps to reproduce:*
> # Fresh install of flink version 1.11.2
> # Change the zookeeper config to start as a quorum
> {code:java}
> server.1=:2888:3888
> server.2=:2888:3888{code}
> # Start zookeeper
> # /bin/zookeeper.sh start-foreground 1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14258: [FLINK-20356][docs] Updated Mesos documentation.
flinkbot edited a comment on pull request #14258: URL: https://github.com/apache/flink/pull/14258#issuecomment-735471233 ## CI report: * 299cc3011151c3efb544fc5c9891c9418e9019ac Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10301) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14249: [FLINK-20391] Set FORWARD_EDGES_PIPELINED for BATCH ExecutionMode
flinkbot edited a comment on pull request #14249: URL: https://github.com/apache/flink/pull/14249#issuecomment-734789081 ## CI report: * 2ce421de97821f5fa47a965362d3c08c4971fa92 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10296) Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10260) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20404) ZooKeeper quorum fails to start due to missing log4j library
[
https://issues.apache.org/jira/browse/FLINK-20404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chesnay Schepler updated FLINK-20404:
-
Component/s: Build System
> ZooKeeper quorum fails to start due to missing log4j library
>
>
> Key: FLINK-20404
> URL: https://issues.apache.org/jira/browse/FLINK-20404
> Project: Flink
> Issue Type: Bug
> Components: Build System
>Affects Versions: 1.11.2
>Reporter: Pedro Miguel Rainho Chaves
>Priority: Blocker
> Fix For: 1.12.0, 1.11.3
>
>
> Upon starting a zookeeper quorum using flink's bootstrapped zookeeper, it
> throws the following exception.
>
> {code:java}
> 2020-11-27 13:13:38,371 ERROR
> org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer [] - Error
> running ZooKeeper quorum peer: org/apache/log4j/jmx/HierarchyDynamicMBean
> java.lang.NoClassDefFoundError: org/apache/log4j/jmx/HierarchyDynamicMBean
> at
> org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.jmx.ManagedUtil.registerLog4jMBeans(ManagedUtil.java:51)
> ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-11.0]
> at
> org.apache.flink.shaded.zookeeper3.org.apache.zookeeper.server.quorum.QuorumPeerMain.runFromConfig(QuorumPeerMain.java:125)
> ~[flink-shaded-zookeeper-3.4.14.jar:3.4.14-11.0]
> at
> org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer.runFlinkZkQuorumPeer(FlinkZooKeeperQuorumPeer.java:123)
> ~[flink-dist_2.11-1.11.2.jar:1.11.2]
> at
> org.apache.flink.runtime.zookeeper.FlinkZooKeeperQuorumPeer.main(FlinkZooKeeperQuorumPeer.java:79)
> [flink-dist_2.11-1.11.2.jar:1.11.2]
> Caused by: java.lang.ClassNotFoundException:
> org.apache.log4j.jmx.HierarchyDynamicMBean
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> ~[?:1.8.0_262]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> ~[?:1.8.0_262]
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
> ~[?:1.8.0_262]
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ~[?:1.8.0_262]
> ... 4 more
> {code}
> This happens because the new flink version is missing a log4j library. This
> can be solved by adding log4j-1.2.17.jar to the classpath, nonetheless the
> bootstrapped zookeepeer version should be compatible with the log4j2
> libraries that come with flink's default installation.
>
> *Steps to reproduce:*
> # Fresh install of flink version 1.11.2
> # Change the zookeeper config to start as a quorum
> {code:java}
> server.1=:2888:3888
> server.2=:2888:3888{code}
> # Start zookeeper
> # /bin/zookeeper.sh start-foreground 1
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14258: [FLINK-20356][docs] Updated Mesos documentation.
flinkbot edited a comment on pull request #14258: URL: https://github.com/apache/flink/pull/14258#issuecomment-735471233 ## CI report: * 299cc3011151c3efb544fc5c9891c9418e9019ac Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10301) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14259: [FLINK-20290][runtime] Fix split duplication issues in Sources
flinkbot edited a comment on pull request #14259: URL: https://github.com/apache/flink/pull/14259#issuecomment-735471312 ## CI report: * eb435ff41287c9bc7d77e0ed5a711e10f8b53385 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10302) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #10847: [FLINK-15578][connectors/jdbc] implement exactly once JDBC sink
flinkbot edited a comment on pull request #10847: URL: https://github.com/apache/flink/pull/10847#issuecomment-573933799 ## CI report: * 693e48ac1e9cd413f54e7cbc4c46b9ae190ce848 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10300) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14259: [FLINK-20290][runtime] Fix split duplication issues in Sources
flinkbot commented on pull request #14259: URL: https://github.com/apache/flink/pull/14259#issuecomment-735471312 ## CI report: * eb435ff41287c9bc7d77e0ed5a711e10f8b53385 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14258: [FLINK-20356][docs] Updated Mesos documentation.
flinkbot commented on pull request #14258: URL: https://github.com/apache/flink/pull/14258#issuecomment-735471233 ## CI report: * 299cc3011151c3efb544fc5c9891c9418e9019ac UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #10847: [FLINK-15578][connectors/jdbc] implement exactly once JDBC sink
flinkbot edited a comment on pull request #10847: URL: https://github.com/apache/flink/pull/10847#issuecomment-573933799 ## CI report: * b55a17559f8bbd427788152fdc5a2e79e273ad7a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10299) * 693e48ac1e9cd413f54e7cbc4c46b9ae190ce848 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10300) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14259: [FLINK-20290][runtime] Fix split duplication issues in Sources
flinkbot commented on pull request #14259: URL: https://github.com/apache/flink/pull/14259#issuecomment-735469674 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit eb435ff41287c9bc7d77e0ed5a711e10f8b53385 (Sun Nov 29 22:43:30 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-20290).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20290) Duplicated output in FileSource continuous ITCase with TM failover
[
https://issues.apache.org/jira/browse/FLINK-20290?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-20290:
---
Labels: pull-request-available (was: )
> Duplicated output in FileSource continuous ITCase with TM failover
> --
>
> Key: FLINK-20290
> URL: https://issues.apache.org/jira/browse/FLINK-20290
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.12.0
>Reporter: Andrey Zagrebin
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.12.0
>
> Attachments: collect-debug-jm.png, collect-debug.png, log
>
>
> If FileSourceTextLinesITCase::testContinuousTextFileSource includes TM
> restarts (after failing TM with TestingMiniCluster::terminateTaskExecutor,
> see testContinuousTextFileSourceWithTaskManagerFailover in
> [branch|https://github.com/azagrebin/flink/tree/FLINK-20118-it]) then
> sometimes I observe duplicated lines in the output after running the whole
> test suite FileSourceTextLinesITCase 5-10 times in IDE:
> {code:java}
> Test
> testContinuousTextFileSourceWithTaskManagerFailover(org.apache.flink.connector.file.src.FileSourceTextLinesITCase)
> failed with:
> java.lang.AssertionError:
> Expected: ["And by opposing end them?--To die,--to sleep,--", "And
> enterprises of great pith and moment,", "And lose the name of action.--Soft
> you now!", "And makes us rather bear those ills we have", "And thus the
> native hue of resolution", "Be all my sins remember'd.", "But that the dread
> of something after death,--", "Devoutly to be wish'd. To die,--to sleep;--",
> "For in that sleep of death what dreams may come,", "For who would bear the
> whips and scorns of time,", "Is sicklied o'er with the pale cast of
> thought;", "Must give us pause: there's the respect", "No more; and by a
> sleep to say we end", "No traveller returns,--puzzles the will,", "Or to take
> arms against a sea of troubles,", "Than fly to others that we know not of?",
> "That flesh is heir to,--'tis a consummation", "That makes calamity of so
> long life;", "That patient merit of the unworthy takes,", "The fair
> Ophelia!--Nymph, in thy orisons", "The heartache, and the thousand natural
> shocks", "The insolence of office, and the spurns", "The oppressor's wrong,
> the proud man's contumely,", "The pangs of despis'd love, the law's delay,",
> "The slings and arrows of outrageous fortune", "The undiscover'd country,
> from whose bourn", "Thus conscience does make cowards of us all;", "To be, or
> not to be,--that is the question:--", "To grunt and sweat under a weary
> life,", "To sleep! perchance to dream:--ay, there's the rub;", "When he
> himself might his quietus make", "When we have shuffled off this mortal
> coil,", "Whether 'tis nobler in the mind to suffer", "With a bare bodkin? who
> would these fardels bear,", "With this regard, their currents turn awry,"]
> but: was ["And by opposing end them?--To die,--to sleep,--", "And
> enterprises of great pith and moment,", "And lose the name of action.--Soft
> you now!", "And makes us rather bear those ills we have", "And thus the
> native hue of resolution", "Be all my sins remember'd.", "But that the dread
> of something after death,--", "Devoutly to be wish'd. To die,--to sleep;--",
> "Devoutly to be wish'd. To die,--to sleep;--", "For in that sleep of death
> what dreams may come,", "For who would bear the whips and scorns of time,",
> "Is sicklied o'er with the pale cast of thought;", "Must give us pause:
> there's the respect", "No more; and by a sleep to say we end", "No more; and
> by a sleep to say we end", "No traveller returns,--puzzles the will,", "Or to
> take arms against a sea of troubles,", "Than fly to others that we know not
> of?", "That flesh is heir to,--'tis a consummation", "That flesh is heir
> to,--'tis a consummation", "That makes calamity of so long life;", "The fair
> Ophelia!--Nymph, in thy orisons", "The heartache, and the thousand natural
> shocks", "The heartache, and the thousand natural shocks", "The slings and
> arrows of outrageous fortune", "The undiscover'd country, from whose bourn",
> "Thus conscience does make cowards of us all;", "To be, or not to be,--that
> is the question:--", "To grunt and sweat under a weary life,", "To sleep!
> perchance to dream:--ay, there's the rub;", "To sleep! perchance to
> dream:--ay, there's the rub;", "When we have shuffled off this mortal coil,",
> "Whether 'tis nobler in the mind to suffer", "With a bare bodkin? who would
> these fardels bear,", "With this regard, their currents turn awry,"]
> at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
> at org.junit.Assert.assertThat(Assert.java:956)
>
[GitHub] [flink] StephanEwen opened a new pull request #14259: [FLINK-20290][runtime] Fix split duplication issues in Sources
StephanEwen opened a new pull request #14259: URL: https://github.com/apache/flink/pull/14259 ## What is the purpose of the change This changes the split duplication problem in the new source framework, as initially observed on the File Source. The issues are independent of the file source and affect all sources implemented against the new framework. This PR combines the changes from - #14256 - Add `resetSubtask()` to `OperatorCoordinator` - #14199 - Add runtime failure tests based on the File Source We need to squash some commits during merging. ## Verifying this change This PR adds significant test coverage via the extension of the `FileTextLinesITCase` with TaskManager and JobManager failures. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): **no** - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: **no** - The serializers: **no** - The runtime per-record code paths (performance sensitive): - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: **yes** - The S3 file system connector: **no** ## Documentation - Does this pull request introduce a new feature? **no** - If yes, how is the feature documented? **not applicableno** This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14258: [FLINK-20356][docs] Updated Mesos documentation.
flinkbot commented on pull request #14258: URL: https://github.com/apache/flink/pull/14258#issuecomment-735468936 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 299cc3011151c3efb544fc5c9891c9418e9019ac (Sun Nov 29 22:36:53 UTC 2020) **Warnings:** * Documentation files were touched, but no `.zh.md` files: Update Chinese documentation or file Jira ticket. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20356) Rework Mesos deployment documentation page
[ https://issues.apache.org/jira/browse/FLINK-20356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-20356: --- Labels: pull-request-available (was: ) > Rework Mesos deployment documentation page > -- > > Key: FLINK-20356 > URL: https://issues.apache.org/jira/browse/FLINK-20356 > Project: Flink > Issue Type: Sub-task > Components: Documentation, Runtime / Coordination >Affects Versions: 1.12.0 >Reporter: Till Rohrmann >Assignee: Matthias >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > Similar to FLINK-20347, we need to rework the Mesos deployment documentation > page. Additionally, we should validate that everything which is stated in the > documentation actually works. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] XComp opened a new pull request #14258: [FLINK-20356][docs] Updated Mesos documentation.
XComp opened a new pull request #14258: URL: https://github.com/apache/flink/pull/14258 First version of the Mesos page. See a screenshot of the page below...  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #10847: [FLINK-15578][connectors/jdbc] implement exactly once JDBC sink
flinkbot edited a comment on pull request #10847: URL: https://github.com/apache/flink/pull/10847#issuecomment-573933799 ## CI report: * b55a17559f8bbd427788152fdc5a2e79e273ad7a Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10299) * 693e48ac1e9cd413f54e7cbc4c46b9ae190ce848 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-20396) Add "OperatorCoordinator.resetSubtask()" to fix order problems of "subtaskFailed()"
[
https://issues.apache.org/jira/browse/FLINK-20396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stephan Ewen updated FLINK-20396:
-
Summary: Add "OperatorCoordinator.resetSubtask()" to fix order problems of
"subtaskFailed()" (was: Replace "OperatorCoordinator.subtaskFailed()" with
"subtaskRestored()")
> Add "OperatorCoordinator.resetSubtask()" to fix order problems of
> "subtaskFailed()"
> ---
>
> Key: FLINK-20396
> URL: https://issues.apache.org/jira/browse/FLINK-20396
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.2
>Reporter: Stephan Ewen
>Assignee: Stephan Ewen
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 1.12.0, 1.11.3
>
>
> There are no strong order guarantees between
> {{OperatorCoordinator.subtaskFailed()}} and
> {{OperatorCoordinator.notifyCheckpointComplete()}}.
> It can happen that a checkpoint completes after the notification for task
> failure is sent:
> - {{OperatorCoordinator.checkpoint()}}
> - {{OperatorCoordinator.subtaskFailed()}}
> - {{OperatorCoordinator.checkpointComplete()}}
> The subtask failure here does not know whether the previous checkpoint
> completed or not. It cannot decide what state the subtask will be in after
> recovery.
> There is no easy fix right now to strictly guarantee the order of the method
> calls, so alternatively we need to provide the necessary information to
> reason about the status of tasks.
> We should replace {{OperatorCoordinator.subtaskFailed(int subtask)}} with
> {{OperatorCoordinator.subtaskRestored(int subtask, long checkpoint)}}. That
> implementations get the explicit checkpoint ID for the subtask recovery, and
> can align that with the IDs of checkpoints that were taken.
> It is still (in rare cases) possible that for a specific checkpoint C,
> {{OperatorCoordinator.subtaskRestored(subtaskIndex, C)) comes before
> {{OperatorCoordinator.checkpointComplete(C)}}.
> h3. Background
> The Checkpointing Procedure is partially asynchronous on the {{JobManager}} /
> {{CheckpointCoordinator}}: After all subtasks acknowledged the checkpoint,
> the finalization (writing out metadata and registering the checkpoint in
> ZooKeeper) happens in an I/O thread, and the checkpoint completes after that.
> This sequence of events can happen:
> - tasks acks checkpoint
> - checkpoint fully acknowledged, finalization starts
> - task fails
> - task failure notification is dispatched
> - checkpoint completes.
> For task failures and checkpoint completion, no order is defined.
> However, for task restore and checkpoint completion, the order is well
> defined: When a task is restored, pending checkpoints are either canceled or
> complete. None can be within finalization. That is currently guaranteed with
> a lock in the {{CheckpointCoordinator}}.
> (An implication of that being that restores can be blocking operations in the
> scheduler, which is not ideal from the perspective of making the scheduler
> async/non-blocking, but it is currently essential for correctness).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-20413) Sources should add splits back in "resetSubtask()", rather than in "subtaskFailed()".
Stephan Ewen created FLINK-20413: Summary: Sources should add splits back in "resetSubtask()", rather than in "subtaskFailed()". Key: FLINK-20413 URL: https://issues.apache.org/jira/browse/FLINK-20413 Project: Flink Issue Type: Bug Components: Runtime / Coordination Reporter: Stephan Ewen Assignee: Stephan Ewen Fix For: 1.12.0 Because "subtaskFailed()" has no strong order guarantees with checkpoint completion, we need to return failed splits in "resetSubtask()" instead. See FLINK-20396 for a detailed explanation of the race condition. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-20290) Duplicated output in FileSource continuous ITCase with TM failover
[
https://issues.apache.org/jira/browse/FLINK-20290?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240375#comment-17240375
]
Stephan Ewen commented on FLINK-20290:
--
Update: A sequence similar to the one described by [~becket_qin] is possible,
where the race is between checkpoint finalization in the Checkpoint
Coordinator, and the subtask failure notifications. See FLINK-20396 for details.
> Duplicated output in FileSource continuous ITCase with TM failover
> --
>
> Key: FLINK-20290
> URL: https://issues.apache.org/jira/browse/FLINK-20290
> Project: Flink
> Issue Type: Bug
> Components: Connectors / FileSystem
>Affects Versions: 1.12.0
>Reporter: Andrey Zagrebin
>Priority: Blocker
> Fix For: 1.12.0
>
> Attachments: collect-debug-jm.png, collect-debug.png, log
>
>
> If FileSourceTextLinesITCase::testContinuousTextFileSource includes TM
> restarts (after failing TM with TestingMiniCluster::terminateTaskExecutor,
> see testContinuousTextFileSourceWithTaskManagerFailover in
> [branch|https://github.com/azagrebin/flink/tree/FLINK-20118-it]) then
> sometimes I observe duplicated lines in the output after running the whole
> test suite FileSourceTextLinesITCase 5-10 times in IDE:
> {code:java}
> Test
> testContinuousTextFileSourceWithTaskManagerFailover(org.apache.flink.connector.file.src.FileSourceTextLinesITCase)
> failed with:
> java.lang.AssertionError:
> Expected: ["And by opposing end them?--To die,--to sleep,--", "And
> enterprises of great pith and moment,", "And lose the name of action.--Soft
> you now!", "And makes us rather bear those ills we have", "And thus the
> native hue of resolution", "Be all my sins remember'd.", "But that the dread
> of something after death,--", "Devoutly to be wish'd. To die,--to sleep;--",
> "For in that sleep of death what dreams may come,", "For who would bear the
> whips and scorns of time,", "Is sicklied o'er with the pale cast of
> thought;", "Must give us pause: there's the respect", "No more; and by a
> sleep to say we end", "No traveller returns,--puzzles the will,", "Or to take
> arms against a sea of troubles,", "Than fly to others that we know not of?",
> "That flesh is heir to,--'tis a consummation", "That makes calamity of so
> long life;", "That patient merit of the unworthy takes,", "The fair
> Ophelia!--Nymph, in thy orisons", "The heartache, and the thousand natural
> shocks", "The insolence of office, and the spurns", "The oppressor's wrong,
> the proud man's contumely,", "The pangs of despis'd love, the law's delay,",
> "The slings and arrows of outrageous fortune", "The undiscover'd country,
> from whose bourn", "Thus conscience does make cowards of us all;", "To be, or
> not to be,--that is the question:--", "To grunt and sweat under a weary
> life,", "To sleep! perchance to dream:--ay, there's the rub;", "When he
> himself might his quietus make", "When we have shuffled off this mortal
> coil,", "Whether 'tis nobler in the mind to suffer", "With a bare bodkin? who
> would these fardels bear,", "With this regard, their currents turn awry,"]
> but: was ["And by opposing end them?--To die,--to sleep,--", "And
> enterprises of great pith and moment,", "And lose the name of action.--Soft
> you now!", "And makes us rather bear those ills we have", "And thus the
> native hue of resolution", "Be all my sins remember'd.", "But that the dread
> of something after death,--", "Devoutly to be wish'd. To die,--to sleep;--",
> "Devoutly to be wish'd. To die,--to sleep;--", "For in that sleep of death
> what dreams may come,", "For who would bear the whips and scorns of time,",
> "Is sicklied o'er with the pale cast of thought;", "Must give us pause:
> there's the respect", "No more; and by a sleep to say we end", "No more; and
> by a sleep to say we end", "No traveller returns,--puzzles the will,", "Or to
> take arms against a sea of troubles,", "Than fly to others that we know not
> of?", "That flesh is heir to,--'tis a consummation", "That flesh is heir
> to,--'tis a consummation", "That makes calamity of so long life;", "The fair
> Ophelia!--Nymph, in thy orisons", "The heartache, and the thousand natural
> shocks", "The heartache, and the thousand natural shocks", "The slings and
> arrows of outrageous fortune", "The undiscover'd country, from whose bourn",
> "Thus conscience does make cowards of us all;", "To be, or not to be,--that
> is the question:--", "To grunt and sweat under a weary life,", "To sleep!
> perchance to dream:--ay, there's the rub;", "To sleep! perchance to
> dream:--ay, there's the rub;", "When we have shuffled off this mortal coil,",
> "Whether 'tis nobler in the mind to suffer", "With a bare bodkin? who would
> these fardels bear,",
[GitHub] [flink] StephanEwen commented on pull request #14143: [FLINK-20223] The RecreateOnResetOperatorCoordinator and SourceCoordi…
StephanEwen commented on pull request #14143: URL: https://github.com/apache/flink/pull/14143#issuecomment-735463122 Merged as part of this change 40bbc174411b1502842ed3cdb9c421d51aa5fc0d This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] StephanEwen closed pull request #14143: [FLINK-20223] The RecreateOnResetOperatorCoordinator and SourceCoordi…
StephanEwen closed pull request #14143: URL: https://github.com/apache/flink/pull/14143 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #10847: [FLINK-15578][connectors/jdbc] implement exactly once JDBC sink
flinkbot edited a comment on pull request #10847: URL: https://github.com/apache/flink/pull/10847#issuecomment-573933799 ## CI report: * Unknown: [CANCELED](TBD) * b55a17559f8bbd427788152fdc5a2e79e273ad7a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=10299) * 693e48ac1e9cd413f54e7cbc4c46b9ae190ce848 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #10847: [FLINK-15578][connectors/jdbc] implement exactly once JDBC sink
flinkbot edited a comment on pull request #10847: URL: https://github.com/apache/flink/pull/10847#issuecomment-573933799 ## CI report: * Unknown: [CANCELED](TBD) * b55a17559f8bbd427788152fdc5a2e79e273ad7a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20412) Collect Result Fetching occasionally fails after a JobManager Failover
[
https://issues.apache.org/jira/browse/FLINK-20412?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17240369#comment-17240369
]
Stephan Ewen commented on FLINK-20412:
--
[~TsReaper] for your attention.
> Collect Result Fetching occasionally fails after a JobManager Failover
> --
>
> Key: FLINK-20412
> URL: https://issues.apache.org/jira/browse/FLINK-20412
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.12.0
>Reporter: Stephan Ewen
>Priority: Critical
>
> The encountered exception is as blow.
>
> The issue can be reproduced by running a test with JobManager failover in a
> tight loop, for example the FileTextLinesITCase from this PR:
> [https://github.com/apache/flink/pull/14199]
>
> {code:java}
> 15335 [main] WARN
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher - An
> exception occurs when fetching query results
> java.util.concurrent.ExecutionException: java.lang.NullPointerException
> at
> java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
> ~[?:?]
> at
> java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) ~[?:?]
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sendRequest(CollectResultFetcher.java:163)
> ~[classes/:?]
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:134)
> [classes/:?]
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:103)
> [classes/:?]
> at
> org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:77)
> [classes/:?]
> at
> org.apache.flink.streaming.api.datastream.DataStreamUtils.collectRecordsFromUnboundedStream(DataStreamUtils.java:142)
> [classes/:?]
> at
> org.apache.flink.connector.file.src.FileSourceTextLinesITCase.testContinuousTextFileSource(FileSourceTextLinesITCase.java:272)
> [test-classes/:?]
> at
> org.apache.flink.connector.file.src.FileSourceTextLinesITCase.testContinuousTextFileSourceWithJobManagerFailover(FileSourceTextLinesITCase.java:228)
> [test-classes/:?]
> at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> ~[?:?]
> at
> jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> ~[?:?]
> at
> jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ~[?:?]
> at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> [junit-4.12.jar:4.12]
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> [junit-4.12.jar:4.12]
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> [junit-4.12.jar:4.12]
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> [junit-4.12.jar:4.12]
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> [junit-4.12.jar:4.12]
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> [junit-4.12.jar:4.12]
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> [junit-4.12.jar:4.12]
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> [junit-4.12.jar:4.12]
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> [junit-4.12.jar:4.12]
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> [junit-4.12.jar:4.12]
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> [junit-4.12.jar:4.12]
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> [junit-4.12.jar:4.12]
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> [junit-4.12.jar:4.12]
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> [junit-4.12.jar:4.12]
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> [junit-4.12.jar:4.12]
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> [junit-4.12.jar:4.12]
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> [junit-4.12.jar:4.12]
> at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> [junit-4.12.jar:4.12]
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> [junit-4.12.jar:4.12]
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
>
[jira] [Created] (FLINK-20412) Collect Result Fetching occasionally fails after a JobManager Failover
Stephan Ewen created FLINK-20412:
Summary: Collect Result Fetching occasionally fails after a
JobManager Failover
Key: FLINK-20412
URL: https://issues.apache.org/jira/browse/FLINK-20412
Project: Flink
Issue Type: Bug
Components: Runtime / Coordination
Affects Versions: 1.12.0
Reporter: Stephan Ewen
The encountered exception is as blow.
The issue can be reproduced by running a test with JobManager failover in a
tight loop, for example the FileTextLinesITCase from this PR:
[https://github.com/apache/flink/pull/14199]
{code:java}
15335 [main] WARN
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher - An
exception occurs when fetching query results
java.util.concurrent.ExecutionException: java.lang.NullPointerException
at
java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
~[?:?]
at
java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1999) ~[?:?]
at
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.sendRequest(CollectResultFetcher.java:163)
~[classes/:?]
at
org.apache.flink.streaming.api.operators.collect.CollectResultFetcher.next(CollectResultFetcher.java:134)
[classes/:?]
at
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.nextResultFromFetcher(CollectResultIterator.java:103)
[classes/:?]
at
org.apache.flink.streaming.api.operators.collect.CollectResultIterator.hasNext(CollectResultIterator.java:77)
[classes/:?]
at
org.apache.flink.streaming.api.datastream.DataStreamUtils.collectRecordsFromUnboundedStream(DataStreamUtils.java:142)
[classes/:?]
at
org.apache.flink.connector.file.src.FileSourceTextLinesITCase.testContinuousTextFileSource(FileSourceTextLinesITCase.java:272)
[test-classes/:?]
at
org.apache.flink.connector.file.src.FileSourceTextLinesITCase.testContinuousTextFileSourceWithJobManagerFailover(FileSourceTextLinesITCase.java:228)
[test-classes/:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:?]
at
jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:?]
at
jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
[junit-4.12.jar:4.12]
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
[junit-4.12.jar:4.12]
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
[junit-4.12.jar:4.12]
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
[junit-4.12.jar:4.12]
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
[junit-4.12.jar:4.12]
at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
[junit-4.12.jar:4.12]
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
[junit-4.12.jar:4.12]
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
[junit-4.12.jar:4.12]
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
[junit-4.12.jar:4.12]
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
[junit-4.12.jar:4.12]
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
[junit-4.12.jar:4.12]
at
org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
[junit-4.12.jar:4.12]
at org.junit.rules.RunRules.evaluate(RunRules.java:20)
[junit-4.12.jar:4.12]
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
[junit-4.12.jar:4.12]
at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
[junit-4.12.jar:4.12]
at
com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
[junit-rt.jar:?]
at
com.intellij.rt.execution.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:67)
[junit-rt.jar:?]
at
[jira] [Closed] (FLINK-20406) Return the Checkpoint ID of the restored Checkpoint in CheckpointCoordinator.restoreLatestCheckpointedStateToSubtasks()
[ https://issues.apache.org/jira/browse/FLINK-20406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen closed FLINK-20406. > Return the Checkpoint ID of the restored Checkpoint in > CheckpointCoordinator.restoreLatestCheckpointedStateToSubtasks() > --- > > Key: FLINK-20406 > URL: https://issues.apache.org/jira/browse/FLINK-20406 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing >Reporter: Stephan Ewen >Assignee: Stephan Ewen >Priority: Major > Fix For: 1.12.0 > > > To allow the scheduler to notify Operator Coordinators of subtask restores > (local failover), we need to know which checkpoint ID was restored. > This change does not adjust the other restore methods of the Checkpoint > Coordinator, because the fact that the Scheduler needs to be involved in the > subtask restore notification at all is only due to a shortcoming of the > Checkpoint Coordinator: The CC is not aware of subtask restores, it always > restores all subtasks and relies on the fact that assigning state to a > running execution attempt has no effect. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (FLINK-20406) Return the Checkpoint ID of the restored Checkpoint in CheckpointCoordinator.restoreLatestCheckpointedStateToSubtasks()
[ https://issues.apache.org/jira/browse/FLINK-20406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephan Ewen resolved FLINK-20406. -- Resolution: Fixed Fixed in - 1.13.0 (master) via 856d4ea3cabce335f25ed163df8ae8eba9680890 - 1.12.0 (release-1.12) via 519c0a6c812c71173939735572fd6865e304e8e4 > Return the Checkpoint ID of the restored Checkpoint in > CheckpointCoordinator.restoreLatestCheckpointedStateToSubtasks() > --- > > Key: FLINK-20406 > URL: https://issues.apache.org/jira/browse/FLINK-20406 > Project: Flink > Issue Type: Sub-task > Components: Runtime / Checkpointing >Reporter: Stephan Ewen >Assignee: Stephan Ewen >Priority: Major > Fix For: 1.12.0 > > > To allow the scheduler to notify Operator Coordinators of subtask restores > (local failover), we need to know which checkpoint ID was restored. > This change does not adjust the other restore methods of the Checkpoint > Coordinator, because the fact that the Scheduler needs to be involved in the > subtask restore notification at all is only due to a shortcoming of the > Checkpoint Coordinator: The CC is not aware of subtask restores, it always > restores all subtasks and relies on the fact that assigning state to a > running execution attempt has no effect. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] rkhachatryan commented on pull request #10847: [FLINK-15578][connectors/jdbc] implement exactly once JDBC sink
rkhachatryan commented on pull request #10847: URL: https://github.com/apache/flink/pull/10847#issuecomment-735452180 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
