[jira] [Updated] (FLINK-18602) Support specific offset for topic list for kafka connector in table api
[
https://issues.apache.org/jira/browse/FLINK-18602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shengkai Fang updated FLINK-18602:
--
Description:
During FLINK-18449, we decide to support topic discovery for kafka connector in
table api. However, we can only use format
{{'partition:0,offset:42;partition:1,offset:300'}} to specify the offset for
single topic. I think the better format is
{{'topic:topic-1,partition:0,offset:42;topic:topic-2,partition:1,offset:300'}}
in topic discovery situation.
was:
During FLINK-18449, we decide to support topic discovery for kafka connector in
table api. However, we can only use format
{{'partition:0,offset:42;partition:1,offset:300'}} to specify the offset for
single topic. I think the better format is {{'topic:topic-1,
partition:0,offset:42;topic:topic-2,partition:1,offset:300'}} in topic
discovery situation.
> Support specific offset for topic list for kafka connector in table api
> ---
>
> Key: FLINK-18602
> URL: https://issues.apache.org/jira/browse/FLINK-18602
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.11.0
>Reporter: Shengkai Fang
>Priority: Major
> Fix For: 1.12.0
>
>
> During FLINK-18449, we decide to support topic discovery for kafka connector
> in table api. However, we can only use format
> {{'partition:0,offset:42;partition:1,offset:300'}} to specify the offset for
> single topic. I think the better format is
> {{'topic:topic-1,partition:0,offset:42;topic:topic-2,partition:1,offset:300'}}
> in topic discovery situation.
>
>
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] xiaolong-sn commented on a change in pull request #12881: [FLINK-18512][kinesis] Introducing RecordPublisher. Refactor ShardConsumer to use PollingRecordPublisher
xiaolong-sn commented on a change in pull request #12881:
URL: https://github.com/apache/flink/pull/12881#discussion_r454251135
##
File path:
flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/proxy/KinesisProxyMarker.java
##
@@ -0,0 +1,28 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.streaming.connectors.kinesis.proxy;
+
+import org.apache.flink.annotation.Internal;
+
+/**
+ * A marker interface for generic Kinesis Proxy.
+ */
+@Internal
Review comment:
What does this interface represent besides the existed
KinesisProxyInterface?
##
File path:
flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/ShardConsumer.java
##
@@ -379,49 +205,18 @@ private void
deserializeRecordForCollectionAndUpdateState(UserRecord record)
}
/**
-* Calls {@link KinesisProxyInterface#getRecords(String, int)}, while
also handling unexpected
-* AWS {@link ExpiredIteratorException}s to assure that we get results
and don't just fail on
-* such occasions. The returned shard iterator within the successful
{@link GetRecordsResult} should
-* be used for the next call to this method.
-*
-* Note: it is important that this method is not called again before
all the records from the last result have been
-* fully collected with {@link
ShardConsumer#deserializeRecordForCollectionAndUpdateState(UserRecord)},
otherwise
-* {@link ShardConsumer#lastSequenceNum} may refer to a sub-record in
the middle of an aggregated record, leading to
-* incorrect shard iteration if the iterator had to be refreshed.
+* Filters out aggregated records that have previously been processed.
+* This method is to support restarting from a partially consumed
aggregated sequence number.
*
-* @param shardItr shard iterator to use
-* @param maxNumberOfRecords the maximum number of records to fetch for
this getRecords attempt
-* @return get records result
-* @throws InterruptedException
+* @param record the record to filter
+* @return {@code true} if the record should be retained
*/
- private GetRecordsResult getRecords(String shardItr, int
maxNumberOfRecords) throws Exception {
- GetRecordsResult getRecordsResult = null;
- while (getRecordsResult == null) {
- try {
- getRecordsResult = kinesis.getRecords(shardItr,
maxNumberOfRecords);
-
- // Update millis behind latest so it gets
reported by the millisBehindLatest gauge
- Long millisBehindLatest =
getRecordsResult.getMillisBehindLatest();
- if (millisBehindLatest != null) {
-
shardMetricsReporter.setMillisBehindLatest(millisBehindLatest);
- }
- } catch (ExpiredIteratorException eiEx) {
- LOG.warn("Encountered an unexpected expired
iterator {} for shard {};" +
- " refreshing the iterator ...",
shardItr, subscribedShard);
-
- shardItr = getShardIterator(lastSequenceNum);
-
- // sleep for the fetch interval before the next
getRecords attempt with the refreshed iterator
- if (fetchIntervalMillis != 0) {
- Thread.sleep(fetchIntervalMillis);
- }
- }
+ private boolean filterDeaggregatedRecord(final UserRecord record) {
+ if (lastSequenceNum.isAggregated()) {
Review comment:
What does this function means?
##
File path:
flink-connectors/flink-connector-kinesis/src/main/java/org/apache/flink/streaming/connectors/kinesis/internals/publisher/polling/PollingRecordPublisher.java
##
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under
[jira] [Closed] (FLINK-18603) SQL fails with java.lang.IllegalStateException: No operators defined in streaming topology
[
https://issues.apache.org/jira/browse/FLINK-18603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexander Filipchik closed FLINK-18603.
---
Resolution: Not A Problem
Issues was in custom format.
> SQL fails with java.lang.IllegalStateException: No operators defined in
> streaming topology
> --
>
> Key: FLINK-18603
> URL: https://issues.apache.org/jira/browse/FLINK-18603
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.11.0
>Reporter: Alexander Filipchik
>Priority: Major
>
> Hi, was playing with 1.11 and found that code that worked in 1.10.1 fails in
> 1.11.0 with :
> {code:java}
> Exception in thread "main" java.lang.IllegalStateException: No operators
> defined in streaming topology. Cannot generate StreamGraph.
> at
> org.apache.flink.table.planner.utils.ExecutorUtils.generateStreamGraph(ExecutorUtils.java:47)
> at
> org.apache.flink.table.planner.delegation.StreamExecutor.createPipeline(StreamExecutor.java:47)
> at
> org.apache.flink.table.api.internal.TableEnvironmentImpl.execute(TableEnvironmentImpl.java:1197)
> at com.css.flink.avro.confluent.table.SqlTest.main(SqlTest.java:53)
> {code}
> code example:
> {code:java}
> StreamExecutionEnvironment bsEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> EnvironmentSettings bsSettings =
>
> EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
> StreamTableEnvironment tEnv = StreamTableEnvironment.create(bsEnv,
> bsSettings);
> String createTable =
> String.format(
> "create table EnrichedOrders ("
> + "name VARCHAR,"
> + "proctime AS PROCTIME()"
> + ") with ("
> + " 'connector.type' = 'kafka',"
> + " 'connector.version' = 'universal',"
> + " 'connector.property-version' = '1',"
> + " 'connector.topic' = '%s',"
> + " 'connector.properties.bootstrap.servers' = '%s',"
> + " 'connector.properties.group.id' = '%s',"
> + " 'connector.startup-mode' = 'earliest-offset',"
> + " 'update-mode' = 'append',"
> + " 'format.type' = 'confluent-avro',"
> + " 'format.schema-registry' = '%s'"
> + ")",
> "avro",
> "broker",
> "testSqlLocal",
> "registry");
> tEnv.executeSql(createTable);
> tEnv.toAppendStream(
> tEnv.sqlQuery(
> "select name, sum(*) "
> + "from EnrichedOrders "
> + "GROUP BY TUMBLE(proctime, INTERVAL '1' MINUTE), name"),
> Row.class)
> .print();
> tEnv.execute("testSql");
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18582) Add title anchor link for file event_driven.zh.md
[
https://issues.apache.org/jira/browse/FLINK-18582?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157881#comment-17157881
]
Congxian Qiu(klion26) commented on FLINK-18582:
---
[~temper] thanks for creating this ticket. could you please share the pages
which ref this page. If currently there is no pages ref to
{{event_driven.zh.md}} maybe we could do this late. what do you think?
>From my side, we always encourage to add title anchor links, but I'm sure
>whether will we accept the change only add the title anchor(if currently no
>page refs to it)
cc [~jark]
> Add title anchor link for file event_driven.zh.md
> -
>
> Key: FLINK-18582
> URL: https://issues.apache.org/jira/browse/FLINK-18582
> Project: Flink
> Issue Type: Improvement
> Components: Documentation, Quickstarts
>Reporter: Herman, Li
>Priority: Major
>
> Translated files should add anchor link for sub-titles.
> Otherwise, links from other documentations may not able to point to this
> specified title. Such as file event_driven.zh.md with title: Side Outputs.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12903: [FLINK-18492][python] Extract the implementation logic of Beam in Operations
flinkbot edited a comment on pull request #12903: URL: https://github.com/apache/flink/pull/12903#issuecomment-658519879 ## CI report: * aa4bb627890b6e46ef36f44528a24aa304bb7064 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4521) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12899: [FLINK-16699][k8s] Support accessing secured services via K8s secrets
flinkbot edited a comment on pull request #12899: URL: https://github.com/apache/flink/pull/12899#issuecomment-658162074 ## CI report: * 77f95464c6036283d77d6e81e2bd21ab27f29d54 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4506) * 2f0285b88d653612277714e12e96d48e2f82fc51 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4522) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-18603) SQL fails with java.lang.IllegalStateException: No operators defined in streaming topology
Alexander Filipchik created FLINK-18603:
---
Summary: SQL fails with java.lang.IllegalStateException: No
operators defined in streaming topology
Key: FLINK-18603
URL: https://issues.apache.org/jira/browse/FLINK-18603
Project: Flink
Issue Type: Bug
Components: Table SQL / Runtime
Affects Versions: 1.11.0
Reporter: Alexander Filipchik
Hi, was playing with 1.11 and found that code that worked in 1.10.1 fails in
1.11.0 with :
{code:java}
Exception in thread "main" java.lang.IllegalStateException: No operators
defined in streaming topology. Cannot generate StreamGraph.
at
org.apache.flink.table.planner.utils.ExecutorUtils.generateStreamGraph(ExecutorUtils.java:47)
at
org.apache.flink.table.planner.delegation.StreamExecutor.createPipeline(StreamExecutor.java:47)
at
org.apache.flink.table.api.internal.TableEnvironmentImpl.execute(TableEnvironmentImpl.java:1197)
at com.css.flink.avro.confluent.table.SqlTest.main(SqlTest.java:53)
{code}
code example:
{code:java}
StreamExecutionEnvironment bsEnv =
StreamExecutionEnvironment.getExecutionEnvironment();
EnvironmentSettings bsSettings =
EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build();
StreamTableEnvironment tEnv = StreamTableEnvironment.create(bsEnv, bsSettings);
String createTable =
String.format(
"create table EnrichedOrders ("
+ "name VARCHAR,"
+ "proctime AS PROCTIME()"
+ ") with ("
+ " 'connector.type' = 'kafka',"
+ " 'connector.version' = 'universal',"
+ " 'connector.property-version' = '1',"
+ " 'connector.topic' = '%s',"
+ " 'connector.properties.bootstrap.servers' = '%s',"
+ " 'connector.properties.group.id' = '%s',"
+ " 'connector.startup-mode' = 'earliest-offset',"
+ " 'update-mode' = 'append',"
+ " 'format.type' = 'confluent-avro',"
+ " 'format.schema-registry' = '%s'"
+ ")",
"avro",
"broker",
"testSqlLocal",
"registry");
tEnv.executeSql(createTable);
tEnv.toAppendStream(
tEnv.sqlQuery(
"select name, sum(*) "
+ "from EnrichedOrders "
+ "GROUP BY TUMBLE(proctime, INTERVAL '1' MINUTE), name"),
Row.class)
.print();
tEnv.execute("testSql");
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-18594) The link is broken in kafka doc
[
https://issues.apache.org/jira/browse/FLINK-18594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jingsong Lee closed FLINK-18594.
Resolution: Fixed
master: bfdb3c8339347ad59ef5debb71fb171ffa87d4c8
> The link is broken in kafka doc
> ---
>
> Key: FLINK-18594
> URL: https://issues.apache.org/jira/browse/FLINK-18594
> Project: Flink
> Issue Type: Bug
> Components: Documentation
>Affects Versions: 1.12.0
>Reporter: Dian Fu
>Assignee: Leonard Xu
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.12.0
>
>
> https://ci.apache.org/builders/flink-docs-master/builds/1897/steps/Build%20docs/logs/stdio
> {code}
> Liquid Exception: Could not find document
> 'dev/stream/state/checkpointing.md' in tag 'link'. Make sure the document
> exists and the path is correct. in dev/table/connectors/kafka.zh.md
> Could not find document 'dev/stream/state/checkpointing.md' in tag 'link'.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] JingsongLi merged pull request #12897: [FLINK-18594][doc] The link is broken in kafka doc
JingsongLi merged pull request #12897: URL: https://github.com/apache/flink/pull/12897 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18539) StreamExecutionEnvironment#addSource(SourceFunction, TypeInformation) doesn't use the user defined type information
[
https://issues.apache.org/jira/browse/FLINK-18539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-18539:
Issue Type: Bug (was: Task)
> StreamExecutionEnvironment#addSource(SourceFunction, TypeInformation) doesn't
> use the user defined type information
> ---
>
> Key: FLINK-18539
> URL: https://issues.apache.org/jira/browse/FLINK-18539
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.11.0
>Reporter: Jark Wu
>Assignee: Jark Wu
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.10.2, 1.12.0, 1.11.1
>
>
> {code:java}
> class MySource implements SourceFunction, ResultTypeQueryable {
> TypeInformation getProducedType() {
>return TypeExtractor.createTypeInfo(SourceFunction.class, this.getClass(),
> 0, null, null);
> }
> }
> DataStream ds = tEnv.addSource(new MySource(), Types.ROW(Types.STRING))
> {code}
> The returned {{TypeInformation}} of {{MySource}} is {{GenericTypeInfo}}, not
> the user given {{RowTypeInfo}}.
> It seems that {{StreamExecutionEnvironment#getTypeInfo}} doesn't use the user
> given {{typeInfo}} in the highest priority.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18593) Hive bundle jar URLs are broken
[ https://issues.apache.org/jira/browse/FLINK-18593?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-18593: Issue Type: Improvement (was: New Feature) > Hive bundle jar URLs are broken > --- > > Key: FLINK-18593 > URL: https://issues.apache.org/jira/browse/FLINK-18593 > Project: Flink > Issue Type: Improvement > Components: Connectors / Hive, Documentation >Affects Versions: 1.11.0 >Reporter: Jingsong Lee >Assignee: Jingsong Lee >Priority: Major > Labels: pull-request-available > Fix For: 1.11.1 > > > we should use [https://repo.maven.apache.org/maven2/] instead -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18602) Support specific offset for topic list for kafka connector in table api
Shengkai Fang created FLINK-18602:
-
Summary: Support specific offset for topic list for kafka
connector in table api
Key: FLINK-18602
URL: https://issues.apache.org/jira/browse/FLINK-18602
Project: Flink
Issue Type: Improvement
Components: Connectors / Kafka, Table SQL / API
Affects Versions: 1.11.0
Reporter: Shengkai Fang
Fix For: 1.12.0
During FLINK-18449, we decide to support topic discovery for kafka connector in
table api. However, we can only use format
{{'partition:0,offset:42;partition:1,offset:300'}} to specify the offset for
single topic. I think the better format is {{'topic:topic-1,
partition:0,offset:42;topic:topic-2,partition:1,offset:300'}} in topic
discovery situation.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12902: [FLINK-18415][python] Support TableResult#collect in the Python Table…
flinkbot edited a comment on pull request #12902: URL: https://github.com/apache/flink/pull/12902#issuecomment-658509076 ## CI report: * 802dfd4a0a2fb3e1b6e9a31fbc850c18f00c5c2b Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4518) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12899: [FLINK-16699][k8s] Support accessing secured services via K8s secrets
flinkbot edited a comment on pull request #12899: URL: https://github.com/apache/flink/pull/12899#issuecomment-658162074 ## CI report: * 77f95464c6036283d77d6e81e2bd21ab27f29d54 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4506) * 2f0285b88d653612277714e12e96d48e2f82fc51 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12866: [FLINK-17425][blink-planner] supportsFilterPushDown rule in DynamicSource.
flinkbot edited a comment on pull request #12866: URL: https://github.com/apache/flink/pull/12866#issuecomment-656576219 ## CI report: * 5eb302884e390b6099859abcd7955e7f1c4e034e Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4517) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17726) Scheduler should take care of tasks directly canceled by TaskManager
[ https://issues.apache.org/jira/browse/FLINK-17726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-17726: Fix Version/s: (was: 1.11.1) 1.11.2 > Scheduler should take care of tasks directly canceled by TaskManager > > > Key: FLINK-17726 > URL: https://issues.apache.org/jira/browse/FLINK-17726 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination, Runtime / Task >Affects Versions: 1.11.0, 1.12.0 >Reporter: Zhu Zhu >Assignee: Nicholas Jiang >Priority: Critical > Fix For: 1.12.0, 1.11.2 > > > JobManager will not trigger failure handling when receiving CANCELED task > update. > This is because CANCELED tasks are usually caused by another FAILED task. > These CANCELED tasks will be restarted by the failover process triggered > FAILED task. > However, if a task is directly CANCELED by TaskManager due to its own runtime > issue, the task will not be recovered by JM and thus the job would hang. > This is a potential issue and we should avoid it. > A possible solution is to let JobManager treat tasks transitioning to > CANCELED from all states except from CANCELING as failed tasks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17075) Add task status reconciliation between TM and JM
[
https://issues.apache.org/jira/browse/FLINK-17075?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-17075:
Fix Version/s: (was: 1.11.1)
1.11.2
> Add task status reconciliation between TM and JM
>
>
> Key: FLINK-17075
> URL: https://issues.apache.org/jira/browse/FLINK-17075
> Project: Flink
> Issue Type: Improvement
> Components: Runtime / Coordination
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Till Rohrmann
>Assignee: Chesnay Schepler
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.10.2, 1.12.0, 1.11.2
>
>
> In order to harden the TM and JM communication I suggest to let the
> {{TaskExecutor}} send the task statuses back to the {{JobMaster}} as part of
> the heartbeat payload (similar to FLINK-11059). This would allow to reconcile
> the states of both components in case that a status update message was lost
> as described by a user on the ML.
> https://lists.apache.org/thread.html/ra9ed70866381f0ef0f4779633346722ccab3dc0d6dbacce04080b74e%40%3Cuser.flink.apache.org%3E
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18479) can not commit partition when set partition time
[
https://issues.apache.org/jira/browse/FLINK-18479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-18479:
Fix Version/s: (was: 1.11.1)
1.11.2
> can not commit partition when set partition time
>
>
> Key: FLINK-18479
> URL: https://issues.apache.org/jira/browse/FLINK-18479
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.11.0
>Reporter: Jun Zhang
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.2
>
>
> when we write streaming data to filesystem, and select the 'partition time' ,
> we can not commit the partition when write finished.
>
> {code:java}
> LocalDateTime partTime = extractor.extract(
> partitionKeys, extractPartitionValues(new Path(partition)));
> if (watermark > toMills(partTime) + commitDelay) {
>needCommit.add(partition);
>iter.remove();
> }
> {code}
> when we set a not UTC zone, and submit the partition, the method 'toMills'
> will get the UTC mills ，for example ，in UTC/GMT+08:00 ，the watermark will
> less than the toMills , so we can not commit the partition forever.
> if we use a local time , not utc , it will be ok in UTC zone and other zone.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18549) flink 1.11 can not commit partition automatically
[
https://issues.apache.org/jira/browse/FLINK-18549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-18549:
Fix Version/s: (was: 1.11.1)
1.11.2
> flink 1.11 can not commit partition automatically
> -
>
> Key: FLINK-18549
> URL: https://issues.apache.org/jira/browse/FLINK-18549
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / Runtime
>Affects Versions: 1.11.0
>Reporter: Jun Zhang
>Priority: Major
> Fix For: 1.11.2
>
>
> I use the sql of flink 1.11, read from kafka and writing to hdfs, I found
> that the partition cannot be submitted automatically. This is my complete
> code。
> My checkpoint interval is 10s. I think it should be normal that there will be
> _SUCCESS file under the partition of hdfs every 10s, but in fact there is no
>
> {code:java}
>StreamExecutionEnvironment bsEnv =
> StreamExecutionEnvironment.getExecutionEnvironment();
> bsEnv.enableCheckpointing(1);
> bsEnv.setParallelism(1);
> StreamTableEnvironment tEnv = StreamTableEnvironment.create(bsEnv);
> String sqlSource = "CREATE TABLE source_kafka (\n" +
> "appName STRING,\n" +
> "appVersion STRING,\n" +
> "uploadTime STRING\n" +
> ") WITH (\n" +
> " 'connector.type' = 'kafka', \n" +
> " 'connector.version' = '0.10',\n" +
> " 'connector.topic' = 'test_topic',\n" +
> " 'connector.properties.zookeeper.connect' =
> 'localhost:2181',\n" +
> " 'connector.properties.bootstrap.servers' =
> 'localhost:9092',\n" +
> " 'connector.properties.group.id' = 'testGroup',\n"
> +
> " 'format.type'='json',\n" +
> " 'update-mode' = 'append' )";
> tEnv.executeSql(sqlSource);
> String sql = "CREATE TABLE fs_table (\n" +
>"appName STRING,\n" +
>"appVersion STRING,\n" +
>"uploadTime STRING,\n" +
>" dt STRING," +
>" h string" +
>") PARTITIONED BY (dt,h) WITH (\n" +
>" 'connector'='filesystem',\n" +
> " 'path'='hdfs://localhost/tmp/',\n" +
> " 'sink.partition-commit.policy.kind' = 'success-file',
> " +
> " 'format'='orc'\n" +
> ")";
> tEnv.executeSql(sql);
> String insertSql = "insert into fs_table SELECT appName
> ,appVersion,uploadTime, " +
> " DATE_FORMAT(LOCALTIMESTAMP, '-MM-dd'),
> DATE_FORMAT(LOCALTIMESTAMP, 'HH') FROM source_kafka";
> tEnv.executeSql(insertSql);
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18376) java.lang.ArrayIndexOutOfBoundsException in RetractableTopNFunction
[ https://issues.apache.org/jira/browse/FLINK-18376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-18376: Fix Version/s: (was: 1.11.1) 1.11.2 > java.lang.ArrayIndexOutOfBoundsException in RetractableTopNFunction > --- > > Key: FLINK-18376 > URL: https://issues.apache.org/jira/browse/FLINK-18376 > Project: Flink > Issue Type: Bug > Components: Table SQL / Planner >Affects Versions: 1.10.0 >Reporter: LakeShen >Assignee: LakeShen >Priority: Major > Labels: pull-request-available > Fix For: 1.11.2 > > > java.lang.ArrayIndexOutOfBoundsException: -1 > at java.util.ArrayList.elementData(ArrayList.java:422) > at java.util.ArrayList.get(ArrayList.java:435) > at > org.apache.flink.table.runtime.operators.rank.RetractableTopNFunction.retractRecordWithoutRowNumber(RetractableTopNFunction.java:392) > at > org.apache.flink.table.runtime.operators.rank.RetractableTopNFunction.processElement(RetractableTopNFunction.java:160) > at > org.apache.flink.table.runtime.operators.rank.RetractableTopNFunction.processElement(RetractableTopNFunction.java:54) > at > org.apache.flink.streaming.api.operators.KeyedProcessOperator.processElement(KeyedProcessOperator.java:85) > at > org.apache.flink.streaming.runtime.tasks.OneInputStreamTask$StreamTaskNetworkOutput.emitRecord(OneInputStreamTask.java:173) > at > org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.processElement(StreamTaskNetworkInput.java:151) > at > org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:128) > at > org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:69) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.processInput(StreamTask.java:311) > at > org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:187) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:487) > at > org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:470) > at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:707) > at org.apache.flink.runtime.taskmanager.Task.run(Task.java:532) > at java.lang.Thread.run(Thread.java:748) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18206) The timestamp is displayed incorrectly
[ https://issues.apache.org/jira/browse/FLINK-18206?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-18206: Fix Version/s: (was: 1.11.1) 1.11.2 > The timestamp is displayed incorrectly > --- > > Key: FLINK-18206 > URL: https://issues.apache.org/jira/browse/FLINK-18206 > Project: Flink > Issue Type: Bug > Components: Table SQL / Client >Affects Versions: 1.10.1 >Reporter: JasonLee >Priority: Minor > Fix For: 1.11.2 > > > I am using the latest Flink version. When I run a scrolling window SQL in SQL > client, the time stamp of the printed result will not be correct > > The results are as follows > > + jason 49 2 2020-06-09T07:59:40 2020-06-09T07:59:45 > + jason 50 2 2020-06-09T07:59:45 2020-06-09T07:59:50 > + jason 50 2 2020-06-09T07:59:50 2020-06-09T07:59:55 > + jason 50 2 2020-06-09T07:59:55 2020-06-09T08:00 > + jason 49 2 2020-06-09T08:00 2020-06-09T08:00:05 > + jason 50 2 2020-06-09T08:00:05 2020-06-09T08:00:10 > + jason 50 2 2020-06-09T08:00:10 2020-06-09T08:00:15 > + jason 50 2 2020-06-09T08:00:15 2020-06-09T08:00:20 > + jason 49 2 2020-06-09T08:00:20 2020-06-09T08:00:25 > + jason 50 2 2020-06-09T08:00:25 2020-06-09T08:00:30 > + jason 50 2 2020-06-09T08:00:30 2020-06-09T08:00:35 > + jason 49 2 2020-06-09T08:00:35 2020-06-09T08:00:40 > + jason 51 2 2020-06-09T08:00:40 2020-06-09T08:00:45 > + jason 50 2 2020-06-09T08:00:45 2020-06-09T08:00:50 > + jason 49 2 2020-06-09T08:00:50 2020-06-09T08:00:55 > + jason 50 2 2020-06-09T08:00:55 2020-06-09T08:01 > + jason 50 2 2020-06-09T08:01 2020-06-09T08:01:05 > + jason 51 2 2020-06-09T08:01:05 2020-06-09T08:01:10 > + jason 49 2 2020-06-09T08:01:10 2020-06-09T08:01:15 > + jason 46 2 2020-06-09T08:01:15 2020-06-09T08:01:20 > + jason 54 2 2020-06-09T08:01:20 2020-06-09T08:01:25 > + jason 50 2 2020-06-09T08:01:25 2020-06-09T08:01:30 > + jason 49 2 2020-06-09T08:01:30 2020-06-09T08:01:35 > + jason 50 2 2020-06-09T08:01:35 2020-06-09T08:01:40 > + jason 50 2 2020-06-09T08:01:40 2020-06-09T08:01:45 > + jason 50 2 2020-06-09T08:01:45 2020-06-09T08:01:50 > + jason 49 2 2020-06-09T08:01:50 2020-06-09T08:01:55 > + jason 50 2 2020-06-09T08:01:55 2020-06-09T08:02 > + jason 49 2 2020-06-09T08:02 2020-06-09T08:02:05 > + jason 51 2 2020-06-09T08:02:05 2020-06-09T08:02:10 > + jason 49 2 2020-06-09T08:02:10 2020-06-09T08:02:15 > + jason 50 2 2020-06-09T08:02:15 2020-06-09T08:02:20 > + jason 50 2 2020-06-09T08:02:20 2020-06-09T08:02:25 > + jason 50 2 2020-06-09T08:02:25 2020-06-09T08:02:30 > + jason 50 2 2020-06-09T08:02:30 2020-06-09T08:02:35 > + jason 50 2 2020-06-09T08:02:35 2020-06-09T08:02:40 > + jason 49 2 2020-06-09T08:02:40 2020-06-09T08:02:45 > + jason 50 2 2020-06-09T08:02:45 2020-06-09T08:02:50 > + jason 50 2 2020-06-09T08:02:50 2020-06-09T08:02:55 > + jason 50 2 2020-06-09T08:02:55 2020-06-09T08:03 > + jason 49 2 2020-06-09T08:03 2020-06-09T08:03:05 > + jason 51 2 2020-06-09T08:03:05 2020-06-09T08:03:10 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18365) The same sql in a batch env and a streaming env has different value.
[ https://issues.apache.org/jira/browse/FLINK-18365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-18365: Fix Version/s: (was: 1.11.1) 1.11.2 > The same sql in a batch env and a streaming env has different value. > > > Key: FLINK-18365 > URL: https://issues.apache.org/jira/browse/FLINK-18365 > Project: Flink > Issue Type: Bug > Components: Table SQL / API >Affects Versions: 1.11.0 >Reporter: xiaojin.wy >Priority: Major > Fix For: 1.11.2 > > > I use the sql-gateway to run this sql. > *The input table is:* > CREATE TABLE `scott_dept` ( > deptno INT, > dname VARCHAR, > loc VARCHAR > ) WITH ( > 'format.field-delimiter'='|', > 'connector.type'='filesystem', > 'format.derive-schema'='true', > > 'connector.path'='/defender_test_data/daily_regression_stream_blink_sql_1.10/test_scalar/sources/scott_dept.csv', > 'format.type'='csv' > ) > *The input data is:* > 10|ACCOUNTING|NEW YORK > 20|RESEARCH|DALLAS > 30|SALES|CHICAGO > 40|OPERATIONS|BOSTON > *The sql is :* > select deptno, (select count(*) from scott_emp where 1 = 0) as x from > scott_dept > *The error:* > In a batch environment, the result value is:10|0\n20|0\n30|0\n40|0 > In a streaming environment, the result value > is:10|None\n20|None\n30|None\n40|None -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-16622) Type ARRAY of table field 'id' does not match with the physical type LEGACY('ARRAY', 'ANY<[Ljava.lang.String;, rO0ABXNyAD......>') of the 'id' field of the Table
[
https://issues.apache.org/jira/browse/FLINK-16622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-16622:
Fix Version/s: (was: 1.11.1)
1.11.2
> Type ARRAY of table field 'id' does not match with the physical type
> LEGACY('ARRAY', 'ANY<[Ljava.lang.String;, rO0ABXNyAD..>') of the 'id'
> field of the TableSource return type.
>
>
> Key: FLINK-16622
> URL: https://issues.apache.org/jira/browse/FLINK-16622
> Project: Flink
> Issue Type: Bug
> Components: Table SQL / API
>Affects Versions: 1.10.0
>Reporter: gkgkgk
>Assignee: Jark Wu
>Priority: Major
> Fix For: 1.10.2, 1.11.2
>
>
> when i define a ARRAY field in table schema and define field in
> 'format.json-schema' like this
> 'format.json-schema' = '{
> "type": "object",
> "properties": {
> "id": {
> "type": "array",
> "items":{"type":"string"}
> }
> }'
> there is a error flow this:
> Exception in thread "main" org.apache.flink.table.api.ValidationException:
> Type ARRAY of table field 'id' does not match with the physical type
> LEGACY('ARRAY', 'ANY<[Ljava.lang.String;, rO0ABXNyADdvc..>') of the 'id'
> field of the TableSource return type.
> the flow is my ddl:
> --source
> CREATE TABLE dwd_user_log (
> id ARRAY,
> ctime TIMESTAMP(3),
> pageId VARCHAR,
> deviceId VARCHAR,
> WATERMARK FOR ctime AS ctime - INTERVAL '10' SECOND
> ) WITH (
> 'connector.type' = 'kafka',
> 'connector.version' = 'universal',
> 'connector.topic' = 'dev_dwd_user_log_02',
> 'connector.startup-mode' = 'latest-offset',
> 'connector.properties.0.key' = 'zookeeper.connect',
> 'connector.properties.0.value' = 'node14.example.com:2181',
> 'connector.properties.1.key' = 'bootstrap.servers',
> 'connector.properties.1.value' = 'node14.example.com:9092',
> 'connector.properties.2.key' = 'group.id',
> 'connector.properties.2.value' = 'dev-group',
> 'update-mode' = 'append',
> 'format.type' = 'json',
> -- 'format.derive-schema' = 'true'
> 'format.json-schema' = '{
> "type": "object",
> "properties": {
> "id": {
> "type": "array",
> "items":{"type":"string"}
> },
> "ctime": {
> "type": "string",
> "format": "date-time"
> },
> "pageId": {
> "type": "string"
> },
> "deviceId": {
> "type": "string"
> }
> }
> }'
> -- 'schema.1.rowtime.timestamps.type' = 'from-field',
> -- 'schema.1.rowtime.timestamps.from' = 'ctime',
> -- 'schema.1.rowtime.watermarks.type' = 'periodic-bounded',
> -- 'schema.1.rowtime.watermarks.delay' = '1'
> -- 'schema.1.from' = 'ctime'
> );
> -- sink
> -- sink for pv
> CREATE TABLE dws_pv (
> windowStart TIMESTAMP(3),
> windowEnd TIMESTAMP(3),
> pageId VARCHAR,
> id ARRAY,
> viewCount BIGINT
> ) WITH (
> 'connector.type' = 'kafka',
> 'connector.version' = 'universal',
> 'connector.topic' = 'dev_dws_pvuv_02',
> 'connector.startup-mode' = 'latest-offset',
> 'connector.properties.0.key' = 'zookeeper.connect',
> 'connector.properties.0.value' = 'node14.example.com:2181',
> 'connector.properties.1.key' = 'bootstrap.servers',
> 'connector.properties.1.value' = 'node14.example.com:9092',
> 'connector.properties.2.key' = 'group.id',
> 'connector.properties.2.value' = 'dev-group',
> 'update-mode' = 'append',
> 'format.type' = 'json',
> 'format.derive-schema' = 'true'
> );
> -- pv
> INSERT INTO dws_pv
> SELECT
> TUMBLE_START(ctime, INTERVAL '20' SECOND) AS windowStart,
> TUMBLE_END(ctime, INTERVAL '20' SECOND) AS windowEnd,
> pageId,
> id,
> COUNT(deviceId) AS viewCount
> FROM dwd_user_log
> GROUP BY TUMBLE(ctime, INTERVAL '20' SECOND),pageId,id;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-17964) Hivemodule does not support map type
[
https://issues.apache.org/jira/browse/FLINK-17964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-17964:
Fix Version/s: (was: 1.11.1)
1.11.2
> Hivemodule does not support map type
>
>
> Key: FLINK-17964
> URL: https://issues.apache.org/jira/browse/FLINK-17964
> Project: Flink
> Issue Type: Bug
> Components: Connectors / Hive
>Affects Versions: 1.10.1
>Reporter: Shuai Xia
>Priority: Major
> Fix For: 1.12.0, 1.11.2
>
>
> {code:java}
> //代码占位符
> Exception in thread "main" scala.MatchError: MAP (of class
> org.apache.flink.table.types.logical.LogicalTypeRoot)Exception in thread
> "main" scala.MatchError: MAP (of class
> org.apache.flink.table.types.logical.LogicalTypeRoot) at
> org.apache.flink.table.planner.codegen.CodeGenUtils$.hashCodeForType(CodeGenUtils.scala:212)
> at
> org.apache.flink.table.planner.codegen.HashCodeGenerator$$anonfun$2.apply(HashCodeGenerator.scala:97)
> at
> org.apache.flink.table.planner.codegen.HashCodeGenerator$$anonfun$2.apply(HashCodeGenerator.scala:91)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
> at
> scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:35) at
> scala.collection.TraversableLike$class.map(TraversableLike.scala:234) at
> scala.collection.AbstractTraversable.map(Traversable.scala:104) at
> org.apache.flink.table.planner.codegen.HashCodeGenerator$.generateCodeBody(HashCodeGenerator.scala:91)
> at
> org.apache.flink.table.planner.codegen.HashCodeGenerator$.generateRowHash(HashCodeGenerator.scala:61)
> at
> org.apache.flink.table.planner.plan.nodes.physical.batch.BatchExecExchange.translateToPlanInternal(BatchExecExchange.scala:182)
> at
> org.apache.flink.table.planner.plan.nodes.physical.batch.BatchExecExchange.translateToPlanInternal(BatchExecExchange.scala:52)
> at
> org.apache.flink.table.planner.plan.nodes.exec.ExecNode$class.translateToPlan(ExecNode.scala:58)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18581) Cannot find GC cleaner with java version previous 8u202
[
https://issues.apache.org/jira/browse/FLINK-18581?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-18581:
Fix Version/s: (was: 1.11.1)
1.11.2
> Cannot find GC cleaner with java version previous 8u202
> ---
>
> Key: FLINK-18581
> URL: https://issues.apache.org/jira/browse/FLINK-18581
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.0
>Reporter: Xintong Song
>Priority: Critical
> Fix For: 1.12.0, 1.11.2
>
>
> {{JavaGcCleanerWrapper}} is looking for the package-private method
> {{Reference.tryHandlePending}} using reflection. However, the method is first
> introduced in the version 8u202. Therefore, if an older version JDK is used,
> the method cannot be found and Flink will fail.
> See also this [ML
> thread|http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Error-GC-Cleaner-Provider-Flink-1-11-0-td36565.html].
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18597) elasticsearch sql insert function cannot set documentid
[ https://issues.apache.org/jira/browse/FLINK-18597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157849#comment-17157849 ] Jark Wu commented on FLINK-18597: - If you want to specify document-id, please refer to the docs: https://ci.apache.org/projects/flink/flink-docs-master/dev/table/connectors/elasticsearch.html#key-handling Lastly, I'd suggest to reach out to the mailing list first. I'd suggest creating an issue only when you have fully analyzed a problem. > elasticsearch sql insert function cannot set documentid > --- > > Key: FLINK-18597 > URL: https://issues.apache.org/jira/browse/FLINK-18597 > Project: Flink > Issue Type: Test >Affects Versions: 1.11.0 >Reporter: xy >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18449) Make topic discovery and partition discovery configurable for FlinkKafkaConsumer in Table API
[
https://issues.apache.org/jira/browse/FLINK-18449?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157848#comment-17157848
]
Shengkai Fang commented on FLINK-18449:
---
Hi, all. I find that FlinkKafkaProducer doesn't support partition discovery
now. So it's may be more clear to rename
{{'topic-partition-discovery.interval'}} to
{{'scan.topic-partition-discovery.interval'}}. Considering Jark's proposal, I
think the following options is better,
{code:java}
'topic' = 'topic'// for single topic
'topic' = 'topic-1, topic-2,..., topic-n', // for list topic
'topic-pattern' = 'topic*' // for use of regex, "topic"
and "topic-pattern" can't coexist
'scan.topic-partition-discovery.interval' = '5s' // for both topic discovery
{code}
> Make topic discovery and partition discovery configurable for
> FlinkKafkaConsumer in Table API
> -
>
> Key: FLINK-18449
> URL: https://issues.apache.org/jira/browse/FLINK-18449
> Project: Flink
> Issue Type: Improvement
> Components: Connectors / Kafka, Table SQL / API
>Affects Versions: 1.11.0
>Reporter: Shengkai Fang
>Assignee: Shengkai Fang
>Priority: Major
> Fix For: 1.12.0
>
>
> In streaming api, we can use regex to find topic and enable partiton
> discovery by setting non-negative value for
> `{{flink.partition-discovery.interval-millis}}`. However, it's not work in
> table api. I think we can add options such as 'topic-regex' and
> '{{partition-discovery.interval-millis}}' in WITH block for users.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12903: [FLINK-18492][python] Extract the implementation logic of Beam in Operations
flinkbot edited a comment on pull request #12903: URL: https://github.com/apache/flink/pull/12903#issuecomment-658519879 ## CI report: * aa4bb627890b6e46ef36f44528a24aa304bb7064 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4521) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18597) elasticsearch sql insert function cannot set documentid
[ https://issues.apache.org/jira/browse/FLINK-18597?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jark Wu updated FLINK-18597: Priority: Minor (was: Critical) > elasticsearch sql insert function cannot set documentid > --- > > Key: FLINK-18597 > URL: https://issues.apache.org/jira/browse/FLINK-18597 > Project: Flink > Issue Type: Test >Affects Versions: 1.11.0 >Reporter: xy >Priority: Minor > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] godfreyhe commented on a change in pull request #12866: [FLINK-17425][blink-planner] supportsFilterPushDown rule in DynamicSource.
godfreyhe commented on a change in pull request #12866:
URL: https://github.com/apache/flink/pull/12866#discussion_r454768224
##
File path:
flink-table/flink-table-planner-blink/src/test/java/org/apache/flink/table/planner/factories/TestValuesTableFactory.java
##
@@ -540,40 +541,55 @@ private boolean
isRetainedAfterApplyingFilterPredicates(Row row) {
private boolean binaryFilterApplies(CallExpression binExpr, Row
row) {
List children = binExpr.getChildren();
Preconditions.checkArgument(children.size() == 2);
- Comparable lhsValue = getValue(children.get(0), row);
- Comparable rhsValue = getValue(children.get(1), row);
- FunctionDefinition functionDefinition =
binExpr.getFunctionDefinition();
+ Object lhsValue = getValue(children.get(0), row);
+ Object rhsValue = getValue(children.get(1), row);
+ // validate that literal is comparable
+ if (!isComparable(lhsValue, binExpr) ||
!isComparable(rhsValue, binExpr)) {
Review comment:
how about check whether the type of a field is comparable in
`shouldPushDownUnaryExpression` method ? then the logic of
`binaryFilterApplies` need not to change
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-18601) TaskManager shut down often in standalone cluster
[
https://issues.apache.org/jira/browse/FLINK-18601?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Echo Lee updated FLINK-18601:
-
Summary: TaskManager shut down often in standalone cluster (was:
Taskmanager shut down often in standalone cluster)
> TaskManager shut down often in standalone cluster
> -
>
> Key: FLINK-18601
> URL: https://issues.apache.org/jira/browse/FLINK-18601
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.10.0, 1.11.0
>Reporter: Echo Lee
>Priority: Major
>
> The situation we encountered is that the job is running on the standalone
> cluster. Cancelling the job occasionally causes the taskmanager to shut down.
> I am not sure whether this is a problem. and Some of the logs are as follows:
> {code:java}
> 2020-07-14 20:16:00.169 [Cancellation Watchdog for KeyedProcess -> (Filter ->
> Map, Sink: FeatureSinkToES) (1/1) (8c10a884bf480714460860f9d0e6131e).] ERROR
> org.apache.flink.runtime.taskexecutor.TaskExecutor - Task did not exit
> gracefully within 180 + seconds.
> org.apache.flink.util.FlinkRuntimeException: Task did not exit gracefully
> within 180 + seconds.
> at
> org.apache.flink.runtime.taskmanager.Task$TaskCancelerWatchDog.run(Task.java:1572)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-14 20:16:00.169 [Cancellation Watchdog for KeyedProcess -> (Filter ->
> Map, Sink: FeatureSinkToES) (1/1) (8c10a884bf480714460860f9d0e6131e).] ERROR
> org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Fatal error
> occurred while executing the TaskManager. Shutting it down...
> org.apache.flink.util.FlinkRuntimeException: Task did not exit gracefully
> within 180 + seconds.
> at
> org.apache.flink.runtime.taskmanager.Task$TaskCancelerWatchDog.run(Task.java:1572)
> at java.base/java.lang.Thread.run(Thread.java:834)
> 2020-07-14 20:16:00.170 [flink-akka.actor.default-dispatcher-27] INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor - Stopping TaskExecutor
> akka.tcp://flink@10.3.67.116:35867/user/rpc/taskmanager_0.
> 2020-07-14 20:16:00.170 [flink-akka.actor.default-dispatcher-27] INFO
> org.apache.flink.runtime.taskexecutor.TaskExecutor - Close ResourceManager
> connection fb5cf1580e59ff78505facc7010b099c.
> 2020-07-14 20:16:00.195 [Canceler/Interrupts for KeyedProcess -> (Filter ->
> Map, Sink: FeatureSinkToES) (1/1) (8c10a884bf480714460860f9d0e6131e).] WARN
> org.apache.flink.runtime.taskmanager.Task - Task 'KeyedProcess -> (Filter ->
> Map, Sink: FeatureSinkToES) (1/1)' did not react to cancelling signal for 30
> seconds, but is stuck in method:
>
> org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.snapshotState(ElasticsearchSinkBase.java:325)
> app//org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:120)
> app//org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:101)
> app//org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:90)
> app//org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:186)
> app//org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:156)
> app//org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:314)
> app//org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointStreamOperator(SubtaskCheckpointCoordinatorImpl.java:614)
> app//org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.buildOperatorSnapshotFutures(SubtaskCheckpointCoordinatorImpl.java:540)
> app//org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:507)
> app//org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:266)
> app//org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$5(StreamTask.java:892)
> app//org.apache.flink.streaming.runtime.tasks.StreamTask$$Lambda$3269/0x7f3f8a9ca560.run(Unknown
> Source)
> app//org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
> app//org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:882)
> app//org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:850)
> app//org.apache.flink.streaming.runtime.io.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:113)
>
[jira] [Created] (FLINK-18601) Taskmanager shut down often in standalone cluster
Echo Lee created FLINK-18601:
Summary: Taskmanager shut down often in standalone cluster
Key: FLINK-18601
URL: https://issues.apache.org/jira/browse/FLINK-18601
Project: Flink
Issue Type: Bug
Components: Runtime / Task
Affects Versions: 1.11.0, 1.10.0
Reporter: Echo Lee
The situation we encountered is that the job is running on the standalone
cluster. Cancelling the job occasionally causes the taskmanager to shut down. I
am not sure whether this is a problem. and Some of the logs are as follows:
{code:java}
2020-07-14 20:16:00.169 [Cancellation Watchdog for KeyedProcess -> (Filter ->
Map, Sink: FeatureSinkToES) (1/1) (8c10a884bf480714460860f9d0e6131e).] ERROR
org.apache.flink.runtime.taskexecutor.TaskExecutor - Task did not exit
gracefully within 180 + seconds.
org.apache.flink.util.FlinkRuntimeException: Task did not exit gracefully
within 180 + seconds.
at
org.apache.flink.runtime.taskmanager.Task$TaskCancelerWatchDog.run(Task.java:1572)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-07-14 20:16:00.169 [Cancellation Watchdog for KeyedProcess -> (Filter ->
Map, Sink: FeatureSinkToES) (1/1) (8c10a884bf480714460860f9d0e6131e).] ERROR
org.apache.flink.runtime.taskexecutor.TaskManagerRunner - Fatal error occurred
while executing the TaskManager. Shutting it down...
org.apache.flink.util.FlinkRuntimeException: Task did not exit gracefully
within 180 + seconds.
at
org.apache.flink.runtime.taskmanager.Task$TaskCancelerWatchDog.run(Task.java:1572)
at java.base/java.lang.Thread.run(Thread.java:834)
2020-07-14 20:16:00.170 [flink-akka.actor.default-dispatcher-27] INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Stopping TaskExecutor
akka.tcp://flink@10.3.67.116:35867/user/rpc/taskmanager_0.
2020-07-14 20:16:00.170 [flink-akka.actor.default-dispatcher-27] INFO
org.apache.flink.runtime.taskexecutor.TaskExecutor - Close ResourceManager
connection fb5cf1580e59ff78505facc7010b099c.
2020-07-14 20:16:00.195 [Canceler/Interrupts for KeyedProcess -> (Filter ->
Map, Sink: FeatureSinkToES) (1/1) (8c10a884bf480714460860f9d0e6131e).] WARN
org.apache.flink.runtime.taskmanager.Task - Task 'KeyedProcess -> (Filter ->
Map, Sink: FeatureSinkToES) (1/1)' did not react to cancelling signal for 30
seconds, but is stuck in method:
org.apache.flink.streaming.connectors.elasticsearch.ElasticsearchSinkBase.snapshotState(ElasticsearchSinkBase.java:325)
app//org.apache.flink.streaming.util.functions.StreamingFunctionUtils.trySnapshotFunctionState(StreamingFunctionUtils.java:120)
app//org.apache.flink.streaming.util.functions.StreamingFunctionUtils.snapshotFunctionState(StreamingFunctionUtils.java:101)
app//org.apache.flink.streaming.api.operators.AbstractUdfStreamOperator.snapshotState(AbstractUdfStreamOperator.java:90)
app//org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:186)
app//org.apache.flink.streaming.api.operators.StreamOperatorStateHandler.snapshotState(StreamOperatorStateHandler.java:156)
app//org.apache.flink.streaming.api.operators.AbstractStreamOperator.snapshotState(AbstractStreamOperator.java:314)
app//org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointStreamOperator(SubtaskCheckpointCoordinatorImpl.java:614)
app//org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.buildOperatorSnapshotFutures(SubtaskCheckpointCoordinatorImpl.java:540)
app//org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.takeSnapshotSync(SubtaskCheckpointCoordinatorImpl.java:507)
app//org.apache.flink.streaming.runtime.tasks.SubtaskCheckpointCoordinatorImpl.checkpointState(SubtaskCheckpointCoordinatorImpl.java:266)
app//org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$performCheckpoint$5(StreamTask.java:892)
app//org.apache.flink.streaming.runtime.tasks.StreamTask$$Lambda$3269/0x7f3f8a9ca560.run(Unknown
Source)
app//org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
app//org.apache.flink.streaming.runtime.tasks.StreamTask.performCheckpoint(StreamTask.java:882)
app//org.apache.flink.streaming.runtime.tasks.StreamTask.triggerCheckpointOnBarrier(StreamTask.java:850)
app//org.apache.flink.streaming.runtime.io.CheckpointBarrierHandler.notifyCheckpoint(CheckpointBarrierHandler.java:113)
app//org.apache.flink.streaming.runtime.io.CheckpointBarrierAligner.processBarrier(CheckpointBarrierAligner.java:137)
app//org.apache.flink.streaming.runtime.io.CheckpointedInputGate.pollNext(CheckpointedInputGate.java:93)
app//org.apache.flink.streaming.runtime.io.StreamTaskNetworkInput.emitNext(StreamTaskNetworkInput.java:158)
app//org.apache.flink.streaming.runtime.io.StreamOneInputProcessor.processInput(StreamOneInputProcessor.java:67)
[jira] [Commented] (FLINK-18600) Kerberized YARN per-job on Docker test failed to download JDK 8u251
[
https://issues.apache.org/jira/browse/FLINK-18600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157844#comment-17157844
]
Dian Fu commented on FLINK-18600:
-
Not sure the reason why we choose oracle JDK and if we could switch to Open JDK
for the tests?
> Kerberized YARN per-job on Docker test failed to download JDK 8u251
> ---
>
> Key: FLINK-18600
> URL: https://issues.apache.org/jira/browse/FLINK-18600
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Dian Fu
>Priority: Blocker
> Fix For: 1.12.0, 1.11.2
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4514=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> + mkdir -p /usr/java/default
> + curl -Ls
> https://download.oracle.com/otn-pub/java/jdk/8u251-b08/3d5a2bb8f8d4428bbe94aed7ec7ae784/jdk-8u251-linux-x64.tar.gz
> -H Cookie: oraclelicense=accept-securebackup-cookie
> + tar --strip-components=1 -xz -C /usr/java/default/
> gzip: stdin: not in gzip format
> tar: Child returned status 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-18600) Kerberized YARN per-job on Docker test failed to download JDK 8u251
[
https://issues.apache.org/jira/browse/FLINK-18600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-18600:
Priority: Blocker (was: Critical)
> Kerberized YARN per-job on Docker test failed to download JDK 8u251
> ---
>
> Key: FLINK-18600
> URL: https://issues.apache.org/jira/browse/FLINK-18600
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Dian Fu
>Priority: Blocker
> Fix For: 1.12.0, 1.11.2
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4514=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> + mkdir -p /usr/java/default
> + curl -Ls
> https://download.oracle.com/otn-pub/java/jdk/8u251-b08/3d5a2bb8f8d4428bbe94aed7ec7ae784/jdk-8u251-linux-x64.tar.gz
> -H Cookie: oraclelicense=accept-securebackup-cookie
> + tar --strip-components=1 -xz -C /usr/java/default/
> gzip: stdin: not in gzip format
> tar: Child returned status 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18600) Kerberized YARN per-job on Docker test failed to download JDK 8u251
[
https://issues.apache.org/jira/browse/FLINK-18600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157843#comment-17157843
]
Dian Fu commented on FLINK-18600:
-
I have checked that I can download the version *8u251* through the oracle
website after accepting the license. However, it's not available if download
through CLI using the same command. I'm wondering if it's because the cookie
has changed or some other reason?
> Kerberized YARN per-job on Docker test failed to download JDK 8u251
> ---
>
> Key: FLINK-18600
> URL: https://issues.apache.org/jira/browse/FLINK-18600
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Dian Fu
>Priority: Critical
> Fix For: 1.12.0, 1.11.2
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4514=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> + mkdir -p /usr/java/default
> + curl -Ls
> https://download.oracle.com/otn-pub/java/jdk/8u251-b08/3d5a2bb8f8d4428bbe94aed7ec7ae784/jdk-8u251-linux-x64.tar.gz
> -H Cookie: oraclelicense=accept-securebackup-cookie
> + tar --strip-components=1 -xz -C /usr/java/default/
> gzip: stdin: not in gzip format
> tar: Child returned status 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (FLINK-18540) Unnecessary retract messages when setIdleStateRetentionTime before converting dynamic table to retract stream
[
https://issues.apache.org/jira/browse/FLINK-18540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157838#comment-17157838
]
WeiqiangSun commented on FLINK-18540:
-
[~libenchao] Mini batch may not work well with retract messages for HBase
connector. Unnecessary retract messages can cause puts and deletes with same
timestamp. As for HBase, deletes can mask puts with the same timestamp.
https://issues.apache.org/jira/browse/HBASE-2256
https://issues.apache.org/jira/browse/HBASE-8721
> Unnecessary retract messages when setIdleStateRetentionTime before converting
> dynamic table to retract stream
> -
>
> Key: FLINK-18540
> URL: https://issues.apache.org/jira/browse/FLINK-18540
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner, Table SQL / Runtime
>Affects Versions: 1.10.0
>Reporter: WeiqiangSun
>Priority: Critical
>
> Unnecessary retract messages when setIdleStateRetentionTime before converting
> dynamic table to retract stream。
>
> {code:java}
> final StreamExecutionEnvironment env =
> StreamExecutionEnvironment.getExecutionEnvironment();
> env.setParallelism(1);
> EnvironmentSettings es =
> EnvironmentSettings.newInstance().inStreamingMode().build();
> StreamTableEnvironment fsTableEnv = StreamTableEnvironment.create(env,es);
> DataStream ds = env.fromElements(5,4,3,2,1);
> // if do not setIdleStateRetentionTime, it will output only one message as
> expected like '(true,5)'
> // if do setIdleStateRetentionTime, it will output unnecessary retract
> messages like '(false,5)/(true,5)'
> fsTableEnv.getConfig().setIdleStateRetentionTime(Time.hours(100),Time.hours(1000));
> fsTableEnv.createTemporaryView("test",ds,"id");
> Table test = fsTableEnv.sqlQuery("select max(id) from test");
> fsTableEnv.toRetractStream(test,Row.class).printToErr();
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #12903: [FLINK-18492][python] Extract the implementation logic of Beam in Operations
flinkbot commented on pull request #12903: URL: https://github.com/apache/flink/pull/12903#issuecomment-658519879 ## CI report: * aa4bb627890b6e46ef36f44528a24aa304bb7064 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18600) Kerberized YARN per-job on Docker test failed to download JDK 8u251
[
https://issues.apache.org/jira/browse/FLINK-18600?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157832#comment-17157832
]
Jark Wu commented on FLINK-18600:
-
Looks like we started to use this JDK link since FLINK-18485. Maybe this link
is not stable.
cc [~dwysakowicz]
> Kerberized YARN per-job on Docker test failed to download JDK 8u251
> ---
>
> Key: FLINK-18600
> URL: https://issues.apache.org/jira/browse/FLINK-18600
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Dian Fu
>Priority: Critical
> Fix For: 1.12.0, 1.11.2
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4514=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> + mkdir -p /usr/java/default
> + curl -Ls
> https://download.oracle.com/otn-pub/java/jdk/8u251-b08/3d5a2bb8f8d4428bbe94aed7ec7ae784/jdk-8u251-linux-x64.tar.gz
> -H Cookie: oraclelicense=accept-securebackup-cookie
> + tar --strip-components=1 -xz -C /usr/java/default/
> gzip: stdin: not in gzip format
> tar: Child returned status 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #12902: [FLINK-18415][python] Support TableResult#collect in the Python Table…
flinkbot edited a comment on pull request #12902: URL: https://github.com/apache/flink/pull/12902#issuecomment-658509076 ## CI report: * 802dfd4a0a2fb3e1b6e9a31fbc850c18f00c5c2b Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4518) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12866: [FLINK-17425][blink-planner] supportsFilterPushDown rule in DynamicSource.
flinkbot edited a comment on pull request #12866: URL: https://github.com/apache/flink/pull/12866#issuecomment-656576219 ## CI report: * 02c81cff07ef76dc20a7c618112cc7a172856720 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4497) * 5eb302884e390b6099859abcd7955e7f1c4e034e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4517) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] lirui-apache commented on pull request #12888: [FLINK-18588] hive ddl create table support 'if not exists'
lirui-apache commented on pull request #12888: URL: https://github.com/apache/flink/pull/12888#issuecomment-658513513 Thanks @wtog for the contribution. Could we also have some test for this in `HiveDialectITCase`? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12903: [FLINK-18492][python] Extract the implementation logic of Beam in Operations
flinkbot commented on pull request #12903: URL: https://github.com/apache/flink/pull/12903#issuecomment-658512879 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit aa4bb627890b6e46ef36f44528a24aa304bb7064 (Wed Jul 15 02:45:44 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] HuangXingBo opened a new pull request #12903: [FLINK-18492][python] Extract the implementation logic of Beam in Operations
HuangXingBo opened a new pull request #12903: URL: https://github.com/apache/flink/pull/12903 ## What is the purpose of the change *This pull request extract the implementation logic of Beam in Operations* ## Brief change log - *Extract the beam related logic of fast_operations to beam_operations* - *rename operations.py to beam_slow_operations.py* ## Verifying this change *current tests are enough* ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18492) Extract the implementation logic of Beam in Operations
[ https://issues.apache.org/jira/browse/FLINK-18492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-18492: --- Labels: pull-request-available (was: ) > Extract the implementation logic of Beam in Operations > -- > > Key: FLINK-18492 > URL: https://issues.apache.org/jira/browse/FLINK-18492 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Huang Xingbo >Assignee: Huang Xingbo >Priority: Major > Labels: pull-request-available > Fix For: 1.12.0 > > > Extract the implementation logic of Beam in Operations, so that the > implementation of general operations and Beam operations can be decoupled -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-17636) SingleInputGateTest.testConcurrentReadStateAndProcessAndClose: Trying to read from released RecoveredInputChannel
[
https://issues.apache.org/jira/browse/FLINK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijiang updated FLINK-17636:
-
Affects Version/s: 1.11.0
> SingleInputGateTest.testConcurrentReadStateAndProcessAndClose: Trying to read
> from released RecoveredInputChannel
> -
>
> Key: FLINK-17636
> URL: https://issues.apache.org/jira/browse/FLINK-17636
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Assignee: Zhijiang
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> CI:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=1080=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=4ed44b66-cdd6-5dcf-5f6a-88b07dda665d
> {code}
> 2020-05-12T11:39:28.7058732Z [ERROR] Tests run: 22, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 2.643 s <<< FAILURE! - in
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGateTest
> 2020-05-12T11:39:28.7066377Z [ERROR]
> testConcurrentReadStateAndProcessAndClose(org.apache.flink.runtime.io.network.partition.consumer.SingleInputGateTest)
> Time elapsed: 0.032 s <<< ERROR!
> 2020-05-12T11:39:28.7067491Z java.util.concurrent.ExecutionException:
> java.lang.IllegalStateException: Trying to read from released
> RecoveredInputChannel
> 2020-05-12T11:39:28.7068238Z at
> java.util.concurrent.FutureTask.report(FutureTask.java:122)
> 2020-05-12T11:39:28.7068795Z at
> java.util.concurrent.FutureTask.get(FutureTask.java:192)
> 2020-05-12T11:39:28.7069538Z at
> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannelTest.submitTasksAndWaitForResults(RemoteInputChannelTest.java:1228)
> 2020-05-12T11:39:28.7070595Z at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGateTest.testConcurrentReadStateAndProcessAndClose(SingleInputGateTest.java:235)
> 2020-05-12T11:39:28.7075974Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-05-12T11:39:28.7076784Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-05-12T11:39:28.7077522Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-05-12T11:39:28.7078212Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-05-12T11:39:28.7078846Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-05-12T11:39:28.7079607Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-05-12T11:39:28.7080383Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-05-12T11:39:28.7081173Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-05-12T11:39:28.7081937Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2020-05-12T11:39:28.7082708Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-05-12T11:39:28.7083422Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-05-12T11:39:28.7084148Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-05-12T11:39:28.7084933Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-05-12T11:39:28.7085562Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-05-12T11:39:28.7086162Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-05-12T11:39:28.7086806Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-05-12T11:39:28.7087434Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-05-12T11:39:28.7088036Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-05-12T11:39:28.7088647Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2020-05-12T11:39:28.7089328Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> 2020-05-12T11:39:28.7090106Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-05-12T11:39:28.7090811Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> 2020-05-12T11:39:28.7091674Z at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> 2020-05-12T11:39:28.7102178Z at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> 2020-05-12T11:39:28.7103048Z at
>
[jira] [Updated] (FLINK-17636) SingleInputGateTest.testConcurrentReadStateAndProcessAndClose: Trying to read from released RecoveredInputChannel
[
https://issues.apache.org/jira/browse/FLINK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijiang updated FLINK-17636:
-
Fix Version/s: 1.11.0
> SingleInputGateTest.testConcurrentReadStateAndProcessAndClose: Trying to read
> from released RecoveredInputChannel
> -
>
> Key: FLINK-17636
> URL: https://issues.apache.org/jira/browse/FLINK-17636
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network, Tests
>Affects Versions: 1.11.0
>Reporter: Robert Metzger
>Assignee: Zhijiang
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.11.0, 1.12.0
>
>
> CI:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=1080=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=4ed44b66-cdd6-5dcf-5f6a-88b07dda665d
> {code}
> 2020-05-12T11:39:28.7058732Z [ERROR] Tests run: 22, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 2.643 s <<< FAILURE! - in
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGateTest
> 2020-05-12T11:39:28.7066377Z [ERROR]
> testConcurrentReadStateAndProcessAndClose(org.apache.flink.runtime.io.network.partition.consumer.SingleInputGateTest)
> Time elapsed: 0.032 s <<< ERROR!
> 2020-05-12T11:39:28.7067491Z java.util.concurrent.ExecutionException:
> java.lang.IllegalStateException: Trying to read from released
> RecoveredInputChannel
> 2020-05-12T11:39:28.7068238Z at
> java.util.concurrent.FutureTask.report(FutureTask.java:122)
> 2020-05-12T11:39:28.7068795Z at
> java.util.concurrent.FutureTask.get(FutureTask.java:192)
> 2020-05-12T11:39:28.7069538Z at
> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannelTest.submitTasksAndWaitForResults(RemoteInputChannelTest.java:1228)
> 2020-05-12T11:39:28.7070595Z at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGateTest.testConcurrentReadStateAndProcessAndClose(SingleInputGateTest.java:235)
> 2020-05-12T11:39:28.7075974Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-05-12T11:39:28.7076784Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-05-12T11:39:28.7077522Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-05-12T11:39:28.7078212Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-05-12T11:39:28.7078846Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-05-12T11:39:28.7079607Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-05-12T11:39:28.7080383Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-05-12T11:39:28.7081173Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-05-12T11:39:28.7081937Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2020-05-12T11:39:28.7082708Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-05-12T11:39:28.7083422Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-05-12T11:39:28.7084148Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-05-12T11:39:28.7084933Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-05-12T11:39:28.7085562Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-05-12T11:39:28.7086162Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-05-12T11:39:28.7086806Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-05-12T11:39:28.7087434Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-05-12T11:39:28.7088036Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-05-12T11:39:28.7088647Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2020-05-12T11:39:28.7089328Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> 2020-05-12T11:39:28.7090106Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-05-12T11:39:28.7090811Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> 2020-05-12T11:39:28.7091674Z at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> 2020-05-12T11:39:28.7102178Z at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:345)
> 2020-05-12T11:39:28.7103048Z at
>
[jira] [Closed] (FLINK-17636) SingleInputGateTest.testConcurrentReadStateAndProcessAndClose: Trying to read from released RecoveredInputChannel
[
https://issues.apache.org/jira/browse/FLINK-17636?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhijiang closed FLINK-17636.
Resolution: Fixed
Merged in release-1.11: 57e57624deded5594749cbccf17a5878c6dc81b8
Merged in master: a9cce5467b3f0628e665cd8c7dd9b2e5958f0fd7
> SingleInputGateTest.testConcurrentReadStateAndProcessAndClose: Trying to read
> from released RecoveredInputChannel
> -
>
> Key: FLINK-17636
> URL: https://issues.apache.org/jira/browse/FLINK-17636
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Network, Tests
>Reporter: Robert Metzger
>Assignee: Zhijiang
>Priority: Critical
> Labels: pull-request-available, test-stability
> Fix For: 1.12.0
>
>
> CI:
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=1080=logs=0da23115-68bb-5dcd-192c-bd4c8adebde1=4ed44b66-cdd6-5dcf-5f6a-88b07dda665d
> {code}
> 2020-05-12T11:39:28.7058732Z [ERROR] Tests run: 22, Failures: 0, Errors: 1,
> Skipped: 0, Time elapsed: 2.643 s <<< FAILURE! - in
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGateTest
> 2020-05-12T11:39:28.7066377Z [ERROR]
> testConcurrentReadStateAndProcessAndClose(org.apache.flink.runtime.io.network.partition.consumer.SingleInputGateTest)
> Time elapsed: 0.032 s <<< ERROR!
> 2020-05-12T11:39:28.7067491Z java.util.concurrent.ExecutionException:
> java.lang.IllegalStateException: Trying to read from released
> RecoveredInputChannel
> 2020-05-12T11:39:28.7068238Z at
> java.util.concurrent.FutureTask.report(FutureTask.java:122)
> 2020-05-12T11:39:28.7068795Z at
> java.util.concurrent.FutureTask.get(FutureTask.java:192)
> 2020-05-12T11:39:28.7069538Z at
> org.apache.flink.runtime.io.network.partition.consumer.RemoteInputChannelTest.submitTasksAndWaitForResults(RemoteInputChannelTest.java:1228)
> 2020-05-12T11:39:28.7070595Z at
> org.apache.flink.runtime.io.network.partition.consumer.SingleInputGateTest.testConcurrentReadStateAndProcessAndClose(SingleInputGateTest.java:235)
> 2020-05-12T11:39:28.7075974Z at
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 2020-05-12T11:39:28.7076784Z at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 2020-05-12T11:39:28.7077522Z at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 2020-05-12T11:39:28.7078212Z at
> java.lang.reflect.Method.invoke(Method.java:498)
> 2020-05-12T11:39:28.7078846Z at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> 2020-05-12T11:39:28.7079607Z at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> 2020-05-12T11:39:28.7080383Z at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> 2020-05-12T11:39:28.7081173Z at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> 2020-05-12T11:39:28.7081937Z at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> 2020-05-12T11:39:28.7082708Z at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> 2020-05-12T11:39:28.7083422Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> 2020-05-12T11:39:28.7084148Z at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> 2020-05-12T11:39:28.7084933Z at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> 2020-05-12T11:39:28.7085562Z at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> 2020-05-12T11:39:28.7086162Z at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> 2020-05-12T11:39:28.7086806Z at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> 2020-05-12T11:39:28.7087434Z at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> 2020-05-12T11:39:28.7088036Z at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> 2020-05-12T11:39:28.7088647Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:365)
> 2020-05-12T11:39:28.7089328Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:273)
> 2020-05-12T11:39:28.7090106Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
> 2020-05-12T11:39:28.7090811Z at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:159)
> 2020-05-12T11:39:28.7091674Z at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:384)
> 2020-05-12T11:39:28.7102178Z at
>
[GitHub] [flink] zhijiangW merged pull request #12879: [FLINK-17636][tests] Fix unstable test SingleInputGateTest#testConcur…
zhijiangW merged pull request #12879: URL: https://github.com/apache/flink/pull/12879 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] zhijiangW merged pull request #12898: [FLINK-17636][tests] Fix unstable test SingleInputGateTest#testConcurrentReadStateAndProcessAndClose
zhijiangW merged pull request #12898: URL: https://github.com/apache/flink/pull/12898 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18599) Compile error when use windowAll and TumblingProcessingTimeWindows
[
https://issues.apache.org/jira/browse/FLINK-18599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
henvealf updated FLINK-18599:
-
Description:
Code:
{code:java}
import org.apache.commons.lang3.StringUtils
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import
org.apache.flink.streaming.api.windowing.assigners.{TumblingProcessingTimeWindows}
import org.apache.flink.streaming.api.windowing.time.Time
import org.apache.flink.streaming.api.scala._
val env = StreamExecutionEnvironment.getExecutionEnvironment
val stream = env.fromElements("a", "b", "c")
stream
.filter((str: String) => StringUtils.isNotEmpty(str))
.map( _ => 1)
.windowAll(TumblingProcessingTimeWindows.of(Time.seconds(5)))
.reduce((a1, a2) => a1 + a2)
.print()
{code}
Compile failed:
{code:java}
error: type mismatch;
found :
org.apache.flink.streaming.api.windowing.assigners.TumblingProcessingTimeWindows
required: org.apache.flink.streaming.api.windowing.assigners.WindowAssigner[_
>: Int, ?]
Note: Object <: Any (and
org.apache.flink.streaming.api.windowing.assigners.TumblingProcessingTimeWindows
<:
org.apache.flink.streaming.api.windowing.assigners.WindowAssigner[Object,org.apache.flink.streaming.api.windowing.windows.TimeWindow]),
but Java-defined class WindowAssigner is invariant in type T.
You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10)
.windowAll(TumblingProcessingTimeWindows.of(Time.seconds(5)))
^
one error found
{code}
What went wrong?
Scala version: 2.11
Flink version: 1.11
Thanks.
was:
Code:
{code:java}
import org.apache.commons.lang3.StringUtils
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import
org.apache.flink.streaming.api.windowing.assigners.{TumblingProcessingTimeWindows}
import org.apache.flink.streaming.api.windowing.time.Time
import org.apache.flink.streaming.api.scala._
val env = StreamExecutionEnvironment.getExecutionEnvironment
val stream = env.fromElements("a", "b", "c")
stream
.filter((str: String) => StringUtils.isNotEmpty(str))
.map( _ => 1)
.windowAll(TumblingProcessingTimeWindows.of(Time.seconds(5)))
.reduce((a1, a2) => a1 + a2)
.print()
{code}
Compile failed:
{code:java}
error: type mismatch;
found :
org.apache.flink.streaming.api.windowing.assigners.TumblingProcessingTimeWindows
required: org.apache.flink.streaming.api.windowing.assigners.WindowAssigner[_
>: Int, ?]
Note: Object <: Any (and
org.apache.flink.streaming.api.windowing.assigners.TumblingProcessingTimeWindows
<:
org.apache.flink.streaming.api.windowing.assigners.WindowAssigner[Object,org.apache.flink.streaming.api.windowing.windows.TimeWindow]),
but Java-defined class WindowAssigner is invariant in type T.
You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10)
.windowAll(TumblingProcessingTimeWindows.of(Time.seconds(5)))
^
one error found
{code}
What went wrong?
Thanks.
> Compile error when use windowAll and TumblingProcessingTimeWindows
> --
>
> Key: FLINK-18599
> URL: https://issues.apache.org/jira/browse/FLINK-18599
> Project: Flink
> Issue Type: Bug
> Components: API / DataStream
>Affects Versions: 1.11.0
>Reporter: henvealf
>Priority: Major
>
> Code:
> {code:java}
> import org.apache.commons.lang3.StringUtils
> import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
> import
> org.apache.flink.streaming.api.windowing.assigners.{TumblingProcessingTimeWindows}
> import org.apache.flink.streaming.api.windowing.time.Time
> import org.apache.flink.streaming.api.scala._
>
> val env = StreamExecutionEnvironment.getExecutionEnvironment
> val stream = env.fromElements("a", "b", "c")
> stream
> .filter((str: String) => StringUtils.isNotEmpty(str))
> .map( _ => 1)
> .windowAll(TumblingProcessingTimeWindows.of(Time.seconds(5)))
> .reduce((a1, a2) => a1 + a2)
> .print()
> {code}
> Compile failed:
> {code:java}
> error: type mismatch;
> found :
> org.apache.flink.streaming.api.windowing.assigners.TumblingProcessingTimeWindows
> required:
> org.apache.flink.streaming.api.windowing.assigners.WindowAssigner[_ >: Int, ?]
> Note: Object <: Any (and
> org.apache.flink.streaming.api.windowing.assigners.TumblingProcessingTimeWindows
> <:
> org.apache.flink.streaming.api.windowing.assigners.WindowAssigner[Object,org.apache.flink.streaming.api.windowing.windows.TimeWindow]),
> but Java-defined class WindowAssigner is invariant in type T.
> You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10)
>
[jira] [Updated] (FLINK-18600) Kerberized YARN per-job on Docker test failed to download JDK 8u251
[
https://issues.apache.org/jira/browse/FLINK-18600?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dian Fu updated FLINK-18600:
Description:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4514=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
{code}
+ mkdir -p /usr/java/default
+ curl -Ls
https://download.oracle.com/otn-pub/java/jdk/8u251-b08/3d5a2bb8f8d4428bbe94aed7ec7ae784/jdk-8u251-linux-x64.tar.gz
-H Cookie: oraclelicense=accept-securebackup-cookie
+ tar --strip-components=1 -xz -C /usr/java/default/
gzip: stdin: not in gzip format
tar: Child returned status 1
{code}
> Kerberized YARN per-job on Docker test failed to download JDK 8u251
> ---
>
> Key: FLINK-18600
> URL: https://issues.apache.org/jira/browse/FLINK-18600
> Project: Flink
> Issue Type: Bug
> Components: Deployment / YARN, Tests
>Affects Versions: 1.11.0, 1.12.0
>Reporter: Dian Fu
>Priority: Critical
> Fix For: 1.12.0, 1.11.2
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=4514=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> + mkdir -p /usr/java/default
> + curl -Ls
> https://download.oracle.com/otn-pub/java/jdk/8u251-b08/3d5a2bb8f8d4428bbe94aed7ec7ae784/jdk-8u251-linux-x64.tar.gz
> -H Cookie: oraclelicense=accept-securebackup-cookie
> + tar --strip-components=1 -xz -C /usr/java/default/
> gzip: stdin: not in gzip format
> tar: Child returned status 1
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (FLINK-18600) Kerberized YARN per-job on Docker test failed to download JDK 8u251
Dian Fu created FLINK-18600: --- Summary: Kerberized YARN per-job on Docker test failed to download JDK 8u251 Key: FLINK-18600 URL: https://issues.apache.org/jira/browse/FLINK-18600 Project: Flink Issue Type: Bug Components: Deployment / YARN, Tests Affects Versions: 1.11.0, 1.12.0 Reporter: Dian Fu Fix For: 1.12.0, 1.11.2 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #12902: [FLINK-18415][python] Support TableResult#collect in the Python Table…
flinkbot commented on pull request #12902: URL: https://github.com/apache/flink/pull/12902#issuecomment-658509076 ## CI report: * 802dfd4a0a2fb3e1b6e9a31fbc850c18f00c5c2b UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12866: [FLINK-17425][blink-planner] supportsFilterPushDown rule in DynamicSource.
flinkbot edited a comment on pull request #12866: URL: https://github.com/apache/flink/pull/12866#issuecomment-656576219 ## CI report: * 02c81cff07ef76dc20a7c618112cc7a172856720 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4497) * 5eb302884e390b6099859abcd7955e7f1c4e034e UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-18599) Compile error when use windowAll and TumblingProcessingTimeWindows
henvealf created FLINK-18599:
Summary: Compile error when use windowAll and
TumblingProcessingTimeWindows
Key: FLINK-18599
URL: https://issues.apache.org/jira/browse/FLINK-18599
Project: Flink
Issue Type: Bug
Components: API / DataStream
Affects Versions: 1.11.0
Reporter: henvealf
Code:
{code:java}
import org.apache.commons.lang3.StringUtils
import org.apache.flink.streaming.api.scala.StreamExecutionEnvironment
import
org.apache.flink.streaming.api.windowing.assigners.{TumblingProcessingTimeWindows}
import org.apache.flink.streaming.api.windowing.time.Time
import org.apache.flink.streaming.api.scala._
val env = StreamExecutionEnvironment.getExecutionEnvironment
val stream = env.fromElements("a", "b", "c")
stream
.filter((str: String) => StringUtils.isNotEmpty(str))
.map( _ => 1)
.windowAll(TumblingProcessingTimeWindows.of(Time.seconds(5)))
.reduce((a1, a2) => a1 + a2)
.print()
{code}
Compile failed:
{code:java}
error: type mismatch;
found :
org.apache.flink.streaming.api.windowing.assigners.TumblingProcessingTimeWindows
required: org.apache.flink.streaming.api.windowing.assigners.WindowAssigner[_
>: Int, ?]
Note: Object <: Any (and
org.apache.flink.streaming.api.windowing.assigners.TumblingProcessingTimeWindows
<:
org.apache.flink.streaming.api.windowing.assigners.WindowAssigner[Object,org.apache.flink.streaming.api.windowing.windows.TimeWindow]),
but Java-defined class WindowAssigner is invariant in type T.
You may wish to investigate a wildcard type such as `_ <: Any`. (SLS 3.2.10)
.windowAll(TumblingProcessingTimeWindows.of(Time.seconds(5)))
^
one error found
{code}
What went wrong?
Thanks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] liuyongvs commented on pull request #12866: [FLINK-17425][blink-planner] supportsFilterPushDown rule in DynamicSource.
liuyongvs commented on pull request #12866: URL: https://github.com/apache/flink/pull/12866#issuecomment-658506250 Hi @godfreyhe . Yes, i find some type is not Comparable, such as Period int getValueAs(Class clazz). Thanks for you review in great details. And I learned a lot This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (FLINK-18598) Add instructions for asynchronous execute in PyFlink doc
Huang Xingbo created FLINK-18598: Summary: Add instructions for asynchronous execute in PyFlink doc Key: FLINK-18598 URL: https://issues.apache.org/jira/browse/FLINK-18598 Project: Flink Issue Type: Improvement Components: API / Python, Documentation Affects Versions: 1.11.0 Reporter: Huang Xingbo Add instructions for asynchronous execute in PyFlink doc -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18597) elasticsearch sql insert function cannot set documentid
[ https://issues.apache.org/jira/browse/FLINK-18597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157811#comment-17157811 ] Yangze Guo commented on FLINK-18597: I think it is fixed by FLINK-18583. Could you take a look? If that is the case, please close this ticket. > elasticsearch sql insert function cannot set documentid > --- > > Key: FLINK-18597 > URL: https://issues.apache.org/jira/browse/FLINK-18597 > Project: Flink > Issue Type: Test >Affects Versions: 1.11.0 >Reporter: xy >Priority: Critical > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-18597) elasticsearch sql insert function cannot set documentid
xy created FLINK-18597: -- Summary: elasticsearch sql insert function cannot set documentid Key: FLINK-18597 URL: https://issues.apache.org/jira/browse/FLINK-18597 Project: Flink Issue Type: Test Affects Versions: 1.11.0 Reporter: xy -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #12902: [FLINK-18415][python] Support TableResult#collect in the Python Table…
flinkbot commented on pull request #12902: URL: https://github.com/apache/flink/pull/12902#issuecomment-658497220 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 802dfd4a0a2fb3e1b6e9a31fbc850c18f00c5c2b (Wed Jul 15 01:44:55 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18415) Support TableResult#collect in the Python Table API to align with the Java Table API
[ https://issues.apache.org/jira/browse/FLINK-18415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-18415: --- Labels: pull-request-available (was: ) > Support TableResult#collect in the Python Table API to align with the Java > Table API > > > Key: FLINK-18415 > URL: https://issues.apache.org/jira/browse/FLINK-18415 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Dian Fu >Assignee: Shuiqiang Chen >Priority: Major > Labels: pull-request-available > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] shuiqiangchen opened a new pull request #12902: [FLINK-18415][python] Support TableResult#collect in the Python Table…
shuiqiangchen opened a new pull request #12902: URL: https://github.com/apache/flink/pull/12902 ## What is the purpose of the change *Support TableResult#collect in the Python Table API to align with the Java Table API* ## Brief change log - * Add collect() method for TableResult* - *add a new class named ClosableIterator to iterate the result of TableResult* - *add a type util to get/transform data between java row object and python object* ## Verifying this change This change added tests and can be verified as follows: - *Add an integration test to test the collect() in test_table_api.py* ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): ( no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: ( no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? ( no) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12901: [FLINK-17285] [API / Python, chinese-translation] Translate "Python Table API" page into Chinese (part 1)
flinkbot edited a comment on pull request #12901: URL: https://github.com/apache/flink/pull/12901#issuecomment-658445990 ## CI report: * 221cb71c264d05e43eb77b5e11d9fb4c8834e7da Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4516) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12901: [FLINK-17285] [API / Python, chinese-translation] Translate "Python Table API" page into Chinese (part 1)
flinkbot commented on pull request #12901: URL: https://github.com/apache/flink/pull/12901#issuecomment-658445990 ## CI report: * 221cb71c264d05e43eb77b5e11d9fb4c8834e7da UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12901: [FLINK-17285] [API / Python, chinese-translation] Translate "Python Table API" page into Chinese (part 1)
flinkbot commented on pull request #12901: URL: https://github.com/apache/flink/pull/12901#issuecomment-658438478 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 221cb71c264d05e43eb77b5e11d9fb4c8834e7da (Tue Jul 14 22:09:45 UTC 2020) ✅no warnings Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-17285) Translate "Python Table API" page into Chinese
[ https://issues.apache.org/jira/browse/FLINK-17285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-17285: --- Labels: pull-request-available (was: ) > Translate "Python Table API" page into Chinese > --- > > Key: FLINK-17285 > URL: https://issues.apache.org/jira/browse/FLINK-17285 > Project: Flink > Issue Type: Improvement > Components: API / Python, chinese-translation >Reporter: Zixuan Rao >Assignee: Zixuan Rao >Priority: Major > Labels: pull-request-available > Original Estimate: 168h > Remaining Estimate: 168h > > This issue is to add Chinese translation to Python Table API. Untranslated > paragraphs currently in ```*-zh.md``` will be replaced with Chinese > translations. Translation will be done with ```Google Translate``` and manual > corrections. I plan to submit a pull request this week. Thanks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] billyrrr opened a new pull request #12901: [FLINK-17285] [API / Python, chinese-translation] Translate "Python Table API" page into Chinese (part 1)
billyrrr opened a new pull request #12901: URL: https://github.com/apache/flink/pull/12901 ## What is the purpose of the change This change is part of many that will translate `*.zh.md` docs for Python Table API to Chinese Simplified. Untranslated paragraphs currently in ```*-zh.md``` will be replaced with Chinese translations. Translation will be done with ```Google Translate``` and manual corrections. NOTE: this change only changes part of the files, and intends to receive feedbacks. ## Brief change log Translate files: - Improve docs/dev/table/python/common_questions.zh.md - Translate docs/dev/table/python/index.zh.md - Translate docs/dev/table/python/python_udfs.zh.md ## Verifying this change This change is already covered by existing tests, such as document build test (if exists). NOTE: some files are translated from `*.md` and others from `*.zh.md`. Please verify that the translated text contains the latest changes, even if this commit is based on updated `*.md` or `*.zh.md` files. NOTE: custom headers should be checked for compatibility and are not applied every where. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? N/A This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12900: [FLINK-18537][table] Remove RowDataTypeInfo
flinkbot edited a comment on pull request #12900: URL: https://github.com/apache/flink/pull/12900#issuecomment-658213136 ## CI report: * ceb895be265bd445b9d36bfc52cf8018a508c333 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4508) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
flinkbot edited a comment on pull request #12846: URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676 ## CI report: * 8c8f594927a2b72c23f8619c1932219585755054 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4509) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18573) InfluxDB reporter cannot be loaded as plugin
[ https://issues.apache.org/jira/browse/FLINK-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-18573: - Description: The service entry is in the wrong directory (service vs service*s* ) It causes the error: 2020-07-13 09:08:46.146 [main] WARN org.apache.flink.runtime.metrics.ReporterSetup - The reporter factory (org.apache.flink.metrics.influxdb.InfluxdbReporterFactory) could not be found for reporter influxdb. Available factories: [org.apache.flink.metrics.slf4j.Slf4jReporterFactory, org.apache.flink.metrics.datadog.DatadogHttpReporterFactory, org.apache.flink.metrics.graphite.GraphiteReporterFactory, org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterFactory, org.apache.flink.metrics.statsd.StatsDReporterFactory, org.apache.flink.metrics.prometheus.PrometheusReporterFactory, org.apache.flink.metrics.jmx.JMXReporterFactory]. 2020-07-13 09:08:46.149 [main] INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - Periodically reporting metrics in intervals of 60 SECONDS for reporter slf4j of type org.apache.flink.metrics.slf4j.Slf4jReporter. was: The service entry is in the wrong directory (service vs service*s*) It causes the error: 2020-07-13 09:08:46.146 [main] WARN org.apache.flink.runtime.metrics.ReporterSetup - The reporter factory (org.apache.flink.metrics.influxdb.InfluxdbReporterFactory) could not be found for reporter influxdb. Available factories: [org.apache.flink.metrics.slf4j.Slf4jReporterFactory, org.apache.flink.metrics.datadog.DatadogHttpReporterFactory, org.apache.flink.metrics.graphite.GraphiteReporterFactory, org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterFactory, org.apache.flink.metrics.statsd.StatsDReporterFactory, org.apache.flink.metrics.prometheus.PrometheusReporterFactory, org.apache.flink.metrics.jmx.JMXReporterFactory]. 2020-07-13 09:08:46.149 [main] INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - Periodically reporting metrics in intervals of 60 SECONDS for reporter slf4j of type org.apache.flink.metrics.slf4j.Slf4jReporter. > InfluxDB reporter cannot be loaded as plugin > > > Key: FLINK-18573 > URL: https://issues.apache.org/jira/browse/FLINK-18573 > Project: Flink > Issue Type: Bug > Components: Runtime / Metrics >Affects Versions: 1.11.0 >Reporter: zhangyunyun >Assignee: Yun Tang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.12.0, 1.11.1 > > > The service entry is in the wrong directory (service vs service*s* ) > It causes the error: > > 2020-07-13 09:08:46.146 [main] WARN > org.apache.flink.runtime.metrics.ReporterSetup - The reporter factory > (org.apache.flink.metrics.influxdb.InfluxdbReporterFactory) could not be > found for reporter influxdb. Available factories: > [org.apache.flink.metrics.slf4j.Slf4jReporterFactory, > org.apache.flink.metrics.datadog.DatadogHttpReporterFactory, > org.apache.flink.metrics.graphite.GraphiteReporterFactory, > org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterFactory, > org.apache.flink.metrics.statsd.StatsDReporterFactory, > org.apache.flink.metrics.prometheus.PrometheusReporterFactory, > org.apache.flink.metrics.jmx.JMXReporterFactory]. > 2020-07-13 09:08:46.149 [main] INFO > org.apache.flink.runtime.metrics.MetricRegistryImpl - Periodically reporting > metrics in intervals of 60 SECONDS for reporter slf4j of type > org.apache.flink.metrics.slf4j.Slf4jReporter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18573) InfluxDB reporter cannot be loaded as plugin
[ https://issues.apache.org/jira/browse/FLINK-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-18573: - Description: The service entry is in the wrong directory (service vs service*s*) It causes the error: 2020-07-13 09:08:46.146 [main] WARN org.apache.flink.runtime.metrics.ReporterSetup - The reporter factory (org.apache.flink.metrics.influxdb.InfluxdbReporterFactory) could not be found for reporter influxdb. Available factories: [org.apache.flink.metrics.slf4j.Slf4jReporterFactory, org.apache.flink.metrics.datadog.DatadogHttpReporterFactory, org.apache.flink.metrics.graphite.GraphiteReporterFactory, org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterFactory, org.apache.flink.metrics.statsd.StatsDReporterFactory, org.apache.flink.metrics.prometheus.PrometheusReporterFactory, org.apache.flink.metrics.jmx.JMXReporterFactory]. 2020-07-13 09:08:46.149 [main] INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - Periodically reporting metrics in intervals of 60 SECONDS for reporter slf4j of type org.apache.flink.metrics.slf4j.Slf4jReporter. was: It causes the error: 2020-07-13 09:08:46.146 [main] WARN org.apache.flink.runtime.metrics.ReporterSetup - The reporter factory (org.apache.flink.metrics.influxdb.InfluxdbReporterFactory) could not be found for reporter influxdb. Available factories: [org.apache.flink.metrics.slf4j.Slf4jReporterFactory, org.apache.flink.metrics.datadog.DatadogHttpReporterFactory, org.apache.flink.metrics.graphite.GraphiteReporterFactory, org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterFactory, org.apache.flink.metrics.statsd.StatsDReporterFactory, org.apache.flink.metrics.prometheus.PrometheusReporterFactory, org.apache.flink.metrics.jmx.JMXReporterFactory]. 2020-07-13 09:08:46.149 [main] INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - Periodically reporting metrics in intervals of 60 SECONDS for reporter slf4j of type org.apache.flink.metrics.slf4j.Slf4jReporter. > InfluxDB reporter cannot be loaded as plugin > > > Key: FLINK-18573 > URL: https://issues.apache.org/jira/browse/FLINK-18573 > Project: Flink > Issue Type: Bug > Components: Runtime / Metrics >Affects Versions: 1.11.0 >Reporter: zhangyunyun >Assignee: Yun Tang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.12.0, 1.11.1 > > > The service entry is in the wrong directory (service vs service*s*) > It causes the error: > > 2020-07-13 09:08:46.146 [main] WARN > org.apache.flink.runtime.metrics.ReporterSetup - The reporter factory > (org.apache.flink.metrics.influxdb.InfluxdbReporterFactory) could not be > found for reporter influxdb. Available factories: > [org.apache.flink.metrics.slf4j.Slf4jReporterFactory, > org.apache.flink.metrics.datadog.DatadogHttpReporterFactory, > org.apache.flink.metrics.graphite.GraphiteReporterFactory, > org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterFactory, > org.apache.flink.metrics.statsd.StatsDReporterFactory, > org.apache.flink.metrics.prometheus.PrometheusReporterFactory, > org.apache.flink.metrics.jmx.JMXReporterFactory]. > 2020-07-13 09:08:46.149 [main] INFO > org.apache.flink.runtime.metrics.MetricRegistryImpl - Periodically reporting > metrics in intervals of 60 SECONDS for reporter slf4j of type > org.apache.flink.metrics.slf4j.Slf4jReporter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18573) InfluxDB reporter cannot be loaded as plugin
[ https://issues.apache.org/jira/browse/FLINK-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-18573: - Description: The service entry is in the wrong directory (service vs services) It causes the error: 2020-07-13 09:08:46.146 [main] WARN org.apache.flink.runtime.metrics.ReporterSetup - The reporter factory (org.apache.flink.metrics.influxdb.InfluxdbReporterFactory) could not be found for reporter influxdb. Available factories: [org.apache.flink.metrics.slf4j.Slf4jReporterFactory, org.apache.flink.metrics.datadog.DatadogHttpReporterFactory, org.apache.flink.metrics.graphite.GraphiteReporterFactory, org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterFactory, org.apache.flink.metrics.statsd.StatsDReporterFactory, org.apache.flink.metrics.prometheus.PrometheusReporterFactory, org.apache.flink.metrics.jmx.JMXReporterFactory]. 2020-07-13 09:08:46.149 [main] INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - Periodically reporting metrics in intervals of 60 SECONDS for reporter slf4j of type org.apache.flink.metrics.slf4j.Slf4jReporter. was: The service entry is in the wrong directory (service vs service*s* ) It causes the error: 2020-07-13 09:08:46.146 [main] WARN org.apache.flink.runtime.metrics.ReporterSetup - The reporter factory (org.apache.flink.metrics.influxdb.InfluxdbReporterFactory) could not be found for reporter influxdb. Available factories: [org.apache.flink.metrics.slf4j.Slf4jReporterFactory, org.apache.flink.metrics.datadog.DatadogHttpReporterFactory, org.apache.flink.metrics.graphite.GraphiteReporterFactory, org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterFactory, org.apache.flink.metrics.statsd.StatsDReporterFactory, org.apache.flink.metrics.prometheus.PrometheusReporterFactory, org.apache.flink.metrics.jmx.JMXReporterFactory]. 2020-07-13 09:08:46.149 [main] INFO org.apache.flink.runtime.metrics.MetricRegistryImpl - Periodically reporting metrics in intervals of 60 SECONDS for reporter slf4j of type org.apache.flink.metrics.slf4j.Slf4jReporter. > InfluxDB reporter cannot be loaded as plugin > > > Key: FLINK-18573 > URL: https://issues.apache.org/jira/browse/FLINK-18573 > Project: Flink > Issue Type: Bug > Components: Runtime / Metrics >Affects Versions: 1.11.0 >Reporter: zhangyunyun >Assignee: Yun Tang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.12.0, 1.11.1 > > > The service entry is in the wrong directory (service vs services) > It causes the error: > > 2020-07-13 09:08:46.146 [main] WARN > org.apache.flink.runtime.metrics.ReporterSetup - The reporter factory > (org.apache.flink.metrics.influxdb.InfluxdbReporterFactory) could not be > found for reporter influxdb. Available factories: > [org.apache.flink.metrics.slf4j.Slf4jReporterFactory, > org.apache.flink.metrics.datadog.DatadogHttpReporterFactory, > org.apache.flink.metrics.graphite.GraphiteReporterFactory, > org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterFactory, > org.apache.flink.metrics.statsd.StatsDReporterFactory, > org.apache.flink.metrics.prometheus.PrometheusReporterFactory, > org.apache.flink.metrics.jmx.JMXReporterFactory]. > 2020-07-13 09:08:46.149 [main] INFO > org.apache.flink.runtime.metrics.MetricRegistryImpl - Periodically reporting > metrics in intervals of 60 SECONDS for reporter slf4j of type > org.apache.flink.metrics.slf4j.Slf4jReporter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-18573) InfluxDB reporter cannot be loaded as plugin
[ https://issues.apache.org/jira/browse/FLINK-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-18573: - Summary: InfluxDB reporter cannot be loaded as plugin (was: plugins/metrics-influx/flink-metrics-influxdb-1.11.0.jar META-INF has no dir named "services" , but "service") > InfluxDB reporter cannot be loaded as plugin > > > Key: FLINK-18573 > URL: https://issues.apache.org/jira/browse/FLINK-18573 > Project: Flink > Issue Type: Bug > Components: Runtime / Metrics >Affects Versions: 1.11.0 >Reporter: zhangyunyun >Assignee: Yun Tang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.12.0, 1.11.1 > > > It causes the error: > > 2020-07-13 09:08:46.146 [main] WARN > org.apache.flink.runtime.metrics.ReporterSetup - The reporter factory > (org.apache.flink.metrics.influxdb.InfluxdbReporterFactory) could not be > found for reporter influxdb. Available factories: > [org.apache.flink.metrics.slf4j.Slf4jReporterFactory, > org.apache.flink.metrics.datadog.DatadogHttpReporterFactory, > org.apache.flink.metrics.graphite.GraphiteReporterFactory, > org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterFactory, > org.apache.flink.metrics.statsd.StatsDReporterFactory, > org.apache.flink.metrics.prometheus.PrometheusReporterFactory, > org.apache.flink.metrics.jmx.JMXReporterFactory]. > 2020-07-13 09:08:46.149 [main] INFO > org.apache.flink.runtime.metrics.MetricRegistryImpl - Periodically reporting > metrics in intervals of 60 SECONDS for reporter slf4j of type > org.apache.flink.metrics.slf4j.Slf4jReporter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-18573) plugins/metrics-influx/flink-metrics-influxdb-1.11.0.jar META-INF has no dir named "services" , but "service"
[ https://issues.apache.org/jira/browse/FLINK-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler closed FLINK-18573. Resolution: Fixed master: 49d3cabb8ae53900138fc1a2f5178dbec46ce378 c3e0d450c82ea8246a7f0abab3a1f86a0f53de7d 1.11: 5c73b0c4a1b50d1b2b3a6a3a3637677a143e598f 60cf377b01595f257992d8d30d1b236d0f6f0103 > plugins/metrics-influx/flink-metrics-influxdb-1.11.0.jar META-INF has no dir > named "services" , but "service" > - > > Key: FLINK-18573 > URL: https://issues.apache.org/jira/browse/FLINK-18573 > Project: Flink > Issue Type: Bug > Components: Runtime / Metrics >Affects Versions: 1.11.0 >Reporter: zhangyunyun >Assignee: Yun Tang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.12.0, 1.11.1 > > > It causes the error: > > 2020-07-13 09:08:46.146 [main] WARN > org.apache.flink.runtime.metrics.ReporterSetup - The reporter factory > (org.apache.flink.metrics.influxdb.InfluxdbReporterFactory) could not be > found for reporter influxdb. Available factories: > [org.apache.flink.metrics.slf4j.Slf4jReporterFactory, > org.apache.flink.metrics.datadog.DatadogHttpReporterFactory, > org.apache.flink.metrics.graphite.GraphiteReporterFactory, > org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterFactory, > org.apache.flink.metrics.statsd.StatsDReporterFactory, > org.apache.flink.metrics.prometheus.PrometheusReporterFactory, > org.apache.flink.metrics.jmx.JMXReporterFactory]. > 2020-07-13 09:08:46.149 [main] INFO > org.apache.flink.runtime.metrics.MetricRegistryImpl - Periodically reporting > metrics in intervals of 60 SECONDS for reporter slf4j of type > org.apache.flink.metrics.slf4j.Slf4jReporter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] zentol merged pull request #12883: [FLINK-18573][metrics] Ensure metric reporters could be loaded via SPI
zentol merged pull request #12883: URL: https://github.com/apache/flink/pull/12883 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18573) plugins/metrics-influx/flink-metrics-influxdb-1.11.0.jar META-INF has no dir named "services" , but "service"
[ https://issues.apache.org/jira/browse/FLINK-18573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chesnay Schepler updated FLINK-18573: - Fix Version/s: 1.12.0 > plugins/metrics-influx/flink-metrics-influxdb-1.11.0.jar META-INF has no dir > named "services" , but "service" > - > > Key: FLINK-18573 > URL: https://issues.apache.org/jira/browse/FLINK-18573 > Project: Flink > Issue Type: Bug > Components: Runtime / Metrics >Affects Versions: 1.11.0 >Reporter: zhangyunyun >Assignee: Yun Tang >Priority: Blocker > Labels: pull-request-available > Fix For: 1.12.0, 1.11.1 > > > It causes the error: > > 2020-07-13 09:08:46.146 [main] WARN > org.apache.flink.runtime.metrics.ReporterSetup - The reporter factory > (org.apache.flink.metrics.influxdb.InfluxdbReporterFactory) could not be > found for reporter influxdb. Available factories: > [org.apache.flink.metrics.slf4j.Slf4jReporterFactory, > org.apache.flink.metrics.datadog.DatadogHttpReporterFactory, > org.apache.flink.metrics.graphite.GraphiteReporterFactory, > org.apache.flink.metrics.prometheus.PrometheusPushGatewayReporterFactory, > org.apache.flink.metrics.statsd.StatsDReporterFactory, > org.apache.flink.metrics.prometheus.PrometheusReporterFactory, > org.apache.flink.metrics.jmx.JMXReporterFactory]. > 2020-07-13 09:08:46.149 [main] INFO > org.apache.flink.runtime.metrics.MetricRegistryImpl - Periodically reporting > metrics in intervals of 60 SECONDS for reporter slf4j of type > org.apache.flink.metrics.slf4j.Slf4jReporter. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Comment Edited] (FLINK-11143) AskTimeoutException is thrown during job submission and completion
[
https://issues.apache.org/jira/browse/FLINK-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156363#comment-17156363
]
Steven Zhen Wu edited comment on FLINK-11143 at 7/14/20, 5:38 PM:
--
[~trohrmann] I am seeing a similar problem *when trying unaligned checkpoint
with 1.11.0*. The Flink job actually started fine. We didn't see this
AskTimeoutException thrown during job submission in without unaligned
checkpoint (1.10 or 1.11).
Some more context about the app
* a large-state stream join app (a few TBs)
* parallelism 1,440
* number of containers: 180
* Cores per container: 12
* TM_TASK_SLOTS: 8
* akka.ask.timeout: 120 s
* heartbeat.timeout: 12
* web.timeout: 6 (also tried larger values like 300,000 or 600,000 without
any difference)
I will send you the log files (with DEBUG level) in an email offline. Thanks a
lot for your help in advance!
{code:java}
\"errors\":[\"Internal server error.\",\" AskTimeoutException is thrown during job submission and completion
> --
>
> Key: FLINK-11143
> URL: https://issues.apache.org/jira/browse/FLINK-11143
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.6.2, 1.10.0
>Reporter: Alex Vinnik
>Priority: Critical
> Attachments: flink-job-timeline.PNG
>
>
> For more details please see the thread
> [http://mail-archives.apache.org/mod_mbox/flink-user/201812.mbox/%3cc2fb26f9-1410-4333-80f4-34807481b...@gmail.com%3E]
> On submission
> 2018-12-12 02:28:31 ERROR JobsOverviewHandler:92 - Implementation error:
> Unhandled exception.
> akka.pattern.AskTimeoutException: Ask timed out on
> [Actor[akka://flink/user/dispatcher#225683351|#225683351]] after [1 ms].
> Sender[null] sent message of type
> "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
> at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
> at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
> at
> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
> at
> scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
> at
> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
> at
> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
> at java.lang.Thread.run(Thread.java:748)
>
> On completion
>
> {"errors":["Internal server error."," side:\njava.util.concurrent.CompletionException:
> akka.pattern.AskTimeoutException: Ask timed out on
> [Actor[akka://flink/user/dispatcher#105638574]] after [1 ms].
> Sender[null] sent message of type
> \"org.apache.flink.runtime.rpc.messages.LocalFencedMessage\".
> at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> at
> org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:772)
> at akka.dispatch.OnComplete.internal(Future.scala:258)
> at akka.dispatch.OnComplete.internal(Future.scala:256)
> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> at
> org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
> at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
> at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
> at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
> at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
> at
> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
> at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
> at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
> at
> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
> at
>
[jira] [Comment Edited] (FLINK-11143) AskTimeoutException is thrown during job submission and completion
[
https://issues.apache.org/jira/browse/FLINK-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156363#comment-17156363
]
Steven Zhen Wu edited comment on FLINK-11143 at 7/14/20, 5:38 PM:
--
[~trohrmann] I am seeing a similar problem *when trying unaligned checkpoint
with 1.11.0*. The Flink job actually started fine. We didn't see this
AskTimeoutException thrown during job submission in without unaligned
checkpoint (1.10 or 1.11).
Some more context about the app
* a large-state stream join app (a few TBs)
* parallelism 1,440
* number of containers: 180
* Cores per container: 12
* TM_TASK_SLOTS: 8
* akka.ask.timeout: 120 s
* heartbeat.timeout: 12
* web.timeout: 6 (also tried larger values like 300,000 or 600,000 without
any difference)
I will send you the log files (with DEBUG level) in an email offline. Thanks a
lot for your help in advance!
{code:java}
\"errors\":[\"Internal server error.\",\" AskTimeoutException is thrown during job submission and completion
> --
>
> Key: FLINK-11143
> URL: https://issues.apache.org/jira/browse/FLINK-11143
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.6.2, 1.10.0
>Reporter: Alex Vinnik
>Priority: Critical
> Attachments: flink-job-timeline.PNG
>
>
> For more details please see the thread
> [http://mail-archives.apache.org/mod_mbox/flink-user/201812.mbox/%3cc2fb26f9-1410-4333-80f4-34807481b...@gmail.com%3E]
> On submission
> 2018-12-12 02:28:31 ERROR JobsOverviewHandler:92 - Implementation error:
> Unhandled exception.
> akka.pattern.AskTimeoutException: Ask timed out on
> [Actor[akka://flink/user/dispatcher#225683351|#225683351]] after [1 ms].
> Sender[null] sent message of type
> "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
> at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
> at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
> at
> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
> at
> scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
> at
> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
> at
> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
> at java.lang.Thread.run(Thread.java:748)
>
> On completion
>
> {"errors":["Internal server error."," side:\njava.util.concurrent.CompletionException:
> akka.pattern.AskTimeoutException: Ask timed out on
> [Actor[akka://flink/user/dispatcher#105638574]] after [1 ms].
> Sender[null] sent message of type
> \"org.apache.flink.runtime.rpc.messages.LocalFencedMessage\".
> at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> at
> org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:772)
> at akka.dispatch.OnComplete.internal(Future.scala:258)
> at akka.dispatch.OnComplete.internal(Future.scala:256)
> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> at
> org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
> at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
> at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
> at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
> at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
> at
> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
> at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
> at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
> at
> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
> at
>
[jira] [Comment Edited] (FLINK-11143) AskTimeoutException is thrown during job submission and completion
[
https://issues.apache.org/jira/browse/FLINK-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17156363#comment-17156363
]
Steven Zhen Wu edited comment on FLINK-11143 at 7/14/20, 5:37 PM:
--
[~trohrmann] I am seeing a similar problem *when trying unaligned checkpoint
with 1.11.0*. The Flink job actually started fine. We didn't see this
AskTimeoutException thrown during job submission in without unaligned
checkpoint (1.10 or 1.11).
Some more context about the app
* a large-state stream join app (a few TBs)
* parallelism 1,440
* number of containers: 180
* Cores per container: 12
* TM_TASK_SLOTS: 8
* akka.ask.timeout: 120 s
* heartbeat.timeout: 12
* web.timeout: 6 (also tried larger values like 300,000 or 600,000 without
any difference)
I will send you the log files (with DEBUG level) in an email offline. Thanks a
lot for your help in advance!
{code:java}
\"errors\":[\"Internal server error.\",\" AskTimeoutException is thrown during job submission and completion
> --
>
> Key: FLINK-11143
> URL: https://issues.apache.org/jira/browse/FLINK-11143
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.6.2, 1.10.0
>Reporter: Alex Vinnik
>Priority: Critical
> Attachments: flink-job-timeline.PNG
>
>
> For more details please see the thread
> [http://mail-archives.apache.org/mod_mbox/flink-user/201812.mbox/%3cc2fb26f9-1410-4333-80f4-34807481b...@gmail.com%3E]
> On submission
> 2018-12-12 02:28:31 ERROR JobsOverviewHandler:92 - Implementation error:
> Unhandled exception.
> akka.pattern.AskTimeoutException: Ask timed out on
> [Actor[akka://flink/user/dispatcher#225683351|#225683351]] after [1 ms].
> Sender[null] sent message of type
> "org.apache.flink.runtime.rpc.messages.LocalFencedMessage".
> at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:604)
> at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
> at
> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
> at
> scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
> at
> scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
> at
> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.executeBucket$1(LightArrayRevolverScheduler.scala:280)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.nextTick(LightArrayRevolverScheduler.scala:284)
> at
> akka.actor.LightArrayRevolverScheduler$$anon$4.run(LightArrayRevolverScheduler.scala:236)
> at java.lang.Thread.run(Thread.java:748)
>
> On completion
>
> {"errors":["Internal server error."," side:\njava.util.concurrent.CompletionException:
> akka.pattern.AskTimeoutException: Ask timed out on
> [Actor[akka://flink/user/dispatcher#105638574]] after [1 ms].
> Sender[null] sent message of type
> \"org.apache.flink.runtime.rpc.messages.LocalFencedMessage\".
> at
> java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
> at
> java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
> at java.util.concurrent.CompletableFuture.uniApply(CompletableFuture.java:593)
> at
> java.util.concurrent.CompletableFuture$UniApply.tryFire(CompletableFuture.java:577)
> at
> java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:474)
> at
> java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1977)
> at
> org.apache.flink.runtime.concurrent.FutureUtils$1.onComplete(FutureUtils.java:772)
> at akka.dispatch.OnComplete.internal(Future.scala:258)
> at akka.dispatch.OnComplete.internal(Future.scala:256)
> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:186)
> at akka.dispatch.japi$CallbackBridge.apply(Future.scala:183)
> at scala.concurrent.impl.CallbackRunnable.run(Promise.scala:36)
> at
> org.apache.flink.runtime.concurrent.Executors$DirectExecutionContext.execute(Executors.java:83)
> at scala.concurrent.impl.CallbackRunnable.executeWithValue(Promise.scala:44)
> at scala.concurrent.impl.Promise$DefaultPromise.tryComplete(Promise.scala:252)
> at akka.pattern.PromiseActorRef$$anonfun$1.apply$mcV$sp(AskSupport.scala:603)
> at akka.actor.Scheduler$$anon$4.run(Scheduler.scala:126)
> at
> scala.concurrent.Future$InternalCallbackExecutor$.unbatchedExecute(Future.scala:601)
> at scala.concurrent.BatchingExecutor$class.execute(BatchingExecutor.scala:109)
> at scala.concurrent.Future$InternalCallbackExecutor$.execute(Future.scala:599)
> at
> akka.actor.LightArrayRevolverScheduler$TaskHolder.executeTask(LightArrayRevolverScheduler.scala:329)
> at
>
[GitHub] [flink] flinkbot edited a comment on pull request #12883: [FLINK-18573][metrics] Ensure metric reporters could be loaded via SPI
flinkbot edited a comment on pull request #12883: URL: https://github.com/apache/flink/pull/12883#issuecomment-657495689 ## CI report: * 25d465768bdf87bd31e0ca9b5749344c35c919d5 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4504) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] twalthr commented on a change in pull request #12900: [FLINK-18537][table] Remove RowDataTypeInfo
twalthr commented on a change in pull request #12900:
URL: https://github.com/apache/flink/pull/12900#discussion_r454506847
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/DataTypeQueryable.java
##
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.types;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+/**
+ * Indicates that a {@link DataType} can be retrieved from a class that
implements this interfaces.
+ */
+@PublicEvolving
+public interface DataTypeQueryable {
Review comment:
I thought about this as well and decided for keeping `WrapperTypeInfo`
in the runtime package. The reason for this is to keep the serializers close to
this utility class in the same package. `WrapperTypeInfo` is not needed in
`table-common` because it is internal. Actually the constructor of
`WrappenTypeInfo` should even be private but this is not possible due to other
failing tests so far. We can mark `DataTypeQueryable` internal if that helps,
but the past around `TypeInformation` has shown that you need such a simple
interface over and over again. It will be more useful in the future once we
introduce a "converter type serializer" where the return data type is not of
internal data structure.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12898: [FLINK-17636][tests] Fix unstable test SingleInputGateTest#testConcurrentReadStateAndProcessAndClose
flinkbot edited a comment on pull request #12898: URL: https://github.com/apache/flink/pull/12898#issuecomment-658161964 ## CI report: * da4c38d2f9cff86990bef5bf357595971ef1cef9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4505) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
flinkbot edited a comment on pull request #12846: URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676 ## CI report: * fbd3e1559f4a80424cbade77ed3034977943bf0b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4463) * 8c8f594927a2b72c23f8619c1932219585755054 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4509) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
flinkbot edited a comment on pull request #12846: URL: https://github.com/apache/flink/pull/12846#issuecomment-654978676 ## CI report: * fbd3e1559f4a80424cbade77ed3034977943bf0b Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4463) * 8c8f594927a2b72c23f8619c1932219585755054 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] nielsbasjes edited a comment on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
nielsbasjes edited a comment on pull request #12846: URL: https://github.com/apache/flink/pull/12846#issuecomment-658267581 Notes about the current set of changes: - Updated the Google dependencies. All Google dependencies no longer specify any versions. So in all cases the inclusion of the `libraries-bom` is needed. At this point I chose to make it such that downstream projects must specify which `libraries-bom` version they need. Although this is an extra step for users I hope this will make updating to newer versions of the Google tools easier. - I fixed almost all cases of Google/Flink api's that have become deprecated over time. One clear exception: The use of the ListCheckpointed. This seemed to become too large ( a full rewrite of the handling of the acknowledgements ) so I think it should be a separate effort. - I replaced the use of the NoCredentials and NoCredentialsProvider with custom implementations because I ran into problems. See also my bug report towards Google https://github.com/googleapis/gax-java/issues/1148 - I added a full end-to-end test that both reads from and writes to PubSub. - I updated the tests to run in parallel 4 (instead of 1) which surfaced a small bug that only appears on "ending" streams (like we only have in test scenarios). Solution: Move the closing of the connection to a slightly later point. This also meant that the `BoundedStringDeserializer` which counted the number of messages could no longer be used. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] nielsbasjes commented on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
nielsbasjes commented on pull request #12846: URL: https://github.com/apache/flink/pull/12846#issuecomment-658270987 @Xeli @becketqin @rmetzger Since you were involved in reviewing this component when originally created I'm tagging you to kindly review these changes. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] nielsbasjes commented on pull request #12846: [FLINK-18448][pubsub] Update Google Cloud PubSub dependencies
nielsbasjes commented on pull request #12846: URL: https://github.com/apache/flink/pull/12846#issuecomment-658267581 Notes about the current set of changes: - Updated the Google dependencies. All Google dependencies no longer specify any versions. So in all cases the inclusion of the `libraries-bom` is needed. At this point I chose to make it such that downstream projects must specify which `libraries-bom` version they need. Although this is an extra step for users I hope this will make updating to newer versions of the Google tools easier. - I fixed almost all cases of Google/Flink api's that have become deprecated over time. One clear exception: The use of the ListCheckpointed. This seemed to become too large ( a full rewrite of the handling of the acknowledgements ) so I think it should be a separate effort. - I replaced the use of the NoCredentials and NoCredentialsProvider with custom implementations because I ran into problems. See also my bug report towards Google https://github.com/googleapis/gax-java/issues/1148 - I added a full end-to-end test that both reads from and writes to PubSub. - I updated the tests to run in parallel 4 (instead of 1) which surfaced a small bug that only appears on "ending" streams (like we only have in test scenarios). Solution: Move the closing of the connection to a slightly later point. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12883: [FLINK-18573][metrics] Ensure metric reporters could be loaded via SPI
flinkbot edited a comment on pull request #12883: URL: https://github.com/apache/flink/pull/12883#issuecomment-657495689 ## CI report: * 93616ee036d921efea80c8003e408d35339cc4ec Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4500) * 25d465768bdf87bd31e0ca9b5749344c35c919d5 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4504) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] dawidwys commented on a change in pull request #12900: [FLINK-18537][table] Remove RowDataTypeInfo
dawidwys commented on a change in pull request #12900:
URL: https://github.com/apache/flink/pull/12900#discussion_r454415036
##
File path:
flink-table/flink-table-common/src/main/java/org/apache/flink/table/types/DataTypeQueryable.java
##
@@ -0,0 +1,30 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.types;
+
+import org.apache.flink.annotation.PublicEvolving;
+
+/**
+ * Indicates that a {@link DataType} can be retrieved from a class that
implements this interfaces.
+ */
+@PublicEvolving
+public interface DataTypeQueryable {
Review comment:
Is this interface really necessary? Especially with `@PublicEvolving`
annotation? How are users supposed to use it? If I understand it correctly you
need it for internal operations. Moreover you need it because the
`WrapperTypeInfo` is in `blink-runtime`, right?
Can't we move the `WrapperTypeInfo` to the `table-common` instead? The
class itself has no runtime dependencies. Only the factory methods need some
runtime classes.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12900: [FLINK-18537][table] Remove RowDataTypeInfo
flinkbot edited a comment on pull request #12900: URL: https://github.com/apache/flink/pull/12900#issuecomment-658213136 ## CI report: * ceb895be265bd445b9d36bfc52cf8018a508c333 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4508) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-18588) hive ddl create table should support 'if not exists'
[
https://issues.apache.org/jira/browse/FLINK-18588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157430#comment-17157430
]
Jark Wu commented on FLINK-18588:
-
[~wtog] I CC'ed someone familiar with this part to review in the PR.
> hive ddl create table should support 'if not exists'
>
>
> Key: FLINK-18588
> URL: https://issues.apache.org/jira/browse/FLINK-18588
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.11.0
>Reporter: wangtong
>Priority: Minor
> Labels: pull-request-available
>
> FlinkHiveSqlParser cant parse sql 'create table if not exists tbl (x int)'
> due to 'if not exists'
> {code:java}
> // code placeholder
> java.lang.RuntimeException: Error while parsing SQL: create table if not
> exists tbl (x int)java.lang.RuntimeException: Error while parsing SQL: create
> table if not exists tbl (x int)
> at
> org.apache.calcite.sql.parser.SqlParserTest$TesterImpl.parseStmtAndHandleEx(SqlParserTest.java:8861)
> at
> org.apache.calcite.sql.parser.SqlParserTest$TesterImpl.check(SqlParserTest.java:8845)
> at
> org.apache.calcite.sql.parser.SqlParserTest$Sql.ok(SqlParserTest.java:9192)
> at
> org.apache.flink.sql.parser.hive.FlinkHiveSqlParserImplTest.testCreateTable(FlinkHiveSqlParserImplTest.java:177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363) at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137) at
> org.junit.runner.JUnitCore.run(JUnitCore.java:115) at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:40)
> at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Iterator.forEachRemaining(Iterator.java:116) at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
> at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:71)
> at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:229)
> at
> org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:197)
> at
> org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:211)
> at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:191)
> at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
> at
> com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:69)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
> at
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
> at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)Caused by:
> org.apache.calcite.sql.parser.SqlParseException: Encountered "not" at line 1,
> column 17.Was expecting one of:
[GitHub] [flink] wuchong commented on pull request #12888: [FLINK-18588] hive ddl create table support 'if not exists'
wuchong commented on pull request #12888: URL: https://github.com/apache/flink/pull/12888#issuecomment-658219194 cc @lirui-apache @JingsongLi This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12900: [FLINK-18537][table] Remove RowDataTypeInfo
flinkbot commented on pull request #12900: URL: https://github.com/apache/flink/pull/12900#issuecomment-658213136 ## CI report: * ceb895be265bd445b9d36bfc52cf8018a508c333 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #12899: [FLINK-16699][k8s] Support accessing secured services via K8s secrets
flinkbot edited a comment on pull request #12899: URL: https://github.com/apache/flink/pull/12899#issuecomment-658162074 ## CI report: * 77f95464c6036283d77d6e81e2bd21ab27f29d54 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4506) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #12900: [FLINK-18537][table] Remove RowDataTypeInfo
flinkbot commented on pull request #12900: URL: https://github.com/apache/flink/pull/12900#issuecomment-658202311 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit ceb895be265bd445b9d36bfc52cf8018a508c333 (Tue Jul 14 14:09:33 UTC 2020) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] twalthr commented on pull request #12900: [FLINK-18537][table] Remove RowDataTypeInfo
twalthr commented on pull request #12900: URL: https://github.com/apache/flink/pull/12900#issuecomment-658201596 CC @wuchong @dawidwys Btw after all this refactoring I was thinking about renaming `WrapperTypeInfo` to `InternalTypeInfo`. This would better match to its purpose. What do you think? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] twalthr opened a new pull request #12900: [FLINK-18537][table] Remove RowDataTypeInfo
twalthr opened a new pull request #12900: URL: https://github.com/apache/flink/pull/12900 ## What is the purpose of the change `TypeInformation` is a legacy class for the sole purpose of creating a `TypeSerializer`. Instances of `TypeInformation` are not required in the table ecosystem but sometimes enforced by interfaces of other modules (such as `org.apache.flink.api.dag.Transformation`). Therefore, we recently introduced `WrapperTypeInfo` which acts as an adapter whenever type information is required. Instances of `WrapperTypeInfo` should only be created for passing it to interfaces that require type information. The class should not be used as a replacement for a `LogicalType`. Information such as the arity of a row type, field types, field names, etc. should be derived from the `LogicalType` directly. This PR updates the code base to not use `RowDataTypeInfo` anymore. `RowDataTypeInfo` was a composite type information with legacy types as field types. This refactoring unblocks FLINK-15803 and reduces potential bugs from type conversion. ## Brief change log See commit messages. ## Verifying this change This change is already covered by existing tests. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: yes - The serializers: yes - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: no - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? no - If yes, how is the feature documented? not applicable This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-18537) Remove RowDataTypeInfo
[ https://issues.apache.org/jira/browse/FLINK-18537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-18537: --- Labels: pull-request-available (was: ) > Remove RowDataTypeInfo > -- > > Key: FLINK-18537 > URL: https://issues.apache.org/jira/browse/FLINK-18537 > Project: Flink > Issue Type: Sub-task > Components: Table SQL / Planner >Reporter: Timo Walther >Assignee: Timo Walther >Priority: Major > Labels: pull-request-available > > FLINK-17000 introduced a TypeInformation class that should replace most of > the type information in the Blink planner. We start with removing > RowDataTypeInfo. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-18588) hive ddl create table should support 'if not exists'
[
https://issues.apache.org/jira/browse/FLINK-18588?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17157401#comment-17157401
]
wangtong commented on FLINK-18588:
--
[~jark] please help to review this pr
> hive ddl create table should support 'if not exists'
>
>
> Key: FLINK-18588
> URL: https://issues.apache.org/jira/browse/FLINK-18588
> Project: Flink
> Issue Type: Improvement
> Components: Table SQL / Planner
>Affects Versions: 1.11.0
>Reporter: wangtong
>Priority: Minor
> Labels: pull-request-available
>
> FlinkHiveSqlParser cant parse sql 'create table if not exists tbl (x int)'
> due to 'if not exists'
> {code:java}
> // code placeholder
> java.lang.RuntimeException: Error while parsing SQL: create table if not
> exists tbl (x int)java.lang.RuntimeException: Error while parsing SQL: create
> table if not exists tbl (x int)
> at
> org.apache.calcite.sql.parser.SqlParserTest$TesterImpl.parseStmtAndHandleEx(SqlParserTest.java:8861)
> at
> org.apache.calcite.sql.parser.SqlParserTest$TesterImpl.check(SqlParserTest.java:8845)
> at
> org.apache.calcite.sql.parser.SqlParserTest$Sql.ok(SqlParserTest.java:9192)
> at
> org.apache.flink.sql.parser.hive.FlinkHiveSqlParserImplTest.testCreateTable(FlinkHiveSqlParserImplTest.java:177)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498) at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325) at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290) at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71) at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288) at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58) at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268) at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363) at
> org.junit.runner.JUnitCore.run(JUnitCore.java:137) at
> org.junit.runner.JUnitCore.run(JUnitCore.java:115) at
> org.junit.vintage.engine.execution.RunnerExecutor.execute(RunnerExecutor.java:40)
> at java.util.stream.ForEachOps$ForEachOp$OfRef.accept(ForEachOps.java:184)
> at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:193)
> at java.util.Iterator.forEachRemaining(Iterator.java:116) at
> java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
> at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:482) at
> java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472)
> at
> java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:151)
> at
> java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:174)
> at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at
> java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:418) at
> org.junit.vintage.engine.VintageTestEngine.executeAllChildren(VintageTestEngine.java:80)
> at
> org.junit.vintage.engine.VintageTestEngine.execute(VintageTestEngine.java:71)
> at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:229)
> at
> org.junit.platform.launcher.core.DefaultLauncher.lambda$execute$6(DefaultLauncher.java:197)
> at
> org.junit.platform.launcher.core.DefaultLauncher.withInterceptedStreams(DefaultLauncher.java:211)
> at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:191)
> at
> org.junit.platform.launcher.core.DefaultLauncher.execute(DefaultLauncher.java:128)
> at
> com.intellij.junit5.JUnit5IdeaTestRunner.startRunnerWithArgs(JUnit5IdeaTestRunner.java:69)
> at
> com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
> at
> com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
> at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)Caused by:
> org.apache.calcite.sql.parser.SqlParseException: Encountered "not" at line 1,
> column 17.Was expecting one of: "ROW" ... "COMMENT" ...
>
[GitHub] [flink] flinkbot edited a comment on pull request #12866: [FLINK-17425][blink-planner] supportsFilterPushDown rule in DynamicSource.
flinkbot edited a comment on pull request #12866: URL: https://github.com/apache/flink/pull/12866#issuecomment-656576219 ## CI report: * 02c81cff07ef76dc20a7c618112cc7a172856720 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=4497) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
