[GitHub] [flink] HuangXingBo commented on a change in pull request #14791: [FLINK-21168][python][table-planner-blink] Add support for general python group window aggregate function in Physical Rule an
HuangXingBo commented on a change in pull request #14791:
URL: https://github.com/apache/flink/pull/14791#discussion_r569200084
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/rules/physical/stream/StreamPhysicalPythonGroupWindowAggregateRule.java
##
@@ -98,7 +98,12 @@ public RelNode convert(RelNode rel) {
FlinkLogicalWindowAggregate agg = (FlinkLogicalWindowAggregate) rel;
LogicalWindow window = agg.getWindow();
-if (window instanceof SessionGroupWindow) {
+List aggCalls = agg.getAggCallList();
+boolean isPandasPythonUDAF =
+aggCalls.stream()
+.anyMatch(x -> PythonUtil.isPythonAggregate(x,
PythonFunctionKind.PANDAS));
+
+if (isPandasPythonUDAF && window instanceof SessionGroupWindow) {
throw new TableException("Session Group Window is currently not
supported.");
Review comment:
This is only temporarily not supported, and there are design solutions
for this part in
[FLIP-137.](https://cwiki.apache.org/confluence/display/FLINK/FLIP-137%3A+Support+Pandas+UDAF+in+PyFlink)
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-21247) flink iceberg table map cannot convert to datastream
[
https://issues.apache.org/jira/browse/FLINK-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1728#comment-1728
]
donglei commented on FLINK-21247:
-
i change code to this and can run normally
print map data can run
sinkStream.addSink(new RichSinkFunction() {
@Override
public void invoke(AttaInfo value, Context context) throws Exception {
Map map2 = value.getEvent_value();
map2.entrySet().stream().forEach(x->{
System.out.println(x.getKey()+":" + x.getValue());
});
}
});
!image-2021-02-03-15-52-12-493.png!
!image-2021-02-03-15-53-18-244.png!
> flink iceberg table map cannot convert to datastream
> ---
>
> Key: FLINK-21247
> URL: https://issues.apache.org/jira/browse/FLINK-21247
> Project: Flink
> Issue Type: New Feature
> Environment: iceberg master
> flink 1.12
>
>
>Reporter: donglei
>Priority: Major
> Attachments: image-2021-02-03-15-38-42-340.png,
> image-2021-02-03-15-40-27-055.png, image-2021-02-03-15-41-34-426.png,
> image-2021-02-03-15-43-19-919.png, image-2021-02-03-15-52-12-493.png,
> image-2021-02-03-15-53-18-244.png
>
>
> Flink Iceberg Table with map
> !image-2021-02-03-15-38-42-340.png!
>
> we want to read the table like this :
>
> String querySql = "SELECT
> ftime,extinfo,country,province,operator,apn,gw,src_ip_head,info_str,product_id,app_version,sdk_id,sdk_version,hardware_os,qua,upload_ip,client_ip,upload_apn,event_code,event_result,package_size,consume_time,event_value,event_time,upload_time,boundle_id,uin,platform,os_version,channel,brand,model
> from bfzt3 ";
> Table table = tEnv.sqlQuery(querySql);
> DataStream sinkStream = tEnv.toAppendStream(table,
> Types.POJO(AttaInfo.class, map));
> sinkStream.map(x->1).returns(Types.INT).keyBy(new
> NullByteKeySelector()).reduce((x,y) -> {
> return x+y;
> }).print();
>
>
> when read we find a exception
>
> 2021-02-03 15:37:57
> java.lang.ClassCastException:
> org.apache.iceberg.flink.data.FlinkParquetReaders$ReusableMapData cannot be
> cast to org.apache.flink.table.data.binary.BinaryMapData
> at
> org.apache.flink.table.runtime.typeutils.MapDataSerializer.copy(MapDataSerializer.java:107)
> at
> org.apache.flink.table.runtime.typeutils.MapDataSerializer.copy(MapDataSerializer.java:47)
> at
> org.apache.flink.table.runtime.typeutils.RowDataSerializer.copyRowData(RowDataSerializer.java:166)
> at
> org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:129)
> at
> org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:50)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:69)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:317)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:411)
> at
> org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:92)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
> at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:241)
>
> we find that iceberg map is ReusableMapData implements MapData
> !image-2021-02-03-15-40-27-055.png!
>
> this is the exception
> !image-2021-02-03-15-41-34-426.png!
> MapData has two default implements GenericMapData and BinaryMapData
> from iceberg implement is ReusableMapData
>
> so i think that code should change to like this
> !image-2021-02-03-15-43-19-919.png!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21247) flink iceberg table map cannot convert to datastream
[
https://issues.apache.org/jira/browse/FLINK-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
donglei updated FLINK-21247:
Attachment: image-2021-02-03-15-53-18-244.png
> flink iceberg table map cannot convert to datastream
> ---
>
> Key: FLINK-21247
> URL: https://issues.apache.org/jira/browse/FLINK-21247
> Project: Flink
> Issue Type: New Feature
> Environment: iceberg master
> flink 1.12
>
>
>Reporter: donglei
>Priority: Major
> Attachments: image-2021-02-03-15-38-42-340.png,
> image-2021-02-03-15-40-27-055.png, image-2021-02-03-15-41-34-426.png,
> image-2021-02-03-15-43-19-919.png, image-2021-02-03-15-52-12-493.png,
> image-2021-02-03-15-53-18-244.png
>
>
> Flink Iceberg Table with map
> !image-2021-02-03-15-38-42-340.png!
>
> we want to read the table like this :
>
> String querySql = "SELECT
> ftime,extinfo,country,province,operator,apn,gw,src_ip_head,info_str,product_id,app_version,sdk_id,sdk_version,hardware_os,qua,upload_ip,client_ip,upload_apn,event_code,event_result,package_size,consume_time,event_value,event_time,upload_time,boundle_id,uin,platform,os_version,channel,brand,model
> from bfzt3 ";
> Table table = tEnv.sqlQuery(querySql);
> DataStream sinkStream = tEnv.toAppendStream(table,
> Types.POJO(AttaInfo.class, map));
> sinkStream.map(x->1).returns(Types.INT).keyBy(new
> NullByteKeySelector()).reduce((x,y) -> {
> return x+y;
> }).print();
>
>
> when read we find a exception
>
> 2021-02-03 15:37:57
> java.lang.ClassCastException:
> org.apache.iceberg.flink.data.FlinkParquetReaders$ReusableMapData cannot be
> cast to org.apache.flink.table.data.binary.BinaryMapData
> at
> org.apache.flink.table.runtime.typeutils.MapDataSerializer.copy(MapDataSerializer.java:107)
> at
> org.apache.flink.table.runtime.typeutils.MapDataSerializer.copy(MapDataSerializer.java:47)
> at
> org.apache.flink.table.runtime.typeutils.RowDataSerializer.copyRowData(RowDataSerializer.java:166)
> at
> org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:129)
> at
> org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:50)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:69)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:317)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:411)
> at
> org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:92)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
> at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:241)
>
> we find that iceberg map is ReusableMapData implements MapData
> !image-2021-02-03-15-40-27-055.png!
>
> this is the exception
> !image-2021-02-03-15-41-34-426.png!
> MapData has two default implements GenericMapData and BinaryMapData
> from iceberg implement is ReusableMapData
>
> so i think that code should change to like this
> !image-2021-02-03-15-43-19-919.png!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21247) flink iceberg table map cannot convert to datastream
[
https://issues.apache.org/jira/browse/FLINK-21247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
donglei updated FLINK-21247:
Attachment: image-2021-02-03-15-52-12-493.png
> flink iceberg table map cannot convert to datastream
> ---
>
> Key: FLINK-21247
> URL: https://issues.apache.org/jira/browse/FLINK-21247
> Project: Flink
> Issue Type: New Feature
> Environment: iceberg master
> flink 1.12
>
>
>Reporter: donglei
>Priority: Major
> Attachments: image-2021-02-03-15-38-42-340.png,
> image-2021-02-03-15-40-27-055.png, image-2021-02-03-15-41-34-426.png,
> image-2021-02-03-15-43-19-919.png, image-2021-02-03-15-52-12-493.png
>
>
> Flink Iceberg Table with map
> !image-2021-02-03-15-38-42-340.png!
>
> we want to read the table like this :
>
> String querySql = "SELECT
> ftime,extinfo,country,province,operator,apn,gw,src_ip_head,info_str,product_id,app_version,sdk_id,sdk_version,hardware_os,qua,upload_ip,client_ip,upload_apn,event_code,event_result,package_size,consume_time,event_value,event_time,upload_time,boundle_id,uin,platform,os_version,channel,brand,model
> from bfzt3 ";
> Table table = tEnv.sqlQuery(querySql);
> DataStream sinkStream = tEnv.toAppendStream(table,
> Types.POJO(AttaInfo.class, map));
> sinkStream.map(x->1).returns(Types.INT).keyBy(new
> NullByteKeySelector()).reduce((x,y) -> {
> return x+y;
> }).print();
>
>
> when read we find a exception
>
> 2021-02-03 15:37:57
> java.lang.ClassCastException:
> org.apache.iceberg.flink.data.FlinkParquetReaders$ReusableMapData cannot be
> cast to org.apache.flink.table.data.binary.BinaryMapData
> at
> org.apache.flink.table.runtime.typeutils.MapDataSerializer.copy(MapDataSerializer.java:107)
> at
> org.apache.flink.table.runtime.typeutils.MapDataSerializer.copy(MapDataSerializer.java:47)
> at
> org.apache.flink.table.runtime.typeutils.RowDataSerializer.copyRowData(RowDataSerializer.java:166)
> at
> org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:129)
> at
> org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:50)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:69)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46)
> at
> org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50)
> at
> org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:317)
> at
> org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:411)
> at
> org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:92)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
> at
> org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
> at
> org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:241)
>
> we find that iceberg map is ReusableMapData implements MapData
> !image-2021-02-03-15-40-27-055.png!
>
> this is the exception
> !image-2021-02-03-15-41-34-426.png!
> MapData has two default implements GenericMapData and BinaryMapData
> from iceberg implement is ReusableMapData
>
> so i think that code should change to like this
> !image-2021-02-03-15-43-19-919.png!
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] amaliujia commented on a change in pull request #14775: [FLINK-20964][python] Introduce PythonStreamGroupWindowAggregateOperator
amaliujia commented on a change in pull request #14775:
URL: https://github.com/apache/flink/pull/14775#discussion_r569195501
##
File path:
flink-python/src/main/java/org/apache/flink/table/runtime/operators/python/aggregate/AbstractPythonStreamGroupAggregateOperator.java
##
@@ -0,0 +1,189 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.table.runtime.operators.python.aggregate;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.state.ValueState;
+import org.apache.flink.api.common.state.ValueStateDescriptor;
+import org.apache.flink.api.common.typeinfo.Types;
+import org.apache.flink.api.java.tuple.Tuple2;
+import org.apache.flink.configuration.Configuration;
+import org.apache.flink.fnexecution.v1.FlinkFnApi;
+import org.apache.flink.runtime.state.VoidNamespace;
+import org.apache.flink.runtime.state.VoidNamespaceSerializer;
+import org.apache.flink.streaming.api.SimpleTimerService;
+import org.apache.flink.streaming.api.TimerService;
+import org.apache.flink.streaming.api.operators.InternalTimer;
+import org.apache.flink.streaming.api.operators.Triggerable;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.functions.python.PythonAggregateFunctionInfo;
+import org.apache.flink.table.planner.typeutils.DataViewUtils;
+import org.apache.flink.table.runtime.functions.CleanupState;
+import org.apache.flink.table.types.logical.BigIntType;
+import org.apache.flink.table.types.logical.RowType;
+import org.apache.flink.table.types.logical.TinyIntType;
+
+import java.util.ArrayList;
+import java.util.List;
+
+/**
+ * Base class for {@link PythonStreamGroupAggregateOperator} and {@link
+ * PythonStreamGroupTableAggregateOperator}.
+ */
+@Internal
+public abstract class AbstractPythonStreamGroupAggregateOperator
+extends AbstractPythonStreamAggregateOperator
+implements Triggerable, CleanupState {
+
+private static final long serialVersionUID = 1L;
+
+/** The minimum time in milliseconds until state which was not updated
will be retained. */
+private final long minRetentionTime;
+
+/** The maximum time in milliseconds until state which was not updated
will be retained. */
+private final long maxRetentionTime;
+
+/**
+ * Indicates whether state cleaning is enabled. Can be calculated from the
`minRetentionTime`.
+ */
+private final boolean stateCleaningEnabled;
+
+private transient TimerService timerService;
+
+// holds the latest registered cleanup timer
+private transient ValueState cleanupTimeState;
+
+public AbstractPythonStreamGroupAggregateOperator(
+Configuration config,
+RowType inputType,
+RowType outputType,
+PythonAggregateFunctionInfo[] aggregateFunctions,
+DataViewUtils.DataViewSpec[][] dataViewSpecs,
+int[] grouping,
+int indexOfCountStar,
+boolean generateUpdateBefore,
+long minRetentionTime,
+long maxRetentionTime) {
+super(
+config,
Review comment:
nit: indents are a bit wired. Seems `config` can be after `super(`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] HuangXingBo commented on a change in pull request #14791: [FLINK-21168][python][table-planner-blink] Add support for general python group window aggregate function in Physical Rule an
HuangXingBo commented on a change in pull request #14791:
URL: https://github.com/apache/flink/pull/14791#discussion_r569195396
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/rules/physical/stream/StreamPhysicalPythonGroupWindowAggregateRule.java
##
@@ -78,11 +78,11 @@ public boolean matches(RelOptRuleCall call) {
.anyMatch(x -> PythonUtil.isPythonAggregate(x,
PythonFunctionKind.PANDAS));
boolean existJavaFunction =
aggCalls.stream().anyMatch(x ->
!PythonUtil.isPythonAggregate(x, null));
+if (existPandasFunction && existGeneralPythonFunction) {
+throw new TableException(
+"Pandas UDAFs and General Python UDAFs are not supported
in used together currently.");
Review comment:
The serializer used by general UDF and pandas cannot be easily used in
the same data channel. So for `General UDF` and `Pandas UDF`, we added a
special logical rule(`PythonCalcSplitPandasInProjectionRule`) to separate them
into the different operator, but for `UDAF`, we can't separate these `UDAF`s
into different operators.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-21060) Could not create the DispatcherResourceManagerComponent
[ https://issues.apache.org/jira/browse/FLINK-21060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Spongebob updated FLINK-21060: -- Attachment: yarn_with_detached_mode.log > Could not create the DispatcherResourceManagerComponent > --- > > Key: FLINK-21060 > URL: https://issues.apache.org/jira/browse/FLINK-21060 > Project: Flink > Issue Type: Bug > Components: Table SQL / Ecosystem >Affects Versions: 1.12.0 > Environment: flink: 1.12.0 > hive: 3.1.2 >Reporter: Spongebob >Priority: Major > Attachments: yarn.log, yarn_with_detached_mode.log > > > I set multi sink to hive table in flink application then deploy it on yarn, > if I run with detached mode, the application can run successful but sink > nothing. And if I run without detached mode, the application would throw this > exception: Could not create the DispatcherResourceManagerComponent. > Attachment is the log of this application run without detached mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21060) Could not create the DispatcherResourceManagerComponent
[ https://issues.apache.org/jira/browse/FLINK-21060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Spongebob updated FLINK-21060: -- Attachment: (was: yarn_with_detached_mode.log) > Could not create the DispatcherResourceManagerComponent > --- > > Key: FLINK-21060 > URL: https://issues.apache.org/jira/browse/FLINK-21060 > Project: Flink > Issue Type: Bug > Components: Table SQL / Ecosystem >Affects Versions: 1.12.0 > Environment: flink: 1.12.0 > hive: 3.1.2 >Reporter: Spongebob >Priority: Major > Attachments: yarn.log > > > I set multi sink to hive table in flink application then deploy it on yarn, > if I run with detached mode, the application can run successful but sink > nothing. And if I run without detached mode, the application would throw this > exception: Could not create the DispatcherResourceManagerComponent. > Attachment is the log of this application run without detached mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] amaliujia commented on a change in pull request #14775: [FLINK-20964][python] Introduce PythonStreamGroupWindowAggregateOperator
amaliujia commented on a change in pull request #14775:
URL: https://github.com/apache/flink/pull/14775#discussion_r569194581
##
File path: flink-python/pyflink/proto/flink-fn-execution.proto
##
@@ -147,6 +147,30 @@ message UserDefinedAggregateFunction {
bool takes_row_as_input = 6;
}
+message GroupWindow {
+ enum WindowType {
+TUMBLING_GROUP_WINDOW = 0;
+SLIDING_GROUP_WINDOW = 1;
+SESSION_GROUP_WINDOW = 2;
+ }
+
+ WindowType window_type = 1;
+
+ bool is_time_window = 2;
+
+ int64 window_slide = 3;
+
+ int64 window_size = 4;
+
Review comment:
+1
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (FLINK-21060) Could not create the DispatcherResourceManagerComponent
[ https://issues.apache.org/jira/browse/FLINK-21060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1726#comment-1726 ] Spongebob commented on FLINK-21060: --- I uploaded the yarn log for the situation of with flink detached mode also. > Could not create the DispatcherResourceManagerComponent > --- > > Key: FLINK-21060 > URL: https://issues.apache.org/jira/browse/FLINK-21060 > Project: Flink > Issue Type: Bug > Components: Table SQL / Ecosystem >Affects Versions: 1.12.0 > Environment: flink: 1.12.0 > hive: 3.1.2 >Reporter: Spongebob >Priority: Major > Attachments: yarn.log, yarn_with_detached_mode.log > > > I set multi sink to hive table in flink application then deploy it on yarn, > if I run with detached mode, the application can run successful but sink > nothing. And if I run without detached mode, the application would throw this > exception: Could not create the DispatcherResourceManagerComponent. > Attachment is the log of this application run without detached mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21060) Could not create the DispatcherResourceManagerComponent
[ https://issues.apache.org/jira/browse/FLINK-21060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Spongebob updated FLINK-21060: -- Attachment: yarn_with_detached_mode.log > Could not create the DispatcherResourceManagerComponent > --- > > Key: FLINK-21060 > URL: https://issues.apache.org/jira/browse/FLINK-21060 > Project: Flink > Issue Type: Bug > Components: Table SQL / Ecosystem >Affects Versions: 1.12.0 > Environment: flink: 1.12.0 > hive: 3.1.2 >Reporter: Spongebob >Priority: Major > Attachments: yarn.log, yarn_with_detached_mode.log > > > I set multi sink to hive table in flink application then deploy it on yarn, > if I run with detached mode, the application can run successful but sink > nothing. And if I run without detached mode, the application would throw this > exception: Could not create the DispatcherResourceManagerComponent. > Attachment is the log of this application run without detached mode. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21247) flink iceberg table map cannot convert to datastream
donglei created FLINK-21247:
---
Summary: flink iceberg table map cannot convert to
datastream
Key: FLINK-21247
URL: https://issues.apache.org/jira/browse/FLINK-21247
Project: Flink
Issue Type: New Feature
Environment: iceberg master
flink 1.12
Reporter: donglei
Attachments: image-2021-02-03-15-38-42-340.png,
image-2021-02-03-15-40-27-055.png, image-2021-02-03-15-41-34-426.png,
image-2021-02-03-15-43-19-919.png
Flink Iceberg Table with map
!image-2021-02-03-15-38-42-340.png!
we want to read the table like this :
String querySql = "SELECT
ftime,extinfo,country,province,operator,apn,gw,src_ip_head,info_str,product_id,app_version,sdk_id,sdk_version,hardware_os,qua,upload_ip,client_ip,upload_apn,event_code,event_result,package_size,consume_time,event_value,event_time,upload_time,boundle_id,uin,platform,os_version,channel,brand,model
from bfzt3 ";
Table table = tEnv.sqlQuery(querySql);
DataStream sinkStream = tEnv.toAppendStream(table,
Types.POJO(AttaInfo.class, map));
sinkStream.map(x->1).returns(Types.INT).keyBy(new
NullByteKeySelector()).reduce((x,y) -> {
return x+y;
}).print();
when read we find a exception
2021-02-03 15:37:57
java.lang.ClassCastException:
org.apache.iceberg.flink.data.FlinkParquetReaders$ReusableMapData cannot be
cast to org.apache.flink.table.data.binary.BinaryMapData
at
org.apache.flink.table.runtime.typeutils.MapDataSerializer.copy(MapDataSerializer.java:107)
at
org.apache.flink.table.runtime.typeutils.MapDataSerializer.copy(MapDataSerializer.java:47)
at
org.apache.flink.table.runtime.typeutils.RowDataSerializer.copyRowData(RowDataSerializer.java:166)
at
org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:129)
at
org.apache.flink.table.runtime.typeutils.RowDataSerializer.copy(RowDataSerializer.java:50)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:69)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46)
at
org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:50)
at
org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:28)
at
org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollect(StreamSourceContexts.java:317)
at
org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collect(StreamSourceContexts.java:411)
at
org.apache.flink.streaming.api.functions.source.InputFormatSourceFunction.run(InputFormatSourceFunction.java:92)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:110)
at
org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:66)
at
org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:241)
we find that iceberg map is ReusableMapData implements MapData
!image-2021-02-03-15-40-27-055.png!
this is the exception
!image-2021-02-03-15-41-34-426.png!
MapData has two default implements GenericMapData and BinaryMapData
from iceberg implement is ReusableMapData
so i think that code should change to like this
!image-2021-02-03-15-43-19-919.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] amaliujia commented on a change in pull request #14791: [FLINK-21168][python][table-planner-blink] Add support for general python group window aggregate function in Physical Rule and
amaliujia commented on a change in pull request #14791:
URL: https://github.com/apache/flink/pull/14791#discussion_r569191629
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/rules/physical/stream/StreamPhysicalPythonGroupWindowAggregateRule.java
##
@@ -78,11 +78,11 @@ public boolean matches(RelOptRuleCall call) {
.anyMatch(x -> PythonUtil.isPythonAggregate(x,
PythonFunctionKind.PANDAS));
boolean existJavaFunction =
aggCalls.stream().anyMatch(x ->
!PythonUtil.isPythonAggregate(x, null));
+if (existPandasFunction && existGeneralPythonFunction) {
+throw new TableException(
Review comment:
Also this might be your design decision before: why throw a exception
than return false to indicate this rule does not match the case when both
general python function and pandas functions appear together?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink-benchmarks] Thesharing commented on a change in pull request #7: [FLINK-20612][runtime] Add benchmarks for scheduler
Thesharing commented on a change in pull request #7:
URL: https://github.com/apache/flink-benchmarks/pull/7#discussion_r569191404
##
File path:
src/main/java/org/apache/flink/scheduler/benchmark/SchedulerBenchmarkUtils.java
##
@@ -0,0 +1,245 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.flink.scheduler.benchmark;
+
+import org.apache.flink.api.common.ExecutionConfig;
+import org.apache.flink.api.common.ExecutionMode;
+import org.apache.flink.api.common.time.Deadline;
+import org.apache.flink.api.common.time.Time;
+import org.apache.flink.configuration.JobManagerOptions;
+import org.apache.flink.runtime.akka.AkkaUtils;
+import org.apache.flink.runtime.blob.VoidBlobWriter;
+import org.apache.flink.runtime.execution.ExecutionState;
+import org.apache.flink.runtime.executiongraph.AccessExecutionJobVertex;
+import org.apache.flink.runtime.executiongraph.DummyJobInformation;
+import org.apache.flink.runtime.executiongraph.ExecutionAttemptID;
+import org.apache.flink.runtime.executiongraph.ExecutionGraph;
+import org.apache.flink.runtime.executiongraph.ExecutionVertex;
+import org.apache.flink.runtime.executiongraph.JobInformation;
+import org.apache.flink.runtime.executiongraph.NoOpExecutionDeploymentListener;
+import org.apache.flink.runtime.executiongraph.TaskExecutionStateTransition;
+import org.apache.flink.runtime.executiongraph.failover.RestartAllStrategy;
+import
org.apache.flink.runtime.executiongraph.failover.flip1.partitionrelease.RegionPartitionReleaseStrategy;
+import org.apache.flink.runtime.executiongraph.restart.NoRestartStrategy;
+import
org.apache.flink.runtime.io.network.partition.NoOpJobMasterPartitionTracker;
+import org.apache.flink.runtime.io.network.partition.ResultPartitionType;
+import org.apache.flink.runtime.jobgraph.DistributionPattern;
+import org.apache.flink.runtime.jobgraph.JobGraph;
+import org.apache.flink.runtime.jobgraph.JobVertex;
+import org.apache.flink.runtime.jobgraph.JobVertexID;
+import org.apache.flink.runtime.jobgraph.ScheduleMode;
+import org.apache.flink.runtime.jobmaster.LogicalSlot;
+import org.apache.flink.runtime.jobmaster.TestingLogicalSlotBuilder;
+import org.apache.flink.runtime.jobmaster.slotpool.SlotProvider;
+import org.apache.flink.runtime.scheduler.DefaultScheduler;
+import org.apache.flink.runtime.shuffle.NettyShuffleMaster;
+import org.apache.flink.runtime.taskmanager.TaskExecutionState;
+import org.apache.flink.runtime.testingUtils.TestingUtils;
+import org.apache.flink.runtime.testtasks.NoOpInvokable;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Collection;
+import java.util.List;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Predicate;
+
+/**
+ * Utilities for runtime benchmarks.
+ */
+public class SchedulerBenchmarkUtils {
+
+ public static List createDefaultJobVertices(
+ int parallelism,
+ DistributionPattern distributionPattern,
+ ResultPartitionType resultPartitionType) {
+
+ List jobVertices = new ArrayList<>();
+
+ final JobVertex source = new JobVertex("source");
+ source.setInvokableClass(NoOpInvokable.class);
+ source.setParallelism(parallelism);
+ jobVertices.add(source);
+
+ final JobVertex sink = new JobVertex("sink");
+ sink.setInvokableClass(NoOpInvokable.class);
+ sink.setParallelism(parallelism);
+ jobVertices.add(sink);
+
+ sink.connectNewDataSetAsInput(source, distributionPattern,
resultPartitionType);
+
+ return jobVertices;
+ }
+
+ public static JobGraph createJobGraph(
+ List jobVertices,
+ ScheduleMode scheduleMode,
+ ExecutionMode executionMode) throws IOException {
+
+ final JobGraph jobGraph = new JobGraph(jobVertices.toArray(new
JobVertex[0]));
+
+ jobGraph.setScheduleMode(scheduleMode);
+ ExecutionConfig executionConfig = new ExecutionConfig();
+
[GitHub] [flink] flinkbot edited a comment on pull request #14841: [FLINK-21232][YARN][Common] Introduce pluggable Hadoop delegation token providers
flinkbot edited a comment on pull request #14841: URL: https://github.com/apache/flink/pull/14841#issuecomment-772182984 ## CI report: * 20eea1289502f47226453012ae600303352f7cb9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12826) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14843: backport [FLINK-21215][task] Do not overwrite the original CheckpointFailureReason in AsyncCheckpointRunnable
flinkbot edited a comment on pull request #14843: URL: https://github.com/apache/flink/pull/14843#issuecomment-772295777 ## CI report: * 450cfb4ea7052e7ac66319aada922bbc465b1795 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12832) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14844: [FLINK-21208][python] Make Arrow Coder serialize schema info in every batch
flinkbot edited a comment on pull request #14844: URL: https://github.com/apache/flink/pull/14844#issuecomment-772295878 ## CI report: * 4c5d3fcfdf5bb2a114964f5cd60fc6743fe331da Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12833) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14799: [WIP] Wrapped StateBackend to forward state changes to StateChangleLog
flinkbot edited a comment on pull request #14799: URL: https://github.com/apache/flink/pull/14799#issuecomment-769554549 ## CI report: * 89066112157569ad58908e401696b45a56f48fc3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12693) * bc4cd266c548473fdd31bad5018ae210048d0472 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12831) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21030) Broken job restart for job with disjoint graph
[
https://issues.apache.org/jira/browse/FLINK-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277769#comment-17277769
]
Zhu Zhu commented on FLINK-21030:
-
I guess I had misunderstood the protocol of
{{CheckpointCoordnator#triggerSynchronousSavepoint(...)}}. Its returned future
just indicates whether the savepoint is successfully taken, ignoring the result
of {{notifyCheckpointComplete}}.
So I agree that the scheduler should take care of the case that a savepoint is
taken but job does not finish in a stop-with-savepoint process.
[~trohrmann]'s solution looks good to me. Yet I think besides triggering a
global failover, we also need to restart the {{CheckpointScheduler}}. There is
such handling in the end of {{SchedulerBase#stopWithSavepoint}} but I think it
does not work in the case that {{notifyCheckpointComplete}} fails when a
savepoint is successfully taken.
> Broken job restart for job with disjoint graph
> --
>
> Key: FLINK-21030
> URL: https://issues.apache.org/jira/browse/FLINK-21030
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Coordination
>Affects Versions: 1.11.2
>Reporter: Theo Diefenthal
>Assignee: Matthias
>Priority: Blocker
> Fix For: 1.11.4, 1.12.2, 1.13.0
>
>
> Building on top of bugs:
> https://issues.apache.org/jira/browse/FLINK-21028
> and https://issues.apache.org/jira/browse/FLINK-21029 :
> I tried to stop a Flink application on YARN via savepoint which didn't
> succeed due to a possible bug/racecondition in shutdown (Bug 21028). Due to
> some reason, Flink attempted to restart the pipeline after the failure in
> shutdown (21029). The bug here:
> As I mentioned: My jobgraph is disjoint and the pipelines are fully isolated.
> Lets say the original error occured in a single task of pipeline1. Flink then
> restarted the entire pipeline1, but pipeline2 was shutdown successfully and
> switched the state to FINISHED.
> My job thus was in kind of an invalid state after the attempt to stopping:
> One of two pipelines was running, the other was FINISHED. I guess this is
> kind of a bug in the restarting behavior that only all connected components
> of a graph are restarted, but the others aren't...
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14740: [FLINK-21067][runtime][checkpoint] Modify the logic of computing which tasks to trigger/ack/commit to support finished tasks
flinkbot edited a comment on pull request #14740: URL: https://github.com/apache/flink/pull/14740#issuecomment-766340750 ## CI report: * c7e6b28b249f85cf52740d5201a769e0982a60aa UNKNOWN * f291c334a6571adc1141a6201e2e531d3b221efa Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12794) * 06f660b73c75590ec1867a578504486531ece624 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-20417) Handle "Too old resource version" exception in Kubernetes watch more gracefully
[
https://issues.apache.org/jira/browse/FLINK-20417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277767#comment-17277767
]
Yang Wang commented on FLINK-20417:
---
Some users send me with private email to share they also come across with this
issue when using {{KubernetesHAService}} for standalone deployment.
> Handle "Too old resource version" exception in Kubernetes watch more
> gracefully
> ---
>
> Key: FLINK-20417
> URL: https://issues.apache.org/jira/browse/FLINK-20417
> Project: Flink
> Issue Type: Improvement
> Components: Deployment / Kubernetes
>Affects Versions: 1.11.2, 1.12.0
>Reporter: Yang Wang
>Assignee: Yang Wang
>Priority: Critical
> Labels: pull-request-available
> Fix For: 1.12.2, 1.13.0
>
>
> Currently, when the watcher(pods watcher, configmap watcher) is closed with
> exception, we will call {{WatchCallbackHandler#handleFatalError}}. And this
> could cause JobManager terminating and then failover.
> For most cases, this is correct. But not for "too old resource version"
> exception. See more information here[1]. Usually this exception could happen
> when the APIServer is restarted. And we just need to create a new watch and
> continue to do the pods/configmap watching. This could help the Flink cluster
> reducing the impact of K8s cluster restarting.
>
> The issue is inspired by this technical article[2]. Thanks the guys from
> tencent for the debugging. Note this is a Chinese documentation.
>
> [1].
> [https://stackoverflow.com/questions/61409596/kubernetes-too-old-resource-version]
> [2]. [https://cloud.tencent.com/developer/article/1731416]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] xintongsong commented on a change in pull request #14647: [FLINK-20835] Implement FineGrainedSlotManager
xintongsong commented on a change in pull request #14647:
URL: https://github.com/apache/flink/pull/14647#discussion_r569188934
##
File path:
flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/DefaultSlotStatusSyncer.java
##
@@ -224,11 +224,12 @@ public void reportSlotStatus(InstanceID instanceId,
SlotReport slotReport) {
LOG.debug(
"Received slot report for unknown task manager with
instance id {}. Ignoring this report.",
instanceId);
-return;
+return false;
}
LOG.debug("Received slot report from instance {}: {}.", instanceId,
slotReport);
+boolean triggerCheckRequirement = false;
Review comment:
The name `canApplyPreviousAllocations ` is not consistent with the value
now
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] rkhachatryan commented on pull request #14829: (1.12) [FLINK-21132][runtime] Don't end input on stop with savepoint
rkhachatryan commented on pull request #14829: URL: https://github.com/apache/flink/pull/14829#issuecomment-772302135 ABP failure is not related (FLINK-21025). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21025) SQLClientHBaseITCase fails when untarring HBase
[
https://issues.apache.org/jira/browse/FLINK-21025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277764#comment-17277764
]
Roman Khachatryan commented on FLINK-21025:
---
1.12:
https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12821=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
[ERROR] Errors:
[ERROR] SQLClientHBaseITCase.testHBase[0: hbase-version:1.4.3] » IO Process
execution ...
> SQLClientHBaseITCase fails when untarring HBase
> ---
>
> Key: FLINK-21025
> URL: https://issues.apache.org/jira/browse/FLINK-21025
> Project: Flink
> Issue Type: Bug
> Components: Connectors / HBase, Table SQL / Client, Tests
>Affects Versions: 1.13.0
>Reporter: Dawid Wysakowicz
>Priority: Critical
> Labels: test-stability
> Fix For: 1.13.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12210=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> [ERROR] Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed:
> 908.614 s <<< FAILURE! - in
> org.apache.flink.tests.util.hbase.SQLClientHBaseITCase
> Jan 19 08:19:36 [ERROR] testHBase[1:
> hbase-version:2.2.3](org.apache.flink.tests.util.hbase.SQLClientHBaseITCase)
> Time elapsed: 615.099 s <<< ERROR!
> Jan 19 08:19:36 java.io.IOException:
> Jan 19 08:19:36 Process execution failed due error. Error output:
> Jan 19 08:19:36 at
> org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlocking(AutoClosableProcess.java:133)
> Jan 19 08:19:36 at
> org.apache.flink.tests.util.AutoClosableProcess$AutoClosableProcessBuilder.runBlocking(AutoClosableProcess.java:108)
> Jan 19 08:19:36 at
> org.apache.flink.tests.util.AutoClosableProcess.runBlocking(AutoClosableProcess.java:70)
> Jan 19 08:19:36 at
> org.apache.flink.tests.util.hbase.LocalStandaloneHBaseResource.setupHBaseDist(LocalStandaloneHBaseResource.java:86)
> Jan 19 08:19:36 at
> org.apache.flink.tests.util.hbase.LocalStandaloneHBaseResource.before(LocalStandaloneHBaseResource.java:76)
> Jan 19 08:19:36 at
> org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:46)
> Jan 19 08:19:36 at
> org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:48)
> Jan 19 08:19:36 at
> org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> Jan 19 08:19:36 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> Jan 19 08:19:36 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> Jan 19 08:19:36 at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> Jan 19 08:19:36 at org.junit.runners.Suite.runChild(Suite.java:128)
> Jan 19 08:19:36 at org.junit.runners.Suite.runChild(Suite.java:27)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> Jan 19 08:19:36 at
> org.apache.flink.util.ExternalResource$1.evaluate(ExternalResource.java:48)
> Jan 19 08:19:36 at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> Jan 19 08:19:36 at org.junit.runners.Suite.runChild(Suite.java:128)
> Jan 19 08:19:36 at org.junit.runners.Suite.runChild(Suite.java:27)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> Jan 19 08:19:36 at
> org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> Jan 19
[GitHub] [flink] amaliujia commented on a change in pull request #14791: [FLINK-21168][python][table-planner-blink] Add support for general python group window aggregate function in Physical Rule and
amaliujia commented on a change in pull request #14791:
URL: https://github.com/apache/flink/pull/14791#discussion_r569187223
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/rules/physical/stream/StreamPhysicalPythonGroupWindowAggregateRule.java
##
@@ -98,7 +98,12 @@ public RelNode convert(RelNode rel) {
FlinkLogicalWindowAggregate agg = (FlinkLogicalWindowAggregate) rel;
LogicalWindow window = agg.getWindow();
-if (window instanceof SessionGroupWindow) {
+List aggCalls = agg.getAggCallList();
+boolean isPandasPythonUDAF =
+aggCalls.stream()
+.anyMatch(x -> PythonUtil.isPythonAggregate(x,
PythonFunctionKind.PANDAS));
+
+if (isPandasPythonUDAF && window instanceof SessionGroupWindow) {
throw new TableException("Session Group Window is currently not
supported.");
Review comment:
Nit: probably `"Session Group Window is currently not supported for
Pandas UDAF"`.
Also why session window does not work when it is Pandas UDAF?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [flink] amaliujia commented on a change in pull request #14791: [FLINK-21168][python][table-planner-blink] Add support for general python group window aggregate function in Physical Rule and
amaliujia commented on a change in pull request #14791:
URL: https://github.com/apache/flink/pull/14791#discussion_r569186393
##
File path:
flink-table/flink-table-planner-blink/src/main/java/org/apache/flink/table/planner/plan/rules/physical/stream/StreamPhysicalPythonGroupWindowAggregateRule.java
##
@@ -78,11 +78,11 @@ public boolean matches(RelOptRuleCall call) {
.anyMatch(x -> PythonUtil.isPythonAggregate(x,
PythonFunctionKind.PANDAS));
boolean existJavaFunction =
aggCalls.stream().anyMatch(x ->
!PythonUtil.isPythonAggregate(x, null));
+if (existPandasFunction && existGeneralPythonFunction) {
+throw new TableException(
+"Pandas UDAFs and General Python UDAFs are not supported
in used together currently.");
Review comment:
Question: so Pandas UDAF and General UDAF cannot work together because
one input is Row and another is non-row?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Updated] (FLINK-21081) Re-compute tasks to trigger when tasks get triggered before finished
[ https://issues.apache.org/jira/browse/FLINK-21081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-21081: Description: When considering checkpoint while tasks get to finished, CheckpointCoordinator computes the tasks to trigger based on the current finish status, then during trigger, some more tasks might finished. In this case, CheckpointCoordinator would need to re-compute tasks to trigger and re-trigger the newly involved tasks. This is a replacement option to [FLINK-21246](https://issues.apache.org/jira/browse/FLINK-21246) was:When considering checkpoint while tasks get to finished, CheckpointCoordinator computes the tasks to trigger based on the current finish status, then during trigger, some more tasks might finished. In this case, CheckpointCoordinator would need to re-compute tasks to trigger and re-trigger the newly involved tasks. > Re-compute tasks to trigger when tasks get triggered before finished > > > Key: FLINK-21081 > URL: https://issues.apache.org/jira/browse/FLINK-21081 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream, Runtime / Checkpointing >Reporter: Yun Gao >Priority: Major > Labels: pull-request-available > > When considering checkpoint while tasks get to finished, > CheckpointCoordinator computes the tasks to trigger based on the current > finish status, then during trigger, some more tasks might finished. In this > case, CheckpointCoordinator would need to re-compute tasks to trigger and > re-trigger the newly involved tasks. > This is a replacement option to > [FLINK-21246](https://issues.apache.org/jira/browse/FLINK-21246) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21081) Re-compute tasks to trigger when tasks get triggered before finished
[ https://issues.apache.org/jira/browse/FLINK-21081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-21081: Description: When considering checkpoint while tasks get to finished, CheckpointCoordinator computes the tasks to trigger based on the current finish status, then during trigger, some more tasks might finished. In this case, CheckpointCoordinator would need to re-compute tasks to trigger and re-trigger the newly involved tasks. This is a replacement option to FLINK-21246 was: When considering checkpoint while tasks get to finished, CheckpointCoordinator computes the tasks to trigger based on the current finish status, then during trigger, some more tasks might finished. In this case, CheckpointCoordinator would need to re-compute tasks to trigger and re-trigger the newly involved tasks. This is a replacement option to [FLINK-21246](https://issues.apache.org/jira/browse/FLINK-21246) > Re-compute tasks to trigger when tasks get triggered before finished > > > Key: FLINK-21081 > URL: https://issues.apache.org/jira/browse/FLINK-21081 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream, Runtime / Checkpointing >Reporter: Yun Gao >Priority: Major > Labels: pull-request-available > > When considering checkpoint while tasks get to finished, > CheckpointCoordinator computes the tasks to trigger based on the current > finish status, then during trigger, some more tasks might finished. In this > case, CheckpointCoordinator would need to re-compute tasks to trigger and > re-trigger the newly involved tasks. > This is a replacement option to FLINK-21246 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21246) Decline Checkpoint if some tasks finished before get triggered
[ https://issues.apache.org/jira/browse/FLINK-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-21246: Description: When considering checkpoint while tasks get to finished, CheckpointCoordinator computes the tasks to trigger based on the current finish status, then during trigger, some more tasks might finished. In this case, CheckpointCoordinator would abort this checkpoint. This is an replacement option to [FLINK-21081|https://issues.apache.org/jira/browse/FLINK-21081] was: When considering checkpoint while tasks get to finished, CheckpointCoordinator computes the tasks to trigger based on the current finish status, then during trigger, some more tasks might finished. In this case, CheckpointCoordinator would abort this checkpoint. This is an replacement option to > Decline Checkpoint if some tasks finished before get triggered > -- > > Key: FLINK-21246 > URL: https://issues.apache.org/jira/browse/FLINK-21246 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream, Runtime / Checkpointing >Reporter: Yun Gao >Priority: Major > > When considering checkpoint while tasks get to finished, > CheckpointCoordinator computes the tasks to trigger based on the current > finish status, then during trigger, some more tasks might finished. In this > case, CheckpointCoordinator would abort this checkpoint. > This is an replacement option to > [FLINK-21081|https://issues.apache.org/jira/browse/FLINK-21081] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21246) Decline Checkpoint if some tasks finished before get triggered
[ https://issues.apache.org/jira/browse/FLINK-21246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yun Gao updated FLINK-21246: Description: When considering checkpoint while tasks get to finished, CheckpointCoordinator computes the tasks to trigger based on the current finish status, then during trigger, some more tasks might finished. In this case, CheckpointCoordinator would abort this checkpoint. This is an replacement option to > Decline Checkpoint if some tasks finished before get triggered > -- > > Key: FLINK-21246 > URL: https://issues.apache.org/jira/browse/FLINK-21246 > Project: Flink > Issue Type: Sub-task > Components: API / DataStream, Runtime / Checkpointing >Reporter: Yun Gao >Priority: Major > > When considering checkpoint while tasks get to finished, > CheckpointCoordinator computes the tasks to trigger based on the current > finish status, then during trigger, some more tasks might finished. In this > case, CheckpointCoordinator would abort this checkpoint. > This is an replacement option to -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21246) Decline Checkpoint if some tasks finished before get triggered
Yun Gao created FLINK-21246: --- Summary: Decline Checkpoint if some tasks finished before get triggered Key: FLINK-21246 URL: https://issues.apache.org/jira/browse/FLINK-21246 Project: Flink Issue Type: Sub-task Components: API / DataStream, Runtime / Checkpointing Reporter: Yun Gao -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21244) UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnLocalAndRemoteChannel Fail
[ https://issues.apache.org/jira/browse/FLINK-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277759#comment-17277759 ] Guowei Ma commented on FLINK-21244: --- thanks! [~pnowojski] > UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnLocalAndRemoteChannel > Fail > -- > > Key: FLINK-21244 > URL: https://issues.apache.org/jira/browse/FLINK-21244 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.11.4 >Reporter: Guowei Ma >Priority: Major > Labels: test-stability > > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12814=logs=2c3cbe13-dee0-5837-cf47-3053da9a8a78=981eced9-6683-5752-3201-62faf56c149b > > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:435) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:209) > > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:159) > > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) > at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) > at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at akka.actor.Actor$class.aroundReceive(Actor.scala:517) > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) > at akka.actor.ActorCell.invoke(ActorCell.scala:561) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) > at akka.dispatch.Mailbox.run(Mailbox.scala:225) > at akka.dispatch.Mailbox.exec(Mailbox.scala:235) > ... 4 more > Caused by: org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint > tolerable failure threshold. > at > org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleTaskLevelCheckpointException(CheckpointFailureManager.java:90) > > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:1889) > > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveDeclineMessage(CheckpointCoordinator.java:996) > > at > org.apache.flink.runtime.scheduler.SchedulerBase.lambda$declineCheckpoint$6(SchedulerBase.java:1026) > > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at java.base/java.lang.Thread.run(Thread.java:834) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-21243) Add Java SDK maven module
[ https://issues.apache.org/jira/browse/FLINK-21243?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai closed FLINK-21243. --- Fix Version/s: statefun-3.0.0 Assignee: Igal Shilman Resolution: Fixed flink-statefun/master: c3660eeafb12081960c73e6a7145fea08b41f68c > Add Java SDK maven module > - > > Key: FLINK-21243 > URL: https://issues.apache.org/jira/browse/FLINK-21243 > Project: Flink > Issue Type: Task > Components: Stateful Functions >Reporter: Igal Shilman >Assignee: Igal Shilman >Priority: Major > Labels: pull-request-available > Fix For: statefun-3.0.0 > > > Add the Skelton maven module that will contain the Java SDK. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-21230) Add protobuf wrapper types for the StateFun SDK types.
[ https://issues.apache.org/jira/browse/FLINK-21230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tzu-Li (Gordon) Tai closed FLINK-21230. --- Fix Version/s: statefun-3.0.0 Assignee: Igal Shilman Resolution: Fixed flink-statefun/master: fa64c6874f1a1f9dc298d02697a614b71a9fbd23 > Add protobuf wrapper types for the StateFun SDK types. > -- > > Key: FLINK-21230 > URL: https://issues.apache.org/jira/browse/FLINK-21230 > Project: Flink > Issue Type: Task > Components: Stateful Functions >Reporter: Igal Shilman >Assignee: Igal Shilman >Priority: Major > Labels: pull-request-available > Fix For: statefun-3.0.0 > > > Add primitive wrapper types to be used for messaging and state as part of the > new type system. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink-statefun] tzulitai closed pull request #197: [FLINK-21243] Add a maven module for the Java SDK
tzulitai closed pull request #197: URL: https://github.com/apache/flink-statefun/pull/197 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] tzulitai commented on pull request #197: [FLINK-21243] Add a maven module for the Java SDK
tzulitai commented on pull request #197: URL: https://github.com/apache/flink-statefun/pull/197#issuecomment-772296232 One thing though (which is easily resolvable): I believe we would need to amend the bundled NOTICE file for this artifact, since it is bundling and shading protobuf. This should be straightforward as Protobuf's license is ASF 2.0 compatible. I'll work on addressing this last bit. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14844: [FLINK-21208][python] Make Arrow Coder serialize schema info in every batch
flinkbot commented on pull request #14844: URL: https://github.com/apache/flink/pull/14844#issuecomment-772295878 ## CI report: * 4c5d3fcfdf5bb2a114964f5cd60fc6743fe331da UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14843: backport [FLINK-21215][task] Do not overwrite the original CheckpointFailureReason in AsyncCheckpointRunnable
flinkbot commented on pull request #14843: URL: https://github.com/apache/flink/pull/14843#issuecomment-772295777 ## CI report: * 450cfb4ea7052e7ac66319aada922bbc465b1795 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14799: [WIP] Wrapped StateBackend to forward state changes to StateChangleLog
flinkbot edited a comment on pull request #14799: URL: https://github.com/apache/flink/pull/14799#issuecomment-769554549 ## CI report: * 89066112157569ad58908e401696b45a56f48fc3 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12693) * bc4cd266c548473fdd31bad5018ae210048d0472 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink-statefun] tzulitai commented on pull request #197: [FLINK-21243] Add a maven module for the Java SDK
tzulitai commented on pull request #197: URL: https://github.com/apache/flink-statefun/pull/197#issuecomment-772292853 Thanks @igalshilman! LGTM, I'll merge this as well as #196. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14844: [FLINK-21208][python] Make Arrow Coder serialize schema info in every batch
flinkbot commented on pull request #14844: URL: https://github.com/apache/flink/pull/14844#issuecomment-772289781 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 4c5d3fcfdf5bb2a114964f5cd60fc6743fe331da (Wed Feb 03 07:12:03 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-21208).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21208) pyarrow exception when using window with pandas udaf
[
https://issues.apache.org/jira/browse/FLINK-21208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-21208:
---
Labels: pull-request-available (was: )
> pyarrow exception when using window with pandas udaf
>
>
> Key: FLINK-21208
> URL: https://issues.apache.org/jira/browse/FLINK-21208
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.12.0, 1.12.1
>Reporter: YufeiLiu
>Priority: Major
> Labels: pull-request-available
>
> I write a pyflink demo and execute in local environment, the logic is
> simple:generate records and aggerate in 100s tumle window, using a pandas
> udaf.
> But the job failed after several minutes, I don't think it's a resource
> problem because the amount of data is small, here is the error trace.
> {code:java}
> Caused by: org.apache.flink.streaming.runtime.tasks.AsynchronousException:
> Caught exception while processing timer.
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask$StreamTaskAsyncExceptionHandler.handleAsyncException(StreamTask.java:1108)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.handleAsyncException(StreamTask.java:1082)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1213)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$null$17(StreamTask.java:1202)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.runThrowing(StreamTaskActionExecutor.java:47)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.Mail.run(Mail.java:78)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.processMail(MailboxProcessor.java:302)
> at
> org.apache.flink.streaming.runtime.tasks.mailbox.MailboxProcessor.runMailboxLoop(MailboxProcessor.java:184)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.runMailboxLoop(StreamTask.java:575)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invoke(StreamTask.java:539)
> at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:722)
> at org.apache.flink.runtime.taskmanager.Task.run(Task.java:547)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: TimerException{java.lang.RuntimeException: Failed to close remote
> bundle}
> ... 11 more
> Caused by: java.lang.RuntimeException: Failed to close remote bundle
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.finishBundle(BeamPythonFunctionRunner.java:371)
> at
> org.apache.flink.streaming.api.runners.python.beam.BeamPythonFunctionRunner.flush(BeamPythonFunctionRunner.java:325)
> at
> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.invokeFinishBundle(AbstractPythonFunctionOperator.java:291)
> at
> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.checkInvokeFinishBundleByTime(AbstractPythonFunctionOperator.java:285)
> at
> org.apache.flink.streaming.api.operators.python.AbstractPythonFunctionOperator.lambda$open$0(AbstractPythonFunctionOperator.java:134)
> at
> org.apache.flink.streaming.runtime.tasks.StreamTask.invokeProcessingTimeCallback(StreamTask.java:1211)
> ... 10 more
> Caused by: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: Error received from SDK harness for instruction
> 3: Traceback (most recent call last):
> File
> "/Users/lyf/Library/Python/3.7/lib/python/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 253, in _execute
> response = task()
> File
> "/Users/lyf/Library/Python/3.7/lib/python/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 310, in
> lambda: self.create_worker().do_instruction(request), request)
> File
> "/Users/lyf/Library/Python/3.7/lib/python/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 480, in do_instruction
> getattr(request, request_type), request.instruction_id)
> File
> "/Users/lyf/Library/Python/3.7/lib/python/site-packages/apache_beam/runners/worker/sdk_worker.py",
> line 515, in process_bundle
> bundle_processor.process_bundle(instruction_id))
> File
> "/Users/lyf/Library/Python/3.7/lib/python/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 978, in process_bundle
> element.data)
> File
> "/Users/lyf/Library/Python/3.7/lib/python/site-packages/apache_beam/runners/worker/bundle_processor.py",
> line 218, in process_encoded
> self.output(decoded_value)
> File "apache_beam/runners/worker/operations.py", line 330, in
> apache_beam.runners.worker.operations.Operation.output
> File "apache_beam/runners/worker/operations.py", line 332, in
>
[GitHub] [flink] HuangXingBo opened a new pull request #14844: [FLINK-21208][python] Make Arrow Coder serialize schema info in every batch
HuangXingBo opened a new pull request #14844: URL: https://github.com/apache/flink/pull/14844 ## What is the purpose of the change *This pull request will make Arrow Coder serialize schema info in every batch* ## Brief change log - *make Arrow Coder serialize schema info in every batch* ## Verifying this change This change added tests and can be verified as follows: - *original tests ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14843: backport [FLINK-21215][task] Do not overwrite the original CheckpointFailureReason in AsyncCheckpointRunnable
flinkbot commented on pull request #14843: URL: https://github.com/apache/flink/pull/14843#issuecomment-772286261 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 450cfb4ea7052e7ac66319aada922bbc465b1795 (Wed Feb 03 07:05:11 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14647: [FLINK-20835] Implement FineGrainedSlotManager
flinkbot edited a comment on pull request #14647: URL: https://github.com/apache/flink/pull/14647#issuecomment-760154591 ## CI report: * 970360401dcd333f1a3b29355392385262953246 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12823) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] pnowojski opened a new pull request #14843: backport [FLINK-21215][task] Do not overwrite the original CheckpointFailureReason in AsyncCheckpointRunnable
pnowojski opened a new pull request #14843: URL: https://github.com/apache/flink/pull/14843 backport part of FLINK-21215. Original fix had two fixes (or one fix and one optimisation, depending how you look at it). As the code has changed too much, I'm back-porting to release-1.11 just one of those fixes (skipping optimisation which I don't see how it can be easily implemented) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21030) Broken job restart for job with disjoint graph
[ https://issues.apache.org/jira/browse/FLINK-21030?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277736#comment-17277736 ] Matthias commented on FLINK-21030: -- Thanks for the analysis, that makes sense. I went ahead with this approach. > Broken job restart for job with disjoint graph > -- > > Key: FLINK-21030 > URL: https://issues.apache.org/jira/browse/FLINK-21030 > Project: Flink > Issue Type: Bug > Components: Runtime / Coordination >Affects Versions: 1.11.2 >Reporter: Theo Diefenthal >Assignee: Matthias >Priority: Blocker > Fix For: 1.11.4, 1.12.2, 1.13.0 > > > Building on top of bugs: > https://issues.apache.org/jira/browse/FLINK-21028 > and https://issues.apache.org/jira/browse/FLINK-21029 : > I tried to stop a Flink application on YARN via savepoint which didn't > succeed due to a possible bug/racecondition in shutdown (Bug 21028). Due to > some reason, Flink attempted to restart the pipeline after the failure in > shutdown (21029). The bug here: > As I mentioned: My jobgraph is disjoint and the pipelines are fully isolated. > Lets say the original error occured in a single task of pipeline1. Flink then > restarted the entire pipeline1, but pipeline2 was shutdown successfully and > switched the state to FINISHED. > My job thus was in kind of an invalid state after the attempt to stopping: > One of two pipelines was running, the other was FINISHED. I guess this is > kind of a bug in the restarting behavior that only all connected components > of a graph are restarted, but the others aren't... -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14842: [FLINK-21238][python] Support to close PythonFunctionFactory
flinkbot edited a comment on pull request #14842: URL: https://github.com/apache/flink/pull/14842#issuecomment-772256636 ## CI report: * e64470869ec5501fbb2b81bc5db796d7c8450ffb Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12828) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Closed] (FLINK-21244) UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnLocalAndRemoteChannel Fail
[ https://issues.apache.org/jira/browse/FLINK-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski closed FLINK-21244. -- Resolution: Duplicate > UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnLocalAndRemoteChannel > Fail > -- > > Key: FLINK-21244 > URL: https://issues.apache.org/jira/browse/FLINK-21244 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.11.4 >Reporter: Guowei Ma >Priority: Major > Labels: test-stability > > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12814=logs=2c3cbe13-dee0-5837-cf47-3053da9a8a78=981eced9-6683-5752-3201-62faf56c149b > > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:435) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:209) > > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:159) > > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) > at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) > at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at akka.actor.Actor$class.aroundReceive(Actor.scala:517) > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) > at akka.actor.ActorCell.invoke(ActorCell.scala:561) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) > at akka.dispatch.Mailbox.run(Mailbox.scala:225) > at akka.dispatch.Mailbox.exec(Mailbox.scala:235) > ... 4 more > Caused by: org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint > tolerable failure threshold. > at > org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleTaskLevelCheckpointException(CheckpointFailureManager.java:90) > > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:1889) > > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveDeclineMessage(CheckpointCoordinator.java:996) > > at > org.apache.flink.runtime.scheduler.SchedulerBase.lambda$declineCheckpoint$6(SchedulerBase.java:1026) > > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at java.base/java.lang.Thread.run(Thread.java:834) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21244) UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnLocalAndRemoteChannel Fail
[ https://issues.apache.org/jira/browse/FLINK-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277728#comment-17277728 ] Piotr Nowojski commented on FLINK-21244: [~maguowei] this is the same issue that you reported before in FLINK-21215, just this time on release-1.11 branch. I will backport FLINK-21215 fix to release-1.11. > UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnLocalAndRemoteChannel > Fail > -- > > Key: FLINK-21244 > URL: https://issues.apache.org/jira/browse/FLINK-21244 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.11.4 >Reporter: Guowei Ma >Priority: Major > Labels: test-stability > > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12814=logs=2c3cbe13-dee0-5837-cf47-3053da9a8a78=981eced9-6683-5752-3201-62faf56c149b > > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:435) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:209) > > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:159) > > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) > at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) > at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at akka.actor.Actor$class.aroundReceive(Actor.scala:517) > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) > at akka.actor.ActorCell.invoke(ActorCell.scala:561) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) > at akka.dispatch.Mailbox.run(Mailbox.scala:225) > at akka.dispatch.Mailbox.exec(Mailbox.scala:235) > ... 4 more > Caused by: org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint > tolerable failure threshold. > at > org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleTaskLevelCheckpointException(CheckpointFailureManager.java:90) > > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:1889) > > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveDeclineMessage(CheckpointCoordinator.java:996) > > at > org.apache.flink.runtime.scheduler.SchedulerBase.lambda$declineCheckpoint$6(SchedulerBase.java:1026) > > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at java.base/java.lang.Thread.run(Thread.java:834) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21244) UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnLocalAndRemoteChannel Fail
[ https://issues.apache.org/jira/browse/FLINK-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277727#comment-17277727 ] Piotr Nowojski commented on FLINK-21244: Another failure caused by FLINK-20675 > UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnLocalAndRemoteChannel > Fail > -- > > Key: FLINK-21244 > URL: https://issues.apache.org/jira/browse/FLINK-21244 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.11.4 >Reporter: Guowei Ma >Priority: Major > Labels: test-stability > > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12814=logs=2c3cbe13-dee0-5837-cf47-3053da9a8a78=981eced9-6683-5752-3201-62faf56c149b > > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:435) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:209) > > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:159) > > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) > at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) > at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at akka.actor.Actor$class.aroundReceive(Actor.scala:517) > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) > at akka.actor.ActorCell.invoke(ActorCell.scala:561) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) > at akka.dispatch.Mailbox.run(Mailbox.scala:225) > at akka.dispatch.Mailbox.exec(Mailbox.scala:235) > ... 4 more > Caused by: org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint > tolerable failure threshold. > at > org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleTaskLevelCheckpointException(CheckpointFailureManager.java:90) > > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:1889) > > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveDeclineMessage(CheckpointCoordinator.java:996) > > at > org.apache.flink.runtime.scheduler.SchedulerBase.lambda$declineCheckpoint$6(SchedulerBase.java:1026) > > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at java.base/java.lang.Thread.run(Thread.java:834) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21245) Support StreamExecCalc json serialization/deserialization
godfrey he created FLINK-21245: -- Summary: Support StreamExecCalc json serialization/deserialization Key: FLINK-21245 URL: https://issues.apache.org/jira/browse/FLINK-21245 Project: Flink Issue Type: Sub-task Components: Table SQL / Planner Reporter: godfrey he Fix For: 1.13.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21215) Checkpoint was declined because one input stream is finished
[ https://issues.apache.org/jira/browse/FLINK-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski updated FLINK-21215: --- Fix Version/s: 1.12.2 1.11.4 > Checkpoint was declined because one input stream is finished > > > Key: FLINK-21215 > URL: https://issues.apache.org/jira/browse/FLINK-21215 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.11.4, 1.12.2, 1.13.0 >Reporter: Guowei Ma >Assignee: Piotr Nowojski >Priority: Major > Labels: pull-request-available, test-stability > Fix For: 1.11.4, 1.12.2, 1.13.0 > > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12691=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2=9146] > > [ERROR] Errors: > [ERROR] UnalignedCheckpointITCase.execute[parallel pipeline with remote > channels, p = 5] » JobExecution > ... 4 more > Caused by: org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint > tolerable failure threshold. > at > org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleCheckpointException(CheckpointFailureManager.java:98) > > at > org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleTaskLevelCheckpointException(CheckpointFailureManager.java:84) > > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:1930) > > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveDeclineMessage(CheckpointCoordinator.java:1007) > > at > org.apache.flink.runtime.scheduler.SchedulerBase.lambda$declineCheckpoint$9(SchedulerBase.java:1009) > > at > org.apache.flink.runtime.scheduler.SchedulerBase.lambda$processCheckpointCoordinatorMessage$10(SchedulerBase.java:1025) > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > > > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-21096) Introduce ExecNodeGraphJsonPlanGenerator to serialize ExecNodeGraph to json plan and deserialize json plan to ExecNodeGraph
[
https://issues.apache.org/jira/browse/FLINK-21096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
godfrey he closed FLINK-21096.
--
Resolution: Fixed
Fixed in 1.13.0: 0150ae471ce1c82404d51be2855f642d91088ad7
> Introduce ExecNodeGraphJsonPlanGenerator to serialize ExecNodeGraph to json
> plan and deserialize json plan to ExecNodeGraph
> ---
>
> Key: FLINK-21096
> URL: https://issues.apache.org/jira/browse/FLINK-21096
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
> Fix For: 1.13.0
>
>
> ExecNodeGraphJsonPlanGenerator is an utility class that can generate json
> plan based on given ExecNodeGraph or generate ExecNodeGraph based on given
> json plan.
> Based on FLINK-21092, FLINK-21093, FLINK-21094 and this jira, we can get the
> json plan for {{insert into MySink select * from MyTable}}, we also can
> submit the json plan and get the execution result.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21215) Checkpoint was declined because one input stream is finished
[ https://issues.apache.org/jira/browse/FLINK-21215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski updated FLINK-21215: --- Affects Version/s: 1.12.2 1.11.4 > Checkpoint was declined because one input stream is finished > > > Key: FLINK-21215 > URL: https://issues.apache.org/jira/browse/FLINK-21215 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.11.4, 1.12.2, 1.13.0 >Reporter: Guowei Ma >Assignee: Piotr Nowojski >Priority: Major > Labels: pull-request-available, test-stability > Fix For: 1.13.0 > > > [https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12691=logs=34f41360-6c0d-54d3-11a1-0292a2def1d9=2d56e022-1ace-542f-bf1a-b37dd63243f2=9146] > > [ERROR] Errors: > [ERROR] UnalignedCheckpointITCase.execute[parallel pipeline with remote > channels, p = 5] » JobExecution > ... 4 more > Caused by: org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint > tolerable failure threshold. > at > org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleCheckpointException(CheckpointFailureManager.java:98) > > at > org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleTaskLevelCheckpointException(CheckpointFailureManager.java:84) > > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:1930) > > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveDeclineMessage(CheckpointCoordinator.java:1007) > > at > org.apache.flink.runtime.scheduler.SchedulerBase.lambda$declineCheckpoint$9(SchedulerBase.java:1009) > > at > org.apache.flink.runtime.scheduler.SchedulerBase.lambda$processCheckpointCoordinatorMessage$10(SchedulerBase.java:1025) > > at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) > at java.util.concurrent.FutureTask.run(FutureTask.java:266) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) > > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) > > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) > > at java.lang.Thread.run(Thread.java:748) > > > > > > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (FLINK-21093) Support StreamExecTableSource json serialization/deserialization
[
https://issues.apache.org/jira/browse/FLINK-21093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
godfrey he closed FLINK-21093.
--
Resolution: Fixed
Fixed in 1.13.0: c37905e8d812d243321c7a32a6f74cbde0294ce0
> Support StreamExecTableSource json serialization/deserialization
>
>
> Key: FLINK-21093
> URL: https://issues.apache.org/jira/browse/FLINK-21093
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
> Fix For: 1.13.0
>
>
> If a source table's ddl is
> {code:sql}
> CREATE TABLE MyTable (
> a bigint,
> b int,
> c varchar
> ) with (
> 'connector' = 'filesystem',
> 'path' = '/tmp',
> 'format' = 'testcsv'
> )
> {code}
> Its corresponding StreamExecTableSource's json representation looks like:
> {code:json}
> {
> "id": 1,
> "description": "TableSourceScan(table=[[default_catalog,
> default_database, MyTable]], fields=[a, b, c])",
> "class":
> "org.apache.flink.table.planner.plan.nodes.exec.stream.StreamExecTableSourceScan"
> "scanTableSource": {
> "identifier": {
> "catalogName": "default_catalog",
> "databaseName": "default_database",
> "tableName": "MyTable"
> },
> "catalogTable": {
> "connector": "filesystem",
> "path": "/tmp",
> "format": "testcsv",
> "schema.0.name": "a",
> "schema.0.data-type": "BIGINT",
> "schema.1.name": "b",
> "schema.1.data-type": "INT",
> "schema.2.name": "c",
> "schema.2.data-type": "VARCHAR(2147483647)"
> },
> "configuration": {}
> },
> "outputType": "ROW<`a` BIGINT, `b` INT, `c` VARCHAR(2147483647)>",
> "inputEdges": [],
> "inputs": [],
> }
> {code}
> All properties of the catalog table will be serialized, so that the
> ScanTableSource instance can be created based on those properties from json.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-21094) Support StreamExecSink json serialization/deserialization
[
https://issues.apache.org/jira/browse/FLINK-21094?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
godfrey he closed FLINK-21094.
--
Resolution: Fixed
Fixed in 1.13.0: cae238841271ae9494c0cf7887ed8e4b0f8f2c84
> Support StreamExecSink json serialization/deserialization
> -
>
> Key: FLINK-21094
> URL: https://issues.apache.org/jira/browse/FLINK-21094
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
> Fix For: 1.13.0
>
>
> Similar to StreamExecTableSourceScan, if a sink table's ddl is
> {code:sql}
> CREATE TABLE MySink (
> a bigint,
> b int,
> c varchar
> ) with (
> 'connector' = 'filesystem',
> 'path' = '/tmp',
> 'format' = 'testcsv'
> )
> {code}
> Its corresponding StreamExecSink's json representation looks like:
> {code:json}
> {
> "id": 2,
> "description": "Sink(table=[default_catalog.default_database.MySink],
> fields=[a, b, c])",
> "class":
> "org.apache.flink.table.planner.plan.nodes.exec.stream.StreamExecSink",
> "dynamicTableSink": {
> "identifier": {
> "catalogName": "default_catalog",
> "databaseName": "default_database",
> "tableName": "MySink"
> },
> "catalogTable": {
> "connector": "filesystem",
> "format": "testcsv",
> "path": "/tmp",
> "schema.0.name": "a",
> "schema.0.data-type": "BIGINT",
> "schema.1.name": "b",
> "schema.1.data-type": "INT",
> "schema.2.name": "c",
> "schema.2.data-type": "VARCHAR(2147483647)"
> },
> "configuration": {}
> },
> "inputChangelogMode": [
> "INSERT"
> ],
> "inputEdges": [
> {
> "requiredShuffle": {
> "type": "UNKNOWN"
> },
> "damBehavior": "PIPELINED",
> "priority": 0
> }
> ],
> "outputType": "ROW<`a` BIGINT, `b` INT, `c` VARCHAR(2147483647)>",
> "inputs": [
> 1
> ]
> }
> {code}
> All properties of the catalog table will be serialized, so that the
> DynamicTableSink instance can be created based on those properties from json.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (FLINK-21092) Introduce getJsonPlan, explainJsonPlan and executeJsonPlan in TableEnvironmentInternal
[
https://issues.apache.org/jira/browse/FLINK-21092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
godfrey he closed FLINK-21092.
--
Resolution: Fixed
Fixed in 1.13.0: e16e45f29e0c8b7dc771f4acb50e5b3910a751e2
> Introduce getJsonPlan, explainJsonPlan and executeJsonPlan in
> TableEnvironmentInternal
> ---
>
> Key: FLINK-21092
> URL: https://issues.apache.org/jira/browse/FLINK-21092
> Project: Flink
> Issue Type: Sub-task
> Components: Table SQL / Planner
>Reporter: godfrey he
>Assignee: godfrey he
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.13.0
>
>
> We will introduce the following methods to support json plan operations in
> TableEnvironmentInternal:
> {code:java}
> /**
> * Get the json plan for the given statement.
> *
> * The statement can only be DML.
> *
> * The json plan is the string json representation of an optimized
> ExecNode plan for the
> * given statement. An ExecNode plan can be serialized to json plan, and
> a json plan can be
> * deserialized to an ExecNode plan.
> */
> @Experimental
> String getJsonPlan(String stmt);
> /**
> * Get the json plan for the given {@link ModifyOperation}s.
> */
> @Experimental
> String getJsonPlan(List operations);
> /**
> * Returns the execution plan for the given json plan.
> */
> @Experimental
> String explainJsonPlan(String jsonPlan, ExplainDetail... extraDetails);
> /**
> * Execute the given json plan, and return the execution result. A SQL
> statement can be
> * converted to json plan through {@link #getJsonPlan(String)}.
> */
> @Experimental
> TableResult executeJsonPlan(String jsonPlan);
> {code}
> and we also introduce getJsonPlan method in StatementSetImpl so that we can
> get the json plan of the all statements and Tables as a batch.
> {code:java}
> /**
> * Get the json plan of the all statements and Tables as a batch.
> */
> @Experimental
> public String getJsonPlan() {
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21244) UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnLocalAndRemoteChannel Fail
[ https://issues.apache.org/jira/browse/FLINK-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Piotr Nowojski updated FLINK-21244: --- Affects Version/s: (was: 1.11.0) 1.11.4 > UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnLocalAndRemoteChannel > Fail > -- > > Key: FLINK-21244 > URL: https://issues.apache.org/jira/browse/FLINK-21244 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.11.4 >Reporter: Guowei Ma >Priority: Major > Labels: test-stability > > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12814=logs=2c3cbe13-dee0-5837-cf47-3053da9a8a78=981eced9-6683-5752-3201-62faf56c149b > > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:435) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:209) > > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:159) > > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) > at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) > at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at akka.actor.Actor$class.aroundReceive(Actor.scala:517) > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) > at akka.actor.ActorCell.invoke(ActorCell.scala:561) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) > at akka.dispatch.Mailbox.run(Mailbox.scala:225) > at akka.dispatch.Mailbox.exec(Mailbox.scala:235) > ... 4 more > Caused by: org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint > tolerable failure threshold. > at > org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleTaskLevelCheckpointException(CheckpointFailureManager.java:90) > > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:1889) > > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveDeclineMessage(CheckpointCoordinator.java:996) > > at > org.apache.flink.runtime.scheduler.SchedulerBase.lambda$declineCheckpoint$6(SchedulerBase.java:1026) > > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at java.base/java.lang.Thread.run(Thread.java:834) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-21132) BoundedOneInput.endInput is called when taking synchronous savepoint
[
https://issues.apache.org/jira/browse/FLINK-21132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277719#comment-17277719
]
Piotr Nowojski commented on FLINK-21132:
{quote}
I think data flow and gateway event are go through different network channel.
{quote}
Data flow and events between TaskManagers are using the same channels. RPCs
from JobManager to the TaskManagers are using a different channel, but they
won't be processed if the Task's thread (mailbox) is blocked. To avoid
deadlocks one would have to carefully thought this through.
> BoundedOneInput.endInput is called when taking synchronous savepoint
>
>
> Key: FLINK-21132
> URL: https://issues.apache.org/jira/browse/FLINK-21132
> Project: Flink
> Issue Type: Bug
> Components: Runtime / Task
>Affects Versions: 1.10.2, 1.10.3, 1.11.3, 1.12.1
>Reporter: Kezhu Wang
>Assignee: Roman Khachatryan
>Priority: Major
> Labels: pull-request-available
> Fix For: 1.11.4, 1.12.2, 1.13.0
>
>
> [~elkhand](?) reported on project iceberg that {{BoundedOneInput.endInput}}
> was
> [called|https://github.com/apache/iceberg/issues/2033#issuecomment-765864038]
> when [stopping job with
> savepoint|https://github.com/apache/iceberg/issues/2033#issuecomment-765557995].
> I think it is a bug of Flink and was introduced in FLINK-14230. The
> [changes|https://github.com/apache/flink/pull/9854/files#diff-0c5fe245445b932fa83fdaf5c4802dbe671b73e8d76f39058c6eaaaffd9639faL577]
> rely on {{StreamTask.afterInvoke}} and {{OperatorChain.closeOperators}} will
> only be invoked after *end of input*. But that is not true long before after
> [FLIP-34: Terminate/Suspend Job with
> Savepoint|https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=103090212].
> Task could enter state called
> [*finished*|https://github.com/apache/flink/blob/3a8e06cd16480eacbbf0c10f36b8c79a6f741814/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/tasks/StreamTask.java#L467]
> after synchronous savepoint, that is an expected job suspension and stopping.
> [~sunhaibotb] [~pnowojski] [~roman_khachatryan] Could you help confirm this ?
> For full context, see
> [apache/iceberg#2033|https://github.com/apache/iceberg/issues/2033]. I have
> pushed branch
> [synchronous-savepoint-conflict-with-bounded-end-input-case|https://github.com/kezhuw/flink/commits/synchronous-savepoint-conflict-with-bounded-end-input-case]
> in my repository. Test case
> {{SavepointITCase.testStopSavepointWithBoundedInput}} failed due to
> {{BoundedOneInput.endInput}} called.
> I am also aware of [FLIP-147: Support Checkpoints After Tasks
> Finished|https://cwiki.apache.org/confluence/x/mw-ZCQ], maybe the three
> should align on what *finished* means exactly. [~kkl0u] [~chesnay]
> [~gaoyunhaii]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [flink] godfreyhe closed pull request #14729: [FLINK-21092][FLINK-21093][FLINK-21094][FLINK-21096][table-planner-blink] Support ExecNode plan serialization/deserialization for `INSERT INTO My
godfreyhe closed pull request #14729: URL: https://github.com/apache/flink/pull/14729 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14842: [FLINK-21238][python] Support to close PythonFunctionFactory
flinkbot edited a comment on pull request #14842: URL: https://github.com/apache/flink/pull/14842#issuecomment-772256636 ## CI report: * e64470869ec5501fbb2b81bc5db796d7c8450ffb Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12828) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14835: [FLINK-21213][task] Degrade log level to INFO when ignore to decline checkpoint as task not running
flinkbot edited a comment on pull request #14835: URL: https://github.com/apache/flink/pull/14835#issuecomment-771620432 ## CI report: * 6ecaa1370de5c2681bef2aa1f84ebaba2623f217 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12798) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14829: (1.12) [FLINK-21132][runtime] Don't end input on stop with savepoint
flinkbot edited a comment on pull request #14829: URL: https://github.com/apache/flink/pull/14829#issuecomment-770746355 ## CI report: * 573520d871f6fcbd2e0cb3468eed08c90bfba1e6 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12821) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14815: [FLINK-21132][runtime] Don't end input on stop with savepoint
flinkbot edited a comment on pull request #14815: URL: https://github.com/apache/flink/pull/14815#issuecomment-769949393 ## CI report: * e3139520c1b415600222869adaf5be2f6b818429 UNKNOWN * b53567a0fdb4a291e4452c1bf00070fa65cb4856 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12820) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14842: [FLINK-21238][python] Support to close PythonFunctionFactory
flinkbot commented on pull request #14842: URL: https://github.com/apache/flink/pull/14842#issuecomment-772256636 ## CI report: * e64470869ec5501fbb2b81bc5db796d7c8450ffb UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14839: [WIP][state] Add DFS-based StateChangelog (PoC)
flinkbot edited a comment on pull request #14839: URL: https://github.com/apache/flink/pull/14839#issuecomment-772060196 ## CI report: * a0d75a1671f3537cf3cf8ef72d08cbffc3d71295 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12818) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14835: [FLINK-21213][task] Degrade log level to INFO when ignore to decline checkpoint as task not running
flinkbot edited a comment on pull request #14835: URL: https://github.com/apache/flink/pull/14835#issuecomment-771620432 ## CI report: * 6ecaa1370de5c2681bef2aa1f84ebaba2623f217 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12798) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (FLINK-12607) Introduce a REST API that returns the maxParallelism of a job
[ https://issues.apache.org/jira/browse/FLINK-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Robert Metzger reassigned FLINK-12607: -- Assignee: John Phelan > Introduce a REST API that returns the maxParallelism of a job > - > > Key: FLINK-12607 > URL: https://issues.apache.org/jira/browse/FLINK-12607 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Affects Versions: 1.6.3 >Reporter: Akshay Kanfade >Assignee: John Phelan >Priority: Minor > Labels: starter > > Today, Flink does not offer any way to get the maxParallelism for a job and > it's operators through any of the REST APIs. Since, the internal state > already tracks maxParallelism for a job and it's operators, we should expose > this via the REST APIs so that application developer can get more insights on > the current state. > There can be two approaches on how we can do this - > Approach 1 : > Modify the existing rest API response model to additionally expose a new > field 'maxParallelism'. Some of the REST APIs that would be affected by this > |h5. */jobs/:jobid/vertices/:vertexid*| > |h5. */jobs/:jobid*| > > Approach 2 : > Create a new REST API that would only return maxParallelism for a job and > it's operators. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-12607) Introduce a REST API that returns the maxParallelism of a job
[ https://issues.apache.org/jira/browse/FLINK-12607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277709#comment-17277709 ] Robert Metzger commented on FLINK-12607: Great! Thanks a lot. I assigned you! > Introduce a REST API that returns the maxParallelism of a job > - > > Key: FLINK-12607 > URL: https://issues.apache.org/jira/browse/FLINK-12607 > Project: Flink > Issue Type: Improvement > Components: Runtime / REST >Affects Versions: 1.6.3 >Reporter: Akshay Kanfade >Assignee: John Phelan >Priority: Minor > Labels: starter > > Today, Flink does not offer any way to get the maxParallelism for a job and > it's operators through any of the REST APIs. Since, the internal state > already tracks maxParallelism for a job and it's operators, we should expose > this via the REST APIs so that application developer can get more insights on > the current state. > There can be two approaches on how we can do this - > Approach 1 : > Modify the existing rest API response model to additionally expose a new > field 'maxParallelism'. Some of the REST APIs that would be affected by this > |h5. */jobs/:jobid/vertices/:vertexid*| > |h5. */jobs/:jobid*| > > Approach 2 : > Create a new REST API that would only return maxParallelism for a job and > it's operators. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (FLINK-19951) PyFlink end-to-end test stuck in "Reading kafka messages"
[
https://issues.apache.org/jira/browse/FLINK-19951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277705#comment-17277705
]
Guowei Ma commented on FLINK-19951:
---
[https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12815=logs=08866332-78f7-59e4-4f7e-49a56faa3179=7f606211-1454-543c-70ab-c7a028a1ce8c]
> PyFlink end-to-end test stuck in "Reading kafka messages"
> -
>
> Key: FLINK-19951
> URL: https://issues.apache.org/jira/browse/FLINK-19951
> Project: Flink
> Issue Type: Bug
> Components: API / Python
>Affects Versions: 1.12.0
>Reporter: Robert Metzger
>Assignee: Shuiqiang Chen
>Priority: Major
> Labels: test-stability
> Fix For: 1.13.0
>
>
> https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=8837=logs=c88eea3b-64a0-564d-0031-9fdcd7b8abee=ff888d9b-cd34-53cc-d90f-3e446d355529
> {code}
> 2020-11-03T08:18:10.2935249Z Nov 03 08:18:10 Test PyFlink DataStream job:
> 2020-11-03T08:18:10.2936216Z Nov 03 08:18:10 Preparing Kafka...
> 2020-11-03T08:18:10.2948091Z Nov 03 08:18:10 Downloading Kafka from
> https://archive.apache.org/dist/kafka/2.2.0/kafka_2.12-2.2.0.tgz
> 2020-11-03T08:18:10.3024006Z % Total% Received % Xferd Average Speed
> TimeTime Time Current
> 2020-11-03T08:18:10.3024610Z Dload Upload
> Total SpentLeft Speed
> 2020-11-03T08:18:10.3024891Z
> 2020-11-03T08:18:10.6563956Z 0 00 00 0 0 0
> --:--:-- --:--:-- --:--:-- 0
> 2020-11-03T08:18:11.6568328Z 0 54.3M0 327680 0 92275 0
> 0:10:18 --:--:-- 0:10:18 92044
> 2020-11-03T08:18:12.6540430Z 11 54.3M 11 6272k0 0 4626k 0
> 0:00:12 0:00:01 0:00:11 4625k
> 2020-11-03T08:18:13.6585146Z 23 54.3M 23 12.6M0 0 5521k 0
> 0:00:10 0:00:02 0:00:08 5521k
> 2020-11-03T08:18:14.6558377Z 36 54.3M 36 19.7M0 0 6018k 0
> 0:00:09 0:00:03 0:00:06 6017k
> 2020-11-03T08:18:15.6593118Z 49 54.3M 49 26.7M0 0 6297k 0
> 0:00:08 0:00:04 0:00:04 6297k
> 2020-11-03T08:18:16.653Z 62 54.3M 62 34.0M0 0 6515k 0
> 0:00:08 0:00:05 0:00:03 6973k
> 2020-11-03T08:18:17.6544951Z 76 54.3M 76 41.8M0 0 6747k 0
> 0:00:08 0:00:06 0:00:02 7322k
> 2020-11-03T08:18:18.2448109Z 91 54.3M 91 49.7M0 0 6923k 0
> 0:00:08 0:00:07 0:00:01 7584k
> 2020-11-03T08:18:18.2450531Z 100 54.3M 100 54.3M0 0 7010k 0
> 0:00:07 0:00:07 --:--:-- 7737k
> 2020-11-03T08:18:20.2751451Z Nov 03 08:18:20 Zookeeper Server has been
> started ...
> 2020-11-03T08:18:22.0064118Z Nov 03 08:18:22 Waiting for broker...
> 2020-11-03T08:18:25.4758082Z Nov 03 08:18:25 Created topic
> test-python-data-stream-source.
> 2020-11-03T08:18:25.8324767Z Nov 03 08:18:25 Sending messages to Kafka...
> 2020-11-03T08:18:35.2954788Z Nov 03 08:18:35 >>Created topic
> test-python-data-stream-sink.
> 2020-11-03T08:18:54.8314099Z Nov 03 08:18:54 Job has been submitted with
> JobID 1b0c317b47c69ee600937e1715ad9cce
> 2020-11-03T08:18:54.8348757Z Nov 03 08:18:54 Reading kafka messages...
> 2020-11-03T08:53:10.5246998Z
> ==
> 2020-11-03T08:53:10.5249381Z === WARNING: This E2E Run took already 80% of
> the allocated time budget of 250 minutes ===
> 2020-11-03T08:53:10.5251343Z
> ==
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (FLINK-21244) UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnLocalAndRemoteChannel Fail
[ https://issues.apache.org/jira/browse/FLINK-21244?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guowei Ma updated FLINK-21244: -- Summary: UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnLocalAndRemoteChannel Fail (was: shouldPerformUnalignedCheckpointOnLocalAndRemoteChannel Fail) > UnalignedCheckpointITCase.shouldPerformUnalignedCheckpointOnLocalAndRemoteChannel > Fail > -- > > Key: FLINK-21244 > URL: https://issues.apache.org/jira/browse/FLINK-21244 > Project: Flink > Issue Type: Bug > Components: Runtime / Checkpointing >Affects Versions: 1.11.0 >Reporter: Guowei Ma >Priority: Major > Labels: test-stability > > > > https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12814=logs=2c3cbe13-dee0-5837-cf47-3053da9a8a78=981eced9-6683-5752-3201-62faf56c149b > > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:435) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:209) > > at > org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) > > at > org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:159) > > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) > at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) > at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) > at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) > at akka.actor.Actor$class.aroundReceive(Actor.scala:517) > at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) > at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) > at akka.actor.ActorCell.invoke(ActorCell.scala:561) > at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) > at akka.dispatch.Mailbox.run(Mailbox.scala:225) > at akka.dispatch.Mailbox.exec(Mailbox.scala:235) > ... 4 more > Caused by: org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint > tolerable failure threshold. > at > org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleTaskLevelCheckpointException(CheckpointFailureManager.java:90) > > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:1889) > > at > org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveDeclineMessage(CheckpointCoordinator.java:996) > > at > org.apache.flink.runtime.scheduler.SchedulerBase.lambda$declineCheckpoint$6(SchedulerBase.java:1026) > > at > java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) > > at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) > at > java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) > > at > java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) > > at > java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) > > at java.base/java.lang.Thread.run(Thread.java:834) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (FLINK-21244) shouldPerformUnalignedCheckpointOnLocalAndRemoteChannel Fail
Guowei Ma created FLINK-21244: - Summary: shouldPerformUnalignedCheckpointOnLocalAndRemoteChannel Fail Key: FLINK-21244 URL: https://issues.apache.org/jira/browse/FLINK-21244 Project: Flink Issue Type: Bug Components: Runtime / Checkpointing Affects Versions: 1.11.0 Reporter: Guowei Ma https://dev.azure.com/apache-flink/apache-flink/_build/results?buildId=12814=logs=2c3cbe13-dee0-5837-cf47-3053da9a8a78=981eced9-6683-5752-3201-62faf56c149b at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRunAsync(AkkaRpcActor.java:435) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleRpcMessage(AkkaRpcActor.java:209) at org.apache.flink.runtime.rpc.akka.FencedAkkaRpcActor.handleRpcMessage(FencedAkkaRpcActor.java:77) at org.apache.flink.runtime.rpc.akka.AkkaRpcActor.handleMessage(AkkaRpcActor.java:159) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:26) at akka.japi.pf.UnitCaseStatement.apply(CaseStatements.scala:21) at scala.PartialFunction$class.applyOrElse(PartialFunction.scala:123) at akka.japi.pf.UnitCaseStatement.applyOrElse(CaseStatements.scala:21) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:170) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) at scala.PartialFunction$OrElse.applyOrElse(PartialFunction.scala:171) at akka.actor.Actor$class.aroundReceive(Actor.scala:517) at akka.actor.AbstractActor.aroundReceive(AbstractActor.scala:225) at akka.actor.ActorCell.receiveMessage(ActorCell.scala:592) at akka.actor.ActorCell.invoke(ActorCell.scala:561) at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:258) at akka.dispatch.Mailbox.run(Mailbox.scala:225) at akka.dispatch.Mailbox.exec(Mailbox.scala:235) ... 4 more Caused by: org.apache.flink.util.FlinkRuntimeException: Exceeded checkpoint tolerable failure threshold. at org.apache.flink.runtime.checkpoint.CheckpointFailureManager.handleTaskLevelCheckpointException(CheckpointFailureManager.java:90) at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.abortPendingCheckpoint(CheckpointCoordinator.java:1889) at org.apache.flink.runtime.checkpoint.CheckpointCoordinator.receiveDeclineMessage(CheckpointCoordinator.java:996) at org.apache.flink.runtime.scheduler.SchedulerBase.lambda$declineCheckpoint$6(SchedulerBase.java:1026) at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] Myasuka commented on pull request #14835: [FLINK-21213][task] Degrade log level to INFO when ignore to decline checkpoint as task not running
Myasuka commented on pull request #14835: URL: https://github.com/apache/flink/pull/14835#issuecomment-772245037 @flinkbot run azure This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14842: [FLINK-21238][python] Support to close PythonFunctionFactory
flinkbot commented on pull request #14842: URL: https://github.com/apache/flink/pull/14842#issuecomment-772240337 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit e64470869ec5501fbb2b81bc5db796d7c8450ffb (Wed Feb 03 05:27:21 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21238) Support to close PythonFunctionFactory
[ https://issues.apache.org/jira/browse/FLINK-21238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-21238: Summary: Support to close PythonFunctionFactory (was: Support to close PythonFunctionFactory manually) > Support to close PythonFunctionFactory > -- > > Key: FLINK-21238 > URL: https://issues.apache.org/jira/browse/FLINK-21238 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Dian Fu >Assignee: Dian Fu >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > PythonFunctionFactory is used to convert a Python class to a Java > PythonFunction representation which could then be used as a user-defined > function. Underlying PythonFunctionFactory, there is a Python process which > is used to perform the actual conversion work. Currently, the Python process > is added to shutdown hook and closed when the JVM exits. The aim of this JIRA > is to provide more flexibility for users by introducing a close method to > PythonFunctionFactory. This will allow PythonFunctionFactory to be closed > manually which is important when integrating this feature in the platforms. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21238) Support to close PythonFunctionFactory
[ https://issues.apache.org/jira/browse/FLINK-21238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-21238: Description: PythonFunctionFactory is used to convert a Python class to a Java PythonFunction representation which could then be used as a user-defined function. Underlying PythonFunctionFactory, there is a Python process which is used to perform the actual conversion work. Currently, the Python process is added to shutdown hook and closed when the JVM exits. The aim of this JIRA is to provide more flexibility for users by introducing a close method to PythonFunctionFactory. This will allow PythonFunctionFactory to be closed manually and periodically which is important when integrating this feature in the platforms. (was: PythonFunctionFactory is used to convert a Python class to a Java PythonFunction representation which could then be used as a user-defined function. Underlying PythonFunctionFactory, there is a Python process which is used to perform the actual conversion work. Currently, the Python process is added to shutdown hook and closed when the JVM exits. The aim of this JIRA is to provide more flexibility for users by introducing a close method to PythonFunctionFactory. This will allow PythonFunctionFactory to be closed manually which is important when integrating this feature in the platforms. ) > Support to close PythonFunctionFactory > -- > > Key: FLINK-21238 > URL: https://issues.apache.org/jira/browse/FLINK-21238 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Dian Fu >Assignee: Dian Fu >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > PythonFunctionFactory is used to convert a Python class to a Java > PythonFunction representation which could then be used as a user-defined > function. Underlying PythonFunctionFactory, there is a Python process which > is used to perform the actual conversion work. Currently, the Python process > is added to shutdown hook and closed when the JVM exits. The aim of this JIRA > is to provide more flexibility for users by introducing a close method to > PythonFunctionFactory. This will allow PythonFunctionFactory to be closed > manually and periodically which is important when integrating this feature in > the platforms. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (FLINK-21238) Support to close PythonFunctionFactory manually
[ https://issues.apache.org/jira/browse/FLINK-21238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-21238: --- Labels: pull-request-available (was: ) > Support to close PythonFunctionFactory manually > --- > > Key: FLINK-21238 > URL: https://issues.apache.org/jira/browse/FLINK-21238 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Dian Fu >Assignee: Dian Fu >Priority: Major > Labels: pull-request-available > Fix For: 1.13.0 > > > PythonFunctionFactory is used to convert a Python class to a Java > PythonFunction representation which could then be used as a user-defined > function. Underlying PythonFunctionFactory, there is a Python process which > is used to perform the actual conversion work. Currently, the Python process > is added to shutdown hook and closed when the JVM exits. The aim of this JIRA > is to provide more flexibility for users by introducing a close method to > PythonFunctionFactory. This will allow PythonFunctionFactory to be closed > manually which is important when integrating this feature in the platforms. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] dianfu opened a new pull request #14842: [FLINK-21238][python] Support to close PythonFunctionFactory manually
dianfu opened a new pull request #14842: URL: https://github.com/apache/flink/pull/14842 ## What is the purpose of the change *This pull request support to close PythonFunctionFactory manually.* ## Brief change log - *Introduce PythonFunctionFactoryImpl which allows to close PythonFunctionFactory* - *Introduce PYTHON_FUNCTION_FACTORY_CACHE which is responsible for closing inactive PythonFunctionFactory periodically.* ## Verifying this change This change can be verified by the existing test cases. ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): (no) - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: (no) - The serializers: (no) - The runtime per-record code paths (performance sensitive): (no) - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (no) - The S3 file system connector: (no) ## Documentation - Does this pull request introduce a new feature? (no) - If yes, how is the feature documented? (not applicable) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21238) Support to close PythonFunctionFactory manually
[ https://issues.apache.org/jira/browse/FLINK-21238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated FLINK-21238: Description: PythonFunctionFactory is used to convert a Python class to a Java PythonFunction representation which could then be used as a user-defined function. Underlying PythonFunctionFactory, there is a Python process which is used to perform the actual conversion work. Currently, the Python process is added to shutdown hook and closed when the JVM exits. The aim of this JIRA is to provide more flexibility for users by introducing a close method to PythonFunctionFactory. This will allow PythonFunctionFactory to be closed manually which is important when integrating this feature in the platforms. (was: PythonFunctionFactory is used to convert a Python class to a Java PythonFunction representation which could then be used as a user-defined function. Underlying PythonFunctionFactory, there is a Python process which is used to perform the actual conversion work. Currently, the Python process is added to shutdown hook and closed when the JVM exits. The aim of this JIRA is to provide more flexibility for users by introducing a close method to PythonFunctionFactory to allow it to be manually closed. ) > Support to close PythonFunctionFactory manually > --- > > Key: FLINK-21238 > URL: https://issues.apache.org/jira/browse/FLINK-21238 > Project: Flink > Issue Type: Improvement > Components: API / Python >Reporter: Dian Fu >Assignee: Dian Fu >Priority: Major > Fix For: 1.13.0 > > > PythonFunctionFactory is used to convert a Python class to a Java > PythonFunction representation which could then be used as a user-defined > function. Underlying PythonFunctionFactory, there is a Python process which > is used to perform the actual conversion work. Currently, the Python process > is added to shutdown hook and closed when the JVM exits. The aim of this JIRA > is to provide more flexibility for users by introducing a close method to > PythonFunctionFactory. This will allow PythonFunctionFactory to be closed > manually which is important when integrating this feature in the platforms. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot edited a comment on pull request #14837: [FLINK-20417][k8s] Create a new watcher when the old one is closed with HTTP_GONE
flinkbot edited a comment on pull request #14837: URL: https://github.com/apache/flink/pull/14837#issuecomment-771773023 ## CI report: * 8e26c4c6349e0d0bd2f078ffcfda63c5b6293bd2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12806) * 2a72efb99f27c99e4bc2e2383e37797c1542b4e0 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12827) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14837: [FLINK-20417][k8s] Create a new watcher when the old one is closed with HTTP_GONE
flinkbot edited a comment on pull request #14837: URL: https://github.com/apache/flink/pull/14837#issuecomment-771773023 ## CI report: * 8e26c4c6349e0d0bd2f078ffcfda63c5b6293bd2 Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12806) * 2a72efb99f27c99e4bc2e2383e37797c1542b4e0 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] MiLk commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
MiLk commented on pull request #14629: URL: https://github.com/apache/flink/pull/14629#issuecomment-772208383 No, I'm doing the following: ``` apiVersion: v1 kind: Pod metadata: name: pod-template spec: containers: - name: flink-task-manager resources: requests: ephemeral-storage: 256Mi limits: ephemeral-storage: 256Mi ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14834: [FLINK-21234][build system] Adding timeout to all tasks
flinkbot edited a comment on pull request #14834: URL: https://github.com/apache/flink/pull/14834#issuecomment-771607546 ## CI report: * bc778f3e205ebde76a63b6163a7fc66699fb1659 Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12816) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14788: [FLINK-21180][python] Move the state module from 'pyflink.common' to 'pyflink.datastream'.
flinkbot edited a comment on pull request #14788: URL: https://github.com/apache/flink/pull/14788#issuecomment-768901227 ## CI report: * fd3c22ae3c320b98c2ed42c4c143cb2d57cad66a Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12824) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] wangyang0918 commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
wangyang0918 commented on pull request #14629: URL: https://github.com/apache/flink/pull/14629#issuecomment-772197835 @MiLk Thanks for your testing. Could you please share how do you set the request and limit for ephemeral storage, like the following? ``` volumes: - name: flink-artifact emptyDir: sizeLimit: "1Gi" - name: flink-logs emptyDir: sizeLimit: "1Gi" ``` This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14841: [FLINK-21232][YARN][Common] Introduce pluggable Hadoop delegation token providers
flinkbot edited a comment on pull request #14841: URL: https://github.com/apache/flink/pull/14841#issuecomment-772182984 ## CI report: * 20eea1289502f47226453012ae600303352f7cb9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12826) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14729: [FLINK-21092][FLINK-21093][FLINK-21094][FLINK-21096][table-planner-blink] Support ExecNode plan serialization/deserialization for `IN
flinkbot edited a comment on pull request #14729: URL: https://github.com/apache/flink/pull/14729#issuecomment-765399004 ## CI report: * 8e7fedb102a0d654ecd59f862e9b912471c2386f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12786) * 8a6de343620a78f1bb9af2a9892a837c7081c704 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12825) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot commented on pull request #14841: [FLINK-21232][YARN][Common] Introduce pluggable Hadoop delegation token providers
flinkbot commented on pull request #14841: URL: https://github.com/apache/flink/pull/14841#issuecomment-772182984 ## CI report: * 20eea1289502f47226453012ae600303352f7cb9 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14788: [FLINK-21180][python] Move the state module from 'pyflink.common' to 'pyflink.datastream'.
flinkbot edited a comment on pull request #14788: URL: https://github.com/apache/flink/pull/14788#issuecomment-768901227 ## CI report: * Unknown: [CANCELED](TBD) * fd3c22ae3c320b98c2ed42c4c143cb2d57cad66a Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12824) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14729: [FLINK-21092][FLINK-21093][FLINK-21094][FLINK-21096][table-planner-blink] Support ExecNode plan serialization/deserialization for `IN
flinkbot edited a comment on pull request #14729: URL: https://github.com/apache/flink/pull/14729#issuecomment-765399004 ## CI report: * 8e7fedb102a0d654ecd59f862e9b912471c2386f Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12786) * 8a6de343620a78f1bb9af2a9892a837c7081c704 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14647: [FLINK-20835] Implement FineGrainedSlotManager
flinkbot edited a comment on pull request #14647: URL: https://github.com/apache/flink/pull/14647#issuecomment-760154591 ## CI report: * 0b72f53fbfe19ef426572f78ac716521bc63ab64 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12769) * 970360401dcd333f1a3b29355392385262953246 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12823) Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] MiLk commented on pull request #14629: [FLINK-15656][k8s] Support pod template for native kubernetes integration
MiLk commented on pull request #14629: URL: https://github.com/apache/flink/pull/14629#issuecomment-772181752 Thank you for your work Following the discussion on the user ML at the beginning of the week, I tried your PR against my workload. My goal was to set the ephemeral storage requests and limits, but it seems the output of the `ResourceRequirementsBuilder` takes precedence over the template, so my config is ignored. However I was able to set a `priorityClassName` without trouble. As discussed in this PR, I think that while options set using flink options should have higher precedence, they should be merged whenever possible with the template. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (FLINK-21232) Introduce pluggable Hadoop delegation token providers
[ https://issues.apache.org/jira/browse/FLINK-21232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17277649#comment-17277649 ] jackwangcs commented on FLINK-21232: Hi [~lirui], thanks. I submited a PoC PR for this. However, it's a bit hard to add tests for delegtion token related code, see: https://issues.apache.org/jira/browse/FLINK-16224. > Introduce pluggable Hadoop delegation token providers > - > > Key: FLINK-21232 > URL: https://issues.apache.org/jira/browse/FLINK-21232 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common, Deployment / YARN >Reporter: jackwangcs >Priority: Major > Labels: pull-request-available > > Introduce a pluggable delegation provider via SPI. > Delegation provider could be placed in connector related code and is more > extendable comparing using reflection way to obtain DTs. > Email dicussion thread: > [https://lists.apache.org/thread.html/rbedb6e769358a10c6426c4c42b3b51cdbed48a3b6537e4ebde912bc0%40%3Cdev.flink.apache.org%3E] > > [https://lists.apache.org/thread.html/r20d4be431ff2f6faff94129b5321a047fcbb0c71c8e092504cd91183%40%3Cdev.flink.apache.org%3E] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] flinkbot commented on pull request #14841: [FLINK-21232][YARN][Common] Introduce pluggable Hadoop delegation token providers
flinkbot commented on pull request #14841: URL: https://github.com/apache/flink/pull/14841#issuecomment-772174441 Thanks a lot for your contribution to the Apache Flink project. I'm the @flinkbot. I help the community to review your pull request. We will use this comment to track the progress of the review. ## Automated Checks Last check on commit 20eea1289502f47226453012ae600303352f7cb9 (Wed Feb 03 02:40:23 UTC 2021) **Warnings:** * No documentation files were touched! Remember to keep the Flink docs up to date! * **This pull request references an unassigned [Jira ticket](https://issues.apache.org/jira/browse/FLINK-21232).** According to the [code contribution guide](https://flink.apache.org/contributing/contribute-code.html), tickets need to be assigned before starting with the implementation work. Mention the bot in a comment to re-run the automated checks. ## Review Progress * ❓ 1. The [description] looks good. * ❓ 2. There is [consensus] that the contribution should go into to Flink. * ❓ 3. Needs [attention] from. * ❓ 4. The change fits into the overall [architecture]. * ❓ 5. Overall code [quality] is good. Please see the [Pull Request Review Guide](https://flink.apache.org/contributing/reviewing-prs.html) for a full explanation of the review process. The Bot is tracking the review progress through labels. Labels are applied according to the order of the review items. For consensus, approval by a Flink committer of PMC member is required Bot commands The @flinkbot bot supports the following commands: - `@flinkbot approve description` to approve one or more aspects (aspects: `description`, `consensus`, `architecture` and `quality`) - `@flinkbot approve all` to approve all aspects - `@flinkbot approve-until architecture` to approve everything until `architecture` - `@flinkbot attention @username1 [@username2 ..]` to require somebody's attention - `@flinkbot disapprove architecture` to remove an approval you gave earlier This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (FLINK-21232) Introduce pluggable Hadoop delegation token providers
[ https://issues.apache.org/jira/browse/FLINK-21232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated FLINK-21232: --- Labels: pull-request-available (was: ) > Introduce pluggable Hadoop delegation token providers > - > > Key: FLINK-21232 > URL: https://issues.apache.org/jira/browse/FLINK-21232 > Project: Flink > Issue Type: New Feature > Components: Connectors / Common, Deployment / YARN >Reporter: jackwangcs >Priority: Major > Labels: pull-request-available > > Introduce a pluggable delegation provider via SPI. > Delegation provider could be placed in connector related code and is more > extendable comparing using reflection way to obtain DTs. > Email dicussion thread: > [https://lists.apache.org/thread.html/rbedb6e769358a10c6426c4c42b3b51cdbed48a3b6537e4ebde912bc0%40%3Cdev.flink.apache.org%3E] > > [https://lists.apache.org/thread.html/r20d4be431ff2f6faff94129b5321a047fcbb0c71c8e092504cd91183%40%3Cdev.flink.apache.org%3E] > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [flink] JackWangCS commented on pull request #14841: [FLINK-21232][YARN][Common] Introduce pluggable Hadoop delegation token providers
JackWangCS commented on pull request #14841: URL: https://github.com/apache/flink/pull/14841#issuecomment-772173966 The PR is just a PoC and still working in progress. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] JackWangCS opened a new pull request #14841: [FLINK-21232][YARN][Common] Introduce pluggable Hadoop delegation token providers
JackWangCS opened a new pull request #14841: URL: https://github.com/apache/flink/pull/14841 ## What is the purpose of the change Introduce a pluggable delegation provider via SPI. Delegation provider could be placed in connector related code and is more extendable comparing using reflection way to obtain DTs. ## Brief change log - *Introduce the HadoopDelegationTokenProvider interface for Hadoop service to obtain delegation tokens* - *Introduce HadoopDelegationTokenManager to manage HadoopDelegationTokenProviders* - *Implement HadoopDelegationTokenProvider for Hadoop Filesystems and HBase* ## Verifying this change This change added tests and can be verified as follows: - Tests are still working in progress ## Does this pull request potentially affect one of the following parts: - Dependencies (does it add or upgrade a dependency): no - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no - The serializers: no - The runtime per-record code paths (performance sensitive): no - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: yes, this will affect Yarn deployment - The S3 file system connector: no ## Documentation - Does this pull request introduce a new feature? yes - If yes, how is the feature documented? not documented now, it's still work in progress This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14788: [FLINK-21180][python] Move the state module from 'pyflink.common' to 'pyflink.datastream'.
flinkbot edited a comment on pull request #14788: URL: https://github.com/apache/flink/pull/14788#issuecomment-768901227 ## CI report: * Unknown: [CANCELED](TBD) * fd3c22ae3c320b98c2ed42c4c143cb2d57cad66a UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [flink] flinkbot edited a comment on pull request #14647: [FLINK-20835] Implement FineGrainedSlotManager
flinkbot edited a comment on pull request #14647: URL: https://github.com/apache/flink/pull/14647#issuecomment-760154591 ## CI report: * 0b72f53fbfe19ef426572f78ac716521bc63ab64 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=12769) * 970360401dcd333f1a3b29355392385262953246 UNKNOWN Bot commands The @flinkbot bot supports the following commands: - `@flinkbot run travis` re-run the last Travis build - `@flinkbot run azure` re-run the last Azure build This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
